EVALUATION LEGAL PROFESSIONAL DEVELOPMENT AND ANTI-CORRUPTION ACTIVITY MIDTERM PERFORMANCE EVALUATION This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc. under the Liberia Strategic Analysis activity. KONAH TARYOR FOR SOCIAL IMPACT ABSTRACT The Legal Professional Development and Anti-Corruption (LPAC) activity midterm performance evaluation examined the appropriateness, performance, and sustainability of LPAC’s new design to promote a more effective and accountable formal justice sector through improved capacity of five legal and regulatory institutions: the James A. A. Pierre Judicial Institute (Judicial Institute), the Louis Arthur Grimes School of Law (Law School), the Liberia Legal Information Institute (LiberLII), the Liberian National Bar Association (LNBA), and the Liberia Anti-Corruption Commission (LACC). The Liberia Strategic Analysis (LSA) evaluation team used a mixed-methods approach to address five evaluation questions (EQs) assessing (1) the original design and redesign; (2) achievement since redesign; (3) investment compared to perceived benefit; (4) government commitment and capacity; and (5) degree of women’s participation. Evaluation methods included a desk review of relevant documents, monitoring and evaluation (M&E) data, key informant interviews (KIIs), focus group discussions (FGDs), and short interviews with stakeholders in Monrovia, Kakata, and Gbarnga. The evaluation found that inaccurate assumptions and planning led to flaws in the original design, which were influenced by the lack of co-design with GOL partners. The management structure and approach was risky and lacked consistent and reliable oversight. The redesign was appropriate in fixing the constraints by integrating components better, promoting cross-pollination, and shifting toward a multi￾agency approach to combating corruption. Since the redesign, LPAC experienced uneven progress working with the legal and regulatory institutions but achieved the greatest results and value with the LNBA and LACC. Participation of women grew by piloting legal aid clinics, increasing Public Service Scholarships and focusing on women’s participation in judicial trainings. iii | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV LEGAL PROFESSIONAL DEVELOPMENT AND ANTI￾CORRUPTION ACTIVITY MIDTERM PERFORMANCE EVALUATION USAID/Liberia Liberia Strategic Analysis Contract No: AID-669-C-16-00002 December 7, 2018 Contact: Liberia Strategic Analysis Social Impact, Inc. UN Drive Adjacent Gate 3 U.S. Embassy Monrovia, Liberia POC: Carla Trippe, ctrippe@socialimpact.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | iv TABLE OF CONTENTS I. EXECUTIVE SUMMARY.....................................................................................................vii Evaluation Methodology............................................................................................................................................ vii LPAC Evaluation: Key Takeaways.......................................................................................................................... viii II. INTRODUCTION................................................................................................................. 1 Context........................................................................................................................................................................... 1 Activity Background..................................................................................................................................................... 1 Evaluation Purpose and Audience............................................................................................................................. 2 Evaluation Questions................................................................................................................................................... 2 III. METHODOLOGY................................................................................................................. 3 Overview ........................................................................................................................................................................ 3 Evaluation Design.......................................................................................................................................................... 3 Data Analysis.................................................................................................................................................................. 5 Limitations...................................................................................................................................................................... 5 IV. FINDINGS AND CONCLUSIONS ..................................................................................... 6 EQ1: To what extent did design and implementation constraints impede decision-making, adaptation, and achievement of results prior to reforming the activity?.............................................................................. 6 EQ2: To what extent has progress been made through the redesign in achieving the objectives under LPAC? Consider timeliness and efficiency of results. ........................................................................................12 EQ3: Considering the investments made by USAID and the perceived benefits of partner institutions, what interventions should be kept and which should be discontinued or scaled back? ...........................16 EQ4: What is the extent of the GOL’s commitment and capacity to sustain Components A and B, both financial and administrative?.....................................................................................................................................24 EQ5: To what extent does the redesign increase participation of women in a meaningful way?...........29 V. RECOMMENDATIONS...................................................................................................... 31 VI. UTILIZATION PLAN......................................................................................................... 35 Annex A: Full Listing of References and Reports Used........................................................... 36 Annex B: Full Listing of Persons Interviewed........................................................................... 37 Annex C: Data Collection Instruments..................................................................................... 38 Annex D: Evaluation Statement of Work................................................................................. 43 Annex E: Disclosure of Any Conflict of Interest ...................................................................... 50 Annex F: Identifying Constraints and Barriers to Entry for Women .................................... 53 Annex G: Brief Overview of LPAC’s Monitoring and Evaluation System ............................. 58 Annex H: Comparing Activity Components and Results Framework Pre-/Post-Redesign . 59 Annex I: Expanded EQ1 Findings .............................................................................................. 62 v | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Annex J: Expanded EQ3 Findings .............................................................................................. 65 Annex K: Limitations to Accessing Justice ............................................................................... 70 TABLE OF FIGURES Table 1: List of Key Informant Interview Types......................................................................................................... 4 Figure 1: Summary of Original Design and Implementation Constraints............................................................. 7 Table 2: Design and Implementation Constraints...................................................................................................... 7 Table 3: Addressing Identified Constraints................................................................................................................10 Table 4: Summary Assessment of Judicial Institute’s Financial Commitment and Capacity...........................24 Table 5: Summary Assessment of LACC’s Financial Commitment and Capacity............................................26 Figure 2: LPAC Results Framework – December 2015.........................................................................................60 Figure 3: LPAC Revised Results Framework – Draft AMELP July 2018.............................................................61 ACRONYMS ABIC Angie Brooks International Center ADS Automated Directive Systems AMELP Activity Monitoring, Evaluation, and Learning Plan AOR Alternate Contracting Officer’s Representative BOC Board of Commissioners CDCS Country Development Cooperation Strategy Checchi Checchi and Company Consulting CLE Continuing Legal Education COP Chief of Party CSO Civil Society Organization DG Democracy and Governance EQ Evaluation Question FGD Focus Group Discussion FIND Foundation for International Dignity FIU Financial Intelligence Unit FY Fiscal Year GOL Government of Liberia HICD Human and Institutional Capacity Development ICT Information and Communication Technology IP Implementing Partner IPT Investigation and Prosecution Team JLA Justice and Legal Advisor KII Key Informant Interview LACC Liberia Anti-Corruption Commission LiberLII Liberia Legal Information Institute LNBA Liberian National Bar Association LNP Liberia National Police LPAC Legal Professional Development and Anti-Corruption Activity LRA Liberia Revenue Authority LSA Liberia Strategic Analysis Activity M&E Monitoring and Evaluation MOC Memorandum of Cooperation MOJ Ministry of Justice MSI Management Systems International NGO Non-Governmental Organization ODC Other Direct Cost PMTP Professional Magistrates Training Program PPCC Public Procurement and Concessions Commission PRS Poverty Reduction Strategies RHRAP Rural Human Rights Activist Program SGBV Sexual and Gender-Based Violence SI Social Impact SOW Statement of Work STTA Short-Term Technical Assistance TO Task Order ToC Theory of Change USAID United States Agency for International Development vii | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV I. EXECUTIVE SUMMARY Through the Liberia Strategic Analysis (LSA) activity, Social Impact (SI) conducted a midterm performance evaluation of the Legal Professional Development and Anti-Corruption (LPAC) activity. LPAC is a five￾year, $20 million activity implemented by Checchi and Company Consulting (Checchi) and Management Systems International (MSI) primarily in Montserrado County, with current grant implementation in Bong, Grand Bassa, Margibi, and Bomi counties. LPAC seeks to promote a more effective and accountable formal justice sector through improving the capacity of five legal and regulatory institutions: the James A. A. Pierre Judicial Institute (Judicial Institute), the Louis Arthur Grimes School of Law (Law School), the Liberia Legal Information Institute (LiberLII), the Liberian National Bar Association (LNBA), and the Liberia Anti￾Corruption Commission (LACC). This midterm performance evaluation provides an independent and in-depth examination of the overall progress of the LPAC activity in Liberia. The evaluation explores the appropriateness and sustainability of the redesign and identifies best practices for the remaining implementation period. The Mission will use findings from the evaluation to inform more strategic USAID investment in the sector. The evaluation focused its instrument design, analysis, and recommendations to address the following evaluation questions (EQs):  EQ1 Assessing the Original Design and Redesign: To what extent did design and implementation constraints impede decision-making, adaptation, and achievement of results prior to reforming the activity?  EQ2 LPAC Achievement Since Redesign: To what extent has progress been made through the redesign in achieving the objectives under LPAC? Consider timeliness and efficiency of results.  EQ3 Assessing Investment vs. Perceived Benefit: Considering the investments made by USAID and the perceived benefits of partner institutions, what interventions should be kept and which should be discontinued or scaled back?  EQ4 Commitment and Capacity: What is the extent of the GOL’s commitment and capacity to sustain Components A and B, both financial and administrative?  EQ5 Increased Women’s Participation: To what extent does the redesign increase participation of women in a meaningful way? In the recommendation, identify what can be done to further increase participation of women in a meaningful way. EVALUATION METHODOLOGY LSA used a mixed-methods approach to conduct the LPAC evaluation. This included the review of relevant documents, aggregation and analysis of the monitoring and evaluation (M&E) data from LPAC, and key informant interviews (KIIs) of LPAC and USAID staff, institutional partners, and LPAC grant recipients in Montserrado County. In addition, site visits were conducted in Kakata and Gbarnga to observe operations at LPAC-supported legal aid clinics and hold KIIs with clinic staff and with magistrates and judges who have attended LPAC-supported Continuing Legal Education (CLE) and Professional Magistrates Training Program (PMTP) sessions, local government representatives, local civil society organizations (CSOs), and short interviews with local community members through a random sampling. For EQ1 and EQ2, the evaluation used the Concept Note on Proposed Revision of LPAC Implementation dated July 21, 2017 as the reference point to compare pre- to post-reform of the activity. For EQ3, LSA conducted a benefit analysis to assess the investment made against results and benefits achieved in LPAC USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | viii interventions to help USAID determine which interventions should be kept, discontinued, or scaled back. The approach assessed the cost of inputs against the perceived benefits of an activity or intervention. LPAC EVALUATION: KEY TAKEAWAYS Design Partnership Gender Integration Successes Shifting LPAC focus away from financial and administrative sustainability to technical focus led to better results with institutional partners. Change in the staffing structure helped reduce silos and increase cross￾pollination of activities. LNBA demonstrated greatest gains relative to cost. Leadership structures are performing well, and the prospect for financial sustainability of the institution is high. Targeted PMTP-3 enrollment contributed to the increased presence of women in the judiciary–particularly in rural areas–and is leading to increased access to judicial services for women in rural areas. Opportunities Set joint vision with institutional partners that defines success and is fit for purpose. Measure that vision through a strong results framework and data that link to planned achievements. Focus engagement with other institutional partners on areas with existing momentum that strategically contribute to the jointly developed vision. At the legal aid clinics, balance pre-trial detention efforts with a strengthened focus on increasing legal services for women in areas such as family and property law. 1 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV II. INTRODUCTION CONTEXT When the Legal Professional Development and Anti-Corruption (LPAC) activity was originally designed in 2011/2012, Liberia was experiencing economic growth, expanding business and investment opportunities, and historically high prices for key commodity exports that underpin the Liberian economy, such as gold, iron ore, and latex. In addition, after 14 years of conflict and civil strife, Liberia was experiencing increasing peace and social and political stability, particularly with the democratic reelection of the Johnson-Sirleaf administration. In this promising environment, the Government of Liberia (GOL) set an ambitious agenda in 2012 to transform Liberia into a middle-income country by 2030.1 However, when the LPAC Task Order (TO) was signed in July 2015, commodity prices had fallen and the Ebola crisis had hit Liberia hard in 2014. Donor funding and the dwindling GOL budget shifted focus to the public health crisis, leaving little resources to cover other public-sector services. Real gross domestic product growth rate fell dramatically from 8.7 percent in 2013, to 0.7 percent in 2014, and to 0 percent in 2015, which is reflected in the GOL’s budget, which has fallen annually by an average of 10 percent since 2014, affecting LPAC counterparts.2 Although the Ebola crisis ended in early 2016, GOL revenue is stabilizing, recent World Bank/International Monetary Fund reports indicate that it will take another four to five years to return to pre-crisis growth and revenue levels.3 ACTIVITY BACKGROUND LPAC is USAID/Liberia’s primary formal justice-sector program and fits within a greater framework of the Mission’s current rule of law and governance sector activities. LPAC was designed to build on key policy and vision platforms of the GOL in its first and second national Poverty Reduction Strategies (PRS), and its Agenda for Transformation. Implemented by Checchi and Company Consulting (Checchi) and Management Systems International (MSI), LPAC is a five-year, $20 million activity that began in July 2015 and targets five legal institutions in Montserrado County: the James A. A. Pierre Judicial Institute (Judicial Institute), the Louis Arthur Grimes School of Law (Law School), the Liberia Legal Information Institute (LiberLII), Liberian National Bar Association (LNBA), and the Liberia Anti-Corruption Commission (LACC). The activity seeks to promote a more effective and accountable formal justice sector through improved capacity of the legal professional training and regulatory institutions, and improved rule of law. Under the original design, emphasis was placed on institutional capacity building to help ensure the financial and administrative sustainability of the five targeted institutions. Recognizing that the mechanisms and scope for implementation were not producing the anticipated results, a proposed redesign of LPAC was provisionally approved in July 2017. The proposed redesign made LPAC leaner, more focused on essential needs, and more cost-effective through a right-sizing of program design and recognition of the constraints faced by Liberian counterparts. LPAC reduced the number of key expat personnel from five to three and shifted focus to hiring local experts and bringing in international short-term technical assistance (STTA) when expertise cannot be identified locally. At the same time, the redesign proposed that the level of 1 World Bank. 2018. Republic of Liberia From Growth to Development: Priorities for Sustainably Reducing Poverty and Achieving Middle-Income Status by 2030; Reed, Kramer. 20 November 2010. “Liberia: to Become a Middle Income Country by 2030 Growth Must Have A Broad Reach.” 2 LPAC Proposed Reform Outline, September 2017. 3 LPAC Proposed Reform Outline, September 2017. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 2 financial sustainability of counterpart institutions be deferred until the Liberian economy and state budget situation improves, understanding that this would likely be beyond LPAC’s end date.4 LPAC’s Theory of Change If LPAC increases the effectiveness of the formal justice sectors and increases demand for quality legal services and anti￾corruption efforts, then Liberians will have better access to justice as required by the laws of Liberia, and public officials entrusted with public resources are more transparent and accountable to the people of Liberia. Since the redesign, LPAC continues to be organized into two components. The first component directs capacity building to the four key Liberian justice sector institutions that educate, regulate and inform legal professionals: the Judicial Institute, Law School, LiberLII, and LNBA. The second component supports the LACC and associated GOL entities in their efforts to investigate, prosecute, and convict perpetrators of corruption-related crimes, and to educate the public regarding the fight against corruption in Liberia. See Annex H for more detail on pre- to post-redesign results frameworks. EVALUATION PURPOSE AND AUDIENCE Human and institutional capacity development (HICD) interventions aim to transform more structural, deeply rooted challenges. Even with strong theories of change (ToCs) underpinning design, these types of interventions are slow to show progress. To ensure that LPAC’s activities under the recent redesign are strategically contributing to outcomes in the most effective way, this midterm evaluation examined the appropriateness, performance, and sustainability of the new design. The evaluation identifies results and lessons learned from implementation and provides succinct, actionable recommendations to determine which components of LPAC to scale up or modify moving forward or for future procurements providing technical advisory support. Evaluation findings and recommendations were shared and discussed with USAID/Liberia and the implementing partners (IPs) during the preliminary findings outbrief. The Mission will also use findings from the evaluation to inform future USAID programming in the sector. In addition, LSA will use the findings from this evaluation to inform and organize the Learning Event with USAID, IPs, and LPAC’s GOL institutional partners, with an aim toward establishing an agreed-upon path forward for the remainder of LPAC’s term. EVALUATION QUESTIONS The evaluation focused on the following questions: EQ1: Assessing the Original Design and Redesign  To what extent did design and implementation constraints impede decision-making, adaptation, and achievement of results prior to reforming the activity? o How appropriate and adequate were the steps taken in addressing pre-reform constraints? EQ2: LPAC Achievements Since Redesign  To what extent has progress been made through the redesign in achieving the objectives under LPAC? Consider timeliness and efficiency of results. EQ3: Assessing Investment vs. Perceived Benefit5 4 Concept Note on Proposed Revision of LPAC Implementation, July 21, 2017, p. 1-2. 5 LSA proposed a revision of EQ3 in the evaluation SOW, which originally stated: “Considering value for money, what interventions should be kept and which should be discontinued or scaled back?” Based on an initial review of activity data available and lack of comparable programs, a cost-benefit analysis was not feasible. USAID approved the new language in EQ3. 3 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV  Considering the investments made by USAID and the perceived benefits of partner institutions, what interventions should be kept and which should be discontinued or scaled back? o Are there alternative options to achieve more effective change, particularly in the area of anti-corruption? o Are there specific lessons learned from LPAC that should be considered as USAID/Liberia prepares its new Country Development Cooperation Strategy (CDCS)? EQ4: Commitment and Capacity  What is the extent of the GOL’s commitment and capacity to sustain Components A and B, both financial and administrative? o Are there specific benchmarks for GOL actions that should be considered as USAID contemplates its investments in this sector? EQ5: Increased Women’s Participation  To what extent does the redesign increase participation of women in a meaningful way? In the recommendation, identify what can be done to further increase participation of women in a meaningful way. The EQs were validated through planning meetings with USAID/Liberia to ensure the above scope was feasible. The original scope of work can be found in Annex D: Evaluation Statement of Work. III. METHODOLOGY OVERVIEW LSA used a mixed-methods approach to conduct the LPAC evaluation. This included the review of relevant documents, aggregation and analysis of the M&E data from LPAC, and KIIs of LPAC staff, USAID staff, institutional partners, and LPAC grant recipients in Montserrado County. Focus group discussions (FGDs) were also held with LPAC Public Service Scholarship recipients to obtain feedback on the quality of their legal education, how the scholarship helps them, and any constraints they face in attending law school and anticipate in entering the legal profession. In addition, site visits were conducted in Kakata and Gbarnga to observe operations at LPAC-supported legal aid clinics and hold KIIs with clinic staff and magistrates and judges who have attended LPAC-supported Continuing Legal Education (CLE) and Professional Magistrates Training Program (PMTP) sessions, local government representatives, local civil society organizations (CSOs), and short interviews with local community members through a random sampling. The evaluation selected Kakata and Gbarnga based on two criteria: (1) the presence of LPAC-supported legal aid/pro bono clinics to assess and compare how the clinics are operating, their demand (client intake) and capacity to respond to demand; and (2) a concentration of circuit and magistrate courts in an effort to assess LPAC’s support to GOL’s goal of increasing access to justice in rural areas. For EQ3, the evaluation examined the activity’s investment (actual and projected) against perceived results and benefits to help USAID determine which interventions should be kept, discontinued, or scaled back. The approach is further described in the Limitations section below. EVALUATION DESIGN Qualitative Data The evaluation relied on three key methods for qualitative data collection and review: desk review, KIIs, and FGDs. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 4 Desk Review During the initial phase of the evaluation, the team conducted a thorough review of background documents related to LPAC and its implementing environment. These documents enabled the team to gain a deeper understanding of the activity’s operations to date, to establish the evaluation design and data collection tools, and to inform the overall evaluative process, including conclusions and recommendations for future programming. The evaluation team reviewed 66 documents and reports, which are listed in Annex A. Key Informant Interviews The evaluation focused the KIIs to be responsive to the data needs of the EQs, with some variance to gain a broader understanding of the rule of law in Liberia, including constraints in women’s participation in its institutions. All KIIs were semi￾structured with probes for follow-up on information related to key questions, thereby providing flexibility in the discussion and a natural conversational flow while following data collection protocols. The evaluation team conducted a series of KIIs in Monrovia, Kakata, and Gbarnga. Table 1 provides a list of key informant types, along with the total number of respondents from each stakeholder group. The evaluation team conducted 73 KIIs involving a total of 67 individuals (21 women; 46 men).6 The sampling frame was developed based on USAID and LPAC recommendations, knowledge of key players in Liberia’s justice sector, and the likely availability of respondents during the data collection period. In addition, the team used snowball sampling to identify additional key informants who would be able to speak on behalf of an institution, in addition to convenience sampling to identify community members to briefly interview at field locations. In cases where several respondents of the same type were available to meet in the same location, a small group discussion format was used to maximize the quantity of perspectives gathered during the data￾gathering period. Focus Group Discussions The evaluation team conducted two FGDs with 18 LPAC-sponsored Public Service Scholarship recipients (11 men and seven women) at the Law School, focusing on the EQs relating to LPAC achievements since 6 In some cases, multiple interviews were required for LPAC staff members. In other cases, multiple participants attended one KII. Joint meetings are reflected in the KII participant list in Annex B. Table 1: List of Key Informant Interview Types TYPE OF INFORMANT # OF KIIS # OF WOMEN # OF MEN MONROVIA KIIS LPAC staff 18 2 6 USAID 3 1 2 Senior Judicial Institute Staff 4 1 4 Senior Law School/LiberLII Staff 3 0 3 LNBA Executive Committee 5 1 7 LNBA Secretariat 2 0 1 LACC Board of Commissioners 2 0 2 LACC Secretariat 7 0 7 Justices 1 1 0 Magistrates 1 0 1 Law Professors 1 1 0 Lawyers 1 1 0 Grantees 4 1 3 Subtotal: 53 9 38 FIELD KIIS Local Government 2 0 2 Magistrate 2 1 1 County Attorneys 2 1 1 Local Police Representatives 2 1 1 LNBA Clinic 1 0 1 Local NGOs 3 2 1 Local Citizens 8 7 1 Subtotal: 20 12 8 TOTAL 73 21 46 5 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV redesign (EQ2) and the degree of women’s participation (EQ5). To help ensure open discussion, FGDs were held separately with men and women. The evaluation used a list provided by LPAC to identify law students who are scholarship recipients. An emphasis was placed on pre-mobilization to make sure participants were available and the discussions did not coincide with classes. Quantitative Data During implementation, LPAC collected data related to learning and capacity development needs and outcomes, as well as assessment tools administered to measure the enhanced capacity of the five legal institutions. These data were examined to determine the level of progress LPAC had made over the implementation period, to assess benefits related to financial investment, and to visually display the connections between the type of intervention LPAC used and performance outcomes. All data reports reviewed are outlined in Annex A. DATA ANALYSIS Data collected during KIIs and FGDs were captured in note form, typed up, and circulated to other team members for additional inputs. The final versions were processed systematically by sorting observations according to a coding system. These data were analyzed by the team using an inductive approach, gathering all of the data before identifying themes and patterns within the data. This process informed development of key findings, conclusions, and recommendations that make up this report. LIMITATIONS Timely Access to Documentation: Access to relevant and highly critical documents was inconsistent and late, which led to an initial evaluation design that did not appropriately reflect LPAC’s redesign. Although the team was provided significant amount of documents to review at the outset, critical documents relating to the redesign, such as the draft contract modification under review by USAID, the proposed revised ToC, and the recently revised draft Activity Monitoring, Evaluation, and Learning Plan (AMELP), were provided to the evaluation team well into the data collection phase. This meant that data collection tools could not reflect the most recent questions about the redesign, such as Component B’s involvement in the redesign process, which led to confusion and misdirection during interviews.7 Weak M&E System: This limitation is based on three fundamental constraints: 1. A lack of data and poor documentation and reporting prior to LPAC’s redesign in July 2017 limited the team’s ability to assess the results and achievements against benchmarks, and to assess relevancy and responsiveness to GOL institutional partners’ needs and USAID’s directives. 2. The lack of baseline data to measure activity progress made measuring for progress and results difficult. 3. A weak performance management system, including a lack of intermediate outcome indicators, made it difficult to assess the relevancy, appropriateness, and benefits of activities or deliverables in achieving LPAC’s goal. Limitations in Data to Compare Costs and Benefits: Budget and cost reports for LPAC are not disaggregated by intervention, nor do they present aggregate costs for headquarters compared to the Liberia office. Based on the time available, analysis for EQ3 was done using the Liberia office budget tool that comprised 7 For example, the evaluation team was not aware of a formal modification submitted to USAID on August 22, 2018, which outlined five key modifications to the contract, including a request to modify the anti-corruption activities (Request 2). USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 6 both actual and projected costs; however, this budget only detailed costs by intervention for a few months for FY2018 and did not include the associated costs for international STTAs who performed training sessions because they are reported under headquarter-level financial reports. While LPAC has made significant improvement to their M&E system by streamlining and enhancing their indicators and standardizing their M&E reporting processes,8 their current indicators do not provide the essential data required to measure benefits from the intervention.9 Given the lack of comparable cost and benefits/results data and indicators, the evaluation assessed the actual and projected costs for FY2018 activities and perceived benefit(s) based on stakeholder feedback. The evaluation was not able to provide comparative analysis of one activity against another. Instead, the analysis examined the quality and relevance of LPAC activities and the benefits produced by these activities as reported during KIIs and in LPAC’s quarterly reports. Limited Institutional Knowledge Leading to Recall Bias: No member of LPAC’s management team was engaged in the activity during the first 16-18 months of implementation and consequently were unable to comment in depth on the first 1.5 years of work. USAID staff members managing LPAC have also turned over since the beginning of implementation. For this reason, this report and evaluation may not contain all information and details about the first months of implementation. To mitigate this limitation’s impact on the evaluation, the team spoke to leadership from the IP and USAID that had the most access to historical documents and any documentation of activity-related conversations and meetings. IV. FINDINGS AND CONCLUSIONS EQ1: To what extent did design and implementation constraints impede decision￾making, adaptation, and achievement of results prior to reforming the activity? FINDINGS To answer EQ1, the evaluation team interviewed stakeholders across the multiple categories noted in Table 1 and asked about the challenges they faced prior to the redesign and what primary factors contributed to the decision to reform the approach. The evaluation tracked frequency of specific challenge or constraint references and found that the primary design constraint identified by respondents was a lack of a clear purpose, stated in results language and supported by a framework. The primary implementation constraint was the resources required to support multiple embedded expats. Additionally, a highly referenced constraint to both design and implementation was the activity’s engagement of and relationship with institutional partners and GOL institutions. These design and implementation constraints are further detailed below. Although the IP’s approach was responsive to the original design in the scope, it was nonetheless based upon a flawed design. The evaluation identified five key constraints to activity progress in the original design, as highlighted in Figure 1—in particular, the lack of evidence of USAID’s co-design with GOL, which in many ways caused other design constraints. The design flaws and performance expectations were then codified in an inflexible contracting mechanism that did not allow USAID or the IP to adjust mid￾course as necessary. While executing the approach, the IP faced five key constraints, which led to five implementation constraints: 8 The M&E team began producing insightful ad hoc reports, such as highlighting the experience of Liberian magistrates. 9 For example, the indicators do not capture data regarding training quality, learning, or behavior change that can be linked to the LPAC-supported/delivered training. 7 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Figure 1: Summary of Original Design and Implementation Constraints Table 2 highlights the design and implementation constraints. Annex I provides more detailed analysis. Table 2: Design and Implementation Constraints Design Constraints Implementation Constraints Impeded LPAC decision-making, adaptation, and achievement of results CONSTRAINTS PRIOR TO REDESIGN DESIGN CONSTRAINT IMPLEMENTATION CONSTRAINT PARTICIPANT FEEDBACK The focus on administrative and financial sustainability in GOL institutional partners • The original design was based on assumptions that did not hold true regarding the political, economic, and social context in Liberia, which led to a poorly defined strategic outlook for the activity. • No evidence that a renewed context analysis was conducted by USAID to check the original activity assumptions. • No evidence that USAID conducted a gender or youth assessment to inform the design. A risky, expat-heavy, and non-integrated LPAC staffing structure to tackle HICD- focused design, and not responsive to operating environment • The implementation approach was not matched to GOL institutional needs, capacities, and mandates. • The staffing approach was personality-driven, institution￾focused, and expat-heavy. • One expat per institution led to high overhead costs and “did not add value” to the activity, according to USAID and IP respondents. • Ineffective (or unknown quality of) technical assistance by advisors. • Little cross-communication, collaboration, and synergy in technical assistance, leading to a silo￾effect in implementation and communication. • LPAC did not weave gender and youth considerations into its strategic implementation approach or address fundamental barriers women face in the justice sector. “There is a design flaw regarding increased [financial and administrative] capacity, which relied on the assumption that improved capacity through better systems and financial and administrative management would lead to better provision of services.” USAID respondent “The advisor was instrumental in establishing the institution’s structural foundation. He was not only helpful, but also necessary. When he arrived in September 2016, the LNBA’s infrastructure capacity was almost non￾existent.” LNBA respondent The Judicial Institute felt the former advisor provided significant added value. He provided guidance on administrative matters and was helpful in developing the capacity of staff on curriculum review, reform, design, and delivery of the quarterly and annual judicial trainings. Judicial Institute respondent “[The former LACC Advisor] was not advising. The [current Advisor] is Design Constraints  Focus on administrative and financial sustainability = poor understanding of context  Lack of co-creation  Tying success to institutional capacity  Pinpointing LACC as primary anti￾graft agency for GOL  Unclear purpose/goal Implementation Constraints  Risky staffing structure  Lack of GOL institutional collaboration on approach  Primary partnership with LACC  Poor M&E plan and systems  High turnover of key staff USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 8 In addition, high IP and USAID turnover during the key first two years of the activity impacted activity momentum and progress: two COPs led LPAC within the first 18 months of the activity, while a new USAID Mission Director, COR, and Activity Manager came on board in 2016 and 2017. This impacted LPAC’s momentum and progress by not providing it the necessary stewardship required to lead a team of embedded international advisors, identify and mitigate design and implementation constraints, and steer the development of an appropriate AMELP and communication framework. CONCLUSIONS While the evaluation found evidence of constraints related to both activity design and implementation, the original activity design proved a bigger and more frequently cited constraint to progress relative to implementation. Furthermore, when respondents explained implementation constraints, they frequently concluded that the constraints resulted from fundamental design issues in the activity. The LPAC Inception Report mirrored the original design outlined in the SOW, reflecting a lack of participation in the design, collaboration, and communication with GOL counterparts during LPAC’s start up months and is considered a missed opportunity to address the key constraints. The lack of testing of the original assumptions led to 18 months of performance that frustrated both USAID (in terms of lack of performance and activity direction) and GOL institutional partners (as the original design was both inappropriate to the institutional mandate while simultaneously unrealistic given budgetary constraints). responsive … he is considered a member of the team.” LACC respondent Lack of co-creation • The lack of co-design interactions with GOL partners. • The selection of GOL institutional partners to achieve the aims of the activity, such as the Judicial Institute and the Law School to achieve administrative and financial sustainability, was uninformed. Lack of collaboration • The LPAC Inception Report mirrored the original design outlined in the Statement of Work (SOW) as opposed to results of a collaborative co-design process with GOL counterparts. • Difficulty in managing institutional partners’ expectations. It was “a lost opportunity for Checchi and USAID that constraints were not addressed in the Inception Report.” LPAC’s former key personnel lacked creativity to provide solutions to address the operating environment, blaming the contracting mechanism that held them to their original design.” USAID respondent TO language infers LACC as the primary anti-corruption agency • By inferring LACC as the primary stakeholder to focus on anti￾corruption work, other institutions fighting corruption were unintentionally marginalized. Strained cooperation with other agencies • Due to LACC’s poor reputation and capacity related to investigative procedures, relevant GOL institutions were reluctant to work with LACC. “Other agencies resented the perception of LACC as an anti-corruption leader. LACC does not have a good reputation among the other anti-corruption agencies.” LPAC respondent Lack of a clear purpose and results language • The original Strategic Framework outlined in the TO’s SOW failed to tie in Components A and B logically and clearly, with no distinct reflection of how success or failure in one component would contribute to the success or failure in the other. • No documented evidence to support approval in the change to the activity’s aim during the Inception Phase to include: “regulatory institutions and improve the rule of law”. Poor M&E plan, systems, data collection, and reporting with minimal results-focused indictors • Weak M&E plan led to uninformed decision-making by the IP and USAID. • The lack of a quality results framework and ToC undermined a performance monitoring system that would inform USAID and LPAC of achievements and critical constraints and flaws in the design and implementation. • M&E requirements and oversight were not robust to ensure quality gender and youth data to be gathered. “There were many deliverables that did not make sense, such as the purchase of infrastructure and transportation. It gave the appearance that LPAC is a slush fund.” USAID respondent “Some of the indicators were not fully defined and were very unclear. The projections were unachievable. It was also not clear how these indicators were going to be measured in the first place. Some of the goals were not realistic, such as autonomous status for the Judicial Institute.” LPAC staff member 9 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Both LPAC and USAID fell short on ensuring gender and youth considerations were woven into the design and implementation because a gender assessment was not conducted, and all relevant monitoring data was not collected and disaggregated by sex and age in addition to other relevant demographic details. Without informed gender considerations, the implementation approach was not set up to address the fundamental barriers that women face in becoming service providers and in their access to legal and judicial services, a point that is more fully discussed in EQ5. The undocumented expansion of LPAC’s goal during the Inception Phase resulted in unclear and unmeasurable linkages between activities and interventions to improve Liberia’s rule of law. Without a clear results framework that reflects how the result under each sub-component ties into the overall objective of the sub-component, it is unclear how each of the sub-components tie together. This not only impacted the evaluation’s ability to conduct a proper cost-benefit analysis and assess the effectiveness of the overall intervention, but it also impacted LPAC’s ability to progress toward a common goal. The high turnover within LPAC, coupled with USAID staffing change, impacted knowledge transfer of strategic actions and management justifications to support decision making. LPAC’s weak performance management system did not accurately measure progress or flag constraints. USAID and LPAC leadership believe the heavy reliance on expat advisors embedded at each institution was a drain on the budget and did not add value to the activity, while one institutional partner underscored lack of effective coordination between LPAC and some institutional partners. EQ1.1: How appropriate and adequate were the steps taken in addressing pre-reform constraints? FINDINGS The evaluation found a mixed level of appropriateness and adequacy in the steps taken to address the identified design and implementation constraints. USAID noted that they were the driving force behind making changes to LPAC, taking the opportunity of turnover in key LPAC personnel to encourage changes and in response the Mission Director’s concerns relating to LPAC’s value for money. However, the evaluation found no clear evidence that USAID and the IP used empirical evidence to determine what and how to change the activity during the redesign process. The shift in LPAC’s strategic approach to support the legal institutions (Component A) was not clearly defined during the redesign, is not clearly reflected in the proposed contract modification under review by USAID,10 and did not include proposed changes to component outcomes. What the redesign targeted was the budget imbalance of expensive overhead costs (expat staffing) against technical assistance, with the aim to cost-effectively improve institutional technical capacity. At the same time in 2017, there was a distinct shift in LPAC’s approach to supporting anti-corruption (Component B), but there was no officially documented strategy outlining how it would change, what it was changing, and what the new outcomes would be.11 This made it difficult to assess what the redesign was improving and how LPAC was improving it. 10 The proposed modification was submitted to USAID on August 22, 2018, amending the draft dated July 16, 2018. The proposal addresses: (1) removing the monthly report requirement; (2) broadening the focus of anti-corruption activities; (3) clarifying the scope of grants under contract; (4) budget realignment; and (5) amending key personnel. 11 USAID and LPAC agreed to hold the decision on Component B until after the election so it does not appear as a political issue and required the IP to submit one-year performance benchmarks on what it will achieve by July 2018. However, the lack of clear technical direction by USAID on how to address their LACC concerns impacted LPAC’s ability to develop a detailed proposed strategic modification, instead proposing a broad concept to focus its anti-corruption component beyond LACC. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 10 Although the redesign was approved in theory while LPAC tested out its approach, many key constraints noted earlier were either not known or not addressed during the redesign. The redesign was therefore not in specific response to the design and implementation constraints noted above. Table 3 highlights whether and how USAID and the IP addressed the design and implementation constraints. Table 3: Addressing Identified Constraints ADDRESSING IDENTIFIED CONSTRAINTS DESIGN CONSTRAINT IMPLEMENTATION CONSTRAINT PARTICIPANT FEEDBACK The focus on administrative and financial sustainability in GOL institutional partners • The shift to an emphasis on technical assistance was appropriate and is shown to have produced results particularly with the Judicial Institute, where the focus on increased quarterly trainings and increased diversification in PMTP-3 candidates has produced positive results. • The revised technical assistance approach fell short for the Law School. It was not responsive to the priorities of the Law School as outlined in its strategic plan. • No evidence that a renewed context, gender, or youth analysis was conducted to inform or test assumptions against the proposed redesign. A risky, expat-heavy, and non-integrated LPAC staffing structure to tackle HICD-focused design, and not responsive to operating environment • Mixed reviews on the change of LPAC’s staffing structure to be leaner and more cost-effective. Successes • LPAC reduced the number of full-time expats from five to three advisors and increased the number of full-time Liberian advisors embedded in GOL partner institutions, while hiring STTAs for support on specific issues. • The new staffing structure increased ‘cross￾pollination’ of planning and implementation. • The two senior-level personnel embedded at the Law School are found to be effective. • LACC officials noted that there was no sense of inclusiveness in decision making and approach with the former advisor. Shortfalls • Staff are stretched and not able to provide adequate technical assistance/skills transfer. • The LPAC-embedded fundraising officer was found to not be effective in raising funds. The Law School also lacks autonomous status. All fundraising proceeds would go directly to the University of Liberia. • All communication with USAID is filtered through the COP, creating inefficiencies in decision making. • LPAC did not adequately take gender and youth considerations into its strategic implementation approach. • LPAC somewhat addressed barriers women face in the justice sector. “Although the staffing has been stressed, we deliver and get it done.” LPAC respondent “[The current Advisor] is disconnected from the priorities and activities of the Judicial Institute and is just handling the procurement and administrative work. She does not have time to get involved and understand our technical mandate.” Judicial Institute respondent LPAC’s overall support and technical assistance “is not transformative.” Law School respondent “Greater emphasis should be placed on providing technical assistance to LACC’s prosecution team of similar caliber and quality as they do for the investigation team. The prosecution team could benefit greatly from a one-on-one approach like the investigation team currently receives.” LACC respondent Lack of co-creation • The reform did not fully address this fundamental design constraint. • A senior-level staff member at the Judicial Institute noted that they were informed about the Lack of collaboration • LPAC did not adequately address managing GOL institutional partners’ expectations with the Judicial Institute and the Law School. • The Judicial Institute prefers a full-time expat Advisor back in their institution, while USAID views the current arrangement to be more effective. “There needs to be an open conversation with LPAC to see how it can make sense of the Law School’s priorities and find a way to meet the Law School midway in responding to our priorities.” Law School respondent12 12 The evaluation notes that LPAC cannot deliver on priorities of the Law School that are not in its scope. 11 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV reform but did not participate in the actual redesign process. • LNBA and LACC noted full collaboration with LPAC during the redesign process. LACC is aware of and understands LPAC’s desire to diversify its beneficiary base. TO language infers LACC as the primary anti-corruption agency • The redesign’s shift from LACC as a primary counterpart toward a multi-institutional approach was appropriate. • The redesign lacks a detailed strategic approach that outlines what the activity intends to achieve and how in combating public corruption in Liberia. Strained cooperation with other agencies • The informal shift to diversifying beneficiaries among the GOL anti-graft agencies produced greater cooperation and coordination between these agencies, while improving LACC’s reputation and leadership. • Joint trainings with other anti-graft agencies are increasing, particularly since the successful prosecution of a former senator. • Increasing opportunity and willingness to conduct joint-investigations involving LACC and other anti-graft agencies. “Investigation reports are more precise and better written, which was commended by the [Board of Commissioners (BOC) Chair]. Their work is more professional. The team has an improved confidence in their work, and morale is much higher on the team.” LACC Commissioner Lack of a clear purpose and results language • This design constraint was not addressed during the activity redesign. • There is no known evidence that USAID and LPAC have worked together to clarify the goal of the activity. Poor M&E plan, systems, data collection, and reporting with minimal results-focused indicators • The redesign’s efforts to restructure and reform the M&E system did not fully address the constraints. • There is a lack of quality indicators to measure progress and performance. • The M&E team is not well institutionalized within LPAC communication and reporting structures. (See Annex G for more details.) LPAC is now able to gather data in a more structured manner. LPAC respondent “They succeeded in trimming down the indicators,” but there is concern about the accuracy of the data. “There are a lot of indicators that basically do not inform us. Improvement is still needed.” USAID respondent LPAC adequately addressed the turnover issue of key personnel. The evaluation found significant support for the new leadership team that came on board in 2017, and key staff retention remains strong. The new Judicial and Legal Advisor received mixed results. CONCLUSIONS The staffing shift was the most significant change that had both positive and negative repercussions. Although the reduction in LPAC activity staff led to increased cost-effectiveness, this also led to an over￾stretching of staffing resources as emphasis was placed on expanding activities instead of hiring additional local staff. In addition, the change in leadership and staffing restructure created a centralized decision￾making and information flow that limits technical information flow between the activity (particularly the technical leads) and USAID. This creates the potential for filtering of information and directives. The Component A redesign addressed some of the constraints identified during 18 months of implementation, but the redesign process was uneven in capturing and fully aligning LPAC’s approach with institutional priorities, thereby creating the perception among some institutional partners of uneven treatment and support. Since there was no formal redesign of Component B, nor a proposed strategic approach outlining what the activity intended to change, it was difficult to assess what shift took place. LPAC staffing at the LACC remained the same, and all direct mentorship and guidance resides primarily with LACC’s investigation team, with the prosecution team to a lesser extent. Despite not having a formal ‘reform process’ for Component B, LPAC’s open collaboration with LACC to shift focus toward a multi-agency approach with support and buy-in from LACC has been lauded by respondents as a positive change in approach. The reform’s effort to improve the M&E plan and systems is adequate, but further efforts are necessary to make it fully aligned with the reform strategy and results-oriented to assess progress towards USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 12 achievement. USAID needs to provide clarity on what they want to achieve through this activity to better inform the final M&E plan. EQ2: To what extent has progress been made through the redesign in achieving the objectives under LPAC? Consider timeliness and efficiency of results. FINDINGS In assessing LPAC’s progress in achieving the objectives under LPAC’s redesign, the evaluation findings reflect uneven progress against the objectives outlined in the TO and LPAC’s Y3 Work Plan. Judicial Institute: LPAC’s support to the Judicial Institute included providing operational support and technical assistance in the delivery of the PMTP-3, quarterly judges and magistrates training, the quarterly public defenders training, and trainings of other judicial personnel. As a result of the redesign, resources were redirected to improve the reach and quality of the training of judges and public defenders with an emphasis on the PMTP-3. The training infrastructure has been improved with the PMTP and quarterly public defenders training. Information and communication technology (ICT) equipment for the computer research and training room have been installed, and internet has been provided to support the PMTP-3 Computer Literacy Improvement training module. However, respondents indicated room for improvement with regard to training quality. According to a senior Judicial Institute official, “Some courses offered by the PMTP are informed by a curriculum [with] modules developed almost five years ago under the ABA program and do not respond to the learning needs of adults undergoing a professional training course.” To diversify the participant pool of the PMTP-3 program, LPAC supported the Judicial Institute to raise awareness in the southeastern counties of Maryland, Grand Kru, River Gee, Sinoe, and Grand Gedeh. According to an LPAC respondent and several Judicial Institute respondents, this resulted in a higher turnout for the September 2017 aptitude exam, from which 60 participants were selected for PMTP-3.13 Other trainings, such as the quarterly public defenders training, the quarterly judges and magistrates training, and the court clerks training, were well-received and timely, although some participants indicated the potential for improvement in the quality of the trainer and relevancy of the training topics. For the quarterly judges training, some JI trainers had no specialty or advanced training in law. This led to several judges boycotting the trainings and then being compelled to attend by order of the Supreme Court. Regarding the magistrate trainings, one participant noted, “The trainings are partly relevant to [our] work such as court room management. However, other topics taught during the training do not respond to my work, such as human trafficking and rape.” These issues do not fall under the jurisdiction of the Magisterial Court to address them. According to both USAID and the IP, the Judicial Institute declined to establish an in-house M&E structure to track the quality and impact of their training courses. Due to the lack of perceived relevancy, LPAC discontinued seconding M&E staff to the partner during the redesign. The issue of institutional autonomy has been stalled. The proposed legislative draft to allow the Judicial Institute to become an independent institution was passed by the Board of Governors in the Fall 2016 13 According to LPAC’s Q1 2018 report, “For the first time, all 60 candidates are either sitting magistrates or college graduates.” When comparing the Judicial Institute’s trained magistrates to the Law School trained students, the Judicial Institute’s trained magistrates had an edge. They have mentors who provide guidance throughout the training. The Judicial Institute sends students to practice and get connected to the job while still in school. Associate Magistrate 13 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV and sent to the Chief Justice of the Supreme Court’s for final review and approval prior to submission to the legislature, according to LPAC quarterly reports. Given the current economic environment and the fact that the Supreme Court created the Judicial Institute, “it is highly doubtful that the Supreme Court will approve this,” according to the IP. Law School: Since the redesign, LPAC’s planned support to the Law School, according to the Y3 Work Plan, included: continuing the public service scholarship; enhancing legal scholarship through the publication of the law journal; promoting scholarly writing by introducing programs that recognize faculty and students for their scholarly efforts; and continuing to design and deliver legal writing workshops that introduce faculty and students to the scholarly writing process from conception to research, editing, and publication. LPAC is supporting the Law School’s financial sustainability by embedding a fundraising officer, but execution of a fundraising plan and associated results has been slow. Law School officials noted, “The fundraising person has been working for two years to support the law school [with] raising funds, but no funds have been raised. The person sits at the school but reports to LPAC.” The embedded fundraising officer is working in a tight economic environment, and his ability to coordinate substantial resource generation is not expected in the short term. Also, since the Law School is not autonomous, all funds raised go straight to the University of Liberia. Since 2015/2016, LPAC has been awarding six law students in each entering class an annual Public Service Scholarship.14 In October 2017, LPAC expanded the number of recipients for the entering 2017/2018 class to nine first-year students, five of whom were female. The scholarship fund now provides scholarships to 20 recipients, eight of whom are women. The redesign was instrumental in ensuring that Law School leadership was more involved in the scholarship process by establishing a selection committee that set up eligibility criteria, developed allocation materials, and developed a transparent application review process. All 18 FGD participants noted that the scholarships helped reduce their financial stress. Law School officials said that the scholarships helped improve learning and “contributed to a higher retention of law students, especially women.” However, one Law School official noted that the scholarship “was not in line with the School’s vision of what scholarship means. It should be merit-based.” LPAC also launched a one-day seminar for all scholarship recipients in December 2017 to provide an orientation and a workshop on ethics in the legal profession. The Clinical Legal Education Program15 (Clinical Program) has not been established at the Law School,16 and efforts to enhance legal scholarship have been constrained due to a lack of financial incentives for law professors and students to conduct independent research.17 The evaluation found that the Clinical Program is in high demand among Law School students, professors, administrators, and members of the LNBA (particularly the Legal Aid Program), who are eager to link with the Clinical Program to mentor 14 The scholarship is a merit-based scholarship. LPAC and the Law School select high performing students who are civic-minded and are willing to serve at least two years in underserved counties outside of Monrovia. 15 The Clinical Program would provide hands-on, practical experience for law students to put into practices the skills they are learning in class. Contents of the program are informed by the outcome of research conducted on topical issues such as implication of legislature oversight on check and balances. The challenge is that the school does not have the financial resources for professors and students to conduct independent research that would produce this content needed for the clinic to be effective and relevant. 16 According to LPAC, the Law School has been inconsistent regarding interest in a Clinical Program. In their Y3 Work Plan, LPAC planned to develop the Clinical Program, which would have been tied to LNBA’s Legal Aid Program but was dropped when the Dean confirmed that UNDP was planning to support a similar activity. However, UNDP support has not materialized. 17 Because almost all professors are part time, they are not prepared to use their extra time out of the classroom to conduct research without financial benefit. In the absence of financial support, they prefer doing other forms of legal practice to earn money. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 14 law students. There was no evidence of the Law School providing legal advice to the University of Liberia, an activity included in the original TO’s scope under Objective A.2.f. LiberLII: Since the redesign, LPAC’s planned support to LiberLII has been put on hold until clarity of structure and management has been restored. While LPAC has planned to focus on making LiberLII more functional, sustainable, and fully integrated in the Law School and the University of Liberia, given the lack of commitment demonstrated by the Board, LPAC ceased all assistance towards this subcomponent until LiberLII’s Board votes on the resolution to transfer control/authority to the Law School. A provisional meeting took place in early October 2018, but planned actions to move forward on this are unknown. LNBA: LPAC’s planned support to LNBA, according to the Y3 Work Plan, included: supporting the annual LNBA convention; trainings to support fundraising and increasing financial sustainability; grant funding toward the LNBA’s legal aid clinic program; formalizing and delivering quarterly CLE courses; and developing and implementing preparatory training courses to better prepare candidates for relevant admission tests. The LNBA is becoming a more professional bar association. The Secretariat demonstrated improvements by filling the Executive Director position and establishing essential financial systems. However, the use of ICT and online sources to communicate and raise awareness is limited. CLE delivery is timely and of high quality, which is reflected in the willingness of members to pay for and attend trainings held on weekends. It has become a critical element to the LNBA’s financial sustainability plan, since the Supreme Court approved the requirement that all lawyers practicing in Liberian courts must be a LNBA “member in good standing.” This requirement involves at minimum 10 CLE courses per year (which will be fee-based as of March 2019) and membership dues to be current. Despite the legal aid clinics’ initial success in achieving the release of 227 pre-trial detainees within their first year of operation, public awareness and access in the satellite offices is still limited, and there is uneven service delivery at each clinic. The locations of the clinics the evaluation team visited in Kakata and Gbarnga were not clearly marked in the local vicinities, and available services beyond addressing pre-trial detainees were not listed. According to an LNBA respondent, “The clinic’s location [in Gbarnga] is too far from most communities and not easily visible.” This limited the public’s understanding of the existence of the legal aid clinics, and the type of legal aid services the clinics can address (e.g., family, property).18 The LNBA respondent also noted a need for a database system to record cases, and to coordinate efforts with other actors who are involved in pro bono work, including the Foundation for International Dignity (FIND) and the Catholic Justice and Peace Commission. LACC: LPAC’s planned support to LACC, according to the Y3 Work Plan, included: increasing overall efficiency (through facilitating the development of strategic policies and infrastructure support); increasing the number of successful investigations and prosecutions; assisting the Education and Prevention Division in an education and public awareness campaign and building the Division’s prevention function. LACC’s capacity to investigate and report findings and recommendations has greatly improved through ongoing skills training, mentorship, and guidance from LPAC. According to Y3 estimates, 47 investigations were completed, of which 23 cases were closed due to insufficient evidence of corruption. LACC identified corrupt conduct in the remaining 24 cases and submitted reports to the BOC for consideration and 18 For example, none of the three citizens interviewed within the vicinity of LNBA’s legal aid clinic in Gbarnga knew it existed. The LNBA has strengthened its reputation through the improved quality and integrity of the CLE program. LPAC’s role has been visible and critical in cementing CLE at the LNBA [through] increased involvement of lawyers in LNBA activities and meetings. Senior Judiciary Informant 15 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV referral to MOJ, a 50 percent increase from Y2. However, the BOC referred only three cases to the MOJ for prosecution, a decline from 10 referrals in Y2. With LPAC’s support, LACC established the only Cyber Crimes Unit in Liberian law enforcement.19 LACC’s capacity to prosecute, however, has been hampered by an overloaded docket in Criminal Court C and a lack of financial capacity to relocate cases to satellite courts. Despite these constraints, LACC prosecuted five corruption cases as either standalone or jointly with MOJ, resulting in one conviction20 with two cases ongoing and two set down, an increase from three prosecutions in Y2. Since the start of implementation, LPAC has been carrying out joint trainings among the key anti-graft agencies, such as the Ministry of Justice (MOJ), Liberia Revenue Authority (LRA), and Financial Intelligence Unit (FIU), with greater emphasis since the redesign on financial and cyber crimes investigations. According to LPAC, “This is a great way to break down barriers and build up an esprit de corps among the officers of these different agencies.” LPAC has not been effective in setting up a workable complaints management system due to a lack of interest within LACC to set up a formal structure. Little action has been taken on administrative and financial sustainability. CONCLUSIONS At the midterm, LPAC has made progress against each of the activity components. While the most substantial progress has been made in Sub-Component A4 of Component A, the efficiency and timeliness of progress in all components show potential for improvement. LPAC is making great strides in its work with LNBA. Its internal financial systems are functioning, but the Executive Committee still needs to approve the institution’s Human Resources plan. The LNBA is increasingly becoming more functional and professionally run and is forecasted to become financially sustainable if it continues on its current path. The CLE is timely and quality driven, and members are willing to pay for the service. Furthermore, the LNBA’s Legal Aid Program has expanded from its original pilot of two clinics in Margibi and Montserrado counties to adding three locations in Bong, Bomi, and Grand Bassa. The LNBA should focus on ensuring greater access for women to legal aid services to maintain balanced legal service provision. LACC’s capacity to investigate and report findings and recommendations has improved tremendously through LPAC’s ongoing skills training, mentorship, and guidance. The establishment of the only Cyber Crimes Unit in Liberian law enforcement is a great achievement, though LACC’s capacity to prosecute has been hampered by an overloaded docket in Criminal Court C and lack of financial capacity to relocate cases to satellite courts. Despite these constraints, LACC led or jointly prosecuted five corruption cases with the MOJ, showing positive movement in tackling corruption cases. The pilot relocation of 19 This resulted from a series of LPAC trainings on digital forensics methods and incorporating them into investigations. LPAC supported opening the Cyber Crimes Unit through infrastructure support (establishing a secure digital forensics room), while providing ongoing training and technical assistance in using these new tools and the Cellerite software purchased with UNMIL funding. 20 LPAC piloted an intervention to relocate one prominent case to a satellite court to prosecute a sitting Senator on procurement offenses. The Senator was convicted in February 2018. According to LPAC’s Q3 2018 report, from the Teahjay conviction, directors of the PPCC requested LPAC to conduct a joint training with LACC and invited LACC investigation officers to attend a five-day training on procurement law and practice. This reflects significant opportunities for joint trainings and growing willingness to work together. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 16 prosecutions has the potential to reduce docket loads in the short term but requires LPAC’s funding to support future corruption prosecutions because those legal resources have not yet been decentralized. Joint trainings are ongoing with other anti-graft agencies, but more can be done on supporting LACC’s Education and Prevention Division. LPAC has achieved uneven progress with the Judicial Institute. It established a modern training venue to teach computer literacy, and the third cycle of the PMTP has increased representatives from southeastern region of Liberia, particularly women. However, some of the curriculum needs to be updated to reflect the current environment in Liberia, which requires LPAC to continue to build staff capacity to produce a revised curriculum.21 The quarterly public defenders training has been timely and relevant, while the relevancy and quality of the quarterly judges and magistrates training has received mixed feedback. The in-house M&E structure, a critical element for any training institution, still has yet to be established. LPAC has also achieved uneven progress with the Law School. Despite LPAC’s progress prior to the reform in establishing the foundation for the Law School to raise funds, its fundraising capability has been slow to start. LPAC’s Public Service Scholarship continues to be highly praised and has increased from six to nine first-year students in October 2017, with the inclusion of five new female scholars, raising the total number of scholarship recipients to 20 students, eight of whom are women. The Clinical Program has not yet started despite the evident high demand for its inception, and there has been lack of interest among law professors and students to conduct independent research without financial incentives. Objective A.2.f aims to enable the Law School to provide legal advice to the University of Liberia through a legal department that does not rely on faculty who are teaching courses. It is out of place and irrelevant in the context of what LPAC aims to achieve with the Law School. LPAC made minimal progress with LiberLII because its Board lacks functionality and commitment. LPAC ceased funding LiberLII in August 2017 and will remain on hold pending the Board’s approval of its Transition Plan. EQ3: Considering the investments made by USAID and the perceived benefits of partner institutions, what interventions should be kept and which should be discontinued or scaled back? As noted in the methodology section, this EQ has been informed by KIIs with LPAC staff, partner institutions and grant recipients, FGDs with scholarship recipients, rapid phone surveys with select beneficiaries, and review of LPAC reports and documents.22 For this exercise, benefits were identified and measured by asking LPAC, partner institutions, select beneficiaries, and select stakeholders if the intended outcomes and benefits of the activity were achieved. Respondents were asked to provide feedback and rate the relevant interventions/activities. Benefits are summarized and presented alongside costs (actuals and projected costs) for FY2018. FINDINGS LPAC has four major interventions with the Judicial Institute that make up 43 percent of the total other direct cost (ODC) budget: the PMTP Cycle II, the quarterly public defenders training, the ICT training, and the quarterly judges and magistrates training. All interventions yielded benefits but at varying degrees. 21 Under the previous advisor, the JI acquired knowledge in training planning and delivery. However, the JI does not have the staff with the requisite knowledge to produce a revised curriculum. 22 Given the limited data available on actual costs, the evaluation focuses on the FY 2018 timeframe. Activities that took place prior to the re-design were not included in this analysis. 17 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV The evaluation team used a scale of Good (i.e., intervention achieved or is on the path to achieve its goal); Fair (i.e., intervention has uneven success in achieving its goal); and Poor (i.e., intervention has failed to achieve its goal) to measure the perceived degree of benefit of each intervention by evaluation respondents. See Annex J for greater detail of the evaluation findings for each intervention and grant listed in the tables. LPAC has four major interventions with the Law School that make up 32 percent of its total activity ODC budget: the Public Interest Scholarships, development of new curriculum courses, the Law School retreat, and legal writing and adult teaching methodology workshops. All interventions yielded positive benefits but at varying degrees. JUDICIAL INSTITUTE INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) PMTP-3 Good: Participants noted they feel better equipped to perform their duty as Magistrates. $74,250 $467,808 43% Quarterly Public Defenders Training Good: All trainees interviewed said that the training improved their courtroom skills, which improved legal defense of indigent criminal defendants. Other respondents noted observed positive change. $20,000 Quarterly Judges and Magistrates Training Fair: All interviewees rated the training either fair or good. Issues were raised over training relevance and quality. Improved case management led to speedy case adjudication in Magistrate Courts and reduced complaints of Magistrate behavior. $35,000 LAW SCHOOL INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Public Interest Scholarship Good: Respondents noted improved balance of school and family obligations, and improved student retention and learning. $8,900 New Curriculum $345,559 32% Good: Positive feedback has been received from School officials and law professors on the curriculum’s relevancy and development. $5,000 Law School Retreat Fair: Outcome of developing annual strategic objectives could have been achieved without this investment. $10,000 USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 18 According to an interview with a female judge and professor at the Law School, “LPAC is on point to push for adult learning and establishing curriculum for every course.” She is not sure if the Law School is convinced that this is a priority, as it is difficult for part-time law professors to do this. She further noted that “skills building is needed across board at the Law School,” and there is no advanced planning for course development. LPAC has two major interventions with the LNBA that make up 15 percent of the total activity ODC budget: quarterly CLE courses and the annual LNBA convention. LPAC also funds the LNBA’s legal aid clinics through a series of direct grants to the LNBA that began in February 2017 totaling $54,370. All interventions yielded ‘good’ benefits according to respondents. LPAC has six major interventions with the LACC. There are other interventions that are smaller but have still produced substantial benefits. LPAC’s interventions with the LACC made up 10 percent of its total programs budget. The interventions yielded benefits but to varying degrees. Legal Writing and Adult Teaching Methods Workshops Fair: Only one workshop was provided in May 2016, reflecting in a high demand among students, but a low demand among professors. $15,000 LNBA INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Annual LNBA Convention Good: To increase member knowledge of trending legal issues in Liberia and discuss/vote on LNBA issues and activities. Members praised the presentations “as very useful to their work,” and noted it greatly improved LNBA’s professional image. $8,000 $164,108 15% CLE Trainings Good: To increase member knowledge of trending legal issues in Liberia. Beneficiaries said that the quarterly CLE trainings increased their knowledge of technical legal areas. $50,000 LACC INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Proceeds of Crime Training Good: Improved investigators technical skills relating to corruption cases. Evidenced by the investigation team’s capacity to and consistency of LACC’s investigation team, and improved cohesion among investigators across anti-graft institutions. $8,900 $111,334 10% Advance interview techniques Good: Improved the quality of evidence collected in criminal cases. Evidenced in the ability of the investigation team to $20,000 19 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Since 2015, LPAC administered five short-term grants to local non-governmental organizations (NGOs) during the Presidential elections. LiberLII’s grant was not extended and will not be until the Board decides on/approves the LiberLII transition plan. The LNBA pro bono legal aid clinics grant is ongoing. differentiate the types of questions and style and manner in which to interview based on the witness or suspect. Satellite courts Good: Improved ability to prosecute corruption cases. A successful pilot activity that allowed LACC to prosecute a corruption case outside of Criminal Court C with reasonable contribution from LPAC. $1,679 Conflict of interest training program for public officials Fair: Improved capacity to inform GOL officials and help identify potential Conflicts of Interest. Mixed reviews within LACC on relevancy and understanding. $20,000 Outreach activities with schools Good: Improved capacity to raise public awareness and increase demand for GOL to combat public corruption. LACC respondent noted it would have been impossible without LPAC support. $5,000 Assisting LACC in conducting risk assessments Good: Improved capacity to conduct corruption risk assessments in GOL agencies. The Prevention Team conducted three risk assessments since training. $2,000 GRANTS INTERVENTION BENEFIT COUNTIES Y1 – Y3 TOTAL GRANTS DISBURSED % OF TOTAL Y1 – Y3 GRANTS DISBURSED LNBA Pro Bono Legal Aid Clinics Good: Improved access to legal services. Opened five legal aid clinics, increasing access women and indigent populations in urban and rural areas. Freed 227 pretrial detainees. Bong, Grand Bassa, Margibi, Bomi and Montserrado $54,370 19% LiberLII Good: Provide operational support to LiberLII. Improved relevancy of Liber LII through ICT and content support. Montserrado $91,298 32% ABIC Good: Educated communities on the rule of law. The Talking Bus targeted 15 counties on the Supreme Court process. Women were particularly targeted on relevant legal issues. Montserrado, Lofa, Grand Bassa, Nimba, Grand Gedeh, Margibi, Bomi, Maryland, and Cape Mount $39,287 14% CENTAL Good: Raising awareness on anti￾corruption issues. Engaged communities through community Montserrado, Grand Bassa, and Bong $37,963 13% USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 20 CONCLUSIONS LPAC’s shift to focus more on technical assistance energized IPs toward achieving greater results, while exposing inefficiencies and quality gaps that can be remedied through adjustments to the approach and trainings. The PMTP-3 training and quarterly public defenders training have yielded the greatest benefits for the Judicial Institute. However, the quarterly judges and magistrates should be adjusted to focus on more relevant legal topics that reflect the cases before the participants. The Public Interest Scholarships and implementation of new curriculum courses have yielded the most benefits for the Law School. A potential successful intervention that has not started but is already in high demand by the Law School students and leadership, as well as the LNBA, is the Law School’s legal clinical program. LPAC’s support to the LNBA’s CLE trainings have yielded substantial benefits, which is reflected in an increase in qualified lawyers in good standing with the LNBA and paying their annual membership fees. Due to LPAC’s support, the most recent LNBA convention was hailed as the most professional to date and is considered to have yielded substantial benefits to LNBA’s reputation and member satisfaction. LPAC support to the establishment of LNBA’s legal aid clinics in five locations has also yielded substantial benefits to women and indigent persons, having helped improve their access to legal aid services, particularly in rural areas. The release of 227 pretrial detainees since the clinics were launched is a significant achievement. A majority of LPAC’s interventions with LACC appear to have yielded good perceived benefits, judging from the responses of the LACC staff, intervention beneficiaries, and other stakeholders. The benefits of five of the six interventions are to increase LACC and partner institutions’ technical skill sets, specifically their ability to investigate anti-corruption cases. While the conflict-of-interest training yielded uneven results, LPAC’s advanced interview techniques, proceeds of crime training, surveillance, digital forensics training sessions and satellite courts interventions produced the greatest benefits.23 These latter 23 Evidence of these benefits included LACC establishing a multi-agency operation to investigate “systemic corruption” targeting GOL officers taking small bribes arising from LPAC’s surveillance and operational planning trainings. forums and radio programs and targeted women’s economic groups. Flomo Theater Good: To improve capacity at local level to fight corruption. Effectively engaged communities through radio drama and skits, live community theater performances, and community forums. Lofa and Margibi $35,910 12% RHRAP Good: To promote transparency and accountability during elections. Engaged communities through radio shows, and linked LACC to communities through community action committees. Nimba $30,815 11% Liberia Media Center Poor: Raising awareness on anti￾corruption issues. Grant was cancelled due to poor performance. Grant was transferred to CENTAL. Montserrado $11,891 4% 21 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV interventions should be scaled up, in addition to smaller interventions, such as LPAC’s outreach to school and risk-assessment interventions. EQ3.1: Are there alternative options to achieve more effective change, particularly in the area of anti-corruption? FINDINGS The evaluation found that with regard to anti-corruption, alternative options do exist that complement ongoing LPAC interventions. All interviewees from LPAC and LACC agreed that there is a need for greater cooperation and coordination to improve information sharing among anti-corruption agencies. In addition, lessons learned need to be shared across anti-graft agencies, especially those lessons concerning the application of new skills, investigative methods and techniques, and the efficient use of agency specializations, mandates and budgets, such as through joint task forces, joint operations, and cross-agency working groups. Evaluation findings also reflect a need to reduce the perceived conflict of interest in the decisions to pursue corruption cases. Over the last year, a new team was formed out of the Financial Crimes Working Group to tackle major economic crimes cases considered a national security interest, called the Investigation and Prosecution Team (IPT). According to two LACC interviewees, the IPT was formed as a direct response from LPAC’s financial crimes trainings. The purpose of the IPT is to create efficiencies in a major economic crimes investigation through liaising between agencies, sharing critical information and lessons learned, developing joint operations at a strategic level or allocating operations to different agencies, and directing the course of an investigation. The IPT is comprised of senior operations officers from the LACC, the LRA, the Transnational and Major Crimes Units of the Liberia National Police (LNP), specialist MOJ prosecutors, and a Liberia National Security Agency officer, although this may change as the IPT becomes more established. The IPT meets every two weeks, where they discuss ongoing investigations and identify economic crimes cases at the national-interest level, according to an LPAC interviewee. The IPT allows strategic thinking and operational decisions about the case to take place prior to commissioner-level approval to proceed. Rather than subvert institutional leadership, the IPT identifies cases and helps establish a strategic investigative framework more efficiently by identifying resources and capacity from multiple institutions before it goes to the relevant institutions’ leadership for approval. LPAC’s embedded Investigation Specialist noted the IPT has the capacity to contact or pull in people from other member’s teams as necessary to complement an investigation’s needs. He also noted that IPT members discuss how they applied or suggest the application of investigative methods and tools learned in LPAC trainings in ongoing investigations. However, as one participant noted, it is a “careful balance to make the IPT functional but not too formal,” which would require Commissioner-level oversight of the group. Impacting the prosecution of corruption cases is Liberia’s low judiciary capacity to adjudicate corruption￾related cases. Fifteen respondents (32 percent) of the Monrovia-based key informants listed in Table 1 At the strategic and operational levels, the IPT senior officers propose to act independently of their agency heads who (in theory) should operate at the policy level only. The officers trust each other and would like to be in a position where they need not consult agency heads about strategic and operational matters - nor should they. LPAC Staff USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 22 noted that low technical knowledge and capacity in the judiciary hinders the adjudication of complex and fluid crimes, such as financial and cyber crimes. The Judicial Institute conducted two cyber crimes￾related trainings at the quarterly judge and magistrates training in April 2017 and 2018, but there is no evidence to support any financial crimes-related trainings were delivered to judges and magistrates. Similarly, there is no evidence to support if the Judicial Institute’s other training programs and LNBA’s CLE program offers trainings on these topics. CONCLUSIONS The judiciary capacity to adequately adjudicate financial and cyber crimes cases will continue to frustrate the GOL’s ability to successfully prosecute corruption cases. The IPT has significant potential to be a mechanism that encourages greater cooperation and coordination in the investigation process, and with technical assistance and logistical support by LPAC, it could lead to greater efficiency and use of existing resources under strained economic times. It also has the strong potential to encourage transparency and accountability in decision-making in pursuing major economic crime investigations and prosecutions. LPAC trainings targeting judges and magistrates are not sufficient to build the capacity of the judiciary to adjudicate financial and cyber crime cases. Overall, LPAC has the foundation in place to support institutional efforts that address corruption issues in the current context, but prevention efforts and building civil society and media are critical elements that require greater emphasis by LPAC and USAID (see Section 3.2 below). EQ3.2: Are there specific lessons learned from LPAC that should be considered as USAID/Liberia prepares its new country development cooperation strategy? Although specific activity design and implementation lessons learned are discussed in the main body of this report, with additional detailed analysis in Annex I, this section highlights lessons learned derived from LPAC’s strategic scope and interventions that USAID/Liberia should consider in developing their CDCS. Additionally, collective feedback from respondents in Monrovia, Kakata, and Gbarnga on ongoing to accessing justice beyond LPAC’s current mandate is summarized in Annex K. FINDINGS Balance needs to be sought between the supply and demand sides of accessing justice: Access to justice remains a major factor underpinning the rule of law and democracy in Liberia. While support toward JI trainings have made some gains in improving the performance of judicial actors, access to justice from a holistic perspective is not adequately addressed in the activity’s ToC and logical model and neglects the demand side of accessing justice. Reduce gaps and inefficiencies in providing Public Service Scholarships to law students: Although LPAC’s Public Service Scholarships have been documented in this report as a significant success, LPAC is unable to release new rounds of scholarships for first year law students in Y4 and Y5 of implementation because these new scholars would be unlikely to complete their studies without support after LPAC closeout. In effect, releasing new scholarships could potentially waste resources and send a poor message about USAID commitment to Liberia. Establish communication mechanisms that collectively introduce new Supreme Court rulings and opinions: Judges, magistrates, and lawyers do not regularly read Supreme Court opinions nor are circuit court judges aware of opinions handed down by their colleagues. With LPAC’s support, the JI and LNBA offer trainings, Complex financial and cyber crimes are becoming increasingly common here, and we need to start recognizing that it is now Liberia’s problem too. Chief Judge Commercial Court of Liberia 23 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV conferences, and other mechanisms to educate judicial actors on cutting edge legal issues, but they are not consistent in informing judicial actors on recent Supreme Court rulings. Improve coordination of legal aid services: The LNBA is addressing the issue of access to legal services, particularly among women and indigent individuals, in its LPAC-sponsored legal aid clinic program. There are local CSOs who provide similar legal aid services but have only become aware of LNBA’s program at a recent forum sponsored by LPAC. According to one CSO representative, there is a need for greater cooperation between the LNBA’s legal aid clinic and local NGOs who provide similar legal aid and support services to help tackle the large number of pre-trial detainees in a coordinated way. Place equal emphasis on prevention and investigation as a more cost-effective way to minimize small- to medium￾scale public corruption: Investigating and prosecuting allegations of public fraud and corruption cases can be costly. According to a senior LACC investigator, “Prevention is critical to our anti-corruption efforts. The high costs of investigating corruption allegations [does not make investigating] claims of petty corruption (e.g., small-scale bribes) cost effective with our ongoing budget issues… But we cannot ignore them either.” Another senior LACC official noted that if LPAC placed greater emphasis on prevention, then LACC would “achieve greater results,” particularly since prevention activities do not directly deal with the other players, such as the MOJ and Judiciary “where the process falls short.” Low capacity and awareness within civil society to improve GOL transparency and accountability: LACC’s education and awareness raising efforts have been less effective, and according to LACC officials, CSOs lack the capacity to develop programs that target transparency and accountability initiatives to support LACC’s anti-corruption efforts. CONCLUSIONS LPAC’s ToC––that better trained judicial actors can contribute to the delivery of justice services––is relevant to rule of law in the Liberian context. Training is a channel to improve performance of judicial actors. Citizens, however, are the ultimate beneficiaries of justice services, and they determine whether the justice system is delivering quality and impartial services. LPAC is correct not to release new scholarships in Y4 and Y5, but this leaves a gap in LPAC’s efforts to support a diverse student body at the Law School and could result in a reduction in female, rural, and less privileged students enrolling. The performance of judicial actors and consistency in the delivery of services would be greatly improved if there was a general understanding of new laws emanating from Supreme Court rulings. A coordinated effort between IPs and CSOs to address the growing need for legal aid services, particularly among poor, vulnerable, and rural populations, would pool resources to raise awareness, improve service delivery, and reach a broader population in need. Prevention is a critical element in the fight against corruption, which should take a greater role in anti￾corruption strategies. It is more cost-effective and sustainable in the long run to inform public officials about conflicts of interest and institutional risks and vulnerabilities toward corruption. Education and outreach are often subjective and difficult to justify when impacts of educational activities are not readily apparent. However, when prevention is linked to investigation activities, such as complaint mechanisms and visible actions arise from these complaints, then awareness activities start to gain traction. Civil society involvement in the anti-corruption effort is critical to reflect the public demand for more transparent and accountable governance. When CSOs lack the capacity to understand and promote the concepts of transparency and accountability, a significant opportunity is lost for agencies such as LACC to reach the local level effectively. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 24 EQ4: What is the extent of the GOL’s commitment and capacity to sustain Components A and B, both financial and administrative? To answer EQ4, the evaluation team interviewed USAID, LPAC, and GOL institutional stakeholders (noted in Table 1) and asked them their perception of institutional commitment and capacity toward achieving the objectives in their relevant component. In addition, the evaluation team reviewed institutional budgets, levels of autonomy, present skills and capacity, LPAC reports, and other resources (beyond budget) to identify and triangulate data. The evaluation team tracked frequency of specific commitment and capacity references and found that the primary implementation constraint identified by USAID and LPAC participants was the lack of resources required to support most LPAC activities currently and beyond the life of the activity. LPAC institutional partners are both public and private institutions. GOL institutions include the Judicial Institute and the Law School. LiberLII was developed and funded by international donors, and the LNBA is a private a membership-based and non-profit organization for professional lawyers practicing law in Liberia. For this question, the commitment and capacity of all institutional partners was assessed despite their status. FINDINGS Measuring the commitment and capacity of GOL’s financial and administrative sustainability remains a daunting challenge. The county’s annual budgets for the past ten years focused largely on salaries, staff benefits, and recurring costs as opposed to operational budget that supports implementation of the Institution’s mandate. Of GOL’s 2018/2019 annual budget (approximately $570 million), $317 million covers compensation for employees (62 percent of recurring expenditure). This is an increase of three percent of the overall budget for 2018/19 versus the 2017/28 budget. Judicial Institute: The Judicial Institute was established in 2010 with a mandate to provide ongoing refresher courses for magistrates, judges, public defenders, and other judicial actors primarily through its PMTP and quarterly trainings for public defenders, circuit court judges and magistrates, and court clerks. It functions under the direct administrative supervision and financial authority of the Supreme Court. The Judicial Institute is one of the few judicial organs that have specific allocation in the Judiciary’s budget supported by the national budget. Despite budget issues noted in Table 4, several senior level staff members at the Judicial Institute noted that a portion of the budget can be used to conduct one to two refresher trainings per year for public defenders, bailiffs, and clerks, but not the PMTP. Table 4: Summary Assessment of Judicial Institute’s Financial Commitment and Capacity 24 In the 2016/17 annual budget, the Judicial Institute was $221,000, increased to $392,000 in the 2017/18 budget and $394,000 in the 2018/19 budget. JUDICIAL INSTITUTE COMMITMENT INDICATOR PERCEPTION RANK OF CAPACITY AND COMMITMENT BUDGET/CAPACITY REALITY  Institutional budget is aligned to contribute toward delivering services Average rank: 2.5  Perception of lack of high-level political will given budget shortfalls in the judiciary, but institutional commitment towards the Judicial Institute is reflected in the steady increase of the Institute’s budget by the Supreme Court24  Steady decrease in the Judiciary’s budget: $19 million in 2016 to $16 million in 2017, from which 15 circuit courts and 100 magisterial courts are funded.  Funding is inadequate to set up new courts, improve infrastructure, and instituting reforms 25 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV With only six in-house trainers, the human resource capacity within the Judicial Institute remains insufficient to meet the needs to deliver its full offering of trainings on a timely basis.25 The in-house trainers also do not have a specialty or advanced training in different aspects of law. The evaluation found this makes them less confident in conducting trainings for circuit court judges. The Law School: The Law School operates under the University of Liberia, and therefore the University through its national budget (approximately $16 million according to the 2018/19 budget), covers the Law School’s operational costs and salaries—though, the percentage of allocation to the Law School is not clear. Budgetary constraints continue to plague the Law School’s ability to hire full-time professors and offer them competitive salaries, in addition to limiting infrastructure improvements. Despite fundraising being a key area of LPAC’s intervention, all tuition and fees generated through student enrollment and fundraising goes directly to the University, not to the Law School’s operational funds. Although the Law School lacks autonomy, it was able to open and obtain signatory authority over a dedicated bank account funded by the University in May 2017, moving the Law School toward budgetary and operational autonomy. According to LPAC, this account has been fully funded and operational since it was opened. Despite its low financial capacity and commitment, the Law School has demonstrated commitment in initiating reforms, including evaluating the performance of professors and instituting a digital system for the submission of grades which reduces fraud and protects the integrity of the grading process. Other reforms include an online database system for submitting student application, a more rigorous and transparent admission process and a revision of the curriculum that has established elective courses to meet international standards.26 LPAC has contributed in the design and delivery of some of the courses. The delivery of new curriculum has been impacted by the low number of full-time professors teaching at the Law School, and given the budget restrictions, it is not able to recruit more. Evidence does not exist to show that professors have the commitment to undertake curriculum development, textbook writing, and independent scholarly research for publications, despite incentives to provide remuneration, nor is there evidence to suggest that this is a priority. LiberLII: LiberLII’s Board is not functional and lacks commitment to its success, which is reflected in a lack of operational capacity since August 2017. On August 28, 2017, LPAC submitted the Restructure and Transition Plan to LiberLII’s Board for review and approval. The Plan calls for the incorporation of LiberLII into the Law School, while LiberLII maintains an autonomous status. In early October 2018, the Board 25 LPAC is filling this gap by covering the cost of five additional in-country lawyers who serve as trainers. 26 Some elective courses include Environmental and Petroleum Law, Children’s Law, and Gender Law.  Cost-share to conduct assessment in southeastern countries reflects commitment, despite budget shortfalls  Chief Justice announced in March 2018 that the Supreme Court is willing to provide funding to increase the number of judges and magistrates to attend future PMTP trainings  Supreme Court seconded three judges to the Judicial Institute’s training program  Supreme Court representative noted the Judicial Institute receives the least allocation of the Judiciary’s budget, which mainly goes toward salaries.  Supreme Court is not confident the 2018/19 budget is sufficient to deploy PMTP-3 graduates. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 26 finally met after more than a year, and provisionally approved the Transition Plan, but details of how it will move forward are unknown. LiberLII has no financial capacity to operate on its own. International development partners have provided financial and technical assistance, but since August 2017, LiberLII has not been operational. LPAC funding was suspended until the Board decided on/approved LiberLII’s transfer under the Law School and other measures to building its foundation for future sustainability. LNBA: The Executive Council makes strategic decisions for the organization and has demonstrated a strong commitment to providing organizational leadership and stewardship. The LNBA’s Executive Director, who heads the Secretariat, rated the Executive Council as highly effective and functional. Through its commitment, the LNBA has a functioning and professionalized Secretariat. CLE includes presentations of research papers on topical issues such as real property law and technical and legal writing. These topics evolved as a result of a survey conducted by the LNBA in partnership with LPAC that targeted lawyers. This professionalizing of the annual assemblies through the presentation of new legal ideas will incentivize lawyers to attend and create opportunities for increasing the income or revenue base of the association. In order to improve sustainability, the LNBA instituted a fundraising strategy which highlights three areas for income generation: annual dues; fees to attend annual assemblies; and fee-based CLE courses. Multiple respondents noted that the LNBA is highly functional. They are able to raise money from multiple sources and are able to pay for activities out of their own budget. LACC: The LACC has financial constraints that hinder the implementation of its mandate. Without resources to cover expenses for core services such as investigation and prosecution, according to LACC and LPAC respondents, the LACC will continue to be constrained to make decisions on which cases to investigate and prosecute based on geographic proximity of LACC’s main office in Montserrado to the courts in other counties. This negatively impacts not only the fulfillment of LACC’s mandate, but also its reputation as a functional anti-graft agency. Budget constraints are particularly relevant as it limits LACC’s venue options to investigate allegations of public corruption and prosecute corruption cases. Criminal Court C, where all financial crimes cases within Montserrado County are heard, has an overloaded docket, causing significant delays in prosecuting cases. Table 5: Summary Assessment of LACC’s Financial Commitment and Capacity 27 Ranked from 1 (poor) to 4 (very good): 1= The Institute’s budget is severely unbalanced, focusing a significant majority on staff salary and basic reoccurring costs, and little on supporting the service delivery; 4= The Institute’s budget is balanced, showing sufficient allocation of funds towards supporting service delivery while providing sufficient funding to cover operational costs and staff salaries. LACC COMMITMENT INDICATOR PERCEPTION RANK OF CAPACITY AND COMMITMENT BUDGET/CAPACITY REALITY Institutional budget is aligned to contribute toward delivering services27 Average rank: 1 (although most respondents refused to rank)  Budget constraints lead to difficult operational decisions on which claims to investigate and which cases to prosecute.  According to LACC technical team members, the bare minimum [in budget] is provided, which is an issue of  Overall, the budget has declined since 2016.  2018/19 budget is $2 million, of which $1.6 million (80%) covers salaries and benefits.  $400,000 is slated for reoccurring operational costs 27 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV In addition to budget constraints that impact LACC’s capacity, a lack of political will and commitment impede progress and perceptions of effectiveness. LACC’s BOC is comprised of Presidential appointees and has the authority to decide which action to take on investigative findings and to decide whether or not a particular case should be prosecuted. Their appointment status may have some influence on their objectivity, which raises concerns that decisions are vulnerable to political bias. LACC technical team members, LPAC and USAID respondents noted that the current BOC is not effective, or as effective as it should/could be. One respondent noted that the BOC’s role is in part policy and strategy, but also technical oversight, where the dysfunction and inefficiency take place. It is more of a “control issue” than of enhancing the division’s effectiveness or efficiency. Furthermore, broader lack of political will impacts LACC’s capacity and commitment. The ongoing delay in the legislature to pass the Whistleblower Bill28 constrains the capacity of the LACC and other graft￾fighting agencies to investigate and prosecute corruption cases. Despite these constraints, LPAC has been strengthening the human resource capacity of the LACC in the area of investigation. Through training provided by LPAC, the skills of investigative team members have increased backed by their commitment to perform, which is reflected in the establishment of joint investigations with other anti-graft agencies and the issuance of two indictments, with two more indictments to be firmed up by the end of 2018. Other technical arms of the LACC have also been strengthened, including the prosecution team who have attended multiple LPAC trainings on financial crimes and other investigative trainings. According to three LACC participants from the IPT, the use of link analysis from an LPAC training session allowed them to expand the net of people allegedly involved in the alleged crime, which has been a significant help in improving investigative approaches and findings and increased the opportunity to propose additional indictments. With increased capacity and commitment, the quality of investigations might improve in their ability to produce credible evidence for future prosecution. CONCLUSIONS The financial and administrative capacity of partner institutions under Component A to sustain interventions supported by LPAC is mixed. The Judicial Institute does not have the financial capacity to conduct large-scale trainings such as the PMTP because its budgetary allocation cannot match the cost. It can, however, conduct within its budget allocation under the judiciary budget a limited number of small￾scale trainings such as public defenders and clerks. Despite demonstrated judiciary commitment to increase capacity of the Judicial Institute by seconding three sitting judges, the number of full-time staff at the Judicial Institute is inadequate to roll out and sustain the Institute’s various training programs. Overall budgetary constraints for the judiciary create uncertainty whether Cycle 3 graduates will be deployed. If this turns out to be the case, LPAC’s intervention could be adversely affected. Like the Judicial Institute, the Law School lacks financial and administrative independence. It receives resources from the overall budget of the University of Liberia, but these funds are allocated toward salaries and operational costs. The Law School, however, has demonstrated commitment to internal reforms that 28 The Whistleblower Bill would give protection to individuals who provide information on corruption practices. political will. If there was political will, greater allocation of resources would be given to help LACC fulfill its mandate.  A member of LACC’s BOC notes that there is no firm commitment by the President.  If you want to fight corruption, you need to invest in [the combating capability]. “This is not happening here,” according to one respondent. and some basic technical activities. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 28 are contributing to the quality of services delivered. LiberLII has zero financial capacity to manage its operations. The LNBA has demonstrated more capacity and commitment to continue implementing LPAC’s objectives compared to the other institutions under Component A. A functional Secretariat and Executive Committee are in place and it is responsive to member needs through CLE courses. Similarly, a vibrant fund-raising scheme is implemented making the LNBA a more financially healthy organization. Under Component B, the steady decline in budget allocations undermines the financial capacity and commitment of the LACC to undertake some core actions like investigation and prosecution that are strategic in implementing its mandate. At the same time, LACC’s institutional and technical capacity to plan and carry out investigations has been greatly enhanced over the last year, and commitment among the technical/operational teams to achieving their mandate is strong in the midst of dwindling resources. EQ4.1: Are there specific benchmarks for GOL actions that should be considered as USAID contemplates its investments in this sector? FINDINGS In addressing the question on benchmarks that institutional partners can undertake that can be considered by USAID as demonstrated commitment, the evaluation team interviewed representatives from USAID, LPAC, and GOL. LPAC’s narrative reports and institutional budgets prescribed in the GOL’s national budget 2018/19 contributed evidence to the interviews. Based on the tasks performed above and given that financial commitment from the GOL may be unforeseen for the near future, other benchmarks obtained by institutional partners could be considered as commitments. First, there is an alignment of institutional budgets to cover costs of some core activities that are required for the fulfillment of their respective mandates. The Judicial Institute noted it is able to conduct a limited number of small-scale trainings such as the public defenders and the court clerk trainings, but not the large-scale PMTP. Second, progressive increase in the number of in-kind contributions such as allocating staff time in the implementation of activities has proven to be considered a form of commitment, as reflected in KIIs with USAID. Third, institutions demonstrating the commitment to cost-share the implementation of key activities such as conducting research at the law school or a change of venue to prosecute corruption cases by the LACC could be considered as benchmarks contributing to commitments. And fourth, the appointment of qualified people with good reputations to key institutional Boards, such as the Judicial Institute, LiberLII, and LACC, would go a long way in reflecting the GOL’s commitment toward achieving the institution’s mandate, as would the voting in of qualified members to the LNBA’s Executive Committee at the next LNBA Convention in November 2018. CONCLUSIONS Some institutions are beginning to demonstrate commitment through in-kind contributions related to staff time. Other forms of commitment such as cost sharing should be considered as a gradual process due to the years of dependency that has characterized the partnership between the past government and its international development partners. 29 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV EQ5: To what extent does the redesign increase participation of women in a meaningful way? To answer EQ5, the evaluation team relied on relevant LPAC reports and interviewed USAID, LPAC, and GOL institutional partners (noted in Table 1) to triangulate reported references to women’s inclusion in the program. In addition, the evaluation team supplemented reported activity data with a brief assessment of the barriers women face in participating in the rule of law sector and in accessing legal services in Liberia. The findings of this assessment can be found in Annex F. FINDINGS Lack of gender inclusion in the original design and implementation approach: The evaluation did not find evidence of a gender assessment conducted by USAID in LPAC’s original design. The original design and implementation approach lacked the nuances that would reflect a studied review of the critical social and economic constraints that women face in actively participating in all levels of the rule of law sector. There is no evidence to support that LPAC developed any policies, procedures, or manuals to ensure gender considerations have been woven into intervention design and implementation. There is also no evidence that LPAC conducted any gender trainings for the GOL institutional partners or grant recipients. There is evidence to support that women participated in and benefited from LPAC interventions, such as the LPAC-sponsored Public Sector Scholarships awards and the pilot LNBA legal aid clinic grant. However, data collection prior to the redesign was skeletal with no breakdown by sex. Lack of strategic focus in redesign: LPAC’s redesign did not address the lack of gender integration in its strategic approach. LPAC continues to lack a strategic or guiding document that states the activity’s approach to gender integration. LPAC did change the way it captures activity data, including detailing the breakdown of beneficiary data – although inconsistent – and how it targets activities that support greater participation of women as judges, magistrates, and lawyers and improved access to justice for women. In addition, USAID has not defined what ‘increased participation of women in a meaningful way’ means or what appropriate parameters or criteria would look like for ongoing or planned activities. Therefore, any current activities or approaches have been developed independently and are based on the best practices supported by LPAC’s leadership team. Data management: Prior to the redesign, LPAC did not disaggregate activity data by sex. The current draft of the AMELP disaggregates data by sex to improve understanding and reporting of women’s participation in LPAC. However, not all LPAC GOL counterparts are capturing data relating to women’s involvement in achieving their mandates. For example, LACC disaggregates by ‘community’ and not by sex, whereas the Law School and the LNBA disaggregates by sex. However, there is no data available to compare female to male participation in the pro bono services offered at the LNBA. Increased participation of women as service providers: Over the last three PMTPs, the Judicial Institute has seen a consistent increase in women’s participation. PMTP-3, for example, has 12 females out of the 60 participants enrolled. This is an increase from six female participants in PMTP-2. Once they graduate, the new magistrates return to their home counties to serve magistrate courts. To increase the enrollment of women and to increase the magistrate candidate pool more generally, in August 2017, LPAC accompanied the Judicial Institute to visit the southeastern region of Liberia to assess women’s participation and interest in entering the judicial or legal sector and raised awareness of the upcoming PMTP-3 cycle. According to a Judicial Institute respondent, “LPAC’s intervention with the awareness campaign benefited the Judicial Institute, by helping them achieve both improved scores in gender balance and [an improved] regional county balance in both northern southeastern regions in their PMTP recruitment processes.” USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 30 As part of its effort to increase technical capacity in the judiciary, LPAC-supported trainings conducted by the Judicial Institute have seen an increase of female participants from 13 in 2017 to 43 in 2018, according to training attendance sheets. The Law School is noting an increased number of women applying each year. In 2018, 15 women have been accepted and enrolled in the new semester. LPAC’s Public Service Scholarship proved to be a significant benefit for female law students. According to FGD participants and the Associate Dean of the Law School, the scholarship helped reduce financial stress, thereby improving learning and contributing to a higher retention of female law students. There are eight female law school students who are currently benefiting from this scholarship, and they have signed an agreement to work in any public service capacity to increase women participation in the legal sector. Since the launch of the LNBA’s legal aid clinics in May 2017 and June 2018, more female lawyers are offering pro bono services at these legal aid clinics. Increased participation of women through access to justice: Since October 2016, LPAC provided the operational and technical support to formalize LNBA’s legal aid services in Margibi, Bong, Bomi, Grand Bassa, and Montserrado. By establishing these clinics, the LNBA improved women’s access to legal assistance, mainly on issues relating to family and property rights. However, the evaluation identified that the legal aid clinics have an uneven approach to increase participation of women with Bong, Bomi, and Margibi, reporting seven female clients to 148 male clients who received pro bono assistance over the period. The Monrovia-based clinic shows a slight increase, with 21 women and 143 men receiving pro bono services, mainly targeting family and property law issues. The clinics in Gbarnga, Bong, Kakata, and Margibi are more focused on pre-trial detainees (mainly men) and less on other legal issues impacting the community. Staffing and technical assistance: The addition of a female Justice and Legal Advisor (JLA) to LPAC helped to increase the participation of women. Since her onboarding in Q1 2017, the JLA has worked with the former Chief Justice of the Supreme Court29 to develop and co-deliver curricula on gender and children laws, and works with the Chief Judge for the Commercial Court on revising the law journal structure and responsibilities to help improve the environment at the Law School, which she deems critical to improving the “yearning for learning” atmosphere within the legal profession. To support the proper oversight and management of the pilot legal aid clinics, the LNBA recruited and appointed a female lawyer as its first Chair. Under LPAC’s guidance, the LNBA ensured female representation at each of the clinics. At minimum, each clinic has one trained female to work with her male counterpart. Other findings: LACC recruited and two female investigators within the last year, while the Law School has qualified female lawyers who are teaching estate and trust, heritage law, practice court and legal ethics, and other curricula. These findings are worth noting here, although it is not possible to measure the contribution that LPAC had to this. CONCLUSIONS Though LPAC has more actively engaged women, particularly through its partners’ activities, the context within the sector has not changed for women. Women continue to face the same challenges and barriers to entry. See Annex F for a brief review of these constraints and barriers. 29 Of note, the former Chief Justice was Liberia’s first female to be appointed Chief Justice of the Supreme Court. 31 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Raising awareness of the PMTP-3 training proved a success for the Judicial Institute in increasing the number of women magistrates in the judiciary. It will also increase the number of magistrates in rural areas where female representation in the judiciary is highly critical at the end of the training year. The significant increase in women participation in quarterly trainings should result in improved adjudication of cases, improved advocacy as public defenders, and encourage greater participation of women in the justice sector – either as service providers or women seeking legal or judicial services. The increasing number of women receiving Public Service Scholarships at the Law School will continue to improve retention and participation of women in their legal education. The legal aid clinics have increased opportunity for women to not only access legal aid, but to provide legal aid services. However, the LNBA should place greater emphasis on diversifying its service portfolio to ensure women’s access to legal aid. V. RECOMMENDATIONS EQ1 Assessing the Original Design and Redesign: Conduct “defining success” workshop: USAID and the IP should jointly plan and conduct a “defining success” workshop during which all key stakeholders can discuss and agree on a) key activity definitions (e.g., what does ‘rule of law’ mean for LPAC?), b) key outcomes/goals, and c) what activity success will look like (and in turn, how to measure this success). This workshop could open by sharing midterm evaluation findings and conclusions and be facilitated by an external consultant, clarifying needs and what can realistically be changed to improve performance for this activity. The outputs of the workshop could be an updated work plan, AMELP, ToC and indicators, partner MOCs, and other supporting documents. Prior to this workshop, USAID and the IP should meet with institutional partners to update their institutional needs and priority assessments to more comprehensively document capacity and resource gaps and determine where LPAC is and is not meeting expectations. In addition, LPAC should review midterm evaluation findings with the institutional partners to ensure all stakeholders are aligned with regard to technical assistance needs and skills transfer. In particular, LPAC should work with institutional partners to define specific skills to be improved and set up measurable milestones for skills transfer. This way, LPAC can manage expectations among institutional counterparts. Prior to the workshop, USAID should also redefine/identify what the Mission wants to achieve through this activity considering changes to the Liberian context that have impacted the original assumptions relating to GOL capacity. This could be undertaken through an internal session to assess, from USAID’s perspective, what is/should be achievable and how it contributes strategically to the CDCS goal. This will refine LPAC’s redesign, while informing the development of the new CDCS. Disengage administrative and financial capacity from the Judicial Institute and Law School scope: Given the constraints of the contracting mechanism, USAID should adapt the language in the scope to reflect a continuation of administrative and financial capacity building for LNBA and LACC, as appropriate, but remove this aspect of the scope from the Judicial Institute and Law School. Revise LPAC’s goal: As a specific step in “defining success” for LPAC and the workshop recommended above, USAID should update the goal statement. The following language change is recommended based on evaluation findings: LPAC seeks to promote a more effective and accountable formal justice sector through improved capacity of legal professional training institutions and regulatory institutions and improve the rule of law. The change should be made official against the TO’s original scope. Retaining the “regulatory institutions” identifier would allow the anticorruption component to expand beyond LACC, while removing the “rule of law” element would reduce the requirement for measuring change. Given that LPAC USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 32 did not conduct a baseline and the differing definitions and understandings among respondents of ‘rule of law,’ it would be difficult to measure this (e.g., through a retroactive baseline) without a significant degree of error. USAID should encourage dialogue with all technical leads on LPAC: Dialogue with all technical leads on LPAC would improve a nuanced understanding of constraints and successes by USAID and clarity of expectations across LPAC. This should include face-to-face meetings and presentations of progress and achievements, in addition to USAID attending and observing events, such as trainings, workshops, and conventions. Conduct a Gender Assessment: USAID should commission an assessment of women’s involvement in rule of law in Liberia to comprehensively identify social and economic barriers and the current environment for women in the legal profession. This could be commissioned through LPAC to use the IP and partners’ knowledge and experience in the sector. The initial findings of this midterm evaluation are based on limited interviews with both men and women working within the legal field, and a more comprehensive approach is needed to understand the challenges women face in their legal education, work environment, and aspirations as legal professionals. Findings from this study should support LPAC interventions and others in the Mission portfolio that tackle increasing women’s participation in Liberia’s rule of law institutions. EQ2 LPAC Achievement Since Redesign: Update the Judicial Institute training curriculum: Given the mixed responses on training relevancy and dated training materials, the IP and the Judicial Institute should conduct an annual assessment of former PMTP students and current quarterly training participants on relevant courtroom topics in the Liberian context, and for recent PMTP graduates, obtain feedback on its usefulness to practice. The findings of the assessment should inform an updated training curriculum annually. More hands-on mentorship at JI: As discussed under both EQ1 and EQ2, more mentorship and guidance are needed to build the capacity of the current JI training team to conduct comprehensive curriculum review and development. LPAC should work with the JI to establish a mutually-accountable training and mentorship framework that would identify gaps, solutions, and targets for JI staff to reflect growing internal capacity. Establish the Clinical Program at the Law School and link it to the LNBA’s legal aid clinics: LPAC should immediately restart communications with the Dean and work with the Law School to develop the Clinical Program concept, structure, and pilot clinical focus areas and develop criteria for student involvement and successful achievements in meeting academic requirements. The IP and the Law School should also involve LNBA in establishing the initial clinical focus areas to ensure synergy with LNBA’s legal aid clinic services. Revisit the fundraising function at the Law School: In its current status, the Law School is not autonomous, and therefore, all fundraising proceeds go straight to the University of Liberia. LPAC should revisit the effectiveness of this intervention. One option is to shift this fundraising mechanism to LiberLII once it is officially under the Law School’s arm. If the intervention is discontinued, USAID should adjust the language in the TO’s Scope under Component A.2.b. Delete Objective A.2.f from LPAC’s TO: For the Law School to provide legal advice to the University of Liberia through a legal department that does not rely on faculty is out of place with what LPAC aims to achieve with the Law School. USAID should remove this from LPAC’s TO. Reinstate LiberLII’s grant: Upon approval of the 2017 Restructure and Transition Plan by the LiberLII Board, LPAC should commit to a one-year grant that would cover operational costs. According to the plan, one year would be “sufficient time” for all stakeholders to engage the GOL to assure continued funding beyond the grant. 33 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Increase women’s access to legal aid services: The LNBA should internally develop and adopt targets based on clinic locations that hold clinics accountable to diversifying their legal services to women with legal issues who cannot afford to hire the services of lawyers. This policy should be communicated to the public through the radio, in print media, and on social media. Continue to relocate prosecution of corruption cases in the short term: To build the capacity of satellite courts and reduce the docket load in Monrovia, the IP should continue to provide logistical and operational support to the prosecution team, whether it is from the LACC or MOJ. USAID and the IP should set criteria for this assistance to clearly identify priority cases and manage GOL expectations. Continue efforts to establish a complaints management and assessment policy: LPAC notes in their draft Y4 Work Plan that they will work with LACC to develop a complaints management and assessment policy. In developing the policy and accompanying procedures, the IP should support efforts to diversify options to lodge complaints or concerns of corruption, which, if established properly, would feed into a data monitoring system that would track perceptions or instances of public corruption, identify trends, and inform the Education and Prevention Team to respond accordingly through education and outreach. The policy should also establish criteria to determine who investigates the claim (LACC or other agency), according to capacity, jurisdiction, and other factors. In addition, the policy should incorporate a “Do No Harm” approach to the system, ensuring that those lodging complaints or notifying LACC of concerns of corrupt activities or abuse of power are not negatively impacted. In this light, ensuring confidentiality will require an extensive and ongoing public education campaign. EQ3 Assessing Investment vs. Perceived Benefit: Scale up successful activities: LPAC’s support toward PMTP-3 and the quarterly public defenders training should be scaled up. This will ensure that these trainings occur every quarter and newly hired public defenders are quickly trained for the courtroom. For recommendations detailing how LPAC trainings could be adjusted, such as the quarterly judges and magistrates training, see EQ1. Although the legal writing and adult learning workshops received mixed reviews, LPAC should expand on this workshop concept within the Law School, as the demand for improved legal-writing skills among students and legal practitioners is high. LPAC may consider shifting this course to the LNBA’s CLE to target legal practitioners if traction at the Law School continues to be slow. Scale up support to new Law School curriculum: LPAC should support the Law School to develop additional courses on new legal topics and offering them to students. Maintain support to the LNBA’s CLE trainings, annual conventions, and legal aid grant: LPAC should maintain its support for the CLE trainings and the LNBA’s annual convention. Lastly, LPAC should continue to support the LNBA legal aid clinics. Additional funding should be provided for the legal aid clinics to diversify their focus beyond pre-trial detainee cases to issues relating to family and property rights. Scale back ineffective activities: The Law School retreat can achieve similar results on a lesser, cost-effective scale by hosting the conference in its facilities or at the University of Liberia if the broader legal community will attend. A smaller scale leadership meeting held at an offsite location would also achieve similar results in developing an annual strategic plan. This would require limiting attendance to the Law School’s leadership and select faculty with a strict agenda and clear deliverables. To achieve more effective change in the area of anti-corruption, USAID should approve a modification to clearly affirm LPAC’s broadened anti-corruption scope: To do this, USAID should rename Component B to “Strengthening the Liberian Anti-Corruption Agencies.” This would then be in line with LPAC’s proposed modification for greater coordination and collaboration with the Liberian Public Procurement and Concessions Commission (PPCC), LRA, MOJ, Transnational Crimes Unit, LNP, FIU, and other relevant USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 34 agencies through joint trainings, working groups, and operations to deepen and broaden assistance to deal with economic crimes. LPAC should provide training and logistical support to the IPT: The support to the IPT should be implemented with sensitivity to avoid the perception that LPAC is pushing a formal task force. Therefore, LPAC should follow through on developing and delivering a management course for senior officers involved with the IPT on operations and joint operations management, targeting issues such as ethics and lawful conduct. LPAC should advise IPT members on the development of standard operating procedures and protocols for the group and identifying ongoing operational needs. Focus financial, economic, and cyber crimes to one or two satellite courts: LPAC should revise and develop curricula and accompanying modules to target financial and cyber crimes to improve understanding and effective adjudication of these cases. LPAC should work with the Supreme Court to identify an appropriate satellite court location to focus the delivery of Judicial Institute’s quarterly judges and magistrates training modules on adjudicating complex financial, economic, and cyber crime cases. Specific criteria should be established between USAID and LACC, such as cost, convenience, and level of propensity for these related crimes. This would allow for efficient delivery of quality and appropriate trainings to judges and magistrates who would most likely hear cases relating to these matters. The establishment of this pilot court would also test the ongoing assumption that corruption cases are not being prosecuted because of the overcrowding of cases in Criminal Court C. Recommendations for USAID to consider as it prepares its new CDCS are outlined in Annex F and K. EQ4 Commitment and Capacity: USAID should clearly communicate cost sharing expectations with its GOL counterparts and broaden the criteria in assessing GOL commitment beyond funding. USAID should distinguish between political-level commitment demonstrating will and technical-level commitment demonstrating capacity when determining whether to continue or pursue new opportunities with GOL counterparts. Given the financial constraints faced by the institutional partners, LPAC and USAID should reach a consensus on priority areas of the partners wherein the partners can make in-kind contributions. Benchmarks: USAID should adopt the following benchmarks for defining commitments: Judicial Institute: (1) The Judicial Institute is able to hire more qualified trainers based on an increased budget allocated to staffing; (2) The Judicial Institute allocates greater operational funds to cover some of the quarterly training costs. LiberLII: (1) The LiberLII Board approves the 2017 Restructure and Transition Plan; (2) the Board relinquishes its legal and autonomy status; (3) an Executive Committee is formed to run the day￾to-day operations of LiberLII; (4) national government provides budget to sustain LiberLII; and (5) a detailed exit strategy is developed that calls for GOL and stakeholder funding. LACC: (1) The BOC is able to make internal adjustments to its budget to make cost-sharing contributions in the implementation of activities; (2) Appointment of qualified Commissioners to the LACC’s BOC, including the Chair appointment in November 2018; (3) The BOC is able to push forward the Whistleblower Law for consideration in the legislature. EQ5 Increased Women’s Participation: Include the Law School in an annual awareness raising tour of the PMTP: LPAC should continue their successful outreach to raise awareness of the PMTP in underrepresented counties, particularly by women magistrates. Invite a representative of the Law School, even LPAC scholars, and showcase roles for women in improving the rule of law in Liberia. 35 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV VI. UTILIZATION PLAN LSA will facilitate a Learning Event one to two months after the evaluation report has been finalized to engage USAID/Liberia, the IPs, GOL, and other relevant stakeholders in the consideration of findings, conclusions, and recommendations. This will be combined with an After-Action Review, prescribed as good learning practice per the Automated Directive Systems (ADS) 201, to support USAID in the incorporation of recommendations into work plans, project or activity design, and the next CDCS. The Learning Event can serve to facilitate a joint working session between USAID and LPAC on finalizing a) key activity definitions, b) key activity outcomes/goals, and c) what activity success will look like (and in turn, how to measure this success). In working one-on-one with each stakeholder, the event can facilitate an internal working group for USAID to redefine/identify what it wants to achieve in the activity given the changes in the implementing environment and GOL constraints. The event could then help facilitate a series of working sessions between LPAC and the institutional partners to work through key recommendations to reach consensus regarding technical assistance needs and skills transfer, define the specific skills to improve, and set up measurable milestones for LPAC’s interventions with institutional partners. This will allow USAID and LPAC to manage expectations among institutional counterparts, while delivering agreed-upon and achievable results for the remainder of the activity. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 36 ANNEX A: FULL LISTING OF REFERENCES AND REPORTS USED Liberia Context Resources. The Security Sector and Rule of Law in West Africa. ROL west_africa_gender_survey.pdf LNBA, Law School, JI. December 2017. Assessment of Court and Case Management Processes, Gaps and Challenges in Liberia – (Assessed October 2017) LPAC. April 2016. LPAC Quarterly Report (Jan – Mar 2016).pdf LPAC. April 2017. LPAC Quarterly Report (Jan – Mar 2017).pdf LPAC. April 2018. LPAC Quarterly Report (Jan – Mar 2018).pdf LPAC. August 2018. Activity Costing.xlsx. Additional Checchi documents LPAC. August 2018. Amended LPAC Budget.xlsx. Additional Checchi documents LPAC. August 2018. Checchi Amended Budget Narrative in Support of the Budget Realignment Request. Additional Checchi documents LPAC. August 2018. Financial Reports and Budgets Amended LPAC Budget.xlsx LPAC. August 2018. LPAC PIDS data.xls LPAC. August 2018. Request for Contract Modification including Budget Alignment (Revised). Additional Checchi documents LPAC. August 2018. Stakeholder Contact List.docx LPAC. December 2015 (Quarter1). Accrual Report. Estimated Expenditures by Line Items LPAC. December 2017 (Quarter1). Accrual Report. Estimated Expenditures by Line Items LPAC. December 2017. LPAC Annual Report (Year 2).pdf LPAC. February 2017. Court Bailiffs’ Capacity and Training Needs Assessment Final Report (2-9-17.docx.) LPAC. Human and Institutional Capacity Development (HICD) Assessment Report. December 2015 LPAC. January 2016. LPAC Quarterly Report (Oct – Dec 2015).pdf LPAC. January 2017. LPAC Quarterly Report (Oct – Dec 2016).pdf LPAC. January 2018. LPAC Quarterly Report (Oct – Dec 2017).pdf LPAC. July 2015. Fully Executed LPAC TO_AID-624-TO-15-00003 LPAC. July 2016. LPAC Quarterly Report (Apr – Jun 2016).pdf LPAC. July 2017. LPAC Quarterly Report (Apr – Jun 2017).pdf LPAC. July 2018. LPAC Quarterly Report (Apr – Jun 2018).pdf LPAC. July 2018. LPAC Draft AMELP LPAC. July 2018. LPAC Organigram. LPAC Organization Chart LPAC. June 2016 (Quarter 3). Accrual Report. Estimated Expenditures by Line Item LPAC. June 2016. Memoranda of Cooperation between USAID/LPAC and JI. Additional Checchi documents LPAC. June 2016. Memoranda of Cooperation between USAID/LPAC and LACC. Additional Checchi documents LPAC. June 2016. Memoranda of Cooperation between USAID/LPAC and Law School. Additional Checchi documents LPAC. June 2016. Memoranda of Cooperation between USAID/LPAC and LNBA. Additional Checchi documents LPAC. June 2017 (Quarter 3). Accrual Report. Estimated Expenditures by Line Item LPAC. June 2018 (Quarter 3). Accrual Report. Estimated Expenditures by Line Item LPAC. March 2016 (Quarter 2). LPAC Accrual Report. Estimated Expenditures by Line Item LPAC. March 2017 (Quarter 2). Accrual Report. Estimated Expenditures by Line Item LPAC. March 2018 (Quarter 2). Accrual Report. Estimated Expenditures by Line Item LPAC. May 2018. LPAC Year Three (3) Work Plan LPAC. October 2015. LPAC Year 1 Work Plan (Inception Report) LPAC. October 2015. REVISED LPAC ME Plan (December 2015 Revised) LPAC. October 2016. LPAC Annual Report (Year 1).pdf LPAC. October 2017. Financial Reports and Budgets LPAC Portfolio Review input Year 3 Activity Table w costs 10 31 2017.xlsx LPAC. October 2018. LPAC Annual Report (Year 3).pdf (DRAFT) LPAC. September 2015. Accrual Report. Estimated Expenditures by Line Item LPAC. September 2016 (Quarter 4). Accrual Report. Estimated Expenditures by Line Item LPAC. September 2016. Year Two (2) Work Plan LPAC. September 2017 (Quarter 4). Accrual Report. Estimated Expenditures by Line Item LPAC. September 2017. LPAC Structuring Organigram. LPAC Organization Chart (Current) LPAC. September 2017. Proposed Reform Outline for USAID Consideration. LPAC. June 2016. Professional Magistrate Training Program Cycle II. Student Baseline Survey Report (surveyed April 2016) LPAC/LACC. Liberia Justice Institute ICT Assessment Report. March 2016.pdf. Reed, Kramer. 20 November 2010. “Liberia: to Become a Middle Income Country by 2030 Growth Must Have A Broad Reach,” https://allafrica.com/stories/201011300115.html. USAID. Liberia CDCS 2013-17.pdf USAID. Liberia Context Resources. ABA Final Report. April 2009. Evaluation of Rule of Law Programs in Liberia.pdf USAID. Liberia Context Resources. April 2010. Liberia Formal and Informal Justice in Liberia CON1222_Accord_23_9.docx. USAID. Liberia Context Resources. September 2015. Liberia Governance and Economic Management Support (USAID/GEMS).pdf World Bank. 2018. Republic of Liberia From Growth to Development: Priorities for Sustainably Reducing Poverty and Achieving Middle-Income Status by 2030. https://openknowledge.worldbank.org/handle/10986/29880. 37 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX B: FULL LISTING OF PERSONS INTERVIEWED The full listing of persons interviewed was submitted separately in line with data de-identification policies. Please contact Carla Trippe, ctrippe@socialimpact.com, to request the data. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 38 ANNEX C: DATA COLLECTION INSTRUMENTS Informed Consent Script Study Title: Legal Professional Development and Anti-Corruption (LPAC) Midterm Performance Evaluation Investigator Funder USAID Introduction of surveyor and his/her affiliation: Hello. My name is _______________. I work with Social Impact. We are conducting some research among selected partners and beneficiaries to ensure that LPAC activities are strategically contributing to the outcomes in the most efficient way. Selection method and invitation to participate: You are invited to participate in this study. You were selected due to your position on or affiliation with the LPAC program. Purpose: We are conducting this research study to examine the appropriateness, performance and sustainability of LPAC’s activities. Procedures: Participation in this study will involve a key informant interview. We anticipate that your involvement will require approximately one hour. Risks and Benefits: People may speculate about what you are talking about or how you responded to the questions. We take confidentiality seriously, and we hold all participant identities confidential. Although this study will not benefit you personally, we hope that our results will add to the knowledge about the program’s efficiency and effectiveness. Confidentiality: All of your responses will be held in confidence/anonymous. Only the researchers involved in this study and those responsible for research oversight will have access to the information you provide. Your responses will be numbered and the code linking your number with your name will be stored in a password protected online server. Voluntary Participation: Participation in this study is completely voluntary. You are free to decline to participate, to end participation at any time for any reason, or to refuse to answer any individual question without penalty [or loss of compensation]. Questions: If you have any questions about this study, you may contact our supervisor in Monrovia, Carla Trippe: +231-888-106151; ctrippe@socialimpact.com; or Leslie Hodel, Member of the Social Impact Ethics Board: +1-240-938-6655; lhodel@socialimpact.com. Agreement to Participate: [if oral consent will be obtained:] Do you have any questions? Do you agree to participate? Enumerator must record consent: 1. Yes 2. No [if written consent will be obtained:] I have read the above information, have had the opportunity to ask any questions about this study and agree to participate in this study. (Printed name) (Date) __________________________________________ 39 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV (Signature) Focus Group Discussion Guiding Questions [Respondents to fill out an attendance sheet that will include demographic questions] Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the USAID/Legal Professional Development and Anti-Corruption (LPAC) Activity, which was implemented by Checchi and Company Consulting, Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you in any way. Your answers will not have any impact on your involvement with the LPAC. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 2 hours. We want all of you to participate in this FGD and we will ask all of you to provide your opinions. We will take notes of our discussion (or record the conversation with their permission). 1. What motivated you to go to law school? 2. What does it mean to you to receive the LPAC Public Service Scholarship? 3. Do you believe you are receiving a quality of education at the Law School? Please explain 4. What are the current challenges affecting or obstacles impacting women’s participation in the justice system? 5. How do you think you will be able to contribute toward strengthening women’s participation in the Liberian justice sector once you graduate from law school? Key Informant Interview-LPAC Title: Name: Date: Sex: Consent Script Interview to discuss consent form and have interviewee sign it. Opener 1. Overall what is one of your team’s greatest success/achievements since project launch and what is your greatest challenge? LPAC Pre-Reform 2. To what extent do you believe USAID and LPAC incorporated the lessons learned from previous rule of law programs to inform the initial Components A and B strategic design in 2015? [EQ: 1] ___ Very Poor ___ Poor ___ Fair ___ Good USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 40 ___ Very Good ___ I do not know Explain: 3. In your view, to what extent do you think that USAID and LPAC’s program leadership at the time took into account the changing political and economic landscape in Liberia in 2015? [EQ: 1] [1] ___ None. [2] ___ Some aspects of the program were adjusted, but we should have adjusted more. [3] ___ Only critical areas of the program were adjusted. [4] ___ We significantly changed the program design. Explain your response: _________________________________________________ 4. Do you believe this contributed to LPAC’s implementation constraints for both Components during its first two years? Yes/No a. If yes, how? Component A Redesign 5. Please describe why, in your perspective, the redesign of Component A was launched. What were the issues that led to the redesign? [EQ: 1] a. Please describe any data/documentation that confirmed these constraints. What evidence was there that verified these issues were challenging the project? 6. Who was driving LPAC’s Component A redesign? [EQ: 1] Who sought out the redesign? 7. To what extent does the re-design of Component A address these constraints that challenged the original design (refer to Q5)? [EQ1] b. Please describe any data/documentation that confirms these constraints were addressed by the redesign. 8. To what extent is the level of data generated by the project sufficient to make informed management decisions today? What additional data is needed to ensure management can make informed decisions? [EQ1] 9. To what extent were the judicial institutions involved in Component A’s re-design? [EQ1] [1] ___ None. They were never consulted in the redesign process. [2] ___ Some but not all, of the justice institutions were involved/consulted. [3] ___ All partner justice institutions were significantly involved/consulted. If option 1 was selected, move ahead to Q9b. a. If they were, to what extent did LPAC incorporate their feedback in Component A’s re￾design? [EQ1] b. If they were not, why were they not involved? 41 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Component B Redesign 10. Please describe why, in your perspective, the redesign of Component B was launched. What were the issues that are leading to its redesign? [EQ: 1] c. Why now as opposed to July 2017 with the Component A redesign? d. Please describe any data/documentation that confirmed these constraints. What evidence was there that verified these issues were challenging the project? 11. Who was driving LPAC’s Component B redesign? [EQ: 1] Who sought out the redesign? 12. To what extent will the proposed Component B’s redesign address these constraints that challenged the original design (as stated in answer to previous question)? [EQ: 1] e. Please describe any data/documentation that confirms these constraints were addressed by the redesign. 13. To what extent was LACC involved in Component B’s proposed re-design? [EQ: 1] [1] ___ None. They were never consulted in the redesign process. [2] ___ Somewhat. [3] ___ All partner justice institutions were significantly involved/consulted. If option 1 was selected, move ahead to Q13b. a. If they were, to what extent did LPAC incorporate their feedback in Component B’s proposed re-design? [EQ: 1] b. If they were not, why were they not involved? Redesign Impacts of Redesign EQ: 1, 2, 4, 5 14. How, if at all, has the redesign impacted the project? 15. I am going to ask you to rank the following aspects of the redesign, using a simple scale to rank/rate the aspect you identified, both before (pre) the reform and after (post) the reform. Second, I will ask you to describe why you selected that rank. In particular, I will want to discuss aspects for which you see a rank/rate change from pre to post reform. For each area, discuss pre and post attributes using the 1-4 Likert scale (1=poor and 4=very good). Fill in table below for each respondent. 1=Poor 2=Fair 3=Good 4=Very Good 0=Not Relevant Pre Reform Post Reform a. Pace of achievement/progress per objective Rank: Comments: Rank: Comments: b. Indicators/measurement of success Rank: Comments: Rank: Comments: c. Clarity of approach, strategy, and goals Rank: Comments: Rank: Comments: d. Effectiveness of management Rank: Comments: Rank: Comments: e. Partnerships and grants (communication and coordination) Rank: Comments: Rank: Comments: USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 42 f. Your day-to-day work/efficiency Rank: Comments: Rank: Comments: g. GOLs capacity and commitment EQ: 4 Rank: Comments: Rank: Comments: h. Women’s participation/involvement (in project activities; in LPAC staff/teams) EQ:5 Rank: Comments: Rank: Comments: 16. At this time (post reform), what are the most significant: [EQ. 1, 2] f. Contributions to project achievement/success? Please explain. g. Remaining challenges/inhibitors/constraints, if any? Please explain. Defining Project Success 17. How do you know when LPAC overall has succeeded for each component? a. What does success look like in achieving the current objectives of Component A in the absence of any component-level indictors to reflect impact or results? [EQ. 1, 2] b. How is that measured, if at all? [EQ. 1, 2] 18. What is the result measure that will determine achievement toward achieving the objectives for Components A and B? [EQ. 2] Increasing Women Participation 19. How does LPAC encourage institutional partner’s support toward improving the participation of women in the justice sector? [EQ: 5] 20. What indicators are you using to measure women participation? [EQ: 5] 21. Over the last year, how has LPAC contributed towards the recruitment of: [EQ: 5] h. Women law professors to the Law School? i. Women leaders at LNBA? j. Circuit Judges? k. Magistrates? 22. Due to social, personal, and/or family commitments, how does LPAC plan to work with the institutional partners to address these constraints of women’s participation in the justice sector? [EQ: 5] Recommendations 23. Do you have any recommendations for the project that could improve performance in the remaining years? In terms of… a. M&E (indicators, measurement of success/achievement, management of data) b. Management c. Coordination/communication d. Strategy/approach e. other Closing and Questions 24. Do you have anything you want to share that we did not ask you about? 43 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX D: EVALUATION STATEMENT OF WORK LEGAL PROFESSIONAL DEVELOPMENT AND ANTI-CORRUPTION (LPAC) ACTIVITY MIDTERM PERFORMANCE EVALUATION STATEMENT OF WORK 1. Background This Statement of Work (SOW) describes the conditions of work and terms of reference for an external evaluation of USAID’s Legal Professional Development and Anti-Corruption (LPAC) activity. LPAC seeks to promote a more effective and accountable formal justice sector through improved capacity of five legal institutions: the James A. A. Pierre Judicial Institute (Judicial Institute), the Louis Arthur Grimes School of Law (Law School), the Liberia Legal Information Institute (LiberLII), the Liberian National Bar Association (LNBA), and the Liberia Anti-Corruption Commission (LACC). The purpose of this evaluation is to provide an independent and in-depth examination of the overall progress of the LPAC activity in Liberia. The evaluation will explore the appropriateness and sustainability of the redesign30 and identify best practices for the remaining implementation period. The Mission will use findings from the evaluation to inform more strategic USAID investment in the sector. Overview of LPAC Activity: Legal Professional Development and Anti-Corruption Contract Number: AID-624-TO-15-00003 Activity Dates: July 17, 2015 – July 16, 2020 Funding: $20 Million Implementing Partner: Checchi and Company Consulting, Inc. Subcontractors: Management Systems International Contracting Officer’s Representative: Thomas Kanneh Alternate Contracting Officer’s Representative: April O’Neill Results Framework: Annex 1 Geographical focus: Montserrado LPAC is USAID/Liberia's primary formal justice sector program and fits within a greater framework of the Mission’s current rule of law and governance sector activities. LPAC was designed to build on key policy and vision platforms of the Government of Liberia (GOL) in its first and second national Poverty Reduction Strategies (PRS), and its Agenda for Transformation. The activity seeks to promote a more effective and accountable formal justice sector through improved capacity of the legal professional training and regulatory institutions and improve the rule of law. An emphasis is placed on institution capacity building to help ensure sustainability of the targeted institutions. LPAC was tasked with implementing an approach with each of the five institutions so that they can achieve sustainability. Recognizing that the mechanisms and scope for implementation were not producing the anticipated results, LPAC was redesigned in July 2017 to make it leaner, more focused on essential needs, and more cost-effective by right-sizing program design. The new design recognizes the constraints faced by Liberian counterparts and takes a more participative, integrated approach to the implementation model. It also 30 The SOW refers to a “redesign”; however, no actual contract modifications have occurred. After a meeting with the LPAC COR, Checchi, and the Contracting Officer on March 27, 2017, it was determined that the proposed changes in how LPAC is being implemented could be done through technical direction. In addition, the “redesign” was memorialized in a document entitled “LPAC Reform Proposal”, which was submitted by Checchi to the LPAC COR on July 21, 2017. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 44 reduces the number of key expat personnel from five to three and shifted focus to hiring local experts and bringing in international short-term technical assistance when expertise cannot be identified locally. Reforms of the implementation mechanisms are designed to more effectively achieve the objectives of LPAC, recognizing that professional legal development and anti-corruption remain essential, but that the activity be focused on essential steps to build the capacity of counterparts with sustainable activities. At the same time, the level of financial sustainability of counterpart institutions envisioned in the LPAC task order must be deferred until the Liberian economy and state budget situation improves – likely beyond the end date of LPAC. The following components of LPAC are linked to the Results Framework in Annex 1. To link the LPAC results framework to the overall USAID/Liberia objectives, refer to the USAID Liberia results framework in Annex II. The first component directs capacity building to the four key Liberian justice sector institutions that educate, regulate and inform legal professionals. The second component supports the LACC and associated GOL entities in their efforts to investigate, prosecute and convict perpetrators of corruption￾related crimes, and to educate the public regarding the fight against corruption in Liberia. • Component A supports human and institutional capacity development (HICD) at four legal educational, information and professional institutions—the Judicial Institute, the Law School, LiberLII, and the LNBA—with the ultimate aim of making those institutions financially and administratively sustainable and able to operate effectively with minimal international donor support. Component A supports direct technical, managerial and financial-capacity building along with formal training with the ultimate purpose of fostering a robust, well-trained and competent legal profession. • Component B supports the Government of Liberia to increase capacity to prevent, investigate, and prosecute corruption, including within the justice sector, and to educate the public regarding the fight against corruption in Liberia. Under this component, LPAC extends technical assistance, capacity development and support to LACC and also supports the Ministry of Justice (MOJ) with respect to coordination with LACC, civil society and media organizations engaged in anti￾corruption initiatives. LPAC also supports the LACC to enhance its organizational structures and systems promoting education, outreach and prevention activities and assists the LACC in its anticorruption leadership and coordination roles. 2. Purpose of Evaluation HICD interventions aim to transform more structural, deeply rooted challenges. Even with strong theories of change underpinning design, these types of interventions are slow to show progress. To ensure that LPAC’s activities under the recent redesign are strategically contributing to outcomes in the most effective way, this midterm evaluation will examine the appropriateness, performance, and sustainability of the new design. The evaluation will identify results and lessons learned from implementation and will provide succinct, actionable recommendations to determine which component(s) of LPAC to scale up or modify moving forward or for future procurements providing technical advisory support. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, the implementing partner, and relevant partners. 3. Evaluation Questions 1. To what extent did design and implementation constraints impede decision-making, adaptation, and achievement of results prior to reforming the activity? o How appropriate and adequate were the steps taken in addressing pre-reform constraints? 2. To what extent has progress been made through the redesign in achieving the objectives under LPAC? Consider timeliness and efficiency of results. 3. Considering value for money, what interventions should be kept and which should be discontinued or scaled back? Provide a prioritized list using cost-benefit analysis. 45 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV o Are there alternative options to achieve more effective change, particularly in the area of anti-corruption? o Are there specific lessons learned from LPAC that should be considered as USAID/Liberia prepares its new Country Development Cooperation Strategy? 4. What is the extent of the GOL’s commitment and capacity to sustain Components A and B, both financial and administrative? o Are there specific benchmarks for GOL actions that should be considered as USAID contemplates its investments in this sector? 5. To what extent does the redesign increase participation of women in a meaningful way? o In the recommendation, identify what can be done to further increase participation of women in a meaningful way. 4. Evaluation Methods The evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. The evaluation team will conduct a desk review of available literature including activity and project documents. The evaluation team will also look at sectoral assessments that were used to determine interventions. Visits to the relevant partners will provide qualitative data for analysis through methods such as in-depth and key informant interviews, focus groups, and direct observation. The qualitative data collection will be supplemented through quantification of qualitative information, especially through participatory ranking exercises that will provide numeric insight on respondent priorities. The cost-benefit analysis will include quantitative and qualitative data across intervention areas to identify and valuate costs, benefits, consequences, and variable scenarios. It is expected that the evaluation team will use data provided by the implementing partner in regular quarterly and annual reports, performance reporting, assessments, and budgets for most of the quantitative data required. The team will also draw on datasets from the Monitoring and Evaluation Unite at the Judicial Institute. Primary collection of quantitative data and large-scale structured surveys are beyond the scope of this evaluation. The USAID/Liberia DG team will provide documents for the desk review, as well as contact information for prospective interviewees. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include but are not limited to: 1. LPAC Contract/Award 2. LPAC Reform Proposal 3. Annual Work Plans 4. Activity Monitoring and Evaluation Plan 5. Quarterly and annual progress reports 6. Performance Indicator Database System data 7. Financial reports and budgets 8. Data Quality Assessment report 9. Baseline assessment 10. Organizational assessments and tools 11. USAID/Liberia CDCS 12. Judicial Institute datasets 13. Agenda for Transformation 14. Other related national data and reports 5. Deliverables and Timeline Evaluation deliverables include: 1. Evaluation Team Planning Meetings 2. Inbrief with USAID/Liberia, Implementing Partner, and GOL 3. Inception Report with work plan and data collection instruments USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 46 4. Debrief with USAID/Liberia, Implementing Partner, and GOL 5. Draft Evaluation Report - A draft report should be submitted to LSA for review, and LSA must submit the draft report to USAID/Liberia within three weeks after the in-country work is conducted. The written report should clearly describe findings, conclusions, and recommendations and conform to USAID requirements outlined below. USAID will provide comments on the draft report within five working days of submission. Then, LSA will submit the draft report to the Implementing Partner for another five working days to provide comments and disclose a statement of difference, if applicable. 6. Final Report - LSA will submit a final report that incorporates USAID and Implementing Partner comments no later than ten days after final, written comments on the team’s draft report have been submitted by all parties. 7. Learning Event – LSA will facilitate a learning event one to two months after the report has been finalized to engage USAID/Liberia, the Implementing Partner, GOL, and other relevant stakeholders in the utilization of recommendations. This will be combined with an After-Action Review, prescribed as good learning practice per ADS 201, to support USAID in the incorporation of recommendations into work plans, project or activity design, and the next Country Development Coordination Strategy. The draft evaluation report should meet the following criteria: 1. The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide. 2. The report should be no longer than 30 pages, excluding executive summary, table of contents, and annexes. 3. The report should include a 3-5 page Executive Summary highlighting findings and recommendations. 4. The report should represent a thoughtful and well-organized effort to objectively respond to the evaluation questions. 5. The report shall address all evaluation questions included in the SOW. 6. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. 7. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). 8. Evaluation findings should be specific, concise and supported by strong quantitative or qualitative evidence. 9. Recommendations should be action-oriented, practical, specific, and evidence-based. 10. The final report should be edited and formatted. 11. Liberia Strategic Analysis must submit the final evaluation to the Development Experience Clearinghouse. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Liberia five working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy: http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Timeline The evaluation should follow the timeline for producing deliverables outlined in the Gantt chart below. The evaluation is estimated to begin September 2018. 47 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Performance Evaluation Gantt Chart ACTIVITY # OF DAYS WEEK 1-2 WEEK 3-5 WEEK 6-7 WEEK 8 WEEK 9 WEEK 10 WEEK 11 WEEK 12 WEEK 13 WEEK 14-15 WEEK 16 WEEK 17 USAID shares SOW with LSA 1 LSA publishes call for CVs 14 LSA recruits and onboards team members 21 Desk Review 5 Team travels to Liberia 1 Team planning meeting - internal 1 Planning Meetings with USAID and IP 1.5 Submit draft Inception Report to USAID In-brief with USAID, IP, and GOL 1 Prepare for field work 2.5 Submit final Inception Report to USAID Field Work 20 Preliminary analysis 1 Debrief with USAID, IP, and GOL 1 Travel from Liberia 1 Analysis and report drafting 10 LSA quality control and revisions 5 LSA submits Draft Report to USAID USAID and IP review the draft report 10 Evaluation team incorporates comments and feedback 5 LSA quality control 5 LSA submits Final Report to USAID LSA facilitates Learning Event with USAID, IP, and other stakeholders 1 1-2 months after PE Prep work by Team Leader and national consultant 1 USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 48 6. Team Composition The evaluation team will be composed of four individuals: team leader, two technical experts, and an evaluation specialist. 1. Team Leader/Senior Evaluation Specialist: S/he will be responsible for coordinating the overall activities of the evaluation team. The Team Leader will approve the final evaluation design, oversee the development of evaluation instruments, integrate the findings of different team members, and coordinate the preparation of the final reports. The Team Leader should have at least ten years of experience in the administration of rule of law or anti-corruption projects in developing countries. Knowledge of HICD in the legal sector is preferred. S/he should have experience in managing multi-disciplinary teams and developing and conducting qualitative evaluations. It is essential that the candidate has the ability to conceptualize and structure evaluation activities and produce high quality evaluation reports in English. Liberia specific experience is strongly preferred. A post graduate degree in law, public administration, management, research, or related field. 2. Economist: S/he should be an economist with at least 8 years of experience working with or evaluating rule of law or anti-corruption activities in developing countries. S/he must have demonstrated experience in the design of cost-benefit analysis, methodology determinations, and data collection instruments. Analysis will include budgets, costing by outputs, and sector report data. The candidate must have exceptional organizational, analytical, writing, and presentation skills. Knowledge of USAID rules, regulations, and procedures in this sector is highly desirable. S/he will contribute to the overall drafting of the evaluation framework and participate in the desk study, interviews, and other data collection; and analyze the data with input from team members and USAID/Liberia to draft the evaluation report. The candidate will hold an advanced degree in economics, law, public administration, or related field. 3. Rule of Law Specialist: The subject matter specialist shall have experience and knowledge about the political context in Liberia, particularly on rule of law, anti-corruption, and governance. Must have strong social analytical skills, specifically in designing and evaluating similar HICD programs. The incumbent must be able to conduct interviews and focus group discussion and analyze the resulting data. Strong American English language speaking and writing skills are required. 4. Logistics Coordinator: The candidate should have at least five years relevant experience in logistics planning for large evaluations or research studies. S/he must have experience in fleet management, scheduling meeting, setting up appointments, and coordinating team travels. The candidate must also have experience in qualitative data collection methods to support interviews as necessary. S/he will serve as the main logistical coordinator for the performance evaluation, and work with local partners to plan travel, data collection, interviews and assessment activities as required. Collectively the team members must have experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in Liberia, all team members are required to familiarize themselves with USAID’s Evaluation Policy, with USAID’s publication outlining a good evaluation report, and with USAID’s checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic, policy and educational context in Liberia. USAID may propose internal staff from USAID/Liberia or from Washington to accompany the team in this evaluation as observers. As observers, their role will be to provide, when asked, background information and to reply to the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. 7. Logistics and Level of Effort 49 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV The evaluation team is anticipated to visit and conduct consultations and data collection visits primarily in the greater Monrovia area. USAID and the IP can assist in providing contact information. LSA is responsible for making meeting and logistical arrangements for the evaluation team including hotel, air travel, and local transportation arrangements. Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than September 2018. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. The evaluation should follow the illustrative level of effort given below. TASK/DELIVERABLE ESTIMATED TIME (DAYS) TEAM LEADER INT’L TEAM MEMBER NAT’L TEAM MEMBER Review background documents & preparatory work (offshore) 5 5 3 Travel to Liberia 2 2 0 Team Planning Meetings in Monrovia with USAID/Liberia and IP 2 Inbrief with USAID/Liberia, IP, and GOL and prepare for field work Submit Inception Report 4 Field work: Data collection and ongoing data analysis 20 (including field travel) Debrief presentation with USAID/Liberia, GOL, and IPs 1 Analysis and report drafting 12 8 7 LSA performs quality assurance check and edits draft report 5 LSA submits draft report to USAID/Liberia 1 USAID/Liberia and IP provide comments on draft report 10 Team Leader revises draft report to incorporate comments and submit final report 5 1 0 LSA performs quality assurance check and submits final report 5 LSA facilitates learning event 1 0 2 Total time required 73 Working days for consultants 52 43 39 USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 50 ANNEX E: DISCLOSURE OF ANY CONFLICT OF INTEREST 51 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 52 53 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX F: IDENTIFYING CONSTRAINTS AND BARRIERS TO ENTRY FOR WOMEN The evaluation conducted a brief assessment of the constraints women face in becoming or being involved in improving the rule of law in Liberia. The team conducted 19 key informant interviews (KIIs), including with one judge, three magistrates, two county attorneys, one private lawyer, two local police representatives, two local non-governmental organizations (NGOs), and eight local citizens. Two focus group discussions (FGDs) targeting 18 LPAC Public Service Scholarship recipients (11 men and seven women) involved questions relating to constraints and barriers women face in studying or practicing law. In addition, the evaluation team wove questions informally into activity-specific KIIs with LPAC staff and institutional partners as the conversation allowed. FINDINGS (1) Women in Law School/Judicial Institute Law School: FGD participants identified three key factors that led them to attend law school. First, as men have historically dominated the legal and judicial sectors, the female FGD participants stated they wanted to join to increase female representation. Second, participants wanted to increase their knowledge to address some of the social injustices faced by women and girls in Liberia, such as sexual and gender-based violence (SGBV). Third, women attended law school because they wanted to be able to provide legal advice to people in their communities. All female FGD participants felt that although they are expected to perform and achieve at the same level as men, women have more responsibilities at home than their male counterparts. They have more family pressure and are considered caregivers for their spouses, children, and parents. This view was shared by half of the male discussants, while the rest thought that male law students experience more challenge because they need to work and take care of the family while at the same time attending classes. FGD participants noted it is difficult to find a balance with home life, spousal relationships, and school responsibilities. Three of the female participants who are married shared that dedicating time and attention to study put a strain on their relationships and introduced tension in the home. In addition, they noted that group study sessions often strained on their relationships with their husbands as well because their husbands would accuse them of having intimate relationships with other male students. Both male and female law school students said that female law school students are more financially challenged than their male counterparts, which serves as a barrier to enrolling and staying in law school. Of note, female FGD participants said that when their friends and family observe them excelling in law school, it is encouraging to their female peers and other young women. Judicial Institute: According to eight interview participants, the awareness that there are educational and career options for women in law and the judiciary is low. According to an Associate Magistrate, “Women should be aware that there is a Judicial Institute and Law School in Liberia. Both institutes should create awareness and encourage the enrollment of more women in the institution.” (2) Perceptions of Capacity and Ability All FGD and interview participants shared that women lawyers cannot provide adequate representation for their clients. This corresponds to the general belief noted by some of the male lawyers and judges interviewed, who indicated that the issue of more women entering the profession is an integral part of a 54 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV larger societal problem that discriminates against women. “This misconception sends a message that the law profession is for men,” according to on male FGD participant. Although there is no consensus over whether barriers or constraints exist for women beyond economic constraints, there is general consensus that women need to be more forceful, have a strong presence, and have thicker skin to be a lawyer. According to an attorney, “There are no impediments for women developing in the legal profession. They need to be more forceful and that would reduce the perception that women cannot represent their clients well.” However, one Stipendiary Magistrate has a different view: “Women do not like to read, they like to sit idle and if you cannot sit down long hours to read and keep your mind working, you cannot be a good lawyer.” He further noted, “Women like to sleep for long hours and a lawyer only sleeps for three hours.” He did not suggest anything to be done to encourage women but he kept emphasizing that women need to take up the challenge just as the men. The Stipendiary Magistrate also noted that women cannot handle the rigors of law school and the legal profession: “The legal profession is a torture chamber, women cannot stand the embarrassment, the way instructors talk to you or your colleague will disgrace you or talk to you in the public. Most women cannot stand it. This has also made other women to leave the profession … Torturing, tension and intimidations are all part of the profession so even if you have the money or the passion, but you cannot stand the tension, you cannot be a lawyer.” However, one Associate Magistrate, underscores what she believes to be the key point in the barrier to women’s participation in the rule of law in Liberia: the perception of men towards women and their lack of desire to have women competing against them. They use old social customs and stereotypes to prevent women from entering the profession. For example, she noted that men tell others that “when a woman becomes a lawyer, she cannot get married” to discourage women and families. She noted, “Men want to be in the front. When they see a woman who has the potential and great prospect, they will do all in their powers to bring her down.” She further noted that violence against women also helps kill a woman self-esteem. (3) Perceptions of Pre-Selected Career Paths All female FGD participants shared their frustration that it is perceived that women can only aspire to be advocacy lawyers and not criminal or corporate lawyers. According to one participant, “Society places female lawyers in a box and expects them to work as advocates in the Association of Female Lawyers of Liberia, advocating for the rights of women and girls.” Over half the KII participants noted that women lawyers tend to opt to engage in alternate law practices, such as advocacy or seeking administrative positions in government or the private sector because they make more money compared to being a judge or courtroom lawyer. According to an LNBA’s Chairwoman, she “sees a lot of women as magistrates, and an increasing number of female law students. However, not a lot of women practice law in the courtroom. They are involved in the justice sector, just not at the frontlines in the courtrooms. They do more mediation, research, corporate, etc., and less on the litigation side.” There are also perceptions that women are weak and chose different career paths because the legal profession is intimidating. According to a Stipendiary Magistrate, “If you are weak, you cannot venture and most women tend to choose different career over the Law Profession.” (4) Intimidation and Bullying All female FGD participants experienced bullying by male law school students and male professors. There is no consensus on why more women are not litigating cases in the court. Two male magistrates and one female lawyer noted that women feel intimidated to perform in the courtroom, thought two USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 55 female magistrates, two female women lawyers (professor and county attorney) disagreed with this view. According to the LNBA’s Legal Aid Chairwoman, the atmosphere is “challenging. Women are scared to go to court. The court has an intimidating atmosphere.” She explained it is not the judges, who she felt were actually very encouraging, but the other lawyers and the male-dominated atmosphere that make it an unwelcoming place for female lawyers. She said, “The men feel the court is for men, and not for women. They try to intimidate the women.” According to a male lawyer the team interviewed, “If he saw a female lawyer in the courtroom, he purposefully lowers his voice to intimidate her. “Older male lawyers use the elder excuse to influence how female lawyers present their cases. Women must be strong and resilient if want to litigate a case … [I]t is not for the faint of heart,” according to the LNBA’s Chairwomen of the legal aid clinic program. (5) Access to Justice and the Need for Female Representation in the Court Fair Hearing/Access to Justice: Of the 37 people interviewed, 21 participants noted that women do not receive a fair hearing, particularly on domestic or SGBV issues. One female law professor noted there is “very little involvement of women” in the rule of law sector in Liberia. “More women need to become magistrates because at the courts, the burden is on women to prove their case when they file a complaint.” By adding that female presence at the court, an Associate Magistrate said it would encourage more women to bring their cases to the court. “Women are at a disadvantage when they bring [domestic violence] cases [before a male magistrate or judge]. After being abused by a man, it discourages them. But if women are [the sitting magistrate or judge], women will bring their cases.” A community justice actor and mediator noted that it is difficult for women to have access to justice in her community because of the lack of female presence in the courts. She said, “If I go to court to seek justice on a domestic violence issue, and [the judge or magistrate is] a man, I will not be happy because, I know that I will not get the justice I want.” She explained that men believe it is culturally acceptable to beat their wives and that women should not take their husbands to court for domestic violence. “But if we know that women are [adjudicating cases] in the court, [more women] will take their cases there and will be confident of getting justice.” Although the demand is high for increasing the female magistrates and judges in the rural areas, some women do not want to take up assignments in their counties of origin, according to one female law professor. Most women come from a poverty background. Therefore, “for these women to relocate back to their county of origin to serve as magistrates or judges, the salaries must be attractive,” the law professor added. CONCLUSIONS Although being a LPAC Public Service Scholarship recipient has relieved the financial pressures for female law students to attend law school, attempts to find the balance between school and family obligations has impacted or will have the potential to impact female students’ academic success in the long run. It also reflects a potential lack of support within the family to the women’s career path, which can impact her future potential in the rule of law sector. The fact that there is no consensus over whether barriers or constraints exist for women beyond economic constraints reflects an overall lack of understanding of or unwillingness to identify the underlying social biases and discrimination that women face in Liberia. The inferences that women are lazy, lack the fortitude to be a lawyer, and will break under pressure all underscore the inherent social bias and discrimination of women. The contention that the perception of men towards women is the key barrier for women’s participation is only part of the broader constraint. The fact that men use “marriageability” as way to influence decision- 56 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV making on whether to attend law school or the PMTP, and pursue a career as a lawyer, magistrate or judge, is insidious, as it pushes the candidate against social traditions and pits the female candidate against family wishes for her to get married. It is clear, according to the analysis: the more women who become magistrates and judges, the more women will use the court system to address sensitive issues, such as domestic violence. RECOMMENDATIONS  Host a roundtable discussion on gender constraints in improving the rule of law Host a roundtable discussion among female leaders, such as Chief Judge Eva Mappy-Brown and Cllr. Joyce Woods of the LNBA, to review the findings and conclusions outlined in this report and discuss how LPAC activities can address them.  Increase capacity and presence of women as judges, magistrate, and mediators Increase presence of female lawyers in the outlying areas by encouraging women to provide pro bono services to the LNBA’s legal aid clinics and in supporting women judges, prosecutors and public defenders to be part of the proposed roaming courts under CDCS. LPAC should support LNBA’s efforts to improve capacity of women mediators through ongoing training and mentorship of clinic staff.  Sensitize male judges, magistrates, and lawyers LPAC should develop curriculum and supporting modules for the PMTP cycles and for the LNBA’s CLE program that helps sensitize male judges, magistrates, and lawyers.  Support a junior mentorship program within the judiciary According to the Associate Magistrate for Gbarnga, she is currently mentoring seven young women to become judicial actors. She said that if each magistrate and judge can mentor a small group of young women, a mindset for young women to have self-confidence and to build their self-esteem is created. Effectively, women who are in a rule of law position should mentor other women.  Raise awareness on career options/paths for women within the rule of law sector in Liberia The GOL should support efforts to create awareness beginning with high schools on career paths in the rule of law sector in Liberia by holding an annual Career Day in each county to raise awareness of the different types of opportunities available for both women and men. Each career area should be supported by at least one female and one male who work in the sector to reflect a gender balance in the messaging.  Raise awareness on gender-rights issues, such as domestic and gender-based violence issues, family law and property rights issues LPAC should re-launch the Talking Bus tour to tackle sensitive gender issues, such as domestic and gender-based violence. The Talking Bus tour could be coupled with community theater skits dramatize what happens at the court so that people can be aware of the legal process and what to expect. LPAC should host radio talk shows that have legal and social welfare specialists speak about gender issues and link the program to the LNBA’s legal aid clinics for one-on-one legal consultation on domestic or violence issues.  Increase women’s enrollment in the Law School and the Judicial Institute’s PMTP cycles USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 57 A Law School professor recommended organizing a female litigator’s clubs, where they can observe/study the performance of women lawyers in the courts. A similar club could be developed for female magistrate candidates at the Judicial Institute.  Internship with the Public Defender’s Office and the LNBA’s legal aid clinics An internship in legal institutions could help expose female interns to indigent cases, criminal cases, and courtroom litigation, while helping them learn basic skills, such as representations, document filing, and court procedures. 58 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX G: BRIEF OVERVIEW OF LPAC’S MONITORING AND EVALUATION SYSTEM FINDINGS Challenges pre-reform: LPAC did not have a functioning monitoring and evaluation (M&E) system at the beginning of the activity. Teams were embedded in some institutions, such as the Judicial Institute, but without an effective M&E system, it appeared that the teams were ineffective. The original indicators were unachievable or not clearly defined, and there appear to be almost no M&E-related data from Year 1. LPAC did not have a M&E data-collection procedures manual. There is no evidence that a baseline study was conducted from which to set targets and measure progress, as was required in the task order’s (TO’s) Statement of Work (SOW). There is no evidence to support USAID intervening and demanding an improved M&E framework until the redesign in July 2017, two years after the activity was launched. Activity Redesign Process: LPAC’s M&E team was not fully involved in the activity redesign process. The LPAC M&E unit was not directly involved in the redesign process. They were involved and led the redesign process of the LPAC M&E system, according to an LPAC respondent. Post Reform: Although a vast improvement from the original framework, the proposed Activity Monitoring, Evaluation, and Learning Plan (AMELP) submitted to USAID in September 2018, requires additional improvements such as standardizing the data capture and reporting process and further improving activity indicators to capture outcomes of interventions, for the revised system to become a useful management tool. LPAC’s M&E team is now able gather data in a more structured and formalized manner, e.g., in capturing trainings attendance; however, the reporting still is not fully standardized. There are multiple sources of attendance data, for example, and collection of robust data related to quality and initial benefit of LPAC trainings is not possible for the M&E team due to the level of sensitivity of the training topics and methods. There is an ongoing lack of understanding of M&E’s role on the activity, which leads to a lack of overall inclusion in activity processes. Key LPAC technical and interventions leads lack an overall understanding and/or appreciation of the role of the M&E team. However, LPAC’s leadership fully appreciates and absorbs the data that M&E collects. CONCLUSION The restructuring of the M&E team during the reform was not appropriate for the activity’s needs for comprehensive and reliable M&E data. The evaluation findings note the ineffectiveness of the M&E team did not lie in the ineffectiveness of the team, but in the overall lack of leadership and an M&E structure and system to collect, manage, and assess data. Since the reform, LPAC has only two staff members who collect activity data, but there is no data management and quality protocols and no apparent linkage between the M&E team and the technical teams besides what appears to be a haphazard activity data dump by the technical team to the M&E team. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 59 ANNEX H: COMPARING ACTIVITY COMPONENTS AND RESULTS FRAMEWORK PRE-/POST-REDESIGN Proposed revisions to the scope of the activity components: the following language changes to the scope of the components make it reflective of the proposed scope in the redesign:  Take out the wording relating to “financially and administratively sustainable” and revise to reflect the inclusion of administrative and financial capacity only as relevant to the institutional partner. Component A supports human and institutional capacity development (HICD) at four legal educational, information and professional institutions—the Judicial Institute, the Law School, LiberLII, and the LNBA—with the ultimate aim of fostering a robust, well-trained and competent legal profession through direct technical and managerial capacity building along with formal training and, as relevant, administrative and financial-capacity building.  Change language in Component B to reflect the provision of technical assistance across GOL anti-graft agencies, as opposed to focusing solely on LACC, and limit the scope of this multi￾agency assistance solely to technical assistance, capacity building and limited support to encourage cross-collaboration and communication. Component B supports the Government of Liberia to increase capacity to prevent, investigate, and prosecute corruption, including within the justice sector, and to educate the public regarding the fight against corruption in Liberia. Under this component, LPAC extends technical assistance, capacity development and limited support to LACC and other GOL anti-graft agencies to improve investigation and prosecution of public corruption and encourage cross-collaboration and communication. LPAC also supports the LACC to enhance its organizational structures and systems promoting education, outreach and prevention activities and assists the LACC in its anticorruption leadership and coordination roles. CHANGES IN RESULTS FRAMEWORK Arising from the redesign, the IP significantly clarified the activity’s results framework to reflect a strategic framework the activity intends to implement. However, the results framework continues to lack the level of detail required to assess progress toward achievement. 60 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Figure 2: LPAC Results Framework – December 2015 USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 61 Figure 3: LPAC Revised Results Framework – Draft AMELP July 2018 GOAL: Strengthened Liberian Institutions Positioned to Drive Inclusive Economic Growth and Poverty Reduction CDCS Development Objective 1 Development Objective 1 More Effective, Accountable, and Inclusive Governance Sub IR 1.2.1: Increase capacity of Legal Professional Training Institutions Sub IR 1.1.2: Increased Public and Civil Society capacity to reduce corruption IR 1.2 Improved Policies, Models and Providers Increase Access to Justice Objective A.1: Improved effectiveness and sustainability of the Judicial Institute Objective A.2: Improved effectiveness and sustainability of the Law School Objective A.3: Improved effectiveness and sustainability of the LiberLII Objective A.4: Improved effectiveness and sustainability of the LNBA Objective B: LACC leads Liberia’s anti-corruption efforts with ` B5-1: : Corruption prevention systems implemented in other GOL agencies with LACC assistance A1-1: Number of specified documents or instruments bought into force to strengthen the operation and autonomy of the Judicial A1-2: Number of courses developed, revised and delivered to PMTP candidates with LPAC support. A1-3: Number of PMTP Students whose grade points improved from Quarter 1- 4 DR 1.3-1: Number of judges and judicial personnel trained with USG assistance DR:2.4-3 Number of people affiliated with non￾governmental organizations receiving USG-supported anti￾corruption training A2-1: Number of clinical legal education program activities conducted as a result of LPAC assistance A2-2: Number of LPAC Supported Law Scholars maintaining 16 credits hours per semester and 2.7 grade point average (GPA) A2-3: Number of courses developed and delivered with LPAC support A2-4: Number of public interest law scholarships awarded by LPAC A3-1: Percentage of legal professional satisfaction of LiberLII website A3-2: Number of visits to LiberLII website A4-1:Percentage of LNBA members attended CLE Training A4-2: Number of CLE modules developed by LNBA as a result of LPAC assistance members who participate in pro bono services as a result of LPAC assistance A4-3: Number of LNBA members who participate in pro bono service as the result of LPAC assistance. A4-4: Number of clients received pro-bono services as a result of LPAC assistance B-1: Number of public complaints made to LACC B2-1: Number of LACC referrals sent to MOJ B2-2: Number of community initiatives as a consequence of LACC outreach and training events B4-1: Number of corruption-related complaints LACC staff received DR:2.4-1: Number of government officials receiving USG-supported anti-corruption training B5-1: : Corruption prevention systems implemented in other GOL agencies with LACC assistance A2-5: Percent of LPAC Law Scholars who obtained 3.00 and above grade point average 62 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX I: EXPANDED EQ1 FINDINGS Design Constraints: There were several constraints to activity progress in the original design, five of which include: (1) lack of a clear purpose and results language; (2) the focus on administrative and financial sustainability in GOL counterpart institutions; (3) lack of co-creation; (4) tying LPAC success to GOL Institutions implementing against their mandate and changing institutional structures and systems without additional assistance; and (5) pinpointing LACC as the primary and leading counterpart to combat public corruption in Liberia. There is a lack of evidence of USAID’s co-design with GOL, which in many ways caused the other constraints to the design. The design flaws and performance expectations were then codified in an inflexible contracting mechanism that did not allow USAID or the IP the ability to adjust mid-course as necessary. Each of these constraints are detailed below.  Design Constraint #1: The original Strategic Framework outlined in the TO’s SOW failed to tie in Components A and B logically and clearly, with no distinct reflection of how success or failure in one component would contribute to the success or failure in the other. In addition, there is no evidence to support approval in the change to the activity’s aim during the Inception Phase, which now reads as follows: To promote a more effective and accountable formal justice sector through improved capacity of legal professional training institutions and regulatory institutions and improve the rule of law.31  Design Constraint #2: The original design was based on assumptions that did not hold true regarding the political, economic, and social environment in Liberia, which led to a poorly defined strategic outlook for the activity. Significant political, economic, and social shifts took place from when the original design was developed (around 2012) to when the contract was signed (in July 2015). Despite evidence of a shift in GOL political and budgetary priorities and the changing economic landscape, there is no evidence that a renewed context analysis was conducted by USAID to check the original activity assumptions.32 In addition, according to one USAID respondent, “There is a design flaw regarding increased [financial and administrative] capacity, which relied on the assumption that improved capacity through better systems and financial and administrative management would lead to better provision of services.”33 In addition to the lack of a revised context analysis, there is also no evidence that USAID conducted a gender or youth assessment as input to the LPAC design. In addition, M&E requirements and oversight, as noted below and in EQ5, were not robust to ensure quality gender and youth data were gathered and recorded properly. LPAC also failed to weave gender and youth considerations into its strategic implementation approach, as was required in the TO’s SOW, although some interventions (such as the LPAC Public Service Scholarships under Component A.2 and anti-corruption campaign activities under Component B) and LPAC’s grants (such as the LNBA’s legal aid clinic grant) supported both gender and youth issues. Evaluation findings, discussed in more detail in EQ5 and Annex F, reflect an inherent bias in Liberia against women’s involvement in the rule of law. Courtrooms are considered men’s domain, and women are often scared or intimidated to go to court. “Older male lawyers use their gender and age to influence how female lawyers present their cases. Women must be strong and resilient if want to litigate a case … [I]t is not for the faint of heart,” according to an LNBA respondent. 31 In bold is what has been added in activity documents, and it was not stated in the TO’s original SOW. 32 USAID respondents could not confirm whether any revised study or analysis was conducted as no one reviewed the internal design documents. 33 This design constraint tied into the assumption that the GOL was on the path to becoming a middle-tiered country by 2030, and therefore would have the capacity and mechanisms in place to firmly establish a well-functioning bureaucracy. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 63 There is no LPAC intervention that addresses these fundamental barriers women face in working in the justice sector and accessing justice in Liberia.  Design Constraint #3: The evaluation found that there was no co-creation process used in designing LPAC’s strategic interventions. A critical element of the design was based on GOL’s capacity and commitment to achieving administrative and financial sustainability. The lack of co￾design interactions with GOL partners is reflected in its selection of inappropriate GOL institutional partners in relation to the required aims of the activity. For example, both the Judicial Institute and the Law School are under the responsibility of the Supreme Court and the University of Liberia respectively, who are responsible for all administrative and financial matters. Therefore, the Judicial Institute and the Law School had no authority to sign a MOC with USAID/LPAC that targets the sustainability of these two institutional functions. If GOL had been involved in co-designing the LPAC aims and approach, this misalignment could have easily been identified and avoided. The lack of co-design is also reflected in the fact that the LPAC Inception Report mirrored the original design outlined in the SOW as opposed to results of a collaborative co-design process with GOL counterparts.  Design Constraint #4: USAID’s expectations of the GOL to contribute financial resources or follow through on activities or recommendations were unrealistic given the GOL budget, economic, and political shifts since 2014, and the lack of institutional mandate to take on these changes in some of the GOL institutions. [Additional background on GOL budget constraints is detailed in EQ4.] According to IP and USAID respondents, the GOL institutional partners became frustrated and demoralized during the first 18 months of LPAC because they were not able to implement the recommendations made by LPAC Advisors due to a lack of funding from their institution’s budget.  Design Constraint #5: By selecting LACC as the primary stakeholder to focus on anti￾corruption work in Liberia, USAID/LPAC unintentionally marginalized the other institutions fighting public corruption, such as the MOJ, FIU, LRA, LNP, the Liberian Drug Enforcement Agency, and the PPCC. According to two IP respondents, due to LACC’s poor reputation34 and capacity, these relevant GOL institutions were either reluctant to work with LACC or refused to do so. One LPAC respondent noted, “Other agencies resented the perception of LACC as an anti-corruption leader. LACC does not have a good reputation among the other anti-corruption agencies.” Implementation Constraints: Although the IP’s approach was responsive to the original design in the scope, it was nonetheless based upon a design that was flawed, as described above. While executing the approach, the IP faced constraints, chief among them included: (1) a risky, expat-heavy, and non-integrated LPAC staffing structure; (2) difficulty in managing GOL institutional partners’ expectations; (3) claims of a focus on administrative and financial sustainability, with less emphasis on technical assistance; (4) high turnover of key personnel at LPAC and USAID; and poor M&E plan, systems, data collection, and reporting with minimal results-focused indictors.  Implementation Constraint #1: The LPAC staffing approach was personality-driven, institution￾focused, and expat-heavy – a risky approach given the high turnover of activity leadership and cultural sensitivities and dynamics. Due to original design constraints, the implementation approach was mismatched and unresponsive to GOL institutional needs, capacities, and 34 Allegations of fraud and corruption on LACC’s Board of Commissioners has hurt its reputation and undermined confidence in the institution’s capacity and willingness to fight corruption. In addition, LACC also had a reputation of poor and unreliable investigative procedures and reporting. 64 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV mandates. Respondents from both USAID and LPAC noted that the one expat per Institution approach led to high overhead costs with mixed results. Poor LPAC leadership was noted across USAID and LPAC respondents. Respondents explained that the former LPAC COPs did not manage the Advisors well enough to ensure cross-communication, collaboration, and synergy in activities and technical assistance, leading to a “silo-effect” in activity implementation and communication.  Implementation Constraint #2: Current LPAC and USAID respondents noted strained relationships between LPAC and GOL institutional partners in the first 18 months of the activity. They noted this was due to: (1) poor presentation of LPAC’s objectives, scope, and mandate; (2) poor management of and unrealistic GOL partner expectations; and (3) ineffective technical assistance by advisors. The evaluation team found LPAC and USAID missed a key opportunity to work with its institutional partners to address design constraints during the Inception Phase, and to manage the expectations of what the activity can/could deliver to them.35  Implementation Constraint #3: According to USAID and the current LPAC leadership, before the redesign, LPAC focused on building financial and administrative systems and related capacity building activities for institutional partners, and less on the technical capacity building, skills building, and knowledge transfer. However, according to activity reports, there is a near-even balance between the building financial/administrative systems and building capacity interventions at the Judicial Institute and LiberLII, as LPAC struggled to finalize MOCs with the Law School, LNBA, and LACC, which might explain LPAC’s lack of progress against its Work Plan for Year 1 and partly in Year 2.  Implementation Constraint #4: Two COPs held LPAC’s reins within the first 18 months of the activity. The lack of consistent and effective activity leadership impacted the LPAC’s momentum and progress by not providing it the necessary stewardship required to lead a team of embedded international advisors, identify and mitigate design and implementation constraints, and steer the development of an appropriate activity monitoring, evaluation, and learning plan (AMELP) and communication framework. In addition, turnover at USAID reduced the COPs’ capability to provide consistent guidance and oversight of the activity.  Implementation Constraint #5: Poor M&E structures and systems led to a lack of informed decision-making by the IP and USAID. Overall, the lack of a quality results framework and ToC undermined any effort to develop a performance monitoring system that would inform USAID and LPAC leadership of achievements and critical constraints and flaws in the design and implementation. 35 During startup, the IP and USAID had an opportunity to study the assumptions underlying the original design and address the design flaws. In October 2015, the IP produced an Inception Report outlining its final LPAC approach and first year implementation plan without taking into consideration the apparent context changes and poor assumptions. There is no evidence that an assessment of the operating environment was conducted by either USAID or the IP to test the original assumptions. USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 65 ANNEX J: EXPANDED EQ3 FINDINGS 36 Respondents corroborated LPAC quarterly reporting, noting: “The quality of these events was recognized as exceptional with the direct participation of the Chief Justice, a former Chief Justice, two sitting Supreme Court Associate Justices, the Chief Judge of the Commercial Court, the Deputy Minister of Justice, the President of LNBA and senior lawyers in private practice in the delivery of substantive training.” JUDICIAL INSTITUTE INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) PMTP-3 Good: Improved understanding of the law and their responsibilities as Magistrates. Participants noted they feel better equipped to perform their duty as Magistrates. According to one respondent, LPAC “helped the Judicial Institute achieve both improved scores in gender balance and regional county balance in their PMTP recruitment processes.” $74,250 $467,808 43% Quarterly Public Defenders’ Training Good: Improved capacity of public defenders from all counties in effective representation of poor litigants. All trainees interviewed said that the training improved their courtroom skills, which improved legal defense of indigent criminal defendants.36 The former Chief Justice of the Supreme Court commended the quality of the training delivery and materials, and the LNBA Chair of Legal Aid noted the improved capacity of public defenders in the courtroom. $20,000 Quarterly Judges and Magistrates’ Training Fair: Strengthened capacity of Liberian Judges and Magistrates to adequately respond to the justice needs of citizens in their respective areas of assignments. Although all interviewees rated the training either fair or good, over 30 percent of participants said that the topics covered could be more relevant to the Liberian context. However, LPAC did clarify that the Supreme Court controls the topics of the trainings and who attends. Overall, magistrates are reducing the issue of writ of arrest in favor of writ of summons to offer the defendants the right to be heard. Simple charges such as menacing and disorderly conduct are disposed without formal trials, thereby reducing the number of pre-trial detainees. Senior Supreme Court officials noted that as a result of the trainings, the court is receiving fewer complaints on magistrates’ behavior in the counties. They also noted improved case management is leading to speedy disposition of cases by magistrates. $35,000 66 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV 37 The 2018 retreat occurred after the data collection period ended. The team was unable to obtain direct feedback from the retreat’s participants and beneficiaries. LNBA INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Annual LNBA Convention Good: To increase member knowledge of trending legal issues in Liberia and discuss/vote on LNBA issues and activities. Both legal and non-legal professionals presented research on various legal topics in the last Convention ranging from Land Rights, Jury Management, and Intellectual Property. Members praised the presentations “as very useful to their work.” The last Convention greatly improved the Professional Image of the LNBA. It was hailed as “the most professional LNBA Convention to date” by multiple interviewees. $8,000 $164,108 15% CLE Trainings Good: To increase member knowledge of trending legal issues in Liberia. Beneficiaries said that the quarterly CLE trainings increased their knowledge of technical legal areas. $50,000 LAW SCHOOL INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Public Interest Scholarship Good: Financial support to motivated and disadvantaged law school students. All female FGD participants mentioned that the scholarships assisted them in better managing their school and family obligations. One Law School official noted improved student retention and learning. Another official believed that the scholarship should be merit-based. $8,900 $345,559 32% New Curriculum Good: Provided opportunities to learn about legal issues of interest, such as Children Law. Positive feedback has been received from School officials and law professors on the curriculum’s relevancy and development. $5,000 Law School Retreat Fair: Provided LAGSL professors with the time and venue to meet and further develop the school’s strategy and the approaching year’s workplan and schedule. The 2017 Retreat was attended by approximately 35 faculty members, School administrators, and LPAC advisors.37 The event was well attended, and participants outlined the $10,000 USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 67 priority LAGSL strategic objectives for the next five years. However, this outcome could have been achieved without this intervention. Legal Writing and Adult Teaching Methods Workshops Fair: To develop critical legal writing and research skills, and to improve skills to transfer legal writing skills to law students. LPAC, law professor and law student respondents did not highlight these workshops as impactful. According to LPAC Quarterly Reports, only one workshop was provided in May 2016, which was well attended by 138 law students (118 males; 20 females), but only two professors attended the Adult Teaching Methods workshop, reflecting a low demand by professors. No additional workshops have been provided since. $15,000 LACC INTERVENTION BENEFIT FY 2018 COST SNAPSHOT FY 2018 TOTAL ACTUALS + PROJECTED % OF TOTAL FY 2018 (ACTUALS + PROJECTED) Proceeds of Crime Training Good: Improved investigators technical skills relating to corruption cases. This is evidenced in the investigation team’s capacity to and consistency in applying the new techniques and lessons learned in their ongoing investigations. Improved cohesion among investigators across anti-graft institutions, which is evidenced in the trainees taking the initiative and establishing an informal working group across relevant anti-graft institutions to share information, discuss cases, and improve knowledge to advance ongoing investigations. $8,900 $111,334 10% Advance interview techniques Good: Improved the quality of evidence collected in criminal cases. This is evidenced the ability of the investigation team to differentiate the types of questions and style and manner in which to interview based on the witness or suspect. $20,000 Satellite courts Good: Improved ability to prosecute corruption cases. A successful pilot activity that allowed LACC to prosecute a corruption case outside of Criminal Court C with reasonable contribution from LPAC. $1,679 Conflict of interest training program for public officials Fair: Improved capacity to inform GOL officials and help identify potential Conflicts of Interest. The evaluation found mixed reviews within LACC, as the concept was confusing to them and to the GOL officials they spoke to. However, one LACC respondent said, “We will continue training and awareness raising, especially to help [GOL] stakeholders identify and prevent conflict of interests situations.” $20,000 68 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Outreach activities with schools Good: Improved capacity to raise public awareness and increase demand for GOL to combat public corruption. According to a LACC respondent, “LACC would not have been able to do outreach anticorruption awareness activities to schools and churches and mosques, without the support from LPAC. These education activities are very important to the LACC.” $5,000 Assisting LACC in conducting risk assessments Good: Improved capacity to conduct corruption risk assessments in GOL agencies. This is evidenced in the Prevention Team’s capacity to conduct risk assessments of LACC’s BOC, and at the Ministry of Health and the National Port Authority. $2,000 GRANTS INTERVENTION BENEFIT LOCATION Y1 – Y3 TOTAL GRANTS DISBURSED % OF TOTAL Y1 – Y3 GRANTS DISBURSED LNBA Pro Bono Legal Aid Clinics Good: Improved access to legal services among indigent populations. Opened five legal aid clinics, increasing access to justice for women and indigent populations in both urban and rural areas. Reduced the number of pre￾trial detainees: freed 227 pretrial detainees. Bong, Grand Bassa, Margibi, Bomi and Montserrado Counties $54,370 19% LiberLII Good: Provide operational support to LiberLII. Supported the upkeep and relevancy of LIBER LII’s website and business model by supporting its upkeep and content updates to its website. In Y2, LiberLII’s website had 228,055 visits, with 269 legal documents uploaded – surpassing its target of 200 legal documents. In Y3, visitors to LiberLII’s website increased to 369,984 by Q2, the last known data collected on LiberLII. Documents uploaded were not tracked in Y3. Montserrado County $91,298 32% ABIC Good: Educated communities on the importance of rule of law during Liberia’s 2018 presidential election process. Through the Talking Bus, ABIC was able to educate the public in 15 counties on the Supreme Court process. Advanced radio broadcasts were used to announce upcoming events, while ABIC used the “Talking Bus” as a mechanism to transport legal professionals to hold community forums on a number of relevant issues. ABIC especially targeted women’s participation, but men were also involved. According to one judicial participant, “The Talking Bus was highly effective in reaching rural women to discuss issues important to them – property and family rights. It was encouraging to see the level of interest in learning about their rights and to help women understand what their rights are… I really hope LPAC continues this and expands its reach.” Montserrado, Grand Bassa, Nimba, Lofa, Grand Gedeh, Margibi, Mary Land, Grand Cape, and Bomi Counties $39,287 14% CENTAL Good: Examined the issues around the elections tied to anti￾corruption and integrity and encourage citizens to hold public officials accountable. CENTAL took over the implementation from Liberia Media Center, and Montserrado, Grand Bassa, and Bong Counties $37,963 13% USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 69 successfully engaged communities through community forums and radio programs and targeted women’s economic groups. Flomo Theater Good: To enhance local expertise and knowledge to fight corruption at the local level. Flomo engaged communities through radio drama and skits, live community theater performances, and community forums. According to the respondents, there was never a woman as town chief in Kona. Under this grant, Flomo Theater discussed female involvement in leadership, which resulted in a woman later being appointed quarter chief running the market. Respondents also noted an increase in cases going to court – local citizens in the targeted regions are increasingly using the formal justice system. There is also “better confidence between citizens, police, and the court system.” Lofa and Margibi Counties $35,910 12% RHRAP Good: To promote transparency and accountability in governance during Liberia’s national elections. RHRAP conducted a situational analysis, engaged in a communication campaign through radio talk show programs, and developed community action committees, which helped link the targeted communities with LACC. Nimba County $30,815 11% Liberia Media Center Poor: Engaged communities to hold politicians accountable by implementing accountability campaigns during Liberia’s 2018 national elections. The grantee did a poor job implementing the activity and could not meet their milestones, respondents noted. The grant was discontinued, and CENTAL picked up the activity and was allocated the remaining budget for the activity. Montserrado County $11,891 4% 70 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV ANNEX K: LIMITATIONS TO ACCESSING JUSTICE The evaluation collected feedback from respondents in Monrovia, Kakata, and Gbarnga on constraints and limitations to accessing justice while implementing LPAC’s objectives. FINDINGS The evaluation found two critical lessons learned from LPAC that should be considered as USAID prepares its new CDCS: (1) expanding LNBA’s legal aid clinic program in remote or rural areas should be accompanied by increasing access to quality judicial services; and (2) the successful release of pre-trial detainees does not address the underlying cause of the number of pre-trial of detentions in Liberia. Difficulty in accessing justice and legal services in remote or rural areas: The LNBA is addressing the issue of access to legal services, particularly among women and indigent individuals, in its LPAC￾sponsored legal aid clinic program. However, according to a representative from the LNBA, access to justice remains limited in rural and remote areas. Qualified judges, magistrates, and lawyers, especially women,38 are not willing to relocate to these areas, and according to two judges and a representative of the Supreme Court, setting up infrastructure is a costly option, particularly as budget resources are already stretched thin. Police are underfunded, and often collect fees to pursue a criminal investigation or civil claim. Legal defense is also sparse in the rural areas. According to a stipendiary magistrate, “There are only two public defenders in Bong County, which is insufficient to cover the nine Magisterial Courts in the county. Two cannot handle it.” A county attorney further noted that public defenders do not follow up on cases of pre-trial detainees. Increasing rates of pre-trial detention: Pre-trial detention was cited as an ongoing issue across the country, both with regard to excessive arrest rates and length of incarceration. There was no consensus on why the orders for pretrial detentions are increasing, citing excessive judicial usage of arrest warrants, excessive bail bond fees, or defendants missing hearing dates due to the inaccessibility of the courts. The LNBA notes that they have seen the judicial authorization of arrest warrants decline since LPAC began. “In the past, judges authorized arrest warrants without due process. There was very little pre-trial conferencing. Plea bargaining has now increased, in addition to the use of writ of summons by judges.” Many respondents in Kakata and Gbarnga noted that the bail bond is too high, and a respondent from FIND attributed the cause to a new bail bond law that was passed in 2013/14. He provided an example that the magistrate can impose a bail bond of $150 for a misdemeanor—well above an affordable level. The lengthy incarceration of pre-trial detainees is blamed on the overloaded docket, as the courts cannot handle the caseload. A FIND representative noted: “Out of the 207 persons in prison, only 87 have been convicted” in Gbarnga. Two respondents noted that there is little transparency in the pre-trial detention process. According to FIND, “Monitoring case flow is a challenge: There is no system or database in place to monitor the case-flow from the police to the courts.” An attorney supported the contention that there is “no tracking system to know case intake.” CONCLUSIONS 38 Annex F addresses ongoing constraints in women’s access to justice and legal services in the rural areas. Most people live far from the courts. Because trials are not speedily done, they become impatient and go back home. Sometimes they do not have enough money to cover the cost of stay longer in Gbarnga. Most times, they abandon the case and they do not follow-up. Community Justice Actor and Mediator, Kakata USAID.GOV LPAC MIDTERM PERFORMANCE EVALUATION | 71 Increasing access to justice for rural populations requires creative solutions to address the human resource and infrastructure gap the judiciary faces, given the GOL’s budgetary constraints. However, it is a critical element in maintaining stability and securing long-term peace after years of civil war. These creative solutions can include improved coordination at the local level between the LNBA’s legal aid clinics and local CSOs who provide similar legal aid services to the local population. Although the administrative fees imposed by the police most likely are necessary for the police to operate, it sends the wrong signal to the population its serves and hinders indigent access to justice. Four key issues contribute to the growing trend of pre-trial detainees: (1) judges and magistrates require further training on the effective adjudication of pre-trial hearings; (2) excessive bail bond fees are being levied against individuals for minor offenses; (3) there is limited accessibility by defendants to the courts; and (4) stretched court dockets and public defender resources contribute to lengthy pre-trial detention, as follow-up on cases is minimal. RECOMMENDATIONS Balancing supply and demand side to accessing justice: Any future programming by USAID should consider striking a balance between the supply and demand sides of justice, including alternative methods to providing judicial services in rural areas. USAID should study the key constraints for rural communities, particularly women and indigent persons, in accessing justice, such as the direct cost of services (fines, time-including delays in resolving cases, and transportation); the structure-related barriers (formality and language, views of justice, and court buildings and personnel); and social barriers (lack of information and perceptions of bias). This study would inform the applicability of options the GOL could consider that would be cost-effective and efficient to reach poor and rural populations. Some respondents suggested establishing a “travelling court” of judges, magistrates, prosecutors, and public defenders who would visit pre-determined locations on a set basis (e.g., quarterly or semi-annual). Address the root cause of pre-trial detentions: USAID can begin the process by conducting a joint review with judiciary counterparts in the Supreme Court of the LPAC assessment of the Court and Case Management System in Liberia from December 2017, in which 17 recommendations highlight key interventions to address the inefficiencies in the current system that contribute to lengthy pre-trial detentions. In addition, this joint session should also address findings relating to the application of bail bonds and other administrative fees that exceed the affordability of most of the accused and the access and reliability of quality legal defense. From this session, USAID and the GOL would identify key constraints and gaps, and jointly identify short- and medium-term solutions to address them over the next five years. In addition, the following recommendations speak to the findings and conclusions presented in the main body of the report regarding access to justice for consideration in CDCS design: Reducing gaps in the delivery of law school scholarships: In developing USAID’s support to promoting the rule of law in the new CDCS for Liberia, USAID should continue LPAC’s success in supporting Public Service Scholarships for law students. If USAID does include supporting Public Service Scholarships in its CDCS and launches a competitive bidding process to implement a follow-on rule of law project to begin in 2020, USAID should include the transfer of all remaining scholarships to the new program once it is operational, thereby allowing LPAC to continue supporting promising law students with Public Service Scholarships in Y4 and Y5. This would also allow LPAC to expand the number of scholarships, as budget and capacity allows. Improving common knowledge of new laws and Supreme Court opinions: USAID should support a communication mechanism where all judicial actors come together to learn and discuss the new laws and Supreme Court decisions. These mechanisms should be held on a regular basis (e.g., semi-annually) to ensure these forums are practical, useful, and cost-effective for both participants and sponsors. 72 | LPAC MIDTERM PERFORMANCE EVALUATION USAID.GOV Establish a legal aid grants initiative targeting local CSOs: Given the limited access of rural populations to legal aid services, USAID should establish a legal aid grants initiative to build the capacity of local CSOs on legal support services, such as local mediation and legal awareness, that compliments the LNBA’s legal aid clinic services. This initiative can be coordinated with the LNBA to ensure synergy and buy-in to the process. Increase prevention activities: In its CDCS, USAID should make prevention a priority for its anti-corruption strategic approach, along with its continued support of improving investigative capacity among the anti￾graft agencies. Prevention activities should continue technical assistance and support toward building capacity and capability to conduct institutional risk assessments. All risk assessments should have an oversight plan to assess the institution’s achievements in implementing the recommendations in the risk assessment report and grade institutions based on their application of these recommendations. This would provide USAID a guide to the government’s willingness and capacity to commit to essential reforms. To complement the risk assessments, USAID should underscore the importance of raising awareness of conflicts of interest by continuing to support training and awareness raising within LACC and among government institutions of conflicts of interest and asset declarations. Raise the capacity of CSOs to build public support toward the fight against corruption: In its CDCS, USAID should highlight the need for greater civil society involvement in the fight against public corruption through greater citizen awareness of and demand for public sector accountability and transparency. USAID should support a grant mechanism that builds CSO capacity for research and analysis, engaging new voices and constituencies, policy advocacy, and public education and awareness raising. Grant concepts could include training workshops for civic activists on transparency and accountability tools and mechanisms; improving voter education, election observation, and effective coordination among CSOs and election officials; and organizing seminars bringing together trainees, government officials, LACC, and other anti-graft agencies to discuss leading and emerging transparent and accountable governance issues in Liberia. U.S. Agency for International Development – Liberia 502 Benson Street Monrovia, Liberia