i MID-TERM EVALUATION OF USAID/MEXICO PROMOTION OF JUSTICE PROJECT EVALUATION REPORT SUBMITTED: MARCH 7, 2018 This document was produced for review by the United States Agency for International Development Mexico Mission (USAID/Mexico) by Linn Hammergren, Diego Gerardo Bolívar, Renato Busquets, and Ernesto Portillo Vargas. MID-TERM EVALUATION OF USAID/MEXICO PROMOTION OF JUSTICE PROJECT EVALUATION REPORT Submission Date: 9/25/2018 (revised final draft) Prepared for the United States Agency for International Development Under AID-OAA-I-15-00024/AID-519-TO-16-00002 USAID/El Salvador Contact: Orlando Hidalgo ohidalgo@usaid.gov USAID/El Salvador Monitoring, Evaluation, and Learning Initiative Contact: Randal Thompson rthompson@engl.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY................................................................................................... i 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS............................ 1 1.1 EVALUATION PURPOSE............................................................................................................. 1 1.2 EVALUATION QUESTIONS....................................................................................................... 1 2.0 PROJECT BACKGROUND........................................................................................ 2 2.1 SETTING THE CONTEXT FOR PROJUST: THE LATIN AMERICAN CRIMINAL JUSTICE REFORMS ........................................................................................................................ 2 2.2 PROJUST: PURPOSE/OBJECTIVE, ORGANIZATION, AND DEVELOPMENT............. 4 3.0 EVALUATION METHODS AND LIMITATIONS................................................... 7 3.1 EVALUATION METHODS .......................................................................................................... 7 3.2 EVALUATION LIMITATIONS .................................................................................................... 8 4.0 FINDINGS AND CONCLUSIONS .........................................................................11 4.1 EFFECTIVENESS ........................................................................................................................... 11 4.1.1 To what extent has PROJUST technical assistance contributed to results, as measured by approved indicators and Qualitative Data?............................................................. 11 4.1.2 Use of PROJUST-Supported Models, Protocols, and Standard Operating Procedures .............................................................................................................................................. 24 4.1.3 Conclusions ............................................................................................................................ 27 4.2 SUSTAINABILITY ........................................................................................................................ 32 4.2.1 What has PROJUST done to promote local ownership (i.e., that the priorities are locally owned, locally resourced, and using rather than supplanting local systems to sustain results? 32 4.2.2 PROJUST promotion of sustainability.............................................................................. 34 4.2.3 What additional actors should PROJUST engage with to promote sustainability? 38 4.2.4 Conclusions ............................................................................................................................ 38 5.0 LESSONS LEARNED................................................................................................41 5.1 EFFECTIVENESS ........................................................................................................................... 41 5.2 SUSTAINABILITY ........................................................................................................................ 45 6.0 RECOMMENDATIONS............................................................................................46 6.1 EFFECTIVENESS ........................................................................................................................... 46 6.2 SUSTAINABILITY ........................................................................................................................ 48 ANNEXES ........................................................................................................................51 ANNEX 1: SCOPE OF WORK FOR THE EVALUATION ............................................................. 52 ANNEX 2: INCEPTION REPORT FOR THE EVALUATION........................................................ 82 ANNEX 3: QUESTIONNAIRE FOR THE SURVEY........................................................................ 108 ANNEX 4: LIST OF INTERVIEWEES ................................................................................................. 115 ANNEX 5: LIST OF PROJECT DOCUMENTS CONSULTED .................................................... 131 ANNEX 6: ADDITIONAL TABLES .................................................................................................... 147 ANNEX 7: CONFLICT OF INTEREST STATEMENTS.................................................................. 152 ANNEX 8: LIST OF ANSWERS PROVIDED IN SPANISH TO QUESTION 6A..................... 156 ANNEX 9: STATEMENT OF DIFFERENCES ................................................................................... 162 FIGURES Figure 1: Tres-Pisos (Three Level) Case Prioritization Model.................................................................. 7 TABLES Table 1: Target States (and Cities) Visited by the Evaluation Team..................................................... 8 Table 2: Indicators 2, 4, and 7 in Targeted States.................................................................................. 12 Table 3: Percentage of Criminal Complaints Resolved Through ADR in Selected State (Indicator 5) ...................................................................................................................................................... 15 Table 4: Percentage of Case Dispositions Reached in Targeted States (Indicator 6).................... 17 Table 5: Pretrial Detention Rates (Indicator 1) and Compliance with Pretrial and Conditional Release Provisions (Indicator 3)................................................................................................ 20 Table 6: Most Important PROJUST Activities by Institution ............................................................... 24 Table 7: Most Important PROJUST Activities by State......................................................................... 24 Table 8: Reported Impacts of PROJUST Activities on Individual Performance............................... 25 Table 9: Perceptions of the Impact of Federal Government Support............................................... 26 Table 10: Answers to Open-ended Question: “Can you specify what your institution is doing to continue the reform on its own (by state, post-PROJUST)?” .......................................... 36 Table 11: Answers to Open-ended Question: “Can you specify what your institution is doing to continue the reform on its own (by institution, post-PROJUST)?”................................. 37 ACRONYMS ADR Alternative Dispute Resolution AG AGO Attorney General Attorney General’s Office CDMX Mexico City CIDAC Center for Research for Development CJM Center for Women’s Justice CONAGO National Confederation of Governors CONAVIM National Commission to Prevent and Eradicate Violence Against Women COPARMEX Mexican Employers’ Confederation CPC Criminal Procedures Code CSO Civil Society Organization ENVIPE ET National Survey of Victimization and Perceptions Evaluation Team FASP FGD FICOSEC FY Contributions Fund for Public Security Focus Group Discussion Trust for Competitiveness and Citizen Security Fiscal Year GOM Government of Mexico INEGI National Institute of Statistics and Geography IPH INL Standardized (homologado) Police Bulletin Bureau of International Narcotics and Law Enforcement Affairs IT Information Technology JASP Justice and Security Support Program LAPOP Latin American Public Opinion Project M&E Monitoring and Evaluation MSI Management Sciences International NCJS New Criminal Justice System NGO Non-Governmental Organization PDO Public Defenders Office PGR Procuraduría General de la República (current name of Mexico’s Federal PPO) PPO Public Prosecutor’s Office PRODERECHO Program to Strengthen Justice in Mexico PROJUST Promotion of Justice Project Q Quarter RRI Rapid Results Institute SETEC Technical Secretariat for the Coordinating Council for the Implementation of the Criminal Justice System SOP Standard Operating Procedure SOW Scope of Work UAME Analysis, Monitoring, and Evaluation Unit UAT “Unidad de Atención Temprana” or Early Attention Unit (intake center) – sometimes called MAT among other names UK United Kingdom UMECA Unit for Precautionary Measures (Pretrial Services) U.S. United States USAID United States Agency for International Development USG United States Government WJP World Justice Project RELATED TERMS Term Description Acta Atenciones Cifra negra Within UATs, registration of misplaced/lost/stolen documents and cell phones for insurance purposes Within UATs, referral to other services or answers to questions not involving a specific complaint “Black number,” the percentage of criminal incidents not reported to the authorities Conditional suspension Deferred prosecution Entidades federativas “Federal entities,” Mexico’s term for the states and CDMX Fiscalía Another term for Public Ministry or Public Prosecutor’s Office Narcomenudeo Sale of small quantities of drugs Procuraduría See Public Ministry Public Ministry In civil law countries, the term for the Public Prosecutor’s Office. Retained in many countries even after a transition to more adversarial codes. Secretariado Executive Secretariat for the National Public Security Program Tres Pisos Three floors or levels, a model introduced by MSI to organized distribution of cases within the Public Ministry or AG) i EXECUTIVE SUMMARY PURPOSE, BACKGROUND, AND EVALUTION QUESTIONS This is the report of the mid-term evaluation of the Promotion of Justice Project (PROJUST). PROJUST is funded by the United States Agency for International Development/Mexico (USAID/Mexico) and supports the Government of Mexico (GOM) in implementing and consolidating a nation-wide comprehensive criminal justice reform. The reform introduces an adversarial, oral procedure, with an emphasis on defendants’ and victims’ rights. PROJUST, implemented by Management Sciences International (MSI) and TetraTech DPK, was approved at $68.2 million and runs from late 2014 to mid-2019. The evaluation assesses PROJUST’s achievements, and provides lessons learned and recommendations for future programming. Throughout Latin America, justice reforms on new criminal procedural codes (CPCs) emphasizing due process rights as a basis have been ongoing since the 1980s. Mexicans pursued these changes for years, culminating in the 2008 constitutional reforms and subsequent enactment of a national CPC that assured meeting a mid-2016 deadline. PROJUST’s objective is to support the “final push toward the 2016…deadline, assisting selected Mexican states to pass the ‘point of no return’ on their…reforms, and support[ing] the [reform’s] consolidation” so as to consolidate the United States Government (USG) and GOM efforts “to mitigate conflict, reduce impunity, increase access, and promote a more transparent and efficient justice system.” As per the Scope of Work (SOW), the evaluation answered three main questions: 1. Effectiveness: To what extent has PROJUST met its objectives relating to the functioning of the Attorney General Offices (AGOs), Courts, Women’s Justice Centers (CJMs), Pretrial Services Units (UMECAs), and the actions of civil society organizations (CSOs) related to the criminal justice reform? 2. Sustainability: To the extent there are results, how sustainable are they in the institutions referenced above? 3. Lessons Learned and Recommendations: What are the lessons learned from PROJUST’s efforts that should be incorporated into a new design? EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation design included document review, analysis of project performance data, interviews and focus group discussions with project stakeholders, and visits to eight target states. The evaluation team (ET) conducted over 250 interviews and 19 focus groups, administered a survey to 150 project-supported justice sector actors, and then compiled and analyzed responses to determine how PROJUST contributed toward improving individual and institutional performance. The primary methodological limitations faced by the ET were limitations in the quality, accuracy, consistency, and comparability of project performance monitoring data. Other limitations included the time and budgetary constraints for preparation, instrument development, pilot testing, and fieldwork; the necessity to rely on MSI and TetraTech DPK for selecting evaluation participants, which could potentially introduce sampling biases; and a lack of a counterfactual that would have allowed the ET to attribute observed changes to project ii activities. In lieu of measuring attribution, the ET looked for evidence about whether and how project activities contributed to observed changes. Below are high-level summaries of the main findings and conclusions, lessons learned, and recommendations from the PROJUST evaluation. More detail is found in the full report. MAIN FINDINGS AND CONCLUSIONS Effectiveness of PROJUST Actions • PROJUST has contributed to important changes in the reception, channeling, and treatment of complaints; treatment of women victims; use of alternative dispute resolution (ADR); programming of hearings; inter-institutional coordination; and case processing. While incremental in nature, these results demonstrate the utility of PROJUST’s approach to reform. • Changes in project performance indicators indicate the project contributed to improved prosecution rates, pretrial services, plea agreements, case disposition rates, and the percentage of cases with sentences exceeding three years—with the caveat that results vary widely across time and state and indicators have several limitations as performance measures. • PROJUST has trained large numbers of justice sector institutions and actors contributing to positive outcomes in terms of increased sector capacity, but the positive effects tend to be more moderate and focused on specialized training relative to general trainings. • PROJUST’s Tres Pisos model has made a significant contribution to how cases are processed at Public Ministries in terms of the reception, filtering, and channeling of cases in the first piso. However, cases channeled to the second piso (ordinary crime) and third piso (high￾impact crimes) tend to stagnate, thus leaving the majority of cases unaddressed. • PROJUST has contributed to the evolution of early attention units (UATS) from simple reception centers to entities that are systematically channeling cases to other offices, albeit with substantial room for improvement in their filtering process. • PROJUST’s Rapid Response exercise contributed to expedited advancement of investigations from Public Ministries to the courts with nearly all target states meeting or exceeding their 100 days goal and all target states processing significantly more cases than before. • PROJUST has contributed to notable improvements in how ADR services function, particularly as measured by the percentage of criminal complaints resolved. Nonetheless, certain functional areas remain weak, including hearing notifications and suspect identification. • PROJUST’s Rapid Results exercise contributed to an acceleration in courts’ hearing schedules and an increase in the number of hearings held and better management of judges’ schedules. However, judicial opposition remains a potential obstacle to further progress in this area. • PROJUST’s support in developing and implementing protocols for conducting preliminary hearings and creating an institutional diagnostic tool to measure institutional progress has iii made limited inroads, in the latter case due in part to a lack of understanding or agreement as to what the assessment criteria are. • PROJUST has contributed to improving the functioning of CJMs as well as to the quality and diversity of their services. CJMs, however, have several limitations, outside of project control, related to limited public access and funding to support infrastructure and operating expenses. • PROJUST contributed a key initial role in developing UMECAs, getting them required and regulated by federal law, and improving their performance. Notwithstanding, the model does not appear to have been based on reliable estimates of demand, caseloads, or budgets. • PROJUST directly contributed to installing Analysis, Monitoring, and Evaluation Units (UAMEs). While these UAMEs continue to operate, they have several systematic weaknesses, including poorly qualified, trained, and motivated staff; limited vision; and a lack of basic statistical analysis skills. • The ET was unable to find evidence as to PROJUST’s contribution to empowering local CSOs to promote justice sector reform. Barriers to greater CSO empowerment include a shortage of CSOs and the existing CSOs’ limited understanding of the criminal justice system. • Most of the models, protocols, and Standard Operating Procedures (SOPs) developed with PROJUST support remain in use and are well-recognized and well-regarded by justice sector actors. However, there is insufficient evidence to conclude that they have had a positive impact on sector actors’ performance. Sustainability of PROJUST Achievements • PROJUST has contributed toward embedding reforms within the justice system. However, for most citizens, these issues are of less interest. To attract their political support, PROJUST will have to incorporate their concerns. • Tres Pisos and Rapid Results proved particularly effective in promoting local ownership and have also contributed to a widening of local ownership to private sector institutions. • PROJUST has been less effective in promoting ownership among state and local governments. Were the federal government to draw down its support to justice system reforms, then it becomes even more critical to cultivate backers in state government. • Extending ownership to a wider portion of the local population has not occurred owing to the public’s limited contact with the system, conflicting interests, distrust of sector institutions, and a limited history of collaboration within civil society. This poses serious risks to the long-term sustainability of the reforms to which PROJUST has contributed. • Obstacles to promoting local ownership include the low priority given to criminal justice reform, poor results achieved when stakeholders get involved, and concerns about the loss of independence among judges. • Several PROJUST-supported justice sector reforms are firmly established and would require a significant counter-reform to reverse. Others are less firmly established and could be iv reversed by a number of factors. Still others remain too new to draw conclusions one way or another. • The evidence is mixed as to whether PROJUST-supported justice sector actors will continue the justice system reform after PROJUST. To the extent sector actors have ideas to continue the reform process, these ideas are inchoate. More work remains for PROJUST to strengthen the institutional capacity of sector actors to continue reforms after PROJUST’s departure. v MAIN LESSONS LEARNED Effectiveness • A reform is not a law; a law is a necessary but insufficient condition for reform implementation. • Training impact hinges on a good definition of needs, how the training is done, adequate coordination with other activities, and systematic evaluation of impacts. • Models, protocols, SOPs, and diagnostic and certification tools can be of little practical utility, especially when they are not based on empirical research and evaluated for their effects. • States and institutions have different capacities for internalizing new ideas and approaches. • The results of changes to complex behaviors, even in pilot units, cannot be reliably assessed in less than five years. • The failure to explore resource requirements for the implementation of new laws and structures hampers the reform process. • Differential case management, prosecutorial discretion, and alterative solutions can be an important sources of change; however, other project or state innovations can be effective. • An emphasis on alternative solutions confronts problems as prosecutors, public and private defenders, and victims and their counsel adapt to their roles. • Inadequate training of the appellate judiciary and federal judges can undercut efforts to combat formalism and promote effective case resolution. • Few institutions can generate reliable statistics on their operations, hampering not only their oversight of their own performance, but also USAID and project efforts to develop indicators. • Although it is easier to improve the work of the courts and public defense, criminal justice reform will never work as intended until police and the Public Ministry improve. • In addressing criminal justice reforms, key actors often think locally rather than globally. Sustainability • All donor-funded projects risk encouraging their clients’ dependence on their continued presence. • Institutional structures and functions created by justice reforms are unlikely to disappear; however, unless they come with increased implementation capacity, they risk stagnating. • It is a challenge to encourage cooperative problem solving among justice institutions, which is made even more difficult by introducing other local actors into the collective deliberations. • Building extra-sectoral support requires appealing to a variety of often conflicting interests. • Sustainability and local ownership both depend on the effective engagement of other parts of state government. vi • Inadequate attention to evidence-based human resource planning negatively impacts sustainability. RECOMMENDATIONS Effectiveness • PROJUST and USAID should reevaluate any remaining work on law drafting, training, and models, protocols, and SOPs against likely impacts. • In any future support to the GOM’s reforms, USAID should insist that adequate analyses of resource needs be conducted, aiding the process as needed. • Some of the largely independent additions by certain state courts and Public Ministries should be tracked to determine their results and, if these are positive, promoted more widely. • Unless it finds states prepared for this approach, PROJUST should stop creating UAMEs because they are beyond the capacity of most institutions/personnel. • In the case of UMECAs, PROJUST should work with local actors and the Secretariado to determine how the essential supervisory function can be better performed. • USAID should turn attention to improving sector institutions’ information technology (IT) systems, especially for case tracking, as well as trying to link better indictors to what they produce. • PROJUST or a future project should work with the National Institute of Statistics and Geography (INEGI) and Secretariado to promote the development of uniform standards for collecting performance statistics. • Regarding impediments posed by appellate and federal judges, PROJUST might work with the former and encourage discussions on eliminating annulments based on procedural errors. • Development of models, protocols, and SOPs should be combined with ongoing or periodic monitoring and reinforcement as a means to ensure both their use and practical utility. Sustainability • PROJUST and USAID should refine their strategy to focus on a set of key outcomes and develop reliable and credible metrics and methods to measure their achievement. • Before introducing more tools and programs, PROJUST should focus on improving what it has already have developed and elaborating plans for its scaling up. • PROJUST should place more attention on building local capacity, improving existing training programs, staff selection, and engaging local governments, civil society, and the private sector. • Now or in a later project, effective means to engage state governments must be found. • USAID in its projects and in discussions with the GOM must begin addressing the costs of full New Criminal Justice System (NCJS) implementation. vii • Any follow-on project must build on and expand PROJUST while recognizing issues of more interest to most Mexicans, including resolution of high-impact and common crimes. 1 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The purpose of the mid-term evaluation of the United States Agency for International Development/Mexico’s (USAID/Mexico’s) Promoting of Justice Project (PROJUST) is to track PROJUST’s results against predefined indicators and targets, assess its sustainability, recommend steps to improve outcomes within the project’s final months, and advise USAID on whether to undertake a new project and, if so, of what type. The evaluation was conducted by an evaluation team (ET) assembled by the USAID/El Salvador’s Monitoring, Evaluation, and Learning Initiative implemented by ME&A. The evaluation covers the period of October 2014 through February 2018 but places greater emphasis on 2016 to the present. The audiences for the report include USAID/Mexico, the PROJUST implementing organizations Management Sciences International (MSI) and TetraTech DPK, and stakeholders ranging from Government of Mexico (GOM) officials to civil society and the private sector. 1.2 EVALUATION QUESTIONS As per the Scope of Work (SOW) (see Annex 1), the evaluation answered the following questions and sub-questions: 1. Effectiveness: To what extent has PROJUST met its objectives relating to the functioning of the Attorney General Offices (AGOs), Courts, Women’s Justice Centers (CJMs), Pretrial Services Units (UMECAs), and the actions of civil society organizations (CSOs) related to the criminal justice reform? a. To what extent has PROJUST technical assistance contributed to results, as measured by approved indicators and qualitative data? b. To what extent are PROJUST-supported models, protocols, and Standard Operating Procedures (SOPs) being used at the state level in the justice sector institutions listed above? If they are being used, to what extent is this attributable to USAID? If they are being used, what aspects of PROJUST’s approach best supported their implementation? If they are not being used, why not? 2. Sustainability: To the extent there are results, how sustainable are these results in the institutions (AGOs, CJMs, UMECAs, and CSOs) referenced above? a. What has PROJUST done to promote local ownership (i.e., that the priorities are locally owned, locally resourced, and using rather than supplanting local systems to sustain results)? b. What has PROJUST done to promote sustainability? Where, if anywhere, could PROJUST go to improve its efforts to promote sustainability? c. What additional actors, if any, should PROJUST engage with to promote sustainability of current or prospective results? 2 3. Lessons Learned and Recommendations: What are the lessons learned from PROJUST’s efforts that should be incorporated into a new design? a. Are there any current areas of support within the programming areas reviewed that should be discontinued? b. Based on the ET’s findings and analysis, are there any other obvious existing or emerging opportunities in Mexico, particularly related to the issue of impunity (defined as the inability of the justice system to investigate, adequately prepare cases for litigation, and adequately present cases before judicial authorities), that USAID should be taking advantage of? 2.0 PROJECT BACKGROUND For over two decades, USAID/Mexico has supported the GOM’s criminal justice reforms and its introduction of more adversarial, oral procedures that incorporate a full recognition of due process rights. Although Mexico is a federal republic, nearly all of USAID’s work has been oriented toward its entidades federativas consisting of the 31 states and the Federal District now called Mexico City, or CDMX. At the federal level, USAID has worked with agencies promoting the reform, leaving the more politically sensitive reorganization of federal justice institutions alone. Within several states (e.g., Baja California, Chihuahua, and Oaxaca), USAID’s assistance under earlier projects helped produce early adoption of new criminal procedures codes (CPCs) and related advances in their full implementation, but it was only in June 2016 that all states graduated to the new system, at least in legal terms. This was accomplished through the adoption of a “uniform, national code” in effect in the federal jurisdiction as well. PROJUST, like USAID’s earlier projects, first focused on legal drafting and mass short-term training. Legal victory, however, did not guarantee operationalization of the code and related laws. For this reason, USAID encouraged MSI and TetraTech DPK to turn their attention to this more momentous task. Discussions began in early 2016, although it was not until March 2017 that a contract modification made the change official and only in October 2017 were the relevant performance indicators approved. Despite these delays, MSI and TetraTech DPK had, by mid-2016, shifted to developing actions aimed at producing results in the operations of sector agencies. 2.1 SETTING THE CONTEXT FOR PROJUST: THE LATIN AMERICAN CRIMINAL JUSTICE REFORMS Similar reforms, based on new procedural codes, have been ongoing in Latin America since the 1980s (Hammergren, 2007). They were inspired by democratic transition in the region, or the passage of nearly all countries in the region from authoritarian to more democratic regimes. The reforms emphasized due process guarantees as well as greater transparency in the laws’ application to counter the traditional use of criminal justice to suppress and punish opposition. Although similar changes were long pursued by Mexican jurists, it was only with the 2008 constitutional reforms that nationwide progress was guaranteed. The reforms stipulated the legal and organizational changes to be in effect by June 2016. While many states moved ahead, the persistence of laggards motivated the 2014 enactment of a single national procedural code, along with complementary legislation, to be universally adopted by the same deadline. 3 In Mexico, initial interest in the change came during a period of relatively lesser violence. This facilitated the acceptance of the due process emphasis. As crime rates surged, however, reaching their peak in 2010-2012,1 it became necessary to stress the code’s positive impact on impunity and security. Simultaneously, CSOs pressed for complementary laws adding, among other things, mechanisms and procedures to combat gender violence, protect victims’ rights, and reduce levels of pretrial detention. None of these affected the principal code’s contents, but they did complicate its implementation, as discussed in later sections. Meanwhile, a series of federal funds were directed at financing aspects of implementation, especially training and infrastructure. From the beginning, the funds required state contributions, but over time states requesting them were also required to adopt various centrally designed models, protocols, and SOPs and, in some instances, CJMs to pursue certification. During its short existence (2008- 2016), the Technical Secretariat for the Coordinating Council for the Implementation of the Criminal Justice System (SETEC) was a major source of funding to the states. Post-2016, funding responsibility shifted to the Executive Secretariat for the National Public Security Program (Secretariado), after which federal funding has been largely limited to police, prosecution, and the entities attached to them (usually pretrial services but occasionally victims’ services and CJMs). These developments, however, posed a number of problems as states attempted to put them into effect: • Continuing lack of agreement as to the code’s purpose, or at least the prioritization of its now multiple objectives of expanding due process guarantees, reducing impunity, providing greater access to more citizens to resolve their criminal complaints, ensuring victim and gender rights, enhancing security, and (least often stressed) a more effective and efficient use of sector resources. • Absence of a criminal justice policy, or even a recognition of its need, at the federal and state level. This policy would help prioritize objectives as well as orient the actions of sector institutions. • Inconsistencies within the New Criminal Justice System (NCJS) legal framework, including the CPC and such laws as those regarding women and other victims). Aside from potentially conflicting party rights, the CPC introduced an adversarial or accusatory system,2 but unlike countries historically following this tradition, it emphasized compromise or conciliatory agreements on case outcomes. As discussed below, lawyers trained in oral advocacy by PROJUST and other United States Government (USG) programs sometimes resist conciliation, preferring to push for full litigation—to trial if needed. These are issues the GOM will eventually have to address, but about which PROJUST can do little. • Ineffectual police reform. Incompetent, poorly trained and supervised, and, often, corrupt police remain the rule in most states and in CDMX. The effects of the recently enacted “Internal Security Law” allowing military forces to fill police roles in some states adds another complication with unknown effects on the reform. 1 The homicide rate (the usual measure) is again rising with some believing it will exceed rates recorded earlier. 2 The United States (U.S.) and other common law systems typically call themselves “adversarial.” For whatever reason, Latin Americans tend to use “accusatory” to describe their new codes. Since some are every bit as adversarial as the U.S., the preference seems more idiosyncratic than analytic. However, in Mexico, some powers enjoyed by judges (e.g., the ability to alter charges and dismiss defense attorneys unilaterally) seem more consistent with modern inquisitorial systems. 4 • Insufficient or non-existent coordination among sector institutions. This is particularly evident in police-prosecution relations but affects others as well. It has been a recognized, but unresolved, problem for decades, far predating the reform. • The retention of many traditional practices, organizations, and vices that do not fit with the new codes, including corruption in accepting and handling complaints, a tendency to prioritize formal processing of cases over resolving them, and the retention of horizontal (non-lineal) organization within many Public Ministries and Public Defenders Offices (PDOs).3 • Development of centrally designed models and protocols intended to guide and set state actions. This is a preferred practice of Mexico’s federal authorities to which PROJUST has contributed, but some of these instruments are widely regarded as poorly conceived or requiring adjustments to local circumstances.4 • Absence of institutional and sector-wide human resource policies that would encourage greater equalization of salaries (which often vary widely among institutions, even within the same state), incorporate new profiles critical to code implementation, and assist organizations in determining the numbers and types of actors needed in each location or jurisdictional specialty. • Late adoption by the federal jurisdiction of new codes, still pending transformation of the Federal Prosecutor’s Office (PGR) to more independent status and continuing failure to define and separate federal and state responsibilities for resolving specific types of crime (a complaint registered by interviewees in all states visited). As discussed in later sections, these issues have affected the implementation of USAID programs, both past and present. However, they are part of the territory within which PROJUST operates and which it can do little to affect. 2.2 PROJUST: PURPOSE/OBJECTIVE, ORGANIZATION, AND DEVELOPMENT PROJUST is the latest in a series of USAID projects aimed at helping Mexico adopt its criminal justice reforms. As initially envisioned by USAID (circa 2012), it aimed at helping states develop and enact new CPCs and related legislation. As stated in the initial contract (523-C-14-00003 m dated July 3, 2014), its overall purpose is to: “[P]rovide customized direct or indirect support at the state and federal level for the final push towards the 2016 Constitutional reform deadlines, assisting selected Mexican states to pass the ‘point of no return’ on their justice system reforms, and therefore contribute to broader USG and GOM efforts to mitigate conflict, reduce impunity, and promote a more transparent and effective justice system.” PROJUST’s purpose was amended in March 2017 to read as follows: 3 What is called horizontal by English-speaking experts is termed non-lineal by Mexicans. What it means is that even in the first instance, a complaint will be passed from one prosecutor or public defender, to another, rather than resting in the hands of only one of them (vertical organization). For further explanation in U.S. practice, see APRI, 2002. 4 For elaboration, see later section on models and protocols. The mechanism mentioned in all states was the uniform police bulletin (IPH). While not supported by PROJUST (except for recent explorations of the use of software to facilitate its completion), it was roundly criticized by prosecutors for its length (even after reduced from 400 to 200 questions), inclusion of details irrelevant to most investigations, and the delays it provoked in the reporting of incidents by police. 5 “[P]rovide customized direct or indirect support at the state and federal level for the final push towards the 2016 Constitutional reform deadlines, assisting selected Mexican states to pass the ‘point of no return’ on their justice system reforms, and support the consolidation of the system past the June 2016 deadline….” [bolding added by the ET] The bolded addition has two simple explanations. First, although not recognized in the initial contract, it was evident by 2014 that Mexico would adopt a single CPC rather than expecting each state to design and implement its own version according to guidelines in the 2008 constitutional amendments. This eliminated the need to help states draft and enact their own code but still required that related state legislation (e.g., for key institutions like the judiciary and the Public Ministry) be adapted to the new requirements, a process that PROJUST continues to support. It also meant the process of putting the laws into operation would likely be delayed for states that already had their own codes and supplementary laws as well as for those that lagged behind. Second, it had become apparent that a legal framework was no guarantee of changes in operations, or what the GOM now calls the reform’s “consolidation,” especially in a climate of escalating violence and consequent demands to scale back some of the due process guarantees. An earlier change also occurred in 2014. Although the original contract was awarded to MSI, a legal protest by TetraTech DPK required that it be divided between the two. This slightly delayed project initiation until early 2015,5 so the five core tasks (each accompanied by a list of two to five results6 ), could be apportioned between the two firms. TetraTech DPK was given tasks relating to augmenting access and increasing public support for the NCJS, while MSI retained tasks having to do with legislation, increasing prosecutorial and judicial effectiveness, and strengthening analytic and quality control capacity within the targeted institutions. In 2017, the five tasks were consolidated into four and divided as follows: 1. Promote increased legitimacy of the NCJS (MSI). 2. Support capacity building for monitoring and evaluation (M&E) of criminal justice system operations (MSI). 3. Promote increased public support for the NCJS (TetraTech DPK). 4. Support increased access to justice and improved victims’ assistance (TetraTech DPK). The project results framework and accompanying indicators changed three times between 2014 and the present.7 For simplicity’s sake, the evaluation focuses on the first and last set of indicators, ignoring the brief interlude between them. During late 2014 to mid-2016, PROJUST focused, in partnership with SETEC, on advancing the states’ adoption of the legislative framework and training to enhance judicial operators’ ability to work in the new system. Activities were carried out in nearly all states, but additional results incorporated in the initial contract were temporarily afforded less attention. 5 Although MSI’s first annual report runs from October 2014, USAID/Mexico staff told the ET they considered the project to have begun in early 2015. 6 No targets were set, however. The quantification of proposed increases or reductions in specified areas, for example, in case disposition times, was left to be determined. 7 This is referenced in the annex to the SOW for the evaluation (Annex 1 of this report). 6 Beginning in early 2016, the shift to producing improvements in organizational performance was to focus on fewer states. For MSI, it was divided into three categories: five states for comprehensive work, two to three for moderate work on selected institutions, and two to three to focus on selected problems as identified by the state. TetraTech DPK (Tasks 3 and 4) was not given specific numbers but was to try to focus on states supported by MSI. When the evaluation SOW was drafted in mid-2017, the implementers were working in 19 states consisting of 12 by MSI and 16 by TetraTech DPK, including seven not covered by MSI, 8 a number that remained stable during the ET’s fieldwork. It took both firms six to 12 months to develop their new activities, although some built-on actions began in 2015 or even earlier (MSI had implemented USAID/Mexico projects from 2004 onward). For MSI, the five initial key actions—improvements to intake centers in the Public Ministries, introduction of plea bargaining (called abbreviated proceeding),9 work with courts on scheduling hearings, creation or strengthening of pretrial services (UMECAs), and introduction of analysis, monitoring, and evaluation units (UAMES) in targeted institutions—were all linked to the “Tres Pisos” (Three Level) model. Although centered in the Public Ministries, the model’s emphasis on differential case management and solutions alternative to a full trial was incorporated in programs for judges and defense attorneys. TetraTech DPK’s further work with alternative dispute resolution (ADR) units within the Public Ministries and UMECAs was linked to Tres Pisos, as the former were part of the first level (first piso) where simple cases might be sent to mediation, and the latter were intended to operate at the second level (second piso), as simpler cases were directed to the courts for preliminary hearings. Work with the CJMs was also related, as many of them had their own intake centers to filter and redirect new cases. However, as gender violence cannot be mediated under Mexican law, complaints that went forward are assigned to one of the higher pisos, either tried or treated with an abbreviated procedure or a “conditional suspension” (deferred prosecution) for which the defendant requires supervision (presumably by an UMECA) to track compliance with judicial orders. Figure 1 shows the Tres Pisos model. The numbers in the figure represent the proportion of cases to be redirected to each level, presumably to be resolved there. They probably should be regarded as illustrative or derived from the experience of the model’s originator in Monterrey. The model’s key point is that complaints should be handled and resolved according to such basic characteristics as: type/severity of crime; information available to the investigators; type of defendant (including criminal antecedents and other pending cases); and willingness of the parties to reach a resolution short of a full trial. Although intake centers already existed in most Public Ministries, their filtering function was an addition with use of the model. To summarize, PROJUST’s purpose remained consistent from 2014 onwards, although its placement in USAID/Mexico’s results frameworks has gone through three iterations. More importantly, the results emphasized, and the indicators of their achievement, have changed from 8 Numbers come from the SOW but were readjusted by the ET, owing to one inaccuracy—San Luis Potosí’s exclusion from the common list although Tetratech DPK (as well as MSI) works there. 9 As explained by a PROJUST staff member, this is really not plea bargaining but sentence bargaining. A defendant willing to plead guilty to the charges levied can, except for a few exempted crimes, receive a prison sentence one to two thirds under the maximum. The agreement is usually reached by the prosecutor, the defendant, and his/her attorney. However, it can be overridden by the judge; the victim and his/her attorney must, in theory, agree as well. 7 a focus on legal change and number of trainees to the improvement of institutional performance. The consolidation of the initial five core tasks into four eliminated the emphasis on the legal framework as of post mid-2016; its achievement was complete except for a few secondary laws in a small number of states. The division of the tasks between MSI and TetraTech DPK did not alter the project’s focus, but after mid-2016 (officially only in 2017), USAID narrowed the number of states in which the tasks would be pursued. Figure 1: Tres-Pisos (Three Level) Case Prioritization Model10 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODS The evaluation was conducted by a team of four consultants—two international and two Mexican—as well as a logistical person to support the ET’s interview agenda, travel, and other logistical details. In conformity with the SOW (see Annex 1), the ET’s approach included: a preliminary document review, three weeks of interviews with project personnel and external experts in CDMX, and visits to eight target states (see Table 1) over a period of five weeks. (The ET visited the first two states together and then divided to visit the remaining six.) 10 Source: MSI Annual Report, October 2016- September 2017; p. 15. 8 Table 1: Target States (and Cities) Visited by the Evaluation Team State (City) San Luis Potosí (San Luis Potosí) Zacatecas (Zacatecas) Tabasco (Villahermosa) Baja California (both Mexicali and Tijuana) Chihuahua (Chihuahua) Coahuila (Saltillo) Guanajuato (Guanajuato) Nuevo León (Monterrey) The methods used in the fieldwork were both qualitative and quantitative (see Annex 2). Qualitative work included observation of organizational practices in the states visited, 14 focus group discussions (FGDs), and over 250 interviews with PROJUST staff, project participants in the states, members of CSOs and private sector groups, and outside experts with knowledge about the reform’s—not necessarily PROJUST’s—progress. Quantitative work included the collection of additional statistics to complement those provided by the PROJUST staff in the form of results indicators and the administration of a questionnaire to 150 members of state institutions who had participated in project activities. The questionnaire (see Annex 3) was based on the SOW questions. It was used to tap respondents’ experience with PROJUST’s activities and their assessment of how PROJUST’s activities affected their performance. Questionnaire responses were converted to a database with primary and secondary coding with the latter used to condense answers to open-ended questions into fewer categories. Tables from the analysis are included later in the text as well as in the annexes. 3.2 EVALUATION LIMITATIONS The primary limitations faced by the ET were limitations related to the quality, accuracy, consistency, and comparability of secondary quantitative data, including data produced as part of PROJUST’s performance monitoring system. To begin with, the proposed use of statistics from courts, Public Ministries, and other institutions (e.g., pretrial services, CJMs) within the eight states or nationwide was frustrated by the poor quality of what was available (see Box 1).11 This meant the evaluation’s principal quantitative data were collected from the questionnaire and methods proposed in the Inception Plan’s Evaluation Design Matrix had to be modified accordingly. Further changes were required by a series of modifications to the SOW (many requested by the ET) and USAID’s instructions on the report’s organization and focus provided after the submission of the initial report draft. 11 See Center for Research for Development (CIDAC) (2017) on similar issues faced by their own researchers. 9 Box 1: A Note on Statistics Within Mexico’s Justice Sector Despite substantial investments in automated case registries (case tracking systems),12 there are few state (or federal) justice agencies that produce performance statistics based on more than manual counts and Excel tables produced periodically by individual work units. Those working in statistical offices are often little more than clerks who compile the Excel sheets into a single database, most frequently used to provide reports required by the National Institute of Statistics and Geography (INEGI) and other federal agencies. Few performance data are available on institutional websites. During its early years, SETEC attempted to collect better data, but, overall, the reform program has not recognized their importance. INEGI does publish basic statistics on justice sector performance, but they come from the same sources. Moreover, their publication is delayed by at least one year; some states do not provide what is requested, and when they do, there are obvious inconsistencies in what and how they count. One fundamental problem involves the base against which all statistics are calculated. In Zacatecas, for example, it only comprises real crimes, excluding actas (registration of misplaced/lost/stolen documents and cell phones for insurance purposes), atenciones (referral to other services or answers to questions not involving a specific complaint), and even referrals to ADR. So far as could be determined, in other intake offices, early attention units or “Unidad de Atención Tempranas” (UATs), all incoming consultations/complaints are registered, meaning the percentage of resolved complaints will be much lower. Reportedly, INEGI’s new director will request more complete data but with no guarantee of improvements. A few state institutions produce, use, and publish better sets of statistics—Guanajuato’s judiciary is one example. However, state Public Ministries are reluctant to share their data or even to indicate what data they collect, how they collect it, and how they use it. With regards to PROJUST’s performance monitoring data, the nine indicators used to track project performance (see Section 4.1.1), as operationalized or defined by the two implementers in coordination with USAID, posed problems for measuring changes in the capacity of supported institutions, for a few reasons. To begin with, some institutions (e.g., CJMs and CSOs) lack indicators, and thus the ET had to rely wholly on qualitative data to assess their progress. Additionally, eight of the nine indicators refer to global progress,13 meaning results depend not on one institution (or on work corresponding uniquely to MSI or TetraTech DPK) but rather on results dependent on two or more organizations. For example, while the courts are the source of information on case disposition rates (Indicator 6), these depend on more than the judiciary. Furthermore, MSI’s annual report for fiscal year (FY) 2017 (the most recent provided to the ET) tracks indicators through 2016. Except for the few institutions that could provide their own updated information, it was thus impossible for the ET to quantify further progress in these specific areas. Moreover, the results tracked, and the states included, changed over time. Although MSI was tracking outcome indicators prior to 2017, the states for which indicators were collected varied across reporting periods, as did the indicators tracked within a single state. Hence, improvements were rarely tracked for the same state and over time. A related 12 These automated systems often did not track cases, but rather assisted case processors in determining which cases had pending actions, included templates for common documents, and predominately allowed text entries (very difficult to code and convert to statistics). Moreover, they often existed on servers within individual court units, and were not connected at the state level, explaining the use of Excel reports on case movements. 13 This is true even of Indicator 8, not reviewed by the ET, as journalists’ perceptions depend not only on training provided by PROJUST (and especially TetraTech DPK) but also on visible progress in many dimensions. 10 problem is that real improvements take time and are subject to fluctuations. For less than a five-year period, it is hard to say whether positive or negative changes really occurred. Finally, three of the nine indicators are operationalized in such a way as to signify less than their name implies. Specifically, prosecution rates are limited only to homicide cases in a specific district, the use of abbreviated procedures initially was limited to a percentage of judgments in homicide cases and is geographically limited,14 and MSI decided to track case disposition rates separately for control and trial courts. As this suggests, all indicators only track results in specific districts, not an entire state. The ET encountered other limitations beyond issues with secondary data. While the ET would have preferred a more rigorous approach to fieldwork, time and budgetary constraints did not permit it, especially in selecting states and interviewees and developing and pilot testing the survey questionnaire. The questionnaire was developed before travel and was approved by USAID three days before the team met in Mexico. There was thus no opportunity for a pilot test, although the ET was able to make a few adjustments (e.g., elimination of two questions, slight changes in wording of other questions) after its initial application in San Luis Potosí. 15 Another limitation was that the evaluation methodology did not allow the ET to credibly attribute observed changes to PROJUST activities. To address this limitation, the state of Guanajuato was initially selected as a “control state,” or state with no recent PROJUST assistance, to serve as a benchmark to compare changes in the target eight states. The ET, however, discovered that Guanajuato had also received inputs from MSI during the last two years and, like all states in the country, had benefited from short-term training and legal assistance before mid-2016, thus negating its usefulness as a benchmark comparison for attribution purposes. On top of this, states not supported, or no longer supported, by PROJUST lacked the relevant statistics to compare changes there to changes in PROJUST￾supported states. In lieu of measuring or estimate attribution, the ET looked for evidence that project activities had contributed to observed changes and the size of their contribution. Finally, the selection of states visited, and key informants interviewed, potentially introduced an unknown level of sampling bias. The ET based its initial proposal on states to be visited according to where the SOW indicated both TetraTech DPK and MSI worked with the targeted institutions. The list was vetted with USAID (which had to approve it), MSI, and TetraTech DPK. At USAID’s and MSI’s suggestion, one state (Morelos) was removed and, after further discussions, another was chosen. The SOW already listed institutional members to be interviewed, but for those not specified by position, the ET had to depend on MSI and TetraTech DPK to provide contacts and set up meetings. The ET acknowledges that this selection process potentially introduced biases in favor of the two implementers. 14 USAID says it requested a change in this indicator, but this did not affect what was reported by MSI in its latest annual report. 15 In retrospect, the parties’ agreement to accelerate initiation of the fieldwork so that it two states could be visited before January 2018 was probably a mistake, contributing to this and other methodological shortcuts described further on. Their acceptance of the ET leader’s suggestion that the team divide to cover the remaining six states in three weeks also proved problematic While an improvement over the initial SOW, which only allowed four weeks for fieldwork, thus implying a division from the start, it forced everyone to return to CDMX for one day between states to compare notes and, even then, did not promote sufficient consistency in approaches. This is mentioned, not as an excuse, but rather as unsolicited advice for any future evaluations of such complex projects. 11 4.0 FINDINGS AND CONCLUSIONS The following sections address the SOW’s principal questions and sub-questions. Conclusions, as requested by USAID, are provided at the end of the section corresponding to each principal question. 4.1 EFFECTIVENESS To what extent has PROJUST met its objectives relating to the functioning of the AGOs (Public Ministries),16 CJMs, Courts, and UMECAs and the actions of CSOs related to the criminal justice reform? FINDINGS 4.1.1 To what extent has PROJUST technical assistance contributed to results, as measured by approved indicators and Qualitative Data? In its comments on the draft report, USAID requested that this question be answered institution by institution. There are, as noted, two sets of indicators, pre- and post-2016, although the second set was not officially approved until late 2017. According to USAID, MSI did not have a contractual indicator before 2017. Up to that point, MSI tracked results related to laws passed, states’ conformity with a special SETEC system for rating advances in legal and organizational implementation, and training done, as well as the number of SETEC M&E initiatives directed to criminal justice reform developed and/or implemented. PROJUST consistently met or exceeded the performance targets for this initial set of indicators; training numbers exceeded workplan targets and, while progress on the SETEC scale varied across states, all showed improvements. An MSI evaluation, 17 conducted six months after the trainings using a combination of FGDs and (except for UMECA staff) a short knowledge retention quiz, found evidence for skills retention among training participants. 18 However, training participants had received different types of training, and skill retention at the six-month mark was lowest among the majority (188 out of 254) that participated in less specialized courses. As the evaluation concludes, “training was helpful but not sufficient on its own.” As opposed to the initial set of indicators, the list of indicators approved in 2017 includes the nine outcome indicators found below. Of these nine outcome indicators, MSI tracks all but Indicators 5, 8, and 9, although it does report results relevant to Indicator 5, even in its 2017 report. The SOW’s evaluation questions exclude Indicators 8 and 9, for which no values exist yet in any event. 1. Pretrial Confinement Rates 2. Prosecution Rates 3. Compliance with Precautionary Measures from Pretrial Service Units 16 Although Mexico currently uses three terms for Public Prosecutors’ Offices (PPOs or AGOs)—Fiscalía, Procuraduria, and Public Ministry— only the last is used here. The difference is important to Mexicans but for comparative specialists, the terms are interchangeable. The structure, organizational placement, relative independence, and functions of these entities do vary considerably from country to country, but not in any way that implies a consistent use of the different titles. 17MSI, “Training Effectiveness Assessment,” July 27, 2016. 18 MSI, “Training Effectiveness Assessment,” July 27, 2016. 12 4. Plea Agreements 5. Use of ADRs 6. Case Disposition Rates 7. Percentage of Cases with Sentences Greater Than Three Years 8. Change of Journalist Perception and Accuracy on Criminal Justice System Reporting 9. Increase in use of Locatel (legal services hotline) 19 These and other performance indicators20 are tracked in the implementers’ quarterly and annual reports. The ET has no reason to believe the reported results are inaccurate. On the whole, and despite downward fluctuations in some cases, the indicators’ results show positive, if sometimes slight, changes for most states (see Tables 2, 3, 4, and 5). The changes in indicators’ values suggest that PROJUST actions contributed to improved outcomes, albeit to an unknown degree owing to the data limitations described in the previous section. Because of these limitations, the ET had to rely extensively on qualitative data to enhance its understanding of the degree to which PROJUST technical assistance contributed to these results. As per USAID’s instructions, the following sections cover advances in the specific institutions listed in the SOW, as evidenced by the approved indicators and complementary qualitative data. AGOs/Public Ministries: Indicators 2, 4, and 7 (see Table 2) and Indicator 5 (see Table 3) are intended to track the strengthening of the Public Ministries due to project support; however, their results also reflect contributions from the courts, public defenders, and victims’ counsel. Victims’ counsel, for example, may advise against mediation and/or the agreements reached in the ADR offices and can also oppose the use of plea bargaining and conditional suspension as opposed to taking a case to trial. Table 2: Indicators 2, 4, and 7 in Targeted States District/State 2. Prosecution Rates (MSI)21 (Indicator 2) 4. Plea Agreement Convictions (MSI)22 (Indicator 4) 7. Sentences Over 3 Years (Indicator 7) 2015 2016 2015 2016 2015 2016 Mexicali, Baja California 15.6 2.5 83.7 83.3 NA NA Macuspana, Tabasco 0 33.3 NA NA NA NA Tepic Nayarit 23 6.3 21.9 100 88.9 NA NA 19 Locatel is a GOM service linking citizens to various government services (not just justice), but its use for this purpose is promoted by TetraTech DPK’s work. 20 Other performance (or “context”) indicators tracked by MSI include conviction rates for cases taken to trial, recommendations of pretrial detention and precautionary measures by UMECAs, and effective debates by PDOs. Those tracked by DPK include percentage of ADR cases reaching agreement and percentage of compliance with agreement. 21 Prosecution rates shown here are still calculated only for homicide cases opened in homicide units. It is not clear whether they include homicides “without a known perpetrator,” whether sent to the homicide unit, or for “cases without a known suspect.” It would also help to list the “N” (the number of cases on which this is calculated), as in some instances (e.g., Macuspana), it is extremely low. 22 Calculated as the percentage of all homicide verdicts reached through plea bargaining (the abbreviated proceeding). 23 It is not clear whether Nayarit is still a target state. Since United States (U.S.) authorities arrested its then Attorney General in March 2017 for drug trafficking, it may have been removed from the list. Also, as Nayarit’s prosecutors have extremely low caseloads (according to CIDAC, 2017, 50 annual filings), it probably is not a good example. The Macuspana office had such low caseloads that it was eliminated as a separate unit. 13 District/State 2. Prosecution Rates (MSI)21 (Indicator 2) 4. Plea Agreement Convictions (MSI)22 (Indicator 4) 7. Sentences Over 3 Years (Indicator 7) 2015 2016 2015 2016 2015 2016 Zacatecas, Zacatecas 1.7 1.4 100 0 94.0 88.0 Saltillo, Coahuila NA 18.5 NA 100 75.0 91.2 Source: For Indicators 2 and 4, MSI Annual Report, 2017, for Indicator 7, USAID Although intake centers already existed in most Public Ministries, their filtering function was an addition with introduction of the Tres Pisos model. In the four states where Tres Pisos was introduced in its current form (Coahuila, San Luis Potosí, Tabasco, and Zacatecas), 24 the model is having a significant effect on initial case processing within the states’ Public Ministries. The most visible improvements of Tres Pisos” remain in the first piso—more effective screening, greater use of ADR, and improved treatment (as reflected in surveys) 25 of the complainant. This is partly attitudinal but is also evident in the organization of the first piso (reception, filtering, and channeling of complaints). However, the second piso (ordinary crime) and third piso (high￾impact crimes) have shown less evidence of having an impact on how the Public Ministries process cases. Absent additional data, however, the ET cannot say with certainty whether such improvements are perceived by system users. [The World Justice Project (WJP) does indicate that victims or complainants perceived receiving better treatment, but these are aggregate national figures.]26 Equally importantly, the four intake centers (UATs) could provide statistics on what they received and to which offices they referred it—ADR, piso 2 or 3, or the unit for “crimes without an identified suspect.” UATs in San Luis Potosí, Tabasco, and Zacatecas were already recording times required to present a complaint and, in Tabasco, what percentage lingered in the first piso. Based on observation and staff interviews, the ET found the UATs in the four states to be professionally organized and actively receiving clientele. A PROJUST subcontractor did criticize Monterrey’s UAT for ineffectual filtering, and Tabasco’s Public Ministry said a next step was to ensure the filtering was done well. On the basis of its observations in the first four states and Nuevo Leon, the ET shares these concerns as to the efficiency of filtering, but the fact that the UATs had evolved from a simple reception center to an entity systematically channeling cases to other offices is an important step forward. The only other problem noted was the occasional proliferation of intake centers in Tabasco and San Luis Postosí (e.g., separate prosecutorial units for homicides or gender violence in CJMs). However, the ET was told, without being able to verify it, that complainants “knew where to go,” and that gender violence in particular, if received 24 In Guanajuato, the “control state,” no interviews were held with the Public Ministry. Interviewees in Baja California, Chihuahua, and Nuevo León said they did not use Tres Pisos. As the model’s development occurred during a time when assistance to these three states had lapsed, this may be a semantic issue. Since Nuevo León is where the model apparently originated, this is the one oddity; however, the “rejection” of the model is likely more symbolic than real, a result of a radical turnover in staff under the previous Attorney General (AG). 25 The most important survey, but of the results of the entire NCJS (not limited to PROJUST states), was funded by the Bureau of International Narcotics and Law Enforcement (INL) and conducted by a WJP grantee. Still the three states featured in the group’s presentation of its findings all received PROJUST assistance: Nayarit, Zacatecas, and Nuevo León with 66.6, 78.8, and 79.2 percent (respectively) of the complainants interviewed reporting that their treatment by the Public Ministry was “good or excellent.” Data provided by Layda Negrete, project director. 26 The WJP project was capable of disaggregation by state but could not provide these figures at the time of fieldwork. 14 in a less appropriate location, would be quickly channeled to where it should have been registered. Regarding further processing of complaints, the critical instrument was MSI’s Rapid Results or “100 Days” exercise (see Box 2), which aimed at moving stalled investigations from the Public Ministries to the courts or to some other form of final resolution. Rapid Results exercises had been conducted or were underway in all target states. The ET’s interviews and FGDs always included their participants. Box 2: Rapid Results Explained Rapidly This is a technique developed by the Rapid Results Institute (RRI) to foster team-based solutions to organizational problems using the “100-day challenge.” As applied by MSI (with RRI’s guidance), it invites representatives of sector institutions (including CSOs and business associations) to identify a problem, remedies, and a 100-day goal (usually a fixed number of cases to be processed). The team meets periodically to assess progress and adjust. The intended results, aside from meeting the goal, are increased cooperation, a sense that problems can be resolved, and the continued application of improved practices at the conclusion of the 100 days. The exercise was very popular among participants (although FGD answers were more positive than questionnaire responses in San Luis Potosí and Zacatecas, see Table 7 below), despite stirring up complaints about increased judicial workloads and the unwillingness of some prosecutors, defense attorneys, and victims’ lawyers to adopt conciliatory approaches to case resolution. Rapid Results was not intended as a permanent fix for processing issues, which remain significant in scope and far outstripping the capacity of Rapid Results or a similar initiative to address comprehensively (see Box 3). Rather, Rapid Results is best seen as a way to jumpstart the process to begin addressing such issues, identify bottlenecks, and encourage inter￾institutional coordination and cooperation in removing them, which it has done. Its immediate results have been important for demonstrating to institutions that they can achieve more collectively and possibly for promoting more permanent changes in how complaints are treated. With only a few exceptions (e.g., San Luis Potosí, which had to reduce its initial 400 case target), states met or exceeded their 100-day goals, and each state processed far more cases than in earlier periods. Box 3: A Numerical Demonstration of Public Ministry (and NCJS) Underperformance Coahuila’s judiciary statistics27 indicate that few complaints are “judicialized” and that the vast majority involve narcomenudeo, or the consumption, possession, and alleged intent to sell small quantities of drugs.28 Of the 5,899 cases brought to the courts in 2017, the breakdown by crime types is as follows: • Narcomenudeo 61 percent • Robberies 12 percent • Family violence 10 percent • Other 12 percent 27 Information from Coahuila, Poder Judicial, “Sistema de Justicia Penal: Análisis 2017” provided to the ET, February 2018. 28 Interviewees in Coahuila (and elsewhere) suggested that what is charged as narcomenudeo is usually only consumption or possession, and that the police exaggerate the quantity of drugs to allow categorization as a crime (which consumption is not, and possession is, only above certain quantities). Obviously, a certain amount of police corruption may also be involved. 15 Although violent crime rates have dropped in Coahuila, as one interviewee said, “it is narcomenudeo that saved the new system,” meaning this minor, if common, crime accounts for most judicial effort and represents a major portion of successful investigative work by the Public Ministry. Moreover, although not only the courts’ fault, the judicial clearance rate (cases resolved/new entries) for 2017 was only 56 percent. Data from a local non-governmental organization (NGO)29 indicate that 80,040 complaints were filed in 2017, of which those judicialized represented only 7 percent, a figure consistent with the Center for Research for Development (CIDAC) 2017 calculations for other states on which it has data. In Tabasco,30 the Public Ministry, with which PROJUST has worked intensively on implementing the Tres Pisos model, reports that of the 5,204 investigations begun in 2017, 83 percent (74 percent if those temporarily archived are considered “resolved”) remained without resolution, meaning they would be carried into 2018. Moreover, 21 percent of the entries in intake centers remained under consideration. Of the 29 percent of all entries sent to the second piso, 72 percent also remained without resolution, as did 63 percent of the 4 percent of cases sent to the third piso (complex crimes). Some of these “unresolved” cases have been judicialized, but this is only a preliminary step in their resolution. The problems documented here are typical of the initial experience of countries transitioning to an “adversarial, oral procedure”—low clearance rates for investigation, few cases taken to the courts, and an emphasis on easy cases (those with in flagrante detainees often accused of minor crimes). Processing easy cases more rapidly is an accomplishment, and part of the organizing the basics approach. However, according to Latin American Public Opinion Project (LAPOP), Latinobarómetro, and other surveys, the general public cares more about crimes that most threaten their security. Notwithstanding the above accomplishments, MSI still could use a design/model/plan for structuring attention within the Public Ministries to channel crimes to the second and third pisos. Interviewees in all states stressed that a majority of complaints stagnated at the second piso for reasons ranging from poor police-prosecutor coordination to the difficulties of getting all parties (including the victim) to agree on the best course forward. In Coahuila, San Luis Potosí, Tabasco, and Zacatecas, the organizational structure for the second piso varied extensively from San Luis Potosí’s proliferation of specialized units for both common and high￾impact crimes to Coahuila’s division of second piso offices according to whether property was affected. Unfortunately, the ET was provided with no data to assess the results of the above activities. Moreover, this still leaves the majority of crimes unaddressed except for general courses and manuals received by all investigators and those provided to all judicial operators on the use of abbreviated proceedings and conditional suspension. The other major area of PROJUST assistance within the Public Ministries is the use of ADRs to resolve minor issues. As seen in Table 3, the percentage of criminal complaints resolved through the ADRs has increased in three of the four states where it is measured, and in each of these cases already, the percentage already exceeds its 2019 goal. Table 3: Percentage of Criminal Complaints Resolved Through ADR in Selected State (Indicator 5) 29 http://rid.ccic.org.mx/. The site also has data for 2011-2016. 30 Information from Tabasco, Fiscalía General del Estados, “Modelo de Gestión y Administración de Casos” provided to the ET, February 2018. 16 State and Average Percentages Baseline Actual (Jan.– Sept. 2017) Actual (Jul.– Sept. 2017) Increase (Baseline vs. Actual, Jan.– Sept.) Increase (Baseline vs. Actual, Jul.–Sept.) Goal 2019 Average 6.4% 7.0% 7.7% 9.4% 32.8% 11.5% Coahuila (Baseline = 2016) 10.2% 12.2% 14.3% 16.4% 28.7% 12.0% San Luis Potosí (Baseline = Quarter (Q)1 2017) 5.1% 4.6% 3.9% –10.9% –30.8% 12.0% Sonora (Baseline = Q1 2017) 7.8% 8.8% 9.8% 11.4% 20.4% 12.0% Tabasco (Baseline = Q1 2017) 2.3% 2.5% 2.6% 6.1% 11.5% 10.0% Source: TetraTech DPK, Annual Report, October 2016-September 2017 Thanks to Tres Pisos, Public Ministries now register the percentage of complaints sent to their ADR offices, but this rarely corresponds with the percentage of complaints resolved there. In Coahuila, for example, while 30 percent of complaints entered in 2016 were referred to an ADR, only 12-14 percent were resolved there.31 There are a number of possible reasons for this outcome (e.g., not all parties want to mediate or not all parties reach agreements), and 100 percent resolution is unrealistic. Notwithstanding, the ET notes that TetraTech DPK is working to increase the percentage of complaints resolved through the ADR offices. Another thing to consider is that the percentage of complaints resolved shown in Table 3 refers to the percentage of agreements (acuerdos) reached. Technically, an agreement is different from and precedes a resolution, which is the successful fulfillment of the agreement. Thus, agreements do not always translate into resolutions. The ET found no evidence, moreover, that ADR offices are making efforts to follow-up to ensure that agreements are actually resolved. The only exception noted by the ET was Coahuila, which said that it was making efforts to contact some victims by phone. Even if, as TetraTech DPK indicates, resolution is immediate in the majority of cases, and thus there is no need for follow-up, Indicator 5 does not differentiate between the two so that the actual percentage of cases reaching a resolution is unknown. The ADR offices in Coahuila, San Luis Potosí, Tabasco, and Zacatecas were well-organized and seemed to have sufficient personnel for potential demand, although in all offices but San Luis Potosí, there were few clients engaged in mediation or awaiting their turn during the ET’s visit. Nonetheless, all ADR directors note that, for a variety of reasons (but especially flawed notification, a notified party’s lack of interest or change of heart, or a lawyer’s recommendation against ADR), defendants often do not arrive for scheduled sessions, many of which had to be 31 And while this was the 2016 number cited by TetraTech DPK and the Coahuila office in Saltillo, in the latter case it was repeated for 2017, suggesting that the office relied on PROJUST to register its advances. 17 cancelled. Notification is an issue in hearings as well, which PROJUST has yet to address. Both MSI (through Tres Pisos) and TetraTech DPK (with diagnostics, models, manuals, training, and accompaniment) have contributed to the improved functioning of ADR services, but there are still fundamental problems (e.g., notification in hearings) that neither has been able to address. The ET also noted a number of cases in which departments or offices received cases without an identified suspect. In Tabasco, however, MSI has helped set up and is working with the unit to enhance its capacity to identify suspects. In this case, the Public Ministry’s investigative police analyze data to identify repeat offenders and, less frequently, perpetrators of individual crimes. This model has already attracted interest from other states. The Judiciary: PROJUST’s post-2016 work with courts, both through Rapid Results and with training and direct technical assistance, has focused on programming hearings to reduce delays and postponements, shorten hearing length, increase the number of daily hearings, 32 and target interventions. Although some of these changes have been recorded by MSI in its annual reports, the most relevant indicator is Indicator 6 for which MSI reports case disposition rates covering 2014-2016 for control courts in Baja California, Morelos, and Zacatecas and for control and trial courts in Saltillo, Coahuila (see Table 4). 33 Table 4: Percentage of Case Dispositions Reached in Targeted States (Indicator 6) State/District 2014 2015 2016 Mexicali, Baja California 55.0 54.4 48.6 Cuernavaca, Morelos 86.1 73.0 73.5 Saltillo, Coahuila NA 53.3 83.5 Zacatecas, Zacatecas 68.0 68.9 68.7 Source: MSI Annual Report, 2017; per USAID instructions, the method for calculating case dispositions has changed. If applied retroactively to include resolved case from earlier years, it should raise the above percentages. As seen in Table 4, case dispositions reached in control courts averaged between 65-70 percent from 2014-2016, although with substantial variation across states. Although the Coahuila rates reported by MSI for both control and trial courts reached 80 percent in 2016, they do not match statistics provided by the Coahuila Tribunal, indicating that overall the disposition rate for all criminal cases was 54 percent. However, the latter figure was statewide, whereas MSI is reporting only on Saltillo. As suggested by USAID, it is also possible that PROJUST and Coahuila’s courts define disposed cases differently. Qualitative data for these and other states provide a more positive picture. The ET interviewed judges in all target states and spoke with judicial administrative officers in Coahuila, Guanajuato, San Luis Potosí, Tabasco, and Zacatecas. The consensus among these judges and officers was that MSI’s Rapid Results exercises have made courts more aware of the potential for accelerating their hearing schedules using Excel tables provided by PROJUST and adopting the recommended scheduling in “blocks”34 so that lengthier trials do not crowd out 32 In Coahuila, for example from 3-5 to 15 or more daily, and a similar increase in Zacatecas, despite the administrator’s appearing less enthusiastic about PROJUST in general. 33 According to USAID, PROJUST reports case disposition rates for control distinct from trial courts in Baja California, Morelos, and Zacatecas so as to “provide more granular” information and because of “variations between the two courts.” However, this makes the results difficult to interpret where a simple mathematical operation could yield a combined rate. In the ET’s opinion, granularity would be better achieved by adding types of disposition (dismissed, principle of opportunity, plea bargaining, trial, and sentence). 34 MSI’s argument here is that the simpler, pretrial audiences should be scheduled in the mornings, and that any full trial, likely to overrun its time, should be placed in the afternoon. While not evident this was a PROJUST recommendation, several states have also decided to assign any 18 the more frequent preliminary hearings. (Tabasco, which said it was doing fine on its own, was the lone exception.) Even those court administrators not fully convinced of these benefits (e.g., San Luis Potosí, Tabasco, and Zacatecas) have, under pressure from Rapid Results, taken steps to increase the daily number of hearings held and to rationalize the formerly unpredictable movement of judges to preside over trials in other districts.35 Where PROJUST has had less success was in convincing judges that this is a necessary practice that could be extended to all cases. For example, in San Luis Potosí, the target for the 100 Days exercise had to be reduced because the judges rebelled against the increased pressures. While targets were maintained in Tabasco and Zacatecas, judges interviewed there had the same complaints. While PROJUST and USAID report that Zacatecas and Chihuahua plan to extend the practice to broader caseloads, this type of judicial resistance poses an obstacle to adoption in the remaining target states. In Coahuila (judiciary) and Tabasco (Public Ministry), MSI assisted in developing protocols for preliminary hearings so as to reduce problems created when actors disagree on which issues should be presented and how they should be addressed. Since those for Coahuila were only approved recently and those in Tabasco were still awaiting judicial acceptance, results are unknown. Here it bears mentioning that the few judiciaries that have moved ahead in this area (Coahuila, Guanajuato), did not use UAMEs (even when installed) but rather depended on the selection of General Administrators already dedicated to this practice. In the others, where the Tribunal did not have the foresight to place such personnel, convincing it to use information proactively continues to be a problem, only part of which can be attributed to inadequate information systems. As applied to the judiciary (and observed by the ET in Guanajuato and Nuevo León), MSI’s new diagnostic model (modelo para excelencia en gestión),36 poses several challenges, including understanding how the diagnostic is scored. 37 In Nuevo León, the Judicial Council representative stated that the model has a “lack of parameters” and cannot easily be adapted to local circumstances, saying the use of ISO 9000 would be preferable. As a self-diagnostic tool, the International Framework always lacked a frame of reference against which courts could measure their performance. MSI’s applies its own version using staff or consultants via individual interviews with court personnel; however, neither of the two courts evaluated could identify what the assessment criteria were. As with other institutions, PROJUST (and especially MSI) continues to provide specialized training to judges, other court staff, and (according to its FY 2019 workplan) local training institutes. It is a concern, however, that training recipients continue to request more MSI training rather than assist their own organizations. A recent initiative in Zacatecas, to train and available judge to pretrial hearings, rather than use the traditional practice of having one judge preside over all hearings related to a specific case. Where adopted, this simple measure has greatly improved scheduling issues, although (Coahuila) it was reported that many judges first resisted its adoption. 35 Mexico is severely “under-judged” as compared to other Latin American countries. However, as few criminal complainants reach the courts, the number of judges, even for the numerous initial hearings, is still adequate. The issue is that in large, multi-district states there may be too few judges in the least busy districts to hold oral trials and thus judges must travel over long distances to fill out the three-judge panel. This makes them unavailable for hearings or trials in the districts where they normally operate. 36 http://www.courtexcellence.com/ 37 As applied by MSI, the institutions receive scores on several dimensions, which are later displayed in a spider graph, indicating where they are weakest and strongest. Presumably the intent is to spur these institutions to make improvements, but the issue is a lack of understanding as to what they should do. 19 certify local trainers may be aimed at this goal, but even the director of the judicial training school there was not aware of how it would be linked to his programs. Six months after the evaluation fieldwork was completed, MSI did provide the ET with diagnostics of the training institutes for PDOs in Baja California and Nuevo León and for the judiciary in Zacatecas done in early or mid-2017, along with excerpts from its FY 2017 workplan on proposed actions for training programs. The diagnostics recommended actions to be taken by the institutes/programs, but the ET’s fieldwork found no evidence that they were acting on those recommendations, nor did MSI provide the ET information about its proposed FY 2018 actions during the evaluation data collection. Women’s Justice Centers: The ET visited or interviewed members of five CJMs, including two in Chihuahua and one each in Coahuila, San Luis Potosí, and Zacatecas and conducted a phone interview with a center in Ciudad Juarez.38 There are, so far, no performance indicators for CJMs, although interviewees were able to provide statistics on women attended, cases resolved, and other services provided. Overall, the ET evaluated the centers and PROJUST assistance provided to them positively. The centers visited varied in their design, services offered, and staffing patterns. However, whatever their available human, physical, and other resources, staff appeared motivated and enthusiastic about their work. In Coahuila and Zacatecas, the centers included facilities to house victims and their children; San Luis Potosí depended on local hostels and refuges to do this. Centers also offered services like education for minor children, job training, and exercise classes. All those observed had staff psychologists and social workers, as well as prosecutors, lawyers, and, occasionally, judges to handle cases. The basic issue with the CJMs, over which PROJUST has no control, is their centralized location; only Coahuila and Chihuahua had additional centers outside the capital city. Given the geographic size of several states, accessibility thus remains limited. The question, which PROJUST cannot resolve, is how Mexico will be able to extend these services throughout the country and where it will find funds to do so. Other problems identified with CJMs also had little to do with PROJUST instead involving GOM policies and forward planning. Although federal funds are available to these programs, they are limited to financing equipment, infrastructure, and training with most operational and salary expenses left up to the states. Consequently, the four CJMs visited had their own buildings; however, necessary equipment (especially vehicles) was frequently in short supply, and both staffing numbers and space were sometimes not up to demand. TetraTech DPK’s integrity model (a TetraTech DPK standard tool predating PROJUST) has been adopted by the Secretariado for use in certifying CJMs. In line with Mexican preferences; however, it focuses on standardizing structures and internal practices rather than measuring outcomes. Notwithstanding TetraTech DPK assistance, the certification process is onerous, requiring that boxes of documents be sent to CDMX where they must be reviewed by the small National Center to Prevent and Eradicate Violence Against Women (CONAVIM) staff. Whether certification improves results and what those results should be are unanswered 38 TetraTech DPK assisted in Tabasco, but there were delays in constructing the center. In Baja California, interviewees said the creation of a CJM required changes to state legislation. 20 questions. TetraTech DPK says it is developing results indicators for the certification, which would be a positive step in validating its utility. Units for Precautionary Measures: UMECAs were first promoted by MSI and are now required and regulated by federal law (https://www.gob.mx/sesnsp/articulos/modelo￾homologado-de-umecas).39 UMECAs have two basic functions: recommendations on pretrial release or detention and monitoring of compliance with imposed conditions (precautionary measures) in the case of supervised release. 40 Its two performance indicators (Indicators 1 and 3) are monitored by MSI (see Table 5). However, only Indicator 3 (compliance with precautionary measures) depends predominantly on the UMECAs’ actions. In its quarterly and annual reports, MSI reports results on Indicator 1 (pretrial detention) for three states covered by the ET. Those registered for compliance cover states that were not visited (Baja California Sur, Tlaxcala, Michoacán, Queretaro, and Oaxaca) and which will apparently not be prioritized. For the most part, the indicators show improvements, some minimal and some substantial. However, there are some immediate issues with what is reported. For example, Oaxaca reports both 100 percent pretrial detention and 100 percent compliance with pretrial or conditional release provisions. 41 TetraTech DPK’s work with UMECAs, for which no indicator exists, involves creating networks of CSOs to assist with supervisions. Table 5: Pretrial Detention Rates (Indicator 1) and Compliance with Pretrial and Conditional Release Provisions (Indicator 3) State42 Pretrial Detention Rate (Adults) Compliance with Pretrial Measures Year 2014 2015 2016 2015 2016 Tlaxcala 71.1 0 NA 95.5 93.6 Baja California Sur 100 73.1 11.4 81.3 88.2 Michoacán 99.8 69.9 NA 83.2 80.6 Nuevo Leon NA 62.4 29.7 NA NA Queretaro 46.2 42.8 NA 85.7 90.4 Oaxaca 100 100 NA 100 100 San Luis Potosí 35.4 32.6 16.2 NA NA Sonora NA 70 58.7 66.7 84.1 Zacatecas NA 30.3 19.5 NA NA Source: USAID and MSI Annual Report, 2017 UMECAs are critical for combatting criticisms of the reform and to PROJUST’s success. For this reason, the ET devoted considerable effort to reviewing their performance, visiting UMECAs or interviewing their staff in seven of the eight target states (all but Guanajuato). The ET notes, however, that the model developed by MSI and the GOM may not have been based 39 Pretrial services have been in effect in other countries, including the U.S., for years, but it was an Open Society Justice Initiative project, financed in part by a previous USAID project (also implemented by MSI), that introduced them in Mexico in 2011 (in Morelos). 40 Two further functions are performed by some UMECAs but remain under discussion—recommendations of specific conditions for pretrial release and monitoring of conditions for deferred prosecution (“conditional suspension”). 41 According to USAID, the inconsistency is due to Oaxaca’s UMECA having been set up only in late 2015. However, that still does not explain the seemingly contradictory figures, even if only a few defendants (two in December 2015 and possibly more in 2016) enjoyed conditional pretrial release. 42 All states except for Nuevo Leon and Zacatecas, Indicator 1 is restricted by district. MSI does not indicate restriction for Indicator 3, but presumably the same applies. 21 on pre-implementation estimates of demand, reasonable caseloads, or budgets.43 The recommendations of the ex-post diagnosis, if implemented, might have resolved deficiencies that emerged after the UMECAs’ creation, but the issue is whether certain of these deficiencies, and others, might have been avoided with a pre-implementation analysis (see Box 4). Box 4: How to Do a Study of Resource Needs (and Costs) for New Procedures and Organizations In-depth analyses of the cost and resource implications have rarely preceded enactment of Latin America’s new criminal procedures codes and related legislation (as well as many laws in other sectors). This kind of analysis is not simple, especially when the laws change the roles and the identity of key institutional actors. The steps in any such analysis involve: 1) defining the workload at present and over time; 2) estimating the human, financial, and other resources needed to handle it, and 3) considering available resources against those needed over time. The most difficult step is step two. Commonly, as is the practice in most Case Weighting Studies (CWS)44 and in Guanajuato’s “Simulator,” analysts start with data on what operators are already doing, but here an important caveat applies: there is no guarantee that current practices maximize efficiency as work does expand to fill the time. This suggests that any initial calculation requires periodic updating (apparently not done for the UMECAs) and as workloads expand, a search for means to increase productivity, for example by improving distribution and organization of tasks. In any event, PROJUST’s attempt to calculate reasonable workloads and staffing for the states where it operates merits mention. The ET, however, did not find evidence that the UMECAs and other NCJS organizations were based on estimates of the costs for nationwide expansion or that PROJUST’s recommendations related to staffing and resource allocations were implemented at visited UMECAs. Interviewees at all seven UMECAs evaluated agreed that insufficient human and material resources constituted the primary challenges facing UMECAs. In Baja California and Nuevo León, for example, each staffer had, respectively, 400 and 130 cases to supervise. For periodic check-ins (usually not in the UMECA office), supervision is not complicated, although as one UMECA director said, “We might not know for a month that the person had not checked in.” In Monterrey, an interviewee mentioned that six months might go by before UMECA staff was informed. Staff turnover was also an issue, especially in Baja California where the passage of the UMECA from the judiciary to the AGO and then back again resulted in the loss of nearly all PROJUST-trained personnel. Additional problems noted in interviews with UMECA staff (but not always reported in each UMECA) were: • Failure of judges to notify the UMECA as to the imposed condition, which was reported in two of the seven UMECAs evaluated. As one director said, “Sometimes we find out from the defendant when he appears in our office.” As a counterpoint, UMECA staff in Zacatecas took time from their principal activities to attend hearings and so ensure that they had the information. 43 The ET’s general impression, based on interviews and observation in seven UMECAs, is that compliance with pretrial and deferred prosecution conditions is, rather like that for compliance with ADR agreements, based on an absence of evidence to the contrary. That is to say that supervision is often so lax that calculations of compliance rates are frequently based on absence of any evidence that the person did not comply, not on a proactive investigation of his/her behavior. Certainly, for an UMECA whose staff supervises 400 cases at a time, it would be hard to imagine anything very different. Even in Zacatecas where supervisors handled an average of 25 cases, staff said they did not have time to do adequate work, especially since one supervisee might have to comply with several conditions. 44 World Bank (2018) Case-Weighting Analyses as a Tool to Promote Judicial Efficiency: Lessons, Substitutes and Guidance.” Washington D.C. The World Bank. 22 • Difficulty of supervising certain conditions in which travel was required (lack of vehicles) or where it involved the person not doing something (e.g., a restraining order). • Where the condition was some sort of therapeutic treatment, lack of available suppliers. • Occasional failure of prosecutors or defenders to request a “revision” hearing to ascertain whether conditions were met, and, where hearings are held, failure to invite UMECA staff. This was only mentioned in two of the seven UMECAs evaluated, but the ET suspects it is found in others given the often-negative comments from other actors about the UMECAs. • While not a supervision issue, the imposition of some conditions (e.g., requiring two￾hour trips to check in or travel prohibition to the district/city where the person works) could be better adapted to the supervisee’s circumstances. For evaluation, which is the less taxing function at UMECAs, there were still issues: • In San Luis Potosí and Baja California, prosecutors complained that the evaluation quality was so poor they did not use it. This complaint was not voiced elsewhere, but interviewees at each of the seven UMECAs evaluated said they usually based evaluations on a short interview with the defendant in a holding cell and a phone interview with his/her family. No staff mentioned any additional verification of this information, contrary to what is recommended in the official model and protocols. • Differences among UMECAs about whether they could suggest precautionary measures. Most did not, but when they did, no defense council or prosecutors said they took them into account. • Although all seven UMECAs used evaluation formats designed by MSI, interviewees could not explain the reasons behind them. As one UMECA director noted, “If someone asked me why a ‘5’ indicated suitability for pretrial release, I could not explain it.” Analysis, Monitoring, and Evaluation Units: Although the SOW did not require these offices be assessed separately, the ET decided they merited assessment, as the issues are similar in all UAMEs visited, regardless in which entity (courts, prosecution, defense) they were located. Moreover, MSI, as the implementer promoting the UAMEs, has included progress in their installation as one of its achievements. MSI reports that 14 UAMEs have been installed to date exceeding the life-of-project target of eight. Within the installed UAMEs, however, the ET observed a number of issues that if MSI addresses could improve operations and outcomes. UAMEs established in Public Defense (Baja California, Coahuila, and Zacatecas) and in Coahuila’s judiciary consisted of one to three staffers did not appear to completely understand the unit’s purpose or how they should use the statistics they received. In the best of cases (Coahuila’s Defense Office), UAME staff were using Excel sheets provided by PROJUST to consolidate statistical information from other offices, but they could not explain how to use it. Two directors (the only UAME staff in their respective units) described their task as supplying required information to INEGI and other federal institutions. With few exceptions, the ET could not identify a UAME office completely capable of doing basic statistical analysis and monitoring. The few exceptions hinged on the presence of staff predating PROJUST who already were interested in collecting and analyzing data to improve performance. 23 Only in Tabasco did PROJUST help develop the department’s organization, especially as regards a new role for the pre-existing five regional coordinators. As noted by a UAME sub-contractor, there were “limited human resources, absence of personnel adept in analysis, lack of interest by higher authorities, and a failure to analyze integrated information systems to improve institutional performance.” The risk, according to this informant, is that these units would remain largely ignored by higher level authorities responsible for elaborating and overseeing institutional development strategies. While the lack of good information is a problem for all UAMEs, their main limitation is staff quality. Even where UAMEs exist, their staff often do not match the required profiles, and those staff that do match the profiles belong to units predating the UAMEs. For example, staff installed at the UAMES were predominantly lawyers that lacked capacity to analyze the information they receive. With better or better-trained personnel, even basic information (e.g., Excel tables from various offices) could be used to good effect, but this was not happening. Staff turnover is also a persistent problem. Given that agency heads are ultimately responsible for selecting UAME staff, the observed staff deficiencies suggest that there are deficiencies and/or constraints at this level that also need to be addressed. 45 Empowerment of Civil Society Organizations to promote the New Criminal Justice System: This area has no corresponding indicator, nor is it mentioned in the objective statements for the initial contract or the March 2017 modification. Notwithstanding, USAID’s current results framework does include it as a lower level result—“participation of civil society in monitoring role increased”—a function addressed by both MSI and TetraTech DPK. Both contract versions and the results framework also refer to measures to improve information to citizens. Before 2016, MSI worked with national and local CSOs to draft and promote new legislation in the majority of states. This information was provided to the ET, but absent contacts with these groups (not provided through PROJUST, but also not requested because USAID’s emphasis was elsewhere), the extent of this cooperation and its effects on “empowering” local-level CSOs could not be verified. After early 2016, both implementers have worked with CSOs in a variety of fashions. MSI, for example, has invited the inclusion of both CSO and private sector associations in the 100 Days exercises. Except for Nuevo León, those that remained most active once the areas of focus were selected tended to be entrepreneurial organizations or the CSOs they sponsored (as in Baja California and Chihuahua). Active participation seemed to hinge on the topic chosen; where it did not interest local CSOs, they appear to have dropped out. TetraTech DPK’s involvement with CSOs took other forms, although until recently this has been with CDMX or Monterrey-based organizations with a national reach. National organizations worked with the public education program (not evaluated here), but the CDMX￾based Creativeria Social and Renace (Monterrey) were contracted to work respectively on the creation of CSO networks to support UMECAs and to develop citizen observatories. Finally, TetraTech DPK recently introduced a small grants program intended for state-based CSOs. As 45 Subsequent to an initial draft of this report, MSI provided additional documentation to the ET related to its work with UAMEs and its outcomes. In the ET’s judgment, this documentation supported the ET’s findings related to a lack of institutional leadership and staff turnover at the UAMEs. This documentation, however, did not address the issue of staff quality, which the ET considers to be the primary challenge facing UAMEs. 24 the program is still being developed, what will be financed is unknown. CSOs in Baja California and Coahuila said they attended the introductory meeting in CDMX and were interested in applying for grants. Except for the associations linked to business groups in Baja California, Chihuahua, Coahuila, and Tabasco, and two already empowered CSOs in Nuevo León, the ET found no evidence that PROJUST actions had empowered local CSOs, except to the extent that they encouraged already empowered private sector organizations to engage in reform issues. The citizen observatories interviewed in Zacatecas (gender), Tabasco, Chihuahua, Nuevo León, and Coahuila did not reference the Renace program or any change to their operations as a result of either PROJUST implementer. As summarized by two very active CSOs in Monterrey, there are several factors working against success elsewhere. First is the shortage of CSOs in many states and/or in their internal districts. Second is the CSOs’ limited understanding of the criminal justice system. Many have a cause they want to promote (e.g., due process or gender rights), but they are not trained to work within the system. This second point may also apply to groups formed or financed by business interests, but here involvement in Rapid Results has provided knowledge on which they can now act. 4.1.2 Use of PROJUST-Supported Models, Protocols, and Standard Operating Procedures The entire set of PROJUST models, protocols, and SOPs was, initially, widely used after its introduction; some of these continue to be widely used and are well-regarded by members of state institutions, but the use of others has declined significantly over time. According to the stakeholder survey, those that continue in use were rated as second in importance among PROJUST’s activities behind short-term training and workshops and ahead of Rapid Results, UMECAs, and ADR (see Tables 6 and 7). Table 6: Most Important PROJUST Activities by Institution Activity All Public Defense Public Ministry Judiciary Others Short-term training 103 14 26 17 46 Help with models, protocols, and SOPs 87 9 19 16 43 Meetings and national workshops 49 8 7 10 24 Tools for M&E 42 8 5 13 16 Rapid Results 37 1 8 7 21 Pretrial detention programs and UMECAs 33 4 8 2 19 ADR 32 5 11 4 12 Other 45 7 9 3 26 Source: Analysis based on answers to question 4, “Which PROJUST activities do you consider most important?” Note: Respondents were given three opportunities to answer so the answers sum to more than 150. Table 7: Most Important PROJUST Activities by State46 Activity All SLP Zac. Tabasco BC Chi Coahuila Nuevo León 46 Abbreviations used here are SLP (San Luis Potisí), Zac (Zacatecas), BC (Baja California), and Chi (Chihuahua) 25 Activity All SLP Zac. Tabasco BC Chi Coahuila Nuevo León Short-term training 103 14 17 10 12 13 18 19 Help with models, protocols, and SOPs 87 13 12 10 9 9 13 21 Meetings and national workshops 49 7 7 5 4 5 12 9 Tools for M&E 42 3 8 1 5 5 11 9 Rapid Results 37 0 0 8 6 9 3 11 Pretrial detention (UMECAs) 33 2 5 5 6 5 2 8 ADR 32 5 4 4 4 5 3 7 Other 45 6 3 7 2 4 7 16 Source: Analysis based on answers to question 4, “Which PROJUST activities do you consider most important?” Note: Respondents were given three opportunities to answer so the answers sum to more than 150. However, the ET thinks the more important questions are not whether but how the instruments are used and with what results, and thus it focused on those issues in its fieldwork. As MSI’s 2016 internal training evaluation concluded, written guidance alone is insufficient to produce desired results. In fact, as seen in Table 8, a much smaller number of survey respondents indicated that the PROJUST models, protocols, and SOPs positively impacted their performance. Table 8: Reported Impacts of PROJUST Activities on Individual Performance Type of Activity or Effect Mentioned All47 Public Defense Public Ministry Judiciary Other Institutions Total responses 158 22 53 26 57 Training and technical assistance 29 4 7 2 16 Efficiency, efficacy, and better results 26 3 13 5 5 Models, protocols, and SOPs 10 2 3 1 4 Organization 9 1 2 2 4 Coordination 8 1 3 0 4 Accompaniment 4 0 1 1 2 Better quality of work 4 0 3 0 1 More rapid work 3 0 1 1 1 Help to train others 2 0 1 1 0 No answer 2 1 0 0 1 Application of good practices 1 0 1 0 0 Other 60 10 18 13 19 Source: Analysis of responses to question 13 “How did PROJUST activities improve your work?” Note: Respondents usually listed the impact but a few stressed the activity. In addition to the stakeholder survey, the ET assessed around three dozen models, protocols, and SOPs developed jointly by PROJUST, SETEC, and the Secretariado (SETEC’s successor).48 47 Numbers sum over 150 as some respondents chose two categories. 48 The ET identified a few exceptions, most specifically the model for ADR departments based on the very effective example from San Luis Potosí, and the UMECAs, as based on a successful pilot in Morelos, However, as elaborated above, the Morelos experience may not have been an adequate basis for the expansion of the UMECAs’ coverage to a much larger adult population. 26 According to key informants, many of these are favored by federal authorities (SETEC, Secretariado, and others) as a means of standardizing operations and for introducing best practices, although others are less useful or not appropriate for the Mexican context. For example, no interviewee indicated that he or she used the bench books or self-teaching guides in the intended manner as doing so took more time than the interviewee had to spare with little visible pay-off. In response to this finding, PROJUST (MSI in particular) said the models, protocols, and SOPs were based on meetings of knowledgeable Mexican experts and in some cases on models from other countries.49 However, the team found that this has not always been the case. In particular, documents provided to the ET did not indicate a Mexican or external example serving as a model for the UAMEs or CJMs, although in the latter case, the model has proven to be reasonably successful. The ET further found that models were at times limited to flowcharts of processes or more academic documents50 with the former in cases honored in the breach (as one CJM director, with model flowcharts posted on the interview room walls, said, “We of course take short-cuts”) and the latter providing at best a discussion of the philosophy behind the operations of an office or entire institution. Models for Homicide Units, UMECAs, UAMEs, and CJMs included organization charts and position descriptions, although they typically featured single designs. Those for UMECAs, UAMES, and CJMs also listed other basic resources required by these units. Unfortunately, the specified resources had no visible empirical basis, whether as estimates of reasonable workloads, costs, or likely demand. The most influential model is clearly Tres Pisos, which was cited as a useful project activity by prosecutors and many others in the four states where it has been implemented. The protocols and SOPs reviewed were typically either checklists or condensations of legal requirements written in plain language, especially for investigations. These tools may be more helpful than the models, although except for homicide units in Baja California and Tabasco and most UMECAs, few interviewees spoke of using them in their work. Some protocols and SOPs, like those for first responders and police detentions, are honored in the breach. Key informants widely agreed that PROJUST’s most effective (or at least most appreciated) work combines protocols with training and accompaniment, either as a several-month presence to work with special units or repeated visits to see how the target office is responding. Moreover, in the newly reintegrated states of Baja California, Chihuahua, and Nuevo León, institutional members frequently mentioned the need for protocols and SOPs that are tailored to local circumstances, not a uniform product. Low demand for uniform criteria is reflected by the responses to another survey question asking about the benefits of federal government support. As seen in Table 9, standardized criteria received a very low score relative to budgetary support. Table 9: Perceptions of the Impact of Federal Government Support 49 This is a common practice in designing new codes and institutions, seen throughout Latin America and in other transitional and developing regions. However, as critics have mentioned, a failure to “contextualize” the practices frequently leads to their adoption in circumstances that do not support them. This is another example where the use of external consultants (and not just visits to other countries) might have aided in identifying the conditions needed for successful adoption. 50 These documents were similar to manuals published under the previous Justice and Security Support Program (JASP). See, for example, MSI’s 2012 publication “Manual on How to Create a Women’s Justice Center” and its 2014 publication “Manual de Coordinación Entre Ministerio Público y Policía.” 27 Impact/Perception All Public Defense Public Ministry Judiciary Other Institutions Is the support of federal government important? Yes 143 19 34 24 66 No 2 1 0 0 1 No answer 5 1 0 0 4 Reasons for its importance: Economic resources and budgetary support 115 15 30 22 48 Standardized criteria 9 2 1 3 3 No answer 7 0 1 0 6 Other 41 6 7 3 25 Source: Results of Questions 11 and 11b of the questionnaire, by institution. Unfortunately, the ET did not have time to check internal operations against the models, protocols, and SOPs guiding them and so had to rely on what personnel said they were using, as well as a few random checks of observed practices against some of the instruments’ content when the opportunity allowed. A final issue indirectly related to the use of written guides is the effect of frequent personnel turnovers and reassignments within targeted institutions on all types of assistance. This finding was emphasized in MSI’s training evaluation as well. Examples include the loss of trained prosecutors in Chihuahua and Nuevo León and assignments to UMECAs and ADR divisions in Tabasco based on who was available and which changed with each new administration. Under these conditions, models, protocols, and SOPs are less likely to be effective because of the constant need to reintroduce concepts and methods to newcomers. 4.1.3 Conclusions In each of the four states where PROJUST introduced more comprehensive programs, there have been important changes in the reception and channeling of complaints, treatment of complainants and women victims, use of ADR, programming of hearings, inter-institutional coordination to facilitate processing of specific cases (and especially those targeted by Rapid Results), and efforts by all supported institutions to quantify their performance, even if only as indicators. States (Baja California, Chihuahua, and Nuevo León) where USAID and/or PPROJUST assistance temporarily lapsed have tended to take a more selective approach to PROJUST offerings, but all want more work with Rapid Results. Compared to the full scope of criminal justice sector reform issues, project achievements have been largely incremental in nature, but they do demonstrate the utility of the mechanisms introduced by the project. In the approximately two years following USAID’s new instructions to the project implementers, PROJUST has contributed to important advances, in addition to consolidating its approach to address basic problems. One could thus characterize the period covered by the evaluation as one of testing the means to put the basics in order. Once that goal is reached, there is the further challenge of how to address more effectively the issues that concern most citizens—successful investigation and prosecution of high impact crimes or just the quick and effective processing of common crimes presumably best resolved by conditional suspension, plea bargaining, or a return to attempted mediation. 28 Specific conclusions related to each of the sub-questions related to project effectiveness are presented below. To what extent has PROJUST technical assistance contributed to results, as measured by approved indicators and qualitative data? In terms of the approved PROJUST performance indicators, the changes in indicator values for the years recorded indicate a general improvement in institutional performance in all six of the approved indicators being tracked by PROJUST at the time of the evaluation, although with moderate to significant variation across time and states: pretrial confinement rates (Indicator 1), prosecution rates (Indicator 2), compliance with precautionary measures from pretrial services (Indicator 3), plea agreements (Indicator 4), case disposition rates (Indicator 6), and percentage of cases with sentences greater than three years (Indicator 7). In addition to this, PROJUST met or exceeded performance targets for all non-approved indicators collected by the project prior to the adoption of the new set of approved indicators in November 2017 and which included the number of laws passed, states’ conformity with the SETEC rating system for legal and organizational advancements, number of SETEC M&E initiatives directed toward criminal justice reform, and the number of trainings completed. Finally, the justice system has improved its treatment of detainees, as perceived by detainees, as a result of the justice system reforms enacted with PROJUST support in terms of the presence of judges, recording of hearings, clarity of proceedings, public presence at proceedings, and the coercion of confessions. Thus the evidence indicates that PROJUST interventions did contribute to the observed improvements in institutional performance as measured by both approved and non-approved performance indicators. This conclusion supports a further conclusion that the interventions implemented by PROJUST merit wider consideration and, possibly, wider implementation. This conclusion, however, includes the caveat that the approved indicators suffer from several limitations in that they include a number of global indicators that are affected by multiple other actors within the justice system, include indicators with data only up through 2016, include indicators for which the results or states measured change from year-to-year, and include indicators that do not necessarily measure what they are intended to measure (e.g., do not capture important dimensions of the underlying result). Another issue related to training is that, while the evidence indicates that the training has, on the whole, contributed to improved functioning at project supported justice institutions, this is less true for less specialized forms of training relative to more specialized skills training. Also, the evidence indicates that supported institutions remain reliant on PROJUST, or other external organizations, to train their staff with few instances of institutions taking responsibility for continuing training on their own in the project’s absence. PROJUST continues to train justice system actors and (according to its FY 2019 workplan) will extend its training local training institutes. It is a concern, however, that training participants continue to request more MSI training rather than assist their own programs. A recent initiative mentioned in Zacatecas, to train and certify local trainers may be aimed at this goal, but even the director of the judicial training school there was not aware of how it would be linked to his programs. 29 Nonetheless, when coupled with the quantitative evidence from approved performance indicators, the qualitative evidence supports the conclusions PROJUST did contribute at the margin to improving the performance of targeted justice system actors, although the extent of the contribution cannot be determined. In terms of the specific justice system actors evaluated by the ET, the following conclusions can be deduced from the evidence presented in the findings section of this report. In each of the conclusions reported below, the ET found PROJUST to have played anywhere from an incremental supporting role to a primary contributing role. AGOs/Public Ministries: Within the Public Ministries, prosecution rates (Indicator 2), plea agreement convictions (Indicator 4), and sentences over three years (Indicator 7) generally improved. However, the variation across time and state, and the short time-frame for measuring change, do not allow a conclusion one way or another about the expected future trend of these results. The Tres Pisos model has had a significant effect on how cases are being processed at Public Ministries in Coahuila, San Luis Potosí, Tabasco, and Zacatecas, particularly in terms of reception, filtering, and channeling of cases within the model’s first piso. Improvements in how Public Ministries process cases within the model’s second piso (ordinary crime) and third piso (high-impact crimes) have yet to emerge to any significant degree. Cases channeled to those levels tend to stagnate due to a variety of factors thus leaving the majority of cases stuck in the system and unaddressed. PROJUST has yet to create/implement a plan for addressing cases channeled to the second piso or third piso. UATs have evolved from simple reception centers to entities that are channeling cases to other offices in a reasonably systematic manner, albeit with substantial room for improvement in how they operate and, in the efficiency and effectiveness of their filtering process. The Rapid Response (or 100 Days) exercise has been a success in expediting the advancement of investigations from Public Ministries to the courts or some other form of resolution with all but one of the eight target states meeting or exceeding their 100 days goals and each of the eight states processing significantly more cases than prior to the start of the exercise. Rapid Response has had the additional effect of demonstrating to states what is possible via collective action and pointing to additional areas for justice system reform. Nonetheless, Rapid Response should be seen as only a start, as the volume of cases that require processing swamps by a large amount the number of cases actually processed, even at the accelerated, expedited rate achieved under Rapid Response. In the ET’s judgment, however, trying to address all of these issues at once would have been a mistake and would not have produced such basic achievements that the project has produced. PROJUST’s methodical, bottom-up approach to management and organizational issues, as exemplified in the Rapid Results exercise, is unique among USAID projects in Latin America and deserves consideration by others. Finally, PROJUST activities have contributed to notable improvements in how Public Ministries’ ADR services function, particularly as measured by the percentage of criminal complaints resolved through the ADR process. Nonetheless, certain functional areas remain weak at the ADRs and require ongoing assistance (being provided by MSI), particularly in terms of providing notifications in hearings and the identification of suspects for cases brought to the ADR. 30 The Judiciary: The one indicator used to assess the performance of the judiciary, Indicator 6 or the percentage of case dispositions in control courts, provides mixed evidence of improved institutional performance, although the omission of trial courts from the indicator’s results in three states makes a more definitive conclusion about institutional improvements in case dispositions impossible. PROJUST’s Rapid Results exercise contributed to an acceleration in courts’ hearing schedules in all but one target state leading to an increase in the number of hearings held and better management of judges’ schedules so as to facilitate a quicker and increased case disposition. Resistance among judges to the increased pressures of expedited case disposition poses a potential obstacle to extending the process to a broader category of cases; however, so far it has not been sufficient to impede the process, and PROJUST is moving forward to scale-up the process. PROJUST support in developing and implementing protocols for conducting preliminary hearings and in creating an institutional diagnostic tool to measure progress in this area (modelo para exelencia en gestíon) have so far made limited inroads with the former awaiting examples of operationalization and the latter suffering from a lack of agreement as to the assessment criteria. Centers for Women’s Justice: PROJUST assistance to CJMs contributed to a notable improvement in the functioning of CJMs and to the quality and diversity of the services they offer. CJMs, however, continue to experience a number of limitations, which are largely outside of project control. These include limited public access due to their centralized locations and limited funding to support physical infrastructure, operating expenses, and the expansion of services to less centralized locations. The project’s model to certify CJMs focuses on structures and processes with comparatively little focus on outcomes and has proven onerous to implement in practice. It is still too early to determine how effective of a tool it is. Units for Precautionary Measures: PROJUST played a key initial role in promoting UMECAs, which have since, due in part to the project’s promotion, become required and regulated by federal law. Approved indicators measuring the performance of UMECAs—Indicator 1 (pretrial detention rate) and Indicator 3 (compliance with pretrial measures) indicate that UMECAs have contributed positively to improvements in both, although their contribution varies over time and by state. Notwithstanding evidence for their positive contribution to improved pretrial detention rates and pretrial compliance, UMECAs face a number of challenges. Primary among these is the lack of evidence found by the ET that that the UMECA model developed and implemented by PROJUST was based on reliable pre-implementation estimates of demand, caseloads, or budgets. Other challenges include supervision difficulties for certain conditions, failures by prosecutors or defense counsel to request revision hearings, a lack of available suppliers, poor evaluation quality, failure to suggest or consider precautionary measures, poorly valued guidelines for pretrial release, and limited resources to supervise defendants on pretrial release. As currently done, moreover, the UMECA’s evaluation of suitability for conditional pretrial release adds little value-added to what a good prosecutor or defense lawyer needs, for which reason it may often be ignored. Supervision, however, is the UMECAs’ unique task. Unless TetraTech DPK can develop NGO networks to do this (with possible assistance from the 31 police), or the resources are augmented for it, supervision will remain spotty, thereby raising the risk that “supervised” defendants will do exactly what the UMECAs were designed to avoid: not appear at hearings, tamper with evidence or witnesses, or continue with criminal activities. Analysis, Monitoring, and Evaluation Units: PROJUST directly contributed to installing 14 UAMEs, exceeding the life-of-project goal of eight. While the installed UAMEs continue to operate, their functioning tends to be poor as a result of variety of systematic weaknesses found across the UAMEs, including poorly qualified, trained, and motivated staff; limited vision as to the UAME’s purpose; and a lack of basic statistical analysis skills. In addition, the evidence points to a lack of interest in the UAMEs, or alternatively a lack of understanding of their roles and importance, among higher judicial sector authorities, a conclusion buttressed by the fact that those authorities responsible for hiring UAME personnel have routinely staffed them with underqualified persons who do not fit the established hiring profiles. Empowerment of Civil Society Organizations to promote the New Criminal Justice System: Unfortunately, the ET was unable to find evidence as to PROJUST’s contribution to empowering local CSOs to promote the NCSJ. In operational terms, CSOs that have shown the greatest interest in working with PROJUST on promoting the NCSJ tend to be either entrepreneurial organizations or the CSOs they support with the level of support depending on the topic chosen for engagement. Barriers to greater CSO empowerment and engagement in this area include the shortage of CSOs in both states and their internal districts and the existing CSOs’ limited understanding of the criminal justice system. Whereas there are CSOs that want to become involved in this area, most are not trained to work within the system. Notwithstanding, those CSOs that participated in the Rapid Results exercises, particularly those with a business or entrepreneurial purpose, now presumably have greater knowledge about the system on which they can act. Use of PROJUST-supported models, protocols, and standard operating procedures: Most of the models, protocols, and SOPs developed with PROJUST support remain in use and are both widely-recognized and well-regarded by justice sector actors. Notwithstanding their relatively widespread use, there is insufficient evidence to conclude that PROJUST models, protocols, and SOPs have had a positively impact on sector actors’ performance. The exception is the Tres Pisos model, which, as documented in this report, has made an important contribution to how cases are processed within the system, at least within the first piso. Aside from Tres Pisos, key informants are generally unable to articulate just how they have used the models, protocols, and SOPs in their work. On top of this, high staff turnover within targeted justice sector institutions means the loss of institutional knowledge related to the models, protocols, and SOPs and the need to undertake constant reinforcement of related concepts and methods. Furthermore, the demand among justice sector actors for models, protocols, and SOPs that are more tailored to their circumstances is a valid expectation that, if met, would likely contribute to their increased use and impact; however, the practical implications of this demand present resource and operational challenges to PROJUST that it must decide how to address moving forward. The range of models, protocols, and SOPs developed with PROJUST support serve multiple purposes, not all of which are appropriate for the Mexican context, and some of which (e.g., bench books or self-teaching guides) add little value to sector actors. To develop the models, protocols, and SOPs, PROJUST evidently drew on knowledge, experience, and models both 32 inside and outside of Mexico; however, specific sources for their development are not always evident. The formats of project-developed models, protocols, and SOPs—models tend to consist largely of flowcharts or academic documents, while protocols and SOPs tend to consist of checklists or summaries of legal requirements—have tended to discourage their usefulness to justice system actors, particularly when not accompanied by ongoing assistance. In contrast, the models, protocols, and SOPs have been most effective when combined with training and accompaniment consisting of ongoing or periodic monitoring and reinforcement. 4.2 SUSTAINABILITY51 To the extent there are results, how sustainable are these results in the institutions (AGOs, Courts, CJMs UMECAs) and CSOs referenced above? FINDINGS 4.2.1 What has PROJUST done to promote local ownership (i.e., that the priorities are locally owned, locally resourced, and using rather than supplanting local systems to sustain results? PROJUST’s work with justice sector institutions has promoted their ownership of the reforms, especially through mechanisms like Tres Pisos and Rapid Results. Local stakeholders outside the sector have been invited to participate in Rapid Results and, when they do, their contributions have been positive, especially when private sector associations are included. This has widened local ownership of the reforms. Still missing, however, is strategic involvement of the rest of state and local government, especially the governor and the legislature, who are the major sources of local funding. PROJUST’s (especially MSI’s) work with state governments was both more extensive and intensive during the earlier law drafting exercises. The ET spoke with few local participants from that period, but interviewees in San Luis Potosí, for example, said that MSI helped pressure the executive and legislature for passage of the critical laws before the mid-2016 deadline. More recently, however, involvement with these state bodies has dropped off, and when it occurs (as reported by PROJUST), it largely responds to specific issues as they arise in each state. 52 Presumably MSI’s local coordinators could take steps toward more systematic engagement, but with the disappearance of the “mini-SETECs,” contact has been less frequent and relatively unproductive. PROJUST faces a couple of obstacles in promoting local ownership. First, outside of state and local governments, the remaining coordinating or implementing bodies are, for the most part, restricted to working with public security (police) and the Public Ministry. Second, their most important reform-related function is to apportion federal monies provided through the Contributions Funds for Public Security (FASPs), especially when they are located in the state Secretariats of Public Security, as most are. These are political decisions determined by the 51 In the following discussion, the ET defines sustainability as the persistence of results and the mechanisms used to produce and improve them beyond the project’s end. 52 According to USAID, PROJUST met “with the governors of Chihuahua, Coahuila, and Tabasco in the last year and the Secretary General of the Nuevo Leon government to deal with specific issues. They also dealt with the Chihuahua Congress on the budget for UMECAs, with the Zacatecas Congress on the new law of the fiscalia, and with the Tabasco Congress on changes to the CPC. In the context of Rapid Results, they have worked with municipal public security officials in Mexicali, Tijuana, Chihuahua City, Zacatecas.” 33 priorities of the state executive and, as an interviewee in Zacatecas noted, “reform lacks political value” (plusvalía). It should be noted that PROJUST’s mandate does not extend to police, although it has made exceptions for the Ministerial Police (those within the Public Ministry), and in some training and technical assistance, it has promoted better coordination between prosecutors and the ordinary police forces. The underlying issue is that whenever local stakeholders focused on justice reform-related issues, they rarely did so in a coordinated fashion or with a single vision. State governors and legislatures did influence or make decisions on appointments, a situation undergoing change, but apart from that, their interest in justice sector reforms was minimal. When uncoordinated, the results of reform efforts were usually not good. For this and other reasons, justice sector institutions traditionally operated on their own, rarely seeking to coordinate with each other. Judges’ protection of their formal independence has produced further resistance to intra-sector planning, as reflected in Tabasco’s difficulties in getting the judiciary to buy into locally￾developed protocols to standardize hearing criteria and in the reluctance of court administrators three states to adopt PROJUST recommendations about programming hearings. Significantly, when judges do take the lead, they can be creative. In both Guanajuato and Coahuila, judges were inventing ways to speed up preliminary hearings, supported by their general managers (see Box 5). Box 5: Some Interesting Judicial Innovations Although judges are often considered the most conservative members of the judiciary, the ET found that they were sometimes the most willing to innovate, even beyond what PROJUST had recommended. For example, in both Coahuila and Guanajuato, first instance judges went beyond PROJUST’s recommendations on block programming to introduce mass control hearings on pretrial detention in which all pending cases were convened and decided within a single morning. Individual defendants were heard separately, but each hearing was reasonably brief. In Coahuila, judges were assigned to any pretrial hearing, as opposed to a former system (still in effect elsewhere) that gave a judge responsibility for all pretrial events for a specific case. Similar procedures had been adopted in Guanajuato, San Luis Potosí, and Zacatecas. Judiciaries in Coahuila and Guanajuato (in the former case building on PROJUST support) are also tracking variables like numbers of hearings cancelled and length of preliminary hearings. In Coahuila, the current Court President keeps a list of cancellations and time overruns and personally queries judges about them. Both states have appointed general administrators who are not lawyers, a radical break with tradition, and both have or are introducing case tracking systems that generate statistics their administrative offices analyze to monitor performance. PROJUST has sensitized judiciaries to the importance of tracking performance, but these courts reported these innovations as their own additions. The situation of other justice sector institutions also influences their willingness to engage more systemically. Depending on what has happened locally, the Public Ministry is either part of the executive or enjoys a semi-independent status, while Public Defense occupies a variety of organizational locations. As for the other sector institutions (e.g., UMECAs, CJMs, legal advisors to victims), their position and resource endowments vary widely from state to state, and while UMECAs are included in all Rapid Results exercises, their contribution varies as well. In San Luis Potosí, Guanajuato, and Zacatecas, there was a reported coalition behind the reforms that also included the current or past governors. However, this is an unusual situation. 34 To the extent local ownership implies funding, Guanajuato was the only significant positive case.53 The Guanajuato state government funded a massive training program from 2008 onward, and sector salaries, which can only be financed locally, are at least twice the level of the other states visited. In Baja California, Chihuahua, and Nuevo León, business groups had financed special initiatives or indicated they were willing to do so, but there were no other examples of state governments currently self-funding justice sector reforms. Moreover, in Chihuahua, under a past governor and attorney general (AG), prosecutors’ salaries were reduced by half, leading to a massive exodus from the Public Ministry and a loss of trained personnel. Prosecutors’ salaries were also cut in Coahuila but have since returned to previous levels. While justice reforms should conceivably attract active engagement by a wider portion of the local population, this has not been the case in Mexico. Problems include citizens’ limited contact with (or even avoidance of) the justice sector, a lack of agreement as to what to expect of it, and the overwhelming mistrust of the police. Moreover, the concerns of those who do engage are often different. As one interviewee in Zacatecas commented, “I am in agreement with due process guarantees,” but the rest of the private sector is most concerned about security (e.g., the rise in kidnappings) and corruption. In Coahuila, for example, representatives of a private￾sector funded CSO, Consejo Cívico de Instituciones de Coahuila, 54 stressed that because violent crime rates had dropped, entrepreneurs’ major concern was corruption. Moreover, in states like Coahuila, much private investment comes from international corporations that are reluctant to get engaged in local politics. TetraTech DPK’s public education programs are largely focused on poor citizens. While potentially reaching a broader audience than the niche group supporting due process, the poor majority is both unorganized and has less clout than the private sector, whose interests lie in other directions. Moreover, so far as the ET could tell, these public education programs do not emphasize the progress made with MSI’s assistance to sector institutions, possibly a missed opportunity for building support for these efforts. 4.2.2 PROJUST promotion of sustainability Promoting the sustainability of PROJUST’s impact entails a number of challenges that vary by reform area and might need different mitigation measures. While there are a few dramatic cases of backsliding in both Chihuahua and Nuevo León (although on results often predating the project), the project continues to move forward. Sometimes by law and sometimes by the acceptability of the message (e.g., Tres Pisos’ emphasis on prosecutorial discretion and alternative solutions as adopted in the AGOs), some changes seem firmly cemented. It would take a major countermovement, for example, to eliminate the CJMs, the public defenders for indigent clients, and the use of ADR. For these activities, the challenge is less sustaining what has been accomplished than developing ways to encourage expansion (see Box 6 on gender). 53 Coahuila’s state government did finance the courts’ very expensive information technology (IT) system, but, as noted, it also tried to cut prosecutors’ (and defenders’) salaries. In Zacatecas, under a previous governor, the principal impact was the construction of a huge judicial center (part of a still larger administrative center) several miles outside of town, and with problems of its own (e.g., limited public transport to or within the center, poor signage, unheated buildings in which secretaries often wore gloves to fend off the cold). A similar center was planned for Chihuahua, but was never built. 54 See https://www.facebook.com/consejocivicodeinstitucionesdecoahuila 35 Box 6: Gender, Sustainability with Some Caveats Mexico has an abundance of mechanisms to combat gender violence and help women victims. Given their political support within and outside the country, they are unlikely to disappear. Because gender is a highly politicized theme, service provision has not been planned strategically. Units, programs, and laws respond to the demands of active lobbies but frequently introduce overlapping or redundant services in some locations while neglecting them in others. While PROJUST’s two implementers assist many of these programs (e.g., training, manuals, and accompaniment), insufficient resources, staffing issues, and inadequate coordination remain as obstacles to their efforts. The mechanisms’ geographic coverage is still limited, and no one has estimated the costs of expanding it. Prosecutors, judges, and defenders serving outlying districts can be and have been trained to deal with gender cases, but the additional services (e.g., CJMs or separate legal assistance to victims) are usually not present there. This situation cannot be attributed to PROJUST, but there may be things it or a later program could do to encourage the GOM to address it. Still as mentioned in an earlier section, PROJUST assistance in improving the quality of the most firmly cemented innovations could be affected by the not-infrequent changes of leadership and staff. Where this has happened, it appears less motivated by opposition to or lack of interest in the programs but simply because there is a position that might go to a favored ally with no objection from within or outside the justice sector. Project accomplishments have not been sufficiently publicized so as to generate significant broad-based opposition to them. Other PROJUST-induced improvements—such as the filtering function of the AGO’s intake centers (UATs), expedited hearing schedules adopted by courts, reductions in pretrial detention,55 and justice actors’ greater use of abbreviated procedures and conditional suspension—could be reversed for a variety of reasons. These include changes in institutional leadership, a new state executive with different priorities, or simply the resistance of institutional actors, such as judges who feel overworked, prosecutors who disagree with judges’ criteria for pretrial detention, or litigators who oppose conciliatory solutions. While PROJUST’s Rapid Results exercises have produced impressive results, they often rely on an unusual degree of effort from participants that is unlikely to be sustained over the long-term, and in some instances has already fallen. The same caveat applies to the improved inter￾institutional coordination elicited by the 100-day goal—whether it can be maintained for the type of cases targeted and more importantly extended to others. There are, moreover, areas where more progress is needed before sustainability even becomes an issue. These include Tetra Tech DPK’s efforts to link CSOs to the supervision tasks of the UMECAs (only feasible in states and districts with sufficient and sufficiently diverse CSOs); the entire UMECA project, for which only Nuevo León provides sufficient resources for it to function adequately;56 and TetraTech DPK’s work on enhancing the sector’s transparency and communication with citizens. As noted above, this last area is also critical to sustaining advances in others and might be directed to this end as well. 55 Already opposed by the National Confederation of Governors (CONAGO) because the reforms are putting “dangerous criminals” back on the streets. Members of CONAGO, with only one exception also support the need to use military forces to combat organized crime. 56 Apparently, Chihuahua may soon have sufficient UMECA staff, but this surmise is only based on MSI’s training of a new group (with real needs still seemingly not estimated). However, even in Nuevo Leon where staffing has been increased, UMECA personnel interviewed said they had 400 cases each to supervise and thus could not do all that was required to track their compliance. 36 One final impediment to PROJUST’s sustainability is the existence of a dependency relationship within the states where it works. This hinders local ownership and sustainability in two ways: perpetuating reliance on PROJUST in defining needs and limiting the incentive to find local solutions. For example, while interviewees consistently emphasized the need for more training from PROJUST, none asked that his or her own programs be taught to provide it. When the ET asked interviewees what would happen once PROJUST ended, those in the states receiving the most comprehensive programs were universally unable to answer the question. The large majority of interviewees in San Luis Potosí and Zacatecas could not imagine PROJUST’s disappearance, with some stating that without PROJUST, they could not “continue the reforms.” In contrast, Coahuila’s judiciary and Tabasco’s Public Ministry did have their own programs; here institutional leaders were grateful for PROJUST support but were more confident about carrying them out on their own. However, this was not the case for interviewees in UAMEs and UMECAs, even in non-comprehensive states. This may be partially a function of the general lack of alternative funding for the functions performed by PROJUST (with the exception of Guanajuato), and it again suggests the need for promoting more state government involvement in the reform process. Three interrelated items on the stakeholder survey addressed issues about post-PROJUST activities. The first asked whether states were doing something to continue the reforms, with 121 respondents saying they were doing something, 12 saying nothing was being done, and 14 not responding. A second question asked whether PROJUST’s activities should be continued with the states’ own resources. This question yielded 128 positive answers, 12 negative answers, and nine non-responses. The third question asked what states and institutions were doing to further the reforms on their own post-PROJUST (see Tables 10-11). Most respondents simply said they would continue with PROJUST, with only three respondents mentioning the need for other resources or funding. Because this was an open-ended question, there were 121 different answers, which the ET consolidated into a smaller number of categories.57 Table 10: Answers to Open-ended Question: “Can you specify what your institution is doing to continue the reform on its own (by state, post-PROJUST)?” Answers All SLP Zac. Tabasco BC Chi Coahuila Nuevo León Totals 121 19 16 14 14 14 23 21 No answer 3 1 0 0 0 0 1 1 Training/certification 22 7 1 1 2 5 4 2 Tres Pisos and other models or protocols 11 0 3 3 1 0 2 2 Apply recommendations and methods 37 3 5 3 5 3 11 7 Rapid results 11 1 1 3 5 1 0 0 Management and 8 1 3 0 0 1 2 1 57 The entire list in Spanish is found in Annex 7. To avoid having some 400 answers, considering those that included several categories, the ET picked the first mentioned or that given most emphasis in each response. 37 Answers All SLP Zac. Tabasco BC Chi Coahuila Nuevo León indicators Coordination 22 2 3 3 1 4 2 7 Identifies new needs 7 4 0 1 0 0 1 1 Table 11: Answers to Open-ended Question: “Can you specify what your institution is doing to continue the reform on its own (by institution, post-PROJUST)?” Answers PD PM Courts UMECA Victim Units CSO CJM Other58 Totals 19 30 20 11 14 10 5 12 No answer 0 1 1 1 0 0 0 0 Training/Certification 4 4 3 6 1 2 1 1 Tres Pisos and other models or protocols 2 6 2 0 0 0 0 1 Apply recommendations and methods 4 11 7 3 9 2 1 0 Rapid results 1 3 0 0 1 0 2 4 Management and indicators 2 0 2 0 1 0 1 2 Coordination 5 5 3 0 1 4 0 4 Identifies new needs 1 0 2 1 1 2 0 0 Tables 10 and 11 highlight interesting differences between states and institutions, even considering the varying number of respondents for each. Even considering the different numbers of respondents in each state, Coahuila and Nuevo León stand out for their lower emphasis on training/certification and no mention of rapid results and for their greater emphasis on applying recommendations and methods. San Luis Potosí’s higher incidence on additional needs may respond to issues of resistance from judges and the court administrator to aspects of the PROJUST project. It may simply be too early to expect states to think beyond PROJUST. Most of the answers in Tables 10-11 came from the rank and file, rather than institutional leadership, and their responses do not necessarily reflect official policy. However, in separate interviews with institutional leaders, none of the target states except Coahuila (judiciary) and Tabasco (Public Ministry) mentioned any plan for what would follow PROJUST. The ET noted examples where PROJUST could do more to strengthen institutional capacity as a precursor for sustainability. For example, PROJUST could do more to integrate its training courses into institutional training programs. In the Zacatecas Public Defense, which has no institutional training program, MSI could suggest arrangements with the courts’ judicial school, 58 The category “other” groups representatives of the private sector (3 responses); members of the Council or Secretariat of Public Security (2); Investigative Police (3), and Implementing organization or council (4) 38 and coordinate its training with the courts’ own program.59 Rapid Results is another example where PROJUST could do more to accelerate the transitioning of responsibility for conducting exercises to the relevant institutions. TetraTech DPK’s continued presence in institutions that have reached acceptable performance levels (e.g., four of the CJMs visited and the ADR unit in San Luis Potosí)60 is yet another example. 4.2.3 What additional actors should PROJUST engage with to promote sustainability? The most obvious answer is other state government actors because without them local funding and other support will not be available. If state governments are not productively engaged, moreover, they can become a negative force, undermining the reform process. The ET has limited knowledge of Mexican judges and prosecutors’ associations, but none were mentioned by interviewees as reform partners. They are not always a positive addition, but at some point, should be drawn in lest they become a source of opposition. The same suggestions and caveats go for local bar associations, universities (as local stakeholders as opposed to grant recipients), and other professional organizations. A final potential engagement to promote sustainability is to reconsider how universities and the press have been involved by both implementers. Although among those invited to the sessions announcing Rapid Results, neither actor appeared to be further engaged in the subsequent exercises in the visited states. (The ET acknowledges that this was due to self-selection rather than an MSI oversight.) The trick in each of the above cases is for justice institutions to convince the relevant actors that the justice reforms are in their interest without PROJUST taking on this responsibility or continuing to act as an intermediary. 4.2.4 Conclusions Sustainability remains a major issue for PROJUST for a variety of reasons. First, PROJUST’s achievements in organizing the basics of justice sector reform will only be sustainable if it, or some other reform promoter, can move the process into the areas of most concern to the public—security, prosecution of high impact crimes, and faster processing of common crimes (e.g., second Piso 2). So far, PROJUST is attacking these challenges crime by crime, but a more comprehensive answer to issues of overall organizational structure is needed. Where the short-term results are the consequence of improved practices, such as Rapid Results, there is the further question of whether these changes will be adopted on a permanent basis elsewhere in the institutions. A longer-term fix is needed that consolidates these gains, such as monitoring by a CSO or some sort of agreement for self-monitoring by the sector institutions. Third, PROJUST initially focused on building coordination and ownership within the justice system, or a sense among the various justice sector institutions that they need to work together. This is one very important step beyond the former isolation of the separate parts in 59 In its response to this comment, MSI said this was because judges and prosecutors have different needs, but “Judicial Academies” elsewhere (Costa Rica, France, Germany, Eastern Europe) do not seem to have this problem, none believe, as MSI added, that any such joint service would violate judicial independence. 60 The ADR unit in San Luis Potosí was the source of TetraTech DPK’s model, but it also has gone beyond the model in adding details like weekly meetings to discuss common problems and solutions. 39 nearly all states but still does not draw in sufficient support from a broader base of justice sector actors. Local CSOs, where they exist, tend to specialize in promoting a narrow set of rights (due process, gender victims, etc.), which limits their utility as a lobby. For the majority of citizens, these issues are of less interest, and to attract their political support, PROJUST will have to incorporate their concerns. So far, PROJUST has done this by including business sector representatives in its Rapid Results exercises, but even they may tire after a third or fourth iteration of the same exercises. Enthusiasm only survives so much repetition and, to be maintained, requires movement to a new stage. Specific conclusions related to each of the evaluation sub-questions related to sustainability are presented below. What has PROJUST done to promote local ownership? PROJUST interventions have successfully promoted local ownership of both the criminal justice system reform process and specific criminal justice reforms. Tres Pisos and Rapid Results proved particularly effective in promoting local ownership and have also contributed to a widening of local ownership to private sector institutions. Notwithstanding the project’s contribution in this area, it has been less effective in promoting ownership among state and local government institutions and officials, particularly governors and legislatures, who are potentially the primary funding sources for criminal justice reforms. Where the project has engaged with state and local governments, the engagement has tended to be relatively unproductive. Were the federal government to draw down its support of justice system reform, then it becomes all the more critical to cultivate backers in state government. Obstacles to promoting local ownership include the low priority given to criminal justice reform among certain stakeholders, including state and local governments; poor results achieved when stakeholders do undertake justice sector reforms; and concerns about the loss of independence among judges, notwithstanding that when judges do become engaged in justice sector reforms, they have shown admirable innovativeness. Extending ownership for justice sector reforms beyond the current group of stakeholders to a wider portion of the local population has not occurred owing to a number of factors, such as the public’s limited contact with the system; conflicting interests and a lack of agreement as to what reforms are needed; distrust of sector institutions, particularly the police; and a limited history of collaboration within civil society actors to jointly work on criminal justice reform issues. The lack of widespread engagement and ownership across a broad cross-section of civil society and government actors poses serious risks to the long-term sustainability of the many criminal justice system reforms to which PROJUST has contributed. Thus, the most pressing challenge for PROJUST and USAID in this regard is to increase engagement with a more diverse set of sector actors and to gradually transition ownership for reform process to these actors. In short, those local actors who are concerned with justice issues are few, and they traditionally pursue their goals independently, keeping their interactions with others to a minimum. To engage them collectively and develop support for reforms (or what USAID calls a local system in justice), PROJUST will have to address a variety of disparate, and sometimes conflicting, 40 interests. At its best, PROJUST has broadened engagement beyond the sector, but in many states, for a variety of reasons, further expansion remains a major task. The problems and the challenges are inherent to Mexico’s reform program. First introduced to expand due process guarantees, the reform’s further progress has foundered on how to interest those with other concerns. Conceivably, PROJUST should do more outreach to groups with unrepresented interests, but this may be a task better left for future programs. In the meantime, the immediate strategic challenge is how to expand reform ownership to a broader cross-section of sector actors. What has PROJUST done to promote sustainability? Where, if anywhere, could PROJUST go to improve its efforts to promote sustainability? The evidence indicates that several of the criminal justice sector reforms to which PROJUST has contributed are firmly established and would require a significant countermovement to reverse. Examples include Tres Pisos, CJMs, ADRs, and public defenders for indigent clients. In such cases, the focus for support is shifting from creation and consolidation to expansion. Where backsliding has occurred, it has focused on reforms that occurred prior to the project. Other reforms to which PROJUST has contributed—case filtering at UATs, expedited hearing schedules, reduced pretrial detentions, and the use of abbreviated procedures and conditional suspension—are less firmly established and could be reversed for a number of factors, such as changes in institutional leadership, a change in institutional priorities, resistance by key sector actors (e.g., judges), public apathy, etc. Conversely, were any of the preceding factors to move in a favorable direction, they could help establish the reforms more firmly within the sector. At this point, there is insufficient evidence to project what direction such factors will take in the future, whether with or without project support. Yet other reforms remain too new to draw a conclusion one way or the other. These include efforts to link CSOs with the UMECA supervisory function; UMECAs in general which, although widely established, are underfunded, poorly resourced, and ill staffed; and work to expand civil society’s engagement with justice sector reform. The ET found mixed evidence as to whether PROJUST-supported justice sector actors have begun to plan for how they will continue with the justice system reform process once PROJUST departs. Survey results support the conclusion that a large majority of actors are planning on continuing the reform efforts post-PROJUST and that they intend to do so with their own (or with their state’s own) resources, although just what they plan to do varies widely both by state and by sector actor. In contrast, qualitative findings support an opposite conclusion in that, in all but two cases, the key informants interviewed had no plans to continue the reform process post-PROJUST. Absent additional information to parse out this issue, the ET interprets the contradictory evidence to indicate that, to the extent sector actors have ideas to continue the reform process post-PROJUST, these ideas are inchoate to the point that, when pressed for details on them, the actors are unable to articulate with clarity what those ideas are. Finally, the evidence cited above supports the conclusion that more work remains for PROJUST to strengthen the institutional capacity of sector actors so that the actors can both continue with the reform process and continue strengthening their own internal capacity absent external support. In terms of capacity development, PROJUST should already be in the process of 41 passing the baton to justice sector actors with PROJUST’s role limited to that of facilitator rather than continuing as a primary provider of capacity development services. What additional actors, if any, should PROJUST engage with to promote sustainability of current or prospective results? As a top priority, PROJUST should engage with state and local government actors to enlist them in the justice sector reform process and, over time, secure their ownership (along with funding and other resource support) for the process. The ET concludes that, absent state and local government engagement and ownership of the reform process, its scope, scale, and sustainability will be significantly constrained and many of the gains that have been achieved will be imperiled. Other potential reform partners, whom PROJUST either has not engaged or has engaged less intensively, include judge and prosecutors’ associations, local bar associations, universities, other professional organizations, and the press. 5.0 LESSONS LEARNED PROJUST (2014-2019), and the preceding Program to Strengthen Justice in Mexico (PRODERECHO) (2004-2007, 2007-2010) and Justice and Security Support Program (JASP) (2009-2014), both of the latter implemented by MSI, have produced a series of lessons learned. Those listed below are organized under the categories of effectiveness and sustainability, but not by sub-questions to avoid redundant explanations. Moreover, USAID’s interest in the following sections was not a further evaluation of PROJUST or intended as a re-recitation of findings, but rather lessons learned for future use. Readers are asked to keep this in mind. 5.1 EFFECTIVENESS A reform is not a law; a law is a necessary but insufficient condition for reform implementation. After years of promoting legal change, this lesson finally seems to have taken hold. The problem is not unique to Mexico or even to Latin America. Many donors, including USAID, have struggled to learn this lesson, and many of their projects continued to stress legal change even after expert opinion turned against it. Admittedly, this was often a response to local demands, exacerbated in Mexico by the partnership with SETEC for which laws and training were the keys to improving performance. In other words, local preferences count but pose a quandary when they favor tactics and strategies that experience (at PROJUST and elsewhere) has proved less effective. Training is another overrated change producer; its impact hinges on a good definition of needs, adequate coordination with other activities, and systematic evaluation of impacts. Interestingly, while questionnaire respondents rated training highly in terms of how they benefited from PROJUST, its perceived importance in shaping performance is much lower. Interpreted in light of the international experts’ experience with projects outside of Mexico, this is easily explained in that there are many reasons to like training (e.g., time off from boring work, intellectually stimulating discussions, certificates and diplomas that can be useful additions to a curriculum vitae, or interesting venues) that have little or nothing to do with improving on-the-job performance. A 2008 study by the World Bank’s Independent Evaluation Office found that only in 50 percent of its projects had training made a visible difference and, even then, its definition of difference was generous. Three knowledgeable external observers interviewed by the ET discounted the impact of the early SETEC or PROJUST mass training, stating that its effect was largely “symbolic” or that its 42 emphasis was inappropriate in that it placed “too much focus on oral litigation techniques rather than topics like investigating a case and developing a case theory.” Unfortunately, SETEC and the states pushed PROJUST into a period of extensive, short-term training, the very type that is increasingly regarded as ineffective. Training’s benefits should not be completely discounted; they depend on how training is done, whether the topics are potentially useful, their match with local staff capacities, and whether content is reinforced by measures to encourage its application. Recent training by both implementers is more consistent with these guidelines, but neither has provided evidence to the ET of adequate evaluation of the results. Models, protocols, SOPs, and diagnostic and certification tools can be of little practical utility, especially when not based on empirical research and evaluated for their effects. The utility of these instruments depends on how they are applied, but simply drafting dozens of them does not necessarily make them useful. There are two issues here. The first is the apparent expectation that target groups will read and use them all. If utilized selectively in conjunction with specialized training and accompaniment, they may strengthen the latter’s impact. If delivered as a personnel guide absent any other support, they will at best, as has occurred in many other projects, fill out personal libraries with little impact on performance. The second issue is how these documents are developed. In several cases (e.g., Monterrey’s Judicial Council, Tabasco’s Public Ministry, Zacatecas’s CJM, Judiciary, and UAT) local authorities suggested that the models required adaptation to local circumstances. The ET’s review of their contents also suggested that some proposed structures and practices were largely the result of guesstimates of what might work (e.g., Protocols UAMEs, n.d. and Manual de Organización de la UAME, n.d as well as models detailing human resource and equipment needs for UMECAs). Of course, those based on pre-existing legal provisions cannot ignore these, but authors might discourage official adoption until their products are fully tested on the ground. In either case, once further tested on the ground, compliance with the contents will require effective monitoring systems within the institutions, which few have developed, except for CJM’s using TetraTech DPK’s certification process. This is not to say the ET favors certification, which it does not, but rather to suggest a less onerous means for tracking compliance at the local level. States and institutions have different capacities for internalizing new ideas and approaches. The most difficult ideas for nearly all participants are how to organize Pisos 2 and 3 of the Tres Pisos model and the functions of the UAME. PROJUST is to be congratulated on transmitting the underlying Tres Pisos argument—that depending on their content and other characteristics (what PROJUST calls relevance), not all complaints will be treated in the same fashion and some will not be pursued at all (prosecutorial discretion). Where PROJUST has operated, the model has been successfully implemented at Piso 1 (intake, preliminary filtering and redirection of some complaints to ADR, or the archiving of those with “unidentified suspects”). What to do with Pisos 2 and 3 remains an issue PROJUST has been addressing by working with specialized units at each level. The question, possibly unanswerable at present, is how improvements there will be transferred to all cases and, especially for Piso 2, give rise to a more efficient organizational structure? 43 The issues confronting UAMEs were elaborated in section 4.1.1. The proposed creation of a unit capable of monitoring performance statistics is important, but its implementation has fallen far short of intent because of the poor quality of those chosen to staff it. Also, without an effort to improve institutional statistics, case tracking systems, and information technology (IT) personnel, the UAMEs’ impact will be limited, although not as much as by poorly chosen personnel. Here, as in instances of the choice and/or replacement of personnel in programs like ADR and UMECAs, a part of the problem may be state leadership’s insufficient understanding of their importance or of the difficulties posed by constant staff changes. The results of changes to complex behaviors, even in pilot units, cannot be reliably assessed in less than five years. This allows time to see whether the behaviors have stuck and whether any upward and downward fluctuations represent permanent trends. Obviously, a mid-term evaluation has to work with what has already happened, but a final evaluation, done three years after the new rules were introduced, will have similar challenges. The failure to explore resource requirements for the implementation of new laws and structures hampers the reform process. This failure stems in large part from the general belief that any justice sector issue can be resolved with more personnel and more infrastructure, encouraging excessive investment in both, and the squeaky wheel syndrome in which the squeakiest wheels receive more than they need, leaving other important components with insufficient resources to do their part. None of this is exclusive to Mexico; however, federal funding in the country (e.g., FASP and that managed by SETEC before its disappearance) exacerbates the problem by its availability only for a few institutions and items, usually infrastructure, equipment, and training. The UMECAs are a useful example of this lesson learned. Except in one state (Nuevo León) and possibly also in Chihuahua (in the near future), all states visited lacked adequate resources (human, financial, and equipment). When located in the state Secretariat for Public Security, the UMECAs receive FASP financing for offices and some equipment but not for staff or for training, as the latter requires a special certification usually applied to police and prosecutors (and possibly irrelevant for pretrial services). The first UMECA, for juveniles in Morelos, worked well, but it covered only a few cases in one district. The expansion to other states and adult defendants has rarely been accompanied by an adequate injection of resources. To function as intended, the UMECAs will need more personnel and other resources, unless TetraTech DPK can create networks of NGOs to do this—so far, an unrealized challenge. The promotion of differential case management, prosecutorial discretion, and alterative solutions (e.g., the Tres Pisos model) can be an important source of change; however, other project innovations, or those developed by the states, can also be effective. Some of the most important examples of the latter from PROJUST include: Coahuila’s use of meetings with mayors to identify crimes most salient in their districts (a form of community prosecution although the judiciary has also been included); the development in Tabasco (with PROJUST assistance) of methods for analyzing cases with unknown perpetrators; and in both Coahuila and Guanajuato (apparently without PROJUST assistance), the elimination of the control-trial judge distinction, thus letting any judge fill either role. An emphasis on alternative solutions confronts problems as prosecutors, public and private defenders, and victims and their counsel adapt to their roles. There is a difficult, but valid, debate as to whether each group should focus on litigating to the end in a zero-sum 44 contest or move to agree on a compromise. The code’s authors favor compromise, but there are many prosecutors, defense attorneys, and victims’ counsel who do not. The extensive victims’ rights in Mexico complicate the situation, but problems would still exist in their absence. This appears less an issue of insufficient training than of how long it will take the actors to reach their own agreements on how to resolve the differences. Where projects focus on first instance courts, delayed/inadequate training of the appellate judiciary, and in a federal system, of its own judges can undercut efforts to combat formalism and promote an emphasis on effective resolution of cases. Even a program of PROJUST’s size cannot do everything at once; moreover, higher instance judges are notoriously difficult to convince that they need training. Perhaps PROJUST can still do more with the state appellate courts, but the federal judiciary is not within its mandate. The federal courts have long been a thorn in the side of Mexico’s state judiciaries because of their willingness to entertain protests (amparos) against interlocutory and final state court rulings. However, state appellate judges’ continued insistence on formal criteria can also lead to the nullification of decisions made where seemingly unnecessary details are not observed. There have been fewer examples where pretrial detention was cited as a violation of the principle of innocence, including the use of amparos against Baja California cases regardless whether the detention met the CPC conditions for its imposition. Notwithstanding state justice sectors’ own IT investments, few institutions can generate reliable statistics on their operations, hampering not only their oversight of their own performance, but also USAID and project efforts to develop indicators. The explanation lies in both human and technical factors. Lack of interest among institutional leaders means that no one checks data entry contributing in turn to data quality and validity problems. In PROJUST’s case, much investment went into systems to facilitate work within judicial offices, such as templates, alarms warning when deadlines were not being met, or screens that told workers what was required each day. Statistics were an often-neglected afterthought as few leaders saw any purpose for collecting them. Some of these systems could be adapted to include a statistical module, but so far no one has pursued this route. MSI’s small IT staff may be able to help; however, PROJUST’s financing of new systems is impossible under present funding arrangements. There are a few case tracking systems that do generate statistics and could be used as models or adopted by other states. Progress here has been slow, and some state institutions (e.g., Guanajuato’s judiciary), are unwilling to share their software, although they are willing to let others examine it. Although it is easier to improve the work of the courts and public defense, criminal justice reform will never work as intended until police and the Public Ministry improve. These two institutions are the motor for criminal justice; if they do not do their job, increasing court efficiency in handling “judicialized” cases is of little consequence. PROJUST can work to improve prosecutorial efficiency, but the many weaknesses of state (and where they exist, municipal) police will still impede its efficacy. The existing gap in defining state and federal competencies for handling criminal cases is a further complication. In addressing criminal justice reforms, key actors often think locally rather than globally. Nearly all staff and consultants are Mexican and PROJUST contents derived largely from the perspectives of Mexican reform proponents. This has been an advantage in work with the federal government and state participants, but it also limits an ability to think outside the 45 (Mexican) box. Both MSI and TetraTech DPK have used lessons from other countries, but each could do more to incorporate external experience (of which there is much) on how to develop this type of criminal justice reform. 5.2 SUSTAINABILITY All donor-funded projects risk encouraging their clients’ dependence on their continued presence. PROJUST has not avoided this tendency. The weaknesses of the remaining state implementing entities exacerbate this trend, especially for institutions excluded from its remit. If PROJUST’s advances are to be sustainable, local institutions must feel both empowered and capable of continuing the reforms on their own. New institutional structures and functions created by justice reforms are unlikely to disappear; however, unless they come with increased capacity to implement the reforms, they risk stagnating. PROJUST has made advances in the treatment of complainants and victims but has made little progress, even in its target states, in augmenting the percentage of all complaints taken to successful resolution (aside from in ADR and the Rapid Results targets). The so-called “revolving door” (repeat offenders continually put on pretrial release) will only augment public perceptions of failure if a greater proportion of common and high impact crimes are not successfully resolved. This is the reform and USAID/PROJUST’s principle challenge. It is a challenge to encourage cooperative problem solving among justice institutions made even more difficult by introducing other local actors into the collective deliberations. In many countries, justice institutions have always worked alone, whether enjoying official independence. Where they resist closer coordination, engaging outside members is even more difficult. Cooperative problem solving is worth attempting where either courts or Public Ministries are interested because it has, in some cases, helped put pressure on more recalcitrant sector organizations. Among the eight states visited, the ET saw a few attempts to encourage cooperative problem solving, but no dramatic successes as of yet. Building extra-sectoral support requires appealing to a variety of often conflicting interests. In Mexico, and in many other countries, the first CPC supporters were due process advocates. They are still present, but their numbers are not expanding. To attract others, the reforms must recognize their interests and seek to accommodate them all, to the extent possible. If reform stakeholders do not perceive a benefit from the reform, the reforms may well take a step backwards. As noted, and despite the CPC’s emphasis on conciliation, early supporters do not help by refusing any addition that might compromise their achievements in shaping the laws. Here independent initiatives by Guanajuato (courses for entrepreneurs) and Coahuila (a variation on community prosecution) are useful innovations to consider. Sustainability and local ownership both depend on effective engagement of other parts of state government. In PROJUST’s case, engagement of state legislatures and executives was most effective during the law-drafting stage. Since then, their interest and involvement vary among the states visited with positive and adverse implications for reform effectiveness and sustainability both within and across states. Inadequate attention to evidence-based human resource planning negatively impacts sustainability. Inadequate attention to resource planning creates several problems that may remain with the system for years to come with impacts even on areas like gender that any 46 counter-reforms are unlikely to target. The same applies to planning for infrastructure and equipment, both of which are also tied to human resources. 6.0 RECOMMENDATIONS This section draws on the findings, conclusions, and lessons learned presented above to offer a set of recommendations for USAID and PROJUST to consider for the remaining life of PROJUST and/or for future projects working on criminal justice sector reform. 6.1 EFFECTIVENESS PROJUST and USAID should reevaluate any remaining work on law drafting, training, and models, protocols, and SOPs against likely impacts; in reference to the latter instruments, the ET suggests that PROJUST halt production and that any future project use a more systematic approach as laid out below. If USAID retains interest in the use/impact of what has been done, it should first ask PROJUST to explain which tools it is currently using, how it is applying them, and what results are expected and achieved. This should clear up any USAID doubts about “use” but also help the parties agree on how it relates to results. Second, USAID should insist that any future project focus on what state entities find most useful with variations on existing instruments tailored to their own needs and available resources. Third, additions to the standard catalogue (under a follow-on project) should be based on a more systematic process, starting with an identification of areas needing guidance and continuing through initial drafting, testing of the effects on behavior, redrafting, retesting, and further revisions or even rejection of the entire product. So far as could be determined by the ET, except for revisions required by legal changes, this process has not been followed. Given the GOM’s apparent preferences, the results of the first step if taken should be shared with it, along with the issues and proposed new approach to producing more of these instruments in any future project. In any future support to the GOM’s reforms, USAID should insist that adequate analyses of resource needs be conducted, aiding the process as needed. These should start with actual and projected demand as well as information on reasonable workloads (based on experience in Mexico and elsewhere) and then develop a series of scenarios indicating mixes of resources to deal with them. Such techniques have been used in more developed countries and while always provoking controversies, even these debates can productively focus attention on “what we need to achieve what we want to do.” External experts are recommended here as this is an underdeveloped area in Mexico (and in fact in most of Latin America). Toward this end, PROJUST, or a future project, should review processes developed by judiciaries in Guanajuato and Coahuila to determine staffing needs and simplify judicial (and possibly prosecutorial and defense) human resource structures. Under any new project, similar attention should be given to estimates of infrastructure and equipment needs. Work by PROJUST and by some institutions on their own (e.g., Guanajuato’s judiciary) makes it apparent that infrastructure needs are often overestimated, especially regarding court hearing rooms. Ideally, and despite the impediments imposed by targeted funds (e.g., FASP), focus could turn as well to sector-wide budgets to ensure that the use of all resources (including personnel) is optimized to improve system outputs and outcomes. 47 Some of the largely independent additions by certain state courts and Public Ministries should be tracked to determine their results, and if these are positive, promoted more widely. As described in the lessons learned, these include Coahuila’s experiment with community prosecution, development of analytic tools for cases sent to “unknown perpetrators” as in Tabasco with PROJUST assistance, and Coahuila and Guanajuato’s elimination of the control-trial judge separation as well as their mega-hearings on pretrial detention. The ET believes all of these are positive, but the results need further examination before being promoted in other states. Unless it finds states prepared for this approach, PROJUST should stop creating UAMEs because they are beyond the capacity of most institutions/personnel. Further work with monitoring via judicial statistics (more than indicators) might continue at a basic level if PROJUST and its subcontractors can simplify their approach. This will require further, and possibly intensive, discussions with leaders of targeted institutions on the importance of analysis and hiring staff capable of numerical analytic work. Thought should be given to conditioning additional assistance, and not only to UAMEs, on providing capable personnel, although in Coahuila (judiciary) and Tabasco (Public Ministry) there is already a department doing analysis (even if it is not called a UAME) that should be sufficient. Moreover, any assistance to UAMEs or other data analysis departments should use consultants who have worked with judicial statistics elsewhere. In the case of UMECAs, PROJUST should work with local actors and the Secretariado (or whatever may replace it) to determine how the essential supervisory function can be better performed. This may require changes to the model or even laws, possibly reducing the number of conditions to be supervised and tailoring them to what an entity like UMECA can reasonably do. If this cannot be accomplished, the entire notion of supervised conditions becomes a dead letter,61 feeding the widespread impression that unsupervised criminals are put back on the streets. Several quick fixes to issues encountered already include: ensuring judges inform the UMECA immediately of the conditions imposed, doing the same with entities other than the UMECA where the periodic check-in is done, and (more difficult) finding more convenient ways (perhaps in a municipal office) for those living at a distance to check in. If not in this project, then in any follow-up, USAID should turn attention to improving sector institutions’ IT systems, especially for case tracking, as well as trying to link better indictors to what they produce. This is a critical need in Mexico and despite what can be done with Excel sheets and “heroic measures” (e.g., casefile analysis) is absolutely essential. As some good systems exist already, USAID’s role, absent funds to finance them, could be to encourage sharing among state institutions. PROJUST or a future project should work with INEGI and the Secretariado to promote the development of uniform standards for collecting performance statistics. The lack of reliable data poses a constant threat to the reforms, and reliability depends, inter alia, on a general agreement on what will be measured and how. This should not preclude states from collecting statistics on issues of most importance to them (e.g., stock rustling in Tabasco, Chiapas, and Campeche), but it does guarantee that certain data of common interest will be collected and reported consistently. (Note that PROJUST mentioned that it tried something 61 A dead letter is a law or treaty that has not been repealed but is ineffectual or defunct in practice. 48 similar to this but found that neither INEGI nor the Secretariado was receptive to its suggestions. Thus, this recommendation may have to wait for a future project.). Regarding impediments posed by appellate and federal judges, PROJUST might work with the former and encourage discussions with judges at the federal level on eliminating annulments based on procedural errors. Any new project should continue this work. These, as well as issues of inconsistent application of laws at the appellate level, are common problems in new code adopters, and, in respect for judicial independence, it appears that the more effective solutions are to start at the top (in this case the Federal Supreme Court) as it often is more open to change than lower level federal and appellate judges. Development of models, protocols, and SOPs should be combined with ongoing or periodic monitoring and reinforcement so that users understand their purpose and how to use or adapt them as a means to ensure both their use and practical utility. As noted above, merely creating models, protocols, and SOPs does not guarantee their use or practical utility. Experience has demonstrated, however, that twinning their development and introduction with either ongoing or periodic monitoring and reinforcement does contribute to their being used and to their perceived usefulness and value-added. 6.2 SUSTAINABILITY In collaboration with PROJUST, USAID must refine its strategy to: 1) better define the linkages among its principal lower-level results (and especially those relating to CSO engagement/empowerment, public education, and greater sector transparency); 2) recommend an order for their introduction and the required minimum achievement levels;62 3) define the expected results (quantitatively if possible) for all activities (not just those covered by indicators); and 4) elaborate the road ahead. To end the project successfully, raise the chances of advances being maintained, and look ahead to any follow-up this modified strategy will be essential. Despite the addition of outcome indicators, too much of the current strategy still revolves around more of everything. Not everything needs an indicator, but there are other ways of suggesting how much of each is needed. Before introducing more tools and programs (or extending mechanisms like the UAMEs to more states), PROJUST should focus on improving what it has already developed and elaborating plans for its scaling up. The focus on the basics is valuable and should be emphasized in those states that have not internalized them, with the exception of those where, for some institutions, this appears to be unachievable. Examples of the latter include all state institutions as regards UAMES except Coahuila’s judiciary and Tabasco’s Public Ministry and possibly also Morelos and Nayarit where recent leadership changes appear to have cut relations with PROJUST. PROJUST should place more attention on building local capacity, whether in conducting Rapid Results, improving existing training programs, selecting staff, or engaging local governments, civil society, and the private sector. PROJUST continues to do much of the heavy lifting here, but justice institutions need weaning from their reliance on the project. The relevant actors may well vary by state, but the emphasis should be on establishing connections that will not require permanent PROJUST intervention, either in continual support of Rapid 62 Here the ET is not referring to project targets, but, rather, to the minimum advance required for the element to have an impact. 49 Results, grants programs, or in some other form. This may be easier in states such as Baja California, Chihuahua, and Nuevo León that, on reestablishing a connection with PROJUST, seem inclined to be more selective about what they accept. It will also require a commitment from PROJUST to transition into more of a facilitator role than an implementation partner in moving forward. Now or in a later project, effective means to engage state governments must be found. Except in states where this has happened on its own, this is the missing element in efforts to focus local stakeholders on justice reform. The current implementing agencies (mini￾Secretariados) are usually marginal participants and, moreover, are a poor channel for reaching governors and legislators, and it is the latter who control budgets and set priorities. Although stated in the previous section, it bears repeating that USAID in its projects and in discussions with the GOM must begin addressing the costs of full NCJS implementation. The lack of such information affects sustainability as well as effectiveness. Given the unlikelihood that the GOM can fund everything at the necessary levels at once, the challenge is to find a reasonable balance among the various needs, possibly leaving the least critical for a later period. If this is not done, USAID support will continue to face issues caused by serious resource deficiencies (as well as some excessive and thereby inefficient resource endowments). USAID could assist in this area and/or make it a condition for assistance to specific institutions and/or states. If USAID/Mexico acquires funds for a follow-on project, it must build on PROJUST’s successes, taking them to new states while recognizing issues of more interest to most Mexicans, including resolution of a higher percentage of high-impact and common crimes. The importance of what PROJUST has done in the end hinges on what follows. If higher level goals are not addressed soon, the sustainability of past accomplishments is also at risk. REFERENCES A full list of documents consulted is found in Annex 5 APRI (American Prosecution Research Institute). 2001. “How Many Cases Should a Prosecutor Handle?” available at http://www.ndaa.org/pdf/How%20Many%20Cases.pdf. CEPEJ (European Commission for the Efficiency of Justice). 2014. European Judicial Systems-- Edition 2014 (2012 data): efficiency and quality of justice. Strasbourg: The Council of Europe. CIDAC. 2017. Hallazgos: Seguimiento y Evaluación de la Operación del Sistema de Justicia Penal en México. CDMX. Hammergren, Linn. 2007. Envisioning Reform: Improving Judicial Performance in Latin American. University Park, Pa: Penn State University Press. INEGI. 2016. ENVIPE available at http://www.beta.inegi.org.mx/proyectos/enchogares/regulares/envipe/2016/ Miles, Thomas J. 2013. “Does the ‘Community Prosecution’ Strategy Reduce Crime? A Test of Chicago’s Experience.” American Law and Economics Review, pages 117-143. Available at http://aler.oxfordjournals.org. 50 World Bank, 2002. The Juicio Ejecutivo Mercantil in Mexico’s Federal District Courts. Washington, D.C., The World Bank, Report No. 22635-ME. World Bank, Independent Evaluation Group (IEG). 2008. Using Training to Build Capacity for Development: An Evaluation of the World Bank’s Project-Based and WBI Training. Washington: The World Bank. Zepeda, Guillermo. 2017. “Impunidad Cero: Índice estatal de desempeño de las procuradurías y fiscalias.” Available at https://www.impunidadcero.org/articulo.php?id=49&t=ranking-de-fiscalias￾y-procuradurias-estatales-en-mexico. 51 ANNEXES Annex 1: Scope of Work for the Evaluation, Including Annexes to the SOW Annex 2: Inception Report Annex 3: Questionnaire for the Survey Annex 4: List of Interviewees Annex 5: List of Project Documents Consulted Annex 6: Additional Tables Annex 7: Conflict of Interest Statements Annex 8: List of Answers in Spanish to Question 6A Annex 9: Statement of Differences 52 ANNEX 1: SCOPE OF WORK FOR THE EVALUATION 53 AID-523-C-14-00003 MEXICO PROMOTING JUSTICE PROJECT (PROJUST) Evaluation Scope of Work 1. INTRODUCTION This is a Scope of Work (SOW) for a mid-term performance evaluation of USAID/Mexico’s Promoting Justice Activity (PROJUST or Activity). The evaluation will be contracted through USAID/El Salvador’s regional Monitoring, Evaluation, and Learning award implemented by ME&A. This evaluation will be conducted by a team of evaluators with significant collective experience in Mexico, rule of law reform, and evaluations. 2. EVALUATION PURPOSE AND OBJECTIVES The main objectives of the evaluation are to assess to what extent has PROJUST’s support been effective and sustainable and provide recommendations for future programming. 3. PROJUST BACKGROUND The Government of Mexico (GOM) is undertaking a complete overhaul of its criminal justice system, replacing closed-door procedures and written arguments with an adversarial system based on public, oral trials. The criminal justice reform seeks to improve the transparency, effectiveness, and efficiency of the criminal justice process, while protecting due process, promoting assistance to crime victims, and strengthening human rights. A 2008 Constitutional amendment required all state and federal jurisdictions to make the transition from the inquisitorial system to the accusatorial system by 2016, and USAID/Mexico is committed to supporting Mexico in this monumental endeavor, which will continue well beyond 2016. USAID’s support for criminal justice reform in Mexico pre-dates the 2008 Constitutional amendment. Currently, PROJUST, implemented by Management Systems International in a teaming agreement with TetraTech DPK, is the vehicle for USAID support. Although both organizations operate under one contract and submit joint documentation in many cases (i.e., financial reports), annual, quarterly and monthly reports are submitted separately. Both organizations are also operating under different timeframes: • MSI: July 16, 2014 to July 15, 2019 TetraTech DPK: January 23, 2015 to July 15, 2019 The period to be covered by the evaluation is from July 2014 to when the Evaluation Team begins its field work. PROJUST has a total estimated cost of $68.2 million. The Constitutional deadline of June 2016 represented a big shift for PROJUST. Before the deadline, the focus of the program was to contribute to the transition towards an adversarial criminal justice system. The effort was centrally led by the Government of Mexico’s Technical Secretariat of the Coordination Council for Implementation of Criminal Justice Reform (SETEC). The PROJUST results framework, indicators, workplans, and reporting reflected these priorities. PROJUST was supporting all but three Mexican states during this period. After the Constitutional deadline, SETEC disbanded and the reform effort is only nominally managed centrally. The federal government still provides limited oversight and funding for states, but states are now responsible for their systems. The Office of the Technical Secretary of the National Public Security System (part of the federal Ministry of the Interior) is now in 54 charge of the “consolidation”63 of the system and is USAID’s key partner at the federal level for Rule of Law. These changes meant that current contract requirements had to be revised to narrow the scope geographically and technically to those states that offered the best conditions—including political will—for successful partnership in the consolidation process. The contract modification had the stated objective of supporting consolidation of Mexico’s Criminal Justice Reforms. 4. PROJUST DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE The changes in the field, the different pace of the reforms in different states, and the shift in federal counterparts have meant that PROJUST’s results frameworks have changed over the years. To date, PROJUST has three results frameworks. PROJUST and USAID will work together to ensure that the new framework and outcome indicators are part of contractual obligations. A. The first results framework is from the USAID/Mexico Country Development Cooperation Strategy, Development Objective 2: “Constitutional criminal justice reforms of 2008 are operational.” 1. Legislative framework strengthened 1.1. Civil society effectively advocates for a stronger legislative framework. 1.2. Capacity of justice sector institutions for drafting and leading the enactment of legislation strengthened. 2. Justice sector actors promote and implement the new criminal justice system 2.1. Justice sector institutional capacity to implement criminal justice reform strengthened. 2.2. CSOs’ ability to promote and monitor criminal justice reforms strengthened. 2.3. Legal profession and education strengthened. See Annex 3 on page 20 for the Results framework under modification. It is also at https://www.usaid.gov/sites/default/files/documents/1869/Mexico%20CDCS%202014%20 2018.pdf, p. 59. B. At the signing of the PROJUST contract in 2014, PROJUST’s theory of change was explicitly linked to the Merida Initiative, a cooperation program that acknowledges the shared responsibilities of the United States and Mexico to counter violence caused by drug trafficking. PROJUST work was conducted under “Pillar II” of the Initiative: “Capacity to sustain the rule of law: Enhance the capacity of Mexican public security, border, and judicial institutions.” The objective of PROJUST was to support Mexico’s transition to the accusatory system, at both federal and state levels, while protecting human rights. The main stated results were: 63 Defined in the 2017 Work Plan as the point where states are able to obtain long-term, sustainable results after the new system is fully functional. 55 1. Legislative framework enacted for criminal justice reform; and 2. Strengthened institutional and human capacity to implement criminal justice reform. See Annex 3, page 19, for results framework. C. The political environment surrounding the consolidation of the reforms has shifted considerably one year after the modification. Mexico is going through its most violent year since deaths related to drug cartel activity started surging during the last decade. The new top-level outcome for USAID’s Justice and Citizen Security Office is “Government of Mexico legitimacy increased.” The current development hypothesis, as articulated in the USAID results framework for rule of law, is as follows: The legitimacy of the government will be increased with a more transparent and responsive justice system achieved through greater accountability, effectiveness, and respect for human rights. This framework was approved in June 2016, and it includes a strong emphasis on measurable progress on a series of key outcome indicators (see section 6 below) which USAID and PROJUST are currently working to include in a revised Monitoring and Evaluation plan. See Annex 3 on page 21 for the current results framework under modification. 5. PROJUST ACTIVITY DESIGN 5.1. Purpose and Description of Activity Up until the 2016 deadline, the objective of PROJUST was to work with the GOM to implement and operationalize a comprehensive criminal justice reform at the state and federal levels, while also ensuring that Mexican and international human rights standards were complied with. The contract’s five tasks were: • Task 1 (MSI): Advance legislative reform; • Task 2: (MSI) Increase prosecutorial and judicial effectiveness; • Task 3 (TetraTech DPK): Increase public support for the new criminal justice system; • Task 4 (MSI): Strengthen analytical and quality control capabilities of targeted justice institutions; and • Task 5 (TetraTech DPK): Increase access to justice and improve victims’ assistance. Since March of 2017, PROJUST implementation is now divided into four key tasks, divided between MSI and TetraTech DPK. A contract modification to revise the scope of the activity was signed in March 2016. • Task 1 (MSI): Promote increased legitimacy of the criminal justice system; • Task 2 (MSI): Support building increased capacity for monitoring and evaluation of criminal justice system operation; • Task 3 (TetraTech DPK): Promote increased public support for the new criminal justice system; and 56 • Task 4 (TetraTech DPK): Support increased access to justice and improved victims’ assistance. For the purposes of this evaluation, the work between the two implementing partners is distinct enough that their work will each be evaluated separately; comparisons may be made only if considered appropriate by the evaluation team. 6. PROJUST LOCATIONS The PROJUST activity is currently being implemented in 19 states throughout Mexico. Annex 2 illustrates which type of PROJUST activity components are being implemented by state and municipality. After the June 2016 implementation deadline mandated by federal authorities, PROJUST has considerably narrowed its focus to the states and institutions presented in Annex 2. Annex 2 reflects only the current four tasks (reflecting that MSI’s Task 4 has been consolidated under Task 2). MSI’s FY2017 approved Work Plan divides the states where it is working into two groups (“first tier” and “second tier”): first tier states are states where support activities are more comprehensive and often include reorganization of the partner institutions. Support for second￾tier states is limited to discreet interventions in selected institutions. TetraTech DPK’s Tasks (3 and 4) are not divided by tiers. For this evaluation, Annex 2 is provided for informative purposes only and to present the full scope of PROJUST. For purposes of this evaluation, the evaluation team is expected to conduct evaluation in eight states, to be confirmed. Priority will be given to states where PROJUST assistance was provided since the start of the Activity. 7. PROJUST PERFORMANCE MONITORING AND EVALUATION USAID will make M&E plans and indicator reporting available to the evaluation team (see Annex 4) for PROJUST’s current outcome indicators (which will be part of a new M&E plan approved before the end of this fiscal year) and its current M&E plan. As mentioned above, the new PROJUST results framework emphasizes the importance of showing substantial progress on indicators relevant to the reduction of impunity. The new Outcome Indicators are (see Annex 4 on page 22 for baseline values and current status): 1. Pretrial Confinement rates 2. Prosecution Rates 3. Compliance with Precautionary Measures from Pretrial Service Units 4. Plea Agreements 5. Use of Alternative Dispute Resolution Mechanisms 6. Case Disposition Rates 7. Time-to-disposition Rates 8. Percentage of cases with sentences greater than three years 9. Change of Journalist perception and accuracy on criminal justice system reporting 8. CRITICAL PROJUST ASSUMPTIONS Three assumptions must be considered by the evaluation team when submitting its proposals and designing and conducting the evaluation. They are: 57 ● No state or jurisdiction of Mexico has enough funding for full provision of all justice sector institution services in all jurisdictions. The evaluation team should work from the assumption that states are working in areas and institutions they feel are critical, and that PROJUST support has been requested in those institutions. ● The evaluation team should not assume that the existence of a justice sector institution is sustainable because it is in operation; turnover in management and political appointments can quickly eliminate completed programs within justice sector institutions. ● PROJUST operates under the assumption that the GOM will continue to commit resources and prioritize criminal justice reform as part of its national policy, and that states will continue to work on strengthening their justice sector institutions, and that they will continue to value USAID support that catalyzes this process. 9. EVALUATION QUESTIONS AND AUDIENCE 9.1. Evaluation Questions The evaluation team will use all planning documents, management documentation, available activity quantitative and qualitative data, and suggested quantitative and qualitative data collection methods referred to in Section 10 below to respond to the below evaluation questions. All questions below must be answered fully and completely in the evaluation, underscoring both positive and negative outcomes. All answers must utilize sex disaggregation for all data collection and analysis, to the extent necessary. PROJUST indicators have varied during its lifetime; the evaluation team must focus on indicators that are relevant to answer the questions below. It then must go beyond the quantitative assessment related to these indicators to carry out a qualitative assessment. 1. Effectiveness. To what extent has PROJUST met its objectives relating to the functioning of the Attorneys General Offices, Women’s Justice Centers, and Pre￾Trial Services Units and the actions of civil society related to the criminal justice reform? a. To what extent has PROJUST technical assistance contributed to results, as measured by approved indicators and qualitative data? b. To what extent are PROJUST-supported models, protocols, and Standard Operating Procedures being used at the state level in justice sector institutions listed above? If they are being used, to what extent is this attributable to USAID? If they are being used, what aspects of PROJUST’s approach best supported their implementation? If they’re not being used, why not? 2. Sustainability: To the extent there are results, how sustainable are these results in the institutions (Attorneys General Offices, Women’s Justice Centers, Pre-Trial Services Unit) and civil society organizations referenced above? a. What has PROJUST done to promote local ownership (i.e., that the priorities are locally owned, locally resourced, and using rather than supplanting local systems to sustain results)? 58 b. What has PROJUST done to promote sustainability? Where, if anywhere, could PROJUST do to improve its efforts to promote sustainability? c. What additional actors, if any, should PROJUST engage with to promote sustainability of current or prospective results? 3. Lessons Learned and Recommendations: What are the lessons learned from PROJUST’s efforts that should be incorporated into a new design? a. Are there any current areas of support within the programming areas reviewed that should be discontinued? b. Based on the evaluation team’s findings and analysis, are there any other obvious existing or emerging opportunities in Mexico, particularly related to the issue of impunity (defined as the inability of the justice system to investigate, adequately prepare cases for litigation, and adequately present cases before judicial authorities), that USAID should be taking advantage of? 9.2. Evaluation Audience The audience for the report includes USAID/Mexico, the PROJUST implementing partners (MSI and TetraTech DPK), and justice stakeholders, ranging from GOM officials to civil society to the private sector. 10. EVALUATION DESIGN AND METHODOLOGY This performance evaluation will be conducted using a mixed-method approach. For more details on definition and description of a performance evaluation, please refer to the USAID Evaluation Policy, page 1, box 1: Concepts and Consistent Terminology, attached in Annex 1. The evaluation team must use quantitative and qualitative data to respond to the evaluation objectives and questions. The evaluation team will be expected to use activity-provided quantitative data [such as OPI results (a component of the activity)] and collect quantitative and qualitative data as data and information resources. We request that the evaluation team provide the most effective evaluation design and methodology in the proposal, including data collection activities based on illustrative data collection methods, data analysis, and disaggregation parameters given below. 10.1. Illustrative Qualitative Data Collection Methods: a. Desk review: contract documents; annual work plans; M&E plans; regular reports to USAID, Teaming Agreement, and program deliverables (protocols, SOPs, etc.) developed by implementers. b. Key informant interviews: The evaluation team shall conduct one-on-one interviews with key informants. Relevant stakeholders may include, but are not limited to, those listed below: o State Attorneys General (AG) and staff members o State Coordinators of Criminal Reform Implementation o Heads of Specialized Units within State Attorney General Offices and staff members o Directors of Women’s Justice Sectors and staff members 59 o Directors of Pre-Trial Service Units and staff members o Heads of Alternative Dispute Resolution Mechanism offices/units and staff members o Spokespeople of Attorneys General Offices o Spokespeople of State Judiciaries o Justice and security section editors/journalists in local newspapers/news channels o Representatives from civil society o Representatives from the private sector [including Mexican Employers’ Confederation (COPARMEX)] o Representatives from academia c) Focus Group Discussions: Focus groups may be comprised of: Investigators and operative staff at specialized units at AG Offices (without the Directors of AG present), Public Defenders and support staff (not the Public Defense heads), Pre-Trial Service Unit staff (without the Directors). o Members of local civil society organizations and guests from the business sector and academia. 10.2.Illustrative Quantitative Data Collection Methods: Quantitative data will be collected through the above-mentioned qualitative interviews, where applicable, and through a desk review. The evaluation team will propose how quantitative data will be used with the qualitative data that will be collected to respond to the evaluation objectives and questions. 10.3.Data Analysis and Disaggregation The exact data analysis methods used may differ for and within each evaluation question and may depend on the data available. The evaluation team will specify the exact methods to be used for each evaluation question in the Inception Report, which will be discussed with and approved by the USAID Contracting Officer Representative. All quantitative and qualitative data collected during the course of the evaluation, both soft- and hardcopy, will be owned by USAID. At the end of the evaluation, all quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the evaluation. Qualitative data (interview and focus group data, observational surveys, etc.) that are the basis of evaluation findings should also be provided without personal identifiable information. When qualitative data are coded a code book and accompanying support documentation should be provided. All data collection and analysis tools used, such as questionnaires, checklists, and discussion guides must be included as an annex to the Evaluation report (see ADS 201mah). 10.4. Evaluation Design Matrix Below is an illustrative evaluation matrix to organize data collection efforts. The evaluation team is requested to complete/refine the below matrix as much as possible based on its proposed evaluation design and methodology, which must be included in the 60 Inception Report. This should be based on the information provided in this PROJUST Evaluation SOW. Table 3: ProJust Evaluation Design Matrix Evaluation Questions Evaluation Sub￾Questions* Illustrative Indicators Data Source/ Collection Methods Sampling/ Selection Criteria Data Analysis Method *Fill this column only if there are sub-questions closely linked to the main question. 10.5.Special Considerations for the Proposed Evaluation During the design, implementation, and finalization of the evaluation and its deliverables, the evaluation team should be cognizant of the following limitations: • There are two different PROJUST implementing entities, working under a single contract: MSI is the main contractor and TetraTech DPK is responsible for implementing two of the now-four PROJUST tasks as specified above, under a Teaming Agreement, which is part of the MSI-USAID contract. • As discussed previously, the geographic nature of PROJUST is complicated: both implementing partners overlap in their work in some states and/or municipalities, but not all; there are two tiers of locations for MSI; different tasks are being implemented in different locations; etc. • Access to data through data collection may be of limited quality or may not be easily accessible, based on where it is located and political will. We ask the evaluation team to identify any other special considerations in the Inception Report and the final Evaluation Report. 11.DELIVERABLES All deliverables must be written and presented in English. However, certain presentations and discussions may be conducted in Spanish. A sixteen-week evaluation schedule should be proposed by the evaluation team. The evaluation team must provide the following deliverables to the USAID Contracting Officer Representative (COR): a) Inception Report: Due on the tenth business day of evaluation implementation. It should be 10 pages maximum and must include: • Detailed evaluation design and methodology; • Identifying data to be collected, including precise explanation of data collection methods that will be used for each evaluation question; • Sampling plan; • Proposed list of key informants and focus group participants; • Data collection tools; 61 • Detailed data analysis plan; • Detailed evaluation schedule and logistics, including debriefing with USAID; and • Roles and responsibilities of each evaluation team member. b) Oral Briefing: The evaluation team will provide an oral exit briefing in English and/or Spanish presenting the team’s preliminary findings and recommendations prior to the departure of the evaluation team’s expatriate members. The oral briefing should not contain a PowerPoint presentation. c) Draft Evaluation Report: Due at the end of Week 13. This should include preliminary findings, conclusions, and recommendations, and should follow the requirements of the Final Report listed below. USAID will submit comments and suggested revisions to the evaluation team by the end of Week 14. d) Final Report: The Final Report will also be submitted by the end of the Week 15, after receiving comments and suggested revisions from USAID at the end of Week 14. The report must be no more than 40 pages and must meet the quality standards outlined in the Evaluation Policy. (https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf) The evaluation team will use the USAID Evaluation Report Template for the final report, which can be downloaded from USAID’s Learning Lab at: https://usaidlearninglab.org/library/evaluation-report-template The Evaluation Report must include: • Acronyms; • Table of Contents; • List of Annexes; • List of Tables; • List of Graphs; • Executive summary: No more than three pages. Brief summary of evaluation purpose and questions; general conclusions; and recommendations; • Evaluation methodology and limitations; • Findings, Conclusions, Lessons Learned and Recommendations: Findings can be presented together; conclusions and recommendations broken down per evaluation question; • Findings should be presented as facts, be concise and supported by strong quantitative and qualitative evidence; and • Appendices: Should include the Evaluation SOW; any SOW amendments; questionnaire formats; and any other information and data that was not required in the Report. e) Final Presentation: the evaluation team’s final presentation on findings, conclusions, and recommendations will be conducted, through a PowerPoint presentation, by the end of Week 16. The presentation sessions should observe the following structure: • Presentation should be no longer than 45 minutes • Presentation will be in English, but Q&A session may be conducted in Spanish 62 11.SCHEDULE AND LOGISTICS The evaluation should be completed in 16 weeks and should start in/around October 2017. Below are the proposed PROJUST Evaluation activities narrative of expected logistics and activities. The evaluation team must create an Evaluation Calendar based on the below activities. A six-day work week is authorized in the field with no premium pay. Following are detailed Activities per week: • Week 1: Planning: Conduct preliminary desktop review and plan evaluation logistics and Inception Report. • Week 2: Planning: On Friday of this 2nd week, the evaluation team will submit its Inception Report. In addition, it will continue its desktop review. • Week 3: On Tuesday of this 3rd week, the evaluation team will have a telephonic conversation with USAID to discuss the Inception Report and any other issues. The Evaluation COR will send Inception Report comments to the evaluation team by Tuesday, close of business. By Wednesday, close of business, the evaluation team must respond to comments, if any. By the end of this week, the Evaluation COR must approve the Inception Report. No other planning is anticipated this week other than the telephone call and any adjustments to the Inception Report based on the written comments provided by the Evaluation COR. • Weeks 4, 5, and 6: Data Collection: The evaluation team will conduct field work and any other data collection work required by the evaluation SOW for approximately three weeks. • Week 7: No work required of evaluation team during this week, other than logistical preparation for weeks 8, 9, and 10. • Weeks 8, 9, and 10: Data Collection: The evaluation team will conduct field work and any other data collection work required by the evaluation SOW for approximately three weeks. • Weeks 11, 12, and13: Data Analysis, Report Writing and Submission of Draft Report: Conduct data analysis and begin drafting the evaluation report. By the end of week 13, the evaluation team should submit the Draft Evaluation Report, which will be reviewed by USAID/Mexico. • Week 14: Review of Draft Report by USAID: USAID/Mexico will review the Draft Evaluation Report and provide comments back to the evaluation team by the end of this week. No work required of the evaluation team this week. • Week 15: Finalization and submission of Final Report: The evaluation team will incorporate comments and submit Final Report by the end of week 15 for the Evaluation COR to clear. ● Week 16: Final Report Presentation: The Team Lead and Evaluation Specialist must conduct an in-person PowerPoint presentation, followed by a question-and-answer session, which should last approximately 1.5 hours. Two other presentations, especially for GOM stakeholders and partners are required; each of these two other presentations will require different presentations and should last one hour. USAID/Mexico will assist in facilitating contact with the relevant implementing partners, GOM, civil society, the private sector, and other donors. The evaluation team should 63 budget for and arrange own transport and accommodation to/from and within the country to meet relevant interviewers. Any travel visas should be obtained before arriving in Mexico. In addition, the evaluation team may request reasonable delays or USAID may extend this schedule to accommodate unforeseen events, i.e., unavailability of interviewees, holidays, USAID staff unavailability. 12.EVALUATION TEAM COMPOSITION The core evaluation team should be comprised of at least five people: the Team Leader; a Senior Rule of Law/Evaluation Specialist; two local Rule of Law/Specialists; and an Administrative Assistant. The Contractor may suggest additional team members, if necessary. All team members should be proficient in utilizing relevant and necessary statistical evaluation packages, and Microsoft Word, Excel, and PowerPoint; the Team Leader and Senior Rule of Law/Evaluation Specialist should have developed skills in statistical analysis and data presentation, must have experience working in Mexico, and must be familiar with USAID evaluation standards and practices. Specific required and desirable skills for each evaluation team member are detailed below. In order to preserve the integrity of the final written work product, none of the consultants will have worked for PROJUST (which means none of the consultants will have been employed by MSI or DPK in Mexico from the time that each company began its implementation) or may not have been part of the design of these projects or activities. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. Evaluation Team Leader: S/he will be responsible for designing, implementing, and managing the evaluation; developing evaluation methodologies and tools; managing the compilation and presentation of all evaluation deliverables, including any writing, editing, and data analysis. He/she will be the principal interlocutor between the evaluation team and USAID/Mexico. The Team Leader (American or Third Country National - TCN) must have: • A Master’s Degree or a Juris Doctor and must be a rule of law specialist; • At least five years of experience participating in rule of law programming; • Experience leading complex, multi-site evaluations, with at least 10 years of overall evaluation experience, including six years of experience evaluating rule of law programs; • Strong presentation experience and skills; • Strong management experience and skills; • Excellent English and Spanish communication skills (level 3/3 Spanish is required); and • Previous experience in the Latin America region. Senior Rule of Law/Evaluation Specialist: S/he will work alongside the Team Leader in preparing all evaluation deliverables and performing data collection and analysis, and other tasks to be assigned by the Team Leader. Senior Rule of Law/Evaluation Specialist (American or TCN) must have: 64 • A Master’s Degree or a Juris Doctor and must be a rule of law specialist; • At least three years of experience participating in rule of law programs; • At least five years of experience in carrying out evaluations of rule of law programs; • Excellent English and Spanish communications skills (level 3/3 Spanish is required); and • Previous experience in the Latin America region. Local Rule of Law Specialists: They will support the Team Leader to perform data collection and analysis and other tasks to be assigned by Team Leader. This may include preparing evaluation deliverables and supporting the Administrative Assistant in arranging meetings. At least one of the Rule of Law Specialists (Mexican citizen) must have: • Academic background or experience in law, a social science, or social research; • University degree in law, a social science, public administration, public policy, or a related field; • Solid demonstrated understanding of political dynamics, political actors, and context at state level; and • At least five years of relevant work experience, to include experience in data entry, compilation, and reporting. Administrative Specialist: S/he is in charge of arranging (with the assistance of the two Rule of Law Specialists, if necessary) all meeting in Mexico City and in the states where the evaluation team will travel. Administrative Specialist (Mexican citizen) must have: • Five years of experience in providing administrative support in activities related to the Mexican government, Mexican civil society organizations, or the Mexican private sector. Annexes: Annex 1: USAID Evaluation Policy Annex 2: List of PROJUST activity sites by State and Municipality Annex 3: PROJUST Results Frameworks and Outcome Indicators Annex 4: PROJUST Performance Indicators 2014 - 2016 65 SOW Annex 2: List of PROJUST Current Activity Sites by State, Recipient Institution, and Municipality Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Task 1: Promote increased legitimacy of the criminal justice system/MSI Baja California Attorney General Office Mexicali Public Defender’s Office Mexicali State Court Mexicali Baja California Sur Inter-Institutional Coordination La Paz Attorney General’s Office La Paz Pre-Trial Service Units La Paz Coahuila Attorney General Office Saltillo Pretrial Service Unit Saltillo Public Defender’s Office Saltillo State Court Saltillo Chihuahua Pre-Trial Service Unit Chihuahua State Court Chihuahua 66 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Morelos Pre-Trial Service Units Cuernavaca State Court Cuernavaca Nayarit Attorney General’s Office Tepic State Court Tepic Nuevo Leon Attorney General Monterrey State Court Monterrey Public Defender’s Office Monterrey Pretrial Service Monterrey San Luis Potosí Attorney General’s Office San Luis Potosí State Court San Luis Potosí Pre-Trial Service Unit San Luis Potosí Public Defender’s Office San Luis Potosí Sonora Attorney General’s Office Hermosillo 67 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Pre-Trial Service Units Hermosillo Tabasco Attorney General’s Office Villahermosa Public Defender’s Office Villahermosa Yucatán Inter-Institutional Coordination Mérida Zacatecas Inter-Institutional Coordination Zacatecas Pre-Trial Service Unit Zacatecas Public Defender’s Office Zacatecas State Court Zacatecas Task 2: Increased capacity for monitoring and evaluation of criminal justice system operation/MSI Baja California Public Defender’s Office Mexicali Coahuila Attorney General’s Office Saltillo State Court Saltillo Public Defender’s Office Saltillo Jalisco Attorney General’s Office Guadalajara 68 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Nuevo León Public Defender’s Office Monterrey Nayarit State Court Tepic San Luis Potosí Attorney General’s Office San Luis Potosí State Court San Luis Potosí Tabasco Attorney General’s Office Villahermosa Zacatecas Attorney General’s Office Zacatecas Public Defender’s Office Zacatecas Task 3: Increase public support for the New Criminal Justice System/TetraTech DPK Baja California Citizen Observatories for Justice Reform Implementation Mexicali Chiapas Citizen Observatories for Justice Reform Implementation Tuxtla Chihuahua Citizen Observatories for Justice Reform Implementation Chihuahua 69 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Coahuila Citizen Observatories for Justice Reform Implementation Saltillo Citizen Platform “Justice for You” (Locatel) Saltillo Estado de Mexico Citizen Platform “Justice for You” (Locatel) Toluca Hidalgo Citizen Observatories for Justice Reform Implementation Pachuca Monitoring of compliance with Pre-Trial Conditions by CSOs Pachuca Michoacán Citizen Observatories for Justice Reform Implementation Morelia Monitoring of compliance with Pre-Trial Conditions by CSOs Morelia San Luis Potosí Increase of Public Support through Journalism San Luis Potosi 70 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Monitoring of compliance with Pre-Trial Conditions by CSOs San Luis Potosi Citizen Observatories for Justice Reform Implementation San Luis Potosi Tabasco Increase of Public Support through Journalism Villahermosa Citizen Platform “Justice for You” (Locatel) Villahermosa Zacatecas Increase of Public Support through Journalism Zacatecas Task 4: Support increased access to justice and improved victims’ assistance - TetraTech DPK Chihuahua Women’s Justice Center Chihuahua, Ciudad Juarez Citizen Observatory Chihuahua Coahuila Alternative Dispute Resolution Mechanisms Saltillo Women’s Justice Center Saltillo 71 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Monitoring of compliance with Pre-Trial Conditions by CSOs Saltillo Hidalgo Monitoring of compliance with Pre-Trial Conditions by CSOs Pachuca Women’s Justice Center Pachuca State of Mexico State Victims’ Assistance Commission Toluca Jalisco Women’s Justice Center Guadalajara Michoacán Citizen Observatory Morelia Monitoring of compliance with Pre-Trial Conditions by CSOs Morelia 72 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Morelos Monitoring of compliance with Pre-Trial Conditions by CSOs Cuernavaca Nuevo León State Victims’ Assistance Commission Monterrey Citizen Observatories for Justice Reform Implementation Monterrey Monitoring of compliance with Pre-Trial Conditions by CSOs Monterrey Puebla Women’s Justice Center Puebla Quintana Roo State Victims’ Assistance Commission Chetumal San Luis Potosí Alternative Dispute Resolution Mechanisms San Luis Potosí Women’s Justice Center San Luis Potosí 73 Activity/Implementer State Partner Justice Sector Institutions/Sub-Task Municipality Citizen Observatories for Justice Reform Implementation San Luis Potosí Monitoring of compliance with Pre-Trial Conditions by CSOs San Luis Potosí Sonora Alternative Dispute Resolution Mechanisms Hermosillo Women’s Justice Center Hermosillo Tabasco Alternative Dispute Resolution Mechanism Villahermosa Yucatán Women’s Justice Center Merida Zacatecas Women’s Justice Center Zacatecas States where MSI is active (States with an asterisk* are states where both MSI and TetraTech DPK are active:) • Baja California* • Baja California Sur • Chihuahua* • Coahuila* • Jalisco* • Morelos* 74 • Nayarit • Nuevo León* • Sonora* • San Luis Potosí • Tabasco* • Zacatecas* States where only TetraTech DPK is active: • Chiapas • Estado de Mexico • Hidalgo • Puebla • Michoacan • Quintana Roo • Yucatan 75 SOW Annex 3: PROJUST Results Frameworks and Outcome Indicators D0 2 RESULTS FRAMEWORKS A. Original PROJUST Results Framework at signing, 2014 76 B. County Development Cooperation Strategy Results Framework “Constitutional Criminal Justice Reforms of 2008 are operational.” 77 C. Revised result framework (2017) “The legitimacy of the government will be increased with a more transparent and responsive justice system achieved through greater accountability, effectiveness, and respect for human rights in the justice system.” 78 Cifra negra “Black Number” refers to the number of crimes that go unreported or unpunished. 79 Projust Outcome Indicator Baseline and Status (Percentage values. The values reflect only data for states where PROJUST is operating on sub￾tasks relevant to the indicator.) 80 Note: Some missing data will be updated; targets are being revised. 81 SOW Annex 4: Performance Indicators 2014-2016 MSI was responsible for implementing Tasks 1, 2, and 4 of the following list and TetraTech DPK was responsible for implementing Tasks 3 and 5. PROJUST’s custom indicators below are presented following the Implementing Partner’s original structure and division of tasks. Description and other characteristics can be found in PROJUST Performance Indicators 2014 - 2016. Task 1: Advance Legislative Reform • 1.1.1 # of targeted states that revise or draft basic secondary laws that are in compliance with the 2008 constitutional reform, HR standards, and best practices • 1.2.1 # of targeted JSIs with procedures aligned to the NCPC and secondary legislation • 1.4.1 # of states receiving assistance from the project to draft secondary legislation Task 2: Increase Prosecutorial and Judicial Effectiveness • 2.1.1 Pre-trial detention rates for youth (14-18 y/o) in targeted jurisdictions • 2.1.2 Pre-trial detention rates for adults in targeted jurisdictions • 2.2.1 Criminal case disposition rates in targeted courts • 2.2.2 # of targeted judges who demonstrate increased capacity to effectively preside over oral hearings • 2.2.3 # of targeted prosecutors who demonstrate proficiency in oral litigation skills • 2.2.4 # of targeted PDs who demonstrate proficiency in oral litigation skills • 2.3.1 Prosecution rates in targeted investigation and prosecution units • 2.4.1 # of performance and training standards produced Task 3: Increase Public Support for the New Criminal Justice System (NCJS) • 1. Number of state SETECs that draft communication plans to promote the reform • 2. Number of state-level institution communication products supported by SETEC • 3. Percentage of users who have heard of the criminal justice system reform in their state Task 4: Strengthen Analytical and Quality Control Capabilities of Targeted Justice Institutions • 4.1.1 # of targeted JSIs that have quality control systems • 4.2.1 # of SETEC Monitoring and Evaluation initiatives directed to Criminal Justice Reform developed and/or implemented Task 5: Increase Access to Justice and Improved Victims’ Assistance • 4. Percentage of cases resolved through alternative justice in targeted states. • 5. Percentage of users satisfied with the services provided by the VAUs and WJCs 82 ANNEX 2: INCEPTION REPORT FOR THE EVALUATION 83 MID-TERM EVALUATION OF USAID/MEXICO Promotion of Justice Project Inception Report SUBMITTED: NOVEMBER 17, 2017 84 Mid-Term Evaluation of USAID/Mexico Promotion of Justice Project Inception Report Prepared for the United States Agency for International Development Under USAID Monitoring & Evaluation & Learning Initiative USAID/El Salvador Contact: Orlando Hidalgo ohidalgo@usaid.gov USAID/ Monitoring, Evaluation and Learning I Contact Prepared for the United States Agency for International Development Contracted under AID-519-TO-16-00002 USAID/El Salvador Monitoring, Evaluation, and Learning Initiative DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 85 ACRONYMS ADRM Alternative Dispute Resolution Mechanisms CDMX Mexico City (formerly, Federal District) CIDAC Center for Research for Development CMIS Case Management Information System CONATRIB National Commission of Superior Tribunals CSO Civil Society Organization Edomex Mexico State (Estado de Mexico in Spanish) GOM Government of Mexico INEGI National Institute of Statistics and Geography JSI Justice Sector Institution KPI Key Personnel Interviews (also Key Performance Indicator, but not so used here)64 LGBT Lesbian, Gay, Bisexual, and Transgender LOP Life of Project M&E Monitoring and Evaluation MSI Management Systems Incorporated NCJR National Criminal Justice Reform PSO Public Sector Organization PROJUST Promotion of Justice Project SETEC Technical Secretariat for the Coordinating Council for the Implementation of the Criminal Justice System SOW Statement of Work TL Team Leader Tt DPK TetraTech DPK WJC Women’s Justice Center 64 Note in the Table in Annex 1, to avoid confusion between the two terms, Key Performance Indicator is KPIc and Key Performance Interviews is KPIv. 86 INTRODUCTION 1.1 PURPOSE OF THE EVALUATION This is a mid-term evaluation of USAID/Mexico’s Promoting Justice Project (PROJUST). PROJUST’s specific objective has been slightly modified over its three years of activity and currently is defined as “to provide customized direct or indirect support at the state level and to federal authorities responsible for leading the implementation of the 2008 Constitutional criminal justice reform, assisting selected Mexican states to pass the ‘point of no return’ on their reform processes, and support the consolidation of the system past the June 2016 deadline.”65 In this context, the evaluation is tasked with: 1) assessing the extent to which PROJUST’s support has been effective in advancing and ensuring the sustainability of the implementation and consolidation of the National Criminal Justice Reform (NCJR); and 2) providing recommendations for future programming. It should be noted that the evaluation will have to deal with several unusual circumstances: • Successive changes in the project’s tasks (or objectives) 66 and results indicators over the 3-year period from inception (mid-2014) to the present; • The division of responsibility for the (currently) four major tasks between two project implementers, Tetra Tech DPK (henceforth Tt DPK) and Management Systems International (MSI), with Tt DPK having entered the project six months after MSI; • Reduction (from 29 to 19) of the states covered during the Life of the Project (LOP), differences in the number and identity of states covered by each implementer, and the further division of those remaining where MSI operates into first-tier states (where activities are more comprehensive) and second-tier states (where support is limited to discreet interventions in selected institutions); and • Disbanding of one principal Government of Mexico (GOM) partner, the Technical Secretariat for the Coordinating Council for the Implementation of the Criminal Justice System (SETEC), in October 2016, meaning that issues of PROJUST contributions to its sustainability (but not effectiveness until that date) are no longer valid.67 1.2 EVALUATION AUDIENCE As indicated by the Scope of Work (SOW), the audience for the evaluation includes USAID/Mexico, the two PROJUST implementers, and justice stakeholders ranging from GOM officials and Justice Sector Institution (JSI) personnel to civil society organizations (CSOs) and the private sector. 1.3 EVALUATIONS QUESTIONS 65 As per the Amendment of Contract between MSI and USAID, dated March 8, 2017. 66 USAID apparently now reserves “objective” for what was once called project “purpose,” and refers to what were once call objectives as “tasks.” This terminology will be used here as well. Task results are now measured with a series of key performance indicators, with at least one and often several for each task. 67 Some but not all the responsibilities of SETEC have been transferred to the Executive Secretariat of the National Public Security System (SESNSP), which no longer works with the judiciary. 87 USAID’s questions (outlined in the table in Annex 1) focus on three general areas: 1) project effectiveness in producing positive changes; 2) sustainability of results; and 3) recommendations and lessons for future work. Although this is not an impact evaluation, the sub-questions for the first two areas focus on attribution or the extent to which and how PROJUST contributed to the changes that did occur. While not specifically asked to do so, this means the team will verify reported advances and identify any unreported positive impacts, as well as determining the PROJUST contribution. 2.0 DETAILED EVALUATION DESIGN AND METHODOLOGY The evaluation will employ a mixed-method methodology, and will be conducted in the following three stages: 1. Document review by team to gather some basic data and general information on project activities, objectives, tasks, and indicators, and reported progress made. While always a part of an evaluation, this phase is particularly important here because of the project’s complex design and history, as noted above. Document review will also extend to material available from other sources, including GOM agencies, CSOs, universities, think tanks, and independent academics and journalists, which publish relevant material. Some of this review will continue through the evaluation. The initial document review is the basis for this inception report, the first deliverable, which must be submitted to and approved by USAID prior to the initiation of fieldwork. 2. Fieldwork: as per USAID instructions, field work will largely be devoted to interviews with a pre-set list (by position) of key personnel, complemented by focus groups, observations and, as available, compilation of additional statistics and other hard data. The fieldwork will have two aims: collection of both quantitative and qualitative data to generate quantitative indicators in response to USAID questions and, of further, more purely qualitative information on project design, structure, impediments, and challenges. 3. Data analysis and report writing. The second and final deliverable is a forty-page paper (plus annexes), to be submitted to USAID for its approval as per the evaluation timeline (Annex 2). Verification of progress indicators, a product of stages 1 and 2 of the evaluation, will be done quantitatively for the most part—reviewing implementer statistics and, to the extent possible, checking them against statistics provided by the JSIs in each state and by other sources [for example, the National Institute of Statistics and Geography (INEGI)]. An initial review of the statistics used as results indicators suggests they need further refinement, as they vary between the two implementers (as well as among other organizations tracking reform outcomes). Where expressed as percentages, it is sometimes unclear as to how the indicators are calculated (i.e., against which base number or denominator). Also, the method of calculations seems to vary from year to year (e.g., case clearance rate calculated either only for cases 88 entered in the same year or for all caseload, including pending).68 Likewise, while some statistics calculate prosecution rates against all denuncias (i.e., complaints), others only calculate them against investigations opened.69 Both figures are important, and if they are not included in the most recent reports, the team will still try to get them. The process of verifying and refining indicators is unlikely to negate results, but it will certainly clarify what they are, and that will be necessary for establishing attribution and sustainability. Given the time and budget allowed as well as the lack of data from states not included in the project’s later stages, a preferred methodology for answering the attribution issues—quasi experimental with control groups—is not possible. It also was not requested by USAID. (It would be difficult in any event to use a quasi-experimental design for two further reasons: the decision by USAID and the two implementers to narrow the initial project focus from 29 states to 19 and the selection for MSI’s concentrated focus of states considered to be “good performers.” A further challenge to addressing attribution issues is separating post-2014 effects from changes already begun under and possibly largely attributable to the prior MSI projects. Issues of Attribution While USAID is understandably interested in the team’s tracking the impact of PROJUST activities on registered positive changes in JSI performance, it has complicated this task in a series of ways—first by insisting that fieldwork be done only in the states and districts within them where PROJUST now operates (thus eliminating any sort of control group for trends that may be nationwide); and second by its emphasis on interviews, which while an excellent source of insights on project operations, threaten various types of bias in measuring their importance. A suggested survey (now removed from the SOW at the team’s request)70 would have faced similar problems of potential response biases as well as the “voting” logic behind both methods—i.e. the notion that majority’s answers count more than minority’s ones. Intensive interviewing often reveals that minority views provide deeper insights on process details than do mass answers to simple questions. The team will nonetheless append a simple questionnaire to all interviews to capture views lending themselves to quantification, responding to USAID desires as expressed in the SOW, and apply a more purely quantitative approach to track impacts against PROJUST activities in each targeted state or district. 68 The latter is the conventional method, but apparently in Mexico, the tendency is to use only dispositions of cases entered in the same year (see for example, CIDAC, Hallazgos 2016). The Mexican method is not only unconventional, but also makes it difficult to evaluate results, since it is virtually impossible to attain a 100 percent clearance rate (the internationally accepted target) given that cases introduced toward the end of the year are unlikely to be disposed (cleared) in the same year. This is particularly true of complex cases (e.g., homicides without a suspect identified at the start) but it also affects even simple, in flagrante cases. This also means that a target clearance rate using the Mexican system should vary depending on the type of case. 69 Part of the problem here is PROJUST’s dependence on statistics presented by JSIs. It has also done its own data collection, but in some instances, uses JSI reports, which unfortunately calculate statistics differently—apparently in some states or institutions with variations from year to year. 70 The elimination of the survey, however, was done because of a series of logistical issues—lack of a database from which to draw a sample, time required for development and testing of questionnaires, and absence of any mention in the SOW of additional personnel to apply the survey in the field. Software like Survey Monkey can be applied electronically, but for that, emails of respondents are required, and even when available, the response rate is typically quite low (in the single or low double-digit range). Email surveys or those simply provided to respondents for self-administration also pose interpretation issues. In one survey conducted recently for a Mission in Africa, among the few who answered the questionnaire, half rated a course they had never taken. 89 2.1 DATA TO BE COLLECTED, DATA COLLECTION TOOLS, AND PROPOSED DATA USE As further detailed in the table in Annex 1, the methodology uses both quantitative and qualitative data collected through several means and analyzed in various fashions. 2.1.1 Quantitative data sources and use Quantitative data will include statistics collected by MSI and Tt DPK on progress on indicators (measured against baselines) as well as their records on project activities (dates, specific activity, locations, participants) and, to the extent feasible, statistics offered by JSIs and other sources. Analysis of quantitative data will focus on two themes: • Verification of reported progress vis-à-vis project indicators and further refinement of their significance; and • A comparison of PROJUST activities in each state with the reported progress as a means of determining the relative efficacy of different types of interventions (e.g., short￾term training versus “intensive” interventions). As suggested above, verification and refinement of progress indicators will be done through “triangulation” (comparison with alternative sources) and interviews with MSI and Tt DPK staff for a better understanding of how indicators are constructed and measured. Although the team has received M&E plans and other relevant documents, it still has questions in this area that must be explored before reaching its determination of what has been accomplished. In its treatment of the earliest indicators of “NCJR implementation” the team will simply count laws enacted post-late 2014,71 as in the implementers’ reports. Indicator verification and refinement will instead focus on the progress of the improved set of results indicators, introduced in 2016, but not officially incorporated (with USAID approval) until 2017. Quantitative data (outcome indicators and types/levels of intervention and, where possible, data from states or districts where PROJUST has operated less or not at all in the last two years) will be used as one means of identifying PROJUST’s contributions to change (attribution). This will be complemented by data from interviews and focus groups as explained below. Initial data provided in the SOW suggest considerable variations in progress among the states tracked on both sets of outcome indicators (i.e., laws and performance). This should permit some form of difference-in-difference analysis of activities’ impacts. If the team can get quantitative data from other sources (JSIs’ own records, sources like INEGI, Center for Research for Development (CIDAC), National Commission of Superior Tribunals (CONATRIB), Jurimetria, and various academic and journalistic publications), the team may also be able to expand this review by comparing progress in states or districts given less or no attention with those targeted by the project. While the period covered is very short, the team will also be looking for indications of sustainability of impacts—whether initial progress appears to continue or drops back 71 It bears mentioning that the SETEC index of reform implementation, also included as an initial PROJUST indicator, also refers only to passage of laws. For both PROJUST and SETEC, actual behavioral change is called reform “consolidation,” which is what the later indicators attempt to track. 90 (“regresses to the mean”) in subsequent years.72 Here, the team will also use both quantitative indicators (budgets, actual or planned expansion of pilot programs, staffing of the same) and qualitative data as described below to assess sustainability of results. 2.1.2 Qualitative Data Sources and Use Qualitative data will come from documents, KPIs (Key Personnel Interviews), focus groups, and observations. In the interest of converting interview results into something more quantified, a short questionnaire will be appended at the end of each interview. The questionnaire, which will also be used as a focus group guide, is provided in Annex 4 as an illustrative example of how this will be done. The AT will not be doing sophisticated, software-aided version of content analysis of interviews (e.g., Grounded Coding), for lack of time and resources for what is a very complex process.73 For the so-called implementation stage (to mid-2016 and referring to the legislative framework), the team will direct interviews and questionnaires toward the issue of the extent of PROJUST impact on these outcomes. We will be unable to determine a counterfactual [except for Mexico State (Edomex in its Spanish acronym), where PROJUST apparently did not operate]—what would have been accomplished without PROJUST input—and will have no time to review “quality” of this input to or the content of any laws; however, the team will, to the extent time allows, attempt to determine whether PROJUST’s input made a difference (i.e., whether, as in Edomex, passage would have occurred without PROJUST’s support). Interviews, observations, and further document review will also be used to inform the team on details of project design, evolution, and specific activities, as well as local contextual factors that may influence adoption of new procedures and practices. This is the technique featured in prior evaluations for USAID and is a rich source of examples and stories that make the results more interesting to the reader. It also is a means of drilling down into details unlikely to emerge in a questionnaire, survey, or pre-structured interview which, of necessity, will have to be designed before the team has done any preliminary fieldwork. (While data collection tools could be modified over the course of the evaluation, much modification will complicate a comparison of answers.) The team will be cautious in the value accorded to anecdotal information, but it is nonetheless important for a fuller understanding of the processes under study. Moreover, given USAID’s insistence on the number of specific key performance interviews (KPIs) to be included, these interviews will occupy the major portion of their time in the field and, thus, will be utilized to the maximum effect. The team will also use focus groups as an additional source of qualitative data, using a preset format (modeled on the questionnaire) to encourage comparability in the results. The issue here is the lack of prior access to lists of potential participants, as well as temporal and budgetary constraints, meaning that these participants will have to be located in the sites visited—in short, the team cannot bring them in from farther afield. The team will also be 72 The regression to the mean phenomenon may also be aggravated by a characteristic often associated with pilot projects. Pilots do better because they feel special (a sort of Hawthorne effect), but, in the process of scaling up, this effect often wears off and both the initial pilot, but especially the later replications, may perform less well, although one hopes better than under the status quo ante. 73 The team also suggests that if USAID is interested in this type of analysis it allows sufficient time and resources (including for software and the transcription of interviews into machine-readable form) for carrying it out adequately. The three weeks allowed in the SOW for data analysis and report drafting could easily have been consumed only by content analysis of an estimated 200 interviews and focus groups. Moreover, it might be well to specify that evaluators be versed in the techniques, although it is unlikely than many/any ROL experts ever use it. 91 dependent on participants volunteered by the implementers or JSIs and has no illusions about their being representative. We will however, insist on as close as possible to a 50-50 split between women and men. Focus group participants will most likely be JSI employees, members of CSOs and private sector organizations (PSOs) that work with the project, or service users identified by any of these agencies. This is unavoidable, given the team’s lack of access to lists of all members or users of these organizations. Interviews and focus groups will also be used to flesh out the analysis through the identification of perceived state-level factors facilitating or impeding progress. For example, these might include budgetary constraints in staffing new offices, changes in governors as well as JSI leadership, and the role of reported judicial resistance to new pre-trial detention programs or to the use of plea bargaining, both of which vary by state—a possible consequence (detectable quantitatively) of when activities were introduced and how many actors were reached, but also possibly attributable to differences in local “judicial” culture. 2.1.3 Sampling plan As the team will not be doing surveys or using comparable mechanisms, there is no sampling plan. Data (statistics, interviews with questionnaires, focus groups, and observations) will be collected within the eight states identified, focusing on the key informants and for focus groups, organizational members as identified by USAID in the SOW. Given time constraints and the number of KPIs stipulated by USAID, there will be no opportunity to select respondents (or states) randomly. The team has been given some leeway in the selection of the eight states where it will do fieldwork. The following table of proposed states uses three criteria: 1) states where both implementers operate (here all but one, Baja California Sur); 2) states with Women’s Justice Centers (four included); and 3) states in MSI’s first tier (necessary because these are the ones where the greatest number of JSIs have been involved). The selection of course biases the results toward the “most successful” or at least most frequent recipients of assistance but, given the perceived differences in their scores on various indicators, the team believes there are sufficient variations to be investigated. Table 1: Team’s suggested list of states to be covered by the evaluation Baja California Morelos Baja California Sur San Luis Potosi Coahuila Tabasco Chihuahua Zacatecas 2.2 GENDER ANALYSIS The SOW requests that the evaluation disaggregate impacts by gender, as appropriate, and as data permit. USAID has not indicated which indicators it wants treated in this manner, so the team will use its own judgment here, based in part on how available quantitative data are presented (i.e., whether tracked by a gender factor). KPIs, the associated questionnaires, and focus groups will record gender of participants, and differences in responses will be identified and reported. For focus groups, the team will attempt a 50/50 ratio of men and women. Participants will not be asked other gender identification as this is a sensitive issue in Mexico (as well as in many other places.). However, we will record any comments on situations affecting 92 Lesbian, Gay, Bisexual, and Transsexual (LGBT) persons (as well as those related to ethnicity, age, or other possibly discriminatory categories). 2.3 PROPOSED LIST OF KEY INFORMANTS, FOCUS GROUP PARTICIPANTS As per USAID instructions, the key informants and focus group members will be the following: • State Attorneys General (AG) and staff members • State Coordinators of Criminal Reform Implementation* • Heads of Specialized Units within State Attorney General Offices and staff members* • Directors of Women’s Justice Centers and staff members* • Directors of Pre-Trial Service Units and staff members* • Heads of Alternative Dispute Resolution Mechanism offices/units and staff members* • Spokespeople of Attorneys General Offices • Spokespeople of State Judiciaries and judges* • Justice and security section editors/journalists in local newspapers/news channels • Representatives from civil society* • Representatives from the private sector (including COPARMEX)*74 • Representatives from academia* • USAID, MSI and Tt DPK staff (added by evaluation team) Asterisks (*) denote possible focus groups, although given time constraints not all will be conducted. Based on the interviews in Mexico City (formerly, Federal District) (CDMX) and the first interviews in a state, decisions will be made as to which groups (or mixes of groups) will be used. This also may vary by state depending on local authorities’ willingness to help recruit focus groups members. 2.4 DETAILED DATA ANALYSIS PLAN See section above and the table in Annex 1 for more details. 3.0 DETAILED EVALUATION SCHEDULE AND LOGISTICS For the timetable of the evaluation, please see Annex 2. Until the list of states is approved by USAID, more detailed information on when each state will be visited cannot be provided. However, the team plans to spend the first fieldwork week in CDMX, interviewing staff of USAID, MSI, and Tt DPK, as well as any former members of SETEC who can be located, and a few other institutions that track reform progress. During the second week of fieldwork, the team will operate together to test data collection tools and techniques in one state. After that, the team will divide into two groups, probably reconsolidating during the last week of fieldwork 74 PROJUST documents suggest that private sector initially was equated with the private bar, but that over time business associations have been added. 93 (now scheduled for the first week of February 2018). This should allow us to cover the eight states required by USAID, three in the first period of fieldwork and five in the second. 4.0 ROLES AND RESPONSIBILITIES OF TEAM MEMBERS Note: the first four team members (but not the logistics expert) will be required to take notes on all interviews and focus groups. At least one team member will be assigned to record answers to the questionnaires. Team Leader • Act as the principal point of communication with USAID and with Monitoring & Evaluation and Learning (M&E&L) Initiative. • Principal responsibility for drafting inception report and draft and final evaluation reports (with inputs for both from other team members) • Design data collection tools and oversight of further analysis • Assign tasks to team members • Coordinate with local logistics person and other team members on travel, hoteling, and other arrangements • Participate in, coordinate, or conduct all interviews, when team works as one, and those under her supervision when team splits into two • Ensure questionnaires are filled out in each interview • Coordinate with local logistics person, USAID, Tt DPK, and MSI on setting interview schedule • Oversee second local specialist in creation of databases and analysis of contents, if not doing own analysis of the same Senior ROL Expert • Participate in all interviews and, when team divides, take charge of the “half” to which will be assigned • When in charge of a team in the field, ensure interviews and focus groups follow agreed upon format and that questionnaires are filled out • Draft portions of the final report • Help Team Leader (TL) with data analysis and provide inputs on inception plan • Analyze, as agreed with TL, quality of laws, manuals, and specific project activities based on own experience in Mexico or other countries • Based on experience in other countries, provide comparative insights into how similar activities and pilot interventions have operated elsewhere, and where they have run into problems Local ROL Expert • Participate in all interviews (except for those conducted by the “other team” when team divides into two parts) 94 • Draft portions of the final report as agreed with TL • Help TL with data analysis and provide inputs on inception plan • Analyze, as agreed with TL, quality of laws, manuals, and specific project activities based on own experience in Mexico or other countries • Help identify CSOs and private groups with which interviews or focus groups should be held • Provide background on reform movement in Mexico and insights into political and cultural issues in each state covered Second Local Consultant • Participate in all interviews with entire team or in relevant team when group splits in two • Assist TL in identifying data available online or from JSIs and other entities • Organize, in coordination with TL, a database for subsequent analysis from available statistics, questionnaires and other sources • Take charge of entering and coding data from the questionnaires • In coordination with TL develop methods for analyzing data and do all or part of data analysis Local Logistics Person • In coordination with M&E&L Initiative, organize transportation (land and air) within Mexico and make reservations as appropriate • In coordination with M&E&L Initiative and TL, make hotel reservations for fieldwork • In coordination with USAID, Tt DPK, and MSI, as appropriate, and with TL, develop interview schedule for each state (and CDMX) and ensure interviewees are contacted • Keep running interview schedule for team and send to team members as updated • Advise TL and the M&E&L Initiative on hotel and transportation options—in the latter case, whether it makes more sense to travel by plane or by land • Function as contact point for interviewees so that any necessary changes in schedule are noted and transmitted to team 95 INCEPTION REPORT ANNEXES 96 INCEPTION REPORT ANNEX 1: TABLE OF INDICATORS TO ANSWER EVALUATION QUESTIONS The table below outlines the data collection and analysis methods to be used in answering the questions posed by USAID. Since the sub-questions for the first two areas are not elaborations of the principal question but, rather, focus on attribution, answers to the two general questions are treated with their own indicators and analytic techniques. For the lessons and recommendations, answers to the general question will be based on data collection and analysis used to answer the two sub-questions. To avoid a 10-page table, the illustrative indicators, data sources, and analytic methods are dealt with in general, not institution by institution. Where relevant, and where data allow, data collection and analysis will be further broken down by gender and by implementer (TT DPK or MSI). See legend below for explanation of acronyms used in table. Evaluation Questions Evaluation Sub-Questions Illustrative Indicators Data Source/Collection Method Sampling/Selection Criteria Data Analysis Method Effectiveness: To what extent has PROJUST met its objectives relating to the functioning of the AG’s Office, Courts,75 Women’s Justice Centers, Pre-Trial Service Units, and CSO activities related to criminal justice reform See below. Changes in in KPIc values for relevant periods (by state and JSI), verified and refined Central records from Tt DPK and MSI and individual ones from any field offices/ers Where possible checked against data from JSIs, and where appropriate interviews/ observations Data from all states on which PROJUST has KPIc data and from local statistics in 8 states selected with USAID Comparison of scores over time and across states/ institutions. Will compare with data (where available) from institutional CMIS. The emphasis is on change, not absolute values as some states clearly were ahead of others in 2014. 75 Although courts were not included in the initial USAID question, several current indicators (case disposition rates; time-to-disposition rates) relate to court performance and thus merit separate consideration. Courts have also been added in the question on sustainability 97 Evaluation Questions Evaluation Sub-Questions Illustrative Indicators Data Source/Collection Method Sampling/Selection Criteria Data Analysis Method To what extent has PROJUST contributed to results as measured by approved indicators (KPIcs)? a. PROJUST activities against changes by state b. % of interviewees or focus group members (in each state, JSI, or CSO, PSO) that mention PROJUST activity as helpful a. PROJUST records b. Interviews and focus groups Data from 8 states and interviews within them—no further sampling a. Compare activities (type, timing) against KPIcs b. Compare % mentions against KPIcs To what extent are PROJUST supported models/protocols/ SOPs used by state JSIs? % of interviewees/focus group members (in each state JSI) that say they have used them Interviews, focus groups, observation Data from PROJUST and interviews/focus groups in 8 selected states. Tabulation of % by state and JSI, supplemented by observation of actual practices 98 Evaluation Questions Evaluation Sub-Questions Illustrative Indicators Data Source/Collection Method Sampling/Selection Criteria Data Analysis Method If models/protocols/SOPs are used, what PROJUST actions helped? If not used, why not? % of specific actions mentioned by interviewees or focus group members (in each state, JSI) as facilitating use. Note taken of actions not mentioned. Interviews, focus groups, observation Data from PROJUST and interviews/focus groups in 8 selected states Tabulation of mentions by state and JSI Prior quantitative analysis (results vs. activities) will also be used here To what extent did PROJUST strengthen local governments, CSOS and private sector to produce/sustain results? % times CSO and PSO interviewees mention PROJUST support as important to their own activities Interviews, focus groups, observation Data from JSI members in 8 selected states Tabulation of percentages by state CSOs and PSOs To what extent are CSOs more empowered? Number of initiatives begun by CSOs to promote NCJR Interviews, focus groups, observation CSOs with which Tt DPK or MSI has worked Tabulation of initiatives by state and JSI Sustainability: To the extent there are results, how sustainable are they in the JSIs (Pre-Trial Service Units, Courts, AG’s Office), Women’s Justice Centers, and CSOs/PSOs See below Local actions in each JSI and CSO to continue PROJUST innovations Records of and interviews with JSIs on budgets, self-financed training, creation, and staffing of new offices Data from KPIvs selected by USAID and JSI records Tabulate by state and JSI or CSO 99 Evaluation Questions Evaluation Sub-Questions Illustrative Indicators Data Source/Collection Method Sampling/Selection Criteria Data Analysis Method What has PROJUST done to promote local ownership? % mentions by JSI, CSO, PSO interviewees and focus group members of specific PROJUST actions as encouraging their support Interviews, focus groups, observation Data from JSI members in 8 selected states Tabulation of mentions in interviews and focus groups, separated by state, JSI, and CSOs, PSOs What has PROJUST done to promote sustainability? # times specific PROJUST actions are mentioned by JSI, CSO, and PSO interviewees as promoting continuation of innovations Interviews, focus groups, observation Data from JSI members in 8 selected states Tabulation of mentions in interviews and focus groups With what additional actors, if any, should PROJUST engage to promote sustainability of current or prospective results? List of recommended new actors (by state) Interviews, focus groups, observation Data from JSI members in 8 selected states Tabulation of mentions within and across states and JSI Lessons Learned and recommendations: What are the lessons learned from PROJUST that should be incorporated in a See below List of recommendations and lessons Based on two sub￾questions (below) but may also incorporate additional information from interviews, focus groups, and observation NA No additional 100 Evaluation Questions Evaluation Sub-Questions Illustrative Indicators Data Source/Collection Method Sampling/Selection Criteria Data Analysis Method new design? Are there any current areas of support within the programming areas reviewed that should be discontinued? List of activities not mentioned as supportive or specifically labeled as non-supportive in interviews and focus groups Interviews, focus groups, observation Data from JSI members in 8 selected states Separate activities by state and JSI Are there existing or emerging opportunities in criminal law and impunity USAID should pursue? List of activities mentioned as needing support or identified by evaluation team based on their own experience Interviews, focus groups, observation Data from JSI members in 8 selected states Separate activities by state and JSI Legend: CMIS = Case Management Information System CSO = Civil Society Organization JSI = Justice Sector Institution KPIc= Key Performance Indicators KPIv= Key Personnel Interviews PSO = Private Sector Organization SOP = Standard Operation Procedures TA= Technical Assistance 101 INCEPTION REPORT ANNEX 2: TIMELINE FOR EVALUATION PROJUST Project 6-10 Nov 13- 17 Nov 20- 24 Nov 27 Nov-2 Dec 4-9 Dec 11- 16 Dec 2-5 Jan 8-13 Jan 15- 20 Jan 22-27 Jan 29 Jan- 2 Feb 5-9 Feb 12- 16 Feb 19-23 Feb 26 Feb-2 Mar 5-9 Mar Desk review and submission of the Inception Report Review Inception Report by USAID Data collection phase 1 Revision and preparation field work phase 2 Data collection phase 2 Data Analysis, Report Writing, and Submission of Draft Report Review Draft Report by USAID Finalization and Submission of Final Report Final Presentation (TL/ES) Note: During week 7, international consultants return home 102 INCEPTION REPORT ANNEX 3: SAMPLE DATA COLLECTION TOOLS Note: the tools are presented in English, and as they will not be given to respondents can remain in that language. However, they may be translated by a team member so as to ensure the same terms are used in Spanish. FORMAT FOR FOCUS GROUPS Explanation: This questionnaire designed for interviews will also serve as the guide for focus groups, with as explained, slightly different details depending on the identity of participants (judge, fiscal, member of PTD, WJC, or MASC unit, CSO, or PSO) and the state or institution from which they come. Although we will collect names, positions, gender, and time in position or institution from focus group members, this will be a separate list. Answers to questions (absolute numbers and percentages) will be tallied for the group. Name of Interviewee or Focus Group Member: Position: Gender: Time in Position and/or Institution: 76 1. Are you familiar with the PROJUST program? Y/N If the answer “no,” try a few other names (MSI, Tt DPK, USAID). If still no, modify following questions accordingly, but it is unlikely that no one will recognize any name. 2. What do you understand as PROJUST’s purpose? (open ended, but we can code) 3. Do you believe PROJUST has helped advance NCJR implementation? Y/N 4. Which PROJUST activities do you believe are most important? (Select up to 3, but if more than 1, rank) a. Short term training for judicial operators b. PTD program c. MASC program d. Help with modules, protocols, SOPs e. Introduction of monitoring and evaluation tools f. Support for drafting of new laws g. Support to CSOs, private sector? h. Citizen education and outreach i. National meetings and workshops j. Study tours k. Other? (please specify) 5. If your state received support from the prior USAID/MSI project (2010-2014)—and all did—how did it compare with post-2014 support from both MSI and TT DPK? a. About the same 76 This is important as in some states and/or institutions there is considerable turnover meaning that many current position holders may not know much of the past history of their organizations. 103 b. Before 2014 was more important c. Post 2014 was more important d. Each was important but worked in different areas—please specify (open-ended, will code) 6. Taking into account areas where PROJUST has provided support successfully, is your state/organization taking steps to continue these programs on its own? Y/N a. Can you specify what it is doing? (open-ended) b. Do you think this kind of continued action (with own funding, resources) is needed? Y/N 7. Are there PROJUST activities you believe were not worthwhile? Y/N 8. If yes, please specific which ones? a. Short term training for judicial operators b. PTD program c. MASC program d. Help with modules, protocols, SOPs e. Introduction of monitoring and evaluation tools f. Support for drafting of new laws g. Support to CSOs, private sector? h. Citizen education and outreach i. National meetings and workshops j. Study tours k. Other? (please specify) 9. Was the problem quantity, quality, or just irrelevance? (Choose one) 10. Are there areas where you think PROJUST should enter to increase its impact? Y/N a. If yes, please specify which ones. 11. Do you think SETEC helped advance the NCJR in your state? Y/N a. Was its disbanding a setback for NCJR implementation? Y/N b. Do you think central support is important? Y/N c. Why? (open-ended, will code answers) 12. Your state scored high on: (Choose area by state—pretrial detention, percentage of plea bargains, percentage of disputes resolved with ADR, criminal case dispositions by court) a. Do you think this is accurate compared to other states? Y/N b. Why? i. Support from judicial operators ii. Support from attorneys iii. Support from public iv. Executive interest and budgets made available v. Exceptional support from SETEC and/or PROJUST 13. Your state scored relatively low on (choose area by state--pretrial detention, percentage of plea bargains, percentage of disputes resolved with ADR, criminal case dispositions by court). a. Do you think this is accurate compared to other states? Y/N b. Why? 104 i. Resistance from judicial operators ii. Opposition from attorneys iii. Opposition public iv. Lack of executive interest v. Insufficient support from SETEC and/or PROJUST 14. For judges/fiscales, members of PTD, WJC, MASC: Have you used any of the following in your work [list protocols, models, SOPs developed by PROJUST (or SETEC)] Y/N 15. For judges/fiscales, members of PTD, WJC, MASC: Has PROJUST activity improved your work? a. How? (open ended; ask to list) b. In what areas? (open-ended; ask to list and we will code answers) 16. For CSOs, local government officials, PS actors: Has PROJUST activity strengthened your organization’s ability to produce results in advancing the NCJR? a. Please provide examples of the most important PROJUST contributions. 17. For CSOs: Has PROJUST activity made you feel more empowered? Y/N a. How? Please provide 3 examples of what activities have supported this. b. Please provide an example of what you are doing differently as a result of PROJUST support. QUESTIONNAIRE FOR INTERVIEWS AND FORMAT FOR FOCUS GROUPS Explanation: This questionnaire will be used for all interviews, with as indicated, slightly different details depending on the identify of person (judge, fiscal, member of PTD, WJC, or MASC unit, CSO, or PSO). The questions can be worked into the interviews and coded sobre la marcha or asked at the end. In any event we need answers as appropriate from all state beneficiaries. Name of Interviewee or Focus Group Member: Position: Gender: Time in Position and/or Institution: 77 1. Are you familiar with the PROJUST program? Y/N If the answer “no,” try a few other names (MSI, Tt DPK, USAID) and if still no reaction, go to 10 or modify questions accordingly. 2. What do you understand as PROJUST’s purpose? (open ended, but we can code) 3. Do you believe PROJUST has helped advance NCJR implementation? Y/N 4. Which PROJUST activities do you believe are most important? (Select up to 3, but if more than 1, rank) a. Short term training for judicial operators b. PTD program c. MASC program 77 This is important as in some states and/or institutions there is considerable turnover meaning that many current position holders may not know much of the past history of their organizations. 105 d. Help with modules, protocols, SOPs e. Introduction of monitoring and evaluation tools f. Support for drafting of new laws g. Support to CSOs, private sector? h. Citizen education and outreach i. National meetings and workshops j. Study tours k. Other? (please specify) 5. If your state received support from the prior USAID/MSI project (2010-2014)—and all did—how would you compare the importance of that support with post 2014 activities by both MSI and Tt DPK ? a. About the same b. Before 2014 was more important c. Post 2014 was more important d. Each was important but worked in different areas—please specify (open-ended, will code) 6. Taking into account areas where PROJUST has provided support successfully, is your state/organization taking steps to continue these programs on its own? Y/N a. Can you specify what it is doing? (open-ended) b. Do you think this kind of continued action (with own funding, resources) is needed? Y/N 7. Are there PROJUST activities you believe were not worthwhile? Y/N 8. If yes, please specific which ones a. Short term training for judicial operators b. PTD program c. MASC program d. Help with modules, protocols, SOPs e. Introduction of monitoring and evaluation tools f. Support for drafting of new laws g. Support to CSOs, private sector? h. Citizen education and outreach i. National meetings and workshops j. Study tours k. Other? (please specify 9. Was the problem: a. Quantity, b. Quality, or c. Just irrelevance? 10. Are there areas where you think PROJUST should enter to increase its impact? Y/N a. If yes, please specify which ones. 11. Do you think SETEC helped advance the NCJR in your state? Y/N a. Was its disbanding a setback for NCJR implementation? Y/N b. Do you think central support is important? Y/N c. Why? (open-ended, will code answers) 106 12. Your state scored high on: (choose area by state--pretrial detention, percentage of plea bargains, percentage of disputes resolved with ADR, criminal case dispositions by court). Do you think this is accurate compared to other states? Y?N Why? a. Support from judicial operators b. Support from attorneys c. Support from public d. Executive interest and budgets made available e. PROJUST support 13. Your state scored relatively low on (choose area by state—pretrial detention, percentage of plea bargains, percentage of disputes resolved with ADR, criminal case dispositions by court) Do you think this is accurate compared to other states? Y?N Why? a. Resistance from judicial operators b. Opposition from attorneys c. Opposition public d. Lack of executive interest e. Insufficient support from SETEC and PROJUST 14. For judges/fiscales, members of PTD, WJC, MASC: Have you used any of the following in your work [list protocols, models, SOPs developed by Projust (or SETEC)]? Y/N 15. For judges/fiscales, members of PTD, WJC, MASC: Has PROJUST activity improved your work? a. How? (open ended; ask to list) b. In what areas? (open-ended; ask to list and we will code answers) 16. For CSOs, local government officials, PS actors: Has PROJUST activity strengthened your organization’s ability to produce results in advancing the NCJR? a. Please provide examples of the most important PROJUST contributions 17. For CSOs: Has PROJUST activity made you feel more empowered? Y/N a. How? Please provide 3 examples of what activities have supported this. b. Please provide an example of what you are doing differently as a result of PROJUST support. 107 INCEPTION REPORT ANNEX 4: DOCUMENTS CONSULTED The following is a list of documents consulted for preparation of the Inception Report. It does not represent all those that will be used in preparation of the evaluation. 167 (or more) project documents provided to the team by USAID CIDAC. 2016. Hallazgos: Seguimiento y Evaluación de la Operación del Sistema de Justicia Penal en México. CDMX. INEGI (Instituto Nacional de Estadística y Geografía). 2016. Censo Nacional de Impartición de Justicia Estatal 2016. CDMX: INEGI Magaloni, Ana Laura and Carlos Elizondo. 2011. “La justicia de cabeza: la irracionalidad del gasto público en tribunales.” CDMX: CIDE, Serie El Uso y Abuso de los Recursos Públicos Cuaderno de debate núm. 10 https://hablandoderecho.files.wordpress.com/2014/10/gasto-publico-en￾tribunales.pdf México Evalúa. 2016. Justicia a la Medida: Siete indicadores sobre calidad de la Justicia Penal. CDMX: México Evalúa. Universidad de las Américas, Puebla. 2016. Índice Global de Impunidad México. Puebla: Fundación Universidad de las Américas Puebla78. Zepeda, Guillermo. 2017. “Impunidad Cero: Índice estatal de desempeño de las procuradurías y fiscalias.” 78 Consulted only for statistics presented. The “Index” is based on some seriously erroneous premises. 108 ANNEX 3: QUESTIONNAIRE FOR THE SURVEY 109 Fecha:_________________ Estado:__________________________________ Nombre del entrevistado/miembro del grupo de enfoque: _________________________________________________________________ Puesto actual: _________________________________________________________________ Género: __________ Tiempo trabajando en puesto actual: ______________ Tiempo trabajando en esta institución: _________________________________ 1. ¿Está usted familiarizado con el programa de PROJUST? En caso de que no reconozca el nombre, mencionar otros nombres como PROJUSTICIA, MSI, Tetra Tech DPK, y USAID Sí No En caso de que la respuesta siga siendo “No” modifique las preguntas correspondientemente. 2. ¿Ha recibido apoyo directo del programa PROJUST? ¿Ha recibido capacitación? ¿Qué tipo de capacitación? Capacitación de corto plazo (hasta 40 horas) Capacitación de largo plazo (más de 40 horas) Capacitación in situ No ha recibido capacitación ¿Ha recibido algún otro tipo de apoyo? Taller o conferencia Conversatorio Acompañamiento Resultados rápidos (100 días) Tours de estudios Otro (por favor especifique) _____________________________________ No ha recibido apoyo del programa PROJUST 110 3. ¿Usted cree que PROJUST ha ayudado en el avance de la implementación de la reforma nacional del sistema judicial y penal? Sí No 4. ¿Qué actividades de PROJUST cree usted que son las más importantes? Seleccione hasta 3 opciones, si selecciona más de una opción, ponga “1”, “2” y “3” para señalar el nivel de importancia (1 es más importante). El listado de actividades está en la siguiente página. Entrenamiento de corto plazo para personas que operan en el sistema judicial Programa de detención preventiva y Unidad de Medidas Cautelares Programa de Mecanismos Alternativos de Solución de Controversias Ayuda con modelos, protocolos y procedimientos estándar de operación Introducción de herramientas de monitoreo y evaluación Apoyo para escribir nuevas leyes Apoyo para las Organizaciones de la Sociedad Civil y el Sector Privado Educación para los ciudadanos y alcance comunitario Reuniones y talleres nacionales Resultados rápidos (100 días) Tours de estudios Otro (por favor especifique) _______________________________________ 5. ¿Usted cómo compararía en importancia el apoyo del proyecto de USAID/MSI en el periodo 2010-2014 con el apoyo actual (post 2014) de las actividades de MSI y Tetra Tech DPK? No Aplica (no estoy familiarizado con periodo 2010-2014) Mismo nivel de importancia Apoyo 2010-2014 fue más importante Apoyo post 2014 es más importante Ambas fueron importantes, pero cada proyecto trabajó en diferentes áreas (por favor especifique) ___________________________________________________ __________________________________________________________________ 111 6. Tomando en cuenta las áreas en que PROJUST ha brindado apoyo de manera exitosa, ¿Está su estado/organización tomando los pasos necesarios para continuar con estos programas por si sola? 6.a. ¿Puede especificar que están haciendo? No está haciendo nada. Hace lo siguiente: _______________________________________________ __________________________________________________________________ 6.b. ¿Usted cree que es necesario tener acciones que continúan con el trabajo de PROJUST con financiamiento y recursos propios? Sí No 7. ¿Usted cree que existen actividades de PROJUST que no valgan la pena? Sí No Pase a la pregunta 10 en caso de que la respuesta sea “No” 8. Por favor especifique que actividades son: Entrenamiento de corto plazo para personas que operan en el sistema judicial Programa de detención preventiva y Unidad de Medidas Cautelares Programa de Mecanismos Alternativos de Solución de Controversias Ayuda con modelos, protocolos y procedimientos estándar de operación Introducción de herramientas de monitoreo y evaluación Apoyo para escribir nuevas leyes Apoyo para las Organizaciones de la Sociedad Civil y el Sector Privado Educación para los ciudadanos y alcance comunitario Reuniones y talleres nacionales Resultados rápidos (100 días) Tours de estudios Otro (por favor especifique) _______________________________________ 112 9. ¿El problema con estas actividades fue la cantidad, la calidad, o simplemente el hecho que fueron irrelevantes? Escoja una opción. Cantidad Calidad Simplemente fueron irrelevantes Otro (por favor especifique) _______________________________________ 10. ¿Existen áreas en las que PROJUST deba entrar para incrementar su impacto? Áreas en las que PROJUST no entró o entró con poca profundidad. Sí. ¿Cuáles áreas? ______________________________________________ _________________________________________________________________ _________________________________________________________________ No 11. ¿Usted cree que SETEC, a nivel federal, ayudó en el avance de la implementación de la reforma nacional del sistema judicial y penal? Sí No 11.a. ¿La disolución de SETEC, a nivel federal, fue un paso atrás para la implementación de la reforma nacional del sistema judicial y penal? Sí No 11.b. ¿Usted cree que el apoyo del gobierno federal es importante? Sí ¿Por qué? Recursos económicos / financieros / presupuestales Otra razón (por favor especifique) _____________________________ ___________________________________________________________ No 113 Pregunta para jueces o fiscales, miembros de UMECAs, centros de justicia para mujeres, mecanismos alternativos de solución de controversias, u otros operadores judiciales. 12. ¿Ha usted utilizado en su trabajo listas de protocolos, modelos, procedimientos estándar de operación desarrollados por PROJUST (o SETEC)? Sí No Pregunta para jueces o fiscales, miembros de UMECAs, centros de justicia para mujeres, mecanismos alternativos de solución de controversias, u otros operadores judiciales. 13. ¿Las actividades de PROJUST han mejorado su trabajo? Sí. ¿Cómo? (por favor enumere) __________________________________ _________________________________________________________________ _________________________________________________________________ ¿En qué áreas? (por favor enumere) ___________________________________ _________________________________________________________________ No Pregunta para organizaciones de la sociedad civil, servidores públicos locales, y miembros del sector privado. 14. ¿Las actividades de PROJUST han fortalecido la habilidad de su organización de producir resultados que contribuyan al avance de la reforma nacional del sistema judicial y penal? Sí. Ejemplos de las contribuciones más importantes de PROJUST (por favor enumere) _______________________________________________ _________________________________________________________________ No Pregunta para organizaciones de la sociedad civil. 15. ¿Las actividades de PROJUST lo han hecho sentir más empoderado? Sí. ¿Cómo? Por favor dé tres ejemplos de actividades que han apoyado en esto _________________________________________________________________ _________________________________________________________________ 114 _________________________________________________________________ Por favor dé un ejemplo de algo que esté haciendo diferente como resultado del apoyo de PROJUST_______________________________________________________ ____________________________________________________________ No 115 ANNEX 4: LIST OF INTERVIEWEES 116 CDMX NAME ORGANIZATION POSITION Armando Islas Ex-SETEC Analist Carlos Ríos Independiente Académico César Chávez Independiente Consultor Gabriela Capó INSYDE Directora Jorge Alberto Jaramillo Tovar Secretariado Director de Asuntos Internacionales José Ramón Cossío Negrete Suprema Corte de Justicia de la Nación Ministro Katia Chávez CONAVIM Directora General Adjunta Layda Negrete World Justice Project Investigadora Lucía Chávez CMDPH Directora del Área de Investigación Marco Lara Klahr Independiente Periodista María de Los Ángeles Fromow SETEC Secretaria Técnica María Novoa México Evalúa Coordinadora del Programa de Justicia Ricardo Corona IMCO Director Jurídico Zobeida García Facultad de Ciencias Forenses, CU Académica Adriana Abreu-Combs MSI Directora de M&E Alejandro Ponce de León MSI Director General Alonso González-Villalobos MSI Director General Adjunto Anaid Segura MSI Coordinadora de Género Anel Pineda MSI Coordinadora de Unidades de Medidas Cautelares Cristian Ortega MSI Coordinador de Capacitación Diana Pardo MSI Especialista en Monitoreo y Evaluación Diego Labougle MSI Especialista en Monitoreo y Evaluación Ivonne Rodríguez MSI Coordinadora de Fortalecimiento de Procuradurías/Fiscalías José Javier Cárdenas MSI Coordinador de Proyectos Normativos y Armonización Legislativa Mariano Pérez MSI Director de Análisis y 117 Monitoreo del NSJP Mariano Pérez MSI Director de Control de Calidad Sandra Vázquez MSI Coordinadora de Fortalecimiento de Tribunales Susana Camacho MSI Directora de Institucional del NSJP Fortalecimiento Susana Camacho MSI Directora de Institucional del NSJP Fortalecimiento Alejandro De Gyves TtDPK Especialista de M&E Ernestina Vega TtDPK Gerente de Donaciones Gabriela Saavedra TtDPK Gerente de Acceso a la Justicia Jania Argüelles TtDPK Gerente de Comunicación Josefina Coutiño TtDPK Directora de Comunicación y Acceso a la Justicia de PROJUSTICIA SAN LUIS POTOSÍ NAME ORGANIZATION POSITION Aarón Eduardo Castro Sánchez Fiscalía Subprocurador Abraham Tobías Zarzoza Unidad de Medidas Cautelares Supervisor Ana Clarisa Huerta Torres Unidad de Medidas Cautelares Evaluadora de Riesgos Procesales Ángel Francisco Requena Aguirre Unidad de Atención Temprana (Fiscalía) Jefe Armando Oviedo Abrego Consejo Estatal de Seguridad Pública de SLP Secretario Ejecutivo Carmen Patricia Ramírez Robles Unidad de Medidas Cautelares Subdirectora encaragda de la Unidad de Medidas Previas al Juicio Edgar Valdez Zavala Defensoría Pública Defensor Elizabeth Rapp Saint Martin Otra Oportunidad, A.C. Directora General Eric Briones Méndez Poder Judicial Gestor Regional Federico Garza Herrera Fiscalía Fiscal General Hugo Alberto Cervantes Villegas Unidad de Medidas Cautelares Evaluador de Riesgos Procesales Irene Guadalupe Ayala Cerda Fiscalía Agente del Ministerio Público (Subprocuraduría de Delitos Sexuales Contra la Familia y Grupos Vulnerables) 118 Jonathan Villalba Lezama Creativería Social Director Julieta Méndez Salas Centr o Mujer es de Justici a par a las Coordinadora general Laura López de Ávila Centr o Mujer es de Justici a par a las Encargada de la Administrativa Coordinacion Luis Lauro Barrueta Niño Encargado Centro de Solución de Controversias (Fiscalía) Director Magdalen a Vega Beatri z Gonzále z Subprocuradora (Fiscalía) Subprocuraduría Especializada para la Atención de Delitos Sexuales Martha Rebeca Fernández Calleja MSI Coordinadora Estatal Projusticia Mayra Isabel Martínez Martínez Unidad de Medidas Cautelares Supervisora Mirna Yadira Castillo Ramírez Centr o Mujer es de Justici a para las Asesora Jurídica Victoria Noyola Romero Defensoría Pública Defensora Yair Jazim Govea Valladares Secret arí a de Salud Responsable Estatal del Programa de Prevención y Atención de la Violencia Familiar y Sexual (Psicólogo) ZACATECAS NAME ORGANIZATION POSITION Carlos Villegas Márquez MSI Coordinador Estatal Projusticia Abraham Ortiz Lara Centro de Prevención y Atención a la Violencia Familiar (Cepavif) Psicólogo facilitador de intervenciones preventivas Adrián Rodríguez Rodríguez Juzgado de Control y Tribunal de Enjuiciamiento Oral Juez de Control Alejandra Samanieg o Sarahy Leyva Unidad de Medidas Cautelares Evaluadora de Riesgo Procesal Alejandro Enríquez Suárez del Real COPARMEX Presidente (Zacatecas) Argelia Alejandra Rodríguez Ayala 119 Christian Altamiran o Manuel Hiriart Módulos de Atención Temprana y Centros de Justicia Alternativa Coordinador General Elisa Magally Sánchez Fano Centr o Mujer es de Justici a par a las Agente del Ministerio Público Everardo Ramírez Aguayo Comisión Ejecutiva de Atención Integral a Víctimas Comisionado Fátima Xochitl Encina Arroyo Centr o Mujer es de Justici a par a las Directora Francisco José Murillo Ruiseco Procuraduría Procurador General Héctor Manuel Martínez de la Cruz Unidad de Medidas Cautelares Director Irma Serrano Esparza Observatorio Ciudadano de Agencias de Género A.C. Integrante Julio Hikichi Enríquez Defensoría Pública Defensor Público Leticia Pacheco Macías Defensoría Pública Directora General María del Socorro Jara Barrios Procuraduría Agente del Ministerio Público (Unidad especializada de Robo con Imputado Conocido) María Magdalena Robles Varela Juzgado de Control y Tribunal de Enjuiciamiento Oral Administradora Mario Muñetones Galindo Poder Judicial Jefe de Planeación Miguel Ángel Rodríguez Estrada Defensoría Pública Titular de la unidad de apoyo técnico y gestión Raúl Carrillo del Muro Poder Judicial Director de la Escuela Judicial Reynaldo Rodríguez Botello Comisión Ejecutiva de Atención Integral a Víctimas Asesor Jurídico Roberto Carlos Carrera Pacheco Comisión Ejecutiva de Atención Integral a Víctimas Asesor Jurídico Rodolfo Moreno Trujillo Juzgado de Control y Tribunal de Enjuiciamiento Oral Juez de Control Salvador Ortiz García Secretariado Técnico Comisión para de la l a Secretario Técnico Pro Tempore 120 Implementación del sistema de justicia penal n u e v o Serguei Gonzál ez Alejandr o Roman o Procuraduría Subprocurador Silvia Lozano COPARMEX Asesora Yolanda Zamora Ramírez Defensoría Pública Defensora Pública TABASCO NAME ORGANIZATION POSITION Jorge Arzubide Dagdug MSI Coordinador Estatal Alfonso Romandía Mesa de Seguridad y Justicia A.C. Coordinador General Aureola Rodríguez Cupil Investigación de Comunes (Fiscalía) Delito s Vice Fiscal Aurora Zurita Gómez Dirección de Inteligencia (Policía de Investigación) Coordinadora de la Unidad de Análisis de Imputado Desconocido César Guercas Policía de Investigación Director de Inteligencia David Sastre Cortazar Unidad de Servicios Auxiliares de Medidas Cautelares Encargado Diozy Gricelda Silvan Cabrera Poder Judicial Administradora Judicial Edmundo Roney Balcázar Reyes Fiscalía Fiscal del Ministerio Público encargado de la fiscalía de homicidios dolosos Fernando García Zamora Policía de Investigación Director General Fernando Valenzuela Pernas Fiscalía Fiscal General Gabriel Oropeza Varela Consejo Coordinador Empresarial de Tabasco Presidente Guadalupe Castro Vidal Fiscalía Delegada de Evaluación y Monitoreo Jaime Sánchez Montaño Centro de Procuración de Justicia Director Javier Darío Chablé Gómez Policía Centr o de Investigació n en Director Javier Vidal Cornelio Fiscalía Fiscal del Ministerio Público (Unidad de Robo a negocio) 121 Jorge Alberto Aguirre Carbajal Secretaría de Seguridad Pública Secretario Julia Arrivillaga Hernández Observatorio Ciudadano Tabasco Directora de Análisis y Estadística Laura López Pérez Unidad de Servicios Auxiliares de Medidas Cautelares Titular de la región 2, 7 y 8 Lorena Díaz del Castillo Pérez Cadena Comercial Oxxo S.A. de C.V. Apoderada legal Manuel Martín Villegas Ortiz Servicios Periciales Director General María Isabel Ulin Velázquez Policía de Investigación Coordinadora Mariana Gabriela Quevedo Tribunal Superior de Justicia Juez de Control Mario Alberto Dueñas Zentella Fiscalía Director General Evaluación De Desarrollo y Mary Fé Díaz Diario ELSOLDE TABASCO Periodista Miguel Herre ra Ánge l De la Fuent e CANACO Villahermosa Presidente Miguel Narváez ONG Observatorio Ciudadano Tabasco A.C. Presidente Mileyli María Wilson Arias Sistema Estatal de Seguridad Pública Secretaria Ejecutiva Norma Elena Martínez Pérez Combate a la Violencia de Género de la Fiscalía Directora Samuel Ramos Torres Tribunal Superior de Justicia (Poder Judicial) Magistrado Tenoch Bastar Ocampo Mecanismos Alternativos de Solución de Controversias Director Yeni Elena González Valenzuela Defensoría de la Región 9 Defensora de la Región 9 BC NAME ORGANIZATION POSITION Martín Bravo MSI Coordinador Estatal Projusticia Alejandra Basaldúa Ayala Procuraduría General de Justicia Coordinadora de Atención a Víctimas y Testigos (Zona Tijuana) Alejandro Peinado Procuraduría General de Justicia Coordinador de Detenido Investigació n co n 122 Alejandro Santos Díaz Defensoría Pública de Mexicali Director Estatal Altagracia Lozano Hernández Procuraduría General de Justicia Coordinadora de la Unidad de Investigación de Delitos contra las Vida y la Integridad Ariel Moreno Soto SSPM Sub Director Técnico Cecilia Navarrete Defensoría Pública Denis Flores Rivera Procuraduría General de Justicia Agente del Ministerio Público Fernando Acevedo Dirección Estatal de Defensoría Coordinador de Defensores Gabriela María Gómez Vidal Unidad de Medidas Cautelares Coordinadora de Evaluación, Supervisión de Medidas Geibel Bolaños Policía General Judicial del Estado Hernán Álvarez Poder Judicial Administración Judicial Hugo Díaz Policía Comercial Sub Director Hymdhar Hezel Romero Defensoría Pública Defensor Público Penal J. Guadalupe Vásquez Corona Defensoría Pública Coordinador de Defensoría de Homicidios Jesús Alejandro Santos Díaz Defensoría Pública Director Estatal Defensoría Pública B.C. Jesús Ferrrat Poder Judicial Administración Judicial Joel Chaidez Defensoría Pública Defensor Público Penal Jorge Alberto Álvarez Mendoza Procuraduría General de Justicia Subprocurador de zona Tijuana José Fernando Acevedo Chavira Defensoría Pública Coordinador NSJP José Martinezno Consejo Ciudadano Juan Manuel Hernández Niebla Consejo Ciudadano Seguridad Pública Estatal de Presidente Leonor Garza Poder Judicial Coordinadora De Jueces de Control Luis Villarreal Administración Judicial Adm inistrador Manuel Guzmán Policía General Judicial del Estado Agente Ministerial Marco Amezc ua Antoni o Sotomay or Seguridad Pública (Tijuana) Municipal Secretario de Seguridad Pública Municipal Maribel Morales Sandoval Defensoría Pública Defensor Público Estatal Mario Alberto Rodríguez Sistema Estatal de Justicia Agente del Ministerio Público 123 García Alternativa Penal (SEJAP) Mario Escobedo CANACO Presidente Miguel Ángel Alvarado Unidad de Evaluación y Monitoreo de la Defensoría Responsable de la Unidad de Monitoreo, Control y Gestión Nelson Iván Garza Policía General Judicial del Estado Supervisor de Unidades de Investigación Paulina Tinoco SEJAP Perla del Socorro Ibarra Ley Procuraduría General de Justicia Procuradora Rodrigo Arias Policía General Judicial del Estado Rubén Ernesto Armenta Zanabia Subsecretaría de Enlace para Asuntos de Justicia, Secretaría General de Gobierno Subsecretario de Enlace con Poder Judicial Susana Alvarado Robles Consejo Ciudadano de Seguridad Pública Secretaria Técnica CHIHUAHUA NAME ORGANIZATION POSITION Alejandra de las Casas Muñoz MSI Coordinadora Estatal Alejandro Legarda Juzgado de Control del Distrito Judicial Morelos, TSJ Juez de Control Alejandro Veliz Titular Unidad de Atención Temprana (Fiscalía del Estado) Coordinador de la Unidad de Atención al Público Ana Luisa Regalado Simental Instituto de Servicios Previos al Juicio Encargada de Vinculación y Enlace Institucional Benjamín Ojeda COPARMEX Consejero Carlos Arturo Luján Olivas Fideicomiso para la Competitividad y Seguridad Ciudadana (FICOSEC) Director Carlos Mario Jiménez Investigación de Persecución del Delito de la Fiscalía Zona Centro Fiscal de Distrito (Zona Centro) Erick Alejandro Rangel Ríos Fiscalía Coordinador de la Unidad de Delitos Contra la Vida 124 Fernando Escamilla Área de Justicia del Observatorio Ciudadano de Seguridad y Justicia de FICOSEC (Fideicomiso para la Competitividad y Seguridad Ciudadana) Investigador Ignacio Israel Ramírez Centeno Consejo de la Judicatura del Poder Judicial Secretario Técnico de la Comisión de Administración Ignacio Manjarrez Ayub COPARMEX Presidente de la Comisión Nacional de Seguridad y Justicia Irma Villanueva Nájera Comisión Ejecutiva de Atención a Víctimas del Estado Coordinadora Isaac Lujan Carreón Defensoría Pública Penal Director Jaime Armendáriz Diario de Chihuahua Periodista José Carlos González Reyes Poder Judicial Asesor Jurídico de Gestión Judicial Juan Carlos Armendáriz Delgado Poder Judicial Encargado de la Unidad Operativa del Sistema Penal Juan Rodríguez Zubiate Centro Estatal para la Consolidación del Sistema de Justicia Penal Secretario Ejecutivo Krishna Martos Fiscalía Agente del Ministerio Público (Coordinadora de Gestión del Ministerio Público) Luisa Estefanía Chávez Hernández Juzgado de Control Jefa de Causa y Gestión (Distrito Judicial Morelos) María del Rosario Ramos Oquita Instituto de Justicia Alternativa del Tribunal Superior de Justicia del Estado Directora Noel Orlando Jiménez Holguín Fiscalía Coordinador Unidad Jurídica Oksana Volchanskaya Instituto Estatal Chihuahua Electoral de Directora de Comunicación Social Patricia Mendías Hernández Defensoría Pública Coordinadora de Defensores de Control Ricardo Márquez Torres Juzgado de Control Juez de Control Rolando Nájera Radionet (Ciudad Juárez) Periodista Rosario Ramos Oquita Dir. Instituto Alternativa De Justicia 125 Sandra Julissa Valenzuela Sígala Fiscalía Agente del Ministerio Público (Unidad Especializada Robos) Sandra Zulema Palma Sáenz Juzgado de Control Juez de Control Susana Bustamante Medina Tania Belkotosky Estrada Instituto de Servicios Previos al Juicio Secretaria Adscrita a Presidencia, encargada implementación Medidas Cautelares Wendy Paola Chávez Villanueva Fiscalía Fiscal Especializada en atención a mujeres víctimas del delito por razones de género GUANAJUATO NAME ORGANIZATION POSITION Ángeles Camacho Poder Judicial Jefa de Causa y Gestión municipio Guanajuato) (juzgad o Cecilia Ángel Morales Defensoría Pública Coordinadora de Juicio Oral (Región 2) Eduardo Padilla Martínez Defensoría Pública Coordinador Estatal Federico Ramírez Cervantes Defensoría Pública Director de la Defensoría Pública Penal Fernando Vázquez Centro Estatal de Justicia Facilitador Galo Carrillo Villalpando Defensoría Pública Director General Pública de Oficio de la Defensor ía Imelda Carbajal Cervantes Poder Judicial Coordinadora de jueces del sistema de oralidad penal María Claudia Barrera Rangel Supremo Tribunal de Justicia y Consejo del Poder Judicial del Estado Presidenta María Cristina Cabrera Juzgado de Oralidad Penal Juez de Control Mauricio Ontiveros Hornelas Poder Judicial Director General de Gestión de Juzgadosde Control y Oralidad Pablo López Zuloaga Consejería Jurídica del Estado Consejero Plinio Emanuel Martínez Tafolla Comisión Estatal para la Implementación de la Reforma en Materia de Justicia Penal y Seguridad Pública Secretario Técnico Tobías García Tovar Escuela de Estudios Investigación Judicial e Director Zaid de Anda Márquez Defensoría Pública Coordinador de Juicio Oral (Región 4) 126 MONTERREY NAME ORGANIZATION POSITION Agustín Bailón Popoca UMECA Titular Aixa Amalia Alvarado Guarany Comisión Ejecutiva Estatal de Atención a Víctimas Titular Alejandra Garza Hernández Defensoría Pública Jefa de la Unidad de Análisis y Gestión Interinstitucional Alejandro Carlín Balboa Poder Judicial Juez de Garantías de Adolescentes Ana Ludyvina Velarde Viccali A.C. Coordinadora Arlene Caritina Ramos Zamora Comisión Ejecutiva Estatal de Atención a Víctimas Coordinadora de Reinserción Social Bernardo Aguayo Obregón Comisión Ejecutiva Estatal de Atención a Víctimas Coordinador del Registro Estatal de Víctimas Daniel Rodríguez Andonie Secretariado Ejecutivo del Consejo de Coordinación del Sistema Integral de Seguridad Pública Director General de Planeación Estratégica David Zendejas González Unidad de Medidas Cautelares Encargado UMECA Adolescentes Diana Hernánd ez Yoland a Ugalde Comisión Ejecutiva Estatal de Atención a Víctimas Asesor Jurídico Dulce María Obregón Tapia Comisión Ejecutiva Estatal de Atención a Víctimas Jefa de Evaluación Interdisciplinaria Edgar Alejandro Márquez Alfaro Comisión Ejecutiva para la Reforma del Sistema de Justicia Penal Director General Francisc o Duarte Javier Hernánd ez Defensoría Pública Director de Defensoría Especializada para Adolescentes Guadalupe Murillo Alternativas Pacíficas A.C. Abogada del Departamento Jurídico Gumaro García González Fideicomiso del Estado de Nuevo León para la Implementación del Sistema de Justicia Penal (SIJUPE) Encargado del área de capacitación Hugo Iván Garza García Fideicomiso del Estado de Nuevo León para la Implementación del Sistema de Justicia Penal Coordinador de Asesores (Violencia familiar) 127 (SIJUPE) Javier Antonio Martínez Sepúlved a Fiscalía Director General de la Fiscalía Especializada para Adolescentes José Agustín Morales Alemán Comisión Ejecutiva Atención a Víctimas Estat al de Coordinador de Políticas Públicas (Investigación y Capacitación) Juan Pablo Raigosa Treviño Consejo de la Judicatura Consejero Karla Yetzabel Armendáriz Lozano Comisión Ejecutiva Atención a Víctimas Estat al de Asesor Jurídico Laura Yajahira Cortez Reyes Comisión Ejecutiva Atención a Víctimas Estat al de Analista Jurídico de la Titular Laura Yolanda Villarreal Zarazúa Alternativas Pacíficas A.C. Trabajadora Social del área de Capacitación y Prevención Marco Polo Olazarán Alvarado Fiscalía Encargado de la Zona Centro de la Fiscalía Especializada en Atención a la Mujer María Angélica Morales Dávila Comisión Ejecutiva Atención a Víctimas Estat al de Directora de Vinculación María de la Luz Rodríguez Balderas Fiscalía Titular de la Fiscalía Especializada en Atención a la Mujer María de Lourdes Ayala Chapa Defensoría Pública Subdirectora General María del Rosario Jara Juárez Comisión Ejecutiva Atención a Víctimas Estat al de Coordinadora de la Asesoría Jurídica Martín Carlos Sánchez Bocanegra Renace A.B.P. Director General Mayra Lizbeth Alvarado Alonso Renace A.B.P. Director Jurídico Nora Selene Balderas Rodríguez Fiscalía Directora Regional Norte de la Fiscalía Especializada en 128 Atención a la Mujer Omar García Morán Unidad de Medidas Cautelares Responsable del Área de Evaluación Raúl Alberto Garza Garza Comisión Ejecutiva Atención a Víctimas Estat al de Coordinador de Evaluación Atención, Gestión y Sandrine Molinard Consejo Cívico de Instituciones de Nuevo León Directora General Thalía Gisel García Román Unidad de Medidas Cautelares Supervisora SALTILLO NAME ORGANIZATION POSITION Héctor Alvear Alvarado MSI Coordinador Estatal Projusticia Alejandra Wade Villanueva Consejo Cívico de las Instituciones de Coahuila Director Alfredo Jiménez Defensoría de Coahuila Director Ana Yadira Flores García Fiscalía Coordinadora de tramitación masiva Anahí Barrón, Unidad de Gestión del Juzgado Penal de Saltillo Tribunal, UAME Directora Ayari Vázquez Rodríguez Juzgado Penal Jefa de Unidad de Causa (Saltillo) Beatriz Adriana Vázquez Juárez Fiscalía Coordinador de Ministerios Públicos Carlos Balderas Defensa Pública Defensor de Oficio Carmen Ibarra Defensoría en Materia Penal Subdirectora Diana Marisol Flores Rivera Mecanismos Alternativos de Solución de Controversias Directora General Edgar Vicente Cárdenas Carrillo Fiscalía Director General Investigación de Unidade s de Everardo Javier Lazo Chapa Fiscalía Delegado Francisco Rodríguez Chapa Unidad de Medidas Cautelares (UMECA) Coordinador Estatal Gerardo García Márquez Fiscalía General de Estado Fiscal General Gina Denisse Valdés Alonso Fiscalía Coordinador de Ministerios Públicos UTMC Griselda Elizalde Castellanos Poder Judicial Juez del sistema acusatorio y oral 129 Javier Rangel Ramírez Poder Judicial Administrador General de los Juzgados Penales Orales José Alfredo Jiménez Guerrero Instituto Estatal de Defensoría Pública Director General José Luis Chapa Comisionado de Seguridad Karla Esmeralda de Rodríguez Ávila Instituto Estatal de Defensoría Pública Defensor Público Penal Katy Salinas Pérez Centros de Justicia para las Mujeres Directora General Luis Alejandro Castillo Santana Instituto Estatal de Defensoría Pública Defensor Público Penal María Antonieta Leal Cota Poder Judicial Juez del sistema acusatorio y oral María del Carmen Hernández Ibarra Instituto Estatal de Defensoría Pública Subdirector de la Unidad de Defensa Penal María Graciela Nava Ramos Poder Judicial Juez del sistema acusatorio y oral Maribel Rodríguez Ramírez Centro de Profesionalización, Acreditación, Certificación y Carrera Directora Marisol Flores Centro de Medios Alternos de la Fiscalía Directora Miriam Cárdenas Cantú Tribunal Superior de Justicia del Estado de Coahuila Presidenta Nadia Mabel García Salcedo Juzgado Penal Jefa de Unidad de Sala Nayeli Anahí López Barrón Juzgado Penal Jefe de Unidad de control y gestión (Saltillo) Odilla Fuentes Unidad de Calidad Óscar Nájera Davis Poder Judicial Magistrado de la Sala Colegiada Penal Paloma Vázquez Chávez Comisión para la Instrumentación del Nuevo Sistema de Justicia Penal (CINSJP) Coordinador de Análisis Jurídico Perla Mata Fiscalía General Raúl Azpeitia Silva Fiscalía Coordinador de Unidad de Atención Temprana (delitos con detenido) Rosalinda Contreras Álvarez Fiscalía Coordinador 130 Vanessa Ramírez Unidad de Calidad Yosdel Isis Gómez Solís Unidad de Medidas Cautelares Evaluadora y Supervisora 131 ANNEX 5: LIST OF PROJECT DOCUMENTS CONSULTED A. Documents Provided byUSAID USAID Folder Document Contract 1. 6089 Mexico PROJUST - MSI DPK CTA Amend 05 MODIFICATION NO. 05 TO CONTRACTOR TEAMING ARRANGEMENT (CTA) 2. Budget 2013-2018 final version.xls USAID Mexico ROL Project 2013-2018. Independent Government Cost Estimate (IGCE) 3. PROJUST Activity contract award - MSI final.pdf AWARD/CONRACT AID-532-C-14-00003 Issued by: USAID/El Salvador Name of Contractor: Management Systems International 4. PROJUST CTA between MSI-Tt DPK 15 Jan 2015 fully signed.pdf CONTRACTOR TEAMING ARRANGEMENT (CTA) Contract Modification 5. Modification final document.pdf AMENDMENT OF SOLICITATION/MODIFICATION CONTRACT Issued by: USAID/Mexico Name of Contractor: Management Systems International 6. MSI Organization Chart - Staffing modification.pdf USAID-PROJUSTICIA-Esquema 7. PROJUST Mod Guidance Paper_06.13.16.docx GUIDANCE PAPER FOR THE PROJUST PROJECT MODIFICATION Core Technical Guidance 8. Coahuila Success Story-30Nov2016_FINAL-v2.docx SUCCESS STORY. Coahuila’s Courts brings closer justice to citizens 9. DPK Findings, 4.30.17.docx ACTION PLAN 10. Factsheet_WJCHidalgoApril2017.docx.pdf Women’s Justice Center in the State of Hidalgo 11. Modelo 3 pisos.docx Three-level Case Prioritization Model 12. PROJUST Diploma Course Gender Based on Violence Oct 2016.pdf SUCCESS STORY. Promoting a gender sensitive perspective on access to justice and its administration 13. PROJUST Michoacan (January 2016).pdf Paving the Way for Mexico’s New Criminal Justice System in Michoacan State 14. PROJUST Nuevo Leon (May 2016).pdf SUCCESS STORY. Innovative Public Defenders in Nuevo Leon Show Results for the Public Core Technical Guidance 15. PROJUST PD Oaxaca (Apr 2016).pdf SUCCESS STORY. The Foundation for a Strong Public Defense in Oaxaca State 16. PROJUST PD Zacatecas (Apr 2016).pdf SUCCESS STORY. Public Defenders Honing Their Skills in Mexico’s Zacatecas State 17. PROJUST PTS Queretaro (Apr 2016).pdf SUCCESS STORY. Upholding the Presumption of Innocence in Mexico’s State of Queretaro 18. PROJUST Snapshot Diploma Course on Gender Violence FY2015.pdf SUCCESS STORY. Mexican Boy Helps in the Fight Against Gender-Based Violence 132 19. PROJUST Success Story Juvenile Justice (September 2015).pdf USAID Supports Civil Society-led Juvenile Justice Reform in Mexico 20. PROJUST Success Story PTSU Hidalgo (September 2015).pdf USAID: Contributing to Mexico´s Criminal Justice System Paradigm Shift 21. PROJUST Success Story Women Justice Centers FY 2015.pdf Supporting Efforts to Reduce Gender Based Violence in Mexico 22. PROJUST Tabasco (June 2016).pdf SUCCESS STORY. Specialized Unit in Tabasco Improves Investigation Techniques 23. PROJUST Victims Topo Chico Apr 2016.pdf SUCCESS STORY. Leveraging technology to improve assistance to victims in Mexico 24. Puebla_briefing_May_2015_FINAL_v2.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT PUEBLA BRIEFING, MAY 2015 25. Quintana_Roo_Briefing_November_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUINTANA ROO BRIEFING, NOVEMBER 2015 26. ROL PROJUST two page with Tom’s comments (3) (4).docx Comunicado de prensa. Mujeres en Frecuencia AC y PROJUSTICIA presentan serie radiofónica para difundir el Sistema de Justicia Penal Acusatorio entre mujeres 27. RoL_Glossy DRAFT_15May2017.docx RULE OF LAW Mexico 28. SLP Success Story_SLP Case Screening-FINAL.docx SUCCESS STORY. Citizens Come First in New Case Screening Unit 29. State_Brief_Coahuila_Updated September_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT COAHUILA BRIEFING, SEPTEMBER 2015 30. State_Brief_Guanajuato_May_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT GUANAJUATO BRIEFING, MAY 2015 Core Technical Guidance 31. State_Brief_Morelos_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT MORELOS BRIEFING, DECEMBER 2015 32. State_Brief_San_Luis_Potosi_May_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT SAN LUIS POTOSI BRIEFING, MAY 2015 33. State_Brief_Tlaxcala_May_2015_FINAL_v2.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT TLAXCALA BRIEFING, MAY 2015 34. State_Brief_Zacatecas_June_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT ZACATECAS BRIEFING, JUNE 2015 35. State_Briefing_Baja_California_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT BAJA CALIFORNIA BRIEFING, JANUARY 2016 36. State_Briefing_Baja_California_Sur_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT BAJA CALIFORNIA SUR BRIEFING, JANUARY 2016 37. State_Briefing_Campeche_November_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT CAMPECHE BRIEFING, NOVEMBER 2015 38. State_Briefing_Chiapas_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT CHIAPAS BRIEFING, DECEMBER 2015 39. State_Briefing_Chihuahua_September_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT CHIHUAHUA BRIEFING, SEPTEMBER 133 2015 40. State_Briefing_Colima_Sept_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT COLIMA BRIEFING, SEPTEMBER 2015 41. State_Briefing_Durango_Updated_ March_2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT DURANGO BRIEFING, MARCH 2016 42. State_Briefing_Federal District_November_ 2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT FEDERAL DISTRICT BRIEFING, NOVEMBER 2015 43. State_Briefing_Guerrero_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT GUERRERO BRIEFING, JANUARY 2016 44. State_Briefing_Hidalgo_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT HIDALGO BRIEFING, DECEMBER 2015 Core Technical Guidance 45. State_Briefing_Jalisco_June_2015_FINAL.pdf USAID/MEX PROMOTING JUSTICE PROJECT JALISCO BRIEFING, JUNE 2015 46. State_Briefing_Michoacan_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT MICHOACAN BRIEFING, JANUARY 2016 47. State_Briefing_Nayarit_Updated_January_2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT NAYARIT BRIEFING, JANUARY 2016 48. State_Briefing_Nuevo_Leon_August_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT NUEVO LEON BRIEFING, AUGUST 2015 49. State_Briefing_Oaxaca_December_2015_FINAL.pdf USAID/MEXICO PROOTING JUSTICE PROJECT OAXACA BRIEFING, DECEMBER 2015 50. State_Briefing_Queretaro_Updated December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUERETARO BRIEFING, DECEMBER 2015 51. State_Briefing_Sinaloa_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT SINALOA BRIEFING, DECEMBER 2015 52. State_Briefing_Sonora_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT SONORA BRIEFING, DECEMBER 2015 53. State_Briefing_State_of_Mexico_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT STATE OF MEXICO BRIEFING, JANUARY 2016 54. State_Briefing_Tabasco_December_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT TABASCO BRIEFING, DECEMBER 2015 55. State_Briefing_Tamaulipas_June_2015_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT TAMAULIPAS BRIEFING, JUNE 2015 56. State_Briefing_Veracruz_January 2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT VERACRUZ BRIEFING, JANUARY 2016 57. State_Briefing_Yucatan_January_2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT YUCATAN BRIEFING – JANUARY 2016 58. Success Story Puebla (July 2016) FINAL.pdf SUCCESS STORY. Delivery Effective Justice in State of Puebla Tec hnic al Gui 59. Success Story_Coahuila_Case Screening Unit_FINAL.pdf SUCCESS STORY. Coahuila’s justice system better serving citizens 134 60. Success Story_Morelos July2017 PTS.pdf SUCCESS STORY. Morelos court administration overhaul benefits citizens 61. Success Story_Zacatecas May2017 PTS.pdf SUCCESS STORY. Fair Justice Provides Citizen a Second Chance 62. Tabasco Success Story-December-FINAL-v3 - marked, DC edits.docx SUCCESS STORY. Justice to benefit from evidence-based decision-making 63. Three level Model on ADRM.docx Three-level Case Prioritization Model 64. USAID contribution to CJR in Mexico (May 2016) Fact Data.pdf Contribución de USAID para la Reforma de Justicia Penal en México 65. USAID contribution to CJR in Mexico (May 2016) Narrative.PDF REFORMA DE JUSTICIA PENAL UN RELATO DE LA HISTORIA Y LA CONTRIBUCIÓN DE USAID A LOS 12 AÑOS DE REFORMA EN MÉXICO 66. Zacatecas Success Story-Conversatorios-FINAL.docx Collective Problem Solving Proves Effective for Justice Sector Grant Programs 67. Mexico PROJUST Grants Manual_v3_2-12-2015.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT GRANTS MANUAL. February 2015 68. Presentación FINAL DONATARIOS.PPTX LAS ORGANIZACIONES DE LA SOCIEDAD CIVIL Y SU CONTRIBUCIÓN A LA REFORMA PENAL M&E 69. Diagnóstico y selección de estados para la recolección_USAID.PPTX Diagnóstico y selección de estados para la recolección directa de datos sobre desempeño Monitoring & Evaluation 70. M&E Plan Indicators Report-7Jan2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT MONITORING AND EVALUATION PLAN INDICATORS REPORT JANUARY – SEPTEMBER 2015 71. M&E Plan Indicators Progress Report_28Oct2016_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT MONITORING AND EVALUATION PLAN INDICATORS PROGRESS REPORT – SEPTEMBER 2016 72. Monitoring and Evaluation Plan (final version).pdf MONITORING AND EVALUATION PLAN USAID/MEXICO O PROMOTING JUSTICE PROJECT 73. Muestro-diagnostico_v2.pdf Diagnóstico y selección de estados para la recolección directa de datos sobre desempeño 6 de abril de 2016 74. PROJUST DPK PMP Report FY 2015 (Year 1).pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 ANNUAL PERFORMANCE MONITORING REPORT (JANUARY – SEPTEMBER 2015) Monitoring & Evaluation 75. PROJUST ME Report FY 2016 Tasks 3 and 5 Final.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 ANNUAL MONITORING & EVALUATION REPORT (OCTOBER 2015 – SEPTEMBER 2016) 76. PROJUST MSI M&E Plan.FINAL.02-29-16.pdf MONITORING AND EVALUATION PLAN USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) 77. PROJUST MSI Report USAID PMP Indicators FY 2015.pdf USAID/MEXICO AND FOREIGN ASSISTANCE INDICATORS REPORT 78. PROJUST Tasks 3 and 5 - Year 1 PMP (Final).03.24.15.pdf 135 PERFORMANCE MONITORING PLAN MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 AND 5 79. Reporte Hallazgos cinco estados 030816.docx Monitoreo del desempeño del nuevo sistema de justicia penal en cinco estados Monitoring & Evaluation 80. State Ranking Table 05.24.pdf USAID/MEXICO STATE RANKING MATRIX 81. USAID State Ranking Criteria.pdf U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT (USAID) SENSITIVE BUT UNCLASSIFIED￾USAID/MEXICO STATE RANKING MEXICO CRIMINAL JUSTICE SYSTEM REFORM MSI & DPK Reports 82. Factsheet_30Oct2017_FINAL.docx 83. M&E Plan Indicators Progress Report_30Oct2017_FINAL.docx MONITORING AND EVALUATION PLAN INDICATORS PROGRESS REPORT – SEPTEMBER 2017 84. PROJUST M&E Annual Report FY3 2016-2017 Tasks 3_4.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT TASKS 3 & 4 ANNUAL M&E REPORT October 2016 – September 2017 MSI & DPK Reports 85. PROJUST Annual Report FY3 2016-2017 Tasks 3_4.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT TASKS 3 & 4 ANNUAL REPORT October 2016 – September 2017 86. PROJUST Annual Report-30Oct2017_FINAL.pdf PROMOTING JUSTICE PROJECT ANNUAL REPORT October 1, 2016 – September 30, 2017 Other Sources / Reference 87. 1. USAID-BASE_Entrevista de Evaluacion de Riesgos Procesales (EERv.4).pdf ENTREVISTA DE EVALUACIÓN DE RIESGOS PROCESALES v.4 88. 1.Manual de Procedimientos MP-CSPJ-01.docx A.MANUAL DE PROCEDIMIENTOS MP-CSPJ-01 89. 2. USAID-BASE-Tabulador de Riesgos Procesales (TRP v.4).pdf TABULADOR DE RIESGOS PROCESALES (v.5) 90. 3. USAID-BASE_Opinion Tecnica de Evaluacion de Riesgos Procesales (OTERP v.3).pdf ASUNTO: OPINIÓN TÉCNICA sobre la Evaluación de Riesgos Procesales Municipio, Estado, a 5 de Diciembre de 2015 91. 142 Ebook Manual Modelo Intervención Comunicadores.pdf MODELO DE INTERVENCIÓN PARA COMUNICADORES DEL NUEVO SISTEMA DE JUSTICIA PENAL 92. 144 Ebook ProtejamosVictimas.pdf ¡PROTEJAMOS A LAS VÍCTIMAS! MANUAL DE NUEVO PERIODISMO JUDICIAL, CON ENFOQUE GARANTISTA MARCO LARA KLAHR 93. AJ E2 Modelo de evaluacion.pdf USAID/MEXICO PROMOVIENDO LA JUSTICIA SISTEMA DE EVALUACIÓN Y SEGUIMIENTO DEL ASESOR JURÍDICO DE VÍCTIMAS EN EL PROCESO PENAL ACUSATORIO ACTUALIZADO CONFORME A LA REFORMA QUE SE REALIZÓ A LA LEY GENERAL DE VÍCTIMAS 136 TAREAS 3 Y 5 94. AJ E3 Plan de Accion.pdf USAID/MEXICO PROJUSTICIA PROMOVIENDO LA JUSTICIA USAID/MEXICO PROJUSTICIA PROMOVIENDO LA JUSTICIA PLAN DE ACCIÓN Y CRONOGRAMA DE CAPACITACIÓN DEL MANUAL DE INTERVENCIÓN DEL ASESOR JURÍDICO EN MODALIDAD PRESENCIAL, ATENDIENDO A LOS LINEAMIENTOS PROPUESTOS POR PROJUSTICIA INSTRUCTOR: SIPP ACT. 21 TAREAS 3 Y 5 95. Formato para captura de datos de expedientes OEMASC 2017.xlsx Pauta de análisis de expedientes en los OEMASC 96. Formato para captura de datos de llamadas usuarios OEMASC 2017.xlsx Encuesta de satisfacción de usuarios 97. Guía_de_Instructor.docx MANUAL DE INSTRUCTOR CAPACITACIÓN INICIAL PARA DIRECTIVOS Y ADMINISTRATIVOS DE LA OSPJ 98. Guia_del_Participante.docx GUÍA DEL PARTICIPANTE CAPACITACIÓN INICIAL PARA DIRECTIVOS Y ADMINISTRATIVOS DE LA OSPJ Other Sources / Reference 99. HALLAZGOS_2015.pdf EVALUACIÓN DE LA IMPLEMENTACIÓN Y OPERACÓN A OCHO AÑOS DE LA REFORMA CONSTITUCIONAL EN MATERIA DE JUSTICIA PENAL CIDAC & USAID 100. HERRAMIENTA TABLA DE MEDICION DE DISCIPLINA DEL MODELO MASC.xlsx TABLERO DE GESTIÓN 101. Manual de procedimientos MASC 2017.docx MANUAL DE PROCEDIMIENTOS PARA EL USO MECANISMOS ALTERNATIVOS EN LAS SOLUCIONES ALTERNAS Y EL REGISTRO Y SEGUIMIENTO DE ACUERDOS REPARATORIOS 102. Manual de Selección por competencias.docx MANUAL DE SELECCIÓN POR COMPETENCIAS. USAID / PROJUST. JUNIO 2016 103. Manual Para la Intervención de Asesores Jurídicos.pdf USAID/MEXICO PROMOVIENDO LA JUSTICIA MANUAL DE INTERVENCIÓN DEL ASESOR JURÍDICO Other Sources / Reference 104. MGO CJM actualización Intervención Vicitmológica.docx USAID/MEXICO PROMOVIENDO LA JUSTICIA PRINCIPIOS PARA LA ACTUALIZACIÓN Y HOMOLOGACIÓN DEL PROCESO INTEGRAL DE ATENCIÓN DE LOS CENTROS DE JUSTICIA PARA LAS MUJERES RESPECTO A LAS DISPOSICIONES PREVISTAS EN LA LEY GENERAL DE VÍCTIMAS. 105. MGO CJM Actualización SJPA.docx USAID/MÉXICO PROMOVIENDO LA JUSTICIA AVANCE DEL MODELO DE GESTIÓN OPERATIVA DE LOS CENTROS DE JUSTICIA PARA LAS MUJERES BAJO EL SISTEMA DE JUSTICIA PENAL ACUSATORIO 106. propuesta de proceso operativo.docx PROCESO OPERATIVO LA OFICINA DE SERVICIOS PREVIOS AL JUICIO. 107. Protocolo de Evaluación base.docx PROTOCOLOS DE ACTUACIÓN EN EL PROCESO DE EVALUACIÓN DE RIESGOS PARA LA OFICINA DE SERVICIOS PREVIOS AL JUICIO 137 108. Protocolo para Monitoreo y Evaluación de los OEMASC 2017.pdf USAID/MEXICO PROMOVIENDO LA JUSTICIA PROTOCOLO PARA EL ANÁLISIS COMPARATIVO DE RESULTADOS DE LOS ORGANOS ESPECIALIZADOS EN MECANISMOS ALTERNATIVOS DE SOLUCIÓN DE CONTROVERSIAS EN MATERIA PENAL TAREA 4 Other Sources / Reference 109. SETEC-METODOLOGIA_OCTUBRE-2015-VF.pptx Resultados de la Nueva Metodología para Clasificación y Estratificación de Entidades Federativas Octubre 2015 Dirección General de Coordinación Interinstitucional Información estratégica para la toma de decisiones basadas en evidencia 110. USAID CEJA Analysis.pdf ARE ORAL TRIALS IN MEXICO WORKING? MONITORING REPORT OF THE IMPLEMENTATION PROCESS OF THE CRIMINAL JUSTICE REFORM IN MEXICO 111.What Explains Criminal Violence in CDMX- 2016.pdf What Explains Criminal Violence in Mexico City? A Test of Two Theories of Crime Carlos Vilalta and Robert Muggah Manual para el Diplomado de Víctimas de Violencia de Género 112.Esquema Curricular.docx DIPLOMADO “LAS VÍCTIMAS DE VIOLENCIA DE GÉNERO EN EL SISTEMA DE JUSTICIA PENAL ACUSATORIO EN MÉXICO” CENTRO DE JUSTICIA PARA MUJERES Manuales de Instructor Diplomado Texto dirigido para el Programa de Capacitación Integral 113. Módulo-Burnout.docx 114. Módulo-Consideraciones intervención psico-jurídica.docx 115. Módulo-Género.docx 116. Módulo-Liderazgo.docx 117. Módulo-Manual-Principios y Directrices para la Asistencia e Intervención.docx 118. Módulo-Multiculturalidad.docx 119. Módulo-Políticas públicas.docx 120. Módulo-Sistema de justicia penal.docx Other Sources / Reference Manual para el Diplomado de Víctimas de Violencia de Género Manual de Instruct or 121. Módulo-Resiliencia.docx Texto dirigido para el Programa de Capacitación Integral Víctimas de Violencia de Género, Trauma, Recuperación y Resiliencia: de Víctima a sobreviviente Materiales Logísitcos 122. CARTA DE BIENVENIDA.docx 123. MÓDULO-BURN OUT.docx Examen Previo del Diplomado: “Las Víctimas de Violencia de Género en el NSJP” Módulo 9: Intervención en el Trauma Vicario y Síndrome de Burn Out) 124. MÓDULO-GÉNERO.docx Examen Posterior del Diplomado: Las Víctimas de Violencia de Género en el NSJP Módulo 1: Género y Perspectiva de Género: el Debate 125. MÓDULO-INTERVENCIÓN PSICOJURÍDICA.docx Examen Posterior del Diplomado: Las Víctimas de Violencia de Género en el NSJP Módulo 7. Consideraciones Generales para la Intervención Psico- 138 jurídica de la Víctima de Violencia de Género. Other Sources / Reference 149. Manual para el Diplomado de Víctimas de Violencia de Género Materiales Logísitcos 126. MÓDULO-INTERVENCIÓN VICTIMOLÓGICA.docx Examen Posterior del Diplomado: Las Víctimas de Violencia de Género en el Nuevo Sistema de Justicia Penal en México. Módulo 9. Principios y Directrices para la Asistencia e Intervención en Víctimas de la Violencia de Género en el Centro de Justicia para Mujeres. 127. MÓDULO-LIDERAZGO.docx Examen previo del Diplomado: Las Víctimas de Violencia de Género en el Nuevo Sistema de Justicia Penal en México Módulo 3: “Liderazgo Organizativo y Desarrollo de Habilidades Gerenciales.” 128. MÓDULO-LOS DELITOS DE VG EN EL SJP.docx Examen Posterior del Diplomado: Las Víctimas de Violencia de Género en el NSJP Módulo 1: Los Delitos de Violencia de Género en el Nuevo Sistema de Justicia Penal. 129. MÓDULO-MULTICULTURALIDAD.docx Examen Posterior del Diplomado: Las Víctimas de Violencia de Género en el NSJP Módulo: El enfoue multi e intercultural en la atención de la violencia de género 130. MÓDULO-NIÑOS Y NIÑAS.docx Examen Previo del Diplomado: Las Víctimas de Violencia de Género en el Nuevo Sistema de Justicia Penal en México Módulo 9: Principios y Directrices para la Intervención en Niños y Niñas Víctimas de la Violencia de Género: Infancia en Riesgo Other Sources / Reference 149. Manual para el Diplomado de Víctimas de Violencia de Género Materiales Logísitcos 131. MÓDULO-POLÍTICA PÚBLICA.docx Examen Previo del Diplomado: Las Víctimas de Violencia de Género en el NSJP Módulo 8: Planeación Estratégica: Políticas Públicas, Género y Justicia. 132. MÓDULO-RESILIENCIA.docx Examen Posterior del “Curso Especializado para la Formación de Mujeres Sobrevivientes de Violencia de Género” 133. MÓDULO-TRATA DE PERSONAS.doc Examen Posterior del Diplomado: “Las Víctimas de Violencia de Género en el Nuevo Sistema de Justicia Penal en México” Módulo 11: “Aproximaciones conceptuales, aspectos básicos de la Trata de Personas. Marco jurídico nacional e internacional.” 134. MÓDULO-VICTIMOLOGÍA.docx Examen Posterior del Diplomado: “Las Víctimas de Violencia de Género en el NSJP” Módulo 3: Victimología y Derechos de la Mujer Víctima de Violencia en el Nuevo Sistema de Justicia Penal. 135. REGLAMENTO.docx REGLAS BÁSICAS DIPLOMADO “LAS VÍCTIMAS DE VIOLENCIA DE GÉNERO EN EL SISTEMA DE JUSTICIA PENAL EN MÉXICO” Reporti ng 136. PROJUST Quarterly Report - Y1 Q2 Jan. 1 - Mar. 31, 2015.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT: FEBRUARY - MARCH 2015 139 137. PROJUST Quarterly Report - Y1 Q3 Apr. 1 - Jun. 30, 2015.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT APRIL 1, 2015 – JUNE 30, 2015 138. PROJUST Quarterly Report - Y2 Q1 Oct. 1, 2015 - Dec. 31, 2015.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT OCTOBER 1, 2015 – DECEMBER 31, 2015 139. PROJUST Quarterly Report - Y3 Q1 Oct. 1, 2016 - Dec. 31, 2016.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT OCTOBER 1, 2016 – DECEMBER 31, 2016 140. PROJUST Quarterly Report Y2 Q1 2015 Tasks 3 and 5.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 QUARTERLY REPORT: OCTOBER – DECEMBER 2015 141. PROJUST Quarterly Report Y2 Q2 2016 Tasks 3 and 5.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 QUARTERLY REPORT: JANUARY – MARCH 2016 142. PROJUST Quarterly Report Y3 Q1 2016 Tasks 3 and 5.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 QUARTERLY REPORT: OCTOBER – DECEMBER 2016 143. PROJUST Quarterly Report Y3 Q2 2017 Tasks 3 and 5.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 QUARTERLY REPORT: JANUARY 1 – MARCH 31, 2017 144. PROJUST Quarterly Report Y3 Q3 2017 Tasks 3 and 4.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 4 QUARTERLY REPORT: APRIL 1 – JUNE 30, 2017 145. PROJUST Y2 Q2 Jan. 1, 2016 - Mar. 31, 2016.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT JANUARY 1, 2016 – MARCH 31, 2016 Reporting 146. PROJUST_Annual Report_FY 2015 - 2016 - MSI -.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT ANNUAL REPORT OCTOBER 1, 2015– SEPTEMBER 30, 2016 147. PROJUST_Monthly_December2014_submitted.docx USAID/MEXICO PROMOTION OF JUSTICE PROGRAM 148. PROJUST_Q2_2017_Task 1_2_27April2017_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT QUARTERLY REPORT JANUARY 1, 2017 – MARCH 31, 2017 149. PROJUST_Q3_2017_Task 1_2_27April2017.PDF PROMOTING JUSTICE PROJECT QUARTERLY REPORT April 1, 2017 – June 30, 2017 150. INFORME DE CARGAS DE TRABAJO PARA PROYECTAR LA ESTRUCTURA ORGÁNICA 140 EN EL ESTADO DE CHIHUAHUA. PROYECTO PROMOVIENDO LA JUSTICIA (PROJUST) USAID, 06 AL 10 DE JULIO DE 2015 151. LÍNEA BASE Y DIAGNÓSTICO INICIAL PARA LA IMPLEMENTACIÓN DE LA METODOLOGÍA DE LOS SERVICIOS PREVIOS AL JUICIO PROYECTO PROMOVIENDO LA JUSTICIA (PROJUST) ESTADO DE COAHUILA 152. CHIHUAHUA-SPJ Propuesta Técnica de Estructura Organica Inicial PROYECTO PROMOVIENDO LA JUSTICIA (PROJUST) JULIO 2015 153. CHIHUAHUA-SPJ Propuesta Técnica de Estructura Organica Inicial PROYECTO PROMOVIENDO LA JUSTICIA (PROJUST) JULIO 2015 154. ANÁLISIS DE LA INFORMACIÓN DE LÍNEA BASE Y DIAGNÓSTICO INICIAL PARA DETERMINAR LAS CARGAS DE TRABAJO DE LA OFICINA DE SERVICIOS PREVIOS AL JUICIO, PROPUESTA DE ESTRUCTURA ORGÁNICA E INVERSIÓN INICIAL 155. Challenges in the implementation and consolidation of M&E Units Annual Report 2017 156. Challenges in the implementation and consolidation of M&E Units Q1 Report (October 1 – December 31) 2018 157. Challenges in the implementation and consolidation of M&E Units Q2 Report (January 1 – March 31) 2018 158. DIAGNÓSTICO DEL ÁREA DE CAPACITACIÓN DEFENSORÍA PÚBLICA DEL ESTADO DE BAJA CALIFORNIA MANAGEMENT SYSTEMS INTERNATIONAL SEPTIEMBRE 2017 159. DIAGNÓSTICO DEL ÁREA DE CAPACITACIÓN DEFENSORÍA PÚBLICA DEL ESTADO DE SAN LUIS POTOSÍ MANAGEMENT SYSTEMS INTERNATIONAL SEPTIEMBRE 2017 160. DIAGNÓSTICO DEL ÁREA DE CAPACITACIÓN DEFENSORÍA PÚBLICA DEL ESTADO DE ZACATECAS MANAGEMENT SYSTEMS INTERNATIONAL AGOSTO 2017 161. DIAGNÓSTICO DEL ÁREA DE CAPACITACIÓN ESCUELA JUDICIAL DEL TRIBUNAL SUPERIOR DE JUSTICIA DEL ESTADO DE ZACATECAS MANAGEMENT SYSTEMS INTERNATIONAL SEPTIEMBRE 2017 162. ENTREGABLE I REPORTE DE VIABILIDAD, IMPLEMENTACIÓN Y SOSTENIBILIDAD ESTADO DE CHIHUAHUA TRIBUNAL SUPERIOR DE JUSTICIA DEL ESTADO DE CHIHUAHUA MANAGEMENT SYSTEMS INTERNATIONAL 163. DIAGNÓSTICO Y ANALISIS DE BRECHAS PARA LA IMPLEMENTACIÓN DEL MODELO DE LA UAME NUEVO LEÓN 141 MANAGEMENT SYSTEMS INTERNATIONAL 164. DIAGNÓSTICO Y ANALISIS DE BRECHAS PARA LA IMPLEMENTACIÓN DEL MODELO DE LA UAME ZACATECAS MANAGEMENT SYSTEMS INTERNATIONAL 165. RESUMEN DIAGNOSTICOS UMECAS ESTADOS OBJETIVO Y NIVEL NACIONAL MH 166. Work with Training Institutes in Current Work Plan, 2017-2018 SETEC Documents 167. Estandares fiscales ACTUALIZADO 2017.docx ESTÁNDARES PARA FISCALES 168. Estandares operativos para defensores.docx ESTANDARES PARA DEFENSORES 169. Estandares operativos para jueces.docx ESTANDARES DE ACTUACIÓN PARA JUECES 170. Instrumento de recolección de acuerdos y seguimiento.xlsx Planilla de seguimiento de acuerdos para conversatorios Medición de eficacia del Conversatorio 171. Ley Modelo 1. Organica del Poder Judicial.docx MODELO DE LEY ORGÁNICA DEL PODER JUDICIAL DE LAS ENTIDADES 172. Ley Modelo 2. Organica de la Fiscalia Modelo 1.docx MODELO DE LEY ORGÁNICA DE LA FISCALÍA GENERAL DE JUSTICIA DE LAS ENTIDADES 173. Ley Modelo 3. Extinción de Dominio.docx LEY MODELO DE EXTINCIÓN DE DOMINIO PARA LAS ENTIDADES 174. Ley Modelo 4. Protección de Personas que Intervienen en el Proceso Penal.docx LEY MODELO PARA LA PROTECCIÓN DE PERSONAS QUE INTERVIENEN EN EL PROCEDIMIENTO PENAL 175.Ley Modelo de Atención a Víctimas SETEC 170914.docx Ley Modelo de Atención a Víctimas 176.Ley Modelo de la Defensoría Pública SETEC.docx LEY MODELO DE LA DEFENSORÍA PÚBLICA 177. Ley Moelo 5. Ley de Administración de Bienes asegurados.docx Ley Modelo para la Administración de Bienes Asegurados, Decomisados o Abandonados SETEC Documents 178. Metodologia para la realización de conversatorios.docx METODOLOGÍA PARA EL DESARROLLO DE LOS CONVERSATORIOS 179. MODELO DE REG ORG DP V 120715.docx REGLAMENTO ORGÁNICO MODELO DE LA DEFENSORÍA PÚBLICA 180. MODELO DE REG ORG DP V 120715.docx REGLAMENTO ORGÁNICO MODELO DE LA DEFENSORÍA PÚBLICA 181. MODELO DE REG ORG FIS V191015.docx 182. MODELO DE REGLAMENTO DE LA LEY ORGÁNICA DE LA PROCURADURÍAFISCALÍA-SECRETARÍA DE JUSTICIA DE LAS ENTIDADES TITULO PRIMERO ESTRUCTURA Y ATRIBUCIONES DE LA FISCALÍA GENERAL 183. MODELO DE REG ORG PJ V191015.docx 142 MODELO DE REGLAMENTO DE LA DE LEY ORGÁNICA DEL PODER JUDICIAL DE LAS ENTIDADES 184. MODELO DE REG SPC DP V191015.docx MODELO DE REGLAMENTO DEL SERVICIO PROFESIONAL DE CARRERA DEL INSTITUTO DE LA DEFENSORIA DEL ESTADO 185. MODELO DE REG SPC MP V 191015.docx MODELO DE REGLAMENTO PARA EL SERVICIO DE CARRERA MINISTERIAL 186. MODELO DE REG SPC PJ V191015.docx MODELO DE REGLAMENTO DEL SERVICIO PROFESIONAL DE CARRERA JUDICIAL Solicitation 187. PROJUST REAL SOLICITATION RFP_No_SOL-523-12-000003 (1).pdf Request for Proposal (RFP) Nº SOL-523-12-000003 188. SOW ROL june 12 2012 w new results - benchmarks - indicators (all changes accepted).docx Promoting Justice Project - PROJUST Section C - Statement of Work C.1 BACKGROUND 1.1 Mexico’s Criminal Justice System Reform 3: Increa se public Suppo rt for the New Crimi nal Justic e Syste 189. CANALONCE JUICIOS ORALES-reporte cuantitativo.pdf REPORTE DE DESEMPEÑO CUANTITATIVO JUICIOS ORALES, JUSTICIA DIFERENTE TASK 4: Increase Access to Justice and Improve Victims’R ights 190. Manual_on_How_to_Create_a_WJC.pdf USAID/MEXICO JUSTICE AND SECURITY PROGRAM MANUAL ON "HOW TO CREATE A WOMEN'S JUSTICE CENTER" TASK 4: Increase Access to Justice and Improve Victims’Rights 191. Orientation_&_informational_booklet_for_users_of_WJC.pdf USAID/MEXICO JUSTICE AND SECURITY PROGRAM ORIENTATION AND INFORMATIONAL BOOKLET FOR USERS OF WOMEN’S JUSTICE CENTERS 192. Presentacion en modelos MASC.pptx Presentación Ejecutiva del Modelo de Gestión de las Investigaciones y del Modelo de Gestión para el Uso de Mecanismos Alternativos en las Soluciones Alternas al Proceso 193. Presentación_semaforizacion_2-USAID.pptx Semaforización de la reforma Resumen de resultados en base al análisis de la información estadística proporcionada por los censos de procuración e impartición de justicia penal del INEGI para el periodo 2010-2014 USAID Internal 194. 2017-06-26_DO2 Results Framework.pptx Government of Mexico Legitimacy Increased 195. OUTCOME INDICATOR SUMMARY FY2017.xlsx Outcome Indiacator Summary Work Plans 196. 6089 Mexico PROJUST - Baseline Study Plan - Final.docx BASELINE STUDY PLAN MEXICO PROMOTING JUSTICE PROJECT (PROJUST) 143 Work Plans 197. PROJUST approved Work Plan 2 10 15.docx USAID/MEXICO PROMOTING JUSTICE PROJECT ANNUAL WORK PLAN FOR PERIOD JANUARY – SEPTEMBER 2015 198. PROJUST DPK Work Plan Year 2 FY 2016 (approved PDF).pdf USAID/MEXICO PROMOTING JUSTICE PROJECT (PROJUST) TASKS 3 & 5 ANNUAL WORK PLAN FOR PROJECT YEAR 2 OCTOBER 2015 – SEPTEMBER 2016 199. PROJUST Tasks 3 and 5 - Year 1 Work Plan (Final).03.24.15.docx USAID/MEXICO PROMOTING JUSTICE PROJECT ANNUAL WORK PLAN FOR PERIOD JANUARY – SEPTEMBER 2015 (TASKS 3 AND 5) 200. PROJUST Tasks 3 and 5 - Year 3 Work Plan (English Version).pdf USAID/MEXICO PROMOTING JUSTICE PROJECT ANNUAL WORK PLAN FOR PROJECT YEAR 3 OCTOBER 2016 – SEPTEMBER 2017 (TASKS 3 AND 5) 201. PROJUST_Amend 1-final.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT ANNUAL WORK PLAN FOR PERIOD OCTOBER 2015 – SEPTEMBER 2016 Work Plans 202. PROJUST_MSI-DPK_States 2017-Tasks-1-5.pptx Type of activities MSI/DPK PROJUST 203. PROJUST_Work Plan_ June-Sept2017_5June_FINAL.pdf USAID/MEXICO PROMOTING JUSTICE PROJECT WORK PLAN FOR PERIOD JUNE – SEPTEMBER 2017 204. Workplan to reduce impunity.pdf Specific PROJUST work to reduce impunity under the June to September 2017 work plan 205. PROJUST EVALUATION 2017 DOCUMENT LIBRARY. Docx PROJUST EVALUATION 2017 DOCUMENT LIBRARY B. Documents Provided by PROJUSTor by State Authorities Source Document Coahuila 190. Coahuila, Poder Judicial, Boletín de Información Judicial. Año 2016, Número 2 191. Coahuila, Poder Judicial, “Plan Integral del Desarrollo y Fortalecimiento, 20 16- 20 17,” Boletín de Información Judicial. Año 2016, Número 1. Tabasco 192. Tabasco, Fiscalía General del Estado, Programa de los 10 0 Días: Resultado de 50 días de trabajo. 16 de febrero de 20 18. (PPT) 193. Tabasco, Fiscalía General del Estado, Modelo de Gestión y Administración de Casos. 2018. (PPT) Guanajuato 194. Guanajuato, Poder Judicial, “SIGO Modulo de generación de proyecciones (Simulación) (PPT) 195. Guanajuato, Poder Judicial, “Secretaria Técnica, Comisión Estatal para la Implementación de la Reforma en Materia de Justicia Penal y Seguridad 144 Pública.” 196. Guanajuato, Poder Judicial, Procuraduría General de Justicia, Defensoría Pública, Secretaría de Seguridad Pública del Estado, Plan Integral de Mejora del Sistema de Justicia Penal del Estado de Guanajuato” 197. Guanajuato, Poder Judicial, “Gestión del cambio,” febrero de 2016. (PPT) Chihuahua 198. Género YSistema De Justicia Penal: Informe Sobre Perspectiva De Género En El Actuar De Agentes Del Ministerio Público Del Estado De Chihuahua 199. Programa 100 Días Presentación Robo A Comercio Al Término (PPT) 200. Perspectiva De Género En El Nuevo Sistema De Justicia Penal Agentes Del Ministerio Público Reporte Comparativo Nacional – Chihuahua (PPT) 201. Perspectiva De Género En El Nuevo Sistema De Justicia Penal, Jueces Y Juezas Reporte Comparativo Nacional – Chihuahua (PPT) 202. Modelo De Gestión De Excelencia Institucional Análisis De ResultadosY Detección De Brechas Poder Judicial Del Estado De Chihuahua Carta Descriptiva Taller de Direcciòn y Conducciòn en la Etapa de Juicio Oral Nuevo León 4 UNIDAD DE ANÁLISIS, MONITOREO YEVALUACIÓN de Nuevo León (PPT) Modelo de Gestión de Excelencia Descripción del modelo (PPT) 1. MANUALDE COORDINACIÓN YCOMUNICACIÓN ENTRE MINISTERIO PÚBLICO Y POLICÍA 2. MÉTODO DE ANÁLISIS DE RIESGOS PROCESALES DE UNIDADES DE MEDIDAS UTELARES 3. Estándares de desempeño para la unidad de investigación y persecución del delito de homicidio 4. Estandares fiscales actualizado 20 17 145 Documentos Enviados PROJUST 5. PROPUESTADE Unidad Especializada de Investigación y persecución del delito de homicidio 6. Protocolo Cadena de Custodia 7. Protocolo de coordinación y conducción 8. Protocolo de entrevista 9. Protocolo de incidente 10. Protocolo de investigación 11. Protocolo de lvantamiento Documentos Enviados PROJUST 12. Protocolo de inspección de persona 13. Protocolo introducción a propiedad privada y orden de cateo 14. Protocolo Rector de Coordinación y Comunicación de LAUIPH 15. Protocolo respuesta inicial a escena del delito 16. Protocolo traslado ofendidos y testigos 17. Protoloco de recolección de evidencia 18. Protocolo Traslado de Detenidos 19. USAID's Technical Note: “The 5Rs Framework in the Program Cycle” 20. List of Work Plan activities implemented 21. Programas de capacitación y pautas de evaluación del programa Projusticia para defensores, jueces y fiscales 22. Análisis general de los resultados cuantitativos y cualitativos del estudio de percepción del sistema de justicia penal en México (ENSIJUP 20 12) 23. USAID's "Local Systems: AFramework for Supporting Sustained Development" 24. USAID's Technical Note: "The 5Rs Fram ework in the Program Cycle" 25. List of Work Plan activities implemented Documentos Enviados Por PROJUST 26. Programas de capacitación y pautas de evaluación del programa Projusticia para defensores, jueces y fiscales 27. Análisis general de los resultados cuantitativos y cualitativos del estudio de percepción del sistema de justicia penal en México (ENSIJUP 20 12) 146 28. Copia de tabla diagnóstico de institutos 29. CUESTIONARIO INSTITUTOS 30. Diagnostico capacidad de capacitar defensoría baja california 31. Diagnostico capacidad de capacitar defensoría slp 32. Diagnostico capacidad de capacitar defensoría zacatecas 33. Diagnostico capacidad de capacitar tribunal zacatecas 34. Diagnostico necesidades defensores zacatecas 35. JUECES DIAGNOSTICO HTSJZ- PROJUSTICIA 36. Modelo de Informe Semanal de las Actividades de Coordinadores 147 ANNEX 6: ADDITIONAL TABLES 148 The following tables come both from the questionnaire and other sources. As noted in the main text, the selection of respondents to the questionnaire was anything but random for three reasons: • First, USAID requested that specific institutions and even staff be interviewed. Between the length of the required list and the maximum of one week in each state, the die was already cast as to whom would be included. • Second, because of the number of states to be covered and time limitations, the evaluation team had to ask PROJUST staff to contact/select interviewees. This led to a selection, beyond those already specified by USAID, of people who had been in contact with the project. • Third, time and budgetary limitations also meant that only individuals in the one (or occasionally two) cities visited could be included. Ideally, the team would have worked with a list of all judges, for example, and selected randomly among them. However, to do so would have required a much longer preparation time, funds to bring in individuals from outside the principal city, and most probably fewer states. Although except for Guanajuato, all states visited had benefitted from intensive PROJUST involvement, the biases introduced by this necessary method were probably not great. However, while the team asked PROJUST to provide focus group members that included: 1) individuals involved in PROJUST activities and 2) those that were not, the results seemed drawn nearly entirely from the first group. Although PROJUST/MIS subsequently objected to a bias in selection, any bias was most likely to be in their favor. The first three tables feature gender breakdown, and Tables 2 and 3 do this for tables already included in the main text. This breakdown was not done there because, as the tables show, gender made little difference in the answers. Table 1 is different, although again not necessarily representative, in that it shows time in position by gender, institution and state. Table 1: Years spent in current position, by institution, state, and gender All Institutions Public Defense Public Ministry Judiciary Other Institutions M F M F M F M F M F Number of respondents 75 75 11 10 19 15 12 12 33 38 Average 7 states 3.23 4.08 4.73 5.46 2.30 6.19 5.44 3.51 2.46 3.07 San Luis Potosí 1.77 6.93 2.00 17.0 1.25 16.0 0.25 ND 2.23 4.19 Zacatecas 4.66 3.68 4.25 6.50 1.00 4.50 10.3 2.92 2.57 1.50 Tabasco 1.66 3.07 ND 5.00 2.75 3.32 ND 1.67 0.78 2.87 Baja California 3.28 5.86 4.40 9.00 3.17 5.00 ND ND 0.71 5.39 149 All Institutions Public Defense Public Ministry Judiciary Other Institutions M F M F M F M F M F Chihuahua 3.41 4.51 ND 0.08 3.02 8.00 3.02 5.28 4.44 2.09 Coahuila de Zaragoza 4.77 2.57 8.75 3.75 1.71 0.96 5.92 3.10 0.67 2.67 Nuevo León 3.02 3.98 2.00 1.50 0.54 15.7 5.04 ND 3.20 2.67 Source: Respondent characteristics from questionnaire. As noted above, the numbers in Table 1are not necessarily representative of each institution but only of the questionnaire respondents. The team is grateful to PROJUST staff for recruiting respondents but could not ask them to do a random selection, as this would have taken too much time. It is nonetheless interesting that for Public Defense and the Public Ministry, female respondents tended to have much longer time in their positions than men, while for the Judiciary and “other institutions” (CJMs, UMECAs, and victims’ attorneys) the results were mixed. However, lacking knowledge of the real distribution in each institution, the most we can say is that these patterns hold for individuals with whom PROJUSTworks. The next two tables, again divided by gender, show little difference in responses to the two questions, except that women were more positive about short-term training and national meetings while men favored ADR and pretrial services more than did women. Table 2: Which PROJUST activities do you consider most important (by gender and institution)? * All Institutions Public Defense Public Ministry Judiciary Other Institutions M F M F M F M F M F Number of respondents 75 75 11 10 19 15 12 12 33 38 Short-term training 47 56 6 8 13 13 8 9 20 26 Help with models, protocols, and SOPs 45 42 4 5 11 8 10 6 20 23 National meetings and workshops 21 28 4 4 4 3 3 7 10 14 Introduction to tools for M&E 20 22 5 3 2 3 9 4 4 12 Rapid Results 19 18 0 1 5 3 3 4 11 10 Pretrial services 20 13 3 1 5 3 1 1 11 8 ADR 20 12 3 2 9 2 1 3 7 5 Other 25 20 3 4 6 3 1 2 15 11 * Numbers add to over 150 because respondents could mention up to three activities Source: Questionnaire, Question 4 150 Table 3: SETEC and its role in the implementation of the NCJS, by gender and institution All Institutions Public Defense Public Ministry Judiciary Other Institutions M F M F M F M F M F Did SETEC help to advance the reform’s implementation? Yes 64 63 9 10 17 12 11 12 27 29 It helped and its disbanding was a step backwards for the reform 39 26 4 5 8 6 7 4 20 11 It helped but its disbanding was not a step backwards 23 31 5 5 9 6 4 7 5 13 Yes, SETEC helped, but no answer as to the impact of its disappearance. 2 6 0 0 0 0 0 1 2 5 No 4 4 1 0 1 1 0 0 2 3 Did not help and its disappearance was not a step backwards 4 4 1 0 1 1 0 0 2 3 Doesn’t know 2 3 0 0 1 0 0 0 1 3 No response 5 5 1 0 0 2 1 0 3 3 Source: Questions11and 11a The following table shows the percentage of court filings under the NCJS. The numbers are less than 10 0 percent even in 2016 because cases filed with courts may have originated several years earlier and so have entered the Public Ministry under the traditional system. Not mentioned in the report, as it is not addressed by PROJUST, is the fact that in most states, a proportion of cases still being processed, entered under the former system, and are seen by judges, prosecutors and defenders still operating under its rules. The Table does indicate which states are further ahead in the transition and which are still dealing with a large backlog of criminal cases begun under the prior system. Table 4: On Cases Registered in Courts Under the NCJS, 2015- 2016 State 2014 2015 2016 Aguascalientes ND ND ND Baja California 22.8% 26.1% 55.6% Baja California Sur ND 0.9% 21.7% Campeche 0.0% 2.3% 25.7% Coahuila de Zaragoza 4.2% 12.6% 83.7% Colima ND ND ND Chiapas 12.2% 12.4% 44.0% Chihuahua 99.2% 99.2% 99.6% 151 State 2014 2015 2016 Ciudad de México ND 6.6% 48.0% Durango 68.9% 90.5% 98.0% Guanajuato 37.3% 53.3% ND Guerrero 1.2% 2.9% 10.3% Hidalgo 0.0% 5.0% 29.7% Jalisco 0.0% 1.7% 8.1% Estado de México 97.4% 82.4% ND Michoacán de Ocampo ND 5.5% 82.9% Morelos ND ND ND Nayarit 0.0% 0.0% 37.2% Nuevo León 37.4% 76.0% 87.4% Oaxaca 33.2% 46.5% 61.5% Puebla 8.7% 13.9% 57.0% Querétaro 2.6% 17.0% 45.8% Quintana Roo 1.3% 6.7% 45.2% San Luis Potosí ND 2.4% ND Sinaloa 0.3% 10.4% ND Sonora 0.0% ND 15.5% Tabasco 2.2% 5.4% 57.1% Tamaulipas 0.0% 2.4% 31.4% Tlaxcala ND 3.4% 15.8% Veracruz 4.3% 7.8% 38.3% Yucatán 9.4% 40.7% 61.8% Zacatecas 38.8% 52.3% 59.0% Source:teamcalculationsbasedonINEGI’s“CensosNacionalesdeimparticióndeJusticiaEstatal”2017,20 16 and 20 15 152 ANNEX 7: CONFLICT OF INTEREST STATEMENTS 153 154 155 156 ANNEX 8: LIST OF ANSWERS PROVIDED IN SPANISH TO QUESTION 6A 157 QUESTION 6A: "WHAT IS YOUR STATE/INSTITUTION DOING TO FURTHER THE REFORM AFTER PROJUST DISAPPEARS?" Respuestas Pregunta 6a Capacitación/certificación 1. Asesorías y capacitación en temas relevantes 2. Busca la certificación en integridad institucional 3. Buscar la certificación de todos los integrantes en las áreas de PROJUST 4. Capacitación a los asesores jurídicos 5. Capacitación aunada a retos que los mismos integrantes conversan en la evolución de los talleres de capacitación. Evaluaciones de la ruta crítica del proceso para cumplir retos. 6. Capacitación constante en contención personal, habilidades y herramientas; capacitación como instructores. 7. Capacitación de personal así como adoptando todas las oportunidades para lograr una mejora al sistema de justicia penal 8. Capacitación jurídica 9. Capacitamos al personal, se les involucra en los cambios para poder soportar el cambio 10. Continúa solicitando capacitaciones al Instituto estatal de las Mujeres para la sensibilización de género, solicita a escuela de la fiscaliza capacitación para el nuevo sistema, mantiene la estadística a través de informes diarios. 11. Continuar con el Sistema de Integridad Institucional 12. Cursos, talleres y actualmente un reto de 100 días 13. El personal de nuevo ingreso recibe la experiencia del personal capacitado 14. Los formadores capacitan defensores 15. Más capacitación y recursos 16. Nos apoya en programa de diagnóstico para detectar áreas de oportunidad y capacitación. 17. Nuestra gente ya capacitada sigue de instructores a nuevos elementos. Creación de protocolos. 18. Replicar la capacitación, seguir con los lineamientos que se nos indique, acatando recomendaciones. 19. Se comunican las instituciones, sin embargo hace falta seguimiento y mayor capacitación para todas las áreas. 20. Se pretende implementar cursos de capacitación 21. Se solicitó capacitación para dar continuidad con los programas 22. Tomando en cuenta algunos perfiles, y se sugiere una capacitación para formador de formadores. Modelos/protocolos 158 Modelos/protocolos 1. Acoplando el modelo de tres pisos a la funcionalidad de la procuraduría creando un manual de operación de módulo de atención temprana para el modelo de tres pisos 2. Aplicamos lo recibido en la capacitación y elaboración de protocolos 3. Creación de la unidad de Homicidios con miras a replicar el modelo en los demás municipios 4. Mantener los protocolos y manuales impartidos a fin de continuar con la productividad. Sin embargo resulta necesaria la continuidad del apoyo. 5. Nos desempeñamos observando los manuales y flujogramas otorgados por PROJUST 6. Se continúa aplicando los protocolos que se elaboraron, trabajando conjuntamente policías, peritos, y Ministerio Público. 7. Se crean UAME en cada institución operadora del sistema. Se esta creando instructores que pueden replicar el conocimiento, mejora de manuales y procedimientos de operatividad. 8. Se están estandarizando los protocolos de acción y capacitando operadores. 9. Se han establecido protocolos de actuación 10. Se sigue el modelo de los tres pisos en la implementación y se vigila y evalúa su avance. 11. Sigue los manuales de USAID y estableciendo los 3 pisos Recomendaciones y metodología 1. Acompañamiento técnico y organizacional 2. Adaptar las recomendaciones tomando en cuenta las condiciones de la capital 3. Analizando los procesos internos para hacer mejoras 4. Aplicación de herramientas impartidas en las capacitaciones, en la atención directa a las víctimas 5. Aplicando los conocimientos adquiridos en las capacitaciones 6. Busca darle un seguimiento 7. Consolidando los programas en el Instituto de Defensoría 8. Continuar con los programas de trabajo 9. Creación de áreas de gestión, formación y de investigación defensiva. 10. Da seguimiento a acuerdos y programas 11. Dando seguimiento a todas las áreas necesarias para lograr los objetivos planteados 12. Dejar registros de las bases (metodología) y de ahí continuar 13. Eficientar las técnicas de trabajo en las intervinientas 14. Entrega de edificio para todas las regiones de UMECA, equipamiento de mobiliario y cómputo para las regiones 159 Recomendaciones y metodología 15. Están trabajando eficazmente para lograr un buen resultado en la implementación del nuevo Sistema de Justicia Penal, gracias a ello se agilizan las gestiones. 16. Fortaleciendo y fomentando las recomendaciones de PROJUST 17. Implementando los conocimientos adquiridos 18. Implementando los programas en el Instituto Estatal de Defensoría Pública, asegurando los pasos necesarios para que sea permanente y sustentable. 19. Incluir las propuestas de PROJUST en su forma de trabajo 20. Institucionalizar las acciones y dándole seguimiento 21. Internamente se ha complementado el proceso de asesoría y representación de las mujeres por parte de las asesoras jurídicas 22. Lleva a cabo cada una de las indicaciones dadas, respetando protocolos. 23. Perfeccionar y eficientar procedimientos para que den los pasos necesarios para atención pronta e inmediata. Dar resultados rápidos (una atención más inmediata). 24. Propuesta de equipamiento y desarrollo de recursos humanos 25. Reforzando cada día los conocimientos que PROJUST nos brindó 26. Se buscan que las actividades tengan uniformidad en todo el sistema de SEJAP, así como la eficiencia en su aplicación. 27. Se continúa el con el seguimiento del programa de manera que éste se haga permanentemente, el mecanismo procura se implemente en otras áreas de la dependencia. 28. Se continúa trabajando bajo el entrenamiento recibido 29. Se encuentra replicando proyectos que ya tuvieron éxito 30. Se implementan las acciones, capacitación, información obtenida. 31. Seguimiento 32. Seguimiento en la vigilancia del actuar de otros actores del sistema con metodología PROJUSTICIA 33. Seguimiento, asistencia. 34. Seguimos aplicando el sistema y mejor aun se replicó en 2 regiones más del estado 35. Sí se aplica el flujo de trabajo de acuerdo a las normativas de USAID 36. Trabajando para continuar trabajando en la misma metodología 37. Tratando de mejorar las áreas de oportunidad e implementar nuevos modelos en la investigación Resultados rápidos 1. Acuerdo de continuidad para seguir programa resultados rápidos 2. Buscamos que las estrategias aplicadas en el proyecto de 100 días se apliquen en otros delitos 160 Resultados rápidos 3. Continuando con el sistema implementado por Resultados rápidos (100 días) 4. Continuar después del Reto de los 100 días 5. En algunas actividades hemos apoyado, como en fomentar la cultura de la denuncia y participar en el programa de los 100 días. 6. En el caso de género se ha mantenido el número de judicializaciones. En el robo a comercio se ha mejorado la investigación y los trabajos de la unidad de imputado desconocido. 7. La procuraduría sigue con capacitación continua, replicando los prácticas que se aprendieron de los 100 días 8. Planear la implementación del programa en otros delitos 9. Programa 100 días 10. Resultados rápidos (100 días) 11. Se va a continuar con la misma forma de trabajar del Programa de cien días, la captura y mecanismo de cada expediente que se abra en el centro será la misma. Gestión 1. Aplicando modelos de gestión en la institución como medición de resultados a efecto de implementar e innovar y condiciones para tener un mejor resultado 2. Control, revisión, mantenimiento y seguimiento de los procesos desde el punto de vista administrativo y jurisdiccional. 3. Creando reformas para lograr estandarización de procesos 4. Desarrollando indicadores de resultados 5. Establece metas, mejora prácticas respecto a agilidad en los procesos 6. Implementación de indicadores y encuestas de evaluación mensualmente 7. Implementación de un mejor sistema de justicia o gestión judicial. Unificación de criterios. Elaboración de protocolos. 8. Implementación del nuevo modelo de gestión, seguimiento en Unidades de Homicidios. Coordinación 1. Aplicación de conversatorios 2. Buscar mecanismos para la terminación de casos 3. Conformación de una mesa de seguridad y justicia 4. Continuar con las estrategias y la comunicación creada 5. Conversatorios entre el tribunal-fiscalía-defensoría así como talleres de buenas prácticas para aplicar en el Nuevo Sistema 6. Conversatorios entre las instituciones del sistema de justicia penal 161 Coordinación 7. Coordina el Sistema Integral de Seguridad Pública, por lo que puede seguir monitoreando el seguimiento de estas unidades. 8. El contacto con los intervinientes para realizar mesas de negociación para privilegiar las salidas alternas 9. Encontrando formas para comprometer a las autoridades a avanzar para mejorar su operación y acompañándolos en el proceso 10. Es un intermediario importante en la consolidación del sistema sociedad civil - gobierno 11. Estamos trabajando en la atención y coordinación con las diferentes áreas de gobierno y sociedad civil 12. Evaluando los pasos a seguir 13. Existe una mejor organización interinstitucional para atender casos de violencia familiar 14. Mantienen comunicación con defensores y fiscales para llegar a salidas alternas 15. Mesas de negociación; continuar con la dinámica de resolver los asuntos en el menor número de audiencias. 16. No perder el ritmo de trabajo y la comunicación 17. Plantear un modelo ágil y eficiente de resolver los asuntos de manera rápida 18. Reuniones de seguimiento por ejemplo 19. Reuniones de trabajo, gestión, capacitación para lograr sostenibilidad 20. Se toman acuerdos con los titulares que operan en el sistema judicial 21. Seguir las prácticas de integración de asuntos, contacto directo con tribunal, concluir en las segundas audiencias el asunto. 22. Sigue en comunicación con las partes técnicas para replicar la información a otras áreas Necesidades 1. Considero que falta continuar con la consolidación del modelo en el resto de las regiones del estado. 2. En el área de género siguen judicializando en cantidad sin que realmente busque calidad, sólo trabaja en base a estadística. 3. Inicio contacto, sin seguimiento 4. Sí se puede continuar pero se necesita el compromiso de todas las partes y la continuidad de los operadores 5. Solicita Alternativas Pacíficas apoyo en diversas convocatorias para fondos de financiamiento 6. Sumarse al proceso, ser parte de él y apropiarse para poder replicar más adelante 7. Tomando y probando sugerencias 162 ANNEX 9: STATEMENT OF DIFFERENCES 200 12th Street South, Suite 1200 Arlington, VA 22202 USA Statement of Difference Management Systems International msiworldwide.com Page 1 of 6 MSI’s Statement of Difference to the Mid-term Evaluation of the USAID/MEXICO Promoting Justice Project Evaluation Report (Final) This statement of difference is in response to the mid-term evaluation report on the USAID/Mexico Promoting Justice Project (PROJUST) delivered by ME&A to USAID on September 25, 2018. USAID/Mexico contracted Management Systems International (MSI) to implement PROJUST from July 2014 - July 2019 with the objective of supporting Mexico’s transition to the accusatory criminal justice system, at both the federal and state levels, while protecting human rights. In January 2015, MSI entered into a 5-year teaming agreement for PROJUST with Tetra Tech DPK. Once the Mexican constitutional deadline requiring all state and federal jurisdictions to make the transition from the inquisitorial system to the accusatorial system was met in 2016, the PROJUST contract was modified (March 2017). This entailed a significant technical adjustment as well as a narrowing of the geographic scope to those states that offered the best conditions for successful partnership in the consolidation process. The contract modification had the stated objective of supporting consolidation of Mexico’s Criminal Justice Reforms. MSI acknowledges and appreciates adjustments ME&A has included in the final version of the evaluation and PROJUST has incorporated a number of the recommendations into its FY19 workplan. However, MSI experienced great difficulties in assessing the quality of the findings and conclusions in every version of the evaluation report and while this version incorporates many of the extensive comments provided by MSI to USAID, it remains far from compliant with USAID guidelines for the generation and presentation of findings and conclusions. MSI could not clearly identify findings or conclusions throughout the body of the report. Indeed, it was rare that the words “finding,” “found,” “conclusion,” or “concluded” were used in an evaluation sense. The “Findings” sections were full of statements that looked like findings, others that looked like conclusions, and still others that looked like recommendations. Many statements of what looked like findings gave no useful indication of what the extent or strength of the evidence base actually was, for example, “Based on observation and staff interviews, the ET found…”, which give readers no idea of whether the finding is supported by multiple observations and multiple interviews with individuals who are knowledgeable about the area in question or an impression or opinion based on slight data from the evaluation team. Statements that looked like conclusions only rarely included text to indicate what findings were used to support the statement. In short, after six months and multiple versions of an evaluation report, MSI still cannot identify how the evaluation team produced findings from the data it collected or generated conclusions from the findings it apparently produced. Statement of Difference Management Systems International msiworldwide.com Page 2 of 6 Contextual and Programmatic Issues This section clarifies key contextual and programmatic errors and misconceptions reported in the evaluation. The number of inaccuracies raise questions as to the ET’s overall understanding of program objectives vis-à-vis the context, project interventions, how and why results are reported, and PROJUST progress and achievements. CONTEXT As indicated in the SOW for the evaluation and the 2017 contract modification, PROJUST’s mandate is to support the consolidation of Mexico’s criminal justice reform process. The evaluation in the background section points to the complexity and challenges facing reform and the Mexican criminal justice system as a whole noting, “… they are part of the territory within which PROJUST operates and which it can do little to affect.” (p.4) These challenges range from “The retention of many traditional practices and vices that do not fit with the new codes…” to constitutional and policy issues. Nonetheless, throughout the evaluation, the ET indicts PROJUST for not addressing these issues that are neither part of PROJUST’s mandate to address nor within its ability to influence. For example, on page v, the report states “Inadequate training of the appellate judiciary and federal judges can undercut efforts to combat formalism and promote effective case resolution.” This ignores the relevance of excessive formalism which stems not only from a lack of training, but to a larger extent from the failure of the Government of Mexico to reform its “Amparo” regime, which is an overarching constitutional mechanism that was developed for the mixed￾inquisitorial model and that jeopardizes the entire accusatorial model – including the flexibility required from trial and appellate and federal judges. This is then reiterated as a recommendation, “Regarding impediments posed by appellate and federal judges, PROJUST might work with the former and encourage discussions on eliminating annulments based on procedural errors.” (page vi) indicating a lack of understanding of the relevance of the Amparo regime as the source of annulments based on procedural errors and thus the possible impact and influence a program focused on structural, rather than policy mechanisms, is intended to achieve. PROGRAM OBJECTIVES AND ACTIVITIES PROJUST interventions are based on a consistent approach aimed at providing State justice institutions with the institutional capacity, systems and tools needed to effectively operate under the accusatory justice system. Working in the federal system of Mexico, this requires an extensive effort in close collaboration with each individual justice institution to adapt all interventions to the local concepts and respond to local needs and priorities. As the examples below indicate, the evaluation does not demonstrate an understanding of how PROJUST works with each State institution, reflecting the limited time the ET spent interviewing and discussing the program with PROJUST technical staff. • Page iii: “PROJUST contributed a key initial role in developing UMECAs, getting them required and regulated by federal law, and improving their performance. Notwithstanding, the model does not appear to have been based on reliable estimates of demand, caseloads, or budgets.” This statement is contradicted later in the report, (p.21) "In any event, PROJUST’s attempt to calculate reasonable workloads and staffing for the states where it operates merits mention. The model developed by PROJUST was produced after extensive Statement of Difference Management Systems International msiworldwide.com Page 3 of 6 empirical, field-based evidence was processed. Moreover, the model is a self￾assessment tool leading to a work plan based on each unit’s strengths and weaknesses. The full operation of the model is purposefully designed to take into consideration the specific realities of each State. • Page iii: “PROJUST directly contributed to installing Analysis, Monitoring, and Evaluation Units (UAMEs). While these UAMEs continue to operate, they have several systematic weaknesses, including poorly qualified, trained, and motivated staff; limited vision; and a lack of basic statistical analysis skills.” Page 47: “Unless it finds states prepared for this approach, PROJUST should stop creating UAMEs because they are beyond the capacity of most institutions/personnel.” It is true that some UAMEs have yielded better results than others, and PROJUST agrees that UAMEs have not yet yielded the hoped-for results. However, the need for this function and for the generation of reliable and systematic data throughout the justice sector, as mentioned repeatedly in the evaluation, is not in dispute. For example, “…there are few state (or federal) justice agencies that produce performance statistics based on more than manual counts and Excel tables produced periodically by individual work units.” (p.9) Additionally, as this is an initiative that has not been previously attempted in Mexico’s criminal justice system, PROJUST believes that this is a process that requires more time and steady effort and abandoning support at this stage would be premature. • Page vi: “Inadequate attention to evidence-based human resource planning negatively impacts sustainability.” This statement suggests that PROJUST has not paid attention to evidence-based resource planning. This is not the case. One example includes a subcontract through which PROJUST is helping Coahuila’s Attorney General to transit to full autonomy by conducting an assessment of human resource needs, one which accounts for workloads and public demand (“Estudio de Volumetria”). The report includes a number of misstatements about the design and implementation of models and the program approach. Of particular note is the lack of understanding around how and to what extent PROJUST has engaged stakeholders including civil society, the private sector and state government. • Page iii: “Extending ownership to a wider portion of the local population has not occurred owing to the public’s limited contact with the system, conflicting interests, distrust of sector institutions, and a limited history of collaboration within civil society. This poses serious risks to the long-term sustainability of the reforms to which PROJUST has contributed.” Acknowledging the need to more fully engage civil society, PROJUST’s modification in 2017 led to the deployment of various initiatives, including the rapid results 100-Day challenges, which have yielded noteworthy results. For instance, the Citizens Security Council in Baja California is already planning to directly fund initiatives to scale the transformational, multi-institutional efforts combating impunity throughout the state, as Statement of Difference Management Systems International msiworldwide.com Page 4 of 6 the ones introduced by PROJUST. Other initiatives include the establishment of local and national CSO networks and the facilitation of cross-sector exchange and dialogue intended to provide civil society with the tools to hold the justice sector accountable as well as engage in the provision of services for victims and the protection of due process. • Page iii: “PROJUST has been less effective in promoting ownership among state and local governments. Were the federal government to draw down its support to justice system reforms, then it becomes even more critical to cultivate backers in state government.” This is an unfounded assertion. There are numerous examples of PROJUST engagement with state and local governments which were included in this version of the report (p. 32) which show the support and participation of state and local governments. • Page v: “It is a challenge to encourage cooperative problem solving among justice institutions, which is made even more difficult by introducing other local actors into the collective deliberations.” The experience of the eight cities involved in the Rapid-Results methodology, some of which were visited by the ET, contradict this finding and represent an important achievement of the project. By involving the private sector and civil society organizations and facilitating the dialogue and team work with and among justice sector public institutions and other local actors, such as Vida Con Calidad, A.C. (VICALLI) in Monterrey and Citizen Council on Public Security of Baja California in Tijuana, impressive results were achieved. After participating in the Rapid-Results initiative both of these CSOs received a grant from PROJUST to continue their work supporting the consolidation of the accusatory system. Rapid Results is also a mechanism through which PROJUST has been able to engage the private sector. For example, setting the stage for an ongoing and growing collaboration with Mexican business association, COPARMEX. PROJUST’s programming is complex, diverse and dynamic and geographic and technical priorities have been adjusted throughout the project as a result of contract modifications as well as in response to needs and effectiveness of programming. The numerous misstatements of program facts found throughout the evaluation indicate findings and conclusions are not based on a full or correct reading of program implementations. A sample of such inaccuracies include: • Page ii (with further explanation on p. 11): “PROJUST has trained large numbers of justice sector institutions and actors contributing to positive outcomes in terms of increased sector capacity, but the positive effects tend to be more moderate and focused on specialized training relative to general trainings.” Training prior to the 2017 contract modification under PROJUST and its predecessor project was indeed generalized training. Evidence is not provided to show that this early “generalized’ training was less effective. Training delivered by PROJUST since the modification is mostly specialized, not general, as is consistent and appropriate for the shift in objectives and activities of the project. • Page ii: “PROJUST’s Rapid Results exercise contributed to an acceleration in courts’ hearing schedules and an increase in the number of hearings held and better Statement of Difference Management Systems International msiworldwide.com Page 5 of 6 management of judges’ schedules. However, judicial opposition remains a potential obstacle to further progress in this area.” All State Courts (Chihuahua, Chihuahua; Monterrey, Nuevo Leon; Villahermosa, Tabasco; and Zacatecas, Zacatecas) benefited through the Rapid Results first round requested the exercises to be repeated and scaled up to other cities. For instance, in Baja California, the exercise was launched for a second time in Tijuana and also scaled up to Mexicali. Similar experiences occurred in Chihuahua and Zacatecas. Moreover, local authorities in Nuevo Leon, including the State Court, decided to incorporate the rapid results methodology as a standard of operation, and, without PROJUST assistance, launched a second 100-day challenge on their own. MSI notes that the discussions with local stakeholders regarding sustainability of the first-round results, along with plans for their scaling up, had just begun when this evaluation was first drafted. This included effectively addressing and overcoming any and all possible reservations from all relevant actors. The Rapid Results methodology is intended to demonstrate the ability of institutions to change the way they work and the impact this can have in order to break down just this type of institutional or cultural barrier to change. • Page 14: “Rapid Results exercises had been conducted or were underway in all target states.” Rapid Results has only been conducted in 5 of PROJUST’s 11 target states. • Page 20: “UMECAs have two basic functions: recommendations on pretrial release or detention and monitoring of compliance with imposed conditions.” The first basic function of UMECAs is to conduct risk evaluations. Only a few UMECAs provide specific recommendations in favor or against pre-trial release or detention, rather they emit technical opinions that can be used by other justice operators to determine the need for pre-trial detention as well as appropriate precautionary measures. • Pages 17 and 29: Referring to ADR services, the report states on page 17, “There are still fundamental problems (e.g., notification in hearings) that neither [MSI nor DPK] has been able to address.” On page 29, “Nonetheless, certain functional areas remain weak at the ADRs and require ongoing assistance (being provided by MSI).” MSI has no mandate to address ADR and does not provide direct assistance to ADRs. • Page18-19: The evaluation refers to, “A recent initiative in Zacatecas, to train and certify local trainers,…” PROJUST does not certify trainers nor training curriculum. • Page vi: “Before introducing more tools and programs, PROJUST should focus on improving what it has already have developed and elaborating plans for its scaling up.” PROJUST has already ceased developing new tools. All existing tools continue to be tested and are being passed to national and state organizations, for them to scale up. MISUNDERSTANDING OR INCORRECT INFORMATION ON INDICATORS AND RESULTS REPORTING MSI performance monitoring data is generated from the USAID approved M&E Plan. The project is now reporting on the third iteration: the first was approved in February 2015; it was Statement of Difference Management Systems International msiworldwide.com Page 6 of 6 adjusted in June 2017 and then again in November 2017, both times with USAID input and approval. Indicators were changed to align with program adaptations to the local context and in accordance with contract modifications. These modifications initiated significant change in technical implementation and necessarily entailed adjustments geographically as well as technically, resulting in indicators that were not tracked after 2017 and others that were newly added to the M&E Plan and may have not produced significant results at the time the evaluation work was conducted given their newness. It should also be noted that the ET only interviewed the PROJECT COP and the DCOP once, and held one, short round of thematic interviews with the technical staff. This took place prior to field visits. After conducting the field visit, the ET lead came back to the office for one very brief visit. The PROJUST team believed that these visits could not have provided the ET the necessary time to access all available project information. The following and numerous mis￾statements related to indicators listed below supports this belief and indicates the ET’s lack of understanding of how the M&E Plan evolved and the implications for reporting of results. o (p.11) The report indicates the project reports against 9 contractual indicators when in fact MSI reports against 13 custom contractual indicators and 7 USAID/Foreign Assistance Indicators. Tetra Tech/DPK reports against additional project indicators. o (page i): The evaluation states, “The primary methodological limitations […] were limitations in the quality, accuracy, consistency, and comparability of project performance monitoring data”. However, this is contradicted on page 12 when the report states, “The ET has no reason to believe the reported results are inaccurate.” and the ET repeatedly notes the difficulty they had due to insufficient and unreliable data from justice sector institutes. The PROJUST team reviews and verifies all information on project indicators it receives from justice sector institutions as part of its reporting. o (p.11) “…MSI did not have a contractual indicator before 2017.” MSI reported results against indicators and targets as laid out in the USAID-approved M&E Plan from February 2016. o (p. 11, 20, 28) The report refers to “Pretrial Confinement Rates” as a contractual indicator. Since 2017 PROJUST has reported on this as a context and not a performance indicator. o (p.12) The report references, “These and other performance indicators…” with footnote #20 which lists contractual indicators as context indicators. o (p.12, 13) The table listing data on Plea Agreement Convictions is inaccurate. Data reported in 2016 was reported as n/a for Tabasco and 0 for Zacatecas. PROJUST reported this as 100% for both states. In both instances, reporting was based on only 1 case. If the ET felt this was not sufficient to merit a finding and therefore changed the way the outcome was reported, accurate reporting would have included an explanation of this change to avoid perception of bias. o (p.22) “MSI reports that 14 UAMEs have been installed to date exceeding the life-of-project target of eight.” The life-of-project target for UAMEs is 18.