PHOTO CREDIT: BELÉN PEÑA FOR USAID. DESCRIPTION: WITHOUT POLITICAL CONNECTIONS, ASTRID SALINAS WAS SELECTED FOR PUBLIC SECTOR JOBS THROUGH A COMPETITIVE PROCESS. FINAL PERFORMANCE EVALUATION USAID/PARAGUAY DEMOCRACY AND GOVERNANCE PROGRAM Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013, Tasking N002 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | i DRG LEARNING, EVALUATION, AND RESEARCH ACTIVITY USAID/PARAGUAY DEMOCRACY AND GOVERNANCE PROGRAM Final Performance Evaluation TASKING N002 OCTOBER 2019 Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013 SUBMITTED TO: USAID/Paraguay SUBMITTED BY: Luis A. Camacho (Principal Investigator) Christopher Hansen (Senior Research Analyst) Renée Hendley (DRG-LER Project Manager and Survey Director) CONTRACTOR: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Renée Hendley Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | ii TABLE OF CONTENTS ACRONYMS............................................................................................................................................................V EXECUTIVE SUMMARY....................................................................................................................................... 1 1. BACKGROUND................................................................................................................................................ 5 Civil Service Reform and Human Resource Management..................................................................... 6 Internal Control Systems.............................................................................................................................. 7 2. EVALUATION OBJECTIVES AND QUESTIONS ..................................................................................... 7 Evaluation Objectives..................................................................................................................................... 7 Evaluation Questions...................................................................................................................................... 8 Sub-questions about Civil Service and Human Resource Management ....................................................8 Sub-questions about Internal Controls.................................................................................................................9 3. METHODOLOGY...........................................................................................................................................10 Desk review....................................................................................................................................................10 Key Informant Interviews............................................................................................................................10 Public Sector Employee Survey..................................................................................................................11 Focus Groups Discussions..........................................................................................................................12 Limitations.......................................................................................................................................................13 4. FINDINGS .........................................................................................................................................................14 Evaluation Question 1..................................................................................................................................14 Evaluation Question 2..................................................................................................................................32 Evaluation Question 3..................................................................................................................................34 Evaluation Question 4..................................................................................................................................37 Evaluation Question 5..................................................................................................................................40 Gender............................................................................................................................................................. 42 5. CONCLUSIONS..............................................................................................................................................44 6. RECOMMENDATIONS................................................................................................................................. 46 BIBLIOGRAPHY ...................................................................................................................................................48 ANNEX A, STATEMENT OF WORK............................................................................................................49 ANNEX B, KEY INFORMANTS.......................................................................................................................59 ANNEX C, EVALUATION DESIGN MATRIX.............................................................................................66 ANNEX D, KEY INFORMANT INTERVIEW GUIDE ................................................................................71 ANNEX E, EMPLOYEE SURVEY DETAILS....................................................................................................75 ANNEX F, EMPLOYEE SURVEY QUESTIONNAIRE (VERSION C)......................................................80 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | iii ANNEX G, CHARACTERISTICS & DIFFERENCES ACROSS SURVEY VERSIONS ...........................99 ANNEX H, FOCUS GROUP DISCUSSION PARTICIPANTS................................................................107 ANNEX I, FOCUS GROUP DISCUSSION GUIDE...................................................................................109 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | iv TABLES Table 1. Summary of progress on MECIP implementation in Executive Branch institutions...........27 Table 2. Percentage of respondents indicating they have observed “Additional payments (cash bribes), gifts, or favors have been accepted in exchange for exceptions to regulations, unwarranted advantages, or privileges” (list protocol)......................................30 Table 3. Percentage of respondents indicating that they have only observed ethical behavior (dichotomous randomized-response protocol)..........................................................................31 Table B.1. Baseline informants............................................................................................................................60 Table B.2. Midline informants .............................................................................................................................62 Table B.3. Endline informants.............................................................................................................................63 Table C.1. Evaluation Design Matrix..................................................................................................................67 Table E.1. Sample Strata.......................................................................................................................................76 Table G.1. Characteristics of Survey Versions A, B, and C....................................................................... 100 Table G.2. Differences between Original Version and CSJ Version........................................................ 105 Table F.1. Participants in FGDs....................................................................................................................... 108 FIGURES Figure 1. Percentage of respondents mentioning the Paraguay Concursa portal as medium to publicize vacancies in their institutions.........................................................................................17 Figure 2. Percentage of respondents indicating merit-based hiring in the public administration of Paraguay increased in the last year...........................................................................................18 Figure 3. Percentage of respondents indicating they receive a formal, written performance evaluation at least once a year........................................................................................................20 Figure 4. Percentage of respondents agreeing with the statement “The highest authorities are committed to the proper operation of the institution” ...........................................................23 Figure 5. Percentage of respondents indicating absenteeism in their institutions is almost non￾existent or low....................................................................................................................................24 Figure 6. Percentage of respondents indicating they are satisfied with their work, overall..............26 Figure 7. Percentage of respondents indicating that women and men have the same hiring, promotion, and pay opportunities in their institutions, by gender........................................42 Figure 8. Percentage of respondents indicating that sexual harassment is a common practice in their institutions, by gender........................................................................................................43 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | v ACRONYMS AECID Spanish Agency for International Development Cooperation (Agencia Española de Cooperación Internacional para el Desarrollo) AGPE Office of the Auditor General for the Executive Branch (Auditoría General del Poder Ejecutivo) CAJ Administration Council of the Judiciary (Consejo de Administración Judicial) CEAMSO Centro de Estudios Ambientales y Sociales CGR Office of the Comptroller General (Contraloría General de la República) CIEJ International Center for Judicial Training (Centro Internacional de Estudios Judiciales) CIRD Centro de Información y Recursos para el Desarrollo CSJ Supreme Court of Justice (Corte Suprema de Justicia) D&G Democracy and Governance DENDE Desarrollo en Democracia DGP Democracy and Governance Program DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity FGD Focus Group Discussion GEAM Gestión Ambiental GoP Government of Paraguay HR Human Resources IADB Inter-American Development Bank ICED Instituto de Estudios para la Consolidación del Estado de Derecho (ICED) IDEA Instituto de Derecho y Economía Ambiental IGP Human Resource Management Index (Índice de Gestión de Personas) INAPP National Institute for the Paraguayan Public Administration (Instituto Nacional de la Administración Pública de Paraguay) KII Key Informant Interviews MAG Ministry of Agriculture (Ministerio de Agricultura) MCC Millennium Challenge Corporation MEC Ministry of Education and Culture (Ministerio de Educación y Cultura) CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | vi MECIP Standardized Internal Control Model of Paraguay (Modelo Estándar de Control Interno del Paraguay) MIC Ministry of Industry and Commerce (Ministerio de Industria y Comercio) MINJUS Ministry of Justice (Ministerio de Justicia) MoF Ministry of Finance (Ministerio de Hacienda) MOPC Ministry of Public Works and Communications (Ministerio de Obras Públicas y Comunicaciones) MSPBS Ministry of Public Health and Welfare (Ministerio de Salud Pública y Bienestar Social) NGO Non-governmental Organization OECD Organisation for Economic Co-Operation and Development PMP Performance Management Plan SEAM Environment Secretariat (Secretaría del Ambiente) SENADIS National Secretariat for Disable People’s Human Rights (Secretaría Nacional por los Derechos Humanos de las Personas con Discapacidad) SFP Civil Service Secretariat (Secretaría de la Función Pública) SIAGPE Integrated System of the Office of the Auditor General for the Executive Branch (Sistema Integrado de la Auditoría General del Poder Ejecutivo) SICCA Integrated Control System for the Administrative Civil Service (Sistema Integrado de Control de la Carrera Administrativa) SOW Statement of Work USAID United States Agency for International Development USAID/Paraguay United States Agency for International Development Mission in Paraguay CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 1 EXECUTIVE SUMMARY As part of the Democracy, Human Rights, and Governance Learning, Evaluation, and Research (DRG-LER) Activity, USAID requested that NORC at the University of Chicago design and carry out a performance evaluation of USAID/Paraguay’s Democracy and Governance Program (DGP). The DGP is a seven-year program (FY2014-FY2020) funded by the United States Agency for International Development Mission in Paraguay (USAID/Paraguay) and implemented through a Cooperative Agreement with Centro de Estudios Ambientales y Sociales (CEAMSO), a non-governmental organization (NGO) headquartered in Asunción, Paraguay. The purpose of the program is to strengthen internal management and governance systems in selected entities of the Government of Paraguay (GoP). The DGP is organized in four components. Component 1 works with selected entities of the Executive Branch and with the Judiciary on issues related to civil service reform, administrative reforms, and service provision improvement. Component 2 works with selected entities of the Executive Branch and with the Judiciary to strengthen transparency and anticorruption efforts. Component 3 works with civil society to advance governance-enhancing legal and policy framework reforms. Component 4 focuses on strengthening CEAMSO as a sustainable and leading Paraguayan civil society organization that manages development projects for quality results and impact. The evaluation focuses on DGP activities aiming to advance civil service reforms and internal control systems, which were respectively undertaken under Components 1 and 2. The main objective of the evaluation is to examine whether DGP activities have resulted in the strengthening of internal management systems in GoP entities, and whether program activities have contributed to changes in the perceptions and behaviors of public sector employees.1 In addition to examining DGP’s results and impacts, the evaluation probes issues related to the program’s operational efficiency, sustainability, and appropriateness of focus. It also identifies lessons learned and priority areas for future USAID programs. NORC’s evaluation follows a mixed-method approach that analyzes qualitative and quantitative data to construct a compelling analytic narrative that addresses the evaluation questions. The analysis draws on documentation and secondary sources, performance monitoring data, and original data collected through key informant interviews (KIIs) with USAID/Paraguay staff, DGP staff, consultants, and subcontractors, government counterparts, and other stakeholders, as well as surveys and focus group discussions (FGDs) with public sector employees. The surveys and FGDs covered five GoP institutions, four of which received DGP support. To examine the DGP’s contribution to changes in the perceptions and behaviors of public sector employees, the surveys were carried out before and after the implementation of the bulk of DGP activities and in one entity that did not receive DGP support. Altogether, these multiple sources of information allow the evaluation team to triangulate findings, ensuring that conclusions are robust and recommendations are sound. Below is a summary of the main findings of the final performance evaluation corresponding to each of the five main evaluation questions. 1 This report uses the term “employee” to refer to those who work in the public sector or a specific institution. The term does not make any distinction between employment type, contractual arrangement, or the like – e.g., civil servants vs. political appointees; executive- vs. mid- and low-level staff; professional vs. non-professional staff; those working under unlimited- vs. limited-term contracts; and those who are paid through payroll vs. consultants or other non-payroll-based arrangement. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 2 EVALUATION QUESTION 1. Have internal management systems within the GoP been strengthened as a result of DGP interventions? Have those improvements led to changes in the perceptions and behavior of public sector employees? The DGP has contributed to strengthening internal management systems. In the area of civil service and human resource management, DGP support was key to strengthening the leadership of the Civil Service Secretariat (Secretaría de la Función Pública, SFP). The DGP provided key resources for the development of the Integrated Control System for the Administrative Civil Service (Sistema Integrado de Control de la Carrera Administrativa, SICCA), including its hiring module and the Paraguay Concursa portal. GoP institutions have widely adopted the portal – with and without DGP support – and more and more vacancies are being advertised and filled using the new system. Support to the SFP has also contributed to improvements in institutional quality of the GoP civil service system. In the area of internal controls, the DGP set out to support the Office of the Auditor General for the Executive Branch (Auditoría General del Poder Ejecutivo, AGPE) and the Office of the Comptroller General (Contraloría General de la República, CGR) in their goal of achieving the full implementation of the Standardized Internal Control Model of Paraguay (Modelo Estándar de Control Interno del Paraguay, MECIP), including it simplified version, MECIP 2015. During implementation, however, the AGPE withdrew its support to the process, and leadership turnover in the CGR slowed the pace of progress. While GoP institutions have been making progress in complying with MECIP regulations and improving their internal control systems, the prospects for the full adoption of MECIP 2015 are uncertain. NORC’s public sector employee surveys and FGDs indicate that these improvements have produced limited changes in the perceptions and behaviors of public sector employees. In the area of civil service and human resource management, employees of all institutions indicate that Paraguay Concursa is increasingly used to publicize vacancies. However, increased usage and visibility of the portal has not been accompanied by similar improvements in perceptions about merit-based hiring. Similarly, in the area of performance evaluation, the evidence indicates that regular written evaluations were taken up in most ministries, but employees do not perceive that this has been accompanied by an increase in evaluation fairness, recognition of hard work and initiative, and other elements that indicate clear movement toward a merit-based system. In the area of internal controls, the evidence suggests that progress in MECIP implementation has likely not contributed to systematically curtailing unethical behavior or acts of corruption, or to increasing accountability for such acts. EVALUATION QUESTION 2. Are inputs provided by the DGP contributing to identifiable changes in processes and systems capacity in its focus areas? How effective has the program been in transferring skills, knowledge, and capacity to GoP staff? The DGP has contributed to identifiable changes in processes and systems in the areas of civil service and human resource management and internal controls. However, the evaluation team has concerns about the sustainability of achievements and about the appropriateness of efforts given the program’s overall objectives. With regard to program effectiveness in transferring skills, knowledge, and capacity to GoP staff, the evaluation team’s assessment is that DGP used an adequate mix of approaches to achieve this goal, but cannot directly assess effectiveness. EVALUATION QUESTION 3. What deficiencies currently exist in the implementation of the program’s objectives and how can these be remedied? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 3 The DGP’s implementation strategy in the areas of civil service and human resource management and internal control systems was not fully adequate to achieve the program’s overarching objectives. The DGP’s implementation strategy was two-pronged: support regulatory institutions in the development and implementation of policies and regulations; and support executing institutions in implementing these policies and regulations – i.e., ministries and other institutions. In supporting regulatory institutions, the program approached its objectives from a mostly technical perspective, not fully taking into account various political economy factors that made implementation inherently complex, especially in the area of internal controls and in work pertaining to the Judiciary Branch. In particular, the DGP was not fully prepared to anticipate and deal with competing interests and preferences within and across institutions, and with over-time change in said interests and preferences. In supporting implementing institutions, the strategy focused on achieving several outputs in multiple institutions, sometimes losing sight of overall objectives. For example, adoption of Paraguay Concursa is necessary but not sufficient to achieve merit-based hiring. Similarly, requiring periodic personnel evaluations is only the first step toward establishing a system that rewards good performance and merit. Supporting the full implementation of reforms in a handful of executing institutions would allow for achieving various outputs that, together, contribute to making substantial progress toward overall objectives. EVALUATION QUESTION 4. Is the current scale and scope of the program appropriate to support sustainability? Should certain aspects or components be prioritized or limited? The DGP scale and scope within regulatory institutions responsible for developing policies and regulations is on target, though sustainability is a concern given the limited enforceability and potential for reversibility of the reforms. The DGP scale and scope within executing institutions responsible for implementing these policies and regulations was not adequate. Given available resources, by choosing to support multiple executing institutions on several lines of work, the DGP was implemented in a diluted, fragmented fashion, achieving various outputs, but not managing to make substantial progress toward the DGP’s ultimate objectives. EVALUATION QUESTION 5. What are the good practices emerging from the program? Are there successes that could be replicated for future projects? There are three good practices emerging from the DGP that future projects should try to replicate. • DGP’s decision to completely overhaul specific processes involving service provision to the citizenry (funciones misionales) under the MECIP – e.g., the scheduling of surgeries at the MSPBS’s Hospital de Trauma or the centralized registry of professional degrees at the MEC. Achieving results in areas that are highly visible to the citizenry in a relatively short period of time is an asset for any program that seeks to advance complex reforms. In line with this good practice, future programs with limited resources should focus on supporting full reform implementation in a handful of executing institutions. • Use of diagnoses to identify specific targets for technical assistance. For example, the 2014-2016 IGP measurement provided inputs to identify priority areas in the area of human resource management. Likewise, within the MECIP 2015 framework, process overhaul was informed by risk assessments and evaluations. Future projects should embrace evidence-based decision-making in their intervention strategies as well as seek to promote it among counterpart staff. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 4 • Promoting the use periodic assessments as a tool to advance reform implementation. For example, the DGP supported the development of Portal MECIP at the AGPE, which allows Executive Branch institutions to conduct internal control self-assessments and submit them to the AGPE for monitoring. Periodic performance assessments not only enhanced the AGPE’s monitoring abilities but also created reputation-related pressures on low-performing institutions to step up compliance efforts. Similarly, the DGP supported the development of IGP Digital with the aim of lowering the cost of IGP assessments and establishing a sustainable model to monitor progress in improving human resource management systems. In light of these findings, the evaluation team recommends the following: • USAID/Paraguay should consider resuming its work with the SFP to support civil service and human resource management reforms. This should include ongoing maintenance of the SICCA and the Human Resource Management Index (Índice de Gestión de Personas, IGP) tools as well as supporting further improvements in the areas of hiring, promotion, and compensation. Given the SFP’s institutional weakness and lack of resources, DGP support was key to sustain normal operations at the SFP. Without support from other donors, and absent increased funding and political commitment from the GoP to strengthen the SFP, it is possible that DGP-supported achievements will be short-lived. • USAID/Paraguay should consider resuming its support in the area of internal controls only if the impasse between the AGPE and the CGR regarding the MECIP can be resolved. If disagreement between institutions cannot be resolved, it would be sensible to prioritize supporting civil service and human resource management reform. • USAID/Paraguay should consider continuing its support to the Administration Council of the Judiciary (Consejo de Administración Judicial, CAJ) and the Supreme Court of Justice (Corte Suprema de Justicia, CSJ) in overhauling internal management systems in the Judiciary. The DGP contributed to the establishment of the CAJ, which was a long-standing, but elusive strategic objective. The aim of this reform is to relieve Supreme Court Justices from having to directly handle internal management and operations – and allowing them to focus on the administration of justice – and to create a better environment for the improvement of internal management and operations themselves. Continued DGP support could help set the CAJ in a trajectory to achieving its mandate and supporting the aims of the reform. • USAID/Paraguay should prioritize depth over breadth in its initiatives seeking to improve internal management systems in the GoP, supporting the ideation, development, and regulatory processes of a very limited number of reforms and their full implementation in a handful of institutions. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 5 1. BACKGROUND As part of the DRG-LER Activity, USAID contracted NORC at the University of Chicago to design and carry out a performance evaluation of USAID/Paraguay’s DGP. The DGP is funded by USAID/Paraguay and implemented through a Cooperative Agreement with CEAMSO, an NGO headquartered in Asunción, Paraguay. The DGP was initially a five-year program (FY2014-FY2018), but was later extended by two additional years (FY2020). It is expected that the program will receive another extension and end in March 2021. The purpose of the DGP is to strengthen internal management and governance systems in selected institutions of the GoP necessary for a responsive democracy. The development hypothesis underpinning this program is that by developing the government’s institutional capacity and strengthening accountability mechanisms, the GoP will govern more effectively. While the short￾term objectives are to enhance transparency and accountability for government institutions and develop capable and responsive government, the long-term objectives are to reduce corruption, increase government responsiveness to citizens’ needs, and ensure a strong institutional foundation for democracy in Paraguay. Embracing key aspects of USAID Forward reforms, the DGP integrates capacity development and systems strengthening to drive sustainable reforms and institutional change in the Paraguayan public sector. It actively promotes local ownership and alignment with host country priorities by focusing on demand-driven interventions, building on the success of country-led initiatives, and providing strategic government-to-government assistance to key public institutions. Importantly, the project also expands and capitalizes on key reform processes and interventions initiated under the USAID￾managed Threshold Programs of the Millennium Challenge Corporation (MCC). USAID wove lessons learned and best practices identified in these programs into DGP’s design. The purpose of the DGP is to strengthen internal management and governance systems in selected public institutions. At inception, the program had three sub-purposes: Sub-purpose 1. Increased institutional capacity of select public institutions; Sub-purpose 2: Accountability and anti-corruption efforts strengthened in select public institutions; and, Sub-purpose 3: Legal and policy framework for effective governance advanced. In 2016, the Cooperative Agreement was modified to include one additional sub-purpose, with the aim of ensuring the sustainability of DGP’s implementer: Sub-purpose 4: CEAMSO strengthened as a sustainable and leading Paraguayan civil society organization that manages development projects to obtain quality results and impact. The DGP is organized into four components. Component 1 is primarily responsible for the achievement of sub-purpose 1; it works with selected institutions of the Executive Branch and with the Judiciary to improve human resource management and the Paraguayan civil service, more generally, budgeting and acquisition processes, and service provision. Component 2 is primarily responsible for sub-purpose 2; it works with selected institutions of the Executive Branch and with the Judiciary to overhaul internal control systems, increase transparency, and strengthen anticorruption efforts. Component 3 is primarily responsible for sub-purpose 3; it works with government and civil society to advocate for the enactment and facilitate the implementation of CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 6 governance-enhancing legal and policy reforms. Lastly, Component 4 works exclusively on sub￾purpose 4, building the institutional capacities of CEAMSO on administration and reporting, human resource acquisition and management, and business development. The evaluation focuses on the subset of DGP activities aiming to improve the civil service and human resource management and internal control systems. These activities were respectively undertaken under Components 1 and 2. The following subsections briefly describe the DGP’s approach to achieving its objectives in these two areas. CIVIL SERVICE REFORM AND HUMAN RESOURCE MANAGEMENT2 A principal objective of the DGP was to advance the professionalization of civil service through human resource reform (i.e., Component 1). The DGP supported the main regulatory agency in this sector, the SFP, in its work to promote transparency and meritocracy across staff recruitment, hiring, performance evaluation, and promotion in selected institutions of the Executive Branch. Specifically, the DGP facilitated the development of human resources software and information systems as well as provided training and other technical assistance. A hallmark of these reforms was the development and implementation of the SICCA. SICCA is a web-based platform that aims to standardize, digitize, and streamline processes in eight core human resource areas, including talent acquisition, training, classification, and compensation. SICCA launched formally in 2014 following a six-year development phase funded initially by the Inter￾American Development Bank (IADB) and then by USAID. SICCA was formally adopted through Presidential Decree 1.212/2014 and further regulated with additional decrees and SFP resolutions. In the context of SICCA implementation, the DGP supported selected institutions in formalizing job descriptions and standardizing titles and pay scales. The Paraguay Concursa portal is a key component of the SICCA reform. It serves as an online portal for institutions to post job announcements and collect applications with the intent of promoting transparent, merit-based hiring. The DGP supported the development of the portal and its adoption in selected institutions of the Executive Branch. With support of the DGP, the Paraguay Concursa portal also has been adapted for the Judiciary Branch and is used for the selection of civil servants who support judges, such as secretaries and paralegals. Another DGP-supported initiative related to human resource management is the digitization of the IGP. The IGP is a self-assessment tool that institutions use to monitor their performance on human resource management. The SFP supported IGP assessments in selected institutions as part of a wider effort to improve the institutional quality of the GoP civil service system. First developed in 2011, IGP contains over 100 indicators designed to assess and measure progress in areas of human resources management. The tool is used to identify weaknesses and develop action plans to address them. The DGP supported the development of IGP Digital, a web-based version of the IGP, and associated support materials, seeking to simplify the complexity of the assessment and increase its uptake and sustainability. 2 Background information on SICCA and IGP in this section comes primarily from the following source: Organisation for Economic Co-Operation and Development (OECD), Estudios de la OCDE sobre Gobernanza Pública: Paraguay: Hacia un desarrollo nacional mediante una gobernanza pública integrada, Paris, 2018. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 7 INTERNAL CONTROL SYSTEMS3 Another principal objective of DGP was to promote accountability and anti-corruption activities in key agencies of the GoP (i.e., Component 2). Specifically, DGP sought to contribute to the institutional strengthening of the GoP’s primary regulatory agencies in the area of internal controlsthe AGPE and the CGRby supporting them in the development of reforms. The DGP then supported reform implementation in selected institutions of the Executive Branch and the CSJ, providing technical assistance and information systems support, and contributing to training. Internal control reform was pursued by formalizing and implementing the MECIP. The MECIP is a complementary set of norms, policies, and procedures designed to promote effectiveness, efficiency, and integrity in public administration, and mitigate the risks of fraud, waste, and abuse. Under Component 2, the DGP supported the implementation of the MECIP in several institutions including the CSJ, the Ministry of Agriculture (MAG), the Ministry of Education and Culture (Ministerio de Educación y Cultura, MEC), the Ministry of Public Health and Welfare (Ministerio de Salud Pública y Bienestar Social, MSPBS), and the Ministry of Public Works and Services (Ministerio de Obras Públicas y Comunicaciones, MOPC). The adoption of the MECIP was facilitated by a series of laws and resolutions. In 1999, Law 1535 for the Administration of Public Finance provided for the development of an internal control framework for public institutions. In 2007, Presidential Decree 10883 empowered AGPE to coordinate and supervise the implementation of the MECIP. In 2008, Presidential Decree 962 approved the implementation of the MECIP in the agencies subject to Law 1535. Following a diagnosis of the factors explaining the lack of progress regarding the MECIP 2008, the AGPE and the CGR decided to update the standard. The newest version of the MECIP was introduced in 2015 and adopted in 2016 by CGR through Resolution 377.4 The 2015 version of the MECIP advances control activities in three areas – planning, management, and evaluation. MECIP 2015 objectives include promoting compliance with agencies’ goals, standardizing codes of ethical conduct among civil servants, strengthening inter- and intra-agency modes of communication, and increasing public access to accurate, timely information. 2. EVALUATION OBJECTIVES AND QUESTIONS EVALUATION OBJECTIVES In line with USAID’s Statement of Work (SOW) presented in Annex A, the evaluation focuses on DGP activities aiming to advance civil service reforms and internal control systems, which were respectively undertaken under Components 1 and 2. The evaluation’s objective is to examine DGP’s results and impacts, namely, whether internal management systems have been strengthened as a result of USAID interventions and if those improvements led to changes in the perceptions and behaviors of public sector employees. The evaluation also probes issues related to the program’s sustainability and appropriateness of focus, and to a lesser extent, operational efficiency and gender. 3 Background information on the MECIP in this section comes primarily from the following two sources: Ministry of Industry and Commerce, “MECIP: 2015.” Asunción, 2015. Accessed April 22, 2019, http://www.mic.gov.py/mic/site/mecip/pdf/Manual%20Mecip-2015.pdf and Secretary of Linguistic Policies (Secretaría de Políticas Lingüisticas), “Sobre MECIP.” Asunción. Accessed April 22, 2019, http://www.spl.gov.py/es/index.php/institucion/mecip. 4 A comparison of the 2008 and 2015 versions of MECIP can be found on pages 40-41 of Ministry of Industry and Commerce, “MECIP: 2015.” Asunción, 2015. Accessed April 22, 2019, http://www.mic.gov.py/mic/site/mecip/pdf/Manual%20Mecip-2015.pdf. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 8 The evaluation ultimately seeks to identify lessons learned and priority areas for future USAID programs. The evaluation has the following main intended users: USAID; CEAMSO; DGP’s counterparts in the GoP; other program stakeholders including non-governmental and civil society organizations; and other international donor agencies active in the democracy and governance and public administration space. EVALUATION QUESTIONS Following a desk review, NORC’s evaluation team developed sub-questions for USAID’s overarching research questions. These sub-questions were refined during a field visit to Paraguay when members of the evaluation team interviewed USAID and DGP staff, government counterparts, and other program stakeholders. Table B.1, Baseline Informants, in Annex B presents the complete list. EVALUATION QUESTION 1. Have internal management systems within the GoP been strengthened as a result of DGP interventions? Have those improvements led to changes in the perceptions and behavior of public sector employees? SUB-QUESTIONS ABOUT CIVIL SERVICE AND HUMAN RESOURCE MANAGEMENT Sub-question 1.1.1. Has the SFP strengthened its leadership in civil service reform? Has it gained influence among Executive Branch institutions? Sub-question 1.1.2. How effectively and extensively is the SICCA being implemented? Sub-question 1.1.3. How effectively and extensively is IGP being implemented? Sub-question 1.1.4. To what extent are hiring and promotions increasingly based on merit? Do employees perceive increasing fairness in hiring and promotions? Are jobs advertised well ahead of time? Are jobs advertised widely enough to assure fair access by the relevant pool of applicants? Is the average number of years of schooling of public employees increasing over time? Sub-question 1.1.5. How regularly are evaluations conducted of employees’ performance? Have all employees been included in some type of evaluation of competency? Are performance evaluations based on set, specified criteria? Sub-question 1.1.6. Do employees feel their hard work and initiative are recognized and rewarded? Do employees see that negative attitudes towards work obligations and laziness are sanctioned? Sub-question 1.1.7. Do employees perceive progress in the institutionalization of a merit system in their agency? Sub-question 1.1.8. Are training actions conducted on a regular basis in all public units? To what extent are training opportunities available? Sub-question 1.1.9. What do employees think about their direct supervisors and their institution’s leadership? Have employees perceptions about their supervisors and their institution’s leadership improved? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 9 Sub-question 1.1.10. Do employees believe their job duties and responsibilities are clearly defined and communicated to them? Do employees feel they are compensated according to their duties and responsibilities? Sub-question 1.1.11. Are the reforms associated with greater job satisfaction and lower absenteeism among federal employees? SUB-QUESTIONS ABOUT INTERNAL CONTROLS Sub-question 1.2.1. Are internal control systems operating effectively? Sub-question 1.2.2. How widely has the MECIP been adopted? Sub-question 1.2.3. To what extent are public sector employees aware of the new systems of internal controls? Sub-question 1.2.4. How often do employees witness unethical behavior or acts of corruption? How often are employees pressured to break rules and engage in acts of corruption? How often are employees forced to unreasonably/illegally redirect benefits to some specific entity/individual? Which are the most common forms of corruption encountered? Sub-question 1.2.5. Is openly unethical/corrupt behavior left unpunished? Sub-question 1.2.6. Do reasonable incentives exist to report the commitment of acts of corruption to authorities? EVALUATION QUESTION 2. Are inputs provided by the DGP contributing to identifiable changes in processes and systems capacity in its focus areas? How effective has the program been in transferring skills, knowledge, and capacity to GoP staff? Sub-question 2.1: Do partner institutions feel inputs provided by DGP have changed the way they function and improve their performance? Sub-question 2.2: Has training provided by DGP improved or changed knowledge, attitudes, and practices of partner staff? EVALUATION QUESTION 3. What deficiencies currently exist in the implementation of the program’s objectives and how can these be remedied? Sub-question 3.1: What are the causes and factors that have hindered the achievement of the DGP’s Performance Management Plan (PMP) targets and progress? Sub-question 3.2: Are any program objectives lagging in results compared to the others? Why? Sub-question 3.3: Have any program themes, cross-cutting elements, or special projects caused controversy or backlash in the country? EVALUATION QUESTION 4. Is the current scale and scope of the program appropriate to support sustainability? Should certain aspects or components be prioritized or limited? Sub-question 4.1: Has the DGP identified key GoP counterparts and is the assistance provided to these on target, based on priority needs? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 10 Sub-question 4.2: Is the scope of DGP assistance flexible to respond to changing priorities or emerging problems within GoP counterparts? Sub-question 4.3: Did the DGP and GoP counterparts have regular “how are we doing” meetings to discuss changing priorities or redundant activities? EVALUATION QUESTION 5. What are the good practices emerging from the program? Are there successes that could be replicated for future projects? Sub-question 5.1: What type of assistance has produced the most improvement? Where and why? Sub-question 5.2: What type of assistance has produced the least progress? Where and why? Sub-question 5.3: What are the emerging lessons that should be kept in mind for future projects? 3. METHODOLOGY NORC’s evaluation follows a mixed-method approach that analyzes qualitative and quantitative data to construct a compelling analytic narrative that addresses the evaluation questions. The analysis draws on documentation and secondary sources, performance monitoring data, and original data collected through KIIs with USAID/Paraguay staff, DGP staff, consultants, and subcontractors, government counterparts, and other stakeholders, as well as surveys and FGDs with public sector employees of five GoP institutions. Altogether, these multiple sources of information allow the evaluation team to triangulate findings, ensuring that conclusions are robust and recommendations are sound. The evaluation relies on the data collection and analysis methods described below. Annex C presents the evaluation design matrix, linking the evaluation questions and sub-questions to these data collection and analysis methods. DESK REVIEW Project documents reviewed include the DGP’s Cooperative Agreement and its modification, work plans, and annual and quarterly progress reports, including the program’s performance monitoring data. Secondary sources include a number of publications analyzing the state of the Paraguayan civil service and the evolution of civil service reforms. The evaluation team used the information contained in these sources for triangulation with original qualitative and quantitative data. KEY INFORMANT INTERVIEWS At endline, NORC’s evaluation team conducted in-depth, semi-structured interviews with a total of 55 informants from USAID/Paraguay, the DGP, including its consultants and subcontractors, government counterparts, and other stakeholders. Thirty-one interviews were conducted in person, during a field visit to Asunción in October 2018; the remaining interviews were conducted via CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 11 teleconference. Some informants were interviewed in groups.5 USAID/Paraguay provided assistance to identify key informants from GoP counterparts and to arrange interviews with them. Both USAID/Paraguay and CEAMSO provided assistance to arrange interviews with DGP consultants and subcontractors. Table B.3 in Annex B lists the informants interviewed at endline. The evaluation team developed an interview guide that included probes on various topics which was adapted for each informant type. The target interview length was 60 minutes. The guide is available in Annex C. KIIs with non-USAID/Paraguay informants were recorded and then transcribed when informants consented to being recorded. The evaluation team used notes and transcripts to synthesize responses to each probe across interviews, identifying recurrent themes. The team also identified appropriate quotes that are included below to illustrate typical and atypical responses. PUBLIC SECTOR EMPLOYEE SURVEY NORC carried out a random sample survey of public sector employees in selected GoP institutions. The survey explores issues related to work climate and human resource management, internal control and compliance with regulations, and job satisfaction. NORC’s original evaluation design called for conducting surveys at baseline and endline, as well as for collecting data in three institutions that would receive DGP support and three comparison institutions that would not receive support. This design would allow for analyzing the over-time evolution of responses across DGP-supported and comparison institutions, providing insights to answer evaluation sub-questions related to DGP’s contribution to changes in employees’ perceptions and behaviors. The following six institutions were included in the baseline survey, which took place between November 2014 and October 2015: the CSJ, the MEC, the MSPBS, the MAG, the Ministry of Industry and Commerce (Ministerio de Industria y Comercio, MIC), and the MOPC. The MAG, the MIC, and the MOPC were initially included as comparison institutions. However, the MAG and the MOPC ended up receiving DGP support, leaving the MIC as the only comparison ministry. In addition, it is worth noting that various difficulties substantially delayed baseline data collection in the MIC, the MOPC, and the CSJ; when data collection activities in these institutions were finally completed in October 2015, DGP interventions were well on their way. Endline data collection took place between October 2018 and February 2019. The MAG was dropped from this wave of data collection because DGP assistance to this institution was discontinued in 2015. NORC subcontracted Economía y Estadísticas para el Desarrollo (E+E), a Paraguay research organization headquartered in Asunción, to support the administration of the surveys. Annex E provides details about the survey design, questionnaire development process, management and quality control arrangements, and administration challenges. 5 An additional 32 informants were interviewed in person, individually or in groups, during an inception field visit to Asunción in August 2014. These interviews were used to develop the evaluation sub-questions and understand DGP’s context, including the state of civil service reforms and internal control systems in Paraguay at the time. Another 24 informants were interviewed in person, individually or in groups, during a midline field visit to Asunción in October and November 2016. These interviews, along with a desk review of DGP and other documentation, and other sources, were used to produce a midterm evaluation report. Table B.1 and Table B.2 in Annex B respectively list the baseline and midline informants. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 12 One novel feature of the questionnaire is the inclusion of three survey experiments or truth elicitation protocols that provide significant degrees of anonymity to respondents and encourage them to provide candid responses to questions related to compliance with regulations and unethical behavior. The characteristics of these protocols are described in detail in Annex E. The implementation of these protocols required developing three different versions of the questionnaires, A, B, and C. Annex F presents version C of the survey instrument and Table G.1 in Annex G lists the characteristics of the various survey versions. Lastly, CSJ-specific versions of the questionnaire were created to include two additional questions requested by CSJ authorities. Table G.2 in Annex G lists the differences between the original and CSJ-specific versions. The analysis below focuses on changes over time in each ministry instead of changes over time by DGP-supported and comparison institutions as proposed in the original evaluation design. Contrasting supported and comparison institutions was no longer feasible because two of the three original comparison ministries received DGP support, and the MIC was the only ministry that served as a comparison institution. For each survey item, the evaluation team compared the proportion of ‘positive’ or ‘desirable’ responses at endline and baseline in a given institution, conducting differences of proportions tests to identify statistically significant changes. The evaluation team also compared responses for all institutions that received DGP support combined. Positive responses refer to those that indicate perceptions or behaviors that are in line with the DGP’s expected results.6 FOCUS GROUPS DISCUSSIONS NORC conducted two FGDs in each of the five institutions included in the employee endline survey, for a total of ten FGDs. Twenty randomly-selected employees were invited to participate in each FGD, with turnout ranging from three to 18 participants. In a given institution, one discussion was dedicated to employees with ‘management’ positions (cargos) and the other, to employees with ‘junior’ or ‘technical’ positions. The intent of grouping participants in this way was to mitigate the risk that junior/technical staff would feel compelled to respond differently in the presence of managers. For convenience and ease of planning, FGDs were held at the ministries during regular work hours. The design section of Annex E provides details about this classification; the sampling procedure followed for FGDs is also described therein.7 Annex H provides details about the participants in each FGD. The evaluation team developed an FGD guide that included probes on topics similar to those examined in the employee survey. The target FGD length was 60 minutes. The guide is available in Annex I. FGDs were recorded and then transcribed. The evaluation team used notes and transcripts to summarize the responses to each probe across groups, identifying recurrent themes. The team also identified appropriate quotes that are included below to illustrate typical and atypical responses. 6 For example, the first question of the survey asks respondents to express their level of (dis)agreement with the statement “The responsibilities of my position were clearly explained to me” using the following response options: strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, and strongly disagree. In this case, agree and strongly agree are considered positive responses. 7 There is one difference between the sampling procedures of the survey and the FGDs. For the survey, the sample is stratified along two dimensions with three levels each – organizational category (cabinet/secretariat, administrative, and operations) and staff category (management, junior, and technical). For the FGDs, the sample is stratified along a single dimension of two levels – staff category (management and junior/technical). It is also worth noting that individuals selected to participate in the survey were removed from the sample frame for the FGDs. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 13 LIMITATIONS The DGP faced great challenges to collect the information necessary to report on several of its performance indicators. Having access to complete performance monitoring data would have allowed NORC’s evaluation team to provide a more thorough assessment of the program. As mentioned earlier, the evaluation team selected key informants from GoP counterparts with assistance from USAID/Paraguay. This selection strategy could have resulted in capturing atypically positive views about the DGP if only satisfied respondents had been interviewed. The evaluation team is confident that this is not the case; interviews were conducted with informants from a broad cross-section of DGP-supported institutions who offered candid, often critical, views about DGP initiatives. Interviews with DGP consultants and subcontractors yielded similar information. With regard to the survey, the baseline report identified social desirability bias as a potential concern. This bias refers to the tendency of survey respondents to answer questions in a manner that will be viewed favorably by others. The evaluation design minimizes the likelihood that this bias leads to erroneous conclusions at endline by analyzing over-time change in responses. If present, social desirability bias should equally affect survey responses at baseline and endline and can be removed by comparing responses across waves. More importantly, the endline survey was conducted in the months following the inauguration of the new administration that occurred in August 2018. During such periods, the appointment of new leadership in Executive Branch institutions is customarily accompanied with significant changes in political appointees (cargos de confianza) and other positions. It is likely that responses to questions related to civil services reforms and human resource management reflect discontent among employees about personnel turnover related to the transition. Lastly, about the FGDs, NORC’s goal was to have eight to ten participants per FGD to allow for meaningful participation from all those present, while covering all topics within the target duration of 60 minutes. As mentioned above, 20 participants were invited to each FGD, with turnout ranging from three to 18 participants – the average was about ten participants. The evaluation team decided that it would be counterproductive to turn away surplus participants as they had incurred opportunity costs to attend the discussion and have a chance to voice their views on issues that were important to them. The moderator encouraged all participants to engage in the discussion and would call on those who had not been as active in the conversation. Due to time constraints, the moderator was not able to cover all probes in each FGD – limiting to some extent the evaluation team’s ability to analyze all themes across groups. The average recording length, which excludes the introduction and consent section of the guide scripted to last five minutes, was 55 minutes. In addition, as noted earlier, two FGDs were held in each of the five ministries, one for management and the other for junior/technical staff. The FGDs were held at the ministries during regular work hours. The FGDs generally included a range of positive and negative opinions; however, the on-site setting and the presence of colleagues may have limited participants’ willingness to fully represent their employment experiences. Lastly, groups were mixed-gender. While this added diversity of perspective to the discussion, it might have impacted women’s willingness to speak candidly about certain topics, including experiences of gender discrimination or perceptions of gender equality in the workplace. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 14 4. FINDINGS This section answers the evaluation questions and sub-questions. It also presents evaluation findings related to the cross-cutting issue of gender. When discussing survey results, item wording and response options are provided in footnotes or as notes to figures. Response options that are considered positive or desirable are underlined. EVALUATION QUESTION 1 HAVE INTERNAL MANAGEMENT SYSTEMS WITHIN THE GOP BEEN STRENGTHENED AS A RESULT OF DGP INTERVENTIONS? HAVE THOSE IMPROVEMENTS LED TO CHANGES IN THE PERCEPTIONS AND BEHAVIORS OF PUBLIC SECTOR EMPLOYEES? The DGP has contributed to strengthening internal management systems. In the area of civil service and human resource management, DGP support was key to strengthening the leadership of SFP. The DGP provided important resources for the development of the SICCA, including its hiring module and the Paraguay Concursa portal. GoP institution have widely adopted the portal – with and without DGP support – and more and more vacancies are being advertised and filled using the new system. Support to the SFP has also contributed to improvements in institutional quality of the GoP civil service system. In the area of internal controls, the DGP set out to support the AGPE and the CGR in their goal of achieving the full implementation of the MECIP, including it simplified version, MECIP 2015. During implementation, however, the AGPE withdrew its support to the process, and leadership turnover in the CGR slowed the pace of progress. While GoP institutions have been making progress in complying with MECIP regulations and improving their internal control systems, the prospects for the full adoption of MECIP 2015 are uncertain. NORC’s public sector employee surveys and FGDs indicate that these improvements have produced only limited changes in the perceptions and behaviors of public sector employees. In the area of civil service and human resource management, employees of all institutions indicate that Paraguay Concursa is increasingly used to publicize vacancies. However, increased usage and visibility of the portal has not been accompanied by similar improvements in perceptions about merit-based hiring. Similarly, in the area of performance evaluation, the evidence indicates that regular written evaluations were taken up in most ministries, but employees do not perceive that this has been accompanied by an increase in evaluation fairness, recognition of hard work and initiative, and other elements that indicate clear movement toward a merit-based system. In the area of internal controls, progress in MECIP implementation has likely not contributed to systematically curtailing unethical behavior or acts of corruption, or to increasing accountability for such acts. The answers to the various sub-questions below support this assessment. SUB-QUESTIONS ABOUT CIVIL SERVICE AND HUMAN RESOURCE MANAGEMENT Sub-question 1.1.1. Has the SFP strengthened its leadership in civil service reform? Has it gained influence among Executive Branch institutions? The SFP is the ministry-level, central office responsible for issuing policies and regulations related to the civil service and the management and development of human resources in the public sector. It is CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 15 also responsible for providing assistance to human resource departments in Executive Branch institutions responsible for implementing policies and regulations. The SFP also is responsible for providing training to human resource professionals in the public sector through the National Institute for the Paraguayan Public Administration (Instituto Nacional de la Administración Pública de Paraguay, INAPP). Between 2014 and 2018, the SFP made great strides developing and implementing two new flagship initiatives supporting civil service reform – the SICCA and the IGP. The DGP provided substantial support for these initiatives as well as for several other SFP priorities, including developing a training curricula for INAPP courses, establishing and implementing a cooperation program with the SFP’s Chilean counterpart, and implementing a digital signature system. Altogether, the launch of these initiatives and DGP support have contributed to strengthening the SFP’s leadership in civil service reform. At the same time, the SFP has improved its positioning as the go-to organization for supporting the implementation of reforms through training and targeted technical assistance. The SFP ultimately is responsible for certifying that the civil service career models of Executive Branch institutions comply with SICCA regulations and Civil Service Law (No. 1626/2000), which further increases its clout within the Executive Branch. Indeed, the SFP has been recognized as a key contributor to the recent improvements in institutional quality of the GoP civil service system, as measured by the Civil Service Development Index of the IADB.8 Sub-question 1.1.2. How effectively and extensively is the SICCA being implemented? Between 2014 and 2018, the SFP made great strides in the development of the SICCA. As described in the background section above, the SICCA is an ambitious and comprehensive tool that seeks to improve the practice and transparency of human resource management from recruitment to termination of employment, including performance assessments, training, and compensation. DGP provided substantial support for the development of software for the various SICCA modules and associated manuals and training materials, for the adoption, dissemination, and use of the portal through communication campaigns, and for managing day-to-day system operations, including troubleshooting. Importantly, the DGP supported the drafting of regulations to implement SICCA including the amendment of the Civil Service Law (No. 1626/2000), which ultimately was not submitted to Congress for consideration. The uptake of the SICCA’s hiring module (modulo de concursabilidad) has been successful. DGP performance monitoring reports indicate a healthy increase in the share of contracts that are issued following competitive selection process across Executive Branch institutions. Though reporting for Year 4 of the program is still pending as of the writing of this report, it is expected that this trend should continue. In addition, DGP progress reports provide evidence indicating that the number of complaints about the portal from applicants has decreased, suggesting the hiring module is operating effectively. This being said, as is discussed below, extensive use of the SICCA hiring module should not be equated with progress in merit-based hiring. NORC’s evaluation team was not able to gather sufficient information to make a definitive assessment of the implementation of other SICCA modules. The limited evidence suggests that ministries with more advanced human resource management systems in place before SICCA, such as the Ministry of Finance (Ministerio de Hacienda, MoF), have been quicker to adapt their systems to 8 Dumas, Víctor. Diagnóstico institucional del servicio civil en América Latina: Paraguay (2017). Nota técnica No. 1345. Washington, DC: Inter-American Development Bank, 2017. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 16 adopt SICCA standards. While compliance with SICCA standards in these institutions is positive, the upgrade of preexisting systems is likely to result in the commitment of resources to stronger, more developed institutions, raising concerns about equity. USAID and the DGP should proceed with caution in this regard. Sub-question 1.1.3. How effectively and extensively is IGP being implemented? The IGP is a self-assessment tool available to institutions to monitor their performance on human resource management. The tool is useful in identifying weaknesses and developing action plans to address those weaknesses. The SFP developed the IGP as part of a wider effort to improve the institutional quality of the GoP civil service system. Between 2014 and 2016, 45 out of a total of 115 eligible institutions conducted IGP assessments,9 nine of which received DGP support. Within DGP-supported institutions, the IGP diagnoses were used to create action plans to guide future DGP interventions. In particular, IGP assessments diagnosed that one key weakness in human resource management was the lack of job profiles, which are considered essential to civil service reforms. Job profiles provide information related to each position’s roles and responsibilities, as well as its experience, qualifications, and skills requirements. DGP efforts in this front focused on the Ministry of Justice (Ministerio de Justicia, MINJUS), the MoF, the MOPC, the MSPBS, the SFP itself, the Environment Secretariat (Secretaría del Ambiente, SEAM), and the National Secretariat for Disabled People’s Human Rights (Secretaría Nacional por los Derechos Humanos de las Personas con Discapacidad, SENADIS). The efforts produced substantial progress, but as discussed below, job profiles do not seem to be available for all positions, at least in the institutions that were included in the FGDs.10 A new round of IGP assessments was planned to start in 2017. With the support of DGP, the SFP developed a digital version of the IGP called IGP Digital, and associated support materials. As mentioned in Section 1, this initiative sought to simplify the complexity of the assessment and increase its uptake and sustainability. As of the writing of this report, however, only four institutions had carried out the assessment with DGP support, including the SFP itself. Sub-question 1.1.4. To what extent are hiring and promotions increasingly based on merit? Do employees perceive increasing fairness in hiring and promotions? Are jobs advertised well ahead of time? Are jobs advertised widely enough to assure fair access by the relevant pool of applicants? Is the average number of years of schooling of public employees increasing over time? When it comes to hiring and recruitment, the employee survey results suggest that some progress has been made. As shown in Figure 1, the percentage of respondents mentioning the Paraguay Concursa portal as a medium used to publicize vacancies in their institutions significantly and substantially increased from baseline to endline in all institutions – and in all DGP-supported institutions combined. Increases range from a whopping 130 percent in the CSJ (increased from 11 to 25 percent) to 29 percent in the MOPC (increased from 44 to 56 percent). 9 Dumas, Víctor. 2017. Diagnóstico institucional del servicio civil en América Latina: Paraguay (2017). Nota técnica No. 1345. Washington, DC: Inter-American Development Bank, 2017, p. 35. 10 Interestingly, most FGD participants indicated not being familiar with the IGP. One respondent from MOPC mentioned that they were aware of SFP’s work on IGP. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 17 Figure 1. Percentage of respondents mentioning the Paraguay Concursa portal as medium to publicize vacancies in their institutions Item wording: Q30. If you have access to the employee recruitment policy, what means are used to publicize vacancies in your institution? Select all that apply. Response options: Institution website; Paraguay Concursa portal; Social media (Facebook, Twitter, etc.); National newspapers; Boards, bulletin or internal gazette; Email; They never publicize vacancies; Don’t know. Base sizes: CSJ baseline, 203; CSJ endline, 188; MEC baseline, 186; MEC endline, 190; MOPC baseline, 183; MOPC endline, 192; MSPBS baseline, 181; MSPBS endline, 184; All DGP baseline, 753; All DGP endline, 754; MIC baseline, 177; MIC endline, 185. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. However, increased usage and visibility of the Paraguay Concursa portal has not been accompanied by similar improvements in perceptions about merit-based hiring. The percentage of respondents who believe that personnel are recruited according to their ability to fulfill job duties “in all cases” and “in most cases” increased significantly only in the MSPBS, from 29 to 39 percent.11 Notably, this percentage decreased significantly in the MIC, from 37 to 26 percent. There was virtually no change in the other ministries. Similarly, the percentage of respondents who believe that announcements about vacancies are “always” or “most of the time” publicized so that everyone who wants to apply can do so with equal opportunity increased significantly only in the CSJ, from 11 to 18 percent.12 Again, this percentage decreased significantly in the MIC, from 55 to 39 percent. The percentage increased from 21 to 26 percent in the MEC, but this increase was not statistically significant. There was virtually no change in the other two ministries. Lastly, the percentage of respondents who believe that the process for filling vacant positions carefully evaluates each candidate’s work history 11 Item wording: Q28. In the institution in which I work, personnel are recruited according to the strict needs of the service, that is, for their ability to fulfill assigned job duties: Response options: In all cases; In most cases; In some cases; Almost never; Never; Don’t know. 12 Item wording: Q29. In my institution, announcements about vacancies are publicized so that everyone who wants to apply for employment can do so with equal opportunity of being hired: Response options: Always; Most of the time; Sometimes; Almost never; Never; Don’t know. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 18 “in all cases” or “in most cases” increased significantly only in the MSPBS, from 28 to 44 percent.13 There were non-significant decreases in the CSJ, MEC, and MIC, and a non-significant increase in the MOPC. The results of an item asking about merit-based hiring through open competitions in the public administration of Paraguay in the last year are quite interesting. While the previous items asked employees about their own institutions, this one asked about public administration of Paraguay in general and is therefore likely to reflect changes in overall mood about this issue. As shown in Figure 2, the percentage of respondents indicating merit-based hiring “increased some” or “increased a lot” in the last year decreased significantly and substantially in all institutions. Decreases range from 46 percent in the MSPBS – percentage decreased from 50 to 27 percent – to 31 percent in the MEC – percentage decreased from 45 to 31 percent. It is likely that this finding reflects the inauguration of the new administration that occurred in August 2018 and the customary changes in personnel that accompany transitions, rather than a reaction to an across-the-board deterioration of hiring practices. Figure 2. Percentage of respondents indicating merit-based hiring in the public administration of Paraguay increased in the last year Item wording: Q32. In the past year, merit-based hiring through open competitions in the public administration of Paraguay. Response options: Has decreased a lot; Has decreased some; Has stayed the same; Has increased some; Has increased a lot; Don’t know. Base size: CSJ baseline, 205; CSJ endline, 188; MEC baseline, 188; MEC endline, 191; MOPC baseline, 183; MOPC endline, 192; MSPBS baseline, 183; MSPBS endline, 184; All DGP baseline, 759; All DGP endline, 755; MIC baseline, 176; MIC endline, 184. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. In conclusion, survey results indicate that while important progress has been made in using the Paraguay Concursa portal to publicize vacancies, this progress does not guarantee an improvement 13 Item wording: Q31. In my institution, the process for filling vacant positions carefully evaluates each preselected candidate’s work history guaranteeing that those best suited for the position are chosen: Response options: In all cases; In most cases; In some cases; Almost never; Never; Do not have information about the recruitment policy. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 19 in perceptions about merit-based, fair, and transparent hiring. Evidence from FGDs supports this conclusion. Most participants indicated they are familiar with the Paraguay Concursa portal and believe its use is a step in the right direction. However, they also believed that hiring was still far from being merit-based. Some participants indicated that job descriptions for vacancies advertised via the portal could be tailored to favor specific applicants; others mentioned that, regardless of management’s increase use of the portal, those in charge of hiring still had plenty of discretion and could favor applicants with family or political connections. Another interesting theme that emerged from FGDs is that most participants took issue with how political appointee positions (cargos de confianza) were filled. Some respondents indicated they were disappointed for not having been considered for a promotion to such a position. Since political appointee positions are a necessity, and most aspects pertaining to them are regulated by law, it would be important for future efforts in the area of civil service reforms to include a communication component explaining this reality to employees and the citizenry. When it comes to promotions, survey results suggest that progress has been modest and localized in the MEC and, to a lesser extent, in the MSPBS. The percentage of respondents agreeing with the statement “I am familiar with the rules that govern promotions in the public administration”14 increased significantly only in the MEC, from 53 to 67 percent. The percentage of respondents agreeing with the statement “In my institution, performance evaluations are used as an input for decisions about promotions” increased significantly in the MEC and the MSPBS. The percentage increased from 24 to 34 percent in the MEC and from 30 to 44 percent in the MSPBS. Observing improvements in the MEC and MSPBS, and not in other institutions, is not surprising. These two ministries employ a large number of frontline service providers whose careers are highly regulated. Any improvements in promotion processes and regulations are likely to be perceived by these employees. More generally, and perhaps more importantly, there were no improvements in perceptions regarding meritocracy in promotions in the public administration as a whole. The percentage of respondents agreeing with the statement “In the public administration of Paraguay, promotions are decided by careful evaluation of professional merit, without personal privileges” did not change significantly in any institution. Evidence from FGDs is in line with this finding, with most participants believing that promotions hinged on family or political connections. A MEC participant mentioned promotions “are mostly political because of the ‘confianza’ issue.” A MIC participant said: “if one doesn’t complain, then one gets nothing; that is the reality. And if you complain, you have to have political ties or ‘work’ a politician.” Finally, with regard to whether the average number of years of schooling of public employees is increasing over time, the survey results show that this seems to be the case in the MSPSB and the MIC. In the MSPSB, the percentage of employees reporting an educational attainment of incomplete licenciatura or lower decreased from 46 to 32 percent, while the percentage of respondents reporting an educational attainment of licenciatura or higher increased from 48 to 62 percent. In the MIC, the percentage of employees reporting an educational attainment of incomplete licenciatura or lower decreased from 47 to 36 percent, while the percentage of respondents reporting an educational attainment of licenciatura or higher increased from 53 to 64 percent. 14 For this and all other agree-disagree items, the response options are: Strongly agree; Somewhat agree; Neither agree nor disagree; Somewhat disagree; Strongly disagree; Don’t know. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 20 Sub-question 1.1.5. How regularly are evaluations conducted of employees’ performance? Have all employees been included in some type of evaluation of competency? Are performance evaluations based on set, specified criteria? The survey evidence related to these questions is mixed. There were improvements when it comes to the uptake of periodic performance evaluations. Figure 3 reports the percentage of respondents indicating that they receive a performance evaluation at least once a year. The percentage increased significantly in the MEC, from 24 to 71 percent, the MOPC, from 61 to 82 percent, and MIC, from 56 to 73 percent. Note, however, that the percentage decreased significantly in the MSPBS, from 85 to 62 percent. Figure 3. Percentage of respondents indicating they receive a formal, written performance evaluation at least once a year Item wording: Q20. How frequently do your supervisors conduct a formal, written evaluation of your performance? Response options: Once a year or more; Less than once a year; Irregularly; I have never had one; Don’t know; Other (Specify). (Other responses indicating a frequency of more than once a year are included in the percentage.) Base size: CSJ baseline, 202; CSJ endline, 186; MEC baseline, 185; MEC endline, 191; MOPC baseline, 183; MOPC endline, 193; MSPBS baseline, 182; MSPBS endline, 189; All DGP baseline, 752; All DGP endline, 759; MIC baseline, 176; MIC endline, 188. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. Similar to what happened in the case of hiring, where increased usage and visibility of the Paraguay Concursa portal was not accompanied by similar improvements in perceptions about merit-based hiring, increased uptake of periodic performance evaluations has not been accompanied by similar improvements in perceptions about the overall fairness of performance evaluations. The percentage of respondents agreeing with the statement “In my institution, the criteria used to evaluate the performance of each employee are objective and reliable” improved significantly only in the MEC, from 38 to 51 percent. The findings about hiring and performance evaluations together indicate that it is easier to produce improvements that involve the uptake of formal requirements – e.g., “from now on, all vacancies have to be announced via Paraguay Concursa” or “from now on, all employees have to have a CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 21 performance evaluation every year” – than changes in norms and underlying practices – e.g., “from now on, job descriptions cannot favor specific applicants” or “from now on, performance evaluations cannot be carried out in a subjective and unpredictable manner.” Sub-question 1.1.6. Do employees feel their hard work and initiative are recognized and rewarded? Do employees see that negative attitudes towards work obligations and laziness are sanctioned? Survey results suggest no gains were made when it comes to rewarding positive attitudes and behaviors. The percentage of respondents agreeing with the statement “In my division, dedication at work is recognized” did not change significantly from baseline to endline. Similarly, there were no significant changes in the percentage of respondents agreeing with the related statement “Creativity and innovation are rewarded in my institution.” When it comes to sanctioning negatives attitudes, the MSPBS is the only institution that seems to have made gains. Therein, the percentage of respondents agreeing with the statement “Lack of dedication or having a bad attitude is sanctioned” increased significantly and substantially, from 12 to 32 percent – a 266 percent increase! FGD participants generally felt that performance evaluations often recognized hard work and initiative, but recognition did not translate into salary increases or promotions. In terms of sanctioning negative attitudes and behaviors, including absenteeism, many participants referenced Law 1626 establishing the rules that govern public employment. These participants indicated that, in spite of the law, practices for punishment and redress are not consistent across or within ministries. Sub-question 1.1.7. Do employees perceive progress in the institutionalization of a merit system in their agency? Survey results suggest progress in this regard is restricted to the MEC. Two survey items are relevant. Q13 asks respondents to state their level of agreement with the statement “The public servants in my institution possess the relevant knowledge and abilities to carry out their work.” Q14 asks respondents to state their level of agreement with the statement “The employees at my institution who have assignments and responsibilities similar to mine receive a total compensation (salary plus all additional payments) that is similar to my total compensation.” The percentage of respondents agreeing with the statements increased significantly in the MEC, from 58 to 68 percent for Q13 and from 30 to 42 percent for Q14. No other changes are statistically significant. Participants in FGDs mostly expressed negative views about this issue, with few participants perceiving improvements. Several participants blamed the turnover associated with political transitions and changes in political appointee positions for this state of affairs. They mentioned that newcomers often lack the required qualifications and that changes in personnel usually translate into changes in work dynamics and policy reversals. A quote from an MOPC participant illustrates the position of those in the minority perceiving improvements: “What I see, is that there is at least people that are more qualified, more knowledgeable, better prepared.” Sub-question 1.1.8. Are training actions conducted on a regular basis in all public units? To what extent are training opportunities available? Survey results suggest no progress has been made in this regard, with the CSJ likely having experienced a deterioration. Two survey items are relevant. Q26 asks respondents to state their level of agreement with the statement “I have been given the opportunity to improve my abilities in my institution through adequate training.” Q27 asks respondents to state their level of agreement CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 22 with the statement “Training resources are allocated fairly and according to service needs.” The percentage of respondents agreeing with the statements decreased significantly in the CSJ, from 54 to 44 percent in the case of both items.15 No other changes are statistically significant. Evidence from FGDs about training is mixed. Some participants indicated that there are training opportunities, while others said there are not. Most importantly, perhaps, the majority of those indicating that there are opportunities thought that there is no equal opportunity to access them. For example, one CSJ participant mentioned that training courses are only available for “the privileged,” while another said: “there is always someone that lets their friend know [of a training opportunity], and they receive this benefit.” An MOPC participant said training opportunities are allocated according to the “personal preferences of those in charge.” Sub-question 1.1.9. What do employees think about their direct supervisors and their institution’s leadership? Have employees perceptions about their supervisors and their institution’s leadership improved? Again, survey results suggest no progress has been made in this regard, with the CSJ likely having experienced a deterioration. Three survey items are relevant. Q10 ask respondents to state their level of agreement with the statement “I have confidence in my immediate supervisor’s capacity.” There were no statistically significant changes in the percentage of respondents agreeing with the statement. Figure 4 presents results for item Q11 gauging respondents’ perceptions about the highest authorities’ commitment to the proper operation of the institution. The percentage of respondents agreeing with the statement decreased in all institutions but the MIC; the decrease is statistically significant only for the CSJ and all DGP-supported institutions combined. This across-the-board deterioration might also reflect the reactions of employees to the inauguration of the new administration that occurred in August 2018 and the corresponding changes in high-level authorities and political appointee positions in the Executive Branch. In the CSJ, deterioration might reflect the changes in the Supreme Court’s composition that happened since the resignation of Justice Pucheta in April 2018 and the impeachment proceedings against two other justices, including the removal from office of one member in November 2018. 15 It is possible that this deterioration is related to the frustrated relaunch of the Centro Internacional de Estudios Judiciales (CIEJ). The DGP continued to support trainings in the CSJ, but the process was fraught with difficulties related to internal conflicts between the CIEJ and the Planning Directorate over leadership in this area. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 23 Figure 4. Percentage of respondents agreeing with the statement “The highest authorities are committed to the proper operation of the institution” Item wording: Q11. The highest authorities are committed to the proper operation of the institution. Response options: Strongly agree; Somewhat agree; Neither agree nor disagree; Somewhat disagree; Strongly disagree; Don’t know. Base size: CSJ baseline, 204; CSJ endline, 185; MEC baseline, 187; MEC endline, 191; MOPC baseline, 181; MOPC endline, 195; MSPBS baseline, 183; MSPBS endline, 187; All DGP baseline, 755; All DGP endline, 758; MIC baseline, 176; MIC endline, 189. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. Lastly, Q4 asks respondents to state their level of agreement with the statement “The objectives and priorities of the institution where I work are well-defined and well-communicated.” This question taps into employees views about leadership, albeit indirectly. There were no statistically significant changes in the percentage of respondents agreeing with the statement. Sub-question 1.1.10. Do employees believe their job duties and responsibilities are clearly defined and communicated to them? Do employees feel they are compensated according to their duties and responsibilities? The survey evidence related to these questions suggests no substantial improvements were made. First, the percentage of respondents agreeing with the statement “My job responsibilities were explained clearly to me” did not change significantly in any institution. In turn, the percentage of respondents agreeing with the statement “The work that I perform corresponds with my position’s formal (written) job description” decreased significantly in the MSPBS, from 82 to 74 percent. Lastly, the percentage of respondents agreeing with the statement “The total compensation (salary plus all additional payments) that I receive for my position corresponds with the duties that I have been assigned” increased significantly in the MEC, from 41 to 53 percent. Evidence from FGDs suggests there is substantive variation across and within ministries when it comes to the standardization of job descriptions or roles. At one extreme, participants at the MOPC agreed that there were written job descriptions specifying roles and responsibilities. At the opposite extreme, participants at the CSJ consistently indicated there are no such descriptions and CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 24 the roles and responsibilities of a given position generally are not clear. Results from the other ministries lie somewhere in between, with participants indicating that job descriptions are available for some positions but not others. Sub-question 1.1.11. Are the reforms associated with greater job satisfaction and lower absenteeism among federal employees? First, with regard to absenteeism, the survey evidence suggests that this undesirable practice might have increased between baseline and endline. Figure 5 presents results pertaining to an item asking respondents about the prevalence of absenteeism in the workplace. As the figure shows, the percentage of respondents indicating that absenteeism is almost non-existent or low in their institutions decreased in all institutions, though not significantly. However, the decrease is significant for all DGP-supported institutions combined. Figure 5. Percentage of respondents indicating absenteeism in their institutions is almost non-existent or low Item wording: Q19. Absenteeism in my workplace is: Response options: Almost non-existent; Low; Moderate; High; Very high; Don’t know. Base size: CSJ baseline, 205; CSJ endline, 187; MEC baseline, 187; MEC endline, 191; MOPC baseline, 183; MOPC endline, 192; MSPBS baseline, 184; MSPBS endline, 189; All DGP baseline, 759; All DGP endline, 759; MIC baseline, 176; MIC endline, 188. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. Several items tap into the overarching construct of job satisfaction. First, the following items ask respondents to state their level of agreement with statements about concrete drivers of job satisfaction: • Q6. All of the resources I need to do my job well are available to me. • Q7. The people with whom I work generally collaborate to get tasks done. • Q8. My workplace is comfortable and pleasant. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 25 • Q9. In the decisions that involve me personally, my opinion is heard and taken into account. The percentage of respondents agreeing with the statement changed significant only for Q6. It increased significantly in all DGP-supported institutions combined, from 53 to 57 percent, and decreased significantly in the MIC, from 69 to 57 percent. This finding is counter to the conventional wisdom on this issue, which is that ministries are lacking in resources. Evidence from FGDs is consistent with this wisdom, however. Participants generally agreed that they do not have the resources necessary to do their job. The resources most commonly cited as lacking were office supplies, including paper, toner, pens, and photocopiers, among other supplies. They also noted weak technological infrastructure (e.g., no internet access or slow connection speeds) and not enough computers for staff use. Participants in two FGDs – MIC junior staff and MSPBS junior staff – stated that their ministries did not provide transportation for work-related travel to other ministry sites in Asunción or the interior. Only in one FGD – MIC management staff – did participants say they generally had the resources necessary to do their job. Three other items explore job satisfaction from a more general angle. The following items ask respondents to state their level of agreement with a statement: • Q5. My work contributes to the achievement of my institution’s objectives. • Q25. My background and experience are utilized in my division. • Q56. I would advise young people to enter the Paraguayan public administration. The percentage of respondents agreeing with the statement did not increase significantly in any case. However, the percentage decreased significantly in a few instances. In the CSJ, the percentage decreased for Q5 and Q56, from 98 to 93 percent and from 74 to 60 percent, respectively. In the MSPBS, the percentage decreased for Q5 and Q25, from 95 to 79 percent and from 81 to 65 percent, respectively. Finally, item Q55 directly asks respondents to state their level of satisfaction with their work.16 Figure 6 presents the percentage of respondents who stated they were very satisfied or satisfied. As the figure shows, employee satisfaction changed significantly only in the CSJ, and for the worse. 16 Item wording: Q55. In general, how satisfied are you with you work? Response options: Very satisfied; Satisfied; Neither satisfied nor unsatisfied; Unsatisfied; Very unsatisfied. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 26 Figure 6. Percentage of respondents indicating they are satisfied with their work, overall Item wording: Q55. In general, how satisfied are you with you work? Response options: Very satisfied; Satisfied; Neither satisfied nor unsatisfied; Unsatisfied; Very unsatisfied Base size: CSJ baseline, 203; CSJ endline, 188; MEC baseline, 187; MEC endline, 188; MOPC baseline, 182; MOPC endline, 194; MSPBS baseline, 181; MSPBS endline, 183; All DGP baseline, 753; All DGP endline, 753; MIC baseline, 175; MIC endline, 182. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. Taken as a whole, the survey evidence pertaining to job satisfaction suggests that morale in the CSJ deteriorated from baseline to endline. This is likely related to the changes in the composition of the courts and impeachment proceedings mentioned under sub-question 1.1.9., which are outside of the control of the program. There is no evidence to suggest that this decline is in any way related to the program or that the program could have done anything to become aware of or prevent this situation. To summarize, available evidence indicates that the reforms have not contributed to greater job satisfaction or lower absenteeism. SUB-QUESTIONS ABOUT INTERNAL CONTROLS Sub-question 1.2.1. Are internal control systems operating effectively? Executive Branch institutions and the CSJ made important progress toward complying with MECIP regulations and improving their internal control systems with support from the DGP. In selected institutions, the DGP supported ministry-wide risk assessments and then provided technical assistance to map processes with the highest risks and improve controls. The DGP also contributed to the effective operation of internal controls by strengthening internal control and forensic audit units in selected institutions; supporting the development of a new audit manual for the GoP and a new electronic system for AGPE to track audits in Executive Branch institutions; and assisting CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 27 institutions in drafting and adopting new codes of conduct. Key informants indicated being very satisfied with the assistance received from the DGP in this area. In addition, two items in the employee survey directly addressed this evaluation question. The items ask respondent to state their level of agreement with the following statements: • Q50. In my workplace, internal controls are effective in improving efficiency and efficacy. • Q51. The audits conducted by the Internal Auditor are effective in limiting fraud. Results suggests no substantial improvements were made; from baseline to endline, there were no significant changes in the percentage of respondents agreeing with the statement in Q50. In turn, the percentage of respondents agreeing with the statement in Q51 decreased significantly in the CSJ, from 51 to 40 percent. Sub-question 1.2.2. How widely has the MECIP been adopted? Overall, Executive Branch institutions seem to have made good progress in implementing the MECIP framework. The AGPE monitors the implementation of the MECIP through institutions’ self￾assessments of internal controls, which are supposed to be conducted every six months. Four subsystems – strategic control, management control, evaluation control, and design phase – are assessed using a scale that ranges from inadequate to adequate. The assessment results in an overall score – known as MECIP rating – for the institution as a whole. While these scores do not measure the implementation of MECIP 2015 itself, the simplified version of the MECIP that the CGR launched in 2016 with DGP support, the scores are nonetheless helpful to track progress in the improvement of internal controls. Table 1 compares the overall scores for the second semester of 2016, the period immediately after the 2015-version of the MECIP was launched, and the second semester of 2018. The table also includes the number of institutions reporting, which is another indicator of adoption. As the table shows, while the scores have improved substantially in this period, the number of reporting institutions decreased. One potential explanation for the decrease in reporting institutions is lack of clarity regarding the ‘valid’ version of the MECIP. This issue is discussed in greater detail under Evaluation Question 3. Table 1. Summary of progress on MECIP implementation in Executive Branch institutions First Semester 2012 Second Semester 2018 Percentage Adequate 40.51 63.01 Percentage Satisfactory 26.58 21.92 Percentage Deficient 16.46 12.33 Percentage Inadequate 16.46 2.74 Number of reporting institutions 79 73 Source: Portal MECIP, AGPE reports on Executive Branch institutions’ self-assessments of MECIP implementation; available at: http://eval.mecip.gov.py/Mecip/reportes/listareportes. The DGP supported the implementation of MECIP 2015 through a two-pronged strategy. On the one hand, it assisted the CGR in developing the necessary policies and regulations as well as CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 28 supporting manuals and training courses. On the other hand, it worked with the CSJ and selected institutions of the Executive Branch to directly support MECIP implementation. While this assistance achieved its overall objectives, evidence from KIIs offers a more nuanced view of MECIP 2015 and its implementation process. This evidence is discussed under Evaluation Question 3. Describing MECIP 2015 implementation, government counterparts explained that it typically began with a ministry-wide risk assessment to identify key processes that corresponded to area(s) of highest risk (i.e., in terms of volume of transactions, monetary value, etc.) or those involving service provision to the citizenry (funciones misionales). Given DGP’s limited resources, processes corresponding to areas of high risk or high visibility to the public were prioritized to receive technical assistance. Once selected, processes were mapped to identify potential operational gaps and documented to promote transparency and consistency in application over time. Integrated into the MECIP is a self-evaluation mechanism through which agencies report publicly on their progress on implementation. The survey evidence suggests MEC employees perceived some improvements related to internal controls. Therein, the percentage of respondents agreeing with two statements related to processes and procedures increased significantly. Agreement with “The processes that my division is responsible for executing are well-specified and regulated” increased from 61 to 71 percent, while agreement with “In my work, I understand the current processes and procedures well” increased from 84 to 91 percent. No other changes are statistically significant. Sub-question 1.2.3. To what extent are public sector employees aware of the new systems of internal controls? Evidence from FGDs indicates employees are generally aware of MECIP 2015 and its implementation process. Like key informants whose views are discussed under Evaluation Question 3, the opinions of employees about MECIP 2015 are mixed. Some focus group respondents spoke positively of MECIP 2015 and cited examples where the framework had helped to improve processes in their ministries. For example, one MEC employee described how the MECIP was used to standardize processes related to how student school supplies and lunches were distributed. Another MEC respondent said: “in certain areas [the MECIP] really works and helps to facilitate processes, that is, [it] establishes certain criteria from which to build on...” Similarly, an employee in the MIC stated that the MECIP helped “tighten things up where we were weak before… [yielding] better results from earlier periods when we didn’t have that type of control.” Still others felt the MECIP had not been fully actualized in their ministries or that it had not been sufficiently adapted for the GoP. For example, a CSJ employee said that, even though MECIP implementation began in 2008, “it still isn’t seen in the day to day.” Similarly, one respondent from the MOPC said that the “MECIP hasn’t really landed; [it’s] still up in the air…” Another MOPC respondent said the “MECIP isn’t really applicable to us...it didn’t work in Colombia either. [MECIP] needs to be transformed and modified for Paraguay.” Others said the MECIP did not truly improve operations or help ministries gain efficiencies because implementation emphasized the documentation of existing processes as opposed to the evaluation of alternatives. As one MIC respondent put it, “for MECIP implementation, what we did was write down what we were doing in the day to day, so we didn’t modify anything. We just wrote down what we were already doing.” Another MIC employee said the MECIP “created more work because we had to fill out more paperwork.” CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 29 Sub-question 1.2.4. How often do employees witness unethical behavior or acts of corruption? How often are employees pressured to break rules and engage in acts of corruption? How often are employees forced to unreasonably/illegally redirect benefits to some specific entity/individual? Which are the most common forms of corruption encountered? The survey evidence related to these questions is mixed. Overall, it indicates that no substantial improvements took place. A set of items asks about ethical and rule-abiding behavior in the workplace. Item Q40 asks respondents for their views about the extent to which their co-workers abide by regulations.17 Item Q35 asks respondents for their views about the extent to which their divisions operate in compliance with current regulations.18 In addition, the items listed below ask respondents to state their level of agreement with a statement: • Q36. My immediate supervisors promote public ethics and rule-abidingness in all administrative acts. • Q41. The highest authorities of the institution in which I work support my supervisors’ effort to maintain an ethical and rule-abiding environment. The percentage of respondents giving positive responses to these items did not increase significantly in any case. However, the percentage decreased significantly in a few instances. In the CSJ, the percentage decreased for Q40 and Q41, from 74 to 59 percent and from 74 to 54 percent, respectively. In the MSPBS, the percentage decreased for Q35, from 81 to 69 percent. Another set of items asks employees about unethical or corrupt behavior in the workplace. Item Q44 asks respondents for their views about the extent to which users have to provide extra payments (bribes), gifts, or favors to get services in their institutions.19 Item Q45 asks respondents for their views about the extent to which they feel pressured to break rules to benefit certain people.20 Lastly, item Q49 asks respondents for their views on the extent to which corruption and administrative fraud have increased or decreased in the past year.21 The only positive and significant change occurred in the MEC, for Q44. Therein, the percentage of respondents responding that users have to provide extra payments (bribes), gifts, or favors to get services in their institutions “never” or “rarely”, increased from 66 to 75 percent. The only negative and significant change occurred in the MIC for Q49. Therein, the percentage of respondents saying that corruption and administrative fraud in the past year “have decreased significantly” or “have decreased somewhat” decreased from 61 to 49 percent. In addition, as mentioned earlier, the employee survey includes three survey experiments or truth elicitation protocols that provide significant degrees of anonymity to respondents and encourage them to provide candid responses to questions related to compliance with regulations and unethical 17 Item wording: Q40. My co-workers abide by regulations. Response options: In all cases; In most cases; In some cases; Almost never; Never; Don’t know. 18 Item wording: Q35. Overall, my division operates in strict compliance with current regulations. Response options: In all cases; In most cases; In some cases; Almost never; Never; Don’t know. 19 Item wording: Q44. In the institution where I work, it is necessary to provide extra payments (bribes), gifts, or favors to receive the services that should be provided. Response options: Never; Rarely; Sometimes; Frequently; Always; Don’t know. 20 Item wording: Q45. I feel pressured to break rules to benefit certain people from both inside and outside of my institution. Response options: Never; Rarely; Sometimes; Frequently; Always; Don’t know. 21 Item wording: Q49. In my opinion, corruption and administrative fraud in the past year: Response options: Have decreased significantly; Have decreased somewhat; Have stayed the same; Have increased somewhat; Have increased significantly; Don’t know. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 30 behavior. The first experiment is a list (or count) protocol in which respondents are asked to write the number of statements in a list that are true – rather than whether each situation is true. One set of respondents is presented with a list of innocuous situations and another set is presented with this same list plus one sensitive – i.e., ethically-questionable or illegal – situation: “Additional payments (cash bribes), gifts, or favors have been accepted in exchange for exception to regulations, unwarranted advantages, or privileges.” By comparing the responses across groups, one can estimate the prevalence of the ethically questionable or illegal situation. Table 2 reports the estimation results. Entries are estimates for the percentage of respondents indicating they have observed the acceptance of additional payments (cash bribes), gifts, or favors. The percentage decreased significantly in the CSJ, from 44 to 12 percent. The percentage increased significantly in the MOPC, from 24 to 53 percent. While the other estimated changes are somewhat substantial, they are not statistically significant.22 Table 2. Percentage of respondents indicating they have observed “Additional payments (cash bribes), gifts, or favors have been accepted in exchange for exceptions to regulations, unwarranted advantages, or privileges” (list protocol) CSJ MEC MOPC MSPBS All DGP MIC % N % N % N % N % N % N Baseline 44.0 152 46.7 120 24.3 121 55.0 129 43.0 518 24.5 118 Endline 12.3 147 17.4 128 53.4 140 38.3 145 30.0 559 38.6 137 Difference -31.7* -29.3 29.1* -16.8 -13.0 14.1 Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. The second experiment is a dichotomous randomized-response protocol in which respondents first have to flip a coin, and then respond to an innocuous or a sensitive question yes-no depending on the outcome of the coin flip. Only the respondent knows the outcome of the roll and which question she or he is answering, encouraging truthful responses. A subset of respondents also separately answers the innocuous question, which allows for estimating the true responses to the sensitive question. Two variations of this protocol were included. The first variation is used to gauge the percentage of respondents who report that their colleagues “behave in an ethical manner.” Those who respond “yes” to this question indicate they have not observed their colleagues act in an unethical manner. The second variation is used to gauge the percentage of respondents who report that they themselves “apply and abide by all established processes and regulations at work.” Those who respond “yes” to this question indicate they have not disregarded established processes and regulations. Table 3 reports the estimation results. Entries are estimates for the percentage of respondents indicating that they have only observed ethical behavior. As the table shows, estimated percentages for the second variant are higher than 100 percent in most instances. The most likely explanation for this is respondents’ unwillingness to self-incriminate even in a “protected” setting like the one 22 In statistical hypothesis testing, a result is statistically significant when the likelihood that it occurred given a null hypothesis is very low. The ability to detect statistically significant results depends on several factors, including the size of the change, the amount of information available to conduct the test (e.g., the number of cases, N), and the amount of variability in estimates. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 31 afforded by the dichotomous randomized-response protocol.23 If present, this bias against self￾incrimination should equally affect survey responses at baseline and endline, which suggests comparisons across waves are valid. Table 3. Percentage of respondents indicating that they have only observed ethical behavior (dichotomous randomized-response protocol) CSJ MEC MOPC MSPBS All DGP MIC % N % N % N % N % N % N “My colleagues behave in an ethical manner” Baseline 79.9 146 67.6 128 76.5 123 59.6 131 70.8 528 95.8 119 Endline 79.4 156 63.5 150 69.7 152 70.1 155 70.6 613 79.2 150 Difference -0.5 -4.0 -6.8 10.5 0.2 -16.6 “I apply and abide by all established processes and regulations at work” Baseline 106.8 147 95.5 126 101. 6 123 122.4 127 106.5 523 117.5 120 Endline 100.5 157 103.6 151 97.4 152 91.5 151 98.1 611 112.0 149 Difference -6.3 8.0 -4.2 -30.9* -8.4 -5.5 Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. The only significant change is for the second variation in the MSPBS. Therein the estimated percentage of respondents indicating that they apply and abide by all established processes and regulations decreased from 122 to 92 percent. A benign interpretation of this finding is that MSPBS employees might have been more willing to deviate from established processes and regulations in light of the formalization and standardization process that should have followed the adoption of the MECIP. The third experiment is a numeric randomized-response protocol in which respondents add the outcome of a die roll to their true response on a scale indicating the frequency of occurrence of various sensitive situations. True responses are estimated taking advantage of the known expected distribution of a die roll. This protocol was included to estimate the prevalence of the following practices: hide or falsify information; obstruct or delay paperwork and procedures; use public property for private benefit; divert public monies; use influence peddling to secure privileges; appoint relatives or co-partisans without merit; authorize trips, trainings, or bonuses arbitrarily or unjustly; neglect institutional property; and steal institutional property. Estimated responses to this protocol are highly anomalous and therefore not reported. Issues with question wording, the high complexity of the task, and a small sample size combined are the likely explanation. Sub-question 1.2.5. Is openly unethical/corrupt behavior left unpunished? 23 More specifically, seeking to eliminate any potential for self-incrimination, a large share of respondents answered “yes” regardless of the outcome of the coin flip (i.e., even if they were answering the innocuous question). This distorted the percentage of “yes” responses to the innocuous question in the dichotomous randomized-response protocol, which in turn, made it differ from the percentage of “yes” responses provided by the subset of respondents who separately answered the innocuous question. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 32 The survey evidence suggests no substantial improvements were made in this regard. From baseline to endline, there were no significant changes in the percentage of respondents indicating that, in their departments, serious administrative faults or mistakes are sanctioned according to regulations “in all cases” and “in most cases.”24 Sub-question 1.2.6. Do reasonable incentives exist to report the commitment of acts of corruption to authorities? While the survey did not directly ask about incentives to report acts of corruption, it includes one question asking respondents about fear of retaliation or sanctions following complaints. Specifically, it asks respondents to indicate their level of agreement with the statement “In my institution, I can file complaints without fear of retaliation or sanctions.” The only significant change in the percentage of respondents agreeing with this statement is in the CSJ, where it declined from 49 to 36. This survey finding contrasts somehow with the evidence from FGDs, where there was consistent awareness about anonymous and confidential mechanisms to file complaints among participants. Awareness about such mechanisms does not imply willingness to use or trust in them, of course, but participants did not share any concerns. It is worth noting that FGD participants also mentioned that such mechanisms were available to members of the public. Altogether, the evidence about questions related to unethical behavior and acts of corruption and the reporting and sanctioning of such acts indicates that no substantial, across-the-board improvements took place between baseline and endline. While some specific results suggest improvements, several others indicate setbacks. Even if the employee surveys did not document improvements in behaviors and practices among employees, internal control reforms are a step in the right direction.25 EVALUATION QUESTION 2 ARE INPUTS PROVIDED BY THE DGP CONTRIBUTING TO IDENTIFIABLE CHANGES IN PROCESSES AND SYSTEMS CAPACITY IN ITS FOCUS AREAS? HOW EFFECTIVE HAS THE PROGRAM BEEN IN TRANSFERRING SKILLS, KNOWLEDGE, AND CAPACITY TO GOP STAFF? The DGP has contributed to identifiable changes in processes and systems in the areas of civil service and human resource management and internal controls. However, the evaluation team has concerns about the sustainability of achievements and about the appropriateness of efforts given the program’s overall objectives. Concerns about sustainability and appropriateness of efforts are discussed under Evaluation Questions 3 and 4. With regard to program effectiveness in transferring skills, knowledge, and capacity to GoP staff, the evaluation team’s assessment is that DGP used an adequate mix of approaches to achieve this goal, but cannot directly assess effectiveness. The answers to the various sub-questions below support this assessment. Sub-question 2.1: Do partner institutions feel inputs provided by DGP have changed the way they function and improve their performance? 24 Item wording: Q43. In my department, serious administrative faults or mistakes are sanctioned according to regulations. Response options: In all cases; In most cases; In some cases; Almost never; Never; Don’t know. 25 Administrative records could provide evidence of improvements in reporting or administrative sanctioning of unethical behaviors and acts of corruption. The evaluation team did not review these records. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 33 To answer this question it is important to differentiate between regulatory institutions that are responsible for developing policies and regulations – i.e., the SFP in the area of civil service and human resource management, and the AGPE and the CGR in the area of internal controls – from ‘executing’ institutions that are responsible for implementing these policies and regulations – i.e., ministries and other institutions. It is also important to differentiate between Executive Branch institutions and the CSJ given that the latter is not under the purview of the SFP or the AGPE. Most key informants at regulatory institutions felt that DGP support had been crucial for enabling them to advance reform initiatives, as well as to strengthen their own capacities. DGP support was also important to enable them to perform their day-to-day activities, which include assisting and assessing the progress of executing institutions in implementing policies and regulations. Some informants also expressed extreme appreciation for concrete financial support from the DGP – e.g., cover materials and supplies for training courses, contract consultancies for updating an institution’s strategic plan, and pay for expenses associated with leaning exchanges. Some of the most important contributions of the DGP to regulatory institutions have been related to the development of software and information systems to support reforms along with the associated institutional capacity to maintain and operate them. These software and systems allow for greater efficiency by automating or speeding up tasks and by promoting standardization. The previously mentioned SICCA software, in general, and the Paraguay Concursa Portal, in particular, are noteworthy examples in the area of civil service and human resource management. Portal MECIP and the Integrated System of the Office of the Auditor General for the Executive Branch (Sistema Integrado de la Auditoría General del Poder Ejecutivo, SIAGPE) are noteworthy in the area of internal controls. Portal MECIP allows Executive Branch institutions to conduct internal control self￾assessments and submit them to the AGPE, which is responsible for monitoring progress. The SIAGPE is an online portal that internal auditors in Executive Branch institutions use to document their planning, execution, and follow-up activities, which are then reviewed by AGPE. Moving on to executing institutions, most key informants felt DGP inputs had been crucial for making progress in the implementation of policies and regulations. In the area of civil service reforms and human resource management, DGP-supported work on developing job profiles, harmonizing pay scales, and implementing competitive recruitment processes were seen as great advances. Informants view these initiatives as laying the foundations for future and more systemic progress towards building a merit-based civil service. In the area of internal controls, informants acknowledged that DGP support was crucial for the implementation of MECIP 2015. However, as discussed under Evaluation Question 3, several key informants expressed reservations about the MECIP itself. Importantly, some informants at executing institutions also expressed a desire for more DGP support, noting that the assistance received was insufficient to achieve full implementation of new policies and regulations. Here it is important to point out that the DGP’s main strategy for providing technical assistance consisted in pairing consultants with partner staff to work together on a subset of the components required to fully implement a given reform. The goal was to create ‘improvement teams’ within GoP counterparts that could advance implementation without DGP support. This strategy is discussed again under Evaluation Question 4 focused on sustainability. Informants at the MEC and MSPBS also spoke highly of DGP support received to fully overhaul specific processes involving service provision to the citizenry (funciones misionales). The processes supported at MEC include the distribution of school kits, the centralized registration of students, and the centralized registration of professional degrees. At the MSPSBS, the processes supported include CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 34 the scheduling of surgeries at the Central Orthopedic Trauma Hospital and the purchasing of medicines and other supplies. At the CSJ, one of DGP’s major achievements was to contribute to the establishment of the CAJ.26 This entity is responsible for internal management and operations at the CSJ, including asset management, administration and finances, budgeting, internal controls, and planning. The aim of this reform is to relieve Supreme Court Justices from having to directly handle internal management and operations – and allowing them to focus on the administration of justice – and to create a better environment for the improvement of internal management and operations themselves. The DGP supported the selection of the CAJ justices and the initial set-up and strengthening of the CAJ. The DGP also supported various initiatives to improve internal management and operations at the CSJ, including the implementation of the MECIP and of competitive selection processes for administrative and other support staff – i.e., staff not included in the judicial track (carrera judicial). Key informants at the CSJ expressed similar opinions to those of informants at executing institutions of the Executive Branch. They felt DGP inputs had been crucial for setting up the CAJ and making progress in improving internal management and operations; at the same time, they expressed a desire for more DGP support. While a step in the right direction, the establishment of the CAJ sparked internal power struggles that made DGP’s work with CSJ more complex. These are discussed in greater detail under Evaluation Question 3. Sub-question 2.2: Has training provided by DGP improved or changed knowledge, attitudes, and practices of partner staff? As mentioned above, the DGP’s main strategy for providing technical assistance consisted in pairing consultants with partner staff to work together on a subset of the components required to fully implement a given reform. The DGP also supported classroom-style trainings, participation in exchange programs, and other training initiatives, provided directly via the program or through partner institutions, such as the CGR’s MECIP trainings. The evaluation team believes these mix of approaches is adequate, but does not have enough evidence to reach a definitive assessment on whether training provided by DGP improved or changed the knowledge, attitudes, and practices of partner staff. The few interviews that we conducted with key informants that worked directly with DGP – as opposed to higher-level staff responsible for supervising those working directly with the program – suggest that training was effective. This is especially the case in the area of internal controls, where MECIP implementation exposed partner staff to new knowledge and skills. In addition, interviews with DGP consultants and subcontractors providing direct technical assistance to GoP institutions suggest that partner staff were generally receptive to the assistance provided and eager to learn from it. EVALUATION QUESTION 3 WHAT DEFICIENCIES CURRENTLY EXIST IN THE IMPLEMENTATION OF THE PROGRAM’S OBJECTIVES AND HOW CAN THESE BE REMEDIED? The DGP’s implementation strategy in the areas of civil service and human resource management and internal control systems was not fully adequate to achieve the program’s overarching objectives. 26 It is worth noting that USAID/Paraguay’s support to the establishment of the CAJ predated the DGP. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 35 The DGP’s implementation strategy was two-pronged: support regulatory institutions in the development and implementation of policies and regulations; and support executing institutions in implementing these policies and regulations. – i.e., ministries and other institutions. In supporting regulatory institutions, the program approached its objectives from a mostly technical perspective, not fully taking into account various political economy factors that made implementation inherently complex, especially in the area of internal controls and in work pertaining to the Judiciary Branch. In particular, the DGP was not fully prepared to anticipate and deal with competing interests and preferences within and across institutions, and with over-time change in said interests and preferences. The answers to the various sub-questions below support this assessment. In supporting implementing institutions, the strategy focused on achieving several outputs in multiple institutions, sometimes losing sight of overall objectives. For example, and in line with what has been discussed under Evaluation Question 1, adoption of Paraguay Concursa is necessary but not sufficient to achieve merit-based hiring. Similarly, requiring periodic personnel evaluations is only the first step toward establishing a system that rewards merit and performance. As discussed under Evaluation Questions 4 and 5, and in the Recommendations, supporting the full implementation of reforms in a handful of executing institutions would allow for achieving various outputs that, together, contribute to making substantial progress toward overall objectives. Sub-question 3.1: What are the causes and factors that have hindered the achievement of the DGP’s Performance Management Plan (PMP) targets and progress? The DGP’s PMP includes two indicators that are directly related to the main focus areas under evaluation. Both indicators pertain to the DGP’s sub-purpose 1.1, human resource management system improved. Indicator 4, “Percentage of civil servants of the (Executive Branch) accessing merit-based civil service,” is operationalized as the number of positions filled through competitive selection divided by the total number of positions filled in DGP-assisted Executive Branch institutions in a given year. Indicator 5, “Percentage of civil servants of the Judiciary contracted through competitive selection processes,” is constructed in an analogous way but with a focus on the Judicial Branch. Targets for Indicator 4 were established for FY 2014 through FY 2018, and were reached in FYs 2014 and 2016. In FY 2015, the target was not reached and, as of the writing of this report, the DGP did not have the information necessary to calculate the indicator in FYs 2017 and 2018. In turn, targets for Indicator 5 were established for FY 2015 through 2018. Targets were not reached in FYs 2015 and 2016, and, as of the writing of this report, the DGP did not have the information necessary to calculate the indicator in FYs 2017 and 2018. Even if incomplete, this information, together with additional evidence, indicates that the DGP was substantially more effective at achieving its targets in the area of human resource management in the Executive Branch than in the Judiciary Branch. The explanation for this disparity lies, for the most part, in political economy factors.27 Power within Executive Branch institutions, the SFP included, is highly concentrated in Ministers. This makes implementing changes easier, once leadership is convinced of the need for it. The DGP’s strategy of working with the SFP to develop and enact new policies and regulations (i.e., the SICCA system and its hiring module), and with specific institutions 27 The discussion that follows is a very simplified picture of the political economy surrounding civil service and human resource management reforms in the Executive and the Judiciary Branches. Carrying out a full political economy analysis is outside the scope of this evaluation. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 36 to support the implementation of such regulations, allowed for making substantial progress in this setting. In the Judiciary Branch, all important decisions pertaining to policies and regulations as well as their implementation are made by the CSJ itself, a collegiate body of nine members. Gathering enough support for carrying out any substantial reform, let alone finding time for the Supreme Court Justices to discuss it, is a daunting task. Given this context, the evaluation team considers that, even if the achievements at the Judiciary are limited when compared to targets, they are quite remarkable and likely to be sustainable. The achievements are also likely to enable future progress, though one should not expect progress to be linear or quick given institutional complexities. Sub-question 3.2: Are any program objectives lagging in results compared to the others? Why? In addition to objectives related to the CSJ, which have been addressed above, the MECIP 2015 reform in the area of internal controls is lagging. As discussed under Sub-question 1.2.2, Executive Branch institutions seem to be making important improvements in internal controls. However, the prospects for the full implementation of MECIP 2015 are uncertain due to the AGPE’s lack of support for the new model. These issues are discussed below. Sub-question 3.3: Have any program themes, cross-cutting elements, or special projects caused controversy or backlash in the country? The updating of the MECIP and the subsequent implementation of this updated version caused controversy, ultimately resulting in a status quo where the prospects for its full adoption and implementation are uncertain. While a number of institutions have made clear progress in the implementation of MECIP 2015, it seems unlikely that these institutions will be able to continue deepening the reform without external support. Similarly, it seems unlikely that all eligible institutions will fully take up the reform. As mentioned by various informants, the process to implement MECIP 2015 is labor intensive and tedious; overworked regular staff do not have an incentive to move the process forward absent external pressure or a clear mandate from upper￾level officials. Moreover, and perhaps most importantly, limited uptake of the reform – within and across institutions – will likely not produce substantially better results than MECIP 2008 when it comes to the effectiveness, probity, and transparency of public administration. At the beginning of the DGP in 2014, the highest authorities at the two regulatory institutions in the area of internal controls, the AGPE and the CGR, were in agreement about the need to update the MECIP. The version of the MECIP valid at the time had been launched in 2008, with the support of USAID, but implementation had not advanced as originally planned. Following a diagnosis of the factors explaining the lack of progress, the two entities decided to update the MECIP with the support of the DGP. Thereafter, two developments chipped away at the original reform impetus. On the one hand, the AGPE authorities’ decided to withdraw from the updating process because of their disagreement with the new model the DGP put forward. On the other hand, turnover in CGR leadership slowed the pace of progress. Despite the DGP’s efforts at encouraging dialogue, the two AGPE and the CGR were not able to come to a consensus regarding the convenience of MECIP 2015. The CGR eventually launched MECIP 2015 by issuing a resolution in May 2016, but without full endorsement from the AGPE, internal control departments at Executive Branch institutions are likely to be slower to take up the new version of the MECIP. Also, since the adoption of MECIP 2015 is not strictly mandatory, it is possible that some institutions might choose to continue operating under the old system. In light of CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 37 this state of affairs, which was likely foreseeable in 2015, the DGP would have been wise to exercise extra caution, reengaging the AGPE in the reform process and working politically to overcome the impasse. In 2018, USAID decided to end its support in the implementation of MECIP in the CSJ, the MEC, the MINJUS, the MoF, the MPOC, the MSPBS, and the SEAM. The developments just outlined and budget constraints are the main reasons behind USAID’s decision. Key informants offered conflicting opinions about MECIP 2015. Some informants spoke highly of MECIP 2015; most of these informants were discussing MECIP 2015 as the tool that was used to completely overhaul specific processes involving service provision. These DGP-supported initiatives were mentioned under Sub-question 2.1. Other informants offered a more nuanced perspective. They reported initially being very enthusiastic about MECIP 2015, hoping to also use it to overhaul internal processes, but subsequently becoming critical during actual implementation. They said that, in their experience, MECIP 2015 implementation had not involved overhauling or reengineering processes; rather, it involved the documentation and formalization of the very inefficient processes that were already in place. Another set of informants believe that MECIP 2015 was in principle a good model, but that it was too complex for public institutions to implement effectively without external support. Importantly, key informants within the CGR also expressed reservations of two types about MECIP 2015. First, uncertainty about the choice of model for MECIP 2015, given that other internal control models could have been chosen. Second, uncertainty about the decision to update MECIP, given that some progress had already been made in the adoption of MECIP 2008. EVALUATION QUESTION 4 IS THE CURRENT SCALE AND SCOPE OF THE PROGRAM APPROPRIATE TO SUPPORT SUSTAINABILITY? SHOULD CERTAIN ASPECTS OR COMPONENTS BE PRIORITIZED OR LIMITED? The DGP scale and scope within regulatory institutions is on target, though sustainability is a concern given the limited enforceability and potential for reversibility of the reforms. The DGP scale and scope within executing institutions responsible for implementing these policies and regulations was not adequate. Given available resources, by choosing to support multiple executing institutions on several lines of work, the DGP was implemented in a diluted, fragmented fashion, achieving various outputs, but not managing to make substantial progress toward the DGP’s ultimate objectives. The DGP’s intervention strategy in the areas of civil service and human resource management and internal controls consisted in working with regulatory institutions to overhaul rules and regulations and then supporting their implementation in specific executing institutions. As stated above, the DGP scale and scope within regulatory institutions was on target, though sustainability is a concern for a number of reasons. First, as mentioned earlier, although civil service reforms related to the SICCA, in general, and to the competitive selection process, in particular, were mostly implemented through presidential decrees, they are still more easily reversible than a law. Likewise, the new version of the MECIP was implemented through a resolution of the CGR, which is less enforceable than a presidential decree or law. Outright cancellation or reversal of the reforms does not seem likely in the foreseeable future, though the fact remains that this could be more easily achieved than if the reforms had been adopted via laws. Indeed, the DGP aimed to advance reforms via laws rather than via decrees or resolutions. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 38 However, this was not achieved due to circumstances that are not within the control of the program. Enforcing compliance with decree- and resolution-level rules and regulations is also a concern. Specifically, it is difficult to set hard deadlines for the adoption of such rules and take action against non-compliers. Moreover, implementing institutions are less likely to be willing to commit substantial resources to implement resolution-level rules and regulations, slowing down reform uptake. Lastly, it is also important to mention that without political commitment at the highest level, executing institutions might not have access to the resources needed to implement mandates. An important contextual factor threatening the sustainability of achievements in the area of civil service and human resource management is the SFP’s institutional weakness. Compared to the AGPE and the CGR, the SFP is severely short-staffed and budget-constrained, as well as heavily dependent on donor support. With support from USAID and other donors, and due also to its former leadership, the SFP has undergone a process of institutional strengthening. As mentioned earlier, this has allowed it to position itself as a key contributor to reform implementation efforts. Without donor support, and absent increased funding and political commitment from the GoP to further strengthen the SFP, it is possible that DGP-supported achievements will be short-lived. Preventing these threats to sustainability, or addressing them after they have materialized, requires political craft rather than technical fixes. Having been conceived of as mostly technical in nature, the DGP was and continues to be ill-equipped to deal with these issues. If working politically is not an option, future interventions should focus on facilitating the implementation of reforms that already have broad and sustained political support. This was the case of DGP interventions in the areas of access to information, open government, and transparency. Moving on to executing institutions, as stated above, the DGP was implemented in a fragmented fashion, achieving various outputs, but not managing to make substantial progress toward the DGP’s ultimate objectives. DGP’s main strategy for providing technical assistance consisted in pairing consultants with partner staff to work together on a subset of the components required to implement a given reform. The goal was to create ‘improvement teams’ within GoP counterparts that could continue advancing reform implementation once support from the DGP came to an end, ensuring sustainability. The appropriateness of this model is questionable on at least three grounds. First, changes in the political priorities of a ministry’s highest authorities constrain the ability of lower-level personnel to advance reforms. Second, high levels of personnel turnaround often require team re-composition and sometimes even result in team disintegration. Third, working with staff who already has a full workload makes it less likely that they will be able to sustain the reform impetus without external support. The evaluation team offers some recommendations about how to address this deficiency under Evaluation Question 5. An exception to this assessment is the work in the Judiciary Branch. As mentioned earlier, therein progress has been disappointing given the DGP’s performance targets. However limited, progress in the CSJ has been the result of long-standing support that was not only technical but also political in nature. Because of these, achievements are likely to be sustainable and enable future progress. Sub-question 4.1: Has the DGP identified key GoP counterparts and is the assistance provided to these on target, based on priority needs? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 39 At the strategy level, the DGP appropriately focused its work in regulatory institutions, but as mentioned above, the DGP chose to work with too many implementing institutions given its level of resources. At an operational level, work within each of the supported institutions was on target and generally driven by priority needs identified in consultation with counterparts. Issues at the strategy level have been addressed above, so the focus here is on the operational level. Key informants indicated that work in a given institution kicked off with high-level meetings to decide on focus areas. The process of identifying specific targets for technical assistance involved additional high-level meetings, with input from diagnoses often informing decisions. For example, the 2014-2016 IGP measurement provided inputs to identify priority areas in the area of human resource management. It is worth noting that some informants indicated that one weakness of this process was that these meetings did not always involve the counterpart staff that worked directly with DGP consultants. Because of this, some decisions were perceived as arbitrary, undermining willingness to work with the DGP. Ensuring that personnel at this level also buy into improvement processes should be a priority. Sub-question 4.2: Is the scope of DGP assistance flexible to respond to changing priorities or emerging problems within GoP counterparts? The scope of DGP assistance was flexible to respond to changing conditions at an operational level. Given its scale and scope, as well as CEAMSO’s ability to quickly mobilize consultants and other resources, the DGP was able to effectively respond to time-sensitive, short-term demands or needs from operational-level counterparts. The program was also effective at adapting and making corrections in their approach to working with operational-level counterparts. The ongoing communication discussed under the next sub-question was key to achieve this. At a strategic level, CEAMSO was not fully prepared to predict or promptly react to changing political circumstances, and there does not seem to have been a strategy in place between CEAMSO and USAID to navigate the GoP’s dynamic political environment. CEAMSO’s prevailing assumption seems to have been that any challenges to DGP implementation would be related to technical or resource availability issues once initial buy-in from high-level counterpart authorities had been secured. Faced with changes in political will, CEAMSO might have been too quick to accept such changes as limitations and turn to USAID for assistance. Similar to CEAMSO, USAID conceived of the DGP as mostly technical in nature. Sub-question 4.3: Did the DGP and GoP counterparts have regular “how are we doing” meetings to discuss changing priorities or redundant activities? The DGP and its GoP counterparts had regular “how are we doing” sessions at both the component and activity levels. Key informants indicated that these sessions were very effective and found CEAMSO staff, consultants, and subcontractors to be interested in feedback and addressing implementation issues that might arise. Informants also indicated that communication was constant, also taking place on an as-needed basis outside regularly scheduled meetings. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 40 EVALUATION QUESTION 5 WHAT ARE THE GOOD PRACTICES EMERGING FROM THE PROGRAM? ARE THERE SUCCESSES THAT COULD BE REPLICATED FOR FUTURE PROJECTS? There are three good practices emerging from the DGP that future projects should try to replicate. The first one is the DGP’s decision to completely overhaul specific processes involving service provision to the citizenry (funciones misionales) under the MECIP – e.g., the scheduling of surgeries at the MSPBS’s Hospital de Trauma or the centralized registry of professional degrees at the MEC. Achieving results in areas that are highly visible to the citizenry in a relatively short period of time is an asset for any program that seeks to advance complex reforms. In line with this good practice, future programs with limited resources should focus on supporting full reform implementation in a handful of executing institutions. This model also avoids the pitfalls of partially implementing reforms in several institutions and relying on improvement teams to finalize implementation. Given bureaucracies general tendency to resist change and the resource constraints of the Paraguayan bureaucracy, completely overhauling a very limited number of institutions is likely to be more effective than initiating reforms in several ones. Such a focus would undoubtedly expose a project to greater risks. At the same, however, it would afford the project greater leverage vis-à￾vis its government counterparts and a chance to demonstrate effectiveness in achieving higher-order results. The second good practice is the use of diagnoses to identify specific targets for technical assistance. As mentioned above, the 2014-2016 IGP measurement provided inputs to identify priority areas in the area of human resource management. Likewise, within the MECIP 2015 framework, process overhaul was informed by risk assessments and evaluations. Future projects should embrace evidence-based decision-making in their intervention strategies as well as seek to promote it among counterpart staff. The third good practice is promoting the use periodic assessments as a tool to advance reform implementation. As mentioned earlier, the DGP supported the development of Portal MECIP at the AGPE, which allows Executive Branch institutions to conduct internal control self-assessments and submit them to the AGPE for monitoring. Periodic performance assessments not only enhanced the AGPE’s monitoring abilities but also created reputation-related pressures on low-performing institutions to step up compliance efforts. Portal MECIP could have been an even more effective tool had the DGP been able to get the AGPE’s full support for MECIP 2015. In a similar vein, the DGP supported the development of IGP Digital with the aim of lowering the cost of IGP assessments and establishing a sustainable model to monitor progress in improving human resource management systems. While it is not yet clear if this initiative will be successful, it is a step in the right direction. In light of the Portal MECIP experience, the evaluation team recommends that the DGP works with the SFP to make the results of IGP Digital assessments available to the public. Although outside the scope of this evaluation, the DGP support to access to information, open government, and transparency reforms must be acknowledged here. The DGP experience shows that donor-sponsored technical assistance can help achieve substantive change in areas where broad and sustained political support is already in place. Future projects with a technical focus could replicate this experience. Future projects open to working politically – and to failing in achieving results – could help create the political support required for technical projects to thrive. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 41 Sub-question 5.1: What type of assistance has produced the most improvement? Where and why? Overall, assistance in the area of civil service and human resource management has produced the most improvement. The DGP’s support to the Paraguay Concursa portal has been particularly effective, contributing to its adoption among executing institutions as well as visibility among citizens. Several factors contributed to this success, including the following. First, Paraguay Concursa offered a relatively-easy-to-use technical solution to conduct selection processes that are compliant with SFP standards. Second, adoption of the portal was not met with great resistance because it does not require that those conducting selection processes give up all the discretion that they have in selecting staff. For example, they can continue writing terms of reference that give preference to certain candidates or giving high weight to subjective factors such as personal interviews in selection processes. Reducing this discretion is arguably one of the hardest changes needed to achieve merit￾based, fair, and transparent hiring. The development and adoption of Paraguay Concursa should be seen as a first step in the right direction upon which future efforts can build. Sub-question 5.2: What type of assistance has produced the least progress? Where and why? The assistance related to MECIP simplification is the one that has produced the least progress. At the output level, this assistance reached its targets; the DGP successfully completed its work in most of the specific processes within supported institutions that it set out to simplify. However, it is not clear that this assistance has substantially contributed to the overarching objective of improving internal controls – and reducing corruption – in the GoP. Moreover, the institutional framework governing internal control systems is less clear than it was in the past, with the prospects for the implementation of MECIP 2015 not being better than those of MECIP 2008. As discussed earlier, the main reason for this state of affairs is political in nature. Sub-question 5.3: What are the emerging lessons that should be kept in mind for future projects? In line with the preceding discussion, there are three main emerging lessons from the DGP: • Any project that aims to effect substantial changes in the way the public sector operates is inherently political. Pursuing projects that contribute to ideation, development, and implementation of reforms has high political risks. Such projects require commitment to high￾order objectives as well as ability to cope with setbacks, non-linear progress, and openness to the possibility of failure. They also require political craft to identify opportunities and risks as well as likely reform allies and blockers, and to develop strategies to advance reforms. When engaging in substantial political work is not possible, projects should focus on facilitating the implementation of reforms that already have broad and sustained political support. • A project that aims to effect substantial changes in the way the public sector operates should prioritize depth versus breadth. This would entail supporting the ideation, development, and regulatory processes of a very limited number of reforms, and their full implementation in a handful of institutions. • A project that aims to effect substantial changes in the way the public sector operates should use evidence both as a tool to diagnose problems and determine focus areas or specific interventions and as a resource to advance project objectives. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 42 GENDER NORC’s evaluation team also explored issues related to gender equality in civil service in the KIIs, employee survey, and FGDs. We asked some key informants about the work they had done with the DGP on gender issues. The support informants highlighted was training and development of gender inclusion plans/policies. Employee survey responses corresponding to two dimensions of gender equality – equal opportunity in hiring, promotion, and pay and perceptions of occurrences of sexual harassment – are represented below in Figures 7 and 8, respectively. With regard to equal opportunity in hiring, promotion, and pay, the employee survey results suggest little progress has been made. In fact, there are statistically significant decreases among women respondents who indicated agreement in the CSJ – from 69 to 46 percent – and the MEC – from 75 to 62 percent. Similarly, there is a statistically significant decrease among women across DGP – from 69 to 55 percent. Among men, there are no statistically significant changes in the percentage of respondents agreeing with the statement. Figure 7. Percentage of respondents indicating that women and men have the same hiring, promotion, and pay opportunities in their institutions, by gender Item wording: Q53. In your institution, women and men have the same hiring, promotion, and pay opportunities. Response options: Institution website; Strongly agree; Somewhat agree; Neither agree nor disagree; Somewhat disagree; Strongly disagree; Don’t know. Base sizes women: CSJ baseline, 129; CSJ endline, 103; MEC baseline, 108; MEC endline, 102; MOPC baseline, 82; MOPC endline, 69; MSPBS baseline, 105; MSPBS endline, 92; All DGP baseline, 424; All DGP endline, 366; MIC baseline, 88; MIC endline, 89. Base sizes men: CSJ baseline, 72; CSJ endline, 84; MEC baseline, 76; MEC endline, 81; MOPC baseline, 98; MOPC endline, 122; MSPBS baseline, 73; MSPBS endline, 86; All DGP baseline, 319; All DGP endline, 373; MIC baseline, 83; MIC endline, 91. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. In turn, the employee survey results suggest no substantial progress has been made with regard to sexual harassment in the workplace. There are no statistically significant changes in the percentage of respondents agreeing with the statement for men or women in any ministry. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 43 Figure 8. Percentage of respondents indicating that sexual harassment is a common practice in their institutions, by gender Item wording: Q54. Sexual harassment is a common practice in your workplace. Response options: Institution website; Strongly agree; Somewhat agree; Neither agree nor disagree; Somewhat disagree; Strongly disagree; Don’t know. Base sizes women: CSJ baseline, 127; CSJ endline, 101; MEC baseline, 105; MEC endline, 101; MOPC baseline, 81; MOPC endline, 67; MSPBS baseline, 102; MSPBS endline, 91; All DGP baseline, 415; All DGP endline, 360; MIC baseline, 86; MIC endline, 90. Base sizes men: CSJ baseline, 71; CSJ endline, 84; MEC baseline, 77; MEC endline, 80; MOPC baseline, 91; MOPC endline, 123; MSPBS baseline, 72; MSPBS endline, 86; All DGP baseline, 311; All DGP endline, 373; MIC baseline, 84; MIC endline, 89. Note: Asterisks indicate difference is statistically significant; * p<0.05; ** p<0.01; *** p<0.001. Participants in FGDs provided generally positive views of gender equality. Across FGDs, there was a general feeling that there exists equality among men and women, specifically with regard to hiring and promotion. Many respondents cited examples of women in managerial positions and equal numbers of men and women staff in certain departments as evidence of a discrimination-free workplace. Still others pointed out that while there may be equal numbers of men and women among junior- and mid-level staff, diversity is not always reflected at the highest levels of management, particularly among political appointees (cargos de confianza) where personal biases may more directly impact selection. One respondent from the MIC stated, “equality between men and women hasn’t materialized…[inequality] continues to be a factor and obstacle that women face – certain disadvantages in relation to men, certain restrictions that women must overcome relative to men.”28 28 As mentioned in Section 3, focus groups were mixed-gender. While this added diversity of perspective to the discussion, it should be acknowledged again that it might have impacted women’s ability to speak candidly about certain topics, including experiences of gender discrimination or perceptions of gender equality in the workplace. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 44 5. CONCLUSIONS The DGP’s intervention strategy in the areas of civil service and human resource management and internal controls consisted of strengthening the capacity of regulatory institutions and providing them with key resources to overhaul rules and regulations. The DGP then supported the implementation of these rules and regulations in various executing institutions. Overall, the DGP has been successful in achieving various output-level results. However, these achievements constitute limited progress toward achieving the DGP’s overall objectives of strengthening the civil service with merit-based hiring and promotion and making internal controls more effective. In the area of civil service and human resource management, the DGP has contributed to strengthening the leadership of the SFP in civil service reform. The DGP provided key resources for the development of the SICCA system, including its hiring module and the Paraguay Concursa portal. The hiring module and Paraguay Concursa are key components of civil service reform, as they provide the tools necessary for GoP institutions to carry out competitive selection processes. The DGP also provide important support for carrying out IGP assessments in various institutions, which were then used to diagnose areas of improvement. The DGP then provided support to create an online tool to lower the cost of future assessments and establish a sustainable model to monitor progress in improving human resource management systems. These achievements are remarkable, but their sustainability is a concern. First, civil service reforms related to SICCA, in general, and competitive selection processes, in particular, were implemented through a decree of the SFP, which makes the reforms more easily reversible than if they had been adopted through a law. Moreover, the ability of the SFP to enforce compliance with decrees and resolutions is quite limited, making the full implementation of reforms less likely. An important contextual factor threatening the sustainability of achievements in the area of civil service and human resource management is the SFP’s institutional weakness. Second, the SFP is severely short-staffed and budget-constrained as well as heavily dependent on donor support. Without donor support, and absent increased funding and political commitment from the GoP to further strengthen the SFP, it is possible that DGP-supported achievements will be short-lived. In the area of internal controls, the DGP set out to support the AGPE and the CGR in their goal of achieving the full implementation of the MECIP. Following a diagnosis of the factors explaining the lack of progress in this direction, the two entities decided to update the MECIP. Thereafter, however, the AGPE withdrew its support to the process and leadership turnover in the CGR slowed the pace of progress. MECIP 2015 was eventually launched with a CGR resolution. Without endorsement from the AGPE, internal control departments at Executive Branch institutions are likely to be slower to take up the new version of the MECIP. Reversibility of reforms is also a concern, given MECIP 2015’s implementation by resolution and the lack of consensus, even within the CGR, about the appropriateness of this tool for GoP institutions. In 2018, USAID decided to end its assistance to the implementation of MECIP 2015 for these reasons. Moving on to executing institutions, the DGP achieved its overall targets in the implementation of the various initiatives. The issue is that DGP’s scope was too wide to be able to effect substantive changes in these institutions. Thus, while Paraguay Concursa has been widely adopted in DGP￾supported institutions, this does not seem to have been accompanied by reductions in the discretion of those making hiring decisions and corresponding improvements in employee perceptions about merit-based hiring. Similarly, while performance evaluations are now more common in DGP￾supported institutions, they have not resulted in an improvement in employees perceptions regarding their appropriateness and fairness. In the area of internal controls, MECIP 2015 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 45 implementation resulted in the overhauling of a handful of specific procedures, with most implementation efforts being directed at formalizing existing processes. The DGP’s main strategy for providing technical assistance in executing institutions was also problematic in the areas of civil service and internal controls. It consisted of pairing consultants with partner staff to work together on a subset of the components required to fully implement a given reform, with the aim of creating ‘improvement teams’ within GoP counterparts that could continue advancing reform implementation once support from the DGP came to an end. This was combined with a permanent dialogue with high-ranked public officers, including Ministers of the Executive Branch, Justices of the Supreme Court, the Controller, and in limited cases even legislators. It is unlikely that ‘improvement teams’ will be effective given that changes in the political priorities of a ministry’s highest authorities could constrain the ability of lower-level personnel to advance reforms; high levels of personnel turnaround could result in team re-composition or disintegration; and personnel generally lack time to devote to reform efforts given full workloads. Prioritizing depth over breadth by supporting full implementation of reforms in a handful of institutions – rather than initial implementation in several institutions – could be a more effective strategy. The overarching explanation for these results is twofold. First, the DGP lacked the focus needed to adequately support the implementation of reforms in executing institutions. Second, the DGP would have benefited from more political work to advance the ambitious reforms it set out to support. In the Judiciary, the DGP’s major achievement was to contribute to the establishment and strengthening of the CAJ, a long-standing goal of USAID/Paraguay and other donors. This reform aims to relieve Supreme Court Justices from having to directly handle internal management and operations and to create a better environment for the improvement of internal management and operations themselves. While the DGP’s results at the Judiciary in the areas of civil service and human resource management and internal controls have been quite limited when compared to targets, the establishment of the CAJ makes future progress more attainable. Still, one should not expect progress to be linear or quick given institutional complexities. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 46 6. RECOMMENDATIONS For years to come, strengthening the civil service with merit-based hiring and promotion and making internal controls more effective will continue to be important challenges for GoP institutions. USAID/Paraguay support will continue to be key to making progress. Building on the DGP experience, the evaluation team makes the following recommendations. First, USAID/Paraguay should consider continuing to support civil service and human resource management reforms in a sustainable manner and considering the political economy risks. Assuming the new administration and SFP leadership are committed to the reforms, sustainability hinges on a better-staffed and adequately-funded SFP that is able to support reform implementation as well as to maintain and upgrade SICCA and IGP systems and tools. When it comes to hiring, the adoption of Paraguay Concursa is only the beginning of a long and challenging reform process that will require developing job descriptions with explicit skill requirements for all positions, strengthening centralized and independent oversight of competitive selection processes, and consolidating a system for grievances against unfair hiring decisions. When it comes to promotions, uptake of periodic performance evaluations must be accompanied by the establishment of a system that fairly, predictably, and transparently rewards good performance and sanctions bad performance. Making substantive progress will also require working on the civil service compensation system as well as changing beliefs, norms, and underlying practices. Advancing these objectives requires a stronger SFP, which, in the short term, can only be achieved with donor support. It is important, however, that the GoP also contribute to efforts by increasing the funding available for this key institution. Second, USAID/Paraguay should consider continuing to support the CAJ and the CSJ in overhauling internal management systems in the Judiciary. A capable, strong CAJ is the necessary precondition for making substantial progress in this regard. Without additional support, the CAJ might not be able to reach its full potential given the challenges associated with its recent establishment. Third, future USAID/Paraguay support in the area of internal controls should only resume if the impasse between the AGPE and the CGR can be resolved. This will likely require substantial political work and has a high potential for failure. Given this, and assuming resources available to a future project will be limited, it would be sensible to prioritize supporting civil service and human resource management reform. Fourth, and finally, future USAID/Paraguay initiatives seeking to improve internal management systems in the GoP should highly consider prioritizing depth over breadth, supporting the ideation, development, and regulatory processes of a limited number of reforms, and their full implementation in a handful of institutions. They should also embrace their inherently-political nature and be better prepared to deal with the political challenges they are bound to face, including their implications on the program’s performance. This will require conducting political economy analyses to assess the situation in a given policy area, identifying political opportunities and risk as well as likely reform allies and blockers, and developing strategies to advance reforms. It will also require a commitment to high-order objectives rather than to achieving outputs, ability to cope with setbacks and non￾linear progress, and openness to the possibility of failure. The DGP offers a number of lessons about strategies that such initiatives could pursue to achieve their objectives: • Overhaul a few specific processes involving service provision to the citizenry (funciones misionales). Achieving results in areas that are highly visible to the citizenry in a relatively short period of time is an asset for any program that seeks to advance complex reforms. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 47 • Use diagnoses and other evidence as a political tool to build support for broad reforms as well as for making technical decisions regarding goals for assistance. • Use periodic assessments as a tool to advance reform implementation. Such assessments not only enhance regulatory institutions’ and the initiatives’ own ability to monitor progress but also create reputation-related incentives for executing institutions to take up reforms. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 48 BIBLIOGRAPHY Dumas, Víctor. “Diagnóstico institucional del servicio civil en América Latina: Paraguay (2017).” Nota técnica No. 1345. Washington, DC, USA. Inter-American Development Bank, 2017. GoP Auditor General of the Executive Branch, “MECIP Portal: Los reportes del MECIP.” Asunción, Paraguay. Accessed April 22, 2019. http://www.mecip.gov.py/mecip/?q=reportes. GoP Ministry of Industry and Commerce. “MECIP: 2015.” Asunción, Paraguay. Accessed April 22, 2019. http://www.mic.gov.py/mic/site/mecip/pdf/Manual%20Mecip-2015.pdf. GoP Secretary of Linguistic Policy, “Sobre MECIP.” Asunción, Paraguay. Accessed April 22, 2019. http://www.spl.gov.py/es/index.php/institucion/mecip. Organisation for Economic Co-Operation and Development (OECD). “Creación de una función pública profesional y eficiente,” in Estudios de la OCDE sobre Gobernanza Pública: Paraguay: Hacia un desarrollo nacional mediante una gobernanza pública integrada. Paris, France. OECD Publishing, 2018. Paraguay Democracy and Governance Program. Annual Work Plans, Annual Reports, Quarterly Reports, Performance Management Plan. United States Agency for International Development (USAID). “Glossary of Evaluation Terms.” Planning and Performance Management Unit, Office of the Director of U.S. Foreign Assistance, 2009. Available at: https://pdf.usaid.gov/pdf_docs/Pnado820.pdf. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 49 ANNEX A, STATEMENT OF WORK CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 50 STATEMENT OF WORK (SOW) USAID/PARAGUAY DEMOCRACY AND GOVERNANCE PROJECT: BASELINE STUDY, MID-TERM AND FINAL PERFORMANCE EVALUATION I. Purpose of the Evaluation In accordance with USAID’s Evaluation Policy, the purpose of this contract is to perform a baseline study, a mid-term and a final performance evaluation of USAID Paraguay Mission Democracy and Governance (D&G) Project. The Democracy and Governance Project will be funded at approximately $25,000,000, beginning in FY 2014 through FY 2018. The evaluation will assess the project at its mid-term junction completed approximately three years after the project initiation, as well as at the end of the five year project. The evaluation will focus on these parameters: 1. Baseline study: An analysis describing the situation prior to a development intervention, against which progress can be assessed or comparisons made 2. Project Performance – Assess the efficiency and effectiveness of the project in achieving the expected objectives, outcomes, and impacts; Assess whether there are heterogeneous project effects between males and females; 3. Appropriateness of Focus – Evaluate the efficacy of the primary project objectives given the country context, as well as USAID and GOP strategy and objectives to support increased government effectiveness; Assess the sustainability and institutionalization of the activities in GOP activities and processes; 4. Cost–effectiveness of the methodologies employed in the project towards achieving project goals and objectives; The evaluation will not measure project performance against a counterfactual or comparison group. The results of the baseline study will provide a thorough analysis of government effectiveness at the start of the project that will facilitate subsequent assessments of progress. The mid-term evaluation will be used by USAID to inform and enhance continued implementation of the D&G project. The final evaluation will inform future D&G programming in Paraguay; results will be shared widely with Government of Paraguay (GoP) personnel and other donors working on DG issues. II. Background Democracy in Paraguay has come far since Andrés Rodríguez launched his coup d’etat in 1989. The Constitution of 1992, which has not since been amended, enshrines pluralism, democracy, human rights, and the rule of law. Five presidential elections have been held, all of which have been free, if not totally fair. There have also been five municipal elections that have led to greater accountability in local government. Human rights are no longer abused as part of state policy, freedom of the press has flourished, new political parties have formed, and civil society has grown in the new spaces that have emerged. Yet, despite such advances in terms of democratic procedures and institutions, the quality of democracy remains low in terms of equal and informed participation, active citizenship and good governance. Paraguay continues to suffer from entrenched clientelism and corruption, political tools used as CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 51 skillfully in the transition to democracy as they were under the Stroessner dictatorship.29 Long-term and pervasive use of these tools has perversely affected the quality of public service delivery, rule of law, political representation and citizen participation. Although important advances have been made, particularly on civil service reforms, the public sector exhibits minimal strategic planning, poor public financial management, and a weak sense of professionalism. Rule of law remains inadequate to guarantee equality of access to justice for all, while the party quota system limits the progress in the independence, effectiveness, and accountability of the judicial system.30 Major political parties seek to capture the state and use their power over public resources to develop client-based relations with citizens, rather than those based on representative dimensions. Similarly, Paraguay’s close party list system provides limited incentives for elected legislative officials to engage and represent the needs of constituents. In Paraguay, economic inequality, which directly undermines political equality, is a core hindrance to democratic governance. 31 High levels of socioeconomic exclusion and inequality lead to lower levels of participation, and inhibit equal consideration of citizen preferences and needs. Although the vast majority of Paraguayans (85.6%) believe it is the State’s role to enact firm policies that reduce inequalities, citizens continue to demonstrate low levels of trust in core state institutions. As a result, unaddressed inequality generates a lower level of support for democracy in Paraguay.32 Socioeconomic exclusion also presents increased opportunities for elite capture, clientelism, and social conflict, all of which are prevalent in Paraguay. The 2008 election of Fernando Lugo was widely applauded as an important step towards elevated political competition and generated high expectations for the expansion of the benefits of democracy to a broader population that was historically excluded or marginalized. President Lugo’s tenure, indeed, installed numerous reforms and demonstrated advances in the fields of social welfare, governmental transparency, and civil liberties. However, some analyses conclude that he did not do enough to tackle corruption or address the great socioeconomic disparity in the country.33 The administration also demonstrated the fragility of coalition governments and continued power of dominant political parties. On June 22, 2012, Congress impeached President Lugo after an overwhelming majority in the Chamber of Deputies (76 out of 80 members) and in the Senate (39 of 45 members) voted in favor of his dismissal. While impeachment is recognized in the Paraguayan constitution, many questioned the express nature of the process and the legitimacy of the outcome. The immediate trigger of President Lugo’s removal is thought to be public dissatisfaction with the way the government handled an ambush between peasants and the police, but it is clear that prior political developments had already distanced the Government from parties represented in Congress.34 This disruption of an elected mandate re-emphasized the critical institutional gaps that continue to hinder true democratic consolidation and undercut opportunities for widespread socio-economic development in Paraguay. As prescribed by the Paraguayan constitution, the Vice-President Federico Franco took office for the transitional period until a new, elected administration takes office in August 2013. With the change in presidency came a whole new set of leaders within the Executive Branch, shifting priorities in some cases and maintaining the predecessor’s strategic focus in others. Important advances within the short transition period include the passing of several long-awaited legislative bills, such as the first-ever Personal Income Tax law, and demonstrated initiative towards countering corruption, 29 Transparency International Report on Paraguay, 2012. 30 USAID/Paraguay DRG Assessment update, 2012. 31 Gini coefficient 32 Lapop 2012 33 Countries at the Crossroads: Paraguay, Freedom House, 2011. 34 OAS report, 2012. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 52 through the creation of a new Anti-Corruption Ministry. Presidential and legislative elections are scheduled for April 2013. III. Project Overview In Paraguay, the culture of corruption and clientelism, as well as the government’s low professional/technical capacity, represent fundamental obstacles to democratic consolidation. The Executive and Judicial branches lack the capacity and incentives to represent and fulfill citizen needs; within these systems, there is a dearth of mechanisms to ensure transparency and accountability for holding government officials to account for poor performance. To overcome the democratic deficits in Paraguay, broad institutional reforms are necessary. In some areas, such as in the judiciary, these reforms may require constitutional and/or legislative modifications. USAID is not in a position to change many aspects of the institutional framework; eliminating the political quota system are beyond the scope of the project. As such, USAID will focus its efforts on the implementation of a series of interventions that target key government priorities and build on previous work conducted under the MCC Threshold program. In particular, USAID’s new DG project will seek to build the Executive and Judicial branches human and technical capacity to effectively respond to citizen needs and strengthen nascent transparency and accountability mechanisms. The development hypothesis underpinning this project is that by developing the government’s institutional capacity and strengthening accountability mechanisms, the Paraguayan State will more effectively govern. While the proximate objectives include enhanced transparency and accountability for public ministries and the judiciary as well as a more capable and responsive government, the distal objectives are to reduce corruption and ensure a strong institutional foundation for democracy in Paraguay. To achieve the project purpose of strengthened internal management and governance systems in select public institutions, the project will seek to: 1) Increase institutional capacity of select public institutions; 2) Strengthen accountability and anti-corruption efforts in select public institutions; and 3) in coordination with inter-agency counterparts and other international donors, advance the legal and policy framework for effective governance. Sub-purpose 1. Increased institutional capacity of select public institutions: Capacity is a key determinant of government performance. Under this sub purpose, the project will develop the institutional capacity of select public institutions to improve internal management, the management of public resources, and in specific cases, service to citizens. Capitalizing on the MCC Threshold programs’ significant investments, the project will prioritize management systems that are critical to efficient and effective government functioning and support the implementation of best practices to improve the government’s ability to serve all Paraguayans more effectively. Although these processes take time to install and institutionalize, advancing key reforms that strengthening human and organizational capacity will ultimately have an impact on the quality of public goods provision and citizen satisfaction with government. Concurrently, the project will also seize windows of opportunity that, while developing public sector capacity, also demonstrate effective and timely government response to high priority needs. These strategic interventions will help mobilize citizen support for public institutions in the short term as they make work to develop sustainable capacity to address citizen demands over the long term. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 53 Expected Outputs for Sub-purpose 1:  Civil service and HR management strengthened with merit-based selection, evaluation, and promotion processes implemented in select Executive Branch entities.  Judicial career strengthened with merit-based selection, evaluation, and promotion processes implemented.  Increased compliance with procurement regulations in select public institutions.  Government budgeting capacity strengthened.  Strategic interventions that improve public sector capacity to address high priority needs implemented.  Institutional capacity of the new People with Disabilities Ministry (SENADIS in Spanish) strengthened. Expected Outcomes for Sub-purpose 1:  Human resource management systems improved.  Public budgeting and procurement systems/processes strengthened.  Capacity to address strategic and high priority needs improved.  Public sector capacity to support and protect vulnerable populations increased. Expected Impacts for Sub-purpose 1:  Enhanced government effectiveness  More effective provision of the public goods of justice and basic services. Sub-purpose 2. Accountability and anti-corruption efforts strengthened in key public institutions. Responsive and functional governments require a plurality of effective accountability mechanisms across multiple levels and sectors to hold those who exercise power accountable at all levels of government. Accountability is important because it is an essential component of effective institutional functioning. Effective accountability mechanisms improve organization performance and enhance the efficiency of public goods delivery by restructuring incentives and altering information sets available to decision-makers (Agrawal and Ribot 2012). To effectively reduce corruption in Paraguay, therefore, the project will foster internal and external accountability. By strengthening public institutions’ internal capacity to monitor and sanction poor performance, government and judicial officials will face strong incentives to act and behave in a more ethical, efficient and effective manner to avoid punishment. Likewise, increased civil society oversight and the risk of negative public perceptions will motivate public officials and institutions to comply with Paraguayan law and regulations. Expected Outputs for Sub-purpose 2:  Internal control systems strengthened within the Executive Branch.  Effective internal control mechanisms reinforced in the judiciary.  Government transparency mechanisms consolidated.  Civil society awareness and oversight increased. Expected Outcomes for Sub-purpose 2:  Effective internal control systems strengthened.  Increased government transparency and external accountability CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 54 Expected Impacts for Sub-purpose 2:  Reduced corruption.  Enhanced government effectiveness  More effective provision of the public goods of justice and basic services. Sub-purpose 3. Legal and policy framework for effective governance advanced. While government effectiveness can be increased within the current institutional framework, many modifications are needed to enhance prospects for real reform and long-term sustainability. Advancing legal and policy reforms is an inherently political process, and therefore, difficult to ensure success within short time frames. To achieve this sub-purpose, the project will work closely with the USG inter-Agency, other international donors, and civil society organizations to influence the process. Expected Output for Sub-purpose 3:  Civil society advocacy efforts for specific reforms supported. Expected Outcome for Sub-purpose 3:  Civil society advocacy for specific legislative and regulatory reforms enhanced. Expected Impacts for Sub-purpose 3:  Reduced corruption.  Enhanced government effectiveness  More effective provision of the public goods of justice and basic services. IV. Scope of Work for the Evaluation The overall objective of the performance evaluation is to assess the performance of the DG project against the project objectives, outcomes and impacts listed above and detailed in the attached full project description. The evaluation will serve as a resource for USAID in designing future programs, as well as the donor community and the Government of Paraguay. In particular, the evaluation would collect data indicative of the success or failure of specific interventions in achieving the expected results identified in the terms of the agreement or contract with the implementing partner. Serving as a point of reference for the mid-term and final evaluations, at the beginning of the project, the performance evaluation team will collect baseline data and provide a baseline report to the Mission. For the mid-term report, the evaluation would assess evidence of project effects (positive or negative) and whether the interventions are being implemented in a manner that will achieve the expected outcomes. Furthermore, data will be disaggregated and presented by gender. The mid-term report will identify and recommend adjustments or corrective measures to be taken in the ongoing project implementation. The final evaluation report will examine whether the project achieved its expected outcomes and impacts. The final evaluation report will address the cost effectiveness of the project and provide technical recommendations to inform future project design in order to maximize the effectiveness of future DG programming in Paraguay. In addition, data will be disaggregated and presented by gender. The evaluation will otherwise comply with the January 2011 USAID Evaluation Policy, available at http://www.usaid.gov/evaluation. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 55 As identified above, the evaluation will focus on the following areas of interest, noting illustrative sub￾questions and topics which will be determined through a collaborative process between the evaluation team and Mission. 1) PROJECT PERFORMANCE  In both the mid-term and final evaluation reports, the performance evaluation team will assess whether, to what extent and why the project has been effective in achieving the expected objectives, outcomes, and impacts.  In both the mid-term and final evaluation reports, the evaluation team will conduct a gender analysis throughout the report, in order to assess heterogeneous treatment effects between men and women, and to examine how women may have faced different barriers or constraints in accessing the project.  In both the mid and final evaluation reports, the evaluation team will address the efficiency of the project along with implementation failures or obstacles to implementation that have hindered the success of the project.  In the mid-term report, the evaluation team will recommend project adjustments to improve the project’s effectiveness.  In the final evaluation report, the team will summarize lessons learned and provide recommendations and guidance to inform future efforts. 2) APPROPRIATENESS OF FOCUS  In both the mid and final evaluation reports, the team will evaluate the efficacy of the primary project objectives given the country context, as well as USAID and GOP strategy and objectives to support as described above.  The final evaluation report will assess the sustainability and institutionalization of the activities in GOP activities and processes. 3) COST–EFFECTIVENESS AND SUSTAINABILITY  In both the mid and final evaluation reports, the team will assess the cost-effectiveness of the methodologies employed in the project towards achieving project goals and objectives.  In the final evaluation report, the team will examine the opportunities or constraints that the GOP faces in terms of its ability to continue or replicate the interventions designed and implemented under the DG project. V. Suggested Evaluation Methods Depending on the question and outcome of interest, the assessment will utilize a mixture of qualitative and quantitative research methods. 1) ILLUSTRATIVE QUALITATIVE METHODS  Key informant interviews  Focus Groups 2) RECOMMENDED QUANTITATIVE METHODS The performance evaluation will utilize a large-N probability survey to assess the project performance. This will consist of a cross-sectional survey of federal bureaucrats in the line ministries receiving the project treatment, as well as judicial staff that are the target of the project’s judicial strengthening components. The survey will be conducted at the baseline, mid-term and end point of CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 56 the project. To rigorously study issues surrounding corruption and accountability, the team will seek to embed experiments in the survey design. Regression techniques and econometric models will be used to analyze the survey data. Depending on the line ministries selected for the project interventions, the performance evaluation may be able to identify a non-treatment line ministry that is similar to one being treated on several critical aspects – including size and function. Given available funding, this similar non-treatment line ministry could afford an opportunity to survey a “comparison” group of federal bureaucrats for a more rigorous evaluation. Here it is important to highlight that the comparison group will not serve as a rigorously constructed counterfactual, but may help to improve some of the research findings and inferences from the evaluation. The development of the survey instruments and sampling plan will take place in collaboration with the Learning Team of USAID/DCHA/DRG. VI. Evaluation Deliverables The primary audiences for the evaluation reports are both external and internal. USAID is committed to making public USG-funded survey data, and believes that the research conducted during the course of the evaluation is of great interest to the broader donor community, the GOP, as well as the academic community. 1) DESIGN AND WORK PLAN Within four weeks of receiving the tasking, NORC shall submit proposed personnel for the evaluation team, an Evaluation Design, and a Work Plan with Gantt chart to the USAID/DRG and USAID/Paraguay. The Evaluation Design shall describe the research design and evaluation methodology and should include draft data collection instruments. The work plan shall include a timeline for data collection, analysis, report writing, submitting drafts of each report to USAID, report revision and finalization. The plan must also include opportunities for briefings of findings to USAID in Paraguay. USAID approval of the Evaluation Design and Work Plan is required before fieldwork can begin. Any requests for deviations from the plan and timeline must be requested in writing and receive approval by the COR. 2) SURVEY INSTRUMENT & SAMPLING STRATEGY The survey instrument and sampling plan used must be reviewed and approved by USAID. 3) ENTRANCE BRIEFING The evaluation team shall provide an entrance briefing to the USAID/Paraguay Mission Director and other interested Mission staff at the beginning of the activity to present to the Mission the Evaluation Team’s objectives and methodology for the evaluation period. 4) BASELINE REPORT Within one month of data collection, the evaluation team shall submit a baseline report to USAID/Paraguay and the COR. The baseline report will provide descriptive data on the survey results, and will include a data analysis plan for the evaluation. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 57 5) DRAFT MID-TERM EVALUATION REPORT Within four weeks of mid-term data collection, the evaluation team shall submit the draft mid-term evaluation report (in English and Spanish), with findings and recommendations, to USAID/Paraguay and the USAID COR for review. The draft report shall contain a maximum of forty pages (not including annexes). 6) MID-TERM EVALUATION REPORT Electronic copies of the mid-term report (in English and Spanish) in PDF and Microsoft Word versions shall be submitted within ten business days following the receipt of comments from USAID. The draft report shall contain a maximum of forty pages (not including annexes). The mid-term report shall incorporate feedback from USAID and shall include an executive summary. Data collection instruments, sources, and the SOW will be included as report annexes. The Mid-term Evaluation Report will be a public document, therefore the contractor shall submit it to the Development Experience Clearinghouse (http://dec.usaid.gov). 7) EXIT BRIEFING The evaluation team shall provide a formal oral exit briefing of the mid-term evaluation findings, conclusions and recommendations to the USAID/Paraguay Mission Director, DRG Office and other interested Mission staff. 8) DRAFT FINAL EVALUATION REPORT Within four weeks of the receipt of final evaluation data, the evaluation team shall submit the draft final evaluation report (in English and Spanish), with findings and recommendations, to USAID/Paraguay and the COR for review. The draft report shall contain a maximum of forty pages (not including annexes). 9) FINAL EVALUATION REPORT Electronic copies of the Final Evaluation Report (in English and Spanish) in PDF and Microsoft Word versions shall be submitted within ten business days following the receipt of comments from USAID. The final report shall contain a maximum of forty pages (not including annexes). The Final Evaluation Report shall incorporate feedback from USAID and shall include an executive summary. Data collection instruments, sources, and the SOW will be included as report annexes. The Final Evaluation Report shall be submitted to the Development Experience Clearinghouse (http://dec.usaid.gov). 10) Fully documented datasets The evaluation team will provide to USAID all documents/data (with all identifiers removed). In addition, the contractor shall submit these documents/data to the Development Experience Clearinghouse. VII. Team Composition PRINCIPAL INVESTIGATOR Role & Responsibilities: CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 58  Leads technical development of design, data collection and analysis, and process to formulate conclusions and recommendations.  Manages the overall research assessment.  Serves as lead author for mid-term and final report.  Ensures technical integrity of study, including development of quality data collection instruments and high standard of fieldwork. Qualifications:  Ph.D. in relevant field preferred (public policy, political science, sociology or public administration)  Extensive experience with governance issues in Latin America, preferably Paraguay  Expertise in quantitative and qualitative research methods  At least 5 years of experience in relevant fieldwork  Fluency in Spanish preferred  Regional knowledge required in public policy, political science, sociology or public administration. TEAM MEMBERS  Experience with governance issues in Latin America, preferably Paraguay, and  Fluency in Spanish (for team members conducting field work). VIII. Evaluation Management The Principal Investigator is expected to oversee all aspects of the work to be performed under this contract. USAID/Paraguay will assist the team in making the necessary arrangements (e.g., identifying interviewees, setting up appointments with public officials) to facilitate their work. The contractor is responsible for arranging travel logistics (e.g., plane tickets, vehicle rental and drivers, translators, hotel accommodations) as needed for site visits and field work and to locate appropriate data collection expertise in order to conduct survey research. USAID will be available to the team for consultations regarding sources and technical issues before and during the evaluation process. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 59 ANNEX B, KEY INFORMANTS CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 60 Table B.1. Baseline informants INSTITUTION NAME TITLE DGP CEAMSO Full team DGP Implementer GOP COUNTERPARTS MEC Gerda Palacio Director, Educational Innovation Center María García Vice Minister of Superior Education Carmen González HR Director General Sonia Paredes Director, Talent Management, Evaluation and Well-Being Celeste Mancuella MECIP Coordinator Dalila Zarza General Director of Planning CSJ María Victoria Rivas Director, Center for Judicial Studies Eugenio Rolón Supreme Court Justice Martinez Simón Supreme Court Justice Gustavo Ocampo Member of the Chamber of Criminal Appeals Mario Liceche Internal Jurisdictional Auditor Rafael Monzón Superintendent Nelly Burgos Director of Department of MECIP Martha Ferrara Executive Director Luis Fernando Bezzano General Internal Auditor AGPE Bertha Rodríguez de Perinetto Minister SFP Humberto Peralta Minister OTHER STAKEHOLDERS Centro de Información y Recursos para el Desarrollo (CIRD) Alvaro Caballero Senior Advisor Leticia Alcaraz Program Assistant Instituto de Estudios para la Consolidación del Estado de Derecho (ICED) Roberto Úbeda Szarán Executive President Semillas para la Democracia Martha Ferrara Executive Director Guzmán Ibarra Coordinator CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 61 INSTITUTION NAME TITLE UNDP Paraguay Camilo Filártiga Callizo Coordinator of Democratic Governance Projects Desarrollo en Democracia (DENDE) Alberto Acosta Garbarino President Instituto de Derecho y Economía Ambiental (IDEA) Patricia Baez Lawyer Fabián Forestieri Director of Social Participation Ezequiel Santagada Executive Director USAID USAID/Paraguay Adriana Casati Development Assistance Specialist – Civil Society Fernando Cossich Mission Director Leanne Webster Director of Projects Alfonso Velázquez Rule of Law Specialist CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 62 Table B.2. Midline informants INSTITUTION NAME TITLE DGP CEAMSO Ofelia Yegros Executive Director Liliana Villalba Coordinator of DGP Components Mary Hogan Chief of Party Enrique Villalba Deputy Chief of Party Fatima Andrada Component 1 Coordinator Margarita Rojas Component 1, IT Specialist Mercedes Argarañaz Component 1, HR Specialist Raúl Quiñónez Legal Framework Coordinator Martín Miranda DGP Monitoring and Evaluation Specialist Cesar Pastore Component 2 Coordinator GOP COUNTERPARTS CSJ Daniel Lobo Corbeta Former Lawyer MEC Dalila Zarza Director SFP Jose Morinigo Bosch Minister OTHER STAKEHOLDERS E+E Cynthia Gonzalez Manager Universidad Católica de Asunción Alberto Ramirez Researcher CIRD Agustin Carrizosa General Manager Gestión Ambiental (GEAM) Jorge Abbate General Manager Semillas para la Democracia Marta Ferrara President No affiliation Julio Dominguez Former Senator No affiliation Patrick Kemper Businessman and politician No affiliation Pablo Lesme Businessman and lobbyist USAID USAID/Paraguay Adriana Casati Program Officer Alfonso Velázquez Rule of Law Specialist Laura Álvarez Monitoring and Evaluation Specialist CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 63 Table B.3. Endline informants INSTITUTION NAME TITLE DGP STAFF, CONSULTANTS, AND SUBCONTRACTORS CEAMSO Fatima Andrada Former Component 1 Coordinator Mercedes Argaña Quintana Component 1, HR Specialist César Pastore Former Component 2 Coordinator Martín Miranda DGP Monitoring and Evaluation Specialist Mary Hogan Chief of Party Raúl Quiñónez Legal Framework Coordinator Integra Servicios Víctor Gómez Director Dataworks Rolando Chaparro Software Developer CIRD María Victoria Rojas Consultant TOP Consulting Jorge Hintze Consultant Benítez, Codas & Asociados Fredy Giménez Consultant Konecta Herman Vottger Project Lead Diana Jara Project Lead Independent Julio Paciello Consultant Independent José Félix Bogado Tábacman Consultant Independent Leopoldo Eduardo Cataldi Sardi Consultant GOP COUNTERPARTS MEC Dalila Zarza Former Director of Planning Mirta Graciela Vera Former Coordinator of Internal Control Martha Lafuente Former Minister Carolina Arrúa Director of Training and Evaluation Sonia Paredes Director of Competitive Selection MIC Carlino Velázquez Institutional MECIP Coordinator Esteban Carloso Auditor General Carolina Rodríguez Director of Monitoring for MECIP Department AGPE Bertha Rodríguez Auditor General of Executive Branch CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 64 INSTITUTION NAME TITLE CAJ (CSJ) Edward Vittone Legal Counsel Luz Antúnez Audit Advisor Juan José Martínez Lawyer Edgar Cicorio Director of Human Resources CGR Gladys Fernandez General Director of the Economy Juan Carlos Cano Cabral Advisor Roy Rodgers Former Comptroller General for the GoP Adriana Hug de Belmont Director of Quality Fabian Forestieri Director of Institutional Affairs Jazmín Vergara Lawyer, Institutional and International Affairs Maria Teresa Anasco Director of Management Ivan Balmore Director of Cooperation MOPC Katya Spiridonoff Former Director of Human Resources Martha Benítez Vice Ministry of Administration and Finance Karina Depps Director of Human Resources Antonia Miranda Institutional MECIP Coordinator MSPBS César Pimienta Director of Human Resources Lázaro Benítez Coordinator of MECIP Patricia Giménez Director of Planning Rocío Lobos Coordinator of the Committee for Continuous Improvement (Trauma Hospital) Luis Carlos Báez Medical Director (Trauma Hospital) SFP Humberto Peralta Minister OTHER STAKEHOLDERS Spanish Agency for International Development Cooperation (AECID) Carmen De Diego Fonseca Program Officer Inter-American Development Bank (IADB) Jean-Eric Theinhardt Modernization of State Specialist CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 65 INSTITUTION NAME TITLE USAID USAID/Paraguay Laura Álvarez Monitoring and Evaluation Specialist Laura McKechnie Democracy and Governance Team Leader Adriana Casati Program Officer Federico Recalde Public Administration Specialist Alfonso Velázquez Rule of Law Specialist CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 66 ANNEX C, EVALUATION DESIGN MATRIX CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 67 Table C.1. Evaluation Design Matrix EVALUATION QUESTIONS AND SUBQUESTIONS DATA SOURCES AND METHODS DESK REVIEW ANALYSIS OF KII DATA ANALYSIS OF EMPLOYEE SURVEY DATA ANALYSIS OF EMPLOYEE FGD DATA PROJECT DOCS. AND SECONDARY SOURCES PERF. MONITORING DATA Evaluation Question 1. Have internal management systems within the GoP been strengthened as a result of DGP interventions? Have those improvements led to changes in the perceptions and behavior of public sector employees? SUB-QUESTIONS ABOUT CIVIL SERVICE AND HUMAN RESOURCE MANAGEMENT Sub-question 1.1.1. Has the SFP strengthened its leadership in civil service reform? Has it gained influence among Executive Branch institutions?  -  - Sub-question 1.1.2. How effectively and extensively is the SICCA being implemented?    - - Sub-question 1.1.3. How effectively and extensively is IGP being implemented?    - - Sub-question 1.1.4. To what extent are hiring and promotions increasingly based on merit? Do employees perceive increasing fairness in hiring and promotions? Are jobs advertised well ahead of time? Are jobs advertised widely enough to assure fair access by the relevant pool of applicants? Is the average number of years of schooling of public employees increasing over time? - - -   Sub-question 1.1.5. How regularly are evaluations conducted of employees’ performance? Have all employees been included in some type of evaluation of competency? Are performance evaluations based on set, specified criteria? - - -   Sub-question 1.1.6. Do employees feel their hard work and initiative are recognized and rewarded? Do employees see that negative attitudes towards work obligations and laziness are sanctioned? - - -   Sub-question 1.1.7. Do employees perceive progress in the institutionalization of a merit system in their agency? - - -   CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 68 EVALUATION QUESTIONS AND SUBQUESTIONS DATA SOURCES AND METHODS DESK REVIEW ANALYSIS OF KII DATA ANALYSIS OF EMPLOYEE SURVEY DATA ANALYSIS OF EMPLOYEE FGD DATA PROJECT DOCS. AND SECONDARY SOURCES PERF. MONITORING DATA Sub-question 1.1.8. Are training actions conducted on a regular basis in all public units? To what extent are training opportunities available? - - -   Sub-question 1.1.9. What do employees think about their direct supervisors and their institution’s leadership? Have employees perceptions about their supervisors and their institution’s leadership improved? - - -  - Sub-question 1.1.10. Do employees believe their job duties and responsibilities are clearly defined and communicated to them? Do employees feel they are compensated according to their duties and responsibilities? - - -   Sub-question 1.1.11. Are the reforms associated with greater job satisfaction and lower absenteeism among federal employees? - - -   SUB-QUESTIONS ABOUT INTERNAL CONTROLS Sub-question 1.2.1. Are internal control systems operating effectively?  -   - Sub-question 1.2.2. How widely has the MECIP been adopted?  -    Sub-question 1.2.3. To what extent are public sector employees aware of the new systems of internal controls? - -   Sub-question 1.2.4. How often do employees witness unethical behavior or acts of corruption? How often are employees pressured to break rules and engage in acts of corruption? How often are employees forced to unreasonably/illegally redirect benefits to some specific entity/individual? Which are the most common forms of corruption encountered? - - -  - Sub-question 1.2.5. Is openly unethical/corrupt behavior left unpunished? - - -   CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 69 EVALUATION QUESTIONS AND SUBQUESTIONS DATA SOURCES AND METHODS DESK REVIEW ANALYSIS OF KII DATA ANALYSIS OF EMPLOYEE SURVEY DATA ANALYSIS OF EMPLOYEE FGD DATA PROJECT DOCS. AND SECONDARY SOURCES PERF. MONITORING DATA Sub-question 1.2.6. Do reasonable incentives exist to report the commitment of acts of corruption to authorities? - - -   Evaluation Question 2. Are inputs provided by the DGP contributing to identifiable changes in processes and systems capacity in its focus areas? How effective has the program been in transferring skills, knowledge, and capacity to GoP staff? Sub-question 2.1: Do partner institutions feel inputs provided by DGP have changed the way they function and improve their performance?  -  - - Sub-question 2.2: Has training provided by DGP improved or changed knowledge, attitudes, and practices of partner staff?  -  - - Evaluation Question 3. What deficiencies currently exist in the implementation of the program’s objectives and how can these be remedied? Sub-question 3.1: What are the causes and factors that have hindered the achievement of the DGP’s Performance Management Plan (PMP) targets and progress?    - - Sub-question 3.2: Are any program objectives lagging in results compared to the others? Why?    - - Sub-question 3.3: Have any program themes, cross-cutting elements, or special projects caused controversy or backlash in the country?  -  - - Evaluation Question 4. Is the current scale and scope of the program appropriate to support sustainability? Should certain aspects or components be prioritized or limited? Sub-question 4.1: Has the DGP identified key GoP counterparts and is the assistance provided to these on target, based on priority needs?  -  - - Sub-question 4.2: Is the scope of DGP assistance flexible to respond to changing priorities or emerging problems within GoP counterparts?  -  - - CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 70 EVALUATION QUESTIONS AND SUBQUESTIONS DATA SOURCES AND METHODS DESK REVIEW ANALYSIS OF KII DATA ANALYSIS OF EMPLOYEE SURVEY DATA ANALYSIS OF EMPLOYEE FGD DATA PROJECT DOCS. AND SECONDARY SOURCES PERF. MONITORING DATA Sub-question 4.3: Did the DGP and GoP counterparts have regular “how are we doing” meetings to discuss changing priorities or redundant activities? -  - - Evaluation Question 5. What are the good practices emerging from the program? Are there successes that could be replicated for future projects? Sub-question 5.1: What type of assistance has produced the most improvement? Where and why?     - Sub-question 5.2: What type of assistance has produced the least progress? Where and why?     - Sub-question 5.3: What are the emerging lessons that should be kept in mind for future projects?    - - CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 71 ANNEX D, KEY INFORMANT INTERVIEW GUIDE CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 72 INTRODUCCIÓN Buenos días. Mi nombre es Luis Camacho. Soy un investigador sénior en NORC at the University of Chicago, un centro de investigación social independiente que se especializa en el análisis y la recolección de datos. Me acompaña mi colega Christopher Hansen, analista de investigación sénior en NORC. USAID está financiado el Programa de Democracia y Gobernabilidad que es implementado por el Centro de Estudios Ambientales y Sociales (CEAMSO). El objetivo del Programa es mejorar los sistemas de gestión de recursos humanos y los sistemas de control interno en diversas instituciones públicas. El programa está próximo a concluir y USAID ha solicitado a NORC realizar una evaluación de desempeño del Programa. Esta es una evaluación independiente; nosotros no somos empleados de CEAMSO o USAID. El objetivo principal de la evaluación es extraer lecciones sobre el Programa, saber, por ejemplo, en qué áreas de trabajo se lograron avances importantes y cuáles otras no se lograron los resultados esperados, qué se hizo bien y qué se pudo haber hecho mejor, y qué áreas prioritarias no pudieron ser atendidas. Toda la información que ustedes nos proporcionen nos será de mucha utilidad. La entrevista debería durar unos 60 minutos. RONDA DE PRESENTACIONES Ahora les pediría que cada uno de ustedes se presente brevemente. CONFIDENCILIDAD Muchas gracias a todos por su colaboración con la evaluación. Nuestro equipo valora mucho las apreciaciones que van a compartir con nosotros el día de hoy. Nuestra conversación es confidencial; ninguna de sus apreciaciones será atribuida a su persona en ningún reporte. Sus apreciaciones serán analizadas junto con aquellas que vamos a obtener a través de otras entrevistas usando un software especializado. Para poder realizar este análisis quisiéramos grabar la conversación. La grabación solo será escuchada por el equipo de evaluación y no será compartida con USAID ni con CEAMSO. ¿Podrían confirmar si están de acuerdo con que la entrevista sea grabada? [Prender la grabadora si los participantes dan su consentimiento.] PREGUNTAS PREGUNTAS INCIALES ¿Cuáles son las principales áreas en las que han trabajado con CEAMSO en el marco del Programa de Democracia y Gobernabilidad? Explorar:  ¿Trabajaron en temas relacionados a la gestión de recursos humanos incluyendo la implementación del Sistema Integrado Centralizado de la Carrera Administrativa (SICCA) o del Índice de Gestión de Personas (IGP)?  ¿Trabajaron en temas relacionados a sistemas de control interno incluyendo la implementación del Modelo Estándar de Control Interno del Paraguay y de portales de transparencia u otras iniciativas de rendición de cuentas? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 73  ¿Trabajaron en temas relacionados a tecnologías de la información y las comunicaciones?  ¿Trabajaron en temas relaciones a contrataciones y adquisiciones?  ¿Trabajaron en temas relacionados a mejorar la equidad entre hombre y mujeres? ¿Cómo fue el contacto inicial con el Programa? ¿Cómo se decidió las áreas de trabajo? ¿CEAMSO les ofreció una gama de posibles áreas de cooperación? Explorar:  ¿Se hicieron líneas de base (usando el IGP o el MECIP) para determinar las áreas que necesitaban atención de manera prioritaria?  ¿Hubieron mecanismos para revisar las áreas de enfoque de manera periódica? PREGUNTAS ESPECÍFICAS SOBRE GESTIÓN DE RECURSOS HUMANOS Y CONTROL INTERNO ¿Cuáles fueron los objetivos que se trazaron? ¿Se lograron todos los objetivos? ¿Cuáles sí y cuáles no? ¿Cuál ha sido el logro más importante? Explorar:  ¿Se observan cambios en los comportamientos y las actitudes del personal del/de la (NOMBRE DE LA ENTIDAD)  ¿Son estos cambios visibles para los ciudadanos? ¿Cuáles son los principales factores que contribuyeron al éxito? Explorar:  ¿Qué tan importante fue la voluntad política para lograr los objetivos? ¿Cuál es el área en la que tal vez no se lograron todos los objetivos? ¿Qué dificultades encontraron? Explorar:  ¿Por qué no se lograron los resultados esperados?  ¿Qué tan importante fue la voluntad política para lograr los objetivos? ¿El trabajo en esta área ha producido algunos resultados contraproducentes que no fueron esperados? PREGUNTAS GENERALES Quisiera conocer sus apreciaciones sobre cómo ha sido el trabajo con CEAMSO y su equipo en el marco del Programa. ¿Qué tan efectivo ha sido el enfoque para proveer asistencia técnica y apoyar la implementación de reformas e iniciativas de mejora? Explorar:  ¿Qué hizo bien y qué pudo haber hecho mejor el Programa en este campo? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 74  ¿CEAMSO y su equipo estuvieron interesados en su retroalimentación a las actividades del Programa? Si sí, ¿de qué manera se hizo la retroalimentación? (Por ejemplo, ¿se hicieron reuniones periódicas, se les pidió comentarios escritos?) ¿Piensan que las mejoras y reformas a las que ha contribuido el Programa van a ser sostenibles en el tiempo? Quisiera que piense en sostenibilidad no solo como mantener las mejoras logradas en el tiempo sino también como continuar avances sin apoyo del programa. Explorar:  ¿Qué hizo el Programa para asegurar la sostenibilidad de las mejoras y reformas?  ¿Qué hizo bien y qué pudo haber hecho mejor el Programa en este campo?s En términos generales, ¿le parece que las áreas en las que se ha enfocado el Programa son apropiadas y prioritarias? Explorar:  ¿Qué otras áreas importantes debería haber cubierto el Programa? ¿Esta necesidad sigue vigente? ¿Cuál es la principal lección o aprendizaje que les deja el trabajo con el Programa en estos años? PREGUNTA FINAL Antes de terminar, quería darles la oportunidad de tocar cualquier otro tema sobre el Programa o CEAMSO que no hemos cubierto en la entrevista. Por favor recuerden que todas sus apreciaciones son confidenciales. Con esto concluimos la entrevista. Muchísimas gracias por su tiempo. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 75 ANNEX E, EMPLOYEE SURVEY DETAILS CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 76 SURVEY DESIGN The public sector employee survey uses a multi-stage, stratified, random sample with proportional allocation to achieve an optimal design – i.e., maximum statistical power given the sample size requested by USAID). The design allows for adequate statistical precision at the level of a given institution, at the level of the staff category (management, junior staff, technical) across the institutions combined, and at the level of the organizational category (cabinet/secretariat, administrative, operations) across the institutions combined.35 NORC drew the samples for the baseline and endline surveys by implementing the steps described below. Each participating institution provided a list of their employees through USAID/Paraguay. These lists were used as the sampling frame after removing all employees not working in the “Asunción” or “Central” districts. STEP 1: PURPOSEFUL SAMPLING OF INSTITUTIONS The primary sampling unit is the institution, for example, the CSJ. The baseline survey included a total of six institutions; the endline survey included five. For each round of data collection, 183 employees were selected from each institution – i.e., equal-sized subsamples from each institution). STEP 2: STRATIFICATION Within each institution, the sample was stratified along two dimensions: (1) organizational category (cabinet/secretariat, administrative, and operations) and (2) staff category (management, junior, and technical).36 Table C.1 provides each combination of organizational and staff categories that forms a stratum, for a total of nine strata, labeled with the letters A through I. Table E.1. Sample Strata DIMENSION 1 DIMENSION 2 MANAGEMENT JUNIOR TECHNICAL Cabinet/Secretariat A B C Administrative D E F Operations G H I To implement this stratification, each departmental subunit – e.g., dirección – was assigned to one of the three organizational categories. Also, each job title (cargo) within each organizational category was assigned to one of the three staff categories. All employees from the sampling frame were allocated to each stratum. STEP 3: SELECTION OF STAFF The secondary sampling unit is the employee. The 183 employees allocated to each institution were equally divided across organizational and staff categories to create a proportionally allocated random sample of approximately 20 employees per stratum. Employees within each stratum were selected through the following process: First, within each stratum, each employee was assigned a random 35 The design does not allow for adequate statistical precision for individual organizational units or job titles within a given institution, which increases respondent anonymity. Preserving respondent anonymity was a chief concern for NORC and USAID. 36 Note that while the categories contain them, the staff categories do not correspond to staff positions (cargos) and the organizational categories do not correspond to the departmental units. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 77 number. Each stratum list was then sorted by the assigned random number within job title. From each list, a certain quantity of staff from each job title within the stratum was selected for inclusion in the sample. That quantity was equal to the proportion of staff in the stratum having that job title.37 For example, if Stratum D in an institution had 100 employees in the sampling frame and 15 were asistentes sociales, then the first three employees (15 percent of the 20 employees assigned to Stratum D) on that part of the randomized list with the title of asistente social were selected for the sample. STEP 4: ADJUSTMENT FOR SMALL SUBPOPULATIONS AND SELECTION OF REPLACEMENTS The procedure outlined above, as is inevitably the case in sample design, had to be adjusted to overcome various realities on the ground. First, some institutions had an insufficient number of employees to fully fill the job-title targets for some strata. Second, some of those employees selected were unavailable to take the survey, generally due to sickness, vacation, conflicting engagements, or office travel. As such, stratum-specific replacements were required. Too few staff for sampling. Ideally, the requisite 20 employees per strata could have been randomly drawn for each of the nine strata in a given institution, weighted according to job title. However, the relative scarcity of staff in some strata job titles – e.g., Stratum C containing “technical experts” in the Cabinet – prevented achieving the stratum’s sample size target for the job title. To compensate, NORC made up the deficit by drawing staff from the same job title in a different organizational category. For example, in some institutions, the Cabinet’s entire sampling frame was smaller than the target subsample size for the stratum and had to be supplemented with people in the same job title from other strata. (The exact algorithm for substitution is a bit more complicated and available upon request). Selection of replacements. To ensure availability of replacements, NORC increased the original sample-size target for each stratum by 25 percent and applied the same procedure described in Steps 1 to 3, above. Hence, instead of 20 employees per stratum, we select approximately 25, for a total of 75 potential respondents per organizational category, and 225 per Institution. Note, therefore, that there was no sampling difference between the original employee selected and her replacement; the HR department in the respective institution simply proceeded to the next employee in the randomized list for that job title within the stratum as needed to achieve targets. QUESTIONNAIRE DEVELOPMENT NORC used the sub-questions presented in Section 2 as a starting point to develop the survey questionnaire. The majority of survey questions asked respondents to indicate whether their level of agreement with a statement using the following response options:  Strongly agree (Muy de acuerdo)  Somewhat agree (De acuerdo)  Neither agree nor disagree (Ni de acuerdo ni en desacuerdo)  Somewhat disagree (Un poco en desacuerdo)  Strongly disagree (Muy en desacuerdo)  Don’t know (No sabe) 37 This was, in fact, a second stratification that ensured that all job titles were adequately represented in the sample. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 78 In addition, the questionnaire used three survey experiments or truth elicitation protocols that provide significant degrees of anonymity to respondents and encourage them to provide candid responses to sensitive, compromising, and even incriminating questions. The first experiment is a list (or count) protocol in which respondents are asked to indicate the number of statements in a list that are true – rather than whether each situation is true. One set of respondents is presented with a list of innocuous situations and another set is presented with this same list plus one sensitive – i.e., ethically-questionable or illegal – situation. By comparing the responses across groups, one can estimate the prevalence of the ethically questionable or illegal situation. The second experiment is a dichotomous randomized-response protocol in which respondents first have to flip a coin, and then respond to an innocuous or a sensitive question yes-no depending on the outcome of the coin flip. Only the respondent knows the outcome of the roll and which question she is answering, encouraging truthful responses. A subset of respondents also separately answers the innocuous question, which allows for estimating the true responses to the incriminating question. The third experiment is a numeric randomized-response protocol in which respondents add the outcome of a die roll to their true response on a scale indicating the frequency of occurrence of various sensitive situations. The distribution of true responses is estimated taking advantage of the known expected distribution of a die roll. The questionnaire was vetted by the evaluation team and reviewed by NORC’s survey specialists for internal logic to ensure that questions are not cognitively difficult, double barreled, culturally insensitive, or include words that may create biased responses. Questions were also analyzed for their ability to link back to key indicators and research sub-questions, and for their contribution to the overall evaluation design. The questionnaire went through an additional round of revisions after a pre-tested with a group of employees from the MEC. Lastly, USAID/Paraguay provided approval for the final version of the questionnaire. Annex F presents version C of the survey instrument. Table G.1 in Annex G lists the characteristics of the various survey versions that were developed to implement the truth elicitation protocols. As mentioned in Section 3, the questionnaires used in the CSJ differed from the ones used in the other institutions. When the baseline survey was to be administered in the CSJ, their leadership insisted on adding a couple of additional questions. These question could not be included in the questionnaires for the other institutions. Table G.2 in Annex G lists the differences between the original and CSJ-specific versions. MANAGEMENT AND QUALITY CONTROL NORC subcontracted Economía y Estadísticas para el Desarrollo (E+E), a Paraguay research organization headquartered in Asunción, to support the administration of the surveys. E+E coordinated closely with the HR departments in each institution to arrange survey-taking sessions at conference rooms in each institution. Selected employees were invited to attend one of these session and self-administer the questionnaire. An E+E field staff member was responsible for distributing questionnaires, providing general instructions to respondents, and answer questions from individual respondents. NORC developed the various versions of the questionnaire and experimental questions, and programmed them using SnapSurvey; developed training materials; provided targeted training to E+E to administer the anonymous survey ensuring complete anonymity; developed the sampling strategy and drew the samples; and analyzed the data. NORC also provided management support. NORC took several steps to ensure data quality, including the following: CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 79  Programming the survey instrument using SnapSurvey software, which combines survey authoring with a built in scanning functionality. Once the survey instrument is developed in SnapSurvey, the scanning application for the instrument is automatically generated and data capture is accomplished by scanning the PAPI forms. SnapSurvey allows review and editing of captured data and preserves scanned records of all PAPI instruments. This approach both speeds data delivery and allows us greater control over data quality by allowing us to easily compare questionnaires with the data file.  The use of SnapSurvey scan data capture to enter responses and regular uploading of the data to the NORC’s Secure File Transfer Protocol (SFTP) ensured that the NORC and E+E team subjected the data to regular review.  NORC’s and E+E were in regular contact to address the challenges that arose during data collection. SURVEY ADMINISTRATION CHALLENGES SURVEY MODE NORC’s original plan was to administer a web-based survey among public sector employees throughout the country. NORC abandoned this plan after much deliberation with human resource representatives from selected institutions because of concerns about the use of email among their employees. We ultimately programmed the survey using Snap Survey, and implemented a self￾administered PAPI survey and restricted the target population to Asunción-based employees. DELAYS IN SURVEY ADMINISTRATION Baseline data collection in the MEC, MSPBS, and MAG was completed by late December 2014, as scheduled. Data collection in the MIC, MOPC, and CSJ, however, was fraught with challenges and not completed until October 2015. Endline data collection in the MEC, MIC, CSJ was completed by December 2018, as scheduled, while data collection in the MOPC and MSPSB was completed by early-February 2019. Reasons for the delays include:  Many employees included in the list provided by the ministries no longer were employed at the ministry, changed jobs, or departments since the list was developed.  There was a lack of communication between those at the higher levels who are in charge and HR employees required support E+E in contacting and making arrangements for employees to participate in the survey.  Likewise, several HR departments were short-staffed or reticent to assist E+E in making arrangements with employees.  Many employees travelled outside of Asunción for their job, making it difficult to arrange group appointments to take the survey.  High-level employee were more difficult to agree to participating in survey-taking sessions.  Sample frame provided by MOPC at baseline included employees outside of Asunción, and required NORC to re-stratify and re-draw the sample.  At baseline, CSJ leadership questioned the survey altogether – challenging the sampling strategy and survey instrument. The CSJ ultimately developed an updated listing of employees which NORC used to re-stratify and re-draw the sample. The CSJ also insisted on revising and adding a few questions (see end of Annex G showing differences in the survey instrument). CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 80 ANNEX F, EMPLOYEE SURVEY QUESTIONNAIRE (VERSION C) 1-3 Encuesta a Funcionarios Públicos NORC, el centro de investigación social de la Universidad de Chicago en los Estados Unidos, le da la bienvenida a esta encuesta, que aspira a conocer las opiniones de las servidoras y servidores públicos paraguayos sobre condiciones y arreglos institucionales que influyen en su labor diaria. Los resultados de esta encuesta serán útiles al proyecto puesto en marcha por el Gobierno del Paraguay y apoyado por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID), para la modernización y fortalecimiento de la administración pública. La información recabada en esta encuesta será utilizada de manera estrictamente anónima y confidencial. Además, como lo podrá confirmar, varias preguntas están formuladas de tal forma que es imposible vincular las respuestas al entrevistado. De todas maneras, NORC la utilizará únicamente con fines estadísticos. No se individualizará ninguna encuesta, y tampoco quedará registrada la identidad, para que cada funcionario o funcionaria pueda responder confiadamente, con sinceridad y veracidad. Su participación en esta encuesta es voluntaria y usted puede elegir no responder a alguna de las preguntas o suspender la encuesta si así lo desea. Sus respuestas son sumamente valiosas para ayudar a mejorar los procesos públicos. Contestar la encuesta en su totalidad toma normalmente 30 minutos. Por favor, tómese el tiempo que necesite, dado que su opinión es muy valiosa para nosotros. Desde ya le agradecemos mucho la dedicación gentilmente ofrecida. ¿Acepta participar en esta encuesta? Sí ........................................................................................................... No.......................................................................................................... 1-3 2-3 Instrucciones Usted también encontrará preguntas donde tendrá que responder con un A continuación, usted encontrará preguntas o afirmaciones a las cuales usted responderá con una "X". Por ejemplo: Usted también encontrará preguntas donde tendrá que responder con “Sí”o “No”. Por ejemplo: número. Por ejemplo: Finalmente, donde usted tenga que especificar o escribir su respuesta, tendrá que hacerlo de la siguiente manera: Por otra parte, habrá algunas preguntas o afirmaciones que indiquen que será posible elegir más de una opción, si en su opinión, más de una es correcta. Recuerde que puede responder confiado dado que sus respuestas serán mantenidas en la más estricta confidencialidad. Si tiene algún problema o dudas sobre el llenado de esta encuesta, por favor, diríjase a los monitores presentes en la sala. 2-3 3-3 Inicio de Encuesta Q0 ¿Cuánto tiempo lleva trabajando en este ministerio? Menos de 3 meses................................................................................ De 3 meses a menos de 1 año ............................................................. De 1 año a menos de 2 años ................................................................ De 2 años a menos de 3 años .............................................................. De 3 años a menos de 4 años .............................................................. De 4 años a más ................................................................................... Q1 Me explicaron claramente las responsabilidades de mi función. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Q2 Las tareas que realizo corresponden a la descripción formal (escrita) de mi puesto. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Mi puesto no tiene una descripción formal, o al menos no la conozco No sabe ................................................................................................. Q3 La remuneración total (salario más todos los adicionales) que recibo por mi cargo se corresponde con las funciones que me asignaron. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q4 Los objetivos y prioridades de la institución en la que trabajo han sido bien definidos y comunicados. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... 3-3 4-3 Q5 Las tareas que realizo ayudan al cumplimiento de los objetivos de la institución en la que trabajo. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q6 Dispongo de todos los recursos que necesito para hacer bien mi trabajo. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q7 Las personas con quienes trabajo generalmente colaboran para que las tareas se realicen. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q8 Mi lugar de trabajo es cómodo y agradable. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Q9 En las decisiones que me involucran personalmente, mi opinión es escuchada y tomada en cuenta: Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q10 Confío en las capacidades de mi jefe inmediato. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... 4-3 5-3 Q11 Las autoridades superiores están comprometidas con la correcta marcha de la institución. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q12 Mi unidad tiene el personal y la supervisión que necesita para cumplir con sus funciones. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q13 Los servidores públicos de mi institución poseen los conocimientos y las habilidades relevantes para realizar su trabajo. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q14 Los funcionarios de mi institución que tienen tareas y responsabilidades parecidas a las mías obtienen una remuneración total (salario más todos los adicionales) similar a mi remuneración. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q15 En mi área se reconoce la dedicación puesta en el trabajo. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q16 La creatividad y la innovación son recompensadas en mi institución. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. 5-3 6-3 Q17 Si se reconoce la dedicación y el empeño, ¿por qué medios se expresa este reconocimiento? Para esta pregunta, elija todas las opciones que desee. Promoción ............................................................................................. Gratificación por desempeño ................................................................ Agradecimiento público......................................................................... Nunca se reconoce ............................................................................... No sabe ................................................................................................. Otro (especificar)................................................................................... Q18 La falta de dedicación o la mala actitud es sancionada. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... Q19 El ausentismo en mi área de trabajo es: Casi inexistente..................................................................................... Bajo ....................................................................................................... Moderado .............................................................................................. Alto ........................................................................................................ Muy alto................................................................................................. No sabe ................................................................................................. Q20 ¿Qué tan frecuentemente sus superiores le realizan una evaluación formal escrita de su desempeño? Cada año............................................................................................... Menos frecuente que cada año............................................................. Es irregular ............................................................................................ Nunca me han realizado una ................................................................ No sabe ................................................................................................. Otro (especificar)................................................................................... 6-3 7-3 Q21 En mi institución, los criterios para evaluar el desempeño de cada funcionario son objetivos y confiables. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q22 En mi institución, las evaluaciones de desempeño sirven de base para las decisiones de promoción. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q23 En la administración pública del Paraguay, los ascensos y las promociones se deciden en función del mérito profesional cuidadosamente evaluado, y sin privilegios personales. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q24 Conozco las reglas que rigen las promociones en la carrera de la administración pública. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... 7-3 8-3 Q25 Mi formación y mi experiencia son aprovechadas en mi unidad de trabajo. Q28 En la institución en la cual trabajo, se recluta el personal en función de las Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Q26 Se me ha dado la oportunidad de mejorar mis habilidades en mi institución con capacitación adecuada. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Q27 Los recursos de capacitación se asignan de manera justa y por razones del servicio. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. estrictas necesidades del servicio, es decir, por sus capacidades para cumplir con la función determinada. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No sabe ................................................................................................. Q29 En mi institución, los anuncios sobre vacantes se publican de manera tal que todo aquel que desee postularse al empleo pueda hacerlo con las mismas oportunidades de acceder al puesto. Siempre ................................................................................................. Muchas veces ....................................................................................... A veces.................................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No sabe ................................................................................................. 8-3 9-3 Q30 En el caso en que usted tenga acceso a la política de reclutamiento de funcionarios públicos, ¿qué medios se utilizan para publicar vacantes en su institución? Elija todos los que se utilicen. Sitio Web institucional ........................................................................... Portal Paraguay Concursa.................................................................... Redes Sociales (Facebook, Twitter, entre otras).................................. Diarios de circulación nacional ............................................................. Tableros, boletín o gaceta interna......................................................... Correo electrónico................................................................................. No se publican nunca............................................................................ No sabe ................................................................................................. No contesta ........................................................................................... Otro (especificar)................................................................................... Q31 En mi institución, los concursos para cubrir las vacantes evalúan cuidadosamente los antecedentes de cada candidato preseleccionado garantizando que se elija al más idóneo para el puesto. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No tiene información sobre la política de reclutamiento ....................... Q32 En el último año, el ingreso por mérito a través de concursos a la administración pública del Paraguay: Ha disminuido mucho............................................................................ Ha disminuido algo................................................................................ Se ha mantenido igual .......................................................................... Ha aumentado algo............................................................................... Ha aumentado mucho........................................................................... No sabe ................................................................................................. Q33 Los procesos que se ejecutan en mi unidad están bien especificados y normados: Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q34 En mi trabajo, conozco bien los procesos y procedimientos vigentes. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... 9-3 10-3 Q35 En general, mi área funciona cumpliendo estrictamente con la normativa vigente. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No conozco la normativa aplicable ....................................................... No sabe ................................................................................................. Q36 Mis jefes inmediatos promueven la ética pública y la legalidad de todos los actos administrativos. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. 10-3 11-3 Q39 En el último mes, ¿usted vió las siguientes situaciones en su institución? Responda simplemente "Sí" o "No" sin usar la moneda. Es criba "Sí" o "No". Un colega mío fue transferido a otro departamento Me reuní con funcionarios de una dependencia descentralizada En mi trabajo, aplico y respeto los procesos y procedimientos establecidos Recibí una carta oficial de otro ministerio del Poder Ejecutivo Hubo al menos tres apagones en mi oficina Recibí en mi oficina a un miembro de mi familia Mis compañeros de trabajo mantienen una conducta ética Mi oficina se quedó sin papel o sin tinta para la empresora Mi compañero de oficina viajó en comisión fuera de Asunción Un compañero de oficina me invitó a salir a almorzar Q40 Mis compañeros de trabajo respetan las normas. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No sabe ................................................................................................. Q41 Las autoridades superiores de la institución en la que trabajo respaldan los esfuerzos de mis jefes por mantener un ambiente de ética y legalidad. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Q42 En mi institución puedo realizar denuncias sin miedo a represalias o sanciones. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. 11-3 12-3 Q43 En mi unidad, las faltas administrativas graves reciben la sanción que corresponde por norma. En todos los casos ................................................................................ En la mayoría de los casos ................................................................... En algunos casos.................................................................................. Casi nunca ............................................................................................ Nunca .................................................................................................... No sabe ................................................................................................. Q44 En la institución en donde trabajo, es necesario realizar pagos extras (coimas), regalos o favores para poder obtener los servicios que deben brindarse. Nunca .................................................................................................... Rara vez ................................................................................................ A veces.................................................................................................. Frecuentemente .................................................................................... Siempre ................................................................................................. No sabe ................................................................................................. Q45 Me siento presionado a incumplir las normas para beneficiar a determinadas personas, tanto de dentro como de fuera de mi institución. Nunca .................................................................................................... Rara vez ................................................................................................ A veces.................................................................................................. Frecuentemente .................................................................................... Siempre ................................................................................................. Q46 En la institución en donde trabajo, se aceptan pago extras (coimas), regalos o favores a cambio de otorgar excepciones a las normas, ventajas injustificadas o privilegios. Nunca .................................................................................................... Rara vez ................................................................................................ A veces.................................................................................................. Frecuentemente .................................................................................... Siempre ................................................................................................. No sabe ................................................................................................. 12-3 13-3 Q48 ¿Con qué frecuencia ha visto las siguientes situaciones en su institución? Respuestas posibles Nunca = 1 Rara vez = 5 Alguna vez = 3 Frecuentemente = 4 Siempre = 2 No sabe = 6 Resultado del cálculo Resultado del cálculo Ocultar o falsear información Nombrar a parientes o correligionarios sin el mérito suficiente Obstruir o demorar tramitaciones o expedientes Otorgar viajes, capacitación o gratificaciones arbitraria o Usar bienes de propiedad del injustamente Estado para fines particulares Desviar recursos públicos Descuidar patrimonio institucional Recurrir al tráfico de influencias para Hurtar patrimonio institucional obtener privilegios 13-3 14-3 Q49 En mi opinión, la corrupción y el fraude administrativo en el último año: Q52 La prioridad de mi área es la satisfacción de los usuarios o beneficiarios Ha disminuido mucho............................................................................ Ha disminuido algo................................................................................ Se ha mantenido igual .......................................................................... Ha aumentado algo............................................................................... Ha aumentado mucho........................................................................... No sabe ................................................................................................. Q50 En mi lugar de trabajo, los controles internos son efectivos a la hora de mejorar la eficiencia y la eficacia. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q51 Las auditorías realizadas por el Auditor Interno son efectivas a la hora de limitar el fraude. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. de los servicios que proveemos. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Q53 En mi institución, las mujeres y los hombres tienen las mismas oportunidades de ingreso, promoción y paga. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... No sabe ................................................................................................. Otra opinión sobre oportunidades basadas en el género (especificar) 14-3 15-3 Q54 El acoso sexual es una práctica frecuente en mi lugar de trabajo. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Tengo otra opinión sobre el acoso sexual en mi lugar de trabajo: ....... Q55 ¿En general, qué tan satisfecho está usted con su trabajo? Muy satisfecho ...................................................................................... Satisfecho.............................................................................................. Ni satisfecho ni insatisfecho.................................................................. Insatisfecho ........................................................................................... Muy insatisfecho.................................................................................... Q56 Aconsejaría a la gente joven a que ingrese a la Administración Pública paraguaya. Muy de acuerdo..................................................................................... Un poco de acuerdo.............................................................................. Ni de acuerdo ni en desacuerdo ........................................................... Un poco en desacuerdo........................................................................ Muy en desacuerdo............................................................................... Muchas gracias por su colaboración. Esta información es muy importante para evaluar el programa de Gobernabilidad y Democracia, desarrollado en el Ministerio. Finalmente, por favor indique los siguientes datos, escribiendo una "X" en el cuadro de la opción que corresponda: Q57 Edad Menos de 18 años................................................................................. 18 - 25 años .......................................................................................... 26 - 35 años .......................................................................................... 36 - 45 años .......................................................................................... 46 - 55 años .......................................................................................... 56-65 años ............................................................................................ Más de 65 años..................................................................................... Q58 Sexo Masculino .............................................................................................. Femenino .............................................................................................. 15-3 16-3 Q59 Último grado de escolaridad alcanzado Primario incompleto............................................................................... Primario ................................................................................................. Grado menor a bachillerato................................................................... Bachillerato............................................................................................ Carrera técnica...................................................................................... Licenciatura incompleta ........................................................................ Licenciatura ........................................................................................... Posgrado (Especialización, Maestría o Doctorado).............................. Posgrado incompleto ............................................................................ Ninguno ................................................................................................. Otro (especificar)................................................................................... Q60 Dedicación a la función Tiempo completo................................................................................... Tiempo parcial....................................................................................... Q61 Modalidad de contratación Temporal................................................................................................ Permanente........................................................................................... Q62 Jurisdicción en donde se desempeña Nacional - Administración Central......................................................... Nacional en Organismo Descentralizado.............................................. Regional ................................................................................................ Municipal o local.................................................................................... Otro (especificar)................................................................................... Q63 Función Directiva/jerárquica ............................................................................... Asesoría ................................................................................................ Supervisión/Coordinación ..................................................................... Administrativa........................................................................................ Atención al público ................................................................................ Servicios generales............................................................................... Operativa (especificar) .......................................................................... Otra (especificar)................................................................................... 16-3 17-3 Q64 Personas que usted supervisa o que dependen jerárquica y directamente Q65 Tamaño de la unidad en la que trabaja de usted Ninguna ................................................................................................. 1 a 5....................................................................................................... 6 a 15..................................................................................................... 16 a 30 .................................................................................................. 31 a 50 .................................................................................................. 51 a 100 ................................................................................................ Más de 100 personas............................................................................ Menos de cinco empleados .................................................................. 6-10 empleados..................................................................................... 11-20 empleados................................................................................... 21-50 empleados................................................................................... 50-100 empleados ................................................................................ Más de 100 empleados......................................................................... FIN DE LA ENCUESTA. GRACIAS POR PARTICIPAR. 17-3 18-3 SOLO PARA USO DE LA OFICINA Ministerio / Entidad Gubernamental Fecha de sesión (dia / mes / año ) / / PARA DIGITACIÓN Nombre de Coordinadora Fecha de digitación (dia / mes / año ) / / Número de encuesta Nombre de digitador Número de sesión - 18-3 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 99 ANNEX G, CHARACTERISTICS & DIFFERENCES ACROSS SURVEY VERSIONS CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 100 Table G.1. Characteristics of Survey Versions A, B, and C ATTRIBUTE VERSION A VERSION B VERSION C Number of Pages 20 20 18 Number of Questions 65 65 62 Question Q37 En el último mes ¿usted vió las siguientes situaciones en su institución? Lanza una moneda y contesta sobre la situación según caiga CARA o CRUZ Ahora, la lanza la moneda. Si cae CARA, contesta sobre la situación 1 (CARA). Si cae CRUZ, contesta sobre la situación 2(CRUZ). Escriba “Sí” o “No” Q37: En el último mes ¿usted vió las siguientes situaciones en su institución? Lanza una moneda y contesta sobre la situación según caiga CARA o CRUZ Ahora, la lanza la moneda. Si cae CARA, contesta sobre la situación 1 (CARA). Si cae CRUZ, contesta sobre la situación 2(CRUZ). Escriba “Sí” o “No” Question not included. Situación 1 - (CARA) Mis compañeros de trabajo mantienen una conducta ética. Situación 1 - (CARA) Mis compañeros de trabajo mantienen una conducta ética. Situación 2 - (CRUZ) Mi compañero de oficina viajó en comisión fuera de Asunción. Situación 2 - (CRUZ) Un colega mío fue transferido a otro departamento. Question Q38 Otra vez. Lanza una moneda. Si cae CARA, contesta sobre la situación 3 (CARA). Si cae CRUZ, contesta sobre la situación 4(CRUZ). Escriba “Sí” o “No” Otra vez. Lanza una moneda. Si cae CARA, contesta sobre la situación 3 (CARA). Si cae CRUZ, contesta sobre la situación 4(CRUZ). Escriba “Sí” o “No” Question not included. Situación 3-(CARA) Un compañero de oficina me invitó a salir a almorzar. Situación 3-(CARA) Recibí en mi oficina un miembro de mi familia. Situación 4-(CRUZ) En mi trabajo, aplico respeto los procesos y procedimientos establecidos. Situación 4-(CRUZ) En mi trabajo, aplico y respeto los procesos y procedimientos establecidos. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 101 ATTRIBUTE VERSION A VERSION B VERSION C Question Q39 En el último mes, ¿usted vió las siguientes situaciones en su institución? Responda simplemente “Sí” o “No” sin usar la moneda. Escriba “Sí” o “No” En el último mes, ¿usted vió las siguientes situaciones en su institución? Responda simplemente “Sí” o “No” sin usar la moneda. Escriba “Sí” o “No” En el último mes, ¿usted vió las siguientes situaciones en su institución? Responda simplemente “Sí” o “No” sin usar la moneda. Escriba “Sí” o “No” -Un colega mío fue transferido a otro departamento -Mi compañero de oficina viajó en comisión fuera de Asunción -Un colega mío fue transferido a otro departamento -Me reuní con funcionarios de una dependencia descentralizada -Un compañero de oficina me invitó a almorzar -Me reuní con funcionarios de una dependencia descentralizada -Recibí una carta oficial de otro ministerio del Poder Ejecutivo -Me reuní con funcionarios de una dependencia descentralizada -En mi trabajo aplico y respeto los procesos y procedimientos establecidos -Hubo al menos tres apagones en mi oficina -Mi oficina se quedó sin papel o sin tinta para la impresora -Recibí una carta oficial de otro ministerio del Poder Ejecutivo -Mi oficina se quedó sin papel o sin tinta para la impresora -Recibí una carta oficial de otro ministerio del Poder Ejecutivo -Hubo al menos tres apagones en mi oficina -Recibí en mi oficina a un miembro de mi familia -Hubo al menos tres apagones en mi oficina -Recibí en mi oficina a un miembro de mi familia -Mis compañeros de trabajo mantienen una conducta ética -Mi oficina se quedó sin papel o sin tinta para la impresora -Mi compañero de oficina viajó en comisión fuera de Asunción -Un compañero de oficina me invitó a almorzar CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 102 ATTRIBUTE VERSION A VERSION B VERSION C Question Q46 A continuación de le presenta una lista de cinco situaciones. ¿Cuántas de estas cinco situaciones usted observó o sabe que sucedieron durante el último mes en su dependencia? No necesitamos saber cuáles, sino solamente cuántas de estas cinco situaciones. A continuación de le presenta una lista de cinco situaciones. ¿Cuántas de estas cinco situaciones usted observó o sabe que sucedieron durante el último mes en su dependencia? No necesitamos saber cuáles, sino solamente cuántas de estas cinco situaciones. En la institución en donde trabajo, se aceptan pago extras (coimas), regalos o favores a cambio de otorgar excepciones a las normas, ventajas injustificadas o privilegios. Nunca Rara vez A veces Frecuentemente Siempre No sabe a. Un colega llega tarde o se va temprano y no compensa nunca el tiempo no trabajado. a. Un colega se toma más tiempo de licencia que lo permitido por la norma. b. Se han aceptado pagos extras (coimas), regalos o favores a cambio de otorgar excepciones a las normas, ventajas injustificadas o privilegios. b. Se han aceptado pagos extras (coimas), regalos o favores a cambio de otorgar excepciones a las normas, ventajas injustificadas o privilegios. c. Un usuario de los servicios de su dependencia presenta una queja escrita. c. Un usuario de los servicios de su dependencia termina tan frustrado que deja de interactuar con ustedes. d. Un colega en su unidad no se le asigna suficiente tarea y permanece ocioso al menos un día entero. d. Exceso de empleados asignados a una tarea que podría haberse realizado con menos personas. e. Un colega o un jefe suyo recibe un reconocimiento oficial por un desempeño extraordinario. e. Un colega suyo trabaja sistemáticamente por sobre las horas reglamentarias requeridas. Cuántas: ……. Cuántas: ……. Question Q47 Otra vez, ¿Cuántas de estas cuatro situaciones usted observó, o sabe que sucedieron, durante el último mes en su dependencia? No necesitamos saber cuáles, sino solamente cuántas de estas cuatro situaciones. Otra vez, ¿Cuántas de estas cuatro situaciones usted observó, o sabe que sucedieron, durante el último mes en su dependencia? No necesitamos saber cuáles, sino solamente cuántas de estas cuatro situaciones. Question not included. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 103 ATTRIBUTE VERSION A VERSION B VERSION C a. Un colega se toma más tiempo de licencia que lo permitido por la norma. a. Un colega llega tarde o se va temprano y no compensa nunca el tiempo trabajado b. Un usuario de los servicios de su dependencia termina tan frustrado que deja de interactuar con ustedes. b. Un usuario de los servicios de su dependencia presenta una queja escrita. c. Exceso de empleados asignados a una tarea que podría haberse realizado con menos personas. c. Un colega de su unidad no se le asigna suficiente tarea y permanece ocioso al menos un día entero. d. Un colega suyo trabaja sistemáticamente por sobre las horas reglamentarias requeridas. d. Un colega o jefe suyo recibe un reconocimiento oficial por un desempeño extraordinario. Cuántas:……. Cuantas: ……. Pregunta Q48 ¿Con qué frecuencia ha visto las siguientes situaciones en su institución? Para cada situación, tire el dado, sume el resultado al número que corresponda a su respuesta, y escriba el resultado final a la derecha. Tenga en cuenta que el resultado de cada cálculo debe caer entre 2 y 12. Respuestas posibles Nunca=1+resultado del dado Rara vez=5+resultado del dado Alguna vez=3+resultado del dado Frecuentemente=4+resultado del dado Siempre=2+resultado del dado No sabe=6+resultado del dado ¿Con qué frecuencia ha visto las siguientes situaciones en su institución? Para cada situación, tire el dado, sume el resultado al número que corresponda a su respuesta, y escriba el resultado final a la derecha. Tenga en cuenta que el resultado de cada cálculo debe caer entre 2 y 12. Respuestas posibles Nunca=1+resultado del dado Rara vez=5+resultado del dado Alguna vez=3+resultado del dado Frecuentemente=4+resultado del dado Siempre=2+resultado del dado No sabe=6+resultado del dado ¿Con qué frecuencia ha visto las siguientes situaciones en su institución? Respuestas posibles Nunca=1 Rara vez=5 Alguna vez=3 Frecuentemente=4 Siempre=2 No sabe=6 Ocultar o falsear información Ocultar o falsear información Ocultar o falsear información Obstruir o demorar tramitaciones o expedientes Obstruir o demorar tramitaciones o expedientes Obstruir o demorar tramitaciones o expedientes CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 104 ATTRIBUTE VERSION A VERSION B VERSION C Usar bienes de propiedad del Estado para fines particulares Usar bienes de propiedad del Estado para fines particulares Usar bienes de propiedad del Estado para fines particulares Desviar recursos públicos Desviar recursos públicos Desviar recursos públicos Recurrir a tráfico de influencias para obtener privilegios Recurrir al tráfico de influencias para obtener privilegios Recurrir al tráfico de influencias para obtener privilegios Nombrar a parientes o correligionarios sin el mérito suficiente Nombrar a parientes o correligionarios sin el mérito suficiente Nombrar a parientes o correligionarios sin el mérito suficiente Otorgar viajes, capacitación o gratificaciones arbitraria o injustamente Otorgar viajes, capacitación o gratificaciones arbitraria o injustamente Otorgar viajes, capacitación o gratificaciones arbitraria o injustamente Descuidar patrimonio institucional Descuidar patrimonio institucional Descuidar patrimonio institucional Hurtar patrimonio institucional Hurtar patrimonio institucional Hurtar patrimonio institucional CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 105 Table G.2. Differences between Original Version and CSJ Version DIFFERENCE ORIGINAL VERSION CSJ VERSION Question Q49 divided into Q49a and Q49b Q49. En mi opinión, la corrupción y el fraude administrativo en el último año: Ha disminuido mucho Ha disminuido algo Se ha mantenido igual Ha aumentado algo Ha aumentado mucho No sabe Q49a. En mi opinión, existe corrupción y fraude administrative en el Poder Judicial. Sí No Q49b. En mi opinión, la corrupción y el fraude administrativo en el último año: Ha disminuido mucho Ha disminuido algo Se ha mantenido igual Ha aumentado algo Ha aumentado mucho No sabe Question Q51 divided into Q51a and Q51b Q51. Las auditorías realizadas por el Auditor Interno son efectivas a la hora de limitar el fraude. Muy de acuerdo Un poco de acuerdo Ni de acuerdo ni en desacuerdo Un poco en desacuerdo Muy en desacuerdo No sabe Q51a. Las auditorías realizadas por el Auditor Interno son efectivas a la hora de limitar el fraude. Muy de acuerdo Un poco de acuerdo Ni de acuerdo ni en desacuerdo Un poco en desacuerdo Muy en desacuerdo No sabe CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 106 DIFFERENCE ORIGINAL VERSION CSJ VERSION Q51b. Las auditorías realizadas por el Auditor Interno son efectivas a la hora de prevenir el fraude. Muy de acuerdo Un poco de acuerdo Ni de acuerdo ni en desacuerdo Un poco en desacuerdo Muy en desacuerdo No sabe CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 107 ANNEX H, FOCUS GROUP DISCUSSION PARTICIPANTS CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 108 Table F.1. Participants in FGDs INSTITUTION STAFF CATEGORY NUMBER OF PARTICIPANTS NUMBER OF FEMALE PARTICIPANTS CSJ Management 3 2 CSJ Junior / Technical 6 1 MEC Management 15 8 MEC Junior / Technical 18 7 MIC Management 14 6 MIC Junior / Technical 11 5 MOPC Management 8 4 MOPC Junior / Technical 9 6 MSPBS Management 5 3 MSPBS Junior / Technical 6 3 Total 95 45 CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 109 ANNEX I, FOCUS GROUP DISCUSSION GUIDE CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 110 PRESENTACIÓN Y CONSENTIMIENTO INFORMADO (5 MINUTOS) Buenos días. Me llamo [NOMBRE DEL MODERADOR] y voy a moderar este grupo focal. Trabajo para una organización especializada en investigación social llamada E+E. Me acompañan [NOMBRE DE PERSONA QUE TOMA NOTAS] de E+E y Luis Camacho y Christopher Hansen de NORC en la Universidad de Chicago, un centro de investigación social independiente que se especializa en el análisis y recolección de datos. La Agencia de los Estados Unidos para el Desarrollo Internacional, USAID, está financiado un programa que busca modernizar y fortalecer la administración pública y es implementado por el Centro de Estudios Ambientales y Sociales, CEAMSO. El programa está próximo a concluir, y USAID ha solicitado a NORC realizar una evaluación del Programa. El objetivo principal de la evaluación es extraer lecciones sobre el Programa; saber, por ejemplo, en qué áreas de trabajo se lograron avances importantes y cuáles otras no se lograron los resultados esperados, qué se hizo bien y qué se pudo haber hecho mejor, y qué áreas prioritarias no pudieron ser atendidas. Esta es una evaluación independiente. Nosotros no somos empleados de CEAMSO o USAID así que por favor siéntanse libres de conversar con nosotros de manera abierta y franca. A continuación vamos a hacerles diversas preguntas sobre sus experiencias laborando aquí. No hay respuestas correctas ni incorrectas; solo nos interesa saber su opinión. Dado que tenemos varios temas que discutir y queremos conocer las apreciaciones y opiniones de todos ustedes, les pido de antemano que me disculpen si los tengo que interrumpir en algún momento para que todos en el grupo tengan la oportunidad de participar. Les pido también que no compartan los detalles de nuestra conversación fuera de este grupo. Nuestro equipo valora mucho las apreciaciones y opiniones que van a compartir con nosotros el día de hoy. Tomaremos notas y grabaremos la sesión para tener un registro detallado de nuestra discusión. Para asegurarnos una grabación de alta calidad, pedimos que sólo una persona hable a la vez. Las notas y grabaciones solo serán usadas por el equipo de investigación; no serán compartidas con USAID, CEAMSO, o ninguna otra persona o institución. Usaremos los registros de este y otros grupos focales, junto con otros datos provenientes de entrevistas, encuestas y diversos reportes para producir un informe final para USAID. No vamos a incluir su nombre en el informe, y solo usaremos la información que ustedes nos den de tal manera que no sea posible atribuirla a su persona. La decisión de participar en esta discusión es totalmente voluntaria. Puede negarse a participar en el grupo focal dejando esta sala en cualquier momento así como negarse a responder cualquiera de las preguntas.  ¿Tienen alguna pregunta para mí?  ¿Todos están de acuerdo en participar en este grupo focal? Antes de empezar la grabación, les quería pedir que se presenten diciendo su primer nombre y cuánto tiempo llevan trabajando en el/la [NOMBRE DE LA ENTIDAD]. Ahora voy a encender la grabadora y comenzar nuestra discusión. [Prender la grabadora.] CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 111 ADMINISTRACIÓN DE RECURSOS HUMANOS (10 MINUTOS) Para empezar, quiero conversar con ustedes sobre sus responsabilidades y funciones y las de las áreas u oficinas en las que trabajan. ¿Les explicaron claramente las responsabilidades y funciones de sus cargos cuando comenzaron a trabajar aquí o cuando fueron asignados a un nuevo cargo? ¿Les explicaron cómo sus cargos contribuyen a los objetivos de las áreas u oficinas en las que trabajan? Explorar:  ¿La información que recibieron respecto a las responsabilidades y funciones de su cargo, fueron suficientes y adecuadas?  ¿Existe un manual de funciones para su cargo? ¿El manual es de buena calidad, es decir, claro y completo, pero también conciso? ¿Cuentan con los recursos necesarios para hacer bien su trabajo? ¿Qué recursos les hacen falta? Pensando ahora en la unidad u oficina en la que trabajan en su conjunto, ¿cuenta con los recursos necesarios para cumplir con sus funciones? ¿Qué recursos le hacen falta? Explorar:  ¿Reciben oportunidades o recursos para capacitación?  ¿Las oportunidades o recursos para capacitación se asignan de manera justa? EVALUACIÓN Y MERITOCRACIA (15 MINUTOS) Ahora quiero que hablemos sobre cómo se evalúa su desempeño. ¿Reciben información sobre su desempeño por parte de sus superiores/supervisores de manera periódica? Explorar:  ¿La retroalimentación se hace de manera informal o formal, es decir, escrita y como parte de un proceso estandarizado? ¿Qué tan seguido se realiza el proceso formal?  ¿La retroalimentación se enfoca tanto en áreas positivas (áreas en las que destacan, están haciendo un buen trabajo) como negativas (áreas en las que deben mejorar)?  ¿Qué aspectos de su trabajo son los que más se toman en cuenta para evaluar su desempeño? ¿Asistencia; cumplir con encargos/tares; calidad de productos; dedicación, creatividad e innovación; trato amable a y disposición a colaborar con colegas?  Si alguno trabaja directamente dando servicios a los ciudadanos, ¿se evalúa este aspecto? ¿Sienten que el buen desempeño se refleja en oportunidades para recibir ascensos o incrementos salariales? ¿De qué otras maneras se premia el buen desempeño? Explorar:  ¿Existen reglas claras para ascensos o aumentos salariales? ¿De qué manera se maneja el mal desempeño? ¿Se da oportunidades y recursos para solucionar problemas de fondo? ¿Se dan sanciones? Explorar:  ¿Existen reglas claras para aplicar sanciones? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 112 De manera general, pensando en toda la administración pública, ¿Se podría decir que el reconocimiento del mérito profesional (o meritocracia) es el criterio sobre el cual se deciden los ascensos y otros premios? ¿Se han dado avances importantes en los últimos años en este campo? ¿Cuáles son esos avances? Explorar:  ¿Han escuchado hablar del índice de gestión de personas? ¿En qué consiste? CONTRATACIONES Y ADQUISICIONES (10 MINUTOS) Conversemos ahora sobre las contrataciones de personal y las adquisiciones y contrataciones de bienes y servicios. Por lo general, ¿cómo se hace el reclutamiento de personas en sus unidades/oficinas? Explorar:  ¿Se hacen concursos públicos en todos los casos? ¿En qué casos sí y en qué casos no?  ¿Su institución hace uso del portal Paraguay Concursa?  ¿Cuáles son los criterios que priman en la toma de decisiones, las cualificaciones y la experiencia profesional u otros criterios como redes de parentesco, afiliación política, etc.? Preguntar de manera general, no específica. En general, pensando en la administración pública en su conjunto, ¿consideran que en los últimos años se han dado avances para hacer más transparentes las contrataciones de personal? ¿Cuáles son esos avances? Por lo general, ¿cómo se hacen las adquisiciones y contrataciones en sus unidades/oficinas? Explorar:  ¿Se hacen concursos públicos en todos los casos? ¿En qué casos sí y en qué casos no?  ¿Se planifican de manera adecuada?  ¿Cuáles son los criterios que priman en la toma de decisiones, las propuestas técnicas y económicas u otros criterios como redes de parentesco, afiliación política, etc.)? Preguntar de manera general, no específica. En general, pensando en la administración pública en su conjunto, ¿consideran que en los últimos años se han dado avances para hacer más transparentes las adquisiciones y contrataciones de bienes y servicios? ¿Cuáles son esos avances? PROCESOS, CONTROL INTERNO Y TRANSPARENCIA (15 MINUTOS) Ahora quisiera conversar con ustedes sobre los procedimientos y procesos que realizan y su reglamentación. ¿Los procesos que se ejecutan en sus unidades están especificados y normados? ¿Son claros para ustedes cuáles son los procesos y procedimientos vigentes? ¿Existen procedimientos en los que no se aplican la normativa vigente, ya sea por omisión o desconocimiento? ¿De ser así, cuales son los más frecuentes? CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 113 ¿Existen mecanismos o procesos que sirvan para realizar denuncias respecto a faltas administrativas, de manera anónima y confidencial? ¿Han escuchado hablar del Modelo Estándar de Control Interno del Paraguay (MECIP)? ¿Qué han escuchado? ¿La implementación del MECIP ha repercutido en sus labores? ¿De qué manera? Explorar:  ¿En su institución se han realizado auditorías de procesos de carácter preventivo, es decir auditorías que buscan identificar riesgos de control para tratar de prevenirlos?  ¿En su institución se han realizado auditorías en reacción a denuncias o sospechas de faltas administrativas o de otro tipo? En general, pensando en la administración pública en su conjunto, ¿consideran que en los últimos años se han dado avances para mejor la gestión de procesos y el control interno? ¿Cuáles son esos avances? En general, pensando en la administración pública en su conjunto, ¿consideran que en los últimos años se han dado avances para mejorar la transparencia y reducir la corrupción? ¿Cuáles son esos avances? TEMAS VARIOS (5 MINUTOS) Antes de terminar, quería tocar con ustedes dos temas adicionales. ¿Consideran ustedes que existe igualdad de oportunidades para hombres y mujeres en la administración pública? ¿Por qué sí o por qué no? Explorar:  ¿Se han dado avances en la búsqueda de igualdad de oportunidades en los últimos años? ¿Qué áreas o temas prioritarios deberían ser atendidos por un futuro programa de Democracia y Gobernabilidad de USAID? Puede ser un área o tema sobre el que hemos conversado hoy, pero también cualquier otro. Con esto concluimos la discusión. Muchísimas gracias por su tiempo. CONTRACT NO. GS-10F-0033M / AID-OAA-M-13-00013 USAID.GOV FINAL PERFORMANCE REPORT USAID/PARAGUAY DEMOCRACY & GOVERNANCE PROGRAM | 114 U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523