REPORT ON THE MID-TERM PERFORMANCE EVALUATION OF THE USAID TRANSPARENCY AND GOVERNANCE PROJECT EL SALVADOR Submitted: 24 December 2012 Data collection: 22 October to 26 November 2012 REPORT ON THE MID-TERM PERFORMANCE EVALUATION OF THE USAID TRANSPARENCY AND GOVERNANCE PROJECT EL SALVADOR DECEMBER 24, 2012 Under Contract Number AID-519-O-13-0001 Development and Training Services, Inc. 4600 N. Fairfax Drive, Suite 402 Arlington, VA USA 22203 Researched and prepared by: Melanie Reimer and Olga Nazario This report was prepared by Development and Training Services, Inc. The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Table of Contents Acknowledgements................................................................................................................................... 1 Executive Summary ................................................................................................................................... 2 A. Introduction: Project Context and Description........................................................................ 7 Context................................................................................................................................................... 7 Previous USAID programming ........................................................................................................... 7 Current Project..................................................................................................................................... 8 Partnership for Growth....................................................................................................................... 9 B. Evaluation Purpose and Methodology........................................................................................10 Evaluation Purpose and Objectives.................................................................................................10 Evaluation Methodology ....................................................................................................................10 General approach ...........................................................................................................................10 Evaluation questions.......................................................................................................................10 Data Collection Methods..............................................................................................................12 Limitations........................................................................................................................................12 C. Findings and Conclusions..............................................................................................................13 Most significant performance results..............................................................................................13 Major factors influencing achievement of objectives...................................................................15 Conclusions......................................................................................................................................17 Effects of the implementation approach on results.....................................................................17 Conclusions......................................................................................................................................19 Involvement and contribution of different sectors of society...................................................21 Steps taken by the Government to control corruption.............................................................22 Institutionalization and Operationalization by the Government..............................................23 Conclusions......................................................................................................................................25 Awareness of USAID funding among beneficiaries......................................................................25 D. Overall Conclusions.......................................................................................................................26 E. Recommendations..........................................................................................................................27 Recommendations for Casals in the short-term (during this Project) ...................................27 Recommendations for USAID in the short-term (during this Project)..................................29 Recommendations for USAID in the long-term (following this Project) ...............................30 Annex A: Evaluation Scope of Work.................................................................................................32 Annex B: Evaluation Plan and Methodology .....................................................................................44 Annex C: List of Informants.................................................................................................................58 Annex D: Analysis of Monitoring and Evaluation Indicators.........................................................61 Annex E: Data Collection Tools.........................................................................................................68 Annex F: Matrix of Documents Reviewed........................................................................................79 Annex G: Disclosure of Real or Potential Conflict of Interest.....................................................84 Annex H: Supplementary Recommendations for Youth Programming .....................................87 Table 1: Guide to Evaluation Questions……………………………………………………………………………………. 11 List of Acronyms AECID Spanish Agency for International Development Cooperation (Spanish) APES Journalists Association of El Salvador APIL Law on Access to Public Information ASI Association of Salvadoran Industrialists CAM Central America and Mexico CASALCO Salvadoran Chamber of the Construction Industry CSO Civil society organization dTS Development and Training Services, Inc. EMPREPAS Empresarias Salvadoreñas por la Responsabilidad Social FESPAD Fundación de Estudios para la Aplicación del Derecho FPEMP Fiscal Policy and Expenditure Management Project (USAID) FUNDE Fundación Salvadoreña para el Desarrollo FUSADES Fundación Salvadoreña para el Desarrollo Económico y Social FY Fiscal Year GET Government Ethics Tribunal GOES Government of El Salvador IACAC Inter-American Convention against Corruption IAPI Institute on Access to Public Information IDB Inter-American Development Bank ISD Iniciativa Social para la Democracia JCAP Joint Country Action Plan (under Partnership for Growth) LAPOP Latin American Public Opinion Project LIDERA Private sector pro-ethics organization that brings together primarily law offices and universities MCC Millennium Challenge Corporation MCP Municipal Competitiveness Project (USAID) MESICIC The Mechanism for Follow-Up on the Implementation of the Inter-American Convention against Corruption M&E Monitoring and Evaluation MoP Ministry of Public Works OAS Organization of American States OIR Office of Information and Response PFG Partnership for Growth PMP Performance Management Plan SOW Scope of Work SSTA Under-Secretariat of Transparency and Anti-corruption TAG Transparency and Governance TI Transparency International UCA Universidad Centroamericana "José Simeón Cañas", El Salvador UAIP Unit for Access to Public Information UN United Nations UNDP United Nations Development Programme US United States of America USAID United States Agency for International Development USG United States Government VMT Vice-Ministry of Transport 1 Acknowledgements The dTS evaluation team would like to thank all those who generously contributed their time and shared their information and ideas with us, during the course of this evaluation. The high level of interest and support from USAID, Casals & Associates, Inc. and all informants – in El Salvador and elsewhere – was above and beyond our expectations, and allowed us to quickly gain a thorough understanding of the project and context. The participation of each and every individual added value to the evaluation, which it is hoped will ultimately contribute to the advancement of transparency and ethics and the fight against corruption in El Salvador. 2 Executive Summary Context The Transparency and Governance Project was designed by the United States Agency for International Development (USAID) in a context in which there were still very limited openings to publicly discuss corruption and transparency problems, in spite of perceptions of pervasive corruption across society. The election of 2009 brought new opportunities for the Government of El Salvador (GOES) and other stakeholders to address this situation. Those circumstances provided fertile ground to support significant advances, though progress has become more difficult over time. The Project The Transparency and Governance Project (“the Project”) is implemented by Casals & Associates, Inc. (“Casals”) through a $7.9 million contract (including option years up to FY2014) focused on controlling corruption in the use of public funds. The contract was signed in May 2009, with a base period that ended in September 2010. The projected end date is March 2014. The Project has two main objectives: 1) to continue efforts to support various levels of the Salvadoran government to increase responsiveness and accountability towards its citizens, through increasing transparency in government and ethical behavior by public officials, reducing conditions for corruption in public administration, strengthening the link between citizens and government, and thereby increasing confidence in the system; and 2) to continue to foster civil society participation, particularly the private sector, in promoting transparency, accountability and control of corruption in the use of public funds. The main client of the Project is the GOES, with a primary focus on the executive branch, particularly the Under-Secretariat of Transparency and Anti-corruption and the Government Ethics Tribunal. The Project has also worked with the private sector, as its secondary target group, and to a lesser extent with civil society. Within the framework of the Partnership for Growth (PFG) signed by the GOES and the United States Government, actions under two institutional strengthening goals (#4 and #6) in relation to the crime and insecurity constraint have been allocated to the Project for implementation. The Evaluation The purpose of this mid-term performance evaluation was to help USAID analyze the Project’s activities, results and approaches to date, in order to use the information for future project design and implementation in the areas of anti-corruption, transparency and governance in El Salvador. The evaluation was carried out by Development and Training Services, Inc. (dTS) under contract with USAID; it covered implementation from the start of the Project in May 2009 through to September 2012. The key evaluation questions were defined by USAID as follows: 1. What recommendations can be made to realign or adjust the Project during the remaining time to optimize chances for success? a. What have been the most significant performance results to date? b. What were the major factors influencing the achievement or non￾achievement of Project objectives? c. How has the current implementation approach positively and/or negatively affected Project results? 3 2. To what extent have different sectors of society been involved and contributed to the Project, including: the Government of El Salvador, civil society, the private sector, other international donors and the media? a. Who should USAID and Casals work with during the remaining time to improve chances of success? 3. To what extent has the GOES taken steps to control corruption during Project implementation? a. How have anti-corruption and transparency initiatives been institutionalized or operationalized in GOES processes? 4. How aware are beneficiaries that this Project is funded by USAID? The evaluation was largely qualitative in nature; it included review of over 50 key documents, interviews with 33 key informants, four focus groups with 31 participants, as well as several site visits. Data collection took place in San Salvador and Usulután from late October until mid-November 2012, followed by a validation workshop with USAID and Casals staff. Key Findings A number of positive or negative factors have influenced the achievement of objectives by the Project, including:  The overall high credibility of the US Government, USAID and Casals in the view of Salvadoran stakeholders.  The change of government that coincided with both the start of the project and a concerted push for reform from civil society.  Lack of access to updated in-depth analysis of corruption in the country.  High levels of politicization and polarization permeating the society.  Declining political will for tackling corruption.  Low level of interest on the part of the private sector.  Diminished impetus from civil society over time. The implementation approach of the Project has also had an impact on results, in the following ways:  Innovativeness, respectful relationships, and dedicated and skilled personnel have all been advantages.  Some human resource constraints have been observed, such as lack of ongoing professional development of the staff and over reliance Hiram Morales for numerous public activities.  The Project design focused on the national government and private sector, which had the consequence of minimizing attention to municipal governments and rural areas in general. The design also meant that the traditional civil society organizations were largely sidelined in favor of business-related organizations.  The approach was very flexible, in accordance with the Task Order, which resulted in work plans that were more akin to guidelines, as compared to concrete activity plans, and a regularly changing menu of activities.  The allocation of several Partnership for Growth lines of action early in 2012 has expanded the focus of the Project beyond what was originally contemplated.  Ethics promotion has been a major emphasis with government and other stakeholders; to that end, a wide range of short awareness-raising events has been conducted, of which the results are difficult to determine. 4  Communication and interaction of the Project with other relevant US Government￾funded projects has been minimal; similarly, links with international actors are weak. The GOES has taken significant strides towards the promotion of transparency, accountability and ethics since 2009, although its direct action on corruption has been less evident. Key advances have included:  Strengthening of the legal framework for anti-corruption  Establishment of the Under-Secretariat of Transparency and Anti-corruption  Establishment of the unified online transparency portal  Systematized accountability reporting by many government institutions. The government has made variable progress towards institutionalizing and operationalizing initiatives supported by the Project, as follows:  The Government Ethics Tribunal and Ethics Commissions are moderately institutionalized and operational, but only in the capital.  The Offices for Information and Response under the new Law on Access to Public Information are also moderately institutionalized and operational, though few have been set up beyond the capital.  The Institute for Access to Public Information is not institutionalized and not yet operational.  The Under-Secretariat of Transparency and Anti-corruption is highly operational, and the government accountability processes are moderately operational in the executive branch—but they are only institutionalized in national policy. Generally, there is a high level of awareness among key Project partners in government, the private sector and civil society that the Project is funded by USAID, which also extends to indirect partners, such as Ethics Commissions, and the international community. Key Conclusions Building on the solid foundation and credibility established by the previous USAID transparency and governance program, the current Project has significantly contributed to placing the topics of ethics, anti-corruption, transparency and accountability firmly on the national agenda. In the conducive environment following the election of Mauricio Funes, it has provided invaluable assistance to the GOES in passing and implementing the Law on Access to Public Information, the new Government Ethics Law and strengthening the Government Ethics Tribunal; and attempting to instill a culture of ethics among public officials. It has also nudged several professions, including lawyers and journalists, into adopting codes of ethics to guide the behavior of their members, including how they should interact with the GOES. Despite the initial hope for dramatic changes, the Project has faced many challenges due to the politicization and polarization of most GOES institutions and organizations. The level of political will has not been consistent, resulting in unfulfilled promises such as the failure to create the Institute for Access to Public Information, and a prolonged suspension of the work of the Government Ethics Tribunal. There has been no solution to the on-going politicization of the Court of Accounts, which has closed off an important avenue for enhancing oversight of public administration. Civil society’s exhaustion from unending controversy over the justice sector and decreasing access to support from USAID and other donors, among other factors, seems to have diminished their capacity and will to advocate for more and faster reforms and engage the population in these efforts. Moreover, the private sector—a traditional partner of previous 5 Salvadoran governments—has undermined its own capacity to be an effective interlocutor due to ongoing conflicts with the Funes administration, and with some exceptions, shown little genuine interest in advancing transparency and combating corruption. Until there is real commitment from the sector for transparency and anti-corruption initiatives, further substantial investment by the Project is not likely to produce sustainable results. Nevertheless, achievements have been significant both for the Funes administration and initiatives supported by the Project. Salvadorans interviewed seem confident that they are on the right track to reduce corruption, for example through the establishment of new government institutions and initiatives such as the Offices for Information and Response, which are dedicated to responding to citizens’ demands. The legal framework for anti￾corruption, while not complete or perfect yet, has improved; there are new and better laws, regulations and procedures on the books. El Salvador still has a long way to go to reduce its levels of impunity, strengthen its institutions and ensure the sustainability of these achievements. Achieving optimal results in the promotion of transparency and fight against corruption will only be possible through balanced engagement by USAID and the Project with government and non-government sectors, and through building in-depth capacity in key entities. The time is ripe for the Project to tighten its focus on consolidation of gains and institutionalization of positive changes, through strategically placed support to the government and civil society during its crucial final year of implementation. Stronger monitoring tools and practices will ensure that the results and lessons of this Project can be captured clearly and capitalized upon in the future. A major challenge remains—extending the benefits of reforms and new services to the majority of the population, whose daily lives are far removed from the capital and national government, but who continue to suffer the effects of corruption on a regular basis. By prioritizing better coordination and collaboration, and by making anti-corruption a cross￾cutting theme of all USAID programming, it will be possible to seek synergies, avoid potential duplication of efforts, and achieve greater and more sustainable impact across the country. Key Recommendations 1. Recommendations for Casals in the short term The Project will end in March 2014, which means that activities will be winding down from December 2013 onwards; that leaves approximately one year for implementation of this set of recommendations.  Establish and cultivate links with other relevant actors and projects, including those funded by US Government and others.  Engage more actively with civil society, to ensure a balanced approach and encourage crucial advocacy for improvements in the legal framework, establishment of the Institute of Access to Public Information, and other reforms.  Provide strategic assistance to the executive branch of GOES, emphasizing the obligations of both sides, consolidating gains, building in-depth capacity, and ensuring that the roles of key entities such as the Under-Secretariat of Transparency and Anti-corruption and the Government Ethics Tribunal are kept from overlapping .  Work with the USAID Economic Growth Office and its implementing agencies to identify suitable approaches to engagement with the private sector. 6  Engage a wide range of relevant experts from the region and beyond to contribute expertise and new ideas, while investing in professional development of staff.  Strengthen monitoring and evaluation within Project and with partners.  Develop activities that will generate visible and tangible short-term wins in the final phase of the Project, to inspire the population and combat impunity.  Support the Under-Secretariat of Governance and Modernization in its efforts to promote a civil service reform. 2. Recommendations for USAID in the short term  Ensure coordination among relevant USAID projects, as a top priority.  Support interaction among the Salvadoran institutions that are key to transparency and anti-corruption in order to improve coordination and resolve areas of apparent overlap of mandates.  Ensure that all USAID projects and key Salvadoran partners are aware of the exact implications of the Partnership for Growth for their work; this is especially important for the private sector.  Update the Performance Management Plan and Project M&E Framework in collaboration with Casals.  Conduct an updated corruption assessment that can support the work of USAID implementers and others. 3. Recommendations for USAID in the long term  Make transparency and anti-corruption a crosscutting theme in the El Salvador mission.  Dedicate resources to transparency and anti-corruption initiatives and entities at municipal level and in rural areas, through both local government and civil society  Simplify M&E systems and indicators.  Consider placing more conditions on assistance and take care not to support institutions known to be corrupt.  Incorporate gender considerations in new programming.  USAID should consider providing more specific anticorruption training to GOES officials and civil society organizations. 7 A. Introduction: Project Context and Description Context The Transparency and Governance Project (“the Project”) subject to this evaluation was designed by the United States Agency for International Development (USAID) in a context in which there were still very limited openings to publicly discuss corruption and transparency problems, in spite of perceptions of pervasive corruption across society. The 2009 Scope of Work described the situation as follows: “The culture of Salvadoran democracy is threatened by unfulfilled expectations for increased economic opportunities, citizens’ overwhelming concerns with violence and crime, the high level of political polarization that impedes public officials from adequately confronting pressing national challenges, and the limited credibility of key democratic institutions and their processes. There also exists dissatisfaction with the lack of integrity and transparency in public decision makers and processes, particularly in areas related to management of public funds. Salvadorans continue to cite corruption as one of the main reasons they do not trust their government.” Those difficulties were compounded by a complicated public procurement process, lack of access to public information, non-transparent legislation, weak internal control systems and a highly politicized budget cycle. The election of 2009 brought new opportunities for the Government of El Salvador (GOES) and other stakeholders to address this situation. During the election campaigns, all presidential candidates publicly committed to a set of transparency and anti-corruption initiatives if elected. After the elections, the Mauricio Funes administration demonstrated early political will to make changes, naming well-known anti-corruption leaders to key positions and creating a Under-Secretariat for Transparency and Anti-corruption. Civil society was also actively engaged in transparency and accountability issues, with some support from the international community. Those circumstances provided fertile ground for the Project to support significant advances, though progress has become more difficult over time. Regional circumstances also complicate the task of fighting corruption as growing transnational organized crime affecting Central America thrives on corruption and inevitably permeates critical institutions, especially in the legislative and judicial sectors. Previous USAID programming USAID implemented the USAID/CAM Anti-corruption, Transparency and Accountability Program from 2004 to 2009 to assist the government, private sector, civil society and other key actors to work together to slow and reverse the spread of corruption in each participating country in the Central American region. From 2006 to 2009, that program supported anti-corruption reforms in El Salvador at the national and local levels, and built advocacy partnerships with some civil society organizations (CSOs). During the program period, the National Assembly passed the Government Ethics Law, and the government established the Government Ethics Tribunal (GET), the Consumer Defense Agency increased its capacity to enforce laws for protecting consumers, and support was provided to development of the government’s report to the Organization of American States (OAS) for the second round of compliance with the Inter-American Convention Against Corruption (IACAC). In the justice sector, initiatives were carried out with the Attorney General’s Office to strengthen its Anti-Corruption Unit and with the Public Defender’s Office to enhance internal procedures. As well, the National Council for the Judiciary was assisted to expand 8 ethics training and to develop judicial transparency indicators, and the Supreme Court was supported to develop a code of ethics for judges. At the local level, USAID provided technical assistance to promote decentralization and municipal transparency. Much of the work during that program provided important foundations for the current Project. Activities in El Salvador under both USAID interventions have been implemented by Casals & Associates, Inc. as lead contractor. Current Project The USAID Transparency and Governance Project for El Salvador, under Contract DFD-I￾01-08-00069-00 with Casals & Associates, Inc. (“Casals”) is a $7.9 million project (including option years up to FY2014) focusing on controlling corruption with the Government of El Salvador (GOES) and civil society in the use of public funds. The contract was signed in May 2009, with a base period that ended in September 2010. The Project has four consecutive option periods; the second option year had just been completed as this evaluation commenced in October 2012. The Project has two main objectives: 1) to continue efforts to support various levels of the Salvadoran government to increase responsiveness and accountability towards its citizens, through increasing transparency in government and ethical behavior by public officials, reducing conditions for corruption in public administration, strengthening the link between citizens and government, and thereby increasing confidence in the system; and 2) to continue to foster civil society participation, particularly the private sector, in promoting transparency, accountability and control of corruption in the use of public funds. The development hypothesis of the Project is: If the Government of El Salvador fights corruption through improved transparency and accountability, then it will not only consolidate the country’s democracy and improve institutional legitimacy, but it will also positively impact El Salvador’s effectiveness to combat crime and insecurity and contribute to economic growth. The main client of the Project is the GOES, with a primary focus on the Executive branch, particularly the Under-Secretariat of Transparency and Anti￾corruption (SSTA) and the Government Ethics Tribunal (GET). The Project has also worked with the private sector, as its secondary target group, and to a lesser extent with civil society. The following three main areas of activity and corresponding sub-activities were defined by USAID in the Scope of Work of the Project. Activity 1: Establishing and promoting good governance and anti-corruption practices within the government Sub-activities: 1.1: Assist the new government to implement transparency, ethics and anti￾corruption measures as related to priorities proposed in its government platform (Plan de Gobierno 2009-2014) 1.2: Continue support to the Government Ethics Tribunal 1.3: Support the GOES to implement the new Transparency and Access to Public Information Law 1.4: Auditing the Court of Accounts Activity 2: Engaging Civil Society in the Fight against Corruption Sub-activities: 2.1: Private sector involvement in the fight against corruption 9 2.2: Establishment of the Transparency International (TI) Chapter in El Salvador 2.3: Independent Court of Accounts 2.4: New generation leadership Activity 3: Provide technical assistance and training in support of other openings for relevant transparency initiatives designed to curb corruption Sub-activities: Have been defined during project implementation Partnership for Growth The Partnership for Growth (PFG) Joint Country Action Plan (JCAP) 2011-2015 for El Salvador and the United States, upon being signed in November 2011, became one of the most important strategy documents governing USAID’s programmatic focus in El Salvador. The JCAP identified two key constraints to growth: crime and insecurity and low productivity in the tradables sector. Within the framework of the PFG and JCAP, several actions under two institutional strengthening goals in relation to the crime and insecurity constraint have been allocated to the Project for implementation. These actions fall within the overall objectives of the original Task Order of the Project, although some areas of work were not specifically anticipated by that Task Order.  Goal #4: Facilitate economic growth by ensuring El Salvador’s labor force is protected from crime while transiting to and from work, and ensuring that the public transportation service providers serving the labor force are protected from crime. Two specific Lines of Action under this goal correspond to this Project: the USG will support GOES as it reviews the current regulations of the public transportation sector and identifies areas to improve transparency and accountability, and USAID will provide technical assistance to help increase the transparency and accountability of the public transport system.  Goal #6: Professionalize El Salvador’s civil service and enhance public confidence in the government. Two specific Lines of Action under this goal correspond to this Project: providing technical assistance to promote and facilitate comprehensive civil service reform (with reform of the law being the specific task assigned to the Project) and supporting GOES in increasing responsiveness to accountability towards its citizens through increasing transparency in government and ethical behavior by public officials. These actions and goals relate directly to USAID/El Salvador’s efforts to promote transparency and governance, particularly through its current Project, but will also direct future programming strategies. Project activity specifically directed towards these two goals commenced in early 2012.1 1 Due to the recent start of these activities, it was premature to attempt to analyze results in relation to those goals. Nevertheless, they have been taken into account by the evaluation, especially in the area of recommendations for future programming. 10 B. Evaluation Purpose and Methodology Evaluation Purpose and Objectives The purpose of this evaluation was to help USAID analyze the Transparency and Governance Project’s activities, results and approaches to date in order to use the information for future project design and implementation in the areas of transparency and governance in El Salvador. This mid-term performance evaluation (as defined in USAID’s Evaluation Policy) was carried out by Development and Training Services, Inc. (dTS), whose evaluation team composition is described in Annex B. The evaluation covered implementation from the start of the Project in May 2009 through to September 2012. Specifically, the objectives of this evaluation were:  To analyze whether USAID is using the right actions and initiatives to control corruption and promote transparency in El Salvador  To assess whether the project has achieved expected results during the base period, as well as option years 1 and 2  To make recommendations on adjustments for optimal implementation during the Project’s remaining time  To make recommendations on with whom USAID and Casals should work to improve chances of success  To gauge the GOES’ response to the Project in terms of anti-corruption and transparency initiatives Evaluation Methodology General approach This performance evaluation was largely qualitative in nature, and based on extensive document and data review, personal interviews with key informants, focus groups and site visits and observations. Informants from various sectors were identified based on suggestions from USAID, from Project staff, and from other informants, complemented by the personal knowledge of the evaluators. Data collection took place in San Salvador and Usulután from late October until mid-November 2012. The original Scope of Work for the evaluation is attached as Annex A, while the full evaluation plan and methodology is at Annex B. In conducting the analyses and reaching conclusions, the evaluators have triangulated findings from documents, interviews, focus groups and site visits. Results were also analyzed in relation to the Performance Management Plan (PMP) of the USAID Democracy and Governance Office and the M&E Framework developed for the Project. Evaluation questions In accordance with USAID guidelines, the evaluation has focused on descriptive and normative questions, including: (a) what did the Project achieve during the implementation period; (b) how is the Project being implemented? c) how is the Project perceived and valued? More specifically, the evaluation addressed the four major and five associated questions and sub-questions set forth in the Scope of Work (SOW) for the evaluation, which are addressed in the report as set out in Table 1 below. 11 The Evaluation SOW written by USAID listed the evaluation questions in priority order, starting with the most important question, as directed by USAID’s Evaluation Policy; however, this order did not follow a logical sequence that translated easily and directly into an evaluation report format. This Evaluation Report presents the findings, conclusions and recommendations corresponding to the evaluation questions in a logical order, with the recommendations at the end of the document. Table 1 is a guide to show how the Evaluation SOW questions (Column 1) evolved into dTS’ Evaluation Plan methodology (Column 2 and Annex B), and finally were presented in this Evaluation Report (Column 3). Table 1. Guide to Evaluation Questions Column 1 Column 2 Column 3 Question # in USAID’s Evaluation SOW Question # in dTS’ Evaluation Plan2 Most Relevant Section of the Evaluation Report 1. What recommendations can be made to realign or adjust the Project during the remaining time to optimize chances for success? 4 Recommendations, page 27 a. What have been the most significant performance results to date? 1 Most significant performance results, page 13 b. What were the major factors influencing the achievement or non-achievement of Project objectives? 2 Major factors influencing achievement of objectives, page 15 c. How has the current implementation approach positively and/or negatively affected Project results? 3 Effects of the implementation approach on results, page 17 2. To what extent have different sectors of society been involved and contributed to the Project, including: the Government of El Salvador, civil society, the private sector, other international donors and the media? 5 Involvement and contribution of different sectors of society, page 21 a. Who should USAID and Casals & Associates work with during the remaining time to improve chances of success? 6 Recommendations, page 27 3. To what extent has the GOES taken steps to control corruption during Project implementation? 7 Steps taken by the GOES to control corruption, page 23 a. How have anti-corruption and transparency initiatives been institutionalized or operationalized in GOES processes? 8 Institutionalization and operationalization by Government, page 23 4. How aware are beneficiaries that this Project is funded by USAID? 10 Awareness of USAID funding among beneficiaries, page 25 A table of the methods and data sources relied upon to answer each question can be found in the Evaluation Plan in Annex B. 2 Question #9 in dTS’ Evaluation Plan (How do stakeholders value and/or perceive the Project’s results?) does not correspond to a specific question in the Scope of Work, and it is not addressed separately in this report, although it did help to guide data collection and analysis. 12 Data Collection Methods  Desk/Document Review: Before arrival in El Salvador, the evaluation team conducted a desk review of relevant Project documents made available by USAID/El Salvador. The document review then expanded to include other materials identified during the evaluation. In total, over 50 documents were reviewed. All documents subject to substantive review by the evaluators are listed in Annex G, indicating which evaluation question each document was used to address.  Key Informant Interviews: The evaluation team conducted 31 interviews involving 33 individuals (25 male and 8 female), primarily in person as well as several by telephone. These key informants included the 12 organizations mentioned in the Scope of Work, such as key personnel of the Sub Secretariat of Transparency and Anti-corruption, Government Ethics Tribunal, Ministry of Public Works, Court of Accounts, and Attorney-General’s Office, as well as others identified by the evaluation team. Categories of informants were developed to ensure a variety of perspectives; they included USAID/El Salvador, Casals staff and consultants, GOES officials, civil society, media, private sector, international agencies and independent experts. The list of informants is included in Annex C, and interview guides can be found in Annex E.  Focus Groups: The evaluators conducted four focus group discussions with a total of 31 Project stakeholders (19 male and 12 female), such as mayors of small towns and members of an Ethics Commission. The list of focus groups appears in Annex C. These discussions explored the interaction of the participants with the Project, their views of its activities and results, and their recommendations. Each group had between six and ten participants. Focus group protocols or question guides were tailored to the composition of each group; an example is attached in Annex E.  Site Visits: In order to directly observe the work of the Project and have informal exchanges with relevant stakeholders, the evaluators made six site visits to venues ranging from the “House of Transparency” at the Ministry of Public Works (MoP) to a workshop for civil society representatives. The list of site visits is found in Annex C. In the case of each site visit, conversations were held with staff or participants at the venue and observations made in relation to how the support of the Project was being utilized.  Validation Workshop: At the close of data collection, the evaluation team organized a workshop to discuss emerging findings and their implications for the Project and future USAID programming. Participants were representatives of Casals and USAID/El Salvador. This participatory workshop assisted the evaluators in interpreting data, refining their findings, pinpointing gaps and new directions, and assessing the feasibility of recommendations. Limitations The evaluation team has experienced the following limiting factors in its data collection and analysis:  Time constraints on data collection, relative to large volume of project documentation and high number of activities over three years.  Flexible nature of the project has involved a number of changes to plans, priorities and activities, which resulted in a complex situation to follow and evaluate.  Lack of concrete data on actual levels of corruption in El Salvador. 13  Unavailability of key informants from judiciary and media during evaluation.  Project semi-annual reports are activity-focused, and analysis is weak in terms of relating activities to results and results to objectives. Reports do not discuss changes in work plans nor targets for monitoring and evaluation (M&E) indicators.  Little information has been gathered by the Project to monitor results of training and awareness-raising, which has limited the ability to make substantive findings on the quality, relevance and impact of training. Due to contractual limitations and restrictions on how USAID had to report progress on indicators to Washington, DC, some indicators and targets in the official PMP and the M&E Framework have not been updated since the beginning of the Project. As a result, some “actual” data on indicators bears little relation to the “target” numbers, which has constrained the evaluation’s use of indicator data in analyzing yearly and overall results. C. Findings and Conclusions In this section, responses are presented to seven of the nine evaluation questions, each in a separate sub-section. Where both findings and conclusions were required to respond to a particular question, the conclusions are presented separately, following the findings for that question. The two remaining questions are answered in the Recommendations section on page 27. Most significant performance results Since its commencement in May 2009, the Transparency and Governance Project has contributed to the achievement of a range of results. These include increased transparency and accountability awareness among various Salvadoran target groups3 , establishment of new mechanisms and procedures, and improvements in the anti-corruption legal framework of the country. The Project increased levels of awareness and openness among Salvadoran citizens and public officials about transparency and anti-corruption and helped place these topics firmly on the public agenda and in the mass media. This awareness is especially prevalent among urban residents, according to informants, as compared to rural citizens with less access to information. Recognition of the importance of transparency and ethics has been heightened among several key professions, including journalists and lawyers, through capacity building and development of codes of ethics. The Project also boosted awareness of ethics and transparency among certain key justice institutions, including through a draft code of ethics for the Judiciary and draft Law on Probity.4 In addition, civil society organizations and public officials have greater awareness of the responsibility of government entities to be actively accountable to the public for their actions and spending. Most executive institutions are now reporting publicly, with various degrees of efficacy, on their activities. A noted case is that of the construction and public works sector, widely seen as one of the most corrupt in the country, which has shown increased awareness and will to tackle corruption through engagement with the Project. 3 There is no quantitative survey data on the specific levels of awareness or capacity mentioned in this section, so the evaluation has relied on the views of informants and documentary review in making these findings. 4 Work of the Project with the justice sector was suspended by USAID in order to avoid overlap with another project focused on rule of law, so these initiatives were never finalized. 14 In terms of the legal framework for transparency and anti-corruption, the Project contributed to passage of the Law on Access to Public Information (APIL) through support to drafting and advocacy by civil society and other activities, which began under the predecessor program. Subsequently, implementation of the APIL has advanced considerably with the support of the Project, primarily through the Under-Secretariat of Transparency and Anti-corruption (SSTA). Important steps have included establishment of at least 80 Information and Response Offices (OIRs), dissemination and socialization of the law, and two public reports on compliance with the law. The work with the SSTA, which has played a key role in advancing the implementation of several laws and mechanisms, has generally contributed to increased availability of and access to public information. The standardization of executive branch websites with official information and creation of the unified government transparency portal were major milestones achieved with Project support. Most recently, the Project has lent support to creation by SSTA of the Virtual Government Training Program, which aims to extend on￾line training opportunities in ethics, transparency, anti-corruption and related topics to a significant portion of public servants in the country. The knowledge base about the APIL, access to public information in general and transparency has been enhanced through establishment and implementation of a 4-month diploma course at a prominent national university. This course, offered at the Universidad Centroamericana (UCA), has primarily targeted central government officials (especially Public Information Officers) to date but also included some civil society representatives. The Project has built capacity of the Government Ethics Tribunal (GET) and the Ethics Commissions in various central government institutions through training, technical assistance, and development of tools and procedures, among other interventions. Reform of the Law on Government Ethics in early 2012, which has strengthened the legal framework considerably, was an important achievement supported by the Project. The new law gives more authority to the GET to pursue cases on its own initiative and allows for anonymous complaints and fines up to nearly US$9000. Transparency in public works procurement has been enhanced through support to development of 26 integrity pacts and other tools, reprinting of the construction industry code of ethics, facilitation of collaborative relations between the industry and the Ministry of Public Works, and an observatory to monitor public works projects. In addition, the Project has made a contribution to increasing the capacity and profile of civil society organizations in order to be credible interlocutors with government and influence legislation and policy, building on the strong foundations laid by the previous program. Work in this area included support to the Foundation for National Development (FUNDE) becoming the national chapter of Transparency International. 15 Major factors influencing achievement of objectives A number of factors had a major influence on the Project’s ability to achieve its objectives, including the significant results outlined above. The credibility of the United States Government (USG) and of USAID among key stakeholders and the public of El Salvador was remarked upon by various informants. USAID is well known and respected as a leading proponent of transparency and accountability initiatives. As well, the previous transparency and governance program had laid a strong foundation for the current Project. Casals and its Chief of Party had already developed strong relationships with key actors, an understanding of the political and cultural context of El Salvador, and a high level of credibility. The Chief of Party is widely recognized as a diplomatic interlocutor who is able to deal with individuals and entities of all political inclinations in a balanced way and skillfully facilitate cooperation even among diverse stakeholders. The election of a new government shortly before the Project commenced paved the way for certain legislative and other reforms, especially given the initial political will for transparency demonstrated by President Mauricio Funes. The change of government coincided with a strong push from a core group of civil society organizations (Grupo Promotor) that already had relevant capacity and had been advocating for a law on access to information and other transparency measures. The new government appointed certain individuals known as Analysis of Results according to PMP and M&E Framework The outputs and results of the Project were also analyzed by the evaluation in relation to the PMP of the USAID/El Salvador Democracy and Governance Office and the more detailed Project M&E Framework. The two indicators on the Strategic Objective level relate to the Corruption Perception Index published by Transparency International and the level of perceived government responsiveness as measured by the biennial surveys of the Latin American Public Opinion Project (LAPOP). With respect to the Index, El Salvador has seen a general downward trend in perception of corruption since 2005, but it has been stable with slight fluctuations from 2009 to 2011. A minor improvement was observed between 2011 and 2012, as the rating climbed from 3.4 to 3.8 (with 10 being the least corrupt). The target set for 2012 was 3.4. As for government responsiveness, the combined proportion of Salvadorans who consider the government to be either “a little” or “a lot” responsive has been fluctuating between 34% and 39% since 2004 baseline, but there is no new data since 2010. It is not possible to draw any conclusions regarding the impact of the Project on these indicators. With respect to activities focused on the government, the indicator data shows that the government has taken various positive steps showing commitment, that about 10,000 participants have been reached by training activities. As for activities with civil society and private sector, key results captured by the indicators are the passage of the APIL, over 10,000 participants in training and awareness raising, and various initiatives and mechanisms supported, though measurement has proven difficult due to unclear definitions. The majority of targets have been met or exceeded by reported results of the Project, although some targets were not updated over time. No indicators were established for Activity 3. A detailed analysis by the evaluation of progress achieved against indicators and measurement challenges is available in Annex D. 16 champions of transparency (for example, Minister of Public Works Gerson Martinez and Undersecretary of Transparency and Anti-corruption Marcos Rodriguez) to influential positions, which they have used to push for greater transparency and ethics and to combat corruption. There were other factors, primarily related to the context of the Project, which have hampered or limited the achievement of objectives. The evaluation team found that the base of in-depth knowledge and analysis of the characteristics of corruption in the country was not optimal; the most recent comprehensive corruption assessment conducted by USAID in 2010 was not available to the Project because the final version was never approved for distribution. Generally, the high level of politicization and polarization that permeates the country makes it difficult to reach compromises and get bipartite or multipartite support for initiatives and key appointments. This is especially true in the case of politically sensitive reforms, such as transparency and anti-corruption actions. The evaluation observed a declining level of high￾level political will in this area, as evidenced by the long delay in appointment of commissioners to GET following the law reform; failure to establish the Institute for Access to Public Information (IAIP) as mandated by the law; failure to appoint a new Attorney General; and inadequate budgets for transparency and anti-corruption entities including the SSTA, GET and IAIP. The continued inefficacy of the Court of Accounts, the body primarily responsible for oversight of the public administration, detracts generally from the credibility of anti￾corruption initiatives and limits the key institutions with which the Project can interact effectively. As well, progress has been constrained by the lack of a professional and highly qualified public administration staff with job security, which could provide continuity and sustainability to the Project’s initiatives. Informants have observed a reduced level of pressure by some civil society organizations (CSOs) and groupings that were very active and outspoken at the outset of the Project. This is attributed to various factors, including financial constraints, exhaustion from previous campaigns, and a closer affiliation with the new government, including through former civil society activists now in government. Disagreement between the leadership of FUNDE and the SSTA in relation to a Project￾supported initiative has led to a rift between those entities and made it difficult for the Project to work simultaneously with these two crucial actors in transparency and anti￾corruption. This situation contributed to the Project focusing on the SSTA and ceasing to work with FUNDE, which has recently become the national chapter of Transparency International. The Project has also faced challenges arising from serious divisions between a large segment of the private sector and the current government, which has stymied efforts by internal and external actors to promote cooperation between the two sectors, including in the areas of transparency and anti-corruption. Overall, the Salvadoran private sector has demonstrated a low level of interest in transparency and anti-corruption, failing to take any real initiative. It appears that Casals headquarters has had a fairly low level of engagement in this Project and high turnover in relevant HQ staff, including among their anti-corruption experts, since the company was acquired by DynCorp International about one year ago. This has limited substantial technical assistance and oversight of the Project since that time. 17 Conclusions The credibility of the USG and USAID in El Salvador has allowed the Project to enjoy significant latitude in its activities and to gain a high level of cooperation, especially with the executive branch. The Project was also able to capitalize on the foundations laid and relationships forged during the previous program, although it also successfully developed new links with a completely new government. Close collaboration with important champions in the Funes government has helped to advance the goals of the Project. With access to the most recent comprehensive data on and analysis of the corruption situation in El Salvador, it would have been easier for the Project to develop strategic interventions in the rapidly evolving context. The inconsistent level of political will at the highest levels for transparency and anti-corruption has held back certain advances, especially in implementation of key legislation and strengthening of the Court of Accounts, which forced the Project to find alternative ways of moving its agenda forward. This growing political reticence casts some doubt on the sustainability of some achievements and reforms. The reduced level of activism by civil society may have been a contributing factor to the slowed pace of change by the government. Conflicts between some key project stakeholders/sectors have also hampered the Project’s efforts to maintain a balanced approach, resulting in a heavier focus on collaboration with government partners such as the SSTA. Effects of the implementation approach on results Innovation Casals and USAID have shown great willingness to consider new ideas, partners and approaches and to take advantage of emerging opportunities. That openness to innovation is evident in the way that the Project has embraced modern technology and incorporated the latest developments in its strategies and activities, such as the virtual training system mentioned above and the use of social media. Respectful relations The Project has adopted a highly collaborative approach with national counterparts and stakeholders, including consistent demonstration of respect for their views and priorities. Alliances and trusting relationships have been established with many key actors in both government and civil society, and efforts to do likewise with the private sector have good potential to show results. Human Resources The dedicated team of Project staff, with proven experience in most areas of the Project and willingness to learn as the Project has evolved, has been an important element in the Project’s performance. However, the personnel have had little opportunity to be exposed to the latest developments and to make new professional contacts in the field of transparency and anti-corruption. Staff members have relied primarily on “self-training” and research to update their knowledge and gain familiarity with the new areas of emphasis under the Partnership for Growth. To complement the staff, the Project has made use of some highly qualified and respected experts, especially to deliver workshops and conferences. Informants voiced their appreciation for the value added by those experts, especially those sourced from other Latin American countries. However, maximum consultant fees set by USAID are low in comparison to typical international rates, and have not proven sufficient to engage some very suitable Latin American experts. 18 Moreover, the Project has relied extensively on a single person, Hiram Morales, to provide required technical expertise. This has been arranged through a part-time employment contract under which he visits El Salvador each month, although he is often referred to as a consultant. While Mr. Morales is widely respected, a number of informants expressed the wish that a wider variety of consultants be used as trainers and speakers. Sectors of Emphasis The Project was designed to focus on the national government, especially the executive branch. That resulted in the Project being unable to support substantive work with municipalities and in rural areas, which many informants noted as a deficit. Although the Project has supported various Internet platforms to facilitate access to information across the country, especially through the SSTA, many Salvadorans still lack effective access to the Internet. The strategic choice of the national government as the primary partner meant that work with civil society was greatly reduced from the levels of the previous project, which has led to a perception among some stakeholders that the Project has “abandoned” civil society while becoming too closely aligned with the central government. Since there was no grants mechanism, most support to CSOs has been either: a) to respond to a need identified by the Project, such as for research, publications or training tools, or b) to finance a specific time-limited activity proposed by a CSO, such as a conference or forum. The Project objectives included a secondary emphasis on the private sector, which has been conflated with civil society to a great extent in the Project design and implementation. Although the divisions are not absolute, traditionally the private for-profit sector is considered to be separate from civil society, each of them (together with the government) constituting one of the three primary sectors of society.5 In transparency projects, the approach is generally to strike a balance among these three sectors, as consistently proposed by Transparency International and most international conventions against corruption. The design of this Project did not reflect such a balance. Planning and Changes to Project The Project has adopted a very flexible approach, continually adapting its activities to take advantage of emerging opportunities and responding to many requests from key partners such as the GET, MoP and SSTA. Flexibility was contemplated by the Task Order and has indeed been demonstrated by both USAID and Casals. Work plans have been considered as guidelines rather than functional plans of action, and as such were often subject to change. Those changes have not been consistently documented. The focus of the Project has recently been expanded considerably by the addition of certain Lines of Action under two goals of the Partnership for Growth agreement. No additional resources have been provided for these newly defined tasks, which fall under the original Scope of Work but were not specifically contemplated at the beginning of the Project. Concern was expressed by some Project personnel about the complexity of the PFG tasks, about what they perceive as the additional burden on the Project’s human and financial resources, and about the capacity of the Project to provide sufficient attention to all priority tasks. 5 The separation of the private sector from civil society is, for example, clearly recognized by the December 2011 USAID Guide to conducting Democracy, Rights and Governance Assessments. The two sectors are also clearly differentiated in the PFG Joint Country Action Plan 2011-2015. 19 Focus on Ethics and Awareness The Project has heavily emphasized the promotion of ethics in an effort to change the culture of El Salvador, where corrupt practices have become quite widely accepted. The Project was tasked with enhancing government transparency, ethics and accountability, but its work on ethics has extended to the private sector and professional associations, and recently to university students through a newly designed course. In its efforts to instill ethics and persuade Salvadorans of the importance of transparency, the Project has carried out a large number of activities that have been listed in reports as “training”, although in many cases they are one-off conferences, meetings or “charlas” (brief lectures). This has allowed for the Project to engage with about 20,000 people, through events that often lasted just a couple of hours; training activities of more than one day’s duration were much less common. External Relations Due to the GOES' weak levels of institutionalization and reliance on individuals rather than systems and processes, the Project has had to rely on personal contacts to develop and maintain close collaboration with key stakeholders. As noted above, the Project staff have shown diplomacy and skills in cultivating useful relationships, especially in government; this has helped to open doors and smooth the path to various areas of collaboration. In contrast to its links with a variety of local stakeholders, the evaluation found that the Project has minimal communication and interaction with other USG-funded programs. For example, those involved in this Project from Casals and USAID were not aware of the activities of (or in contact with the agency implementing) the Fiscal Policy and Expenditure Management Project (FPEMP), although it includes a major component on fiscal transparency. Similarly, the Project has had little contact with the Municipal Competitiveness Project (MCP), which is offering training and materials to targeted municipalities on transparency and anti-corruption, especially on the new access to information law. Their work has included cooperation with the SSTA and close links with private enterprise. There has not been any interaction between the Project and the Millennium Challenge Corporation (MCC), which recently completed its first compact in El Salvador and is reportedly close to signing of a new compact. Interaction with other international actors working in or supporting work in the transparency and anti-corruption field has been very limited, and informants report that there is no coordinating mechanism for agencies implementing activities in these areas.6 Conclusions The willingness of Casals and USAID to innovate and consider new ways of working towards Project goals has been a definite asset, especially considering the evolving context. The flexibility of the Project has generated some positive results when opportunities have panned out. However, the undefined “opportunistic” activities outlined in the project often resulted in many diverse activities in response to requests from Salvadoran actors, rather on more focused initiatives. Due to the ambitious menu of tasks and corresponding time 6 The UNDP has recently taken steps to engage in transparency issues by launching an initiative to measure El Salvador’s progress in transparency and anti-corruption, and by creating a Consultative Group with key government and civil society representatives, among others. The national cooperation agency of Spain (AECID) includes transparency as a cross-cutting theme, notably supporting the justice sector with minor engagement with civil society and the SSTA. The World Bank and the Inter-American Development Bank (IDB) are reportedly preparing to fund the SSTA. Transparency International recently accepted FUNDE as its national chapter. 20 pressures, in many cases activities were not followed up with additional attention to the same participants. The shift of emphasis by the Project to fulfill its mandate under the PFG is presenting serious challenges as the small team works to meet all of the Project’s commitments without additional human or financial resources. Lack of investment in ongoing human resource development may have limited the ability of the Project staff to provide technical leadership in all thematic areas as the Salvadoran and regional context has evolved. Use of a wider variety of consultants might have resulted in greater impact, especially in terms of sharing the experience of different countries and contexts. Low ceilings on consultant rates and the time-consuming process of seeking waivers of those limits seem to have restricted the pool of expertise, especially for activities that were planned in a short time frame. The Project design meant that it was unable to support substantive work in rural areas, thus leaving out a large segment of the country’s population. The design also allowed for no real opportunity for civil society to secure support for a genuine project with strategic objectives and series of activities with potential to achieve longer term impact, especially in the important area of advocacy. The fusion of civil society and the private sector in the Project’s design and implementation has hindered the definition of a clear strategy for engagement with these two elements of society, which have very distinct characteristics and roles. The goal of a more ethical society is commendable and the promotion of ethics activities by the Project may bear fruit in the longer term; however, the immediate results are not tangible, and this far-reaching behavior change campaign is consuming resources that could be devoted to achieving short-term results with measurable impact. While probably useful in terms of raising general awareness of the issues and laying the foundation for later interventions, short informational events that lecture to large groups of people are not likely to build the skills and expertise of participants or of key institutions in a sustainable way. Since monitoring of changes in awareness and knowledge has not been done by the Project on a systematic basis, it is difficult to determine the effect of these activities on participants or their actions. Personal contacts and relationships, especially with recognized “champions” of transparency and anti-corruption within the government, have paid some dividends for the Project. However, this approach carries a corresponding risk that changes in personnel would necessitate the re-establishment of contacts and working relationships, which takes time and hinders sustainability. Sustainability of reforms may also be less assured. The emphasis on personal relations has posed a challenge to the Project when partner institutions are in conflict. For example, the serious rupture in relations between the directors of the SSTA and FUNDE, the Transparency International national chapter, has led to a standstill in high priority areas of activity. The weak links of the Project with other USG-funded programs represent a serious deficit. There are significant potential synergies with the FPEMP and MCP, as well as serious potential for overlaps and inconsistencies if due care is not taken.7 It seems that programs are compartmentalized within the USAID mission, including within the Democracy and 7 For example, an internal document describing the FPEMP indicates that they will be working with project partners such as the SSTA and working in areas that are very closely linked with this project, as follows: FPEMP will support the Undersecretary of Transparency of the Presidency in the implementation process, by supporting the definition of a comprehensive system for fiscal data dissemination and tools for civil society participation, which includes the implementation of a citizens audit system. 21 Governance Office, and this is reflected in the low degree of information sharing and coordination among projects and relevant USAID staff. Although no other major bilateral programs target this subject as a primary area of intervention, and the United Nations Development Programme (UNDP) has only recently begun an intervention in this sector, the evaluation found definite opportunities for synergy and sharing of lessons learned among international donors and implementers, especially since several have been asked for support by the SSTA and the same prominent CSOs. Involvement and contribution of different sectors of society This section identifies the sectors of Salvadoran society that have been involved in and/or contributed to a significant extent to the Project since its commencement in 2009. Government The Project has provided significant support to several government institutions, notably the GET and SSTA, which could be considered the key partners of the Project. Considerable assistance has also been given to the Ministry of Public Works, with lesser support to other ministries in response to opportunities or requests. The justice sector was the target of some early activities, notably to develop a draft code of ethics. The Project helped to push for an independent audit of the Court of Accounts, but little other action has been undertaken to bolster the Court. The legislative branch has received limited attention, although they played a major role in Project achievements by passing key legislation. The Project has included minimal assistance to the municipal and department levels of government, primarily consisting of support to establishment of OIRs in 14 governors’ offices and one mayor’s office with a pilot OIR. Indirect assistance has been provided through the SSTA, which has included some municipalities in awareness-raising activities. Private sector Although the private sector was identified in Objective 2 as a key target group, the concrete activities and results of the Project have been limited. This can be attributed to a combination of Project staffing problems (frequent turnover and limited capacity/experience of assigned staff), lack of interest among most major unions and associations, and the serious rift between sector leaders and the government. Some informants reported that the USAID requirement of 50% cost-sharing has been a disincentive to some business leaders. In spite of those constraints, the Project has worked closely with the construction industry via CASALCO, the leading association in the industry to build awareness and facilitate anti￾corruption measures. The Project has supported the development of codes of ethics by several professional associations, and is currently developing a working relationship with the Association of Salvadoran Industrialists (ASI), an association of 500 businesses. Civil Society The design of the Project de-emphasized civil society. The Project did not include a grant mechanism for CSOs, although some funding was provided to help pay for CSO-organized events. Accordingly, the benefit to civil society has been modest, especially in terms of support for advocacy work. However, the passage of the APIL represented the culmination of sustained efforts by civil society, supported in the final stages by the Project. As well, organizations such as Fundación de Estudios para la Aplicación del Derecho (FESPAD) and Iniciativa Social para la Democracia (ISD) have carried out important research and other technical work under sub-contracts with the Project. 22 Media The Project has worked with the Journalists Association of El Salvador (APES) to raise awareness and draft a code of ethics, which was adopted by the organization after a series of consultations. With respect to the APIL, the Project supported development of several information booklets, including one specifically aimed at journalists. The media has played a key role in publication of Project events and messages, and given ever-increasing airtime and newspaper space to stories and opinions on transparency and anti-corruption. Women The Project was not designed to make specific efforts to address gender or reach out to women. While it has undertaken some events with “Empresarias por la Responsabilidad Social” (EMPREPAS), an association of women entrepreneurs promoting social responsibility, it has otherwise not focused on this area. The number of male and female participants in training and similar activities has been tracked (although data is not reported by percentage as required by the Task Order). It appears that overall female participation has been approximately 50% in training and awareness activities, although there is great variation among events. Public information products supported by the Project appear to be gender￾balanced. Youth One sub-activity of the Project was designed to promote “second generation party leadership” as part of a regional USAID effort to cultivate young leaders in government. The Project aimed to develop a younger generation of political party leaders while inculcating principles of transparency and ethics. Various awareness-raising activities were held with youth groups, which led to an attempt to create a national Youth Alliance on Transparency, which foundered due to “lack of unity and leadership”, according to Project reports. The scope of this sub-activity has since been adjusted to promote ethics and transparency among youth, especially university students. The Project has supported youth forums via Fundación Democracia, Transparencia, Justicia (Fundación DTJ) and information sharing with youth groups about the APIL. Customized information booklets on the APIL targeting students and teachers were developed with Project support. Recently, the Project designed a Culture of Ethics Youth Training Program that has been accepted by five universities.8 Steps taken by the Government to control corruption The Government of El Salvador has made significant strides towards the promotion of government transparency, accountability and ethics during the Project period, although direct action on corruption has been less evident. The following is a brief outline of the key steps taken since 2009. It should be noted that these findings are limited to areas where the Project has been involved on some level, and do not necessarily include all GOES efforts.  Raised awareness of the population about corruption and showed that the Government of Mauricio Funes proposed to govern differently from previous administrations; this has given many Salvadorans a sense that, despite all shortcomings, the process toward greater transparency and accountability for which civil society struggled for many years, has started. 8 National University of El Salvador, University José Matías Delgado, Technological University, University Francisco Gavidia and University Alberto Masferrer. 23  Made progress towards compliance with at least 8 of the 20 transparency and anti￾corruption commitments that Funes signed with Transparency International and FUNDE as a presidential candidate  Furthered the legal framework for anti-corruption through the passing of the APIL, the new Law on Government Ethics, and others  Established and supported the SSTA as an entity to promote transparency and combat corruption throughout government  Systematically reported to the Organization of American States’ Mechanism for Follow-Up on the Implementation of the Inter-American Convention against Corruption (MESICIC)  Systematized a process of accountability reporting for many government institutions  Established the unified transparency portal for executive branch information  Conducted an audit of the Court of Accounts  Established OIRs and/or UAIPs in many government institutions, especially in the executive branch  Instituted various transparency and ethics initiatives and mechanisms in the Ministry of Public Works Institutionalization and Operationalization by the Government The evaluation also examined the extent to which the government has institutionalized and/or operationalized initiatives supported by the Project. Specific conclusions linked to these findings are set out at the end of this section. Government Ethics  Law and Tribunal – Although the Tribunal existed previously, this Project contributed to reform of the law to give more powers of investigation and prosecution to the GET, among other changes. The Project has also offered considerable technical and financial aid to the GET. The Tribunal still lacks the financial and human resources to address the current backlog of cases, and could be undermined by an inadequate budget or changes in leadership. There is overlap between the roles of the GET and the Attorney General’s Office but mechanisms for coordination and referral of cases do not yet exist. As well, operations are constrained by the lack of legal protection for “whistleblowers”, which discourages people from filing and pursuing complaints.  Ethics Commissions – These groups of government employees have been established in numerous government entities (primarily ministries and national level institutions) in compliance with the Law on Governmental Ethics. Their primary functions are to receive and transmit complaints of violations of ethics to the GET and to raise awareness among employees of the institution. The Project has supported the GET in developing materials and procedures for those Commissions and in training over 400 members on the new law. The level of functioning of those Commissions is variable, and they operate on a voluntary basis with limited budget from the GET. Thus, their operational effectiveness depends on the level of motivation of each Commission and their ability to secure resources (in-kind or otherwise) from their respective institutions – which in turn depends on the leadership of each institution. To date, at least 83 Commissions have been created, 24 though none exist in municipalities, which indicates low operationalization beyond San Salvador. Access to Public Information – The passage of the APIL (ratified in March 2011) was a major step forward, though most relevant support was provided by the previous program. The Law is regarded as a good model; it was recently ranked as 5th best out of 89 countries with legislation on this subject.9 Its adoption has provided the legal foundation for a steadily increasing operationalization of access to public information. Goal #6 of the PFG specifies that the GOES will provide the necessary resources to implement the APIL; will select the members of the Access to Information Institute in a merit-based fashion, in accordance with established law and regulations; and will provide adequate budgetary resources for the functioning of the Institute and the Access to Information Units. The first six-month scorecard on the PFG, issued in July 2012, as well as the second one issued in November 2012, observed that progress on this goal was behind schedule, noting the failure to appoint the members of the IAIP.  Institute for Access to Public Information – This is an autonomous state entity mandated by the APIL to support citizens with difficulties in obtaining information and to sanction government officials who are not complying with the law. As noted above, the Institute has not yet been created. After a participatory selection process by civil society, in May 2012 President Funes rejected all of the candidates put forward, and the situation remains at a standstill as legal challenges are pursued. Its operational effectiveness in future could be subject to budget constraints, especially if political will is weak.  Units of Access to Information (UAIP) and Offices of Information and Response (OIR) – The APIL mandates the creation of UAIP within all government institutions to disseminate official information and handle information requests within strict deadlines. Soon after adoption of the law, the SSTA decided to support creation of OIRs, entities intended to fulfill the role of UAIP but with an additional mandate of supporting citizen participation and accountability processes and promoting ethics. While the narrower UAIP model has been adopted by a few ministries, OIRs predominate; they now exist in at least 73 executive branch institutions as well as 14 governor’s offices and one mayor’s office. The three OIRs visited briefly by the evaluation team seemed to be operational in terms of staffing, equipment and office space, although the numbers of information requests were still quite limited, which is consistent with global data on all OIRs.10 OIRs barely exist outside of San Salvador, and there are currently no avenues for appeal by citizens who are not satisfied with the government response. Under-Secretariat of Transparency and Anti-corruption – The creation of the SSTA within the Presidency, although not prompted by this Project, is an indication of priority being given to Project issues. The progress made by that office is largely thanks to the experience, persuasiveness and commitment of the Undersecretary, according to informants. SSTA activity has extended into areas that appear to lie within the mandate of the GET and the Institute for Access to Public Information, in an effort to fill gaps in the functioning of those bodies. The SSTA could be eliminated by presidential decision with no need for consultation with other branches of government. That could lead to the 9 http://www.rti-rating.org/index.php 10 During the first three months of implementation of the APIL, 2,314 requests for information were received by government agencies. 25 discontinuation of most or all SSTA-led initiatives, although informants consider such a step would be politically costly. Government Accountability - The government has established and is annually expanding a system of annual accountability reporting. More than 60 ministries and semi-autonomous government bodies are issuing annual reports and carrying out information sharing activities with the public. There is no legal framework for this mechanism, although informants believe there would be serious political repercussions if the system were eliminated. Some consider the reporting processes to be relatively superficial. Ministry of Public Works Reforms – The Ministry has adopted a series of policies and procedures in order to promote transparency and ethics and combat corruption. The reforms are very dependent on the proactive leadership of the Minister, who is making efforts to ensure their sustainability, including with Project support. Conclusions Government Ethics Law and Tribunal – moderately institutionalized and operationalized, subject to various threats  Ethics Commissions – moderately institutionalized, operationalization varies, barely active beyond the capital Access to Public Information  Institute for Access to Public Information – weakly institutionalized and not operationalized  UAIPs and OIRs – UAIPs are moderately operational (mostly as OIRs), though primarily in central government entities. UAIPs are well institutionalized, but full mandate of OIR (including citizen participation, accountability and ethics) is not legislated and is still weakly operationalized. Under-Secretariat of Transparency and Anti-corruption – highly operationalized, weakly institutionalized Government Accountability – moderately operationalized (within executive branch), weakly institutionalized Ministry of Public Works Reforms – moderately operationalized, weakly institutionalized Awareness of USAID funding among beneficiaries Generally, there is a high level of awareness among key Project partners in government, the private sector and civil society that the Project is funded by USAID, which also extends to indirect partners, such as Ethics Commissions. International actors were also well aware of the funding source of activities. However, the Casals name is often used by stakeholders to refer to the Project, which is also closely identified with the Chief of Party. The use of the Casals name is likely linked to some confusion over the name of the Project. Some documents, including official Project and USAID documents, refer to the Democracy Strengthening Program, while others call it the Transparency and Governance Project. When asked about the effects of USAID branding, informants were uniformly positive and had no concerns about any negative impact. Some indicated that the inclusion of the USAID logo was important to CSOs as it adds extra “weight” to their voice when conducting advocacy. Certain minor issues have been noted by the evaluation team with the branding of 26 Project activities and publications; however, these seem to be the exception rather than the rule. D. Overall Conclusions Building on the solid foundations established by the previous USAID transparency and governance program, the current Project has significantly contributed to placing the topics of ethics, transparency and accountability firmly in the national agenda. In the conducive environment following the election of Mauricio Funes, it has provided invaluable assistance to the GOES in passing and implementing the Law on Access to Public Information, strengthening the Government Ethics Tribunal; and attempting to instill a culture of ethics among public officials. It has also nudged several professions, including lawyers and journalists, into establishing codes of ethics to guide the behavior of their members, including how they should work with the GOES. Despite the initial hope for dramatic changes, the Project has faced many challenges due to the politicization and polarization of most GOES institutions and organizations. The level of political will has not been consistent, resulting in unfulfilled promises such as the failure to create the Institute for Access to Public Information, and a prolonged suspension of the work of the Government Ethics Tribunal. The government has been unable to resolve the politicization of the Court of Accounts, which has closed off an important avenue for enhancing oversight of public administration. Civil society’s exhaustion from unending controversy over the justice sector and decreasing support from USAID and other donors, among other factors, seems to have diminished their capacity and will to advocate for more and faster reforms and engage the population in these efforts. Moreover, the private sector—a traditional partner of previous governments—has undermined its own capacity to be an effective interlocutor due to ongoing conflicts with the Funes administration, and shown little genuine interest in advancing transparency and combating corruption. Until there is real commitment from the sector for transparency and anti-corruption initiatives, further substantial investment by the Project is unlikely to produce meaningful and sustainable results. Nevertheless, achievements have been significant both for the Funes administration and initiatives supported by the Project. Salvadorans interviewed seem confident that they are on the right track to reduce corruption, for example through the establishment of new government institutions and initiatives such as the Offices for Information and Response, which are dedicated to responding to citizens’ demands. The legal framework for anti￾corruption, while not complete or perfect yet, has improved; there are new and better laws, regulation and procedures on the books. El Salvador still has a long way to go to reduce its levels of impunity, strengthen its institutions and ensure the sustainability of these achievements. Achieving optimal results in the promotion of transparency and fight against corruption will only be possible through balanced engagement by USAID and the Project with government and non-government sectors, and through building in-depth capacity in key entities. The time is ripe for the Project to tighten its focus on consolidation of gains and institutionalization of positive changes, through strategically-placed support to the government and civil society during its crucial final period of implementation. Stronger monitoring tools and practices will ensure that the results and lessons of this Project and other initiatives can be captured clearly and capitalized upon in the future. 27 A major challenge remains—extending the benefits of reforms and new services to the majority of the population, whose daily lives are far removed from the capital and national government, but who continue to suffer the effects of corruption on a regular basis. Through placing a higher priority on coordination and collaboration, and by making anti￾corruption a cross-cutting theme of all USAID programming, it will be possible to seek synergies, avoid potential duplication of efforts, and achieve greater and more sustainable impact across the country. E. Recommendations Recommendations for Casals in the short-term (during this Project) The Project will end in March 2014, which means that activities will be winding down from December 2013 onwards; that leaves approximately one year for implementation of this set of recommendations. Approach and Methods of Implementation  The Project should concentrate on consolidating gains already made and strengthening bonds with other projects and actors, in order to promote sustainability of reforms and build deeper capacity in key institutions.  Work plans should closely target strategic areas of activity, eliminating areas where little progress is likely in order to focus on high-impact activities. A carefully planned approach including exit strategies should take precedence over spontaneous response to opportunities and requests.  Work more even-handedly with government and civil society, as well as the private sector (to the extent that it demonstrates genuine interest).  Strengthen tools and systems for monitoring and evaluation, especially of capacity building and awareness-raising activities. Project partners such as the GET and SSTA should be required to report regularly on their use of Project resources and on results, and be assisted to develop their own M&E tools and systems as needed.  Provide key institutions with specific types of technical assistance and intensive mentoring and training, designed in a consultative and strategic manner. Focus on IAIP once created. One objective is to strengthen levels of institutionalization so as to reinforce sustainability in important entities such as the SSTA, GET and MoP.  Utilize a wider range of experts to bring fresh perspective and experiences to the Project, working with USAID to address consultant fee ceilings to secure the most relevant expertise. Casals headquarters should facilitate access to relevant experts.  Support professional development of the Project team, especially in specific themes under PFG. It is important for key individuals to interact with others working in similar programs, for example through regional conferences, in order to have updated knowledge and a broader set of tools from which to choose. If the Project budget cannot accommodate these activities, then other means should be sought by Casals to address this need. 28 Priority Stakeholders  Cultivate contacts and establish regular communication and coordination with other USG-funded programs that include or have impact on transparency and anti￾corruption. The Project (and USAID) should also actively engage with the UNDP and other international actors in order to explore synergies and achieve optimal impact, possibly through establishment of coordinating mechanisms. The UNDP’s Consultative Council may be one useful conduit.  A specific strategy for engagement with civil society should be developed immediately, in consultation with the core group of experienced organizations and coalitions with demonstrated capacity and commitment. In addition to financial assistance, the Project could build capacity in management of networks and coalitions, negotiation skills and advocacy. Steps should be taken to build bridges between FUNDE and SSTA, since they are two major actors in transparency who currently have severely limited communications and cooperation.  With the executive branch of government, focus on fulfillment of PFG commitments, through a balanced approach that emphasizes the obligations of the government and provides strategic assistance that does not allow the government to avoid its responsibilities. In relation to the Ministry of Public Works, base support on an updated strategic evaluation of needs, and emphasize consolidation and institutionalization of steps taken to date.  With the Under-Secretariat of Transparency and Anti-corruption, analyze needs carefully and confine support to activities that have not been allocated by law to other entities, such as the GET or the Institute for Access to Public Information, which could detract from the full operationalization of those bodies. Focus on sustainability of the SSTA and its work in priority areas.  Assist the Government Ethics Tribunal to consolidate its legal framework and structures and develop the capacity to carry out its broadened mandate.  Work with the Economic Growth Office of USAID to identify entry points and strategies under the PFG that could genuinely motivate the private sector to become a partner in transparency and anti-corruption. One such entry point may be through FUSADES, a respected think tank with close private sector and civil society ties.  Encourage professional groups and others who have already adopted codes of ethics to move towards methods for monitoring compliance, reporting and sanctions. In the words of one informant: “The ethics code should be a means and not a goal.” The future ethics code should include strict compliance mechanisms and sanctions for non-compliance.  Work with universities to promote the type of in-depth capacity building being offered by the course in Access to Public Information.  Mobilize the energy of youth for advocacy in collaboration with other CSOs. Activities should be aligned with the new USAID Youth in Development Policy. Involve youth in consultations on new legislation and policies. Additional recommendations for working with and for youth are included in Annex H.  Engage women and women’s groups in consultations on new anti-corruption legislation and policies, and ensure that gender equality is promoted and women’s rights protected by the new civil service law and regulations.  Support the independent media, notably to enhance investigative reporting. 29 Priority Areas of Activity Advocacy - Support CSOs and coalitions (including the private sector, as appropriate) to advocate in the following key areas:  Legal Framework for Transparency and Anti-corruption. Changes are needed to provide a more secure foundation for reforms, in accordance with MESICIC recommendations and the UN Convention against Corruption. Priorities include laws on illicit enrichment and conflict of interest, and strengthening the APIL.  Other key legislation. Priorities include a law on political party financing11 and reform of the civil service law, already in Project plans, as well as laws to protect “whistleblowers” who report on corruption.  Establishment, operationalization and funding of the Institute for Access to Public Information.  Consolidation and funding of the Government Ethics Tribunal. Assessments - In order to inform this Project as well as future interventions, conduct the following as soon as practicable.  Assessment of the structure and functions of the Court of Accounts, to identify key bottlenecks and potential entry points that could lead to meaningful reforms.  Evaluation of implementation of the Government Ethics Law, to assess compliance with MESICIC recommendations (already in the 2012-13 work plan).  Analysis of internal controls and risk mapping of the Vice Ministry of Transport, in line with Goal #4 of PFG (already in the 2012-13 work plan). Short-term Wins - Concentrate on generating some short term wins and publicly visible gains. These could occur on a high level to create prominent examples of the price that should be paid for corrupt behavior, or on the “ordinary citizen” level to show that it is possible to combat the corruption that affects their daily lives. Opportunities have opened up to go beyond addressing the conditions for corruption to actually assisting Salvadoran institutions to:  Take punitive action against corrupt officials, for example through prosecution of symbolic cases by the GET; and  Reduce corruption in targeted institutions or processes, for example by cleaning up the process for obtaining driving licenses. Recommendations for USAID in the short-term (during this Project)  Place a high priority on coordination among USAID staff in various offices and among projects whose work touches on transparency and anti-corruption, insisting on regular exchange of information and taking an active role in identifying opportunities for collaboration and risks of overlap. Meetings of relevant contractors should be held at least monthly. USAID also should coordinate with MCC in order to identify the best way of leveraging the promise of new MCC funding to advance the transparency agenda at this crucial moment. 11 The Project plans to support development of a new law on political parties’ financing. Due to the complexity of this area and need for specialized expertise, the Project should limit its involvement to drafting and advocacy for the law, unless additional resources allow for a broader engagement in reforms. 30  Ensure that all USAID initiatives are fully informed of the PFG’s implications for their work, and are aware of who has been tasked with which goals and activities. Implementers need to ensure that key partners receive clear information about the PFG, and should be offered support as needed to do so. In parallel, those charged with overseeing implementation of the PFG should develop a strategy to inform the private sector about the PFG and engage them in its execution.  Review the Project M&E framework and relevant parts of the Performance Management Plan together with Casals, in parallel with finalization of the 2012-13 work plan, to clarify (or delete) indicators as needed and bring targets into line with current plans.  Conduct a thorough corruption assessment to identify key points for strategic interventions and recommend which USAID (and other USG) programs are best suited to carry out which actions. This should include analysis of how women and youth can be specifically engaged and empowered in efforts to reduce corruption.  Work to bring together the diverse Salvadoran institutions (Attorney General’s Office, GET, IAIP, SSTA, Office of Probity, Court of Accounts, etc.) that are key to transparency and anti-corruption, in order to resolve bottlenecks in their collaboration and confusion over division of roles and responsibilities. Recommendations for USAID in the long-term (following this Project)  Make transparency and anti-corruption a crosscutting theme in the El Salvador mission, in order to make sure that all programs and sectors are taking these issues into account and sending a consistent message to Salvadoran stakeholders.  Allocate resources and design complementary projects to ensure that transparency and anti-corruption at municipal level, both through local government and CSOs. Promote linking of urban and rural civil society organizations, and involve churches and faith-based organizations to leverage their significant reach at grassroots level and influence at national level.  Simplify M&E systems and tools, using less indicators and clearer definitions, and provide thorough training to relevant USAID and implementing staff.  Develop strategies to address corruption and transparency issues within law enforcement institutions, either within USAID or beyond.  Several informants recommended that USAID (and the USG in general) place more conditions on assistance. In particular, care should be taken not to support (or be perceived as supporting) institutions widely known to be corrupt, in order to maintain a consistent message on transparency and anti-corruption. USAID should support the strengthening the internal control mechanisms in government institutions, which are crucial to identifying corruption-offenses.  Incorporate gender considerations in line with the new USAID Gender Equality and Female Empowerment Policy, including through the following illustrative activities:  Involve women and gender experts in the design of projects in transparency and anti-corruption.  Provide gender training and relevant information materials (including the Policy) to all implementing staff. 31  Ensure that gender equality and female empowerment are considered in the drafting of laws and policy documents, as appropriate.  Adapt information products for the public to low-literacy audiences, and make them available to those who are tied to the home and children without access to Internet or decision makers.  Include women-oriented or women-run CSOs and private sector groups in consultations and provide equal opportunities for them to qualify for funding.  Develop creative ways of getting women more involved in citizen participation and other transparency mechanisms, especially in relation to projects and policies that affect their lives directly.  Disaggregate monitoring data on training to identify any differences between satisfaction and learning of women and men, and devise strategies to address imbalances.  USAID should consider providing more specific anticorruption training to GOES officials and civil society organizations. These trainings should be conducted as course and/or workshops. They should be designed and targeted to the needs of the entity and/or tasks being performed by the officials. They should focus on prevention, reduction and sanctioning of corrupt activities rather than on ethic awareness. Ethic awareness could be part of a broader anticorruption training course/workshop but, anticorruption efforts cannot rely primarily on raising citizens and officials' ethics standards. Training should be clearly identified as such. Lectures and conferences, while attendees will be able to learn, cannot be considered trainings. Also, lectures and conferences on anticorruption would benefit from a greater diversity on topics and participation of national and international experts. 32 Annex A: Evaluation Scope of Work Mid-term Performance Evaluation Scope of Work Transparency and Governance Project in El Salvador April 20, 2012 I. Purpose of the Evaluation The purpose of this evaluation is to help USAID analyze the Project’s activities, results and approaches to date in order to use the information for future project design and implementation in the areas of transparency and governance. This mid-term performance evaluation (as defined in USAID’s Evaluation Policy, see below) will cover implementation from the start of the project in FY 2009 through the third quarter of FY 2012, or a three￾year period before the third and fourth option years are exercised. The objectives of the evaluation are:  To analyze whether USAID is using the right actions and initiatives to control corruption and promote transparency in El Salvador  To assess whether the project has achieved expected results during the base period, as well as option years 1 and 2  To make recommendations on adjustments for optimal implementation during the Project’s remaining time  To make recommendations on with whom USAID and Casals & Associates should work to improve chances of success  To gauge the Government of El Salvador’s response to the Project in terms of anti￾corruption and transparency initiatives The main participants in the evaluation will be Government of El Salvador officials, civil society members and private sector representatives. The principal audience of this evaluation will be USAID/El Salvador and Casals & Associates. The evaluation will provide these two parties with information that they can use to adjust (if necessary) and implement the Transparency and Governance Project during the following two option years, if extended, as well as evaluate what has or has not worked well to-date. II. Background Information on the Project Project Name: Transparency and Governance Project for El Salvador Contract Number: DFD-I-01-08-00069-00 Implementing Partner: Casals & Associates Award Dates: Base period: May 2009-September 2010, currently in Option Year 2, possible extension through option years until FY 2014 Funding: $7.9 million The USAID Transparency and Governance Project for El Salvador, under Contract DFD-I￾01-08-00069-00 with Casals & Associates, is a $7.9 million project (including option years up to FY2014) focusing on controlling corruption with the Government of El Salvador (GOES) and civil society in the use of public funds. The contract was signed in May 2009, with a base period that ended in September 2010. The Project has four consecutive option 33 periods; the second option year is currently being implemented. The evaluation will cover project implementation from May 2009 to March 2012. USAID implemented an initial 5-year program from 2004 to 2009, entitled USAID/CAM Anti-corruption, Transparency and Accountability Program, to assist the government, private sector, civil society and other key actors in working together to slow and reverse the spread of corruption in each participating country in the Central American region, including El Salvador. In this five-year multi-country program, USAID supported anti-corruption reforms at the national and local levels, and built advocacy partnerships with some civil society organizations (CSOs). As a result, the National Assembly passed the Government Ethics Law, and the GOES established the Government Ethics Tribunal to lay a solid foundation for creating a culture of ethical conduct and combating corruption in the public sector; the Project strengthened the Consumer Defense Agency (Defensoría del Consumidor) to enforce laws for protecting consumers; and the Project supported development of the GOES report to the Organization of American States (OAS) for the second round of compliance with the Inter-American Convention Against Corruption (IACAC). In the justice sector, several transparency initiatives were carried out with the Attorney General’s Office (Fiscalía General de la República) to strengthen its Anti-Corruption Unit; the Public Defender’s Office (Procuraduría General de la República) to assess and enhance its own internal procedures; the National Council for the Judiciary, (Concejo Nacional de la Judiciatura) to expand ethics training by incorporating a permanent module on ethics into its curriculum, and to develop judicial transparency indicators; and the Supreme Court to develop a Code of Ethics for judges. At the local level, USAID and the donors group,(Red de Cooperantes para el Desarrollo Local) provided technical assistance to the National Commission for Local Development (Comisión Nacional para el Desarrollo Local), and the Corporation of Salvadoran Municipalities (Corporación de Muncipalidades de El Salvador) in decentralization and municipal transparency. See Annex 1: Final Report – El Salvador. This evaluation covers the Project that followed the abovementioned 5-year regional program in El Salvador. Much of the work that was done in El Salvador during implementation of the 5-year program provided the foundations for the current bilateral Project with Casals & Associates. The Partnership for Growth (PfG) Joint Country Action Plan (JCAP) 2011-2015 (See Annex 2: JCAP) for El Salvador and the United States, signed in November 2011, is one of the most important strategy documents governing USAID’s programmatic focus. The JCAP identified two key constraints to growth: crime and insecurity and low productivity in the tradables sector. The JCAP states that “crime and insecurity also negatively affect the legitimacy of the authority and institutions of the government. The limitations of the state to combat and prevent crime can erode the confidence of the people and can undermine good governance.” Goal #4 under Institutional Strengthening in the Crime and Security constraint is: 34 Facilitate economic growth by ensuring El Salvador’s labor force is protected from crime while transiting to and from work, and ensuring that the public transportation service providers serving the labor force are protected from crime. Actions under Goal #4 include using special prosecutors and courts to combat crime, identifying necessary legal changes and institutional reforms in the government, technical training and assistance, among others. Goal #6 under Institutional Strengthening in the Crime and Security constraint is: Professionalize El Salvador’s civil service and enhance public confidence in the government. Actions under Goal #6 include implementing the new Access to Public Information Law, technical assistance to increase accountability and transparency and promote other reforms, such as the approval of a new Civil Service Law. These actions and goals relate directly to USAID/El Salvador’s efforts to promote transparency and governance, particularly through its current Project, but will also direct future programming strategies to provide support to the Government of El Salvador in these areas. Current Scope of Work from Casals & Associates The Transparency and Governance Project subject to this evaluation is a follow on to the USAID/CAM Anti-corruption, Transparency and Accountability Program outlined above and has two main objectives: 1) to continue efforts to support various levels of the Salvadoran government to increase responsiveness and accountability towards its citizens, through increasing transparency in government and ethical behavior by public officials, reducing conditions for corruption in public administration, strengthening the link between citizens and government, and thereby increasing confidence in the system; and 2) to continue to foster civil society participation, particularly the private sector, in promoting transparency, accountability and control of corruption in the use of public funds. These objectives apply to all option years implemented. The main client of the Project is the Government of El Salvador (GOES), with a primary focus on the Executive branch, particularly, the Under-secretariat of Transparency and Anticorruption (Under-secretaría de Transparencia y Anticorrupción) and the Government Ethics Tribunal (Tribunal de Etica Gubernamental). The other client is civil society. One significant result of the Project to date is the support that was provided to the Government of El Salvador to draft, promote and ratify (in 2011) the new Access to Public Information Law (Ley de transparencia y acceso a la información pública). This Law mandates greater transparency through the publication of governmental budgets, processes and results in the public realm. Recently, Casals & Associates has started working with the private sector. One important achievement was the adoption of a new Ethics Code by the country’s Construction Association (CASALCO), which has been used as an example for other sectors in the country and abroad. However, this work has been challenging since many private sector players view corruption as an issue that should be dealt with by the government. 35 The Transparency and Governance Project has three main activities: Activity 1: Establishing and promoting good governance and anti-corruption practices within the government 1.1: Assist the new government to implement transparency, ethics and anti￾corruption measures as related to priorities proposed in its government platform (Plan de Gobierno 2009-2014) 1.2: Continue support to the Government Ethics Tribunal 1.3: Support the GOES to implement the new Transparency and Access to Public Information Law 1.4: Auditing the Court of Accounts Activity 2: Engaging Civil Society in the Fight against Corruption 2.1: Private sector involvement in the fight against corruption 2.2: Establishment of the Transparency International (TI) Chapter in El Salvador 2.3: Independent Court of Accounts 2.4: New generation leadership Activity 3: Provide technical assistance and training in support of other openings for relevant transparency initiatives designed to curb corruption See Annex 3: Casals TAG Task Order for more information about each Project activity and expected results. See Annex 4: Monitoring and Evaluation Plan on indicators used to monitor project activities. The Contractor will be responsible for collecting all monitoring data on Project indicators from Casals & Associates, analyzing it using cross tabulations and triangulation (or cross examinations from several data sets collected using different methods) and reporting on it in the Final Evaluation Report as evidence to answers for the evaluation questions. Most Project indicators have a baseline of zero (0) despite the fact that this Project follows the previous multi-country program. Exceptions include international indices that have data available from prior years (for example, Transparency International’s Corruption Perception Index). Therefore, reported indicator results refer only to Project implementation and are not cumulative since the multi-country effort. One weakness of the indicator data that is important to highlight is that the Project is dealing with topics of transparency and governance, which are hard to calculate and measure. In addition, several indicators have to do with perceptions, which may or may not be a true measure of reality in the country. This is one reason why a review of indicator data alone cannot, and should not, be an indication of Project results, and why USAID expects the Contractor to employ other data collection methods and analyses to triangulate data and answer the evaluation questions. The development hypothesis of the Transparency and Governance Project is: If the Government of El Salvador fights corruption through improved transparency and accountability, then it will not only consolidate the country’s democracy and improve institutional legitimacy, but it will also positively impact El Salvador’s effectiveness to combat crime and insecurity and contribute to economic growth. 36 III. Evaluation Questions The Contractor must comply with USAID’s Evaluation Policy. (See Annex 5: USAID Evaluation Policy, or online at: http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf.) The following evaluation questions, in priority order, have been identified by USAID/El Salvador. They should be answered as a result of this evaluation and clearly presented in the Final Report in terms of how they relate to the evaluation purpose. 1. What recommendations can be made to realign or adjust the Project during the remaining time to optimize chances for success? a. What have been the most significant performance results to date? b. What were the major factors influencing the achievement or non￾achievement of Project objectives? c. How has the current implementation approach positively and/or negatively affected Project results? 2. To what extent have different sectors of society been involved and contributed to the Project, including: the Government of El Salvador, civil society, the private sector, other international donors and the media? a. Who should USAID and Casals & Associates work with during the remaining time to improve chances of success? 3. To what extent has the GOES taken steps to control corruption during Project implementation? a. How have anti-corruption and transparency initiatives been institutionalized or operationalized in GOES processes? 4. How aware are beneficiaries that this Project is funded by USAID? IV. Data Collection, Analysis and Methodology The data collection plan for this evaluation will include at a minimum: a desk review of relevant documents; interviews and/or focus groups; and direct observation through site visits and/or attendance at planned events occurring during the evaluation period. The results of these interviews and reviews will be analyzed for content on a qualitative and quantitative basis. The Contractor may propose additional data collection methodologies in the Evaluation Plan.  Desk review of relevant documents o USAID/El Salvador will provide the Contractor with all relevant Project specific documents, such as scopes of work, reports, prior assessments, etc. The Contractor should review the documents before meeting with local stakeholders for interviews. The Contractor is expected to review these and create a Document Review Matrix to be delivered to USAID/El Salvador using the following illustrative format, which may be improved by the Contractor: Evaluation Question # Methodological Data Collection Tool(s) Used Source(s) (Documents and others) Any comments 37 Example: 1.c How has the current implementation approach positively and/or negatively affected project results? Example: 1. Document review of Semi-annual Reports 2. Focus group Example: 1. Casals & Associates, 2. Chief of Party, Government officials from Offices X and Y Focus group participants were selected at random from the participating offices. o USAID/El Salvador can provide the following information as data sources for this evaluation: 1. Partnership for Growth Joint Country Action Plan (See Annex 2) 2. Casals TAG Task Order (See Annex 3) 3. Casals Monitoring and Evaluation Plan (See Annex 4) 4. National democracy surveys conducted during 2006-2010 (See Annex 6) 5. Semi-annual reports from Casals & Associates, containing monitoring data (See Annex 7) 6. Monitoring and indicator tracking tables (See Annex 8).  Consulting stakeholders in the country o Key Informant Interviews, Group Interviews, Focus Groups. The Contractor will interview the identified individuals (18) below at a minimum. The Contractor may also conduct focus groups or group interviews with individuals from the organizations proposed below: 1. USAID/El Salvador: Maria Antonieta Zelaya, Contracting Officer Representative 2. Casals & Associates: Carlos Guerrero, Chief of Party; Otto Vidaurre, Deputy Chief of Party 3. Project Consultant: Hiram Morales, Anti-Corruption Expert 4. The Government’s Secretariat of Transparency: Marcos Rodríguez, Sub-Secretario de Transparencia (Deputy Secretary of Transparency) 5. The Government’s Ethics Tribunal: Lizzette Kury de Mendoza, former President; Jennifer Giovanna Vega Hércules, Member 6. Ministry of Public Works: Gerson Martínez, Minister of Public Works 7. FUNDE (leading the Transparency International (TI) chapter in El Salvador): Roberto Rubio, President 8. Grupo Promotor of the Access to Information Law (FUSADES/APES/FUNDE/ANEP/IIDC/UCA): Claudia Umaña, former head of FUSADES' ROL Department; Laura Rivera, former Transparency Initiatives Coordinator, now in the UNDP; Javier Castro, FUSADES; José Luis Benitez, UCA 9. CASALCO (Salvadoran Construction Association): Ismael Nolasco, Executive Director 10. Organizations included in the regional program and other USAID Projects: Andrés Amaya, Attorney General's Office; Sidney Blanco or Florentín Meléndez, Magistrates of the Supreme Court of Justice 11. LIDERA: Lilian Zelaya, President 38 12. Iniciativa Social para la Democracia (ISD): Ramón Villata, President. The Contractor may talk to other related individuals or organizations that the evaluation team may identify as relevant to this evaluation. Specific interview and focus group questions will be prepared in advance and finalized during the team planning meeting with USAID/ El Salvador; the questions should be recommended by the Contractor with the specific purpose of answering the evaluation questions listed in this Scope of Work and must be limited in number. The questions asked in interviews and/or as part of focus groups should be different and targeted to the specific audience. A sampling plan of who is selected for interviews, whether purposeful sampling, random or a combination of approaches, must be developed in advance and summarized in the Final Evaluation Report. o Site visits with organizations listed above or other relevant ones identified by the Contractor. USAID/El Salvador and/or Casals & Associates can make suggestions and will help the Contractor gain access, if needed; however, the Contractor will make the final determination of which sites to visit. Site visit selection should be based on a sampling plan developed in advance and summarized in the Final Report. This could include purposeful sampling methods, random sampling, or some combination of approaches. o The Contractor may attend events hosted or sponsored by the Project during the evaluation period in El Salvador to conduct direct observations. The Contractor can use these meetings to talk with other Project stakeholders and collect additional data as evidence to answers to the evaluation questions. USAID/El Salvador and/or Casals & Associates will provide the Contractor with a list of events once the evaluation team has arrived in country.  Team planning meetings o An initial team planning meeting will be held between USAID and the Contractor before the evaluation begins so that USAID can clarify any questions from the Contractor, expectations and guidelines. The expected results of this meeting are to:  Confirm each team member’s role and responsibilities  Confirm timeline and deliverables  Finalize questions to be asked during interviews  Finalize other data collection tools and methodologies  Identify communications logistics and how the Contractor, USAID/El Salvador and Casals & Associates will communicate with each other  Finalize and approve the Evaluation Plan o A second team planning meeting will be held among USAID, the Contractor and Casals & Associates in country before the evaluation begins so that the Contractor can clarify the evaluation methodology and initiate contact with the Casals & Associates. 39 The analysis of the data collected is just as important as the actual collection. The Contractor must triangulate data collected in order to have sound evidence for the findings and conclusions in the Final Evaluation Report based on the data presented. In the Final Report, the Contractor should list any biases or limitations that exist for both data collection and analysis. In addition, all real or possible conflicts of interest must be disclosed by each member of the evaluation team in writing (See Annex 9: Disclosure of Conflict of Interest Form). All data should be disaggregated and analyzed by gender. V. Deliverables The Contractor will deliver the following to USAID: 1. Timeline and/or Milestone Plan in Word or Excel on/about 2 weeks (Week 2) after the award. 2. An Evaluation Design and Work Plan in Word with the methodologies and Sampling Plan in Word or Excel on/about 2 weeks (Week 2) after the award. 3. Weekly bullet reports of activities in Word, particularly for Weeks 1-6 after the award, due every Monday by the close of business. 4. Document review matrix in Word or Excel (see format above in Section IV) to be completed on/about 6 weeks (Week 6) after the award. 5. A draft of the Final Report in Word for review due in Week 6 after the award to USAID. USAID will provide comments within one week. The draft report should be submitted in English. It will include an Executive Summary no longer than 3 pages, stating the methodologies, findings, conclusions and recommendations of the evaluation. (The final Executive Summary will be presented in both English and Spanish.) 6. A Final Report in Word no longer than 30 pages excluding annexes, identifying methodologies, findings, conclusions and recommendations. The Final Report should be presented in English and Spanish and have incorporated USAID’s comments, as appropriate. USAID/El Salvador and/or Casals & Associates may attach a Statement of Differences as an annex to the Final Report if any differences remain in the final version. The Contractor should turn in two copies in print in each language and 1 copy in electronic version with both languages (DVD or flash drive). The Final Report will be due to USAID one week after the Contractor receives comments on the draft, in Week 8 after the award. USAID will review and then approve the Final Report if comments have been incorporated satisfactorily. 7. Any raw data (qualitative or quantitative) collected in electronic form (DVD or flash drive, in original format of Word, Excel, etc.) due in Week 8 after the award. 8. A Final Presentation with Power Point slides to USAID and Casals & Associates upon acceptance of the Final Report due in Week 9 after the award. Only the Team Leader needs to be present for the Final Presentation; however, local evaluation team members may also attend. 40 All reports and papers will be considered draft versions until they are approved by USAID. VI. Reporting Requirements The Final Report in English and Spanish will become public documents for distribution among the project’s key stakeholders, including high-level U.S. government policy-makers and officials, host country government officials, the private sector and civil society and other audiences. The main users of the evaluation will be USAID/El Salvador and Casals & Associates, to adjust Project implementation tools if necessary, and USAID/El Salvador, to plan for future programming designs in the areas of transparency and governance. The Final Report will include the following sections: 1. Executive Summary 2. Project Identification Cover Page 3. Table of Contents 4. List of Acronyms 5. Background 6. Evaluation Purpose and Objectives 7. Methodologies and Limitations (including sampling and data collection and analysis methodologies) 8. Findings 9. Conclusions 10. Recommendations 11. Appendices a. Copy of the final Evaluation Scope of Work (SOW) and Purchase Order (PO) – required b. Copy of the final Evaluation Plan - required c. A Statement of Differences, regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team – if needed d. Copies of all tools used, such as checklists, surveys, questionnaires – required e. A list of all sources of information, properly identified – required f. Any other Appendices The Contractor will submit the Final Report in English and Spanish to the Development Experience Clearinghouse (DEC): http://dec.usaid.gov within three months after final approval. VII. Team Composition The Contractor is recommended to propose a multidisciplinary team with a combination of international and local experts. To minimize costs and maximize the potential for capacity￾building, it is recommended that the team utilizes local expertise to the maximum extent possible. The team should include at a minimum the following recommended key positions: 1. Team Leader Education: Bachelors university degree in development or a related development field, such as Economics, Political Science, Public Administration, Business 41 Administration, or other discipline related to development assistance is required. Advanced degree preferred. Formal training in monitoring and evaluation is preferred. Language Proficiency: Spanish Level III and English Level IV Work Experience: At least 8 years of relevant prior experience conducting development evaluations and monitoring projects and programs, and preferably Latin or Central America. At least 5 years of project management experience. Some familiarity with USAID’s objectives, approaches and operations, particularly as they relate to evaluations, is a plus. Position Description: The Team Leader must be someone external to USAID. The Team Leader will be the lead on Monitoring and Evaluation; therefore, s/he should have knowledge and experience in evaluation methodologies and practical applications. The Team Leader will be responsible for overseeing and coordinating all activities related to this evaluation and for ensuring the production and completion of a quality and professional report, in conformance with this SOW. 2. Transparency Expert Education: Bachelors university degree in development or a related development field, such as International Law, Economics, Political Science, Public Administration, Human Rights or other discipline related to development assistance is required. Advanced degree is preferred. Language Proficiency: Spanish Level IV and English Level III Work Experience: At least 5 years of experience in the areas of transparency, governance or anti-corruption in Latin America. It is preferred for the Transparency Expert to have relevant prior experience in evaluations of development projects. Position Description: The Transparency Expert will provide local support and know-how to the Team Leader, specifically, knowledge about transparency and governance in Latin America. S/he should have some expertise in the Salvadoran political environment, contextual factors and local players and dynamics. The Transparency Expert will be involved in planning, data collection and analysis and drafting the Final Reports, as well as any other tasks determined by the Team Leader. Having a Transparency Expert who has worked with the Project at any time would represent bias and a conflict of interest in the evaluation; therefore, anyone who has worked with the Project should not be considered for this position. 3. Logistics Coordinator Education: High School degree is required. Bachelors university degree in administration or a related development field, such as Economics, Political Science, 42 Public Administration, Business Administration, or other discipline related to development assistance is preferred. Language Proficiency: Spanish Level IV and English Level II Work Experience: At least 5 years of relevant work experience. Participation in or knowledge about evaluations of development projects is a plus. Position Description: This is not a full-time position. The Logistics Coordinator will be responsible for coordinating with USAID/El Salvador, Casals & Associates and the Contractor to schedule local meetings, transportation and other administrative logistics. The Logistics Coordinator will be involved in planning and should help with data collection and analysis. In addition, each team member should have, at minimum, the following skills and experience:  An understanding of the country context  Demonstrated skill in written and oral communications  Ability to work effectively and communicate with a diverse set of senior governmental officials and professionals VIII. Logistics The Contractor will be responsible for all logistical support under this Purchase Order and for complying with provisions set forth in this SOW. IX. Evaluation Criteria The criteria listed below are presented by major category in descending order of importance, so that Offerors will know which areas require emphasis in the preparation of information. Offerors should note that these criteria serve as the standard against which all technical proposals will be evaluated, and serve to identify the significant areas that Offerors should address.  Technical Approach Clear description of how the evaluation will be conducted and deliverables submitted on time. Demonstrated approach that is feasible, innovative and appropriate to the culture and context of El Salvador.  Management Plan and Personnel Quality and mix of the staffing plan to undertake the proposed approach successfully. Demonstrated interpersonal, technical, administrative, educational and language skills of the key individuals proposed.  Institutional Capacity and Past Performance Demonstrated experience and quality in conducting similar work. Satisfactory past performance working in development project evaluations. X. Annexes – Available from the USAID/El Salvador Program Office (SDO) 1. Final Report – El Salvador 2. Partnership for Growth Joint Country Action Plan 2011-2015 3. Casals TAG Task Order 43 4. Casals Monitoring and Evaluation Plan 5. USAID Evaluation Policy 6. National Democracy Surveys 2006-2010 7. Semi-annual Reports 8. Monitoring and indicator tracking tables 9. Disclosure of Conflict of Interest Form 44 Annex B: Evaluation Plan and Methodology dTS Evaluation Plan October 2012 The USAID Transparency and Governance Project in El Salvador The Transparency and Governance Project has two main objectives. One is to continue efforts to support various levels of the Salvadoran government to increase responsiveness and accountability towards its citizens, through increasing transparency in government and ethical behavior by public officials, reducing conditions for corruption in public administration, strengthening the link between citizens and government, and thereby increasing confidence in the system. The second objective is to continue to foster civil society, particularly the private sector, in promoting accountability and control of corruption in the use of public funds. The Project focuses on three main areas: 1: Establishing and Promoting Good Governance and Anticorruption Practices in Government 2: Engaging Civil Society in the Fight against Corruption 3: Providing technical assistance and training in support of other openings for relevant transparency initiatives designed to curb corruption. Purpose and Objectives of this Evaluation The purpose of this evaluation is to help USAID analyze the Project’s activities, results and approaches to date in order to use the information for future project design and implementation in the areas of transparency and governance. This mid-term performance evaluation (as defined in USAID’s Evaluation Policy) will cover implementation from the start of the Project in FY 2009 through the third quarter of FY 2012, or a three-year period before the third and fourth option years are exercised. More specifically, the objectives of this evaluation are:  To analyze whether USAID is using the right actions and initiatives to control corruption and promote transparency in El Salvador  To assess whether the project has achieved expected results during the base period, as well as option years 1 and 2  To make recommendations on adjustments for optimal implementation during the Project’s remaining time  To make recommendations on with whom USAID and Casals & Associates should work to improve chances of success  To gauge the Government of El Salvador’s response to the Project in terms of anti￾corruption and transparency initiatives Evaluation Team To implement the evaluation, dTS will provide a team consisting of two senior experts in anti-corruption and governance programs who are also seasoned USAID evaluators, plus a local logistics coordinator. As can be seen in the table below, the Team Leader, Melanie Reimer, is responsible for overseeing and managing all activities for this evaluation, while the Transparency Expert, Dr. Olga Nazario, will support Ms. Reimer in analyzing and assessing the Project and in making recommendations to ensure full compliance with the Projects’ 45 goals and objectives. The team will be assisted by Logistics Coordinator Ana Maria Ordoñez and supported by dTS’ home office Project Management Team (PMT) under the leadership of Director Dr. Malcolm Young with the assistance of Sandra Medina. Table 1: Team Composition Position Name Responsibility Evaluation Team Leader Melanie Reimer Technical: Lead day-to-day implementation of the evaluation, work planning, lead/develop data collection protocols, lead/conduct review of reports and documents, lead validation workshop and USAID de-briefing, report writing. Management: Maintain communications with the COR and other USAID representatives, management of the evaluation team, coordinate field and HO communications. Transparency Expert (Anti￾corruption & Governance) Olga Nazario, PhD Provide support in evaluation design; provide transparency sector and Salvadoran contacts and context expertise; conduct key informant and survey interviews; review reports and documents; contribute to validation workshop and debriefing, draft and final report writing. Logistics Coordinator Ana María Ordóñez Arrange interviews; provide transportation and other logistical support; assist in summarizing documents and interview data. Home Office Project Management Team Director Mal Young Support evaluation design/reporting, technical oversight. Associate Sandra Medina Contract administration, financial management, CO liaison, mobilization, logistics, administrative support, report production. A hallmark of dTS’ management approach is maintaining regular and consistent communications with USAID and important USAID stakeholders. No activity will be undertaken without explicit dialogue with our USAID COR. Our primary point of contact with the COR will be our Evaluation Team Leader. During the Washington-based activities, our Team Leader will be in electronic communication with the COR and during the field phase, she will provide the dTS HO and COR with a weekly status report. The dTS PMT Director and Project Associate will maintain 24/7 availability to respond to any unanticipated needs or emergencies. Evaluation Design General approach 46 This performance evaluation will be tailored to the context and culture of El Salvador. It will draw heavily on our team’s knowledge of the country, theoretical and practical understanding and experience with transparency and governance programming, and will be based on extensive document and data review, personal interviews with key informants, focus groups and site visits and observations. The evaluation team will ensure that all primary beneficiaries of this Project, including government officials, leaders of business associations and civil society organizations are interviewed by the evaluation team, either in focus groups or individually. Specifically, the evaluation team will interview the head of the Government’s Secretariat of Transparency, members of the Government Ethics Tribunal, the Minister of Public Works, the Court of Accounts, Magistrates of the Supreme Court and if possible, Deputies who played an important role in the passage of transparency legislation, such as the Access to Information Law and the Government Ethics Law. With respect to civil society organizations and business associations, the evaluation team will interview members of the Fundación Nacional para el Desarrollo (FUNDE), Fundación Salvadoreña para el Desarrollo Económico y Social (FUSADES), Fundación LIDERA, Iniciativa Social para la Democracia (ISD), the Universidad Centroamericana (UCA), the Salvadoran Journalists Association (APES), the umbrella business organization, Asociación Nacional de la Empresa Privada, (ANEP), and the Salvadorian Construction Association (CASALCO), among others. In conducting our analyses and reaching our conclusions, we will triangulate the findings from documents reviews of existing reports and surveys, GOES legislation and policies, interviews with GOES and other actors in the formal political process, and interviews and focus groups with knowledgeable observers from the media, academic institutions and private sector. Evaluation questions, data sources and data collection techniques In accordance with USAID evaluation guidelines, the evaluation will focus on descriptive and normative questions: (a) what did the Project achieve during the implementation period; (b) how is the Project being implemented? c) how is the Project perceived and valued? More specifically, the evaluation will address the four major and five associated questions and sub-questions set forth in the SOW, which are listed in the table below. Essentially, the dTS team will examine whether the targets established by the Project were met, whether the results and overall objectives were achieved during the time frame being evaluated, and what recommendations can be made to realign or adjust the Project during its remaining to time optimize its chances for success. It will also assess the extent to which different sectors of society, including women, have been involved and contributed to the Project; assess the extent to which the Government of El Salvador (GOES) has taken steps to control corruption; and through the interviews assess the extent to which beneficiaries are aware that the Project is funded by USAID. Table 2, below, lays out the main evaluation questions along with the associated data sources and the collection techniques for each question. The table includes all the questions set forth in the SOW, although in some cases slightly reordered and supplemented by related questions that are also important to address. The evaluation team will use these questions as the basis for developing appropriate data collection instruments. 47 Table 2: Evaluation Questions, Methods and Data Sources Evaluation question Methodological data/collection tool Sources (documents and others) 1. What have been the most significant performance results to date?  Document review  Key informant interviews  Focus groups  Project PMP and Results Framework  Semi-annual Project Reports  Surveys (Democracy Survey, LAPOP, etc.)  USAID Documents  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Key stakeholders, including Project staff, USAID Officials, Project consultants, leaders of business and civil society organizations and selected academics. 2. What were the major factors influencing the achievement (or non￾achievement) of Project objectives?  Document Review  Key informant interviews  Focus Groups  Semi-annual Project Reports  Project PMP and Results Framework  Selected academic, international and civil society reports on corruption and transparency in El Salvador  USAID officials, Project staff, and Project Consultants  Key stakeholders—government, civil society and business organizations (such as FUNDE, Grupo Promotor of the Access to Information Law; GOES Secretariat for Transparency and the Government Ethics Tribunal).  Selected academics, journalists, and civil society organizations that were not direct beneficiaries of this Project but that have valuable opinions about transparency and accountability problems. 3. How has the current implementation approach positively and/or negatively affected the Project’s results?  Document review  Key informant interviews  Focus groups  Site visits  Semi-annual Project Reports  Project PMP and Results Framework  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Project Staff and USAID officials  Government officials, civil society organizations and business associations that worked with this Project  Selected academics, journalists, international and civil society organizations that were not direct beneficiaries of this Project but that have valuable opinions about transparency and accountability problems. 48 Evaluation question Methodological data/collection tool Sources (documents and others) 4. What recommendations can be made to realign or adjust the Project during the remaining time to optimize chances for success?  Document review  Key informant interviews  Focus groups  Selected academic, international and civil society reports on corruption and transparency  LAPOP Surveys  Democracy Surveys  USAID officials, Project staff, and selected key informants 5. To what extent have different sectors of society been involved and contributed to the Project, including GOES, civil society, private sector, other international donors, and the media?  Document review  Key informant interviews  Focus groups  Site visits  Semi-annual Project Reports  Project PMP and Results Framework  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Media reports  USAID officials, Chief of Party, Project staff, and Project consultants  Key beneficiaries—government, civil society and business organizations  Selected civil society, media and international organizations that were not direct beneficiaries of this Project but who have valuable opinions about transparency and accountability problems. 6. Should other stakeholders be involved during the remaining time to improve the Project’s results?  Document review  Key informant interviews  Focus Groups  Site visits  Semi-annual Project Reports  Project PMP and Results Framework  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Interviews with USAID officials, Project staff, and Project Consultants  Key beneficiaries—government, civil society and business organizations  Selected civil society, media and international organizations that were not direct beneficiaries of this Project but who have valuable opinions about transparency and accountability problems 7. To what extent has the GOES taken steps to control corruption during Project implementation?  Document review  Key informant interviews  Focus groups  Project PMP and Results Framework  Semi-annual Project reports  GOES official documents  USAID documents  Surveys (Democracy Survey, LAPOP, etc.)  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Media reports  USAID, Project team, and selected key informants in government and civil society 49 Evaluation question Methodological data/collection tool Sources (documents and others) 8. How have the Project’s anti￾corruption and transparency initiatives been institutionalized or operationalized in the GOES’ processes?  Document review  Key informant interviews  Focus groups  Project PMP and Results Framework  Semi-annual Project reports  GOES official documents  USAID Documents  Surveys (Democracy Survey, LAPOP, etc.)  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Media reports  Key stakeholders, including Project staff, USAID Officials, GOES officials, Project consultants, and leaders of business and civil society organizations. 9. How do stakeholders value and/or perceive the Project’s results?  Document review  Key informant interviews  Focus groups  Site visits  Surveys (Democracy Survey, LAPOP, etc.)  Media reports  Semi-annual Project reports  Selected academic, international and civil society reports on corruption and transparency in El Salvador  Media reports  Project beneficiaries, including government officials, business associations and civil society organizations  Other national and international stakeholders 10. How aware are beneficiaries of this Project that it is funded by USAID?  Document review  Key informant interviews  Focus groups  Site visits  Semi-annual Project reports  Project Branding Strategy  Project-supported materials (press releases, publications, etc.)  Project beneficiaries, including government officials, business associations and civil society organizations Data Collection Methods in Detail Desk/Document Review: Before arrival in El Salvador, the dTS evaluation team will conduct a desk review of all relevant Project documents made available by USAID/El Salvador and other relevant information available in Washington. At a minimum, these will include: (1) Partnership for Growth Joint Country Action Plan; (2) Casals TAG Task Order; (3) Casals Monitoring and Evaluation Plan (4) National democracy surveys conducted during 2006- 2010 (5) Semi-annual reports from Casals & Associates, containing monitoring data; (6) Monitoring and indicator tracking tables; and (7) additional documents such as the Performance Monitoring Plan, Branding Implementation Plan and annual Project Workplans provided to the team by USAID and Casals during the initial days of the evaluation. The document review will also include other materials identified during the course of the 50 evaluation. All relevant documents will be listed as an annex in the evaluation final report. Based on this review, our team will create a document review matrix, which will contain a list of core documents reviewed in relation to the evaluation questions. A final version of the matrix will be included as part of the final report. Key Informant Interviews: The evaluation team will gain insights and gather data through 25- 30 individual meetings with a representative cross section of stakeholders. These will include representatives from the 12 organizations cited in the SOW, such as USAID/El Salvador, Casals & Associates and Project consultants, GOES officials, civil society organizations, international agencies and independent academic and media experts. They will also include informants identified during the course of the in-country data collection, including some non-beneficiaries to gain a broad perspective of this Project’s results. Key informants will be selected on the basis of their role within relevant organizations or their reputation as knowledgeable or influential individuals. An initial list of key informants is included as an annex to this Evaluation Plan, and USAID will be informed of additional interviewees as they are identified. In conducting the key informant interviews the primary interview style will be a semi-structured interaction, following interview protocols consisting of key issues/questions to be covered yet encouraging interviewees to talk freely about their experiences, observations, concerns, and recommendations. A sample interview guide is attached as Annex 3; other tools will be finalized in the first stages of data collection. This will allow interviews to take unanticipated, often productive directions that are less often achieved in more structured interactions. The interview guides will be tailored to the different types of respondents. Some interviews may be carried out by telephone, especially where the informant is located outside of El Salvador. Focus Groups: We also plan to conduct at least three focus group discussions with Project stakeholders that will be identified once the team is in country. (Where a full-fledged focus group format is not feasible, then a more flexible method will be used to discuss evaluation questions with small groups of informants.) These will explore in greater depth the short and intermediate term results of the Project’s interventions, perceptions of the Project, as well as on-going needs for assistance that the Project, USAID or others could provide. Our approach to conducting focus groups is guided by the practical experience of our team and best practices in qualitative research worldwide. Experience shows that the optimum size is seven to 10 discussants that are carefully selected representatives of the target population. Each focus group protocol will be tailored to the composition of the group and the specific issue(s) being addressed. Our evaluators will serve as the moderators, and careful notes will be taken to ensure that all responses are captured accurately. Site Visits: We will hold interviews and focus group discussions during visits to offices of GOES ministries and offices and to civil society and private sector organizations. To ensure representation of informants from outside the capital, we propose to visit up to three departments outside of San Salvador where we will interview GOES officials and civil society or media representatives. These will be one-day visits with pre-arrangements made in coordination with USAID/El Salvador or the appropriate GOES ministry. As well, we will try to attend any events or Project activities that may be occurring during the field work period, in order to directly observe the work of the Project team and have informal exchanges with relevant stakeholders, which may bring to light new informants to interview 51 or new documents to review. Reports on all such visits will be written up by the team immediately after the visit in order to capture the content of conversations with stakeholders as well as more general impressions. Validation Workshop: At the close of the field phase, the evaluation team will organize a validation workshop to assess the extent of agreement on emerging findings from the evaluation across different groups and to discuss their implications for the Project or future USAID programming. Participants in the workshop will be representatives of the Casals & Associates team, USAID/El Salvador, and GOES stakeholders that USAID deems appropriate. Such workshops can make an evaluation more constructive as implementers and stakeholders have the opportunity for facilitated reflection, building ownership and involvement in the evaluation findings and recommendations. This workshop will assist dTS in further interpreting findings, pinpointing gaps and/or new directions, and assessing likely reactions and implications to potential recommendations. It will be held shortly before the exit briefing by the evaluation team. Implementation Plan dTS will implement the evaluation design in three major phases of Project activity as follows. Phase 1 - Preparation: The work of the evaluation team began with a six-day working period of review, planning, and initial data collection prior to arrival in El Salvador. The first step, within two weeks of contract award, was a conference call amongst the USAID/El Salvador, the COR and CO, and dTS’ home office project management team (PMT). It included a discussion of USAID expectations for the evaluation and its results to make certain of a common understanding, and agreement on the overall evaluation schedule. The second step was to prepare for the fieldwork by reviewing available materials and starting to prepare the Document Review Matrix. Concurrently, our team has finalized arrangements with our logistics coordinator, assembled a draft list of key informants and/or organizations to interview (see Annex 1), and prepared a list of additional documents to be reviewed. We then prepared drafts of interview guides, focus group protocols, and document review protocols, and began making logistical arrangements. A draft Evaluation Plan was submitted to USAID shortly after the team’s arrival in El Salvador. Phase 2 - Field Data Collection and Analyses: dTS’ two evaluators and the logistics coordinator will devote approximately three weeks to collecting and analyzing data in El Salvador. The first step upon the evaluators’ arrival in San Salvador will be for the team to meet with USAID. This will provide the team an opportunity to discuss the Evaluation Plan and to have more detailed discussions regarding the Project with appropriate USAID staff. Following the initial meetings at USAID, our team will meet with the Casals & Associates leadership, obtain and review additional documentation, and confirm logistical arrangements for interviews and visits to organizations and Project beneficiaries. During these meetings we will explain our evaluation approach, clarify expectations and arrangements, obtain additional background documents and lists of activities and beneficiaries, and identify related information regarding entities and individuals to be consulted. The first week in El Salvador will be devoted to additional document reviews, and key 52 informant interviews and site visits in San Salvador. The following two weeks will include additional interviews and focus groups in San Salvador, and visits to up to three departments where we will interview GOES local officials and representatives of civil society. Analysis and validation activities will occur throughout the data collection period. At the end of each week our team will submit a weekly report to USAID. The final week in El Salvador will include a more in-depth synthesizing of findings and preparation of a draft set of recommendations that will be vetted as part of a validation workshop in the final week. Based on the results of the validation workshop our team will prepare a debriefing for USAID/El Salvador which will be delivered one day prior to the team’s departure. Phase 3 - Draft and Final Reporting: Within 7 business days following the validation workshop, the team will provide a draft final report to USAID. It will be in English, consisting of no more than 30 pages excluding annexes. The report will clearly identify methodologies, findings, conclusions and recommendations. It will also include an Executive Summary no longer than 3 pages. Our final report will incorporate comments from USAID on the draft, and be submitted to USAID within one week after we have received comments on the draft. Two copies of the Final Report will be submitted in English and Spanish along with an electronic version with both languages. At the time of submission we will also provide raw qualitative and quantitative data collected during the course of the evaluation. Upon acceptance of the Final Report by USAID, our Team Leader will return to El Salvador and make a presentation of findings, conclusions and recommendations to USAID and Casals & Associates. The presentation will include PowerPoint slides to be provided to USAID in English and Spanish. If needed, dTS will make final adjustments to the report following that presentation. Table 3: Draft Timeline Task Tentative Dates Award signed by dTS 17 October Phase 1: Preparation, including initial discussions with USAID, review of documents, and preparation of Evaluation Plan and travel 17-22 October Phase 2: Initial meeting with USAID and finalize Evaluation Plan (in El Salvador) 23-24 October Collect & analyze data, continue document review, weekly reporting to USAID (every Monday) 24 October-26 November Analyze data, complete document review, develop recommendations, prepare for validation workshop 12-19 November Validation workshop with USAID and Implementing Partner(s) 27 November Exit briefing with USAID and travel home 28-29 November Phase 3: Prepare Draft Report 29 November – 4 December Submit Draft Report in English 5 December Revise Draft Report (assuming that comments are received by COB 11 December) 12-17 December Submit Final Report in English 17 December Finalize presentation and travel of Team Leader to El Salvador 18-19 December Final Presentation (in El Salvador) 20 December 53 N.B. Minor final revisions to the report and/or presentation may be required, and will be completed no later than 15 January 2013. Limitations The evaluation will be subject to certain limitations, which will be described in full in the final report. On a preliminary basis, dTS has identified the following constraints that may affect the evaluation process:  time constraints on data collection and document review activities  flexible nature of the project that has involved a number of changes to plans, priorities and activities  difficulty of finding concrete data on actual levels of corruption  possible unavailability of key informants during the evaluation period  possible lack of candor from some respondents due to sensitive nature of the subject  lack of sufficient knowledge of project activities by some informants. Annex 1 - Potential Informants for Interviews Note: Black text indicates informants that will be interviewed as a top priority, while red text indicates possible informants that have been identified so far by the dTS team. Those are subject to discussion and approval by USAID, which will also apply to any other informants who are identified in the field. In some cases, individual informant names have not yet been identified. Category of Informant Informant Organization Informant Name Donor and implementin g agency (4-6 interviews) USAID  Maria Antonieta Zelaya, COR  Mauricio Herrera Casals staff and consultants  Carlos Guerrero, COP  Otto Vidaurre, Public Sector Expert  Hiram Morales, Anti Corruption Expert  Marta Saprissa, Finance Manager  Erich de la Fuente, Public Awareness Consultant, Miami Government officials (6-8 interviews) Sub Secretariat Transparency and Anticorruption  Marcos Rodriguez, Subsecretary Government Ethics Tribunal  Jenniver Giovanna Vega Hercules, GET Member Ministry of Public Works  Gerson Martínez, Minister 54 Category of Informant Informant Organization Informant Name Attorney General’s Office  Andrés Anaya, Anticorruption Unit Constitutional Court  Sidney Blanco or Florentin Meléndez, judges Government Ethics Tribunal  Lizzette Kury de Mendoza, former president National Assembly  Deputies who could comment on access to info law, Sigfrido Reyes, of FMLN, new president of the Assembly Consumer Defense Agency  TBD Court of Accounts  Marco Gregorio Sánchez, former Public Defender Civil society (private sector) (2-3 interviews) Salvadorian Construction Association (CASALCO)  Ismael Nolasco, Executive Director Asociación Nacional de la Empresa Privada (ANEP)  Javier Siman, President Chamber of Commerce TBD Industrial Society (ISA) TBD Women Entrepeneurs Association TBD Civil society (other) including coalitions or networks (6-8 interviews and/or FGD) FUNDE (TI Chapter)  Roberto Rubio, President or Raul Torres FUSADES/ROL (Grupo Promotor)  Claudia Umaña, formerly FUSADES, now Fundación DTJ FUSADES (Grupo Promotor)  Javier Castro UCA (Grupo Promotor)  José Luis Benitez LIDERA  Lilian Zelaya, President Iniciativa Social para la Democracia (ISD)  Ramón Villata, President FESPAD  María Silva Guillen FUNDAUNGO  Ricardo Córdova, Director Media (2-3 interviews) Salvadoran journalist  Amílcar Mejía, periodista de sección Política, La Prensa Gráfica or  Álvaro Cruz Rojas, Editor Jefe de Diario El Mundo or  Katlen Urquilla, periodista de sección Política, El Diario de Hoy 55 Category of Informant Informant Organization Informant Name El Faro  Carlos Dada Journalists Association (APES) (Grupo Promotor) TBD International Community (2-3 interviews) OAS TBD UNDP  Laura Rivera Transparency International (HQ)  Alejandro Salas Government and CSOs outside San Salvador (3-4 interviews) TBD TBD Other experts, committees, think tanks, training providers (2-4 interviews) Anticorruption expert (founder of Probidad)  Jaime López (alternate - Manuel Castillo) Justice/security expert  Henry Campos Justice/security expert  Pedro Cruz Annex 2 - Draft Final Report Outline Executive Summary Cover Page Table of Contents Acronyms and abbreviations A. Introduction – Purpose & Scope of Evaluation B. Background: Overview of Transparency and Governance Project in El Salvador C. Evaluation Methodology & Limitations D. Major Findings 1. Most significant performance results 2. Major factors influencing achievement of results and effects of current implementation approach on results 3. Involvement of different sectors of society (such as women, civil society, government, private sector, youth) 4. GOES efforts to control corruption and institutionalization/operationalization of transparency and anti-corruption initiatives 5. Awareness of beneficiaries that Project is funded by USAID E. Conclusions & Recommendations 1. Adjustments to optimize chances for Project success 56 2. Recommendations for future projects ANNEXES: (a) Evaluation Scope of Work; (b) Final Evaluation Plan; (Statement of Differences (if needed); (c) Data collection tools and schedule; (d) Information Sources; (e) Evaluation Team composition and qualifications; (f) Disclosure of Conflict of Interest forms; (g) Document Review Matrix. Annex 3 - Sample Interview Guide Government Officials Interview Questions Note: Some introduction to the Project may be required for informants who are not very familiar with its activities and objectives. Questions of lower priority are indicated in grey shading. General Knowledge/Results 1. What interaction (if any) has your ministry/department/unit had with the Project since 2009? (Probe for various ways.) a. Did you collaborate in any way? b. Coordinate? 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? If not, why not? b. What have been the obstacles/challenges? 3. Considering the broad picture of transparency and corruption in El Salvador, which activities were most useful? a. Why? b. In your view, which activity has been the least useful? c. Why? (Probe to see if anything considered harmful or negative, as needed) Work with Government (Objective 1) 4. The Project has provided a lot of training, much of it targeting government personnel. What do you see as the results of the training? a. If you participated yourself, did you learn something useful? b. Can you mention specific examples of change? c. How was the learning of training participants measured (testing, post-training follow up, etc.)? d. How could the Project deliver better training and TA in the future? (Or should the Project offer less training, to focus on other types of support?) 5. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? a. Has the Project helped to strengthen political will for greater transparency? b. How can support to the government be more effective, especially to ensure long term impact? 6. Going beyond the scope of the Project, what do you consider to be the most important steps taken by the government in recent years to combat corruption and promote transparency? 7. Which anti-corruption and transparency initiatives have been institutionalized in the GOES’ processes in recent years? a. What are some examples? 57 b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 8. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground b. how could the Project ensure that policy and legislation translates into effective action? Work with Civil Society (Objective 2) 9. What are the main weaknesses of civil society in general in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? 10. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? Other Beneficiaries and Stakeholders 11. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. How could the Project be more effective in this area? 12. Has the Project involved the appropriate people and organizations/entities in its work? a. Why or why not? b. If not, with whom should it work in the future? 13. Do you think that most stakeholders in government and civil society are aware that the Project was funded by USAID? a. Why do you think so? Recommendations/Other 14. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? 58 Annex C: List of Informants A. Key Informants Interviewed Category of Informant Informant Organization Informant Name 1. Donor and implementing agency USAID El Salvador  Maria Antonieta Zelaya, COR  Mauricio Herrera Coello, Coordinator of Prevention of Violence Program, COR of former project Casals & Associates, Inc.  Carlos Guerrero, Chief of Party  Hiram Morales, Senior Anti Corruption Expert  Otto Vidaurre, Senior Technical Expert, Public Sector  Nohemy Rivera, Technical Expert, Public Sector  Edward Wollants, Technical Expert, Private Sector Government authorities Sub Secretariat Transparency and Anticorruption, within Secretariat for Strategic Affairs  Marcos Rodriguez, Undersecretary Government Ethics Tribunal  Lic. Jennyffer Giovanna Vega Hercules  Dr. Salvador Eduardo Menéndez Leal  Lic. Luis Romeo García Alemán Ministry of Public Works  Gerson Martínez, Minister Attorney General’s Office  Andrés Anaya, Specialized Unit on Corruption Court of Accounts  Dr. Marcos Gregorio Sánchez Trejo, President National Assembly  David Reyes, Diputado Civil society (private sector) Salvadorian Chamber of the Construction Industry (CASALCO)  Ing. Ismael Nolasco, Executive Director 59 Category of Informant Informant Organization Informant Name Salvadorian Association of Industrialists (ASI)  Jorge Arriaza, Executive Director Civil society (non private sector) FUNDE (National Chapter of Transparency International)  Roberto Rubio-Fabián, Executive Director Fundación DTJ  Claudia Umaña Araujo, President UCA (Central American University “José Simeón Cañas”)  Dr. José Luis Benitez, Head of Communications and Culture Department LIDERA  Lic. Lilian Zelaya, President Iniciativa Social para la Democracia (ISD)  José Ramón Villalta, Executive Director Fundación de Estudios para la Aplicación del Derecho (FESPAD)  María Silvia Guillen, Executive Director Media Journalists Association of El Salvador (APES)  Dr. José Luis Benitez, President (also listed under Civil Society) ContraPunto, Digital Newspaper  Juan José Dalton, journalist International Community UNDP  Laura Rivera Marinero, Program Officer, Democratic Governance RTI (Municipal Competitiveness Project of USAID)  José Luis Trigueros, Project Director Spanish Agency for International Cooperation  Roberto Rodriguez Meléndez, Manager of Justice and Governance Program Government and CSOs outside San Salvador Municipality of Usulután  Dr. Francisco Meardi Guevara, Mayor of Usulután Other experts, committees, think tanks, training providers FUSADES  Javier Castro de León, Director of Legal Research Anticorruption expert (formerly of NGO Probidad)  José Manuel Castillo Pérez Gómez Instituto Iberoamericano de Derecho Constitucional  Dr René Fortin Magaña Security sector  Henry Campos Government Ethics Tribunal  Lizeth Kury de Mendoza, Former president 60 B. Focus Groups Conducted 1. Members of Association of Municipalities of Los Nonualcos (6 men, 1 woman) 2. Mayors of Usulután Department (8 men, 3 women) 3. Staff of Information Unit, Under-Secretariat for Transparency and Anticorruption, plus two Information Officials from government institutions (2 men, 5 women) 4. Members of Ethics Commission, Ministry of Governance (3 men, 3 women) C. Site Visits Made 1. Office of Information and Response – Ministry of External Affairs 2. Center for Legal Advice on Anti-corruption (ALAC) – at FUNDE office 3. Transparency House – Ministry of Public Works 4. Training workshop for NGOs conducted by SSTA 5. Government Ethics Tribunal 6. Office of Information and Response – Mayor’s Office of Usulután D. Gender Disaggregation of Informants Type of Informant Male Female Total Key Informant 25 8 33 Focus Group Participant 19 12 31 TOTAL 44 20 64 61 Annex D: Analysis of Monitoring and Evaluation Indicators The tables below contain an abridged version of all M&E indicators of the Transparency and Governance Project (definitions and data sources have been omitted), including in the far right column various observations by the evaluators on discrepancies or challenges in utilization and interpretation of the indicators, as appropriate. Following each major section of the framework, there is a brief analysis of the results as measured by those indicators. Strategic Objective Level Indicator Source Target Actual Observation by Evaluation Team Corruption Perception Index Score TI Annual CPI Report FY2009=4.4 FY2010=3.6 FY2011=3.7 FY2012=3.4 FY2009=3.4 FY2010=3.6 FY2011= 3.4 FY2012=3.8 Targets in PMP have been modest (not clear how they were set); target for 2012 was equal to actual in 2011. Level of perceived government responsiveness LAPOP Survey FY2009=n/a FY2010=39.4% FY2011=n/a FY2012=41.5% FY2009=n/a FY2010=38.87% FY2011= N/A FY2012 =TBD As above. Results of survey should be out soon. Query whether those indicating “a little” and “a lot” of responsiveness should be combined. Analysis of Results  Corruption Perception Index – Following a general downward trend since 2005, the rating was basically stable from 2009 to 2011. Targets set by Project were met in 2010 and 2012.  Level of perceived government responsiveness – The proportion of those surveyed indicating little or a lot of responsiveness has been fluctuating between 34% and 39% since 2004 baseline, but there is no new data since 2010. Activity 1: Establishing and promoting good governance and anticorruption practices within the Government Indicator Source Target Actual Observation by Evaluators Corruption Victimization in the Courts LAPOP Survey FY2009=n/a FY2010=6.0% FY2011=n/a FY2012=6.0% FY2009=n/a FY2010=4.3% FY2011= N/A FY2012=TBD Target for 2012 set above the actual in 2010, unclear why. Corruption Victimization LAPOP Survey FY2009=n/a FY2010=13.4 % FY2011=n/a FY2012=11.0 % FY2009=n/a FY2010=11.4% FY2011= N/A FY2012=TBD Target set for slight reduction. Level of perceived corruption in public employees LAPOP Survey FY2009=n/a FY2010=67.0 % FY2011=n/a FY2012=63.0 % FY2009=n/a FY2010=64.60% FY2011= N/A FY2012=TBD Target set for slight reduction. 62 GOES commitment to key Anticorruption/ Transparency Initiatives as demonstrated on the USAID Commitment Index. GOES publication and Casals verification Y0=0 (2006) FY2009=137.5 FY2010=162.5 FY2011=200 FY2012=237.5 FY2013=275.0 FY2009=40.63 FY2010=130.39 FY2011= ? FY2012=? Lack of clarity about how to count the achievements on this index. Have omitted the details of Commitment Index in this table. Sub-Activity 1.1 Assist new Government to implement transparency, ethics and anti corruption measures as related to priorities proposed in its Government Platform Indicator Source Target Actual Observation by Evaluators 1) Number of Government Officials receiving USG-Supported Anti-Corruption Training Records of trainings, workshops, and conferences results kept by USAID/DSP M&E Program Officer (Casals & Associates). FY2009= 2,000 FY2010=2,000 FY2011=2,000 FY2012=2,000 FY2013=2,000 FY2009= 1,721 FY2010=2,806 FY2011= 1,907 FY2012= 3,462 Targets not revised since beginning of project, for various indicators from here on. 2) Number of GOES officials receiving information on access to information mechanisms to assure increase of transparency. USAID/DSP records on number of manuals, brochures, CDs delivered to GOES officials on the new transparency mechanisms put in place. FY2009= 400+ FY2010=450 FY2011=500 FY2012=500 FY2012=500 FY2009= 35 FY2010= 377 FY2011= 975 FY2012= 406 3) # of Mechanisms for external oversight of public resource used supported by USG assistance Information provided by GOES, MOU, press news, independent reports, websites, International Organization’s Reports (OAS, UNDP, WB, IDB) and others. FY2009=5 FY2010=3 FY2011=3 FY2012=3 FY2013=3 FY2009 =3 FY2010= 3 FY2011= 4 FY2012 = 4 This refers to government mechanisms only. Difficult to distinguish between mechanism and measure. 4) Number of USG￾supported Anti￾corruption Measures Implemented Information provided by GOES, MOU, press news, independent reports, websites, International Organization’s Reports (OAS, UNDP, WB, IDB) and others. FY2009= 6 FY2010=5 FY2011=3 FY2012=3 FY2013=3 FY2009=6 FY2010=6 FY2011= 6 FY2012 = 4 63 Indicator Source Target Actual Observations by Evaluators 5) # of recommendations of the MESICIC review acted upon or completed by the GOES. Subsequent GOES MESICIC Report and response by the OAS Committee of Experts. GOES public pronouncements on the progress of fulfilling OAS recommendations. Y1= 15/30 2005 Y1= 10/41 2007 Y2=16/30 2005 Y2=11/30 2007 Y2= 0/40 2009 Y3=30/30 2005 Y3=21/41 2007 Y3=5/40 2009 *Pending definition of indicator and target FY2011= Idem, to be defined considering the Guatemalan experience. FY2012 = 4 Indicator has proved difficult to measure, hard to verify government compliance. Does not seem that all reported 2012 actions took place in 2012. Sub-Activity 1.2 Continue to support the Government Ethics Tribunal (GET) Indicator Source Target Actual Observations by Evaluators 1A) # of non￾judiciary public officials trained in ethics and Governmental Ethics Law Records of trainings, workshops, and conferences results kept by USAID/DSP M&E Program Officer (Casals & Associates) FY2009=300 FY2010=400 FY2011=400 FY2012=300 FY2013=300 FY2009=1,372 FY2010=1256 FY2011= 424 FY2012= 1,067 Targets not revised since beginning of project, for various indicators from here on. 1B) # of judiciary public officials trained in ethics and Governmental Ethics Law. Records of trainings, workshops, and conferences results kept by USAID/DSP M&E Program Officer (Casals & Associates) FY2009=300 FY2010=100 FY2011=100 FY2012=100 FY2013=100 FY2009= 110 FY2010= 121 FY2011= 0 *(Justice System will be addressed by other USAID program) FY2012= 70 Should be dropped or merged with 1A above, no need to distinguish. 2) A functioning general communication strategy and dissemination tool. GET Management Information System reports; media reports and dissemination materials produced FY2009=Y FY2010=Y FY2011=Y FY2012=Y FY2012=Y FY2009= YES FY2010= YES FY2011=YES FY2012=YES Difficult to define how functional the system must be to qualify for YES. 3) # of additional transparency/ethics mechanisms introduced GET, outside sources, amendments to law, newspaper, etc. FY2009=2 FY2010=2 FY2011=2 FY2012=2 FY2013=2 FY2009=3 FY2010= 3 FY2011= 2 FY2012= 2 May no longer be relevant to introduce more mechanisms. Hard to decide what is a mechanism. 64 Indicator Source Target Actual Observations by Evaluators 4) Number of cases or complaints presented for investigation to the GET. GET records from the complaint section FY2009=5 FY2010=3 FY2011=3 FY2012=3 FY2013=3 FY2009= 184 FY2010= 300 FY2011= 438 FY 2012 = 149 (89 denounced and 60 notifications) Targets need to be updated with GET. 5) Percent of cases or complaints presented result in resolution. GET records from the complaint section. FY2009=Baseline TBD FY2010=TBD FY2011=TBD FY2012=TBD FY2013=TBD / TBD by the TEG FY2009= 267 FY2010= 135 FY2011= 294 FY2012 = 2 No targets being set with GET, and should be reported as a %. Sub-Activity 1.3 Support Passage of the Access to Information Law (APIL) Indicator Source Target Actual Observations by Evaluators 1) # of Salvadorian actions in support of the passage of the APIL. Newspaper articles, aide￾memoires of meetings, visits, letters, press releases and PSAs. FY2009=10 FY2010=n/a FY2011=n/a FY2012=n/a FY2013=n/a FY2009= 14 FY2010= APIL approved FY2011= N/A FY2012= N/A Indicator achieved, but could be usefully updated to track actions in support of the law’s implementation. Sub-Activity 1.4: Auditing of the Court of Accounts Indicator Source Target Actual Observations by Evaluators 1) # of advocacy actions to create an enabling environment to implement the Court audit. Newspaper articles, aide￾memoires of meetings, visits, letters and press releases. FY2009=0 FY2010=1 FY2011=1 FY2012=1 FY2013=1 FY2009=0 FY2010=3 FY2011=3 FY2012= 2 Court was audited, so indicator seems to be achieved unless a further audit is sought. Activity 1 – Analysis of Results  Several indicators on corruption victimization and perceptions were last measured by LAPOP Democracy Survey in 2010, hard to draw any conclusions until new figures are released.  GOES commitment index tracks progress on a wide variety of steps that provide evidence of commitment to transparency and anti-corruption. Many achievements have been recorded since 2009, such as legal reforms and others noted above.  Sub-activity 1.1 - Targets for numbers of officials trained and informed, number of mechanisms and measures implemented by government, etc. have generally been met or exceeded, although targets were not always kept up to date. About 10,000 officials participated in capacity building to date. 65  Sub-activity 1.2 – GET training targets far exceeded. Other indicators were problematic in terms of tracking reporting and resolution of cases, so no findings can be made on results.  Sub-activity 1.3 – APIL passed, indicator met.  Sub-activity 1.4 – Audit of Court of Accounts done, indicator met. Activity 2: Engaging Civil Society in the Fight against Corruption Sub-Activity 2.1: Engage the Private Sector in the Fight against Corruption Indicator Source Target Actual Observations by Evaluators 1) # of transparency and accountability initiatives promoted/implem ented by private sector organizations. Information provided by ANEP, CASALCO, FUSADES, FUNDEMAS, LIDERA, ASI, ASIA, CEDES, AmCham, Chamber of Commerce, education centers, universities, etc.- MOU signed with private sector organizations. Systematization reports. Press news. FY2009=2 FY2010=3 FY2011=3 FY2012=3 FY2013=3 FY2009=2 FY2010=2 FY2011=3 FY 2012 = 9 Use of the term “initiative” now added to mechanism and measure and action, gets confusing to define and measure. 2) Dollar value of transparency and accountability initiatives promoted/implem ented by private sector organizations. Provided by private sector entities and verified by Casals. FY2009=50K FY2010=50K FY2011=50K FY2012=30K FY2013=30K FY2009=$24.3K FY2010= $44,700K FY2011= $35.500 FY2012= $32,448 3) Number of people affiliated with Non￾Government Organizations receiving USG Supported Anti￾corruption Training. Records of trainings, workshops, and conferences results kept by USAID/DSP M&E Program Officer (Casals & Associates). FY2009=200 FY2010=300 FY2011=300 FY2012=300 FY2013=300 FY2009=1,316 FY2010=3,394 FY2011= 2,636 FY2012 = 3,022 Targets not revised since beginning of project, disconnect can be seen clearly in this case. Definition seems to cover any kind of meeting and any duration, which may be inflating the numbers. 66 Indicator Source Target Actual Observations by Evaluators 4) Number of mechanisms for external oversight of public resource used, supported by USG assistance. Casals records and reports verified by Program Officer. FY2009= 6 FY2010= 5 FY2011= 3 FY2012= 5 FY2013= TBD FY2009=3 FY2010= 5 FY2011= 4 FY2012 = 7 This indicator refers to civil society and private sector mechanisms (as compared to the previous one). Sub-activity 2.2: Establishment of the TI Chapter in El Salvador Indicator Source Target Actual Observations by Evaluators 1) # of actions carried out by FUNDE/TI to qualify for designation as a TI Chapter. Information provided by FUNDE, TI, USAID/DSP. FY2009=5 FY2010=5 FY2011=0 FY2012=0 FY2013=0 FY2009 = 1 FY2010 = 2 FY2011= 3 FY2012 = 1 These two indicators seem to duplicate each other. Project link with FUNDE was suspended before ALAC established. In any event, FUNDE is now TI chapter so these will no longer be tracked. 2) # of initiatives carried out by FUNDE/TI to qualify for designation as TI Chapter. Information provided by FUNDE, TI, USAID/DSP FY2009= 5 FY2010= 2 FY2011= 5 FY2012= 5 FY2013= 5 FY2009=4 FY2010=1 FY2011=0 FY2012 = 1 As above. Sub-activity 2.3: Independent Court of Accounts Indicator Source Target Actual Observations by Evaluators 1) # of actions to create an enabling environment for a more independent Court of Accounts or for the creation of a Contraloría General. Newspaper articles, aide-memoires of meetings, visits, letters and press releases. FY2009=1 FY2010=2 FY2011=3 FY2012=3 FY2013=3 FY2009 = 1 FY2010= 3 FY2011=3 FY2012= 2 Difficult indicator to measure, unclear link to project. Activity 2 Analysis of Results  Sub-activity 2.1 – targets exceeded for civil society initiatives and oversight mechanisms, as well as for number of personnel trained (over 10,000). Some definitions are hard to interpret and training targets were very low.  Sub-activity 2.2 – FUNDE was accredited as the Transparency International chapter in 2012, so indicator achieved, though recent support has been limited.  Sub-activity 2.3 – little action has been taken by civil society towards a more independent Court of Accounts 67 Activity 3 – Provide technical assistance and training in support of other openings for relevant transparency initiatives designed to curb corruption  No indicators have yet been developed for this area of activity. 68 Annex E: Data Collection Tools Seven data collection interview guides or questionnaires are presented in this annex. Each questionnaire is headed by the category of informant for which the questions were designed. A. Casals Staff/Consultants - Interview Questions General Results/Challenges 1. What do you consider to be the most important achievements of the Project since 2009? (Probe for specifics and reasons.) 2. Considering the whole picture of transparency and corruption in El Salvador, which Project strategies and activities were most useful? a. Why? b. Which activity has been the least useful? c. Why? (Probe to see if anything considered harmful or negative, as needed) 3. What have been the major challenges or obstacles to achieving the objectives of the Project? (Probe for both internal factors, such as process or implementation difficulties, design/strategy of the Project, as well as external factors that may have prevented activities from having their desired effect, such as lack of government cooperation.) a. Are there specific areas in which the Project has not been able to make as much progress as expected? b. Why? c. Is the Project on track to meet all of its PMP indicators? d. Do the PMP indicators adequately measure the results? 4. Has the Project been able to fulfill its work plans? a. If not, what are the shortfalls and reasons? b. How can these be addressed? 5. The Project has provided a lot of training to various stakeholders. a. What do you see as the results of the training? b. How was the learning of training participants measured (testing, post-training follow up, etc.)? c. Can you mention specific examples of change? d. How could the Project deliver better training in the future? e. Or should the Project offer less training, to focus on other activities? Work with Government (Objective 1) 6. How would you describe the Project’s relationship with the Government? a. Have they been supportive of the activities and objectives? (Probe for examples of support or obstacles.) 7. Do you believe that the Project has helped to build the government’s capacity and responsiveness in the areas of transparency and anticorruption? (Probe for examples.) a. Has the Project helped to strengthen political will for greater transparency? b. How can support to the government be more effective, especially to ensure long term impact? 8. To what extent have the Project’s anti-corruption and transparency initiatives been institutionalized in the GOES’ processes? 69 a. What are some examples? b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 9. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground? b. how could the Project ensure that policy translates into effective action? Work with Civil Society (Objective 2) 10. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? 11. What additional support does civil society (private sector and otherwise) need to work effectively to promote transparency and combat corruption? a. What needs to be done, that has not yet been done? b. Or what needs to be followed up with more work? 12. Has the Project had any effect on coalition-building among CSOs – either in a positive or negative way? a. How? b. How could the Project better support coalitions in civil society? Other Beneficiaries and Stakeholders 13. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications, video spots, etc. In your view, how effective were those activities? a. Why do you think so (any data or evidence?) b. How could the Project be more effective in this area? 14. Has the Project involved the appropriate people and organizations/entities in its work? a. Do you think the balance between working with/supporting government, with civil society (including media and private sector) and targeting the general public has been suitable? b. Should that balance be changed in some way to have more impact? 15. Did the Project consistently coordinate its work with other stakeholders, including other US agencies, Salvadoran and international bodies that are directly or indirectly involved in the sector? a. How, for example? b. Did the activities of the Project complement the work of others, or was there some overlap or confusion? 16. Do you believe that women and men have had equal chances to participate in (and benefit from) Project activities? a. Has the Project made any special efforts to reach out to and include women – in CSOs, in the public, in government, etc.? b. Is there any need for additional effort in that area? 17. What steps did the Project take to ensure that beneficiaries and others were aware of USAID support for its activities? a. Were those steps effective? 70 b. Why or why not? Recommendations/Other 18. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? B. Civil Society Organization and Network/Coalition Interview Questions General Results/Challenges 1. How has your organization (or network) participated in the Project? [Probe for all different types of interaction (training, technical assistance, grants, special events, collaboration, etc.) 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? b. If not, why not? c. What have been the obstacles/challenges? 3. Which Project activity has been the most useful to your organization specifically? a. Why? b. Now considering the broader picture of transparency and corruption in El Salvador, which activities were most useful? Why? c. In your view, which activity has been the least useful? Why? (Probe to see if anything considered harmful or negative, as needed) Work with Civil Society (Objective 2) 4. What are the main weaknesses of civil society in El Salvador in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? (Probe about lack of grants since 2009.) 5. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? 6. The Project has provided a lot of training to various stakeholders. What do you see as the results of the training? a. If you participated yourself, did you learn something useful? b. Can you mention specific examples of change? c. What was the result of that change? d. How was the learning of training participants measured (testing, post-training follow up, etc.)? e. How could the Project deliver better training and TA in the future? f. Or should the Project offer less training, to focus on other types of support? 7. Has the Project had any effect on cooperation and coalition-building among CSOs – either in a positive or negative way? a. How? b. How could the Project better support the work of coalitions in civil society? (Probe for sustainability after the Project is over.) Work with Government (Objective 1) 8. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? 71 a. Has the government been supportive of the Project (and more generally of the push for more transparency)? b. Has the Project helped to strengthen political will for greater transparency? c. How can support to the government be more effective, especially to ensure long term impact? 9. To what extent have the Project’s anti-corruption and transparency initiatives been institutionalized in the GOES’ processes? a. What are some examples? b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 10. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground, and how could the Project ensure that policy translates into effective action? b. Has your organization made any requests for information under the new Law? Other Beneficiaries and Stakeholders 11. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. Why do you think so (any data or evidence?) b. How could the Project be more effective in this area? 12. Has the Project involved the appropriate people and organizations/entities in its work? a. Do you think the balance between working with/supporting government, with civil society (including media and private sector) and targeting the general public has been suitable? b. Should that balance be changed in some way to have more impact? 13. Do you believe that women and men have had equal chances to participate in (and benefit from) Project activities? a. Have you observed any special efforts made by the Project to reach out to and include women – in CSOs, in the public, in government, etc.? b. How could they be improved? 14. Did the Project coordinate its work with other stakeholders, including your organization (or network), other international agencies, other US-funded initiatives? a. How, for example? 15. Before this meeting, were you aware that this Project was funded by USAID? a. How did you know that? b. Do you think that many people know that USAID is supporting these efforts? Recommendations/Other 16. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? 17. Who else do you suggest that we talk with during our evaluation? C. Government Officials Interview Questions 72 General Knowledge/Results 1. What interaction (if any) has your ministry/department/unit had with the Project since 2009? (Probe for various ways.) a. Did you collaborate in any way? b. Coordinate? 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? b. If not, why not? c. What have been the obstacles/challenges? 3. Considering the broad picture of transparency and corruption in El Salvador, which activities were most useful? Why? a. In your view, which activity has been the least useful? Why? (Probe to see if anything considered harmful or negative, as needed) Work with Government (Objective 1) 4. The Project has provided a lot of training, much of it targeting government personnel. What do you see as the results of the training? a. If you participated yourself, did you learn something useful? b. Can you mention specific examples of change? c. How was the learning of training participants measured (testing, post-training follow up, etc.)? d. How could the Project deliver better training and TA in the future? e. Or should the Project offer less training, to focus on other types of support? 5. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? a. Has the Project helped to strengthen political will for greater transparency? b. How can support to the government be more effective, especially to ensure long term impact? 6. Going beyond the scope of the Project, what do you consider to be the most important steps taken by the government in recent years to combat corruption and promote transparency? 7. Which anti-corruption and transparency initiatives have been institutionalized in the GOES’ processes in recent years? a. What are some examples? b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 8. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground? b. how could the Project ensure that policy and legislation translates into effective action? Work with Civil Society (Objective 2) 9. What are the main weaknesses of civil society in general in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? 10. How do you view the efforts of the private sector to combat corruption? 73 a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? Other Beneficiaries and Stakeholders 11. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. How could the Project be more effective in this area? 12. Has the Project involved the appropriate people and organizations/entities in its work? a. Why or why not? b. If not, with whom should it work in the future? 13. Do you think that most stakeholders in government and civil society are aware that the Project was funded by USAID? a. Why do you think so? Recommendations/Other 14. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? D. International Stakeholders Interview Questions General Knowledge/Results 1. What interaction (if any) has your organization had with the Project since 2009? (Probe for various ways.) a. Did you collaborate in any way? Coordinate? b. In general, did the Project coordinate its work with other stakeholders? c. What could be improved in this area? 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? b. If not, why not? c. What have been the obstacles/challenges? 3. Considering the broad picture of transparency and corruption in El Salvador, which activities were most useful? Why? a. In your view, which activity has been the least useful? Why? (Probe to see if anything considered harmful or negative, as needed) Work with Government (Objective 1) 4. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? a. Has the government been supportive of the Project (and more generally of the push for more transparency)? b. Has the Project helped to strengthen political will for greater transparency? c. How can support to the government be more effective, especially to ensure long term impact? 5. To what extent have the Project’s anti-corruption and transparency initiatives been institutionalized in the GOES’ processes? a. What are some examples? 74 b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 6. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground? b. How could the Project ensure that policy and legislation translates into effective action? Work with Civil Society (Objective 2) 7. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? 8. What are the main weaknesses of civil society in El Salvador in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? (Probe about lack of grants since 2009.) 9. Has the Project had any effect on cooperation and coalition-building among CSOs – either in a positive or negative way? a. How? b. How could the Project better support the work of coalitions in civil society? (Probe for sustainability after the Project is over.) Other Beneficiaries and Stakeholders 10. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. How could the Project be more effective in this area? 11. Has the Project involved the appropriate people and organizations/entities in its work? a. Why or why not? b. If not, with whom should it work in the future? 12. Do you think that most stakeholders know that the Project is funded by USAID? a. How did the Project make that fact known? Recommendations/Other 13. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? E. Media and Thematic Experts Interview Questions General Knowledge/Results 1. What interaction (if any) have you had with the project since 2009? If the informant has little knowledge of the Project, skip to Question 4 and adjust the questions to remove the references to the Project. 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? 75 b. If not, why not? c. What have been the obstacles/challenges? 3. Considering the broad picture of transparency and corruption in El Salvador, which activities were most useful? Why? a. In your view, which activity has been the least useful? Why? (Probe to see if anything considered harmful or negative, as needed) Work with Government (Objective 1) 4. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? a. Has the government been supportive of the Project (and more generally of the push for more transparency)? b. Has the Project helped to strengthen political will for greater transparency? c. How can support to the government be more effective, especially to ensure long term impact? 5. To what extent have the Project’s anti-corruption and transparency initiatives been institutionalized in the GOES’ processes? a. What are some examples? b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 6. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground, and how could the Project ensure that policy and legislation translates into effective action? b. Have you personally (or your organization) requested information under the new Law? c. What has been your experience (or the experience of others)? Work with Civil Society (Objective 2) 7. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? 8. What are the main weaknesses of civil society in general in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? 9. Has the Project had any effect on cooperation and coalition-building among CSOs – either in a positive or negative way? a. How? b. How could the Project better support the work of coalitions in civil society? (Probe for sustainability after the Project is over.) Other Beneficiaries and Stakeholders 10. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. How could the Project be more effective in this area? 76 11. Has the Project involved the appropriate people and organizations/entities in its work? a. Why or why not? b. If not, with whom should it work in the future? 12. Do you think that most stakeholders know that the Project is funded by USAID? a. How did the Project make that fact known? Recommendations/Other 13. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? F. Private Sector Interview Questions General Results/Challenges 1. How has your organization participated in the Project? [Probe for all different types of interaction (training, technical assistance, grants, special events, collaboration, etc.)] 2. Overall, do you think that the Project has had any significant results since 2009? a. If so, what are those results? b. If not, why not? c. What have been the obstacles/challenges? 3. Which Project activity has been the most useful to your organization (and its members) specifically? a. Why? b. Now considering the broader picture of transparency and corruption in El Salvador, which activities were most useful? Why? c. In your view, which activity has been the least useful? Why? (Probe to see if anything considered harmful or negative, as needed) Work with Civil Society (Objective 2) 4. How do you view the efforts of the private sector to combat corruption? a. Has the Project been effective in supporting private sector groups/associations in this area? b. Why or why not? 5. What are the main weaknesses of civil society in general in terms of combating corruption and promoting transparency? a. What needs to happen for CSOs to work more effectively in these areas? (Probe about lack of grants.) 6. The Project has provided a lot of training to various stakeholders. What do you see as the results of the training? a. If you participated yourself, did you learn something useful? b. Can you mention specific examples of change? c. What was the result of that change? d. How was the learning of training participants measured (testing, post-training followup, etc.)? e. How could the Project deliver better training and TA in the future? f. Or should the Project offer less training, to focus on other types of support? 7. Has the Project had any effect on cooperation and coalition-building among CSOs – either in a positive or negative way? a. How? 77 b. How could the Project better support the work of coalitions in civil society? (Probe for sustainability after the Project is over.) Work with Government (Objective 1) 8. Do you believe that the Project has helped to build the government’s capacity in the areas of transparency and anticorruption? a. Has the government been supportive of the Project (and more generally of the push for more transparency)? b. Has the Project helped to strengthen political will for greater transparency? c. How can support to the government be more effective, especially to ensure long term impact? 9. To what extent have the Project’s anti-corruption and transparency initiatives been institutionalized in the GOES’ processes? a. What are some examples? b. What are the difficulties that have been experienced (or that you anticipate) in this area? (Probe on allocation of budget for institutions/implementation/services, appointments, etc.) 10. The Project supported advocacy for the new Access to Public Information Law, among other legislative and policy initiatives. Do you believe that those changes have brought (or will bring) results? a. What are the obstacles to real change on the ground? b. How could the Project ensure that policy translates into effective action? c. Has your organization made any requests for information under the new Law? Other Beneficiaries and Stakeholders 11. The Project supported several activities to help the public to better understand and mobilize to combat corruption, via the media, publications (e.g. citizens’ manual), posters, video spots, etc. In your view, how effective were those activities? a. Why do you think so (any data or evidence?) b. How could the Project be more effective in this area? 12. Has the Project involved the appropriate people and organizations/entities in its work? a. Do you think the balance between working with/supporting government, with civil society (including media and private sector) and targeting the general public has been suitable? b. Should that balance be changed in some way to have more impact? 13. Do you believe that women and men have had equal chances to participate in (and benefit from) Project activities? a. Have you observed any special efforts made by the Project to reach out to and include women – in CSOs, in the public, in government, etc.? b. How could they be improved? 14. Before this meeting, were you aware that this Project was funded by USAID? a. How did you know that? b. Do you think that many people know that USAID is supporting these efforts? Recommendations/Other 15. What recommendations do you have for future USAID programming in transparency, ethics and anticorruption? 16. Who else do you suggest that we talk with during our evaluation? 78 G. Example of Focus Group Questions This set of questions was used with the staff of the Under-Secretariat of Transparency and Anti￾corruption (Oficina de Transparencia). 1. What are the main activities that each of you carries out to promote transparency and combat corruption? a. Follow up questions to clarify, as needed 2. What have been the results of the work that you are doing in the SSTA? How have those results been measured or evaluated up to now? a. Websites, portals b. Training c. Awareness raising d. Accountability reports e. Other 3. What are the areas in which the SSTA has not been able to achieve the results it was aiming for? a. What have been the obstacles? 4. What do you think about the support received by the SSTA from the USAID project implemented by Casals? a. How could that support be more effective? b. What other support does the SSTA need? 5. On a more general level, what needs to be done by the project and by the SSTA in order to promote transparency and combat corruption in El Salvador? 79 Annex F: Matrix of Documents Reviewed Evaluation Questions Document Name What have been the most significant performance results to date? What were the major factors influencing the achievement (or non- achievement) of Project objectives? How has the current implementation approach positively and/or negatively affected the Project’s results? To what extent have different sectors of society been involved and contributed to the Project, including GOES, civil society, private sector, other international donors, and the media? Should other stakeholders be involved during the remaining time to improve the Project’s results? To what extent has the GOES taken steps to control corruption during Project implementation? How have the Project’s anti- corruption and transparency initiatives been institutionalized or operationalized in the GOES’ processes? How aware are beneficiaries of this Project that it is funded by USAID? What recommendations can be made to realign or adjust the Project during the remaining time to optimize chances for success? USAID Evaluation Policy X X Checklist for Assessing USAID Evaluation Reports X X Evaluation Report How To Note X X USAID Youth in Development Policy X X X X USAID Gender Equality and Female Empowerment Policy X X X X Task Order for Project X X X Partnership for Growth Joint Country Action Plan 2011-15 X X Project Branding Implementation and Marking Plan X Project Annual Work plans 2009-10, 2010-2011, 2011-2012 X X X X X Project Annual Work plan 2012-13 (draft) X X X Project Semi-annual report #1 April-Sept 2009 X X X X X X X X 80 Project Semi -annual report #2 Oct 2009 - March 2010 X X X X X X X X Project Semi -annual report #3 April -Sept 2010 X X X X X X X X Project Semi -annual report #4 Oct 2010 - March 2011 X X X X X X X X Project Semi -annual report #5 April -Sept 2011 X X X X X X X X Project Semi -annual report #6 Oct 2011 - March 2012 X X X X X X X X Project Semi -annual report #7 April -Sept 2012 X X X X X X X X Final Report of Transparency and Governance Project 2006 - 9 X X X X Project Monitoring and Evaluation Plan X X USAID Democracy and Governance Office Performance Management Plan Update May 2012 X X X X Project M&E Framework as of Oct 2012 X X X X Transparency International Corruption Perceptions Index (2009, 2010, 2011) X X Cultura política de la democracia en El Salvador: 2006, by LAPOP X X X Cultura política de la democracia en El Salvador, 2008: El impacto de la gobernabilidad, by LAPOP X X X Consolidación democrática en las Américas en tiempos difíciles: El Salvador, 2010, by LAPOP X X X X X Diagnóstico de Necesidades, Ofertas de Servicios y Estrategias de Formación en Anticorrupción y Transparencia, by ISD X X X Dossier de Modalidades de Formación en Transparencia y Anticorrupción, by ISD X X 81 Ley de Acceso a la Informacion Publica - Explicada (Version Borrador), by FESPAD X X Informe de Consultoría. Asistencia Técnica para el Diseño y Puesta en Marcha de las Oficinas de Información y Respuesta en Entidades del Órgano Ejecutive, by FEPADE X X X X MESICIC: Republica de El Salvador; Informe Final; Septiembre 14, 2012 X X X X Análisis de la Nueva Ley de Ética Gubernamental de EL Salvador, by Hiram Morales X X X X Investigación sobre la Tipología de la Información Pública relevante para distintos Sectores de la población, by CID Gallup X X Sistematización de Experiencias y Buenas Prácticas en Transparencia y Anticorrupción desarrolladas por Instituciones del Órgano Ejecutivo, by ISD X X X X Various public information materials produced or funding by the project, such as illustrated APIL for journalists and students, Codes of Ethics, etc. X X X X Reports produced by the Usulután OIR on requests received and handled by the Information Officer X X X X X X Documento de Trabajo sobre Balance en el tercer año de la gestión del Ministerio de Obras Públicas. Administración 2009 - 2014, by FUNDE X X X X X X INFORME DE MONITOREO DE TRANSPARENCIA AL TERCER AÑO DE GESTIÓN DEL GOBIERNO DEL X X X X 82 PRESIDENTE MAUIRICIO FUNES, by ISD Informe General de Cumplimiento Acceso a Informacion – Junio Agosto 2012 by SSTA X X X X Ley de Acceso a la Informacion Publica X X X X Ley de Ética Gubernamental (2011) X X X X Farah, Douglas, "Organized Crime in El Salvador: The Homegrown and Transnational Dimensions," Feb. 2011. X X X UNDP Project Document: Evaluando la Gobernabilidad Democrática en El Salvador: Transparencia y Rendición de Cuentas X X X X Marco de Asociación para el Desarrollo entre El Salvador y España 2010 -2014 X X X X USAID RFP for Justice Sector Improvement Project X X X X Índice de Competitividad Municipal 2011, El Salvador X X X X Summary of Fiscal Policy and Expenditure Management Project of USAID X X X X Compromisos ante la Ciudadan ía para la lucha contra la corrupción y la promoción de la transparencia en la administración pública, by Mauricio Funes X X X X World Bank Worldwide Governance Indicators, Country Report for El Salvador, 1996 -2011 X X X Various media reports, including articles collected by the project, and relevant items in El Diario and La Prensa during data collection in El Salvador X X X X X X X 83 Various websites, including Transparencia Activa, Grupo Promotor, Casals & Associates, USAID/El Salvador, Sub￾secretaria de Transparencia y Anticorrupción, Millennium Challenge Corporation, AECID, Government Ethics Tribunal, etc. X X X X X X X X X Various mission summary reports prepared by consultants engaged by the Project X X X X 84 Annex G: Disclosure of Real or Potential Conflict of Interest Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Melanie Reimer Title Team Leader Organization dTS Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) Contract Number AID-519-O-13-0001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Transparency and Governance Project in El Salvador Implemented by Casals & Associates, Inc. DFD-I-01-08-00069-00 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 4 December 2012 85 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Olga Nazario Title Consultant Organization dTS Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract Number AID-519-O-13-0001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Transparency and Governance Program in El Salvador Implemented by Casals & Associates, Inc. DFD-I-01-08-00069-00 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I was a full time employee of Casals and Associates, Inc. from 1999 to 2010. I served as a Senior Technical Advisor and in that capacity, was aware of the programs Casals had implemented for USAID/El Salvador. I visited the previous Casals program to assist in the closing in 2008 or 2009 and may have contributed briefly to the Casals proposal for the current project. I was, however, dedicated fully to a USAID/Colombia Program implemented by Florida International University and Casals from 2006 to 2010, and had no involvement in the current project. Moreover, I left Casals shortly after the company was sold to DynCorp International in 2010 and have no relationship, awareness, or contacts with the new leadership. All former directors left the company. Over the years, despite leaving Casals, I have kept in touch with Casals’ COP in El Salvador, Mr. Carlos Guerrero. We occasionally exchange small talk over emails. We have not discussed the Program during these contacts, although he may have mentioned satisfaction with an event or accomplishment. I had not seen Mr. Guerrero since the International Conference on Anti-Corruption, held in Greece, in 2008. To guard against any perception of bias, at no time during the evaluation did I meet or communicate with Mr. Guerrero without the involvement of the Team Leader. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date December 2, 2012 x  x x x 86 Disclosure of Conflict of Interest for USAID Evaluation Team Members 87 Annex H: Supplementary Recommendations for Youth Programming In furtherance of the USAID Youth in Development Policy, recommendations for future programming are as follows. 1. Consult with youth and youth organizations in the design of future projects and activities in the transparency and anti-corruption sector. 2. Work with youth and experts to develop exciting themes that will capture and hold their attention, as well as activities that will make short-term concrete differences for youth and their families. 3. Take advantage of the latest technology, especially via mobile phones, to communicate with and mobilize youth at relatively low cost. 4. Develop grant mechanisms to accommodate youth-led organizations, and provide assistance to overcome gaps in capacity and inexperience. 5. Disaggregate monitoring data by age groups, to be able to track the participation of youth and identify needs for action.