Theory of Change August 2011 March 31, 2014 This publication was produced by Amex International, with sub-contractor Panagora Group, for the United States Agency for International Development under Contract No. AID-RAN-I-00-09-00008/AID-438-TO-14- 00002. Written by Milos Cavic, John Doane with the participation of Oyun-Erdene Chuluunbaatar and Zoljargal Naranbaatar. USAID MONGOLIA LOCAL GOVERNANCE MID-TERM EVALUATION FINAL REPORT USAID MONGOLIA LOCAL GOVERNANCE MID-TERM EVALUATION: STAGE AND APPEAL PROJECTS FINAL REPORT DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i TABLE OF CONTENTS ABBREVIATION AND ACRONYMS........................................................................................................................... ii EXECUTIVE SUMMARY.................................................................................................................................................. 1 General Findings and Conclusions............................................................................................................................ 5 General Recommendations........................................................................................................................................ 5 INTRODUCTION............................................................................................................................................................ 6 BACKGROUND ............................................................................................................................................................... 8 Country Context .......................................................................................................................................................... 8 Background on STAGE and APPEAL Projects....................................................................................................... 8 STAGE Project.......................................................................................................................................................... 9 APPEAL Project ........................................................................................................................................................ 9 METHODOLOGY ............................................................................................................................................................ 9 Evaluation Assessment and Methodology ............................................................................................................... 9 Document Review.................................................................................................................................................... 9 Key Informant Interviews....................................................................................................................................... 9 Focus Group Interviews........................................................................................................................................ 10 Performance Monitoring Plan Analysis.............................................................................................................. 10 Limitations of Design and Methodology ........................................................................................................... 10 Evaluation Questions.................................................................................................................................................. 10 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................................................................ 12 General Findings, Conclusions, and Recommendations..................................................................................... 12 General Findings and Conclusions in Relation to the Projects’ Implementation Mechanisms ............ 12 General Findings and Conclusions in Relation to the Project Participatory and Consultative Processes.................................................................................................................................................................. 13 General Findings and Conclusions in Relation to the Project Relevance ................................................. 14 General Recommendations.................................................................................................................................. 15 STAGE Findings, Conclusions, and Recommendations...................................................................................... 17 Partnership in Transparency Initiative (PTI) Targeted Analysis .................................................................. 18 General Administrative Law (GAL) and Administrative Procedure Law (APL)...................................... 19 Targeted Analysis of IAAC Activities ................................................................................................................ 20 STAGE PMP Analysis.................................................................................................................................................. 21 STAGE Recommendations........................................................................................................................................ 25 APPEAL Findings, Conclusions, and Recommendations.................................................................................... 25 Budget Law Implementation, Impact, and Significance................................................................................... 28 Roles and Responsibilities of SGUs: New Budget Law, Decentralization, and Autonomy................... 28 Focus Group Results ............................................................................................................................................. 29 APPEAL PMP Analysis................................................................................................................................................ 33 APPEAL Recommendations...................................................................................................................................... 37 ANNEXES ......................................................................................................................................................................- 1 - ANNEX A: SURVEY INSTRUMENTS.....................................................................................................................- 2 - ANNEX B: METHODOLODY AND ANALYSIS.................................................................................................- 6 - ANNEX C: REFERENCES ....................................................................................................................................... - 18 - ANNEX D: MEETING AND FIELD TRIP NOTES............................................................................................ - 22 - ANNEX E: FIELD TRIP ITINERARY..................................................................................................................... - 35 - ANNEX F: STATEMENT OF WORK.................................................................................................................. - 36 - ii ABBREVIATION AND ACRONYMS AC Anti-Corruption ACAP Anti-Corruption Action Plan ACJWG Anti-Corruption Joint Working Group Aimag: Province or Subnational unit of Government APL Administrative Procedure Law APPEAL Active Partnerships and Public Engagement for Accountable Localities Bagh Sub-unit of Soum BCM Business Council of Mongolia CC Criminal Code CDCS Country Development Cooperation Strategy CPI Corruption Perception Index CPC Criminal Procedural Code CS Civil Society CSO Civil Society Organization COI Conflict of Interest DCHA Bureau for Democracy, Conflict and Humanitarian Assistance DG Democracy and Governance DOC Development Outreach and Communications Officer DOD Department of Defense DP Democratic Party DPO Disabled Persons’ Organization DUC Democratic Union Coalition EG Economic Growth EIU Economist Intelligence Unit EPP Elections and Political Process Division EU European Union FOIA Freedom of Information Act FSN Foreign Service National FSO Foreign Service Officer FY Fiscal Year GAL General Administrative Law GCC General Council of Courts GDP Gross Domestic Product GEC General Election Commission GoM Government of Mongolia GONGOs Government-Oriented Nongovernmental Organizations GOSWS General Office of Social Welfare GPO General Prosecutor’s Office GTZ Deutsche Gesellschaft für Technische Zusammenarbeit HFMD Hand, Foot, and Mouth Disease HRC Human Rights Commission IAAC Independent Authority Against Corruption ICT Information and Communication Technology IEC Information Education and Communication Materials IMF International Monetary Fund IRI International Republican Institute IT Information Technology iii IRs Intermediate Results JICA Japanese International Cooperation Agency KOICA Korean International Cooperation Agency LGBT Lesbian, Gay, Bisexual, Transgender LGU Local Government Unit MALA Mongolian Association of Local Authorities MCA Millennium Challenge Account MCC Millennium Challenge Corporation MDG Millennium Development Goal MNB Mongolian National Broadcaster MNTR Mongolian National Television and Radio MONPAC Mongolian Parliamentarians Against Corruption MONEF Mongolian Employers’ Federation MoPDSP Ministry of Population Development and Social Protection MP Member of Parliament MPRP Mongolian Peoples’ Revolutionary Party MoJ Ministry of Justice MWLA Mongolian Women Lawyers Association NACS National Anti-Corruption Strategy NED National Endowment for Democracy NDI National Democratic Institute NGO Non-Governmental Organization NLI National Legal Institute OCI Organizational Capacity Index OSF Open Society Forum OSCE Organization for Security and Cooperation in Europe OT Oyu Tolgoi Mine PAR Procedural Administrative Regulation PANI Public Administration and New Initiative PSC Personal Service Contractor PMP Performance Monitoring Plan PTI Public Transparency Initiative SAF Strategic Assessment Framework SDP Soum Development Plan SDF Soum Development Fund SGH State Great Hural (Parliament) SGU Sub-national Governmental Unit SOE State-Owned Enterprise Soum Sub-unit of an aimag (county) SMF Sant Maral Foundation SMP Soum Master Plan SPEAK Survey on Perceptions and Knowledge of Corruption STAGE Strengthening Transparency and Governance STOPP Study of Private Perceptions of Corruption TAF The Asia Foundation TT Tavan Tolgoi Mine UNDP United Nations Development Program USAID United States Agency for International Development UB Ulaanbaatar, the Capital City USD United States Dollars iv USG United States Government USGS United States Geological Survey WSP Women for Social Progress 1 EXECUTIVE SUMMARY Mid-term performance evaluations were conducted on two projects funded by USAID/Mongolia: Strengthening Transparency and Governance (STAGE), implemented by The Asia Foundation (TAF), and Active Partnerships and Public Engagement for Accountable Localities (APPEAL), implemented by Mercy Corps. On July 27, 2012, USAID awarded a competitive 26 months cooperative agreement to TAF and Mercy Corps for their respective projects. Together, the two projects have a total ceiling of $6,999,988 ($3.5 million for STAGE and $3,499,988 for APPEAL), and both are scheduled for completion September 30, 2014. Following the Task Order award, the evaluation team—composed of team members from AMEX International, Inc. (Prime Contractor) and Panagora Group (Subcontractor)—began work on January 27, 2014. The work for these evaluations was performed over a period of six weeks. The purpose of these mid-term evaluations was to measure the performance-to-date of two interconnected governance programs, to assess whether and to what extent the planned objectives, outcomes, and results of the projects have been or are being achieved as a result of the implementing partners’ work. To gather the necessary information on each project, the evaluation team conducted an extensive review of relevant documents and reports, designed multiple survey instruments, conducted interviews with key stakeholders, lead six focus groups, and visited six aimags (provinces) and seven soums (counties). Section I presents an overview of the mid-term performance evaluations of the APPEAL and STAGE projects. Section II provides a country overview and background on the two projects. USAID/Mongolia’s Country Development Cooperation Strategy (CDCS) is based on a comprehensive understanding of the Mongolian context, particularly in the economic growth (EG) and democracy and governance (DG) sectors. In 2012, USAID initiated two projects dedicated to achieve the Development Objective— Government Effectiveness and Accountability Improved—through implementation of activities under two Intermediate Results (IRs). The STAGE project is implemented at the national level, while the APPEAL project is implemented in 15 aimags and 60 soums. The locally-oriented APPEAL project is a counterpart to, and coordinated with, the center-focused STAGE project. Together the implementing partners’ goal is to strengthen anti-corruption efforts and improve the administrative reform process. The goal of the STAGE project is to strengthen democratic governance by building a more transparent and accountable regulatory and legislative environment as well as ensuring the principles of checks and balances. Among the primary beneficiaries of the project are parliamentarians, government officials, civil society organizations (CSOs), legal professionals, as well as public and private sector businesses. The goal of the APPEAL project is to promote democratic reforms, economic growth, and to improve legal, institutional, and administrative processes in targeted governmental agencies. The primary objectives of the project are to strengthen the awareness and understanding of relevant legal frameworks related to anti-corruption and good governance as well as supporting engaged, informed action in localities thereby animating good governance through Mongolia’s legal framework. Section III presents the methodology of the mid-term evaluations and its four methods of acquiring information: a) Document Review; b) Key Informant Interviews; c) Focus Group Interviews; and, d) Performance Monitoring Plan Analysis. The evaluation methodology combined quantitative and qualitative data collection and evaluation techniques. Following the initial document review, the evaluation team engaged both internal and external stakeholders (i.e., USAID technical and project staff, implementing partners, Government of Mongolia ministry staff, regional and local government staff, business leaders, civil society and community leaders, average citizens, and the media) in the design, collection of data, and interpretation of findings. 2 Section IV details findings, conclusions, and recommendations drawn from the two mid-term project evaluations. Both STAGE and APPEAL are well conceived democracy and governance projects that are being implemented successfully by their respective implementers (See Figures I and II below). There are many positive findings regarding information dissemination, participation, anti-corruption efforts, good governance practices, and gender role equality. This section also describes STAGE and APPEAL Performance Indicators (Figure 5 and Figure 9). The resulting score of 75% reveals that the STAGE project has either fully or partially achieved targets set against indicators for FY 2013. For two of the indicators the planned result was not achieved as Year 2 was a baseline year (no target, no expected results). The resulting score of 64% reveals that APPEAL project has either fully or partially achieved targets set against indicators for FY 2013. For four of the indicators the planned result was not achieved. Annex A provides three survey instruments (STAGE survey instrument, APPEAL survey instrument, and Focus group survey instrument) that the evaluation team used to collect information from primary sources. Annex B contains the detailed methodological approach, theoretical background, data collection and analysis, as well as project performance indicators and PMP analysis. References are contained in Annex C. Annex D includes all the meeting notes and field trip notes. The field trip itinerary appears as Annex E. Figure 1. STAGE Mid-Term Summary Table Requirement/Goal Result Intended Result Achieved Quarterly Reports --- Submitted PMP --- Conducted Branding Strategy --- Substantial compliance Marking Plan --- Substantial compliance Objective 1. Promoting Good Governance to Reduce Corruption and Increase Transparency • Result 1.1: Increased knowledge of the modalities of grand corruption • Result 1.2: Strengthened legal and policy environment for transparency and good governance • Result 1.3: Increased capacity of relevant AC agencies to fulfill their mandate • Result 1.4: Increased private sector engagement in anti-corruption • Result 1.5: Increased capacity of civil society to monitor anti-corruption in the public and private sectors through innovative modalities Considerable Progress: Planned activities have been initiated and many are substantially completed. Existing anticorruption legislation has been used for raising awareness and knowledge among stakeholders on corruption and anticorruption activities. Empowerment of AC agencies has been useful for coordinating anticorruption activities as well as for the agencies’ involvement in the promotion of anticorruption and good governance activities. 3 Objective 2. Improving Administrative Reform • Result 2.1: Improved understanding on administrative practice in key agencies • Result 2.2: Strengthened legislative framework for administrative reform • Result 2.3: Increased support for and awareness of administrative reform. • Result 2.4: More accountability, transparency, and procedural clarity in pilot administrative agencies Partial Progress: Actions initiated but some have not been completed due to dependence on passage of administrative legislation. While understanding on administrative practices in key agencies has been improved and existing anticorruption legislation has been substantially implemented according to the planned activities, the majority of activities related to strengthening legislative framework for administrative reform and support for and awareness of administrative reform have been postponed due to delays in adoption of GAL and APL. Further progress has been hindered by failure of Parliament to enact proposed reform laws 4 Figure 2. APPEAL Mid-Term Summary Table Requirement/Goal Result Intended Result Achieved Quarterly Reports Submitted PMP Conducted-issues outstanding Branding Strategy Substantial compliance Marking Plan Substantial compliance Objective One—Improve key elements of Mongolian legal frameworks related to anti-corruption and good governance • Result 1.1 Legislation related to anti-corruption and good governance understood and strategized at the local level • Result 1.2 APPEAL community stakeholders understand good governance and anti-corruption initiatives related to specific government authorities at local level • Result 1.3 APPEAL and STAGE stakeholders exchange findings on substantive and sectorial policies and practices to advance anti-corruption and good governance throughout Mongolia Implementation is continuing. Most planned activities in support of the Objective have been initiated. Increased public participation is evident; Transparency has increased; There has been significant progress toward the 3 intended Results, in large part due to increased transparency resulting from FOI, and new Budget Law Objective Two—exercise of authority and delivery of government services reflect transparent and accountable administrative processes, provide effective checks and balances, and engage informed communities to optimize quality and impact of governance at local level • Result 2.1 Improved consistencies of procedural administrative regulations (PARs). • Result 2.2 Civil society has increased capacity to plan and deliver high-impact anti￾corruption and good governance initiatives. • Result 2.3 Local governance is more transparent and inclusive. Implementation is continuing. Most planned activities in support of the Objective have been initiated. There has been significant progress toward the 3 intended Results due largely to FOI law, increased transparency and anti-corruption campaigns. Implementation of the new Budget Law has empowered SGUs, which encouraged public interest in Aimag and Soum activities and administration. 5 GENERAL FINDINGS AND CONCLUSIONS • STAGE effectively promotes good governance at the national level by encouraging transparency, monitoring, and auditing; however, more needs to be done by GoM to encourage greater applicability of the anti-corruption legislation to agencies and to increase incidents of criminal prosecution for violators, but this is beyond the scope of STAGE to control. Similarly, increased transparency at national and sub-national levels of government, encouraged by Parliamentary legislation on freedom of information, has occurred; however, the failure of Parliament to enact proposed administrative reform legislation (i.e., GAL and APL) has deterred further progress regarding anti-corruption and increased transparency. • Good governance at subnational levels in aimags and soums is promoted effectively by APPEAL and has gained momentum. Citizens’ expectations of transparency, participation, and the implementation of further desired results are enduring a highly significant transformation in comparison to the former system. Across all six focus groups, the evaluation team found that the majority of participants feel that the APPEAL project engaged stakeholders and beneficiaries in various aspects of governance, especially through the activation of Citizens Chambers, development of SMPs and SDFs as well as in process of monitoring of implemented ACAPs. Overall conclusion is that APPEAL contributed to the increased knowledge and understanding of good governance and anti-corruption. • Branding and Marking policies of USAID are substantially respected and implemented by both projects in the majority of cases. Nonetheless, there are a few cases of non-compliance with the Branding and Marking policies that are presented in the main analysis of this report. • Both projects addressed some needs of common citizens. As result of both projects’ activities those national and local institutions that are involved in the projects are more transparent. Furthermore, through participation in Citizens Chambers, citizens have the opportunity to articulate their needs and influence on local government. • STAGE monitoring of activities are consistent with the work plan requirements. It should be noted that all interviewed parties expressed satisfaction with TAF’s management of the STAGE project activities. In same manner, communication between the TAF and its institutional and civil society partners was assessed as timely and efficient. • APPEAL sub-grantees are numerous and monitored through regional offices which are in turn monitored by the Headquarters in Ulaanbaatar. This method of monitoring is more cost effective than if the local project offices and sub-grantees are monitored by the headquarters as well. To date, APPEAL, in coordination with sub-grantees, has been able to successfully promote public awareness, civic participation, responsive government, and anti-corruption to large segments of SGU populations. • STAGE and APPEAL have successfully utilized findings from the two USAID funded public surveys on corruption perceptions conducted by the SMF: SPEAK and STOPP. Both projects and IAAC have broadened their anti-corruption activities on those institutions that were indicated by the surveys to be most corrupted. • STAGE has successfully launched and coordinated, together with IAAC, activities of Public Transparency Initiative (PTI) in order to strengthen anti-corruption and good governance efforts and gain momentum on implementation of already existing anti-corruption legislation, COI law, and Criminal Code. GENERAL RECOMMENDATIONS • As the projects near conclusion, implementers should make a concerted effort to emphasize the sustainability of good governance after the projects end. Additionally, the evaluation team encourages implementing partners to identify opportunities to provide some level of continued consultation for partners after the conclusion of the two projects. 6 • As the STAGE project is nearing its end and adoption of NACS, CC, and CPC will be decided during the Parliament’s spring session, it is recommended that advocacy efforts toward their adoption be terminated consistent with the end of that parliamentary session. Consider reallocation of STAGE project funds/resources from MONPAC and efforts to enact proposed COI, GAL, Criminal Code, Criminal Procedure, and Administrative Reform Law to more viable reform prospects such as enacting legislation providing broader powers related to anti-corruption to IAAC. STAGE should review, in consultation with GoM and other relevant stakeholders, the likelihood of GAL, COI, and Administrative Procedure reform laws being enacted and determine whether its continued efforts are a useful expenditure of limited resources during the remainder of the project’s duration. It may be more useful to focus on an alternative approach to reform by advocating another legal mechanism such as broadening the scope of IAAC’s powers to monitor, audit, and initiate legal action for suspected corrupt activities in Ministries in addition to the limited jurisdiction currently authorized. • In the last round of SPEAK and STOPP surveys, TAF and SMF should include control questions regarding STAGE and APPEAL projects. The purpose of the control questions would be to find out the level of awareness about the projects and their activities among the general population. Rational for such amendments to the surveys lies in the fact that, when STAGE and APPEAL projects come to their end, the control questions would provide USAID with a solid reference point for the final impact assessment. • As the APPEAL project is nearing its end, it is recommended that Mercy Corps and other project partners continue previous work on capacity building of local CSOs. This is an area that should receive special attention. APPEAL should be training local CSOs on mechanisms and strategies of fundraising as knowledge of fundraising is crucial for the survival of local CSOs. This will ensure the sustainability of anti-corruption and good governance activities to the greatest extent possible, even after termination of APPEAL project. • APPEAL should clarify project implementation success by using nontechnical language to describe program results and impact. It should engage the services of a public relations expert to better convey pertinent information of this well-executed program. • There needs to be greater emphasis on sustainability of selective and identifiable good governance practices after the USAID funded projects end. • There should be continued cooperation between STAGE, APPEAL, and IAAC on supporting implementation of ACAPs at the national and local level, as well as monitoring of implementation of already adopted ACAPs. • Documentation of APPEAL PMP indicators would benefit from general improvement in quality and clarity. For the APPEAL project, it is challenging to reconcile indicators in the PMP with furnished documentation as some are not adequately supported by documentation. INTRODUCTION Mid-term performance evaluations were conducted on two projects funded by USAID/Mongolia: Strengthening Transparency and Governance (STAGE), implemented by The Asia Foundation (TAF), and Active Partnerships and Public Engagement for Accountable Localities (APPEAL), implemented by Mercy Corps. Together, the two projects have a total ceiling of $6,999,988 and are scheduled for completion in September 2014. The purpose of these mid-term evaluations was to measure the performance-to-date of two interconnected governance programs, to assess whether and to what extent the planned objectives, outcomes, and results of the projects have been or are being achieved as a result of the implementing partners’ work. The evaluation findings provide USAID/Mongolia and its partners with the opportunity to better understand: 7 • The extent to which the planned and related objectives, outcomes, and results of the projects are being achieved • The efficiency and effectiveness of the projects in achieving their objectives, outcomes, and results; and assess areas in need of improvement and areas of success • Project activities that have had the most tangible results and benefits from the perspective of key stakeholders and beneficiaries • Areas of overlap in activities of other donors and opportunities for donor coordination The work for these evaluations was performed over a period of six weeks, beginning January 27, 2014. The evaluation team began by gathering, reviewing, and analyzing relevant background information for both projects and their respective implementers, including project assessments, cooperative agreements between USAID and implementing partners, performance monitoring plans (PMPs), yearly work plans, quarterly reports, statements of work, surveys, gender studies, website posts, and other relevant materials developed by USAID and its implementing partners. The evaluation methodology combined quantitative and qualitative data collection and evaluation techniques. Following the initial document review, the evaluation team engaged both internal and external stakeholders (i.e., USAID technical and project staff, implementing partners, Government of Mongolia ministry staff, regional and local government staff, donor groups, business leaders, civil society and community leaders, average citizens, and the media) in the design, collection of data, and interpretation of findings. Using three survey instruments, the evaluation team collected information from primary sources through a combination of key informant interviews and focus groups. At the state level (STAGE), the evaluation team conducted interviews with available officials from the Government of Mongolia’s ministries and agencies, business leaders, representatives from the Cabinet Secretariat and Civil Service Council, judges and court staff, and members of legal professional organizations. During field visits (APPEAL), the team conducted interviews with key beneficiaries in the provinces to assess ongoing projects, collect evidence of collaborative efforts by implementing partners, and to gauge the perceptions of those involved in individual activities. The team engaged regional government officials and representatives of local professional, civil society, and anti-corruption organizations. The team also conducted six focus groups with local Civil Society Organizations (CSOs), beneficiaries, and non￾beneficiaries. The evaluation team then analyzed the collected data to develop evidence-based findings—and conclusions based on analysis of those findings—in terms of project results and sustainability, efficiency and effectiveness of project activities, inclusivity for various identity categories, and other evaluation parameters. These are presented in terms of cross-cutting findings and conclusions, as well as project￾specific findings and conclusions. The evaluation team also developed concrete recommendations for the current activities based on their findings and conclusions that are action-oriented, practical, specific, and that provide defined responsibility for the recommended actions. These recommendations are also presented in terms of cross-cutting recommendations, as well as project-specific recommendations, for future activities. The intended audience of this mid-term evaluation report is the USAID/Mongolia Mission, specifically the democracy and governance team, and the implementing partners (TAF and Mercy Corps). 8 BACKGROUND COUNTRY CONTEXT USAID/Mongolia’s Country Development Cooperation Strategy (CDCS) is based on a comprehensive understanding of the Mongolian context, particularly in the economic growth (EG) and democracy and governance (DG) sectors. Once a Soviet satellite state, Mongolia transitioned into a parliamentary republic shortly after the collapse of the Soviet Union. The transition from the former system to the present system has experienced many of the problems encountered by other emerging states, including corruption, lack of transparency, political favoritism, immature public and private institutions of governance, lack of effective economic regulation, and immature civil society organizations. Mongolia is at a critical turning point. In the past 20 years, Mongolia has progressed significantly in its transformation from a centrally-planned state to a democratic market economy. Transparency International ranked Mongolia 83rd out of the 175 countries surveyed in its 2013 Corruption Perception Index, showing notable gains over its rank the previous two years (94th in 2012 and 120th in 2011). However, rapid economic growth, led by the mining sector (especially gold, copper, and coal), has the potential to threaten previous gains in governance, environmental protection, the social sector, gender equality, and capacity. Current strategies and decisions will have significant implications for the country’s future. Mining revenues should translate into significant new financial profits for both the GoM as well as the private sector. However, the GoM faces significant impediments toward successfully managing this windfall, including limited administrative capacity, an inefficient bureaucracy, declining government accountability, a vulnerable and undiversified economy, an undereducated and youthful population, and growing income disparities. While Mongolia’s rich endowment of mineral resources promises major economic benefits, it also brings even greater opportunities for mismanagement and corruption. In order to maximize its assets, Mongolia needs to manage its mineral wealth, harnessing its transformative properties to lower poverty, raise household incomes, increase political participation (particularly among women and the poor), and steadily improve living standards for all Mongolians. This can be accomplished through improved governance, policy reform, and enhanced private sector and civil society participation. BACKGROUND ON STAGE AND APPEAL PROJECTS In 2012, USAID initiated two projects dedicated to achieve the Development Objective - Government Effectiveness and Accountability Improved - through implementation of activities under two Intermediate Results (IRs): • IR 2.1 Anti-Corruption Efforts Strengthened • IR 2.2 Legal Reforms Enhanced The two projects funded by USAID/Mongolia are: Strengthening Transparency and Governance (STAGE), implemented by The Asia Foundation (TAF), and Active Partnerships and Public Engagement for Accountable Localities (APPEAL), implemented by Mercy Corps. On July 27, 2012, USAID awarded a competitive 26 months cooperative agreement to TAF and Mercy Corps for their respective projects. Together, the two projects have a total ceiling of $6,999,988 ($3.5 million for STAGE and $3,499,988 for APPEAL), and both are scheduled for completion September 30, 2014. The STAGE project is implemented at the national level, while the APPEAL project is implemented in 15 aimags (provinces) and 60 soums (counties). The locally-oriented APPEAL project is a counterpart to and coordinated with the center-focused STAGE project. Together the implementing partners’ goal is to strengthen anti-corruption efforts and improve the administrative reform process. 9 STAGE Project The goal of the STAGE project is to strengthen democratic governance by building a more transparent and accountable regulatory and legislative environment as well as ensuring the principles of checks and balances. To achieve this overarching goal, the two primary objectives for the STAGE project are to promote good governance, thereby reducing corruption, and improve administrative reforms. Among the primary beneficiaries of the project are parliamentarians, government officials, civil society organizations (CSOs), legal professionals, as well as public and private sector businesses. APPEAL Project The goal of the APPEAL project is to promote democratic reforms, economic growth, and to improve legal, institutional, and administrative processes in targeted governmental agencies. The primary objectives of the project are to strengthen the awareness and understanding of relevant legal frameworks related to anti-corruption and good governance as well as supporting engaged, informed action in localities thereby animating good governance through Mongolia’s legal framework. Both of the projects evaluated reflect USAID’s continuing commitment to and comparative advantage in, providing assistance to develop Mongolia’s democratic government. Sustainable improvements in good governance, including the greater participation of women, the disabled, and other marginalized groups at all levels of government and civil society, are necessary to strengthen democratic principles of pluralism, equality, and representation in Mongolia’s government. This approach will enhance the capacity of the government to transparently advance national development goals including sustained economic growth and reductions in poverty. METHODOLOGY EVALUATION ASSESSMENT AND METHODOLOGY USAID’s Evaluation Policy requires that performance evaluations should be evidence-based. This evaluation is designed to meet that requirement by employing a mixed methods approach to searching for and examining available evidence of changes attributable to the two projects evaluated and to addressing the key evaluation questions. The evaluation methodology predominantly used qualitative and to a lesser extent quantitative data collection and evaluation techniques as support to the qualitative nature of this research. The evaluation team worked with USAID/Mongolia to identify the full range of STAGE and APPEAL supported activities and organizations at the national and local levels. Specifically, the evaluation team’s approach to gathering data and information to assess the two projects includes the following methods. Document Review The evaluation team examined project documentation related to the design, objectives, and achievements of each project. The evaluators assessed: a) the degree to which each projects’ objectives and intermediate results are articulated qualitatively and quantitatively; and b) the quality of data collected to measure indicators of progress toward achieving targeted results. In addition to the primary project documentation, the evaluation team reviewed and analyzed additional documents from both implementing partners and other relevant, publicly accessible documents. Key Informant Interviews In order to obtain in-depth knowledge of both projects, the team constructed two semi-structured interview survey instruments (Annex A). In coordination with USAID, the evaluation team compiled a list of public officials and other key informants to obtain their observations on the respective projects’ impact on management, outcomes, and participation. In the city of Ulaanbaatar, for both STAGE and APPEAL projects, the team gathered information using the STAGE and APPEAL survey instruments, thereby allowing the team to tailor each interview to the context and participant. The data collected from these interviews was then summarized in short reports and served as the foundation for further 10 interviews and comparisons with other data (Annex D). For the remaining interviews, based on logistical constraints and the recommendations of USAID, the evaluation team selected one aimag from each region. In total, the evaluation team visited six aimags and seven soums while conducting the APPEAL project evaluation. (For more on site selection, see Annex E). For these interviews with local officials, the evaluation team used the APPEAL survey instrument to gather information at the Sub-Governmental Unit (SGU) level and explore the connection between the APPEAL project and observed impacts on management, outcomes, and participation at the SGU (aimag or soum) level. Finally, the evaluation team consolidated the information gathered from both survey instruments, providing a summary of the findings and allowing for its disaggregation (see Figure 3). Focus Group Interviews The evaluation team organized six focus groups, one in each of the aimags visited, and worked to ensure that each focus group interview was designed and conducted in accordance with USAID guidelines. The aim of each was to identify participant groups’ perceptions on the APPEAL project and its activities. Questions for the focus group were categorized into following areas: 1) Knowledge, 2) Management, 3) Outcomes, and 4) Good Governance. The final category was the subdivided into: i) Participation, ii) Government Transparency, iii) Government Accountability, iv) Government Effectiveness, v) Sustainability, and vi) Gender Equality/Inclusion. At the conclusion of each portion of a discussion, the facilitator would ask participants to evaluate the APPEAL project, as a group, using a Likert scale. Each group’s perception was recorded in a matrix and then further aggregated in a single matrix that included all six focus group interviews (Figure 6). When they occurred, evaluation team members recorded dissenting opinions within a group in their notes. It should be noted that data collected from the focus groups is qualitative in nature and that the matrix functions as a supplementary and complementary instrument to the qualitative analysis of the focus groups. Performance Monitoring Plan Analysis Analysis of the PMP indicators includes baseline data, targets, and actuals for Fiscal Year 2012 and Fiscal Year 2013 for both projects. As Fiscal Year 2012 is truncated (three months), our PMP analysis for both projects concentrated on Fiscal Year 2013 (October 1, 2012 through September 30, 2013). Limitations of Design and Methodology The timeline and budget for the project dictated certain limitations for the collection of data. The evaluation team predominantly used qualitative methods of data collection and analysis. Primary data collection of country statistics was not feasible; however, the evaluation team carefully scrutinized secondary sources of quantitative data (e.g., government statistics, legislative reports, donor and NGO data, and surveys conducted by project implementers, partners, and local civic and professional organizations). Where possible, the data was disaggregated by gender, age, geographic location, and other important characteristics to observe group-specific impacts of the projects. Similarly, site selection and participants were subject to the limitations of time and availability. For example, during the mid￾term evaluation, Dornogobi Aimag had a quarantine imposed to limit the spread of Hand, Foot, and Mouth Disease (HFMD) discovered among its cattle herds. Arkhangai Aimag was selected as an alternate location for that region. EVALUATION QUESTIONS Even though the evaluation team used three different survey instruments to gather necessary information about APPEAL and STAGE projects, we found that the common denominator or starting point of all three surveys can be summarized within three main topics: Project Implementation Mechanisms, Project Participatory and Consultative Processes, and Project Relevance. It is important to mention that these questions were adjusted, altered, or even expended across all three surveys in order to create the most suitable approach for data collection. In interviews conducted with STAGE and 11 APPEAL partners on the national level, Survey Instrument 1 was used because it is designed to extract more general data. For interviews conducted with local officials related to the APPEAL project, the team constructed Survey Instrument 2 to collect more detailed data about SGU and project performance. When it comes to the focus groups, once again we altered the original set of questions to adjust them to the needs of a focus group (i.e., we needed to scrutinize the number of questions and generalize even further in order to foster participant’s conversation). Source questions that were drawn upon for the various survey instruments include: Project Implementation Mechanisms: • How effectively and efficiently are the projects being implemented? • What are the primary activities of each project that contribute to the achievement of the intended results? • From the activities already undertaken, what activities have not revealed any measurable intended results? • To what extent are the projects’ management structures effective? • To what extent are the projects reaching/benefiting all sectors of the target population, and what obstacles, if any, have they encountered? • What are the projects actually doing and delivering, and how does that compare with their respective project design? • What have the projects delivered so far (at different milestones) and how does that compare with the project design? • Are the projects being implemented and managed in a cost-effective manner? Project Participatory and Consultative Processes: • How effectively has each of the activities engaged STAGE and APPEAL stakeholders at the local and national level? • To what extent is the project being implemented in a participatory way? • Are all appropriate stakeholders being properly consulted, and what are the methods used to consult with various stakeholders? What, if any, impediments have prevented stakeholders from being consulted? • How satisfied are different stakeholders with the project so far? • What role did gender play in the implementation and results of the project? • To what extend is the project gender equitable? • Has the project integrated gender considerations into its activities? • Has the project developed any measures to enhance women’s participation in governance? Project relevance: • What do beneficiary groups see as the goals of the project? • Who received services, and what services did they receive? • What was the situation of the target population before the project began? • To what extent does the project respond to their identified needs? 12 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The following section presents findings, conclusions, and recommendations drawn from the two mid￾term project evaluations. Below are broad, cross-cutting finding, recommendations, and conclusions. Following this section are findings, conclusions, and recommendations that are project specific. GENERAL FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Both STAGE and APPEAL are well conceived democracy and governance projects that are being implemented successfully by their respective implementers. There are many positive findings regarding information dissemination, participation, anti-corruption efforts, good governance practices, and gender role equality. The methodology and analysis related to the findings is elaborated upon in Annex B. Branding and Marking policies of USAID are substantially respected and implemented by the both projects in majority of cases. General Findings and Conclusions in Relation to the Projects’ Implementation Mechanisms STAGE sub-grantees, such as MALA, WSP, MWLA, PANI, BCM, SMF and the Zorig Foundation, are implementing programs and activities funded by USAID in an efficient and effective manner. Additionally, institutional partners, such as IAAC, MoJ, GCC, have successfully accomplished various activities under the scope of the STAGE project regarding the education of public officials on anti-corruption activities and legislation as well as on institutional cooperation for fighting corruption. Joint activities of STAGE and APPEAL targeting administrative reform and fighting corruption are only partially accomplished. That is to say, STAGE and APPEAL successfully facilitated, in cooperation with other institutional and civil sector partners (MWLA), the drafting of the GAL, APL, and the NACS; however, in spite of STAGE’s strong support, these laws and strategy have not yet been adopted. As a result, the team concluded that administrative reform and building capacity for fighting corruption has been only partially accomplished. Nonetheless, STAGE coordinated activities on drafting the laws and strategy at the national level as well as actively encouraging their adoption, and APPEAL facilitated local, public discussions on the laws and prepared recommendations. STAGE monitoring of activities are consistent with the work plan requirements. It should be noted that all interviewed parties expressed satisfaction with TAF’s management of the STAGE project activities. In same manner, communication between TAF and its institutional and civil society partners was assessed as timely and efficient. STAGE has successfully utilized findings from the two USAID funded public surveys on corruption perceptions conducted by the SMF: SPEAK and STOPP. For example, SMF pointed out that the land utilization, property ownership, and procurement as the areas where grand corruption most frequently occurs. As a result, STAGE and IAAC broadened their anti-corruption activities on those institutions that were indicated by the surveys to be most corrupted. In order to strengthen anti-corruption and good governance efforts and gain momentum on implementation of already existing anti-corruption legislation, COI law, and Criminal Code, STAGE has successfully launched and coordinated, together with IAAC, activities of the Public Transparency Initiative (PTI), which is bolstered by a network of 18 CSOs among which are major STAGE sub￾grantees such as the Zorig Foundation, Globe International, and MALA. STAGE effectively promotes good governance at the national level by encouraging transparency, monitoring, and auditing; however, more needs to be done by GoM to encourage greater applicability of the anti-corruption legislation to agencies and to increase incidents of criminal prosecution for violators, but this is beyond the scope of STAGE to control. Similarly, increased transparency at national and sub￾national levels of government, encouraged by Parliamentary legislation on freedom of information, has occurred, but the failure of Parliament to enact proposed administrative reform legislation (i.e., GAL and APL) has deterred further progress regarding anti-corruption and increased transparency. 13 APPEAL sub-grantees are numerous and monitored through regional offices and regional offices are monitored by the Headquarters in Ulaanbaatar. This method of monitoring is more cost effective than if the local project offices and sub-grantees are monitored by the headquarters. To date, in coordination with sub-grantees, APPEAL has been able to successfully promote public awareness, civic participation, responsive government, and anti-corruption to large segments of SGU populations. Although, the adoption of the Budget Law in 2012 was not part of APPEAL and STAGE objectives, nor the consequence of APPEAL and STAGE activates, it is important to mention that the majority of APPEAL objectives would have a decreased likelihood of being accomplished if this law was not adopted. The Budget Law has had a profound impact upon subnational units of governance. It requires all revenue sources and expenditures by aimags and soums be shown in an annual budget document that is transparent and publicly available; it also requires public participation during the budget adoption process and financial reports explaining how public funds are expended. The SGUs that were visited during this evaluation had all implemented the law in 2013. Therefore, some credit for the implementation of the Budget Law can be attributed to the APPEAL project. For example, Citizens Chambers existed before the APPEAL project, but they were inactive institutions. Now they are places where public opinion regarding local budgets and Soum Master Plans (SMP) is formed and presented to local governments. Similarly, before the APPEAL project, the local civil sector did not actively and consistently monitor the local governments’ activities, nor did it pressure local governments for more transparency; this was the case even though there were legislative frameworks in place that would allow for such actions (Freedom of Information Act, The Budget Law). As such, current governance at the local level is byproduct of synergy between the existing legal framework and activities of APPEAL project. Another successful mechanism of APPEAL and IAAC for preventing corruption is the adoption of 75 Anti-Corruption Action Plans (ACAPs) in 21 aimags and 54 soums, which exceeded the initial target of 15 aimags that were included in the project. In order to foster good governance and anti-corruption activities, APPEAL needed to build mechanisms for how to improve the capacity of local civil sectors to monitor activities of local governments as well as facilitate anti-corruption actions. For this reason, in cooperation with PANI, APPEAL has assessed 58 local sub-grantees using Organizational Capacity Index and provided trainings to those CSOs with the aim of improving institutional capacity in human resources, financial standards, and project management. There have been visible improvements in the capacity building of local civic organizations. General Findings and Conclusions in Relation to the Project Participatory and Consultative Processes Good governance at subnational levels in aimags and soums is promoted effectively by APPEAL and has gained momentum. Citizens’ expectations of transparency, participation, and the implementation of further desired results are enduring a highly significant transformation in comparison to the former system. Across all six focus groups, the evaluation team found that the majority of participants feel that the APPEAL project engaged stakeholders and beneficiaries in various aspects of governance, especially through the activation of Citizens Chambers but also through increased knowledge and understanding of governance and anti-corruption, and through new legislation such as the Budget Law. To the extent that it may impact the findings, it should be noted that the majority of representatives of the civil sector that participated in focus groups were women. With regards to the both the STAGE and APPEAL projects, the following are observed activities for encouraging wider public participation: 14 • Sub-grantees of both projects were paying attention that both genders are represented when conducting trainings on anti-corruption and good governance. • BCM, STAGE’s sub-grantee, conducted several trainings and seminars on anti-corruption within the Mongolian business community. • Various MoJ officials, judiciary members, and Ulaanbaatar city officials were participating in trainings and seminars facilitated by STAGE’s sub-grantees. • Youth participation in anti-corruption and good governance activities was encouraged through activities organized by Globe International, a sub-grantee of both APPEAL and STAGE. The Zorig Foundation, a STAGE sub-grantee, has conducted activities regarding building awareness on the conflict of interest law and anti-corruption law using social media among youth. Progress on gender issues is uneven and may have been favorably influenced, but not demonstrably caused, by STAGE or APPEAL related activities. The GoM Ministries interviewed tend to be professionally staffed with both men and women. Similarly, employees of aimags, soums, and participating NGOs with whom they work are well represented by members of both genders; however, it is not clear where members of each gender in their respective roles fall within the hierarchy of administrative decision making. Additionally, elections continue to result in the selection of more men than women. General Findings and Conclusions in Relation to the Project Relevance From beneficiaries’ perspective, both STAGE and APPEAL heavily contributed to the strengthening of anti-corruption efforts through various direct activities targeting national and local governance institutions. For example, these benefits can be seen through the implementation of ACAPs on both the national and local levels, improved coordination between state agencies (IAAC, GPO, and judiciary) on the corruption investigation processes, and an increased awareness and knowledge of corruption (STAGE and APPEAL) and knowledge of anti-corruption legislation and activities. Beneficiaries of the APPEAL project are of the opinion that the project has relevant positive impacts on strengthening good governance. Consequently, participants of all focus groups rated the impacts of the APPEAL project on local governments’ transparency, accountability, effectiveness as satisfactory. Beside this, CSOs are aware that only a developed civil society can control and monitor the governments’ actions on both the national and local level. They see STAGE and APPEAL projects as opportunity for civil society capacity building. In this sense, STAGE and APPEAL, to some extent, contributed to the capacity building of civil society through PTI (CSOs networking) and PANI (in charge of local CSO capacity building project) activities. Both projects also addressed some of the needs of common citizens. As result of both projects’ activities those national and local institutions that are involved in projects are more transparent. Furthermore, through participation in Citizens Chambers citizens have the opportunity to articulate their needs and influence on the local government. The civil sector, citizens, and even public officials did not have much knowledge of anti-corruption and good governance mechanisms before STAGE and APPEAL were implemented. It can be said that ignorance and confusion ruled among public, private, and civil sectors of the society. In this sense, the achievement of STAGE and APPEAL was that they articulated needs of the society in a structured manner by educating all targeted stakeholders, actively participating in the implementation of existing anti-corruption and administrative legislation (FOIA, COI, and Budget Law), and at the same time that they worked on administrative reform. Additional information on the overall response to STAGE and APPEAL can be seen in the matrix below (Table 1). Particularly noteworthy is the overwhelmingly favorable responses that participants provided 15 regarding the positive impact of the two projects. The higher incidence of participant recognition of APPEAL in comparison to STAGE is likely attributable to the scope of their respective projects. General Recommendations As the projects near conclusion, implementers should make a concerted effort to emphasize the sustainability of good governance after the projects end. Additionally, the evaluation team encourages implementing partners to identify opportunities to provide some level of continued consultation for partners after the conclusion of the two projects. Both STAGE and APPEAL can enhance the utility of project implementation status reports by using nontechnical language to describe program results and impacts. While both TAF and Mercy Corps submit quarterly reports which document the technical requirements and mandated activities in their respective programs, they should engage the services of public relations experts to better convey their message. At a minimum, it is recommended that they use the Executive Summary of each Quarterly Report to describe their achievements in commonly used, non-technical language. Figure 3. Disaggregated Response Matrix for Knowledge and Impact of Projects Matrix of Contacts and Responses (Disaggregated) Knowledge of Positive impact upon # Organization or interviewee STAGE APPEAL USAID Funding Anti￾Corrupt Legal frame work Trans￾parent Civil Society Improve GoM Admin Services 1 MALA Yes Yes Yes Yes Yes Yes Yes Yes 2 Globe International Yes Yes Yes Yes N/A Yes Yes Yes 3 PANI Yes Yes Yes Yes Yes Yes Yes Yes 4 Women’s Lawyer Association Yes Yes Yes Yes Yes Yes Yes Yes 5 Women for Social Progress Yes N/A Yes Yes N/A Yes Yes Yes 6 IAAC Yes Yes Yes Yes Yes Yes Yes Yes 7 MoJ Yes N/A Yes N/A Yes Yes Yes Yes 8 City Municipality Yes N/A No Yes N/A Yes N/A Yes 9 Business Council Yes N/A No Yes N/A Yes Yes Yes 10 Judicial General Committee Yes N/A Yes N/A Yes Yes N/A Yes 11 Zorig Foundation Yes N/A Yes Yes N/A Yes Yes Yes 12 Umnugovi Aimag Chairman of CRK N/A Yes No N/A N/A Yes Yes N/A 13 Togt-tsetsii Soum, UG N/A Yes Yes Yes N/A Yes Yes Yes 14 Focus group UG N/A Yes Yes Yes N/A Yes Yes Yes 15 Gobi-Altai Aimag Chief of Governor Office N/A Yes Yes Yes N/A Yes Yes Yes 16 Sharga Soum, GA N/A Yes No Yes N/A Yes Yes Yes 17 Focus group GA N/A Yes No Yes N/A Yes Yes Yes 18 Uvs Aimag Chairman of CRK N/A Yes Yes Yes N/A Yes Yes Yes 19 Tarialan Soum, Uvs N/A Yes Yes Yes N/A Yes Yes Yes 20 Focus group Uvs N/A Yes Yes Yes N/A Yes Yes Yes 21 Uvurkhangai Aimag Governor N/A Yes Yes Yes N/A Yes Yes Yes 16 Matrix of Contacts and Responses (Disaggregated) Knowledge of Positive impact upon # Organization or interviewee STAGE APPEAL USAID Funding Anti￾Corrupt Legal frame work Trans￾parent Civil Society Improve GoM Admin Services 22 Tugrug Soum, UH Chief of Governor Office N/A Yes Yes Yes N/A Yes Yes Yes 23 Uvurkhangai Legal Department N/A Yes Yes Yes N/A Yes Yes Yes 24 Uvurkhangai Social welfare N/A Yes Yes Yes N/A Yes Yes Yes 25 Arvai kheer Soum, UH Governor N/A Yes Yes Yes N/A Yes Yes Yes 26 Focus group UH N/A Yes Yes Yes N/A Yes Yes Yes 27 Khentii Aimag Deputy Governor N/A Yes No Yes N/A Yes Yes Yes 28 Khentii Aimag Chairman of CRK N/A Yes Yes Yes N/A Yes Yes Yes 29 Kherlen Soum KE Governor N/A Yes Yes Yes N/A Yes Yes Yes 30 Kherlen Soum KE Chairman of CRK N/A Yes Yes Yes N/A Yes Yes Yes 31 Focus group KE N/A Yes Yes Yes N/A Yes Yes Yes 32 Arkhangai Aimag Secretary of CRK N/A Yes Yes Yes N/A Yes Yes Yes 33 Arkhangai Aimag Chief of Governor Office N/A Yes Yes Yes N/A Yes Yes Yes 34 Tsenkher Soum AR Deputy Governor N/A Yes Yes Yes N/A Yes Yes Yes 35 Focus group AR N/A Yes Yes Yes N/A Yes Yes Yes 36 Sant Maral Foundation Yes Yes Yes Yes N/A Yes Yes Yes Total # Yes 12 30 30 33 6 36 34 35 Total # No 0 0 6 0 0 0 0 0 Total # N/A 22 6 0 3 30 0 2 1 All of the participants are aware of the STAGE and APPEAL projects and the most of them are directly participating in project activities. Most of the participants (83%) are aware that USAID provided funding for these projects; however, a few participants (17%) have no clear idea who is providing funding for these projects. For example, Mr. Ser-Od, Vice director of Business Council of Mongolia, had no clear idea of USAID’s involvement with the STAGE project. There was no document provided to determine if there was compliance with the Branding and Marking policies of USAID. Even though, all printed brochures and handbooks have USAID logos on them, local government officials of Sharga Soum in Gobi-Altai aimag believe that Mercy Corps provides funding for the APPEAL project. Most of the participants (more than 90%) believe that both projects impacted positively on anti￾corruption, transparency, civil society, and the government’s administrative services. About 17% of the interviewees were directly involved in the legal framework through organizations such as the MoJ, IAAC, MWLA, and Judicial General Committee. 17 Figure 4. Summary Response of Knowledge and Impact of Projects in Numbers STAGE FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The goal to strengthen democratic governance by building a more transparent and accountable regulatory and legislative environment, and ensuring principles of checks and balances has been substantially advanced through the implementation of STAGE activities in support of that goal. Due to the significant importance for the STAGE project, the analyses of IAAC and PTI activities and the analysis on adoption of the GAL and the APL are each presented in separate sections within objective 1 and 2 of this report. Objective 1. Strengthen Anti-Corruption Efforts: Anti-corruption has been effectively addressed as a public issue as a result of programmatic activities organized and supported by TAF with its partners. The SPEAK and STOPP surveys designed and executed by SMF and STAGE can be described as starting points for all other anti-corruption efforts. The nature of these surveys is educational as well as practical. Practical application of the findings from these two surveys includes: • As the source of information on corruption activities that directed MWLA’s facilitation of public discussions on drafting NACS • For education of the participants of Citizens Chambers about corruption’s modus operandi • For education of the business community and public officials about corruption’s modus operandi • To focus STAGE’s and PTI’s attention on crucial areas where grand corruption occurs, thus indicating which public institutions should be monitored more thoroughly and frequently. For this reason, STAGE has cooperated with the Municipality of Ulaanbaatar on implementation of anti￾corruption action plans (ACAPs) for its four agencies: the City Property Office (that issues land ownership permit), the General Planning Office (that issues construction permit), the Office for Education, and the Office for Health. The evaluation team found out that these two surveys have been conducted in an efficient manner. Currently, they have been repeated three times (every six months) and the fourth round of surveying is expected in middle of March 2014. The effectiveness of the SPEAK and STOPP surveys is critical due to 18 comprehensive utilization of their findings by the STAGE project for the implementation of other anti￾corruption activities. One of the major anti-corruption activities was certainly the training of public officials on the good governance spectrum and ways to implement laws regarding conflict of interest, transparency, and income and asset declaration that was conducted by the Women for Social Progress (WSP) in coordination with IAAC. In the first phase, WSP and IAAC developed a set of training materials (training methodologies and guidebooks for trainers). In the second phase, the WSP trained 31 trainers (19 from ministries, 12 from CSOs and other organizations). These 31 trainers conducted 10 trainings in which over 250 government officials were trained on anti-corruption legislation. This is a clear indicator of the existing willingness for cooperation between public and civil sector on implementing good governance and anti-corruption activities. From the prospective of strengthening the capacity of the judiciary to adjudicate corruption cases and prevent judicial corruption several positive actions were undertaken. With technical and financial supports from USAID and STAGE, the Judicial General Council has recruited an international consultant, Professor Pim Albers from the Netherlands, to develop a performance evaluation system for judges. Dr. Albers successfully accomplished his tasks. The Judicial Performance Evaluation study for the selection of judges was developed. As the judiciary plays an important part in combating corruption, the evaluation and the study has strengthened the significance of performance evaluations of the key actors in the judiciary: judges. Additionally, a software program for the selection of judges was developed and is currently being used. All these activities are in compliance with the new law regarding the courts of Mongolia, which requires regular reviewing of a judge’s performance and authorizes both the Judicial General Council and the Qualifications Committee to conduct evaluations of judges. Therefore, the evaluation team has concluded that there have been improvements in relation to the strengthening the capacity of the judiciary to adjudicate corruption cases. As for STAGE’s role to include the business sector of Mongolia in anti-corruption activities, the activities were mostly educational with one effort to include the business sector in combating corruption. Both BCM and MONEF organized trainings and seminars on anti-corruption legislation and practices. MONEF together with IAAC plans to promote incorporation of anti-corruption action plans and a code of ethics within companies, but the team can’t confirm whether this action will occur or not as it is planned for March 2014. From the interviews with the BCM and SMF, the evaluation team noted that the majority of representatives of foreign owned companies were reluctant to participate in any kind of anti-corruption activities of the STAGE project. Partnership in Transparency Initiative (PTI) Targeted Analysis In relation to the support of civil society advocacy for implementation of anti-corruption laws (the FIOA, the COI Law, the CC, and the CPC) the evaluation team discovered that the STAGE project has been successful and effective. By establishing the PTI, STAGE has succeeded in synchronizing activities of 18 different CSOs toward the promotion of, education regarding, and implementation of anti-corruption laws. The goal of PTI is to reinforce the importance of the role citizens play in ensuring government agencies make all-encompassing policies and implement them. They do this through initiatives such as supporting and constructing partnerships with state agencies. One clear example of cooperation between a state agency and civil sector is the training curriculum developed by IAAC and WSP, and used by PTI members to train more than 1,500 citizens, civil servants, citizens’ representatives, and representative of the business sector on the structure and functioning of the anti-corruption laws. 19 Another major activity of PTI is monitoring governance activities of public institutions, from schools to government agencies, with special attention toward land utilization, property ownership, and procurement as pointed out by the SPEAK and STOPP surveys, since these areas were indicated as of the sectors demanding immediate improvements. One of the most effective examples of PTI’s monitoring activities can be seen in the work of PANI. PANI has monitored budget transparency of the Ministry of Health and the Ministry of Education and Science in order to find out whether or not they are in compliance with the FOIA. Based on the findings from these monitoring activities, PANI has been focusing on raising awareness and knowledge on transparency and anti-corruption. To date, PANI has produced and disseminated more than 1,000 information, education, and communication materials (IEC). After thorough investigation of project documentation and interviews conducted with recognized partners of the PTI (Zorig Foundation, Globe International, MALA, and PANI) and IAAC, the evaluation team came to conclusion that the STAGE project has ensured a higher rate of visibility and effectiveness of its anti-corruption activities by introducing PTI. Objective 2. Improve Administrative Reform: Insufficient progress has been made to improve administrative reform at the national level. The reason for this minimal progress can be attributed to long-standing, poor practices and a general lack of political will among elected officials to make changes. There is increased public recognition of such poor practices, partially as a result of STAGE sponsored activities; nonetheless, instances of favoritism, cronyism, corruption, and non-transparent administrative practices continue to exist in many Ministries, GoM offices, and Parliamentary proceedings. It remains to be seen whether the efforts of STAGE and other reforms will be sufficiently influential to overcome vested interests resistant to such change. Overall, STAGE is a well-conceived democracy and governance project that is being implemented successfully by TAF. STAGE effectively promotes good governance at the national level by encouraging transparency, monitoring, and auditing. More needs to be done by GoM to encourage greater legal applicability to agencies and to increase incidents of criminal prosecution for violations, but that is beyond the scope of STAGE to control. Increased transparency at national and sub-national levels of governmental operations supported by STAGE, and enabled by Parliamentary legislation on freedom of information, have occurred. STAGE has supported activities that, while significant, have not resulted in parliament enacting laws. The General Administrative Law (GAL) and Administrative Procedure Law (APL) are two examples. This has inhibited further progress. STAGE has contributed to increased awareness and knowledge of corruption and its corrosive impact upon society and the body politic. This increased knowledge has been partially responsible for new legislation on transparency and freedom of information as well as increased media attention for governance activities. Progress on gender issues is uneven and not demonstrably related to STAGE. Women were well represented at the various governments and non-governmental agencies the evaluation team interviewed; however, it is unclear if labor categories tend to be populated with one gender or another, the division of tasks, the relative authority, or any disparity in wage. For elected positions, men continue to have a greater presence than women. General Administrative Law (GAL) and Administrative Procedure Law (APL) Since the beginning of the STAGE project, there were delays in the drafting processes for both the General Administrative Law (GAL) and the Administrative Procedure Law (APL). Currently, the draft GAL and APL are in the hands of the MoJ and are expected to be passed to a Parliamentary working group. 20 STAGE exerted a great deal of effort drafting the GAL and APL as well as lobbying for their adoption. In cooperation with MWLA, MoJ, and other partners, the STAGE project has organized various activities in relation to the drafting and adoption of these laws, such as: coordinating wide public discussions that provided numerous recommendations for the draft laws, funding expert studies on administrative reform, employing translators and experts on foreign countries’ administrative laws, and funding field visits for public officials outside of Mongolia. Nonetheless, all of these activities only partially achieved their main objectives: drafting the laws and adoption of the laws. While laws were finally drafted with the inputs from experts, CSOs, and even citizens, the drafted laws remain with the MoJ and have yet to be passed. Therefore, the evaluation team’s finding is that the effectiveness of these efforts is low relative to the cost. Ultimately, the desired outcome is beyond the control of the STAGE project. Due to the fact that the drafting process of the GAL and APL took more time than was originally planned, the evaluation team is not able to confirm whether, in following spring session of the Mongolian Parliament, these laws will be adopted. Another consequence is that the majority of the activities listed below, except activity 2.2.2, of the STAGE project were postponed as well: • Activity 2.2.4: Build public awareness of the implications of the Law on Administrative Procedures • Activity 2.2.5: Coordinate action on the legislative implications of the Law on Administrative Procedures • Result 2.3: Increased support for and awareness of administrative reform • Activity 2.3.1: Raise public awareness on administrative reform and collect public comments on proposed legislative changes • Activity 2.3.2: Organize conferences on administrative reform • Activity 2.2.2: Support the Parliamentary Working Group on the Law on Administrative Procedures: Support activity continues despite lack of progress. Targeted Analysis of IAAC Activities The role of IAAC in relation to the STAGE and APPEAL projects can be divided into several categories: Category 1: Raising awareness and knowledge on anti-corruption legislation among public officials, civil servants, civil sector and private sector employees, students, and educators through a series of trainings and public discussions through the following project activities: • Activity 1.2.4 a): Integrate AC curriculum for secondary schools into the national curriculum • Activity 1.2.4 b): Design and introduce anti-corruption curriculum for University students • Activity 1.2.3 (a): Support training and selected research on Anti-Corruption (AC) laws and amendments after passage • Activity 1.4.1: Train private sector on AC laws and policies, and the benefits of pro-active private sector engagement in AC These successfully implemented activities raised knowledge and awareness on corruption and anti￾corruption legislation, mechanisms, and ongoing anti-corruption activities among various stakeholders. Category 2: Facilitation in drafting of the National Anti-Corruption Strategy (NACS) and amendments to the AC laws (Conflict of Interest Law, Criminal Code, and Criminal Procedure Code) and advocacy for their adoption and implementation (NACS, Criminal Code, and Criminal Procedure Code) occurred through the following project activities: 21 • Activity 1.2.1: Support Anti-Corruption Working Group (ACJWG) to build political and public support for the National Anti-Corruption Mid-Term Strategy • Activity 1.2.2: Support civil society advocacy for implementation of the COI Law and Criminal Code and support the passage of draft amendments to the Criminal Procedure Code IAAC was partially successful in the implementation of these two activities. A great deal of effort and funding has been given toward the adoption of NACS and amendment of the two AC laws (Criminal Code and Criminal Procedure Code). All three documents have been drafted, widely circulated for public discussion, have an abundance of support, and have been sent to Parliament for adoption. Previously stated limitations regarding STAGE activities and the ultimate adoption of supported policy remain. Implementation of existing COI Law has been fostered successfully thought various activities. Category 3: Implementation and monitoring of Anti-Corruption Action Plans (ACAPs) through the following project activities: • Activity 1.2.5: Support more effective monitoring practices of ACAPs • Activity 1.2.6: Identify and support pilot agencies to implement ACAPs Most planned activities were successfully implemented. Both cases of implementation of ACAPs and their respective monitoring by IAAC and STAGE have been accomplished. There were some comments on lack of cooperation of IAAC on this matter, but they were resolved. Category 4: Cooperation of IAAC with other agencies (GPO, Judiciary,) on AC investigations through the following project activities: • Activity 1.3.1: Support inter-agency coordination and collaboration among IAAC, GPO and the Judiciary. There have been some improvements in relationships among these three agencies; however, the evaluation team was unable to determine the degree to which the various relationships had improved as it would require a separate, detailed study with quantifiable findings and conclusions, which the evaluation team couldn’t conduct due to the scope and timeframe of this evaluation project. One of the tangible areas in which cooperation between three agencies has been noted is the jointly compiled study on 300 corruption and abuse of office cases. Another result of their cooperation was a survey conducted among IAAC investigators, GPO prosecutors, and judges. Findings from this survey were used to draft recommendations for amendment of following laws: Criminal Code; Criminal Procedure Code; rejection of amnesty for grave corruption cases; procedure for reverting cases, by courts, for reinvestigation; and amendments to the investigation processes standards. Category 5: Strengthen capacity for conducting corruption investigations through the following project activities: • Activity 1.3.2: Support IAAC to advocate for amendments to laws that limit their investigative authority IAAC has been recently empowered to conduct corruption investigation in several GoM agencies. Strong support toward widening jurisdiction of MoJ and IAAC to conduct investigations and/or prosecution of alleged corruption is needed. STAGE PMP ANALYSIS Per contractual requirements, the Quarterly Performance Reports have documented interim progress on achieving the targets set out for the indicators. The summary of results of the performance evaluation at the end of FY 2013 shows that: 22 • For 8 indicators (67%), the planned result was achieved as the target number was met • For 1 indicator (8%), Indicator #9: Number of CSO employees trained on AC laws and monitoring skills; the planned result was partly achieved as progress was made toward meeting the target number. This indicator’s target in Year 2 was 50, however the actual result was 44. • For 1 indicator (8%), Indicator #6: Decrease in the number of inter-agency procedural and practical issues, conflicts, and differences; the planned result was not achieved as no progress was made toward meeting the target number. This indicator’s target in Year 2 was 3; however actual result in Year 2 was 0 (zero). • For 2 indicators (17%), Indicator #7: Number of companies adopting AC plans, rules, and regulations to improve corporate governance, and Indicator #11: Number of trends across agencies that affect performance improvement in key agencies, there was no target and no expected results. The resulting score of 75% reveals that the STAGE project has either fully or partially achieved targets set against indicators for FY 2013. For two of the indicators the planned result was not achieved as Year 2 was a baseline year (no target, no expected results). Figure 5 details the actuals for the STAGE performance indicators for FY 2012 and FY 2013. 23 Figure 5. Actuals of STAGE performance indicators 24 Figure 5. Actuals of STAGE performance indicators 25 STAGE RECOMMENDATIONS Recommendations deemed applicable to enhancing the STAGE project are: In general, it is recommended that STAGE continue the current level of effort and effectiveness for implementation. Regarding implementation of the COI Law, it is recommended that STAGE continue its previous activities. As it relates to PTI, a continuation of the monitoring surveys of public institutions it is recommended, with the goal of identifying ambiguities in the processes pertaining to land utilization, property ownership, procurement, and all other prime corruptive areas that might be indicated by the results of the last round of SPEAK and STOPP surveys. Upon publishing results from the fourth round of surveys, TAF and SMF should encourage the media outlets that are using findings from the surveys (television, radio stations, and newspapers) to promote SPEAK and STOPP surveys according to branding and marking policies of USAID. Consider reallocation of STAGE project funds/resources from MONPAC and efforts to enact proposed COI, GAL, Criminal Code, Criminal Procedure, and Administrative Reform Law to more viable reform prospects such as enacting legislation providing for broader IAAC powers related to anti-corruption. STAGE should review, in consultation with GoM and other relevant stakeholders, the likelihood of GAL, COI, and Administrative Procedure reform laws being enacted and determine whether its continued efforts are a useful expenditure of limited resources during the remainder of the project’s duration. It may be more useful to focus on an alternative approach to reform by advocating another legal mechanism such as broadening the scope of IAAC’s powers to monitor, audit, and initiate legal action for suspected corrupt activities in Ministries in addition to the limited jurisdiction currently authorized. If GAL or APL are not enacted in the spring parliamentary session it is advisable to stop activity 2.2.2 “Support the Parliamentary Working Group on the Law on Administrative Procedures” and leave the passage of these laws for some other project, instead concentrating on other anti-corruption activities as well as support toward implementation of the Budget Law within the Ministries Freedom of Information Act enforcement and National Anti-Corruption Strategy. As the STAGE project is nearing its end and adoption of NACS, CC, and CPC will be decided upon during the Parliament’s spring session, it is recommended that advocacy efforts toward their adoption be terminated consistent with the end of the that parliamentary session. If new AC laws and NACS are not adopted at that time, the STAGE project will be terminated before further advocacy can have a meaningful impact. APPEAL FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The goal to promote democratic reforms, economic growth, and improve legal, institutional, and administrative processes in targeted Mongolian government agencies has been substantially advanced through the implementation of APPEAL activities in support of that goal. There have been some complaints regarding the APPEAL project, but they tend to be unsubstantiated or normal for the size, scale, and complexity of the project. Engagement and local participation through its aimag and soum focused activities is substantial. Objective 1. Knowledge - Improve key elements of Mongolian legal framework related to anti-corruption and good governance: There are encouraging signs that key elements of the Mongolian legal framework related to anti-corruption and good governance are better understood and made relevant to key local stakeholders and the general public as a result of programmatic activities organized and supported by Mercy Corps. 26 In collaboration with IAAC, APPEAL has contributed to the implementation of ACAPs in local government institutions at both the aimag and soum level. According to IAAC, they have been implementing 75 ACAPs in 21 aimags and 54 soums. According to this finding it can be said that APPEAL has exceeded primer target of 15 aimags that are the part of the project. It should be noted that each of implemented ACAPs could not have been achieved without cooperation between Mercy Corps, the IAAC, and local governments to strengthen transparent, accountable, and participatory governance at the local level. However, as could be seen from the PMP analysis of APPEAL, the evaluation team didn’t receive complete documentation from Mercy Corps that can support these findings; rather Mercy Corps provided documentation that only confirmed incorporated ACAPS in 6 aimag governments. Furthermore, the APPEAL project can be credited for wider civil society and public participation in ACAPs’ implementation. Explicitly, APPEAL has included various local CSOs to work together with IAAC on development, promotion, and dissemination of new ACAPs as well as monitoring the implementation of already existing ACAPs in their local communities. Moreover, good governance Master Trainers, trained through the APPEAL project, are facilitating Citizens Chambers where, through public discussions, ACAPs are assessed and recommendations are given to the local government. Mercy Corps admitted that they have not been successful when it comes to implementation of ACAPs with the local departments of the Ministry of Population Development and Social Protection (MoPDSP). According to PMP indicators, only 5 out of 15 local MoPDSP departments have incorporated ACAPs. However, corrective actions were undertaken. APPEAL, together with IAAC and the General Office of Social Welfare Service (GOSWS), has established working groups whose goal is to support the development of ACAPs in local MoPDSP departments. Whether this action will have positive impact has yet to be seen. In relation to awareness raising, the evaluation team has discovered that IAAC and other CSO partners such as the Zorig Foundation have creatively promoted knowledge of and interest in anti-corruption and good governance among Mongolian youth. Particularly compelling are the development of a website (www.xamtdaa.mn) and the successful use of social media. In addition to the new Budget Law and public information law, there is a recent movement toward Civil Society Organizations (CSOs), media coverage of government activities, legally sanctioned financial audits, and the requirement for designated governmental agencies and entities to publicly disseminate financial reports. Consequently, 58 of APPEAL’s sub-grantees have organized more than a thousand anti-corruption and good governance activities such as trainings, public discussions, anti-corruption promotion activities, and monitoring activities across all 15 aimags and 60 soums. One of the reasons why the evaluation team assessed the APPEAL project as successful is due the diversity of methods by which APPEAL encouraged and mobilized local citizens to participate. The first is through the successful utilization of local Citizens Chambers. The second method that APPEAL has used is facilitating the development of aimags and soums’ ACAPs as well as those for the local MoPDSP. The third method for mobilization of local citizens is through the development of Soum Master Plans (SMPs) and the assessment of Soum Development Funds (SDF). Finally, through APPEAL, around 5,000 (five thousand) people in 15 aimags were introduced to and educated about Procedural Administrative Regulations (PARs) such as the Freedom of Information Act, Conflict of Interest law, the Budget Law, GAL, APL, and other anti-corruption laws. APPEAL’s sub￾grantee MWLA had a crucial role in facilitation of these trainings. Educational activities of local public officials were part of APPEAL’s curriculum as well. MALA has trained 912 people, from various backgrounds, on initial participatory planning and how to conduct 27 SWOT analysis on local socio-economic development. The majority of participants (746) were local public officials. Objective 2. Action- Exercise of authority and delivery of government services reflect transparent and accountable administrative processes, provide effective checks and balances, and engage informed communities to optimize quality and impact of governance at local level: APPEAL effectively promotes good governance at the sub-national level by encouraging transparency, monitoring, and auditing; however, more needs to be done by GoM to require routine, competent financial audits of SGUs annually. MWLA facilitated 15 public discussions on PARs in Citizens Chambers cross various aimags with the help of APPEAL. As result, MWLA provided more than 600 citizens’ recommendations on GAL and APL and other PARs to the MoJ. This is a clear indicator of citizens’ participation in creation of legislation. The expected result of the training provided by MALA to local officials on participatory planning and local socio-economic development is the increased adoption of local SMPs. This is yet to be seen. Activation of Citizens Chambers is one of the most prominent of APPEALs achievements. The importance of Citizens Chambers lies in the fact that, by participating in the work of these institutions, common citizens have a chance to articulate their opinion about matters of great importance to them and can influence policymakers across a wide spectrum of fields. The nature and range of discussions that occurred in Citizens Chambers differed depending on the topic that was on the agenda. Some examples include ACAPs, local budgets, SDF use, SMP formulation, good governance, and other local development topics. In the majority of aimags and soums visited, civil society and public sector representatives have confirmed that Citizens Chambers play an active role in local development. APPEAL has achieved remarkable success in the aimags and soums. The Soum Development Plans and new Budget Law have been, and continue to be, implemented to a degree that exceeds expectations. This has resulted in marked improvement for both public knowledge and participation. Good governance has progressed as a result of recent empowerment enabling SGUs to provide services in response to perception of public needs and demands, within constraints of budgeted resources. Empowerment for decision making at the provincial and local levels is now consistent with acceptable standards of decentralization. In addition, the effective engagement of CSOs in anti-corruption advocacy has resulted in sensitivity not formerly present. While not sufficient to eliminate corruption, it has been reduced at the SGU level due to increased awareness, transparency, and monitoring by CSOs. While the evaluation team has determined that a decrease in NGO activity without USAID funding sources, numerous APPEAL client NGOs predict the continuation of program initiatives without project funding, perhaps at reduced levels. Progress on gender issues is uneven and not demonstrably caused by APPEAL activities. Employees of aimags, soums, and participating NGOs with whom they work are well represented by members of both genders but it is not clear where the respective roles of members of each gender fall in the hierarchy of administrative decision making. Elections continue to result in the selection of more men than women. APPEAL contributed to increased awareness and knowledge of corruption and its corrosive impact upon society and the body politic. This increased knowledge has been partially responsible for promoting new legislation on transparency, freedom of information, as well as increased media attention to activities related to governance. Findings are based upon document review, Survey Instrument Reports from SGU (aimag and soum) site visits, and analysis from the Matrix of Contacts and Responses–Disaggregated (Table 1). Since there is 28 pervasive commonality regarding what and how these SGUs operate it is useful to generalize our findings relating to the APPEAL program in the aimags and soums, rather that present iterative reports on each. Budget Law Implementation, Impact, and Significance Subnational units of government in Mongolia are in part administrative districts of the national government, but have important attributes of local influence in decision making, particularly since the implementation of the Budget Law enacted by Parliament in 2012. Functional powers and responsibilities of subnational governmental units (SGUs) are not always clearly defined by law and are not uniform in practice. Some functions are shared between or among state, aimag, soum and bagh. Often this occurs with no clear mandate for any single entity to be responsible or accountable. Such shared responsibility for various functions is neither unusual nor inherently problematic, but it can make it difficult to hold a single governmental unit accountable for results. Nonetheless, there are indicators that suggest the APPEAL program has had significant positive impact upon good governance by SGUs. Roles and Responsibilities of SGUs: New Budget Law, Decentralization, and Autonomy The Budget Law enacted by Parliament in 2012 and implemented during 2013 by aimags and soums has enabled SGUs to operate with a degree of decentralized decision-making not previously allowed. This significant degree of increased autonomy is the essence of meaningful sub-national governance, and is necessary to have public participation in such governance. Decentralization of government is often a requisite feature for responsive and accountable delivery of local, public services related to the APPEAL project’s second objective. There are numerous examples of implementation or support for modes of decentralization that are consistent with the new mode of SGU operation in Mongolia. For example, the Council of Europe, USA, GTZ, European Union, OSCE and other international organizations frequently express support for the concept of devolution of government power from the state to the Local Governmental Units (LGUs) consistent with practices prevalent in western democracies. The Council of Europe has promulgated a set of guidelines, or standards, against which legislation enacted by transitional states are evaluated as one of several conditions for consideration of admission to EU membership. Since the early 1990’s most nation states in eastern and central Europe and republics of the former Soviet Union have adopted constitutional and/or legislative provisions, ostensibly for that purpose. However, many of those legislative actions, when compared with the Council of Europe criteria for decentralization, are undermined by the existence of other laws and practices that make effective decentralization impossible in practice. Consequently, LGU’s control or significantly influence only a miniscule percentage of public expenditures, the vast majority of which remained under the centralized purview of the state. Merely reading the LGU enabling laws and constitutional provisions is not sufficient to evaluate actual practices. Since LGUs historically had little ability to deliver meaningful public services or impact the quality of life in the local jurisdiction, there was little reason for the citizens in such a jurisdiction to have much interest in participation in a local political process that achieves no results. Mongolia recently enacted a major reform of its public finance system by devolving budget authority to SGUs. The resources SGUs need to meet their responsibilities must be raised through mechanisms such as taxes, fees, and subventions. Citizens are now able to hold governments accountable for the use of such resources. To achieve this goal, reforms have occurred in fiscal operations. These include: legal authority to raise revenues and allocate them among levels and sectors of government and society; the 29 development of appropriate accounting, information, management, and auditing systems to minimize inappropriate resource use and expose corruption; the involvement of NGO partners in governmental activities, and the training and equipping of managers and leaders to work with new and reformed systems and procedures. Collectively, these reforms contribute to improved governance and public sector performance. Decentralization of governmental administration is consistent with and supportive of Objective Two. The new Budget Law enacted by Mongolian Parliament in 2012 and implemented in 2013 has had a profound impact upon the ability of aimags and soums to respond to local demands for services. In addition, it has given CSOs and individual citizens a reason to support anti-corruption activities at the sub-national level in their respective jurisdictions. This represents an essential form of decentralization that is important for the future of Mongolian governance. Conditions necessary for decentralization include SGUs that are: • Meaningful to citizens • Elected • Able to allocate public resources • Able to generate local revenue • Able to own assets • Able to provide public services • Transparent • Accountable To varying degrees SGUs in Mongolia meet the above criteria. Since the Parliament of Mongolia enacted the new Budget Law that devolves significant power to SGUs; it had tremendous impact upon the range of governmental functions performed by aimags and soums. Moreover, the substantial degree of increased public awareness, interest and participation through Citizens Councils has been dramatic in both scope and impact. Improved performance and increased accountability are the inevitable results of such an important change in the ability of aimags and soums to respond to local, public demands. Another useful tool provided to SGUs by the GoM is the incentive to produce Master Plans for Soum Development. There is visible evidence posted in citizen’s halls of such plans already in existence, or in process of development. Based upon generally acceptable criteria applicable to decentralization of governance and meaningful local participation, Mongolia has recently achieved considerable success, much of which has been effectively supported by APPEAL and its partners. Focus Group Results The most common purpose of a focus group interview is to provide an in depth exploration of a topic. Working to conduct focus groups in accordance with USAID guidelines, the evaluation team organized six focus groups: one in each visited aimag. The aim of focus groups was to identify the group’s perception on the APPEAL project and its activities as well as participants’ perception of the projects potential impact on local governance. Questions for the focus group were categorized as follows: 1) Knowledge, 2) Management, 3) Outcomes, and 4) Good Governance. The final category was the subdivided into: i) Participation, ii) Government Transparency, iii) Government Accountability, iv) Government Effectiveness, v) Sustainability, and vi) Gender Equality/Inclusion. 1) Knowledge: In all six aimags, participants were very well aware of project activities and the most of them are directly participating. The evaluation team also discovered that, upon implementation of the APPEAL project, overall public awareness and knowledge about anti-corruption legislation and activities, 30 as well as good governance and procedural legislation and activities have improved. Participants were largely of the opinion that this was the result of numerous trainings and six-months of field visits/trainings conducted in these aimags as part of APPEAL project. 2) Management: The majority of focus group participants agreed that the APPEAL project’s management structure is effective and executed in a way that is consistent with objectives. In their opinion, Mercy Corps has been successful in facilitation of anti-corruption and good governance activities, Citizens Chambers, dissemination of information about project activities, and as a facilitator of local working groups consisting of local officials, CSOs representatives. For example, participants of the Gobi Altai focus group repeated several times that the best way to fight corruption is to prevent corruption, attributing APPEAL for introducing methods of corruption prevention. The evaluation team didn’t find any serious management or communication issues in the execution of APPEAL. There were some minor remarks regarding delays in CSO reports, the strict policy of Mercy Corps about reimbursement/accounting, and the financing of local CSOs’ activities. When participants of the focus groups were asked if they knew who was financing this project, five of the six focus groups responded that the funding came from USAID. Only the Gobi Altai focus group provided an ambiguous answer predominantly attributing the money to the United States, but not to USAID specifically. 3) Outcomes: The evaluation team found that every focus group had very similar conclusions regarding outcomes and achievements. The most notable outcomes with the greatest success were: • Increase in citizens’ knowledge on good governance and corruption • Increased participation and engagement of citizens • Better relationship between local government and civil sector • Development of anti-corruption action plans • Development of Soum Master Plans • Increased knowledge of laws and regulations due to frequent trainings provided by APPEAL • Improvement of CSOs’ capacity due to trainings provided by APPEAL APPEAL succeeded, in combination with the new Budget Law and the Freedom of Information Act, to motivate and activate civil sector monitoring of local government work. While not as ubiquitous a response as those bulleted above, the focus group from Uvs Aimag emphasized that one of the important outcomes of the APPEAL project was the education of adolescent citizens about corruption and good governance. 4) Good Governance: The evaluation team found that the majority of focus group participants expressed positive perceptions in relation to APPEAL’s impact on good governance. Evidence for focus groups’ positive attitude toward good governance can be seen using the following calculation: Taking the average satisfaction rate from the sum of all satisfaction rates within the good governance category (e.g., participation, transparency, etc.), we get value of 3.64 (satisfactory) for the category as a whole. As the idea of good governance is complex, the team has disaggregated it into more basic notions with an aim of discovering the common opinions of participants from all six focus groups: i) Participation: The evaluation team found that majority of focus group participants is of opinion that the project engaged stakeholders and beneficiaries in various aspects of governance, especially through the activation of Citizens Chambers. Similar responses were found across all six focus groups. They were: • Increased citizens’ participation as result of increased knowledge and understanding of governance and anti-corruption 31 • Increased citizens’ participation as result of Citizens Chambers activation • Increased citizens’ participation due to new legislation (Budget Law) Participants from the Khenti focus group emphasized that before the APPEAL project, Citizens Chambers were inactive institutions and that credit for increased participation should be given to the APPEAL project. ii) Government Transparency: Upon conducting all six focus groups, the evaluation team came to the conclusion that APPEAL activities made local governments more transparent. The main reason is that local governments are aware that citizens have a better understanding of their political and civil rights. A second reason is increased participation of CSOs and citizens through Citizens Chambers. A final reason is that, according to the new Budget Law, local government needs to consult with its citizens when making decisions regarding the budget. Evidence that the budget and budget process have become more transparent can be seen on public boards within government institutions as well as government websites, both of which provide access to the budget. A participant in the Khenti focus group stated that corruption has become more sophisticated as anti￾corruption actions and legislation have taken place. iii) Government Accountability: Similar to transparency, participants of the various focus groups were of opinion that the APPEAL project contributed to more accountable local governance. Put simply, local government is now more responsive to requests from local CSO’s. Proof of increased local government accountability is the formation of joint working groups that monitor anti-corruption and good governance activities. iv) Government Effectiveness: Focus groups expressed that there have been improvements in relation to the local government effectiveness as result of the APPEAL project. For example, as result of APPEAL’s efforts in relation to the active implementation of ACAPs, FOIA, COI law, Budget law, and by organizing Citizens Chambers, local governments are encouraged to make decisions that address, to a greater extent, the needs of local population. Additionally, participants are of opinion that by building the capacity of civil sector, CSOs will be more capable and equipped to pressure local governments to work more effectively. A few participants were of the opinion that additional legislation is needed in order for local governments to be more effective. v) Sustainability: Regarding the sustainability of anti-corruption and good governance activities that have emerged as a result of the APPEAL project, participants tend to favor a conception of the future where project activities will survive the termination of APPEAL. As citizens have become more educated there are certain activities that they may need less; however, many of those activities are widely supported and may survive because individuals prefer, for example, to voice their interests and needs through CSOs even as they are aware of other methods. The evaluation team noticed upon receiving answers from participants of the focus groups, that part of the civil sector will decrease activities toward monitoring and controlling of local governments due to lack of funding. Participants expressed fear that if CSOs start receiving funds from local governments then that will represent conflict of interest. vi) Gender Equality/Inclusion: In relation to the gender equality and inclusion, the evaluation team came to the conclusion that there was no direct activity which fostered greater gender equality. Participants stated that gender was a component of every training curriculum. The team came to conclusion that there was no direct activity which fostered greater gender equality, although, there were plenty of activities that targeted students and schools. Participants expressed their views that gender equality was part of each training curriculum and in that sense it can be said that APPEAL contributed toward increase of awareness about gender equality. 32 The matrix below consolidates and aggregates the information gathered from the various focus groups (Table 2). It should be noted that data collected from the focus groups is qualitative in nature and that the matrix functions as a supplementary and complementary instrument to the qualitative analysis of the focus groups. Figure 6. APPEAL Focus Group Response Matrix Matrix - Satisfaction Rate of APPEAL Performance Overall performance Good Governance Aimag/ focus group Knowl. MGMT Outcome Part. Gov. Trans. Gov. Acc. Gov. Eff. Sustain Gender Equality Avg. satisfaction of Aimag focus groups with APPEAL’s performance Uvs 4 3 4 4 4 4 4 4 4 3.88/Satisfied Gobi Altai 5 4 4 4 4 3 4 4 3 3.88/Satisfied Uvurkhangai 5 4 4 4 4 4 4 3 4 4/Satisfied Omnogobi 5 4 4 2 4 3 3 4 3 3.55/Satisfied Khenti 5 5 4 4 4 4 4 4 4 4.22/Satisfied Arkhangai 4 4 4 2 4 4 3 4 2 3.44/Neither Sat. nor Dissat. Average satisfaction of all Aimag focus groups per topic 4.66/ Highly Sat. 4/Sat. 4/Sat. 3.33/ Neither Sat. nor Dissat. 4/Sat. 3.66/ Sat. 3.66/ Sat. 3.83/ Sat. 3.33/ Neither Sat. nor Dissat. Average satisfaction with good governance 3,64/Satisfied Overall satisfaction of all groups 3.83/Satisfied [5 = Highly satisfied // 4 = Satisfied // 3 = Neither satisfied nor dissatisfied // 2 = Dissatisfied // 1 = Highly dissatisfied] All focus group participants were highly satisfied (average 4.7 points) with APPEAL project activities and the most of them are directly participating in APPEAL project activities. As can been seen from Table 2, the majority of focus group participants are satisfied (average 4.0 points) with the APPEAL project’s management and the outcome, to date, of the project. They all believe that the management structure is effective and consistent with the projects objectives. Most of the focus group interviewees (67%) were satisfied with the APPEAL project participation; however, approximately 33% of the interviewees were not satisfied with project participation. All of the focus group participants agreed that citizens’ participation increased enormously due to the APPEAL project; however, focus group interviewees were neither satisfied nor dissatisfied (average 3.3 points) with the participation of the APPEAL project. All participants were satisfied (average 4.0 points) with the APPEAL project activities geared towards more transparent government. They all concluded that local governments are now more transparent than before as any information of local government activities, such as financial reports and procurement information, are available to the public through various channels. Most of the participants were satisfied (average 3.7 points) with the APPEAL project’s contribution toward accountability and effectiveness within the local governments. Participants were satisfied (average 3.8 points) with the sustainability of the APPEAL project. They are optimistic about the sustainability of the projects as citizens have become more educated and have a better understanding of their civil rights, thus they will continue to be involved in local governments’ activities such as asking to publish the budget to public. Focus group participants were neither satisfied nor dissatisfied (average 3.3 points) with the APPEAL project’s activities towards gender equality. As can been seen in Figure 7, the majority of NGO participants were female (68%) and only about 32% were male. Figure 8 shows that most of the government officials were men (66%) and only about 34 % were women. Therefore, it is possible to conclude from our observations that more women are involved in NGOs and more men are involved in political and governmental positions. 33 Figure 7. NGOs by Gender Figure 8. Government official by Gender APPEAL PMP ANALYSIS Per contractual requirements, the Quarterly Performance Reports have documented interim progress on achieving the targets set out for the indicators. The summary of results of the performance evaluation at the end of FY 2013 shows that: • For 4 indicators (33%), the planned result was achieved as the target number was met. • For 3 indicators (25%), the planned result was partly achieved as progress was made toward meeting the target number. − Indicator #1. Number of aiming governments with timely, relevant anti-corruption action plans. The target for this indicator was 10 in Year 2; however, according to the supporting documents provided by Mercy Corps, only six ACAPs were submitted in Year 2. Mercy Corps claims that 15 ACAPS were submitted, but the evaluation team was not provided any documentation for the remaining nine ACAPs. 34 − Indicator #2. Number of line departments that incorporate Ministry of Human Development and Social Welfare anti-corruption strategy into local plans. According to the quarterly report, as of September 2013, five action plans were submitted; however, Mercy Corps has not provided any supporting documents to back up this indicator. The target for this indicator was 10 in Year 2, but only five action plans were submitted according to Mercy Corps. − Indicator #4. Number of media and CSO reports and articles reflecting local government plans and actions relevant to APPEAL. The target for this indicator was 700 in Year 2. As of September 2013, 526 media reports (TV, radio, newspaper articles) shared information about government plans and actions relevant to APPEAL. • For 3 indicators (25%), the planned result was not achieved as no progress was made toward meeting the target number. − Indicator #5. Percentage change in public confidence perception about government and public institutions’ effort in combating corruption. The target was 3% in Year 2; however, Mercy Corps did not provide any documentation to support this indicator. The evaluation team consulted with the SPEAK longitudinal findings in order to try and find the proper answer, but there wasn’t any question that exactly refers to this indicator. The survey finding that has the closest meaning to the targeted indicator was presented in figure 7.1 and 7.2 of the SPEAK longitudinal findings (December 2013, p. 20). The specific question was: “In your opinion, who should organize and lead the effort to combat (grand) corruption”. In both figures results for local governments were not reaching the indicator’s target of 3% change. In figure 7.1 (in relation to corruption), public perception on a local government taking a leading role in fighting corruption jumped from 2.6% in November 2012 to 3.2% in September 2013. In figure 7.2 (in relation to grand corruption), public perception of a local government taking a leading role dropped from 1.8% in March 2013 to 0.8% in September 2013. − Indicator #9. Percentage change in citizens’ perceptions of quality and consequence of citizen’s halls. The target in Year 2 was 10%; however, Mercy Corps did not provide any documentation to support this indicator. − Indicator #10. Number of new Soum Master Plans developed and ratified by soums citizen’s halls. The target was 40 in Year 2. While it is an ongoing activity, Mercy Corps did not provide any documentation to support this indicator. • For 2 indicators (17%), there was no target and no expected results. − Indicator #6. Number of best practices introduced in Good Governance Week for potential replication in non-target areas. − Indicator #8a. Percentage of Civil Society Organizations rising by one level in Organizational Capacity Index (OCI). The resulting score of 58% reveals that APPEAL project has either fully or partially achieved targets set against indicators for FY 2013. For three of the indicators the planned result was not achieved. Figure 9 details the actuals for the APPEAL performance indicators for FY 2012 and FY 2013. 35 Figure 9. Actuals of APPEAL performance indicators 36 Figure 9. Actuals of APPEAL performance indicators 37 APPEAL RECOMMENDATIONS Recommendations deemed applicable to enhancing the APPEAL project are: Mercy Corps should continue the current level of effort and enthusiasm for implementation of project. Emphasize sustainability of selective and identifiable good governance practices after the USAID funded project ends. Encourage and identify opportunities for continuing reforms in consultation with implementing partners. APPEAL should enhance the utility of project implementation status reports by using nontechnical language to describe program results and impacts. While Mercy Corps submits quarterly reports which document the technical requirements and mandated activities in their respective programs, they should engage the services of public relations experts to better convey their message. At a minimum, it is recommended that they use the Executive Summary of each Quarterly Report to describe their achievements in commonly used, non-technical language. Sustainability challenges of CSOs will arise when the APPEAL project terminates. It is recommended that Mercy Corps and other project partners continue previous work on building internal capacity among local CSOs. An area that should receive special attention from APPEAL is the training of local CSOs on fundraising mechanisms and strategies. Knowledge of fundraising is crucial for the survival of local CSOs and such training will ensure, to the greatest extent possible, the sustainability of anti￾corruption and good governance activities even after the APPEAL project is terminated. It is recommended that APPEAL, in coordination with IAAC, continue to support the implementation of ACAPs among SGUs. Additionally, the monitoring of implementation of already adopted ACAPs in SGUs should be continued. Special attention and support should be given to the new working group formed by APPEAL, IAAC, and GOSWS to ensure the development and incorporation of ACAPs in local MoPDSP departments. Improvement and clarification of the APPEAL project PMP indicators is needed. Currently, for the APPEAL project, it is challenging to reconcile indicators in the PMP with furnished documentation as some are not adequately supported by documentation. - 1 - ANNEXES - 2 - ANNEX A: SURVEY INSTRUMENTS 1. STAGE SURVEY INSTRUMENT Evaluation Questions. Based on the priorities outlined in the Statement of Work and limitations of time in country, AMEX has grouped the evaluation questions into the following categories: 1) project management; 2) outcomes and results; and 3) engagement and local participation. This will enable the AMEX team to work with USAID/Mongolia staff and project officers to develop useful evaluation tools that will produce findings and recommendations that address issues of relevance, sustainability, efficiency, and effectiveness related to project performance. Each set of questions, which will be finalized in collaboration with USAID staff, will be used to assess both STAGE and APPEAL activities as appropriate. Entity Interviewed: __________________________________ Date: ______________________________________________ Conducted by: ______________________________________ 1. Management questions A. To what extent is the project management structure effective? B. To what extent were project designs and activities consistent with their objectives? C. What have been the major challenges faced by the project? D. What obstacles have there been to reaching all sectors of the target population? E. Has the project been managed and implemented in a cost effective manner? F. How are your sub-contractors and sub-grantees complying with the USAID's Branding Strategy and Marking Plan? What measures are you taking in order to ensure your sub-contractors and sub-grantees are complying with the requirement? 2. Outcome and results questions focus on achievement of goals and objectives. A. What primary activities are contributing to the achievement of the intended results? B. What activities have not led to significant results and why? C. What assumptions about activities were justified or misplaced? D. What corrective actions did project implementers take? E. What factors can be attributed to a lack of results? F. What are the greatest achievements/successes of the project since its launch? G. What are the lessons learned major from the project? 3. Engagement and local participation questions focus on the project’s relevance and inclusiveness. A. To what extent is the project reaching/benefiting all citizens? B. Has the project engaged stakeholders and beneficiaries at the local and national levels? C. To what extent has the project been implemented in a participatory way? D. What obstacles have there been to reaching the targeted populations? E. How were those obstacles overcome? F. Are all appropriate stakeholders being properly consulted? G. How have the lives of targeted groups improved? Space reserved below for extended notes on above questions - 3 - 2. APPEAL SURVEY INSTRUMENT (Revised Feb 8, 2014) On Site Interview Name of Aimag (or Soum): _____________________________________ Location/venue of interview: ___________________________________ Date of Visit/Interview: _______________________________________ Name of Person Interviewed: ___________________________________ Information or Document Requested (Notation code: Est. = Estimated; N/A = Not available; UK = Unknown) 1. Population of SGU 2. Current or recent Budget for SGU 3. Current of recent Financial Report 4. Does SGU have Bank Account? 5. Organizational chart or description for SGU 6. Number of Employees of SGU 7. Names and titles of Elected officials of SGU 8. Names and Titles of Highest level Appointed officials 9. Do you have an inventory of SGU property? 10. How do citizens receive information about SGU meetings and actions? 11. Does SGU have any contact with NGOs? 12. Does the APPEAL program serve a useful purpose for your Aimag? 13. If so, what are the most useful elements of the APPEAL program? 14. What are the least useful aspects of the Appeal Program? 15. Have any staff persons from Mercy Corp visited here? 16. Has there been any project activity in any Soums within the Aimag? Please elaborate. 17. Do you know anything about the APPEAL program and its activities? If so, what do you know? 18. Please identify the groups, organizations, and other beneficiaries of APPEAL? 19. What are the biggest problems with your participation in APPEAL? 20. What field monitoring has occurred by Mercy Corps, _____ How often and by whom? 21. Do you know this project is funded by USAID, or who is funding this project? 22. Other comments/observations: Comments and Notes on other topics of concern: Table of SGU(and State Level) functional responsibilities and activities shown below. If the function is a shared responsibility between or among more than one level of government mark all boxes applicable. Comment on nature of any shared responsibilities (e.g.: who delivers service and who pays for it, etc.) - 4 - Current allocation of functional responsibilities of Mongolian Sub-national governmental units (SGUs) Function/Level State (Central) Province/ Aimag District/ Soum Sub￾District Other Other Other (egg, Private Sector) Military Defense Law Enforcement Judicial Admin Health Soc Welfare Education Electricity Telephone Gas Postal Service Water Sewer Solid Waste Parks, Open Space Maint. Environmental Regulation Streets Pub Trans Land Use BsnsLic BldgInsp Tax Assess Tax Collection Procurement Disbursement Treasury Mgmt Budget Accounting Financial Reporting Personnel. Function Public Info Other Interview conducted by Team Member ___________________________________________ - 5 - 3. FOCUS GROUP SURVEY INSTRUMENT (Feb 8, 2014) Entity Interviewed __________________________________ Date ______________________________________________ Conducted by ______________________________________ 1. Do you have knowledge of the APPEAL Project? 2. Is the project management structure effective and consistent with objectives? 3. What have been the major challenges faced by the project? 4. What corrective was taken? 5. Do you know who is financing this project?_______; How? 6. What was greatest success of the Project since beginning? 7. What lessons were learned from the project and what should be improved? 8. Has the project engaged stakeholders and beneficiaries at the local level? 9. What obstacles have there been to reach the targeted population? 10. How were those obstacles overcome? 11. How have the lives of target groups improved? 12. Do you think that APPEAL supported transparency in governance? 13. Do you think that APPEAL contributed to more accountable governance? 14. Do you think that APPEAL contributed to more effective governance? 15. Will you be able to continue good governance and anti-corruption activities after APPEAL project terminates? 16. Has APPEAL project contributed to more gender equality and inclusion? Other: Note: Above questions were selectively used in each focus group to elicit program relevant information. - 6 - ANNEX B: METHODOLODY AND ANALYSIS 1. Theoretical Background Starting point in the evaluation of two USAID good governance projects (STAGE and APPEAL) was of the explanatory nature. In this sense, the evaluation strived to have a research approach in which findings and conclusions were made after and in accordance with a data collected. The team had neither assumptions nor hypothesis prior to the projects evaluation. Therefore, the most appropriate research principle for this evaluation can lean to the principle of the Grounded Theory. In Grounded Theory a researcher avoids theoretical preconceptions when examining an observed phenomenon. From this perspective we can say that the evaluation which leans on Grounded Theory attempts to “get though and beyond conjecture and preconception to exactly the underlying processes of what is going on, so that professionals can intervene with confidence to help resolve the participant's main concerns” (Glaser 1978) . Hence, the summarized characteristics of an evaluation/research based on principles of the Grounded theory are following Charmaz (2006): • simultaneous collection and analysis of data • creation of analytic codes and categories developed from data and not by pre-existing conceptualizations • discovery of basic social processes in the data • inductive construction of abstract categories • writing analytical memos 2. Methodological approach AMEX’s evaluation approach engaged internal and external stakeholders (USAID technical and project staff, implementing partners, Government of Mongolia ministry staff, regional and local government staff, business leaders, media, civil society and community leaders, and average citizens) in the design, data collection, and interpretation of findings. This ensured that the final evaluation report reflected the different perspectives, concerns, and needs of USAID, project implementers, key stakeholders, and beneficiaries as well as non-beneficiaries. The evaluation methodology blended qualitative (e.g. pre- and post-activity descriptions or conditions or people’s perceptions of the extent any positive changes in those condition) and quantitative (e.g. numbers and percentages) data collection and evaluation techniques. The evaluation team worked with USAID/Mongolia to identify the full range of STAGE and APPEAL supported activities and organizations at the national and local levels. After reviewing relevant documents, the evaluation team used three survey instruments, shown in Appendices as Exhibits A, B and C, respectively. Information was collected from primary sources through key informant interviews and focus groups composed of project beneficiaries, as well as non￾beneficiaries and interested observers who might have been in a position to observe how well an activity was implemented. STAGE – At the state level (STAGE), the team conducted interviews with officials from the Government of Mongolia’s ministries and agencies, business leaders, representatives from the Cabinet Secretariat and Civil Service Council, judges and court staff, and members of legal professional organizations. APPEAL – During field visits for the APPEAL project, we conducted interviews with key beneficiaries in the provinces to assess ongoing projects and collect evidence of collaborative efforts by implementing partners, and the perceptions of those involved in individual activities. The team engaged regional and local government officials, and representatives of local professional, civil society, and anti-corruption - 7 - organizations. The team also conducted six focus groups with local CSOs, beneficiaries and non￾beneficiaries in each visited aimag. Data Collection in brief: During the initial phase of our work, we reviewed project assessments, cooperative agreements between USAID and implementing partners, performance monitoring plans for both STAGE and APPEAL, yearly work plans, quarterly reports, surveys, gender studies, statements of work, website posts, and other relevant materials developed by USAID and its implementing partners. We were looking for the most relevant and current data on legislative and regulatory reforms, governance, corruption, and civic engagement as well as indicators and targets for STAGE and APPEAL activities. To document evidence of the project’s main areas of impact and to identify activities and results, the evaluation team used a semi-structured questionnaire with open-ended questions to guide informant interviews and to serve as the basis of meeting summaries. Additionally, a standard checklist in case of PMPs for STAGE and APPEAL was used as part of the documentation review to assess the degree to which each project’s objectives and intended results have been articulated qualitatively and quantitatively; what base line and benchmark data have been collected and tabulated to assure completeness, timeliness, accuracy and relevance (PMP analysis). Data Analysis and Disaggregation in brief: The collection of relevant quantitative and qualitative data was used to assess the effectiveness and efficiency of STAGE and APPEAL activities. During analysis of the data, the team was led by the maxim that “not everything that is important can be counted, and much that can be counted is not worth knowing.” The evaluation team reviewed the activities of The Asia Foundation and Mercy Corps and has used the feedback received from beneficiaries and key stakeholders for the purpose of identifying any mid-course changes they may have undertaken. The team examined the program’s effectiveness by comparing actual achievements with respective work plans and goals. The evaluation Team emphasizes the collection of reliable empirical data indicating success or failure as opposed to anecdotal evidence. Finally, because of time limitation of mid-term evaluation the team predominantly used qualitative method of data collection and analysis. However, due to importance of quantitative data collection and analysis for the method of triangulation statistical data was used to support/oppose findings assembled through qualitative methods of analysis. For that reason, a basic spreadsheet program (Microsoft Excel) was used to analyze and display data collected as part of our evaluation process. Lastly, our goal was to ensure that the findings and conclusions related to the effectiveness of STAGE and APPEAL activities are based on data that are both accurate and reliable, and that information gathered is representative and reasonably reflects results actually achieved. Design and Methodology Limitations. Due to the limited time and funding for the evaluation, primary data collection of country statistics was not feasible. Therefore, the evaluation team carefully scrutinized secondary sources of quantitative data, such as government statistics and legislative reports; data collected by the USAID, The Asia Foundation, and Mercy Corps; and surveys conducted over time by the project implementers, partners, and local civic and professional organizations. Where possible, the data was disaggregated by gender, age, geographic location, and other important characteristics to observe group-specific impacts of the project. 2.1 Data Collection and Analysis When conducting an evaluation of any kind of project, data collection and analysis are the processes which cannot be strictly divided into two timely separate actions. Moreover, we can say that if a research principle is constructed under conditions that can be found in the Grounded Theory then data - 8 - collection and analysis are conducted in timely parallel manner and complement each another. After each session of data collection the team analyzed data in order to make a clear picture of doings of STAGE and APPEAL projects and structure a better approach for a next session of data collection. This approach also helped the team in selection of relevant data and elimination of irrelevant information in relation to the needs of the projects’ evaluation. This also can be called as a constructivist approach, where understanding of STAGE and APPEAL projects was constantly upgraded throughout entire period of data collection. Another methodological principle that was used in collection and analysis of data was method of triangulation where secondary collected quantitative data (reports, surveys, PMPs,) was used as a support/oppose factor to the primary collected qualitative data (key informant interviews, field visits, focus groups). By triangulating qualitative and quantitative data the evaluation team was able confirm its findings and pin point whether STAGE and APPEAL projects’ objectives were accomplished in successful, efficient and effective manner. In this way, data collection and analysis can be divided in three segments: Table 1. Data Collection and Analysis timeframe Week1 Week2 Week3 Week4 Week5 Documentation review and analysis Interviews, field visits, focus groups, and analysis PMP analysis 2.1.1 Documentation review and analysis Main purpose of documentation review and analysis is to get general knowledge of a project activities, objectives, obstacles and outcomes. It serves as introductory point for further evaluation activities, as well as benchmark against which the evaluation team compares other findings, obtained with different methods of data collection. In this sense, the evaluation team reviewed the activities of The Asia Foundation, Mercy Corps in relation to the STAGE and APPEAL projects and has used the findings from reviewed documentation (yearly work plans, quarterly reports and cooperation agreements with USAID) in order to compare and verify documented data with the feedback received from beneficiaries and key stakeholders. Additional purpose of reviewing respective documentation was to identify any mid-course changes occurred and/or to detect any unanticipated problems. Beside this, the team examined the program’s effectiveness by comparing actual achievements with respective work plans and goals. Additional to the primary project documentation the team has reviewed and analyzed additional documentation received from Mercy Corps and The Asia Foundation field offices as well as other publicly accessible documentation relevant to the projects 2.1.2 Key informant Interviews In order to obtain in-depth knowledge of both STAGE and APPEAL projects the team has constructed two semi-structured interview survey instruments (see exhibit A and B ). The purpose of the semi￾structured interview is to extract from an interviewee the most useful information. Its questions are open ended, which allows (to some extent) new ideas to be brought up during the interview as a result of what the interviewee says. The interviewer in a semi-structured interview generally has a framework of themes to be explored. In case of both STAGE and APPEAL the evaluation team highlighted Management, Outcomes and Participation topics as most relevant and it strived maintain interviews within set boundaries. - 9 - Even if both questionnaires are designed to help the evaluation team in comprehension of STAGE and APPEAL projects, they differ in level of detail. In case of interviews conducted with public officials and other key informants in City of Ulaanbaatar, for both STAGE and APPEAL projects, the team used exhibit A survey instrument, which has fewer questions and served more as interview guideline. Generally, it helped the evaluation team to focus an interview on the topics (Management, Outcomes, and Participation) at hand without constraining them to a particular format. This freedom assisted the evaluation team to tailor the questions to the interview context/situation, and to the people we were interviewing. Data collected from Ulaanbaatar interviews was summarized in short reports (see Table below) and served as ground point for findings and comparison with other data. Table 2. Structure of short report Name of Organization Name & Titles Date& Location Brief Description of activities Relevance for the STAGE/APPEAL in terms of good governance Relevance for the STAGE/APPEAL In terms of objectives Achievements/Outcomes Reason for failing to perform (if any)/ problems Management/Communication structure Management / Communication issues (if any)? Cost effectiveness? Complying with the Branding and Marking policies of USAID? Additional comments/clarifications/ complaints/ relevant peculiarities As regard to the exhibit B survey instrument, it was used only in evaluation of APPEAL project. This survey instrument strives to portray picture of Sub Governmental Unit (Aimag or Soum) in order to help the evaluation team in exploration of existing relation between a SGU and APPEAL project. Namely, by using this survey instrument the evaluation team wanted to find out how APPEAL project performed regarding the Management, Outcomes and Participation areas in observed SGU. Finally, the matrix below consolidates information gathered from use of both Survey Instruments, summarizes it and enables its dis-aggregation. - 10 - Table 3. Sample Matrix of Contacts and Responses (Disaggregated) Contact Knowledge of Positive impact upon # Organization or interviewee STAGE APPEAL USAID Funding Anti￾Corrupt. Legal frame work Transparent Civil Society Improve Government administration Services 1 MoJ yes Yes Yes yes yes yes -- yes 2 3 4 5 6 7 8 9 1 0 1 1 1 2 Total # Yes -- -- Total Yes -- Total N/A -- 2.1.3 Focus Groups A focus group interview is an inexpensive, rapid appraisal technique that can provide managers with a wealth of qualitative information on performance of development activities, services, and products, or other issues. A facilitator guides a group of 7 to 11 people in a discussion of their experiences, feelings, and preferences about a topic. The facilitator raises issues identified in a discussion guide and uses probing techniques to solicit views, ideas, and other information. Sessions typically last one to two hours. (USAID, 2011) The evaluation team strived to conduct Focus Groups in accordance with USAID guidelines. In this sense, the team organized six focus groups, one in each visited aimag. The aim of focus groups was to identify participant group’s perception on APPEAL project and its activities. Questions for the focus group were comprised into following areas: Knowledge, Management, Outcomes, and Good Governance. In the first place, the team strived to find out: Knowledge – How well APPEAL project is known as well as how APPEAL project contributed to overall awareness and comprehension of anti-corruption legislation and activities, administrative reform legislation and activities and good governance activities. Management – How APPEAL project managed and organized its activities in accordance with the set objectives, as well as how civil sector perceived the facilitation by the Mercy Corps. Outcomes – What were the outcomes of the APPEAL’s activities and how civil sector perceived those outcomes? Good Governance – Additionally, the team wanted to detect participants’ perceptions regarding the impacts of the APPEAL project on good governance in SGUs. There are numerous definitions of the notion of good governance that have been created by eminent international organizations such as the UN, UNDP, World Bank, ADB. Though all of these definitions are different, they encompass the same principles for defining good governance: equal participation, transparent government, accountable government, effective government, sustainable government, and gender equality. In order to get more - 11 - precise information on good governance the evaluation team has disaggregated good governance into these basic principles, which can be defined as follows: Participation – A meaningful opportunity for all members of society exists to have an informed and consequential voice in decisions that affect them. Participation can be either direct or through legitimate intermediate institutions or representatives. Here, the team also wanted to know about participation in APPEAL project Transparency – Decisions and their enforcement follow established rules and regulations and all people affected have access to easily understandable and timely information. In regard to the transparency the team wanted to know perception and satisfaction of participants about APPEAL project’s activities toward a more transparent local government Accountability – Officials have the responsibility and capacity to answer for actions and decisions to people affected by the actions or decisions of government agencies and institutions. In regard to the accountability, the team wanted to know perception and satisfaction of participants about APPEAL project’s activities toward a more accountable local government Effectiveness – Processes and institutions produce results that meet the needs of society while making the best use of resources at society’s disposal. In regard to the effectiveness the team wanted to know perception and satisfaction of participants about APPEAL project’s activities toward a more effective local government. Sustainability – Will outcomes of APPEAL project and activities initiated during the project period outlive the date of termination of the project? Gender Equality – Both genders feel that they have a stake, do not feel excluded, and have the opportunity to participate in decisions and activities that interest them. In regard to the gender equality the team wanted to know perception and satisfaction of participants about APPEAL project’s activities toward a more gender inclusive local government Generating group decision After each discussion on a certain area (e.g. Management) the facilitator asked participants to scale, as a group, APPEAL's performance in relation to specified topics. Scaling of a groups’ perception about APPEAL's performance was: 5 points = Highly Satisfied 4 points = Satisfied, 3 points = Neither Satisfied nor Dissatisfied 2 points = Dissatisfied 1 point = Highly Dissatisfied In this way, the team was able to aggregate and present a group’s perception (see table below). Although, reaching consensual opinion of the group was main target, the team respected and noted difference in opinions among participants. Detailed information on the focus group deliverables can be found in attached reports/ notes. - 12 - Table 4. Group’s Satisfaction with the APPEAL project performance and impact on governance (5) - Highly Satisfied (4) -Satisfied (3) - Neither Satisfied nor Dissatisfied (2)- Dissatisfied (1) - Highly Dissatisfied APPEAL’s performance in relation to objectives Knowledge X Management X Outcomes X Participation X Total/Average 18/4.5 APPEAL’s performance in relation to the good governance Gov. Transparency X Gov. Accountability X Gov. Effectiveness/ Efficiency X Sustainability X Gender Equality X Total/Average 18/3.6 Finally, the table/matrix below consolidates and aggregates information gathered from use of Focus Groups survey instrument. Matrix served as the instrument for catching overall satisfaction rate with APPEALS performance in relation to all 9 areas/topics of and across all six focus groups. Data from the matrix can also be disaggregated by areas/topics of performance, and by each aimag focus group. It is important to know that this matrix served for simplifying and summarizing findings from the focus groups. However, data collected from focus groups is of qualitative nature and serves to obtain in-depth knowledge of the observed phenomenon. Therefore, the team obtained and explained each area/topic in the focus group analysis, which can be found in the body of the report. In this sense, the matrix served as a supplementary and complementary instrument to the qualitative analysis of the focus groups. Table 5. Sample Satisfaction Rate Matrix Sample Matrix - Satisfaction Rate of APPEAL performance Overall performance Good Governance Aimag / focus group Know. Manag Outcome Particip. Gov. Trans Gov. Acc. Gov. Effe. Sustain. Gender Equality Avgsatisfaction of aimag focus groups with APPEAL’s performance Uvs 4 3 4 4 4 4 4 4 4 Gobi Altai 5 4 4 2 4 3 4 4 4 Uvurkhangai 5 4 4 4 3 4 4 3 4 Omnogobi 5 4 4 2 4 3 3 4 3 Khenti 5 5 4 4 4 4 4 4 5 Arkhangai 4 5 4 2 4 4 3 4 2 Average satisfaction of all aimag focus groups per topic Overall Satisfaction of all groups - 13 - 2.1.4 PMP ANALYSIS This section analyzes the project monitoring plans (PMP), performance indicators including baseline data, targets and actuals of Fiscal Year 2012 and Fiscal Year 2013 for both APPEAL and STAGE projects. As Fiscal Year 2012 covers only three months, our PMP analysis for both projects will be concentrated on Fiscal Year 2013 ( October 1, 2012 through September 30, 2013). This is the second fiscal year￾based work plan of the contracts. Monitoring progress and evaluating results are key management functions in any performance-based management plan. Performance monitoring is an on-going process that allows managers to determine whether or not an activity is making progress towards its intended results. Performance information plays a critical role in planning and making decisions. Evaluation is the periodic assessment of a project’s relevance, performance, efficiency, and impact—both expected and unexpected—in relation to stated objectives. The strength of monitoring and evaluation lies in its ability to provide timely performance information which enables USAID and project staff to manage for results and to improve project performance. A. Critical assumptions To achieve the life-of-project results identified in both APPEAL and STAGE projects’ Results Framework, several critical assumptions must hold true. First, project partners in government agencies, ministries, and parliament must continue to cooperate and engage with the project on activities to achieve progress towards identified results. Second, some degree of continuity of key counterparts in GoM agencies would be required for project activities to be completed in a timely manner to minimize delays in implementation. As these factors are beyond project control, we will exercise our best efforts to adjust and adapt to the changing environment as needed to keep the proposed activities on track. B. Performance indicators The corresponding performance indicators for both STAGE and APPEAL projects are shown in Exhibit II-1 and Exhibit II-2, respectively. Exhibit II-1: STAGE Project Performance Indicators for FY 2013 Goal: To strengthen democratic governance by building a more transparent and accountable regulatory and legislative environment and ensuring principles of checks and balances 1 Improved legal and regulatory environment designed to increase transparency and accountability 2 Number of GoM institutions/employees involved in USG supported good governance activities Objective 1: Promoting good governance to reduce corruption and increase transparency Result 1.1: Increased knowledge of the modalities of grand corruption 3 Number of breakthrough anti-corruption activities are ongoing in Mongolia, with USG support 4 Number of citizens, Government of Mongolia (GoM) and CSO employees informed about knowledge of types and methods of grand corruption, with USG assistance Result 1.2: Strengthened legal and policy environment for transparency and good governance 5 Number of practical efforts, including proposals, recommendations and draft decisions, put forward to strengthen the legal & policy environment for AC are ongoing, with USG assistance - 14 - Result 1.3: Increased capacity of relevant AC agencies to fulfill their mandate 6 Decrease in the # of inter-agency procedural and practical issues conflicts and differences Result 1.4: Increased private sector engagement in anti-corruption 7 Number of companies adopting AC plans, rules, regulations to improve corporate governance Result 1.5: Increased capacity of civil society to monitor anti-corruption in the public and private sectors through innovative modalities 8 Number of cases of CSOs effectively monitoring actions of government agencies 9 Number of CSO employees trained on AC laws and monitoring skills Objective 2: Improving Administrative Reforms 10 New and more detailed set of legal and regulatory environment created Result 2.1: Improved understanding on administrative practice in key agencies 11 Number of trends across agencies that affect performance improvement in key agencies 12 Number of government agencies assessed for current practice - 15 - Exhibit II-2: APPEAL Project Performance Indicators for FY 2013 Goal: Improved legal, institutional and administrative processes in targeted Mongolian government agencies promote democratic reforms and economic growth in the country Objective 1: Key elements of Mongolian legal frameworks related to anti-corruption and good governance are understood by and relevant to government institutions, civil servants, civil society and the public at the local level Result 1.1 Legislation related to anti-corruption and good governance understood and strategized at the local level 1 Number of aimag governments with timely, relevant anti-corruption action plans 2 Number of line departments that incorporate Ministry of Human Development and Social Welfare anti-corruption strategy into local plans Result 1.2 APPEAL community stakeholders understand good governance and anti-corruption initiatives related to specific government authorities at local level 3 Percentage change in public who have access to and understand local anti-corruption and governance documents 4 Number of media and CSO reports and articles reflecting local government plans and actions relevant to APPEAL Result 1.3 APPEAL and STAGE stakeholders exchange findings and recommendations to further advance anti-corruption and good governance practices and policies 5 Percentage change in public confidence perception about government and public institutions effort in combating corruption 6 Number of best practices introduced in Good Governance Week for potential replication in non-target areas Objective 2: Exercise of authority and delivery of government services reflect transparent and accountable administrative processes, provide effective checks and balances, and engage informed communities to optimize quality and impact of governance at local level Result 2.1: Improved consistency of procedural administrative regulations (PARs) 7 Number of multi-sector stakeholders understand PARs and governance laws application to promote good governance and stop corruption Result 2.2 Civil society has increased impact in anti-corruption and good governance initiatives 8 a Percentage of Civil Society Organizations rising by one level in Organizational Capacity Index (OCI) 8 b Number of Civil Society Organizations and community groups engaged Result 2.3 Local governance is more transparent and inclusive 9 Percentage change in citizens’ perceptions of quality and consequence of citizen's halls - 16 - 10 Number of new Soum Master Plans developed and ratified by soums citizens hurals 11 Number of ICT innovations to reach clients and communities tested and adopted in APPEAL soums and aimags. C. Actuals of performance indicators Exhibit IV-1 and Exhibit IV-2 for STAGE and APPEAL projects at the end of section IV details the actuals for the performance indicators under each objective and result. This summary exhibit contains projected life-of-project targets, baseline data, targets and actuals set by yearly and a cumulative total projected target and actuals for FY 2012 and FY 2013. D. Summary of PMP results STAGE Project Per contractual requirements, the Quarterly Performance Reports have documented interim progress on achieving the targets set out for the indicators. The summary of results of the performance evaluation at the end of FY 2013 shows that: • For 8 indicators (67%), the planned result was achieved as the target number was met • For 1 indicator (8%), Indicator #9: Number of CSO employees trained on AC laws and monitoring skills; the planned result was partly achieved as progress was made toward meeting the target number. This indicator’s target in Year 2 was 50, however the actual result was 44. • For 1 indicator (8%), Indicator #6: Decrease in the number of inter-agency procedural and practical issues, conflicts, and differences; the planned result was not achieved as no progress was made toward meeting the target number. This indicator’s target in Year 2 was 3; however actual result in Year 2 was 0 (zero). • For 2 indicators (17%), Indicator #7: Number of companies adopting AC plans, rules, and regulations to improve corporate governance, and Indicator #11: Number of trends across agencies that affect performance improvement in key agencies, there was no target and no expected results. The resulting score of 75% reveals that STAGE project has either fully or partially achieved targets set against indicators for FY 2013. For two of the indicators the planned result was not achieved due to the Year 2 was a baseline year (no target, no expected results). APPEAL Project Per contractual requirements, the Quarterly Performance Reports have documented interim progress on achieving the targets set out for the indicators. The summary of results of the performance evaluation at the end of FY 2013 shows that: • For 4 indicators (33%), the planned result was achieved as the target number was met. • For 3 indicators (25%), the planned result was partly achieved as progress was made toward meeting the target number. − Indicator #1. Number of aiming governments with timely, relevant anti-corruption action plans. The target for this indicator was 10 in Year 2; however, according to the supporting documents provided by Mercy Corps, only six ACAPs were submitted in Year 2. Mercy Corps claims that 15 ACAPS were submitted, but the evaluation team was not provided any documentation for the remaining nine ACAPs. - 17 - − Indicator #2. Number of line departments that incorporate Ministry of Human Development and Social Welfare anti-corruption strategy into local plans. According to the quarterly report, as of September 2013, five action plans were submitted; however, Mercy Corps has not provided any supporting documents to back up this indicator. The target for this indicator was 10 in Year 2, but only five action plans were submitted according to Mercy Corps. − Indicator #4. Number of media and CSO reports and articles reflecting local government plans and actions relevant to APPEAL. The target for this indicator was 700 in Year 2. As of September 2013, 526 media reports (TV, radio, newspaper articles) shared information about government plans and actions relevant to APPEAL. • For 3 indicators (25%), the planned result was not achieved as no progress was made toward meeting the target number. − Indicator #5. Percentage change in public confidence perception about government and public institutions’ effort in combating corruption. The target was 3% in Year 2; however, Mercy Corps did not provide any documentation to support this indicator. The evaluation team consulted with the SPEAK longitudinal findings in order to try and find the proper answer, but there wasn’t any question that exactly refers to this indicator. The survey finding that has the closest meaning to the targeted indicator was presented in figure 7.1 and 7.2 of the SPEAK longitudinal findings (December 2013, p. 20). The specific question was: “In your opinion, who should organize and lead the effort to combat (grand) corruption”. In both figures results for local governments were not reaching the indicator’s target of 3% change. In figure 7.1 (in relation to corruption), public perception on a local government taking a leading role in fighting corruption jumped from 2.6% in November 2012 to 3.2% in September 2013. In figure 7.2 (in relation to grand corruption), public perception of a local government taking a leading role dropped from 1.8% in March 2013 to 0.8% in September 2013. − Indicator #9. Percentage change in citizens’ perceptions of quality and consequence of citizen’s halls. The target in Year 2 was 10%; however, Mercy Corps did not provide any documentation to support this indicator. − Indicator #10. Number of new Soum Master Plans developed and ratified by soums citizen’s halls. The target was 40 in Year 2. While it is an ongoing activity, Mercy Corps did not provide any documentation to support this indicator. • For 2 indicators (17%), there was no target and no expected results. − Indicator #6. Number of best practices introduced in Good Governance Week for potential replication in non-target areas. − Indicator #8a. Percentage of Civil Society Organizations rising by one level in Organizational Capacity Index (OCI). The resulting score of 58% reveals that APPEAL project has either fully or partially achieved targets set against indicators for FY 2013. For three of the indicators the planned result was not achieved. - 18 - ANNEX C: REFERENCES This includes bibliographical documentation, meetings, interviews, and focus group discussions. Charmaz K (2006) “Constructing Grounded Theory: A Practical Guide through Qualitative Analysis” London Sage Publications Glaser BG (1978) Theoretical sensitivity, California, The Sociology Press Globe International (2013) “Report of STAGE activities Mar-Sep 2013” Khentii Aimag Trainers (2013) “Report of Good governance trainings” Mongolian Association of Local Authorities (2014) “Report of APPEAL project” Mongolian Association of Local Authorities (2014) “Report of STAGE project” Mercy Corps (2012) “APPEAL 1st Quarterly Report Jul-Sep 2012” Mercy Corps (2012) “APPEAL 2nd Quarterly Report Oct-Dec 2012” Mercy Corps (2013) “APPEAL 3rd Quarterly Report Jan-Mar 2013” Mercy Corps (2013) “APPEAL 4th Quarterly Report Apr-Jun 2013” Mercy Corps (2013) “APPEAL 5th Quarterly Report Jul-Sep 2013” Mercy Corps (2013) “Performance Monitoring Plan Status as of September 2013” Mercy Corps (2012) “Work Plan-Year One APPEAL” Mercy Corps (2012) “Work Plan-Year Two APPEAL” Mercy Corps (2012) “Work Plan-Year Three APPEAL” Mercy Corps Khentii Aimag Office (2014) “Report of APPEAL activities in Khentii Aimag” Mercy Corps (2011) “Product catalogue: Made in countryside of Mongolia” Mercy Corps, TAF and USAID (2013) “Gender Desk review” Ministry of Justice (2013) “Draft General Administrative Law of Mongolia” Newspaper “News of Kherlen Soum” No. 3(003)1 January 2014, Khentii Newspaper “Colour of Uvs” newspaper, No. 03/44/ January 25, 2013, Uvs Newspaper “Colour of Uvs” newspaper, No. 09/50/ April 15, 2013, Uvs Newspaper “Colour of Uvs” newspaper, No. 12/53/ June 7, 2013, Uvs Newspaper “Colour of Uvs” newspaper, No. 14/55/ July 5, 2013, Uvs Newspaper “Colour of Uvs” newspaper, No. 20/61/ November 15, 2013, Uvs Newspaper “Colour of Uvs” newspaper, No. 21/62/ December 12, 2013, Uvs Newspaper “Tsetsii News”, No. 1, January 2014 Oyuntungalag, O., (2013) “Our beginning of success” Togt-tsetsii soum, Umnigobi aimag Power, Greg (2013) “Parliament Ethics and Handbook for Parliamentarians” (translated into Mongolian in 2013) - 19 - Public Administration New Initiative & The Asia Foundation (2013) “Handout: Law on Information Transparency and right to obtain information” The Asia Foundation (2013) “Performance Management Plan for STAGE” The Asia Foundation (2012) “STAGE 1st Quarterly Report Jul-Sep 2012” The Asia Foundation (2012) “STAGE 2nd Quarterly Report Oct-Dec 2012” The Asia Foundation (2013) “STAGE 3rd Quarterly Report Jan-Mar 2013” The Asia Foundation (2013) “STAGE 4th Quarterly Report Apr-Jun 2013” The Asia Foundation (2013) “STAGE 5th Quarterly Report Jul- Sep 2013” The Asia Foundation & Sant Maral Foundation (January 2013) “Study of Private Perceptions of Corruption” The Asia Foundation & Sant Maral Foundation (September 2013) “Study of Private Perceptions of Corruption” The Asia Foundation & Sant Maral Foundation (December 2013) “Study of Private Perceptions of Corruption” The Asia Foundation & Sant Maral Foundation (March 2013) “Survey on Perceptions and Knowledge of Corruption” The Asia Foundation & Sant Maral Foundation (September 2013) “Survey on Perceptions and Knowledge of Corruption” The Asia Foundation & Sant Maral Foundation (December 2013) “Survey on Perceptions and Knowledge of Corruption” The Asia Foundation & Sant Maral Foundation (December 2013) “Corruption at business environment” The Asia Foundation (2012) “Work Plan-Year One STAGE” The Asia Foundation (2012) “Work Plan-Year Two STAGE” The Asia Foundation (2012) “Work Plan-Year Three STAGE” Tumur, Munkhtungalag (2013) “For Parliamentarians” Ulaanbaatar USAID (2011), “Automated Directive Service (ADS)”, Chapter 203, Conducting Focus Group Interview USAID (2012) “Cooperative Agreement No AID‐438‐A‐12‐00004 APPEAL” USAID (2012) “Cooperative Agreement No AID‐438‐A‐12‐00005 STAGE” USAID (2010) “Executive summary-Mongolia Democracy and Governance Assessment” Women for Social Progress (2013) “An Introduction” Internet sources www.arkhangai.gov.mn www.asiafoundation.org www.gobialtai.gov.mn www.iaac.mn www.khentii.gov.mn www.mercycorps.org - 20 - www.umnugobi.gov.mn www.uvs.gov.mn www.uvurkhangai.gov.mn www.undurkhaan.com Meetings in UB, Site Visits Conducted Date Meetings with Venue/note Feb 3 USAID-M Foster, Ms. Orgilmaa Zorigt US Embassy Feb 3 The Asia Foundation – CoP Basanta Pokharel and staff TAF office Feb 4 MALA – Exec Director Mr. Manaljav and officials MALA office Feb 4 Globe International – CEO Mrs. Naranjargal, Coordinator Narantsetseg and Gundegmaa Globe International office Feb 4 PANI – Exec Director Mrs. Tserenpurev PANI office Feb 5 Mongolian Women’s Lawyers Association- President Mrs. Dolgor MWLA office Feb 5 Women for Social Progress – Exec Director Oyuntuya S. WSP office Feb 5 Mercy Corps – Chief of Party Mrs. Urana and staff MC office Feb 6 IAAC –Mrs. Bat-Otgon B., Head of Dept of Prevention and Public Awareness IAAC building Feb 6 UB City Hall – Mr. Ganbold U., Head of Public Admin Deptartment and other staffs City Hall Feb 6 Ministry of Justice – Mrs. Selenge E., Legal Reform Policy Dept Specialist MoJ office Feb 7 Business Council of Mongolia - Mr. Ser-Od Ichinkhorloo , Vice President of BCM BCM office Feb 7 Judicial General Council - Mrs. Munkhzul N., Head of Judicial Qualifications JGC office Feb 7 Zorig Foundation – Mr. Badruun G., Exec Director Mr. Amar B., Project Development Manager Zorig Foundation office Feb 10 Gobi-Altai Aimag Mr. Batzorig, Chief of Staff of Governor Office, Mr. Battaivan, Senior legal officer Mr.Rentsendorj, Secretary of Citizens Representative Khural; Mr. Ayurzana, Member of working group Zola and Milos Feb 11 Gobi-Altai Aimag, Sharga Soum Ms. Narantuya, Secretary of Citizen’s Representative Khural of Sharga soum Mr. Byambaa, Deputy Governor Ms. Oyunchimeg, Chief of Staff Ms. Gerelmaa, Treasury officer Zola and Milos Feb 11 Focus group – Gobi-Altai Aimag Zola and Milos Feb 11 Mercy Corps – Gobi-Altai Aimag Zola and Milos Feb 10 Umnugobi Aimag Mr. Batchuluun L., Chairman of Citizens Representative Khural of Umnigobi aimag Oyunaa, Saikhana and John Feb 10 Focus Group - Umnogobi Aimag Oyunaa, Saikhana and John Feb 11 Umnigobi aimag, Tsogt-tsetsii Soum Mrs. Gerelmaa Kh., Deputy Governor Mr. Munkhdelger Ts., Chairman of Citizens Representative Khural Oyunaa, Saikhana and John Feb 13 Uvurkhangai Aimag Mr. Togtohsuren, Governor of Aimag Mr. Puntsag, Head of Legal Department Ms. Otgontsetseg, Officer of Legal Department Ms. Uuriintuya, Officer of Legal Department Ms. Tsetsegmaa, Head of Social Welfare Department Mr. Ishdorj, Governor of Arvaikheer soum Zola and Milos Feb 14 Focus group - Uvurkhangai aimag Zola and Milos Feb 14 Uvurkhangai aimag, Tugrug Soum Ms. Narmandakh, Head of Staff Zola and Milos Feb 13 Uvs Aimag Oyunaa, Saikhana and John - 21 - Mr. Chimed, Chairman of Citizens Representatives Khural of Uvs Aimag Mr. Erdenebat, Consultant of Governor Feb 13 Uvs Aimag Mr. Bolor-Erdene G, Chairman of Governor Office Oyunaa, Saikhana and John Feb 14 Uvs Aimag, Tarialan Soum Mr. Tsolmon S., Governor of Tarialan Soum Mr. Bayanzul B., Deputy Governor Oyunaa, Saikhana and John Feb 14 Focus Group – Uvs Aimag Oyunaa, Saikhana and John Feb 17 Khentii Aimag Mr. Purevjav S., Chairman of Citizens Representative Khural of Khentii Aimag Mr. Bayart-Od P., Deputy Governor of Khentii Aimag Mrs. Oyuntuya D., Senior Accountant of Aimag’s Budget at Governor Administration Office Mr. Bat-Erdene S., Trainer Team Feb 17 Focus group - Khentii Aimag Team Feb 18 Khentii Aimag, Kherlen Soum Mr. Bazarkhand N., Governor of Kherlen Soum Mr. Batbayar Z., Chaiman of Citizens Representative Khural Mr. Enkhbayar A, Deputy Chairman of Citizens Representative Khural Team Feb 18 Mercy Corps – Khentii Aimag Team Feb 20 Arkhangai Aimag Mr. Lkhagvasuren B, Head of Governor Office Mr. Purevsuren S, Head of Legal Department Mr. Otgonbaatar L., Secretary of Aimag’s Citizens Representative Khural Team Feb 20 Focus group – Arkhangai Aimag Team Feb 21 Arkhangai Aimag, Tsenkher Soum Mr. Erdenetsogt J., Deputy Governor Team March 3 Sant Maral Foundation - Sumati Luvsandendev Milos - 22 - ANNEX D: MEETING AND FIELD TRIP NOTES MALA – Mongolian Association of Local Authorities Name & Titles Date & Location Manaljav G , Deputy Head Yanjinlkham P, Head of Dept. Sainbileg M, Officer Bileg-Urnukh G, Project Manager, Mercy Corps Sanjaa Ts, Project Manager, The Asia Foundation 4th of February 2014, Ulaanbaatar Brief Description of Activities APPEAL – MALA assisted local governments to draft local development plans. For this, MALA trained 4 consultants and 120 training facilitators. Over 2,000 people were trained. STAGE – MALA organized workshops for citizens and government officials to promote transparency, prevent conflict of interest, and build awareness of the anti-corruption laws. As a result, some districts of Ulaanbaatar allocated funds for anti-corruption activities in their budget. Relevance for STAGE/APPEAL in terms of Good Governance APPEAL – Develop SMP which would contribute to transparent governance and more accountable, active public participation STAGE – Support civil society in implementing the conflict of interest laws and encourage citizens’ participation to fight corruption Relevance for STAGE/APPEAL in terms of Objectives APPEAL – Strengthening public engagement in community development STAGE – Anti-corruption efforts Achievements/Outcomes APPEAL – MALA signed a MOU with Mercy Corps to develop 17 aimags’ SDP. For this propose, MALA trained 4 national consultants and 120 facilitators. 120 facilitators’ trained 10-15 people so over 2,000 people were trained on how to develop SMP. STAGE – MALA organized workshops in districts of Ulaanbaatar for citizens and representative Khural members on how to fight corruption. Reason for Failing to Perform (if any) APPEAL – N/A STAGE – N/A Management/Communication Structure APPEAL – direct communication (via email and phone) STAGE – direct communication (via email and phone) Management/Communication Issues (if any) N/A Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID APPEAL/STAGE – MALA is fully aware of USAID involvement in this project and USAID’s logo was present on their published project related documents Additional Comments MALA is very thankful both to APPEAL and STAGE projects. Only future improvement would be to reduce financial requirements. In the countryside, it is hard to collect adequate financial receipts. - 23 - Globe International Name & Titles Date & Location Mrs. Naranjargal Kh., President and CEO of Globe International Mrs. Narantsetseg Kh., Coordinator of STAGE Ms. Gundegmaa O., Coordinator of APPEAL 4th February 2014, Ulaanbaatar Brief Description of Activities APPEAL – Advertising and strengthening good governance to youth: 1)The “Good governance-Youth action” campaign has been implemented for 15 aimags; 2) The “Good governance-Youth action” webpage (www.facebook.com/pages/good￾governance-youth-action) has been developed; 3) The “Good governance-Youth action” handouts are printed; 4)Various youth events and activities such as workshops, contests, rallies, projects, and surveys have been taking place. STAGE – Impacting by information and advertisement for implementation of Anti-corruption laws and legislation project: 1) Anti-corruption materials prepared and broadcast throughout the country (5 training videos and 3 audio recordings); 2) A play on conflict of interest was prepared and broadcast on the radio as well as a training of journalists regarding COI law; 3) Anti￾corruption laws and legislation public information and advertising plan developed based on suggestions from CSOs to strengthen the transparency of public entities; 4) COI law and anti-corruption legislation website established for journalists, journalism students, and the general public; 5) Anti-corruption television series (20 episodes) developed and production commenced. Relevance for STAGE/APPEAL in terms of Good Governance APPEAL – advertisement of good governance, youth participation to local governments, transparency of information, reach out to minority (Kazakhs) STAGE – promotion of anti-corruption legislation and COI law, inclusion of CSOs on strengthening transparency and rising public awareness of anti-corruption activates , especially targeting journalists Relevance for STAGE/APPEAL in terms of Objectives APPEAL – transparent governance for youth throughout the country STAGE – Main goal of the TV shows, radio programs, and trainings is to promote investigative journalism and enable the use of resource materials, videos, and other forms of Information, Education, and Communication (IEC) materials to raise public awareness. Another objective is to promote CSOs’ participation in anti-corruption and COI activities. Achievements/ Outcomes APPEAL – Under the scope of the activities of “Good governance-Youth action”, 15 aimags took part in these activities, including the production of 65 videos to raise anti-corruption awareness and 3 workshops (shirt design and poem contests, a discussion, and a rally to identify the problems with high ranked officials). Within 61 soums, a total of 900 young adults and 2400 students and passersby participated. Other achievements include: - Webpage is online place where for young people can express their complaints and suggestions for the local governance, in relation to corruption and anticorruption activities. - “Good governance-Youth action” printed, and subsequently delivered to the public, 500 copies of handout - Globe International has developed the youth radio program in Kazakh language in Bayan-Ulgii aimag and this program has been aired at least 840 times as a result of Kazakh people’s frequent requests. STAGE – In order to raise public awareness of corruption, Globe International has initiated (with the support of PTI and IAAC) the stage program “No!” 67 attendees representing government agencies, universities, media, PTI, and other organizations participated. The 30 minutes long program (SOCHI staged reading performed by professional actors) was followed by discussions on COI issues. This staged reading was produced in close partnership between Globe International and the IAAC’s Prevention and Public Awareness Department. It was covered by three major national newspapers and four TV channels. Mongolian National Broadcast TV broadcasted the whole staged reading throughout the country. It highlighted deeper issues in conflict of interest leading to corruption. Globe International collaborated with the IAAC to develop a stage-script, public service announcements, and slide shows on COI and the COI Law for public viewing and mobilization. It also organized training workshops for 24 journalists (6 major daily newspapers, 5 television stations, 6 websites, and 3 freelancers) in Ulaanbaatar on COI and the COI Law and provided technical knowledge to journalists so that they can write interesting anti-corruption pieces. As a result, a number of independent articles appeared in major newspapers. The journalists will continue to benefit from the Globe’s online education materials for journalists. The website is: http://www.journalism-edu.mn/conflictofinterest. Globe International Center is working with NTV to produce a 20-episode television series portraying the investigation of fictional corruption cases. It should be noted that IAAC refused to support the TV series because it doesn’t portray real-life cases and it is not in accordance with existing laws of Mongolia. Nonetheless, the TV series aims to raise awareness and knowledge on corruption and anticorruption activities and its tentative topics are: abuse of office, private sector corruption, giving and taking bribes, money laundering, budget misuse, neglect of duty, illegal enrichment, and conflict of interest. Reason for Failing to Perform (if any) N/A N/A - 24 - Management/Co mmunication Structure APPEAL – Globe International claims that they lack support from Mercy Corps’ coordinators. Also, Globe International claims that the time pressure for conducting tasks under its scope was a problem, as they require more time. N/A Management/ Communication Issues (if any) APPEAL – There are no budgets to evaluate the local aimag and soums’ activities. The initial budget was 50 million tugrugs, but Globe International has received only 8.5 million tugrugs for the APPEAL project since Mercy Corps has asked Globe International to reduce its funding for project’s activities. The aimags usually request more and equal activities across their individual aimags, but the funding for tje 15 aimags and their soums are not enough. STAGE – Globe International has an issue with IAAC regarding selection of the topics for the anti-corruption TV series. IAAC officials were not cooperative in providing the ideas for corruption cases. In addition, the production team for the TV series has been facing budgetary constraints as it relates to completing the intended 20 episodes. Its initial budget was 156 million tugrugs, but Globe International would prefer some additional funding. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID Yes, Globe International is well aware of the USAID branding. Additional Comments None - 25 - PANI – Public Administration New Initiative Name & Titles Date & Location Tserenpurev. S , Executive Director 4th of February 2014, Ulaanbaatar Brief Description of Activities APPEAL – CSO capacity building project: 1) conducted a ToT (Training of Trainers) for CSOs and Mercy Corps staff on organizational capacity index measurement (March to April 2013); 2) CSOs use the social accountability tools such as public budgeting, social auditing, and citizen participation in assuring more transparent and accountable government. STAGE – PANI monitored budget of Ministry of Education and Ministry of Health and their line agencies at (UB) district level and Darkhan-Uul aimag Relevance for STAGE/APPEAL in terms of Good Governance APPEAL – CSOs capacity building to better monitor and influence work of local governments, thus achieving principles of good governance: more accountable, transparent government and active public participation Possible indicators for success: The number of requests for reviewing/auditing budgets, the number of CSO’s AC complaints to local gov., and the number of CSOs and community groups engaged. STAGE – Budget transparency and anti-corruption efforts Relevance for STAGE/APPEAL in terms of Objectives APPEAL – Address the capacity of local CSOs to grow as effective organizations, recognizing the importance of the CSOs’ role in mobilizing and strengthening public engagement in community development. STAGE – Build the capacity of and provide support to civil society actors to monitor corruption Achievements/Outcomes APPEAL – PANI conducted a four-day intensive ToT from April 22-25 of April 2013 for 30 CSO representatives (two representatives from 15 APPEAL project provinces), as well as for 15 Mercy Corps aimag partnership facilitators. The purpose is to introduce an organizational development assessment tool named Smart Growth. After returning home, the trainers held two-day sessions in May on organizational development and citizen participation for the 58 APPEAL sub-grantees that are currently implementing sub-grant projects. The participants of the two-day training then conducted self-assessments. Findings: The majority of CSOs lacked skills in: fundraising, financial management, marketing, and human resources management. STAGE – The Ministry of Health uploaded the budget information on its website (after deadline). The problem was that The Ministry of Education posted on its website the statement that it had adopted participatory approach to finalize fiscal year 2013 budget, which PANI assessed as incomplete and non-transparent statement in terms of respecting the Freedom of Information Act and that the Ministry should publish the budget on its website. Also, PANI has developed and distributed 1,000 Information, Education, and Communication (IEC) materials to raise public awareness. Reason for Failing to Perform (if any) APPEAL – Out of 58 CSOs 1 CSO failed to comply with MC requirements STAGE – N/A Management/ Communication Structure Mercy Corps is practicing ICT (Information and Communications Technology) applications with PANI to help local CSOs (e.g., WEBINAR held in MAY 2013) STAGE – Direct communication, frequent visitation from STAGE project officers Management/ Communication Issues (if any) N/A Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID APPEAL/STAGE – PANI is fully aware of USAID involvement in this project and USAID’s logo was present on their published project related documents. Additional Comments PANI expressed satisfaction working with both STAGE and APPEAL. However, when asked which project PANI preferred more, its representative gave her preference to STAGE because in cooperation with STAGE PANI was able to be directly involved in anti-corruption activities. - 26 - Mongolian Women Lawyers Association Name & Titles Date & Location Ms.Dolgor, president of MWLA February 5th 2014, Ulaanbaatar Brief Description of Activities APPEAL – MWLA piloted the training of Master Trainers of Good Governance (GG) for 33 Mercy Corps field staff (December 2012). After that, they updated training curriculum and conducted a training of 64 Master Trainers from 15 aimags on key governance legal frameworks and procedural administrative regulations (good governance laws). The training curriculum on good governance was aiming to educate on: Conflict of Interest Law, Anti-corruption Law, FIOA, the new Budget Law, Gender Equality Law and other PAR laws. Those 64 trainers have conducted 109 trainings in 15 target aimags and 60 soums, training 4,960 individuals (more than the targeted 3000) on GG laws and PAR laws. APPEAL/STAGE – MWLA facilitated a nationwide public consultation on the general Law of Procedural Administration and hosted a webinar using Blackboard technology that brought together 138 people from 20 aimags for a 3 hour training to discuss this draft law. Following this training, the local facilitators conducted a public discussion on this law in 20 aimags. Also, in coordination with IAAC and STAGE/APPEAL, MWLA facilitated public consultations on The National Strategy on Combating Corruption at the local level in 15 APPEAL project aimags, six non-project aimags, and 9 districts of Ulaanbaatar city (with help from STAGE). STAGE – Similarly, MWLA facilitated discussion on GAL (General Administrative Law). It also trained 60 trainers on facilitation GAL public discussion. Relevance for STAGE/APPEAL in terms of Good Governance APPEAL/STAGE – MWLA is aiming to educate the public on GG, Anti-corruption, PAR, and GAL, thus contributing to the notions of participation and a more transparent and accountable governance. Relevance for the STAGE/APPEAL in terms of Objectives APPEAL/STAGE – MWLA is participating in achieving both objectives : Knowledge/Understanding (facilitating public discussions and conducting trainings) and Activities (involved in drafting Laws on GAL and APL) Achievements/Outcomes APPEAL – Exceeded expectation on the number of individuals trained and on the quality of trainings on laws. APPEAL/STAGE – The feedback from public discussion was compiled by MWLA and Mercy Corps and submitted to the MoJ in April 2013 for recommendation on the APL (more than 600 ideas and recommendations). MWLA succeeded in broadening public discussion on national strategy for anti-corruption to local communities by using its local network of members and CSOs. Interviewee emphasized that for the first time in Mongolian history broad public/expert discussions on drafting laws occurred, and the credit for this can be given to APPEAL and STAGE. STAGE – 1075 people participated in discussions on GAL and more than 800 recommendations were sent to MoJ (in charge of drafting GAL). Reason for Failing to Perform (if any) N/A Management/Communicatio n Structure APPEAL – ToT was just a pilot project conducted in coordination with Mercy Corps. After its successful implementation, the curriculum for ToT was updated and then used to train and educate a broader population of stakeholders. When APPEAL/STAGE ordered to MWLA to organize a public discussions on GAL and APL, this organization executed its duties in following order: 1) train facilitators for a discussion; 2) conduct broad discussion on the laws; 3) analyze discussion and summarize findings to help inform recommendations ; 4) deliver recommendations to MoJ Management/Communicatio n Issues (if any) APPEAL – 1) Difficulties in legal knowledge transfer because the majority of trainers/facilitators are not legal experts they sometimes have hardships when conducting trainings; 2) Low per diem fees for trainers (traveling expense and working fees), but only for APPEAL. Cost Effectiveness APPEAL – With existing budget MWLA exceeded targets APPEAL/STAGE – Using Blackboard software for communicating online, cost of training were significantly reduced. According to Mercy Corps, traditional training would cost 26,000 USD and online training cost was only 500 USD for beverages and snacks. Complying with the Branding and Marking Policies of USAID MWLA is compliant with the Branding and Marking policies of USAID. Additional Comments Gives credit to both projects, but slightly prefers cooperation with STAGE than APPEAL because interviewee thinks that STAGE project is a bit more important. - 27 - WSP – Women for Social Progress movement Name & Titles Date & Location Oyuntuya S, Executive Director Munkhnaran A, Manager Elberel D, Project officer Emily K, AYAD Volunteer 5th of February 2014, Ulaanbaatar Brief Description of Activities APPEAL – N/A STAGE – WSP has collaborated with TAF on the following: 1) To determine if there is need for trainings and handbooks on anti-corruption; 2) To develop and print handbooks; 3) To train 25 master trainers; 4) To train 600 high level government officials. Relevance for STAGE/APPEAL in terms of good Governance APPEAL – N/A STAGE – Trainings will enhance the understanding of government officials on various aspects of good governance and impart skills to implement laws relating to conflict of interest, transparency, and income and asset declaration. Relevance for STAGE/APPEAL in terms of Objectives APPEAL – N/A STAGE – Promote good governance to reduce corruption and increase transparency Achievements/Outco mes APPEAL – N/A STAGE – The following activities have been achieved: 1) Two types of handbooks were developed with the help of IAAC. 1000 copies of each book were printed. IAAC is very thankful for these handouts for trainings. Therefore, IAAC asked UN to help them to print another 1000 copies. 2) Trainers of the training were chosen by IAAC. So far 31 trainers were trained; 19 from ministries, 12 from NGOs and management academy and other training organizations. 31 trainers conducted 10 trainings and over 250 government officials were trained on anti-corruption legislation. The STAGE Project impacted: 1) IAAC now has its own handbooks on anti-corruption laws; 2) the capacity of IAAC trainings has improved; 3) IAAC has become more open to public and IAAC’s training methodology content have improved greatly; 4) as a direct consequence of these trainings, IAAC has changed its approach and plan of activities to belter address good governance and anticorruption curriculum; 4) government officials’ attitude towards NGOs has improved; 5) more NGOs are now participating in anti-corruption activities Reason for Failing to Perform (if any) APPEAL – N/A STAGE – N/A Management/ Communication Structure APPEAL – N/A STAGE – Daily direct communication (via email and phone) Management/ Communication Issues (if any) WSP needs to get approval from TAF on every stage of the project. For example, WSP needed to get approval from TAF on the handbook cover picture. It slows down the planned activities. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID APPEAL/STAGE – WSP is aware that USAID is funding this project Additional Comments WSP is very thankful to USAID for STAGE project. - 28 - IAAC Name & Titles Date & Location Mrs. Bat-Otgon B., Commissioner and Head of Prevention and Public Awareness Department 6th of February 2014, Ulaanbaatar Brief Description of Activities APPEAL – Mercy Corps and IAAC have signed the MoU in Dec 2012 to collaborate on three distinct initiatives including: 1) creating Anti-Corruption Action Plans; 2) developing and launching a new website; and 3) preparing for nation-wide consultations on the draft National Anti-Corruption Strategy. STAGE – IAAC have collaborated with the TAF on the following: 1) conducting a series of consultative meetings with civil society, professional associations, private entities, media, the judiciary, prosecutors’ office, national government, and local government agencies in Ulaanbaatar and four regions (Khuvsgul, Umnugovi, Khovd, and Sukhbaatar aimags); 2) procuring services from international experts (critique strategy); and 3) contributing to discussions in the working group as and when required. Relevance for STAGE/APPEAL in terms of Good Governance APPEAL – Anti-corruption Action Plans define local authorities’ commitment to transparency and accountability. Since the launch of the new youth website, the information of local governance is more transparent for youth and the broader public as well. STAGE – The trainings are to help inform government officials on various aspects of good governance and impart skills to implement laws relating to conflict of interest, transparency, and income and asset declaration. Relevance for STAGE/APPEAL in terms of Objectives APPEAL – IAAC has coordinated with APPEAL for conducting regional consultation meetings regarding anti-corruption awareness as well as reaching out to youth through advertising and anti-corruption activities. STAGE – The assessment of anti-corruption action plans have improved the development, monitoring, and implementation of the anti-corruption activities throughout the state organizations. Achievements/ Outcomes APPEAL/STAGE – IAAC together with the TAF, Mercy Corps’ APPEAL project team, and MWLA formulated a plan of action to hold public consultations that provide feedback on the draft National Anti-Corruption Strategy. The draft National Anti-Corruption Strategy has been submitted for final approval to the Parliament by the President’s Office in December 2013. APPEAL – Under the scope of the project to develop and implement Anti-Corruption Action Plans, aimag and soum administrations have created 30 new ACAPs. According to the most recent IAAC report, 125 ACAPs now exist across Mongolia. As a result of the MOU between APPEAL and the IAAC, there are now 75 ACAPs in 21 provinces and 54 soums. IAAC launched its youth-focused website (www.xamtdaa.mn) to promote youth awareness about corruption prevention with support from APPEAL. Since the launch of the website, IAAC joined forces with other national NGOs to create an advocacy plan called “Together” (xamtdaa). IAAC and APPEAL have produced a comic book on COI. STAGE – The Voters Education Center and the IAAC developed Controlling Corruption: Handbook for Parliamentarians. Specifically, the Handbook is an anthology of strategies and information that will strengthen and deepen the parliamentary engagement with the COI Law, Freedom of Information Act, Income and Asset Declaration Law, and so forth in compliance with the UNCAC. The WSP and the IAAC developed a training package that included training methodologies and a handbook for master trainers. The ToT was held in November for 25 master trainers. IAAC, with the TAF, has completed the assessment of anti-corruption action plans of 20 government agencies in 2012 and 105 additional agencies in 2013. Reason for Failing to Perform (if any) N/A Management/ Communication Structure APPEAL – Since IAAC does not have local branches in aimags, Mercy Corps has supported IAAC’s local visits. Despite the poor management from the previous Mercy Corps’ officials, IAAC has better relations with Jennifer Bjelman and Ms. Urana now. STAGE – IAAC has a positive, collaborative relationship with the TAF and there were not any issues regarding management and communication. Management/ Communication Issues (if any) APPEAL – There were a couple of issues with Mercy Corps. First, Mercy Corps has not been efficient with information exchange. When Mercy Corps has planned something with IAAC, they do not follow the plan, failing to contact or do follow ups. Ms. Bat￾Otgon claims that the management of Mercy Corps is poor as she has not even been informed, for 3-4 months, about the new appointment of the new CoP, Ms. Urana. The changes of the MC management team have happened at least 3 times (Munkhzaya, interim CoP, and Urana). However, According to Ms. Bat-Otgon the relationship between IAAC and MC is getting better now. Second, the comic books were printed without being reviewed by the IAAC officials and with many vocabulary errors. According to IAAC, this was very unprofessional as the book indicated that it has been reviewed by an IAAC official. That said, Ms. Urana took the responsibility of re-printing these comic book. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID IAAC is aware of that USAID’s fundingof this project. Additional Comments N/A - 29 - Ulaanbaatar City Municipality Name & Titles Date & Location Unench Ganbold, Head of Public Admin Dept. and other staff Feb 6, 2014; 1100 Governors office in UB “City Hall” Brief Description of Activities As a pilot project, Ulaanbaatar City Municipality was recently included in the STAGE project and has since worked together with IAAC and TAF on implementing anti-corruption action plans (ACAPs ) in the municipality’s four agencies: 1) City Property Office (that issues land ownership permit); 2) General Planning Office (that issues construction permit); 3) Office for Education; and 4) Office for Health. It should be noted, this is still at an early phase of launch at the time of this evaluation. Relevance for STAGE/APPEAL in terms of Good Governance By directly involving local administration of the capital city in the implementation of anti-corruption activities, STAGE is contributing to the development of more accountable and transparent capital city government. Relevance for STAGE/APPEAL in terms of Objectives Relevant for the first objective of STAGE: Anti-corruption activities Achievements/Outcomes Training of officials that are working in local priority public organizations on AC plans. Project named “Glass budget” was created in coordination with TAF. The aim of the project is to help and encourage local public entities to make their budgets more transparent. For example, 125 information boards, containing schools’ budget information, were published on the walls of the city schools. Also, there have been public awareness advertisements on AC. The Education and General Planning offices separately organized contests on civil servants’ knowledge on anti￾corruption. Reason for Failing to Perform (if any) N/A Management/Communication Structure Direct communication and cooperation with IAAC Indirect communication with TAF (STAGE) through IAAC Management/Communication Issues (if any) They are of opinion that indirect communication with TAF (through IAAC) is fine. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID? Interviewees didn’t know that USAID is financing the project activities. Later they stated that they will put the USAID logo on the school boards. Additional Comments N/A - 30 - Ministry of Justice Name & Titles Date & Location Mrs. Selenge E., Specialist of Legal Reform Policy Department of Ministry of Justice 6th February 2014, Ulaanbaatar Brief Description of Activities Ministry of Justice has been working primarily with the TAF regarding the draft of the new General Administrative Law and the Administrative Procedure Law, fostering their adoption by the Parliament of Mongolia. In order to draft a new GAL, MoJ has conducted the following activities: 1) established a professional working group for drafting the law; 2) translated the relevant international laws into Mongolian; and 3) conduct various meetings, discussions, and seminars to discuss the draft laws that include judges, legal specialists, state organizations, and local citizens. An additional and separate activity of MoJ was to conduct the Administrative Processes, Procedures, and Performance Study. Relevance for STAGE/APPEAL in terms of Good Governance The new General Administrative Law would be the first law to regulate the activities of the administrative organizations, to unite the regulations of the activities of law enforcement agencies and to control the relations between these state organizations. Therefore, it is the most important legal act for strengthening good governance. Relevance for STAGE/APPEAL in terms of Objectives Transparent administrative activities and international democracy experiences would be adopted in Mongolia. Achievements/Outcomes With support from STAGE and USAID funding, the translation of the draft law on Administrative Procedures from Mongolian into English was conducted. The administrative laws and the administrative procedure laws of four countries-Latvia, The Netherlands, Japan, and Spain have been translated into Mongolian. The TAF on behalf of the MoJ has organized two consultative meetings. The first meeting was held among 16 District Administrative and Appellate Capital City Court judges and the second meeting was organized among five Supreme Court Justices of the Administrative Chamber and their five assistants. MoJ, Management Academy, and the TAF have completed the Administrative Processes, Procedures, and Performance Study. Based on the review of eight different government agencies. The study report will further support the Parliamentary Committee to finalize the General Administrative Law (GAL) and the Administrative Procedure Law (APL). The Working Group on GAL has completed a Draft of General Administrative Law and Minister of Justice Mr. Temuujin has delivered the draft law to the Cabinet Secretariat on 9 January 2014 for introducing it in spring session of the Parliament. The working group is in the drafting process for APL and about to finish. Reason for Failing to Perform (if any) N/A Management/ Communication Structure Legal Reform Policy Department of Ministry of Justice has no problem with current TAF management and communication. Management/ Communication Issues (if any) N/A Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID MoJ is aware of that USAID is funding this project. However, MoJ prefers not to use the USAID logo on the official documents relating to the draft laws as to avoid any potential conflict of interest. Additional Comments N/A - 31 - Business Council of Mongolia Name & Titles Date & Location Mr. Ser-Od Ichinkhorloo, Vice director February 7th 2014, Ulaanbaatar Brief Description of Activities In collaboration with TAF (STAGE), Business Council of Mongolia conducted 2 anti-corruption seminars for representatives of multi-national or international companies in Mongolia. They also organized lectures on anti-corruption at the Mongolian National University (MUIS) and the Institute of Finance and Economics. The Business Council of Mongolia provided STAGE with its business contacts for the STOPP survey Relevance for STAGE/APPEAL in terms of Good Governance Indirect relevance for GG, because they are involved in training of businesses on anti-corruption activities. Therefore, we could connect BCM’s efforts only to the participatory aspect of good governance Relevance for STAGE/APPEAL In terms of Objectives Train the private sector on AC laws and policies as well as the benefits of proactive private sector engagement in AC Achievements/Outcomes BCM developed a proposal to train representatives from multi-national companies (April Jun 2013), held two seminars and two lectures, but has yet to organize two events on anti-corruption that were planned to occur in January Reason for Failing to Perform (if any) BCM explanation for failing to organize two events on anti-corruption involved the Mongolian New Year (end of January). They felt that gathering people for two events wouldn’t be possible; however, even though it is late, they said that they will organize two anti-corruption events in upcoming period. Management/Communication Structure BCM has direct communication and cooperation with TAF. There is no agreed budget on activities, rather BCM is reimbursed for organizing activities and paid afterwards. Management/Communication Issues (if any) Mr. Ser-Od complained that STAGE’s CoP is pressuring them to write the report both in Mongolian and English. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID The interviewee was confused about USAID’s role in the project. Also, the evaluation team wasn’t provided with any documentation in order to determine if there was compliance with the Branding and Marking policies of USAID. Additional Comments Evaluators’ observation (Milos): The interviewee has a general lack of interest in the project because BCM does not play big role in it. BCM should not have additional responsibilities. Simply put, they are satisfied with the current situation and level of involvement in the project. - 32 - Judicial General Council Name & Titles Date & Location Mrs. Munkhzul N., Head of Judicial Qualifications Committee Mr. Battulga P., Specialist of Judicial Qualifications Committee 7th of February 2014, Ulaanbaatar Brief Description of Activities The Judicial General Council has conducted the following activities with technical and financial support from USAID and the STAGE project: 1) recruited an international consultant to develop a performance evaluation system (methodology, criteria, evaluation process) for judges; 2) developed a software program for judges’ selection; 3) developed the Judicial Performance Evaluation study for judges’ selection regarding ethics, management and leadership skills, and the talent of nominees; and 4) translated the handout “Transparent Courts and Implementation Evaluation of Court Authority” Relevance for STAGE/APPEAL in terms of Good Governance The Judicial General Council has completed the Judicial Performance Evaluation study with the support of STAGE to evaluate judges’ performance. Since the Judiciary plays a vital role in reducing corruption, the exercise has reinforced the importance of performance evaluation of the key profession in the Judiciary. Moreover, the inclusive nature of the study has already addressed the resistance against the implementation of the evaluation plan. In addition, the new Law on Courts of Mongolia currently requires regular reviewing of a judge’s performance and authorizes both the Council and the Qualifications Committee to do so. Relevance for STAGE/APPEAL in terms of Objectives STAGE – Strengthen the capacity of the judiciary to adjudicate corruption cases and prevent judicial corruption. Achievements/Outcomes Within the scope of the STAGE project, TAF and the Judicial General Council recruited Professor Pim Albers from the Netherland to act as an international consultant and develop a performance evaluation system for judges. The consultant held discussions with many of the concerned authorities from the judiciary including the Council, Supreme Court, and Courts of Appeal on September 2013 and subsequently submitted a report. The Judicial Performance Evaluation study has been successfully developed by a professional researcher. The software program for judges’ selection is completed and the translated hand-outs are in use. Reason for Failing to Perform (if any) N/A Management/ Communication Structure The Judicial General Council has regular, unproblematic communication with TAF, and the activities have been implemented on time. Management /Communication Issues (if any) There was an issue regarding the payment of the professional fee for the researcher who completed the Judicial Performance Evaluation study. According to the researcher’s job performance agreement, the researcher shall be paid 9 million tugrugs for his/her job; however, the TAF had questions about the researcher’s total income, other jobs and salary. These queries and “investigations” caused uncomfortable and unprofessional situations for the researcher and Judicial General Council. Finally, the TAF has paid 8 million tugrugs to the researcher instead of 9 million. Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID This organization knows that USAID finances the STAGE program and USAID logo has been used in their Power Point presentation and the translated handouts. Additional Comments N/A - 33 - ZF- Zorig Foundation Name & Titles Date & Location Badruun G, Executive director Amar Baatartsogt, Project development manager 7th of February 2014, Ulaanbaatar Brief Description of Activities APPEAL – N/A STAGE – Zorig Foundation has collaborated with TAF to build awareness on the conflict of interest law and anti￾corruption law using social media among youth Relevance for STAGE/APPEAL in terms of Good Governance APPEAL – N/A STAGE – Supports civil society advocacy for the implementation of the COI Law, Amended Criminal Code, and the passage of draft amendments to the Criminal Procedure Code. Relevance for STAGE/APPEAL in terms of Objectives APPEAL – N/A STAGE – Promote good governance to reduce corruption and increase transparency Achievements/Outcomes APPEAL – N/A STAGE – The following activities have been achieved: 1) built awareness among youth on anti-corruption and conflict of interest themes using Facebook page; 2) organized a workshop on conflict of interest and corruption issues among university students with the help of IAAC; 3) organized an online competition on conflict of interest among youth Reason for Failing to Perform (if any) APPEAL – N/A STAGE – N/A Management/ Communication Structure APPEAL – N/A STAGE – Daily, direct communication (email and phone) Management/ Communication Issues (if any) N/A Cost effectiveness? N/A Complying with the Branding and Marking Policies of USAID APPEAL/STAGE – Zorig Foundation is aware of that USAID is funding this project Additional Comments N/A - 34 - Sant Maral Foundation – Ulaanbaatar Name & Titles Date & Location Sumati Luvsandendev , Director of SMF 3rd of March 2014, Ulaanbaatar Brief Description of Activities In coordination with TAF (STAGE), SMF has designed, executed, and (to some extent) disseminated two longitudinal surveys/research on corruption perception, grand corruption as well as on perceptions and knowledge regarding administrative practices and corruption in business sector: SPEAK (Survey on Perceptions and Knowledge of Corruption) and STOPP (Study of Private Perceptions of Corruption). Both SPEAK and STOPP have been repeated three times and the fourth and final round of surveys is scheduled for mid-March 2014. Relevance for STAGE/APPEAL in terms of Good Governance STAGE/APPEAL – SMF has direct impact on knowledge of corruption. The general public, as well as public and private sector, now have a clearer picture of how corruption is conducted and the major institutions perceived as the most corrupt. This information is useful in relation to counter measures in anti￾corruption strategies and action plans. Relevance for the STAGE/APPEAL in terms of Objectives STAGE – SMF public surveys research on grand corruption helped STAGE as well as IAAC in seeing which public agencies and institutions are perceived as the most corrupt and direct their course of action toward these institutions. The clear example for such directed activity is the recent inclusion of the Municipality of Ulaanbaatar into the STAGE project. Namely, the interviewee stated that the SPEAK survey found out that the public perceives land utilization as a major area where corruption happens, the most corrupted public agencies are those that decide what will be done with the public land on a national, regional, and local level. From this and previous interviews and upon seeing the results of the SPEAK and STOPP surveys, the evaluator concluded that STAGE strived to include into the project Ulaanbaatar municipal agencies, among which are City Property Office (that issues land ownership permits) and the General Planning Office (that issues construction permits) in order to tackle land utilization corruption. The interviewee stated that STOPP and SPEAK surveys had direct impact on IAAC policy making and influenced on IAAC’s direction of investigations. Achievements/Outcomes Due to easy access (website of SMF and TAF) surveys are publicly available for free. This has resulted in a change in knowledge and awareness of corruption among general public as well as the public and private sector. These surveys have a possible influence on policy making process, especially indirectly as they are widely disseminated through media (newspapers and television). Due to the fact that both surveys are longitudinal (same method of data collection on with same aim is used over period of time), it is possible to see changes in perceptions of corruption. This could have a positive impact on other researches and re-direct the policy process. Reason for Failing to Perform (if any) SMF has experienced pressure from some political actors related to their surveys, however there were no problems caused by STAGE project. There is no firm evidence that SPEAK and STOPP surveys, as well as STAGE project, are widely known in public, despite the interviewee stating that findings from these surveys are widely known. Management/Communication Structure SMF had only direct communication and cooperation with TAF in execution of its duties. However, SMF was little included in dissemination of survey findings. SMF didn’t have any direct communication or cooperation with other STAGE or APPEAL partners Management/Communication Issues (if any) N/A Cost Effectiveness N/A Complying with the Branding and Marking Policies of USAID The interviewee was fully aware that USAID is funding STAGE project Additional Comments The interviewee expressed concerns that after termination of the project he doesn’t know who will continue to fund these two surveys. However, the evaluator is of opinion that SMF will succeed in conducting corruption surveys after the termination of the STAGE project, because: 1) SMF has collaborated with TAF and USAID since 2006 on corruption public polls, long before STAGE project started; 2) SMF is one of only a few NGOs that are doing public polls, and the only one that is dealing with public polls on corruption which puts SMF if unique and favorable position; 3) SMF has “know how” of designing methodology for complex researches and it knows how to execute it; 4) The director of SMF was previously representative of Transparency International in Mongolia. All these reasons lead us to the conclusion that SMF will find new donors, or donors will find them (due to monopoly position on the market), as well as means to continue with corruption perception surveys/researches. - 35 - ANNEX E: FIELD TRIP ITINERARY Aimag and Soums – Site Visits # Aimag Soum 1 Uvs Tarialan soum, Uvs 2 Khenti Kherlen soum KE 3 Arkhangai Tsenkher soum AR 4 Umnugobi Togt-tsetsii soum, UG 5 Uvurkangai Arvai kheer soum, UH 6 Uvurkangai Tugrug soum, UH 7 Gobi-Altai Sharga soum, GA We followed the logic of selecting one Aimag from each region, with input from USAID by adding Uvs and Gobi-Altai Aimags in lieu of Khuvsgul and Bayan-Ulgii. We also considered the geographical position so that we could arrange the most efficient visitation itinerary. 1. Uvs – the selection was based upon USAID request. It has been several years since USAID has visited this Aimag. It will therefore be an excellent choice for the evaluation purposes. The Team will arrange transportation. 2. Khentii – the reason for choosing Khentii Aimag from eastern region is that it’s closer to Ulaanbaatar and easy to travel by car. 3. Arkhangai - was substituted for Dornogobi because of the imposition of quarantine for “hoof-and￾mouth” disease in cattle herds there. 4. Umnugobi – the reason for choosing Umnugobi from Gobi region is that Umnugobi Aimag is rich in mineral deposits. So much are happening in Umnugobi Aimag in terms of development as a result of the mining boom. A wide range of community development programs are being implemented in order to stimulate local economic development. It’s important to see how the governance projects are being implemented. 5. Uvurkhangai – the reason for choosing Uvurkhangai Aimag from Uvurkhangai region is that Uvurkhangai Aimag is located on the way to Ulaanbaatar from Umnugobi Aimag. 6. Gobi-Altai – the selection was based upon USAID request. Gobi-Altai Aimag is a regional office. It has been several year since USAID has visited this Aimag. It will therefore be an excellent choice for the evaluation purposes. The Team will arrange transportation. We selected one Aimag from each region, with input from USAID by adding Uvs and Gobi-Altai Aimags in lieu of Khuvsgul and Bayan-Ulgii. In addition to the 6 Aimags, 7 Soums were included in the SBU survey in order to ascertain what governmental functions were performed at the sub-aimag level. We also considered the geographical position so that we could arrange the most efficient visitation itinerary. Due to travel distances and time constraints the evaluators divided into two sub-teams to conduct evaluation of program activities in the Aimags and Soums. After planning the field visit itinerary, it was necessary to substitute Arkhangai Aimag for Dornogobi Aimag due to travel restrictions imposed by quarantine for “hoof-and-mouth” disease among cattle herds. The evaluators divided into two sub-teams. Each sub-team conducted the evaluation in 3 Aimags. As schedule permitted, selected Soums were included during visits to respective Aimags. - 36 - ANNEX F: STATEMENT OF WORK - 37 - - 38 - - 39 - - 40 - - 41 - - 42 - U.S. Agency for International Development 1300 Pennsylvania Ave., NW Washington, D.C. 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov