DISCLAIMER This work plan is made possible by the support of the American people through the United States Agency 
for International Development (USAID). The contents are the sole responsibility of the authors and do not 
necessarily reflect the views of USAID or the United States Government.
PHOTO CREDIT: USAID/NORTH MACEDONIA CEP FACEBOOK
SUMMATIVE PERFORMANCE EVALUATION
OF CIVIC ENGAGEMENT PROJECT IN 
NORTH MACEDONIA
Evaluation Report
This publication was produced at the request of the United States Agency for International Development
(USAID). It was prepared independently by IMPAQ International, LLC, under IDIQ Contract No. AID￾167-I-17-00004, Task Order No. 72016521F00002.
December 2021
Implemented by:
IMPAQ International
1325 G Street, NW
Suite 900
Washington, DC 20005
ABSTRACT: This Civic Engineering Project (CEP) is a USAID-funded activity in North Macedonia that
began in 2016 prior to the formation of the new government in 2017 and the pursuit of policies of 
European Union (EU) accession and North Atlantic Treaty Organization (NATO) membership. CEP is 
one of several activities contributing to USAID’s 2020-2025 Country Development Cooperation 
Strategy (CDCS) Development Goal, “USAID Partnerships Promote a Prosperous, Well-Governed, and 
Resilient North Macedonia Committed to Euro-Atlantic Integration”. The project is implemented by 
prime implementer East-West Management Institute (EWMI) in close collaboration with local 
consortium partners: 1) Association for Democratic Initiatives (ADI); 2) Foundation for Internet and 
Society “Metamorphosis” (Metamorphosis); 3) Foundation Open Society – Macedonia (FOSM); and 4) 
National Youth Council of Macedonia (NYCM). 
In 2021, IMPAQ International (IMPAQ) received a contract from USAID/North Macedonia to conduct a 
summative performance evaluation of CEP. IMPAQ, supported in country by Strategic Development 
Consulting, North Macedonia (SDC) and Women Influencing Health, Education and Rule of Law (WI￾HER), conducted a use-focused evaluation that integrates quantitative and qualitative methods to enrich 
USAID’s understanding of the effectiveness of supported civil society organizations (CSOs) and their 
government transparency, anti-corruption, and rule of law interventions towards EU accession. As a 
result, IMPAQ provided to USAID/North Macedonia actionable recommendations to improve the 
efficiency and sustainability of its transparency, anti-corruption, and rule of law programming moving 
forward.

CONTENTS
ABBREVIATIONS AND ACRONYMS V
ACKNOWLEDGMENTS VI
EXECUTIVE SUMMARY I
INTRODUCTION 1
CIVIC ENGAGEMENT PROJECT BACKGROUND 1
EVALUATION PURPOSE 2
EVALUATION METHODOLOGY 3
DATA ANALYSIS 4
KNOWN LIMITATION AND MITIGATION STRATEGIES 4
FINDINGS AND CONCLUSIONS 6
CONTEXT FINDINGS 6
CONTEXT CONCLUSIONS 11
EQ 1 FINDINGS 11
EQ 1 CONCLUSIONS 26
EQ 2 FINDINGS 28
EQ 2 CONCLUSIONS 36
LESSONS LEARNED AND RECOMMENDATIONS 37
LESSONS LEARNED 37
RECOMMENDATIONS FOR USAID FOR FUTURE PROGRAMMING 37
ANNEX A: EVALUATION STATEMENT OF WORK 40
ANNEX B: EVALUATION METHODOLOGY 45
ANNEX C: DATA COLLECTION INSTRUMENTS 51
ANNEX D: SOURCES OF INFORMATION 70
ANNEX E: KEY INFORMANTS AND FOCUS GROUP PARTICIPANTS
74
ANNEX F: ANALYSIS OF CEP KEY PERFORMANCE INDICATORS 77
ABBREVIATIONS AND ACRONYMS
ADI Association for Democratic Initiatives, North Macedonia
CAFT Coalition of Civic Associations All for Fair Trials
CCC Association of Citizen’s Center for Civil Communications, North Macedonia
CDCS Country Development Cooperation Strategy
CEP Civic Engagement Project
CO Contracting Officer
COR Contracting Officer’s Representative
COVID-19 Coronavirus disease of 2019
CSO Civil Society Organization
DPM Deputy Prime Minister
EQ Evaluation Question
EU European Union
EWMI East-West Management Institute, Inc.
FGD Focus Group Discussion 
FINANCE THINK FINANCE THINK Institute for Economic Research and Policy
FOSM Foundation Open Society – North Macedonia
GESI Gender Equality and Social Inclusion
GONM Government of North Macedonia
IR Intermediate Result
J2SR Journey to Self-Reliance
KII Key Informant Interview
LCDF Local Community Development Foundation
LYPYP Law on Youth Participation and Youth Policies 
MEL Monitoring, Evaluation, and Learning
MELP Monitoring, Evaluation, and Learning Plan
MISA Ministry of Information, Society, and Administration
NATO North Atlantic Treaty Organization
NYCM National Youth Council of North Macedonia
OM Outcome Mapping
PG Partnership Grant
PMU Project Management Unit
SDC Strategic Development Consulting, North Macedonia
SSG Strategic Support Grant
TI Transparency International, North Macedonia
U-FE Utilization-Focused Evaluation
USAID United States Agency for International Development
VCA Vulnerability to Corruption Assessment
WI-HER Women Influencing Health, Education, and Rule of Law
WHO World Health Organization
ACKNOWLEDGMENTS
The evaluation team wants to express their sincere gratitude to those who provided input and advice 
throughout the USAID/North Macedonia CEP evaluation.
We have sincere respect for the representatives of the North Macedonia government for taking the 
time to speak to the evaluation team and participating in the online survey. We thank all central and 
municipal government officials who provided thoughtful insights.
We are grateful to USAID/North Macedonia. We also acknowledge the cooperation of all former and 
current staff of the implementing partner East-West Management Institute and CEP partner and grantee 
civil society organizations that made themselves available for interviews, focus group discussions,
workshops, and the provision of project information and data.
The evaluation team offers thanks to the logistics coordinator and translator, Gorjan Boshkovski. We 
are also grateful for the home office support of IMPAQ International throughout all stages of the 
evaluation.
i | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
EXECUTIVE SUMMARY
The United States Agency for International Development (USAID) commissioned IMPAQ International, 
LLC (IMPAQ) to conduct a summative performance evaluation of the Civic Engagement Project (CEP) in 
North Macedonia. The $9.4 million project commenced in August 2016, and will conclude in August
2022, reflecting a one-year no-cost extension. CEP is implemented by lead partner East-West 
Management Institute (EWMI) in coordination with four civil society organization (CSO) partners: 
Association for Democratic Initiatives (ADI), the Foundation for Internet and Society “Metamorphosis” 
(Metamorphosis), the National Youth Council of Macedonia (NYCM), and the Foundation Open Society 
Macedonia (FOSM) along with a team of five CSO Grantees.
The evaluation team assessed the extent to which and how CEP-supported Government of North 
Macedonia (GONM) agencies and CSOs helped to advance North Macedonia’s European Union (EU) 
accession in regard to improving transparency, anti-corruption, and rule of law, and to understand the 
prospects for government sustainability. The findings from this evaluation will help inform USAID/North 
Macedonia, EWMI, and other stakeholders involved in strengthening governance practices to design 
future activities in North Macedonia. 
FINDINGS AND CONCLUSIONS
CEP faced several external and extenuating challenges in 2020: 1) a major restructuring of its 
programming at the mid-way point (a pivot) from CSO capacity building to government technical 
assistance initially for an implementation period of 18 months;, 2) the emergence of the COVID-19 
pandemic from March 2020 onward; 3) Parliamentary elections in July 2020 with changes and additions 
to public institutions and agencies; 4) a 4-month ‘‘uncertainty” period in 2021 for the approval of a one￾year no-cost extension (granted 2 months before its initial end-of-project date); and 5) local elections in 
October 2021. These challenges forced CEP to continually adapt its programming and delivery. Changing 
its programming and shifting from a CSO and citizen focus to one that also included a focus on high-level 
central government and municipal assistance, dedicated solely to transparency, anti-corruption, and rule 
of law interventions, also required changes to its entry points and CEP’s monitoring plan to measure its
progress and success with new indicators and targets. 
Some of CEP’s CSO Grantees were most affected by the programming changes, as it required them to
close their grants before the initial contractual end dates. The government was positively affected, due 
to receiving a wide range of technical assistance. This evaluation takes into account the challenges and
achievements to date, and makes recommendations for the remaining implementation period to August 
2022, and for future similar USAID activities. 
The objective of technical assistance in support of EU accession through more transparent 
and accountable government institutions was highly ambitious. While CSO Grantees and 
citizens felt the burden of the confusion and fragmentation of CEP’s pivot and new strategy, including 
the loss of funding, the government benefited through new technical assistance from CEP’s 
Evaluation Question 1: To what extent and how have CEP-supported CSOs helped to advance 
North Macedonia’s EU accession in areas related to rule of law, transparency, and anti-corruption? 
What advanced or impeded this approach to supporting the GONM in these areas? 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | ii
programming shift. The focus on the ideals of transparency, anti-corruption, and rule of law, through 
technical support to municipalities and public institutions, increased CEP’s visibility with the GONM at
the central level. All GONM evaluation informants at the central level were conversant with CEP’s 
strategy. The informants maintained that they knew the implicit project goal of advancing EU accession, 
yet also acknowledged the explicit goal of countrywide awareness and interventions advancing good 
governance through transparency, anti-corruption, and rule of law. 
CEP tailored its technical assistance to government requests and priorities, resulting in 
high usage rates, with much of the advice and support implemented immediately. The 
support was provided in a way for the government to adopt, adapt, modify, and establish cycles of 
operational activities. The support was provided through short to medium duration tasks and projects 
that have achieved what they intended to do. However, the next stage is to follow up, where relevant,
through locally led or donor funded programs or through CSO activities on the implementation, 
monitoring, or provision of evidence-based statistics or reports to inform evidence-based decision 
making and policy development. 
CEP has established a foundation for institutional change in the processes and procedures 
for government transparency, anti-corruption, and rule of law, particularly in government 
institutions opening their data to the public. Of more importance, CEP trained the government 
on making the data usable for different audiences. At the same time, CEP created demand from citizens 
for public municipal data, as well as good governance, such as fair trials and engagement with citizens. 
While significant progress was made at the central and municipal levels, making sustainability highly likely 
due to the processes and technology in place, “there is still a long way to go.”1
CCC has effectively created momentum, providing tools and intervening in legislation to 
encourage increased transparency in public procurement. However, more work remains.
The findings demonstrate broad consensus that CCC has been instrumental in creating the necessary 
conditions for improved transparency in public procurement. However, the improvements achieved to 
date have not sufficiently translated into a practice of transparent tenders. CCC has provided tools and 
mobilized stakeholder support to use these tools, including the Index of Active Transparency and public 
procurement monitoring methodologies, to galvanize public pressure that leads public institutions to 
enact reforms. CCC’s work has led to increased awareness of transparency issues and some cases of 
changed practices within institutions. 
CCC has further positioned itself as a respected CSO with unique expertise, so much so 
that public institutions open their doors, extend invitations, and invite recommendations 
to legislative and regulatory reforms. Despite significant momentum, and increased objective 
scores observed through the Active Transparency Index, the end-user of public procurements – the 
business sector – has not seen a significant change in tender procedures. The process of change, through 
1 Interview MN02, CSO, October 2021.
Evaluation Question 2: To what extent and how has the sub-grantee, Center for Civil 
Communications (CCC) increased transparency in public procurement? What lessons learned from 
CCC’s work could be valuable and applicable to other efforts to advance government transparency, 
anti-corruption, and rule of law? 
iii | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
public procurement monitoring, is lengthy and slow. Informants also perceive that public officials’ 
impunity and the lack of penalties for public spending corruption cases that do make it to court, have
contributed to the slow pace of changes. Ongoing monitoring of public procurements is required, and it 
may need to remain with civil society for the time being, as North Macedonia’s institutions are not yet 
ready to adopt these accountability protocols. However, CCC has provided a host of tools and 
methodologies for a broader coalition of CSOs and stakeholders to engage in this mandate, effectively 
building the infrastructure for transparency monitoring. 
Public spending for monitoring risks remains ineffective without robust adherence to the 
rule of law and free media in North Macedonia. The findings noted that enforcement of rule of 
law regarding public procurement violations is inconsistent and is rendered potentially ineffective in 
flagging offenses. Furthermore, when other public officials observe that corruption cases do not result in 
penalties, there is little incentive to adhere to transparency and anti-corruption practices. Lack of law 
enforcement is a significant deterrence to effective transparency and anti-corruption work. In addition, 
the findings indicate important concerns regarding the freedom of the press in North Macedonia. Media 
is a key avenue to “voice” and flagging potential corruption issues in public spending. Therefore, freedom 
of press to investigate and report issues without interference from business or political interests is 
imperative. Stakeholders emphasized a diminished role of investigative journalism practice and an 
increased presence of political or business influences in mainstream media. If freedom of traditional 
media outlets continues to diminish, CSOs in their roles as watchdogs will require additional capacity 
and resources for mass communication campaigns to raise public awareness of corruption cases. 
RECOMMENDATIONS
Recommendation 1: Strengthen public and independent institutions with roles related to 
transparency, anti-corruption, and rule of law, and state explicitly the roles and rules of engagement for 
each stakeholder. Specific examples of government and independent institutions or entities for 
continued, new and/or reinvigorated support include: Academy for Judges and Public Prosecutors; 
Agency for Protection of the Right to Free Access to Public Information; Anti-Corruption Platform; 
Cabinet of the Deputy Prime Minister for Anti-Corruption, Sustainable Development and Human 
Resources; Finance Intelligence Directorate; Local Courts; Ministry of Information Society and 
Administration; Ministry of Justice; State Audit Office; Municipal Councils; State Commission for 
Prevention of Corruption; and State Judicial Council. 
Recommendation 2: Strengthen the focus on outcome and impact indicators, targets, and results and 
be clear about expected changes in consultation with the government and the targeted public or 
independent institutions. Ensure the alignment of the activity/project objective indicator to the objective 
to ensure measurability, monitoring, and reporting. Ensure the selection of appropriate indicators to 
measure progress and success at output and outcome levels, including the measure of change. Activities 
should be clearly in line with national policies and plans. For example, include qualitative indicators on 
knowledge, attitudes, and perceptions across targeted groups, particularly citizens’ perceptions of 
government accountability, to measure change in awareness and changes in behavior against the 
institutions’ expectations, goals and strategies. 
Recommendation 3: Strategically select key interventions for systemic change, given the multiple 
layers within each area of transparency, anti-corruption, and rule of law. The Theory of Change should 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
iv
identify sectors and synergistic pathways toward measured and verified change, particularly rule of law 
interventions. 
Recommendation 4: Support long-term technical assistance (LTTA) with independent and neutral in￾house (in-person) international or national experts who are resilient to influence to play a role in 
facilitating and monitoring the implementation of USAID-funded institutional support. This could be a 
team of LTTA advisors in both government and independent institutions, particularly to facilitate the 
creation of independent compliance control mechanisms, such as USAID-funded independent CSOs or 
other mechanisms, to ensure impartiality. The country has systems in place but requires more 
concentrated efforts on implementation. For example, the Ministry of Information Society and 
Administration (MISA) could be supported with a LTTA advisor to ensure implementation of strategies, 
such as the introduction of the merit system in employment. Other important links in the corruption 
chain include courts and prosecutions through the Court Council and the Prosecution Council because 
they appoint judges and prosecutors. Therefore, an independent LTTA advisor could adopt the role of 
reviewing past performances and current procedural compliance of the Court Council and the 
Prosecution Council. An LTTA in the State Audit Office (SAO), for example, should provide assistance 
to enable it to become a fully legitimate, respected, and authoritative agency.
Recommendation 5: Support short-term technical assistance (STTA) that focuses on systemic 
processes that are cyclical or periodic, adaptable, replicable, and sustainable for CSOs or government 
products or services. For example, the process of conducting citizen surveys on their perception of 
government accountability – with processes that are step-by-step including data collection, data analysis, 
monitoring, reporting, actions for improvement and learning (whole cycle), as well as a schedule for 
repeated surveys. The in-person, local or international STTA advisors with direct knowledge about the 
country’s political and governance narrative, should provide guidance and mentoring for specific, 
targeted, government-identified tasks. Promote a culture of evaluation, in which the government 
conducts internal and external assessments of their work for continuous improvements.
Recommendation 6: Include more comprehensive and inclusive engagement of a wide range of 
stakeholders, particularly public institutions – directly or indirectly – on transparency, anti-corruption, 
and rule of law interventions. This engagement includes youth, particularly given that the government 
adopted the law on youth participation and youth policies (LYPYP) in January 2020, which obliges 
municipalities to involve youth in decision making processes. Strong support for the media, specifically 
investigative journalism, to monitor a range of areas, such as public procurement, should continue to 
enable the necessary public pressure that motivates institutions to change their practices, legislation, or 
regulations.
Recommendation 7: Strengthen youth and youth groups on transparency, anti-corruption, and rule 
of law issues, their leadership, and their communication skills. Identify change agents and monitor their 
capacity to conduct actions or engagements with relevant CSOs and local government. 
Recommendation 8: Continue support for open and transparent public procurement and 
contracting, in line with a recommendation from Transparency International’s 2020 Corruption 
Perceptions Index report, to assist in the identification of conflicts of interest and opportunities for fair 
pricing to prevent the diversion of public resources. 
v | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Recommendation 9: Re-focus the support of CSOs towards monitoring, rather than implementation 
of governance activities; i.e., to act as government watchdogs and monitors of transparency, anti￾corruption, and rule of law that complement the CSOs’ and government’s work. Civil society should 
continue to facilitate citizen awareness on transparency, anti-corruption, and rule of law, and to facilitate 
two-way linkages between citizens and government on local issues for monitoring, policy development, 
and decision making.
Recommendation 10: Consider supporting CEP follow-on activities to ensure the application of 
technical assistance. Follow-on activities could include monitoring recommendations of Vulnerability to 
Corruption Assessments; monitoring the use of the online Open Data platform, particularly for citizens; 
continued support for amendments to the law on use of data from the public sector (Open Data law); 
and monitoring the municipalities with integrity plans.
1 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
INTRODUCTION
CIVIC ENGAGEMENT PROJECT BACKGROUND
The Republic of North Macedonia has been a candidate for accession to the EU since 2005, but due to 
issues such as contestations from its neighbors, widespread corruption, and democratic governance 
backsliding, EU accession negotiations have stalled. The Government of North Macedonia (GONM) has 
achieved some notable good governance successes, thanks to a series of reforms ushered in by the new 
government formed in 2017, which prompted EU member states to grant permission for accession 
negotiations in 2020. However, more work is needed to gain accession to the EU. On March 27, 2020, 
North Macedonia became a member of NATO when all 29 national parliaments voted to ratify its 
membership in February 2019, after the country’s long-standing contributions to Euro-Atlantic security.
In November 2020, an EU-member country blocked the start of North Macedonia’s EU accession talks, 
which has still not been lifted despite international efforts from EU-member countries and the US.
CEP, a USAID-funded activity in North Macedonia, began in 2016 prior to formation of the new 
government in 2017 and the pursuit of policies of EU accession and NATO membership. CEP is one of 
several activities contributing to USAID’s 2020-2025 Country Development Cooperation Strategy 
(CDCS) Development Goal, “USAID Partnerships Promote a Prosperous, Well-Governed, and Resilient 
North Macedonia Committed to Euro-Atlantic Integration.”2 EWMI is the project’s prime implementer 
in close collaboration with local consortium partners: 1) Association for Democratic Initiatives (ADI); 
2) Foundation for Internet and Society “Metamorphosis” (Metamorphosis); 3) Foundation Open Society 
– Macedonia (FOSM); and 4) National Youth Council of Macedonia (NYCM). 
CEP has undergone two phases. In its first phase, at project inception in 2016, the program had two 
objectives: 
1. Strengthen constructive engagement by civil society organizations (CSOs) on policies and issues of 
public concern.
2. Increase and sustain youth engagement in public life in North Macedonia.3
After transition to a new government, the project entered its second phase when USAID/North 
Macedonia shifted its focus in response to findings from the IMPAQ-led Project Appraisal Document 
evaluation, priorities of the newly elected government in North Macedonia, and the launch of USAID’s 
Journey to Self-Reliance (J2SR) Framework. As of 2020, CEP pivoted away from civil society capacity 
building, and toward strengthening GONM institutions’ ability to execute reforms in three important 
areas of work toward their goals of EU accession and NATO membership: anti-corruption; 
transparency; and rule of law. As a result, 10 grants did not focus on these high-priority areas were 
closed before their initial contractual dates, while seven grants that did focus on these areas continued 
2 USAID/North Macedonia (2020). Country Development Cooperation Strategy: September 2020 - September 
2025, p. 1.
3 EWMI (2020, September 30). CEP, Year 5 Work Plan, p. 7.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
2
to be funded. In January 2020, USAID confirmed CEP’s shift in focus with its sixth modification to CEP’s 
contract, which also increased the contract’s budget from $5,662,300 to $7,678,924.
As a result of this shift, CEP revised its Theory of Change. CEP’s revised Theory of Change posits that: 
“IF North Macedonian public institutions, CSOs, CSO alliances, and citizens, including youth, are 
provided the resources, cooperation mechanisms, and technical assistance for coordinated and long-term 
joint actions in the areas of rule of law, transparency, and anti-corruption, THEN they will be more 
capable to contribute to North Macedonia’s accession and integration in EU and NATO.”2F
4
Thus, CEP’s new goal is to:
“Strengthen constructive engagement by civil society organizations (CSOs) on policies and issues targeted 
at anti-corruption, transparency, and rule of law, and sustain youth engagement in public life in North 
Macedonia.” 
CEP aims to achieve its purpose through focusing on four intermediate results (IR), as shown in the 
CEP’s revised Results Framework (Exhibit 1).3F
5
EXHIBIT 1. CEP RESULTS FRAMEWORK
EVALUATION PURPOSE
USAID/North Macedonia commissioned IMPAQ International, LLC (IMPAQ) to conduct a summative 
(final) performance evaluation of the USAID-funded CEP in North Macedonia. The evaluation provides 
to USAID/North Macedonia a full, evidence-based, and independent assessment of the Project’s 
effectiveness, including an in-depth analysis of the extent to which and how CEP-supported GONM 
4 EWMI (2020, September 30). CEP, Year 5 Work Plan, p. 7.
5 EWMI (2020, September 30). CEP, Year 5 Work Plan, p. 8.
3 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
agencies and CSOs helped advance North Macedonia’s EU accession in regard to improving 
transparency, anti-corruption, and rule of law. 
The findings from this evaluation will help inform USAID/North Macedonia on governance activities in 
North Macedonia that could be replicated and inform activities under the new USAID’s North 
Macedonia CDCS 2020-2025 on anti-corruption leading to EU accession. The audience for the 
evaluation report is USAID/North Macedonia, its implementing partner, EWMI, GONM, and other 
international donors and development partners.
In conducting the final performance evaluation, the team focused on two evaluation questions (EQs) 
specified in the evaluation’s scope of work, found in Annex A:
1. EQ1: To what extent and how have CEP-supported CSOs helped to advance North Macedonia’s EU 
accession in areas related to rule of law, transparency, and anti-corruption? What advanced or 
impeded this approach to supporting the GONM in these areas?
2. EQ2: To what extent and how has the sub-grantee, Association of Citizens Center for Civil 
Communications (CCC) increased transparency in public procurement? What lessons learned from 
CCC’s work could be valuable and applicable to other efforts to advance government transparency, 
anti-corruption, and rule of law?
EVALUATION METHODOLOGY
To conduct the summative, final performance evaluation of CEP, the evaluation team used a utilization￾focused approach, incorporating complementary qualitative and quantitative methods of data collection 
and analysis, including online data collection tools tailored for different stakeholder groups. Before 
collecting data, the evaluation team conducted a thorough review of the documentation related to the 
EQs to gain an in depth understanding of CEP’s activities and to assess the availability and quality of 
CEP’s monitoring and evaluation data to answer the EQs. 
From October 18 through November 23, 2021, the evaluation team collected qualitative and 
quantitative data on the perspectives and experiences of key stakeholders involved in CEP. The primary 
research methods included key informant interviews (KIIs) focus group discussions (FGDs), an online 
survey, and a case study approach to capture each stakeholder group. Due to COVID-19-related public 
health restrictions, the evaluation team conducted all data collection activities remotely by telephone 
and web-based conferencing technology. 
The following summarizes the data collection activities to inform EQ1:
● Key Informant Interviews: The evaluation team conducted online interviews with 36 individuals 
(20 women and 16 men) including: 1) USAID and other USG stakeholders; 2) the CEP staff; 3) the 
CSO grantee staff; 4) GONM central level officials; and 5) media outlets and journalists. Each KII 
followed an interview protocol adjusted for different respondent types and was conducted in English 
or Macedonian, depending on the key informant’s preference.
● Focus Group Discussions: The evaluation team conducted 4 online FGDs with a total of 17
participants (12 women and 5 men), tailored to 3 main stakeholder groups including: 1) justice 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
4
system direct beneficiaries, including legal officers and trial monitors trained by CSO Grantees; 2) 
sub-CSO members from non-grantee CSOs that are part of a network of community CSOs managed 
by a CEP-supported CSO Grantee; and 3) citizens engaged by CEP-supported CSO Grantees. By 
conducting FGDs with specific stakeholder groups, the evaluation team could target lines of inquiry 
to gather the most relevant information. The evaluation team sought to ensure gender parity, to the 
extent possible.
● Online Survey: The evaluation team disseminated an online survey to 88 staff members
representing the local government level or municipalities. A total of 42 individuals completed the 
survey, for a response rate of 48 percent. The respondents represented 19 municipalities across 7 
regions, including 4 rural municipalities5F
6 and 20 female staff members and 22 male staff members.
For EQ 2, the evaluation team focused solely on the CSO CCC and its activities to enhance the 
transparency of public procurement. Taking a case study approach, the evaluation team conducted a 
workshop with 3 CCC staff members, key informant interviews with 10 individuals, and a focus group 
discussion with 4 participants from the business sector. 
For the detailed methodology, refer to Annex B. For the data collection protocols, refer to Annex C. 
DATA ANALYSIS
The evaluation team employed several data analysis methods and approaches to identify key findings 
from the data, draw conclusions, and make recommendations. The team triangulated the available 
performance monitoring data with qualitative data collected through document review, KIIs, FGDs, and 
the online survey. Evaluation team members independently reviewed the evaluation notes and tallied 
occurrences of key themes, comparing data from the various sources described above, to determine 
whether findings were divergent or convergent. The extent to which multiple informational streams 
provided consistent findings informed the certitude and internal validity of the conclusions. The 
evaluation team conducted content analysis by reviewing and coding KII and FGD data to identify and 
highlight notable examples of CEP’s successes, or lack of success, that contributed to, or hindered, 
achievement of its objectives. Throughout the analysis, the evaluation team disaggregated all KII and 
FGD data by sex and age and analyzed them for possible differences in respondents’ perspectives 
associated with gender and age. 
KNOWN LIMITATION AND MITIGATION STRATEGIES
The team identified several challenges and limitations in preparing for and conducting the evaluation, 
including cognitive bias as well as challenges associated with remote data collection. Additionally, the 
evaluation team experienced challenges in facilitating data collection due to the October 2021 local 
elections in North Macedonia affecting the team’s ability to employ the planned outcome mapping 
research methods to inform EQ2.
6 North Macedonia has 8 statistical regions and 81 municipalities (10 municipalities constitute the City of Skopje). 
CEP engaged with at least 32 municipalities.
5 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Cognitive bias: Key informants and FGD participants constituted a key source for answering all EQs, 
but interview data is prone to cognitive biases, including recall and social desirability, as well as 
respondent bias when implementers facilitate contact with beneficiaries. To mitigate this risk, the 
evaluation team began each KII and FGD with a protocol that reviewed CEP’s objectives. Additionally, 
the team explained the evaluation purpose and how the data can be used, while ensuring confidentiality. 
Challenges with remote data collection: Given the ongoing COVID-19 pandemic, the evaluation 
team collected all data remotely. Potential challenges associated with remote data collection included 
the risk of technology failures, diminished communication quality, and increased challenges in arranging 
meetings with respondents. To mitigate these limitations, the evaluation team had a variety of remote 
communication tools (Skype, WhatsApp, Zoom, mobile conferences, and regular mobile phone 
communication) available. Overall, the evaluation team encountered very few issues with connectivity 
and was able to mitigate this challenge. 
October 17, 2021 Local Elections in North Macedonia: Data collection was expected to occur 
from October 18 to November 12, 2021. Due to the local elections on October 17, the evaluation team 
found it difficult to arrange interviews with central and municipal government representatives. A second 
round of elections was held 2 weeks later on October 31. The evaluation team launched the online 
survey with municipal GONM officers immediately and followed-up with emails twice. The evaluation 
team also extended the data collection period to November 23. However, the planned number of KIIs 
and FGDs still was not met, resulting in an additional extension of the data collection period.
Outcome Mapping Approach: For EQ2, an innovative approach was planned, involving outcome 
mapping, a participatory method of evaluation that focuses on a review of the Theory of Change to 
identify behavioral and relationship changes and how a project intervention (the CCC public 
procurement transparency intervention for GONM ministries and municipalities) influenced those 
changes. The proposed outcome mapping approach included a series of workshops of about 3 hours 
over 2 days, and a follow up workshop to validate findings. The participatory process is long, requiring 
lengthy preparation to review documents, carefully select “boundary partners” to interview, and to build 
trust with CCC. Due to delays in the start of data collection activities in combination with the CSO’s 
lack of available time to fully invest in the evaluation’s participatory approach due to the work 
commitments of their staff, the original design was no longer feasible. The evaluation team adapted the 
approach to focus on KIIs and FGDs and an abridged outcome mapping technique, which enabled 
triangulation of information.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
6
FINDINGS AND CONCLUSIONS
CEP’s Background and Operating Environment
CONTEXT FINDINGS
STRATEGY AND IMPLEMENTATION APPROACH
CEP launched in a difficult operating environment, but overcame many challenges to 
establish a strong CSO alliance in North Macedonia. The USAID-funded CEP was awarded in 
2016 to the EWMI, prior to the change of the elected government in 2017. CEP was designed with the 
overarching goal to “strengthen the influence of CSOs’ and “increase and sustain youth engagement” in 
North Macedonia predominantly through capacity building interventions.7 Inherent challenges for a civic 
engagement project to operate in a country where CSOs were viewed with government suspicion were 
present from the start of CEP. CEP staff and CSO partners and grantees describe the early operating 
environment of CEP as “a bad time” as CSOs had to “keep a low profile, because, before, people were 
beaten, there were court trials, phone tapping, threats … it led to some CSO staff having psychological 
problems.”8 In efforts to mitigate these challenges and to demonstrate CEP’s commitment to “open[ness] 
and transparen[cy] about [its] activities,” CEP regularly extended invitations to engage with government 
counterparts and willingly shared information about its activities. Despite the varying degree of success 
in CEP’s government engagement approach, as described in more detail under EQ1, CEP built a strong 
CSO alliance, at several levels, with partners, grantees, and synergistic activities. Following the 2017 
elections and change in the elected government, general mistrust and suspicion resolved. 
After 3 years of implementation, or Phase 1 of CEP, CEP adapted its overarching objective
in response to new evidence generated from USAID/North Macedonia’s 2018 Project 
Appraisal Document (PAD) and to align with USAID’s J2SR framework representing the 
second phase of CEP. The PAD indicated that USAID should take a more targeted approach to 
supporting democratic transition focusing on North Macedonia’s key gaps in working towards EU
accession, namely transparency, anti-corruption, and rule of law. These findings from the PAD coupled 
with the recent change of GONM, generated an ideal situation for USAID to pilot its newly developed 
J2SR framework and was launched in October 2018. The J2SR framework outlines USAID’s approach to
“working with host country governments and partners to achieve locally-sustained results, helping 
countries mobilize public and private revenues, strengthening local capacities, and accelerating 
enterprise-driven development.”9 As a result of these contextual factors, USAID reviewed CEP’s original 
design and mandate. While CEP was recognized as a champion in supporting CSOs in North Macedonia 
and the country’s progression toward a democratic transition, CEP’s original mandate was broader and 
thus had only a small number of grants and activities focusing on the three areas identified by the PAD 
as gaps towards North Macedonia’s EU accession. By January 2020, USAID modified CEP’s contract to 
solely and strategically focus on transparency, anti-corruption, and rule of law activities. This shift, 
7 USAID/North Macedonia (2016, August). Solicitation, Offer and Award, Activity for Civic Engagement (ACE), 
East-West Management Institute Inc., AID-165-C-16-00003, p. 8. 8 Interview MN01, CSO, October 2021.
9 https://www.usaid.gov/selfreliance
7 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
considered the end of CEP’s Phase 1, resulted in closing 10 of its 17 grants that were not related to 
these priority issues. Of the remaining seven grants, two were also not related but had a near-term 
close date. The last five grants were already aligned to the priority issues and thus were extended 
through 2021 as CEP entered its Phase 2. 
The resulting shift in strategy and implementation approach was cataclysmic for CEP’s 
implementing partners and CSOs. CSO partners and CSO grantees referred to Phase 2 in a 
number of ways, including “the pivot,” “the redesign,” “the restructure,” “the reorientation,” and “the 
disruption” among others. Some partners and grantees sensed that a change was coming, but none 
predicted its severity and the shock waves that it caused. CSOs regarded the pivot as “a serious and 
severe blow” to the project.10 The decision was perceived as abrupt and simultaneously, “unexpected” and 
“unexplainable”.11 The sudden closure of the 10 grants, left grantees shocked and financially 
disadvantaged. CSO partners had made plans, recruited and trained staff, and looked to a future of 3 to 
4 years of USAID-funded support. One grantee said, “We had a grant for former drug addicts that were 
being reintegrated into society and they were abandoned because of the pivot.” CEP staff had to manage the 
expectations of government, its partners, and grantees. The stress for CEP staff was palpable and for 
some overwhelming, resulting in their resignation.
Entering into its second phase of implementation, CEP found itself in a new operating 
environment, one predominantly occupied by the EU and their work on anti-corruption.
Under its new mandate, CEP had to shift from a project designed to provide grants and capacity building 
support to a broad range of local CSOs to one that provided technical assistance to national and local
government institutions directly and through a smaller number of CSO grants already working on these 
issues. CEP had to shift its beneficiary target from CSOs and citizens – a large target audience at the 
grassroots level – to a new beneficiary pool of government agencies. CEP partners, including EWMI, 
ADI, and FOSM, had to move “from being grant managers to becoming implementers.” Upon engaging with 
the government agencies, CEP found that EU-funded activities were already heavily present which meant 
CEP needed to find synergies in the activities. 
To achieve its stated objectives under Phase 2, CEP’s prime implementer, EWMI, works 
with two levels of CSOs: CEP CSO partners (sub-contractors), and CSO grantees (now 
under Service Agreements and not grants). EWMI commenced in 2016 with four CEP CSO 
Partners: 1) ADI; 2) Metamorphosis; 3) FOSM; and 4) NYCM. FOSM had its capacity development 
assistance component terminated in June 2019, as well as 10 CEP grants, when activities were realigned 
under a USAID/North Macedonia re-design for Phase 2 of implementation.
For this evaluation, per USAID’s instruction and as outlined in the evaluation scope of work, the NYCM 
youth component is not included because its focus is outside the scope of transparency, anti-corruption, 
and rule of law. Thus, the evaluation includes an assessment of EWMI’s partners ADI, Metamorphosis, 
and FOSM (Exhibit 2). 
10 Interview MN07, CSO, November 2021. 11 Interview MN02, CSO, October 2021.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
8
From January 2020, to achieve this new sole objective of advancing North Macedonia’s accession to the 
EU, CEP completed its grants mechanism; 10 CSO partnership grants (PGs) and strategic support grants 
(SSGs) that did not focus on the 3 high-priority areas were closed earlier than planned, and 7 CSO 
grantees that did focus on these areas continued to receive funding. Of these seven CSOs, five had a 
strategic focus on anti-corruption, transparency, and rule of law. The pivot resulted in CEP working with 
five CSOs and their local sub-partners for the remaining 2 years of the project until July 2021, which was 
extended to August 2022 for some CSOs (Exhibit 2).12 By July 2020, CEP finalized and closed all grants
regardless of their end dates, and issued no further grants. After this time, the grantees were issued 
Service Agreements (not grants) if their activities were extended. However, for convenience, this 
evaluation report refers to these CSOs as grantees, rather than Service Agreement CSOs.
EXHIBIT 3: CSO GRANTEES ON TRANSPARENCY, ANTI-CORRUPTION, AND RULE OF LAW
TYPE CSO DURATION GOAL FOCUS
PG CAFT – Coalition of Civic 
Associations All for Fair Trials 
June 2018 –
June 2021
Improve access to justice and quality of 
justice for civil trial cases with a focus on 
vulnerable groups seeking justice (disabled, 
low socio-economic status, etc.). [EQ1]
Rule of law
12 The five prioritized CSO grantees were reviewed during a Design Workshop with USAID/North Macedonia,
held on September 27, 2021, and were further refined through CEP grantee lists.
EXHIBIT 2: CEP CSO PARTNERS
TYPE CSO DURATION GOAL FOCUS
Lead
EWMI – East-West 
Management Institute, 
Inc.
August 2016 – August 2022
Overall management of CEP, including liaison at the 
central government level. All
Partner ADI – Association for 
Democratic Initiatives
September 
2016 –
December 
2021 
ADI was responsible for short-term grants, e.g.,
Strategic Support Grants (SSGs), rapid response 
grants, and youth engagement support grants (YES). 
From January 2020, ADI worked on Component 4 
on anti-corruption at the local level with 
municipalities and citizens. ADI conducted 
vulnerability assessments.
Anti￾corruption
Partner
Metamorphosis (Meta) 
– Foundation for 
Internet and Society 
“Metamorphosis”
September 
2016 – June 
2022
Metamorphosis managed long-term Partnership 
Grants (PGs) – 3-year grants. From January 2020, it 
continued with the Ministry of Information, Society 
and Administration (MISA) on the Open Data 
system, working with municipalities and citizens.
Transparency
Partner 
FOSM – Foundation 
Open Society 
Macedonia
September 
2016 –
December 
2021 
FOSM strengthened municipalities’ resilience (i.e.,
capacity building) in corruption and conflicts of 
interest, focusing on reported cases and providing 
legal support. From January 2020, FOSM worked 
with five municipalities that already had integrity 
systems and an additional five municipalities to 
conduct integrity assessments. [The IP does not 
consider FOSM to be an official partner from 2020.] 
Anti￾corruption 
9 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
EXHIBIT 3: CSO GRANTEES ON TRANSPARENCY, ANTI-CORRUPTION, AND RULE OF LAW
TYPE CSO DURATION GOAL FOCUS
PG 
Citizens Association 
FINANCE THINK Institute 
for Economic Research and 
Policy
May 2017 –
September 
2020 
Enhance local-stakeholders’ practice and 
participation in the local budgeting process 
and increase the effectiveness of the 
municipal-money spending. [EQ1]
Transparency
SSG LCDF – Local Community 
Development Foundation 
March 2018 –
February 2021
Improve the quality of local public policies 
by strengthening the capacities of the 
municipal councils and local grassroots 
CSOs in policymaking and monitoring. 
[EQ1]
Transparency
PG TI – Transparency 
International
May 2017 – 
May 2020 
Enhance the impact of the CSOs related to 
anti-corruption progress in Macedonia. 
[EQ1]
Anti-corruption
SSG
CCC – Association of 
Citizens’ Center for Civil 
Communications 
April 2018 –
June 2021
Enhance effectiveness in public spending 
(public procurement) by reducing 
corruption and increasing transparency. 
[EQ2]
Anti-corruption 
& Transparency
THE COVID-19 PANDEMIC
CEP, along with the rest of the world, had to adjust its implementation approach in 
response to the COVID-19 pandemic. The World Health Organization (WHO) declared the 
outbreak of COVID-19 a pandemic on March 11, 2020.13 To reduce the cycle of COVID-19 
transmission and the high number of deaths, governments around the world, including the GONM, 
instituted countrywide lockdowns at periodic intervals, and strict public safety and hygiene measures 
such as physical distancing, wearing face masks in designated areas, and restrictions on domestic and 
international air travel. Initially, from March to May 2020, government and businesses closed – “everyone 
stopped breathing” – until a gradual re-opening commenced in June of the same year. This created 
significant challenges for CEP’s renewed mandate, as government counterparts, such as the Commission 
of Judges, effectively ceased operations for 6 months. COVID-19 posed significant challenges to the 
development community, and CEP had to formulate alternative ways to implement its activities remotely 
by online meetings, not knowing when the “unknown force” would end. This technological change 
brought its own challenges to local and central government operations. 
NATIONAL AND LOCAL ELECTIONS
CEP and the continuity of its activities throughout its period of performance were affected 
on numerous occasions by national and local elections in North Macedonia. In June 2020, just 
as the fear of the pandemic became manageable, early Parliamentary elections commenced, creating 
another challenge or “roadblock” for CEP to overcome. After months of working to build brand new 
relationships with government institutions that CEP had not worked with before, Parliamentary 
elections led to a pause in activities and finally some shifts in government counterparts, requiring re￾13 COVID-19, Coronavirus Disease of 2019, was first identified in December 2019.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
10
engagement. Ultimately, the same party remained in power, which was a benefit to CEP as this party
continued the agenda of anti-corruption due to international donors’ influence. Following elections and 
continued donor influence, the GONM formed a new Cabinet and appointed a Deputy Prime Minister
(DPM) for Anti-Corruption, Sustainable Development, and Human Resources (the Cabinet of the DPM) 
with two additional staff.
The GONM proceeded to hold municipal elections in October 2021. During the course of these local 
municipal elections, CEP had to stop engagement with mayors and municipality citizens through the 
respective partners and CSO grantees, such as progressing with the Integrity Plans. In October 2021,
following the opposition’s obtainment of mayoral seats and a second round of defeat, the Prime Minister 
resigned on October 31.14 In addition to the change of mayors in some CEP-support municipalities, the 
election had the lowest representation of women in mayoral seats since 2009, when no women were 
elected. In the October 2021 local elections, only two female mayors (2.5 percent) of the 81 municipal 
heads were elected into office. These shifts in municipal governments mean that partners and CSOs will 
have to work with new counterparts, some of which will present challenges to the CEP agenda. One 
interviewee shared that one new mayor has 15 criminal charges against him, lamenting the challenges 
this will impose on progressing with municipal integrity plans. 
CEP’S RESPONSE
CEP, hampered by COVID-19 and elections, and with an ambitious implementation plan, 
received a 1-year, no-cost extension of the project’s period of performance through August 
2022. Noting the many contextual factors hampering CEP’s ability to make progress, USAID approved a 
one-year no-cost extension. Given the ambitious mandate on anti-corruption, transparency, and rule of 
law, with only an initial two-year period of performance, CEP welcomed this extension, and it provided 
some stability and confidence across the partners. 
Despite these contextual challenges, CEP adapted and 
restructured its programming, devised alternative
implementation strategies, and, from the onset of COVID￾19, adjusted its communication approach. During evaluation 
interviews with current and former project staff and CSO grantee 
staff, the shock was evident, but most staff were reflective, 
acknowledging difficulties and proud of their progress under severe 
disruption. 
The pivot enabled the project to become narrower and deeper, focusing on specific activities –
predominantly technical and process-oriented activities at the GONM central level – moving away from 
general activities, such as broad CSO capacity building.15 GONM evaluation informants said the change 
of programming was well received. Although the majority of the CSO grantees whose grants were cut 
short felt the greatest pain, CEP staff indicated that eventually the shift was ultimately “good” because it 
14 Euronews (2021, November 1), https://www.euronews.com/2021/11/01/north-macedonia-s-prime-minister￾resigns-after-local-election-results 15 Interview MN07, CSO, November 2021.
We adapted very quickly 
because we had to. A lot 
was happening in the 
country at the time of the 
pivot – new government, 
new spirit, new energy.
– CEP Staff
11 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
demonstrated CEP’s commitment to the big issues of transparency, anti-corruption, and rule of law. CEP 
staff members attribute a smooth transition in part to much support from USAID/North Macedonia. 
Among the CSO grantees that no longer received support from 
CEP following the pivot, some continued to work on their 
previous activities, seeking funding elsewhere or restructuring
their efforts. One former CEP CSO staff member said, “We 
finished 18 months ago, but we are continuing our activities. So, that 
shows sustainability.” The informant confirmed the continued CEP
engagement with and support the CSO alliance through its CSO 
partners, developed before the pivot: “The support from ADI has 
not ended; and we also continue to work with Metamorphosis on 
activities to promote the Openness Index. CSOs are working together 
as a result of CEP. It’s a never-ending partnership.”16 
CONTEXT CONCLUSIONS
CEP faced numerous external and extenuating challenges from the project onset, forcing CEP to 
continually adapt its programming and delivery. Changing its overarching mandate midway through 
implementation, shifting from a project focused on civil society and citizen engagement to one focused 
on engaging central and municipal level government stakeholders in efforts to promote transparency, 
anti-corruption, and rule of law, was tantamount to starting a new project. Despite the resulting 
challenges, CEP staff successfully shifted its focus to appropriate entry points and maintained
commitment from the remaining CSO grantees to its new objectives. The following sections take into 
account these challenges, the achievements to date, and what to expect for the final months of the 
project and beyond.
The purpose of EQ1 was to examine if and how CEP-supported CSO partners, including EWMI, ADI, 
FOSM, and Metamorphosis and the CSO networks supported by these grants, advance efforts in anti￾corruption, transparency, and rule of law, which are necessary precursors to EU accession. Additionally, 
in response to EQ1, the evaluation team sought to understand what internal and external factors 
advanced or impeded CEP’s ability to demonstrate progress toward these objectives. 
EQ 1 FINDINGS
CEP’S ENTRY POINTS TO THE GONM
The Cabinet of the DPM, a newly-elected entity and the first of its kind in North 
Macedonia, became CEP’s primary central level government counterpart or entry point.
To fulfil its new mandate and in alignment with USAID/North Macedonia’s new CDCS, which placed
16 Interview MN10, CSO, November 2021.
Evaluation Question 1: To what extent and how have CEP-supported CSOs helped to advance 
North Macedonia’s EU Accession in areas related to rule of law, transparency, and anti-corruption? 
What advanced or impeded this approach to supporting the GONM in these areas? 
We used the CSO alliance 
networking approach for its 
solidarity … a sense of belonging 
with the same cause and mission, 
sharing and exchanging our 
knowledge and resources. From the 
disruption, the pivot, the CSOs 
were monitoring and coaching, 
making videos … but after the 
pivot, they continued to work 
together.
– CEP CSO Partner Representative
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
12
anti-corruption at the heart of its strategy, CEP focused on providing technical assistance to the Cabinet 
of the DPM to build its role within the GONM. This entry point was considered particularly important 
because as a new institution, there was significant opportunity to influence the determination of its role 
within the anti-corruption system. As stated by one interviewee, “There was room for creativity.” 
Additionally, in taking a systems-based approach, CEP simultaneously focused on 
complimentary entry points, government, and independent agencies at the central level, 
including the following:
1. Transparency – Ministry of Information, Society, and Administration (MISA) and the Agency for 
Protection of the Right to Free Access to Public Information.
2. Anti-corruption – (independent) State Commission for Prevention of Corruption (SCPC).
3. Rule of law – State Judicial Council, and the Academy for Judges and Public Prosecution. 
Respondents considered all entry points successful. However, some interviewees felt that the full 
landscape of anti-corruption was not able to be engaged. Also, CEP’s mandate on rule of law was limited
to small interventions. Interviewees felt that it was an important aspect of the anti-corruption landscape 
to engage for transparency and anti-corruption work to be effective. 
CEP continued working with its existing entry points such as CSOs, municipalities 
(including council members, mayors, and operational staff), media organizations and 
journalists, and local citizens. All of these stakeholders indicated that they understood “their role and 
relevance in the project”17 towards CEP’s goal to “increase GONM responsiveness to citizens’.18
The evaluation team encountered no external criticism of CEP as a project, and 
government entities at central and municipal levels were generally positive, supportive, 
engaged, and ready to receive support. Government informants indicated that public servants are 
generally young, and while they may be inexperienced, they indicated that they were “enthusiastic”.
TRANSPARENCY
CEP’s transparency activities focused on increasing GONM responsiveness to citizen’s needs, 
strengthening the ethical competence of public servants, and improving relevant mechanisms to 
enhancing “professional ethics”. In efforts to strengthen targeted public institutions to act more openly 
and increase accountability, CEP through CSO partner Metamorphosis and grantees FINANCE THINK 
and LCDF, focused on strengthening municipal councils to involve citizens in local decision making
processes while simultaneously building citizens’ capacity in their municipalities to become effective 
watchdogs of local issues. The following findings summarize the key achievements of CEP’s programming 
designed to strengthen transparency in North Macedonia.
17 Interview VD02, CSO, October 2021. 18 USAID/North Macedonia (2020, January). Solicitation, Offer and Award, Activity for Civic Engagement (ACE), 
East-West Management Institute Inc., AID-165-C-16-00003, Modification #06.
13 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP, through its partnership with Metamorphosis, contributed to enhancing the 
transparency and accountability of government institutions by increasing accessibility and 
use of open data at the national and local levels. CEP’s technical assistance designed to improve 
access and use of open data at the national and local levels has achieved high results with 106 new 
government open datasets released by CEP-supported institutions against a target of five. Notably, one 
ministry indicated that they did not input data for a year, showing that commitment is a long process. 
While the evaluation findings suggest that the established target was low, CEP set this target cautiously 
as this was the first time a foundation for open data has been 
introduced to public institutions in North Macedonia. 
Additionally, of all 42 respondents, 26 (62 percent) confirmed 
that their municipality had opened up their datasets or 
information repositories to the public as a result of the CEP 
project’s technical assistance. Five (12 percent) said they had not 
opened up their datasets yet, and 11 (26 percent) did not 
know.19
In January 2020, Metamorphosis’ Project Management Unit (PMU) began providing open data technical 
assistance to the MISA. For sustainability, the assistance focused on amending the law on the use of data 
from the public sector (Open Data Law), adopted in February 2020 to align the national data protection 
legislation with the general data protection regulation (GDPR), Regulation EU 2016/679. The MISA 
Open Data Working Group, in its first meeting in October 2021, expected to submit the draft by the 
end of 2021 for adoption by March 2022.
Furthermore, CEP successfully engaged and facilitated collaboration among 63 stakeholders in the design 
of a new Open Government Partnership National Action Plan (OGP NAP), which contributed to the 
development of an online OGP platform to provide access to government data for public. In 
collaboration with MISA, Metamorphosis oversees the use of the online OGP platform. 
Metamorphosis also published the first monitoring report of the implementation of the GONM 
Transparency Strategy (2000-2021) as part of the country’s efforts to comply with EU accession 
requirements, providing strategies that enable citizens to freely access public information. The 
monitoring report was developed in alignment with the Open Data Strategy (2018-2020), 
complementary to the Public Administration Reform Strategy (2018-2022), and the OGP NAP. The 
report samples and monitors 50 GONM institutions and documented limited implementation of the 
Government’s Transparency Strategy. 
CEP Metamorphosis PMU continues to provide training to civil servants and data officers 
to support use of the OGP platform. The main objective of these online trainings is to provide to
data officers across the target municipalities an understanding and appreciation of open data, a new 
concept in North Macedonia. As of September 2021, CEP trained 60 government data officers on open 
data quality in 17 municipalities. As some of the targeted municipalities lacked access to computers or 
adequate staffing, CSOs indicated that the training was “slow and inconsistent”.20 Hence, this training only 
19 Online Survey for Municipalities, Question 3.5 Has your municipality opened up its data sets or information 
repositories as a result of the project’s support?
20 Interview MN02, CSO, October 2021.
Sixty-two percent of survey 
respondents (26 out of 42) 
confirmed that their municipality had 
opened up their datasets or 
information repositories to the public 
as a result of the CEP’s assistance.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
14
represents half of the established target of 120. Training has continued and remains ongoing.21
Together, MISA and CEP Metamorphosis produced 10 video lessons (an automated webinar) for open 
data, with topics ranging from identification to cataloguing to prioritizing datasets. To promote broader 
use and uptake of the training materials, the OGP platform will soon publish the videos, making them 
available for government officers, CSO staff, and the general public. Notably, the webinar videos also 
contain a self-assessment for viewers to evaluate their readiness to engage in managing or using datasets.
MISA representatives are confident of the sustainability 
of open data processes, policies, and laws. MISA regarded 
the online OGP platform, which the GONM adopted in October 
2021, as a “great” monitoring tool providing free access to 
government information. Of about 1,200 institutions, MISA has 
reached 150 (12.5 percent), with open data received from 60 
institutions (5 percent).22 MISA’s aim is not to reach all 
institutions simultaneously, but to prioritize them. For example, 
MISA has worked with the SCPC since 2020 with data systems. 
The SCPC is experiencing technological issues and expects to 
launch a new open data system by the end of 2021.
Following the completion of CEP, MISA plans to conduct video training on the online OGP platform for 
all users in early 2022, after fixing technical issues. CEP’s technical support is regarded, not as a short￾term project, but as an ongoing “extraordinary process” that will ensure continuity throughout the 
country’s political changes. 
Local government training participants expressed high levels of satisfaction with the 
training facilitated by CEP. Of 42 respondents to the online survey for municipality staff, 91 percent 
of respondents were “very satisfied” or “mostly satisfied” with a range of trainings provided through 
CEP.23 Only one person (2 percent) was “a little” satisfied, and three did not answer. Refer to Exhibit 4.
Of the survey participants, 27 respondents confirmed that they had received targeted training from CEP 
on publishing data in different formats for different end users.24 Nine training participants provided 
feedback on the quality and usefulness of the training. Seven respondents indicated that the training was
“solid and effective” and that the training “was useful as there was an exchange of opinions and insights from 
several municipalities.” Two participants indicated that the training courses were provided too late, and 
that the trainings “should have been at the very beginning of the mandate so that we could use the knowledge 
during the term of office”. Additionally, these participants felt that the trainings lacked “practical examples”.
21 Interview MN02, CSO, October 2021. 22 Interview MN17, GONM, November 2021. 23 Online Survey for Municipalities, Question 3.4 How satisfied are you with the training support provided to you 
by the project? 
24 Online Survey for Municipalities Question 3.1 Have you been trained by the project to publish data or 
information in different formats for different users? and Question 3.2 Comment on the quality and usefulness of 
the training. Of 42 respondents, 27 had been trained, 8 had not been trained, and 7 didn’t answer.
The online OGP platform was used 
straight away. It’s a great 
monitoring tool, a supersonic portal 
for CSOs, institutions, businesses, 
and chambers of commerce. It’s free 
access to government information.
– Government Representative
15 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Furthermore, when municipal staff members were asked to comment through the online survey on 
tangible changes in their workplace that they have adopted due to the CEP transparency training, five 
respondents noted that:25
• “I pay more attention to the decisions to be voted on.”
• “After training, [we] decide to make all the decisions, not partisanship.”
• “All decisions made online and live broadcasts of council sessions were published.”
• “Several types of open-format data on the municipal website have been published.” 
• “[We have] greater discipline and responsibility for publishing data and information.”
CSOs cooperation on activities extend beyond CEP’s funding. The objective to improve 
transparency of the governments stems from engaging local CSOs (sub-CSOs) and citizens through 
partnerships. These partnerships are not solely concentrated on transparency, but through a 
combination of transparency, anti-corruption, and rule of law synergies. CEP CSO partners and grantees 
cooperate on activities not funded by CEP. For example, CEP set a target of eight synergies, and one has 
been achieved – Metamorphosis cooperated with Accountability, Technology Network of Institutional 
Openness in Southeast Europe (a regional network) to compare the Openness Index data for central 
government institutions across the region to gain lessons learned. 
25 Online Survey for Municipalities, Question 3.3 Comment on what you do now in your job that you did not do 
before – e.g., tangible or behavioral changes you have observed/experienced.
50%
41%
2% 0%
7%
Very Satisfied Mostly Satisfied A little Satisfied Not at all Satisfied No Response
Exhibit 4: How satisfied are you with the training support provided to you by the 
project? (n=42)
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
16
ANTI-CORRUPTION
CEP identified and drew on two main entry points to implement activities to combat corruption at the 
central level including the Cabinet of the DPM and the SCPC. Additionally, by strengthening local 
government resilience and conflict of interest through integrity planning and vulnerability to corruption 
assessments (VCA), CEP also sought to combat the effects of corruption in North Macedonia at the 
local level working in close collaboration with CSO partners ADI and FOSM and grantee Transparency 
International (TI). Additionally, through CEP’s collaboration with CCC, CEP sought to enhance the 
effectiveness of public spending by reducing corruption and increasing transparency. EQ2 presents an in￾depth analysis of CCC’s programming.
CENTRAL LEVEL
CEP support was critical to establishing the Cabinet of the DMP. CEP provided short-term 
technical assistance (STTA) to the newly formed Cabinet of the DPM to strengthen its institutional 
capacity, particularly in defining its roles and responsibilities. The Cabinet of the DPM, while not a public 
institution or agency and without executive powers, provides a coordination role for government anti￾corruption activities. Evaluation informants explained that initially the roles of the two entities were 
perceived to be competitive, but formal collaborations commenced and is ongoing. DPM staff maintain
regular communication with the SCPC and hold bi-monthly meetings. The government also approved 
eight new employments and seven promotions in the SCPC; approved funding for the development and
procurement of an assets register (of elected officials); almost doubled the SCPC budget from 2019 to 
2021; and provided new office space for the SCPC.
CEP employed a 3-K (3-C) strategy to address public institutional corruption focusing on
Collaborate, Communicate, and Coordinate.26 For example, to understand its audience – the 
citizens of North Macedonia – CEP assisted the Cabinet of the DPM to establish a public opinion 
research mechanism, which led to the Cabinet conducting its first national opinion poll and focus group 
discussions in 2021. CEP also assisted the Cabinet of the DPM with a software upgrade project for 
assessing citizen complaints with government institutions, which is ongoing. The Cabinet of the DPM 
described CEP’s governance support as “significant”, helping staff to be more professional, cohesive, and 
focused on teamwork.
[CEP] helped us a lot, [CEP] helped us get organized, restructured, motivated, and [CEP] provided help 
with our next ideas [projects]. It was important that [CEP] was here in person … [CEP] provided 
objective and independent advice. [CEP] helped to make the Cabinet fully functional. 
Furthermore, CEP worked together with the Cabinet of the DPM on a code of ethics for 
public officials for which CEP designed a curriculum that included six-90 minute modules
and conducted online training. The target was 15 training sessions with 20 people per training for a
total of 300 public officials trained. By the end of September, the Cabinet of the DPM had extended 
invitations to 400 public officials and 188 have received training.27 CEP also integrated a risk identity 
26 In the Macedonian language, the words collaborate, communicate, and coordinate begin with the letter K. 27 Interview MN09, November 2021.
17 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
methodology and integrity testing within the training program in two parts: 1) psychological testing, and 
2) real case examples. 
According to evaluation informants, two modules on the code of ethics were the most useful: 1) ethical 
leadership; and 2) ethical filtering during decision making. Another topic of use was the principles of 
ethical behavior, which included receiving gifts and citing research material. While participants perceived 
the training and testing as informative, many expressed an interest in follow up activities to better 
understand how to apply the knowledge gained. Specifically, one informant said, “We trained lots of 
officials [on the code], but how they will apply the code is a big question mark… it is okay in theory, but I don’t 
know how sustainable it is in practice. It needs further understanding.”28 At the time of the evaluation, CEP’s 
internal results of the training and testing were pending. 
The Cabinet of the DPM appreciated the rapid turnaround of CEP’s management advice 
and assistance. “There is a quick turnaround, but quicker for training project ideas than for the software 
upgrade. In comparison to other donor projects, CEP and USAID were concrete, specific, and quick.”29 At the 
time of the evaluation, the Cabinet of the DPM has already submitted another proposal to CEP for 
continued support. With the current political turmoil, the sustainability of the Cabinet of the DPM is 
uncertain. However, CEP plans to continue its support to the Cabinet until its status becomes clearer. 
Cabinet staff will sustain their experience and continue influencing anti-corruption activities and 
strategies. Stakeholders, and the evaluation team, maintain that CEP’s continued support to the Cabinet 
is appropriate due to the Cabinet’s willingness to own and sustain the supported approaches. 
In addition to working with the Cabinet of the DPM, CEP provided technical assistance to
the SCPC according to the Commission’s capacity building priorities, such as the 
assessment of bank statements and accounts. This assistance resulted in a manual on processes. 
The SCPC asked for assistance with reporting on political party financing during election campaigns as 
part of Article 32 of the law for prevention of corruption and conflict of interest. In April 2021, at 
SCPC’s request, CEP supported a VCA in Public Employment, which identified the government’s 
“growing commitment … [and] notable progress in legislative improvement, including the law on institutions and 
law on public enterprises, reducing the number of Board members and increasing the requirements for 
professional qualifications.”30 TI conducts quarterly monitoring reports on how selected public institutions
implement their vulnerabilities. SCPC uses the reports as evidence-based findings to take corrective 
actions as part of its independent role in monitoring government institutions. 
LOCAL LEVEL
CEP provided anti-corruption training focused on topics including conflict of interest and
training and integrity assessments at the local government level. These trainings were received 
favorably with 90 percent of surveyed municipality staff indicating that they were “mostly” or “very 
28 Interview MN09, November 2021. 29 Interview MN16, GONM, November 2021. 30 EWMI (2021, October). Quarterly Progress Report, July-September, #20, p. 6. Another CEP CSO Partner 
conducted VCAs in five municipalities: Bitola, Gostivar, Kavadarci, Kumanovo, and Shtip in 2020.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
18
satisfied” with the training. Refer to Exhibit 5.31 Specifically, several respondents described the trainings 
as “useful,” and that the training content was “well conveyed [and] easy to adopt.” As a result, one 
participant noted that they can now “recognize/identify different forms of corruption and bribes.” 
Source: Evaluation Team, Online Survey, October-November 2021.
Beyond satisfaction and knowledge gained, when municipal staff were asked to comment on the online 
survey about tangible changes in their workplace that they have adopted due to the CEP anti-corruption 
training, 32 participants noted that “For somethings, I now know where to turn or ask for help for certain 
information” and that they are practicing “better/more frequent communication.” On the other hand, a few
training participants indicated that they “didn’t need the intervention, but I can recognize and intervene as 
needed” and that they “have no engagement as a result of training.”
To strengthen government resilience to corruption and conflict of interest, CEP initiated 
Municipality Integrity Plans (MIP) over a period of 10 months from May 2020 to March 
2021, for five municipalities: Dojran, Karposh, Ohrid, Prilep, and Zhelino. Despite the CSO’s
low expectations for the initiative,33 the multi-party Municipal Council in three of the five municipalities 
have formally adopted the Integrity Plans: Dojran, Karposh, and Zhelino.34 The representative from 
Karposh confirmed this output in the evaluation’s online survey, describing it as “quality crafted and 
useful”.35 Two Integrity Plans are still in progress, although the Ohrid informant did not know its status, 
and neither did the CSO because the country’s Census interrupted the last month of the initiative.
31 Online Survey for Municipalities, Question 4.1 Have you been trained by the project on preventing corruption 
or provided with resources/support on preventing corruption? Of 42 respondents, 17 had been trained, 12 had 
not been trained, and 13 didn’t answer. Question 4.4 How satisfied are you with the training and support provided 
to you by the project? Question 4.2 Comment on the quality and usefulness of the training or engagement.
32 Online Survey for Municipalities, Question 4.3 Comment on what you do now in your job, after training or 
engagement with the CSO, that you did not do before.
33 Interview MN05, CSO, October 2021.
34 EWMI (2021, July). Quarterly Progress Report, April-June, #19, p. 25. 35 Online Survey for Municipalities Question 4.5 Have you prepared a Municipality Integrity Plan? and Question 4.6 
Comment on the quality and usefulness of the Integrity Plan.
37%
53%
0% 0%
10%
Very Satisfied Mostly Satisfied A little Satisfied Not at all Satisfied No Response
Exhibit 5: How satisfied are you with the training and support provide to you by the 
project? (n=30)
19 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Prior to CEP’s engagement, these five municipalities had never received training on the conflict of 
interest law before. The amendment to the prevention of corruption law subsumed the previously 
separate conflict of interest law, and both laws were covered in the MIP training. The 2-day training 
involved identification of local risks for each municipality. Although the risks were similar across 
municipalities – such as public procurement – the rankings for the likelihood of each risk was different. 
A combined workshop enabled information exchange among the municipalities. 
While the next step is the follow up and implementation of MIPs to use as tools for establishing and 
verifying the integrity of local government units as a basis for their policies, standards, and procedures, 
the CSO was not tasked with this continued support. To facilitate implementation, the CSO worked 
with the Strategic Planning and Integrity Sector of the SCPC to inform them of the MIPs. The SCPC 
liked the one simple, clear document with actions aligned to national policies, outlining roles and 
responsibilities.36 The result, to ensure sustainability, was a Memorandum of Understanding with the 
SCPC, the endorsement of the activity, and formation of a working group to follow up this activity.37
As a result of different CEP’s interventions, the majority of assisted municipalities 
demonstrated an understanding and respect for ethical considerations. The online evaluation 
survey for municipalities included three ethical dilemma questions. All 42 respondents answered each 
question (Exhibit 6). The two corruption questions (B and C) showed a clear choice stating that people
should always report a case of corruption, with no significant difference for the sex of the claimant (95 
percent for a male claimant and 93 percent for a female claimant). The promotion dilemma question (A) 
demonstrated differences in the selection of options. The respondents had a possibility of a promotion 
and the options included taking the promotion with a bribe, not reporting the person who got the 
promotion with a bribe, reporting the person who got the promotion with a bribe, or not knowing what 
to do because they did not trust the system. The majority (59 percent) did not know what to do 
because they had no confidence in the system. Thirty-six percent chose to report the promoted person 
who paid a bribe, and 5 percent took the promotion after paying a bribe. 
EXHIBIT 6. ETHICAL DILEMMA QUESTIONS – RESULTS (N=42)
DILEMMA NUMBER %
A: there is a possibility for a promotion that comes with a condition. What would you choose to do?
NO 1: A higher position that supports my family, but I have to pay the hiring officer a fee to influence 
the result to get the job.
2 5%
NO 2: A lower paid job even though I know that the person who got the promotion paid to influence 
the result, but I am not going to say anything because I might lose my job.
0 0%
NO 3: I didn’t get the promotion. The person who was promoted paid a fee to influence the result, so I 
am going to report it to the appropriate authority.
15 36%
36 Interview MN05, CSO, October 2021; and Interview MN14, November 2021. 37 Interview MN05, CSO, October 2021.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
20
EXHIBIT 6. ETHICAL DILEMMA QUESTIONS – RESULTS (N=42)
DILEMMA NUMBER %
NO 4: I don’t know what to do because I don’t have confidence in the system. 25 59%
B. When do you think it is justifiable for a woman not to report a case of corruption to the police?
NO 1: Never, she should always report it. 40 95%
NO 2: If she is in danger from the police because they are corrupt, and they may harm her. 0 0%
NO 3: A woman should not report a case of corruption to the police, but she should go to another 
authority or CSO/NGO that will support her.
1 2.5%
NO 4: She should always keep quiet for her own safety. 1 2.5%
C. When do you think it is justifiable for a man not to report a case of corruption to the police?
NO 1: Never, he should always report it. 39 93%
NO 2: If he is in danger from the police because they are corrupt, and they may harm him. 0 0%
NO 3: A man should not report a case of corruption to the police, but he should go to another 
authority or CSO/NGO that will support him.
2 5%
NO 4: He should always keep quiet for his own safety. 1 2%
Source: Evaluation Team, Online Survey, October-November 2021.
RULE OF LAW
CEP, in coordination with CSO grantee the Coalition of Civic Associations All for Fair Trials, conducted 
rule of law interventions at the bottom-up and top-down level by raising citizen awareness on their 
access to justice.
CEP’s central level interventions focused on improving the legal and regulatory 
mechanisms that hold the government accountable and strengthening the capacity of 
GONM institutions and ministries, such as MISA, the Academy for Judges and Public 
Prosecutors (AJPP), and the State Audit Office (SAO).38 In the judiciary system, CEP 
38 EWMI (2021, July). Quarterly Progress Report, April-June, #19, p. 6.
21 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
particularly focused on trial monitors working in civil and criminal cases centrally and regionally. CEP 
also established eight Advisory Bodies engaging citizens. CEP’s focus was on three pillars: education of 
the judicial ethics committee, strengthening the visibility of the association of judges, and peer-to-peer 
knowledge exchange with peers from other countries, particularly in ethical principles and mechanism 
for integrity. To date, the results of the Advisory Bodies are not substantial. Informants suggested a 
more effective approach would be citizen perception surveys conducted by GONM at the local level, 
such as the Cabinet of the DPM at the central level has commenced for anti-corruption.
CEP conducted trainings for judges and public prosecutors (staff of the SCPC and SAO), in collaboration 
with CSOs, media, and the Anti-Corruption Platform (ACP) – an informal entity – on reporting 
corruption; for example, public finances, conflict of interest, the law on protection of whistleblowers, 
and financing political campaigns. The training materials focused on criminal offenses in financing political 
parties during election campaigns for the AJPP related to the work of the SCPC, and training materials 
for the SAO, can be used for internal trainings to conduct their own capacity building. Although the 
evaluation team made an effort to interview some of the 15 ACP members, but were not successful, 
making it unclear how their capacities have improved, and the likelihood of its sustainability after the 
completion of CEP.
Notably, CEP supported the government to monitor civil trials, with the possibility of 
extending to criminal cases. Trial monitors received training on the methodology of monitoring civil 
and criminal cases, and they implemented their learnings according to court schedules, but also selected 
against criteria such as “a citizen who complains about too long procedure, no transparent trial, or no fair 
trial”.39 The citizen approached CEP’s CSO Grantee, which subsequently assigned a monitor to the case.
The trial monitors said the training was “very specific”, which enabled their effective monitoring. They 
added that “The transparency of trials improved significantly and also public access to the trials.” The monitors
noted a change in the behavior of some judges, such that “Now, the judges very rarely deny the presence of 
the public in the courtroom. This is the main benefit from the project; the 
public is more present in the courts.”40 However, the general view is 
that the court system needs more improvement, since not all 
participants and public institutions are involved in trainings or willing 
to engage: “My grade of transparency is 4 out of 10” said the 
evaluation informant. When asked if CEP was instrumental for EU 
accession in North Macedonia, the informant said, “Absolutely yes. 
The analysis and reports are useful for judiciary development. The project 
contributed a lot for the improvement of the transparency in courts.”
The main concerns about the trial monitoring intervention were that it was ad hoc and not organized 
with a clear focus. Trial monitors may attend a case on any civil issue, such as building rights, and that 
the trials are “long and complex, and coming to a trial in the middle of the procedure can mislead the monitor 
since [the monitor] is not familiar with the whole case.” Regarding CEP’s public institution monitoring 
reports, the same informant commented that, ‘‘The analysis provided by the project was good content-wise, 
but what was missing was clear conclusions and recommendations to overcome the identified issues…I assume 
39 Interview VD04, GONM, October 2021. 40 Interview VD04, GONM, October 2021.
Monitoring of court cases is a 
good idea since it enables better 
transparency and makes judges 
… provide fair trials and 
complete them in a reasonable 
time, not to be extended for 
many years without any valid 
reason.
– Government Representative
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
22
that findings are the result of a number of trial monitorings, and my opinion is that they are not very useful for 
the goal of improving transparency or rule of law.”41 Therefore, the informants questioned its sustainability 
and thought it would be more effective if there was established infrastructure for recording trials 
electronically, which could be disseminated, if appropriate, to interested CSOs and citizens as part of 
the open access dialogue.
PROGRESS TOWARDS EU ACCESSION
Assisting a country to reach EU accession is a highly ambitious goal and lengthy process
that requires strategic and coordinated support across several sectors and numerous 
stakeholders. To join the EU, a country must demonstrate its respect for EU’s fundamental values 
established in Article 2, including respect for democracy, rule of law, and human rights, as well as 
protection of minorities. The applying country must then receive approval from all existing member 
states. There are four basic steps to EU accession: 1) application, 2) candidate status, 3) negotiations, 
and 4) accession. While the average time to complete this process, from the start of negotiations to 
accession, is 5 years;42 Croatia, the last country to succeed in joining the EU in 2013, completed the 
process in 10 years from the start of its application. In 2004, North Macedonia applied for accession and 
was accepted 16 years later into the negotiations stage. 
CEP’s approach was not focused on the ultimate long-term outcome to gain EU accession, 
but on systematic step-by-step processes for transparency, anti-corruption, and rule of law 
that are, in themselves, the goal of many citizens of North Macedonia. While the objectives of 
transparency, anti-corruption, and rule of law are crucial to the systematic process of EU accession, and 
North Macedonia has made progress toward the objectives, 
the majority of stakeholders do not see accession as 
imminent. Of the 42 respondents participating in the online 
municipality survey, 25 respondents (60 percent) believed 
North Macedonia would gain EU accession within the next 5
to 10 years, while the remaining 17 (40 percent) disagreed, 
indicating that it would not.43 In March 2020, North 
Macedonia gained membership into the North Atlantic 
Treaty Organization (NATO), seen as a precursor to EU 
accession. One CSO member stated, “We are now in NATO, 
so the country has done it and should be congratulated. In terms 
of CEP, we are halfway along our Theory of Change.”44
However, in November 2020, an EU-member country blocked the start of North Macedonia’s EU 
membership talks, and, despite international efforts from EU-member countries and the United States, 
the blockade has not been lifted to date. Furthermore, while some informants thought the government 
was addressing corruption issues, Transparency International’s Corruption Perceptions Index (CPI) 
shows that North Macedonia scored 35 in 2020 of 180 countries with an average score of 43 out of 
41 Interview VD03, GNOM, October 2021. 42 UK in a Changing Europe, https://ukandeu.ac.uk, accessed December 5, 2021. 43 Online Survey for Municipalities, Question 6.1. 44 Interview MN15, CSO, November 2021.
EU Accession – no, that’s unlikely. The 
goal was too ambitious, but we provoked 
discussion and awareness of 
citizens…with the new government in 
2017 anti-corruption was raised higher 
and processes started, so even if we go 
backwards after this year’s elections, we 
won’t go back to zero. Even the EU is in 
doubt about what the country will look 
like in the future.
– CEP CSO Partner Representative
23 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
100, which places them towards the corrupt continuum (a zero score is highly corrupt and 100 is clean). 
North Macedonia’s CPI ranking has worsened from 93 in 2018 to 106 in 2019 and 111 in 2020.45
APPROACHES THAT ADVANCED OR IMPEDED CEP SUPPORT TO THE GOVERNMENT
The beginning of the Findings and Conclusions section of the report presents a series of contextual 
factors that created an extremely challenging environment in which to work. Despite these factors, CEP 
managed to deliver a wide array of interventions and adapt to the changing context. The challenges were 
rare for implementers to face, mid-stream, in the course of their contract. Nevertheless, CEP had to 
make implementation decisions for the remaining 18 months (January 2020 to August 2021) before they 
sought and received an extension to August 2022. 
Under the conditions CEP faced, sensitivities around communication and engagement were noted. The 
evaluation team found two main internal challenging factors that may have impeded CEP’s approach to 
achieving its own objectives, as well as supporting the government in its goal towards EU accession: 1) 
engagement of stakeholders in terms of missed opportunities, and 2) its monitoring, evaluation, and 
learning plan. This is followed by an identified factor that advanced CEP’s support to the government
through in-person STTA.
MISSED OPPORTUNITIES
CEP was not expected to partner with a wide range of government institutions over the short period of 
time. However, this narrowed the scope of institutions, particularly for rule of law and anti-corruption 
interventions. A larger range of government institutions could have been reached through a wider 
communication or engagement strategy to learn from each other in a neutral environment, which would 
indirectly capture a full spectrum of institutions that were not part of CEP’s mandate. 
CEP did not use youth engagement as an entry point to improve transparency, anti￾corruption, and rule of law in North Macedonia. Component 3 which focuses on strengthening 
youth engagement, with its existing network of active youth interventions, was not maximized after the 
pivot. The 18 to 25-year-old group has potential to contribute to generational change in good 
governance. The Assembly of the Republic of North Macedonia adopted the law on youth participation 
and youth policies (LYPYP) in January 2020. The new law obliges municipalities to involve youth in 
decision making processes, and also includes a Manual for Local Youth Councils, revised in 2020 in 
accordance with the LYPYP.46
“Media has the power to increase citizen awareness and understanding of their civic rights.”
CSOs believe that, despite extensive coverage of CSO activities in the local and national 
media and online, a wider entry point is required to maximize the media, particularly 
investigative journalism that exposes violations.47 Reforms to the rule of law, and the 
engagement of the judiciary and legal system to advance effective penalties for violations, are limited 
within a short-term project, and therefore targeted strategies could be considered.
45 TI (2021). Corruption Perceptions Index 2020, p.3; and compared with the two previous years. 46 Interview MN19, CSO, November 2021. 47 Interview MN11, CSO, November 2021.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
24
Despite the many, and appropriate, entry points CSOs engaged, CSOs noted that they 
“cannot fill in for [public] institutions.”48 Transparency, anti-corruption, and rule of law, particularly 
to advance EU accession, must come from institutional change through improved processes and 
procedures that remain stable throughout political change, as well as through the country’s citizens 
being knowledgeable and aware of good governance to hold the government to account. 
MONITORING, EVALUATION, AND LEARNING PLAN 
CEP updated its Monitoring, Evaluation, and Learning Plan (MELP) to adapt to changes in 
its mandate and its extended period of performance. After USAID/North Macedonia’s decision 
to restructure the project, CEP had to plan for new stakeholders, interventions, strategies, approaches, 
and activities. These actions had to be included in CEP’s MELP and had to reflect the changes with new 
indicators and targets, at the same time that the COVID-19 pandemic struck in March 2020. The revised 
MELP was approved in September 2020, and by February 2021, CEP applied for a one-year extension, 
which was approved shortly thereafter. In light of this extension, CEP had to again adjust its approach 
and MELP for another year of programming. 
As a result of continued uncertainty and subsequent revisions, CEP’s final MELP, in part, 
monitored and measured its progress and success. To some extent, CEP adapted its MELP to 
measure government engagement and demonstrate government commitment to the processes of 
transparency and professional ethics. However, CEP’s revised MELP did not establish clear processes to 
measure change, effects of grantee interventions, or citizen engagement. With more time, less 
uncertainty, and few external challenges, CEP could have included measurement of change at the 
outcome level and a citizen perception survey or study.
Given that CEP began as a CSO-strengthening, CSO-alliance building, and CSO-engagement project, the 
project objective and indicators are appropriate and measurable, but not ambitious. The overall project 
objective and its indicators aim to measure strengthened CSO engagement and the number of joint 
activities in cooperation (i.e., engagements) with government institutions. 
After the programming pivot and the introduction of Component 4, CEP’s indicators and targets 
remained mostly output-based, and not outcome-based. The indicators measure activity results rather 
than outcome indicators that measure change. As noted in the Project’s Modification #06, CEP’s 
mandate expanded to include the objective, “technical assistance in support of EU Accession” through 
more transparent and accountable government institutions, for the following results: Intermediate 
Result (IR) 4.I Increased GONM responsiveness to citizens; and IR4.2 Strengthened ethical competence 
of public servants, and strengthened mechanisms to support “professional ethics.”49 Component 4 
introduces the end-consumer – citizens – and a focus on the government’s responsiveness to them, but 
not the measure of citizen confidence to engage with the government (despite the many CEP activities 
that support citizens and government linkages and engagements).
48 Interview VS03, GONM, October 2021. 49 USAID/North Macedonia (2020, January). Solicitation, Offer and Award, Activity for Civic Engagement (ACE), 
Modification #06, East-West Management Institute Inc., AID-165-C-16-00003, p. 2.
25 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Under Component 4, there are five performance output indicators that measure the number of 
trainings, the number of trained civil servants/data officers, the number of stakeholders engaged in 
creating the OGP National Action Plan, and the number of representatives participating in trainings. One 
indicator is documented as an outcome indicator, but it is an output indicator for the number of new 
open datasets released by each supported institution. The indicator does not measure change, such as 
the frequency of use of open datasets, who is using them, for what purpose, and to what effect or 
impact. However, the indicator does demonstrate the commitment of public institutions to the process 
of transparency.
Furthermore, from the CSO Grantees point of view, many of their activities are pilot projects or new 
methodologies, approaches, or activities by themselves or their sub-CSO partners. Having collective or 
aggregated indicators, or some measure of success, at the CEP project level in its MELP, would help to 
highlight the most effective approaches and to learn from the innovations and experimentations.
Due to the lack of outcome-level indicators that measure change at the central 
government, local government (municipality), civil society, and citizens levels, as well as 
the follow-up of the learnings and CSO-government-citizen engagements, it is difficult to 
fully understand CEP’s progress and achievements through an evidence-based approach to 
its MELP. CEP’s progress toward success is detailed in the narratives of its quarterly reports. These 
narratives focus less on achievements against a range of sequential indicators with realistic targets that 
measure the objectives, theory of change’s intermediate results, and all three areas of support 
(transparency, anti-corruption, and rule of law). For example, CEP’s measurements did not focus on 
defining success as measured by the Project’s technical assistance (to central and municipal 
governments), increased responsiveness to citizens (and citizens’ perceptions of the government 
response), and changes resulting from CEP’s interventions on anti-corruption and rule of law in addition 
to government transparency and professional ethics.
IN-PERSON STTA 
One approach that advanced CEP’s support to the GONM functioned within the Cabinet 
of the DPM in which both STTAs were present, working with the staff, visibly, in-person, 
and providing technical assistance and mentoring. CEP’s governance STTA, starting with three 
people in the Cabinet, supported and strengthened the staff by creating a team that grew to six and 
more officers. A Residential Retreat for the Cabinet in May 2021 further strengthened their teamwork. 
Through this teamwork approach, with clearly defined aims and objectives aligned to the government’s 
national plan, and both individual and team roles and responsibilities, the Cabinet of the DPM achieved a 
significant amount of work in a short period of time, as noted by representatives during evaluation KIIs.
The code of ethics STTA conducted training remotely but was present in country for ongoing contact 
and support. 
Staff members of the Cabinet stressed the importance of having physically present experts--one a local 
university professor; the other a retired international expert in democracy living in North Macedonia--
who were readily available for “quick response’ support”. The CEP management team was also described 
as being “cooperative” and able to provide support and quick turnaround response times to queries. 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
26
EQ 1 CONCLUSIONS
The objective of technical assistance in support of EU accession through more transparent 
and accountable government institutions was highly ambitious. While CSOs Grantees and 
citizens felt the burden of the confusion and fragmentation of CEP’s pivot and new strategy, including 
the loss of funding, the government benefited through new technical assistance from CEP’s programming 
shift. The focus on the ideals of transparency, anti-corruption, and rule of law, through technical support 
to municipalities and public institutions, increased CEP’s visibility with the GONM at central level. All 
GONM evaluation informants at the central level were conversant with CEP’s strategy. The informants
maintained that they knew the implicit project goal of advancing EU accession, yet also acknowledged 
the explicit goal of countrywide awareness and interventions advancing good governance through 
transparency, anti-corruption, and rule of law. 
CEP tailored its technical assistance to government requests and priorities, resulting in 
high use rates, with much of the advice and support implemented immediately. The support 
was provided in a way for the government to adopt, adapt, modify, and establish cycles of operational 
activities. The support was provided through short to medium tasks and “projects’ which have achieved 
what they intended. However, the next stage is to follow up, where relevant, through locally led or 
donor-funded programs, or through CSO activities on the implementation, monitoring, or provision of 
evidence-based statistics or reports to inform evidence-based decision making and policy development. 
CEP has established a foundation for institutional change in the processes and procedures 
for government transparency, anti-corruption, and rule of law, particularly in government 
institutions opening up their data to the public. Of more importance, CEP trained the 
government on making the data usable for different audiences. At the same time, CEP created demand 
from citizens for public municipal data, as well as good governance such as fair trials and engagement 
with citizens. While significant progress was made at the central and municipal levels, making its 
sustainability highly likely due to the processes and technology in place, “there is still a long way to go.”50
The evaluation team took a case study approach, reviewing in depth the transparency work of one of 
CEP’s grantees, CCC. The evaluation team sought to form a multi-faceted understanding of CCC’s work 
and its effects, while collecting information and perceptions from representatives of institutions that 
CCC intended to influence, as well as key people in the public procurement process.
CEP GRANT TO CCC: PUBLIC SPENDING FOR PUBLIC GOOD
CCC is a well-established CSO in North Macedonia, operating since 2005. It has focused on promoting 
transparency, accountability, and anti-corruption in public institutions, conducting and publishing 
research to flag potential corruption issues, working with the media to broadcast these findings, and 
providing recommendations and advisory services to government initiatives, legislation, and procedures 
50 Interview MN02, CSO, October 2021.
Evaluation Question 2: To what extent and how has the sub-grantee, Center for Civil 
Communications increased transparency in public procurement? What lessons learned from CCC’s 
work could be valuable and applicable to other efforts to advance government transparency, anti￾corruption, and rule of law? 
27 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
to reduce corruption and increase transparency. CCC received a grant in 2018 under the CEP Strategic 
Support Grant Program titled “Public Spending for Public Good” (Exhibit 7). 
EXHIBIT 7. CCC GRANT
PUBLIC SPENDING FOR PUBLIC GOOD
Implementation: April 1, 2018 – June 30, 2021
Duration: 3 years
Total Budget: USD 168,320.24
Purpose: 
Enhance effectiveness in public spending by reducing corruption and increasing transparency.
1. Improve public procurement’s legal framework and best practices toward reduced corruption 
and enhanced effectiveness in public spending. 
2. Improve implementation of regulation related to transparency and accountability of public 
institutions on a national and local level.
3. Enable journalists, CSOs, and citizens to fulfil their oversight role in public spending and 
increase the pressure on public institutions for greater transparency and reduction of 
corruption.
4. Increase organizational capacity for oversight of public spending and expanding activities.
CCC approached its work by conducting the following activities: 
• Public spending research and dissemination: CCC monitors central and municipality level 
public spending activities and attends open tender procedures to produce databases, research 
reports, and information on an on-going basis. CCC engages with stakeholders to widely disseminate 
this information, ensuring it is publicly available on the CCC website for reference. For the grant, 
CCC analyzed public spending activities including the following subjects: 1) vehicles owned by 
executive branch institutions; 2) firefighting equipment; 3) winter street maintenance contracts; 4) 
New Year’s decoration purchases; 5) COVID-19 response procurements; 6) property tax collection; 
and 7) payments to municipal councils. 
• Index of Active Transparency: CCC conducts the annual Index of Active Transparency
assessment to rank the performance of 98 public institutions representing government agencies, line 
ministries, and municipalities. For the grant, CCC published the Index in 2019, 2020, and 2021. The 
Index often prompts public institutions to reach out to CCC for consultation or assistance on 
improving their score, which is a service provided on an as-needed basis. 
• Journalist Help Desk: CCC provides ad hoc and demand-based assistance and consultations to 
journalists requesting information about the misuse of public funds by public officials and data to 
corroborate potential corruption in public spending. Further, CCC reaches out to their journalists’
network to flag important corruption risks or hindrances to transparency, for news reports.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
28
• CSO training and engagement: CCC mobilizes and trains CSOs within their network on how 
to use CCC’s available public spending research and the Index of Active Transparency to put 
pressure and provide support to public institutions within their sphere of influence. CCC conducts 
trainings for CSOs to do their own public procurement monitoring, equipping them with the 
methodology and skills. For the grant, CCC trained 31 CSO representatives from across the country. 
• Annual business community survey: CCC conducts an annual survey which it distributes to 
companies to comment on their experience participating in public procurements. CCC uses this 
information to inform their engagements with public institutions. 
• Engagement in GONM-led working groups and roundtables: CCC is an active participant 
and civil society representative in government-led working groups, expert roundtables, and 
consultations. Through these working groups and roundtables, CCC provides key recommendations 
to support legislative and regulatory changes, as well as GONM strategies and programs. 
EQ 2 FINDINGS
All stakeholders interviewed cited that CCC’s research is high quality, user-friendly, and 
impactful. Each informant highlighted that CCC produces digestible and methodologically sound 
research and information on public procurement monitoring that easily incorporates into their 
respective work. CSOs shared that they use CCC’s databases for monitoring spending activities at the 
municipality-level. Journalists said that CCC’s research is “easily accessible to us to produce news reports 
that put pressure on the institutions51”. The journalists emphasized that, in addition to the content being 
sound, the way CCC presents the findings – through data visualizations and information visuals – enables 
a non-procurement expert to understand the transparency issue under analysis. Meanwhile, public 
institutions shared that the “CCC very precisely and practically conveys the analysis and knowledge to all 
institutions” and that the research is “important and statistically substantiated52”. Public institutions 
acknowledged that research is rarely translated immediately into legislation, but CCC’s research has 
served an important function in sparking a debate within and across public institutions. Further, in 
comparison to other CSO’s work on monitoring government activities, one public institution asserted 
that CCC’s “criteria and methodology is crystal clear, whereas other CSOs don’t always apply clear 
methodologies to measure transparency.53” A notable strength of the methodologies is that CCC does not 
rely on primary data collection with the government. Instead, CCC uses publicly available information or 
requests for information that should be publicly available to conduct their work. Finally, journalists 
emphasized that this data and information is critical in producing compelling investigative stories that can 
lead to important prosecution of corruption, reminding public institutions that CSOs are monitoring 
their work, and that there are penalties for corrupt practices. 
The Index of Active Transparency is widely considered a particularly useful tool for CSOs, 
journalists, and public institutions. In discussing CCC’s work, all stakeholders interviewed with the 
exception of the business community, highlighted the Index of Active Transparency as an important 
51 Interview CC01, Journalist, November 2021 52 Interview CC03, Public institution, November 2021 53 Interview CC08, Public institution, November 2021
29 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
contribution made by CCC to North Macedonia’s fight against corruption. As a tool, different 
stakeholders used the Index in the following ways: 
• CSOs use the Index as a benchmarking tool to put pressure on largely municipal-level government 
institutions and correct behaviors. 
• Journalists use the Index to publicly demonstrate institutions’ progress on transparency, including 
highlighting top performers and naming those falling behind. 
• Public institutions also use the Index as a benchmarking tool, as well as a targeting tool, to identify 
areas that need improvement to increase their Index score. 
Public institutions most widely discussed the effects of the Index, likely because of its use by CSOs and 
journalists to structure institutions’ progress on transparency. The institutions said that when the Index
is published, they review and assess their score, their deficiencies, and sometimes reach out to CCC to 
request advice on actions to improve their score for the next assessment. As described by the Ministry 
of Defense informants, which experienced a major shift in their Index scores over the years: 
A few years ago, the Ministry of Defense was in 45th position on the list … Last year we were in second 
place, and this year, we are first on the rating list, with a maximum 100 points…. When we had 98
percent, we asked CCC where this 2 percent noncompliance was and they provided us a clear answer 
and we have corrected and now have 100.
Stakeholders noted that the Index’s success is due to the methodology; in particular, CCC’s 
transparency around the methodology. CCC ensures the specific data sources, means of verification, 
and analyses are open and public for review and scrutiny. In addition, CCC staff avail themselves the 
details with institutions regarding their scores. These aspects contribute to the overall success of the 
Index. 
Public institutions stressed CCC’s contribution to educating and raising awareness on 
transparency issues. When describing changes in attitudes or behaviors experienced by public 
institutions because of their engagement with CCC, they most highlighted improved understanding 
about free access to information and open public procurement processes. Public institutions shared a 
renewed understanding of what “classified’ information is, and that sometimes this designation has been 
Success Case A: The Active Transparency Index as a Tool for Government Monitoring
An independent CSO described how they used the Index to conduct continuous monitoring of 
municipal public institutions in the Eastern region. In 2016, the Eastern region ranked in the middle 
out of eight regional groups whose municipality instructions were assessed through the Index. The 
CSO began their monitoring work at that time, sending numerous letters and indications of how and 
what should be changed and specific information that needs to be published or publicly available to 
help the public institutions in the region improve their individual and collective scores. “We are 
putting a sort of constructive positive pressure.” The CSO uses the Index as its way of framing 
transparency issues as well as being explicit about what the institutions need to do to ensure they 
have a better score the next year. Today, the region is in the top position, meeting 70 percent of 
criterions. The CSO continues its work, in an effort to move the region closer to 100 percent.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
30
improperly used to obscure public spending information54. Previously, information thought to be
sensitive was legally entitled to be open to the public. For example, the Public Procurement Bureau 
shares its experience observing this change in understanding: 
Discussions have been initiated related to CCC’s demands for free access to information … I see it from 
the perspective of authorities that feared sharing the information … they turned with questions to us 
and asked if it was allowed or not … an awareness has been developed with the institutions in terms of 
the fact that there are no secret data.
In addition, the Ministry of Defense describes the difference they have experienced in the success case 
below. One journalist interviewed echoed the public institutions’ sentiments, stating that in recent 
requests for information, public institutions are “more open than previously, when asked they usually provide 
the information”. However, informants also noted that the information comes slowly, often the day 
before the deadline. They noted that the public institutions’ websites are now much more informative,
and information can be found around open finance and public spending. 
Most stakeholders agreed that CCC’s engagement was instrumental to key public 
procurement legislative and regulatory changes. CCC notes its contributions and that public 
institutions accepted their recommendations to the following legislative and regulatory changes during 
their engagement under CEP: law on public procurement; law on free access to public information; 
open government partnership action plan; North Macedonia’s 5-Year anti-corruption strategy; draft 
strategy for government transparency; and public finances management program. 
From discussions with stakeholders, the most referenced contributions CCC made to legislative and 
regulatory changes as it relates to transparency included: 1) the law on public procurements; 2) the 
electronic system for tenders; and 3) the public finances management program. As noted in the 
description of work, CCC can provide concrete inputs into these government processes because its 
members are invited to take part in working groups and roundtables. CCC brings to these engagements 
its recommendations from its research, monitoring, and analytical work. According to one stakeholder,
“it was the CCC that had the greatest impact for the recent changes to the law [on public procurement] … it 
was a long process and pressure was made and there was evidence based on practice.
55” Meanwhile, the 
business sector lauded CCC’s work to support creating an electronic system for tenders as “good 
54 Interview CC08, Public institution, November 2021 55 Interview CC05, CSO, November 2021
Success Case B: Ministry of Defense’s Change in Behaviors
The Ministry of Defense described the difference they have experienced from their engagement with 
CCC and interactions around the Index of Active Transparency. This engagement and interactions
are presented in their own words. “We made changes. We have changed the quantity of information we 
publish. It is now a massive amount of information we make public. We have changed the internal rules and 
behavior within the Ministry, which provided this result. Active transparency was raised to the highest level, as 
one of the most important functions in the Ministry. As a result, transparency was given top priority and all 
the staff respect this direction for improved transparency. This is not overnight activity but slowly we progress 
and now we are the first on the list [of the Index of Active Transparency].”
31 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
practice, which provides better transparency.56” Finally, another stakeholder highlighted how CCC’s high￾quality work has earned it a reputation as the go-to CSO to engage in producing official documents or 
legislation related to transparency: 
[CCC] is practically called upon as a CSO to the drafting and monitoring of the implementation of the 
Public Finances Management program, which is the most significant program document at the level of 
the Ministry of Finance.’
CSOs manifest an increased understanding of transparency and skills to conduct public 
procurement monitoring as a result of their engagement with CCC. CCC galvanizes other 
CSOs to adopt CCC’s research and methodologies to conduct their own public spending monitoring 
activities as related to their geographic coverage. CCC’s procurement monitoring research is at the 
central level and the municipal level, but CCC tends to focus on applying pressure to the system as a 
whole or the central level. Thus, CCC collaborates with other CSOs and invests in training CSOs on 
CCC methodologies so that CSOs can replicate the monitoring on issues of concern in their 
municipalities. 
CSOs described being trained multiple times during the CEP grant and before, as well as more generally 
engaging with CCC through CSO networks and relevant platforms. CSOs shared that, through the CCC 
trainings, they “acquired knowledge that will enable them to easily spot any instances of corruption”, and while 
engaged with CCC they benefited from “the networking of all local organizations and expert consultations57”.
Overall, CSOs reported positive views of these trainings and engagements, emphasizing that they applied 
the new knowledge and skills daily. One CSO described examples of how its staff translated what they 
learned from CCC into their work and interventions: 
What we learned in training helped us start a local tribune for anti-corruption where we announced our 
findings from public procurement monitoring of a municipality’s hospital. We used this to mobilize the 
local community and inform them about spending problems ... for example, charging the public for 
vaccines when they are free of charge.
While important public procurement legislative and regulatory changes have occurred, 
there are mixed reviews on how this has translated into changing government practices. In 
addition, important progress has been made in reforming legislation and regulations about transparency 
in public procurement as described above. However, according to representatives from business 
associations and chambers, the business sector has not yet seen this transform into concrete changes in 
public tenders. As explained by one business community member:
Even though many changes have been introduced to public procurement, the situation has not 
significantly changed. The law itself cannot prevent corruption and lack of transparency without rigid 
control and sanctions. So, the control is missing and many tenders are not transparent. 
The business sector acknowledged that progress in transparency has been made, particularly through 
introduction of the electronic tender system. However, gaps remain: “transparency is improved, but all the 
56 FGD CC06, Business community, November 2021 57 Interview CC02, CSO, November 2021
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
32
manipulations are invisible58.” The business sector feels ill equipped to recognize signs of corruption in 
tenders. Another example of progress is the change in the law, but that the law is insufficient without 
controls to ensure the law is followed. The business sector mentioned cases where tenders are all 
legally correct, but the tender criteria are manipulated such that only a small group of businesses can 
participate. Furthermore, access to tenders for small companies remains limited. 
Unfortunately, the challenges faced in the public tender process have reduced the motivation for some 
companies from participating in publicly funded work. In an evaluation FGD, the business sector 
commented that “Most companies do not even follow the calls for tenders, since they believe that all the 
tenders are corrupted, mostly politically, and do not bother to apply at all.59”
The majority of stakeholders said that CCC is effective because of its reputation. CCC’s 
long history of work on monitoring public spending and conducting methodologically sound and 
transparent research, has earned it a positive reputation among all stakeholders interviewed during the 
evaluation. Informants stressed that this is reason why CCC is almost always invited to represent civil 
society in government working groups and consultations around legislation, procedures, or programs 
related to transparency. As noted by one CSO, “All the CCC employees are highly committed and 
specialized, which is also proven by their attendance in all government consultations60”. All the informants
described the ways CCC worked with them on key legislation or regulation, with about half citing that 
they do not cooperate in this way with any other CSO. According to one person, “CCC has made quite 
an impression, and in our opinion, it has the highest score.61” Another public institution member said: 
We often invite CCC to support us in preparation of some document. It is invited and always welcome, 
since it is a relevant and reliable partner who can provide good support. It has integrity and objectivity 
that is very much appreciated by all … [CCC’s] opinions about different issues are important for us.
Many stakeholders opined that CCC is effective because of its relationship with the media 
and journalists. While all stakeholders affirmed that CCC has “an excellent cooperation with the 
media62”, some believed that this was the key ingredient to its success alongside its reputation. One 
public institution stakeholder highlighted that “CCC is founded by two journalists … they were excellent 
journalists and they left TV wanting to make a difference63.” Informants drew the conclusion that the 
background of the CCC founders provides them with greater ties to the media, knowledge on how to 
work with the media and journalists, and knowledge of how to present data and information in a way 
that can be used by the media. Other informants highlighted that a broad network of national and local
media is aware of CCC and frequently publish its activities and some even conducted their own public 
procurement monitoring. According to CEP staff, of all of the CEP grantees, media most frequently 
picked up CCC’s work and most frequently disseminated it. This was considered notable given the 
media’s role in galvanizing public pressure. 
58 FGD CC06, Business community, November 2021 59 FGD CC06, Business community, November 2021 60 Interview CC07, CSO, November 2021 61 Interview CC03, Public institution, November 2021 62 Interview CC02, CSO, November 2021 63 Interview CC08, Public institution, November 2021
33 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CCC’s central strategy is to bring instances of corruption or risks of corruption in 
government spending to light to put pressure on public institutions to change legislation, 
regulation, and practices. This pressure relies on reaching a broad audience, which CCC has 
accomplished through ensuring the media has the information necessary to publish relevant news 
stories. Stakeholders shared that this was an important strategy because the media has “power in 
influencing public institutions64”. However, CSOs, public institutions, and journalists raised concerns about 
the freedom of the media, and whether there is sufficient funding, support, and independence for 
investigative journalism. According to one public institution, the capacity and availability of journalists are 
insufficient: 
The media is not as they used to be. Previously, each journalist was assigned to a specific topic and 
specialized in that field, such as judiciary, health or something else … Today, journalists are only looking 
for the information to publish without in-depth knowledge of the sector, so they only provide basic 
information. There are not many strong journalists. 
Both journalists and CSOs agreed that insufficient journalists are doing investigative journalism, in part 
due to a lack of capacity, but more crucially due to a lack of financial independence and concerns over 
media’s freedom. 
In our country, this type of [investigative] journalism has been existing in a restricted matter due to a 
number of multifaceted reasons ... It requires financial independence. However, in our market and 
society, the conditions are not there to support strong investigative journalism … investigative journalism 
based merely on our resources [in North Macedonia], is impossible. – Journalist, KII
I do not know how feasible it is in these times to work with the media on transparent governance, 
because they are mainly funded by the state. You open a channel and everyone is saying the same thing
... There is not enough support for the journalists and there is no continuity. It is a very difficult time now. 
Journalists cannot speak if it is not in line with editorial policy. – CSO, KII
Stakeholders expressed that the media seems to be aligned with commercial incentives, with limited 
incentives and funding available for investigative journalism. Most stakeholders agreed that international 
donors play an important role in ensuring the freedom of the press. At the same time, stakeholders re￾affirmed the role that free press “critically monitors processes in society and the political system65” and the 
risk that if media is less free, the impact of the work of CSOs, such as CCC, will diminish. CCC has few 
staff members, only about five individuals, which reflects the typical staffing among CSOs in North 
Macedonia. If the media were diminished, CSOs would need to rely on their internal communications, 
marketing, and dissemination capacities, which tend to be limited. 
Stakeholders agree that weak rule of law is a significant constraint to CCC’s work and 
progress towards transparency and anti-corruption. CSOs, the business sector, and public 
institutions agreed that enforcement of the rule of law when corruption is identified in public 
procedures, including penalties for officials and businesses that engage in corrupt practices, forms a
significant gap to enforcing the improvements in the Law of Public Procurement and moving towards a 
64 Interview CC10, Journalist, November 2021 65 Interview CC01, Journalist November 2021
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
34
more transparent and accountable government. CSOs feel frustrated when reporting corruption cases 
to the State Commission for Prevention of Corruption, because it “doesn’t have any outcome66” or “there 
were no court trials started as a result of our submissions67.” Public institutions reiterated this sentiment: “In 
the end, we do not see many results in the form of open court cases or people or companies sentenced.” The 
business sector confirmed that “the tenders are corrupted, we all know this, there is impunity, and no one is 
penalized68.” Some public institution stakeholders felt that there wasn’t support from authorities such as 
the prosecution office69. 
The business sector mentioned another fallout from the weak enforcement of the rule of 
law, pertaining to the health and vitality of small and medium enterprises (SMEs). The
business sector said that it is not uncommon for public institutions to neglect to pay companies or 
follow their contract agreement’s payment schedule. While companies will bring these cases to court, 
and at times win the case, there isn’t an enforcement mechanism to ensure companies get paid, or 
penalties enacted for non-payment or significantly delayed payment. As a result, chamber
representatives asserted that “This is a reason for the bankruptcy of a number of companies that are not 
liquid … only very rich companies can afford to wait for months or even years for payment.70” Delayed or non￾payment further demotivates SMEs to participate in public procurement and reduces the vitality of the 
private sector’s ability to engage in public works. 
Stakeholders agree that CCC’s work needs to continue and note risks to sustainability. All 
stakeholders agreed that CCC plays an important monitoring and accountability function around public 
spending that must continue. One CSO noted that the work needs to be constant, and progress is slow, 
but ongoing monitoring will ensure ongoing pressure that leads to change. There are still “very serious 
problems in public procurements, due to politically-based arrangements”, and transparency work is not 
achievable in a short period of time. CCC’s work comes at odds with donor approaches to projects, 
which are “time-limited and if the funding is discontinued, will be a serious problem71” in CCC’s or other 
CSO’s abilities to continue this role. The State Commission for Prevention of Corruption further 
emphasized this fact when commenting on CCC’s work: 
I have doubts about sustainability. It [CCC] is dependent on funds provided by national or international 
sources. A CSOs’ work is usually only temporary and stops the activities when the project funding is 
stopped. What if CCC stops? Who will continue this work? Because of this, I believe that this 
transparency measurement needs to be institutionalized and performed by a public institution. I am not 
sure which one but in this way, it is quite risky.
CEP project staff and stakeholders echoed the sentiment that, in an ideal world, CSOs would not need 
to play this role, which could be a function for a public institution. However, given the weak 
enforcement of rule of law and nascent capacity around transparency, stakeholders agreed that current 
conditions are not optimal for such an arrangement. One public institution suggested a more 
decentralized approach, where at least one person per institution has this role within their own 
66 Interview CC02, CSO, November 2021 67 Interview CC07, CSO, November 2021 68 FGD CC06, Business community, November 2021 69 Interview CC09, Public institution, November 2021 70 FGD CC06, Business community, November 2021 71 Interview CC02, CSO, November 2021
35 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
institutions and receives CCC training and coaching: “It is very important for institutions to have persons who 
would be in charge of integrity.”72
Finally, despite the concerns expressed about CCC’s sustainability to conduct this work indefinitely, CEP 
staff members said that CCC is skilled at acquiring funding and aligning it to their mission, rather than 
aligning its mission to donor agendas. In this way, CCC has been able to remain hyper-focused on its
mission, which contributed to cultivating its strong reputation as an expert in public spending 
transparency. However, stakeholders noted an opportunity for CCC to grow from a small CSO to a 
larger organization solidifying CCC’s status while expanding its reach. However, as a small CSO, CCC 
requires targeted organizational capacity development to expand and it is not clear that CCC is 
interested in growing its operations. 
A cross-section of stakeholders felt that CCC could engage more widely with the business 
sector. The public procurement stakeholder landscape includes public institutions issuing tenders, 
CSOs monitoring tenders, journalists disseminating corruption risks or flags, and businesses applying for 
and receiving contracts. Stakeholders felt that the business sector is generally treated as passive 
stakeholders within public procurement monitoring. For example, CCC conducts an annual survey of 
companies to understand their experiences with procurement and bring them to discussions and 
working groups, but this is the limit of CCC’s engagement. One CSO described the results of a lack of 
CCC’s engagement with the business community as follows: 
The business community has very little engagement, it is not active [in public procurement monitoring]. If 
a company loses a tender, it can’t learn why it lost because of a lack of connections. There is no attempt 
by companies to complain and this is because they are fearful that public institutions will punish them or 
call for inspection. 
The business sector echoed this fearful sentiment, stating that:
They are frightened to talk loudly, since as owners of the companies, they are afraid that authorities will 
send them to inspections and “find something wrong’ in their operations. This will lead to punishment or 
exclusion from future tenders or work with institutions. 
Stakeholders felt that these fears are valid, but that an opportunity exists for a greater role for the 
business sector, whether through organizations that represent business interests, such as chambers of 
commerce or business associations, or otherwise. During the FGD with business sector representatives, 
they shared that they are “not satisfied” with the role they play, which is “minimal in this process and needs 
to be improved significantly”73. Associations and chambers described how they organize regular press 
conferences and, during these, they comment on government corruption and non-transparency, but 
their issues and critiques could be more targeted and specific to bring these issues to light. 
72 Interview CC02, Public institution, November 2021 73 FGD CC06, Business community, November 2021
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
36
EQ 2 CONCLUSIONS
CCC has been effective in creating momentum, providing tools, and intervening in 
legislation to encourage increased transparency in public procurement, however more 
work remains. The findings demonstrate broad consensus that CCC has been an instrumental force in 
creating the necessary conditions for improved transparency in public procurement, but the 
improvements achieved to date have not sufficiently translated into a practice of transparent tenders. 
CCC has provided the tools and mobilized stakeholder support to use these tools, including the Index 
of Active Transparency and public procurement monitoring methodologies, to galvanize public pressure 
that leads public institutions to enact reforms. CCC’s work has led to increased awareness of 
transparency issues and some cases of changed practices within institutions. 
CCC has further positioned itself as a respected CSO with unique expertise, so much so 
that public institutions open their doors, extend invitations, and invite recommendations 
for legislative and regulatory reforms. Despite significant momentum and increased objective 
scores observed through the Active Transparency Index, the end-user of public procurements – the 
business sector – has not seen a significant change in tender procedures. The process of change, through 
public procurement monitoring, is lengthy and slow. Informants also perceive that public officials’ 
impunity and the lack of penalties for public spending corruption cases that do make it to court, have 
contributed to the slow pace of changes. Ongoing monitoring of public procurements is required, and it 
may need to remain with civil society for the time being, as North Macedonia’s institutions are not yet 
ready to adopt these accountability protocols. However, CCC has provided a host of tools and 
methodologies for a broader coalition of CSOs and stakeholders to engage in this mandate, effectively 
building the infrastructure for transparency monitoring. 
Monitoring public spending risks becoming ineffective without robust adherence to the 
rule of law and free media in North Macedonia. The findings noted that enforcement of rule of 
law about public procurement violations is inconsistent and is rendered potentially ineffective in flagging 
offenses. Furthermore, when other public officials observe that corruption cases do not result in 
penalties, there is little incentive to adhere to transparency and anti-corruption practices. Lack of law 
enforcement is a significant deterrent to the effectiveness of transparency and anti-corruption work. In 
addition, the findings indicate important concerns about the freedom of the press in North Macedonia. 
Media is a key avenue to ‘voice’ and flag potential corruption issues in public spending and, therefore, 
freedom of press to investigate and report issues without interference from business or political 
interests is imperative. Stakeholders emphasized a diminished role of investigative journalism practice 
and an increased presence of political or business influences in mainstream media. If freedom of 
traditional media outlets continues to diminish, CSOs in their roles as watchdogs will require additional 
capacity and resources for mass communication campaigns to raise public awareness of corruption 
cases. 
37 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
LESSONS LEARNED AND RECOMMENDATIONS
LESSONS LEARNED 
The following lessons learned resulted from the evaluation team’s key observations of CEP’s effective 
practices. 
Lesson Learned 1: Technical assistance that is in-person, client-identified, client-prioritized, process￾oriented, specific, concrete, adaptable, and modifiable is effective to ensure rapid adaptation and 
ownership, as well as sustainable in changing contexts. Short-term technical assistance by reputable 
experts with both context and country expertise led to effective results. 
Lesson Learned 2: Technical assistance that focuses on identifying gaps in expertise, knowledge, 
implementation, management and/or operations, and establishes simple steps relevant too short to 
medium processes and procedures to fill the gaps, adapt to change, account for contingencies, and 
prepare the foundation for future assistance is effective for all levels of support for all stakeholders.
Lesson Learned 3: Process-driven tools and methodologies can be applicable and replicable to other 
contexts, across public institutions and agencies, and across levels – top-down and bottom-up. CCC’s 
Index of Active Transparency presents a key example. 
Lesson Learned 4: Partners that are visible, independent, integrity-based, mission-focused, have 
explicit and adaptable expertise, and a longer-term presence have greater chances to gain entry into, and 
engage with, the government on legislation and regulatory issues. These partners are more likely to 
regularly receive invitations to working groups, expert roundtables, and other opportunities to provide 
recommendations to legislative or regulatory reforms, strategies, or new activities, as was evidenced in 
the CCC case study.
RECOMMENDATIONS FOR USAID FOR FUTURE PROGRAMMING
Recommendation 1: Strengthen public and independent institutions with roles related to 
transparency, anti-corruption, and rule of law, and state explicitly the roles and rules of engagement for 
each stakeholder. Specific examples of government and independent institutions or entities for 
continued, new and/or reinvigorated support include: Academy for Judges and Public Prosecutors; 
Agency for Protection of the Right to Free Access to Public Information; Anti-Corruption Platform; 
Cabinet of the Deputy Prime Minister for Anti-Corruption, Sustainable Development and Human 
Resources; Finance Intelligence Directorate; Local Courts; Ministry of Information Society and 
Administration; Ministry of Justice; State Audit Office; Municipal Councils; State Commission for 
Prevention of Corruption; and State Judicial Council. 
Recommendation 2: Strengthen the focus on outcome and impact indicators, targets, and results and 
be clear about expected changes in consultation with the government and the targeted public or 
independent institutions. Ensure the alignment of the activity/project objective indicator to the objective 
to ensure measurability, monitoring, and reporting. Ensure the selection of appropriate indicators to 
measure progress and success at output and outcome levels, including the measure of change. Activities 
should be clearly in line with national policies and plans. For example, include qualitative indicators on 
knowledge, attitudes, and perceptions across targeted groups, particularly citizens’ perceptions of 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 
38
government accountability, to measure change in awareness and changes in behavior against the 
institutions’ expectations, goals and strategies. 
Recommendation 3: Strategically select key interventions for systemic change, given the multiple 
layers within each area of transparency, anti-corruption, and rule of law. The Theory of Change should 
identify sectors and synergistic pathways toward measured and verified change, particularly rule of law 
interventions. 
Recommendation 4: Support long-term technical assistance (LTTA) with independent and neutral in￾house (in-person) international or national experts who are resilient to influence to play a role in 
facilitating and monitoring the implementation of USAID-funded institutional support. This could be a 
team of LTTA advisors in both government and independent institutions, particularly to facilitate the 
creation of independent compliance control mechanisms, such as USAID-funded independent CSOs or 
other mechanisms, to ensure impartiality. The country has systems in place, but require more 
concentrated efforts on implementation. For example, the Ministry of Information Society and 
Administration (MISA) could be supported with a LTTA advisor to ensure implementation of strategies, 
such as the introduction of the merit system in employment. Other important links in the corruption 
chain include courts and prosecutions through the Court Council and the Prosecution Council because 
they appoint judges and prosecutors. Therefore, an independent LTTA advisor could adopt the role of 
reviewing past performances and current procedural compliance of the Court Council and the 
Prosecution Council. A LTTA in the State Audit Office (SAO), for example, should provide assistance to 
enable it to become a fully legitimate, respected, and authoritative agency.
Recommendation 5: Support short-term technical assistance (STTA) that focuses on systemic 
processes that are cyclical or periodic, adaptable, replicable, and sustainable for CSO or government 
products or services. For example, the process of conducting citizen surveys on their perception of 
government accountability – with processes that are step-by-step including data collection, data analysis, 
monitoring, reporting, actions for improvement and learning (whole cycle), as well as a schedule for 
repeated surveys. The in-person, local or international STTA advisor with direct knowledge about the 
country’s political and governance narrative, should provide guidance and mentoring for specific, 
targeted, government-identified tasks. Promote a culture of evaluation, in which the government 
conducts internal and external assessments of their work for continuous improvements.
Recommendation 6: Include more comprehensive and inclusive engagement of a wide range of 
stakeholders, particularly public institutions – directly or indirectly – on transparency, anti-corruption, 
and rule of law interventions. This engagement includes youth, particularly given that the government 
adopted the LYPYP in January 2020, which obliges municipalities to involve youth in decision making 
processes. Strong support for the media, specifically investigative journalism, to monitor a range of 
areas, such as public procurement, should continue to enable the necessary public pressure that 
motivates institutions to change their practices, legislation, or regulations.
Recommendation 7: Strengthen youth and youth groups on transparency, anti-corruption, and rule 
of law issues, their leadership, and their communication skills. Identify change agents and monitor their 
capacity to conduct actions or engagements with relevant CSOs and local government. 
Recommendation 8: Continue support for open and transparent public procurement and 
contracting, in line with a recommendation from Transparency International’s 2020 Corruption 
39 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Perceptions Index report, to assist in the identification of conflicts of interest and opportunities for fair 
pricing to prevent the diversion of public resources 
Recommendation 9: Re-focus the support of CSOs towards monitoring, rather than implementation 
of governance activities; i.e., to act as government watchdogs and monitors of transparency, anti￾corruption, and rule of law that complement the CSOs’ and government’s work. Civil society should 
continue to facilitate citizen awareness on transparency, anti-corruption, and rule of law, and to facilitate 
two-way linkages between citizens and government on local issues for monitoring, policy development, 
and decision making.
Recommendation 10: Consider supporting CEP follow-on activities to ensure the application of 
technical assistance. Follow-on activities could include monitoring recommendations of Vulnerability to 
Corruption Assessments; monitoring the use of the online Open Data platform, particularly for citizens; 
continued support for amendments to the law on use of data from the public sector (Open Data law); 
and monitoring the municipalities with integrity plans.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 40
ANNEX A: EVALUATION STATEMENT OF WORK
C.1 SUMMARY OF PROJECT INFORMATION
SUMMARY OF PROJECT INFORMATION
ACTIVITY NAME CIVIC ENGAGEMENT PROJECT
Implementer East-West Management Institute
Contract # AID-165-C-16-00003
Total Estimated Ceiling of 
the Evaluated Contract 
(TEC)
$9,495,923
Life of Activity August 2016 – August 2021/2
Activity Geographic Regions Countrywide
Development Objective North Macedonia is a prosperous, self-reliant, and inclusive democratic society
USAID Office General Development Office
C.2 BACKGROUND
A. DESCRIPTION OF THE PROBLEM AND DEVELOPMENT HYPOTHESIS
The Civic Engagement Project (CEP) was designed and began implementation during a period in which
the political space was closing, and the civil society sector was confronted with many challenges. CEP
is one of several activities contributing to the “Improved Democratic Participation and Processes”
project. The problem statement for that project was: “A progressive constriction of competitive space
in the country is leading to a decline in voice, participation, inclusion, and accountability; worsening
ethnic tensions; and erosion of basic rights.” The Development Hypothesis for the project was: IF
citizen engagement is strengthened AND more inclusive checks on executive authority are functioning,
THEN democratic participation and processes will better connect state and society and create a more
responsive, accountable, and resilient political system.” The results framework for the project was:
41 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
The Development Hypothesis for the CEP when it began was, “IF North Macedonia’s CSOs, CSO 
alliances, and citizens, including youth, are provided the resources, incentives, and platforms for
coordinated capacity development that meets best national and international standards, THEN they will
be more capable of actively participating in North Macedonia’s democratic development, including
engaging in constructive dialogue with government partners at all levels of decision making.” The
original results framework for the CEP, which aimed to support the Improved Democratic Participation
and Processes Project, Intermediate Result (IR) 1.1, More Inclusive Citizen Engagement, is shown
below:
The original contract with EWMI for the Civic Engagement Project included two objectives: (1)
Strengthened influence of CSOs; and (2) Increased and sustained youth engagement. The first objective
included two components: (1) Strengthen civil society organization alliances; and (2) Strengthen CSOs.
The second objective included two components: (1) Strengthen youth engagement; and (2) Strengthen
youth activism.
A political crisis which began in 2015 gripped the country until mid-2017. A new government formed
on June 1, 2017 and began to pursue policies of EU accession and NATO membership, and opening up
the political space for citizen engagement.
In 2018, USAID/North Macedonia commissioned an independent evaluation of its Project Appraisal
Document (PAD) for the “Improved Democratic Participation and Processes” project and associated
IRs, including More Inclusive Citizen Engagement and Functioning Checks on Citizen Authority. The
evaluation recommended a rephrased problem statement to: “As the development of strong democratic
institutions is still in its infancy, citizens remain wary of the new government’s capacity to deliver
services and stimulate an enabling environment for employment creation, while continued ethnic
divisions remain perilous to national unity.” Among several recommendations, the evaluators
suggested that USAID should support public administration reform and judicial system enhancement,
while continuing to support and promote civil society.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 42
B. SUMMARY OF CIVIC ENGAGEMENT PROJECT
In January 2020, USAID/North Macedonia amended EWMI’s contract for CEP to de-emphasize civil
society capacity building, add direct support to help GONM institutions to become more transparent
and accountable to citizens, and emphasize anti-corruption, transparency, and rule of law, three key
priority reforms for North Macedonia’s EU accession. The rationale for the changes was an assumption
that the civil society sector was sufficiently strong and that with a more reformist government in place,
USAID should provide direct support to the Government of North Macedonia in areas that would
advance its EU accession and NATO membership. The new development hypothesis posed by EWMI
is: “IF North Macedonian public institutions, CSOs, CSO alliances, and citizens, including youth, are
provided the resources, cooperation mechanisms, and technical assistance for coordinated and long
term joint actions in the areas of rule of law, transparency, and anti-corruption, THEN they will be more
capable to contribute to North Macedonia’s accession and integration in EU and NATO.” The EWMI’s
new results framework for CEP is as follows:
After the contract modification, EWMI ended ten of the grants to CSOs that were not focused in areas
related to anti-corruption, rule of law, or transparency. It continued to support seven grantees that 
were focused in these high priority areas.
CEP has been implemented countrywide. The activity’s story map highlights activities by location.
Please see this link:
https://uploads.knightlab.com/storymapjs/14a3ab5dd5cf608596120bf6ae3288f9/test/index.html.
C. SUMMARY OF THE CEP’S MONITORING, EVALUATION, AND LEARNING (MEL) PLAN
There is an original MEL Plan dated November 14, 2016 reflecting the original CEP design and a
revised MEL Plan dated September 30, 2020 reflecting the amended design.2 The September 2020
MEL Plan includes both custom and standard F performance indicators to track performance at the
Goal, Result, and Sub-Intermediate result level (as per EWMI’s revised Results Framework for CEP
shown above).
43 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
C.3 SCOPE OF WORK
The contractor will conduct the evaluation of the Civic Engagement Project in line with the evaluation
questions, design and methodology below:
C.3.1 EVALUATION QUESTIONS (IN ORDER OF PRIORITY)
a) To what extent and how have CEP-supported CSOs helped to advance North Macedonia’s
EU accession in areas related to rule of law, transparency, and anti-corruption? What
advanced or impeded this approach to supporting the GONM in these areas?
b) To what extent and how has the sub-grantee, the Association of Citizens Center for Civil
Communications (CCC) increased transparency in public procurement? What lessons
learned from CCC’s work could be valuable and applicable to other efforts to advance
government transparency, anti-corruption, and rule of law?
C.3.2 EVALUATION DESIGN AND METHODOLOGY
This summative performance evaluation will utilize mixed methods including both quantitative and
qualitative research.
CSO Grantees Working on Rule of Law, Transparency, and Anti-Corruption: As mentioned
earlier, of the 17 multi-year CSO grants that were in place at the time of the January 2020 contract
modification, ten of those grants were ended at that time, while the seven that worked in the new 
priority areas were continued. This evaluation will focus on the seven CSOs that were supported in 
both phases of the CEP and take a deeper look at the activities of one of those CSOs, the Citizens 
Center for Civil Communications.
OUTCOME/
RESULT
ILLUSTRATIVE 
INDICATORS
SUGGESETD DATA 
SOURCES
SUGGESTED 
DATA 
COLLECTION 
METHODS
SUGGESTED 
DATA 
ANALYSIS 
METHODS
North 
Macedonia’s EU 
accession
advanced
• % of government interlocutors 
(of all those surveyed) who 
state that CEP-supported CSO 
assistance was helpful or very 
helpful
• % of government interlocutors 
(of all those surveyed) who 
state that CEP-supported 
CSOs brought new
information, knowledge, or 
skills that helped to inform 
policy or other governmental 
changes
• Description of how CSO- provided support brought new
information, knowledge, or 
skills to help inform policy or 
other governmental changes
The 7 CSO grantees
whose grants continued
after the January 2020
contract amendment;
GONM interlocutors
who were supported by
CSOs; EWMI CEP 
project staff
Closed-ended
surveys using
Likert scales; open￾ended surveys;
interviews
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 44
OUTCOME/
RESULT
ILLUSTRATIVE 
INDICATORS
SUGGESETD DATA 
SOURCES
SUGGESTED 
DATA 
COLLECTION 
METHODS
SUGGESTED 
DATA 
ANALYSIS 
METHODS
Transparency in
public 
procurement
increased
• Extent to which CCC 
public procurement 
monitoring reports
increased transparency in 
public procurement
• % change in Index of Active
Transparency (of ministries 
and municipalities) from 
year to year
• # of government policies,
procedures, or programs 
improved by CCC support
• Extent to which research 
on various public 
procurement topics has
increased transparency in 
public procurement
CCC staff; EWMI CEP
program staff; municipal 
and ministry officials
Case study; closed￾ended and open￾ended surveys; 
Index of Active
Transparency
C.4 EVALUATION SCHEDULE / DELIVERABLES
The contractor may follow the below schedule and must submit listed deliverables according to the
below timeline:
PROPOSED ACTIVITIES DELIVERABLES TIMELINE DUE
Preparation of the work plan and evaluation design Workplan 14 days after award
In-Briefing In-Briefing Within 5 days after workplan is 
accepted
Preparations for data collection
Data Collection
Mid-term Briefing Mid-Term Briefing Within 30 days after In-Briefing
Data Analysis
Final Presentation Final Presentation 14 days after Mid-Term Briefing
Report Writing Draft Report 14 days after Final Presentation
USAID review and approval of draft report; USAID shares draft 
report with IP
Incorporate USAID comments and prepare/submit final report Final Report 14 days after USAID approval of 
Draft Report
C.5 TEAM COMPOSITION
The evaluation team will consist of three experts and a translator. The team will be led by the
international development and evaluation expert, with support from the local experts, as specified in
Section F.4.
45 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
ANNEX B: EVALUATION METHODOLOGY
To answer the two EQs framing this CEP summative evaluation, the evaluation team conducted a mixed￾method performance evaluation, integrating quantitative and qualitative data collection and analysis 
methods, and encompassing both primary and secondary data analysis. The mixed-method evaluation 
design allows the evaluation team to present a comprehensive account of CEP’s results, including trends 
in good practices across CSO programming and lessons learned about the effectiveness of a particular 
model or activity. 
• For EQ1, the evaluation focused on the work of the CEP Partners who implemented activities 
on transparency, anti-corruption, and rule of law, including EWMI, ADI, FOSM, 
Metamorphosis, and the CSO grantees supported by these partners. The evaluation team used 
KIIs, FGDs, and an online survey. Respondents included USAID, CEP staff, CEP partners, CEP 
grantees, and beneficiaries, including central government, local government municipalities, 
journalists from independent media outlets, justice sector beneficiaries and citizens. 
• For EQ2, the evaluation conducted an in-depth case study, focused solely on the transparency 
in public spending work of the CEP CSO grantee, CCC. In following a case study approach, the 
evaluation team sought to form a multi-faceted understanding of CCC’s work and its effects, 
which included collecting information and perceptions from representatives of institutions 
CCC intended to influence, as well as key players in the public procurement process. The case 
study included a workshop with CCC staff, KIIs with central government institutions, 
journalists from independent media outlets, CSOs engaged by CCC, and an FGD with business 
community representatives. 
MIXED METHODS APPROACH
The evaluation team applied a utilization-focused evaluation (U-FE) approach.74 A U-FE approach is 
based on the principle that an evaluation should be judged by its usefulness to its intended users and that 
integrating users into the evaluation process generates more relevant findings that are likely to be used. 
The U-FE features that were applied through this evaluation included75: 1) a design workshop with 
USAID; 2) a kick-off and input gathering session with CEP; 3) a validation workshop; and 4) producing a 
digestible format for conveying results of the evaluation through an infographic. 
U-FE FEATURES: IMPAQ held a 1.5-hour virtual Design Workshop with USAID on September 27, 
2021 to inform the final design. The workshop used participatory facilitation techniques to identify 
expectations; refine the EQs and develop sub-EQs; prioritize the issues of greatest interest and learning 
needs; and garner input to the evaluation’s approaches, stakeholders, and tools. Following the 
workshop, the team held a Kick-Off Meeting on October 6, 2021 with CEP staff to introduce them to 
74 Michael Quinn Patton (2008). Utilization-Focused Evaluation, 4th edition, Thousand Oaks, California: Sage.
75 The evaluation team originally planned to conduct a co-interpretation workshop; however, this was not possible 
due to time constraints in the evaluation process, limited availability for such a workshop with CEP, and limited 
availability from USAID. A co-interpretation workshop typically is several-hours long. 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 46
the evaluation, gather their inputs, coordinate outreach, and foster buy-in to the process. Outputs from 
these two events were used to develop the final work plan and methodology plan.
Following the completion of data collection and in support of determining findings, conclusions, and 
recommendations, the team held a virtual 1.5-hour validation workshop on December 7, 2021 with CEP 
staff. During this workshop, the evaluation team presented the preliminary findings, sourced inputs to 
clarify respondent trends, and sought information to further contextualize the findings. In addition, the 
CEP team provided additional feedback and clarifications for the evaluation team’s consideration in 
written form. In addition to the validation workshop, the team conducted a 1-hour preliminary findings 
presentation for USAID on December 6, 2021. During this presentation, USAID also provided inputs 
into the findings, including further contextualization. The inputs from these two meetings were 
subsequently weaved into the reports’ findings, conclusions, and recommendations. 
DATA COLLECTION
Data collection took place over 23 days between October 18 and November 23, 2021. While data 
collection was originally slated to end by November 19, the local municipal elections and ensuing 
political crisis delayed the evaluation team’s progress, particularly with government stakeholders. The 
evaluation team approached CEP stakeholders and participants at all levels to enable triangulation of 
information from diverse perspectives. 
KEY INFORMANT INTERVIEWS – EQ1
The evaluation team developed semi-structured KII guides, to collect perceptions from key CEP 
stakeholders including the consortium, the five CSO grantees, select central-level GONM interlocutors, 
and journalists engaged through CEP-supported activities. The evaluation team conducted a total of 36 
virtual KIIs across these stakeholders as demonstrated in Table B1 below. 
Table B1: EQ 1 KIIs by Stakeholder Type Respondents
USAID/North Macedonia 2
CEP Project Staff 16
Civil Society Organizations, Grantee Staff 8
Government of North Macedonia Interlocutors 8
Journalists from Independent Media Outlets 2
Total 36
FOCUS GROUP DISCUSSIONS – EQ1
The evaluation team developed separate FGD guides for three primary stakeholder groups, including 
engaged citizens, trial monitors trained by a CSO under the rule of law programming, and sub-CSOs 
(CSO networks). The evaluation team conducted a total of remote 4 FGDs, representing 17 
participants, across these stakeholders as demonstrated in Table B2 below. 
47 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
Table B2: FGDs by Stakeholder Type Respondents 
Citizens (2 FGDs) 7
Trial Monitors 7
Sub-CSOs 3
Total 17
ONLINE SURVEY – EQ1
The evaluation team constructed and administered a 17-question online survey to provide quick, 
comparable data from government municipality stakeholders engaged through CEP’s partners or 
grantees. Because of the diversity of activities implemented under CEP in relation to the three thematic 
areas and variety of municipality stakeholders engaged, the survey focused on the quality of the 
assistance or support from CEP grantees and partners; the value of the support in bringing new 
information, knowledge, and skills; and the situational questions to assess views as it relates to 
transparency and anti-corruption. The survey used a mix of fixed-response, open-ended, and situational 
judgement questions.
The survey was constructed on the SurveyMonkey platform. On October 25, it was distributed to a list 
of 88 government stakeholders’ emails compiled by the CEP partners and grantees from their 
monitoring data. The survey was kept open for a three-week period and closed on November 12. The 
evaluation team pursued a multi-pronged approach to maximize response rates including two email 
reminders from the evaluation team, one email reminder from the CEP Chief of Party, and follow-up 
phone calls by the evaluation team to non-respondents to request responses. 
As a result of these efforts, the survey received a total of 42 responses, or a 48% response rate. 
CASE STUDY – EQ2 
To respond to EQ2, the team conducted a case study about the CSO CCC, in relation to their CEP 
grant “Public Spending for Public Good” focused on increasing transparency in public procurement. A 
case study is a detailed examination of a single person, group, organization, community or event that is 
exemplary or provides insights around a trend. It involves data collection from multiple sources and 
points of views to create a narrative of the case. In the case of CEP, CCC was one of the CSO grantees 
that was considered to be doing exemplary work in transparency and from whom lessons could be 
learned to apply to future governance programming. 
The evaluation team’s original design for the case study included a methodology called outcome mapping 
(OM). OM is a participatory and complexity-aware method that focuses particularly on identifying the 
behavioral and relationship changes among “boundary partners”76 that lead to bringing about observed 
76 “Boundary partners” refers to institutions in a project’s direct sphere of influence, whose behaviors and actions 
the project seeks to change. For the case study, boundary partners would include GONM interlocutors who CCC 
worked with directly and GONM ministries and municipalities that CCC sought to influence a change in practice.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 48
outcomes and traces how a project influenced those behavioral and relationship changes. The 
methodology implicates a series of workshops with the evaluand (in this case, CCC), typically lasting a 
full-day or two half-days at the start of data collection and at the end. At the start of data collection, the 
method seeks to review in-depth the project theory of change, identify key stakeholders, map influence 
pathways, articulate behavior change goals, and link to long-term goals. At the end of data collection, the 
method seeks to co-interpret the data collected and draw conclusions in a participatory way. As a 
result, significant buy-in and engagement is required from the case being investigated for the success of 
the methodology. Data is collected from “boundary partners” around progress markers of behavior 
change identified through these workshops. 
The evaluation team was unable to implement the methodology as originally design and in fidelity to the 
approach. The evaluation’s timeframe coincided with a heavy workload for CCC due to the political 
climate. In addition, CCC is a small organization with only five staff, three of whom were engaged on the 
CEP grant, and multiple competing priorities. The proposed time commitment of the OM methodology 
was extensive for their availability, they were only able to provide us with 1.5 hours of their time for the 
envisioned series of workshops. As a result, the evaluation team adapted the approach to a more 
traditional case study but including some of the principles OM, such as reviewing collaboratively 
reviewing the project theory of change and areas of influence, as well as collecting data from 
representative boundary partners identified with CCC. 
The evaluation team began with an abridged 1.5-hour workshop with the three CCC staff to understand 
the theory of change, intended areas of influence, and “boundary partners”. With boundary partners 
identified, the evaluation team developed KII guides for each stakeholder and one FGD guide based on 
the information gathered through the workshop on CCC’s areas of influence, approaches towards the 
stakeholder, and intended goals for each stakeholder group. The boundary partners identified are those 
with roles in the public procurement transparency landscape, including included public institutions who 
issue tenders, journalists from independent media who publish about potential corruption, CSOs who 
monitor government spending, and the business community who participate in procurements. 
The evaluation team conducted 10 KIIs and one FGD with them. The CCC case study thus included 
perspectives from a total of 17 respondents as exhibited in Table B3. 
Table B3: EQ 2 Case Study by Stakeholder Respondents
Association of CCC, Workshop 3
Public Institutions, KIIs 5
Media/Journalists, KIIs 2
Civil Society, KIIs 3
Business Community, FGD 4
Total 17
49 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
 EVALUATION DESIGN MATRIX
EVALUATION QUESTION SUB-EVALUATION QUESTIONS INDICATORS TO SUPPORT EQ RESPONSE DATA SOURCES / DATA COLLECTION 
METHODS
1. To what extent and how have 
CEP-supported CSOs helped to 
advance North Macedonia’s EU 
accession in areas related to rule 
of law, transparency, and anti￾corruption? What advanced or 
impeded this approach to 
supporting the GONM in these 
areas?
o What were CEP’s GONM 
intervention approaches at 
the central government level 
and the local government 
level (municipalities, etc.) and 
how effective was intervening 
at these different levels to 
affect broader GONM change 
as it relates to anti￾corruption, transparency, or 
rule of law? 
• Percent of GONM interlocutors who state 
CEP and CEP-supported CSOs brought new 
information, knowledge or skills that helped 
inform policy or procedure changes 
• Percent of non-CEP respondents perceive 
improved GONM transparency/ accountability
• Desk Study of secondary data including CEP 
program documents, CEP partner documents, 
5 CSO grants, monitoring, evaluation, and 
learning (MEL) data, and CSO research
• Key Informant Interviews (KIIs) with US 
government officials, CEP staff and partners, 5 
CSO grantees, selection of GONM 
interlocutors, and journalists
• Focus Group Discussions (FGDs) with CSO 
network members (sub-CSOs) and engaged 
adult citizens o What were CEP’s most • Online Survey with GONM interlocutors 
effective entry points (people, 
committees, groups, 
networks, etc.) to advocate, 
monitor, or support GONM 
to change behavior, policies, 
or laws relevant to rule of 
law, transparency, or anti￾corruption? 
o How did CEP’s activities 
affect the lives of engaged 
citizens of North Macedonia, 
perceptions of the GONM, 
and ongoing civic 
participation? 
o How did CEP’s various rule 
of law, transparency and anti￾corruption activities 
implemented by grantees and 
partners complement or 
support one another to 
achieve outcomes? 
o What contextual factors 
(political, COVID-19, others) 
helped or hindered the 
success of CEP’s activities?
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 50
 EVALUATION DESIGN MATRIX
EVALUATION QUESTION SUB-EVALUATION QUESTIONS INDICATORS TO SUPPORT EQ RESPONSE DATA SOURCES / DATA COLLECTION 
METHODS
2. To what extent and how has 
the sub-grantee, CCC increased 
transparency in public 
procurement? What lessons 
learned from CCC’s work 
could be valuable and applicable 
to other efforts to advance 
government transparency, anti￾corruption, and rule of law?
o How did CCC partner with 
GONM institutions and 
foster an openness to 
collaborate on transparency 
in public procurement? 
• Percent change in the Index of Active 
Transparency among GONM institutions 
engaged through the grant (2018 results as 
baseline) and CCC’s contributions to changes 
• Number of GONM policies, procedures, & 
programs improved by CCC support
• Case study with outcome mapping (OM) 
techniques – includes CCC staff, GONM 
ministry and municipal officials, and journalists 
engaged through CCC programming
• Index of Active Transparency 2018, 2019, and 
o How sustainable are CCC’s 2020 reports
activities and how can USAID 
or others build upon the 
work to support continuity? 
51 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
ANNEX C: DATA COLLECTION INSTRUMENTS
CEP EVALUATION KEY INFORMANT INTERVIEW GUIDE
INTRODUCTORY SCRIPT FOR ALL KIIs
Thank you for taking the time to talk to me. My name is [INSERT NAME]. I am from IMPAQ 
International, a U.S.-based international policy research, technical assistance, and evaluation company. 
IMPAQ is conducting a summative, final evaluation of the USAID’s Civic Engagement Project or CEP. 
CEP is a five-year USAID-funded activity in North Macedonia designed to support USAID’s broader 
efforts to “Promote a Prosperous, Well-Governed, and Resilient North Macedonia Committed to 
Euro-Atlantic Integration.”1 Specifically, CEP seeks to “strengthen constructive engagement by civil society 
organizations (CSOs) on policies and issues targeted at anti-corruption, transparency, and rule of law, and 
sustain youth engagement in public life in North Macedonia.” The project is implemented by prime 
implementer East-West Management Institute (EWMI) in close collaboration with local consortium 
partners (1) Foundation for Internet and Society “Metamorphosis” (Metamorphosis); (2) Association 
for Democratic Initiatives (ADI); (3) National Youth Council of Macedonia (NYCM); and (4) 
Foundation Open Society – Macedonia (FOSM). 
The purpose of the evaluation is to examine to what extent and how CEP-supported Civil Society 
Organizations (CSOs) helped to advance North Macedonia’s European Union (EU) accession in 
regard to improving transparency, anti-corruption, and rule of law. I would like to ask you questions 
related to your experience with the project. 
This interview will last approximately one hour (1h). With your permission, I would like to audio 
record this session for report writing and analysis purposes only. The recordings will be destroyed 
once we complete our analysis. Is this okay with you? 
□ Yes 
□ No
In the evaluation report, your name will not be mentioned except in an annex in a list of people 
consulted and this list will only be shared with the donor (USAID). The evaluation team will make 
every effort to protect the anonymity and confidentiality of our discussion and hope you will feel 
comfortable providing honest feedback on your experiences and points of view. Participation is 
voluntary; also, you may decline to respond to our questions or end the interview at any time. Do 
you have any questions? Can we start now?
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 52
CEP EVALUATION KEY INFORMANT INTERVIEW GUIDE
CIVIL SOCIETY ORGANIZATIONS
EVALUATION QUESTION 1
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
QUESTIONS
1. How is your CSO supporting the CEP project – what skills does it bring to the project?
2. What specifically were you doing at the central level and at the local municipal level – and how did 
you differentiate the type of support and interventions between the two levels?
3. Which approach is more effective – targeting the central or the local level – and why? Which 
approach is more sustainable and why?
4. One of the indicators was for the government to receive new information, knowledge, or skills to 
inform policy or procedure change – how is this indicator being met?
5. What did you see were the biggest changes in the government/courts as a result of your 
organizations’ work?
6. How did you work with other CSOs (networks or individual CSOs)? How did your work training 
CSOs or coordinating networks of CSOs contribute to your grants’ achievements in supporting 
[rule of law, transparency, anti-corruption]? 
7. What were the most effective entry points to influence the government or courts to change their
behavior or practice (e.g., individuals, committees, groups, networks)? What were the most 
challenging entry points? Is it worth persevering with the challenging entry points?
53 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
8. Which actors, leverage points or structures were the most willing/effective partners and what 
where the factors facilitating or limiting their engagement?
9. How were citizens engaged? What were their perceptions of their engagement and how it affected 
them?
10. How has the project affected the wider community, such as people not directly involved in CEP 
activities? 
11. In general, how do citizens view the transparency and accountability of the government? Is there a 
difference in perception between the citizens your grant engaged directly and non-project-engaged 
citizens in their views of the government?
12. Did you work or collaborate with the other CSOs being funded by CEP? How do you believe this 
affected CEP’s ability to reach the project’s outcomes? 
13. How have the right people (government, individuals, CSOs, networks) been targeted for support? 
Are there any missed groups?
14. How did your CSO take gender and social inclusion into consideration or was it not considered 
important?
15. What internal or external factors helped or hindered the project’s success? (e.g., politics, COVID￾19, etc.) How can you prove “success”?
16. Was your organization part of the consultative process for the drafting of the EU progress report? If 
yes, did activities and achievements within CEP were communicated to the EUD?
17. What were the most effective activities for sustainability? What else could have been done? Who 
will continue to work on this activity?
18. In going forward, what would be useful for USAID/North Macedonia to be aware of for future 
programming in transparency/anti-corruption/rule of law?
19. Do you have anything else to add?
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 54
CEP EVALUATION KEY INFORMANT INTERVIEW GUIDE
USG AND CEP PROJECT STAFF
EVALUATION QUESTION 1
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
QUESTIONS FOR USG RESPONDENTS:
1. How do you think the grantees and their interventions work cohesively together and support 
each other, coordinate with each other? 
2. Modification 06 occurred in January 2020, before some 2020 EU-NATO decisions were officially 
announced. For example, the government-EU discussions in which the EU member states granted 
permission for accession negotiations in 2020 and North Macedonia’s membership into NATO in 
March 2020? What was the reason for the pivot in 2020, in addition to pre-empting the 
announcements? Were there strategic decisions or mitigating/external circumstances that led to 
the pivot?
3. After the pivot, CEP was expected to engage with, and provide technical assistance to high-level 
government institutions, such as the office of the Deputy Prime Minister for Anti-Corruption. 
What is your confidence level in the CEP implementers to work at this level, given that EU 
accession has challenging for any country seeking membership – and it’s a very long process 
external to project contributions?
4. CEP grantees work at the municipality and citizen level, so the project goes beyond direct EU 
accession support and includes wider benefits for communities, so what are your expectations at 
the grassroots level?
5. How do you think CEP has adapted and developed during this pivot? Did you expect challenges, 
and if so, how did USAID/NM support the project to address these challenges?
55 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
6. Detail how your expectations have been met, or otherwise, since the pivot, to date? 
7. What now defines your success for the project to August 2022 apart from meeting its goal and 
indicator outcomes? [Strengthen CSOs and their alliances]
8. What is your view on how COVID-19 or other external factors affected the overall project’s 
progress and performance? 
QUESTIONS FOR CEP PROJECT STAFF:
1. What was your direct work on transparency/anti-corruption/rule of law? (What were you 
contracted to do?)
2. Can you explain the challenges for you and your grantees in addressing the pivot towards only 2 
components and the focus on transparency, anti-corruption, and rule of law?
3. How has CEP addressed these challenges? How has CEP adapted to the change? Was the change 
of programming strategic, opportunities, or mitigating/external circumstances? 
4. How has USAID supported the pivot? And how have the implementers supported the grantees? 
And how have the grantees supported the sub-CSO? Where is the weakest link? Where is the 
strongest link?
5. What contextual factors (political, COVID-19, others) helped or hindered the success of CEP’s 
activities?
6. How did the project address gender and inclusivity issues?
7. How did CEP’s various rule of law, transparency and anti-corruption activities implemented by 
different grantees and partners complement or support one another to achieve outcomes?
8. Which actors/participants (government, CSOs, individuals, etc.) have been the most engaged in 
CEP – i.e., advocates of change?
9. How do citizens know about CEP? What is their view of the program? What are the indirect 
benefits or concerns for them? 
10. What evidence is there that citizens support transparency, anti-corruption, and rule of law – and 
what differences are there between CEP-supported citizens and non-CEP-supported citizens?
How did the 5 grantees partner with GONM institutions? Were any deemed more effective than 
others? How did your organization partner with GONM institutions? 
11. How do you think the grantees and their interventions work cohesively together and support 
each other, coordinate with each other? 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 56
12. How have linkages between people and the government increased – has people’s involvement in 
local politics increased? Do you see any difference in citizen engagement at the central versus 
municipal levels? 
13. Where are citizen leaders and change agents in society now? How, in what way, under what 
circumstances? What makes an effective leader? What evidence do you have that citizens have the 
capacity, or otherwise, to become effective leaders? What external factors could challenge their 
roles as leaders? 
14. How satisfied are you with the grants process and the type of grants being awarded? What were 
the challenges/complaints? What determined the success of the grants? What determined the 
weaknesses? 
15. How confident do you feel about the goal of the program and meeting the component objectives 
and outcomes? 
16. What strategies, approaches, tactics worked and what didn’t, and why? What did you learn from 
the process? 
17. What are the program’s plans for sustainability? Are they feasible and achievable – in the short 
term and the long term? What would you change to improve the program? 
18. How sustainable are the 5 CSO activities and how can USAID or others build upon the work to 
support continuity?
19. CEP had a large focus on training CSOs to monitor government activities at the central and 
municipal levels. Do you believe this was an effective strategy? How did the non-funded CSOs 
apply the learnings from the trainings? What were some notable successes? 
20. How confident are you that the work your CSO did, in combination with other CEP-supported 
CSOs, will contribute transparency/anti-corruption/rule of law? How have these project activities 
helped the country to gain EU accession regardless of which party is governing the country? For 
example, what accountability processes are in place? Or is there still a lot to do?
21. In going forward, what would be useful for USAID/North Macedonia to be aware of for future 
programming in transparency/anti-corruption/rule of law?
22. Do you have anything else to add?
57 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP EVALUATION KEY INFORMANT INTERVIEW GUIDE
GONM INTERLOCUTORS
EVALUATION QUESTION 1
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
Central / Municipality
Location
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
QUESTIONS
1. What’s your involvement with the CEP? How is the CEP supporting you? What groups in the CEP 
project did you engage with? [management – i.e., EWMI]. How effective is their support?
2. What were your original views of CEP and have they changed over time? 
3. Does this type of program lead to building transparency, anti-corruption, and/or rule of law?
4. What are the strengths of the program? What are the challenges for you? 
5. How has the work with [insert relevant grantee or partner] been helpful for you? Has it changed 
your department’s knowledge and attitudes? In what ways? What elements of the training or 
assistance are you using/implementing? 
6. Explain the relationships – and their changes, if any – between the municipalities and central 
government – before the program and now? How is this likely to change – positive or otherwise –
to the end of the project in August 2022?
7. What contextual factors (political, COVID-19, others) helped or hindered the success of CEP’s 
activities?
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 58
8. Do you know what communities need in order to support the government to gain EU accession 
and overall improvements in increasing transparency and rule of law and decreasing corruption? 
9. What were the most effective ways [insert grantee or partner] worked with your office and/or 
staff? Would you recommend to USAID that these ways of working with the government should 
continue or do you think a different way is more effective? 
10. What are your department’s / your municipality’s plans – for sustainability of program processes, 
components, or activities – which ones will you adopt and why, or how will you modify what you 
have learned from the program? 
11. What was the most significant change you saw in your department/your municipality from your 
engagements with [insert relevant grantee or partner] or CEP more broadly? 
12. In going forward, what would be useful for USAID/North Macedonia to be aware of for future 
programming in transparency/anti-corruption/rule of law?
13. Is there anything else you would like to add? 
59 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP EVALUATION KEY INFORMANT INTERVIEW GUIDE
MEDIA
EVALUATION QUESTION 1
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
CSO associated with
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
QUESTIONS
1. How familiar are you with the project activities? And its terminology – transparency, anti￾corruption and rule of law?
2. What type of training did you receive? How relevant was it? What did you gain from it to use in 
your work?
3. What has been your role, or the role of media outlets in particular, to support the project?
4. Are awareness-raising campaigns the best way to inform citizens about transparency, anti￾corruption and rule of law? Do you know whether there a significant change in the number of 
court trials related to corruption?
5. Overall, what is view of a program like CEP?
6. What are the drivers/enablers of change in your opinion, and is CEP addressing change effectively? 
7. What actors have been the most effective in facilitating change? 
8. What gaps exist in addressing the program objectives to increase transparency, anti-corruption, or 
rule of law? What approaches are best to meet those gaps? 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 60
9. How did the program address gender and inclusivity issues? 
10. What are the priority areas that require continued support? 
11. Is the government capable of moving towards EU accession regardless of which party is governing 
the country? For example, is there accountability processes in place?
12. What would be useful for USAID/North Macedonia to be aware of for future programming in 
transparency/anti-corruption/rule of law?
13. Is there anything else you would like to share with us?
61 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP EVALUATION FOCUS GROUP DISCUSSION GUIDE
INTRODUCTORY SCRIPT FOR ALL FGDs
Thank you for taking the time to talk to me. My name is [INSERT NAME]. I am from IMPAQ 
International, a U.S.-based international policy research, technical assistance, and evaluation company. 
IMPAQ is conducting a summative, final evaluation of the USAID’s Civic Engagement Project or CEP. 
CEP is a five-year USAID-funded activity in North Macedonia designed to support USAID’s broader 
efforts to “Promote a Prosperous, Well-Governed, and Resilient North Macedonia Committed to 
Euro-Atlantic Integration.”1 Specifically, CEP seeks to “strengthen constructive engagement by civil society 
organizations (CSOs) on policies and issues targeted at anti-corruption, transparency, and rule of law, and 
sustain youth engagement in public life in North Macedonia.” The project is implemented by prime 
implementer East-West Management Institute (EWMI) in close collaboration with local consortium 
partners (1) Foundation for Internet and Society “Metamorphosis” (Metamorphosis); (2) Association 
for Democratic Initiatives (ADI); (3) National Youth Council of Macedonia (NYCM); and (4) 
Foundation Open Society – Macedonia (FOSM). 
The purpose of the evaluation is to examine to what extent and how CEP-supported Civil Society 
Organizations (CSOs) helped to advance North Macedonia’s European Union (EU) accession in 
regard to improving transparency, anti-corruption, and rule of law. I would like to ask you questions 
related to your experience with the project. 
This focus group will last approximately one hour (1h). With your permission, I would like to audio 
record this session for report writing and analysis purposes only. The recordings will be destroyed 
once we complete our analysis. Is this okay with you? 
□ Yes 
□ No
In the evaluation report, your name will not be mentioned except in an annex in a list of people 
consulted and this list will only be shared with the donor (USAID). The evaluation team will make 
every effort to protect the anonymity and confidentiality of our discussion and hope you will feel 
comfortable providing honest feedback on your experiences and points of view. Participation is 
voluntary; also, you may decline to respond to our questions or end the interview at any time. Do 
you have any questions? Can we start now?
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 62
CEP EVALUATION FOCUS GROUP DISCUSSION GUIDE
SUB-CSO GROUPS
EVALUATION QUESTION 1
SSG LCDF Mar 18–Feb 21 Public policies (municipal councils & local 
CSOs)
Transparency
CSO networks, such as the Platform of CSOs in the Fight Against Corruption; People Powered; and the Fiscal 
Accountability, Sustainability, and Transparency Network. FGDs will seek to understand the value and 
effectiveness of the network structure in moving forward actions related to transparency, anti-corruption, and 
rule of law, the tangible achievements of these networks, the perceived successes of a network-driven intervention 
vs. an individual CSO-driven intervention, the differences in national-level vs. local-level engagement, the best 
practices in managing a network that intervenes with GONM, build consensus around the right entry-points 
engage with the GONM, and document challenges. 
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
CSO associated with
Location
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
RESPONDENT
#: SEX NAME OR 
INITIALS POSITION SUB-CSO (IF DIFFERENT FROM 
ABOVE)
01
02
03
04
05
06
07
08
63 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
WARM UP (IF RELEVANT)
First, I’d like everyone to introduce themselves. 
QUESTIONS
We will begin with questions about the CSO network structure.
1. Can you comment on the benefits or challenges of a network of CSOs (against individual CSOs) –
i.e., the effectiveness and value of a network-driven structure? How do you build consensus 
around the right entry points to engage with the GONM? What were some of the documentation 
and reporting challenges, if any? What are some “best practices” in managing a network that 
intervenes with GONM?
2. For a general, over-arching question, what is your overall impression of the CEP activity in terms 
of its objective to support the government in increasing transparency and rule of law and reducing 
corruption?
3. What do you think are the biggest drivers of corruption and lack of transparency in this country? 
4. Do you think there are different drivers for men and women, and for young and old, and for 
urban and rural people? Have these drivers been met by CEP activities?
5. How are CEP strategies tailored at the local level? Are the strategies relevant, responsive, and 
appropriate? What are the differences in national-level versus local-level engagement? How does 
the government – local and national – assist or lead with strategies to prevent corruption?
6. How have CEP interventions been adopted at community/citizen-level to date? How are the 
interventions contributing to increasing transparency and rule of law and reducing corruption? Is 
CEP addressing all the factors that lead to corruption and lack of transparency in an effective way 
– expand on your response? 
7. How would you define “success’ in terms of this project? [How will you know when you have 
succeeded?] Has there been evidence of CEP strategies that have actually started to be 
“successful” and, for example, discouraged corruption? 
8. How are you monitoring your progress and is it effective? [Where does the data come from, is it 
reliable, on time, relevant, and useful?]
9. What are the tangible benefits of this project – the tangible benefits of the CSO network? Of all of 
the specific CEP prevention interventions that you have been involved in, or are knowledgeable 
about, which ones have met your expectations? How have they differed from your expectations 
(exceeded or have not met expectations)? 
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 64
10. How would you change anything to improve the CSO network structure – i.e., how would you 
improve CEP so that it could advance the goals of the CSO network? Can you comment on any 
gaps that the CSO network could have addressed but didn’t?
CLOSING THE FGD 
Thank you for participating. This has been a very successful discussion and your opinions will be a valuable 
asset to our evaluation of the Civic Engagement Project. We hope you have found the discussion 
interesting too. I would like to remind you that any comments featuring in this report will be anonymous 
and confidential. 
Thank you.
65 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP EVALUATION FOCUS GROUP DISCUSSION GUIDE
ENGAGED CITIZENS AND JUSTICE SYSTEM
EVALUATION QUESTION 1
INTERVIEW INFORMATION
Date
Interviewer Name
Primary Notetaker Name
RESPONDENT INFORMATION
Respondent Name
Respondent Title
Respondent Organization
CSO associated with
Location
Sex of Respondent □ Male
□ Female
□ Other
□ Refuse
RESPONDENT
# SEX NAME OR 
INITIALS POSITION SUB-CSO (IF DIFFERENT FROM 
ABOVE)
01
02
03
04
05
06
07
08
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 66
WARM UP (IF RELEVANT)
First, I’d like everyone to introduce themselves. 
QUESTIONS
1. How is the CSO supporting you with activities on transparency, anti-corruption, or rule of law?
2. What activities are the most effective and why? 
3. Describe how citizens are able to engage with and monitor their governments’ work to ensure 
transparency. Describe the re-dress, remediation, or complaints mechanism if cases of corruption 
or lack of transparency occur?
4. How confident are you in speaking up against violations? In the process? Is there a better process 
or improvements in the process that you would suggest?
5. Are there different issues for different groups – men and women, children, young and old, urban 
and rural, marginalized people? And how does the project help to address issues from different 
groups? Are there groups that the project is missing, that could be included?
6. How will you know when you have succeeded? Has there been evidence of the project strategies 
that have actually started to be “successful” and, for example, discouraged corruption? 
7. What are the tangible benefits of this project for you and your community?
8. How does the government – local and national – assist or lead with strategies to prevent 
corruption? Do you feel that the government is actively engaged?
9. What is your view on your municipality’s level of transparency and corruption following your 
engagement with the project? What is your view of transparency and corruption at the central 
level of government? Where have you seen greater progress, if any? 
10. How would you change anything to improve the project or their strategies? 
11. Is there anything you would like to add?
67 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
CEP EVALUATION ONLINE SURVEY
GOVERNMENT MUNICIPALITIES
INTRODUCTORY TEXT
The purpose of CEP working with government municipalities is to advance good governance at the local 
level by strengthening local governments’ resilience to corruption, conflict of interest, transparency, and 
rule of law through support and activities, such as the introduction of integrity systems in municipalities; 
increased civil participation, especially of vulnerable categories of citizens. 
SURVEY QUESTIONNAIRE
QUESTIONS RESPONSE SKIP RULE
SECTION 1: DEMOGRAPHIC INFORMATION
1.1. Title
1.2. Department
1.3. Municipality
1.4. Sex 
0 = Female
1 = Male
2 = Other
3 = No response / Do not want to answer
1.5. Age 
0 = Under 30
1 = 31-50
2 = Over 50
3 = No response / Do not want to answer
SECTION 2: WORK STATUS
2.1. Years of service in the 
government? (Single Answer)
0 = One year or less
1 = 1-5 years 
2 = More than 5 years 
SECTION 3: TRANSPARENCY
3.1 Have you been trained by the 
project to publish data or 
information in different formats for 
different users?
0 = I have been trained
1 = I have not been trained yet
2 = No response 
If you have not been 
trained, skip to 3.4
3.2 Comment on the quality and 
usefulness of the training.
3.3 Comment on what you do now 
in your job that you did not do 
before – e.g., tangible or behavioral 
changes you have 
observed/experienced due to 
project training or engagement 
[i.e., new/amended policy, a shift in 
attitudes, change in practice, etc.]
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 68
QUESTIONS RESPONSE SKIP RULE
Please describe the change as much 
as possible
3.4 How satisfied are you with the 
training support provided to you by 
the project
1. Not 
at all
2. A 
little
3. 
Mostly
4. 
Very
5. I 
don’t 
want to 
answer
3.5 Has your municipality opened 
up its datasets or information 
repositories as a result of the 
project’s support?
0 = No
1 = Yes
2 = I don’t know 
SECTION 4: CORRUPTION
4.1 Have you been trained by the 
project on preventing corruption
or provided with 
resources/support on preventing 
corruption?
0 = I have been trained or engaged
1 = I have not been trained or engaged yet
2 = No response 
If you have not been 
trained, skip to 4.5
4.2 Comment on the quality and 
usefulness of the training or 
engagement.
4.3 Comment on what you do now 
in your job, after training or 
engagement with the CSO, that 
you did not do before.
4.4 How satisfied are you with the 
training and support provided 
to you by the project
1. Not 
at all
2. A 
little
3. 
Mostly
4. 
Very
5. I 
don’t 
want to 
answer
4.5 Have your prepared a 
municipality Integrity Plan?
0 = No
1 = Yes, but it is still in progress
2 = Yes, and it is completed 
3 = Yes, and it is formally adopted
4 = I don’t know 
5 = I don’t want to answer
If you have not been 
trained, skip to 5.1
4.6 Comment on the quality and 
usefulness of the Integrity Plan
SECTION 5: DILEMMA QUESTIONS
5.1 Promotion Dilemma
There is a possibility for your 
promotion that comes with a 
certain condition. What would you 
choose to do? From the following 
scenarios, which one do you relate 
to the most? (Single Answer)
a. A higher position that supports my family 
but I have to pay the hiring officer a fee 
to influence the result to get the job.
b. A lower paid job even though I know that 
the person who got the promotion paid 
to influence the result, but I am not going 
to say anything because I might lose my 
job
c. I didn’t get the promotion. The person 
who was promoted paid a fee to 
influence the result, so I am going to 
report it to the appropriate authority.
69 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
QUESTIONS RESPONSE SKIP RULE
d. I don’t know what to do because I don’t 
have confidence in the system. 
5.2 Corruption Dilemma
When do you think it is justifiable 
for a woman not to report a case 
of corruption to the police?
1 = Never, she should always report it.
2 = If she is in danger from the police because 
they are corrupt, and they may harm her.
3 = A woman should not report a case of 
corruption to the police, but she should go to 
another authority or CSO/NGO that will 
support her.
4 = She should always keep quiet for her own 
safety.
5.2 Corruption Dilemma
When do you think it is justifiable 
for a man not to report a case of 
corruption to the police?
1 = Never, he should always report it.
2 = If he is in danger from the police because 
they are corrupt, and they may harm him.
3 = A man should not report a case of
corruption to the police, but he should go to 
another authority or CSO/NGO that will 
support him.
4 = He should always keep quiet for his own 
safety.
SECTION 6: OPINION
6.1. Which of the following 
statements do you most agree 
with?
1. North Macedonia will gain membership to 
the EU within the next 5-10 years.
2. North Macedonia will not gain membership 
to the EU within the next 5-10 years.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 70
ANNEX D: SOURCES OF INFORMATION
BIBLIOGRAPHY
CAFT (2021, July). Civic Engagement Project in North Macedonia, Final Narrative Report: Fair for One, 
Fair for All. Coalition All for Fair Trials.
Center for Civil Communications (2021, February). 2020 Monitoring of Public Procurement, Report No. 
34 (January-June). USAID.
Center for Civil Communications (2019, May). 2019 Monitoring of Public Procurement, Report No. 33 
(July-December). USAID.
Center for Civil Communications (2020, May). Index of Active Transparency 2020. USAID.
Coalition of Civil Associations: All for Fair Trials (2019, June). 2019 Annual Report for the Work of the 
Coalition: Annual Narrative and Financial Report for the Work of the Coalition for 2019.
EWMI (no date). Civic Engagement Project in North Macedonia, Scope of Work, Konekt: Strengthening 
the Financial Intelligence Directorate of the Government of North Macedonia. East-West Management 
Institute.
EWMI (2021). Civic Engagement Project in North Macedonia, Scope of Work, Association for 
Democratic Initiatives (ADI): East-West Management Institute.
EWMI (2021). Civic Engagement Project in North Macedonia, Scope of Work, Foundation for Internet 
and Society Metamorphosis. East-West Management Institute.
EWMI (2021, October). Civic Engagement Project, Quarterly Report #20, July-September. East-West 
Management Institute.
EWMI (2021, August). Civic Engagement Project, Year 6 Work Plan. East-West Management Institute.
EWMI (2021, July). Civic Engagement Project, Quarterly Report #19, April-June. East-West Management 
Institute.
EWMI (2021, April). Civic Engagement Project, Quarterly Report #18, January-March. East-West 
Management Institute.
EWMI (2021, January). Civic Engagement Project, Quarterly Report #17, October-December. East￾West Management Institute.
EWMI (2020, October). Civic Engagement Project, Quarterly Report #16, July-September. East-West 
Management Institute.
EWMI (2020, September). Civic Engagement Project, Year 5 Work Plan. East-West Management 
Institute.
EWMI (2020, September). Revised Civic Engagement Project, Monitoring, Evaluation, and Learning Plan 
(MELP). East-West Management Institute.
EWMI (2020, July). Civic Engagement Project, Quarterly Report #15, April-June. East-West Management 
Institute.
71 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
EWMI (2020, May). Civic Engagement Project in North Macedonia, Scope of Work Modification: My 
Money, My Responsibility-Participative Budgeting in the Municipalities in Macedonia. Finance Think. East￾West Management Institute.
EWMI (2020, April). Civic Engagement Project, Quarterly Report #14, January-March. East-West
Management Institute.
EWMI (2020, February). Civic Engagement Project in North Macedonia, Scope of Work Modification, 
Coalition All for Fair Trials (CAFT): Fair for One, Fair for All. East-West Management Institute.
EWMI (2019, November). Civic Engagement Project, Technical Proposal – for Modification #06. East￾West Management Institute.
EWMI (2019, April). Civic Engagement Project, Quarterly Report #10, January-March. East-West 
Management Institute.
EWMI (2018). Civic Engagement Project in North Macedonia, Scope of Work, Coalition All for Fair 
Trials (CAFT): Fair for One, Fair for All. East-West Management Institute.
EWMI (2018). Civic Engagement Project in North Macedonia, Scope of Work, Local Community 
Development Foundation Shtip (LCDF): Effective Mechanism for Civic Participation. East-West 
Management Institute.
EWMI (2017, May). Civic Engagement Project in North Macedonia, Scope of Work, Finance Think: My 
Money, My Responsibility-Participative Budgeting in the Municipalities in Macedonia. East-West 
Management Institute.
Finance Think (2020, October). Civic Engagement Project in North Macedonia, Final Narrative Report: 
My Money, My Responsibility-Participative Budgeting in the Municipalities in Macedonia. Finance Think 
Institute for Economic Research and Policy. USAID.
Finance Think (2020, June). Town Hall Meetings: A Tool Kit for Including Citizens in the Process of 
Participatory Budgeting, Study No. 28, Finance Think Institute for Economic Research and Policy. 
USAID.
Foundation for Internet and Society Metamorphosis. (2021, July). Assessment of Good Governance in 
North Macedonia and the region through the Openness Index.
Foundation for Internet and Society Metamorphosis. (2020, June). Assessment of Good Governance in 
North Macedonia and the region through the Openness Index.
LCDF (2021, March). Civic Engagement Project in North Macedonia, Final Narrative Report: Effective 
Mechanism for Civic Participation. Local Community Development Foundation Shtip.
Transparency International (2021). Corruption Perceptions Index 2020. 
OSCE (2021). Manual for Local Youth Councils. Organization for Security and Co-operation in Europe, 
Mission to Skopje.
UNODC (2018). Manual on Corruption Surveys: Methodological Guidelines on the Measurement of 
Bribery and Other Forms of Corruption through Sample Surveys. United Nations Office on Drugs and 
Crime.
USAID/North Macedonia (2021). North Macedonia Business Ecosystem Project: Quarterly Progress 
Report April-June. The Palladium Group.
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 72
USAID/North Macedonia (2020). Country Development Cooperation Strategy (CDCS): September 
2020-September 2025.
USAID/North Macedonia (2020). 2020 Democracy and Governance Survey – North Macedonia.
USAID/North Macedonia (2020, January). Solicitation, Offer and Award, Activity for Civic Engagement 
(ACE), East-West Management Institute Inc., AID-165-C-16-00003, Modification #06.
USAID/North Macedonia (2019). Democracy and Governance Survey – North Macedonia: Survey 
Report, Results of survey wave in 2019.
USAID/North Macedonia (2019, July). USAID/North Macedonia Gender Analysis Report. Banyan Global.
USAID/North Macedonia (2018). E-Newsletter, USAID/North Civic Engagement Project in Macedonia 
(CEP), Issue IV, July-December.
USAID/North Macedonia (2017). E-Newsletter, USAID/North Civic Engagement Project in Macedonia 
(CEP), Issue II, April-December.
USAID/North Macedonia (2016, August). Solicitation, Offer and Award, Activity for Civic Engagement 
(ACE), East-West Management Institute Inc., AID-165-C-16-00003.
DATA SOURCES – INTERNATIONAL INDEX MEASURES 
CORRUPTION
• https://www.transparency.org/en/cpi/2020/index/mkd
• https://risk-indexes.com/global-corruption-index/
• https://seldi.net/publications/reports/corruption-assessment-report-macedonia-2016/
• https://dksk.mk/en/
• http://www.antikorupcija.mk/eng/index.php?id=2&mv=1
• https://www.coe.int/en/web/greco/-/north-macedonia-publication-of-fifth-round-compliance-report
• https://ti-defence.org/gdi/countries/north-macedonia/
TRANSPARENCY
• https://cea.org.mk/budget-transparency-index-in-municipalities-in-north-macedonia/?lang=en
• https://aspi.mk/en/home-en/ - The Agency for Protection of the Right to Free Access to Public 
Information
• https://opennessindex.actionsee.org/country/north-macedonia
• https://www.internationalbudget.org/open-budget-survey/country-results/2019/macedonia
73 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
• https://www.internationalbudget.org/wp-content/uploads/OBS2015-CS-Macedonia-English.pdf
• https://www.oecd.org/gov/regulatory-policy/governance-indicators.htm
RULE OF LAW
• http://www.esem.org.mk/en/index.php/what-we-do-now/3274-rule-of-law-index-for-2020.html
• https://worldjusticeproject.org/rule-of-law-index/country/2021
• https://freedomhouse.org/countries/nations-transit/scores
• https://freedomhouse.org/country/north-macedonia
• http://info.worldbank.org/governance/wgi/Home/Reports
• https://bti-project.org/en/reports/country-dashboard/MKD
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 74
ANNEX E: KEY INFORMANTS AND FOCUS GROUP 
PARTICIPANTS 
EVALUATION QUESTION 1: KEY INFORMANT INTERVIEWS
Stakeholder Position Organization
F USG CEP Contracting Officer’s Representative USAID/North Macedonia
M USG Monitoring & Evaluation Officer USAID/North Macedonia
M CEP Project CEP Chief of Party 
East-West Management Institute 
(EWMI)
F CEP Project CEP Former Chief of Party EWMI 
M CEP Project CEP Former Deputy Chief of Party EWMI 
F CEP Project Senior Grants Manager EWMI 
M CEP Project Rule of Law Specialist EWMI 
F CEP Project M&E and Communications Specialist EWMI 
F CEP Project Former M&E Officer EWMI 
F CEP Project Code of Ethics Specialist (Technical Advisor) EWMI 
M CEP Project Governance Specialist (Technical Advisor) EWMI 
F CEP Project Director Metamorphosis 
M CEP Project Program Manager Metamorphosis 
F CEP Project Former M&E Officer Metamorphosis 
F CEP Project Director
Association for Democratic 
Initiatives (ADI) 
F CEP Project Program Coordinator Foundation Open Society 
Macedonia (FOSM)
F CEP Project Program Assistant Foundation Open Society 
Macedonia (FOSM)
F CEP Project International Affairs & Policy Coordinator
National Youth Council of 
Macedonia (NYCM/NMS)
F CEP CSO Grantee Executive Director Finance Think 
F CEP CSO Grantee Executive Director Finance Think 
F CEP CSO Grantee Executive Director Local Community Development 
Foundation, Shtip (LCDF)
M CEP CSO Grantee Project Manager LCDF
F CEP CSO Grantee Executive Director
Coalition of Civil Associations All 
for Fair Trade (CAFT)
M CEP CSO Grantee Legal Analyst CAFT
M CEP CSO Grantee Project Coordinator Transparency International
M CEP CSO Grantee Journalist Transparency International
M GONM Central Head of Cabinet
Cabinet of the Deputy Prime 
Minister (DPM)
M GONM Central Special Legal Advisor Cabinet DPM
F GONM Central State Advisor Ministry of Information, Society, & 
Administration (MISA)
F GONM Central Judge Civil Court, Skopje
F GONM Central Judge Criminal Court, Skopje
75 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
EVALUATION QUESTION 1: KEY INFORMANT INTERVIEWS
Stakeholder Position Organization
M GONM Central Court President Basic Court, Kavadarci
F Public Institution Head of Section for Strategic Planning & 
Integrity
State Commission for the 
Prevention of Corruption (SCPC)
M Public Institution Officer for Strategic Planning & Integrity SCPC
M Media Journalist TV Telma 
M Media Journalist Prizma
EVALUATION QUESTION 1: FOCUS GROUP DISCUSSIONS
FGD 1: 
RULE OF LAW
Justice System Direct Beneficiaries:
Trial monitors, capital & regional, conducting 
criminal & civil cases.
Trained by CSO Grantee CAFT.
Skopje – 3
Veles – 2
Ohrid – 1
Shtip - 1
• 3 males
• 4 females
FGD 2: 
TRANSPARENCY
Justice System and Engaged Citizens: 
Members of 3 Advisory Bodies. 
CSO Grantee LCDF established 8 Advisory Bodies
Kocani – 1
Shtip – 1
Zrnovci – 1 
• 1 male
• 2 females
FGD 3: 
TRANSPARENCY
Sub-CSO Groups:
CSOs from a network of CSOs.
Trained by CSO Grantee LCDF.
Probishtip – 1
Shtip – 1
Vinica - 1
• 0 males
• 3 females
FGD 4: 
ANTI-CORRUPTION
Engaged Citizens:
Citizens participating in Public Dialogues.
Organized by CSO Partner ADI.
Gostiva – 1
Kavadarci – 1
Kumanovo – 1
Shtip – 1
• 1 male
• 3 females
EVALUATION QUESTION 2: WORKSHOP – CENTER FOR CIVIL COMMUNICATIONS
Stakeholder Position Organization
M CSO Grantee President CCC 
F CSO Grantee Program Director CCC 
F CSO Grantee Project Coordinator CCC 
EVALUATION QUESTION 2: KEY INFORMANT INTERVIEWS
Stakeholder Position Organization
F GONM Chief Communication Officer Ministry of Defense 
M GONM Communication Officer Ministry of Defense 
F GONM Head, Public Procurement Division Public Enterprise Macedonian 
Forests
F GONM Head, Normative Affairs, Training & 
International Relations Division
Public Procurement Bureau
M Public Institution Commission Member
State Commission for the 
Prevention of Corruption (SCPC)
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 76
EVALUATION QUESTION 2: KEY INFORMANT INTERVIEWS
Stakeholder Position Organization
F CSO President
Echo Educational Humanitarian 
Organization, Shtip 
M CSO President
Association Initiative of 
Unemployed Intellectuals, Vinica
F CSO Program Director Stanica P.E.T. Prilep
M Media Editor-in-Chief 360 Stepeni
F Media Country Director & Chief Editor
Prizma, Investigative Online 
Platform & Balkan Investigative 
Reporting Network
EVALUATION QUESTION 2: FOCUS GROUP DISCUSSIONS
FGD 1: 
BUSINESS 
COMMUNITY 
Employer associations & chambers of 
commerce
Skopje • 2 males
• 2 females
77 | SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT USAID.GOV
ANNEX F: ANALYSIS OF CEP KEY PERFORMANCE INDICATORS
Performance Indicator Tracking Table
Performance Indicator FY 2021 Target QR20 Actuals
Number of CSOs receiving USG assistance engaged in 
advocacy interventions. 25 27
R 1: Strengthened CSOs engagement with and oversight of public institutions
Sub-IR 1.1: GoNM and CSOs implement joint activities to advance and/or implement 
urgent reforms related to RoL, anti-corruptions and/or transparency
Number of joint activities undertaken by CEP partners and 
grantees in cooperation with government institutions. 8 23
Number of trained representatives of the local CSOs and 
LGUs on the provisions of the new Law on Prevention of 
Corruption
60 22
Sub-IR 1.2 Improved public policies and their implementation related to RoL, anti￾corruption and transparency
Number of Public Policies introduced, adopted, repealed, 
changed, or implemented consistent with citizen input 20 2
R 2: CSO Sustainability Strengthened
Sub IR 2.1 Increase number of CSOs working together within the civil society sector
Number of synergies between CEP partners, grantees and 
CSOs related to RoL, anti-corruption and transparency. 8 1
Number of citizens participating in the events related to 
targeted areas 680 311
Number of media articles on CSOs activities. 500 299
Sub-IR 2.2 Increase number of CSOs partnerships across sectors (government and 
private sector and public)
Number of advocacy initiatives undertaken by CEP grantees 
together with different stakeholders based on a joint 
partnership
29 8
R 3: Increased and sustained youth engagement in North Macedonia
Sub-IR 3.1 Increased Leaderships/Advocacy Skills of the Youth
Percentage of CEP-supported youth that remains active in 
public life beyond the initial CEP support. 70% N/A
Number of CEP-supported youth that acquires new 
knowledge, skills, and networking opportunities. 25 N/A
Sub-IR 3.2 Intensified youth-led grass root level engagement
Number of young people participating in the initiatives 
undertaken by YCEA winners. 100 17
Number of young people that participate in LEAD Fellows' 
activities. 500 N/A
R 4: More transparent and accountable government institutions
USAID.GOV SUMMATIVE PERFORMANCE EVALUATION OF CEP, EVALUATION REPORT | 78
Performance Indicator Tracking Table
Performance Indicator FY 2021 Target QR20 Actuals
Sub-IR 4.1 Increased GoNM responsiveness to citizens
Number of trained civil servants/data officers under relevant 
state institutions 120 22
Number of new open data sets released by each supported 
institutions 5 0
Number of stakeholders engaged on creation of new Open 
Government Partnership National Action Plan (OGP NAP) 
2021
50 1
Sub-IR 4.2 Strengthened ethical competence of public servents, and strengthened 
mechanisms to support "professional ethics"
Number of training and/or capacity building activities 
conducted with CEP assistance 24 9
Number of representatives participating at CEP supported 
trainings and/or workshops 15 71