October 2014
This evaluation was made possible by the support of the American People through the United States Agency
for International Development (USAID).The contents of this evaluation are the sole responsibility of its
authors Keri Culver, Edna Bonilla Seba, Paola Benítez and Management Systems International and do not
necessarily reflect the views of USAID or the United States Government.
Final Performance Evaluation of the
Public Policy Program
FINAL PERFORMANCE
EVALUATION:
PUBLIC POLICY PROGRAM
Management Systems International
200 12th Street South, Suite 1200
Arlington, VA 22202
www.msiworldwide.com
November, 2014
Contracted under AID-514-C-13-00003
USAID/Colombia EVAL (Evaluation and Analysis for Learning)
The cover image was created via www.Tagxedo.com as a frequency word cloud of the content of
the Findings/Conclusions/Recommendations table (Annex 1) compiled from this evaluation.
DISCLAIMER
This evaluation was made possible by the support of the American People through the United
States Agency for International Development (USAID).The contents of this evaluation are the
sole responsibility of its authors Keri Culver, Edna Bonilla Seba, Paola Benítez and Management
Systems International and do not necessarily reflect the views of USAID or the United States
Government.
CONTENTS
Acronyms......................................................................................................................................... i
Executive Summary........................................................................................................................ ii
Evaluation Purpose & Evaluation Questions.................................................................................. 1
Project Background......................................................................................................................... 2
Evaluation methods & limitations .................................................................................................. 4
Design.......................................................................................................................................... 4
Sampling...................................................................................................................................... 4
Respondent samples ................................................................................................................ 4
Activity Sample ....................................................................................................................... 4
Methods....................................................................................................................................... 5
Document review..................................................................................................................... 5
In-depth interviews.................................................................................................................. 5
Case studies ............................................................................................................................. 5
Data analysis ............................................................................................................................... 5
Limitations .................................................................................................................................. 5
Findings, Conclusions & Recommendations.................................................................................. 6
1. Relationships and Influence between the GoC, AECOM and USAID................................ 6
2. International Best Practices.................................................................................................. 8
3. PPP’s effectiveness ............................................................................................................ 10
4. Quality of PPP’s Technical Assistance.............................................................................. 14
5. Institutional Strengthening................................................................................................. 16
Additional Findings: Project Management ............................................................................... 18
i
ACRONYMS
CONPES Consejo Nacional de Política Económica y Social; GoC regulatory document
EVAL
GoC
Evaluation and Analysis for Learning
Government of Colombia
INCODER Colombian Rural Development Institute
MARD Ministry of Agriculture and Rural Development
PMP Performance Management Plan
PPP Public Policy Program
PWS Program Work Statement
SOW Statement of Work
TA Technical assistance
USAID U.S. Agency for International Development
ii
EXECUTIVE SUMMARY
EVALUATION PURPOSE
The purpose of the evaluation was to extract institutional strengthening and policy reform lessons
from the experience of the USAID-funded Public Policy Program (PPP). Results will inform future
Mission project design and implementation efforts.
PROJECT BACKGROUND
The USAID-funded Public Policy Program, implemented by AECOM, helped the Government of
Colombia (GoC) develop policy and build institutional capacity from 2010 to 2014. PPP provided
technical expertise to form, adopt and support the implementation of public policies, focused on
the following thematic pillars: Land and Rural Development, Victims, and Consolidation. It also
supported the following cross-cutting issues considered critical to each pillar: mechanisms at the
territorial level; sustainable livelihoods and access to finance; environment, biodiversity and
climate change; and gender.
DESIGN, METHODS AND LIMITATIONS
The evaluation design was systematically qualitative. The team reviewed activity documentation
to understand PPP’s interventions, clients, modes of operation, accomplishments and challenges.
The team then aligned the many policy and institutional strengthening contracts executed against
the higher-order objectives they sought to address. The team interviewed a select set of GoC
clients; USAID Contracting Officer’s Representative and pillar-based Technical Monitors; and
AECOM staff and consultants. Five case studies highlight lessons learned and best practices.
The team addressed the following challenges:
(1) PPP was executed in an environment where priorities, political relationships, and
USAID’s own targets and goals were in constant flux. This complexity and dynamism
means that the multiple facets and perspectives around PPP’s achievements can be
contradictory or unclear, even after extensive data collection and analysis.
(2) Some actors (key GoC stakeholders, AECOM staff and contractors, and interested parties
at USAID) have changed positions since the project was implemented. Some individuals
sought for interview were unreachable; and
(3) The absence of a Performance Management Plan and baseline data for the complex
activities meant the team had to rely on the Program Work Statement (PWS) as a
performance measurement framework. FINDINGS AND CONCLUSIONS
PPP generally achieved its objectives. Many of its implementation approaches should be integrated
into future USAID/Colombia policy reform initiatives. AECOM recruited and managed a highly
competent staff and an unusually large group of consultants, managed GoC relations with aplomb,
was very highly regarded by the GoC, and achieved a great deal. While administrative matters held
the project back at first, these issues were largely overcome in time. USAID/AECOM relations
were productive, though rocky and stressful at times, with a compressed agenda (from five to three
years, eight months for implementation) and demanding, politically contentious thematic areas.
3
PPP experience indicates technical assistance in Colombia is most effective when it links senior
GoC officials with senior consultants as a committed change team over the significant time periods
required to affect change. Expatriate assistance is helpful to integrate international best practices,
but this is most effectively accomplished through a thoughtful integration of Colombian expertise
and professional linkages in adapting those practices to the complex Colombian context. USAID
can most effectively achieve its policy reforms when those targets are aligned with GoC priorities
and when those policies are realistically scaled: fewer GoC entities, focused institutional reform
or establishing institutions from scratch, linking national and regional levels, etc.
Policy reform projects – entering as they do into inner government sanctums and touching on
important national (and minor individual) political interests –require USAID sensitivity in
implementation. USAID must clearly define its policy priorities, based on its Country
Development and Cooperation Strategy; understand which of these coincide with GoC interests
through direct and ongoing USG/GoC dialogue; and work with the implementing partner to
develop strategies to achieve them. After that, a light touch to USAID intervention may be called
for, enabling the IP and GoC units to navigate the sensitive processes. For this scheme to work,
the IP must invest heavily in proactive communication so that USAID can be kept abreast of
developments and to build IP/USAID confidence and collaboration. The importance of adept
diplomacy and effective project management cannot be overstated.
Themes for intervention must be selected carefully. USAID should consider periodically (perhaps
annually) convening a project advisory committee – including GoC, thematic experts and civil
society leaders – to revisit the policy agenda, discuss challenges and plan for the coming period.
Designating themes as “cross-cutting” can be fatal, particularly if not aligned with expert staff, IP,
GoC and USAID commitment and budgetary resources. This has been the fate with themes such
as “environment” and “gender” in many projects and it occurred with PPP, too. If environment,
for example, is important, make it an objective unto itself – or recognize that attention will falter.
Policy projects often are thought of as finite, seeking to have a quick impact and then leave. In
contrast, PPP demonstrates how effectiveness must be seen as navigating a river of local policy
and political dynamics for change. Many of the talented staff came from prior USAID efforts and
many of the themes began in prior projects and continue with later projects (such as work with
victims, reconciliation, and land restitution to support peace.) Thus, while accountability and
results orientation remains paramount, success must be considered in the context of this longer
effort to support peace in Colombia. It should be managed by USAID as a portfolio of medium￾term investments that need prolonged tending over time rather than quick purchases with an iron
grip. All stakeholders involved in each policy effort need to examine realistically the obstacles and
potential timelines of a given effort, and plan accordingly.
In this context, the Mission should integrate any specific policy projects into the rest of its work.
Thus, firewalls between sectors, between policy and institutional strengthening and policy change,
and between national dynamics and reaching local government units must be carefully managed
by USAID and communicated to implementing partners to achieve maximum efficiency and
impact.
1
EVALUATION PURPOSE &
EVALUATION QUESTIONS
EVALUATION PURPOSE
The purpose of this performance evaluation is to extract lessons from the experience of the
USAID-funded Public Policy Program (PPP) regarding institutional strengthening and policy
reform in Colombia. Key questions on government-level relationships, appropriate management
structures within USAID and the implementer, quality and effectiveness are intended to provide
information on policy reform and institutional strengthening processes and results, to improve
future Mission project design and implementation efforts.
The evaluation is particularly opportune in that further policy-related activities are being designed
and solicitations issued at the time of writing. As a result, the evaluation should be useful for the
program and technical offices for whom policy efforts are current or imminent.
This evaluation also meets the USAID requirement to conduct at least one performance evaluation
of activities that are larger than average (in dollar terms) for a given Mission Development
Objective.
EVALUATION QUESTIONS
The evaluation questions around which this evaluation was based are as follows. (For a full
description, please see the evaluation’s Statement of Work in Annex V.)
A. To what extent were international best practices introduced vs. relying on Colombian
expertise and models? How effective was the approach?
B. To what extent did AECOM’s technical assistance feed into policy development and
adoption?
C. How effective was PPP in assisting the GoC to adopt policy changes?
D. What was the level of quality in assistance provided?
E. What characteristics of AECOM’s work supported institutional strengthening most
effectively and which were less useful?
2
PROJECT BACKGROUND
PPP was intended to provide technical management expertise to form, adopt and implement public
policies and GoC structural reforms designed to break the cycle of poverty that leads marginalized,
poor communities in areas with weak state presence to undertake illicit crop production and illegal
activities, fueling violence and the further degradation of Colombia's social, economic and
environmental resources. AECOM supported a wide variety of GoC policy initiatives, including,
but not limited to, land reform, access to financial services, labor and vulnerable populations.
PPP was designed at a time when peace negotiations and a potential end to conflict did not seem
to be an immediate possibility. The design was focused on general policies addressing important
issues of application to all Colombians, intended to improve the economic and social conditions
of vulnerable Colombians through effective rights-based institutional presence.
In its first phase, from October 2010 until March 2012, PPP worked through six components:
1. Land policy;
2. Access to finance;
3. Policies toward conflict-affected populations;
4. Support for decentralization and improved public economic management;
5. Labor issues; and
6. Environmental policy.
As the political climate in Colombia changed over time, so did USAID objectives in the
development of public policies and the urgency in program implementation. USAID decided to
pursue policy areas critical to consolidation of good governance throughout the country, many of
them specifically related to new GOC policies and the peace agenda. This shift in USAID
objectives was expressed by the modification of the implementation contract concluded in
September 2012, providing additional resources as well as accelerating implementation. Contract
Modification 3 (March 1, 2012) changed the results established within the Performance Work
Statement (PWS) to a new three-pillar structure: Land Policy, Victims Policy, and Consolidation
Policy. The modification also replaced key personnel and realigned the budget.
PPP attempted to focus on strategic interventions and promoting synergies across a wide range of
GoC and USAID priorities and regional programs. It supported a number of cross-cutting issues
considered critical to each pillar to ensure integration of policy implementation at the regional
level. These included budgeting and coordination mechanisms at the territorial level; sustainable
livelihoods and access to finance; environment, biodiversity and climate change; and gender.
PROGRAM IDENTIFICATION DATA
Program Title: Public Policy Program (PPP)
Award Number: AID-514-C-11-00001
Award Dates: October 2010 – October 2015, accelerated to May 2014
Funding: USD $22,487,555, increased to $27,387,555
Implementing Organization: AECOM
COR: Norberto Martinez
3
The GoC demands and needs that drove the PPP model were complex. The GoC needed to
establish the Victims’ and Land Restitution Units, ensure their capacity to implement new
legislation and policy frameworks, and improve coordination with departmental and municipal
governments. Regional offices were still being created and staff trained while having to process
massive numbers of restitution and reparations claims. Similarly, the Consolidation & Territorial
Reconstruction Unit was still hiring personnel in the regions and faced major challenges to
respond rapidly to community needs and coordinate service provision from different GoC agencies
and local governments. Provision of social services, housing, and support for income generation to victims and
communities in regions affected by conflict remained a challenge. Despite significant budget
allocations identified in the Victims’ CONPES regulatory document and the revised National
Consolidation Plan, GoC service delivery did not respond effectively to the needs of victims, ethnic
minorities, and consolidation zones. Novel funding and service delivery methods were needed to
reach vulnerable populations and regions devastated by decades of internal conflict. The GoC had
to eliminate rigidities that restricted service provision, create conditions for communities to build
their own infrastructure, and provide services that took into account gender and ethnic disparities
as well as victims’ physical, psychological and social rehabilitation needs. To help GoC institutions reach their goals, PPP:
1. Supported the Victims’, Land Restitution, and Consolidation Units, and INCODER and
MARD to strengthen their operations;
2. Worked with institutions to coordinate across government and to extend from national
policy formulation to implementation at regional and local levels; and
3. Harmonized sectorial policies to promote coordinated, effective programming for priority
populations and regions.
With frequent changes to its mission, and the output-oriented nature of the PWS structure (PPP
never had a PMP), PPP seems to have been focused on executing its many deliverables. PPP served
as a vehicle to meet the diverse policy demands of the GoC and various technical offices within
USAID. Although international consultants and subcontracts to Colombian firms were also used,
PPP chiefly met demand by issuing contracts to individual Colombian consultants to work
individually, or in teams, to complete assigned deliverables. Over the course of the contract, PPP
issued contracts to over 400 individuals and firms, with some receiving more than one contract.
PPP closed during the course of the evaluation.
4
EVALUATION METHODS & LIMITATIONS
Design
The evaluation design was systematically qualitative. PPP’s work to support policy and
institutional strengthening was complex, with multiple stakeholders and potential spoilers, in a
context of evolving political will and interests. Program effects are nationwide and diffuse, making
quantitative study less helpful than probing the how and why. The evaluation therefore included
in-depth, semi-structured interviews, document review, and case studies.
Resultant data are necessarily subjective: each respondent is likely to have a unique perspective
on the topic under discussion. To be able to review those variances and weigh the values placed
on them, the evaluation design is based on triangulating sources. For each case study, for example,
perspectives were sought from GoC, USAID, and AECOM staff and consultants.
Evaluation methods are described fully in Annex II, including the team’s Getting to Answers
matrix, describing how data were collected and analyzed to respond to the evaluation questions.
Sampling
Annex III includes the full list of sources consulted, and a disaggregation of the sample. The
samples were selected per the following parameters.
Respondent samples
The evaluation team conducted 36 in-depth interviews (19 women; 17 men) including 19 AECOM
staff and consultants, five USAID staff, and 12 GoC actors. Of the individuals consulted, 25 were
related to the case studies. To the extent possible, responses of individual interviewees are
anonymized in the report, to ensure candor. Activity Sample
In concert with USAID technical and program offices, the team selected a purposive sample of
PPP activities for the evaluation. Cases were classified by their predominant area of effort (policy
or institutional strengthening) and whether impressions of program results tended toward the
positive, or toward the negative. The interviews captured a fuller picture of each case and,
especially, its lessons learned. This sample includes the following case studies:
Table 1: Case studies
More positive Less positive
Policy effort Access to financial services Rural development law
Institutional strengthening Victims’ Unit Consolidation Unit
In addition, while data collection was underway, USAID requested that the evaluation team
examine a fifth case study, on income generation activities undertaken with the GoC.
5
Methods
Document review
The team reviewed work plans, quarterly reports, and PWS for the life of PPP. Deliverables for
each case study were also read closely. Each case study also required a review of policy and/or
institutional strengthening documentation.
In-depth interviews
The team triangulated data from in-depth interviews from with GOC, AECOM and USAID
perspectives. Interview guides were tailored for each respondent to answer the relevant evaluation
questions from the evaluation design, per the individual respondent’s role in or with the program. Sample instruments are included in Annex IV. The team adapted these questions in the field as
themes emerged in an individual interview or across a set of interviews. Case studies
As a method, case studies are tailored to fit the purpose for which they are used. In this evaluation,
cases integrate perspectives on a particular policy reform or institutional strengthening effort, about which actors shared varying opinions and experiences – even contradictory ones. Case
studies also required secondary data review, including policy documents and deliverables, results
data, institutional strengthening manuals and work plans.
Data analysis
The evaluation team answered the evaluation questions by systematically analyzing interview
transcriptions to discern patterns, convergence and divergence of opinions and experience, and
trends. Interview data were coded according to the themes and dynamics most important for
answering the evaluation questions. The team also noted emergent codes – that is, themes that
occurred in the responses that had not been considered in the evaluation questions. In this way, the
team attempted to capture unintended consequences of the PPP activities.
Limitations
The team faced the following challenges:
(1) PPP was executed in an environment where priorities, political relationships, and USAID’s
own targets and goals were in constant flux. This complexity and dynamism means that
the multiple facets and perspectives around the Program’s achievements can be
contradictory or unclear, even after extensive data collection and analysis.
(2) Some actors (key GoC stakeholders, for example, or AECOM staff and contractors, or even
interested parties at USAID) have changed positions since the project was implemented.
Some individuals sought for interview were unreachable; and
(3) The absence of a Performance Management Plan and baseline data for the complex
activities meant the team had to rely on the Program Work Statement (PWS) as a
performance measurement framework.
6
FINDINGS, CONCLUSIONS &
RECOMMENDATIONS
This section discusses the key findings, conclusions and recommendations from the evaluation
research. These elements are organized by a set of key themes that emerged from the field data,
and which respond to the evaluation questions that the research was designed to answer. Also
included are case studies of five PPP interventions, as described in the section on sampling. These
are highlighted in text boxes, with particular attention to the ways these particular experiences
exemplify the work of PPP, highlight challenges, or answer one or more of the evaluation
questions.
A table listing the detailed Findings, Conclusions and Recommendations is in Annex I.
1. Relationships and Influence between the GoC, AECOM and USAID
Evaluation question 1: To what extent did AECOM influence the GoC in defining its policy
decisions in the content of the project deliverables?
Summary: AECOM/ GoC/USAID relations were important for success in PPP efforts. The
working style AECOM fostered – flexible, collaborative, high-level working groups – was
widely appreciated. AECOM was able to influence policy through the quality of its teams.
Success was more difficult with complex and cross-agency policies and could become
variable as political commitment waxed and waned over time.
FINDINGS
Government respondents roundly appreciated the PPP consultant teams with whom they worked.
The PPP teams brought innovation and worked collaboratively with government partners to reach
consensus. One called this style of work “useful, pertinent and very productive.” Others reported
that they valued the consultants’ skills, the extended timelines of support and, particularly, the
collaborative style of work. Evidence pointed to genuine collaboration, rather than solutions
imposed from outside. For example, respondents reported that the GoC always gave feedback on
the deliverables, allowing for a follow up and control of the technical assistance (TA) provided by
PPP.
PPP leadership, staff and consultants used their high-level connections to open doors, and create
good relationships from the start of PPP activities. Respondents frequently noted the need to work
with the leaders of various entities, rather than with mid-level functionaries, because decision- making is centered in the heads of offices. At the same time, having such high-level and close
contacts occasionally resulted in an awkward situation where USAID staff felt excluded from the
communications.
The following challenges to influencing policy were repeatedly mentioned:
(1) The wider the participation of multiple GoC agencies, in general, the more challenging the
7
influence. There was competition for resources among agencies and varying levels of
commitment from each. In the Income Generation mission, for example, respondents
reported that various partners in the working group – Ministries of Work and Commerce,
for example, and the national apprenticeship office (SENA) – did not fully incorporate their
work with the round table back into their home institutions. This limited the long-term
effects of the income generation activity.
(2) Vacillating political priorities during implementation greatly affected the influence PPP
had on its GoC interlocutors’ decision-making. Several respondents noted that productive
discussions around the Rural Development Law affected how development is conceived
and planned. However, for what were cited as political reasons, the law was not passed.
CONCLUSIONS
Diverse experts brought innovation and interesting debates to the decision-making (such as
Colombian and international best practices; see section below for greater detail on this theme.)
Factors that supported successful outcomes included:
1) Extended efforts
2) Building consensus through collaborative work (rather than imposing solutions from
without)
3) PPP staff and consultant connections with high-level GoC actors with decision-making
power
Factors that inhibited success or increased the likelihood of difficulties included changes in GoC
political priorities and activities in which multiple agencies were responsible for pieces of a reform.
RECOMMENDATIONS
Future policy activities should replicate PPP’s staffing patterns of high-level staff and consultants,
working closely with high-level government contacts, to help empower GoC staff to act. Future
projects should replicate PPP’s practice of working hand in hand with the GoC on each activity,
over time, seeking consensus, as a team. Seek common ground between USAID and GoC priorities
(see the following recommendation) in selecting thematic areas and priorities.
Implementing Partners (IPs) for future policy projects should support a high-level, ongoing round
table with the GoC that provides a space for proposing thematic areas for joint action. If a given
theme is not part of both USAID and GoC priorities, it may well be best to drop or postpone it
rather than continue without consensus. Let this process set the Mission’s expectations and goals.
IPs and USAID must plan for the likelihood that any effort that involves inter-agency arrangements
will require more time, more effort, more consensus, and more political will. Such efforts are an
order of magnitude more challenging than working with just one entity.
8
2. International Best Practices
Evaluation question 2: To what extent were international best practices introduced, versus
relying on Colombian expertise and models? How effective was the approach?
Summary: International best practices (IBPs) were important in many of PPP’s policy and
institutional strengthening efforts. They came in the form of international round tables and
seminars, consultant experiences, literature review, and specific cases known to teams or
to individual consultants. Respondents were also nearly unanimous in saying that IBPs
were not sufficient tools on their own, and had to be adapted substantially to meet
Colombia’s unique institutions and conflict contexts. FINDINGS
Consultants, GoC, USAID, and PPP staff reported that PPP activities sought out IBPs when
relevant to their work. One USAID respondent was concerned that this consultation was not
systematic or thorough, perhaps thereby missing an opportunity to place Colombia’s efforts in a
wider context and choose the optimal policy response. PPP staff and consultants, as well as GoC
respondents, did not describe systematic attention to IBPs, but rather reviews that differed by
activity and thematic area. There were, for example, international seminars about relevant issues, literature reviews, use of consultants’ experiences overseas, and specific practices from other
countries known by to the consultant or GoC teams, or researched by them. There were reports
also of the use of national best practices, such as Antioquia’s experience with victims’ reparations.
Respondents noted several specific examples. In PPP’s work with the Consolidation Unit, for
example, the team made use of international cases from Brazil, Mexico and Chile. With the
Victims’ Unit, the team examined the Antioquia model as well as examples from former
Yugoslavia. A respondent from the Ministry of Agriculture cited best international practice cases
that were reviewed in their PPP activities from Brazil, Peru, Chile, Africa and Southeast Asia and
were analyzed to determine what could be incorporated in Colombia.
The Income Generation Mission
The Mission was led by the Department for Social Prosperity (DPS). It exemplifies the strengths and limitations of PPP support
in a specific policy effort. Key success factors included:  A space for discussion and consensus, based on national and international research.  Efficient work style, well-connected experts with international experience, and professional, committed DPS team.  Identification of policy or institutional gaps and acceptable solutions.  Concrete diagnosis and deliverables on management, rural/ urban implementation, institutional coordination, and
monitoring.
Limitations included:  Results varied. GOC entities (SENA, MARD, ADR, Ministries of Interior and Commerce) participated weakly and did not
take shared agreements back to their home institutions.  Six months was too brief to address this complex theme and accompany interventions. Action planning left much
“pending.”
The strong relationship with DPS and their constant participation was key to the process. If the GOC counterpart is not this
involved or capable, it is unlikely to be successful.
9
Respondents frequently noted, however, that IBPs were insufficient on their own as inputs to the
planning process in Colombian institutions. “Forums, events and studies are probably less useful
elements, we really need more tangible support,” recounted one. A government respondent said
the IBPs that were consulted for their work were too abstract and with insufficient detail, while a
USAID respondent added that when an international case provided good detail, those details would
not work in Colombian legal or social conditions.
One such case involved international experts helping with the rural cadaster. Their help when
visiting Colombia was useful, but their international experience (according to the USAID
respondent) was less useful. Another government respondent related a similar mismatch with IBPs
on income generation. When respondents were asked why they felt the IBPs were not adaptable
for Colombia, they mentioned Colombia’s particularities because of the ongoing conflict and the
overarching importance of addressing victims’ concerns here, which is an order of magnitude
larger than in other environments. The experience of providing services to victims in Antioquia
was cited by PPP staff and government as particularly useful since it was a proven experience in a
related context.
CONCLUSIONS
IBPs were generally effectively integrated to PPP activities. There was variation in the degree to
which IBPs were incorporated, in part because of the differences in how IBPs were brought to the
table. As a result, it seemed there was a variable level of confidence on USAID’s part about
whether there was systematic consideration of options from international cases. The focus
appeared to be on inclusion of IBPs as part of deliberations on policy design and formulation, with
less attention given to IBPs that supported aspects of implementation.
GoC partners valued the support in cutting-edge themes that IBPs put on the table, while also
noting that these inputs were not always sufficiently tangible and detailed. PPP staff and
consultants also brought national best practices, and the ability to incorporate IBPs in ways that
were appropriate to the Colombian context. This came from their high-level technical expertise
and knowledge of the institutional environment. IBPs were useful inputs in some cases, were perhaps not considered systematically in others, and
were rarely definitive responses in any case. RECOMMENDATIONS
Future projects should always scan international experience for applicable best practices to address
policy changes. If USAID is concerned that IBPs are not adequately being considered, it should
monitor their systematic inclusion activity by activity, and include in work planning that each
policy or institutional strengthening activity effort be accompanied by a systematic review of
relevant options. The options pursued should go beyond policy designs to focus instead on the
ways in which new and reformed policies can best be implemented in challenging environments.
At the same time, future projects should remain flexible about how these IBPs are sourced and
incorporated. Remember that the IBPs put into practice must be relevant, instructive, and apt for
incorporation into Colombian institutions and contexts.
10
3. PPP’s effectiveness
Evaluation question 3: How effective was the Public Policy Program in assisting the GoC to
adopt policy changes?
Summary: Effective adoption of policy changes varied by thematic pillar, as these
coincided with the political environment and GoC interests. Cross-cutting themes, which
were to be applied to each pillar, were quite variable in their effectiveness. It is important
to examine effectiveness in terms of USAID’s long-term policy efforts, and their particular
focus on national and regional implementation of the reforms. In sum, there is strong
evidence of important successes in PPP’s support to new GoC policies and reforms, as well
as a set of valuable lessons learned. The evaluation’s answer to this question is linked to
the next question, on the quality of PPP assistance.
FINDINGS
The three pillars
Timing was propitious for PPP, which began operations alongside a new government in Colombia. Many government offices were highly motivated to take advantage of PPP’s opportunities for
technical assistance (TA) and support to new policies. At the same time, there were circumstantial
challenges that affected the new government and PPP’s effectiveness. An ola invernal, a particular
winter storm, caused a humanitarian crisis near the start of the program; agrarian strikes applied
pressure for particular kinds of land reform; peace talks were in the offing; and other national and
regional circumstances opened windows of opportunity, or closed them. Working hand in hand
with government interests and requests for assistance, PPP was often poised to take advantage of
opportunities that arose. Government respondents to this evaluation frequently lauded this
THE LONG-TERM NATURE OF POLICY REFORM
Findings: PPP staff worked on the same policy efforts over time: from ADAM/MIDAS to PPP to The Land Project, building
political will for significant institutional change over time. The more contentious and political a contemplated reform, the
longer it takes to build a coalition for change. Some complex changes initiated under PPP (such as rural development and
financial service policies) are only now coming to fruition, a year after work ended in PPP. Another example is from the peace talks in Havana, where the first point discussed was rural development and the
integrated, “territorial” approach, vital issues in a post-conflict situation. “We brought those people together and now our
proposal on land and rural development comprises the discourse.”
Shortening the program to three years, eight months contradicted this need to tackle long-term, controversial themes with
many interests that need to be included if a reform is to succeed.
Conclusion: The ultimate impacts of USAID-funded policy projects are often not clear during the project period. Sometimes
they come to fruition later. But, the analysis and processes begun are vital to subsequent project generations. This is also
true of the professionals who work on these projects, who are hired by subsequent projects and/or move into public service.
They are important resources.
Recommendation: USAID’s policy reform efforts should be thought of as a portfolio of initiatives that mature at different
times. USAID should monitor the progress of various initiatives and accelerate or pause activities accordingly – recognizing
that ultimate impact may go beyond the project period.
11
flexibility.
This flexibility is notable in the context of AECOM’s contract and PWS and work planning. It was
a balancing act to maintain flexibility before government counterparts, while also maintaining the
standards and benchmarks demanded by USAID in these documents (and changed substantially
after the close of the first year.) Nevertheless, there is every indication that PPP was able to balance
these two sets of overlapping, but not completely coincident interests. First, the PWS in Year One
required workshops and papers, but the teams were able to adapt the content and the duration of
TA support as long as the document or event was delivered as promised. Quarterly reports show
the achievement of milestones almost always precisely as planned, even when plans changed.
Second, USAID CPARS (Contractor Performance
Assessment Reporting System) ratings were consistently
high for the first three years of the contract, receiving
consistent Exceptional and Very Good ratings for
product quality and management of key personnel. PPP
was also praised for the acceleration of its goals in line
with opportunities provided by the Santos government.
On the other hand, the most frequent obstacles were
political in nature. The Consolidation Unit, for example,
was unable to secure agreement on a CONPES document
because of political disagreements over its content and
role. “A technical document became a political football,”
one respondent reported. When leadership changed
hands, work carried out with PPP was in at least three
cases shelved by incoming leadership with different
ideas. INCODER was reported to have resisted reform at
its regional levels, because of political appointees who
were protecting their interests, and the Agriculture
Ministry was said to have a limited absorptive capacity.
These political obstacles were also clear in the tendency
for wider reforms – that is, across agencies – to stumble.
It was difficult to work jointly across agencies with
competing interests and timetables. These inter￾institutional commitments were called “very fragile” by
one respondent, and another said that participants in a
multi-agency mission for income generation did not take
the agreed-upon actions back to their institutions for
implementation.
Smaller, concrete reforms within one institution proved
more straightforward to implement and their effects less
subject to political challenges. One example is the
computerization of INCODER’s files; another was the
strategic planning, indicator dashboard and index for the
Consolidation Unit.
Respondents were asked to rate the TA they received, in
Case study
Consolidation Unit, Institutional
Strengthening
PPP’s support to the Consolidation Unit
demonstrated that requirements to coordinate
agencies and institutions limits effectiveness, due to fragile and varying commitments and
resource constraints. Also, working with an
existing entity was politically challenging, as
entrenched actors resisted change and the loss
of influence. PPP support helped to adjust administrative and
political bases for the Unit. An indicator
dashboard was designed and implemented with
PPP support. The Respuesta Rapida program
was reported to be useful to the Unit, as was the
Unit’s introduction into consolidation zones.
Challenges included:
Consolidation Index: The Consolidation Index
was perhaps overly sophisticated for the Unit.
Fundamental issues such as contracts and
administrative issues were not resolved, making
the large number of indicators not functional.
AECOM helped to reduce the number from 700
indicators to 80, something more manageable
for the Unit.
Interagency coordination was difficult with
changes inside the Unit and in its many
relationships with DPS and other institutions.
The Consolidation Unit couldn’t get CONPES
because of a lack of political support and
direction. Sector stakeholders inhibited the work
if their political and resource needs were not
addressed.
The Unit’s basic institutional capacity and a
lack of political interest were important
limitations. According to its own staff, the Unit
needs help administratively, technically and with
respect to its budget. These issues needed to be
resolved first in order to absorb the TA.
12
a Likert scale from one to five, with one being very poor and five being very good. Rating the
effectiveness of the PPP TA, respondent scores averaged 4.5; when asked about the results of the
guidelines or practices they were provided, the average score was also 4.5.1
Where both USAID and the GoC had a strong interest in a given reform, this shared interest also
supported success, even where there were competing political interests, as in the case of the
creation of the Social Inclusion Sector and the Victims’ Unit. Similarly, with these two bodies,
PPP activities were related to the creation of new institutions, rather than reforming existing ones – which, by definition, would be challenged by existing political interests. Respondents involved
with the Victims’ Unit also noted that the GoC lacked budget for this work, which made the PPP
institutional strengthening all the more welcome.
Cross-cutting themes
PPP was designed to include themes across pillars – gender, the environment, good governance,
access to financial services, and bridging the national-regional divide. In terms of gender, there
was a spoken commitment to what is called the differential approach2 within several PPP activities.
This was incorporated in Victims’ Unit definitions and priorities, but had less traction in the other
two pillars, according to respondents. PPP’s Gender specialist worked with the President’s High
Commission on Gender-Based Violence, but this work was not linked to PPP pillars. The Gender
specialist noted that her expertise was not, strictly speaking, in gender, and that having such
expertise on staff would likely have improved the way PPP addressed the theme.
Other evidence points to a lack of real commitment to the cross-cutting issues. The USAID
technical monitor on environmental issues had $70M in programming but no budget at all under
PPP; one respondent reported that as a result, PPP was not high priority. In fact, all budget for
cross-cutting themes had to come out of a pillar’s budget, which meant that the technical lead for
that pillar had to value those themes against the pillar-specific activities being undertaken. One
respondent who worked on cross-cutting issues said, “I had to convince the pillar leaders that they
needed me, because I had no budget. The cross-cutting matrix looks good on paper, but I had to
force myself on the pillars, where they were not at all focused on these themes. They didn’t even
know they needed certain support.”
In addition, the Chief and Deputy Chief of Party for PPP committed their own time across the three
pillars, though neither reported having been (nor were they reported to have been) drivers for the
cross-cutting themes. Their leadership on cross-cutting issues appears to have been weaker than
1 The interview questions were used as guides for protracted conversations with respondents, and as such not all
respondents answered all questions (as not all questions pertained to the work with all respondents.) However, the
evaluation team feels that these “scores,” though subjective, are indicative across the verbal responses in the
interviews even when numeric responses were not provided. 2 Differentiated attention to women, ethnic minorities, people with disabilities and other groups, as a standard for all
government interventions.
13
their very targeted participation in the implementation of
the pillars. In this environment the cross-cutting themes
became less transversal and more opportunistic.
Respondents who worked on the theme of access to
financial services reported benefiting from a USAID
technical monitor who was tirelessly supportive. At the
same time, respondents said the USAID Mission
leadership held financial services for rural development
in low esteem, and cut the budget throughout the project.
Positive outcomes from this cross-cutting theme included
coordinating government entities and the private sector,
which showed significant interest in participation.
In the environment cross-cutting theme, much effort was
focused on response to victims of the natural disaster that
occurred near the start of PPP. When efforts to restore
housing for the crisis victims had ended, this component
was closed, in line with the restructuring of the PPP
contract and pillars.
CONCLUSIONS
The three pillars
Windows of political opportunity and commitment were crucial dimensions of successful PPP
efforts. USAID and PPP were frequently flexible in responding to GoC concerns and priorities.
Political pressures could limit common ground within the GoC or between USAID and the GoC.
Where GoC priorities and PPP work themes were aligned, activities were more effective, and had
more lasting effects. Other characteristics of success included working on targeted, smaller-scale
reforms, within one agency rather than various. However, where commitment was high – such as
in the case of the Social Inclusion Sector and the Victims’ Unit – PPP offered effective support
despite challenges that arose.
Impediments to adoption of PPP recommendations included:  Changes in GoC staff, priorities and interests  Multiple agency coordination, with greater or lesser commitment from each agency
 Corruption and regional politicking
 Lack of institutional capacity or maturation
 External shocks such as the agrarian strikes, fiscal challenges, the peace process  Delays due to AECOM and USAID procedures
Cross-cutting themes
Cross-cutting themes in USAID’s initial design did not become fully cross-cutting as implemented
by PPP. Lack of appropriate expertise or influence on the team (environment, gender), and political
challenges or lack of mandate (nation-region and good governance) limited integration in the pillar
structure. PPP’s leadership focused on the pillars and their deliverables, which meant that cross- cutting themes were put on the back burner. Neither USAID, GoC, or PPP appeared to place gender
Policies on Access to Financial Services (AFS)
PPP sought to improve vulnerable rural
population’s AFS, to facilitate their inclusion in
regional development and growing prosperity.
Policy and regulatory instruments were designed
to expand financial networks, and leveraged
US$38 million in public and private resources.
Success factors included:  An early start in the GOC, making
consultants’ project time more productive.  A high quality technical team to develop
the program’s objectives.  GoC willingness to implement proposed
changes.  Positive pilot outcomes, allowing for more
precise policy proposals.
Access to financial services is bound up with
Colombia’s structural issues and rural
development as a whole. As a cross-cutting
issue, AFS worked closely with the three pillars,
but its lower priority meant challenges in budget,
coordination and implementation.
14
or environment high on their priorities. A strong champion for a given cross-cutting effort (such as in the case of access to financial
services, as noted above) increased impact, even while the theme was given less official priority. The lack of specialist team members who are also strong advocates limited PPP’s ability to bring
(for example) a gender lens to bear on a range of efforts with competing priorities. RECOMMENDATIONS
Pillar areas
Work closely with the GoC to identify policy reforms that are high priority for both, in line with
USAID’s CDCS and with GoC priorities in the National Development Plan, potential peace
accords, or other high-level areas. Remain flexible during implementation to take advantage of
windows of opportunity. Planning should include attention to targeted, single-agency efforts as
well as over-arching, inter-institutional reforms. Where the latter are contemplated, build in more
time, effort and resources.
Cross-cutting themes
If a theme is important enough to include as cross-cutting, USAID and IPs should ensure that it is
explicitly covered, with appropriate expertise, with its own budget, on relevant topics, with
sensitive M&E targets, and adequate resources. IPs must be explicitly engaged with these themes
or they are not likely to be addressed adequately. USAID should monitor that cross-cutting themes
are genuine priorities during implementation, beyond any proposal language.
4. Quality of PPP’s Technical Assistance
Evaluation question 4: What was the level of quality in assistance provided?
Summary: High technical quality supported PPP’s successes, and PPP was largely
successful in meeting its deliverables and contractual requirements. Across respondents,
NATION-REGION
Findings: The need to bring reform to all Colombian territories was a PPP cross-cutting theme– as well as a Mission-wide
USAID priority. There were successes within the pillars: working with the Victims’ Unit and Social Inclusion Sector to ensure
territorial presence; orienting financial products toward micro-business people while expanding coverage; and integrating
regional level indicators into INCODER’s planning. But PPP did not have a contractual mandate to operationalize and
implement at the regional level. Rather, other IPs were charged with addressing regional needs.
Conclusion: PPP’s mandate was at the national level only. But implementation at the regional level is vital for impact. PPP
and region-based IPs appear to have worked more in isolation. Relative to its priority at USAID, the territorialization of
results was not as successful or as visible as desired.
Recommendation: Include regional participation and coordination to support GoC decision making within and among
USAID interventions. Set goals for regional implementation and uptake where the key geographies for execution are
regional rather than national. Link national and regional USAID programs better, or combine national and regional activities
for one pillar in one contract, so that the IP can strategize and implement vertically in partnership with the GoC.
15
the evaluation team heard extensive and specific praise of PPP consultants and staff. For
their part, the PPP staff and consultants also lauded their government counterparts, and all
agreed that the TA was a joint effort.
FINDINGS
PPP was largely successful in meeting its contractual requirements with high quality deliverables. The deliverables were reviewed by the technical staff and then by the contracting office to ensure
compliance with the contract’s scope of work. A knowledge management staff member also
reviewed and classified the deliverables, making them available to various audiences when this
was useful. Moreover, the deliverables were almost unanimously lauded by government respondents; one
called them “pertinent and flexible,” while another noted the value of the continuous support in
the context of an immature institution advancing very rapidly. Respondents were asked, in
interviews, to rate the quality of the technical assistance and of the technical tools provided, in a
Likert scale from one to five, with one being very poor and five being very good. Rating the
technical tools they received from PPP, respondent scores averaged 4.5; when asked about quality
of the TA, those who responded with a numeric score all gave PPP the highest score of 5.
Many specific examples of quality in PPP processes and deliverables were cited in evaluation
interviews. PPP was successful in the creation, institutional capacity building and policy
implementation at the Victims’ Unit because of the quality of people and deliverables, according
to one government respondent. Another government official said that PPP provided clarity and
vision for the Unit “from day one” on the registry, assistance and reparation themes. A USAID
respondent noted that the PPP designed the unit and trained Unit staff in a way that changed the
trainees’ views of victims – from people looking for a handout, to a reparative and differentiated
approach. Another government respondent remarked that the entire social inclusion sector is well
positioned, in part because of the quality of PPP’s work with the Victims’ Unit.
Government and consultant respondents said that the work in access to financial services was of
very high quality, including their work in transferring knowledge and supporting strategic planning
with the Banca de Oportunidades. PPP staff applied their awareness of the issues and limitations
their government counterparts faced to put feasible plans on the table that contemplated those
limitations. This was due to the broad range of consultants with USAID experience as well as
experience in and outside the GOC. PPP team members were routinely lauded as “high-level,”
experienced, adaptable and collaborative.
CONCLUSIONS
An important part of PPP’s success was the excellent selection of staff and consultants. The
personal relationships with the GoC, high-level national consultants, quality in the deliverables
and commitment were key elements consistently identified by the GoC. Frequently cited
complements included: “committed people”, “effective results”, “able to work point by point with
their GoC counterparts over extended periods.” PPP’s staff conducted strict quality control on
deliverables. They knew what the client expected and ensured it. (Please see also the section on
relationships with the GoC, above.)
16
RECOMMENDATIONS
IPs should continue to use strong and connected Colombian individuals and teams, to ensure the
consistently high levels of quality attained in PPP. IPs must cover the thematic areas of the program
with this kind of quality technical support in staff and consultants, and with cross-cutting themes
as well. Staff should be predominantly Colombian. Where international expertise is sought and included,
ensure that that international presence is balanced with Colombians. The connections and
understanding of local context are important for what they bring to the work.
5. Institutional Strengthening
Evaluation question 5: What characteristics of AECOM’s work supported institutional
strengthening most effectively and which were less useful?
Summary: As with the policy efforts described above, institutional strengthening worked
best with targeted efforts, collaborative work styles, fewer political interests at risk, and
agreement between GoC and USAID on priorities. But institutional strengthening faced
more operational obstacles, not least of which was the lack of contractual mandate to
engage in these activities. Also, institutional maturity and absorptive capacity limited the
degree to which institutional strengthening activities took hold.
The Rural Development Law policy effort
PPP supported the GOC in drafting the Rural Development Law, currently in consultation with communities. The effort enjoyed a
high quality, high-level technical team and an in-depth review and integration of international best practices on the subject. The
draft was innovative and caused wide debate on the integrated model, financial integration and differential approach.
These and other elements of the law emerged from the wide debate, and changed how GOC and other practitioners have
conceptualized rural development. The law’s innovations were important for discussions with the FARC in Havana on point one of
the peace talks agenda in 2013.
The success of PPP’s technical and policy support cannot be fully evaluated, as the Law has not yet been enacted for political
reasons. However, it is worth noting how work on the Law supported last year’s land formalization program (Decree 1465 of 2013).
USAID may wish to further support the passage of the Law. To do so, any follow-up work would need to address the entrenched
political interests related to rural development issues, and who oppose its passage. If a peace accord is signed, the passage of
the Law may become even more important, and more challenging, given the dimensions of the changes it proposes.
17
FINDINGS
According to one USAID respondent, “PPP’s
success was in funding receptive government agents
who needed help, every day for a year.” A PPP
staffer said that institutional strengthening worked
best in the form of “focused efforts within particular,
amenable offices of the government.” A PPP
consultant said that PPP’s diagnostic work helped to
“identify failure in the institutional order and make
recommendations.” Where institutional
strengthening was carried out, such as in a targeted
civil service reform effort, the work was valued by
respondents across the evaluation. At the same time, the ongoing challenges inherent in
institutional strengthening – as opposed to passing a
policy reform – were evident: “It was an
achievement to mount the structures of the Victims’
Unit and Center for National Historical Memory.
But much harder to put them into action, with
institutional strengthening.” Competition for
resources, political positioning, staff rotation, and
low capacity in the target agency were cited as
obstacles to success in the Victims’ Unit, Ministry of
Agriculture, INCODER, Consolidation Unit, and
others.
CONCLUSIONS
There is reversion to the mean in highly transient
bodies of the GoC: reforms and institutional
strengthening activities are diluted over time and
where staff are transient or leadership changes occur.
Institutional strengthening worked better in a new
institution like Victim’s Unit than in an existing one,
like INCODER, where reform raised hackles and
was resisted vociferously. Political pressures were a key obstacle to efforts, as
it was difficult to obtain consensus within the GoC
or between the GoC and USAID. Activities were
limited by GoC political priorities. PPP did
diagnostics and wrote recommendations, however
the adoption of these proposals depended on the
government’s will to implement them. The
impediments to effective work in policy reform,
listed above, apply here as well (changes in GoC
Case Study:
Institutional strengthening of the Victim’s Unit
Support to the Unit illustrates these success factors:  A strong technical team with high-level
connections achieved shared goals;  PPP worked hand-in-glove with the GoC and came
to consensus, rather than imposing pre-designed
activities;  Both GoC and USAID had political will and
interest in establishing a Victims’ Unit;  PPP helped the GoC gather political support and
resources, and draft regulations, to operate the
Unit; and
 The Unit had capacity to absorb PPP support. For the GoC and USAID, the passage the Victims’ Law
(1448/11) opened a window of opportunity for priority
support. PPP’s institutional strengthening added value to
the Unit’s launch of a new entity in Colombia’s complex
institutional context.
PPP’s institutional strengthening included strategic
planning, complex interagency mapping, technical
design, and differential approaches for vulnerable
populations. Challenges included:  USAID respondents perceived as “lack of control”
what PPP felt was “maneuvering room.” The work
moved quickly, and PPP did not always share these
advances quickly with USAID.  The design includes significant inter-agency roles
that have proven problematic in practice.  On at least two occasions PPP’s team was not able
to come to consensus with the National Planning
Department (DNP), and the Victims’ Unit’s General
Secretariat, during the design of an “End to
vulnerability” index and in work to make the
application of the Unit’s offerings more flexible. Not coming to consensus, however, indicates as well
that the PPP team was on an equal footing with GoC
partners. Rather than a failure of PPP, this respectful
group work environment is a significant part of the overall
success. The Unit continues to work with the consultant
team after PPP’s support ended (now under IOM’s
Victim’s Program, Maximizar).
Gender considerations were integrated into the VU
mandate in outreach and in differentiated approaches for
victim reparation. PPP’s technical gender specialist was
not involved, as PPP’s technical specialist on victims
assumed responsibility for gender within the pillar.
18
staff, priorities and interests; corruption and regional politics; lack of institutional capacity;
external shocks; and delays due to AECOM or USAID procedures.) If anything, these affected
institutional strengthening even more than policy reform efforts, because of the ongoing nature of
institutional work.
RECOMMENDATIONS
USAID must make explicit any goals for institutional strengthening just as for policy. If these
reforms are expected to be operationalized, including at regional levels, this must be clearly
designed, budgeted, monitored and carried out. As with policy goals, it is not worth the expense
or effort if the GoC is not fully on board. USAID should choose these activities in full consultation
with the GoC and include an assessment of whether the institution has the absorptive capacity for
the reform, as well as the political commitment to follow through with it. Additional Findings: Project Management
Summary: Technically, PPP set a high standard for management, including seeking and
contracting high-level contractors with knowledge of government, and putting them to use
in collaborative work groups with government. Administratively, the contracting process
to make this technical side work was daunting. USAID’s decentralized technical
monitoring aided project activities but also had drawbacks. Most important for USAID’s
management were issues around communication, timing, and trust. PPP’s staff could move
quickly because of their high-level working relationships with their GoC contacts;
sometimes their speed outstripped that of their USAID leadership. When communication
on these fast-moving activities lagged, USAID felt cut out of the loop. At the same time,
USAID’s slower pace sometimes caused PPP to miss windows of opportunity.
FINDINGS
PPP’s management
PPP managed its work efficiently and with attention to technical needs, offering a collaborative
structure among GoC, staff and consultants. Its staff and consultants were selected for their
experience, knowledge of the themes, familiarity with the functioning of the GoC, and ability to
open doors at the highest level of relevant government offices. The high level of the consultants
helped to guarantee direct communication with the GoC. Government respondents told us that PPP
consultants adapted to their changing priorities, but also were frank enough to say “no” when
necessary. They also knew the strengths and weaknesses of the GoC and were able to situate policy
and institutional reforms within that context.
AECOM used hundreds of contracts, with individuals and firms. These required both home office
and USAID approval. The pace and paperwork of contracting were frequently mentioned as
problematic. “We lost one valuable consultant who could no longer wait,” said one government
respondent. Contracting was tedious and complicated, and took too long, according to government,
contractor and PPP respondents. Even after contracts were issued, the process used multiple
deliverables as a way to control payments. This meant “a lot of deliverables to review and rate,”
19
according to a PPP respondent.
Cost-effectiveness
The flip-side of the slow contracting process was that the project appeared to be very cautious with
expenses, and remained audit-ready at all times. Government respondents reported that the
expenses were low compared to the high return they received. This was mentioned in particular
regarding the income generation mission as well as the work in creating the social inclusion sector
and its components. One USAID respondent asserted that the return-on-investment was much
higher than USAID’s regional-level programs, because PPP was able to affect policy across the
country all at once.
USAID’s guidance
Given the high level of PPP’s teams, their existing relationships with government, and the
autonomy under which they worked, the degree to which they communicated their progress with
USAID varied. Where their USAID technical monitors were confident in the process and the
degree to which their wishes had been communicated to the teams, this more distant relationship
ran smoothly. As one USAID respondent put it, “Where there was trust top to bottom – Mission
Director, COTR, Technical Monitor, AECOM, GoC – things worked. Without it, nothing else
mattered.” There were instances where PPP pursued courses of action in concert with GoC independently,
when the consultant team and GoC were in sync, when they had approval from USAID for the
general course of action, and where the effort was reflected in the work plan. It does not appear
that such actions were committed (thinking particularly of the Victims' Unit work under very
capable leadership) without approval. The level of trust was higher with some consultant teams
than others. More trusted teams seem to have been granted greater latitude. This depended largely
on the consultant team, the thematic area, USAID technical leadership, and the window of
opportunity offered by the GoC. Often these were in equilibrium, while other times (such as a
change in USAID's technical leadership around land and rural development) new equilibria were
sought with the new conditions.
Outside of USAID, the dominant perspective seemed to be that less involvement from USAID was
better. Though one government respondent told the evaluation team, “The relationship between
USAID and the DPS was positive and helpful, they [USAID] didn’t appear much once we [the
DPS and PPP consultants] were working well together,” others were less positive. There were
some GoC environments in which active USAID participation was not welcome. A USAID
respondent told the evaluation team, “If the USAID person is in the room, the meeting is not the
same,” and a government respondent said, “I don’t want a bunch of Americans coming in here.”
Two GoC respondents spoke about unprofessional behavior on the part of USAID actors who
participated in PPP activities. A USAID respondent reported that two complaints were raised by
the GoC about such behavior, which they called disrespectful.
Although USAID succeeded in procuring well-connected, high-level staff on the PPP team, there
were two competing perspectives about the Mission’s management of the project. USAID
respondents tended to report that PPP staff acted on their own and failed to inform USAID; at the
same time, PPP and government respondents said USAID’s management sometimes tied the hands
of the PPP experts, insisting on closer and closer control. Over time, USAID staff exerted
increasing control. While the Mission Director who launched PPP was said to have given AECOM
20
“free rein,” later leadership took the reins in hand. Technical management
USAID built a structure of six technical monitors who supervised different aspects of PPP’s
operations and one COR who managed things administratively. Communication channels
included meetings with the technical monitors, quarterly reports, and weekly and biweekly
reports, where the main advances in politics and the main challenges were shared. The biweekly
reports and meeting were said to require a lot of work by PPP. They were also considered a
valuable opportunity to study the context, update the current policy debates, and show program
advances. USAID respondents said these communication methods worked well and that PPP was
good at delivering reports while ensuring quality control.
Three USAID respondents discussed the disadvantages of the system, however. The technical
monitors competed for program resources to support their technical areas. Communication
among six to eight people involved with technical matters was hard to manage. The three pillars
were, essentially, different mandates. Technical differentiation was necessary but difficult to
manage. Also challenging were the timelines for review within the Agency. “Where we lost out was when
the whole thing took too long, and the window of opportunity closed. The difficulty here was
more with USAID,” said one of the USAID technical team that monitored a PPP pillar. A
government respondent lamented a long-awaited decision from USAID on regional support that
PPP would be allowed to provide to a Contrato Plan office. When USAID eventually denied the
request for funding, the GoC respondent blamed USAID. PPP staff may also have made
promises to the GoC that were not yet approved by USAID. In either case, the relationship with
the GoC suffers when the USAID and IP relationship is not well-defined. There were cases
where the USAID response timeline was longer than the GoC was prepared to wait.
USAID requested that no large close-out event be held across the project. As a result, PPP
conducted handovers with each GoC partner individually, over a period of six months. This
included handing over the technical reports to each partner. However, many respondents –
government, USAID and PPP – said that the lack of a visible ceremony or handover of the
activity at close out was an oversight. It was felt that this impeded handover to the government,
as well as from an outgoing government lead to one who is incoming – because no formal packet
of the inputs and deliverables, by agency, was made available. Respondents also said that a
formal ceremony would have underscored the importance and contributions of the interventions.
CONCLUSIONS
PPP management
PPP got many things quite right: it was flexible in its support, adapting to USAID and GoC
needs in many instances. PPP successfully identified the right professionals, who:  Were of high technical quality and knowledge level;  Had important connections with GoC; and
 Were able to identify policy and/or institutional gaps and propose acceptable solutions. PPP’s teamwork model helped to secure consensus (see also the findings in the first section on
21
the relationship between GoC and PPP.)
The contracting model used in PPP had many important technical benefits, such as having the
right people for the tasks at hand, at the right moment. (Please see also the findings in the section
on how PPP worked with the GoC.) However, AECOM’s contracting processes were tedious and
complicated the work.
PPP did not uniformly inform USAID of advances with sufficient time for their participation.
This eroded trust with USAID and resulted in USAID wanting even closer control. Cost-effectiveness
Though PPP’s expenditure data per activity were not available, indications from government,
USAID and the contractor indicate (1) close attention to spending rules and regulations; (2)
attention to national-level priorities which leveraged or attempted to leverage effectiveness at
more local levels; and (3) impressionistic evidence of policies and reforms implemented
successfully, at reasonable cost. USAID management
As Mission leadership and management rotated, they changed the rules of engagement for the IP.
In some respects the change was significant and affected the IP’s ability to do its work with
sensitive GoC offices. Very close USAID oversight created some problems. There was some desire expressed among
GoC counterparts to do the work with less direct USAID presence. GoC representatives raised
formal complaints, and it is unknown the extent of the damage to the relationship this may have
caused. The lack of a significant ceremonial “closure” – some kind of formal handoff from
USAID and AECOM to the GoC – was keenly felt. USAID has an obligation to oversee its contractors, technically as well as administratively. And
PPP’s reform agenda must hew to USAID priorities. If not managed with aplomb, efforts to
control can create tensions in a country, and with a consultant team, with a high level of capacity
(see findings on the quality of staff above.) If there is uncertainty or lack of trust between
USAID and the IP, there can be a negative effect on the IP’s ability to manage its high-level
relationships with GoC and to get things done. Increased communication, though time-consuming, appeared to have been both appreciated by
all respondents who spoke to the topic, and useful for USAID’s maintaining technical oversight, perhaps with a bit of strategic distance. The decentralized technical management provided
detailed technical focus. Technical monitors provided timely and contextualized guidance to the
IP. At the same time, the model is not perfect, as it resulted to some extent in competing silos. Taking advantage of windows of opportunity was successful. As noted in government
relationship findings above, the new GoC presidential administration coincided with PPP and
USAID was able to create a lot of synergy through PPP. RECOMMENDATIONS
PPP’s model of flexible, consensus-oriented teamwork that responded agilely to the GoC should
be replicated as closely as possible for future efforts. Similarly, its recruiting criteria should be
used as a basis for team member selection: experience, technical knowledge and relationship
22
with GoC are all important characteristics. It is essential to get the contracting process right, early in the program, with a minimum of layers
of approval and potential for delay. IPs should look for ways to minimize contracting
requirements for consultants, or at least for alternative (such as interim) hiring mechanisms to
avoid losing good team members
USAID should look for a happy medium between an iron grip on policy efforts and allowing IPs
free rein. Particularly where relationships with GoC are strained, or where GoC offices have
expressed misgivings about “gringo involvement,” find ways to work at a distance through the
IP. Keep in mind the ultimate goal of successful policies: if this can be accomplished through a
trustworthy partner, the presence of USAID in individual meetings is not as crucial. USAID should replicate the structure with thematic technical monitors – including for cross- cutting themes – who have day-to-day contact with and control over IP activities in each pillar,
and extend it to include territorial–level activities that fall under the same topic area. USAID will
have to avoid creating isolated silos, but the technical focus is worth the effort. Hold biweekly
technical meetings (by pillar) and monthly cross-pillar meetings. IPs should have a planned closure (with budget, timing and activities) to show the results,
deliverables, and future recommendations to the government offices. This way the program leaves
documentation, a presentation and/or a final report for every GoC office served by USAID support.
23
ANNEXES
CONTENTS
Annex I: Findings, Conclusions and Recommendations Table .................................................... 24
Annex II: Design, Methods and Limitations................................................................................. 58
Design........................................................................................................................................ 58
Sampling.................................................................................................................................... 58
Respondent samples .............................................................................................................. 58
Activity Sample ..................................................................................................................... 59
Methods..................................................................................................................................... 60
Document review................................................................................................................... 60
In-depth interviews................................................................................................................ 60
Case studies ........................................................................................................................... 60
Data analysis ............................................................................................................................. 61
Limitations ................................................................................................................................ 61
Getting to Answers Matrix........................................................................................................ 62
Evaluation questions................................................................................................................. 62
Annex III: Information sources..................................................................................................... 64
Annex IV: Instruments.................................................................................................................. 67
Annex V: Evaluation Statement of Work ..................................................................................... 68
Annex VI: Conflict of Interest Forms........................................................................................... 79
24
ANNEX I: FINDINGS, CONCLUSIONS AND
RECOMMENDATIONS TABLE
Note to reader: the findings are comprised of quotes and paraphrased statements from the interview respondents. The origins
of the findings are indicated after each finding, by a letter G (GoC), A (AECOM), C (Consultants) or U (USAID). Where
multiple respondents from the same type of source are referenced, a number follows the letter (e.g., G2 or A3.) This protects
sources’ confidentiality while offering information on how responses were triangulated.
Findings Conclusions Recommendations
RELATIONSHIPS AND INFLUENCE: USAID – GoC – AECOM
How PPP worked with the GoC
Extended and in-depth group work
Having a large number of consultants fostered innovation and
consensus (A) More than PPP, “the Mission (DGI) did it” [By “the
Mission”, this respondent referred to PPP’s income generation
work in which he participated, comprised of a team of GoC
representatives from DPS, DNP, SENA and other bodies, plus
the AECOM PPP consultants.] (G).
Government respondents greatly appreciated extended,
collaborative work style (“useful, pertinent and very productive”)
(G)
PPP provided a comprehensive but small work team of
consultants, and this team had good relations with the GoC (A)
The GoC always gave feedback on the deliverables, which
allowed carrying out a follow up and control of the technical
Diverse experts brought
innovation and interesting
debates to the decision- making (such as Colombian
and international best
practices; see section below
for greater detail on this
theme.)
Efforts of extended duration,
in collaboration with the GoC, were an essential success
factor. Consensus-building was
critical to success.
Future projects should
replicate PPP’s practice
of working hand in hand
with the GoC on each
activity, over time,
seeking consensus, as
a team. Seek a common
ground between USAID
and GoC priorities (see
the following
recommendation) to
decide on thematic
areas and priorities.
Future Implementing
Partners (IPs) should
support a high-level,
25
assistance provided by PPP. (A)
GoC respondents valued the consultant skills, the extended
timelines of support (G2, U2) and, particularly, the collaborative
style of work (G4).
Please also see below the section on the process and success of
thematic selection, for a discussion of the importance of political
and thematic considerations that lead, in part, to the
recommendations at right.
ongoing round table with
the GoC that provides a
space for proposing
thematic areas for joint
action. Follow the
suggestions and
requirements expressed
there. If a given theme
is not part of both
USAID and GoC
priorities, it may well be
best to drop it rather
than continue without
consensus. Let this
process set the
Agency’s expectations
and goals.
Future activities should
replicate PPP’s staffing
patterns of high-level
staff and consultants,
with appropriately high￾level government
contacts, to ensure that
functionaries involved in
policy activities are fully
empowered to act.
IPs and USAID must
plan for the likelihood
that any effort that
involves inter-agency
arrangements will
Other important aspects of PPP’s way of working
One of PPP’s strengths was that it worked across different levels
of institutional maturity and functionality (U, G2)
To work successfully in GoC it is essential to work with the heads
of the relevant entities, not with mid-level staff (U2, A). PPP could
operate at high political levels. The staff’s close relationships with
GoC seemed to go around USAID at times. Also having a
Colombian COP was very useful; the Colombian staff were able
to open doors (U)
The fact that PPP could say “no” to GoC entities requesting
support provided considerable negotiation leverage and flexibility.
(U) At the same time, another respondent said that PPP did not
critique the GoC enough (A)
PPP staff’s personal ties were at the foundation of successes. (U2, A2)
There were two cases in the Victims’ Unit work when [AECOM
Staffer] couldn’t reach consensus with the DNP and with the
Unit’s General Secretariat. These were when [her team was]
trying to make their offering more flexible, and when [they] were
working on an index to measure the end of vulnerability. (A)
PPP staff had excellent
connections, at very high
levels, within the GoC. These
personal ties were part of the
foundation of PPP
successes. High level GoC
staffers are defined as the
ones in charge of policy
decision-making and budget
allocation, and PPP’s
connections at that level
made it possible to insert the
program where its activities
could be most successful.
Challenges and limitations:
26
The most the consultants can do is make the best
recommendations they can and hope that the GoC adopts them.
The consultants cannot guarantee implementation of the
recommendations. (C) For example, the Law on Rural
Development that PPP developed but that did not pass. At the
same time, the collaboration and discourse on the “territorial
approach” advocated by PPP and its GoC interlocutors has now
become part of the general discussion on how to better
incorporate rural productive areas post-conflict, including in
Havana. (U2, A2, C, G)
Interagency work was a challenge (C). When the new Unit was
created, it was in the midst of a battle for resources, because the
Social Inclusion Sector was also being created. (G) It would have
been better if our table of experts and representatives of SENA,
MinTrabajo, MinCom would have taken what we did together
back into their institutions. (G)
Politics always played a part (discussed in further detail below)
require more time, more
effort, more consensus,
more political will. Such
efforts are an order of
magnitude more
challenging than
working with just one
entity.
27
INTERNATIONAL BEST PRACTICES (IBPs)
International best practices were important in many of PPP’s efforts (C, A,
G)
A USAID respondent doubted that activities systematically considered
IBPs, saying that PPP shouldn’t “waste time reinventing the wheel.” (A)
Using IBPs successfully would have been systematic in the search for the
right response to use here, now, not improvising or working just from
theory (A)
Examples of IBPs in use
International seminars about rural development models were carried out,
and were integrated to the rural development management model. (C)
IBPs were brought in through literature review, seminars, consultants’
experiences, and specific and relevant examples from other countries
(known to the consultant or GoC teams, or researched by them.) National
examples were also used, such as Antioquia’s experience with victims’
reparations. (C) Victims’ Unit also looked at examples from former
Yugoslavia. (G)
The Consolidation Unit activity with PPP made use of international cases
from Brazil, Mexico and Chile. (G) I’m not sure if it was all AECOM’s doing,
but in the Vice Ministry we looked at best international practice cases like
Brazil, Peru, Chile, Africa and Southeast Asia and analyzed what could be
incorporated here. But forums, events and studies are probably less useful
elements, we really need more tangible support. (G)
“The IBPs were brought to the table in the form of my experience”. (C)
PPP staff brought a great deal of international experience and IBPs to the
table (A)
In the change from Acción Social to DPS, experts studied and offered IBP
information on specific themes like planning, M&E, methodologies and
work procedures. However, local expertise was fundamental to work within
the local context. (A)
IBPs were generally
effectively integrated to
PPP activities. There was
variation in the degree to
which IBPs were
incorporated, in part
because of the
differences in how IBPs
were brought to the table.
As a result, it seemed
there was a variable level
of trust on USAID’s part,
about whether there was
systematic consideration
of IBP options. The focus
appeared to be an inclusion
of IBPs as part of
deliberations on policy
design and formulation,
with less attention given to
IBPs that supported aspects
of implementation.
GoC partners valued the
support in cutting-edge
themes that IBPs put on
the table, while also
noting that these inputs
were not always
sufficiently tangible and
detailed. PPP and its
consultants also brought
Future projects
should always
scan international
experience for
applicable best
practices to
address policy
changes. If USAID
is concerned that
IBPs are not
adequately being
considered, it
could monitor their
systematic
inclusion activity
by activity, and
include in work
planning that each
policy or
institutional
strengthening
activity effort be
accompanied by a
systematic review
of relevant options.
The options pursued
should go beyond
policy designs to
focus instead on the
ways in which new
and reformed
policies can best be
28
IBPs had to be closely contextualized
IBPs were not uniformly easy to adopt due to the particularities of the
Colombian conflict (U) and the particularity of the topic of victims in
Colombia. (G, A) At times they were too abstract, not concrete enough,
with insufficient detail or, when there was detail, it was not immediately
adaptable to Colombian needs. (U, G) IBPs were taken as reference and
studied but not always incorporated. The experience of providing services
to victims in Antioquia was included since it was a proven experience in a
related context. (A, G)
Very good, important international help. But, it’s important to note that their
knowledge didn’t always translate to the Colombian environment (as in the
cases of the rural cadaster, and the income generation work with DPS) (U,
G)
national best practices,
and the ability to
incorporate IBPs in ways
that were appropriate to
the Colombian context.
This came from their
high-level technical
expertise and knowledge
of the institutional
environment.
As a result, IBPs were
useful inputs in some
cases, were perhaps not
considered systematically
in others, and were rarely
definitive responses in
any case.
implemented in
challenging
environments.
At the same time,
future projects
should remain
flexible about how
these IBPs are
sourced and
incorporated.
Remember that
the IBPs put into
practice must be
relevant,
instructive, and
able to be
incorporated into
the Colombian
institutions.
29
THE THEMATIC DEVELOPMENT OF PPP ACTIVITIES
Politics and the selection process
30
USAID was an ally, not only an aid partner. (A)
Deep interest on the part of the new president
was vital, and coincided with the start of the
PPP. (U)
Context mattered: Internal problems, like
agrarian strikes, affected the development of
PPP activities. (C)
When the mechanism is good, one will use it more
[as did the Office of the Presidency with PPP.] Their
[PPP’s] flexibility was key... it was what we needed,
when we needed it. They [PPP] were present. (G)
We had meetings, so many meetings with GOC in
the beginning, and we wrote the PWS listening to
them and to what USAID demanded. That’s how we
were able to meet our goals – we set them together.
(A) The first PWS was very, very detailed, every
comma and period was listed. In later years the
benchmarks were higher-order. (A)
According to AECOM interviews and the evaluation
team’s document review, PPP met all or nearly all of
its contract benchmarks (which changed
substantially after the close of the first year.) The
PWS in Year One required workshops and papers,
but the teams were able to adapt the content and the
duration of TA support as long as the document or
event was delivered as promised. Quarterly reports
show the achievement of milestones almost always
precisely as planned, even when plans changed.
Second, USAID CPARS (Contractor Performance
Assessment Reporting System) ratings were
consistently high for the first three years of the
Windows of political opportunity and commitment
were crucial dimensions of successful efforts.
Alignment of GoC priorities and PPP work themes
was important to the success of the project. When
PPP responded to GoC demands the activity
functioned better, and had more lasting effects.
PPP gave the government the impression of
flexibility, while also meeting their PWS targets
with USAID – this balancing act was deftly
managed.
Political pressures could limit common ground
within the GoC or between USAID and GoC,
reducing success. Policy and institutional
strengthening are limited (or enabled) by GoC
political priorities. PPP would diagnose issues
and draft recommendations. But, adoption of
these suggestions depended on the government’s
will to implement them.
Impediments to adoption of PPP
recommendations included:  Changes in GoC staff, priorities and interest  Corruption and regional politicking
 Lack of institutional capacity/maturation
 Context/external shocks such as the agrarian strike, fiscal
challenges, peace process, etc.  Delays due to AECOM and USAID procedures (please see
section on project management.)
These affected institutional strengthening even
more than policy; challenges were pervasive
through certain bodies such as INCODER,
MADR, and the Consolidation Unit.
Some policies and institutional efforts required
coordinating government agencies. Such efforts
USAID should start
from its CDCS to
identify thematic
areas of interest –
niches – where there
is a shared interest
with the GoC, and a
commitment from the
GoC. This
commitment and level
of interest (both at
USAID and GoC)
should continually be
tracked for changes,
because they can
and do change
unexpectedly. IPs
must identify
responsible parties
with sufficient interest
for each effort, across
the agencies affected
by a proposed
reform, and involve
them in decision- making on policies to
be targeted.
Where reforms or
activities affect GoC
vertically, through
regional levels, each
level must be
considered and
31
contract (Exception and Very Good for product
quality and Management of Key Personnel,
consistently.) PPP was also praised in the 2011
CPARS for accelerating progress dramatically, in
taking advantage of a window of opportunity with
the new Santos government. In response, USAID
shortened the timeline for completion of the contract,
while adding new funding, to address more GOC
requests and themes.
Political pressures
We couldn’t get past regional [INCODER]
leadership. They’re politically appointed and not
technically capable. Everything we did came
from Bogota-based people going to the field (U,
G) The changes in INCODER’s administration
blocked the complete adoption of the proposed
normative changes. (C) MADR just didn’t have
the capacity to do what [PPP consultant team]
were working on with them (U)
Consolidation Unit couldn’t get a CONPES
because of a lack of political support. A
technical document became a political football.
We came in in the middle, the hand-off was not
very smooth (G) Sector stakeholders made the
work nearly impossible. (G) The Respuesta
Rapida program was well done and it’s useful to
the Unit, however, the strategic planning wasn’t
used due to the lack of interest of the Director’s
Unit. (A)
Inter-agency challenges
VU’s interagency structure: Victim’s Unit
were the most problematic, due to weak
commitment, differing interests, and lack of
resources. Uncommitted partners perceive the
programs as just “more work.”
Creating new entities was less politically
challenging than changing an existing body, due
to entrenched interests and practices in the latter.
Land issues are politically sensitive, for the
number of interests involved. As the number of
GoC offices involved in a reform increased, the
effort tended to become more challenging, and
ultimately less successful.
included in planning
and execution. See
also the nation-region
section for additional
findings on this
recommendation.
Each reform effort
should have
concentrated,
politically savvy
attention.
32
depends on other institutions offerings to serve
the victims. The inter-institutional commitments
are very fragile. (C)
It was difficult to get participants from different
GoC offices (SENA, MADR, MinInt, MinComm)
to take the agreed-upon actions back to their
institutions and get them implemented. (G)
Where some things did not change, it was due
to trying to work jointly across competing
agencies interests. (G)
33
Selecting Themes
34
Overall
Nearly all PPP respondents mentioned the
change in PPP structure as an important
rupture. They report that it was driven by desire
from the top Mission leadership that all topic
areas had to conflict–related. (A) That change
elevated some themes and diminished others;
at times without resources to ensure their
successful completion. (U, A) There was a lot of
pressure to do a lot. PPP had strong leadership
and some very important themes to work on, in
which there was plenty of GoC interest. The
three pillars helped to focus the program (C) At
the same time it was challenging to respond to
three major pillars, with significant reforms, in
the same program. (U)
Large reforms were fraught with competing
definitions, unclear goals, and old institutions’
baggage. Smaller targeted efforts were more
fruitful. (G) We were able to re-do and update all
the archives – which were a mess – and
computerize it all. Now going to INCODER is
pleasant, not a cave of papers and files (G)
Some products just sit on the shelf at the GoC.
(U)
“Politics” refers also to USAID switching
priorities (A2, U)
Specific themes
PPP’s Victims pillar was more proactive, with
more visible outputs or products. Others were
PPP’s three pillars helped to focus the
program. However, each pillar was
extensive and worked across multiple GoC
institutions. Because of this inter-agency
nature, the work was at times more difficult
to take root.
Smaller, targeted efforts were less
problematic, and needed less time, money
and effort to reach consensus, than did
overarching reforms, especially where
these crossed agencies. Supporting GoC reforms that were not yet
funded sufficiently proved successful,
where political will for the reform was
sufficient.
Themes that aren’t high short-term GoC
priorities are difficult to develop.
PPP's mandate did not extend to regional
implementation. Full piloting would likely
have run up against important problems - capacity gaps, contracting needs, special
auditing and anti-corruption needs, how to
make budgets and resource flows
compatible from these differing bodies - in
short, the types of problems that any
complex, inter-agency program could be
expected to face in implementation. But
PPP's mandate was to design the
mechanism and not to implement it; it is of
course unknowable what adjustments, and
what learning, would have taken place had
Work closely with the GoC to
identify policy reforms that are
high priority for both, in line
with USAID’s CDCS and with
GoC priorities in the National
Development Plan, potential
peace accords, or other high￾level areas. Working with a
presidential administration that
is favorable to desired reforms
means tailoring the contracting
process to coincide with that
administration’s own timing.
Remain flexible during
implementation to take
advantage of windows of
opportunity. Planning should
include attention to targeted,
single-agency efforts as well as
over-arching, inter-institutional
reforms; where the latter are
contemplated, build in more
time, effort and resources.
35
reactive. Access to Financial Services and
Labor had strong results (U) From the approval
of Law 1448/11 PPP was side-by-side with the
GoC, with a team dedicated to the effort. GoC
had no budget for this. (G)
One of DPS’ needs was the Income Generation
Mission (G) For Consolidation Unit, strategic
planning, index, and dashboard were the “spine”
of cooperation (G)
There were contradictions between the rural
development law and environmental rules. The
land and development pillar met only 10% of its
goals as a result. (U)
The Contratos Plan (CP) mechanism was
designed with close attention to Colombian
systems and weaknesses, while offering a
profound change and very new mechanism with
integrated goals from national and territorial
levels, for 8-10 year time horizons. Importantly,
the GoC (DNP) took ownership of the design,
showing their commitment to making it work and
implanting their own criteria, and building on
Colombian-based theoretical precepts, such as
those of Orlando Fals Borda, inlcuding the
regionalization of social and state institutional
problems, and their localized resolution.
However, the DNP did not take on board PPP's
key recommendation, to ensure that the
creation of an independent (not national, not
territorial) management system. As a result, the
constituent bodies were left in charge of
PPP taken the pilot to conclusion. But
these problems rather must be worked out
in practice, using the experience as a
proving ground to address the unique
Colombian challenges Don mentions here.
No policy mechanism could foresee how
the challenges would play out, especially
considering the significant regional
differentiation of governance and power in
Colombia.
36
implementation (with their own agendas, for
example.)
PPP worked with the pilot Contratos Plan
programs only to the point of pre-accords.
PPP's COR cut support to the regional level
design and preparation (in Nariño) before the
pilot was done.
With the Victims’ Law, the Constitutional Court
requested a costing of the Law after passage,
which is the correct regulatory sequence for
Colombia, and the Ministry of Hacienda carried
out the study. The GoC requested further
support from PPP to plan the CONPES
document that would allot resources. PPP thus
worked to write the regulatory documents for (1)
the creation of the institutional framework
necessary to carry out the law, and (2) the
development of the policy, in the CONPES. In
this work, PPP estimated 80MM COP for
implementation of the law, of which the GoC
allotted 54MM COP. (D, A)
37
Cross-cutting themes
There was a spoken commitment to the
differentiated approach across PPP’s pillars.
This was incorporated more easily in the
Victims’ Unit than in other pillars, including
attention to ethnic communities. (A3) The
cross-cutting topic managers worked without
budget, convincing the pillar leads of the need
for addressing themes related to the themes,
but goals and expectations for these themes
were unclear. (A2)
Those cross-cutting themes that were pillars in
the first AECOM contract are slightly easier to
tease out - totally a bit over $2MM COP in
contract-by contract allocation. But Good
governance, Nation-Region, Gender and post- modification Access to Financial Services, are
components of given contracts that are alloted
to pillars in the AECOM expenditure data we
received. We are just wholly unable to estimate,
for example, how much of a given contract (like
1.2.3 for Benchmark Diseño de Medios de Vida
Sostenible en el Marco de Políticas) on Land
Restitution should be attributed to gender, how
much to good governance, how much to Nation- Region, and how much to the pillar area itself.
Other evidence points to a lack of real
commitment to the cross-cutting issues. The
USAID technical monitor on environment had
$70M in programming compared to no budget
with PPP – this project was not his priority. (U)
All budget for cross-cutting issues had to come
Cross-cutting themes in USAID’s initial
design did not become fully cross-cutting
as implemented by PPP. Lack of
appropriate expertise or influence on the
team (environment, gender), and political
challenges or lack of mandate (nation￾region and good governance) limited
integration in the pillar structure. Achieving
success in the pillars was demanding,
even without including gender, for
example.
PPP’s leadership focused on the pillars
and their deliverables, which meant that
cross-cutting themes were put on the back
burner. Neither USAID, GoC nor PPP
appeared to place gender or environment
high on their priorities.
In some PPP activities (for example,
strengthening certain institutional
structures in the Consolidation Unit), cross- cutting themes were not included because
they did not apply.
Changing USAID priorities based on high￾level interest or lack of interest in a given
theme made for instability in the PPP
team’s planning and implementation.
A strong champion for a given cross- cutting effort (such as financial inclusion)
can have a big impact. The lack of
specialist team members who are also
strong advocates, limited PPP’s ability to
If a theme is important
enough to include as cross- cutting, USAID and IPs
should ensure that it is
explicitly covered, with
appropriate expertise, with
its own budget, on relevant
topics, with sensitive M&E
targets, and adequate
resources. IPs must be
explicitly engaged with
these themes, with agreed
upon targets, or they are not
likely to be addressed
adequately.
USAID should monitor that
cross-cutting themes are
genuine priorities during
implementation, beyond any
proposal language.
38
out of a pillar’s budget. They had no resources
of their own. (A2) Who at AID cared about
gender? Who had one meaningful conversation
about it? (U) The program mandate did not
include implementation, or taking the policy
work down to the regional level; the contract
said nothing about institutional strengthening.
(U2)
I had to convince the pillar leaders that they
needed me, because I had no budget. The
transversal matrix looks good on paper but I had
to force myself on pillars where they were not at
all focused on these themes. They didn’t even
know they needed certain support. (A2)
PPP’s COP had particular interest in two pillars,
and the DCOP in the third pillar. (A2, U) The
DCOP is mentioned as having a particular
interest in bringing reforms to the territorial
levels with appropriate institutional reform. (A,
U) Neither reflected deeply on any of the other
cross-cutting themes, nor were mentioned as
drivers for them.
For some of its cross-cutting themes, PPP
lacked appropriately skilled persons to
champion them. The pillars were all-consuming,
with enormous responsibilities, and gender and
environment, for example, were very low
priorities for government and PPP. (U, A2)
Cross-cutting activities became smaller and not
actually transversal. (A)
For Consolidation Unit, the cross-cutting themes
bring (for example) a gender lens to bear
on a range of efforts with competing
priorities.
39
didn’t fit in with the work substantially. For
example, with the Index, we understand the
need to disaggregate, but our institutional
strengthening didn’t really have anything to do
with the differential approach at the national
level. (G)
Financial inclusion suffered from having a small
budget that was cut time and time again.
Mission priorities changed, and rural
development work was no longer valued at high
levels in the Mission. At the same time, this PPP
team was able to accomplish a great deal with
limited resources, in part because of strong
commitment and tireless work from the
Mission’s technical monitor. (U) PPP’s
assistance was very good in articulating
different actors like government’s entities and
the private sector. This was carried out with a
fund for which private sector partners competed,
and awareness events with the financial sector.
(G)
Gender did not have such traction. It worked
with the Alta Consejería on Gender-Based
Violence, which was not linked to PPP’s pillars.
(A) PPP lacked technical specialists in two
cross-cutting themes: gender and environment
(A). Environment focused on response to
victims of a natural disaster. When those efforts
to restore housing for the crisis victims had
ended, this component was closed. (A)
Nation-region (detailed in its own section below)
40
Long-term nature of reform
Several PPP staff and consultants worked on
the same policy efforts over time, both before
PPP in ADAM/MIDAS and after it in the Land
activity. They noted that the policy efforts they
were working on were of necessity long-term –
significant institutional change was challenging
and took time, and building political will for
policy change also took time. The more
contentious and political the policy proposal,
they noted, the longer it would take to build up a
coalition to make the change. But their work
was coming to fruition, with both land and rural
development and access to financial services
policies still making progress a year after work
on those areas had finished (A3, U) In the
discussion of the first point of the peace talks,
there were strong links to the work we had done
prior to that on rural development and the
integrated, “territorial” approach that has been
so well received by rural producers and others,
and touching on such important issues that we
will face in post-conflict. (G, A) What we worked
on with ADAM MIDAS laid down the basis for
what we did in PPP. (A) They’re about to pass
the financial access rules we advocated. (A)
We now work on land and rural development
law. What we worked hard on – a territorial
approach – is part of the discourse here and in
Havana [i.e., at peace talks.] (A)
We brought those people together and now our
proposal on land and rural development
The ultimate impact of USAID-funded
policy projects are often not clear during
the project period. Sometimes they come
to fruition under a later project when the
timing is more propitious as the process
matures. But, the analysis and processes
begun are vital to subsequent project
generations.
This is also true of the professionals hired
to work on these projects, who often are
hired by subsequent projects and/or move
in and out of public service with the GoC.
They constitute important resources.
USAID’s policy reform
efforts should be thought of
as a portfolio of initiatives
that mature at different
times, based on issues
discussed elsewhere in this
document. USAID should
monitor the progress of
various initiatives and
accelerate or pause
activities accordingly –
recognizing that ultimate
impact may go beyond the
project period, if the
initiatives are powerfully
seated within the GoC.
41
comprises the discourse. (A) We had to bring so
many people into it- all the thinkers on these
topics who otherwise would have shot us down,
if they hadn’t been included. (A, U)
Shortening the program to 3 years was a
mistake. We had to run as if in a marathon, but
policy efforts are more long-term, particularly on
controversial, robust themes with many interests
that all have to be included in the discussions.
(A2)
42
Nation-Region
Examples PPP’s efforts, successes and
challenges:
PPP worked with the Victims’ Unit to develop
the theme of territorial presence (expanding
upon a prior vision that was restricted to a
national level.) Though PPP proposed this link
the team did not have the contractual mandate
to operationalize and implement it. (G) In the
social inclusion sector, PPP provided important
inputs to the final structure of the sector,
including its regional presence. (G)
PPP helped orient financial products toward
low-income populations and micro-business
people, and helped to increase and improve
coverage. (G)
PPP successfully promoted integration of
regional level indicators into INCODER’s
territory-level plans. (G)
In the design of the interagency coordination
system, PPP team helped in the coordination of
regional entities for the supply of state services. Still, what is lacking today is how to address
regional differences. (G)
Though PPP proposed the importance of
territorial linkages (see above) it did not have
resources or mandate to operationalize and
implement it. (G, U)
It was difficult for this national-level project to
see the experiences from more local levels,
PPP’s mandate was at the national level
only. At the same time, getting the policies
implemented – especially down to the
regional level – was important for eventual
impact. Other USAID programs were
tasked with regional work. However they
and PPP did not work as closely together
as might have been required for success. Results are therefore mixed. Some
successes are noted but relative to the
level of USAID priority, the territorialization
of results was not as successful or as
visible as desired.
Include regional
participation and
coordination to support GoC
decision making within
USAID policy projects, and
among the family of USAID
interventions, in future
USAID-supported reform
efforts. At the same time,
set goals for regional
implementation and uptake
where the key geographies
for execution are regional
rather than national.
USAID should link national
and regional USAID
programs better, or combine
national and regional
activities for one pillar in one
contract, so that the IP can
strategize and implement
vertically in partnership with
the GoC.
43
especially with respect to policy implementation.
Nevertheless, regional-level impacts were a
priority for USAID. Other IPs were charged with
addressing regional needs. PPP had to
abandon efforts to reach the regional level. It
was beyond its mandate (U)
44
INSTITUTIONAL STRENGTHENING AND TA STRATEGIES
45
Examples:
The Institutional Strengthening Component was highly
valued, such as in civil service reform. This was a smaller,
targeted effort. (A) There were other such efforts, all sort of
juggled by [name redacted]. These were focused efforts
within particular, amenable offices. (A, G, U)
PPP’s success was in funding receptive government agents
who needed help, every day for a year, in much larger ways
than international best practices language would suggest (U)
The identification of failure in the institutional order and the
recommendations are important to the needed modifications
of the agrarian processes and will be very useful to the GoC
in the future. (C)
It was an achievement to mount the structures of the Victims’
Unit and Center for National Historical Memory. But much
harder to put into action, with institutional strengthening. An
example is the Unit’s targets, which seemed to be based on
political concerns. (U)
PPP was not charged with bringing policy down to the level of
the government user, in implementation. PPP did institutional
gaps diagnostics for us. But the effort was lost, and impact
low, because there were limits to GoC absorption. (U) The
Victim’s Unit design is inefficient. It has no procedures or
budget. It cannot offer direct services to the victims; it has to
coordinate with other entities. (C) The results in the
institutional strengthening were limited in cases where the
GoC counterpart was not strong. (C) Results in INCODER
were strong on activities like the archiving project, as it was
no threat to political interests. But institutional reform never
passed, for the opposite reason. (G)
The lack of technical skills in the MADR has kept the
Post facto, there is reversion to the
mean in highly transient bodies of the
GoC: reforms and institutional
strengthening activities are diluted
over time and where staff are
transient.
Smaller, targeted efforts with real
constituencies – especially with one
or two agencies, rather than a
multitude – were more successful.
Institutional strengthening worked
better in a new institution like Victim’s
Unit than in an existing one, like
INCODER, where reform raised
hackles and was resisted
vociferously. Political pressures were a key
obstacle to efforts, as it is difficult to
obtain consensus within the GoC or
between the GoC and USAID.
Activities are limited by GoC political
priorities. PPP did diagnostics and
wrote recommendations, however the
adoption of these proposals
depended on the government’s will to
implement them.
Impediments (as noted above in the
thematic selection section) included:  Changes in GoC staff, priorities and interest  Corruption and regional politicking
 Lack of institutional capacity/maturation
USAID must make
explicit any goals for
institutional
strengthening just
as for policy, if such
efforts are
prioritized. If these
reforms are
expected to be
operationalized,
including at regional
levels, this must be
clearly designed,
budgeted,
monitored and
carried out. As with
policy goals, it is not
worth the expense
or effort if the GoC
is not fully on board.
USAID should
choose these
activities in full
consultation with the
GoC and include an
assessment of
whether the
institution has the
absorptive capacity
for the reform, as
well as the political
commitment to
follow through with
46
institution from completely appropriating and using the
information from the consultancies. (C) Similar comment on
the Consolidation Unit. (C)
In at least one occasion, USAID stopped PPP’s support (to
Contratos Plan.) “Adoption” could have been more complete
had USAID allowed PPP to work regionally. (A) USAID did
not have institutional strengthening as a standalone
requirement in the contract. No budget; no staff. If USAID
wanted it, they should be able to point to it in the contract. (U)
 Context/external shocks such as the agrarian
strike, fiscal challenges, peace process, etc.  Delays due to AECOM a/o USAID procedures
(please see section on project management.)
These affected institutional
strengthening even more than policy;
challenges were pervasive through
certain bodies such as INCODER,
MADR, and the Consolidation Unit.
it.
47
High technical quality supported PPP’s
success
PPP was largely successful in meeting its
deliverables and contractual requirements
(Desk review) PPP’s deliverables were of high
quality. (G4) The deliverables were reviewed by
the technical staff and then by the contracting
office to make sure of the TOR’s compliance.
Also, there was a knowledge management
office to review and classify the deliverables. (A)
PPP was successful in creation, institutional
capacity building and policy implementation at
the Victims’ Unit, because of the quality of
people and deliverables. (A) Today the sector
is well positioned, in part because of the quality
of the work from PPP – like Alzate in the VU.
(G)
The quality of the work in access to financial
services (later cross-cutting) were very high,
despite limited budget (G). The private sector
was very interested. (A, G, C) Banca de
Oportunidades: Qualified consultants and staff
transferred knowledge, defined priorities, and
strategies with the staff of BdO. All this
improved access. (G)
The most important success was PPP’s clarity
and vision from day one, on the [VU] registry,
assistance and reparation. (G) In designing the
unit & training staff, PPP changed the VU’s view
of victims, from assistentialism to a reparative
and differentiated approach, in line with the new
PPP’s staff quality-controlled the
deliverables. They knew what the client
expected and ensured it.
An important part of PPP’s success was
the excellent selection of staff and
consultants. The personal relationships
with the GoC, high-level national
consultants, quality in the deliverables and
commitment were key elements
consistently identified by the GoC. Committed people, effective results, able to
work point by point with their GoC
counterparts, over extended periods.
Mutual confidence between PPP staff and
consultants and their GoC counterparts
was an important element of successful
initiatives. (See also the section on
relationships with GoC, above.)
IPs should use strong and
connected Colombian
teams, including individuals
with significant USAID
project experience. IPs must
cover the thematic areas of
the program with this kind of
quality technical support in
staff and consultants, and
with cross-cutting themes as
well.
The staff as described
above should be
predominantly Colombian.
Where international
expertise is sought and
included, ensure that that
international presence is
balanced with Colombians.
The connections and
understanding of local
context are important for
what they bring to the work.
48
law. (U)
PPP’s leadership, technical and managerial
staff quality, and strong staff relationships with
high-level GoC were important to PPP’s
successes. High level team with good
experience. PPP was willing to adapt to the
GoC needs and support them. (G) PPP hired
the best consultants and consultancy firms. (A2)
We had good, useful debates. (A)
USAID involvement was not always welcomed
(U, A4, C2; see further comments below on
USAID management, guidance, trust) such as a
high-level GoC staffer saying they didn’t want “a
bunch of Americans to work on this.”
PPP almost never suggested an activity with
which the GoC wasn’t already on board. (U, A)
The staff were up on the issues and on their
counterparts’ interests and limits.
PPP’s teams provided high quality documents,
they were pertinent and flexible. (G)
I don’t want to score the support [on the scale of
1-5] because I came into the project late, but I
will say the support was very valuable because
it came at a time when our very young institution
was growing very rapidly. PPP followed up with
us continuously and were very helpful.
On the 1-5 scale, with 1 being very poor and 5
being very good, government respondents rated
PPP with the following scores:
The totality of the technical tools: 4.5. (6 voters)
49
Results of PPP guidelines and practices: 4.5 (4
voters)
Effectiveness of the TA: 4.5 (7 voters)
Quality of the TA: 5 (6 voters)
Quality of the institutional strengthening, and
quality of the use of international best practices:
4.2 (2 voters each)
The interview questions were used as guides for
protracted conversations with respondents, and
as such not all respondents answered all
questions (also, not all questions pertained to
the work with all respondents.) However, the
evaluation team feels that these “scores” are
quite indicative of the verbal responses in the
interviews even when numeric responses were
not provided.
50
PROJECT MANAGEMENT
PPP’s management
51
Successes and good practices
Work was well-planned. (A2). Efficient support,
flexible, fast. (U, C, G) PPP’s way of working was
in teams with GoC, staff and consultants. (C)
Without good management, the GoC can be
dysfunctional (A) PPP had high-level staff close to
the decision-making positions in the GoC. (C)
PPP assumed high risks and was successful. PPP
faced vested interests and there were political
costs involved in tackling these themes in
Colombia. (C)
PPP selected a team with experience, knowledge
of the themes, people in GoC and local context,
some government insiders and others from
outside. (A, G) Efficient support thanks to the
consultants’ high levels, which guaranteed direct
communication with the GoC representatives. PPP
team adapted to changing priorities of government,
and knew its weaknesses. (G) They were frank
with us, and told us “no” when they thought the
government shouldn’t be outsourcing some piece
of work. (G)
Challenges
Administrative and financial themes in PPP made
work difficult; we lost one valuable consultant who
could no longer wait. (G3) The dynamics in the
Ministry require promptness, but there were long
delays in the administrative and contracting
process. (G) There were no clear rules to follow in
the admin process. (A) Contracting was tedious,
and complicated. (G, C, A) Contracting took too
long. The rules were too many and somewhat
PPP got many things quite right: it was
flexible in their support, adapting to
USAID and GoC needs in many
instances. PPP successfully identified
the right people for the job (staff and
consultants.) They:  Were of high technical quality and
knowledge level  Had important connections with GoC
 Were able to identify policy and/or
institutional gaps and propose
acceptable solutions
PPP’s teamwork model also helped to
secure consensus (see also the findings
in the first section on the relationship
between GoC and PPP.)
The contracting model used in PPP had
many important technical benefits, such
as having the right people for the tasks
at hand, at the right moment. (Please
see also the findings in the section on
how PPP worked with the GoC.)
However, AECOM’s contracting
processes were tedious and
complicated the work. It is essential to
get the contracting process right, early
in the program, with a minimum of
layers of approval and potential for
delay.
PPP teams did not uniformly inform
USAID of advances with sufficient time
PPP’s model of flexible,
consensus-oriented
teamwork that responded
agilely to the GoC should be
replicated as closely as
possible for future efforts.
Similarly, their recruiting
criteria should be used as a
basis for team member
selection: experience,
technical knowledge and
relationship with GoC are all
important characteristics.
IPs should look for ways to
minimize contracting
requirements for
consultants, or at least for
alternative (such as interim)
hiring mechanisms to avoid
losing good team members.
52
absurd. (A) AECOM’s contracting process was not
efficient. Consultants had to provide a lot of
deliverables as a way to control payments and this
meant a lot of deliverables to review and rate. (A)
At times PPP did not reconcile its work with what
USAID wanted. It’s a huge challenge. (U) The
program used hundreds of contracts, with
individuals and firms. (A2, U2) requiring home
office & USAID approval, and could be so slow as
to miss programmatic windows of opportunity.
for their participation. This eroded trust
with USAID and resulted in USAID
wanting even closer control.
53
Cost-Effectiveness
Administrative costs were appropriate for a
PPP. (A)
How much have we dropped into consolidation
municipalities, for very low ROI? 10:1 for PPP
versus other USAID projects (U) Compared to
my current project, the expenses for PPP were
fractional. (A)
Contracting slow, justifying expenses. They
were very cautious with expenses and being
audit-ready. That took a lot of time. Also took
additional time every time they had to correct
the work of a subcontractor. (G)
Some consultants were guided to leave after
being paid. Should be more commitment. (G)
Others took less than their normal rate, just to
participate (C, A)
Worthy. Contributed a lot to us in only 6 months
that the (income generation) Mission lasted. (G)
This was a project with low resources and very
significant impact for us, deploying nationally in
very little time. (G)
Though PPP’s expenditure data per activity
were not available, indications from
government, USAID and the contractor
indicate (1) close attention to spending
rules and regulations, (2) attention to
national-level priorities which leveraged or
attempted to leverage effectiveness at
more local levels, and (3) impressionistic
evidence of policies and reforms
implemented successfully, at reasonable
cost.
54
USAID Management
Guidance and trust
There were communication channels through meetings
with the technical monitors, quarterly reports, weekly and
biweekly reports, where the main advances in politics and
the main challenges were shared. (A)
The Biweekly reports/meeting were a lot of work for PPP, but at the same time, were a good opportunity to study the
context, be updated to the current policy debates in
different topics and show the current advances in the PPP.
(A2, U2)
Where they had trust top to bottom – MD, COTR, TM,
AECOM, GoC – things worked. Without it, nothing else
mattered. (U)
First Mission Director gave PPP free rein; the second took
it back (A, U). If USAID person is in the room, the meeting
is not the same. (U, A) USAID’S presence in the meetings
was sometimes counterproductive. In ADAM MIDAS it was
different: the team had USAID’s total confidence in the
meetings with the GoC. (A)
Although USAID succeeded in procuring well-connected,
high-level staff on the PPP team, the Mission sometimes
tied those experts’ hands, insisting on closer and closer
control. USAID changed guidance about relations between
AECOM and USAID. At the beginning the team was
accorded broad freedom. Over time USAID staff exerted
increasing control and was more interventionist. (A3, U)
One example: The relationship between USAID and the
DPS was positive and helpful, they [USAID] didn’t appear
much once we were working well together. But they always
As Mission leadership and
management rotated, they changed
the rules of engagement for the IP. In
some respects the change was
significant and affected the IP’s ability
to do its work with sensitive GoC
offices.
USAID oversight created some
problems. There is a desire among
many GoC counterparts to do the work
without USAID’s explicit presence.
GoC representatives raised formal
complaints, and it is unknown the
extent of the damage to the
relationship this may have caused.
USAID has an obligation to oversee its
contractors, technically as well as
administratively. And PPP’s reform
agenda must hew to USAID priorities.
If not managed with aplomb, efforts to
control can create tensions in a
country, and with a consultant team,
with a high level of capacity (see
findings on the quality of staff above)
and the ability to manage their high￾level relationships with GoC to get
things done.
Increased communication, though
time-consuming, appeared to have
been both appreciated by all
USAID should look
for a happy medium
between an iron grip
on policy efforts and
allowing IPs free rein.
Particularly where
relationships with
GoC are strained, or
where GoC offices
have expressed
misgivings about
“gringo involvement”,
find ways to work at a
distance through the
IP. Keep in mind the
ultimate goal of
successful policies: if
this can be
accomplished
through a trustworthy
partner, the presence
of USAID in individual
meetings is not as
crucial.
55
worked in a spirit of support when they were there. (G2) A
more challenging example: USAID, (…) were not
professional, even lacked respect for GoC counterparts
(G). A couple of complaints were raised by the GoC with
the USAID MD. (U) At times we receive invitations for an
event in which the Ministry’s presence is announced right
in the invitation, without USAID ever agreeing with us what
kind of participation we’re supposed to be bringing to the
event. They over-reach with us. (G) PPP team members
entered their upcoming meetings in OneSource, to which
USAID had access and both USAID and AECOM had
approval rights. (A)
With Economia Urbana, there appeared to be a very
serious case of mistrust with respect to a PPP staff
member… you can’t climb out of that trust deficit. Both
sides documented it poorly. (U)
respondents who spoke to the topic,
and useful for USAID’s maintaining
technical control, perhaps with a bit of
strategic distance.
There appeared to be a precipitous
decline in trust during the third year of
the program. This deteriorating
relationship had effects on activity
outcomes in some cases.
As noted in the findings of the
management section that follows, the
lack of “closure” – some kind of formal
handoff from USAID and AECOM to
the GoC – was keenly felt.
56
Technical management: structure
Newsletter, Weekly Meetings, Briefers were good
communication methods. (U) PPP was good at delivering
reports and had a good monitoring and control system.
The multi-pillar meetings at USAID provided valued
integration space for USAID managers, but only at the end
of the project. (U) The methods adopted to encourage
cross-pollination worked, though apparently these were
adopted late in the project (U2)
Multiple projects, instead of one, fought for resources.
There was no meaningful self-evaluation of the project. No
common vision of success. There were benchmarks for
GoC, but not for us. (U)
Better to have one COR with one mandate. (U) Hard to
communicate with 6, 8 heads. (U)
The decentralized technical
management provided detailed
technical focus. Technical monitors
provided timely and contextualized
guidance to the IP. At the same time,
the model is not perfect, as it resulted
to some extent in competing silos.
Taking advantage of windows of
opportunity was successful. As noted
in government relationship findings
above, the new GoC presidential
administration coincided with PPP and
USAID was able to create a lot of
synergy through PPP.
On the other hand, USAID did on
occasion then lose those windows
through inefficient internal
management. There were cases
where the USAID response timeline
was longer than the GoC was prepared
to wait.
USAID should
replicate the structure
with thematic
technical monitors –
including for cross- cutting themes – who
have day-to-day
contact with and
control over IP
activities in each
pillar, and extend it to
include territorial –
level activities that fall
under the same topic
area. USAID will have
to avoid creating
isolated silos, but the
technical focus is
worth the effort. Hold
biweekly technical
meetings (by pillar)
and monthly cross- pillar meetings.
IPs should have a
planned closure (with
budget, timing and
activities) to show the
results, deliverables,
and future
recommendations to
the government
offices. This way the
program leaves
Technical management: quality
USAID is opportunistic, in a good way, taking advantage of
the windows of opportunity offered by the GoC. (C)
Where we lost out was when the whole thing took too long,
and the window of opportunity closed. The difficulty here
was more with USAID. PPP’s consultants rarely produced
something not already agreed with the GoC. (U) One
respondent reported 42 days’ wait for a contract approval.
(A) Internal USAID response times and procedures
stretched out; products arrived late to GoC. (U2) On one
occasion, AECOM said it would manage a regional office
[of Contrato Plan]. AECOM discussed it with USAID, who
asked for justification and eventually denied it. We lost a
lot of time waiting for the decision. (G) USAID requested
no large close-out event be held across the project. PPP
57
handed over with each GoC partner individually, over six
months'. This included handing over the technical reports
to each partner. (A)
Success requires clear, logical procedures: dialogue with
the GoC; procedures for planning and execution; internal
consensus on objectives; timelines. (U)
There was a lack of closure, no ceremony or handover to
the GoC at program end. This made it feel less
accountable to the GoC. (U, G,A4)
IOM and PPP both worked on the Victims Unit – it was
very inefficient. (U)
documentation, a
presentation and/or a
final report for every
GoC office served by
USAID support.
58
ANNEX II: DESIGN, METHODS AND
LIMITATIONS
Design
The evaluation design was systematically qualitative. Activities undertaken in PPP to support
policy and institutional strengthening environments are complex, with multiple stakeholders and
potential spoilers, in an evolving context of political will and interests. Further, program effects
are nationwide but quite diffuse. These conditions make quantitative study less helpful than
understanding the how and why of program practices and lessons. Methods used in the evaluation
therefore included in-depth, semi-structured interviews, document review, and case studies.
The data from such a study are by definition subjective: each respondent is likely to have a
unique perspective on the topic under discussion. To be able to review those variances and weigh
the values placed on them, the evaluation design is based on triangulating sources. For each case
study, for example, perspectives were sought from the GoC, USAID, AECOM, and consultants.
This annex relates the complete set of evaluation methods, along with the team’s completed
Getting to Answers matrix. The latter shows, in tabular form, the relationship between the
methods used for data collection and analysis, and each of the evaluation questions listed above.
Sampling
Respondent samples
The team reviewed project documentation to understand the project’s activities, its clients, its
modes of operation, its accomplishments and its challenges. The team then aligned the many
policy and institutional strengthening contracts executed over the life of PPP against the higher- order objectives they sought to address. Against this universe of PPP activity the team:
A. Purposively selected a set of GoC clients with substantial involvement with PPP to
interview;
B. Interviewed USAID staff (Contracting Officer’s Representative and thematic Technical
Monitors);
C. Interviewed key AECOM staff and consultants; and
D. Purposively selected five case studies into which to delve more deeply, to illustrate
lessons learned and best practices.
The evaluation team conducted 36 in-depth interviews. AECOM staff and consultants comprised
19 of these; along with five USAID staff and twelve GoC actors. Of the individuals consulted, 25
were related to the case studies. Of the respondents, 19 were women and 17 men. To the extent
possible, the responses of individual interviewees are anonymized in the report, to ensure candor.
The full list of respondents is included at Annex III. The evaluation used qualitative methods to
extract lessons learned for the Mission in terms of institutional strengthening and policy reform
59
programming. Based on the evaluation questions, the team designed the interview protocols to
capture different perspectives regarding each activity under study.
Activity Sample
In concert with the USAID technical and program offices, the evaluation team selected a
purposive sample of PPP activities to include in the evaluation. Cases were classified by their
predominant area of effort (policy or institutional strengthening) and the impressions of program
results – whether tending toward the positive, or toward the negative. The range of interviews
provided a more robust picture of each case and its nuances and, especially, its lessons learned.
This sample includes the core of the four case studies, as follows, and a set of ancillary results
presented in a second table:
Table 1: Case studies
More positive impressions Less positive
impressions
Policy effort Access to financial
services
Rural development law
Institutional
strengthening
Victims’ Unit Consolidation Unit
Table 2: Additional themes examined in the sample:
Thematic area/
transversal axis
Specific result set
Victims’ pillar The treatment of gender in the
unit
Consolidation pillar Consolidation Index
Land and rural
development pillar
INCODER strategic planning
process
Nation-region, good
governance
Contrato-Plan, royalties law
In addition, while data collection was underway, USAID requested that the evaluation team
examine a fifth case study, on income generation activities undertaken with the GoC.
60
Methods
Document review
There are a wide range of documents of import to the evaluation. The team reviewed work plans,
quarterly reports, and PWS for the life of the PPP. Deliverables for each case study were also
read closely. Each case study also required a review of policy and/or institutional strengthening
documentation as suggested in the table below. For those activities selected for the further in- depth case studies, this document review expanded to include all relevant documentation as part
of the case study.
Table 3. List of documents consulted
Implementer documents Policy documents Institutional strengthening
 Work plans  Quarterly reports  Deliverables  PWS
 Contract and activity
databases and
spreadsheets
 Conpes  Regulations  Decrees  Circulars
 Process manuals  Strategies and missions  Functional plans and
strategies or mission
statements  Diagnostic documents
In-depth interviews
In-depth interviews were used in the evaluation of the PPP to examine the activities from
different perspectives. Evaluating an institutional strengthening activity, for example, included
interviews with the implementer and contractors, as well as the group of government staff and
decision-makers involved. To the degree possible, EVAL interviewed those individuals who
were involved at the time of the activity, as well as any replacement personnel, to understand
both the conduct of the activities and any ongoing results the present staff were be able to cite.
The interview guides for various parties were constructed to answer the relevant evaluation
questions from the evaluation design, per the individual respondent’s role in or with the program.
The team also adapted these questions in the field, when themes became emergent in an
individual interview or across a set of interviews.
Case studies
Case studies are a way to look at activities, opinions, institutions or other phenomena in human
social relations, as part of social science research. They are qualitative and provide insight for
individual cases rather than a generalizable experience. In social science research, case studies
are often best used in four circumstances:
1. There are a few examples or cases, rather than a large sample.
2. The researchers or others need to understand the nuances of social phenomena, such
as how a change occurs within a given institution. This is contrasted with
understanding simple but observable behaviors across a wide sample.
61
3. There is a complexity to the phenomena being studied that does not lend itself to
quantitative study. This might include closely-held opinions, networks of political
influence, or other themes that may emerge from discussion.
4. No comparator exists – such as a government agency – within the same or a very
similar context.
As a method, case studies must be tailored to fit the purpose for which they are used. They
include various perspectives, coming from interviews or discussion groups or both, of a
phenomenon (such as public policy reform or institutional strengthening in the current study)
about which actors may have various opinions and experiences – even contradictory ones. Each
case study for this evaluation also required further documentary or secondary data review, such
as with policy documents or other deliverables, results data, and institutional strengthening
manuals and work plans.
Data analysis
The PPP evaluation team conducted a systematic analysis of the interview transcriptions,
examining patterns, convergence and divergence of opinions and experience, and trends analysis,
to answer each of the evaluation questions. Notes and transcriptions from interviews were coded
according to the themes and dynamics most important for answering the evaluation questions.
The team also paid attention to emergent codes – that is, themes that occurred in the responses
that had not been considered in the evaluation questions. In this way, the team attempted to
capture unintended consequences of the PPP activities.
Limitations
Three particular limitations for this evaluation should be made explicit. First, the program
activities were performed in a highly dynamic environment, with changes in priorities, political
challenges, and USAID’s own changes in targets and goals. This complexity and dynamism
means that the multiple facets and perspectives around the Program’s achievements can be
contradictory or unclear, even after extensive data collection and analysis. Second, some actors
(key GoC stakeholders, for example, or AECOM staff and contractors, or even interested parties
at USAID) had changed positions since the project was implemented. Some individuals sought
for interview were unavailable. And finally, the absence of a Performance Management Plan
(PMP) and baseline data for the activities undertaken, the evaluation team focused on the
Program Work Statement (PWS) results as a measure of performance.
62
Getting to Answers Matrix
EVALUATION QUESTIONS Data collection Sampling or
selection criteria
Analysis plan
Source(s) Method(s)
1. To what extent did
AECOM influence the
GoC in defining its policy
decisions in the content
of the project
deliverables?
 GoC officials and functionaries  USAID staff  Implementer staff and contractors  Reports  Deliverables  Administrative and legislative acts
and policies  Documentary references
 In-depth
interviews  Document
review
 Case studies  In-depth
interviews  Document
review
 Case studies
 Definition of
criteria for the
case studies  Related GoC
officials, USAID
staff and IP staff  Relevant
documents LOP
 Content
analysis  Analyses of
convergence
and
divergence
2. To what extent where
international best
practices introduced vs
relying Colombian
expertise and models?
How effective was the
approach?
3. How effective was the
Public Policy Project
assisting the GoC to
adopt policy changes?
 GoC officials and functionaries  USAID staff  Implementer staff and contractors  Reports  Deliverables  Administrative and legislative acts
and policies  Database of PWS and deliverables
 In-depth
interviews  Document
review
 Case studies  Review of PWS
and deliverables
data from IP
 Criteria for the
case studies  Related GoC
officials, USAID
staff and IP staff  Relevant
documents for
the life of the
project
 Content
analysis  Frequencies
from among
project
deliverable
data
63
EVALUATION QUESTIONS Data collection Sampling or
selection criteria
Analysis plan
Source(s) Method(s)
4. What was the level of
quality in assistance
provided?
 GoC officials and functionaries  USAID staff  Implementer staff and contractors  Reports  Deliverables  Administrative and legislative
policies
 In-depth
interviews  Document
review
 Case studies
 Criteria for the
case studies  Related GoC
officials, USAID
staff and IP staff  Documents
 Content
analysis
5. What characteristics
of AECOM´s work
supported institutional
strengthening most
effectively and which
were less useful?
 GoC officials and functionaries  USAID staff  Implementer staff and contractors  Reports  Deliverables  Administrative and legislative
policies
 In-depth
interviews  Document
review
 Case studies
 Criteria for the
case studies  Related GoC
officials, USAID
staff and IP staff  Relevant
documents for
the life of the
project
 Content
analysis
64
ANNEX III: INFORMATION SOURCES
Type Name Title PPP activity area
USAID Norberto Martinez COR PPP in its entirety
USAID Angela Suarez Technical COR Victims pillar
USAID Thea Villate Technical ACOR Victims pillar
USAID Marcela Chavez Technical Monitor Land and rural development pillar
USAID Amy Meyers Technical Monitor Land and rural development pillar
AECOM Jaime Andrés Niño Chief of Party General; rural development emphasis
AECOM Diego Bautista Deputy COP General; consolidation, contrato-plan
AECOM Adriana Velez Technical Lead Land and rural development pillar
AECOM Diana Parra Technical Lead Access to financial services
AECOM Lina Ibañez Technical Lead Consolidation pillar
AECOM Sandra Alzate Technical Lead Victims pillar
AECOM Francisco Osuna Knowledge Mgt. Cross-pillars
AECOM Marcela Bustamante Technical team Victims pillar
AECOM Juan Guillermo Rubio M&E Monitoring and Evaluation
AECOM Patricia Henao Cross-cutting Gender and environment
AECOM Ana Maria Rivera Cross-cutting Institutional strengthening
CONS Carlos Rodríguez Consultant Contratos Plan
CONS Santiago Tobón Consultant Land and rural development pillar
CONS Eduardo Noriega Consultant Land and rural development and
access to financial services
CONS Arturo García Consultant Land and rural development pillar
CONS Diana Medrano Consultant Victims pillar
CONS Consuelo Corredor Consultant Income generation mission
CONS Jaime Arteaga Consultant Consolidation pillar
CONS Carlos Sandoval Consultant Access to financial services
GoC Paola Buendía Subdirector Office of the Presidency
GoC Anibal Fernández Soto Vice Minister Min. Agr. and Rural Development
GoC Myriam Villegas Former director INCODER
GoC Ricardo Sabogal Director Land Restitution Unit
GoC Germán Chamorro Director Consolidation Unit
65
GoC Juliana Alvarez Director Banca de Oportunidades
GoC Angela Penagos Director National Planning Department
GoC Iris Marín Subdirector Victims’ Unit
GoC Mariana Escobar Subdirector DPS, income generation mission
GoC Julio Abril Subdirector DPS, income generation mission
GoC Carlos Parodi Subdirector Planning, Consolidation Unit
GoC Carlos Córdoba Manager Contratos-Plan
BIBLIOGRAPHY
FARR, James; HACKER, Jacob S.; y, KAZEE, Nicole, (2006) “The Policy Scientist of
Democracy: The Discipline of Harold D. Lasswell.” The American Political Science
Review, Vol. 10, No. 4 (November 2006)
HOWLETT, Michael; RAMESH, M.; y PERL Anthony, (2009) Studying Public Policy.
Policy Cycles & Policy Subsystems. New York: Oxford University Press, 2009),
capítulos, 1, 2 y 3.
LINDBLOM, Charles E. (1991). El Proceso de Elaboración de Políticas Públicas.
Madrid. Ministerio para las Administraciones Públicas.
MENY, Ives y THOENIG Jean-Claude. (1992) Las Políticas Públicas. Barcelona:
Editorial Ariel S. A., parte 1
MORAN, Michael ; RIEN, Martin; and GOODIN, Robert E., eds. “The Historical Roots
of the Field.” (2006) The Oxford Handbook of Public Policy New York: Oxford
University Press. Chapter 2, “The Development of the Policy Sciences,” DELEON,
Peter.
PNUD. 2011. Colombia rural. Razones para la esperanza. Informe Nacional de
Desarrollo Humano 2011. Bogotá: INDH PNUD, Septiembre, Bogotá.RAVALLION
Martin, 2007. Evaluación de Programas Contra la Pobreza, Washington. Banco
Mundial, mimeo.
ROTH, André-Noël, 2010. Enfoques para el análisis de Políticas Públicas. Bogotá.
Universidad Nacional de Colombia.
Also reviewed: Program reports and documents provided by AECOM, including:  Program Work Statements (PWS) 2011, 2012, 2013
 Contractor Performance Assessment Reporting System (CPARS) reviews 2011,
2012, 2013
 Work Plans 2011, 2012, 2013
66
 Quarterly reports (project timeline inclusive, through Q12.)
67
ANNEX IV: INSTRUMENTS
The interview instruments were highly customized for each respondent, given their singular roles
and experiences with the PPP. All instruments (with these substantial variations) are included in
the supplementary disk submitted as part of this report, along with anonymized interview data.
68
ANNEX V: EVALUATION
STATEMENT OF WORK
Statement of Work
Performance Evaluation of the Public Policy Project (PPP)
1. THE ACTIVITY TO BE
EVALUATED
The Public Policy Project is being implemented through a cost-plus-fixed-fee contract (Contract
No. AID-514-C-11-00001) to AECOM. The original contract award amount of USD
$22,487,555 was increased on September 2012 by $4,900,000, to $27,387,555, through
Modification #7.
The initial period of performance was from 21 October 2010 to 20 October 2015 (a five-year
program). However, project reports indicate that because of Colombian government (GOC)
demand, completion of activities was accelerated. Thus, the project is now scheduled to close
July 2014, at the end of its third year of operations.
2. EVALUATION PURPOSE AND
USE
Purpose: To extract lessons from the experience of PPP regarding institutional strengthening
and policy reform. The evaluation results will be used by USAID to guide future Mission
project design and implementation efforts. This evaluation also meets the USAID requirement to
conduct at least one performance evaluation of activities that are larger than average (in dollar
terms) for a given Mission Development Objective.
3. EVALUATION QUESTIONS
A. To what extent were international best practices introduced vs. relying on Colombian
expertise and models? How effective was the approach?
B. To what extent did AECOM’s technical assistance feed into policy development and
adoption?
C. How effective was PPP in assisting the GOC to adopt policy changes?
D. What was the level of quality in assistance provided?
69
E. What characteristics of AECOM’s work supported institutional strengthening most
effectively and which were less useful?
4. PROJECT BACKGROUND
The PPP was originally a five-year program (reduced to three years) that was intended to provide
technical management expertise to form, adopt and implement public policies that would
improve the business environment, promote sustainable income generation, strengthen state
presence and improve the provision of social services to poor and vulnerable groups throughout
Colombia. It was also intended to gather and support reform priorities from Colombia Strategic
Development Initiative (CSDI) regional and other USAID programs.
PPP's main goal was to support GOC structural reforms designed to break the cycle of poverty
that leads marginalized, poor communities in areas with weak state presence to undertake illicit
crop production and illegal activities, fueling violence and the further degradation of Colombia's
social, economic and environmental resources. AECOM supported a wide variety of GOC policy
initiatives, including, but not limited to, land reform, access to financial services, labor and
vulnerable populations.
PPP’s strategy changed frequently during its relatively short lifetime. The original purpose of the
contract was to improve the advocacy, capacity and soundness of the private financial sector
through targeted reforms and institutional building and improvement of the quality and
availability of finance and business education. (The original Program Work Statement – PWS –
is too large to include in this document; but will be made available to the evaluation team.) In the
first phase, from October 2010 until March 2012, PPP worked through six components:
7. Land policy;
8. Access to finance;
9. Policies toward conflict-affected populations;
10. Support for decentralization and improved public economic management;
11. Labor issues; and
12. Environmental policy.
The original organizational chart is presented in Figure 1.
70
Figure 1: Public Policy Project Initial Organizational Chart
Contract Modification 3 (March 1, 2012) restructured PPP by replacing the PWS, replacing key
personnel and realigning the budget. PPP began focusing its capacity building and technical
assistance to support the key pillars of the GOC strategy: Land Policy, Victims Policy, and
Consolidation Policy.
PPP continued supporting the creation and strengthening of key national institutions in support
of the GOC strategy and development of key policy frameworks and their programmatic design
at the national level. PPP increased its focus on designing mechanisms to enhance planning,
budgeting and monitoring and evaluation cycle in national-level institutions.
PPP attempted to focus on strategic interventions and promoting synergies across a wide range of
GOC and USAID priorities and regional programs. PPP supported a number of cross-cutting
issues considered critical to each pillar to ensure integration of policy implementation at the
regional level. These included budgeting and coordination mechanisms at the territorial level;
sustainable livelihoods and access to finance; environment, biodiversity and climate change; and
gender.
Knowledge management had two objectives: (1) to support strategic communications related to
GOC policy reform and PPP achievements and (2) to build a knowledge base regarding the
policy change processes employed by the project.
The revised staff chart is presented in Figure 2; the revised PWS as Figure 3.
71
Figure 2: Public Policy Project Second Organizational Chart
72
Figure 3: Public Policy Project Second Program Work Statement (PWS)
The GOC demands and needs that drove the PPP model were complex. The GOC needed to
strengthen the Victims’ and Land Restitution Units to ensure their capacity to implement new
legislation and policy frameworks, and improve coordination with departmental and
municipal governments. Regional offices were still being created and at the same time had to
process massive numbers of restitution and reparations claims, while personnel were being
trained. Similarly, the Consolidation & Territorial Reconstruction Unit was still hiring
personnel in the regions and faced major challenges to rapidly respond to community needs and
coordinate service provision from different GOC agencies and local governments. Provision of social services, housing, and support for income generation to victims and
communities in regions affected by conflict remained a challenge. Despite significant budget
allocations identified in the Victims’ CONPES document and the revised National
Consolidation Plan, GOC service delivery did not respond effectively to the needs of victims, ethnic minorities, and consolidation zones. Novel funding and service delivery methods were
needed for vulnerable populations and regions devastated by decades of internal conflict. The
GOC had to eliminate rigidities that discouraged provision of services, and had to create
conditions for communities to build their own infrastructure and provide services that took into
Quality
Assurance/
Quality
Control
Land Restitution Unit established l Expert panel
Inputs to Compensation decree l Expert panel
Inputs to Land Restitution for Ethnic Minorities decree l Expert panel
Inputs to Law for Rural Development l Expert panel
Design of Unit for Rural Land Use Planning l Expert panel
Design of Sustainable Livelihoods Policy Framework l Expert panel
Design of Regional Rural Development Areas
Programs l Expert panel
Design of formalization program l Expert panel
Regulatory framework for land formalization l Expert panel
Design of institutional structure for formalization
program l Expert panel
Design of Department of Social Prosperity (DAPS) l Expert panel
Design of Victims’ Unit l Expert panel
Design of Regional Victims’ Assistance Centers l Expert panel
Design of Center for Historical Memory l Expert panel
National Plan for victims’ reparations in place l Expert panel
Design of regional reparations plans l Expert panel
2.3 Victims and civil society stakeholders participate in the implementation of the Victims’ Law Victims and stakeholder participation in policy design l
Expert panel,
Meeting
evaluations;
stakeholder
surveys
Revised NCP framework l Expert panel
Regional Consolidation Action Plans l Expert panel
Design of interagency and inter-governmental budget
and coordination mechanisms l Expert panel
Design of Consolidation Unit l Expert panel
Design of Consolidation Fund l Expert panel
3.2 Institutional structure designed to implement the NCP at the national and regional level
 NEW PERFORMANCE WORK STATEMENT (PWS)
PILLAR THREE: CONSOLIDATION POLICY
3.1 National Consolidation Policy (NCP) framework revised
2.1 Victims’ Law institutions designed
2.2 National plan for Victims’ humanitarian assistance and reparations
1.3 Rural property formalization program designed, including regulatory framework and
institutional infrastructure
PILLAR TWO: VICTIM’S POLICY:
1.1 Land restitution chapter of the Victim’s Law implemented
1.2 Land and Rural Development Law drafted
Year 4 Year 5
Program Goal: Improved economic and social conditions of vulnerable Colombians through effective rights-based
institutional presence
PILLAR ONE: LAND POLICY
Results/Performance Standards Delivery Requirements/Milestones Year 1 Year 2 Year 3
73
account gender and ethnic disparities, and victims’ physical, psychological and social
rehabilitation needs. In this new phase of work, PPP addressed the aforementioned challenges to ensure previously
supported reforms and policies would become a reality. In order to help GOC institutions reach
their goals, PPP:
6. Supported the Victims’, Land Restitution, and Consolidation Units, and INCODER and
MARD to strengthen their operations;
7. Worked with institutions to coordinate across government from the national policies
level to implement at regional and local levels; and
8. Harmonized sectorial policies to ensure coordinated, effective programming for priority
populations and regions.
As a result, the PWS was replaced, as shown in Figure 4.
74
Figure 4: Public Policy Project Third PWS
75
With frequent changes to its mission, and the output-oriented nature of the PWS structure (in
fact, it never had a PMP), PPP seems to have been focused on executing its many deliverables.
Reports indicate that it served as a vehicle to meet the diverse policy demands of the GOC and
various technical offices within USAID. Although international consultants and subcontracts to
Colombian firms were also used, PPP chiefly met demand by issuing contracts to individual
Colombian consultants to work individually, or in teams, to complete assigned deliverables.
Over the course of the contract, PPP issued contracts to over 400 individuals and firms, with
some receiving more than one contract.
The contract is currently in close-out mode, with the staff having dwindled from approximately
forty men and women to five.
5. EVALUATION METHODS
The approach will be systematically qualitative. The team will begin by reviewing project
documentation to understand the project’s activities, its clients, its modes of operation, its
accomplishments and its challenges. The team will then align the many policy and institutional
strengthening contracts executed over the life of PPP against the higher-order objectives they
sought to address. Much of the raw data will be arrayed by AECOM, as described in Section 6,
below. Against this universe of PPP activity the team will:
E. Purposively select a set of GOC clients with substantial involvement with PPP to
interview;
F. Interview USAID staff (COR, PROG, Technical Office Heads;
G. Interview key AECOM staff and ex-staff;
H. Purposively select two policy case studies and two institutional strengthening case studies
into which the team will delve more deeply, possibly using Social Network Analysis
method, among other approaches.
For budget and timeline purposes, EVAL estimates 40 in-depth interviews. One quarter to one￾third of these would be with current and former AECOM staff, and the remainder with
participating GOC partners and USAID. Case studies will also comprise at least one-third of the
interviews.
These data will be analyzed and reported to USAID using a standard Findings, Conclusions, and
Recommendations matrix that will inform a draft report to be submitted to USAID as well as a
debriefing. Based on feedback from the briefing and draft report, the Evaluation Team will
submit a final report.
76
6. EXISTING PERFORMANCE
AND OTHER PROJECT
INFORMATION
The following information will be made available to the evaluation team by AECOM:  All strategic documents, such as PWS and records of achievement against these goals  All work plans  All quarterly and annual reports  A table listing all subcontracts, including:
o person/entity awarded;
o period of performance;
o amount of contract;
o link to the completed product (if a document);
o if readily available:  assessment of performance (if any); and
 assessment of use of the product;  For each of the policy initiatives, a list including:
o Name of policy initiative
o Contact details of GOC participants
o List of contracts to support the policy
o Outcome of the initiative.  For each institutional strengthening initiative, a list including:
o Name of organization to be strengthened
o Contact details of GOC participants
o List of contracts to support the policy
o Major outputs of the support
o Outcome of the initiative.  A file containing all technical documents completed.  A list of current and past AECOM staff, with current email and cell phone contacts.  A listing of complete AECOM staffing patterns as of the end of each Fiscal Year.
7. TEAM COMPOSITION
The team will be comprised as follows:
A. The Team Leader will be a Senior Public Policy/Public Administration Expert, with
experience implementing policy reform projects that have a high technical assistance
component. S/he will also be an experienced evaluator.
B. The Senior Researcher will be an experienced analyst with significant experience
gathering and analyzing data and excellent English speaking and writing skills.
77
C. Junior Data Analyst with experience managing quantitative and qualitative data.
Combined, the team will have the following experience:  Leading and participating in performance evaluations;  Supporting policy processes in Colombia and elsewhere;  Working with USAID and understanding its practices;  Managing field projects;  Supporting capacity development of government institutions;  Evaluating policy and capacity building projects;  Working directly with GOC institutions;  Experience working in a wide range of sectors, such as agriculture, micro-enterprise,
environment, Colombian process of consolidation, among others; and
 Excellent written and spoken English and Spanish.
NOTE: A Senior Development Expert, with significant experience evaluating and managing
USAID field projects, will be added to serve on a temporary basis, if the above staff do not have
sufficient familiarity with USAID.
8. SCHEDULING, TIMING, AND
DELIVERABLES
Subject to various approvals, the team is expected to begin work in March, completing
deliverables as outlined in the following section.
DELIVERABLE DESCRIPTION Timing (calendar days)
Work plan and
evaluation design
A written evaluation design and detailed operational work plan
will be prepared and submitted to USAID for review prior to
fieldwork.
14 days after Start Date
(subject to availability for
interviews with USAID
and implementers)
Data collection
and analysis tools
All draft instruments and the analysis plan will be prepared and
submitted to USAID for review prior to fieldwork.
21 days after SOW Start
Date
Field data
collection
Weekly electronic reports of the progress made covering key
scheduled activities, completion status, constraints identified
with approaches to address constraints.
Throughout data
collection and analysis
period
First draft report
Rough draft of the report submitted to the USAID COR, who
will provide preliminary comments – one unified document
from whatever sources are required within USAID – to
facilitate preparation of the debriefing.
60 work days after SOW
Start Date
Debriefing with
USAID
EVAL will present the evaluation findings to USAID through a
presentation and discussion of findings, conclusions and
recommendations. The team will consider USAID comments
and revise the draft report, as appropriate.
Ten work days after
submission of draft
Final report Final report incorporating written comments received from
USAID and issues identified by USAID during the debriefing.
Seven work days after
receipt of final comments
from USAID
78
9. REPORT SECTIONS AND
CONTENT
The report will not exceed 30 pages, excluding table of contents, executive summary, acronyms
list, references and annexes. The format will be consistent with the 2011 USAID Evaluation
Policy and November 2012 USAID How-to Note on Evaluation Reports.
At a minimum, the report will include the following sections:  Executive Summary (3 to 4 pages);  Evaluation Purpose and Questions (1 to 2 pages);  Project Background (1 to 3 pages);  Evaluation Methods and Limitations (1 to 3 pages, with full version in an annex);  Findings, Conclusions and Recommendations (15 to 25 pages); and,  Annexes.
The report may include additional content, split the sections up differently, or present the
sections in a different order.
79
ANNEX VI: CONFLICT OF INTEREST
FORMS
Please see additional file in PDF form, “Signed PPP COI forms”
80
U.S. Agency for International Development
1300 Pennsylvania Avenue, NW
Washington, DC 20523