MIDTERM EVALUATION
OF THE
LATIN AMERICA AND THE CARIBBEAN
REGIONAL HEALTH SECTOR REFORM INITIATIVE
VOLUME I
By:
Gary Robinson
Maria Helena Jaen
Sergio Jellinek
Submitted by:
LTG Associates, Inc.
and
TvT Associates, Inc.
Submitted to:
The United States Agency for International Development
Under Contract No. HRN-I-00-99-00002-00
March 2000
Midterm Evaluation of the Latin America and the Caribbean Regional Health Sector Reform Initiative was
prepared under the auspices of the U.S. Agency for International Development (USAID) under the terms of
the Monitoring, Evaluation and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00.
The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG
Associates, TvT Associates, or USAID.
Information about this and other MEDS publications may be obtained from:
Monitoring, Evaluation, and Design Support (MEDS) Project
1101 Vermont Avenue, N.W., Suite 900
Washington, DC 20005
Phone: (202) 898-0980
Fax: (202) 898-9397
scallier@ltgassociates.com
ACRONYMS AND FOREIGN TERMS
C-NGOS Contracting NGO Services
CORE Cost revenue analysis tool
DAR Decentralization Applied Research
DDM Data for Decision-Making
DR Dominican Republic
DTF Dissemination task force
FPMD Family Planning Management Development
FY Fiscal year
G/PHN Bureau for Global Programs, Field Support and Research
Center for Population, Health and Nutrition
GPP Guidelines for policy process
HSR Health sector reform
IADB Inter-American Development Bank
Initiative Latin America and the Caribbean Regional Health Sector Reform Initiative
IR Intermediate Result
KPAB Knowledge, patterns, attitudes and behavior
LAC Latin America and the Caribbean
MHR Framework for Monitoring Health Reform
MOF Ministry of Finance
MOH Ministry of Health
MOST Management and Organizational Sustainability Tool
MPI Master plans of investment
MSH Management Sciences for Health
NGO Nongovernmental organization
NHA National health accounts
PAHO Pan American Health Organization
PHR Partnerships for Health Reform
PYS Performance incentive system
RSD Office of Regional Sustainable Development
SO Strategic Objective
TAG Technical advisory group
TPI Tool for plan investment
USAID United States Agency for International Development
WHO World Health Organization

CONTENTS
ACRONYMS AND FOREIGN TERMS
EXECUTIVE SUMMARY ................................................................................................. i
I. BACKGROUND AND INTRODUCTION ............................................................1
II. METHODOLOGY ..................................................................................................5
III. INITIATIVE FRAMEWORK .................................................................................9
A. Introduction........................................................................................................9
B. Intermediate Result Consistency and Indicator Focus.....................................11
C. Comparing the Health Reform Results Package Indicators
with the Actual Initiative Framework..............................................................12
D. Validity and Feasibility of the Initiative Framework.......................................12
IV. EFFECTIVENESS.................................................................................................15
A. Progress and Achievements.............................................................................15
B. Intermediate Results Analysis..........................................................................15
C. Summary of Findings and Analysis of the Strategic Objective.......................31
V. INITIATIVE MANAGEMENT EFFICIENCY AND ADMINISTRATION.......35
A. Initiative Leadership ........................................................................................35
B. Initiative Structure ...........................................................................................36
C. Initiative Management and Administration .....................................................39
VI. SUSTAINABILITY...............................................................................................45
VII. RECOMMENDATIONS.......................................................................................49
A. Initiative Framework and Sustainability...........................................................49
B. Effectiveness....................................................................................................50
C. Initiative Management and Administrative Efficiency....................................58
D. Methodology and Study Design.......................................................................58
E. Specific Considerations for the Next Initiative Work Plan .............................59
VIII. REFERENCES ......................................................................................................61
TABLES
Table 1: Country Index of Tools, Methodologies, Instruments, and Exchanges
March 2000 ...........................................................................................................6
Table 2: LAC/RSD Health Reform Results Package........................................................10
Table 3: Perceived Strengths and Weaknesses of USAID
Leadership and the Management Structure.........................................................37
Table 4: Initiative Contracting Arrangements...................................................................38
Table 5: Initiative Timeline...............................................................................................38
Table 6: PAHO Work Plan 2000 ......................................................................................43
Table 7: Factors Hindering Institutionalization and Sustainability
of Tools, Methodologies, and Exchanges...........................................................46
Table 8: Proposed Checklist for Assessing the Introduction and Use of Tools,
Methodologies, Experiences, and/or Exchanges ................................................50
FIGURES
Figure 1: Prototype Structure ..........................................................................................36
Figure 2: Perceived Effectiveness of Initiative Coordinating and Task Forces..............39
Figure 3: Establishing the Level of Risk of the Assumptions.........................................47
APPENDICES
A. Scope of Work .............................................................................................................67
B. Health Sector Reform Results Package Indicators, February 2000.............................75
APPENDICES IN VOLUME II
C: Methodology for Collecting and Analyzing Data on Effectiveness
(Intermediate Results 3.4.1, 3.4.3 and 3.4.4)
D: Interview Formats Model
E: Conceptual and Methodological Framework
F: Results Framework Analysis. IRs, Indicators and Expected Results of Tools,
Methodologies, Instruments, Experiences, Forum and Study Tours (March 2000)
G: Criteria for Assessing Communications Components in Development Programs
H: Database IRs 1, 3 and 4 (Raw Data)
I: Tools, Methodologies, Experiences, Exchanges, Forum, and Study Tour Evaluation
Results: Effectiveness by Tool or Experience and Country Evaluation Results IRs 1, 3
and 4
J: Results on Dissemination Strategy
K: HSR LAC Newsletter Concept Paper
L: Hits Recorded–LAC/RSD Web Site
M: Official Documentation of the Strategy (Messages Defined by the Strategy)
N: Traditional Communications Approaches
O: Partners’ Opinions on Audiences and Media (Opinion de Socios Sobre Audiencias y
Medios)
P: The Development Communications Strategic Cycle
i
EXECUTIVE SUMMARY
The Latin America and Caribbean (LAC) Health Sector Reform Initiative seeks to
positively influence health sector reform in 13 Latin American and Caribbean countries
through the development, testing, and dissemination of methodologies, tools,
experiences, and/or exchanges. Its goal is to facilitate targeted countries’ ministries of
health and others in the health area to integrate lessons learned into their health reform
processes and to increase their institutional capacity.
The United States Agency for International Development (USAID), through a contract
with Monitoring, Evaluation and Design Support (MEDS), engaged a 3–person team to
conduct a midterm review of the LAC Health Sector Reform Initiative. Its goal was to
assess midterm progress towards accomplishment of the following four Intermediate
Results (IRs) and to reflect on whether progress to that point would enable achievement
of the Strategic Objective of more effective delivery of basic health services and policy
interventions. The four IRs are:
ß IR 1: Methodologies and tools developed, tested and disseminated for
analysis and design, implementation and monitoring of country health sector
reforms.
ß IR 2: Information on health reform efforts and experience gathered and made
widely available to interested parties in LAC countries and to health sector
donors.
ß IR 3: Reform processes, specifically related to equitable access, and outcomes
monitored and feedback provided to countries, donors and other partners.
ß IR 4: Opportunities and means to share experience and advice among
countries are established.
The evaluation team reviewed the validity of the Initiative framework and addressed
three primary questions. The first is related to the effectiveness of the Initiative in
achieving the Intermediate Results. The second is concerned with its management and
administrative efficiency, and the third is related to the feasibility of scale-up and long￾term sustainability. Recommendations deriving from the findings are presented.
The Initiative framework was found to be valid but with suggested changes in
indicators and increased emphasis on mitigation plans for managing contingencies of the
assumptions underlying it. A more focused and consistent approach of measuring results
is suggested along with the recommendation that the Initiative track different types of
indicators at each level. At country level 2, the Initiative should track indicators that
reflect changes in behavior that occur from or through use of the methodologies, tools,
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
ii
and experiences. At the regional level, the Initiative should focus on tracking indicators
of the effective delivery of Initiative products and services.
An examination of the progress and achievements addresses the first evaluation
question of effectiveness. A case study was determined to be the most effective way to
carry out the country-specific study of methodologies, tools, and exchanges. The
objective was to understand and describe the relationship between variables in each case
and the dynamic processes that were occurring in each country. The first step, described
above, was an examination of the Results Framework used by USAID in its program
management of the Initiative. The second step was a study of the product life cycle of
design, development, test, analysis, implementation, and monitoring in three selected
countries—the Dominican Republic, Honduras, and Peru. The third step was a study and
analysis of the dissemination and communication of Initiative tools, methodologies, and
exchanges. Two hypotheses were developed and guided the conceptual framework of the
evaluation design:
ß The success of the Initiative, that is, the achievement of its Intermediate
Results, depends not only on the effect of variables under its control but also
on factors beyond its control but which could be influenced by Initiative
efforts if reflected in a systematic way in Initiative management systems; and
ß The evaluation of achievements and progress is a function of not only the four
Intermediate Results and indicators that pertain to them, but also the expected
results for each tool, methodology, experience, forum, or study tour at the
midpoint of the Initiative.
Primary source information was gathered and analyzed from key stakeholders and
Initiative partners through specifically designed interview formats. The identification and
selection of individuals was based on the criteria of key actors who had participated in or
were knowledgeable of Initiative activities, were active participants in the health reform
process, or were in a position to influence health sector reform. The number of key
actors interviewed was 17 in Honduras, 18 in Peru, and 20 in the Dominican Republic.
Secondary source information from Initiative and partner documents were reviewed and
analyzed.
The Initiative was found to be making varying degrees of progress towards its four
intermediate goals relative to the level of development and/or use of the different tools
and methodologies. Factors found promoting and inhibiting use of the tools and
methodologies were:
ß the degree of institutional capacity of ministries of health (MOH) and other
involved organizations,
ß political will,
ß the existence of supportive national and MOH coalitions,
EXECUTIVE SUMMARY
iii
ß international organization alignment with national efforts, and
ß a national communication and dissemination strategy.
Field interviews revealed that the most successful experiences were those in which the
Initiative influences design and implementation factors that affect country utilization of
regional tools. In other words, the most successful experiences were those in which the
regional approach and intervention strategy was accompanied by an in-country initiative.
Moreover, the most effective institutional strategy targeted different public institutions;
that is, not just the Ministry of Health but also other ministries, special reform
commissions, or nongovernmental organizations (NGOs) working together with the
public sector. Factors that were within Initiative control before the dissemination of the
tools, methodologies, and experiences were:
ß the quality of the experience and the selection of themes that are demand
driven,
ß the development of a training process that takes into account national
expectations,
ß the agreement of key actors at the country level on the best and most feasible
institutional strategy for implementation purposes, and,
ß the achievement of partner alignment at the headquarters level on the different
tools, methodologies, or experiences.
In addition, the report identifies factors which are beyond Initiative control but which the
Initiative should strive to influence. The most important are local or national adaptation
of the tool, a national communication and dissemination strategy, and horizontal
alignment of partners at the country level. The stability of key actors and the definition
of a public policy on health reform are vital to achieving national commitment to the
health sector reform agenda.
Initiative management was found to have taken an innovative approach that uses the
unique talents of four partners in a synergistic way to execute its plan. A steering
committee and various task forces or work teams were used to manage the various
Initiative themes. The steering committee was very effective in its work, while the task
groups ranged from marginal to effective. Contract completion dates of three of the
contractors pose the risk of a hiatus in Initiative continuity; greater steering committee
emphasis is needed for the management of task groups and risk factors. Findings
regarding financial resources were mixed. Initiative financial resources total $10.2
million, of which USAID contributes $7.4 million and the Pan American Health
Organization (PAHO) contributes $2.8 million. PAHO is also the recipient of $2.5
million (of the $10.2 million), giving it responsibility for $5.3 million of Initiative
resources. Some partners believe that the Initiative does not have enough resources given
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
iv
the enormity of its task, while data showed that the Initiative was actually underspending
with respect to its planned expenditures.
Changes that are needed for long-term sustainability of the Initiative include an effective
risk assessment of Initiative assumptions and activities and management of high-risk
areas through risk mitigation. These high-risk areas include weakness in MOH
institutional capacity for health sector reform and lack of or weak political will to move
the health reform process forward. Specific factors in the latter include the lack of a
health sector reform public policy, lack or weakness of health sector reform leadership,
inability to make political decisions supportive of health sector reform or political
interference in decision-making at the MOH, process management difficulties, and
parallel, duplicative, and competitive reform units within the health sector.
Recommendations to increase Initiative potential in facilitating the integration of lessons
learned and building increased institutional capacity for health reform in targeted
countries are provided. The major recommendations are summarized below.
Methodology: A representative sample would have provided stronger data and while it
might require more evaluation team preparation time, it is recommended for future
evaluations.
Initiative Framework and Sustainability: The Initiative framework and sustainability
are two interconnected components. To increase sustainability and to strengthen the
Initiative framework, it is recommended that the Initiative Results Framework be
adjusted by operationally defining key actors working on health reform, Intermediate
Results be measured through a redesigned tracking system for some Intermediate Results
and their indicators, and the level of risk be monitored periodically, using the
recommended (or a similar) methodology to assess movement in the level of risk
exposure.
Effectiveness: Intermediate Results 1, 3, and 4: To increase effectiveness and progress
and achievements, it is recommended that the proposed checklist or a variation be used
for assessing the introduction and use of tools, methodologies, experiences, and/or
exchanges; commitment of other stakeholders be obtained for support and technical
assistance of the design and development of effective and efficient training processes; a
strategy be created for garnering support from different public stakeholder institutions;
partner alignment be achieved at the headquarters level on the different tools,
methodologies, experiences, or exchanges; additional Initiative influence be exercised
over design and implementation factors, particularly partnerships, that affect country
utilization of regional tools; and plans be developed for managing factors beyond
Initiative control.
Effectiveness: Intermediate Result 2: A shift in paradigm from dissemination to
development communications is recommended. This will affect the type of products and
services provided by the strategy and management and operational arrangements.
Moreover, it will be more horizontal in approach with respect to information flow, more
EXECUTIVE SUMMARY
v
inclusive in terms of targeted audiences, more demand driven from the perspective of
messages crafted and delivered, more global in terms of the media to be used, and more
corporate in order to guarantee a new equilibrium between stronger branding and
partners’ individual identities. This will also have the positive effect of mitigating key
risks associated with the development of political will and mobilization of resources for
health sector reform in the region and will contribute to the building of coalitions for
health sector reform at the national as well as the regional level. The recommendations
section presents a detailed list of steps to be taken in order to broaden the present
dissemination strategy.
Initiative Management and Administrative Efficiency: The recommendations include
clarifying roles and responsibilities of the steering committee, partners, and task forces;
reexamining the PAHO partnership; enhancing management, including preparing
milestone charts for each activity and task force and reviewing them on a defined steering
committee schedule linked to Intermediate Results; reviewing the 2000 work plan and
requiring recovery plans when the schedule of deliverables is not adhered to, particularly
where the Initiative is spending less than the planned expenditures; and rescheduling the
technical advisory group to coincide with the planning cycle, and if it is willing, use it to
help the steering committee manage the assumptions of the Results Package and
mitigation plans.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
vi
1
I. BACKGROUND AND INTRODUCTION
Countries throughout Latin America and the Caribbean (LAC) are introducing reforms
that can profoundly influence how basic health services are provided and who receives
them. Health system reform is being undertaken to reduce inequities, improve quality,
and correct inefficiencies in current systems.1
Governments in the region identified the need for a network to support health reform
through analysis, training, and other capacity-building measures at the Summit of the
Americas in 1994, and again at a Special Meeting on Health Sector Reform convened by
an interagency committee of the United Nations and other multilateral and bilateral
agencies in 1995.
In response, the United States Agency for International Development (USAID) and the
Pan American Health Organization (PAHO) launched the Latin America and Caribbean
(LAC) Regional Health Sector Reform Initiative. This Initiative promotes more equitable
and effective delivery of basic health services through support of regional activities that
support informed decision-making on health policy and management, health financing,
health service improvement, decentralization and institutional development.
The Health Sector Reform Initiative is one of five initiatives which support the LAC
Regional Strategic Objective (SO), “more effective delivery of selected health services
and policy interventions.” It focuses on “sustainable country health sector reforms
(designed to increase equitable access to high quality, efficiently delivered, basic health
services), to increase country capability to assess health sector problems, and to design,
implement and monitor reforms and solutions.”
The LAC Regional Health Sector Reform Initiative is implemented by the Pan American
Health Organization, the Partnerships for Health Reform (PHR) Project, the Family
Planning Management Development (FPMD) Project and the Data for Decision- Making
(DDM) Project. PAHO is an international public health agency headquartered in
Washington, D.C. It serves as the specialized organization of the Inter-American system
for health and as the regional office for the Americas of the World Health Organization.
Partnerships for Health Reform is a five-year USAID-funded contract that builds
capacity in policy foundation and implementation, health economics and financing, and
organization and management of health systems.
Family Planning Management Development is a five-year USAID-funded cooperative
agreement that helps national and local family planning and health programs and
organizations develop their capability to plan and manage sustainable programs.
 
1
 This section of the report is taken from the scope of work for this midterm evaluation.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
2
Data for Decision-Making is a five-year USAID-funded cooperative agreement that
supports health sector reform and assists leadership in developing countries in making
informed policy and financing decisions.
The Initiative provides regional support to activities in Bolivia, Brazil, the Dominican
Republic, Ecuador, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico,
Nicaragua, Paraguay, and Peru.
The five-year Initiative was authorized on July 29, 1997, for a total of $7.4 million in
USAID funding. PAHO also committed $2.8 million of its funds to the Initiative, for a
total value of $10.2 million. The allocation of USAID funds envisaged was $2.5 million
for PAHO through a public international organization grant, $3.2 million for the
Partnerships for Health Reform Project, $900,000 for the Data for Decision-Making
Project and $750,000 for the Family Planning Management Development Project. Of the
$7.4 million in USAID funds, $6 million was new funding approved at the time of
authorization and $1.4 million was fiscal year (FY) 1995 carryover field support funding
($1.1 million of PHR’s planned allocation and $300,000 of DDM’s planned allocation).
In order to provide management oversight to an innovative approach and to enhance the
achievement of the LAC Initiative, a midterm evaluation was planned. Its goal was to
assess progress and achievements to date and present recommendations for adjustment,
redirection, or reconfiguration. To accomplish this, the steering committee established
six evaluation objectives:
ß to determine if the Results Framework is valid and feasible and to recommend
adjustments to the Results Framework and activities as appropriate;
ß to examine progress towards achieving the SO and IRs as planned, evaluating
whether the results/indicators will be met in a timely and effective manner, to
identify specific internal/external constraints which may limit their
accomplishment or success, and to recommend adjustments based on findings
and conclusions;
ß to assess how the Initiative structure as configured is working, including both
the delegation of specific technical areas and activities to the partners as well
as their coordination, working relationships, and collaboration in the
implementation of Initiative activities;
ß to assess the administration of the Initiative by USAID and the implementing
partners within their organizations. The focus should be on Initiative
management, meeting deadlines, monitoring, and technical assistance;
adjustments in implementation based on findings and conclusions should be
recommended;
ß to review the resources management of the Initiative, including the use of
human, material, and financial resources to achieve IRs; and
I. BACKGROUND AND INTRODUCTION
3
ß to examine the feasibility of scale up or sustainability and mechanisms for
ensuring sustainability after external funding ceases.
The evaluation team began its work in mid-February, with field work being conducted
February 19 through March 17, 2000. The full scope of work is contained in appendix A
and includes key questions to be answered that were instrumental in guiding the
evaluation design. Appendix B contains the health sector reform Results Package
indicators.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
4
5
II. METHODOLOGY
The methodology used was designed to gather necessary and sufficient data in order to
make inferences about effectiveness in achieving Intermediate Results and their attendant
impact on health policy. It also examines management of the Initiative and the
sustainability of accomplishments and benefits. Recommendations based on these
analyses are offered for midterm reconfiguration and changes.
All Initiative documentation was reviewed, including annual reports, partner reports,
Initiative products, working papers, and other pertinent documents. On the basis of this
review and conversations with partners, a three-part methodology was developed. The
first part involved gathering data in Washington, D.C., from the Initiative partners and
others with authoritative knowledge of the Initiative. A standardized interview format
was constructed and used that focused on the LAC Initiative in general, its operations,
and its management. The second part involved gathering data from the three countries
selected by USAID and the steering committee for in-country visits (appendix C):2
 the
Dominican Republic, Honduras, and Peru. A standardized interview format, adjusted for
country differences, was used to gather information from individuals who had experience
with or were in a position to offer authoritative commentary on Initiative tools,
methodologies, and exchanges. In a separate but parallel effort, the third part of the
methodology involved gathering data through questionnaires regarding the Initiative
dissemination and communication strategy. (Appendix D contains the interview formats
model.)
A case study was determined to be the most effective way to carry out the country–
specific study to evaluate methodologies, tools, and exchanges related to Intermediate
Results 1, 3, and 4. Table 1 (on the following page) was used as the guide in the
observation and collection of data. The different stages and degrees of progress of the
Initiative’s implementation were also taken into account.
The evaluation design addresses a key issue deriving from the nature of midterm
evaluations; that is, the type of results that could reasonably be expected at this point in
time since accomplishment indicators are written in terms of final results. The 3
countries, however, are not used as a sample of all 13 countries involved in the Initiative.
Rather, the goal of the case study was to examine how factors that affect the Initiative’s
results operate in the three countries. The objective was to understand and describe the
relationship between variables in each case and the dynamic processes that were
occurring in each country. In this way, implementation patterns and trends regarding
Initiative performance, progress, and achievements could be ascertained. Thus, the
evaluation design (a case study) and sampling size (only key respondents) influence the
 
2Appendix C also contains specific descriptions and complementary tables regarding the case study in the
Dominican Republic, Honduras, and Peru.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
6
nature of the findings. The findings reflect the situation in the countries studied; they are
not meant to be representative of the Initiative’s work in its entirety.
Table 1: Country Index of Tools, Methodologies, Instruments, and Exchanges
(March 2000)
TOOLS, METHODOLOGIES, STUDY TOURS,
AND EXCHANGES PARTNER HONDURAS DOMINICAN
REPUBLIC PERU
Tools and Methodologies
1. Tool for Plan Investment (TPI) PAHO x
2. National Health Accounts (NHA) DDM/PHR/
PAHO
x x x
3. Methodology for Monitoring Health 
Sector Reform (HSR)
PAHO x x x
4. Performance Incentive Systems (PYS) FPMD x
5. Cost Revenue Analysis (CORE) FPMD x
6. Management and Organizational 
Sustainability Tool (MOST)
FPMD x x
7. Guidelines for Policy Process; Health Policy
Process Applied Research (GPP)3
DDM x
8. Decentralization Applied Research (Decision
Space) (DAR)
DDM x
9. Contracting NGO Services (DR) (C￾NGOS)
PHR x x
10.NGO in Health Reform (Policy Process) 
(Bolivia) (NGOS-HSR)
PHR x x
Study Tours
1. Panama PAHO x x
2. Colombia PHR x x x
3. Canada PAHO x
Exchanges
NGOs in Decentralization FPMD x
NGOs Quality Assurance FPMD x
Forum on Provider Payment Mechanisms PAHO x x
Andean Region Forum in Bolivia (Santa Cruz,
Bolivia, 1999)
PAHO x
 
3
 The health sector analysis tool currently is being used in two countries (Paraguay and Nicaragua). It was
not evaluated since these two countries were not part of the case study. Tools and methodologies 7 to 10
are current as of this evaluation; additional guidelines, tools, or papers on the development and testing
process are in progress.
II. METHODOLOGY
7
 The following steps were taken to gather data regarding Intermediate Results 1, 3, and 4:
ß A draft analytical framework was designed for the collection and analysis of
data.4
ß In consultation with partners both in Washington and in the field, key
respondents were identified who had participated in one or more of the
experiences (i.e., methodologies, tools, and exchanges) and who were in a
position to provide authoritative commentary on them. Once in-country, these
lists were reviewed and adjusted consistent with local conditions. (See
appendix C, table C-1, for a detailed list of country institutions and key
stakeholders.)
ß Interviews were carried out in the three selected countries. The number of key
stakeholders interviewed was 17 in Honduras, 18 in Peru, and 20 in the
Dominican Republic.
ß The conceptual framework for data analysis was completed (appendix E).5
ß To measure midterm results, a set of expected results was postulated by which
the individual tools, methodologies, experiences, study tours and exchanges
could reasonably be assessed. (See appendix F for Results Framework
analysis, IRs, indicators, expected results, and indicators for the midterm
evaluation of tools, methodologies, instruments, exchanges, fora, and study
tours.) These projected midterm results are used to measure midpoint progress
and achievements. The following steps were taken for data consolidation and
analysis of Intermediate Results 1, 3, and 4:
· The results from the three-country case study were organized,
transcribed, consolidated, and aggregated by Intermediate Result, tool,
methodology, exchange, and key stakeholders.
· Data analysis was guided by the conceptual framework and relevant
aspects and specific questions in the scope of work for the evaluation.
The conceptual framework was used to analyze the relationship between
variables and expected results for each tool, methodology, experience, or
exchange.
 
4
 This framework focused on the design of interview formats, which were used to obtain data on variables
and factors that could affect the progress and achievement of Initiative Intermediate Results. Factors that
could influence the institutionalization and sustainability of Initiative tools, methodologies, and exchanges
were also identified and incorporated into the interview formats.
5
 Appendix E contains a description of the conceptual and methodological frameworks used for data
analysis. It also presents the tables for factors and results with their indicators, operational definition,
sources of information and interview questions, and the tables for operational definitions of additional
categories for data analysis.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
8
· Findings were reviewed with the partners in a debriefing session. The
completion of findings, conclusions, and recommendations took the
partners’ suggestions and comments into consideration.
In the separate but parallel effort focusing on the communications/dissemination strategy,
three questionnaires were administered to gather data to be used for the subsequent
analysis of Intermediate Result 2. The first questionnaire was administered to the partners
to assess the level of consensus on the communications/dissemination strategy and its
different components, such as targeted audiences, crafted messages, and media used. The
second questionnaire was directed to individuals interviewed in the three countries to
measure name recognition of the Initiative, message relevance, audiences to be targeted,
and media to be used, and to capture new insights. The third questionnaire was directed
to the same group and was used to obtain relevant data on information needs, the use of
strategy information services relevant to Intermediate Result 2, and the use of Initiative
electronic services compared with other available sources. These questionnaires were
also distributed to 10 individuals in other countries and key personnel at the World Bank
and the Inter-American Development Bank (IADB). The following process was used for
consolidating and analyzing the data:
ß Field test results from the three countries were consolidated and aggregated.
Data gathered from partners were consolidated.
ß Consolidated data were reviewed within the conceptual framework (appendix
G: Assessment Criteria) and relevant aspects and specific questions of the
scope of work for the evaluation.
ß Basic findings were noted and validated with data received from other
countries and key stakeholders in Washington.
ß Findings were reviewed with the partners in a debriefing session. The
completion of the findings, conclusions, and recommendations took partners’
suggestions and comments into consideration.
9
III. INITIATIVE FRAMEWORK
A. INTRODUCTION
The internal logic of the Initiative framework is consistent. However, the distinction
made between regional and national levels requires additional refinement in terms of the
metrics used for tracking results, strategies for mitigating risk, and management of the
Initiative through a process of continued performance improvement.
The validity of the framework is contingent upon the assumptions made regarding its
potential success. (This evaluation covers this in detail in the section on sustainability.)
Assessing the inherent risks derived from the assumptions and developing mitigation
plans for high-risk assumptions deserves increased attention.
The Framework6
The Latin America and Caribbean (LAC)/Office of Regional Sustainable Development
(RSD) health reform package of December 8, 1997, provides a hierarchy of objectives
from a Strategic Objective (SO) level to levels of Intermediate Results (IR) at the country
and regional levels. (See table 2 on the following page.) The IRs are within the specific
mandate of the Initiative, while the SO is focused on the end users of Initiative products
and services (with which the Initiative is only indirectly connected). In short, the
regional mandate is for the Initiative to develop, test, and disseminate for use at the
country level, whereas the IRs seek to change behavior through the use of the
methodologies, tools, experiences, and/or exchanges.
At the regional level (Intermediate Results, level 3), the framework establishes four IRs
that represent regional products and services under Initiative control. This approach is
consistent in terms of causal linkages and the supplier/customer chain and includes
significant relationships.
However, the Intermediate Results that were part of the scope of work (regional level 3 of
the original design) and those that the Initiative used for monitoring during 1998 and
1999 were inconsistent in the way results were measured, due largely to a lack of
indicator focus. This is discussed in detail in the following sections.
 
6
 See table 2 for the health reform Results Package indicators used by the Initiative until FY 99.
Table 2: LAC/RSD Health Reform Results Package
Agency Goal World’s Population Stabilized and Human Health Protected
in a Sustainable Fashion
LAC/RSD Strategic
Objective
More effective delivery of selected health services and
policy interventions
Selected policy intervention Sustainable country health sector reforms (designed to
increase equitable access to high quality, efficiently
delivered basic health services)
Intermediate Results Level 1
(Country–level)
In–country capability to assess health sector problems, and
to design, implement, and monitor reforms
Intermediate Results Level 2
(Country–level)
2.1: Methodologies and tools
adapted and used for analysis and
design, implementation, and
monitoring of country health sector
reforms
2.2: Information on health reform efforts and experience
accessible and used by interested parties in LAC countries
2.3: Internal feedback used to
monitor/revise reform efforts;
complete reports provided to
regional collection point on
schedule
2.4: In-country interested parties** share
experience and advice between countries
Intermediate Results Level 3
(Regional)
3.1: Methodologies and tools
developed, tested and disseminated
for analysis, design, implementation
and monitoring of country health
sector reforms
3.2: Information on health reform efforts and experience
gathered and made widely available to interested parties in
LAC countries and to health sector donors
3.3: Reform processes and
outcomes monitored and
feedback provided to countries,
donors, and other partners
3.4: Opportunities and means to share
experience and advice between countries are
established
Timeframe: 1996–2001**
Partners: DDM/Harvard, Abt
Associates, PAHO, Management
Sciences for Health
Timeframe: 1997–2001
Partners: PAHO, DDM/Harvard, Abt Associates,
Management Sciences for Health
Timeframe: 1997–2001
Partners: PAHO, In-country
interested parties,* Abt
Associates
Timeframe: 1997–2001
Partners: PAHO, In-country interested
parties, Abt Associates, DDM/Harvard,
Management Sciences for Health
Projects used to implement
activities
598-0825, Health Priorities Project
938-5974.13, Partnerships for
Health Reform
938-5991.01, Data for Decision￾making in the Health Sector
938-3055, Family Planning
Management Development
598-0825, Health Priorities Project
938-5974.13, Partnerships for Health Reform
938-5991.01, Data for Decision-making in the Health
Sector
938-3055, Family Planning Management Development
598-0825, Health Priorities
Project
938-5974.13, Partnerships for
Health Reform
598-0825, Health Priorities Project
938-5974.13, Partnerships for Health Reform
938-5991.01, Data for Decision-making in the
Health Sector
938-3055, Family Planning Management
Development
Critical Assumptions
(for this Results Package)
Government and nongovernment health providers,
professional societies, university faculty willing to participate
together in reform efforts
In-country interested parties can mobilize political will to re–
direct resources (time, personnel, and money) to reforms that
increase equitable access to basic health services
Donors continue to fund capital costs for country health
reform design and implementation, including technical
assistance, studies, and systems design/implementation
Source: USAID PHN/LAC/RSD, July 1, 1997
*In-country interested parties will vary by country; many LAC countries have a Health Sector Reform Team with members from the government, NGOs, universities, and/or professional societies—and in some
cases, international agencies.
**Following the presentation of this Results Framework in the 1996 R4, PHR and DDM began using FY 95 x-G field support funds for these activities.
III. INITIATIVE FRAMEWORK
11
B. INTERMEDIATE RESULT CONSISTENCY AND INDICATOR FOCUS
All of the regional IRs are within the Initiative’s range of capability. However, the
indicators used to measure IR 1 and IR 4 go beyond the measurement of delivered
products and services. These indicators seek to measure change in client behavior. Those
indicators related to the number of countries using tools and methodologies (IR 1) and the
indicator referring to the application of lessons learned from the study tours (IR 4) are
measured by observed behavioral change. That is, are the clients doing something
different as a result of Initiative activities?
The concept of use in the IR 1 indicator is different for the NHA methodology. It seeks
not only use but also institutionalization. This distinction clearly indicates two levels of
use: which use is to be measured by the Initiative and for which methodology or tool.
Should all or only some be institutionalized?
Further complicating the measurement of IR 1 is the definition of key actors.
Operationally, key actors are defined as those who can significantly affect health sector
reform in each of the 13 targeted countries. This includes the selection of government
versus nongovernment individuals for implementation arrangements.
It is not clear why two NGO papers (PHR7
) are included in IR 1: Tools and
methodologies, and two papers (FPMD8
) are included in IR 4: Exchange or sharing
experiences.9
 This expected result appears to be more related to IR 4 than to IR 1, to
sharing experiences, creating networks, and generating scenarios for cooperation.
One indicator used to measure IR 2 also goes beyond the availability of information
services by analyzing the use of electronic services by Initiative customers. This same
type of problem is found with the indicator measuring achievement of IR 3. Instead of
measuring the feedback provided by countries, the indicator measures the use of the
methodology for monitoring health reform at the country level.
The indicator selected for measuring the achievement of IR 4 does not measure the result
as such (opportunities for sharing established). Instead, the indicator measures the effect
that the sharing has had on the participants in terms of applying lessons learned (taking
steps based on lessons learned).
 
7
 PHR: NGOs’ Role in Health Sector Reform Policy and Public Sector Contracting of NGOs for the
Provision of Health Services.
8
 FPMD: Public/NGO Partnerships in Response to Decentralization and Public/NGO Partnerships for
Quality Assurance.
9
 This observation is even more important if it is understood that the four NGO papers were produced under
an overall, joint NGO strategy. The regional meeting and the papers’ preparation are not isolated sets of
activities. They were intended to gather regional experience with public/NGO partnerships that would
provide input to a larger meeting between public sector and NGO participants. This latter meeting took
place in Managua, April 3–7, 2000.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
12
C. COMPARING THE HEALTH REFORM RESULTS PACKAGE INDICATORS10
WITH THE ACTUAL INITIATIVE FRAMEWORK
Two changes were made in the Strategic Objective. The first changed the goal to be
achieved from 50 percent of Initiative countries to all target countries. The second
modified the denominator from “Initiative” to “Results Package,” changed
“methodologies” to “methods,” and used “sharing” instead of “exchanges.” That new
definition of the Strategic Objective indicator has not overcome the inconsistencies found
by the midterm evaluation.
In IR 1, the indicator is changed from methodologies and tools “used” to “actively used”
by key stakeholders. Consequently, the limitations mentioned above with respect to this
IR and its indicator have not been overcome. However, the shift from “used” to “actively
used” could be interpreted as recognition by the steering committee of the difficulty in
operationally defining and assessing the use of tools and methodologies by countries.
“Actively used” needs to be defined.
The IR 2 indicator measuring user satisfaction with electronic services was moved to IR
4. This reflects a positive change recognizing the interactive scenario provided by the
web site and related services and is consistent with findings in the section on
dissemination/communication strategy in this report.
D. VALIDITY AND FEASIBILITY OF THE INITIATIVE FRAMEWORK
The evaluation data suggest that a more refined approach than currently exists for
measuring results is needed. The original Results Framework is still consistent in its
design but different types of indicators should be tracked at each level.
At the country level (level 2), indicators should be tracked that reflect changes in
behavior, such as use of tools and methodologies, use of information and communication
services, or steps taken according to lessons learned, and that will generate the necessary
inputs in order to create macro indicators: x number of countries or x percent of countries
using y methodology by z time.
At the regional level (level 3), the focus should be on tracking indicators for efficient
delivery of products and services, such as:
ß x number of methodologies developed, tested, ready to use, and ready to
disseminate;
ß number and type of publications produced and received by targeted audiences;
ß articles printed and distributed;
 
10 See appendix B, Health Sector Reform Results Package Indicators, Approved in February 2000.
III. INITIATIVE FRAMEWORK
13
ß electronic services provided; and
ß a forum on reform relevant subjects organized.
An example for IR 1 would be the following:
ß Level 2 (country): Methodologies and tools adapted and used for analysis and
design, implementation, and monitoring of country health sector reforms.
· Type of indicator: x number of countries using y methodology by z time.
ß Level 3 (regional): Methodologies and tools developed, tested, and
disseminated for analysis, design, implementation, and monitoring of country
health sector reforms.
· Type of indicators:
® x type of methodology developed by z time,
® x type of methodology tested in y number of countries by z time, and
® x type of methodology disseminated to y number of key stakeholders
in z countries in t time.
Additionally, a set of criteria is needed for determining the type of information necessary
for tracking the status of the indicators, including customer feedback and the methods
used to obtain and objectively measure the data.
There are also external factors that are relevant to achieving expected results at the
national level. The direct linkage between regional and national levels can be
jeopardized if these factors are not taken into account, and in some cases, actively
managed. For example, national institutional capacity is external to the Initiative, but it
can be managed (influenced). For instance, creating a baseline through a rapid
institutional assessment and then incorporating the results into the country Initiative
design could be one step toward improving institutional capacity.
In summary, the Initiative framework is a workable instrument, but only if it is refined in
scope and measures what it is responsible for measuring. There is a tendency to confuse
the Initiative’s regional mandate with the desire for results at the country level.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
14
15
IV. EFFECTIVENESS
A. PROGRESS AND ACHIEVEMENTS
This analysis focuses on the factors affecting the effectiveness of the Initiative in
facilitating targeted countries to integrate lessons learned in terms of building
institutional capacity for health reform. A key hypothesis guided the analysis of the data,
namely that a set of factors or variables (some of them within the control of the Initiative
and others beyond its control [see appendix E for the conceptual and methodological
framework]) affects the achievement of the Initiative’s IRs and SO. The evaluation of
achievements and progress at this point in time focuses not only on the four Intermediate
Results and indicators but also on the expected results for each of the Initiative’s
methodologies, tools, experiences, and exchanges (see appendix F). Using proxy
indicators, the following section examines each of the methodologies, tools, experiences,
and exchanges and reports findings regarding progress and achievements (see appendix H
for the database with data for each activity by country, institution, and key informants
individually).
B. INTERMEDIATE RESULTS ANALYSIS
INTERMEDIATE RESULT 1: Methodologies and tools developed, tested, and disseminated for
analysis and design, implementation, and monitoring of country health sector reforms.
INDICATOR 1: All methodologies and tools used by key actors in 50 percent of countries
where introduced.11
National Health Accounts (PAHO, PHR, and DDM)
NHA is successfully used by two of the three countries studied but is not being
institutionalized in any of them. It is a demand-driven methodology that is considered to
be of good to high quality, with strong partnership at the central level. Its implementation
has complemented the regional approach with country orientation and an
interinstitutional strategy. The use of NHA by the Ministry of Health (MOH) in two of
the countries is due mostly to NHA being part of the reform agenda and supported by a
consensus of key stakeholders. In one case, NHA is being used exclusively by the
Ministry of Finance (MOF) because the MOH does not have the political will to promote,
support, and use NHA. Continuity and stability of key staff as well as an Initiative
 
11 See appendix I for evaluation results by each activity by IRs 1, 3, and 4 and their indicators with detailed
data by indicator, specific analysis, and outcomes discussion. Most of the results are the expression of key
informants’ perception (see the operational definition of each indicator in appendix E). Although the health
sector analysis tool actually is being used in two countries (Paraguay and Nicaragua), it was not evaluated
because these two countries were not part of the case study.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
16
partnership supports its use at the national level. Headquarters and local partner offices
are aligned in supporting NHA efforts. Difficulties in institutionalization appear to be
related to the lack of a health reform policy, political interference in decision-making,
MOH institutional weaknesses, and limited adaptation of the tool to national
requirements.
Framework for Detailed Implementation Plans of HSR and Master Plans for Investment
(MPI) (PAHO)
This tool is in the development and testing stage in Honduras and key MOH staff is
knowledgeable of it. In this country, there is a political coalescence regarding the need
for the tool. It is on the reform agenda, is perceived as high quality, and is a demand￾driven methodology. There is stability and continuity of key actors favoring the use of
MPI and alignment between PAHO headquarters and country office partnership efforts.
Lessons learned from the analysis of experiences for implementation and
institutionalization purposes are that centralized training should be accompanied by
decentralized technical assistance, the MOH needs to be strengthened, an Initiative
partnership needs to exist at the country level, and the methodology needs to be adapted
to country circumstances. Because information was not available to the evaluation team
regarding the experience in Nicaragua, broad conclusions could not be drawn and
patterns or general trends could not be determined.
Decentralization Applied Research (Decision Space) (DDM)
The tool is in the development and testing process and is assessed on that basis. It is not
possible to make a final evaluation of the degree of knowledge and application of this
technical assistance based only on the data from Honduras. The small number of
respondents does not lend itself to general conclusions or trends. In Honduras, the tool is
perceived more as technical assistance than as a tool per se. Even though it is not broadly
known and used by MOH staff, it is a research tool that appears to have value in
promoting and supporting decentralization of the health system. The dissemination of this
experience through the World Bank Flagship courses may facilitate broad regional
application of the research findings to health system decentralization. This is because
participation in these courses by MOH staff and reform unit consultants is encouraged
and gives them the opportunity to learn and analyze the research findings on
decentralization from various countries.
DDM used the preliminary results from the decentralization and policy process applied
research projects to develop the decentralization module for the World Bank Flagship
course in Washington and in regional courses in China, Thailand, Malaysia, Hungary,
and Chile. The course is used by the World Bank to build capacity for health reform
around the world—targeted toward middle and high-level officials and private sector
leaders. The Economic Development Institute of the World Bank (Human Resource and
Poverty Division) has promoted the Flagship course since November 1997. It is a 1–
month course in Washington but usually 10 days in regional courses. “The Flagship
course aims to provide an integrated and coherent learning program covering health
IV. EFFECTIVENESS
17
sector reform and financing options…” This learning is expected to improve the
performance of client country participants in their work in their countries, as well as the
related work of bank staff participants. It is also expected to “stimulate participation and
sharing of experiences in health economics and financing networks, via the Internet”
(World Bank–EDI, “Flagship Course on Health Sector Reform and Sustainable
Financing,” October 1997).
Health Policy Process Applied Research (DDM)
As with the above activity, this tool is in the development and testing process and is
assessed on that basis. It is not possible to make a definitive evaluation based on data
from a small number of respondents in Honduras. Specifically, the results show that it is
known and used by a small number of key MOH staff of Honduras and that it is seen as
providing technical assistance that supports the policy process. It appears to have
potential value for promoting and supporting national leadership and stewardship of
HSR. Consistent with the experience with the previous activity, the dissemination of the
research findings through World Bank Flagship courses may facilitate the introduction
and application of policy process analysis in promoting health sector reform on a regional
basis.
Sustainability Tool (MOST), Cost Revenue Analysis Tool (CORE) and Management
Organizational Performance Incentive System (PYS) (FPMD)
These are tools that have been field tested and successfully introduced in a number of
organizations around the world. The Initiative’s contribution was to provide for
translation into Spanish, dissemination, and training a cadre of facilitators. Due to the
small sampling size, general conclusions cannot be made about awareness or the use of
these tools. They cannot be evaluated on the same basis as the rest of the tools and
experiences because the approach and institutional strategy for the test and
implementation of these tools is very different. In this case, the partner does not follow
the regional or even a country approach or institutional strategy (for example, MOH
participation). Because the focus is the NGOs but without the participation of the public
sector, it gives a different perspective to the evaluation process and results. It is not
possible to use the same type of respondents used for other methodologies in each
country since the key actors in health reform knew neither the NGOs nor the tools. The
only exception is INSALUD in the Dominican Republic. Since it was not possible to
identify at the country level other respondents for the evaluation of these tools, the
findings are restricted to application of the tools at the NGO level, which yielded less
valid results concerning these tools.
In summary, respondents evaluated CORE and PYS as useful for improving the
organization of their NGOs; they did not find MOST useful. However, an overriding
issue with respect to NGOs is that they are only incorporated in a limited way into the
national health system. Although the tools are valuable in increasing NGO capacity, to
increase their value to the health sector, NGOs need to be incorporated into the country’s
health system.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
18
Management and Organizational Sustainability Tool (MOST)
There have not been any MOST training workshops held in the three countries visited
and only two informants were knowledgeable of this tool. One of the respondents
interviewed perceived the quality of the tool as fair to bad. He had not received any
training and was not aware of specific application of the tool to his NGO. The second
respondent explained that his organization is using another tool for management and
organizational purposes (Evaluación de Desarrollo Gerencial [EDG]) and does not have
an informed opinion regarding MOST.
Cost Revenue Analysis Tool (CORE) and Performance Incentive System (PYS)
CORE and PYS were evaluated in one NGO in Honduras that is using the tools. They
were considered to be of high quality. Staff was provided with proper training and the
tools were meeting and satisfying the NGO’s need.12
Initiative NGO Strategy
The two NGO papers of the Public Heath Reform Project (PHR), NGOs’ Role in Health
Sector Reform Policy and Public Sector Contracting of NGOs for the Provision of Health
Services, are associated with IR 1. The two FPMD papers Public/NGO Partnerships in
Response to Decentralization and Public/NGO Partnerships for Quality Assurance are
within activities of IR 4. Since these four NGO papers were produced under a joint NGO
strategy, all the NGO–related activities are reported on a one-by-one basis here. In some
cases, lessons learned show different country results with the same Initiative experiences.
Moreover, it would be useful to define the specific need for technical assistance by the
NGO as it relates to the country policy on NGO participation. Another is related to the
Initiative institutional strategy for working with NGOs. Respondents believed that it
would be better to have public officials and NGO managers together in workshops.
However, this requires weighing the trade-off between the present practice of including
only NGO representatives in the meetings to facilitate an open exchange of views among
NGO participants and achieving joint participation of public sector and NGO
organizations. It would be useful if partners could agree at both the central and country
levels on the strategy for inclusion of private sector and NGO participation in the health
system and health sector reform.
NGOs in Health Reform Policy (PHR)
This tool is in the development and testing process and is assessed on that basis. The
findings show significant differences between the two countries with the same Initiative
experience. In one country, there is sharing of information with other workshop
participants and workshop results are being used to help incorporate NGOs into the
 
12 It is not possible to estimate the impact that a separate endeavor of FPMD has on the use and
applicability of these tools.
IV. EFFECTIVENESS
19
health system. An important number of respondents acknowledged the use of workshop
results for NGO participation in HSR. In all likelihood, the participants had circulated
and disseminated results of the workshop widely among key actors. In this case, the
workshops have been used to support a country process and respondents see the
experience as highly useful to the incorporation of NGOs into the health reform process.
However, in the other country, although respondents participated in this same workshop,
they did not perceive these experiences as useful for supporting community participation
efforts in health care. It is notable that in both countries, respondents perceive weak
institutional capacity on the part of both the MOH and NGOs that limits NGO
incorporation into HSR. Contributing to this is a lack of local partnership in tool
development in this area. In fact, one of the partners (PAHO) perceived the NGO strategy
as a PHR/USAID initiative.
Public Sector Contracting of NGOs for the Provision of Health Services (PHR)
The results for this NGO experience are similar to the previous one. This tool is in the
development and testing process and is assessed on that basis. There were similar
significant differences with the same Initiative experience in the two participating
countries visited by the evaluation team. In one country, there is sharing of information
with other workshop participants and workshop results are being used to support the
inclusion of NGOs in the health sector as health providers. An important number of
respondents acknowledge the use of workshop results and see the experience as highly
useful. However, in the other country, although respondents participated in the same
workshop, they did not perceive these experiences as useful for what they are doing in
community participation in health care. Again, in both countries there is a perception of
weak institutional capacity in both the MOH and NGOs. This limits the MOH in
contracting NGOs for health service delivery. Similarly, there was a lack of local
partnership in the development of a tool in this area.
NGOs in Decentralized Systems and NGO Quality Assurance (FPMD)
It is not possible to show general trends and patterns regarding these two experiences
because there was only one key informant in the case study. As with the two tools above,
they are in the development and testing process. The respondent identified the Initiative
institutional strategy of directing actions exclusively to NGOs as limited to the
implementation of workshop results and the opportunity and means to share experiences
and advice among countries.
NGOs in Decentralized Systems (FPMD)
As a result of the workshop, NGOs in Decentralized Systems, the respondent has shared
information with other NGOs and also has taken concrete steps to promote NGO
participation in a decentralized health system. He evaluated this workshop as being of
high quality and needed for the national decentralization process. He affirmed that only
the working group on NGOs in a decentralized system had been useful.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
20
NGOs Quality Assurance (FPMD)
As a contrast, he perceived the NGO Quality Assurance workshop as being of poor
quality and not demand driven at this moment. Nevertheless, he affirmed that it could be
very useful to the current health reform process in his country. He believes the lack of use
and sharing of information gained from this workshop is what contributed most to its
poor quality.
INTERMEDIATE RESULT 2: Information on health reform efforts and experiences gathered
and made widely available to interested parties in LAC countries and to health sector
donors.
INDICATOR 2a: At least five different institutions in each of the LAC Initiative target
countries report receiving Initiative materials.
INDICATOR 2b: Fifty percent of electronic network users surveyed report finding network
services useful. 13
IR 2 is related to actions of the dissemination or communications strategy. This section
discusses the level of achievement of each indicator and subsequently introduces
concepts and mechanisms to broaden the dissemination effort towards a new
development communications paradigm. It also discusses specific questions raised in the
scope of work.
Five Institutions in Each Country Report Receiving Initiative’s Materials
According to 1999 figures (see the Initiative 1999 report) and data gathered (see
appendix J on communication issues) by the evaluation team in the three countries, this
indicator has been achieved. While 50 percent of respondents reported receiving the
materials, 30 percent reported irregular distribution.
The Initiative changed the focus of this indicator from individuals to institutions in 1999.
However, it requires further refinement in order to measure it. Distribution lists continue
to be based on key individuals at the institutions. Moreover, the indicator does not specify
the number of individuals needed to report receiving materials to assume that the
institution is receiving materials.
The indicator is incomplete as presently written in that it does not take into consideration
a key aspect of IR 2—distribution to the donor community related to health sector
reform. Beyond that, this indicator does not measure what clients do with the materials.
In spite of this, the findings do substantiate the usefulness of printed materials. Within the
group of respondents who reported receiving printed materials from the Initiative, more
than 60 percent perceive them as useful or very useful and report that their main use is for
improving performance, sharing experiences, increasing knowledge on the reform, and/or
 
13 See appendix J for the results on the dissemination strategy.
IV. EFFECTIVENESS
21
becoming acquainted with new developments. There is a clear demand for printed
materials (see appendix J).
A different type of productæthe newsletteræis an effort that goes beyond the
dissemination view and that could be an effective tool for offering an updated status of
reform initiatives, successes, and challenges, if its content is improved with different
types of stakeholders expressing their views. A clear effort in designing the newsletter
with a more inclusive approach than it has currently is perceived, even if the first edition
is still reflecting a top-down approach (see appendix K).
Fifty Percent of Electronic Network Users Surveyed Report Finding Services Useful
This indicator is intended to measure client satisfaction with the electronic provision of
materials. The Initiative used different ways to measure client satisfaction with this
service in 1999. The primary way was a survey that did not give a clear picture of the
level of usefulness of the services. While 78 percent of individuals who responded to the
survey are reported to have found the services useful, this is based only on a 10 percent
return of those surveyed (more than 300). From a statistical point of view, this low a
level of responses makes it difficult to validate the results for the universe selected.
Another way to assess the usefulness of electronic services is to use a proxy indicator to
record the number of hits on the web site. The assumption, not entirely accepted in the
industry but used by PAHO in this instance, considers that an average of seven hits can
be considered as constituting a user. This would lead to the conclusion that 100,000 users
were visiting the site. This figure needs to be viewed with some skepticism in as much as
several hits can be recorded by just opening the front page of the web site. Additionally,
the use of proxy servers makes it difficult to create a clear relationship of cause and effect
between the number of hits and the number of users. (See appendix L for data gathered
by PAHO for the numbers of hits/users on the LAC/RSD web site.)
According to data from the three countries and the responses received from other
countries to validate them, the LAC HSR web site has a 6 percent average share of the
market of Internet users interested in health reform–related issues. However, when this 6
percent is asked about its perception of the Initiative’s electronic services, a clear
majority rates them as “good” to “very good.” This means that even if the number of
users is low, the regular clients of the web page value the services as useful.
In some countries, such as Peru, the World Bank’s web site has almost twice the share
(11 percent) of the market than the Dominican Republic and Honduras; the preferences
are clearly linked to the use of search engines as a way to obtain the information needed
(21 percent). In some cases, the PAHO or PHR web pages received a higher score than
the Initiative’s web site.
Another consideration is the open medium nature of the Internet. PAHO data suggest
that there are other groups of possible users beyond the core group who could be attracted
to it for different reasons. The evaluation data show that there is a need for broader
marketing in order to increase the number of users of the Initiative’s web page. New
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
22
services could make the web page more attractive to potential users and increase its share
of the market.
Beyond IR 2: Is the dissemination strategy working? Is it contributing to a
sustainable behavioral change?
Official Initiative documents characterize the effort of promoting knowledge and related
activities about its tools and methodologies indistinctly as dissemination, information, or
communications strategy.
According to available documents, the process of establishing such a strategy began in
1997, continued during 1998, and was refined in 1999. Yet, even with this effort, data
show that the conceptual framework of the dissemination strategy was not clearly
deployed. The terms communications, dissemination, and information continued to be
used in a quasi-interchangeable way (see appendix M).
This conceptual lack of specificity has an impact on the relationships between the
recipients of the information and the way they interact. Dissemination implies a vertical
flow orientation from the sender to preselected recipients. Communications implies the
establishment of at least a two-way information flow whereby the targeted audiences
interact within the space of different selected media. Development communications is a
synthesis of social marketing, public relations, and media advocacy applied to
development projects (see appendix N).
The scope of work and the specific questions contained in it along with comments from
Initiative partners at the debriefing meeting (March 16, 2000), suggest that the analysis of
IR 2 should go beyond its accomplishments as noted above and answer the basic
evaluation questions: Is there a better way? Is it sustainable?
Examination of the differences between the dissemination strategy generated by the
project and the traditional categories or approaches used in this field constitute a basic
starting point. They reinforce the understanding that the needs of the Initiative are not
solved by one simple framework (see appendix N).
The Results Framework and the delivery of information and/or communications products
and services are related to changes in knowledge, patterns, attitudes, and behavior
(KPAB)14 in key selected audiences. The use of information/communication services in
relation to certain tools contributes to the accomplishment of IR 1. At the SO level,
generating capacity building at the national level will result in the use of these services
for policy development, decision-making, and other strategies.
Questionnaires used by the evaluation team focused on how the partners and their clients
defined the targeted audiences, the messages selected, the media to be used for
channeling the messages, the branding of the strategy, and the way it is managed. These
are discussed in greater detail below.
 
14 Composite index used as a standard within communication components in health-related projects.
IV. EFFECTIVENESS
23
The Audiences15
There is substantive agreement among the partners that the following are key targeted
audiences:
ß key officials involved in health sector reform,
ß academics working on health sector reform–related issues,
ß key decision-makers at the multilateral regional and international institutions,
and
ß other governmental officials related to the area.
Additional groups revealed through evaluation data to be targeted include:
ß key decision-makers at NGOs,
ß journalists covering health issues,
ß private sector decision-makers,
ß national legislators concerned and working with health issues, and
ß public opinion and health personnel.
Evaluation data document that different perceptions of audiences to be targeted will
affect the type of communications products and services to be delivered and the kind of
media to be used in an outreach effort such as this.
There is a broad perception that the audiences are much more segmented than originally
thought and that other audiences need to be taken into account and given attention by the
strategy.
Beyond the audiences defined by the Initiative, a clear majority of respondents agree that
the following new audiences should be taken into account:
ß national legislators dealing with health issues,
ß key individuals in the media who work with health issues and who have
access to the targeted audiences,
ß private sector representatives involved with health, and
ß professionals (physicians and health personnel).
Many of the respondents included public opinion as a whole as a new audience.
 
15 See Guidelines for Dissemination and Communication, April 1998, and appendix O for partners’
opinions on targeting audiences and media selected by the Initiative.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
24
Main Messages and Media Defined by the Strategy
The first step in message development derives from audience screening or stakeholder
analysis that defines the type of problems and topics that interest each selected audience
in relation to specific themes relevant to health sector reform. Audience screening
provides important data for developing key messages to be delivered through the strategy
and different media channels.
The basic criteria should be that messages crafted and delivered respond to audience
problems and interests. If not, a communications gap or blockade between the sender and
the recipient(s) may result.
In the case of the dissemination strategy (see appendix J), partner interviews indicated
that both the messages and the selected dissemination media were the result of the
dissemination content of the strategy. This means that the messages created by the
strategy are general, for the most part, and not tailored to the information needs of the
audiences. Most contain a basic explanation of the Initiative’s services. Some messages
are related to a brief explanation of specific tools and methodologies developed by the
Initiative (see appendix M).
From a dissemination point of view, the idea is to make information on Initiative
products and services available to a core list of defined Initiative targeted audiences. The
assumption is that targeting potential users would increase access and facilitate the use of
information on Initiative tools and methodologies and, in the long term, increase health
sector capacity and service delivery.
From the customer’s point of view, evaluation data reveal two main findings regarding
their information needs. Respondents report that
ß key areas of concern related to health reform are aligned with the areas
covered by Initiative tools and methodologies; and
ß the following additional information needs are not covered by Initiative
information or should be covered in greater detail:
· information related to experiences of health reform,
· health information systems,
· reproductive health,
· legal framework, and
· social participation.
Even assuming that the tools/methodologies developed are customer demand driven, the
changes in KPAB expected within the key audiences seem to be related to a more
sophisticated set of factors, that is,
IV. EFFECTIVENESS
25
ß how other decision-makers (political, opinion leaders, civil society) interact
and if they can create synergy favoring a positive scenario for the use of
Initiative tools and methodologies;
ß how these influencers affect the core health sector reform group;
ß how the use of information/communication tools increases name recognition
and bargaining capacity of key actors who are willing to move forward but
need to ease local resistance; and
ß the adequacy of the media selected for carrying the messages. (This is
discussed further in the following sections.)
The media selected by the strategy include printed series, brochures, an Internet web
page, and newsletters.
Some individual and sporadic interventions leading to publications in international
media, although encouraging, were not part of the original strategy (reports from CNN–
En Español, channeled through PAHO).
Of the four media selected, both the printed materials and the brochures are consistent
with the dissemination view. However, the web page and the Internet are interactive
communication tools and should be designed and managed accordingly.
The potential of the Internet as a communications media has not been fully exploited by
the web site. Strategy weaknesses here, similar to findings in printed media, derive from
a restricted view of targeted audiences and a lack of audience screening and stakeholder
analysis for message development.
In terms of the media to be used in the future, interview data show that all partners and a
clear majority of respondents in the three countries favor including other types of media
(see appendix J). Other media selected, in order of priority, were
ß opinion editorials in regional and national newspapers and magazines,
ß short videotapes,
ß participation in television programs,
ß teleconferences, and
ß radio programs.
Brand Recognition
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
26
The evaluation team was asked to assess whether Initiative brand recognition is a
contributing factor for attaining the Initiative and Intermediate Results. Partners and
respondents in the three countries concurred that there is weak brand recognition of the
Initiative (see appendix J and data processed from interviews with the partners).
A successful communications strategy needs to establish consistency between targeted
and key messages to be delivered, media to be used, and corporate image. Brand
recognition is vital to communicating the Initiative’s comparative advantage in the health
sector market and in differentiating its services and products from others. Some of the
reasons for weak brand recognition are discussed below.
USAID and PAHO have established regional and sectorwide brand recognition. Project
leadership and the dissemination task force perceive the task of creating brand
recognition mainly as communicating the cooperative effort between the Initiative and its
partners. The goal is for customers to recognize the involvement of PAHO/USAID and
their partners and to attach importance to the cooperative and value-added scope and
nature of the Initiative. However, many customers already have established working
relations with one or more Initiative partners. This limits present Initiative brand
recognition as a unique regional framework because in some instances, a partner’s brand
is more recognized than the Initiative’s brand.
The majority of Initiative partners, with the exception of PAHO, believe that brand
recognition is occasionally present. Only PAHO staff perceives that the Initiative
presents a corporate image. Country data show that brand recognition is weak and that
branding is primarily related to Initiative partners and secondarily with the Initiative
itself. An unresolved (and challenging) issue is how to develop a corporate branding
while preserving individual partner identity. (This is discussed in the recommendations
section.)
Health sector reform champions are not identified and used for branding. A way to
strengthen the Initiative brand and to promote Initiative identification is related to the
identification of reform champions in the differently targeted audiences. The words and
actions of those champions will echo strongly at the regional level. Interaction at that
level would have an impact at the national level. That will further reinforce identification
and perceptions about the Initiative and its unique regional advantage.
The marketing strategy has a limited scope. Consistent with the dissemination view, the
marketing component includes the web site, the newsletter, study tours, forums, and the
efforts conducted by PAHO and USAID at the national level. However, there was no
structured marketing campaign addressing regional media, regional professional
magazines, national media outlets as unpaid advertisements (press releases distributed to
selected media), institutional exchanges, and others. The lack of this type of marketing
diminishes the probability of clear brand positioning.
In order to develop brand recognition, the logo and acronyms are important elements. If
this is an Inter-American project, it also includes North America and the Caribbean. The
logo map gives the impression of a South American project. In addition, the use of
IV. EFFECTIVENESS
27
acronyms is heavy and not self-explanatory, making brand recognition even more
difficult. LAC is not an acronym used in Latin America and its combination with RSS
makes it more complicated to understand. To achieve the best results, both the logo and
the acronym need to be explicit, comprehensive in terms of messages conveyed, and self￾explanatory.
A simple logo idea such as the above, even if it does not relate immediately to the reform
issue, is clear in terms of the scope of the project and the countries involved. It is easier to
remember and promotes the Strategic Objective related to improved health in the whole
region.
The colors selected for the brochures and other printed materials are part of the branding
and are a key input in the type of messages that the strategy would like to convey. An
aggressive message is consistent with the colors selected (pinkish orange/blue); a
persuasive message will be more consistent with varieties of beige or similar colors. The
three triangles used in the printed documents do not convey any special message and may
actually confuse the reader.
Lessons Learned at the National Level
Even if the dissemination strategy is regional in nature, developments at the national level
have shown that communication actions can accelerate the process of achieving IR 1. In
the Dominican Republic and Honduras, there is an ongoing process of consultation. It
targets several audiences with the objective of building a national coalition to establish a
participatory approach for health sector reform and related development. This is
especially true in the case of the Dominican Republic, where the impact in the national
media reinforces and contributes to the process of building a national consensus among
decision-makers.
Furthermore, the number of audiences targeted in the Dominican Republic includes new
groups beyond those previously defined by the strategy. Actions associated with these
new groups include
ß consultation workshops,
ß development of newsletters,
ß media coverage,
ß opinion editorial articles, and
ß television coverage.
Lessons emerging from the field show that a broad communications policy at the regional
level can have an effect on strengthening coalition building at the national level.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
28
INTERMEDIATE RESULT 3: Reform processes, specifically related to equitable access, and
outcomes monitored, and feedback provided to countries, donors and other partners.
INDICATOR 3a: All LAC Initiative countries monitor health sector reform using framework
established by PAHO. 16
 
INDICATOR 3b: Regional monitoring system for comparative analysis for health sector
reform institutionalized by PAHO. 17
Monitoring Health Reform (PAHO)
The framework for monitoring health reform (MHR) is well known but is used in only
one of the three countries evaluated—the Dominican Republic—where it is being
adapted to national conditions and the Dominican health reform process. The
methodology is perceived as needed and of high quality. The training and technical
assistance met national expectations, and its implementation has complemented the
regional approach with a country orientation along with a participatory consensual
development process with key stakeholders. There is a strong alignment of Initiative
partners at the local level and an alignment of headquarters and local partner efforts. The
methodology is part of the country’s partner-consultant specific tool-related terms of
reference. There is a national consensus on the need for monitoring health reform and the
key stakeholders acknowledge that the theme is on the reform agenda. Stability and
continuity of key actors support the use of the MHR. Specifically in this country, the
MOH is perceived as having the institutional capacity to drive and perfect the use of the
methodology. The framework has not been institutionalized in any of the three visited
countries. The general impression is that lack of institutionalization appears to be related
to weak political will to monitor health reform, absence of a public policy on health
reform, and political interference in decision-making at the MOH.
Some divergence was found between the midterm evaluation results and the Initiative’s
FY 99 annual report. The evaluation shows that one country out of three is employing the
PAHO framework. But the Initiative’s annual report states that 11 countries have
achieved this goal: “The LAC Initiative has had a significant impact on the target
countries’ capability to assess health sector problems and to design, implement and
monitor reform.”18 To support this statement, the Initiative uses the approval of this
methodology by the ministers of health in Resolution CD41.R12 by the 41st Directing
Council of PAHO (San Juan, Puerto Rico, September 199919). Two hypotheses are
 
16 See appendix I for the evaluation results by activity by IR 3 and its indicator with detailed data by
indicator, specific analysis, and outcomes discussion. Most of the results are the expression of key
informants’ perceptions (see operational definition of each indicator in appendix F).
17 This indicator was not considered because it is in the development stage.
18 LAC Health Sector Reform Initiative, Annual Report FY 99, p. 1.
19 This resolution states: 1) recognize the efforts to design and implement the methodology for monitoring
and evaluating health sector reform in Latin America and the Caribbean (methodology that was designed
and implemented by the LAC Initiative) and 2) urge member states to institutionalize the monitoring and
evaluation of their health sector reform processes and apply policies that take into account the results of
those processes.
IV. EFFECTIVENESS
29
offered to explain this contradiction. The first one is related to the indicator that is used
by the partners to track the use of the methodology. Thus, while there is recognition of
the need for the methodology by the ministers of health—as shown by the 41st Directing
Council of PAHO—this has not been translated into acceptance or action by their
ministry teams. Using acceptance of the resolution by the ministers of health is the
prerequisite for tracking the use of this methodology. Indeed, this could bias results and
brings into question whether the right metric for use or full application is being tracked.
The second hypothesis is that the structure or quality of the instrument and the process
through which it is deployed may have inhibited implementation and institutionalization.
INTERMEDIATE RESULT 4: Opportunities and means to share experience and advice among
countries are established.
INDICATOR 4: Fifty percent of participants surveyed report taking steps based on lessons
learned in fora. 20
Payment Mechanisms Forum (PAHO)
Two of the three countries—Peru and the Dominican Republic—participated in the
workshop. Respondents reported that no steps had been taken as a result of this forum. In
the Dominican Republic, two informants were aware of the actual implementation of new
payment mechanisms in the health system. In Peru, even though a significant number of
informants were aware of the implementation of new payment mechanisms to providers,
only one actor recognized the forum as input, that is, having provided information, for the
implementation of these mechanisms. The evaluation case study results show a tendency
that deserves further study: namely, that no steps had been taken using forum experience
as input. Respondents perceived that the forum was not demand driven. In fact, they
reported that the regional workshop did not take into account their national needs,
previous technical conditions, and health system particularities. There was no clear
institutional strategy in choosing the participants for the forum, and no communication or
dissemination strategy subsequently at the national level. There were indications that at
the time of the forum, headquarters and country partners were not aligned. The fact that
one country is implementing new payment mechanisms and the other has not yet initiated
any action can be associated with the following factors: in the former, there has been a
political decision to support payment mechanisms; it has put the topic on the health sector
reform agenda and achieved a national consensus regarding the need for new payment
mechanisms in the health system. Moreover, there is an Initiative partnership at the
national level regarding payment mechanisms. Key actors of both countries, however,
perceived a weak legal framework and/or institutional capacity for introducing new
payment mechanisms in health services.
Study Tour to Colombia (PHR)
 
20 See appendix I for the evaluation results by activity by IR 4 and its indicator with detailed data by
indicator, specific analysis, and outcomes discussion. Most of the results are the expression of key
informants’ perceptions (see operational definition of each indicator in appendix F).
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
30
MOH officials and hospital directors from two of the three countries evaluated perceived
that they were promoting hospital autonomy and reported taking steps based on lessons
learned on the study tour. Some possible explanations of this outcome are that the study
tour was perceived in this country as pertinent and applicable, there was an Initiative
partnership at the national level on hospital autonomy, and there was a political decision
and national policy promoting hospital autonomy. Moreover, the tour was perceived as
being of high quality. In the other participating country, MOH officials stipulated that
they were working on hospital autonomy but not as a result of the study tour experience.
In this case, respondents perceived that the quality of the tour was between fair to good.
In both countries, the hospital autonomy issue is on the health reform agenda and there is
headquarters and local hospital alignment in the hospital autonomy effort. However, there
was no communication and dissemination strategy at the national level. Moreover,
respondents did not perceive that they had a legal framework to support hospital
autonomy nor was the study tour designed to be adapted to national needs for
implementation purposes.
Study Tours to Panama (PAHO)
The key actors in two participating countries in the study tour to Panama perceived
themselves as promoting and taking actions to increase hospital autonomy. In both cases,
the key informants recognized the use of the study tour as providing input for
implementing steps in hospital autonomy. Some of the possible explanations of this
outcome are that, in both cases, the hospital autonomy issue is on the health reform
agenda and the tours were perceived by MOH officials and hospital directors as being of
high quality, pertinent, and applicable. Moreover, there is an Initiative partnership at the
national level and also a headquarters and local alignment regarding the hospital
autonomy effort. But, as with the study tour to Colombia, there is no national
communication and dissemination strategy. Moreover, respondents perceived that there
was no legal framework to support this policy and that the study tour was not designed to
be adapted to their countries, thus limiting implementation.
Study Tour to Canada (PAHO)
Only one case was reviewed and although the experience is evaluated as being of high
quality and applicability, perceived constraints at the country level make application of
the study tour experience difficult. Among them are the lack of a public policy on health
reform and the limited institutional capacity of the MOH. The informants also perceived
an additional constraint; the study tour experience was not accompanied by an in-country
orientation for implementation purposes.
C. SUMMARY OF FINDINGS AND ANALYSIS OF THE STRATEGIC OBJECTIVE
Is the Initiative facilitating countries in achieving effective policy interventions and
delivery of selected health services? Have countries studied integrated lessons learned
from the Initiative? Have they been useful in building institutional capacity for health
IV. EFFECTIVENESS
31
reform and consequently delivering country health reforms more effectively and
providing selected health services and policy interventions in a more effective manner
than they had been? (It is important to note that the evaluation team was asked to focus
the evaluation design on the progress towards Intermediate Result achievement more than
on accomplishment of the Strategic Objective.) With this guidance, no evaluation of
changes in health service delivery was made in the three countries. In most cases, results
are useful for drawing meaningful lessons regarding factors that have affected attainment
of Intermediate Results and are valuable for guiding the second half of the
implementation of the Initiative towards the accomplishment of the Strategic Objective.
Although the evaluation did not explicitly focus on service delivery, one can still examine
whether achievement of the Intermediate Results is leading towards the accomplishment
of the Strategic Objective. The process of capacity building that is aimed at improving
service delivery can be analyzed. From this perspective, the following commentary can
be made.
The results of the midterm evaluation with respect to the introduction and use of NHA
suggest the hypothesis that this is a tool that has improved the institutional capacity for
informed decision-making on health financing and health policy. Two countries are
working on the definition of a specific unit responsible for NHA at the MOH. In addition,
the NHA methodology has been introduced in the three countries studied and two of them
have increased their use of data for health policy decisions and interventions as a result of
using the NHA tool.
The tools and methodologies—Master Plans for Investment, Decentralization Using
Decision Space, and Guidelines for Assessing the Policy Process—are in the
development and testing stage; experience in only one country was evaluated. For this
reason, it is not viable to state an opinion regarding their impact in building institutional
capacity for health reform. However, the evaluation results suggest the hypothesis that
these three tools have potential value in strengthening MOH capacity for delivering more
effective investment planning, to promote a decentralization process of the health system,
and to endorse and support leadership and stewardship of HSR, respectively.
Because there was a small number of respondents with knowledge or experience with the
Management Organizational Sustainability tool, the Cost Revenue Tool, and the
Performance Incentive System, it is not possible to substantively comment on them.
However, the evaluation results suggest that CORE and PYS are valuable in increasing
NGO institutional capacity. An important question for Initiative Strategic Objective
accomplishment is whether it is valid and reasonable to consider this approach and
institutional strategy for the introduction of these tools: do they contribute to institution
and capacity building for HSR? This analysis is not related to the degree to which NGOs
are major stakeholders in health sector reform, nor are they viewed as having the
potential to be major stakeholders in the health sector. Some stakeholders do not consider
some of the NGOs studied as part of the country health system. The Initiative would be
more effective in achieving its Strategic Objective and more efficient in using its
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
32
resources if it facilitates NGO active incorporation into health system reform as providers
of innovative services.
The Initiative has produced four papers21 to promote public sector/NGO partnership
contributions to HSR and to create the conditions for public sector understanding and
awareness as well as NGO capacity to respond. It is not feasible to accurately judge the
impact of the introduction of these experiences because they are in the development and
testing process. However, the evaluation showed differences in the process of building
institutional capacity at the MOH and in NGOs that are related with the Initiative’s
institutional strategy. In this sense, the Initiative could be more effective in achieving its
Intermediate Results and Strategic Objective when NGOS are incorporated into health
system reform, when workshops for NGOs support specific country reform processes,
and when workshops have the private and public sectors participating and collaborating
together.
The dissemination of printed materials and electronic services (IR 2) is making available
a range of information services that, if used, will provide impetus for policy decision￾making. However, an expansion of the dissemination effort into a development
communications strategy will contribute to more sustainable achievement of raised levels
of knowledge, attitudes, patterns, and behavior associated with the successful
introduction of health reform in the region. It will develop a different scenario that is
more interactive, open to all stakeholders in health sector reform, and regionally supports
and reinforces the development of local coalitions that favor health sector reform in each
of the targeted countries.
The midterm evaluation showed that one of the key tools needed for delivering more
effective country health reforms is a methodology for monitoring and follow up.
Acknowledging the relevance of the PAHO framework, the structure or quality of the
instrument and the process through which it is deployed could explain its apparent lack of
implementation and institutionalization. In the country where the framework for
monitoring HSR is being adapted and applied, it is employed as a means of obtaining
consensus on HSR policy. In this case and for this country, this is an important step for
the delivery of more effective health services.
The midterm evaluation of the effect of the Payment Mechanisms Forum in two countries
showed that the majority of the key stakeholders who were knowledgeable of this forum
did not recognize it as input for the implementation (“taking steps based on lessons
learned”) of new payment mechanisms in the health system. Using this indicator, it can
be said that this activity is not perceived as strengthening public institutional capacity for
health financing. Moreover, while the Initiative’s annual report (FY 99) located fora
among networking activities, no data were collected during the evaluation to verify
whether the forum supported the opportunities and means to share experiences and advice
among countries on this topic.
 
21 NGOs’ Role in Health Sector Reform Policy, Public Sector Contracting of NGOs for the Provision of
Health Services, Public/NGO Partnerships in Response to Decentralization and Public/NGO Partnerships
for Quality Assurance.
IV. EFFECTIVENESS
33
The Initiative has been innovative in using the study tours to expand knowledge of best
practices within the LAC region. A substantial number of key informants recognized the
use of the study tour as providing input for taking steps toward hospital autonomy. The
evaluation team perceived indications that these experiences were being used for building
hospital institutional capacity and, by extrapolation, may increase health care services.
However, to improve hospital institutional capacity and consequently to deliver health
services more effectively, participation in the study tours requires MOH executive
commitment to action afterwards and a dissemination/communication strategy.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
34
35
V. INITIATIVE MANAGEMENT EFFICIENCY AND
ADMINISTRATION
This section discusses whether the effectiveness, progress, and achievements described in
the previous sections are being achieved efficiently. Initiative leadership, structure,
management, and resource utilization are reviewed.
A. INITIATIVE LEADERSHIP
Initiative leadership includes vision, a mission, and the value-added nature of the
Initiative. Initiative leadership is shared among the different partners, with USAID taking
the lead. Partners described USAID leadership as excellent, fair, equitable, and one that
listens. It was not found to be systematic and concern was expressed regarding its
dependence on one USAID person.
There is no published Initiative vision or mission statement. However, project documents
and partner comments appear to reflect the Initiative vision as “sustainable country health
sector reforms in effect in the Americas that increase equitable access to high-quality,
efficiently delivered, basic health services.” Complementing this vision is a mission that
makes new tools and methodologies available, provides information on health reform
efforts and experiences, monitors national reform processes and outcomes, provides
feedback to countries, donors, and other partners, and provides the means to network and
share experiences regarding health reform best practices.
Partners’ comments regarding the vision ranged on a continuum from a sense that the
vision was not definitive to one of an extremely productive context. One partner
summarized it by stating that “the Initiative is to promote health sector reform activities
that lead to improvements in equity, efficiency, and quality of services. It is to do this by
developing and disseminating knowledge about effective health reform activities and by
partnering with local actors who participate in the design and deployment of health
policy.”
The value-added nature of the mission was perceived as the synergistic result derived
from bringing together unique partner strengths and adding to the greater whole. It
communicates to country officials that they are not alone in their pursuit of health sector
reform and complements other efforts, such as those by the World Bank and the Inter￾American Development Bank. In summary, the value added element is creation and
dissemination of Initiative knowledge, tools, and activities to promote an improved
regional means of designing, adopting, and implementing health sector reform.
Partners also perceived that Initiative leadership was important because of the existing
competitive relationships of the partners.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
36
B. INITIATIVE STRUCTURE
Initiative structure examines the strengths and weaknesses of the management structure.
This includes the delegation of specific technical areas and activities to partners as well
as their coordination, working relationships, and collaboration in the implementation of
Initiative activities.
The Initiative structure represents a unique and innovative approach to the challenge set
forth in its vision and mission. The actual structure of the Initiative is not graphically
represented in any of the documents and one would be useful (see figure 1). In the
interest of seeking to provide further clarity to this innovative approach, the evaluation
team and USAID staff have collaborated and offer the following prototype structure as
one that can be improved over time. It does not adequately address the feedback loop and
how the Initiative can learn from itself. This is identified as an issue for further
deliberation.
Figure 1: Prototype Structure
LAC/RSD/
PHN
PAHO
G/PHN
PHR DDM FPMD
Regional Activities
Country Beneficiaries
PAHO 
Country 
Offices
USAID 
Country 
Missions
TAG
Steering Committee
TAG = Technical Advisory Group
G = Bureau for Global Programs, Field Support and Research (Global Bureau)
PHN = Center for Population, Health and Nutrition
Indirect
relationship
Direct
relationship
V. INITIATIVE MANAGEMENT EFFICIENCY AND ADMINISTRATION
37
The structure, while perceived by partners as innovative and creative, was also perceived
as creating problems, or as some prefer, opportunities. Opportunities exist at a number of
levels. At the USAID level, there is a perceived competitiveness between the LAC Office
and the Global Bureau. This is seen in USAID country Missions acceptance or lack of
acceptance of Initiative efforts. Missions in two of the three countries perceived the
Initiative to be a value-added contribution to their health reform efforts. The other
country perceived it to be a distraction and detour of resources that should have been
invested into its national efforts instead of into a regional Initiative.
The structure also creates some interesting partnership concerns. There are two major
partners of the $10.2 million Initiative: USAID and PAHO. USAID provides $7.4
million and implements through various contractual and grant arrangements. PAHO
contributes $2.8 million and is also the recipient of $2.5 million of the $7.4 million
USAID funding. This gives it management responsibility for $5.3 millionæmore than
half the Initiative funding. While this should be a win-win situation, the structure creates
ambiguity regarding the partnership issue. USAID believes that it is the lead partner
while PAHO believes that it is not given sufficient credit or status for its investment.
From USAID’s perspective, it is a USAID Initiative. From PAHO’s perspective, it is a
joint partnership, even if the partnership is not equal in terms of monetary contribution.
This ambiguity is not a serious threat to the Initiative. However, it creates unnecessary
distraction to the Initiative’s purpose. Lack of understanding by the other partners of
who has authority and where decisions are made are other results that derive from this
issue. Partner perceptions of Initiative leadership and structure effectiveness are shown in
table 3.
Table 3: Perceived Strengths and Weaknesses of USAID Leadership
and the Management Structure
Strengths Weaknesses
USAID leadership is excellent, fair, equitable, and
listens.
Leadership is not systematic; it is very personalized
and poses a danger if the leader leaves.
The steering committee and ad hoc task forces
appear to be an innovative approach.
Different task forces operate at different levels of
quality resulting in micromanagement at times.
The management structure fosters participation and
collaboration.
Lines of authority are sometimes ambiguous and
leave open to question who has approval authority.
There is a complementarity in the development of
annual work plans.
There is a lack of clarity about how decisions are
made.
Part of the reason for this sense probably derives from the complexity of contracting
arrangements, shown below in table 4. By its very nature, it is more complex because it
must take into account two funding sources. However, USAID implements through
contractual and grant arrangements while PAHO has direct implementing responsibilities.
Moreover, three other partners have implementing responsibility under the different types
of USAID contracting arrangements.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
38
Table 4: Initiative Contracting Arrangements
Organization Contracting Arrangement Responsibility
ABT Associates/PHR Project Performance-based contract through
USAID’s Global Bureau
Public health reform tools,
methodologies, and exchanges
Harvard University/Data Dimensions
Management Cooperative agreement
Research on health systemsæ
decentralization, policy process kit,
and comparative analysis of health
sector reforms
Management Sciences for Health
(MSH)/Family Planning Management
Development (FPMD)
Cooperative agreement Nongovernmental management and
organizational development
Pan American Health Organization Public international organization
grant
Monitoring and investment
methodologies, collection and
dissemination of information
regarding tools, methodologies, and
exchanges; networking
The question that arises is why complicate a fairly straightforward mission with this kind
of arrangement. The answer is opportunity. U.S. government contracting is a
complicated and involved process requiring an infrastructure. The LAC Initiative has
minimal staff and compensates for this through leadership of the Initiative and the
various implementation arrangements. The validity of the adage that “necessity is the
mother of invention” is borne out by these arrangements. Innovative contracting
arrangements and resources already in place were used to fund Initiative projects under
its responsibility. The four partners bring existing products and services to the Initiative
as well as an infrastructure for delivering them. The challenge for management is the
creation of interorganizational cooperation in an arena where partners have been and will
continue to be competitors.
However, as one partner commented, “the fact that all these contracts are working with
different fiscal years with different expiration dates poses a problem that should be
resolved.” These are further described in table 5 below.
Table 5: Initiative Timeline
Year
Partner
1997 1998 1999 2000 2001 2002
PAHO -----July June
PHR ----------- Sept March*
DDM ----------- June
FPMD ----------- Sept March*
*There is a potential for extension of the contract with no increase in funding.
PAHO was the first partner to begin Initiative work, followed by PHR, DDM, and
FPMD, who already had existing Global Bureau contracts prior to the start of the
Initiative.
V. INITIATIVE MANAGEMENT EFFICIENCY AND ADMINISTRATION
39
C. INITIATIVE MANAGEMENT AND ADMINISTRATION
Initiative management and administration examines the coordination of partners and
Initiative work; meeting deadlines; monitoring whether reports, work plans, and other
products have been of adequate quality and on time; new work; and scale up and
sustainability of the Initiative.
In order to manage effective coordination, various mechanisms were created. The
Initiative established a steering committee of all partners that meets once every 6 weeks.
The steering committee is supported by a technical advisory group (TAG) and other ad
hoc task forces set up to guide individual areas, such as information dissemination, NGO
participation, indicators, study tours, and information systems. Each partner is assigned
certain responsibilities and takes the lead for a particular tool or methodology that it may
have designed and/or for which it is the owner or copyright holder. Figure 2 below
shows the partners’ collective sense of how well these are working, with a favorable
overall impression of the steering committee.
Figure 2: Perceived Effectiveness of Initiative Coordinating and Task Forces
The TAG has met once and is reported to have been useful in helping the Initiative reset
priorities. It recommended focusing on Initiative coordination for completing activities
already initiated. However, the TAG meets in the spring of the year and is out of
sequence with the planning that occurs in the latter part of the year. It is perceived to
have the right membership, except for one member who did not attend the first meeting.
Most agree that the TAG has not been incorporated into the Initiative to the degree it
should be in order to be as effective as possible.
The steering committee is perceived to be highly effective. However, it does this in an ad
hoc manner and was not always perceived as providing administrative guidance to the
various task forces or groups it uses for operational management of various parts of the
Initiative. Lack of complete delegation to the various task forces, lack of a clear mandate
for decision-making, and slow approval were cited as concerns. Moreover, it is not
always clear to the partners how steering committee meetings are set nor how the
Not 
Effec tive Outs tanding
Info. Disse m.
Tas k Force
NGO
Tas k Force
Indica tor
Tas k Force
Steering 
Com mittee
TAG
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
40
participants are selected. Even with these limitations, work is accomplished. At times, it
is perceived that the steering committee has focused on operating details to the detriment
of coordination.
The performance of the dissemination task force (DTF) is a case in point in the use of
task forces. Partners perceived it as the weakest of all the structures created by the
Initiative. It was ranked between noneffective and moderately effective. Reasons given
were that the DTF dedicated too much time defining technical questions resulting in
delayed implementation, and as a result, limited effectiveness.
Adjustments were made to the work through the assignment to PAHO and PHR of
different work packages to be developed cooperatively. However, even with these
assignments, the DTF confronted several constraints in its work:
ß changing partner representation and participation in the DTF (skills of the
DTF and in PAHO were heavily oriented to information technology instead of
development communications);
ß implementation arrangements were the result of a weak overall strategy
design;
ß no management tools were found for planning, implementing, monitoring, and
evaluating the strategy by the DTF;
ß the decision-making process in the DTF was slow, and in several cases, the
DTF was micromanaged by the steering committee; and
ß the DTF confused its roles and responsibilities and ended up being neither an
implementing nor a decision-making unit.
In summary, this task force was found to have a poorly defined scope of work and
suggests review of other existing or new task forces.
Initiative administrative management also covers areas such as quality of reports, systems
to track output or interim outcome measures and achievement of Intermediate Results,
shaping annual work plans, defining new areas of work, and scale up and sustainability of
the Initiative.
Quality of Reports
Quality of reports and products submitted to USAID range from excellent to adequate
and are very good on average, particularly the technical reports. However, timing has
presented a problem from several perspectives. For research projects, there has been
delay due to conflicting demands and the desire to assure quality research. For work
plans, the rounds of revisions and reviews are perceived to take an inordinately long time.
As was noted in the section on effectiveness, Initiative customers perceived printed
V. INITIATIVE MANAGEMENT EFFICIENCY AND ADMINISTRATION
41
papers on various topics, including tools, methodologies, experiences, and other
instruments, as one of its most valuable products. Some partners believe that the process
is long, cumbersome, and frustrating.
Tracking Systems
Opinion varied considerably regarding whether systems are in place that can track output
or interim outcome measures. The project is at the point where major products are being
completed. The clearinghouse and web site tracking were noted as two major methods
being used for measurement. The evaluators’ additional perspective is that the annual
report provides the forcing function to track accomplishment.
Management of Intermediate Result Achievement
Closely related to systems for tracking outcome measures is the partners’ perception of
their achievement of Intermediate Results by successfully accomplishing specific
activities in their work statements. In confirmation of the progress and achievements
reported in the section on effectiveness, partners believe that extensions and follow-up
contracts will be important to achieve the Intermediate Results. In addition, work
completed with the World Bank and the Inter-American Development Bank will
contribute to the sustainability of results. It was noted that research projects have taken
longer than anticipated to complete and achieving measurable results was not anticipated
until the completion of and dissemination of the research projects. One partner
commented that preliminary results have been used in a variety of unanticipated ways
that have contributed to project achievement: “Specifically, the research design for
decentralization studies has been published in a well-respected journal (Social Science
and Medicine), preliminary results have been incorporated in the well-attended World
Bank Flagship course both in Washington and in other regional centers, and preliminary
results have been presented in symposia and conferences in Washington.”
Scale Up of Initiative Efforts
Partners perceived that a number of the tools are sustainable and that their use could be
increased. These include national health accounts, the policy tool kit, and others where
partners have a long-term relationship with national institutions. One partner expressed
the belief that the knowledge gained on policy process and decentralization should be
sustained through important publications and incorporation in the World Bank Flagship
course. Other training materials will contribute to health reforms for many years to come.
Increased sharing between countries and monitoring and evaluation of health sector
reform should reinforce increased scale up.
Resource Management
Partners acknowledged the importance of stewardship of Initiative resources in meeting
the challenges of its mission. According to project documents, the Initiative is operating
within the resources allocated to it, although some partners believe that the statement of
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
42
work is greatly under resourced, that is, the magnitude of effort, particularly for creating
behavior change, eclipses the available resources. They also commented that while
financial resources are important, nonfinancial resources may be more critical to the
Initiative’s ultimate success. The human resources involved will determine how robustly
the Initiative continues to garner resources for its efforts.
The project has accumulated an account of social capital or good will with its various
partners. It is at the point where they should begin to pay off in a significant way.
Ironically, because of the unique manner in which the Initiative was structured, it is
limited now because of the expiration of contracts and cooperative agreements which
fund three of the partners. DDM will expire in June and cannot be renewed or extended.
FPMD and PHR expire in September and can be extended until March with no increase
in funding. Given the positive response to most of the tools, it will be costly to the
Initiative to have a hiatus.
USAID staff reports that procurement for follow-on activities for PHR and FPMD are
under way. Once these are completed, the Initiative will choose appropriate mechanisms
for its continuation.
In reviewing the Initiative management efficiency section, some overriding issues bear
reemphasizing. These are summarized below and commented on in the recommendations
section.
Think Regionally, Act Locally
The Initiative is based on the concept that tools and methodologies could be developed,
tested, and disseminated at the regional level. Once these have been completed, there are
choices the Initiative must make to assure adaptation at the country level. This is at
variance with the findings of this evaluation. Only in one instance was there a truly
regional result and that was with the use of some of the instruments in the Flagship
course of the World Bank. Greater attention is required for successful analysis, design,
implementation, and monitoring of health sector reforms at the country level; to create
greater alignment between headquarters and field offices and at the country level; and
with onsite partner staff terms of reference for a successful life cycle of tools or
methodologies.
Project management requires constant stewardship. The detailed review by the evaluation
team during its visit of the use of Initiative tools at the country level appeared as a
management function.22 A country review at least annually or semiannually would be of
added value to Initiative management.
 
22 Of particular value was insight provided about health sector reform across country borders. Individuals
interviewed were directed to other country experiences as well as to resources they had not considered. The
experience indicated that more Initiative management of this type would be helpful.
V. INITIATIVE MANAGEMENT EFFICIENCY AND ADMINISTRATION
43
Initiative Administration Could Be Improved
Plans for the year 2000 for some of the tools are instructive with respect to this point.
Consider the PAHO tools, methodologies, and activities that in many instances indicate
work in progress and, in others, expansion of Initiative activities (see table 6).
Table 6: PAHO Work Plan 2000
Tool or Methodology Work Planned Budget
Health Sector Analysis Two subregional workshops $107,416
Master Plans of Investment Two subregional workshops 115,000
Dissemination Publications/web site 105,592
Monitoring Health Reform Two subregional workshops 102,385
Monitoring Equitable Two subregional workshops 91,640
Networking Subregional forum 50,299
Exchange of Experiences Study tour to Canada 56,929
Financing Social Insurance Two country workshops 41,000
Impact of Social Security Expert panel 122,500
Social Security Reform Expert meeting 45,000
Research Half-day professional meeting in Los Angeles 30,000
Intensive course on HSR Merge with World Bank or affiliate with Latin
American institution
No budget given
This is an imposing workload. It includes an intensive course on health sector reform and
a new focus on social security with the establishment of a new task force to develop it.
Available documents do not indicate that all are scheduled or funded, but at the same
time, do indicate that it is underspending planned expenditures.
Initiative Visibility
The Initiative is not marketed well. This is covered in the dissemination and
communication section but bears repeating here.
Partner Alignment
Greater partner alignment is needed than currently exists. Where USAID Missions and
PAHO offices work together in managing Initiative activities, the potential for success is
much greater than when they do not work together. When this alignment is
complemented by crossfunctional work with other partners at the country level, it is an
even more potent force. This is particularly important in creating the capacity for
continuing to manage the health sector reform effort when the external funding ceases.
There are other stakeholders in the field. Other organizations, such as FunSalud, in
cooperation with the Inter-American Development Bank, were found to be working on
similar efforts. Work is reportedly underway in trying to create an alliance at this level to
increase the benefits of these complementary efforts to client countries. This is a positive
step and the Initiative should use it in moving the level of interaction up another level
from partnership to alliance.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
44
45
VI. SUSTAINABILITY
Sustainability of Initiative services and product delivery is related to the assumptions
used in formulating the Initiative framework and design. While these assumptions are
largely outside the Initiative’s control, this does not prevent the Initiative from vigorously
pursuing ways to influence them as a means to ensure long-term sustainability (see
appendix E for factors affecting project success within and beyond the control of the
Initiative). A more effective due-diligence sustainability analysis is critical to the
successful deployment of Initiative Intermediate Results and the Strategic Objective.
The three critical assumptions identified were:23
1. Government and nongovernment health providers, professional societies, and
university faculty willing to participate together in reform efforts;
2. In-country interested parties can mobilize political will to redirect resources
(time, personnel, and money) to reforms that increase equitable access to basic
health services; and
3. Donors continue to fund capital costs for country health reform design and
implementation, including technical assistance, studies, and systems
design/implementation.
While the Initiative is regional in scope, the expected achievements take place mostly at
the national level. Common external factors that influence the institutionalization of the
Initiative’s tools, methodologies, and results of exchanges were identified through field
interviews. (Institutionalization is the incorporation of these tools, methodologies, or
results of exchanges into routine or everyday use in the health sector.) Table 7 on the
following page contains the factors that affect the probability of sustained capacity.
The table reflects two main aspects from the majority of key respondents: concern over
the weakness in MOH institutional capacity and/or lack of or weak political will to move
the health reform process forward. These were further broken down into four factors
perceived as most difficult to overcome in making Initiative benefits sustainable over
time:
ß lack of a health sector reform public policy: definition and formulation of
problems and internal and external constraints to public policy formulation;
ß lack of health sector reform leadership, political decisions, and process
management difficulties;
 
23 See Results Framework, December 8, 1997.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
46
ß limitations of MOH institutional capacity in the area of health sector reform;
and
ß parallel, duplicative, and competitive reform units within the health sector and
political interference in decision-making at the MOH.
Table 7: Factors Hindering Institutionalization and Sustainability* of Tools,
Methodologies, and Exchanges
Requirements for Institutionalization and Sustainability
Tool, Methodology, or Experience** Peru Dominican
Republic Honduras
Master Plans for Investment
1. MOH institutional capacity
2. MOH political will
3. Agencies’ support for technical assistance
National Health Accounts
1. Information, communication,
dissemination, and training
2. Institutional capacity
3. Political will
1. Institutional capacity
2. Political will
Monitoring Health Reform Political will MOH institutional capacity
NGOs in HSR, NGOs Contracting, Health Policy
Process Applied Research, and Decentralization
Applied Research
1. Political will
2. Institutional capacity
1. Information systems on costs and production
2. Implementation systems of payment in place and functioning
Provider Payment Mechanisms (Peru and Honduras
only)
1. Political will
Study Tours on Hospital Autonomy 1. Technical
assistance
2. Study tours
for MOH
officials
Political will 1. MOH strengthening
2. MOH and hospital capacity 
building
*See appendix H for responses regarding requirements for sustainability.
**The three interviewees for MOST pointed out that this methodology has not been applied in their NGOs.
The interviewee for NGOs in Decentralized System and NGOs Quality Assurance pointed out that the key
for sustainability is that the NGOs continue the process and increase the demand to the public sector. The
two interviewees for CORE identified two factors affecting the sustainability of the tool: a general
agreement and commitment with USAID about the tool’s implementation and the availability of cost data
from regional offices. Finally, the interviewee for Use of Performance Incentive System identified the
availability of financial resources as the key requirement for sustainability.
The three Initiative assumptions need to be updated and refined using these factors.
Making them more explicit will allow the Initiative to influence them and ensure the
achievement of the Intermediate Results and Strategic Objective.
Factors that contribute to long-term sustainability and effective risk assessment of
Initiative efforts include the following:
ß Make assumptions as specific as possible. Instead of using a general
assumption (for example, donors continue to fund capital costs for country
health reform design and implementation, including technical assistance), a
more specific assumption using performance indicators such as the following
VI. SUSTAINABILITY
47
would be better: The World Bank and the Inter-American Development Bank
commit at least $xxx over the next five years to the design and
implementation of health sector reform.
ß Be more realistic in Initiative risk assessment. This could be accomplished
by analyzing the probability that the assumption will hold. Assumptions
having a high failure probability represent the higher risk. Figure 3 proposes a
methodology for assessing risk for the Initiative.
Figure 3: Establishing the Level of Risk of the Assumptions
The shaded area indicates high risk and where mitigation strategies by the steering
committee would be useful in increasing the effectiveness of the Initiative. By
establishing a mitigation strategy, management can better influence the assumption, thus
lowering risk to the Initiative.
The suggested development communications strategy, described earlier in the section on
effectiveness, used an expansion of the dissemination strategy as an example of how risk
could be mitigated. It promotes coalition building at the national level by increasing key
stakeholder participation and commitment.
Such a strategy would also be more clearly connected to the Results Package by
addressing the creation of viable political regional scenarios, favoring coalition and
consensus-building efforts at the national level and reinforcing the capacity of those who
can champion the reform at the national level. The role of international and regional
cooperation agencies and banks is critical and the definition of a course of action to
influence decision-makers in those organizations would be necessary.
R
e 
l 
e 
v 
a 
n 
c 
e
Probability of Failure 
100%
H
75%
S
50%
M
25%
L
0%
0% L 25% M 50% S 75% H 100%
High relevance, low 
to moderate risk
Low to moderate 
relevance, moderate 
to high risk
Low to moderate 
relevance, low to 
moderate risk
High relevance, high
risk
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
48
49
VII. RECOMMENDATIONS
The recommendations are divided into the following sections:
ß Initiative framework and sustainability;
ß effectiveness (this section is divided into two parts to provide continuity: it
first considers the tools, methodologies, experiences, and/or exchanges
included in Intermediate Results 1, 3, and 4, and then discusses IR 2, which
covers a different but directly related areaædissemination, information, and
communication);
ß Initiative management and administrative efficiency;
ß methodology and study design; and
ß considerations for the next work plan.
A. INITIATIVE FRAMEWORK AND SUSTAINABILITY
The Initiative framework and sustainability are two interconnected components.
Sustainability is a result of the degree to which the Initiative framework and the
assumptions it is based on hold.
1. Adjust the Initiative Results Framework by operationally defining key stakeholders
working on health reform, working with Initiative policy tools or other policy reform
analysis24 methodologies in each of the 13 targeted countries.
2. Redefine Intermediate Results and indicators as well as their tracking system to
establish the difference between indicators that generate impact (i.e., use of services)
and those that reflect basic delivery of services. It is one thing to create and deliver a
tool on a timely basis and another to have it used by clients.
3. Redefine assumptions, making them as specific as possible, including measurement
indicators.
4. Identify and analyze assumptions or external factors influencing attainment of the
next level of the hierarchy.
 
24 These include the PHR Policy Tool Kit (June 2000), the Harvard policymaker (M.R. Reich and D. M.
Cooper, 1996-98), and the PAHO Lineamientos para la realización de un análisis estratégico de los
actores de la reforma en salud (Junio 2000).
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
50
5. Consider using the risk mitigation model to generate mitigation strategies to be
incorporated into the framework as new actions. In some cases, for example, the
communications strategy is part of the Intermediate Results.
6. Periodically monitor the level of risk, using the recommended (or other) methodology
to assess movement in the level of risk.
B. EFFECTIVENESS
Intermediate Results 1, 3, and 4
1. Reassess each tool as part of the 2001 work plan exercise in October 2000. The
proposed checklist (see table 8) for assessing the introduction and use of tools,
methodologies, experiences, and/or exchanges or some variation would be helpful in
this regard. The scores, with adjustments, could be applied to the risk assessment
model (described in the section on sustainability).
Table 8: Proposed Checklist for Assessing the Introduction and Use of Tools,
Methodologies, Experiences, and/or Exchanges
Values
Factor Low-------------------------High
1 2 3 4 5
Quality of methodology
Need for the tool, methodology, experience and/or exchange
Training and technical assistance support planned and consistent with
national expectations
Potential for national adaptation
Regional approach complemented by country implementation
Alignment of Initiative partnership at headquarters
Alignment of Initiative partnership at the country level
Alignment of headquarters and local staff
Country partners/consultants have specifically related terms of
reference
A national participatory, coalition-building development process that
favors use of the tool
Stability and continuity of key stakeholders supporting Initiative
activities
The activity, tool, methodology, or experience is included on the health
sector reform agenda
The MOH will commit to introducing and using the tool, methodology,
and/or experience
Total Points
Score
32 and lower 33-40 41-48 49-54 55-60
Low Potential Fair Potential Average Potential Good Potential High Potential
2. Use the reassessment of each tool in achieving consensus regarding future plans and
the collective investment of Initiative resources. For example, given the level of
VII. RECOMMENDATIONS
51
resources and contract schedules, consider whether some should be dropped. If the
goal is institutionalization of the tools, then create a plan for this to occur.
3. Reassess the indicators used to measure the methodology for monitoring health sector
reform because the present ones are not accurate. Use of the tool at the country level
by MOH staff and others should be the metric.
4. Develop a process that will assure that tools, methodologies, and exchanges respond
to the health priorities of targeted countries.
5. Obtain commitment of other stakeholders for support and technical assistance25 for
the design and development of effective and efficient training processes.
6. Create a strategy for garnering support for the LAC Initiative in Health Sector
Reform from different public stakeholder institutions: national institutions, such as
the MOH, Ministry of Finance, reform units, NGOs, and other governmental
organizations, such as Congress and state reform units; international institutions, such
as other donors and agencies (i.e., the World Bank and the IADB).
7. Create agreements with key stakeholders at the country level on the best institutional
strategy and plan for implementation. This reinforces an Initiative design premise of
subsequent assumption of regional work by others for country-level implementation
and adaptation.
8. Exercise more Initiative influence over design and implementation factors that affect
country utilization of regional tools. The Initiative’s approach in this area crucially
and significantly affects its success. Midterm evaluation data reveal that the most
successful experiences are those in which the regional approach and intervention
strategy are accompanied by an in-country initiative. The regional nature of the
Initiative requires that it influence country utilization of regional tools. This factor
and the means to influence it should be considered in the design of Initiative work
plans.26
9. Identify and influence factors beyond Initiative control that affect Initiative results.
Identify and use alliances and agreements with key stakeholders in health sector
reform in target countries to assure that:
ß tools, methodologies, and experiences are adapted and/or customized to local
or national requirements;
ß there is a national communication and dissemination strategy; and
 
25 The design of the training mechanisms is managed by the Initiative but the technical assistance is not.
Under the mandate of a regional Initiative, the partners are prohibited from providing technical assistance
at the country level.
26 Another option is a second-generation projectæa follow-on Initiativeæthat could provide country￾specific technical assistance with Mission funding as part of a regionally organized scheme.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
52
ß a stronger alliance is forged among Initiative partners and other agencies, such
as the World Bank and the Inter-American Development Bank.
10. Initiative partnerships are a critical strategy for Initiative implementation. Four kinds
of increased partnership alignment are recommended:
ß an Initiative partnership at headquarters: the Initiative’s approach to designing
and introducing NHA is a good example of such an alignment;
ß an Initiative partnership at the national level:27 two examples that were
mentioned were the adaptation of monitoring health reform in the Dominican
Republic through the strong alignment of Initiative partners PAHO and PHR,
and the Initiative partnership among PAHO, USAID, and PHR regarding the
introduction, use, and institutionalization of NHA in Honduras;
ß PAHO and USAID headquarters aligned with local partnership efforts for
each tool and activity: an example is the strong USAID position of the NGO
strategy in Honduras and the Dominican Republic and PAHO alignment with
PHR in support of monitoring health reform in the Dominican Republic; and
ß country partners’ consultants (PHR or FPMD) with specific tool-related terms
of reference provide stronger support for Initiative activities: for example, the
PHR consultant in the Dominican Republic has terms of reference specifically
related to the MHR methodology.
11. Develop plans to manage factors beyond Initiative control. Among these are plans
for
ß increased stability and continuity of key stakeholders favoring Initiative
activity; for example, the Initiative could encourage continued involvement of
some key stakeholders through special agreements or contracts with them for
specific products or services that would also move the agenda of health sector
reform forward;
ß increased channeling of Initiative efforts through partners to increase support
of specific country programs; for example, the Initiative could help expand
use of the Master Plans of Investment tool in Honduras through other partners
in other countries with similar multicountry programs to increase health
coverage;
 
27 Based on the evaluation process, it is not possible to discern whether the country coordination among the
partners (i.e., USAID and PAHO, PAHO and PHR, or USAID and PHR) is a result of unrelated bilateral
activities or a result of Initiative efforts. There is also no way to demonstrate whether coordination among
the partners already existed or whether it came with Initiative activity. What the report suggests is that the
Initiative’s partnerships at headquarters and the local level, and the alignment between headquarters and
local staff, affect the achievement of the Initiative’s objectives.
VII. RECOMMENDATIONS
53
ß influencing political health reform decisions regarding Initiative tools,
methodologies, or experiences; for example, at the Initiative level, partners
could agree on a common strategy that would be implemented by partners
through channels available to them individually or collectively in a given
country, such as relationships with key health officials and bilateral projects;
ß strengthening political will to introduce Initiative tools, methodologies, or
experiences to bring change to the health system; following through on many
of the recommendations on dissemination would help strengthen political will
through increased public awareness and influence on key political figures;
ß ensuring that Initiative activities are on the national health reform agenda in
targeted countries; for example, by strategizing together at the regional level
and then using partners’ linkages in individual countries; and,
ß assessing the level of institutional capacity at the country level before
introducing the tools, methodologies, experiences, and/or exchanges, and
periodically monitoring the degree to which it has strengthened national
institutional capacity for health reform, such as trained personnel,
strengthened public administration, and institutional design. Examples are
found in the recommendations on effectiveness and in particular, specific use
of the proposed checklist (page 50).
Intermediate Result 2: Dissemination and Communications Strategy
1. The steering committee should adopt a new methodological approach for
dissemination and communication. The products and services already provided
should be continued, improved, and focused accordingly on new objectives, and a set
of new services should be added. This requires transforming the dissemination effort
into a development communications strategy that would improve both regional and
national impact. This key recommendation is based on customer and partner
perception of ways to strengthen and reinforce interaction between Intermediate
Results (described elsewhere in the report). The Initiative should shift paradigms to
increase achievement of the Strategic Objective. Such a shift would mean a more
horizontal communications structure between the Initiative and the targeted
audiences, increased feedback, and a more interactive web site, newsletter, and other
media. It represents a dramatic conceptual move to make messages more consistent
with Initiative offerings through expanding media used for channeling messages and
reorienting them to problems and interests of the audiences. (See step 12 of the
development communications strategic cycle in appendix P.) Moving through the
different steps of the cycle would refocus and expand the strategy.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
54
2. Apply the cycle (see appendix P) to gain
ß closer linkage between the strategy and the overall Results Framework, thus
mitigating the risk of failure of critical assumptions related to political will;28
ß redefinition of targeted audiences;
ß redefinition and tailoring of messages to redefined audiences;
ß broadening of selected media;
ß change in the content of some existing media, mainly the web site and
newsletter;
ß establishment of a management system based on clear institutional and
implementation arrangements;
ß development of an Initiative learning system through transparent monitoring
and evaluation to improve performance; and
ß improving brand recognition while maintaining partner identity.
This development communications package is described in greater detail below.
Audiences
Focus the strategy on decision-makers, including the new groups identified (legislators,
journalists, private sector representatives, health professionals). However, it should not
focus directly on the public or public opinion, as suggested by many respondents in the
field testing. This goes beyond the scope and customer base of the Initiative. The process
of influencing public opinion is national in character and should be addressed at that
level. Targeting the new groups referenced above given their frequent interaction with
the core group of decision-makers could generate scenarios for national and regional
coalition building.
Messages
Hold a message development session based on the new audience screening and its
identification of key interests and problems. The new messages should be thematic and
tailored to each audience. The following format is recommended:
ß a short basic statement (5 lines),
ß factual support, and
ß anecdotal and/or historical/geographic linkage with the audience.
 
28 For a broader discussion of risk mitigation, see the section on sustainability.
VII. RECOMMENDATIONS
55
Each message should be classified by theme and audience and made available for
Initiative spokesperson(s) so that everyone will have the same core messages.
Media Channels
Web Site
The web site should be more interactive than it is and should include the recent initiatives
from the PAHO implementation team. Other ideas that could be explored are:
ß creation of a news section on health reform-related issues in the Americas;
ß create a section with articles from regional health reform leaders to foster the
idea of champions (voices of the reform);
ß develop a network of journalists that specialize in health issues, targeting key
media in the region; the web site could be used for access and interaction by
the new journalist network; and
ß consider the development of chat rooms using health reform personalities,
question and answer sessions, and other interaction alternatives.
Television and Radio
Prepare newsworthy materials for regional television outlets, such as CNN en Español,
Televisa, Telemundo, or Univision. They are increasingly interested in health and now
have segments dedicated to medical and health issues. The Initiative has many attractive
stories to offer to these carriers. Organizing regional and national press conferences and
interviews with television/radio outlets in connection with field visits, workshops,
forums, and other activities should be planned as part of the strategy.
Videotape Production
Produce an institutional videotape (4-5 minutes) that presents the present health reform
situation and uses local and national key stakeholders to show how Initiative tools and
methodologies have been used successfully in mitigating this situation.
Produce 30–second spots for distribution through regional television stations and other
regional media that accept public information messages at little or no cost.
Opinion Editorials in Regional and National Magazines/Newspapers
Generate articles written by both regional and national Initiative allies that support the
development of health reform champions. They could be instrumental in creating positive
scenarios for debate and generating favorable trends on specific issues. Regional
specialized magazines and health sections in newspapers should also be targeted.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
56
Additional New Direction Actions
An Annual Report on Health Sector Reform Status
Develop an annual report on the status of health sector reform in the Americas. This
would improve Initiative brand recognition and reinforce its regional character. Ideally, it
would become a yearly announcement of the state of health sector reform in the
Americas. A press conference in Washington with simultaneous events in two or three
cities in the region should be implemented.
The Annual Journalist Prize on Health Reform
Award an annual health reform journalist prize to a member of the new journalist
network. This would accent the Initiative’s comparative regional advantage, reinforce
links with journalists throughout the region, and increase knowledge of the Initiative’s
unique regional brand.
Organize Workshops for Journalists on Health Reform
Organize subregional or national workshops based on problems and results achieved to
gain recognition and presence in the media and to generate simultaneously knowledge on
health reform issues.
Develop and introduce a Tool Kit on Communications for Health Reform in Each
Targeted Country
Have some of the partners produce a simple tool kit to improve capacity and increase
local impact. The tool kit should address issues ranging from how to design a national
communications strategy for health reform to how to implement different types of actions
and manage the strategy. An accompanying operations manual should be a substantive
part of the tool kit. Technical assistance may be required to introduce it and assure
sustainability.
Increasing Brand Recognition29 and Maintaining Partners Identities
Formulate a new strategy for increased brand recognition. Using the Initiative’s corporate
communications strategy umbrella for proposed activities would contribute to
 
29 Brand recognition will certainly be improved if the new proposed activities are conducted under the
umbrella of the Initiative. The formulation of a new strategy for increased brand recognition is dependent
upon the horizon (projected longevity) of the Initiative. If the Initiative is going to be renewed, the best
alternative would be to redesign the branding with new logo colors, design, etc. It would be best to
combine the corporate branding in a different way, establishing a new equilibrium between corporate
branding and individual (institutional) name recognition.
VII. RECOMMENDATIONS
57
establishing a new equilibrium between the corporate brand and partner name
recognition.
The following approach and course of action is suggested for rolling out a new phase of
the Initiative for the next two years and for increasing brand recognition in the process:
1. Define a set of actions that will be presented in the future as entirely corporate (press
releases, press conferences, institutional videotape, television spot).
2. Create a new Initiative news service sponsored by USAID and PAHO to deploy the
new Initiative corporate communications strategy.
3. Do not radically alter the logo and colors. Use the colors of the newsletter instead of
the orange/blue color combination of the logo. Make the logo more consistent by
taking away LAC and adding an I, so that it will be IRSS (Iniciativa Regional del
Sector Salud or Iniciativa Reforma del Sector Salud).
4. Boost brand recognition through definition of a leading phrase to be used in the
different campaigns.
5. Develop a set of complementary communications actions in which the name
recognition of the different partners is emphasized and include them as part of the
yearly work plan to maintain a desirable equilibrium between institutional identity
and Initiative branding.
A New Strategy Design and Implementation Arrangements
Reorient the strategy from dissemination to a development communications approach by
sequencing the different steps of the cycle in the following way (see appendix P):
STEP 1
Organize a communications development mini-workshop for key stakeholders of the
Initiative and the dissemination task force (DTF) in order to:
ß move from Step 12 to 7 in the development communications cycle; thus, the
sequence is from audience screening to message development, media
identification, expected KPAB impact, product, and services to be delivered,
and the first draft of an implementation plan;
ß add internal or external development communications skills to a re-appointed
PAHO team;
ß redefine the statement of work of the new implementation team and its
relationship with the steering committee; establish clear roles and
responsibilities and allow the new team to outsource services, if needed; and
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
58
ß reorient the budget in accordance with new priorities (pricing of these changes
is included in appendix P).
STEP 2
Require the new implementation team to develop a monitoring and evaluation system and
an overall management system, including a work breakdown structure, Gantt charts and
responsibility charts. Technical assistance should be provided as needed.
STEP 3
The steering committee approves the new strategy, the implementation plans for the first
year, the monitoring and evaluation processes, and its annual events and assigns a budget.
C. INITIATIVE MANAGEMENT AND ADMINISTRATIVE EFFICIENCY
1. Clarify the roles and responsibilities of the steering committee, partners, and task
forces. Use responsibility charts to show who is responsible, who needs support, who
must be communicated with, and the latitude of decision-making authority.
2. Reexamine the PAHO partnership. Given the completion dates of USAID agreements
with other partners, PAHO may need to assume increased responsibility for Initiative
activities.
3. Prepare milestone charts for each activity and task force, review them on a defined
schedule in steering committee meetings, and link them to Intermediate Results. A
standardized set of charts that shows what is on schedule and what is not on schedule,
what is within the budget and what is not within the budget, should be included.
Where there is schedule slippage on deliverables, require a recovery plan. Begin with
a review of the 2000 work plan.
4. Reschedule the TAG to coincide with the planning cycle and if it is willing, use it to
help with the recommended mitigation plans for managing assumptions.
5. Use the steering committee to manage the assumptions of the Results Package and
mitigation plans described above.
6. Simplify and expedite the publications process.
D. METHODOLOGY AND STUDY DESIGN
Use a representative sample approach for succeeding evaluations. In the present LAC
Initiative midterm evaluation, USAID and the steering committee, for various reasons,
limited the field study to three countries, which prevented generalizing conclusions
drawn from the three-country case study. If representative findings of the regional
VII. RECOMMENDATIONS
59
activity are desired to evaluate the majority of the products produced, in future
evaluations it would be advisable to visit more countries and to work with a
representative sample of informants by activity.30 (An alternative to the present study
would have been to send two people to three countries each to gather data. However, to
have done so would have required more evaluation team preparation time.)
E. SPECIFIC CONSIDERATIONS FOR THE NEXT INITIATIVE WORK PLAN
The work plan for FY 00 continues with ongoing activities that the Initiative has
implemented during the past two years and new activities that it is developing in the areas
of NHA, social insurance, research on health sector reform, and capacity building. To
support the implementation of ongoing activities, the following priorities are suggested:
1. Introduce the PAHO framework for monitoring health reform at the national level of
the MOH (and in some cases in reform units) through a participatory development
process, building a coalition that favors the use of the framework.
2. Institutionalize NHA by incorporating the following requirements identified in the
midterm evaluation for institutionalization and sustainability: promotion of MOH
political will; institutional capacity; reinforcement of information, communication,
dissemination, and training on NHA at the country level and reinforcement of the
Initiative network or alliance among the World Bank, IADB, PAHO, and USAID that
has been operational since 1995.
3. Increase emphasis on crossfunctional deployment of policy tools in the development
process: guidelines for assessing the policy process/DDM and policy tool kit/PHR.
4. Expand the dissemination effort into a development communications strategy.
5. Reinforce the Initiative joint NGO strategy, applying it to the experiences related to
public/NGO partnership and also to the introduction of managerial tools to NGOs.
6. Work with clients in the development, testing, dissemination, and implementation of
tools, methodologies, fora, and study tours, with the understanding that other donors
and agencies at the central and local levels (i.e., World Bank and IADB) are among
the clients for the Initiative’s actions).
7. Use the proposed checklist (table 9) to assess risk when initiating new activities, such
as the process for social insurance practice, guidelines and technical assistance, and
the new study tour to Canada.
 
30 When the sample of key informants is too small, for example, in the case of key stakeholders interviewed
for the evaluation of MOST, it would have been suitable to interviewæby telephone or e-mailæa wider
group of participants in the Initiative activity. This evaluation approach would have yielded more definitive
results than it did.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
60
8. Incorporate tools, methodologies, and experiences in the World Bank Flagship course
to increase dissemination and technical assistance.
61
VIII. REFERENCES
Berman, P., Arellanes, L., Henderson, P., and Magnoli, A. Health Care Financing in
Eight Latin American and Caribbean Nations: The First Regional National Health
Accounts. Partnerships for Health Reform. April 1999.
Bossert, Thomas J. Decentralization of Health Systems: Decision Space, Innovation, and
Performance. Data for Decision Making. November 1999.
Bossert, Thomas J. World Bank Flagship Course. Module 9: Decentralization of Health
Systems. Background Paper. Harvard School of Public Health. 1999.
Bossert, T., Larranaga, O., Infante, A., Beauvais, J., Espinosa, C., and Bowser, D.
Applied Research on Decentralization of Health Systems in Latin America: Chile Case
Study. Data for Decision Making Project. December 1999.
Bossert, T., Larranaga, O., and Meir, F. Decentralization of Health Systems in Latin
America. Pan American Journal of Public Health. Special Issue on Health Sector
Reform.
Family Planning Management Development. Latin American and the Caribbean Health
Sector Reform Initiative.
______. Health Sector Reform in the Americas: Equitable Access to Basic Health
Service. Proposal to Strengthen NGOs Capability in the Health Sector Reform Initiative.
Submitted to the United States Agency for International Development. 1997–98.
______. Strengthening NGO Contributions to Health Sector Reform. Work Plan
prepared by FPMD. April 1998.
______. Strengthening NGO Contributions to Health Sector Reform. FPMD/MSH FY00
Work Plan. December 23, 1999.
______. Strengthening NGO Capacity to Support Health Sector Reform: Sharing Tools
and Methodologies.
______. FPMD/MSH FY99 Work Plan.
______. FPMD Annual Accomplishments—FY98 LAC Health Reform Initiative. October
1998.
Kolehmainen-Aitken, R. State of the Practice: Public-NGO Partnerships in Response to
Decentralization. Family Planning Management Development Project. Boston:
Management Sciences for Health, February 2000.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
62
Latin America and the Caribbean Regional Health Sector Reform Initiative.
Clearinghouse on Health Sector Reform in Latin America and the Caribbean.
http://www.americas.health-sector-reform.org/english/clearinghouse.htm.
______. Methodology for Monitoring and Evaluation of Health Sector Reform in Latin
America and the Caribbean. Washington, D.C. December 1998.
______. Base Line for Monitoring and Evaluation of Health Sector Reform in Latin
America and the Caribbean. Washington, D.C. (undated)
______. Clearinghouse on Health Sector Reform. Washington, D.C. (undated)
______. Annual Report. 1998 and 1999.
______. Meeting Minutes, LAC HSR Initiative Steering Committee for meetings held on
December 6, October 13, September 8, and July 19, 1999.
Letter from USAID to PAHO approving the grant, Health Reform in the Americas:
Equitable Access to Basic Health Service. August 20, 1997.
Maceira, Daniel. Provider Payment Mechanisms in Health Care: Incentives, Outcomes,
and Organizational Impact in Developing Countries. Major Applied Research 2,
Working Paper 2. August 1998.
Pan American Health Organization. Health Sector Reform in the Americas: Equitable
Access to Basic Health Service. Proposal to the United States Agency for International
Development, 1997–2002. Washington, D.C. July 17, 1997.
______. Work Plan, July 1997–June 1998. Washington, D.C. October 10, 1997.
______. Latin America and the Caribbean Health Sector Reform Initiative. Washington,
DC.
______. Regional Forum on Provider Payment Mechanisms, Lima, Peru, 16–17
November 1998. Final Report. Washington, D.C. March 1999.
______. Cuentas Nacionales de Salud: Bolivia; Washington, D.C. (undated)
______. Technical Advisory Group Meeting Report. Washington, D.C. (undated)
______. LAC Health Sector Reform Initiative: Annual Report FY99.
______. LAC Health Sector Reform Initiative: Work Plan FY00.
VIII. REFERENCES
63
______. Canadian Health Insurance and the Structure and Function of the Canadian
Health System. Study Tour Report, September 20–24, 1999.
______. Foro Subregional Andino sobre Reforma Sectorial en Salud. Informe de
Relatoria, Santa Cruz, Bolivia, 5 a 6 Julio de 1999. Enero de 2000.
______. Lineamientos para la realización de un análisis estratégico de los actores de la
reforma en salud. Junio 2000.
PHR/LAC Initiative Work Plan, October 1, 1999–September 30, 2000. Updated February
1, 2000.
Partnerships for Health Reform. NGOs’ Role in Health Sector Reform Policy.
______. Public Sector Contracting of NGOs for the Provision of Health Services.
______. Policy Tool Kit. June 2000.
Rosenthal, G. State of the Practice: Public-NGO Partnerships for Quality Assurance.
Family Planning Management Development Project. Boston: Management Sciences for
Health, February 2000.
Rossetti, A., and Bossert, T. Comparative Analysis of Policy Processes: Enhancing the
Political Feasibility of Health Reform. Data for Decision Making. November 1999.
United States Agency for International Development. LAC HSR Initiative: DDM 98
Work Plan. (undated)
______. Guidelines for Dissemination and Communication. April 1998.
______. LAC/RSD Health Reform Results Package. July 1, 1997.
______. Results Framework: LAC/RSD Health Services Objective. July 1, 1997.
Winnett, L. and Wallack, L. Advancing Public Health Goals through the Mass Media.
Journal of Health Communication,Vol 1, pp173-196, 1996.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
64
APPENDICES
A: Scope of Work .............................................................................................................67
B: Health Sector Reform Results Package Indicators, February 2000.............................75

APPENDIX A
SCOPE OF WORK
(FROM USAID)
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
68
Scope of Work
Midterm Evaluation
Health Priorities Project, 598-0825
LAC Regional Health Sector Reform Initiative
August 30, 1999
Background
Countries throughout Latin America and the Caribbean are introducing reforms that can
profoundly influence how basic health services are provided and who receives them.
Health system reform is being undertaken to reduce inequities, improve quality and
correct inefficiencies in current systems.
Governments in the region identified the need for a network to support health reform
through analysis, training and other capacity-building measures at the Summit of the
Americas in 1994 and again at a Special Meeting on Health Sector Reform convened by
an interagency committee of the UN and other multilateral and bilateral agencies in 1995.
In response, the United States Agency for International Development (USAID) and the
Pan American Health Organization (PAHO) launched the Latin America and the
Caribbean Regional Health Sector Reform Initiative. This Initiative seeks to promote
more equitable and effective delivery of basic health services by supporting regional
activities. The regional efforts of the Initiative support informed decision-making on
health policy and management, health financing, health service improvement,
decentralization and institutional development.
The Health Sector Reform Initiative is one of five initiatives which support the LAC
Regional Strategic Objective (SO) "more effective delivery of selected health services
and policy interventions', by focusing specifically on "more effective delivery of
sustainable country health sector reforms (designed to increase equitable access to high
quality, efficiently delivered basic health services). In country capability to assess health
sector problems, and to design, implement and monitor reforms and solutions".
The LAC Regional Health Sector Reform Initiative is implemented by the Pan American
Health Organization, the Partnerships for Health Reform Project, the Family Planning
Management Development Project and the Data for Decision Making Project. PAHO is
an international public health agency headquartered in Washington, D.C. It serves as the
specialized organization of the Inter American System for health and as the Regional
Office for the Americas of the World Health Organization.
Partnerships for Health Reform is a five–year USAID-funded initiative that builds
capacity in policy foundation and implementation, health economics and financing, and
organization and management of health systems.
APPENDICES
69
Family Planning Management Development is a five–year USAID-funded initiative that
helps national and local family planning and health programs and organizations develop
their capability to plan and manage sustainable programs.
Data for Decision Making is a five–year USAID-funded initiative that supports health
Sector reform and assists leadership in developing countries to make informed policy and
financing decisions.
The Initiative provides regional support to activities in Bolivia, Brazil, the Dominican
Republic, Ecuador, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico,
Nicaragua, Paraguay and Peru.
The five–year Initiative was authorized on July 29, 1997, for a LOP of $7.4 million in
USAID funding. PAHO also committed $2.8 million of its funds to the Initiative for a
total value of $10.2 million. The allocation of USAID funds envisaged was $2.5 million
for PAHO, $3.2 million for the Partnerships for Health Reform Project, $900,000 For the
Data for Decision Making Project and $750,000 for the Family Planning Management
Development Project. Of the $1.4 million in USAID funds, $6.0 million was new funding
approved at the time of authorization and $1.4 million was FY95 carryover field support
funding ($1.1 million of PHR's planned allocation and $300,000 of DDM's planned
allocation).
Results and Performance Indicators
The Strategic Objective of the Health Priorities Project is: More effective delivery of
selected health services and policy interventions.
The aspect of the SO that is the objective of this Initiative is: More effective delivery of
sustainable country health sector reforms (designed to increase equitable access to high
quality, efficiently delivered basic health services).
The indicators for monitoring and assuring the results of the Initiative are:
Strategic Objective 3 Indicator:
ß 50% of Initiative countries' reform processes substantially integrate lessons
learned from the Initiative (methodologies and tools, information, monitoring
and sharing).
The Intermediate Results and their corresponding indicators are:
Intermediate Result 3.4.1: Methodologies and tools developed, tested and disseminated
for analysis and design, implementation and monitoring of country health sector reforms.
Indicator 3.4.1: All methodologies and tools used by key actors in 50% of countries
where introduced.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
70
Intermediate Result 3.4.2: Information on health reform efforts and experiences
gathered and made widely available to interested parties in LAC countries and to health
sector donors.
Indicator 3.4.2.a: At least 5 different institutions in each of the LAC Initiative target
countries report receiving Initiative materials.
Indicator 3.4.2.b: 50% of electronic network users surveyed report finding network
services useful.
Intermediate Result 3.4.3: Reform processes, specifically related to equitable access,
and outcomes monitored and feedback provided to countries, donors and other partners.
Indicator 3.4.3.a: All LAC Initiative countries monitor health sector reform using
framework established by PAHO.
Indicator 3.4.3.b: Regional monitoring system for comparative analysis for health sector
reform institutionalized by PAHO.
Intermediate Result 3.4.4: Opportunities and means to share experience and advice
among countries are established.
Indicator 3.4.4: 5O% of participants surveyed report taking steps based on lessons
learned in fora.
Evaluation Objectives
ß To determine if the results framework is valid and feasible and to recommend
adjustments to the results framework and activities as appropriate.
ß To examine progress towards achieving the SO and IRs as planned evaluating
whether the results/indicators will be met in a timely and effective manner. To
identify specific internal/external constraints which may limit their
accomplishment or success and to recommend adjustments based on findings
and conclusions.
ß To assess how the Initiative structure as configured is working including both
the delegation of specific technical areas and activities to the partners as well
as their coordination, working relationships and collaboration in the
implementation of initiative activities.
ß To assess the administration of the initiative by USAID and the implementing
partners including the status of coordination and communication within their
organizations specifically between headquarters and field operations and host
country governments. The focus should be on initiative management, meeting
deadlines, monitoring, and technical assistance. The team should recommend
adjustments in implementation based on findings and conclusions.
ß To review the resources management of the initiative including use of human,
material and financial resources to achieve IRs.
APPENDICES
71
ß To examine the feasibility of scale-up or sustainability and mechanisms for
ensuring sustainability after external funding ceases.
Key questions to be answered
The evaluation team should answer the following questions and make recommendations
as to how USAID can improve and/or strengthen the activity to increase the likelihood of
achieving objectives/results as defined in these questions. Final revisions to the
evaluation questions will be made within two days of signature of the amendment to the
task order.
Results Framework
Does the results framework (intermediate results or indicators) need to be changed or
adjusted in any way?
What recommendations does the evaluation team have about changes?
Are the partners tracking the right indicators at the local level to measure intermediate
changes that occur before the results are achieved? Are systems in place at the
demonstration sites that can track output or interim outcome measures?
Progress and Achievements
Has the initiative made good progress toward achievement of the IRs and results by
successfully accomplishing specific activities as described in key project documents?
What is the vision or value-added of this initiative?
What is the feasibility of sca1e-up or what if any elements should be or have the
possibility of being sustained?
Has the project had an impact or is it likely to on regional, national, and local policy
Initiative Structure
What are the strengths and weaknesses of the management structure as implemented and
how responsive have the partners been to USAID management, USAID missions, and
each other during the course of Initiative implementation?
To what extent are related structures functioning adequately e.g., TAG? NGO Task
Force? Information Dissemination Task Force? Indicator Task Force? Study Tour
working group? Has the work of these groups moved the initiative forward in its
achievement of the IRs?
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
72
Have different components been coordinated? Has there been coordination among the
partners.
Administration
How effective has the management and oversight of the Initiative been by USAID/LAC?
Has there been coordination with USAID both in the regional bureau and in the missions?
Is there good communication between the headquarters and the field staff or all partner
organizations? Is there good communication between partner staff and host country of
officials?
Have reports, work plans and other products been of adequate quality and submitted to
USAID on time with appropriate levels of information?
Use of Resources
Have the project resources been used appropriately and effectively e.g. staff and other
inputs? Have all planned staff been hired and have staff and job descriptions been
approved by USAID?
Materials and Procedures
1. Data Sources
The LAC HSR Initiative evaluation team will review relevant documentation including
but not limited to the following; The results package, Authorization, Grant to PAHO,
R4s, Indicators Paper, Memorandum of Understanding, Trip Reports, Monthly Reports,
Quarterly Financial Reports, Committee (TAG, NGO Task Force, Information
Dissemination Task Force) Minutes and Reports and other project documents and
publications, etc. Interviews will be conducted with staff from USAID/LAC, USAID/G,
PAHO, PHR, DDM, FPMD, USAID field missions, PAHO field offices, NGOs involved
with the initiative and the MOH.
2. Methods of Data Collection
This will be primarily a process evaluation. The data will be collected through document
review, key informant interviews, e-mail surveys, telephone interviews, field site visits
and group discussions. The LAC HSR Initiative evaluation team will interview USAID,
PAHO, PHR, DDM and FPMD staff and health reform stakeholders. The team will travel
to 3 countries in the LAC region and interview field staff, MOH counterparts, community
members and visit project field sites.
APPENDICES
73
3. Duration and Timing of the Evaluation and LOE
The final schedule will be agreed with the consultants selected prior to initiation of the
assignment; the evaluation process will include:
-Review of project documentation
-Meetings with Initiative staff and US-based stakeholders, final planning of field visits
(in Washington DC)
-Visits to 3 Initiative countries to meet with USAID, PAHO, PHR, DDM, PPMD staff (as
available), representatives of key institutions working on health sector reform.
-Clarifications/consultations with US-based Initiative staff, processing of information and
preparation of draft evaluation report. (Washington DC)
-Presentation of draft evaluation report to LAC HSR Initiative Steering Committee
(Washington DC)
-Steering Committee comments on draft report.
-Preparation of final evaluation report
Team Composition
We anticipate that the evaluation team will consist of three members, each of whom
would be fluent in English and Spanish. The additional skills and expertise listed below
are necessary but may be fulfilled in a different team configuration:
1. A Health Sector Reform Specialist with expertise in the design, implementation,
and/or evaluation of substantial health sector reforms in Latin America and the
Caribbean. The ideal candidate would have a combination of at least seven years
of national and international experience in health sector reform in the region with
an emphasis on the policy process and the management of change, be able to read
and write fluently in both Spanish and English and be familiar with USAID's
policies and priorities in PHN.
2. A Project Management Specialist with a track record of successful management
of complex international (non-engineering) projects with multiple implementing
agencies, including financial management and the development and
implementation of project tracking systems. This person would have at least 5
years of experience overseeing projects with programming of at least $1 million
annually or extensive experience as an international management consultant.
Ideally, this person would have a master's degree in business or public
administration.
3. An International Communications Specialist with expertise in the development
and evaluation of information dissemination strategies, design and production of
print and electronic materials for reaching international audiences and design and
implementation of audience research. This person would have at least 5 years of
experience in international communications, preferably in the LAC region.
Alternatively, the set of tasks related to assessing the project's information
dissemination strategy could be subcontracted to a public opinion firm.
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
74
Funding and Logistical Support
All funding and logistical support for the assessment will be provided through this SO
using a Task Order to the MEDS (Monitoring, Evaluation, and Design Services) activity.
Activities that will be covered include, recruitment of the team members; distribution of
background documentation provided by USAID; arrangements for travel and logistical
support; secretarial and office assistance as needed, support for all personnel and business
expenses related to the evaluation; and assistance as required with editing, printing, and
distributing the final report.
APPENDIX B
HEALTH SECTOR REFORM RESULTS PACKAGE INDICATORS
FEBRUARY 2000
(FROM USAID)
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
76
HEALTH SECTOR REFORM RESULTS PACKAGE INDICATORS,
2/2000
BASELINE and PLANNED VALUES IDENTIFIED
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions.
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: SO #3; Health Sector Reform Results Package: More effective delivery of
sustainable country health sector reforms (designed to increase equitable access to high quality,
efficiently delivered basic health services)
SO 3 INDICATOR 6: Target countries with reform processes that substantially integrate lessons
learned from this results package (methods and tools, information, monitoring, and exchanges).
UNIT OF MEASURE: Number
SOURCE: PAHO Reports using information from PHR,
DDM and FPMD.
INDICATOR DESCRIPTION:
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia, Brazil,
Dominican Republic, Ecuador, El Salvador, Guatemala,
Haiti, Honduras, Jamaica, Mexico, Nicaragua, Paraguay
and Peru.
1
Ecuador, Guatemala, Paraguay, Dominican Republic,
Honduras, and Jamaica.
YEAR PLANNED ACTUAL
1997 NA NA
1998 (B) 0
1999 4 6
1
2000 6
2001 7 (T)
 
1
1
APPENDICES
77
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.1; Health Sector Reform Results Package:
Methodologies and Tools developed, tested and disseminated for analysis, design,
implementation and monitoring of country health sector reforms.
INDICATOR 3.4.1: Methodologies and tools actively used by key actors in 50% or more of
target countries where introduced.
UNIT OF MEASURE: Percent of methodologies and
tools
SOURCE: Each partner agency will report on the
methodologies it introduces (DDM, FPMD, PAHO &
PHR).
INDICATOR DESCRIPTION: The denominator
includes only methodologies and tools that were
introduced by the year prior to measurement.
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia, Brazil,
Dominican Republic, Ecuador, El Salvador, Guatemala,
Haiti, Honduras, Jamaica, Mexico, Nicaragua, Paraguay
and Peru.
1National health accounts are being used in Bolivia,
Ecuador, Guatemala, Mexico, and Nicaragua, but not in
Peru, Dominican Republic, or El Salvador.
YEAR PLANNED ACTUAL
1997 100% 100%
1998 100% 100%
1999 100% 100%1
2000 100%
2001 100% (T)
 
1
1
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
78
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.2; Health Sector Reform Results Package:
Information on health reform efforts and experience gathered and made widely available to
interested parties in LAC countries and to health sector donors.
INDICATOR 3.4.2: Target countries where at least 5 different institutions report receiving
materials through this RP.
UNIT OF MEASURE: Number
SOURCE: PAHO, PHR
INDICATOR DESCRIPTION:
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia, Brazil,
Dominican Republic, Ecuador, El Salvador, Guatemala,
Haiti, Honduras, Jamaica, Mexico, Nicaragua, Paraguay
and Peru.
Partner agencies surveyed a sample of people on their
mailing lists to test for recall of receipt of materials
disseminated.
YEAR PLANNED ACTUAL
1997 --
--
1998 5
5
1Bolivia, Brazil, Dominican Republic, Ecuador, El
Salvador, Mexico, Nicaragua, and Peru.
1999 8
8
1
2000 11
2001 13 (T)
 
1
1
APPENDICES
79
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.3; Health Sector Reform Results Package: Reform
processes and outcomes monitored and feedback provided to countries, donors and other
partners.
INDICATOR 3.4.3a: Regional monitoring system for comparative analysis of health sector
reform institutionalized by PAHO.
UNIT OF MEASURE: Level of development (see
Key)
SOURCE: PAHO Reports
INDICATOR DESCRIPTION:
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia, Brazil,
Dominican Republic, Ecuador, El Salvador, Guatemala,
Haiti, Honduras, Jamaica, Mexico, Nicaragua, Paraguay
and Peru.
Key: 1=System has been initiated.
2=System is functional.
3=System has been institutionalized.
The regional monitoring system, operated by PAHO,
will analyze and report data on principal indicators of
health sector reform process and outcomes to country
program managers, donors, and other partners.
YEAR PLANNED ACTUAL
1997 (B) --
1998 -- 0
1999 1 0
2000 2
2001 3 (T)
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
80
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.3; Health Sector Reform Results Package: Reform
processes and outcomes monitored and feedback provided to countries, donors and other
partners.
INDICATOR 3.4.3b: Target countries that monitor health sector reform using framework
established by PAHO.
UNIT OF MEASURE: Number
SOURCE: PAHO assessment
INDICATOR DESCRIPTION:
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia, Brazil,
Dominican Republic, Ecuador, El Salvador, Guatemala,
Haiti, Honduras, Jamaica, Mexico, Nicaragua, Paraguay
and Peru.
Monitoring health sector reform using the framework
established by PAHO means full application of the
methodology including the entire regional monitoring
framework. PAHO will assess annually whether each
target country is monitoring reform using the regional
framework.
YEAR PLANNED ACTUAL
1997 (B) --
1998 0 0
1999 6 11
2000 10
2001 13 (T)
APPENDICES
81
ADDITIONAL INDICATOR
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.4; Health Sector Reform Results Package: Opportunities
and means to share experience and advice between countries are established.
INDICATOR 3.4.4a:Electronic network users surveyed who report finding network services
useful.
UNIT OF MEASURE: Percent
SOURCE: Each partner agency which operates an
electronic network.
INDICATOR DESCRIPTION: Implementing partners
operating electronic networks will survey users annually
to identify what percentage find them useful.
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia,
Ecuador, Peru, Paraguay, Brazil, Mexico, El Salvador,
Guatemala, Honduras, Nicaragua, Dominican Republic,
Jamaica, and Haiti.
YEAR PLANNED ACTUAL
1997 -- --
1Network of key actors on health reform was not in
operation as of the end of FY 98.
1998 50% 0%1
2
46 of 59 persons surveyed found the Results package's
Web services useful.
1999 50% 78%2
2000 50%
2001 50% (T)
 
1
1
2
2
MIDTERM EVALUATION OF THE LATIN AMERICA
AND THE CARIBBEAN REGIONAL HEALTH SECTOR REFORM INITIATIVE
82
OBJECTIVE: Strategic Objective #3: More effective delivery of selected health services and
policy interventions
APPROVED: 7/29/97; Health Sector Reform Results Package
COUNTRY/ORGANIZATION: LAC/RSD-PHN
RESULT NAME: Intermediate Result 3.4.4; Health Sector Reform Results Package: Opportunities
and means to share experience and advice between countries are established.
INDICATOR 3.4.4b: Participants surveyed who report taking steps based on lessons learned in
workshops and study tours.
UNIT OF MEASURE: Percent
SOURCE: Each partner agency which organizes
workshops and study tours.
INDICATOR DESCRIPTION: Participants in
workshops and study tours will be surveyed in the
following year to identify the percent taking steps based
on the lessons learned in such activities sponsored by
this Results Package.
COMMENTS: Target countries are the USAID
presence countries with PHN objectives: Bolivia,
Ecuador, Peru, Paraguay, Brazil, Mexico, El Salvador,
Guatemala, Honduras, Nicaragua, Dominican Republic,
Jamaica, and Haiti.
YEAR PLANNED ACTUAL
1997 -- --
1No workshops or study tours took place during FY 98. 1998 50% 0%1
1999 50% 91%
2000 50%
2001 50% (T)
 
1
1