Prepared under the Democracy and Governance Analytical Services Indefinite Quantity Contract, #DFD-I-00-04-00229-00
Submitted to: 
USAID/Serbia 
Prepared by: 
JUNE 2014
This publication was produced for review by the United States Agency for
International Development. It was prepared by Democracy International, Inc.
ALBANIA
PERFORMANCE EVALUATION OF THE 
ENABLING EQUITABLE HEALTH 
REFORM PROJECT
FINAL REPORT
Submitted to: 
USAID/Albania
 
Prepared by: 
Tim A. Clary, Ph.D., EMBA, M.S., M.A.
Lindita Çaçi, M.Sc.
Contractor: 
Democracy International, Inc.
7600 Wisconsin Avenue, Suite 1010
Bethesda, MD 20814 
Tel: 301-961-1660 
Email: info@democracyinternational.com
www.democracyinternational.com
PERFORMANCE EVALUATION OF 
EEHR FOR USAID/ALBANIA 
FINAL REPORT
JUNE 2014
The views expressed in this document are the authors' and do not necessarily reflect the views of the 
United States Agency for International Development or the United States Government
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA
TABLE OF CONTENTS
ACRONYMS............................................................................................................................................................................ 1
EXECUTIVE SUMMARY.................................................................................................................................................. 2
BACKGROUND & OVERVIEW ................................................................................................................................. 5
METHODOLOGY............................................................................................................................................................... 8
FINDINGS AND CONCLUSIONS ....................................................................................................................... 10
RECOMMENDATIONS................................................................................................................................................ 21
ANNEX A: STATEMENT OF WORK............................................................................................................... A-1
ANNEX B: EVALUATION QUESTION MATRIX..................................................................................... B-1
ANNEX C: TEAM COMPOSITION................................................................................................................... C-1
ANNEX D: DOCUMENTS REVIEWED.......................................................................................................... D-1
ANNEX E: INDIVIDUALS CONTACTED...................................................................................................... E-1
ANNEX F: STATEMENTS OF CONFLICT OF INTEREST ................................................................... F-1
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 1
ACRONYMS
ADHS Albanian Demographic and Health Survey
BCC Behavior change and communication
COP Chief of Party
EEHR Enabling Equitable Health Reform (USAID-funded project)
GoA Government of Albania 
HA Health Authorities
HCs Health Centers
HII Health Insurance Institute
HIS Health/Hospital Information System
HRISG Health Reform Implementation Support Group
INSTAT National Institute of Statistics
IPH Institute of Public Health
KII Key informant interview
M&E Monitoring and evaluation
MCH Maternal and child health
MoH Ministry of Health
MoLG&D Ministry of Local Government and Decentralization
NCCE National Center of Continuing Education
NCQSA National Centre of Quality, Safety and Accreditation
PBMP Performance Based Management Plan
PHC Primary Health Care
PHRplus Partners for Health Reform plus (USAID-funded project)
PPC Progress per Civilizim
SOW Scope of Work
TA Technical assistance
TFR Total fertility rate
TQM Total Quality Management 
USAID United States Agency for International Development
USG US Government
WHO World Health Organization
 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 2
EXECUTIVE SUMMARY
Although there have been many barriers to health sector reforms in Albania, the government has charted a 
course that includes a single payer system, operated by the Health Insurance Institute (HII), and the separa￾tion of the provision of health services from the financing of health services, through HII. By 1995, HII had 
become a major funder of doctor salaries as well as drugs for primary health care (PHC) services. Beginning 
in 1998, as part of the government’s decentralization initiative, the Ministry of Local Government and De￾centralization channeled budgets for operating and maintenance costs of PHC facilities (previously funded 
by the Ministry of Health) through block grants to local governments, which then determined how much 
was allocated to PHCs.
Steady progress has been made in health sector reform during the past 20 years, though, many challenges 
remain. These include: lack of political consensus around the development and/or implementation of poli￾cies; chronic underfunding of the health sector; difficulty in enforcing universal participation in the HII system; 
corruption in the form of “informal” out-of-pocket fees and procurement of commodities; frequently chang￾ing institutional leadership with many political appointees, including hospital directors and, consequently, 
hospital staff; the greater complexity of the hospital sector compared with PHC; and other many other ex￾ogenous factors.
Beginning in 2000, USAID planned and funded a series of technical assistance (TA) projects in the health 
sector, including: a) Partners for Health Reform Plus (PHR plus) which provided technical assistance to the 
Government of Albania (GoA) from 2001 to 2005 in the design and implementation of a subset of the 
GoA’s health sector reform strategy plans, focused on strengthening the PHC sector; and, b) PRO Shëndetit
which developed a basic package of services at PHC centers that was adopted nationwide, and now serves 
as a basis for funding of PHC by the HII. During this period, USAID also supported family planning initiatives, 
including behavior change communication (BCC) through the C-Change project and improvement of post￾partum and post-abortion care through the ACCESS-FP project. 
Both prior to and following the completion of the PRO Shëndetit project, USAID worked on the design of 
another flagship project for the health sector. The Health Sector Review that supported the design of the 
new project observed that returns on investment in the Albanian PHC system were beginning to decline for 
USAID, and that another approach, would be: 
“to target USAID assistance … on efforts to assist the GoA implement its own health reform agen￾da. … That ambitious agenda includes a wide array of strategies, policies, laws and regulations 
which, if enacted and implemented, would affect significant and lasting improvement in the quality 
and responsiveness of the Albanian health care delivery system. Yet, despite years of discussion, de￾bate and donor exhortation, virtually none of those nominal reforms have been finalized and im￾plemented.”
This planning work came to fruition in the Enabling Equitable Health Reform (EEHR) Project, awarded to 
Abt Associates in September 2010. Phase I of the contract was allotted up to two years, and was envi￾sioned as the period “to help narrow the health reform agenda to a feasible set of priority actions that will 
likely have the greatest impact nationally.” Based on this feasible set of priority actions, the main objective of 
Phase II was “to use the produced tools and mechanisms in designated pilot region(s) field testing this way 
the defined feasible set of priority reform actions and b) to produce evidence to guide national implementa￾tion of health reforms”. However, a joint decision was made by USAID and EEHR staff to shift the focus of 
Phase II to practical interventions that could be piloted as a set of reforms in the hospital sector rather than 
the broader regional scope with subsequent evidence limited to a sector rather than an agenda of overall 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 3
national health reform. Beginning in early 2012, EEHR began to implement a set of activities focused on the 
hospital sector in three pilot hospitals that, in part, paralleled earlier strategies employed in the PHC sector. 
USAID commissioned a performance evaluation after slightly more than three years of EEHR being imple￾mented. To fully complete the analysis for this performance review, two evaluation teams were fielded. The 
first team conducted fieldwork in Albania from November 4 to November 19, 2013. The second team
conducted additional key informant interviews (KIIs) from March 31 – April 4, 2014. Prior to beginning 
fieldwork, the evaluation teams reviewed a variety of background materials, and then following the initial in￾country meetings with USAID and EEHR staff, the first evaluation team held in-depth interviews with key 
stakeholders at the Ministry of Health; Health Insurance Institute; National Center of Quality, Safety and Ac￾creditation of Health Institutions; National Center of Continuing Education; World Health Organization 
(WHO); and the World Bank. The team made site visits to the three pilot hospitals and held 19 focus 
group discussions or interviews with 73 stakeholders from the hospitals as well as regional offices of the 
Public Health Institute and HII. The second evaluation team focused on interviewing USAID staff, EEHR pro￾ject staff, and members of the Health Reform Implementation Support Group (HRISG).
The teams based their data collection and development of subsequent semi-structured interview guides on 
the following five evaluation questions, per the SOW:
1) To what extent have EEHR activities resulted in removing key barrier/obstacles to the health re￾form process?
2) How successful was the project in producing tools and mechanisms that are tested as successful 
in pilot regional hospitals?
3) Based on the perception of outside stakeholders, do the selected regional hospitals have better 
performance as a result of the approaches and implemented activities, than they otherwise would 
have had?
4) How successful was the contractor in designing and implementing the small grants program?
5) What were the challenges (internal and external) faced by the program and lessons learned?
Similar to most evaluations that are conducted well after activities have been implemented, this evaluation 
may be limited by recall bias. Additionally, interviewees may self-select by either making themselves available 
for interviews or in the amount of time they allot for the interview. Further, the main approach for this eval￾uation was qualitative and the full capture and analysis of qualitative data is difficult. Finally, it should be not￾ed that by the time that the second evaluation team was fielded, a decision had been made to end the pro￾ject earlier than originally anticipated. This knowledge may have influenced the final group of interviewees 
(primarily USAID and EEHR staff) in their responses to the evaluation questions. It is believed by the evalua￾tion teams, though, that the robustness of the various methods employed will allow the team to present 
solid findings, conclusions, and recommendations.
The main findings and conclusions from this evaluation are that EEHR has had limited impact in removing 
key barriers/obstacles in the health reform process and most of the project’s success should be measured by 
the activities done at the pilot hospital level. However, the sustainability of the management and technical 
changes done at the pilots is questionable given the potential turnover in hospital leadership (hospital direc￾tors are political appointees). While almost all key informants were unanimous in that the pilot hospitals 
were the project’s key success, the findings and conclusions for the small grants program point to it being 
the weakest component of EEHR. Only approximately 10% of the original budgeted amount of $400,000 
will be expended by the time of the project’s closure. Compounding the technical challenges that the pro￾ject faced were additional management challenges with all stakeholders. During the life of the project, there 
were six individuals who served either as a temporary, acting or permanent EEHR Chief of Party, four Minis￾ters of Health, and an uncertainty on the part of USAID, per key informant interviews, whether it had a 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 4
shared vision of EEHR with project staff and, similarly, how the project’s success should be measured. Finally, 
it should be noted, that to definitively determine the answers to some of the evaluation’s questions, primari￾ly those aimed at better understanding the outcomes at the pilot hospitals, additional methods should be 
utilized, including facility-based and patient/client satisfaction surveys.
Currently, there is a World Bank project in its initial phase of feasibility and scoping studies and, if possible, 
USAID and EEHR staff should collaborate with the World Bank on the design of its new project and con￾sider how initiatives undertaken or planned during Phase II of EEHR might be sustained as potential catalysts 
and models for reform in Albania’s health sector. Further, any relevant research which was conducted as 
part of Phase 1 of EEHR should be shared freely with the World Bank. During the few remaining months of 
the project, no new programmatic initiatives should be undertaken. Ongoing initiatives should be reinforced 
so that they are more likely be sustained. New initiatives, especially those for which plans have been made, 
should be suggested for the World Bank health sector project design. USAID might also consider avenues 
for sustaining the local capacity for technical assistance that has been developed which itself builds on previ￾ous USAID projects, through supporting the possible creation of a local NGO that houses the existing ca￾pacity, or through other mechanisms that might sustain the capacity in some institutional form. Finally, while 
the Project was noted for building the local counterpart capacity in monitoring and evaluation, its internal 
level of M&E requirements was set quite low. Had additional resources been dedicated to measurements at 
the higher outcome level, perhaps, stronger findings conclusions, and recommendations would be forthcom￾ing from any external evaluations conducted.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 5
BACKGROUND & OVERVIEW 
In the early- to mid-1990s, Albania entered a new phase of political and social change. It transitioned from a 
totalitarian to a democratic system and shifted gradually to a free market economy, a process that contrib￾uted to changes in various demographic and health characteristics throughout the country. During the next 
two decades, the total fertility rate (TFR) fell to 1.6 births per woman, a rate consistent with surrounding 
countries. While the contraceptive prevalence rate rose to 69 percent, the use of modern methods re￾mained at only 11%. Antenatal and delivery care by a trained provider achieved more than 98 percent cov￾erage and 95 percent of children aged 18-29 months were fully vaccinated. Other important child survival 
indicators improved, including nutritional status and low levels of anemia for both children and women 
(ADHS 2010).
Albania’s strategies in developing its health sector have aligned more with those of Western Europe than 
with the United States, the one Western nation with a health system run primarily through the private sec￾tor. Although there have been many barriers to health sector reforms in Albania, the government has chart￾ed a course that includes a single payer system, operated by the Health Insurance Institute (HII), and the 
separation of the provision of health services from the financing of health services, through HII. 
During the 1990s, the Albanian health system went through organizational and budgetary changes that shift￾ed important roles to relatively new agencies. By 1995, HII had become a major funder of doctor salaries as 
well as drugs for primary health care (PHC) services. Beginning in 1998, as part of the government’s decen￾tralization initiative, the Ministry of Local Government and Decentralization (MoLG&D) channeled budgets 
for operating and maintenance costs of PHC facilities (previously funded by the Ministry of Health) through 
block grants to local governments, which then determined how much was allocated to PHCs (Fairbank and 
Gaumer, 2003).
Although steady progress has been made in health sector reform during the past 20 years, particularly in 
primary health care, many challenges remain. These include: lack of political consensus around the develop￾ment and/or implementation of policies; chronic underfunding of the health sector; difficulty in enforcing 
universal participation in the HII system (especially within the informal labor market); corruption in the form 
of “informal” out-of-pocket fees and procurement of commodities; frequently changing institutional leader￾ship with many political appointees, including hospital directors and, consequently, hospital staff; the greater 
complexity of the hospital sector compared with PHC; and other many other exogenous factors.
USAID SUPPORT TO THE ALBANIAN HEALTH SECTOR
While these changes in Albania’s health sector were taking place, USAID and other donors partnered with 
the Government of Albania (GoA) to rehabilitate and re-equip many health facilities that had been damaged 
during the civil unrest earlier in the 1990s. Beginning in 2000, USAID planned and funded a series of tech￾nical assistance (TA) projects in the health sector.
The USAID-funded Partners for Health Reform Plus (PHR plus) provided technical assistance to the Gov￾ernment of Albania (GoA) from 2001 to 2005 in the design and implementation of a subset of the GoA’s 
health sector reform strategy plans, focused on strengthening the PHC sector. An integrated PHC service 
delivery model was designed, implemented, and evaluated in two districts to inform national health policy
and to be replicated on a wider scale. The model integrated five major components: (1) family medicine 
training for PHC providers; (2) a facility-based health information system; (3) service-delivery reorganization 
and quality improvement; (4) financing reforms; and (5) community involvement. The model addressed the 
low quality of PHC services, lack of any data on PHC patients or costs, the bypassing of PHC clinics for spe￾cialty polyclinics or hospitals, fragmented financing and management of PHC, and the absence of community 
involvement (Cook et al., 2005).
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 6
This earlier work in PHC was built upon and reinforced by the USAID flagship project PRO Shëndetit. PRO 
Shëndetit provided a foundation for the World Bank’s “Albania Health Sector Modernization” Project, 
which required GoA adoption of several tools and interventions developed under the PRO Shëndetit pro￾ject as pre-conditions for World Bank assistance. PRO Shëndetit developed a basic package of services at 
PHC centers that was adopted nationwide, and now serves as a basis for funding of PHC by the HII. This
approach provided a model for the MoH and HII to establish a comparable “basic services package” of care 
for hospitals. PRO Shëndetit also worked to turn GoA Health Centers (HCs) into autonomous facilities re￾sponsible for the delivery of PHC services to the population based on contracts executed between the HCs 
and the HII. PRO Shëndetit also developed and installed a pilot Health Information System (HIS) in five re￾gions that is now in use nationwide, and it serves as the primary basis for HII reimbursement to HCs and 
pharmacies for services and drugs, respectively. During this period, USAID also supported family planning 
initiatives, including behavior change communication (BCC) through the C-Change project, managed out of 
Washington, D.C. (USAID/W), and improvement of post-partum and post-abortion care through the AC￾CESS-FP project, also managed by USAID/W (Bowers and Johnson, 2010). 
Both prior to and following the completion of the PRO Shëndetit project which ended in September 2009, 
USAID worked on the design of another flagship project for the health sector. The Health Sector Review 
that supported the design of the new project observed that returns on investment in the Albanian PHC 
system were beginning to decline for USAID, and that another approach, which the review proposed and 
was “endorsed by the leadership of the MoH and HII”, would be: 
“to target USAID assistance … on efforts to assist the GoA implement its own health reform agen￾da. … That ambitious agenda includes a wide array of strategies, policies, laws and regulations 
which, if enacted and implemented, would affect significant and lasting improvement in the quality 
and responsiveness of the Albanian health care delivery system. Yet, despite years of discussion, de￾bate and donor exhortation, virtually none of those nominal reforms have been finalized and im￾plemented.” 
The review proposed that the USAID assistance strategy take on the tasks left undone by the GoA (Bowers 
and Johnson, 2010). This planning work came to fruition in the Enabling Equitable Health Reform (EEHR) 
Project, awarded to Abt Associates in September 2010. The original contract noted some of the challenges 
that EEHR would face: 
“The Contractor will convert ambitious but unrealistic and unfunded reform mandates into action 
plans supported by key stakeholders.… The Contractor shall provide technical assistance to over￾come a number of complex factors that have hampered progress including: institutional and political 
rivalries, lack of transparency and accountability, diffusion of responsibility, lack of reliable data, and 
relative absence of evidenced-based integration of lessons learned in strategic and action plan￾ning.…The Contractor shall provide technical assistance to promote inter-agency cooperation in 
strategic and operational planning in order to advance the reform agenda. The communication bar￾riers between institutional silos will be lowered, enabling collective efforts to address priorities, iden￾tify sequential actions with timetables, resources, and ensure participation of stakeholders” (USAID, 
2010). 
Phase I of the contract was allotted up to two years, and was envisioned as the period “to help narrow the 
health reform agenda to a feasible set of priority actions that will likely have the greatest impact nationally.” 
Based on this feasible set of priority actions, the main objective of Phase II was “to use the produced tools 
and mechanisms in designated pilot region(s) field testing this way the defined feasible set of priority reform 
actions and b) to produce evidence to guide national implementation of health reforms”. However, a joint 
decision was made by USAID and EEHR staff to shift the focus of Phase II to practical interventions that 
could be piloted as a set of reforms in the hospital sector rather than the broader regional scope with sub￾sequent evidence limited to a sector rather than an agenda of overall national health reform. Beginning in 
early 2012, EEHR began to implement a set of activities focused on the hospital sector in three pilot hospi-
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 7
tals that, in part, paralleled earlier strategies employed in the PHC sector. The three hospitals chosen 
through a consultative process and assessment were Queen Geraldine Maternity Hospital (Tirana), Korca 
Regional Hospital, and Lezha Regional Hospital.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 8
METHODOLOGY
To fully complete the analysis for this performance review, two evaluation teams were fielded. The first 
team conducted fieldwork in Albania from November 4 to November 19, 2013. The second team con￾ducted additional key informant interviews (KIIs) from March 31 to April 4, 2014. During both periods of 
fieldwork, the teams held in-briefs and out-briefs with USAID with the first team having additional meetings 
with USAID staff at approximately the mid-point of the field work. 
Per the Task Order’s Statement of Work (SOW), a detailed work plan was submitted to USAID as part of 
the original proposal. The work plan included the evaluation’s methodology and activities, an illustrative list 
of individuals and organizations the evaluation teams planned to interview, along with a preliminary schedule. 
Prior to beginning fieldwork, the evaluation teams reviewed a variety of background materials, including re￾ports on the Albanian health sector, previous USAID projects in the sector, project documents, and other 
documents produced by EEHR. 
Following the initial meetings with USAID and EEHR staff, the first evaluation team held in-depth interviews
with 19 key stakeholders at the Ministry of Health; Health Insurance Institute; National Center of Quality, 
Safety and Accreditation of Health Institutions; National Center of Continuing Education; World Health Or￾ganization (WHO); and the World Bank. The team made site visits to the three pilot hospitals and held 19 
focus group discussions or interviews with 73 stakeholders from the hospitals as well as regional offices of 
the Public Health Institute and HII. 
A full listing of focus group members is given in Annex E: Individuals Contacted under Korca Regional Hos￾pital, Lezha Regional Hospital, and Queen Geraldine Maternity Hospital. In general, focus groups were 
comprised of hospital senior administration, department heads, physicians and other care providers, and 
representatives from relevant departments (human resources, environmental services, infection control, and 
monitoring and evaluation). The evaluation team’s local member also participated in one of the teleconfer￾ence sessions that the project organized between the three hospitals. The second evaluation team focused 
on interviewing four members of USAID staff, EEHR project staff, and members of the Health Reform Im￾plementation Support Group (HRISG).
The teams based their data collection and development of subsequent semi-structured interview guides1 on 
the following five evaluation questions, per the SOW:
1) To what extent have EEHR activities resulted in removing key barrier/obstacles to the health re￾form process?
2) How successful was the project in producing tools and mechanisms that are tested as successful 
in pilot regional hospitals?
3) Based on the perception of outside stakeholders, do the selected regional hospitals have better 
performance as a result of the approaches and implemented activities, than they otherwise would 
have had?
4) How successful was the contractor in designing and implementing the small grants program?
5) What were the challenges (internal and external) faced by the program and lessons learned?
 
1 The original question matrix from the evaluation work plan is included in Annex B.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 9
Both teams were careful not to prompt or lead interviewees to specific opinions or conclusions. Though 
interviews were structured around the five evaluation questions, if subjects raised an issue or topic that was 
outside the interview guide, the teams would probe the subject matter further and during the interviews 
would ask if there were any particular topics not covered during structured portion. For external interviews, 
no USAID or EEHR staff members were present for any of the interviews or focus group discussions. 
USAID and EEHR staff were interviewed individually to avoid any influence that might have occurred due to 
lines of reporting. 
The evaluation teams cross-checked what was heard, read, or observed to ensure consistency and accuracy 
of fact finding. The teams considered preliminary findings and, when needed, requested additional infor￾mation or confirmation of interpretations. As with any primarily qualitative evaluation, the teams’ methodol￾ogy relied on the experience and training of evaluation team members for the analysis and interpretation of 
findings. After all data collection was complete, the teams reviewed and synthesized their findings, drew 
conclusions, and made recommendations based on the findings and conclusions. 
LIMITATIONS 
Similar to most evaluations that are conducted well after activities have been implemented, this evaluation 
may be limited by recall bias. In other words, EEHR was awarded in September 2010 and began activities 
shortly thereafter. The first team was fielded more than three years later and the second team began inter￾views approximately three and a half years after the project’s start. This lag may influence how well inter￾viewees can recall certain events and activities and result in either a positive or negative bias. Additionally, 
interviewees may self-select by either making themselves available for interviews or in the amount of time 
they allot for the interview. Persons with stronger vested interests in the results of the evaluation (either 
negatively or positively) may spend more time with the interviewers to ensure that the evaluation results 
skew in their favor. As noted previously, the main approach for this evaluation was qualitative and the full 
capture and analysis of qualitative data is difficult. Evaluation team members may fail to record important 
data for a variety of reasons and, subsequently, that data will not be part of the findings, conclusions, or rec￾ommendations. Finally, it should be noted that by the time that the second evaluation team was fielded, a 
decision had been made to end the project earlier than originally anticipated. This knowledge may have in￾fluenced the final group of interviewees (primarily USAID and EEHR staff) in their responses to the evalua￾tion questions. Some of the limitations noted have been mitigated by the number and scope of interviews, 
focus groups, and use of available quantitative data, when available. It is believed by the evaluation teams 
that the robustness of these various methods will allow the team to present solid findings, conclusions, and 
recommendations.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 10
FINDINGS AND CONCLUSIONS
EVALUATION QUESTION 1: To what extent have EEHR activities resulted in 
removing key barriers/obstacles to the health reform process?
FINDINGS 
1) Staff from EEHR and USAID interviewed as key informants were unanimous that EEHR activities 
had and would have limited effect in removing key barriers/obstacles in the national health reform 
process; though, the activities implemented within the pilot hospitals could serve as focal points for 
technical and managerial lessons learned in moving the reform process forward within the hospital 
sector. However, none of the KIIs were certain whether the changes instituted at the hospital level 
would be sustainable after the project’s end date. The most cited reason for doubting the sustaina￾bility of the hospital-level changes was because hospital directors are political appointees and there 
has been and, most likely, will continue to be frequent turnover in their positions. Thus, the proba￾bility that a succeeding director would support his or her predecessor’s position on reform was un￾known.
2) The Health Reform Implementation Support Group, a key body supported by EEHR and a poten￾tial structure which could continue to move the reform agenda forward, currently has high member 
turnover due to approximately 70% of its membership being GoA officials and subject to replace￾ment by successive governments. Further, due to membership turnover and some members with 
little prior experience in the health sector, the capacity of HRISG members to address barriers and 
obstacles in the health reform process varies substantially.
3) EEHR did reach its Year 2 targets, per the Performance Based Monitoring Plan (PBMP), of: a) having 
two of the HRISG decisions enacted; b) producing a policy brief for the MoH based on HRISG rec￾ommendations; and, c) having three decisions, policies, and plans drafted with EEHR assistance. 
4) By the second quarter of its third year of implementation (March 2013), EEHR had: a) the Hospital 
Board Bylaws approved for recommendation to the Minister of Health; b) begun preparations for 
the national roll-out of the human resources policies, tools for job descriptions, as well as, the new 
employee orientation; and, c) had approved the National Guidelines for Incident Reporting. 
5) Responses by key informants regarding the usefulness of the research documents produced during 
Year 1 / Phase 1 in moving the health reform process forward were mixed. Some key informants 
believed that they helped focus EEHR project activities and generated new information while other 
respondents believed some of the research was repeating work previously completed. This “re￾peated” work included the general governance review, some of the monitoring and evaluation stud￾ies, and some of the special studies focused on identifying the socio-economic, health, and demo￾graphic characteristics of proposed project beneficiaries.
6) More than three-quarters of the EEHR, USAID and other key informants stated that the sharpening 
of the focus from the larger-scale goals as stated in the original SOW within the contract to a more 
specific set of objectives and activities was needed by the end of Year 1. This was in order to pro￾duce more tangible results (versus on-going formative research) to ensure constituency buy-in from 
all stakeholders, as well as, provide evidence for potential expansion of pilot efforts. It also necessi￾tated some re-working of the Results Framework and PBMP. Specifically, as can be seen in Figures 1 
(Year 1 Results Framework) and Figure 2 (Year 2 Results Framework) below, many of the lower 
level expected results have been reformulated to focus specifically on the hospital sector while the 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 11
third “intermediate result” (see far right branch) was rewritten to highlight the role of non-state ac￾tors participation in supporting the project’s overall goal.
Figure 1: EEHR Results Framework (Year 1 Annual Report) 
Figure 2: EEHR Results Framework (Year 2 Annual Report) 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 12
7) The project was able to at least partially address gender considerations both through the overall 
work it did at the three pilot hospitals, as well as, establishing working groups within the three pilot 
sites as shown in Table 1. 
Table 1: Sex disaggregation of pilot hospital working groups and employees 
Working Groups Employees
Females Males Females Males 
Korca Regional 
Hospital
27 21 402 186
Lezha Regional 
Hospital
29 20 211 64
Tirana Maternity 29 11 322 50
Totals / Percent 85 (62%) 52 (38%) 935 (76%) 300 (24%)
 
8) Two consultants engaged in the development of the EEHR tools also worked on the World Bank 
health sector project, which should provide continuity between donor agency projects on one spe￾cific aspect of the health sector.
9) Per KIIs, all three pilot hospitals are sharing their experience and lessons learned through telecon￾ference and live meetings arranged with support from the EEHR project. 
CONCLUSIONS
EEHR has had a limited effect in removing key barriers/obstacles in the health reform process. The project 
was originally envisioned to have a much broader impact as described by the following statement in the 
contract SOW: 
“Specifically, the purpose of this project is to provide access to health services for the poor 
by a) helping remove the existing barriers and constraints to the reforms at the national 
level and b) field testing approaches and tools that define a feasible set of implementable 
reforms…”. 
Based on the key informant interviews conducted, these goals were more narrowly focused near the end of 
the first year of project implementation when both USAID and EEHR recognized that the original goals of 
large-scale national health reform might be too ambitious to be accomplished during the life of the project.
Thus, while Phase 2, per the original SOW within the contract, did envision a narrowing of objectives, the 
focus on pilot hospitals was a more indirect approach to increasing access to health services for the poor 
and removing constraints to reforms at the national level.
Further, it is unclear whether any of the piloting activities (up to 11 different activities given in Question 2 
below) done at the hospital level will be sustainable or rolled-out. Because of changes in GoA counterparts 
which effected individuals from the Minister of Health to the hospital director level, it is difficult to decisively 
conclude that there has been any true “country ownership” of the process. This was also reflected in the KII 
results in which no respondent could give a definitive answer about the project’s sustainability. 
Additionally, given changes in Ministry of Health leadership and varying levels of support for the project’s 
health reform goals by GoA officials, a joint decision, according to the KIIs, was made to pilot tools and 
mechanisms at the regional/hospital level to examine whether these changes might serve as a broader cata￾lyst of change similar to previous projects which had worked at the primary health care level. Per the Year 2 
Annual Report PMP, as well as, the most recent Quarter Report (Q2, Year 3, March 2013) PBMP provided 
to the evaluation team, the project appeared to be on track in terms of meeting output level targets.
Finally, EEHR made considerable progress in addressing gender considerations as given in the contract’s 
SOW which stated: 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 13
“To the greatest extent possible, the Contractor shall seek to include both men and wom￾en in all aspects of this program including participation in leadership in [e.g., meetings, train￾ings, etc.].” 
As noted in Table 1 above, the majority of individuals with whom the project interacted at the hospital level, 
either via working groups or more generalized training were, indeed, female. 
EVALUATION QUESTION 2: How successful was the project in producing 
tools and mechanisms that are successful in tests at pilot regional hospitals? 
FINDINGS
1) Implementation of the various tools and mechanisms did not fully begin until April 2012; thus, it is 
likely too soon to fully assess whether they were successful in being fully adopted. Additionally, to 
fully confirm whether the tools and mechanisms were successful, additional information would need 
to be gathered via a client/patient satisfaction survey to ascertain whether the practices had any im￾pact on the experience of the end-user of hospital services. While EEHR’s PBMP and the hospital 
composite indicator contain several indicators to measure improved management practices within 
the pilot hospitals, the project did not include metrics to assess client/patient satisfaction with the 
new tools, mechanisms, and practices being implemented. Hospital staff feedback via focus groups 
about some of the tools as given in 6a (Improved human resources management) and 6c (Incident 
reporting) below point to improvements in, at least, management practices around patient care and 
a strengthened understanding of roles and responsibilities.
2) The process undertaken to design and implement the tools and mechanisms began with analysis 
and consultant reports developed by project staff and/or consultants, and culminated in customized 
operational tools (e.g., job descriptions, guidelines, manuals, etc.) and analysis (e.g., M&E and incident 
reporting) developed by local counterparts, with support from the project’s staff and consultants. 
The project helped train members of these groups and engaged them in a learn-by-doing process 
to develop tools and mechanisms uniquely designed for their hospital setting, considering what was 
required and drawing on models of tools and mechanisms that had been developed for and used in 
other country contexts; thus, avoiding the duplication of efforts.
3) By the end of Year 2 of implementation, EEHR had produced four tools and/or mechanism exceed￾ing its target of two, had engaged 81 persons at the pilot hospitals to implement proposed inter￾ventions exceeding its target of 15, and had provided training to 128 persons exceeding its target of 
30. All of these targets, likewise, appear to have continued to have been exceeded in terms of ac￾tual results by the end of the second quarter in the third year of implementation (March 31, 2013). 
That these targets were greatly exceeded despite the on-going variations in project and MoH lead￾ership can either be interpreted as a need to better set (higher) project targets or that at the out￾put level project staff were able to continue implementation even with leadership challenges.
4) EEHR staff were unanimous in asserting that this part of the project was perceived as quite success￾ful. Specifically, EEHR staff noted that the greatest achievement was a change in how hospital staff 
are working in teams, have more autonomy to make decisions, and that the silos between depart￾ments have been lowered. They also cited the improvement in environmental services (hospital 
cleanliness) which should lower nosocomial infections.
5) Likewise, the approximately 100 stakeholders interviewed either individually or in focus groups 
were nearly unanimous in indicating that this component was successful. A large number of stake￾holders cited specifically the strengthening of M&E capacity and its ability to assist in lowering com￾munication barriers between institutional silos in a constructive and strategic manner to encourage 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 14
evidence-based decision-making as a key outcome. During focus group sessions, stakeholders often 
stressed that the “mentality” of the staff or the culture within the pilot hospitals was changing. 
6) Specific tools and mechanisms produced for piloting at the regional hospitals included:
a. Improved human resource management: This involved the development of job descrip￾tions, a system for performance review, and the introduction and utilization of identity 
badges. In some cases there were existing job descriptions for staff, but they were 
inaccurate. Having job descriptions was a legal requirement for all hospitals, but 
the requirement was theoretical - i.e., on paper but not enforced or implemented 
in practice. Staff members whose jobs were to be described were involved in the 
process, sometimes as the primary authors of the descriptions and sometimes 
through a review process. More than 70 job descriptions were developed at Kor￾ca Hospital, which had the largest number of staff (approximately 600). In parallel 
with the development or refinement of job descriptions, the project helped staff 
develop a system of performance review. This had been implemented in one re￾gional hospital and was about to be implemented in the other at the time of the 
evaluation. 
b. A training program in support of all interventions: More than 2,000 credit hours of training 
were provided to 306 people over a period of 12 months, after which the project was fully 
launched into the implementation of the selected mechanisms and tools beginning in mid￾2012. As shown in Table 2 below, almost all of the training was multidisciplinary, with phy￾sicians, nurses, and administrators involved.
Table 2: EEHR-Supported Trainings 
c. The introduction of incident reporting: Incident reporting in hospitals is a variant of total 
quality management (TQM) in that the staff took ownership for identifying and analyzing 
the real or potential problems and developing solutions to those problems. Incident report￾ing was a formal obligation of the hospitals under MOH regulations, but it was with the 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 15
help of EEHR that the requirement was implemented. The form introduced for reporting 
incidents was approved by the MoH, and the MoH has distributed it to all hospitals and 
made it a requirement for hospitals throughout Albania. The focus group members de￾scribed how they had to break down resistance from the staff because of the fear 
that reporting incidents (or potential incidents) generated and the tendency to feel 
that reporting incidents would result in blame or criticism of staff members. Ac￾cording to focus group members, the number of incidents reported monthly had 
been rising, indicating that the fear of reporting was going down. 
d. Better space allocation: The project engaged an architect with expertise in hospitals to re￾view the current configuration of the two regional hospitals and make recommendations 
for appropriate changes (Hoey 2012). This work resulted in plans to make changes in such 
things as consolidation of the storage areas for pharmaceuticals and creating one or more 
reception areas. For example, Lezha Hospital had four small pharmaceutical warehouses, 
which have now been consolidated into one, which allows more efficient management and 
the reduction of “leakage” of products. 
e. Improving visitor management: This initiative encompassed behavior modification, not only 
for staff, but for the visitors from the general public. The reception areas were designed to 
provide an introduction or orientation for both patients being admitted and visitors, and a 
channel of communication with the public where people could be informed about visiting 
hours. In Lezha Hospital, visiting hours were established; visitors were informed about the 
new rules, and congestion had gradually been controlled or stopped. In the Queen Gerald￾ina Maternity Hospital, magnetic cards are being provided to visitors and the hospital is not 
only limiting the visiting hours but the number of visitors allowed at any given time. The 
evaluation team could observe the difference in the Lezha Hospital because the 
ambulatory section of the hospital was still heavily congested, as visiting hours 
did not apply there.
f. Environmental Services and Infection Control: The pilot hospitals were each divided into 
three zones: high-risk for infection control, medium-risk, and low-risk. Different degrees of 
attention or different procedures for infection control were applied in the three different 
zones. A new human resources department for Environmental Services was created, and all 
cleaning staff members were assigned to this department rather than being assigned to a 
particular ward. Manuals were developed and staff members were trained in new proce￾dures, from how to clean a toilet properly to how to dispose of biological or other high-risk 
waste. All aspects of this intervention were seen by hospital staff as very important in con￾trolling nosocomial infection within the hospitals, in reducing other problems related to the 
hospital environment, and in giving cleaning staff a new sense of professionalism. However, 
there were also some limitations of funding for basic equipment and cleaning supplies that 
were practical barriers to full implementation. 
g. Developing better public relations: EEHR supported work in customer and public relations. 
In addition to more traditional training, staff from the hospitals received training directly in 
the hotel industry, where the concept of customer relations is most relevant to hospitals. 
Hotel managers who provided the training came to visit the hospitals so as to have a better 
idea of what would be most relevant for hospital staff. Further, Lezha Hospital has created 
an informal board with 15 representatives from different organizations and districts in 
Lezha. This “Citizens’ Advisory Panel” may be able to both improve public relations and al￾so build the kind of platform that would support strong and active community involvement 
on a future hospital board. The hospital’s website had also been made more functional with 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 16
support from EEHR, and the Lezha Hospital staff organized a fundraising event, which is not 
a normal practice in the MoH system. 
h. Cost accounting and payment mechanisms: Although the hospitals have been involved with 
this aspect of EEHR’s work, the discussion of this initiative and its ownership of it came 
primarily from staff working with the Health Insurance Institute. As with other interventions, 
the feedback from HII stakeholders was supported by EEHR documentation (Kenny 2013, 
three documents). If the project had proceeded further with this activity, such as defining 
and reaching formal agreement on a package of services to be reimbursed from HII and the 
payment mechanism to be utilized, then linking the services and costing via a Health Infor￾mation System, could have led to improved performance in this area. 
i. Outsourcing of hotel and food services: Both of the regional hospitals had outsourced 
laundry services, and both had done preliminary analysis and are planning to outsource 
meal services. Tirana’s Queen Geraldina Maternity Hospital had outsourced both laundry 
and meal services and the hospital staff was pleased with the quality and cost of the out￾sourced services in comparison with the prior arrangement. 
j. Patient Medication Management and Pharmaceutical Supply: The mini-warehouses for 
pharmaceuticals inside Lezha Hospital were consolidated and record-keeping for pharma￾ceuticals prescribed and dispensed to patients was introduced. EEHR has recently complet￾ed a technical report that analyzes the situation and provides clear recommendations that 
range from the consolidation and enforcement of treatment protocols in the hospitals, to 
adherence to regulations, to improved stock and inventory management in the hospitals, 
and finally to the financing and procurement of pharmaceuticals by HII (Li Bassi, 2013). 
k. Monitoring and Evaluation (M&E): M&E core working groups were created with EEHR sup￾port and the M&E Directorate within the MoH can now provide analysis and evidence that 
can be used to prioritize health sector efforts and better inform policy formulation. M&E 
reports are shared with all institutions, not just the MOH. In terms of the pilot hospitals, 
they are collaborating with the regional offices of the Public Health Institute and the Health 
Insurance Institute to improve and strengthen monitoring and evaluation at the regional 
level. The hospital-level M&E working groups came together to: (1) clarify and better define 
for their reporting purposes the indicators they collected; (2) eliminate duplication of re￾porting of the same indicators from the three different institutions; (3) consolidate the re￾porting of indicators into a single report submitted to the MoH Directorate of M&E at the 
central level; and (4) better analyze and report the data to the MoH. Each institution has 
responsibility for one section of the consolidated regional M&E report, but the three hospi￾tals work collaboratively on the analysis and presentation of the report. This work has been 
actively supported by EEHR, including assistance with graphics and lay-out.
l. MoH’s M&E Directorate strengthening: While not directly related to the activities done at 
the pilot hospitals, the MoH M&E Directorate has also been strengthened by the project. 
For the first time, this unit of the MoH has a clear mandate and is actively collaborating with 
other institutions at the central and regional level. The directorate is also serving as a secre￾tariat for the HRISG. 
CONCLUSIONS
Per both the objective measure of PBMP targets versus actual results (primarily output-level indicators) and 
key informant interviews, the development of tools and mechanisms for the pilot regional hospitals should 
be considered one of the more successful elements of EEHR. However, as noted previously, this statement 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 17
cannot be considered definitive as the various tools and mechanisms were only implemented two years pri￾or to this evaluation and higher-level outcomes were not examined during this assignment nor were they 
tracked as part of the project’s PBMP.
Similar to behavior change initiatives, the adoption of new practices via the development and introduction 
of tools would require dissemination of knowledge (usually via trainings), a change in attitude away from 
previous practices, and consistent and sustained usage of the new practices. This would require more than 
the two year timeframe between the introduction of the tools and mechanisms and the initiation of this 
evaluation2. 
Both the process for the development of the tools and their subsequent utilization and, per the focus 
groups and the unanimous opinion of the EEHR KIIs, improvement of staff performance, point toward the 
potential for sustained usage and a longer-term effect. Anecdotal evidence in terms of shared consultants 
for EEHR and the upcoming World Bank health project further this conclusion, as well as, the pilot hospitals 
sharing lessons learned in the usage of the tools. However, any sustainable use might be undermined by a 
change in a hospital’s leadership (i.e. the Hospital Director) and, thus, sustainability of this part of EEHR’s 
activities is not ensured.
EVALUATION QUESTION 3: Based on the perceptions of outside stakehold￾ers, do the selected regional hospitals have better performance as a result of the ap￾proaches and implemented activities than they would otherwise have?
FINDINGS 
1) In general, the findings for this question are uncertain because of the need for objective measures in 
addition to the limited outside key informant interviews which were conducted as part of this eval￾uation. Current indicators within the PBMP do not capture this information and, most likely, an ad￾ditional study, such as a facility and staff satisfaction survey, outside the scope of this evaluation, 
would be able to make this determination.
2) The limited number of outside stakeholders (four persons chosen in conjunction with USAID rep￾resenting HRISG members, other technical agencies, and donors) queried by the evaluation teams 
do believe that there is better performance at the pilot hospitals both in terms of before and after 
EEHR’s pilot interventions, as well as, in comparison to other similar current hospital settings in Al￾bania. For example, at least one MoH Deputy Minister interviewed as part of the evaluation, re￾quested the expansion of this initiative to all Albanian hospitals. The two Directors of professional 
orders who were interviewed, likewise, believed the activities have benefited the pilot hospitals’ 
performance.
3) EEHR staff were unanimous in their opinion that the pilot activities have resulted in better hospital 
performance; however, their feedback must be qualified in that they are not outside stakeholders.
4) The large number of pilot activities implemented by EEHR as cited in Question 2 above should, in 
theory, result in better hospital performance. 
 
2 Hospital Quality Improvement: Strategies and Lessons from U.S. Hospitals, Sharon Silow-Carroll, Tanya Alteras, 
and Jack A. Meyer, April 2007, The Commonwealth Fund.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 18
CONCLUSIONS
Unfortunately, there were insufficient outside stakeholders interviewed to draw any definitive conclusions 
for this question; thus, a number of theoretical assumptions must be made. Based on the number and types 
of activities implemented by EEHR, it can be reasonably concluded that, theoretically, the regional hospitals 
should have better performance post project interventions. The small number of outside stakeholders who 
were interviewed did, indeed, cite that they believed that the three hospitals were functioning better after 
the project activities and were also better in comparison to similar hospitals in Albania. Additionally, the 
questioning of outside stakeholders alone would not allow definitive conclusions to be made. A facility￾based survey along with a pre- and post- patient satisfaction survey are needed to examine hospital perfor￾mance more thoroughly and answer this question. The PBMP did include an indicator “Improved operations 
and resource management”, which was based upon the hospital composite indicator and baselines were set. 
While some progress was seen by the end of Year 2 and by Quarter 3 of the third year of implementation, 
which appears to exceed the targets set, this is an insufficient measure. Without the aforementioned surveys 
along with an established counterfactual to determine how much improvement is due to EEHR efforts, the 
most robust conclusions can only be theoretical.
EVALUATION QUESTION 4: How successful was the contractor in designing 
and implementing the small grant program?
FINDINGS 
1) The implementation of the small grants programs was delayed by more than one year than its origi￾nal start date, per the first year work plan. The reasons for this delay are unclear as different key in￾formants attributed to the delay to different factors. Some KIIs cited that EEHR staff needed to be 
more pro-active in seeking civil society organizations with which to work (in addition to building 
their capacity), while other KIIs cited an overly bureaucratic grant procurement process, and other 
KIIs noted that the size of the potential pool of small grantees had been greatly overestimated.
2) A total of three grants were awarded. Per EEHR staff, two additional grants were considered for 
award, but not funded due to the early close of the project. The three grants awarded were as fol￾lows:
a. IDRA Research and Consulting: for qualitative research on perceptions and attitudes 
toward public hospitals among residents of Tirana, Korca and Lezha regions.
b. Together for Life: an NGO of journalists to strengthen the role of media in informing 
the population on issues of health reform, work with journalists to strengthen their 
knowledge on health issues, and engage young people through social media and other 
Internet tools on health and health reform topics. 
c. Progress Per Civilizim (PPC): for activities related to maternal and child health referral, 
including current status assessment of problems in the perinatal referral system, catego￾rization of perinatal services, preparation of guidelines and referral algorithms for the 
MCH/perinatal system.
3) The original budgeted amount for the small grants program was $400,000. The actual monies ex￾pended by April 2014 (nearly 3.5 years into project implementation) were approximately $34,537 
or 8.7% of the original amount. It was projected by EEHR staff that when the final payment is made 
under the remaining grant, the total monies expended will be approximately $40,577 or 10% of the 
original budgeted amount. 
4) The EEHR Small Grants manager had multiple job titles and duties, including occasionally serving as 
the acting Chief of Party during part of the first two years of project implementation. Additionally, 
during the key informant interview, the Small Grants Manager was unaware of some relatively 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 19
common granting procedures which would have allowed a more unbiased and, potentially quicker,
granting procedure (i.e. discarding the high and low scores from the grant review panel), as well as, 
expressing dissatisfaction with the granting process in general.
5) Three formative research documents3 produced by international consultants highlighting the chal￾lenges of engaging with civil society on the issue of public health and health services were finalized 
in March 2012, and May and June 2013, approximately 1.5 and 2.5 years after EEHR began imple￾mentation. Some of the barriers noted lack of capacity among civil society organizations, a general 
public distrust of non-governmental organizations, and cultural barriers to civil society engagement 
and volunteerism.
CONCLUSIONS
This element of the project can be considered the weakest of those examined by this evaluation. Only a 
small portion of the original budgeted amount for grants was utilized, the awarding of grants started much 
later than anticipated, and the capacity of EEHR staff dedicated to this activity was, most likely, insufficient 
both in terms of quantity (one individual with multiple duties) and quality (inadequate experience managing 
granting procedures). Though two of the grants (IDRA and Together for Life) could be considered as having 
a direct link to the project’s goal of influencing health care reform, the linkage of EEHR’s goals with the third 
grant (PPC) is unclear.
Although three documents were produced as formative research by EEHR which noted some of the chal￾lenges which might be encountered in trying to engage non-state actors into the health sector, the first of 
these documents was finalized in March 2012 or 1.5 years into project implementation. Thus, the findings 
from the research were, most likely, not timely enough to allow EEHR to mitigate any challenges it might 
encounter in the small grants activity.
EVALUATION QUESTION 5: What were the challenges (internal and exter￾nal) faced by the program and lessons learned?
FINDINGS 
1) At least six individuals served as either an official, temporary or acting EEHR CoP during the life of 
the project; though, EEHR has had a consistent CoP since February 2012. All key informants inter￾viewed noted that each CoP had a different management style and vision for the project.
2) EEHR staff varied in their opinions on the usefulness of Phase 1/Year 1. All agreed that Phase 
2/Year 2 was a positive turning point for EEHR.
3) EEHR staff varied in their opinion on the working relationship and partnership with USAID with the 
emergent theme that the partnership worked best on the practical, activity-based level and faltered 
at the higher conceptual level. This was particularly cited as an issue around the concept of “country 
ownership”. All EEHR staff cited examples of USAID not providing rationales behind its decisions.
All also cited the delays with the work plan approval process as a hindrance to project implementa￾tion.
 
3 Joanne Jeffers and Palushaj, Ornela , March 22, 2012. Increasing Non-State Actors’ Engagement in Health System Governance, Technical Report. 
Bethesda, MD. Enabling Equitable Health Reform Project in Albania, Abt Associates, Inc.
IDRA Research and Consulting, May 08th
, 2013. Health Care System in Albania: A Formative Research with Consumers to Increase Non-State Ac￾tors Engagement in Health System Governance. Bethesda, MD. Enabling Equitable Health Reform Project in Albania, Abt. Associates Inc.
O’Sullivan, Gael. June 25, 2013. Civil Society Engagement and Communication Strategy and Action Plan - Technical Report. Bethesda, MD. Enabling 
Equitable Health Reform Project in Albania, Abt Associates Inc.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 20
4) No USAID staff could easily articulate what should be the benchmark for determining the project’s 
success or measurement of performance. EEHR staff responded more readily to this line of ques￾tioning citing the pilot hospitals as the primary source for measuring performance.
5) No USAID staff interviewed was certain if they and EEHR had a shared vision for the project. Like￾wise, no EEHR staff was completely certain of a shared vision, though, some cited the work plan 
approval process as a proxy for a shared vision. 
6) The SOW of in the original contract instructed the implementing partner that “All activities should 
be conducted in close cooperation and partnership with local counterparts to ensure their full 
ownership and transition, future sustainability and growth of targeted sectors”. During the life of the 
project, there were four Ministers of Health. Additionally, the political appointee system in Albania 
extends down to the hospital level, including Hospital Directors; thus, there was leadership turnover 
at the pilot hospitals as well.
7) The HRISG (and to a lesser extent hospital boards) were established by the project to, among oth￾er activities, assist in mitigating the MoH turnover and, thus, weak country ownership. However, the 
HRISG was primarily composed of GoA officials and suffered from the same turnover and, most 
likely, a lack of institutional memory.
8) Continued underfunding of the health sector was not a large obstacle for the project, but it was an 
obstacle. The project was able to provide some financial support such as the minor renovation 
work in the hospitals. However, limitation of funding to the sector could undermine the project’s 
potential for rolling out reforms to all hospitals.
CONCLUSIONS
The three major stakeholders for this project, EEHR staff, USAID, and Government of Albania / Ministry of 
Health officials, all had significant challenges which needed to be addressed in order to better ensure suc￾cessful project implementation. It can be reasonably concluded that the number of individuals which served 
in the Chief of Party role was detrimental in the project forming a cohesive vision until the final CoP arrived 
in February 2012 or nearly 1.5 years after the project’s start. Further, this turnover in senior leadership and 
changing project vision combined with Phase 1 activities which were designed to better clarify the vision of 
the remaining life of the project, could also reasonably be assumed to have resulted in project delays. Until 
there was a focal point of responsibility for ensuring an internal unified vision for EEHR, any activities under￾taken as part of a work plan could be considered tentative. Finally, the working relationship with USAID, the 
project’s donor, may have needed strengthening as the EEHR staff interviewed cited examples of not being 
provided enough information, along with, uncertainty of whether there was a shared vision of project suc￾cess with USAID.
USAID may have had during the initial phases of the project’s lifecycle, including the project design phase, a 
clear vision of what would be considered a successful project. However, that vision appears to have evolved 
and become more unclear during actual implementation. Further, perhaps, more frequent and higher-quality 
communication was needed with EEHR staff as USAID KIIs were uncertain whether there was a shared vi￾sion and all EEHR staff shared examples of not being fully informed of the rationale behind USAID decisions. 
Finally, it should be noted that the underlying pillar for EEHR’s success was the idea of “country ownership”.
In other words, it was anticipated that Government of Albania counterparts, primarily Ministry of Health 
officials, would take a strong leadership role in moving forward with needed reforms. However, given that 
there were four Ministers of Health within the 3.5 years of the project’s life, it would be difficult to conclude 
that this ever occurred. The development of the HRISG may have been one activity designed to mitigate 
the changing MoH leadership; but, given that the HRISG was primarily composed of GoA officials, it too 
probably did not contribute significantly to instituting country ownership of the reform process.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 21
RECOMMENDATIONS 
1) Prior to the second evaluation team’s departure, USAID had an initial meeting with the World Bank 
regarding a proposed health sector project that the MoH had requested from the World Bank.
Though the World Bank project is still in its initial phase of feasibility and scoping studies, if possible, 
USAID and EEHR staff should collaborate with the World Bank on the design of its new project 
and consider how initiatives undertaken or planned during Phase II of EEHR might be sustained as 
potential catalysts and models for reform in Albania’s health sector. The focus should be on: (1) 
hospital autonomy (both legislation and building blocks needed to prepare hospitals for autonomy); 
(2) HII’s reimbursement mechanism for the hospitals; (3) conceptualizing nationwide roll-out of the 
strengthening activities undertaken in the pilot hospitals; and (4) M&E as instrumental to supporting 
informed policy formulation through the HRISG. Further, any relevant research which was conduct￾ed as part of Phase 1 of EEHR should be shared freely with the World Bank. By building upon the 
lessons learned from EEHR, the primary objective of the EEHR project of identifying barriers and 
obstacles to more effective health policy and reforming implementation to increase access to health 
services, particularly for the poor, could be built upon. This would further efforts to ensure sustain￾ability of the project’s goals and the three years of EEHR implementation combined with the multi￾ple years of activities under the World Bank project should lead to higher level impacts given suffi￾cient time.
2) During the few remaining months of the project, no new programmatic initiatives should be under￾taken. Ongoing initiatives should be reinforced so that they are more likely be sustained. New initia￾tives, especially those for which plans have been made, should be suggested for the World Bank 
health sector project design. Those initiatives currently successfully underway in the hospitals in￾clude: human resource management activities; incident reporting and analysis of reports; space allo￾cation renovations; visitor management activities; environmental services and infection control activi￾ties; contracting of hotel-type services; visitor management and public relations (including activities 
designed to engage the community and eventually support community involvement in an independ￾ent hospital board). The ongoing work in strengthening M&E at the regional and central levels 
should also be supported, along with the HRISG’s continuation.
3) USAID should consider continued funding for the HRISG via one of its on-going governance pro￾jects since the HRISG does function as a quasi-governing advisory board for the Ministry of Health.
However, prior to the continued funding being granted, USAID along with the World Bank and 
other development partners should consider pressing the GoA to reform the HRISG such that the 
total percentage membership of the GoA on the HRISG is no more than 33%. Given the continued 
turnover in GoA leadership, this reduced GoA percentage may allow a greater institutional memory 
to develop if the remaining 67% were divided equally between civil society /patients’ rights organiza￾tions and development/technical assistance partners. Further, as has been shown with similar gov￾ernance mechanisms, such as the Global Fund’s Country Coordinating Mechanisms, the strong 
presence of civil society organizations and health sector beneficiaries provides an important feed￾back mechanism in health reform.
4) USAID can also continue to support health reform activities by engaging in a dialogue (along with 
the World Bank representative) with the new Minister of Health about: (1) the pending legislation 
on hospital autonomy; (2) the M&E work, and especially sustaining the HRISG as an inter￾institutional policy-making group; and (3) in supporting future work with the HII on reimbursement 
mechanisms for the hospital sector, as well as, improving participation in the HII, particularly in the 
informal sector. 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA 22
5) Time and funding permitting, it would be useful to hold a conference or similar event prior to the 
end of the project, or as an end-of-project event, that engages those in positions of leadership in a 
discussion about the findings and recommendations in key project documents. In this manner, 
USAID could support the new leadership within the MOH and HII by ensuring that the analytical 
work produced under the EEHR project is provided to those in new positions of leadership. The 
substance of materials created by the project could be converted to “teaching case studies” to sup￾port an analytical decision-making process around topics such as the basic package of hospital ser￾vices and HII payment mechanisms. 
6) USAID might also consider avenues for sustaining the local capacity for technical assistance that has 
been developed under the EEHR Project, which itself builds on previous USAID projects, through 
supporting the possible creation of a local NGO that houses the existing capacity, or through other 
mechanisms that might sustain the capacity in some institutional form, whether public or the private 
sector. The support of indigenous organizations is a worldwide trend for USAID, and apparently 
other USAID-funded projects have been converted to indigenous NGOs in Albania, which may 
provide useful models.
7) While the Project was noted for building the local counterpart capacity in monitoring and evalua￾tion, its internal level of M&E requirements was set quite low. The great majority of PBMP indicators 
were at the output level and very few internal evaluative activities were planned. Though the 
broader sector-wide scope of the project was scaled back to focus on reforming and improving ter￾tiary (hospital) management and care through pilots, this does not relieve any of the stakeholders 
from the obligation of ensuring that outcomes are monitored and evaluated. To definitively deter￾mine the answers to some of the evaluation’s questions and for the improvement of the project’s 
performance during implementation (primarily those aimed at better understanding the outcomes 
at the pilot hospitals), additional methods should have been utilized. This could have included facili￾ty-based surveys, staff surveys, verification of improved staff practices and use of improved infra￾structure through direct observation, community surveys, and patient/client satisfaction surveys. 
Further, by engaging the communities being served by the facilities in the monitoring and verification 
of results, project beneficiaries would not only have been better educated and informed about their 
health rights, but, potentially, become advocates for those rights as well. Had additional resources 
been dedicated to measurements at the higher outcome level, perhaps, stronger findings conclu￾sions, and recommendations would be forthcoming from any external evaluations conducted.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-1
ANNEX A: STATEMENT OF 
WORK
SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK
C.1 PURPOSE
The Contractor will conduct two performance evaluations of the USAID-funded EEHR and PLGP Projects 
during the period of September 2013 to November 2014 in Albania.
C.2 BACKGROUND
Albania has made many notable improvements in rule of law and good governance in the last twenty-two 
years but – coming out of decades of harsh communist rule and a tradition of fragmentation and localism – 
the country has not been able to move forward with the momentum required to introduce strong and sus￾tainable democratic institutions that form the bedrock of representative, stable, prosperous, and equitable 
societies. In late 2011, USAID began implementation of a new Country Development Cooperation Strategy 
(CDCS) for Albania, covering the period 2011-2015. Under this strategy, the Mission will continue to en￾gage in areas of past investment, including democracy and governance, and economic growth to help ensure 
sustainability of reforms and progress in these areas. 
USAID/Albania’s overarching goal for the 2011 - 2015 strategy is “European Integration through strength￾ened democratic institutions and inclusive economic growth.” In support of this goal, USAID has set two 
primary objectives:
• Strengthened Rule of Law and Improved Governance
• Conditions Created for Broad Based, Sustainable and Inclusive Economic Growth
The synergies between these two objectives are exceptionally strong, in that success in achieving greater 
transparency and reduced corruption positively impacts the private sector and, looking at it the other way, 
economic growth helps to create a stable environment for a free and open democracy. Greater emphasis 
will be put on increasing sustainability of development impacts and potential legacy partners through en￾hanced use of host-country systems and local institutions and organizations.
In compliance with the Automated Directives Systems (ADS) 203.3.2 and the New Evaluation Policy,
USAID Albania will conduct two performance evaluations to determine progress of these programs imple￾mented in the final phase of Albania’s aspirations for EU candidacy.
C.3 PURPOSE OF THE EVALUATIONS 
The purpose of the two performance evaluations subject to this Statement of Work (SOW) is to provide 
USAID with external evidence for development outcomes and performance of the defined interventions in 
the two projects mentioned in the above summary. Each evaluation will be conducted at the mid-point of 
program implementation, which will allow for a thorough assessment of activities as well as enough time to 
modify/improve the approaches for the remaining period before the end of program. These evaluations will 
assist USAID in determining the degree of the success in current programs based on collected evidence and 
external analysis of the models of cause and effect using a broad range of qualitative, quantitative, or mixed 
methods.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-2
This is an umbrella SOW that covers two performance evaluations and provides overall guidance for the 
potential implementer. More specificity is provided in the separate statements of work for each individual 
evaluation.
C.4 METHODOLOGY
The Contractor, in collaboration with USAID/Albania, will finalize the overall evaluation methodology once 
in-country. However, the Mission expects that, at a minimum, the team will:
• use methods that generate the highest quality and most credible evidence that corresponds to the 
questions being asked;
• take into consideration time, budget, and other practical considerations; and
• consider both qualitative and quantitative methods as they both yield valuable findings.
Depending on the SOWs of each of the two performance evaluations where the purpose and key ques￾tions of the evaluation are identified, the design and the types of methodology used may be relatively simple 
or more complex.
C.5 EVALUATIONS TEAM
Evaluation Team Leader: Each of the teams will be comprised of one independent evaluation expert who 
has experience with the subject matter of the particular evaluation with no relations to the project whatso￾ever in order to ensure unbiased reporting and avoid even a perception of a potential conflict of interest. 
The team leaders of each of the two evaluations should have significant experience in evaluating develop￾ment assistance and working on or evaluating projects aimed at improving the specific subject matters.
Local team members: Each of the evaluation teams will have one or two local experts (depending on the
specific scope for the evaluation) who have excellent understanding of the subject matters and are able to 
establish contacts and communicate effectively with both government officials, businesses, non￾governmental organizations and other independent entities as necessary. They must be proficient in English. 
To avoid conflict of interest, none of the local team members should have current or past business relation￾ships with the Project whose evaluation will be conducted.
Assistants: Each of the evaluation teams will have one assistant responsible for translation with counterparts 
and logistical support.
While the contractor is responsible for assembling the team described above, USAID/Albania may include a 
member of the staff to participate in the evaluation team. Although led by the external evaluation expert, 
the evaluation teams should consider including customers and partners (implementing partners, host￾country government partners, other donors’ staff, etc.). More details on each team will be provided in the 
specific SOWs for each of the performance evaluations.
C.6 SCHEDULE AND LOGISTICS
The evaluation efforts will commence as scheduled in the individual SOWs for each of the two evaluations. 
The table below provides tentative timeframes for the different phases of the evaluations. For each evalua￾tion, the Independent Team Member should arrive in Tirana, Albania and be prepared to begin work as 
soon as the individual SOW is issued. USAID/Albania will provide the team with input and guidance in set￾ting up a schedule of interviews and site visits, but the responsibility for the schedule for each evaluation 
resides with the Contractor.
Enabling Equitable Health Reform September – November 2013 – Field work of the 
Performance Evaluation
Planning and Local Governance Program September – November 2014 – Field work of the 
performance evaluation
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-3
The schedules should be defined as much as possible before the Independent Team Member arrives in 
country and should be finalized as soon as possible after the full Team is assembled in Albania. Draft sched￾ules should be ready for review and discussion at the initial team planning meetings with USAID, which 
should take place within two days of when the team first convenes in Albania.
Prior to beginning of the field work in Albania, all team members will review background program docu￾ments to gain a firm understanding of the situation in Albania and the two USAID Projects subjects to eval￾uations.
The Independent Team Member will spend at least two weeks in Albania. The team will interview key 
USAID and Project staff, beneficiaries of USAID’s assistance, representatives of the Government of Albania 
(GOA), other appropriate donor organizations providing assistance in the local governance, health sector 
management and civil society. Additionally, in its response, the Contractor shall propose its plan for selecting 
a representative number of Project activities for the evaluation team to assess based on the evaluation ques￾tions.
The Contractor is encouraged to identify and visit additional Albanian organizations and groups, both formal 
and informal, based on its review of materials.
C.7 DELIVERABLES
For each evaluation, the Contractor will provide the following:
1. The work plan for approval, which should include:
a. Evaluation Design Document including: a detailed evaluation design matrix (the key ques￾tions, methods and data sources used to address each question and the data analysis plan 
for each question), draft questionnaires and other data collection instruments or their main
features, known limitations to the evaluation design, a Final Evaluation Report outline and a 
dissemination plan.
b. The anticipated schedule and logistical arrangements, delineating the roles and responsibili￾ties of members of the evaluation team to ensure coverage of all elements of the State￾ment of Work. The work plan shall include briefing with USAID and timing of deliverables. 
Work Plan for the overall evaluation shall be completed by the lead evaluator within two 
weeks of the award of the contract and presented to the M&E Specialist and AOR.
2. All proposed team members must provide a signed statement of interest attesting to a lack of con￾flict of interest or describing an existing conflict of interest relative to the project being evaluated.
3. Regular briefings for USAID representatives during the field work.
4. An out brief for USAID/Albania at the end of the field work, including written materials (a power 
point presentation or an outline of the Evaluation Report – no more than 5 pages) which should in￾clude data from the baseline survey, as well as data from field visits, background of the local context 
and the projects being evaluated, the main evaluation questions, the methodology or methodolo￾gies, and preliminary findings.
5. Final Evaluation Report. The following sections shall be included in the document:
i. Table of Contents
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-4
ii. An Executive Summary – (2-3 pages) containing a clear, concise summary of the most criti￾cal elements of the report, including the recommendations/impact.
iii. Evaluation Findings (no more than 15 pages), which provides analysis and answers the 
questions listed in the specific Statements of Work including baseline data and counterfac￾tual data collected.
iv. Detailed Recommendations for the interim period between the data collection and the fi￾nal evaluation.
v. Report Appendices, including:
a. A copy of the evaluation Statement of Work;
b. Cross-reference guide listing the evaluation questions and specifying on which page the 
questions are answered in the report.
c. Team composition and study methods (2 pages maximum);
d. A list of documents consulted, and of individuals and agencies interviewed; and
e. More detailed discussions of methodological or technical issues as appropriate.
6. Turn over all data collected while conducting the evaluations to USAID Albania at the end of the 
contract. All the records from the evaluation (e.g., interview transcripts or summaries) must be pro￾vided to the Task Order Contracting Officer’s Representative (TOCOR.) All quantitative data col￾lected by the evaluation team must be provided in an electronic file in easily readable format agreed 
upon with the TOCOR. The data should be organized and fully documented for use by those not 
fully familiar with the project or the evaluation. USAID/Albania will retain ownership of the survey 
and all datasets developed.
The Contractor will be responsible for providing the final deliverables to USAID/Albania no later than ten 
working days after the departure of the Evaluation Team leader from Albania. This period includes also the 
feedback from USAID mission to the draft of the Evaluation Report. 
The Contractor will be responsible for providing the final deliverables to USAID/Albania via email and in 
hard copy. The Contractor, upon approval of the final Evaluation report by USAID/Albania, will also provide 
an electronic copy to DEC, the database of the USAID Development Experience Clearinghouse (DEC) 
within 30 days of completion. http://dec.usaid.gov
C.8 PROPOSED LEVEL OF EFFORT
Specific guidance on the level of effort/team composition will be provided in each of the two separate 
SOWs for the two evaluations.
C.9 SPECIAL CONSIDERATIONS
Duty Post: Tirana, Albania
Access to Classified Information: The Contractor will not have access to any Government classified material.
Workweek: A six-day workweek is authorized. 
Logistical Support: The Contractor is responsible for providing all logistical support. Office space shall not be 
provided by USAID. The Contractor will be responsible for providing office supplies, equipment, computers, 
copiers, printers, etc. Translation services and vehicle rentals are the responsibility of the contractor.
USAID/Albania will provide assistance with scheduling appointments and meetings, at the possible extent.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-5
Supervision: The team will work under the technical direction of the USAID/Albania TOCOR, i.e., the Moni￾toring & Evaluation specialist assisted by the Contracting Officer Representative (COR) of the specific pro￾ject undergoing evaluation.
C.10 SCOPE OF WORK FOR EEHR
Enabling Equitable Health Reform in Albania
C.10.1 GENERAL DATA
TITLE ENABLING EQUITABLE HEALTH REFORM (EEHR) 
AWARD TYPE AND # CONTRACT # 182-C-00-10-00104-00 
IMPLEMENTER ABT ASSOCIATES 
AWARD VALUE $8,605,712 
LIFE OF PROGRAM OCT. 01, 2010 – SEP 30, 2015 
COR AGIM KOÇIRAJ 
I. BACKGROUND
EEHR Project is designed to directly support USAID’s health governance program area in Albania.
The EEHR Project aims to strengthen the Albanian health sector’s capability through the implementation of 
equitable health reforms, moving from strategies into implementation action plans, by providing instruments 
and tools and testing those in selected districts.
The purpose of this project is to provide access to health services for the poor by a) helping remove the 
existing barriers and constraints to the reforms at the national level and b) field testing approaches and tools 
that define a feasible set of implementable reforms in Albania.
The expected result under this project is to produce progress toward equity for the poor in the following 
priority areas:
• The first priority is to improve access, identify costs, reduce informal payments, and improve pro￾vider responsiveness at primary and secondary levels of care.
• The second priority is health systems strengthening to improve overall quality of care and efficiency 
of the health delivery systems:
• The third priority is to improve overall performance of the health care delivery system by better 
linking performance based health financing with improved resource allocation, improvement of ca￾pacity and enforcement of standards.
The above results are expected to be partially achievable at a national level during the life of the project and 
beyond.
Nevertheless, they are first expected to be tested at a regional level to become the driving force which 
practically and conceptually directs strategic and operational planning at a national level. The EEHR project is 
designed to be implemented in two phases:
Phase One Objectives are: a) to help narrow the health reform agenda to a feasible set of priority actions 
that will likely have the greatest impact nationally and b) to produce tools and mechanisms that will enable 
the implementation of action plans.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-6
Phase One - Expected results
• Removed key barriers/obstacles to the reform processes;
• Established Institutional framework for implementation plans;
• Defined feasible parameters that include but are not limited to:
a. Standard packages of PHC and hospital care to be covered by health insurance;
b. Referral mechanisms between different levels of care;
c. Costing of the defined services;
d. Performance measures needed to for health delivery system;
e. Standards and protocols developed etc.
Phase Two Objectives are: a) to use the produced tools and mechanisms in designated pilot region(s) field 
testing this way the defined feasible set of priority reform actions and b) to produce evidence to guide na￾tional implementation of health reforms.
The phase II of the project is planned to start by the end of the second year of the project life, following a 
thorough review process of the phase I. A commonly agreed plan for regional implementation will be deliv￾ered and approved.
Phase II - Expected Results
• Improved access, coverage and utilization of health services at the pilot region (s);
• Increased technical and financial efficiency;
• Improved health performance and outcomes through integrated services;
• Ensured sustainability;
• Removed barriers to institutional collaboration;
• Improved GoA’s capacity to implement reforms at the regional levels;
• Provided guidance to national government reform efforts; and
• Expected results related to the material assistance:
o Visitor control; system installed and functional at Tirana Maternity Hospital
o Software deployed; IT infrastructure improvements completed in Lezha Hospital
o IT equipment delivered, configured and installed in Lezha and Korça hospitals
o Refurbishment work completed in Korça and Lezha Hospitals.
C.10.3 PURPOSE 
The Purpose of this performance evaluation is to provide rigorous, evidence-based and independent analysis 
on EEHR’s performance at the mid-term of the project and inform the Mission of continuing unmet needs 
in the sector. Specifically, the performance evaluation will serve to examine the processes, outcomes and 
the effectiveness of the project activities implemented to date; to determine whether the project has 
achieved the expected results according to the plan, to identify gaps in performance against targets, and to 
provide broad recommendations to the Mission for future interventions.
The evaluation will focus on both management and technical aspects of the program and its implementa￾tion. The evaluation will assist USAID in determining the degree of the success in current programs and 
identifying lessons learned for the development of future programs. 
Qualitative and quantitative data will be used to illustrate the processes and outcomes of the program. Data 
generated from the evaluation should be specific to each of the programs and provide solid evidence for 
conclusions and recommendations that will be made by the evaluation team(s). 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-7
It’s expected that the Evaluation Team will not simply provide an accounting of performance against targets, 
but provide an independent analysis on why targets were met or not achieved. Of particular interest will be 
the analysis of the reasons behind any gaps between expected and actual performance, including the identi￾fication of the likely source(s) of these gaps (design, project management, changes in operating environment, 
success factors, etc.). This analysis will inform future decisions regarding the project's focus by the Mission, 
particularly by the Democracy and Governance Office.
C.10.4 EVALUATIONS QUESTIONS 
In conducting the performance evaluation for the EEHR program, the Evaluation Team will provide detailed 
answers to the following questions:
1. To what extent have EEHR activities resulted in removing key barriers/obstacles to the health re￾form processes;
2. How successful was the project in producing tools and mechanisms that are tested as successful in 
pilot regional hospitals?
3. Based on perception of outside stakeholders, do the selected regional hospitals have better per￾formance as a result of the approaches and implemented activities, than they otherwise would 
have?
4. How successful was the contractor to design and implement the small grant program?
5. What were the challenges (internal and external) faced by the program and lessons learned?
The report will summarize the findings from interviews, discussions, and evaluations of relevant EEHR re￾ports, studies and assessments. Pitfalls and gaps, if any, should be analyzed, justified and addressed by rec￾ommendations. The evaluators are encouraged to be as specific as possible in its recommendations, so as to 
best inform our Mission for the sake of prompt reaction if needed. Recommendations should be action￾oriented, practical and specific, with defined responsibility for the action.
The evaluation (data collection, analyses and reports) should reflect the cross-cutting issues of gender and 
environmental considerations.
C.10.5 SUGGESTED METHODOLOGY AND STUDY TASKS 
The Evaluation is described in the section C.4 at page 4 above. In addition to that the Mission is looking for 
additional suggestions on the methodologies, data collection instruments for conducting the evaluation, and 
it is anticipated that the Contractor will provide a more detailed explanation of the proposed methodology, 
for carrying out the work and appropriate to the Evaluation questions.
It is anticipated that the evaluation team will first complete a desk study that will be used to establish an un￾derstanding of EEHR’s activities and environment before the field work.
The desk study will include, (but not limited to) the following steps:
Review of the relevant portions of the project’s contract (original and modified).
Review of all relevant project reports, assessments, annual and life of project work plans.
Review of the project’s grants documentation.
Review of the project’s annual and life of project performance management plans (PMP).
Review of USAID/Albania Strategy.
Meetings with Abt Associate, relevant staff in USA (by phone) as necessary.
Briefings with USAID/Albania’s relevant staff (Program Office, project COR, etc.)
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA A-8
Based on this understanding, the team will prepare a work plan that will be presented to the Mission prior 
to arrival in-country. The work plan will include a design matrix that demonstrates how the team plans to 
answer each evaluation question (data collection methods, sources, methods of analysis, limitations, etc. The 
methodology will include a mix of tools appropriate to evaluation’s research questions.
C.10.6 TEAM COMPOSITION 
The evaluation team will consist of three members: two technical specialists, and an interpret￾er/administrative assistant. The US expert will serve as the Evaluation Team Leader.
The evaluation experts shall demonstrate familiarity with USAID’s Evaluation Policy
http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf
Specific responsibilities and requirements for each of the Evaluation team members include the following:
a) Evaluation Team leader/Social scientist: The incumbent must have extensive and documented rele￾vant prior experience in leading evaluation of development projects and programs, designing evalua￾tions, writing evaluations reports. S/he should possess experience and knowledge related to similar 
health reforms in countries in transition, preferably Western Balkans countries. The proposed indi￾vidual will meet the minimum level of academic and the work experience qualifications required for 
the Senior level expert in the health field.
b) Evaluation Team member/Social scientist (locally hired): S/He should have experience in evaluating 
development programs especially those of health governance and management focus. This team 
member is expected to have excellent knowledge and professional background that demonstrates 
extensive experience and understanding of Albanian context, Albanian health sector, and related 
policies and management systems. S/he should be fluent in English and in Albanian. S/He will meet 
the minimum level of academic and the work experience qualifications required for the Junior level 
expert.
c) Interpreter/administrative assistant (locally hired): This team member will provide logistical, adminis￾trative, clerical and translation support to the team throughout the evaluation. Special attention 
should be paid to this team member’s ability to translate technical language as it relates to the 
health systems management and health policies terminology.
C.10.7 SCHEDULE AND TIMELINE 
USAID/Albania anticipates having this evaluation conducted in fiscal year 2014 during the period September 
2013 – November 2013. The number of working days will include up to four days of prearrival preparation 
and up to ten days of post in-country work for writing and finalizing the evaluation report. A six-day work￾week for the Evaluation Team(s) and travel days will be authorized accordingly for expatriate Evaluation 
Team members.
USAID/Albania will provide assistance with scheduling appointments and meeting. The Level of Effort for 
the technical team members is 18 days and for the Team Leader 27 days. Deadline for submission of deliv￾erables and a suggested breakdown is shown in Table 1.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA B-1
ANNEX B: EVALUATION QUES￾TION MATRIX
Research Question Evaluation Tools Specific Data Sources Data Analysis Plan 
1. To what extent 
have EEHR activities 
resulted in removing 
key barriers/obstacles 
to the health reform 
processes? 
EEHR document re￾view
Informant interviews 
Focus group discus￾sions 
EEHR reports, existing health￾related legislation, EEHR part￾ners (Ministry of Health, Insti￾tute of Public Health; HII; Na￾tional Center for Continuing 
Education; the National Center 
for Quality, Safety and Accredi￾tation; Regional Public Health 
Departments; Regional HII Of￾fices; and health centers), doc￾uments produced by and those 
working in other health-related 
assistance projects, NGOs and 
other organizations involved 
with health reform.
Identifying key barri￾ers/obstacles in place 
prior to EEHR launch. 
Matching EEHR pro￾gram activities (and their 
intended outputs and 
outcomes) to particular 
barriers/obstacles. Ana￾lyzing the incentives and 
relative influence of key 
stakeholders. Assessing 
the current status of 
identified barriers.
2. How successful was 
the project in produc￾ing tools and mecha￾nisms that are tested 
as successful in pilot 
regional hospitals? 
 
EEHR document re￾view
Informant interviews
Focus group discus￾sions
Site visits to three re￾gional testing hospitals 
and surrounding urban 
and rural areas
EEHR performance monitoring 
plan and reports, results 
framework, M&E reports, test￾ing hospital documents and 
personnel, EEHR partners, oth￾er key project stakeholders
Assessing the process of 
designing and imple￾menting tools. Evaluating 
collaboration among all 
relevant stakeholders. 
Assessing whether tools 
address specific, identifi￾able health care prob￾lems. Assess ongoing 
implementation of tools 
(obstacles, successes, 
etc.)
3. Based on percep￾tion of outside stake￾holders, do the select￾ed regional hospitals 
have better perfor￾mance as a result of 
the approaches and 
implemented activities, 
than they otherwise 
would have? 
EEHR document re￾view 
Informant interviews 
Focus group discus￾sions
Site visits to three re￾gional testing hospitals 
and surrounding urban 
and rural areas
EEHR partners, testing hospital 
documents and personnel, ur￾ban and rural citizens near test￾ing hospitals
Develop baseline un￾derstanding of situation 
prior to pilot testing. 
Assess current status of 
health care access, fi￾nancing, and insurance 
at testing sites, and re￾late that to data from 
comparable non-testing 
sites.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA B-2
4. How successful was 
the contractor in de￾signing and implement￾ing the small grant 
program? 
EEHR document re￾view
Informant interviews
Focus group discus￾sions
EEHR small grant recipients 
(including project objectives 
and activity reports)
Identify objectives of 
grants, and match them 
with recipient advocacy 
efforts and campaigns. 
Assess the application 
process as well as the 
systems of accountabil￾ity. Analyze the added 
value of the grants to 
NGO capacity. Assess 
prospect for sustainabil￾ity of NGO advocacy 
following the program
5. What were the 
challenges (internal 
and external) faced by 
the program and what 
are the lessons learned 
to date? 
Document Review
Informant interviews
Focus groups
Site visits to three re￾gional testing hospitals 
and surrounding urban 
and rural areas
EEHR documents and reports, 
M&E plans and reports, key 
stakeholders and implementing 
partners, citizens who interact￾ed with testing hospitals and 
other EEHR activities; popular 
press articles on health sector 
reform.
Aggregating data from 
all relevant stakeholders, 
partners, and beneficiar￾ies, and synthesizing 
their impressions of the 
health care environ￾ment, its challenges, 
needed reforms, and 
obstacles. Assess the 
degree of collaboration 
among all relevant ac￾tors, and their commit￾ment to health sector 
reform. Identify the de￾gree of administrative 
and political will among 
key stakeholders, and 
their relative influence. 
Assess level of internal 
teamwork and EEHR 
project efficiency, and 
the project’s interactions 
with and influence with￾in the operating context. 
Assess major assump￾tions, linked to a theory 
of change, of the exist￾ing health sector reform 
environment.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA C-1
ANNEX C: TEAM COMPOSITION
EVALUATION TEAM LEADER 
Tim Clary served as the Evaluation Team Leader for the second portion of the evaluation, with primary re￾sponsibility for overall conduct of the evaluation process and report. He was the team’s principal point of 
contact with USAID/Albania and assumed responsibility for coordinating evaluation activities. He also assist￾ed with the development and refinement of the revised work plan and evaluation methodology and lead 
the preparation and delivery of all presentations and briefings to USAID and other stakeholders. Dr. Clary
led key informant interviews and other data collection activities and served as the lead author of the revised 
evaluation report. Dr. Clary is an expert in the health sector, with over 14 years of experience. He holds a 
Ph.D. and an M.S. in Epidemiology and Geography from UCLA, an EMBA in Business Administration from 
the Instituto de Empresa in Madrid, an M.A. in Geography from UCLA, and a B.S. in Journalism from the 
University of Florida.
EVALUATION TEAM MEMBER
Lindita Çaçi served as Local Team Member for the evaluation. She assisted the Team Leader in the overall 
management of the evaluation. As a health care policy professional in Albania, Ms. Çaçi provided subject 
matter expertise and local country contextual knowledge, and she identified additional documents for re￾view and was particularly instrumental in arranging for meetings with interview candidates on short notice. 
The evaluation team drew on her expertise to design the evaluation methodology and data collection in￾struments as well as to inform subsequent analysis, conclusions, and recommendations. Ms. Çaçi conducted 
background research, assisted with the development and refinement of the instrument design and evaluation 
methodology, and participate in fieldwork. Ms. Çaçi also worked with the Team Leader to draft and edit the 
evaluation report and present key findings and recommendations to USAID Mission staff and other key 
stakeholders.
Ms. Çaçi brought both formal academic training and first-hand experience with the Albanian MOH to bear 
on the findings and recommendations of the evaluation. She has a M.Sc. in Health Institutions Management 
from the University of Montreal. Following an earlier stage of her career in the Economic Department of the 
MOH, she served as the Director of the Policy and Planning Department of the MOH for four years before 
relocating to Canada.
LOCAL TRANSLATOR AND LOGISTICS ASSISTANT
Laura Kolaneci served as the team’s local translator and logistics assistant. In addition to her role as a transla￾tor for the Team Leader, she provided logistical support for meetings, interviews, site visits, daily travel, and 
other administrative tasks as needed. Ms. Kolaneci’s role went beyond these activities. She also has Master’s 
level training in Health System Management, and she actively participated in all team sessions, beginning with 
the translation of interviews and focus groups, and continuing through to the subsequent discussion and 
analysis of findings with the two main technical members of the team.
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA D-1
ANNEX D: DOCUMENTS RE￾VIEWED
ADHS. 2010. Albanian Demographic and Health Survey: 2008-2009. Institute of Statistics, Institute of Public 
Health Tirana, Albania, and ICF Macro, Calverton, Maryland, US. March, 2010.
Bowers, Gerard and William Johnson. 2010. USAID/Albania Health Sector Review, January 23 – February 
10, 2010.
Chee, Grace and Joanne Jeffers. 2011. The Albania Health Sector Governance Study: Technical Brief. Abt 
Associates, Bethesda, MD. July 15, 2011.
Cook, Margaret, Mark McEuen, and Jan Valdelin. February, 2005. Primary Health Care Reform in Albania: A 
Pilot Project to Provide Evidence for Health Policy. Abt Associates, Partners for Health Reformplus 
Project.
EEHR. 2011. Enabling Equitable Health Reforms Project in Albania: First Annual Report. Abt Associates, Be￾thesda, MD. October 14, 2011.
EEHR. 2012. Enabling Equitable Health Reforms Project in Albania: Second Annual Report. Abt Associates, 
Bethesda, MD. October 15, 2012.
EEHR. 2011. Year One Work Plan (October 1, 2010 – September 30, 2011). Abt Associates, Bethesda, 
MD. October 14, 2010.
EEHR. 2011. Second Year Work Plan (October 1, 2011 – September 30, 2012). Abt Associates, Bethesda, 
MD.
EEHR. 2012. Third Year Work Plan (October 1, 2012 – September 30, 2013). Abt Associates, Bethesda, 
MD.
EEHR. 2012. Initiation of Hospital Improvement Plans: Space Programming. (Trip Report) Abt Associates, 
Bethesda, MD. July 30, 2012.
Fairbank, Alan and Gary Gaumer. 2003. “Organization and Financing of Primary Health Care in Albania: 
Problems, Issues, and Alternative Approaches.” Abt Associates, Partners for Health Reformplus Pro￾ject. April, 2003.
Hashem, Ahmad. 2012. Information Systems Development at Selected Albanian Hospitals: Technical Re￾port. April 20, 2012. Enabling Equitable Health Reform Project in Albania, Abt Associates, Bethesda, 
MD.
Hoey, Arthur. 2012. Trip Report: Initiation of Hospital Improvement Plans: Space Programming, June 25 – 
July 11, 2012. Enabling Equitable Health Reforms Project in Albania, Abt Associates, Bethesda, MD.
July 30, 2012.
Hsiao, William C. 1995. “Abnormal economics in the health sector.” Health Policy. Vol. 32: 125-139. 
Huff-Rousselle, Maggie. 2012. “The Logical Underpinnings and Benefits of Pooled Pharmaceutical Procure￾ment: A Pragmatic Role for our Public Institutions?” Social Science and Medicine. Vol. 75, Issue 9 
(2012) 1572-1580. 
IDRA Research and Consulting. 2013. Health Care System in Albania: A Formative Research with Consum￾ers to Increase Non-State Actors Engagement in Health System Governance. Enabling Equitable 
Health Reforms Project in Albania, Abt Associates, Bethesda, MD. May 8, 2013. 
Jeffers, Joanne and Ornela Palushaj. 2012. Increasing Non-State Actors’ Engagement in Health Systems Gov￾ernance, Technical Report. Enabling Equitable Health Reforms Project in Albania, Abt Associates, 
Bethesda, MD. March 22, 2012.
Kenny, Steve. 2013. Package of Hospital Services in Albania. Trip Report. Enabling Equitable Health Reforms 
Project in Albania, Abt Associates, Bethesda, MD. April 26, 2013. 
Kenny, Steve. 2013. Hospital Funding in Albania. Technical Report. Enabling Equitable Health Reforms Pro￾ject in Albania, Abt Associates, Bethesda, MD. September 25, 2013. 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA D-2
Kenny, Steve. 2013. Package of Hospital Services in Albania. Technical Report. Enabling Equitable Health 
Reforms Project in Albania, Abt Associates, Bethesda, MD. September 26, 2013.
Kongoli, Zyhrada. 2012. Analysis of Legal and Regulatory Framework for Health Facility/Hospital Autonomy 
in Albania. Enabling Equitable Health Reform Project in Albania, Abt Associates, Bethesda, MD. Oc￾tober 9, 2012. 
Li Bassi, Luca. 2013. Supply and Demand Management for Inpatient Pharmaceuticals. Equitable Health Re￾form Project in Albania, Abt Associates, Bethesda, MD. October 18, 2013. Enabling
Myers, Louise. 2012. The Albania Pilot Hospitals Baseline Survey. Technical Report. Abt Associates, Bethes￾da, MD. March 16, 2012.
O’Sullivan, Gael. 2013. Civil Society Engagement and Communication Strategy and Action Plan. (Technical 
Report) Enabling Equitable Health Reform Project in Albania, Abt Associates, Bethesda, MD. June 
25, 2013. 
Post, Susan E. Pritchett. 2013. The Albanian Health Sector Leadership Study. Technical Report. Enabling Eq￾uitable Health Reform Project in Albania, Abt Associates, Bethesda, MD. November 17, 2013. 
Purvis, George, Ainura Ibrahimova and Flora Hobdari. 2011. Albania Health Insurance Institute Review: Chal￾lenges and Opportunities. (Technical Report) Enabling Equitable Health Reform Project in Albania, 
Abt Associates, Bethesda, MD. July 15, 2011. 
USAID. 2010. Contract No.182-C-00-10-00104-00 with Abt Associates Inc. for the EEHR Project, in Alba￾nia, dated October 1, 2010, and subsequent modifications 1 through 4, with last amendment dated 
April 11, 2013.
World Bank. 2011. Out-of-Pocket Payments in Albania’s Health System: Trends in Household Perceptions 
and Experiences 2002-2008. Report No. 64803-AL. The World Bank, Human Development De￾partment Europe and Central Asia Regional Office. March 28, 2011. 
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA E-1
ANNEX E: INDIVIDUALS CON￾TACTED
ABT ASSOCIATES
1. Steve Kenny, International Consultant, Hospital Financing
2. Mark McEuen, Vice President, Abt Associates (telephone interview)
3. Louise Myers, International Consultant, Hospital Management
EEHR PROJECT 
1. Ervis Bregu, Manager for Hospital Information Systems
2. Blerina Dudushi, Manager for Engaging and Communicating with Non-State Actors
3. Mirela Cami, Policy Process/M&E Manager
4. Ornela Palushaj, Project Information Manager
5. Julian Simidjiyski, Chief of the Party, EEHR Project
6. Zamira Sinoimeri, Policy Senior Advisor
7. Filip Vila, Site Manager
HEALTH INSURANCE INSTITUTE (HII)
1. Astrit Beci, General Director, appointed 3 weeks prior to interview
2. Nora Horanlliu, Chief of Budgeting Department, formerly Chief of Development and Policies, Hos￾pital Directorate
3. Naun Sinani, Chief of Hospital Service Directorate, formerly Advisor
HEALTH REFORM IMPLEMENTATION SUPPORT GROUP (MEMBERS)
1. Din Abazaj, President Order of Physicians
2. Sabri Skenderi, President, Albanian Nursing Order
KORCA REGIONAL HOSPITAL
1. Entela Bardhi, Economist, Head of Hospital Biostatistics Office
2. Edlira Bode, Physician, Hospital Coordinator (for EEHR Project)
3. Loreta Bode, Physician, PHC, Health Insurance Institute, Regional Directorate
4. Vasilika Cuti, Specialist, HR Sector
5. Mirela Dhembi, ES Monitoring Specialist
6. Anila Dishnica, Head of Finance Office
7. Piro Dishnica, Specialist, M&E Sector
8. Doriana Frasheri, Head Nurse, Infectious Disease Unit
9. Netreta Gjoshe, Specialist, M&E Sector, Regional Public Health Institute Office
10. Ligor Golka, General Head Nurse
11. Stela Guci, Specialist, HR Sector
12. Bashkim Ibi, Physician, Head of ICU
13. Margarita Jazxhi, Nurse in Emergency Room
14. Edion Kollcinaku, Neonatalist, Head of Neonatology Unit
15. Rezarta Korance, Nurse at Cardiology Service
16. Para Kosta, Head Nurse, Emergency Ward
17. Entela Madhi, Head of Hostelry Office
18. Anastas Mitace, Head Nurse, Pediatrics Unit
19. Ylli Qirinxhi, Hospital Director
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA E-2
20. Edmond Remacka, Head of Maintenance Office
21. Ornela Ruco, Physican, Head of Infectious Disease Unit
22. Anila Sedo, Nurse, IPC Unit
LEZHA REGIONAL HOSPITAL
1. Ardinana Barbullushi, Lawyer
2. Nevruz Bare, Hospital Director
3. Vjollca Begaj, Secretary
4. Erjona Brungaj, Chief of Costing Service
5. Tonin Bushi, Chief of Finance
6. Suada Danaj, M&E Sector, Regional Public Health Department
7. Donina Doda, Head Nurse, Obstetrics Ward
8. Aida Gega, Statistics, Regional HII
9. Aferdita Gjoni, Site Coordinator (EEHR)
10. Bardha Gjoni, Physican, Quality Coordinator
11. Roza Hilaj, Chief of Pathology Ward
12. Edvin Hoxha, Head of Environmental Services
13. Denjola Kadija, Head Nurse for Hospital
14. Almir Keli, Head of M&E, Regional
15. Pranvera Lushi, Head Nurse, Intensive Care
16. Mark Marku, Chief of Pediatric Ward
17. Martin Marku, Chief of Infection Service
18. Teuta Marku, Head of Technical Department
19. Vladimir Martini, Chief of Policlinic
20. Taulant Mergjyli, Nurse, Emergency Ward
21. Angje Ndoci, Head Nurse, Surgery Ward
22. Silva Nikolli, Head of Human Resources, Hospital
23. Valentina Nikolli, Economic Director
24. Valbona Prenga, Chief of Emergency Ward
25. Tonin Rumija, Physican, Hospital Coordinator for EEHR Project
26. Valdete Sherri, Head Nurse of Pathology
MINISTRY OF HEALTH
1. Halim Kosova, Former Director of Maternity Hospital (15 years), Former Minister of Health (6 
months); current Member of Parliament
2. Jonela Leka, Specialist, M&E Sector, MOH
3. Petro Mersini, Director, Hospital Planning Directorate, MOH
4. Silva Novi, Head of Hospital Sector, Hospital Planning Directorate, MOH
5. Pellumb Pipero, Former Director of Policy and Planning, MOH; current MD at University Hospital
6. Klodian Rjepaj, Vice Minister of Health, Former Director of Institute of Public Health
7. Petraq Shtrepi, Head of M&E Sector, MOH
8. Ledia Xhafa, Specialist, M&E Sector, MOH
NATIONAL CENTER OF CONTINUING EDUCATION (NCCE)
1. Entela Shehu, Director 
NATIONAL CENTRE OF QUALITY, SAFETY AND ACCREDITATION OF 
HEALTH INSTITUTIONS (NCQSA-HI)
1. Professor Isuf Kalo, Director
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA E-3
QUEEN GERALDINE MATERNITY HOSPITAL
1. Aurora Bajraktari, General Head Nurse
2. Vera Beca, Microbiologist
3. Dhurata Boka, General Head Midwife
4. Maksim Gjoni, Deputy Director of Quality and Accreditation, and staff physician
5. Eda Hushi, Pharmacist
6. Adriana Kopani, Human Resources Specialist
7. Mirlinda Krasniqi, Health of Human Resources
8. Blenard Nonaj, Director 
9. Ervina Nuri, Psychologist/Social Worker
10. Marizeta Oili, Public Relations Specialist
11. Anxhela Palla, Attorney
12. Vera Pashollari, Head Midwife
13. Altin Pasko, Economic Director
14. Mimoza Pipero, Receptionist
15. Robert Qirko, Former Director (for 7 months in 2013), on-going role as staff physician
16. Anjeza Sadiku, Midwife
17. Lirie Shehaj, Head Nurse
18. Fuat Zhabjaku, Head of Technical Sector
19. Ermira Zhupa, Midwife
USAID
1. Jim Barnhart, Mission Director
2. David Cowles, Acting Mission Director, and Senior Private Enterprise Advisor
3. Marc Ellingstad, General Development Officer
4. Agim Kociraj, Health Specialist & COR for EEHR Project
5. Clare Masson, Program Officer
6. Alken Myftiu, Monitoring and Evaluation Specialist
7. Zhaneta Shatri, Deputy General General Development Officer (seconded to USAID/Iraq)
8. Laura Slaughter, Regional Contracting and Agreement Office
WORLD HEALTH ORGANIZATION
1. Vasil Miho, WHO Representative
WORLD BANK
1. Lorena Kostallari, Senior Operations Officer, World Bank
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA F-1
ANNEX F: STATEMENTS OF 
CONFLICT OF INTEREST
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA F-2
PEFORMANCE EVALUATION OF EEHR FOR USAID/ALBANIA F-3
U.S. Agency for International Development
1300 Pennsylvania Avenue, NW
Washington, DC 20523
Tel: (202) 712-0000
Fax: (202) 216-3524
www.usaid.gov