April 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Joseph M. Anderson and Carl Ulbricht, under contract to Social Impact. The Way Forward: End of Project Performance Evaluation of the Armenia Pension and Labor Market Reform Project i COVER PHOTO: PENSION REFORM IMPLEMENTATION COMPONENT, FEATURING PALM EXPERTS AND GOVERNMENT EMPLOYEES AT THE ROUND TABLE ENTITLED “PENSION REFORM PROCESS: SHIFTING TO THE FUNDED SYSTEM”, JUNE 2, 2012, GOVERNMENT SESSION HALL, YEREVAN, REPUBLIC OF ARMENIA ii THE WAY FORWARD: END OF PROJECT PERFORMANCE EVALUATION OF THE ARMENIA PENSION AND LABOR MARKET REFORM PROJECT USAID/ARMENIA June 2013 Contract # AID-RAN-I-00-09-00019 Task Order # AID-111-TO-12-00002 Evaluation of Five USAID/Armenia Activities DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS Acronyms.................................................................................................................................................................................................. 1 Executive Summary................................................................................................................................................................................. 3 Evaluation Purpose & Evaluation Questions.................................................................................................................................... 7 Project Background ................................................................................................................................................................................ 9 Evaluation Methods & Limitations ....................................................................................................................................................12 Findings, Conclusions & Recommendations...................................................................................................................................14 Evaluation Question 1.....................................................................................................................................................................14 Findings..........................................................................................................................................................................................14 Conclusions..................................................................................................................................................................................18 Recommendations......................................................................................................................................................................18 Evaluation Question 2.....................................................................................................................................................................18 Findings..........................................................................................................................................................................................18 Conclusions..................................................................................................................................................................................20 Recommendations......................................................................................................................................................................20 Evaluation Question 3.....................................................................................................................................................................20 Findings..........................................................................................................................................................................................20 Conclusions..................................................................................................................................................................................24 Recommendations......................................................................................................................................................................24 Evaluation Question 4.....................................................................................................................................................................25 Findings..........................................................................................................................................................................................25 Conclusions..................................................................................................................................................................................28 Recommendations......................................................................................................................................................................28 Evaluation Question 5..................................................................................................................................................................... 28 Findings..........................................................................................................................................................................................28 Conclusions..................................................................................................................................................................................30 Recommendations......................................................................................................................................................................31 Gender issues....................................................................................................................................................................................31 Annexes...................................................................................................................................................................................................32 Annex I: Evaluation Statement Of Work...................................................................................................................................33 Annex Ii: Evaluation Methods And Limitations ........................................................................................................................37 Annex Iii: Data Collection Instruments......................................................................................................................................40 Annex Iv: Sources Of Information: List Of Persons Interviewed.......................................................................................43 Annex V: Contract Statement Of Work: Goals, Tasks, Expected Results.......................................................................46 Annex Vi: Palm Project Performance Monitoring Plan ..........................................................................................................53 Annex Vii: Disclosure Of Conflicts Of Interest.......................................................................................................................59 iv Acronyms AMPI Armenian Mandatory Pension Information System CBA Central Bank of Armenia CDA Central Depositary of Armenia CEE Central and Eastern European Countries COP Chief of Party DEC Development Experience Clearinghouse DH&SR Democracy, Health and Social Reform FBCF Financial Banking College Foundation FED Finance for Economic Development FY Fiscal Year GOAM Government of Armenia IT Information Technology IOPS International Organization of Pension Supervisors ISP Internet Service Providers IT Information and Technology LIFE Livelihood Improvement through Functional Employment M&E Monitoring and Evaluation MoF Ministry of Finance MLSA Ministry of Labor and Social Affairs MLSI Ministry of Labor and Social Issues NSS National Statistical Service NGO Non-Government Organization OECD Organization for Economic Co-operation and Development PALM Pension and Labor Market Reform Project PAYG Pay As You Go PMP Performance Management Plan 1 PPL/LER Bureau for Policy, Planning, and Learning’s Office of Learning, Evaluation, and Research PRIMB Pension Reform Implementation Management Board PRIP Pension Reform Implementation Project PRIWG Pension Reform Implementation Working Group PSAC Pension System Awareness Center PW Programmed Withdrawal RA Republic of Armenia SPSS Social Protection Systems Strengthening Project SESA State Employment Service Agency SOW Statement of Work SRC State Revenue Committee SSSS State Social Security Service STTA Short Term Advisor ToR Terms of Reference UITE Union of Information and Technology Enterprises USAID United States Agency for International Development UTPRS Unified Income Tax and Personified Record Keeping System YPOC Youth Professional Orientation Center 2 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This report is an end-of-project performance evaluation of the Armenia Pension and Labor Market Reform (PALM) project. The major objectives of the PALM Project were: (1) to build the Government of Armenia (GOAM) institutional and human resource capacity to design and implement a multi-pillar pension system and (2) to improve labor markets and employment through promoting new linkages at the local level between the private sector, local authorities, and vocational training institutions. Five major questions were addressed in this evaluation: 1. What do stakeholder groups see as the achievements of the program? 2. To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? 3. To what extent has the support given to the Government of Armenia (GOAM) to build the foundation of Multi-Pillar Pension System contributed to the preparedness of the system to be implemented? 4. To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency (SESA) to match labor market demand and to develop/implement active labor market policies? 5. How adequate was the project Performance Management Plan (PMP) in reporting on project progress? PROJECT BACKGROUND Since 2000, with the support of USAID, the GOAM has been engaged in efforts to improve the existing Pay-As-You-Go (PAYG) universal pension system and, beginning in 2006, to design a new, funded pension savings system. In November 2008, the government adopted a Pension Reform Program that recommended a multi-pillar approach to reforming the country's pension system. To provide continuing support for the pension reform and its implementation, in 2009 USAID designed a third project, the Pension and Labor Market Reform Project (PALM). USAID also has supported programs to improve the Armenian labor market and employment since 2000. The PALM Project included a component to support efforts to improve the Armenian labor market and provide improved employment opportunities. The PALM Project was implemented from November 2009 through September 2012. The budget was $5.4 million, of which $5.1 million had been spent through September 25, 2012. EVALUATION METHODS AND LIMITATIONS The evaluation methods included document review, key informant interviews, and site visits. The evaluation had several limitations. All of the principal counterparts and PALM Project staff who were interviewed were identified by USAID mission staff or by PALM Project staff. Given the short time in country (2 weeks) the evaluation team was not independently able to identify individuals who might be knowledgeable and more independent of the PALM Project. 3 Given lack of data and the short time available, it was not possible to analyze employment data for specific regions where labor market and employment programs were implemented to determine if they had an effect on employment. FINDINGS, CONCLUSIONS & RECOMMENDATIONS EVALUATION QUESTION 1 Findings The PALM project was engaged in a wide range of activities which were valued by the relevant counterparts. These activities included, inter alia, facilitating debate on pension reform, assistance in drafting and revising legislation, submitting concept papers and reports, developing actuarial models and analysis, and assistance in developing the Personified Record Keeping System. In addition to these achievements and contributions, some Armenian government counterparts perceived deficiencies in the project. Conclusions The PALM Project had many significant achievements that were recognized and appreciated by counterparts. It made a significant contribution to the advancement of pension reform in Armenia in numerous areas. It contributed in recognizable ways to the activities of the SESA. Communication among stakeholders regarding PALM management, budget, and priorities was sometimes imperfect, leading to increased transaction costs and frustration. Recommendations USAID staff and implementation partners should re-double efforts toward clear communication with counterparts at every level from the outset of any future project in the pension area concerning the purpose and mandate of the project, the priorities, and the constraints that exist in terms of budget and other resources, laws, rules and regulations. EVALUATION QUESTION 2 Findings PALM played an important role in supporting the Ministry of Labor and Social Issues (MLSI), State Revenue Committee (SRC), Ministry of Finance (MoF), and Central Bank of Armenia (CBA) in drafting and revising pension laws and other legal acts to establish the legal framework of the new multi-pillar pension system in Armenia. Additional legal resources and attention will be needed to refine the existing legal framework. Conclusions The contribution of the PALM Project to the development of pension legislation has been significant. The legal basis for each of the four pillars of the proposed pension system has been established and most required regulations have been drafted. Recommendations The follow-on pension reform project should have a strong, national, and full-time pension legislation expert. 4 EVALUATION QUESTION 3 Findings The PALM Project has provided support for the Multi-Pillar Pension System in numerous areas. This is recognized by the GOAM counterparts that are involved with pension reform. Conclusions The support provided by the PALM Project has been very significant and constructive and has provided the foundation for the Multi-Pillar Pension System to be implemented. Although the MLSI has the formal leadership role in the pension reform effort, it has not developed an adequate staff below the senior level to effectively carry out the continuing analysis of pension reform issues and direction of the pension reform. Recommendations USAID should work with MLSI to help develop staff below the senior level to serve as active counterparts to the follow-on pension reform project, including a skilled pension modeling expert. Important analytical issues are still unresolved. The follow-on pension reform project should continue and expand pension public awareness and public education programs. EVALUATION QUESTION 4 Findings The PALM Project provided considerable support to the SESA and to the Youth Professional Orientation Centre (YPOC). While the senior staff of SESA had a positive view of the PALM Project and USAID support, they expressed frustration that the PALM Project was not able to do more. This was an area in which improved communication about the mandate, role, funding, and capacity of USAID and PALM might have been helpful. CONCLUSIONS The PALM Project engaged in many activities and tasks designed to enhance the ability of the SESA to match labor market demand, to develop/implement active labor market policies, and to support the programs of the YPOC. A number of ongoing programs and a significant level of capacity were developed in active labor market policies. RECOMMENDATIONS If in the future USAID determines that it should develop significant labor market or employment programs in Armenia, an evaluation component should be built into the projects from the beginning, including the collection of relevant data before the beginning and as the project is being implemented. This should be done in conjunction with the counterpart and as part of the technical assistance. EVALUATION QUESTION 5 Findings The Performance Management Plans (PMPs) for PALM, including the final PMP, correspond well to the Expected Results in the contract Statement of Work (SOW). The PALM Project, for both Components 1 (implementation of the pension system) and 2 (reform of SESA), demonstrated significant accomplishments and progress on indicators and accomplished most of the results specified in the PMP. However, the Component 2 Goal, Task 1, and most of the associated Illustrative Activities are all vague. Component 2 was carried out by engaging in several activities that could be expected to promote 5 improved labor markets and facilitate employment, but no measurable criteria for either improved labor market outcomes, or even reform of SESA, were specified in the SOW. Even given these measurement difficulties, it is clear that PALM did not realize all of its planned outcomes and impacts for either component. This raises the broader question of how appropriate PALM’s PMP results and indicators were in measuring progress firmly linked to achieving its outcomes and impacts. There was no indication in interviews with USAID staff or PALM project staff that these stakeholders felt the project did not achieve its main outcomes. Moreover, there is no written indication that the SOW or work plan of the project was changed during the project period to account for new circumstances or to adjust PALM’s two goals and their related impacts. Conclusions The project PMPs were useful in measuring and communicating project progress to counterparts, to USAID, and to outside evaluators. They demonstrated in a clear and timely fashion that the PALM Project attained many of the specific results called for in the SOW. These significant achievements, while reported adequately by the PMP, did not lead fully to the PALM outcomes desired, suggesting that the linkages between outputs and outcomes were not sufficiently specified in the PALM results framework. Recommendations Care should be taken in developing the SOW for future USAID projects to identify linkages between results and expected outcomes in as concrete a way as possible and provide some guidance concerning how the contractor and the USAID mission at the outset of the project may be able to develop measures of accomplishment. 6 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE This report is an end-of-project performance evaluation of the Armenia Pension and Labor Market Reform (PALM) project. The major pension system institutions are still under development, so there are no measurable outcomes in terms of changes in individual well-being to measure. In concept, some labor market outcomes could be measured, but the project interventions are small relative to the much larger structural and macroeconomic forces that have determined employment and labor market outcomes during the project period. Consequently, the evaluation describes and assesses the development of institutions and institutional underpinnings, such as policies, laws, regulations, training, public education, and analytical capabilities, during the project period. Development of these institutions, it is hoped, will ultimately help achieve the objectives of improving the economic and social well-being of the Armenian population and the Armenian economy. September 2012 marked the conclusion of the third major USAID project focusing on pension reform and labor market improvement.1 Going forward, the pension reform efforts will be supported under three separate projects. One project, which will support the tax authorities and continue the development of the Information Technology (IT) system for the reformed pension system, began in fall 2012. Two other new projects, focusing on financial institutions and pension system implementation, will begin in spring 2013. Consequently, this is a good time to take stock of what has been accomplished during the 2009–2012 period to assess what has been achieved, whether the USAID design and approach was appropriate for Armenia, and how effective were the interventions carried out by USAID and its implementing partner. USAID may use the findings of the performance evaluation to ascertain whether the project design and approach was appropriate and effective for Armenia and should be continued or modified, whether the overall efforts and activities of its implementing partners were effective, what specific activities and approaches were effective or ineffective, and what lessons can be learned that can be applied to follow￾on projects, both on the part of implementing partners and USAID. The primary audience for this evaluation is the staff and leadership of the USAID mission in Armenia. The evaluation should be designed to help them in providing guidance to future implementing partners and in facilitating improved communication and interaction between the mission, the implementing partners, and the Armenian government counterparts. A secondary audience is the staff and management at USAID/Washington, particularly the Bureau for Policy, Planning, and Learning’s Office of Learning, Evaluation, and Research (PPL/LER). The findings and conclusions of the evaluation could potentially be used to inform and guide the development of new projects, to provide guidance that may help improve communication between USAID and implementing partners and Armenian government counterparts, to inform and guide the continuing design of the IT system, and to assess the need for public relations/public education/training. . 1 USAID has provided social and humanitarian assistance programs to Armenia for more than 18 years. Previous pension reform and labor market assistance was provided under the Armenia Social Transition Project (2000–2005) and Social Protection Systems Strengthening Project (2007–2009). 7 Examples of specific decisions that could be informed by the evaluation include the design of the USAID/Armenia management structure for ongoing pension reform assistance; specific aspects and roles of implementing partners in the design and development of the IT system; how much additional public relations, public education, and training should be carried out; how specifically that should be done; and what models and approaches have been effective. EVALUATION QUESTIONS Five major questions were addressed in this evaluation in response to the evaluation SOW: • What do stakeholder groups – the Ministry of Labor and Social Issues (MLSI), the Central Bank of Armenia (CBA), the Ministry of Finance of Armenia (MoF), State Revenue Committee (SRC), State Employment Service Agency (SESA), Pension System Awareness Center (PSAC) Foundation, Youth Professional Orientation Center (YPOC) non-government organization (NGO) – see as the achievements of the program? • To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? • To what extent has the support given to the Government of Armenia (GOAM) to build the foundation of Multi-Pillar Pension System contributed to the preparedness of the system to be implemented? • To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency (SESA) to match labor market demand and to develop/implement active labor market policies? • How adequate was the project Performance Management Plan (PMP) in reporting on project progress? 8 PROJECT BACKGROUND From the Soviet Union, Armenia inherited an unfunded Pay-As-You-Go (PAYG) universal pension system. The Armenian pension system is both inadequate and financially unsustainable because of the structure of the current system and its aging population. The pension system is characterized by a deteriorating system dependency ratio (the share of beneficiaries to contributors), low and inadequate benefits, relatively low retirement ages, high tax evasion, lack of reliable individual records, and lack of a strong relationship between contributions and benefits. With the support of USAID, since 2000 the GOAM has been engaged in efforts to improve the existing PAYG system and, beginning in 2006, to design a new, funded pension savings system. In November 2008, with assistance from the USAID Social Protection Systems Strengthening Project (SPSS), the government adopted a Pension Reform Program that recommended a multi-pillar approach to reforming the country's pension system. By February 2009, four pension laws to implement the reform had been drafted and presented to the government by the Pension Reform Implementation Working Group (PRIWG). The multi-pillar pension system outlined in the pension reform framework included: • "0" Pillar – A non-contributory minimum social pension at 80% of the basic pension for those with less than 10 years of insured service. • 1st Pillar – Mandatory public pension system based primarily on years of service (minimum 10 years) without a link to wages and funded from the general budget. Employees will pay 26% of wages as income and social contribution taxes, known as the unified tax. The 1st Pillar consists of two parts: 1) a flat basic pension, which is the basis for all other benefits, and 2) an amount reflecting years of contributions prior to the reform. • 2nd Pillar – Mandatory individual retirement savings system financed by contributions of 10% of wages, 5% paid by workers and 5% by the state, up to a ceiling. • 3rd Pillar – Voluntary individual retirement savings system. The proposed Pension Reform Program originally called for the introduction of a funded pillar in 2010 coupled with increases in state-funded pensions. USAID recognized that the proposals needed careful assessment and continuing support. USAID also recognized that a new funded pension system would require significant infrastructure development and transparency, as well as investment institutions and policies that adhered to international standards and best practices. To provide continuing support for the pension reform and its implementation, USAID designed a third project in 2009, the PALM Project. The fiscal pressure for pension reform in Armenia is linked to high and increasing expenditures resulting from deteriorating system dependency ratios. According to the National Statistical Service, in 2008 Armenia had about 1.2 million economically active people (labor force), but social insurance contributions were regularly paid by 482,000 workers who supported 522,700 pensioners. In November 2012, the Head of the State Social Security Service (SSSS) estimated, that there were about 500,000 beneficiaries of the PAYG system and about 455,000 contributing employees. USAID also has supported programs to improve the Armenian labor market and employment since 2000. Even during a period of relatively rapid economic growth before 2009, growth was uneven in the regions and across various economic sectors. For many years Armenia has experienced “job￾less” growth. As a result, the economic growth yielded only modest gains for the population in terms of increased employment and earning power. 9 In 2009 Armenia was hit hard by the global economic crisis, resulting in dramatic increases in the number of unemployed "working poor" and those in vulnerable employment. USAID therefore wanted to design and provide continuing support for efforts to improve the Armenian labor market and provide improved employment opportunities, particularly to try to help mitigate the 2009 economic crisis. Under the SPSS, SESA began to move from passive labor market programs to active programs. The PALM Project included a component to support these employment policy and program reforms. The ultimate objective of the PALM project is to assist individuals, households, and communities to better manage social risks or needs, such as those arising from old age, disability, death of the bread￾winner, unemployment, and to improve labor market functioning and employment opportunities. The project aimed to accomplish these objectives by: 1. Building institutional and human resource capacity to design and implement a multi-pillar pension system that will provide an adequate income for its elderly and disabled population and that is fiscally sound and sustainable with a reformed current PAYG scheme and a newly created private retirement savings system, and 2. Improving labor markets and employment by promoting new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps and by improving the country’s workforce competitiveness. The components, tasks and expected results of the PALM Project from the contract Statement of Work (SOW) are described in Annex IX. The PALM Project PMP is shown in Annex X. Comparison of Annex IX and Annex X shows that the PALM Project carried out almost all of the originally proposed tasks with no major changes. The most important change was that the new multi-pillar pension system was not implemented in 2010, as proposed in 2009. It is now being implemented in 2013 and 2014. The ultimate beneficiary population of the project will be the people of Armenia, and particularly the workers and their families who will enjoy more adequate and secure social security benefits and improved employment opportunities. The immediate beneficiaries are the agencies of the GOAM that are responsible for formulating and implementing pension and employment policies. The project covers all of Armenia. All public and private sector workers will be covered by the new pension system. In particular, employment service centers have been assisted in all 10 marzes (provinces). Pension reform public education programs and roundtables have been conducted in Yerevan and many other marzes. The developmental logic for support for pension reform includes several elements. The pension system is an important element of the social protection system of the country and vital to the welfare of the population. However, the existing system is inadequate, inefficient, and unsustainable. It is a major source of fiscal instability. Pension expenditures were 2.6% of GDP in 2007. Before the 2008 election, benefits were increased 60%, raising expenditures to 3.8% of GDP. Pension expenditures were expected to reach 4.3% of GDP in 2012. At these rates of increases, the pension system is not sustainable and is a major cause of the country's high and rising budgetary deficit. In view of the difficulty of collecting taxes, the GOAM may have to freeze increases to the already low budgeted health and education social expenditures. A funded pension system with individual contributions that workers consider to be savings may reduce pressure on wages and labor costs, thereby increasing Armenian competitiveness. The accumulation of funded pension accounts can contribute to capital formation and the development of financial institutions. The development logic for labor market and employment programs also has several elements. Unemployment in Armenia is high, resulting in loss of incomes and waste of human resources. The GOAM is aware that decisive and coordinated actions are needed to avert social conflict that could 10 lead to political instability. USAID and the GOAM believed in 2009 that labor market interventions were necessary to help the government and the most vulnerable individuals mitigate the economic and social consequences of the global economic crisis through skills enhancement programs, improvements in the efficiency of the labor market, and the implementation of active employment measures. 11 EVALUATION METHODS & LIMITATIONS METHODS The methods used in this evaluation included document review, key informant interviews, and site visits. Specifically, this performance evaluation comprised of: • Review of the contract SOW, the PALM Project PMPs (notably, the PMP of the Final Report), the PALM Quarterly Reports, and the PALM Final Report; • Interviews with principal counterparts, PALM project staff, USAID staff, and others • Site visits to SESA, YPOC, the Financial Banking College Foundation (FBCF), and others The PALM evaluation project was conducted from October 2012 to March 2013. The evaluation team conducted research and fact-finding in Yerevan, Armenia, from October 28 to November 10, 2012, primarily through interviewing former PALM Project staff, GOAM and non-government counterparts, and USAID/Armenia mission staff as well as site visits to the YPOC, FBCF, AM Partners Consulting Company, PSAC Foundation, and SESA. An initial briefing with USAID mission staff was conducted October 29, 2012, and an exit briefing and presentation of preliminary findings and recommendations was conducted November 9, 2012. In total, 32 interviews were conducted, involving eight PALM ex-staff and officials from 14 counterpart organizations. Selection of the interviewees was based on recommendations from the USAID Mission: the evaluators were given a list of contacts to enable them to arrange interviews. LIMITATIONS All of the principal counterparts and PALM Project staff who were interviewed were identified by USAID mission staff or by PALM Project staff. Given the short time in country (two weeks), the evaluation team was not able to independently identify persons who might be knowledgeable and independent of the PALM Project. All efforts were focused on interviewing PALM ex-staff and counterparts and, given that the combined total of staff and counterparts was quite large, the evaluators were kept busy conducting the 32 interviews within the time available. Evaluations based on interviews with a selected group of interested participants in a project are subject to two forms of bias: recall bias and selection bias. We did not consider recall bias to be a serious problem because the activities and events being described occurred in the recent past. The PALM Project ended in September 2012 and the interviews were conducted in late October and November 2012. Some of the PALM project staff are now working on the follow-up tax reform project. Armenia government counterparts are all still engaged in pension reform or labor market activities. Selection bias was mitigated by interviewing participants who had different perspectives on the same issue, such as government counterparts, PALM Project staff, and USAID staff, comparing their answers, and returning to interview the same person when discrepancies appeared. Selection bias was also mitigated by comparing versions of past events with the Quarterly Reports written at the time and the 12 exit reports of consultants who had been involved in activities about which differences of opinion emerged. Two key Armenia government counterparts were not interviewed because they were unavailable during the time the evaluation team was in Armenia. These were Artem Asatryan, Minister of Labor and Social Issues, and Nerses Yeritsyan, Deputy Chairman of the CBA. The Chairman of the CBA was also not interviewed. Senior staff of the MLSI and of the CBA closely involved with the pension reform were interviewed, as was a member of the Board of the CBA who served on the Pension Reform Implementation Management Board (PRIMB). Evaluation questions 2, 3, and 4 seek an assessment of the project’s impact in specific areas; the phrase “to what extent did the project contribute…” is used twice. The evaluators felt that this called for a qualitative rather than quantitative assessment. Although in certain cases specific figures and lists of activities could be included as evidence, it was not possible to put a numerical value on the project’s achievements. Thus in responding to the questions, the evaluators have tried as much as possible to qualify the extent of impact. Given the short time available and lack of data, it was not possible to analyze employment data for specific regions where labor market and employment programs were implemented to determine if they had a measurable effect on employment. In recognition of the many factors that determine employment conditions in any given area, such an analysis would require collection of considerable local data for many different localities and the use of multivariate statistical techniques to analyze the data. (One of the recommendations of this evaluation is that if USAID is to support major employment programs in the future, such an evaluation be built into the project at the beginning, including the collection of relevant data before the beginning and as the project is being implemented.) It was not possible to interview employers to determine if they used the PALM Project-supported programs. All information about labor market programs came from interviews with PALM Project staff and GOAM agency heads. 13 FINDINGS, CONCLUSIONS & RECOMMENDATIONS EVALUATION QUESTION 1 What do stakeholder groups – the Ministry of Labor and Social Affairs (MLSA), the Central Bank of Armenia (CBA), the Ministry of Finance of Armenia (MoF), State Revenue Committee (SRC), State Employment Service Agency (SESA), Pension System Awareness Center (PSAC) Foundation, Youth Professional Orientation Center (YPOC) non-government organization (NGO) – see as the achievements of the program? FINDINGS Major achievements of the PALM Project identified by stakeholders were: • Brought counterparts together in a more informal setting than the Pension Reform Implementation Management Board (PRIMB), facilitating communication and coordination and acting as an “honest broker” among sometimes competing views. • Promotion of the reform and development of the overall GOAM IT system. Up to the present there has been no data connectivity among GOAM government agencies. The PALM IT consultants and their counterparts and other contractors are designing and developing data exchange procedures and mechanisms that will apply broadly to GOAM functions, including pensions, social assistance, tax collection and reporting, and others. • Assistance drafting and revising legislation and regulations. The PALM Project began in November 2009. Its first key achievement was the promotion of intensive review and discussion of five laws establishing and regulating a new multi-pillar pension system on State Pensions, on Fully Funded Pensions, on Personified Record Keeping System, on Unified Income Tax, and on Investment funds. These laws were adopted by the National Assembly in December 2010. After the adoption of the main laws, the GOAM began drafting laws and regulations related to different aspects of the new multi-pillar pension system. The PALM Project led discussions and drafted 40 regulations and by￾laws. (Laws and regulations drafted or assisted by PALM are listed in Annex IX. PALM support for legislation is discussed further under Question 2.) • Concept papers and reports on pensions and youth employment issues. The PALM team developed policy papers on different aspects of the multi-pillar pension system that provide analysis of policy impacts and policy recommendations and highlight possible risks and challenges. PALM provided a review of international practice on providing guarantees within the Pillar II, mandatory, defined contribution (DC) private pension systems. PALM examined the supply of investable instruments existing in Armenia, comparing the experience of other countries with developed capital markets. PALM provided technical assistance for improving the privileged pension system of Armenia by providing a comprehensive review of international experience and by collecting information on privileged pension systems of foreign countries, including former socialist countries, and proposals for improving the privileged pension policy. PALM provided an analysis of the design of the payout phase of the system. PALM also developed an important concept paper on youth professional orientation and careers. 14 • PALM provided education, training, information for counterparts and staff, such as examples of “Best International Practice” or practices of specific other countries (e.g., Estonia, Serbia, Russia, Chile). PALM provided numerous reports and briefings to senior counterparts and staff on international practice in pension systems and labor market policies. The PALM project brought a Chilean government official who was a leading proponent and designer of the multi-pillar Chilean pension system to Armenia to provide seminars and meet with Republic of Armenia (RA) government officials. The Chilean pension reform has served as a model for pension reforms in many countries, and the visit of this official was reported to be highly informative and motivational. • PALM organized and conducted a study tour to Serbia for staff of the SESA and YPOC, which was appreciated. (Other examples of training are provided in Question 3.) • Development models and adaption of existing actuarial models and analysis, modeling and simulation, projection of pension systems, and analysis of specific issues (e.g., pension payout scenarios.) PALM Project experts assisted the GOAM by providing comprehensive estimates and projections for reform of the pension system. In addition to the existing PROST model (developed by the World Bank) and the SPSS model (developed by the USAID SPSS Project), the PALM Project developed two new actuarial models, the New Statistical Method of Pension System and the Voluntary Funded III Pillar Model, in April 2011 in order to complete the modeling and projection tasks. These new models allow projections of various economic, labor, and financial scenarios in the future and thus contribute to informed decision making. Two expert actuaries revised the existing models and developed the new models, provided training, and produced a manual on “Statistical Model of the Pension System.” A PALM national expert continued to enhance the models and provide training. • Training in use of models for MoF, CBA, and MLSI staff. PALM project, through the international experts and with active support of local experts, provided two comprehensive training programs on the new actuarial models to staff from the MLSI, MoF, the Central Bank, as well as actuaries from the banks, insurance companies, and the Actuarial Mathematics department of the Yerevan State University. PALM staff reported that the MoF and CBA counterparts gained good proficiency in use of the models, but that MLSI staff did not. • Concept, functional specification, data exchange specification for Personified Record Keeping System and other IT assistance (SRC, MoF). PALM assisted the GOAM to implement the Unified Income Tax and Funded Contribution Personified Record Keeping System (UTPRS), covering employees’ personal Appreciation of PALM There was widespread appreciation of the work of the PALM project, as evidenced by these quotes: “Initial PALM funding helped us to survive” ~PSAC Foundation Representative “I am 90-100% satisfied with PALM. If PALM hadn’t existed, CBA would have found another source of assistance but that would have taken time and money. The advantage of PALM was that it had an already specialized team and that there was experience in USAID.” ~Ex-Board Member of CBA “Flexibility of projects can be a problem in a fast-moving reform process. Nevertheless the project was efficient.” ~International Department Representative, MLSI 15 and financial information collection, taxes and pension contributions, information accessibility and management (see Question 3). • Public opinion and public awareness surveys. In order to develop and implement its outreach and public education proposals and activities, the PALM project initiated and funded the Baseline Survey of the General Population’s Awareness of the Government’s Proposed Pension Reform (see Question 3). • Public relations, public awareness, public education programs, partners, and regional outreach initiatives. PALM, in cooperation with counterparts and implementing partners, made significant public awareness efforts: visiting different Marzes in Armenia and meeting different target groups, providing presentations, and communicating information and messages that are important to deliver to the public. PALM supported the new PSAC Foundation and assisted the Foundation in organizing the First Annual Pension Conference. The RA Prime Minister and other government officials presented the achievements and challenges of the pension reform to the media and audiences from the public and private sector (see Question 3 for more discussion of PR /PE). • Capacity building – especially at MoF, CBA, YPOC, PSAC and other Public Relations institutions. PALM organized more than 50 policy discussions, trainings, and retreats to help counterparts prioritize the tasks within the pension reform process, to set up deadlines for each activity, to address challenges, to discuss urgent and difficult topics of the pension reform, and to provide extensive advisory assistance to counterparts based on international best practice. PALM provided extensive training in the use of the actuarial models (see Question 3). • Renovation of SESA headquarters offices. SESA asked PALM’s assistance to renovate the SESA headquarters offices. SESA staff reported that this renovation improved morale and staff efficiency by improving working conditions for about 40 headquarter staff. In addition to these achievements and contributions, which were mentioned by the relevant sectoral counterparts and were not contradicted in any of the interviews, some Armenian government counterparts perceived deficiencies in the project. Deficiencies specifically cited included: • Communication concerning the work plan, proposals from counterparts, commitments. In some cases counterparts reported being frustrated that USAID and the PALM Project could not or would not perform specified activities or respond to requests. Often this occurred either because the budget did not have sufficient resources to meet the request or the request was outside or inappropriate for the scope of the project. Quality of the Foreign Experts and Their Reports Some criticisms of the project were focused on the output of the short term experts: “In one or two cases it would have been good to have more experienced foreign experts. Fewer experts but better quality would be my recommendation. Overall, 85% of the time the advice was OK. But sometimes it would be good to know why certain countries adopted a certain policy.” ~Capital Asset Management Representative “Sometimes expectations are unrealistic – e.g. wanting better documents from the international experts, with more detail on the local solution. But the reality is that local experts (including MLSI) then have to work on the document.” ~International Development Representative, MLSI “Experts were not of high quality. E.g, they would give the example of Romania, which was hardly inspiring given that it’s a very corrupt country.” ~State Social Security Service Representative 16 • Appropriateness and expertise of international experts, cited by both PALM staff and counterparts. Some counterparts reported that some international experts contracted by PALM had limited expertise or less expertise in the given subject matter than counterparts or PALM local staff. Respondents reported that thanks to USAID and other donor projects, many Armenians now have considerable expertise in pension reform. Armenians involved in pension reform, it was reported, do not need general knowledge of pension reform issues or concepts from international experts; rather, they are looking for specific expertise that they lack. Two specific example are as follows: - Some counterparts reported that some Short Term International Experts took a long time to become familiar with the Armenian situation. By the time they understood Armenia and the requirements for the pension reform, their time was up. - One international expert provided a work product that was perceived by some counterparts to not be useful and resulted in waste of counterpart time. This resulted in part from incorrect phasing of the project. At the beginning of the PALM Project, a specification for an IT system was delivered before the legislative basis for that system was developed adequately. When the legislative issues were resolved, the IT system was inappropriate. Leadership of the State Social Security Service (SSSS) cited specific frustrations with PALM: 1. Early in the PALM Project, an expert provided a report describing development of the technical specification of a Personified Record Keeping IT system. The proposed design did not apply to the Armenian legal or institutional framework. Armenian officials spent much time discussing the proposal but accomplished nothing. 2. Assistance to the SSSS Pension Management System was not adequate. Software designed by a PALM expert was deficient. “In designing the Project Management System, the consultants used IT security standards which were 10 years out of date.” SSSS did not use the work product. In 2011, SSSS wanted to do a Pension Management System IT project design but did not have enough internal expertise. They requested funds from PALM to hire local experts, but were told PALM had no resources. SSSS staff drafted a Terms of Reference (ToR) for the IT system, and they used state resources to design it. They are developing and implementing the pension management system software with World Bank support. 3. Furniture and equipment for 52 SSSS offices is old and must be replaced. PALM provided furniture for one office. SSSS requested more but the request was refused. 4. Fifty-two managers must be trained; PALM project has not provided assistance for training office managers. Leadership of the State Employment Services Agency (SESA) expressed a positive view of PALM and USAID staff, yet also expressed frustration that due to budget reductions and activities had been agreed upon in the project work plan (in their opinions) had to be reduced or eliminated. They attributed this to the fact that funds were reallocated from labor market interventions to pension reform. According to officials, activities that were reduced or curtailed included: (1) elimination of the 2012 employer survey, (2) reduction of activities to improve the GORTS employment database, (3) reduction in the number of Job Clubs supported, (4) reduction in staff training and monitoring and evaluation, (5) elimination of support for translation and printing of a report, and (6) elimination of a project to adapt 20 work places for disabled workers. After the long-term expert on labor market issues left in 2011, little was done. 17 CONCLUSIONS The PALM Project accomplished many significant achievements that were recognized and appreciated by counterparts. It made a significant contribution to the advancement of the pension reform in Armenia in numerous areas, including bringing counterparts together in productive ways, reforming the GOAM IT system; strengthening the conceptual basis; understanding of pension reform issues, legal, and regulatory framework; information technology; capacity building; and public awareness and understanding. It contributed in recognizable ways to the activities of the SESA. The PALM Project efforts devoted to labor market and employment reform in terms of staff and resources were more limited, and the overall achievements were less significant. Interviews with RA government counterparts, as well as with PALM staff and USAID staff, suggest that dissatisfaction on the part of some SSSS and SESA officials may have arisen from misunderstandings on the part of some counterparts concerning what the PALM project was able and mandated to do. It is noteworthy that, for the most part, RA counterparts in policy-making institutions had a very positive view of PALM, whereas some in implementation agencies expressed dissatisfaction regarding some lack of clarity between stakeholders about project budgets, priorities, and allocation of responsibilities. There is often a tendency in many governments for there to be tension between those formulating broad policy and proposing legislation and regulations and those responsible for implementing programs on the ground. It is also possible that donors and foreign experts find it more interesting to deal with broad and far-reaching policy and institutional changes than with mundane day-to-day tasks such as providing furniture for offices or printing reports. There is some difficulty finding and recruiting experts on short notice for pension related assignments in Armenia that have the proper knowledge and orientation. RECOMMENDATIONS USAID staff and implementation partners should redouble efforts toward clear communication from the outset of every project with counterparts at every level concerning the purpose and mandate of the project, the priorities, and the constraints that exist in terms of budget and other resources, laws, rules, and regulations. It is advisable to have a written understanding at the beginning of a project of what USAID and the implementing partner are permitted and committed to do and what they are not permitted to do or do not have budget to do. EVALUATION QUESTION 2 To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? FINDINGS The Project had two attorneys who worked on legislation and regulations from April 2010 through the end of the project in September 2012. Four major pieces of legislation were drafted with the assistance of the SSPS project in February 2009. These laws required considerable revision and augmentation. In some cases they were internally inconsistent or inconsistent with other laws of RA. The two PALM Project attorneys worked on the legislation, in conjunction with staff of the MoF, MLSI, SRC, CBA, PRIMB, and the Standing Committee on Social Issues of the Parliament of the Republic of Armenia in 2010 and particularly intensely during October–December 2010. On December 22, 2010, the Parliament of Armenia passed the legislative package, which provides the legal ground for the new multi￾pillar pension system. The package of pension laws includes five main laws: on State Pensions, on Funded Pensions, on Investment Funds, on Unified Income Tax, and on Personified Record Keeping. 18 After the enactment of the Pension Laws by the Parliament in 2010, the development of business process maps revealed that the Law on Personified Record Keeping was not consistent with the design of the personified record keeping system. PALM international and local experts provided comments on Funded Pension and Investment Fund Laws, indentifying mistakes and inconsistencies in the laws. Notable among those comments was the assertion that the laws should be amended; the PALM team prepared a summary document of inconsistencies in the pension laws and delivered it to counterparts. In August 2011, two working groups (legal-methodological and technical) were established by the government. Specialists from MLSI, SSSS, MoF, CBA, and SRC were in the groups, which also included PALM legal and IT experts. The task of the legal methodological group was to finalize the discussions and develop draft laws to amend the pension laws. The task of the technical group was to make the laws consistent with the requirement and design of the IT system, especially for personified record keeping. PALM local and foreign experts actively participated in the discussions and provided the working groups with comments on the draft amendments on the Fully Funded Pension Law, on Investment Funds Law, on Personified Record Keeping Law, and on Unified Income Tax law. PALM experts provided the working group with comments showing experience of Eastern Europe and Latin American countries and comparative approaches on pension policy. At the end of March 2012, the draft amendments were finalized and submitted to the government. The draft laws with the amendments on Personified Record Keeping, Funded Pensions, Unified Income Tax, and Investment Funds were approved by the government on April 26, 2012. Presentation of the laws to the Parliament was delayed by the Parliamentary elections, which took place on May 6, 2012. As of November 2012, the package of pension laws was on the agenda of the current session of the Parliament and was in the process of parliamentary hearings. Regulations and by-laws were drafted and enacted along with the package of pension laws. In total, 170 references to the need for subsidiary legislation were identified in the pension laws. Accordingly, PALM prepared a list of required government decrees and regulations for the implementation of the pension laws. The list provided information on the number of decrees and regulations to be drafted and what state institutions should be in charge of drafting and approving them. In total, 40 regulations were required, and the PALM team either advised on drafts or took the initiative in drafting, depending on the counterparts’ requests. By the end of the project, all the regulations were in place. Quality of the team Many counterparts had good words to say about the Chief of Party and local team: “I must mention Rosa’s involvement – I met her several times and was able to benefit from her international experience. Rosa was able to deal with enquiries more than satisfactorily.” ~ Social Affairs Committee Representative, National Assembly “Rosa was very dedicated, trying to do her best. Very professional. Susanna Karapetyan is a great specialist. Arsen Manukyan went very deep into matters. He has a long experience.” ~ Ex-Board Member of CBA “I had a good experience of working with Gayane Dallakyan. She was practical, as opposed to the SPSS expert who designed an ideal system that didn’t work in Armenia.” “Rosa Chiappe was very good – smart and practical. A decisionmaker. Rusanna and Arman were also good.” ~ IT Department Representative, State Revenue Committee “Good aspects of the project were e.g., Arsen Manukyan's work, as he knows the legislation very well and is able to take the advice of international experts and adapt it to the local situation.” ~ State Social Security Service Representative 19 The laws and regulations developed with assistance of the PALM Project are listed in Annex VII. CONCLUSIONS The contribution of the PALM Project to the development of legislation has been significant. PALM played an important role in supporting MLSI, SRC, MoF, and CBA in drafting and revising pension laws and other legal acts to provide the legal framework of the new multi-pillar pension system in Armenia. The legal basis for each of the four pillars of the proposed pension system has been established, and most required regulations have been drafted, although more legal input will be required to complete and sustain the new pension system. The PALM legal team also contributed significantly to capacity building in the areas of legal drafting and understanding of financial markets regulation, especially at the MoF and the CBA. One PALM lawyer expressed the opinion that GOAM counterpart personnel could now draft many of the regulations that will be required in the future. This assertion seems reasonable on the basis that the basic laws and subsidiary regulations are in place and future amendments will be within the capacity of GOAM and the CBA. The process of considering and enacting the laws, amendments, and regulations prepared in support of pension reform is dynamic and not complete. There will be continuing needs to adjust the content of these laws and regulations as they are debated, passed, and implemented, and some new laws and regulations will be needed. For example, new legal and regulatory issues will arise with the roll-out of the Unified Income Tax in 2013 and the 2nd Pillar mandatory contributory pension in 2014. A particularly valuable contribution of PALM was facilitated by the close collaboration among the PALM legal experts and the PALM IT experts. The IT system required a well-articulated legal basis. However, the legal basis could not be designed without an understanding of the technical requirements, features, and limitations of the IT system. The IT system could not be designed without an understanding of the business process requirements and functions of the pension system and the tax system and the legal restrictions on data collection and reporting. The fact that PALM legal experts and IT experts worked in close proximity and had frequent and extensive communication facilitated this close collaboration. In conclusion, the key PALM contributions were in advising on the draft laws, preparing amendments to the adopted laws, and assisting in the drafting of 40 regulations. Although pension reform is a moving target, and there will always be further legislative requirements as the system evolves, the project clearly did what was required to ensure that the legislative basis was in place for operation of the system. RECOMMENDATION The follow-on pension reform project should have a strong national, full-time pension legislation expert to assist in refining existing legislation and regulations. EVALUATION QUESTION 3 To what extent has the support given to the Government of Armenia (GOAM) to build the foundation of Multi-Pillar Pension System contributed to the preparedness of the system to be implemented? FINDINGS Support for Multi-Pillar Pension System Building on the SPSS project, the PALM Project has provided support for the multi-pillar pension system in numerous areas. This is recognized by the GOAM counterparts that are involved with pension reform. Areas where PALM was cited as having provided significant support include: 20 • Concept papers, reports, discussion of design issues of the multi-pillar pension system (see Question 1 above); however, some important analytical issues are still unresolved (e.g., pension payout phase). • Information on international practice and specific examples/descriptions of “Best International Practice” (see Question 1). • Assistance drafting, analyzing, amending, and revising legislation and regulations (see Question 2 and Annex IX). • Education and training of counterparts staff (see Question 1). • Support for senior level pension reform advocates within the GOAM, e.g., presentations to the Prime Minister. • Analysis, model development, and enhancement of existing models (model developed by SPSS project and PROST model developed by World Bank); actuarial modeling; and simulation/projection of pension systems and model training of MoF and CBA staff (see Question 1). • Concept, functional specification, data exchange specification for Personified Record Keeping System, and other IT assistance (SRC, MoF). With respect to this area, PALM assisted the GOAM to implement the Unified Income Tax and Funded Contribution Personified Record Keeping System (UTPRS), covering employees’ personal and financial information collection, taxes and pension contributions, and information accessibility and management. More specifically, PALM: ˗ Organized meetings and discussions with local internet service provider (ISP) representatives regarding the connectivity assessment for successful introduction of UTPRS; ˗ Reviewed proposals and incorporated the results in the data exchange document; ˗ Developed business process maps, declaration and registration forms; ˗ Developed detailed descriptions for each business process; ˗ Submitted business process maps and descriptions to local and international experts and counterparts for review and comments; developed the architecture design of the UTPRS system; ˗ Developed a comprehensive concept paper for UTPRS, including business process maps, descriptions, and risks; ˗ Developed a data exchange document with detailed information on data structure, transfer methods, data integrity checks, and algorithms; ˗ Developed the UTPRS system requirements functional specification; and ˗ Submitted the package of business process maps and descriptions to local and international experts and counterparts for review and comments. The final package of the concept paper, functional specification, and data exchange documents were submitted to the Prime Minister, MLSI, MoF, SRC, CBA, and CDA. PALM provided technical assistance to the IT company developing the UTPRS. Public Awareness and Capacity Building • Baseline survey of public pension awareness and opinion (AM Partners). In order to develop and implement its outreach and public education proposals and activities, the PALM project initiated and 21 funded the Baseline Survey of the General Population’s Awareness of the Government’s Proposed Pension Reform. The purpose of this initiative was to obtain quantitative data on questions that were to be raised during the public awareness campaign to promote the implementation of the pension reform. The Baseline Survey was conducted by AM Partners Consulting Company LLC during the period October 2010 – March 2011. The results of the survey were used by PALM to develop communication strategies and increase public awareness of the reform and to enhance the level of financial literacy of the society. • Public financial literacy (AM Partners and Financial Banking College Foundation/FBCF). The PALM Project’s partner, AM Partners Consulting Company, provided financial literacy seminars to support the pension reform. The company held seminars for opinion leaders in six marzes of Armenia (Tavush, Lori, Vayots Dzor, Syunik, Shirak, and Kotayk). A “Brochure on Financial Knowledge within the Private Pension System” was published and distributed to seminar participants as well as in State Social Security Services (SSSS) in all marzes and Yerevan (51 offices), 18 ministries, universities, associations, and institutions involved in the pension reform implementation process (CBA, SRC, and NAZDAQ OMX Armenia). • Public relations, public awareness, public education programs, regional public information programs, advertising (FBCF, PSAC Foundation). PALM employed tools such as brochures, leaflets, posters explaining the new pension system. The PALM project produced different material for different audiences and disseminated them broadly (to regions and Yerevan City, government institutions, universities, associations, etc.). In addition to the printed materials that were broadly distributed, targeted public appearances, presentations, and discussions were held and organized for different audiences with specific focuses and messages, as well as conferences and interviews. During the initial stage of the campaign, PALM directed its efforts to inform professional associations such as the Armenian Society of Actuaries, Armenia Bar Association, Union of Banks of Armenia, Association of Accountants & Auditors of Armenia, and Union of Manufacturers and Businessmen (Employers) of Armenia. Speed of the Reform Process Pension reform is a moving target, and during the interviews various counterparts reflected on whether things might have been done differently or at a greater speed: “Overall, pension reform was slower than I expected, but this was mainly due to the government, tax authorities, etc.” ~ Capital Asset Management Representative “The Government wanted to finish pension reform in 2011. I opposed this … it was important that the reform should be accessible for all, and therefore the reform schedule was negotiated between the National Assembly and government. As a result they gained more than they lost. In the time gained many questions were answered.” ~ Social Affairs Committee Representative, National Assembly “If I could start the process again, I would do public awareness for a year and then do the legislative reforms.” ~ idem “I always say in meetings: ‘don’t think this will be the last amendment: there will be others’.” ~ idem 22 • PALM collaborated with the FBCF to raise public awareness on the pension reform in the regions of Armenia. Eleven meetings were held and targeted at raising awareness among the population in the 11 regions of Armenia. • The PALM Project put significant efforts in capacity building in public awareness. In August 2011 the GOAM created the Pension System Awareness Center (PSAC) Foundation. This new institution had the task of raising awareness and educating the population of Armenia on the new pension System. PALM provided support to build capacity in the new institution. The project supported the PSAC Foundation with financial resources and technical support. • The project also conducted training and seminars on pension reform for journalists. This was part of an effort to introduce the private pensions system to the public. • The project contributed to capacity building, especially at MoF, CBA, PSAC, and other public relations institutions. PALM organized more than 50 policy discussions, trainings, and retreats to help counterparts to prioritize the tasks within the pension reform process, to set up deadlines for each activity, to address challenges, to discuss urgent and difficult topics of the pension reform, and to provide extensive advisory assistance to counterparts based on international best practice. PALM provided extensive training in actuarial modeling and the use of the models developed for the project. According to some PALM staff, MLSI has lagged in capacity building. MLSI has fewer qualified staff working on the technical aspects of pension reform and able and willing to take advantage of PALM training and other capacity building efforts. As a result, MLSI does not have adequate pension reform analytical staff. MLSI reportedly does not have capability to use pension models. Relationship with USAID • Gap in USAID support for pension reform. During the period of this evaluation and report there was a gap in USAID assistance because PALM ended September 30, 2012, and the follow-on Finance for Economic Development (FED) program, which is to support the financial institutions and markets required for the mandatory 2nd pillar and voluntary 3rd pillar funded components of the pension reform, and the Pension Reform Implementation Project (PRIP) contract/teams were neither selected nor in the field by March 2013. Some counterparts were concerned that there would be no support or inadequate support during the interim period. It will take time to “train” a new team. One counterpart asserted that “Six months will be lost... This is a disaster.” • Coordination of future USAID pension reform assistance that is to be delivered via three different projects. After the completion of the PALM Project in 2012, the tasks performed by the PALM Project will be carried out through three separate projects. The tax reform, support for the State Revenue Committee (SRC), and IT system for personified record keeping for income taxes will be done under the Tax Reform Project (TRP) (which began in January 2013). Support for the CBA and for the development and regulation of financial institutions will be provided under the FED program, which has not begun as of March 2013. Support for the MLSI and the implementation of the pension reform will be provided under PRIP, which has not begun as of March 2013. There will be three different project teams and, potentially, there could be three different contractors. The three projects will be managed by two different USAID Offices: Democracy, Health and Social Reform (DH&SR) and Economic Development. There will also be three different lead counterparts – each with a different mandate and orientation: PRIP via the MLSI; tax reform/personified record keeping/IT through the SRC; and pension asset management/Financial management by the CBA. 23 Synergy The PALM project provided synergy among concept/design, modeling/analysis, legal drafting and support, IT, and public education. Public Awareness, Public Information, and Public Relations Public Information support is to be provided by PRIP, but the key requirement for public education may be the Personified Record Keeping System, which is managed by the SRC and supported by the Tax Reform Project. Several counterparts and PALM staff stressed that public understanding and support of the new Multi-Pillar Pension System is key. The RA Prime Minister established a working group on public education and public information about the pensioner reform. CONCLUSIONS The support provided by the PALM Project has been very significant and constructive and has provided the foundation for the multi-pillar pension system. It is clear from the long list of activities set out in the “Findings” section above that considerable efforts have been made to ensure that the multi-pillar pension system is ready to be implemented. Some failures have been highlighted by counterparts but they have been muted in comparison to the overall level of effort. As pension reform is a moving target, it has become clear that certain aspects – such as public awareness – have grown in importance, and so with hindsight it would be tempting (but probably unfair) to say that more could have been done on that issue. That is a comparatively minor quibble. Although the MLSI has the formal leadership role in the pension reform effort, it has not developed an adequate staff below the senior level to effectively carry out the continuing analysis of pension reform issues – including pension modeling – and direction of pension reform, including addressing important unresolved analytical issues. With the benefit of hindsight, it would have been useful to commit more resources to capacity building within MLSI. A strong pension reform management structure is needed and will enhance GOAM’s ability to implement follow-on pension reform successfully. RECOMMENDATIONS USAID should work with MLSI to help develop staff below the senior level to serve as active counterparts to the follow-on pension reform project. After the conclusion of the project a former PALM Project attorney became the head of the MLSI legal department. It may be possible to build on that capability to develop the required capacity at MLSI. The follow-on pension reform project should have a strong pension modeling expert and other staff capacity to address the remaining unresolved analytical issues. The follow-on pension reform project should continue and expand pension public awareness and public education programs with strong support from USAID and PRIP. GOAM, USAID, and others should create a management structure for the follow-on pension reform program that ensures coordination among stakeholders and resolve conflicts pro-actively. A pension reform management representatives of all GOAM counterparts, all project teams and relevant partners, and USAID may serve this purpose. (The existing PRIMB could in theory play a coordinating role, but it ordinarily meets only one time per month and does not include USAID or contractor personnel as official members.) USAID should also designate a USAID “Pension Czar” to resolve disputes and make resource allocation decisions. It is important to provide appropriate content/presentations in regions and villages using simple, attention-getting, engaging content and formats. The implementing partner and the public relations/public education counterparts should experiment and compare experiences to determine the 24 appropriate media. Different approaches are appropriate in different contexts. More than one approach should be used, e.g., town meetings, round tables with local and national opinion leaders, radio/TV programs and advertisements, text messages, posters, repeat visits, different media, and non￾government spokespersons. USAID, PRIP, and the FED program can foster and facilitate interaction/relationships among private sector entities that must provide the infrastructure for funded pensions – both 2nd and 3rd pillars. These could include asset management skills and companies and actuarial skills and actuarial consulting companies (e.g., Society of Actuaries of Armenian for accounting and reporting and the Association of Accountants and Auditors of Armenia). EVALUATION QUESTION 4 To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency (SESA) to match labor market demand and to develop/implement active labor market policies? FINDINGS The PALM Project provided considerable support to SESA and the YPOC in several activities: • Employment Strategy Paper developed in cooperation with counterparts. From the beginning of the PALM project, a priority for the MLSI was the development of an employment strategy paper. The Ministry requested PALM to help with assistance in developing monitoring and evaluation (M&E) concepts and employment promotion tools for youth and persons with disabilities. A PALM international expert reviewed and commented on the draft performance measurement indicators for employment programs. Based on this and interviews with MLSI Labor and Employment Department and SESA leadership, a paper outlining M&E issues and recommendations was developed and submitted to MLSI. In addition, two PALM local experts worked with MLSI on development of the concept paper and an action plan. • Support for Two Employer Surveys. SESA has conducted employer surveys for labor market analysis and forecasting since 2005. The surveys were conducted by SESA regional offices staff twice a year. PALM assisted SESA to improve its survey capacity and conduct the biannual employer surveys, bringing them closer to European standards. The PALM Project also assisted SESA with dissemination of information to stakeholders at the Republic and Marz levels, assisting with finalization, printing and dissemination of survey reports. • Assessment and improvement of the GORTS employment data system, including connectivity and reporting, search capabilities, and resolving database inconsistencies, and implementation of the system at self-service kiosks. The GORTS IT system was developed by the World Bank in cooperation with the MLSI NORK Center to provide SESA data, job matching and referral functions. The GORTS system is the internal data and information reporting system operated by NORK center for SESA. The GORTS program is the only PC data package used in SESA’s 51 employment offices. SESA leadership indicated that GORTS was not meeting the increased needs of local offices. A PALM short-term IT expert spent two weeks assessing GORTS and formulating recommendations to improve the system. The short term adviser (STTA) expert provided a report outlining three issues: connectivity within 19 offices, software design glitches within the GORTS program, and the self service Kiosks’ lack of access to internet. 25 • GORTS system hardware improved. Based on PALM recommendations, SESA upgraded operational memories for 150 SESA computers for GORTS system effective functioning. • Support for Integrated Social Service Centers. • Establishment or support for 8 Job Clubs (of total of 17, covering about 1/3 of the country). In the first year of the project, PALM organized a seminar for 12 existing job clubs and five new offices where job clubs were planned, discussing new tools and methods to increase their effectiveness. The first Job Club training identified the strengths and weaknesses in the implementation of job clubs. Five new SESA centers staff were provided guidance on how to start up job clubs. Thirty-four SESA employment office directors and program facilitators along with four SESA headquarters staff participated in the first Job Club Networking and skills training sessions. After the training, five new job clubs were started during the first year (three after renovations) making the total of 17 operating job clubs in the system. Twice a year PALM has has brought all 17 Job Club staff together, building a Job Club network involving up to 40 SESA staff. • Support for establishment of Employer Councils. Employer Councils provide market-based approaches (focusing on demand by employers) to labor market issues and assist SESA in building strategic alliances with employers, community leaders, and training/academic institutions. PALM prepared and implemented a three-day training program for 34 SESA staff on how to start-up and implement Community Employer Councils. PALM experts presented and discussed the concept, benefits, and methodology of Employer Councils. An Employer Council guide was provided. At the end of the training, the centers came up with action plans which included an organization description, participant list, and timetable to start Employer Councils. PALM’s initial training resulted in 12 SESA Community Employer Councils being formed. PALM conducted a follow-up training to strengthen this program and to improve SESA staff skills in providing employer services through the Employer Councils. Twenty-eight staff from 12 SESA offices which run Community Employer Councils and six SESA headquarter staff were trained. More than 50 employers and 20 Vocational Training representatives participated in these council meetings for addressing community workforce development and workforce mobilization issues. • Support for the organization of Job Fairs in regional employment centers. • Public-Private Partnership between SESA and the Union of Information and Technology Enterprises (UITE). PALM organized a public-private partnership between UITE and SESA representatives for matching employer job openings with the unemployed (a mini IT job fair) during the DIGITEC Expo, which was held October 2011. This was the first public-private partnership to enhance job matching for the IT sector. PALM assisted with the promotion campaign and provided on-going technical assistance. The fair brought 58 employers offering 208 vacancies to 5000 job seekers with 104 job seekers placed over a three month period. • International study tour organized and conducted. The MLSI requested PALM to organize a study tour for the staff from SESA and YPOC. The PALM project selected Serbia to study three programs: two employment programs (one for people with disabilities and the other for youth out of school with no work experience) and one on M&E. The study tour to Serbia was conducted for nine staff from MLSI, SESA, and YPOC as well as one USAID and three PALM staff. Participants met the Serbian National Employment Services staff and learned about targeting employment promotion programs for youth and people with disabilities and the M&E system. Participants reported positive results from the Serbian study tour. 26 • Support for development of Self-Service Kiosks. Six self-service booths were made operational in the second year of the project. Support for GORTS system in the self-service kiosks provided self￾service employment matching function. • Outfitting and refurbishing of three SESA regional offices and Headquarters office. PALM renovated three local Employment Centers in Armenia’s southern region of Syunik. New self service kiosks and meeting rooms are used to enhance services to job seekers and employers. SESA asked PALM’s assistance to renovate the SESA headquarters offices. SESA staff reported that this renovation improved morale and staff efficiency by improving working conditions for about 40 headquarters staff. To support the implementation of SESA job clubs, PALM equiped the job clubs in Vanadzor and Martuni offices with necessary furnture, chairs, and equipment. IT and printing equipment was provided to MLSI Employment Division. • Finalization of Labor Safety Regulations. PALM experts worked with the First Deputy Minister and the Head of the Labor and Employment Department to finalize the Labor Safety regulations developed by the USAID SSPS project. • Support for the YPOC. • Review of international best practice in youth employment and employment of disabled people. Two PALM international experts supported the development of a youth career management policy and a methodology for career guidance policy. Experts provided advice on the organizational design the YPOC should adopt. In cooperation with YPOC, PALM organized a series of meetings and discussions with all stakeholders with the experts and YPOC representatives on the draft report, including 25 participants from SESA, YPOC, and university career center staff. Training on job search skills was held at YPOC. PALM experts trained the participants on how to customize job search for youth. In addition PALM discussed elements of a successful youth career event. PALM procured training and public presentation equipment to allow the youth center to provide services to the Marz and rural schools. A computer, printer, video camera, screen, monitor, and bulletin board were provided to the youth center. • Vocational Guidance concept paper developed. The Prime Minister instructed MSLI to develop a youth vocational guidance concept paper. MLSI requested PALM to provide international technical assistance to develop the concept paper. A PALM Vocational Guidance STTA drafted the Vocational Guidance Concept Paper. Local PALM experts provided background information and helped to draft the concept paper to assure the vocational guidance program was customized to Armenian needs. Meetings with the MLSI Deputy Minister and MLSI staff, the Vocational Guidance Working Group (including a representative from the Ministry of Education and Science), and the SESA head were held to gain input and consensus. One result of the design is that the YPOC became the leader in implementing this concept. • Youth Career Fair. PALM cooperated with SESA and YPOC to organize a youth career fair in May 2011. The career fair was the first to address youth unemployment. Sixty-five employers, seven university career centers, seven vocational education institutions, and representatives from the event’s 10 partner organizations participated to provide job vacancies, internships, career advising, and education opportunities. About 3,500 young people of all ages participated in the event, meeting with employers, applying for jobs, and learning which careers are in demand. Based on data received from participant employers provided by the organizational committee members, there was a 53% placement rate on available internships and jobs. 27 PALM Project resources and staff devoted to labor market and employment policy were more limited than those for pension reform. For the last year of the project, PALM did not have a long term international expert working full time with labor market/employment policy. With the limited time and resources of the evaluation, it was not possible to gather the data that would be required or do the analysis to estimate how PALM Project interventions affected employment and labor market conditions in specific towns and regions. Given the many factors that determine employment conditions in any given area, such an analysis would require collection of considerable local data and the use of multivariate statistical techniques. Nor was it possible to interview employers to determine if they used the PALM Project-supported programs. All information about labor market programs came from interviews with PALM Project staff and GOAM agency heads. While the senior staff of SESA had a positive view of the PALM Project and USAID support, they expressed frustration that the PALM Project was not able to do more (see Question 1 above). They expressed the opinion that the pension reform took priority over labor market assistance. This was an area where respondents cited improved communication about the mandate, role, funding, and capacity of USAID and PALM might have been helpful. With the end of the PALM Project, USAID does not have a large labor market/employment policy project underway, although the LIFE (Livelihood Improvement through Functional Employment) project has many labor market components and is supporting SESA activities. There are many other international donors, including the World Bank, supporting active labor market and employment policy efforts in Armenia. CONCLUSIONS The PALM Project engaged in many activities and tasks designed to enhance the ability of the SESA to match labor market demand and to develop/implement active labor market policies and to support the programs of the YPOC. A number of ongoing programs and a significant level of capacity were developed in active labor market policies. It was not possible within the confines of the evaluation to assess the outcome of those activities in terms of actual improvement in employment conditions in the regions affected. RECOMMENDATIONS If USAID in the future determines that it should develop significant labor market or employment programs in Armenia, an evaluation component should be built into the projects from the beginning, including the collection of relevant data before the beginning and as the project is being implemented. This should be done in conjunction with the counterpart and as part of the technical assistance. EVALUATION QUESTION 5 How adequate was the project Performance Management Plan (PMP) in reporting on project progress? FINDINGS Eight PALM Project Work Plans and eight Performance Management Plans (PMPs) were provided to the evaluation team and reviewed, including the Final Report PMP. The PMPs correspond closely to the Illustrative Activities and Expected Results in the contract Statement of Work (SOW) (see Annex IX), providing mostly quantitative indicators. The PALM Project demonstrated significant accomplishments and progress for the indicators identified in the PMP. The Final Report PMP is provided in Annex X. 28 The three major Tasks and the Illustrative Activities and Expected Results specified in the contract SOW for “Component 1: Pension Reform Implementation” are primarily stated in terms of activities and results. The Expected Results are reflected in the PMP and the PALM project accomplished almost all the results set out there. However, the Expected Result, “Pension system administration modernized and business processes improved according to international standards and best practices,” was not achieved because the multi￾pillar pension system was not implemented during the project and the business processes were not implemented. The Head and Deputy Head of SSSS indicated that the administrative processes of the O Pillar system (the existing pay-as-you-go benefit system) had been improved by the implementation of a new software system, but they stated that that was not done by the PALM project. In fact, they expressed frustration that PALM did not support their efforts to improve the business management software. A broader question is how appropriate the PMPs were in terms of assessing the progress PALM made in realizing its desired outcomes. While most of the results of Components 1 and 2 were achieved, the outcomes of Component 1 were only attained in part and some in Component 2 were too vague to measure. Thus, while indicators measured progress toward realization of Component 1 and 2 results and those results were mostly achieved, the outcomes of the project were only attained in part. The contractor SOW for Component 1 specified: • Expected Impact: Implementation of the new multi-pillar pension system that provides an adequate income for Armenia’s elderly population in an environment that is fiscally sound and sustainable, with minimized labor market distortions and negative fiscal impacts, leveraged positive financial market impacts, responsive to diverse kinds of shocks, and little subject to the risks of political interventions that create adverse consequences for short-term political gains. USAID recognizes that the successful implementation of a multi-pillar system in Armenia will rest heavily on modernizing several administrative and operational systems in a number of pension institutions and related entities, particularly the SRC core systems (such as those related to tax filings and payment, audit, and collection enforcement). • Goal: The GOAM implemented a multi-pillar pension system that will provide an adequate income for Armenia’s elderly population in an environment that is fiscally sound and sustainable. This Expected Impact and its planned Goal were not fully accomplished by the project, nor were they specified in the PMP. It is not clear whether accomplishment of the outcome and impact was expected by USAID. There was no indication in interviews with USAID staff or PALM project staff that USAID felt that the project did not accomplish its major objectives, but there is no written indication that the SOW of the project was adjusted during the project period. 29 It should be noted that the package of key laws for the pension system was not adopted by the National Assembly until December 2010, more than a year after the start of the project. PALM cannot be blamed for the delay. Further, a key stage in the preparations for implementation of the pension system was the introduction of IT and management changes to ensure smooth operation of the unified income tax and personified record keeping systems. At the time of the fieldwork in November 2012, many counterparts described the intensive preparations that this involved to the evaluation team, not least the need to inform all employers prior to the intended start date of January 2013. Unlike a standard logical framework, the PMP did not include any list of risks and assumptions upon which successful implementation was to be contingent. For Component 2, Labor Market Interventions, the contractor SOW specified: • Expected Impact: A reformed State Employment Services Agency, along with its regional offices, as one of the key players regarding labor market mediation and workforce development. This institution, along with the private sector, will provide high quality employment services taking into account appropriate flexibility and responsiveness to ensure customer satisfaction among the majority of those who come in contact with these employment service providers, including employers and job seekers. These activities are intended to impact the business environment by improving the job mediation process, reducing unemployment, and improving the quality of the labor force. They are also intended to improve the effectiveness of job mediation and workforce development for the disadvantaged and other under-served groups. PALM’s results are intended to influence directly human capital development and better skills matching in Armenia. Ultimately, USAID’s technical assistance will impact Armenia’s ability to increase employee productivity, reduce employer costs, and allow enterprises to become more profitable and competitive all in the interest of greater economic growth that will improve the standard of living for the citizens of Armenia. • Goal: The GOAM is institutionally capable to help the most vulnerable individuals to mitigate the economic and social consequences of the global economic crisis and improve the country’s workforce readiness and competitiveness. The Component 2 Goal, Task 1, and the most of the Illustrative Activities all are vague. No measureable outcomes, in terms of changes in employment or the labor market itself, were specified, even for the first three illustrative activities. Task 2 is to “support State Employment Services Agency,” but most of the Illustrative Activities are also vague. However, the five Expected Results were incorporated into the PMPs and most were achieved. CONCLUSIONS The project PMPs were useful management tools and mechanisms for communicating to counterparts, to USAID, and to outside evaluators in reporting project progress and accomplishments. The PMPs demonstrated that the PALM Project achieved nearly all desired project results set out in the SOW and PMP. However, these significant accomplishments did not fully lead to the outcomes desired, suggesting that the linkages between outputs and outcomes were not sufficiently specified in the PALM results framework. The process of preparation for implementation of the pension system included several key steps, delays in any one of which would impact on the whole process – in effect there were bottlenecks. The initial delay in adopting the key laws is the most obvious example. Neither the PMP nor the Project Description document included a “risks and assumptions” section which would have weighed the political and technical risks involved. The closest the latter document comes to acknowledging the issue is in the following passage: “USAID recognizes that the successful implementation of a multi-pillar system in Armenia will rest heavily on modernizing several administrative and operational systems in a number of pension institutions and related entities, particularly the SRC core systems (such as those related to tax filings and payment, audit, and collection enforcement).” 30 Accordingly, the goal of the Pension Reform Implementation component – to build GOAM institutional and human resource capacity to design and implement a multi-pillar pension system – was accomplished only in part because there was an expectation that the PALM Project would result in significant implementation of the new pension system. The SOW was not modified to change this objective during the course of the project, but as reflected in the answers to Evaluation Questions numbers 1 and 3 above, many important activities and tasks required for the ultimate achievement of this objective were accomplished. The counterparts did not lay the blame for delays at PALM’s door, rather there were political and technical difficulties which could have been recognized as risks affecting the overall goal of Component 1. For the Labor Market Interventions component, the description of Task 1 and three of its Illustrative Activities were too vague to permit development of measureable criteria for success. The Expected Results that could be measured, which were outputs rather than outcomes, were mostly achieved so that PALM built a strong basis for realizing the desired outcome. RECOMMENDATIONS Care should be taken in developing the SOW for each USAID project to identify expected outcomes in as concrete a way as possible (rather than language like “foster greater attention to…“) and provide some guidance concerning how the contractor and the USAID mission at the outset of the project may be able to develop measures of accomplishment. The overall objective of each of the project components should be stated at the beginning of each PMP, and effort should be made to indicate progress toward its achievement. Risks and assumptions should be clearly set out in project documentation. GENDER ISSUES An adequate, sustainable, universal pension system is one of the most important social policy institutions to support the economic and social welfare of women. Because of several factors, women are especially vulnerable to the hazards of old age, disability, or loss of the family bread-winner. Women have a longer average expectation of life, so are more likely to live into old age and be widowed. Women on average have lower earnings than men and less opportunity to save or accumulate an adequate pension. Because of child-rearing responsibilities and/or discrimination, women often have shorter or interrupted work careers. Design of an adequate and sustainable pension system, taking into account these factors, can play a major role in increasing the well-being of women. For many of the same reasons, adequate labor market and employment policies enhance the opportunities and welfare of women. In addition to the vital social and economic role for women of the pension system being developed, the PALM Project provided a strong example of women in substantive technical and leadership positions. The COP and senior pension expert during most of the project was a woman. The senior long-term international labor expert was a woman. The international actuary and the majority of the local experts on the pension system team, were women. 31 ANNEXES 32 ANNEX I: EVALUATION STATEMENT OF WORK Evaluation of USAID/Armenia Health Systems Strengthening Project, Civil Society and Local Government Support Program, Alternative Resources in Media Project, Assistance to the Energy Sector to Support Energy Security and Regional Integration Program, Pension and Labor Market Reform Project Summary USAID/Armenia requires performance evaluations of the following activities: Health Systems Strengthening Project (HS-STAR), Civil Society and Local Government Support Program (CSLGSP), Alternative Resources in Media Project (ARM), Assistance to the Energy Sector to Support Energy Security and Regional Integration Program (ESRI), and Pension and Labor Market Reform Project (PALM). The purposes of this Task Order are to evaluate the success of these projects in their relevant technical areas and to assess the overall effectiveness of the projects in achieving set programmatic goals and USAID/Armenia’s strategic objectives. Three of the five planned evaluations are designed as mid￾term performance evaluations (HS-STAR, CSLGSP, and ARM), while the remaining two are designed as end-of-project performance evaluations (ESRI and PALM). The purposes of the mid-term evaluations are to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Findings from the mid-term evaluations must inform future work plans of the relevant projects, as well as designs of future similar activities. The purpose of the end-of-project evaluations is to assess the effectiveness of resources spent and to inform design and development of future strategies and projects. The evaluation findings must be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. The respective project AORs/CORs will develop plans for incorporation of relevant recommendations from the evaluations in their future work plans. Contractor Responsibilities and Projects The evaluations should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, guided by the evaluation questions formulated for each individual activity. Each evaluation question must be answered empirically, relying on factual evidence, and must be addressed distinctly in the final reports. PALM Oct 2009-Sep 2012, $5.4 million PALM is USAID’s primary activity aimed to assist individuals, households, and communities to better manage social risks or needs, provide support to the vulnerable employed and jobless individuals (particularly the low-skilled, women, and the disabled), and give greater attention to countering the effects of the global economic crisis. The program is focused on: (1) building institutional and human resource capacity to implement a multi-pillar pension system that will provide an adequate income for its elderly population in an environment that is fiscally sound and sustainable, with reformed current pay-as-you-go (PAYG) scheme and the newly created private retirement savings system; and (2) countering the effects of the economic downturn, through catalyzing new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps, and through improving the country’s workforce competitiveness. Evaluation Questions: The evaluation questions for mid-term evaluations are designed to help the projects adjust and/or modify future implementation and work plans, as well as designs of future similar projects. The questions for end-of project evaluations will help assess the effectiveness of the resources spent and will help 33 inform the design of future projects and strategies. The contractor must review and summarize the data collected from document review and from field work to answer the following evaluation questions: PALM • What do stakeholder groups - the Ministry of Labor and Social Affairs, the Central Bank of Armenia, the Ministry of Finance of Armenia, State Revenue Service, State Employment Service Agency, “Pension System Awareness Center” foundation, “Youth Orientation Center” NGO – see as the achievements of the program? • To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? • To what extent has the support given to the GOAM to build the foundation of a Multi Pillar Pension System contributed to the preparedness of the System to be implemented? • To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency to match labor market demand and to develop/implement active labor market policies? • How adequate was the project PMP in reporting on project progress? USAID’ S Role in the Evaluation The USAID Mission in Armenia will: • provide relevant programmatic and budgetary information to the Contractor; • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor must: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Identify data collection methodology to provide the best possible evidence to answer the evaluation questions, also considering feasibility issues. 3. Identify informants and stakeholders, samples and/or other relevant data sources. 4. Prepare a field work plan. 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 34 7. Revise the draft reports addressing comments by USAID and submit final reports to USAID/Armenia for acceptance. 8. Address implementing partner comments within one week as necessary after USAID/Armenia shares the final reports with implementing partners and if partners choose to submit “Statements of Differences”. The proposed methodology should address the need for data collection from qualitative and quantitative sources and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. All evaluation questions need to be answered empirically; therefore, the data collection methods should be tailored to ensure that adequate evidence is collected to answer each of the questions in a definitive manner. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. Data should come from facts, rather than be based on anecdotal evidence. Conclusions should be based on findings received from multiple sources, and strengths and limitations of the methodology should be explicitly communicated. All people-level data should be disaggregated by sex to allow analysis of findings by sex. Baseline data for all projects is available from their monitoring data. A sample of indicators used for monitoring of each of the projects is provided in the Annex. Some of the baseline data sources include surveys, official statistics, automated information systems, and project records. Deliverables The Contractor’s deliverables must include: 1. Written methodology plans (evaluation designs and work plans). The evaluation designs must include detailed evaluation design matrices (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final designs require COR approval. The work plans must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation teams. The methodologies will be shared with relevant stakeholders, including the implementing partners for their comments before finalizing. 2. Verbal debriefing The evaluation teams must meet with USAID/Armenia upon arrival. The teams must also provide oral briefings of findings and recommendations to the USAID/Armenia senior management and relevant technical teams prior to departure. 3. Draft Evaluation Reports Prepare draft evaluation reports which must analyze data and summarize key findings, conclusions, and recommendations. The Findings section must list all facts and evidence received from desk research and from field work in relation to each evaluation question. The Conclusions section must analyze the facts specified in the Findings section and must discuss what worked, what did not work, and why. The Recommendations section must address issues of what can be improved for future programming, based on Findings and the analysis provided in the Conclusions section. More specifically the Recommendations section must address the following: for PALM - what recommendations can be made for design of future pension reform projects. 35 The Evaluation Reports must at a minimum contain: 1) a 3-5 page Executive Summary summarizing key points (purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations); 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action-oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work; all evaluation tools; all sources of information properly identified and listed; any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team; disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Draft reports must be submitted to USAID/Armenia within three weeks after completing the fieldwork for each evaluation. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. Final Reports The Contractor must submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The reports shall follow USAID branding procedures. The reports must include an executive summary and not exceed 30 pages (excluding appendices). The final reports must meet the following quality standards (Please see the USAID Evaluation Policy): a) The reports must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports must address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings must assess outcomes and impact on males and females; f) Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; and j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. Evaluation Data The Contractor must submit Evaluation data to USAID/Armenia along with the final reports for record keeping and future use by the Mission. The data must be in an easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Schedule The anticipated duration of all evaluations is not to exceed nine months. 36 ANNEX II: EVALUATION METHODS AND LIMITATIONS The SOW for the evaluation provides five specific questions to be answered. Below is a description of the proposed methodologies to answer these questions. • What do stakeholder groups - the Ministry of Labor and Social Affairs, the Central Bank of Armenia, the Ministry of Finance of Armenia, State Revenue Service, State Employment Service Agency, “Pension System Awareness Center” foundation, “Youth Orientation Center” NGO – see as the achievements of the program? • Interviews with senior and mid-level Ministry and other stakeholder officials. • Review of PMP and other documents and reports. • Interviews with USAID Mission personnel. • Focus groups, including groups with participants from several different stakeholders. Method: Key informant interviews; focus group discussions; document review Source: Relevant USAID program reports; relevant stakeholders • To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? • Description of legislative and regulatory framework at beginning of PALM project. • Review of legislation and regulations proposed and passed during the project period. • Interviews with senior and mid-level Ministry and other stakeholder officials. • Review of PMP and other documents and reports. • Interviews with USAID Mission personnel. • Focus groups, including groups with participants from several different stakeholders. Method: Key informant interviews; focus group discussions; document review; legislation review Source: Relevant USAID program reports; relevant stakeholders • To what extent has the support given to the GOAM to build the foundation of Multi Pillar Pension System contributed to the preparedness of the System to be implemented? • Review of background to PALM project: ˗ Legacy of Soviet Union pension system and social welfare and labor market institutions. ˗ Pension system and institutions at beginning of PALM project. ˗ Achievements and shortcomings of previous USAID pension reform projects. • Description of current pension system and institutions ˗ Legislation and regulations 37 ˗ Institutions • Basic pension • 1st pillar • 2n pillar • 3rd pillar • Revenue collection • Benefits calculation, award, appeals • Benefits payment • Funds management • Regulatory framework for 3rd pillar • Information technology • Interviews with senior and mid-level Ministry and other stakeholder officials. ˗ Role and contribution of PALM project • Review of PMP and other documents and reports. • Interviews with USAID Mission personnel. • Focus groups, including groups with participants from several different stakeholders. Method: Key informant interviews; focus group discussions; document review Source: Relevant USAID program reports; relevant stakeholders • To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency to match labor market demand and to develop/implement active labor market policies? • Review of background to PALM project: ˗ Labor market and employment institutions at beginning of PALM project. ˗ Achievements and shortcomings of previous USAID pension reform projects. • Description of current institutions, functions, and capabilities of the State Employment Service Agency. • Role and contribution of PALM project. • Interviews with senior and mid-level Ministry and other stakeholder officials. • Review of PMP and other documents and reports. • Interviews with USAID Mission personnel. • Focus groups, including groups with participants from several different stakeholders. Method: Key informant interviews; focus group discussions; document review Source: Relevant USAID program reports; relevant stakeholders • How adequate was the project PMP in reporting on project progress? • Review of PMP for each project year and other documents and reports. 38 • Interviews with USAID Mission personnel. • Interviews with senior and mid-level Ministry and other stakeholder officials. Method: Key informant interviews; document review Source: Relevant USAID program reports; relevant stakeholders Known Limitations to Evaluation Design: The feasibility of the above evaluation methodology depends on the availability and usefulness of relevant program reports, as well as the availability of key informants for interview. Given the condensed time frame of evaluation (two weeks in the field), securing program stakeholders for participation in interviews and focus groups on short notice may prove difficult, posing a known limitation. The evaluation team will attempt to mitigate this limitation by identifying and contacting interview subjects as early as possible, with assistance from the Mission, at the time of the PALM team in-brief. A local logistician (Kara Hakobyan) will also aid in the timely scheduling of appointments and overall evaluation-related logistics. 39 ANNEX III: DATA COLLECTION INSTRUMENTS The evaluation was undertaken through a combination of document review, interviews with relevant stakeholders, and focus group discussions. Below are illustrative interview questions for: i) the recipients of the assistance - in this case, primarily, government ministries and pension and employment organizations, ii) other stakeholders, iii) USAID. Questions for focus group discussions were based on the interview questions for recipients Interview questions for PALM project assistance recipients: 1. What is the nature of your organization? What functions does your organization undertake? By what mandate? 2. What is your relationship with USAID? 3. What has been the nature of the assistance being provided? 4. What have been the results to date? 5. How satisfied are you with the experience with the USAID assistance? How would you rate your level of satisfaction on a scale of 1 – 10 (10 highest)? 6. Without the PALM project assistance, what other alternatives would have been available to you? 7. How effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? 8. Can you provide the evaluation team with information to help us understand the impact of the assistance on your activities? 9. How sustainable are the impacts of the assistance? Do you anticipate that your organization will continue with the same practices after the USAID assistance has finished? 10. Do you see the assistance as being relevant in light of the current employment situation in Armenia? 11. To what degree is cooperation achieved with other donor agencies operating in these and related sectors? 12. What, in your opinion, is the main constraint acting on USAID assistance in Armenia? 13. What, if any, training has been provided? What kind of training? Should this training be augmented? How? 14. In your view, how effective was the USAID assistance? Do you believe that it has achieved what it set out to do? 15. In your opinion, how could the assistance be improved? 16. What other types of assistance could be offered by USAID for pension and employment services reform and development? 17. Do you have a specific recommendation that the evaluation team could provide to USAID to help it improve future assistance? In light of your experience, what advice would you give USAID? 18. Would it help to make changes to the design of the next pension reform project in order to deliver assistance more effectively or to respond better to current challenges? Please explain in detail. Interview questions for other stakeholders: 1. Can you please summarize your organization’s mandate, role, and functions? 2. Are you involved in other assistance projects in Armenia other than those with USAID? What are they? 3. In your opinion, how appropriate is the USAID assistance? What are the strongest points of USAID assistance in pension reform and employment services? Are any improvements needed? 4. To what degree is cooperation achieved with other donor agencies operating in these sectors? 40 5. In your opinion, how successful is USAID assistance in these sectors? How well has it met the country’s needs? 6. How effective do you consider the work to be of implementing the assistance provided? What are your success stories and what are your failures? 7. How sustainable are the improvements brought about by the assistance? Will the changes continue once the assistance has ended? 8. Are there any areas in which provision of assistance might be more effective? 9. What, in your opinion, is the main constraint acting on USAID assistance in these sectors in Armenia? 10. Do you see the particular USAID assistance highly appropriate and relevant, in light of the current regulatory and legal environment in Armenia? Do you believe it fits well within the desired development strategy for Armenia? 11. Are the objectives of the assistance being accomplished? Please elaborate. 12. Does your component of the project(s) have a human capital development component or an institutional strengthening component or both? If not, in your opinion, should there be? 13. Can you recommend any individuals, groups, or organizations in Armenia the evaluation team should be sure to contact? 14. On a scale of 1 – 10 (10 highest), how would you rate USAID assistance to the Armenian pension system and/or employment services? 15. Would it help to make changes to the design of the next pension reform project in order to deliver assistance more effectively or to respond better to current challenges? Please explain in detail. 16. Do you have a specific recommendation that the evaluation team could provide to USAID to help develop similar assistance in the future? In light of your experience, what advice would you give USAID? Interview questions for USAID: 1. How does USAID/Armenia anticipate using this assessment? What are the special areas of concern? What should the assessment be sure to cover? 2. What were USAID’s objectives for the PALM project a. Pension reform b. Employment services 3. May we obtain the original SOW for the PALM project, as well as a timeline of modifications to the work over the life of the project? 4. Can the evaluation team obtain a timeline of the cost summary of the project? 5. Can you please provide the evaluation team with copies of the technical reports produced by the PALM project? 6. How do the individual components of the PALM project fit with USAID’s development strategy for Armenia? 7. Can you briefly summarize USAID’s objectives for the future pension reform project? 8. Are there policy or other issues that currently represent an obstacle to the achievement of USAID’s objectives, or cause disagreement between USAID and stakeholders in these sectors, including GOA? 9. What are the respective roles and responsibilities of USAID/Armenia, USAID/ Washington, and the Armenia implementing agency in terms of project implementation? What were the respective roles in the design of the project? 10. In your view, how sustainable are the changes brought about by the projects? 41 11. How effective was the project’s contractor? How effective are the targeted recipients of the assistance? 12. What, in your opinion, were the main constraints in effectively carrying out the assistance? 13. In your opinion, does the project have a sufficient human capital development component or an institutional strengthening component? If not, should these support activities be augmented? 14. Who are the key organizations or people in Armenia that USAID feels that the evaluation team should meet while conducting the evaluation? 42 ANNEX IV: SOURCES OF INFORMATION: LIST OF PERSONS INTERVIEWED N Name Position Organization Address Meeting Date Team Members 1. Stephen M. Brager Director, Democracy and Social Reform Office USAID USAID HQ 29.10.12 JA, CU 2. Tatshat Stepanyan Project Management Specialist Social Reform Office USAID USAID HQ 3. Mariam Gevorgyan Monitoring and Evaluation Specialist USAID USAID HQ 4. Susanna Karapetyan Pension Specialist PALM 30.10.12 JA, CU 5 Arsen Manukyan Senior Legal Specialist PALM 30.10.12 JA, CU 6 Arman Zaratsyan IT specialists PALM 30.10.12 JA 7 Ruzanna Marukhyan IT specialists PALM 8 Karen Hakobyan Head of Securities Market Methodology Division Financial System Reg Dept Central Bank of Armenia 6 Vazgen Sargsyan str., 0010, Yerevan, Armenia 31.10.12 JA 9 Lilit Manvelyn Securities Market Expert Financial System Reg Dept Central Bank of Armenia 6 Vazgen Sargsyan str., 0010, Yerevan, Armenia 10 Naira Avchyan Securities Market Expert Financial System Reg Dept Central Bank of Armenia 6 Vazgen Sargsyan str., 0010, Yerevan, Armenia 11 Stella Mikaelyan Financial Lawer PALM 31.10.12 JA, CU 12 Gayane Dallakyan PALM 31.10.12 JA, CU 13 Tigran Tananyan PALM 14 Haykuhi Gevorgyan Director Youth Professional Orientation Center 2 Baghyan str. Nor Nork, 0056 Yerevan 01.11.12 JA 15 Sona Harutyunyan Head of SESA State Employment Services Agency K. Ulnetsu 68, 0069 Yerevan 01.11.12 JA 16 Artak Simonyan Deputy Head of SESA State Employment Services Agency K. Ulnetsu 68, 0069 Yerevan 01.11.12 17 Vahe Mambreyan Director AM Partners Consulting Company Ulnetsi 62/1, apartment 1 01.11.12 JA 18 Armen Abrahamyan Director STATE SOCIAL SECURITY SERVICE 13 Nalbandyan St., 0010, Yerevan, Armenia 01.11.12 JA, CU 19 Arsen Manukyan Senior Legal Specialist PALM Hotel Golden Tulip 01.11.12 JA 20 Vardan Aramayan Deputy Minister Ministry of Finance of Republic of Armenia 1 Melik-Adamyan str. 02.11.12 JA 43 N Name Position Organization Address Meeting Date Team Members 21 Karen Tamazyan Head, Licensing Agency Ministry of Finance of Republic of Armenia 1 Melik-Adamyan str. 02.11.12 22 Armen Saroyan Director "Financial Banking College" Foundation 26 Amiryan, 0002, Yerevan, RA 02.11.12 JA 23 Hayk Voskanyan Adviser to the President Capital Asset Manager 27 Amiryan str., Yerevan 0002, RA 02.11.12 JA 24 Artak Ghazaryan CEO PSAC Foundation Mher Mkrtchyan str., Building 8 05.11.12 JA,CU 25 Hakob Hakobyan Social Affairs Committee Chair NA of RA, Chairman National Assembly of RA 19 M. Bagramyan Ave., Yerevan 0095, Armenia 05.11.12 JA,CU 26 Karine Minasyan Deputy Minister, Member of Pension Reform Board Ministry of Economy 5 Mher Mkrtchyan St., 0010, Yerevan, Armenia 05.11.12 JA 27 Smbat Saiyan First Deputy Head of SSSS SSSS / Ministry of Labor and Social Issues 13 Nalbandyan St., 0010, Yerevan, Armenia 05.11.12 JA 28 Tatshat Stepanyan Project Management Specialist, Social Reform Office USAID Golden Tulip 06.11.12 JA, CU 29 Nune Kochinyan M&E Specialist PALM Golden Tulip 06.11.12 JA 30 Arsen Manukyan Senior Legal Specialist PALM Hotel Golden Tulip 06.11.12 JA 31 Vahtang Mihrumayan Deputy of Chairman State Revenue Committee of RA 3 M. Khorenatsi street, Yerevan, Armenia 07.11.12 JA 32 Susanna Karapetyan Pension Specialist PALM Hotel Golden Tulip Tel 094 781 800 07.11.12 JA 33 Tigran Tananyan Labor/Employment Specialist PALM Hotel Golden Tulip 07.11.12 JA 34 Aharen Chilingaryan Former Board Member Central Bank of Armenia Hotel Golden Tulip Tel 077 800 650 08.11.12 JA,CU 35 Arman Zaratsyan IT specialists PALM 099 127 001 Hotel Golden Tulip 08.11.12 JA 36 Anahit Martirosyan Head of International Department Ministry of Labor and Social Issues Republic Square, Government Building N3 09.11.12 JA, CU 37 Artak Simonyan Deputy Head of SESA State Employment Services Agency K. Ulnetsu 68, 0069 Yerevan 09.11.12 JA, CU 38 Karen R. Hilliard USAID Mission Director USAID USAID HQ 09.11.12 JA, CU 39 Stephen M. Brager Director, Democracy and Social Reform Office USAID USAID HQ 40 Tatshat Stepanyan Project Management Specialist, Social Reform Office USAID USAID HQ 41 Michael Pashayan Head of IT Department State Revenue Committee of RA 3 M. Khorenatsi street, Yerevan, Armenia 10.11.12 JA Not Interviewed Artem Asatryan Minister Ministry of Labor and Social Issues Republic Square, Government Building N3 N/A 44 N Name Position Organization Address Meeting Date Team Members Nerses Yeritsyan Deputy Chairman Central Bank of Armenia 6 Vazgen Sargsyan str., 0010, Yerevan, Armenia N/A 45 ANNEX V: CONTRACT STATEMENT OF WORK: GOALS, TASKS, EXPECTED RESULTS EXECUTIVE SUMMARY The social protection1 challenge for Armenia is to strike the right balance between providing an adequate social safety net and promoting economic growth. To that end, Armenia should strive to develop a competitive labor market that promotes growth as well as fosters affordable public and private systems for individuals to deal with short-term and long-term social risks (sickness, disability, old age, survivors, family/children, and unemployment) that promote equity. The goals of the USAID Pension and Labor Market (PALM) Reform Program in Armenia are to: support pension reform implementation and labor market interventions aimed to assist individuals, households, and communities better manage social risks or needs, provide support to the vulnerable employed and jobless individuals (particularly the low-skilled, women, and the disabled), and give greater attention to countering the effects of the global economic crisis. To achieve this, the PALM program will provide technical assistance and support for human and institutional capacity building, and will target two key social protection areas: • Pension reform implementation – build the Government of Armenia (GOAM) institutional and human resource capacity to implement a multi-pillar pension system that will provide an adequate income for its elderly population in an environment that is fiscally sound and sustainable, with reforms to the current pay-as-you-go (PAYG) scheme and the creation of a private retirement savings system; and • Labor market interventions – help the GOAM and the most vulnerable individuals to mitigate the economic and social consequences of the global economic crisis by assisting the government and the private sector to: 1) give greater attention to countering the effects of the economic downturn which significantly affected the already inefficient Armenian labor market, 2) catalyze new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps, and 3) improve the country’s workforce readiness and competitiveness with an integrated economic and social approach. STATEMENT OF WORK Overall Program Objectives The goals of the USAID Pension and Labor Market (PALM) Reform Program in Armenia are to: support pension reform implementation and labor market interventions aimed to assist individuals, households, and communities better manage social risks or needs, provide support to the vulnerable employed and jobless individuals (particularly the low-skilled, women, and the disabled), and give greater attention to countering the effects of the global economic crisis. To achieve this, the PALM program will provide technical assistance and support for human and institutional capacity building, and will target two key social protection areas: • Pension reform implementation – build the Government of Armenia (GOAM) institutional and human resource capacity to implement a multi-pillar pension system that will provide an 1 A modern concept of social protection includes three key objectives that should: (1) assure minimum well-being through a guarantee of essential goods and services that provide protection against life contingencies for all people; (2) adopt proactive strategies and policies to prevent and protect against risks; and (3) promote individual and social potentials and opportunities. The foundation of these objectives would be to promote poverty reduction and sustainable economic development in a participatory manner. 46 adequate income for its elderly population in an environment that is fiscally sound and sustainable, with reformed current pay-as-you-go (PAYG) scheme and the newly created private retirement savings system; and • Labor market interventions – help the GOAM and the most vulnerable individuals to mitigate the economic and social consequences of the global economic crisis by assisting the government and the private sector to: 1) give greater attention to countering the effects of the economic downturn which significantly affected the already inefficient Armenian labor market, 2) catalyze new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps, and 3) improve the country’s workforce readiness and competitiveness with an integrated economic and social approach. By the end of the PALM project, the Contractor shall have increased the capacity of the GOAM, local authorities, and other key stakeholders, as appropriate, to: (a) implement, administer, and further refine a new multi-pillar pension system that provides adequate retirement income in a fiscally affordable and sustainable way; (b) create pension reform developmental effects by minimizing negative impacts (e.g. labor market) and leveraging on positive impacts (e.g. financial market development); (c) prepare adequate labor market policy responses, develop contingency plans and targeted interventions needed to counter the effects of the economic downturn and to address labor market gaps; (d) increase the employability of the workforce and support the vulnerable unemployed and workers (particularly low-skilled workers, labor migrants who lost their jobs, women, and people with disabilities). Anticipated tasks and services to be accomplished by the Contractor shall include technical assistance to GOAM and other counterparts for, but not limited to: • Addressing institutional and human resource capacity building needs necessary to develop, implement, and administer modern social protection system • Developing and implementing pension and labor market policy actions, using leading-edge analyses and providing timely and appropriate responses • Addressing consequences of global economic and jobs crisis by combining emergency action with the important structural reforms and support to innocent victims of the crisis (e.g., stimulus package for wider coverage of unemployment benefits and insurance schemes, public investment in infrastructure and housing, SME support, etc.) • Applying modern information and communications technology (ICT) solutions • Building political commitment and ownership by linking project activities to strategic priorities of the country and the government • Raising public awareness, public education, and information Component 1: Pension Reform Implementation The objective of Component 1 is to strengthen Armenia’s human resource and institutional capacity to enable the government and other key stakeholders to introduce and complete the implementation of the new multi-pillar pension system. Under the PALM project, USAID’s technical assistance will continue shifting from predominantly policy analysis, legal drafting, and basic education and training to assisting the GOAM implement its reformed pension system, including, but not limited to, providing intensive public education and information campaign technical assistance. 47 Component One Goal: The GOAM implemented a multi-pillar pension system that will provide an adequate income for Armenia’s elderly population in an environment that is fiscally sound and sustainable. Task 1. Help the GOAM strengthen its administrative capacity and improve business operations to introduce a new multi-pillar pension system Illustrative Activities: • Identify measures and establish a timeline to address the political, administrative, financial, and other challenges, necessary to launch and implement the new pension system successfully and within the timeframes set out in the governing laws and regulations • Conduct a comprehensive estimate of the new pension system’s costs that should include: (i) the cost of the zero (universal social pension) and first (employment-related) pillars which, as planned, are to be totally financed from the general budget; (ii) the cost of pension increases for other than occupational pensioners, such as military officers and soldiers; (iii) the administrative costs of the required institutional changes to implement and manage the new pension system (first 5 to 7 years); and (iv) the costs of providing detailed pension reform information and customer service to the public before and during the reform • Finalize the business process design for the SRC and other affected entities such as the CDA after the pension reform laws are finalized and adopted2 • Recommend modern information and communications technology (ICT) solutions • Support GOAM to improve the revenue performance of the unified income tax system with operational improvements of SRC’s and other relevant entities’ business and administrative processes • Conduct ongoing pension modeling activities on different reform scenarios • Minimize negative labor market and fiscal impacts, whenever possible, and reduce administrative and other compliance costs to businesses and insured persons • Develop, tailor, and implemented capacity building training programs for policy makers and others responsible for the implementation, regulation, and administration of the multi-pillar pension system3 Task 2. Finalize the legal and regulatory framework related to public pension system refinement (social assistance Pillar 0 and pay-as-you-go Pillar 1) and the introduction of mandatory and voluntary individual retirement savings (funded Pillars 2 and 3) schemes Illustrative Activities: • Identify significant missing, conflicting, or incomplete elements that must be addressed in future laws and regulations to improve the effectiveness and efficiency of the multi-pillar pension system • Draft regulations and by-laws needed to successfully implement the new pension system 2 The business process design for SRC and other affected entities such as the CDA can be finalized only when the pension reform laws are finalized and adopted by the National Assembly (parliament). Pension reform successful implementation will rest heavily on modernizing SRC core systems (particularly those related to filing and payment, audit, and collection enforcement), which will take significant time and resources. 3 For instance, redeploy staff to SRC from SSSS, which now collects personified data. 48 • Develop recommendations to address the payout phase of the mandatory retirement savings Pillar 2 system, based on international standards and best practices Task 3. Improve public understanding and confidence in the new multi-pillar pension system Illustrative Activities: • Develop and implement outreach and public education campaigns that provide detailed and easily understood public messages and customer service to the public prior to and after the new pension system is introduced, crucial to the success of the new pension system, with a particular focus on key messages to young people • Provide financial literacy education, especially as it relates to retirement savings and investing, with the special focus on the first cohort of individuals who will start contributing to the new pension system • Involve a broad cross section of the community in the pension reform dialogue to ensure the successful outcome of the reform • Conduct public opinion polls to assess public perception of the multi-pillar pension system Expected Results The Contractor’s performance shall be measured by its achievement of the following results by the end of the project: • Pension system administration modernized and business processes improved according to international standards and best practices • A full cost-benefit analysis of the multi-pillar pension system developed and addressed by the GOAM • Significant missing and contradictory elements in pension reform and other laws identified and addressed in draft legislative amendments to advance the implementation of the new multi￾pillar pension system • Comprehensive assessment of the quality of administrative readiness to execute the pension account numbering, information transfers, and asset management conducted and submitted to GOAM prior to the new pension system implementation date4 • Integrated pension processes and MIS/IT operations system developed and fully operational to handle registrations and centralized management of all aspects of the new pension system • At least two public opinion polls conducted – one at the beginning and one at the end of the project – to evaluate changed public perception of the multi-pillar pension system Expected Impact Final outcome of all the activities under Component 1 will be the implementation of the new multi￾pillar pension system that provides an adequate income for Armenia’s elderly population in an environment that is fiscally sound and sustainable, with minimized labor market distortions and negative fiscal impacts, and leveraged positive financial market impacts. A primary objective of the 4 This result will be expected if and only if the introduction of the mandatory retirement accounts system (Pillar 2) will be postponed until after January 2010. Otherwise, such an assessment would not likely be relevant as a matter of timing. 49 pension reform is to develop a pension system that responds efficiently to diverse kinds of shocks and is less subject to the risks of political interventions that create adverse consequences for short￾term political gains. USAID recognizes that the successful implementation of a multi-pillar system in Armenia will rest heavily on modernizing several administrative and operational systems in a number of pension institutions and related entities, particularly the SRC core systems (such as those related to tax filings and payment, audit, and collection enforcement). Component 2: Labor Market Interventions The overall objective of Component 2 is to give greater attention to countering the effects of the global economic downturn which significantly affected the already inefficient Armenian labor market, and help the GOAM improve country’s workforce readiness and competitiveness on the basis of integrated economic and social approach. The Contractor shall help the GOAM and the most vulnerable individuals to mitigate the economic and social consequences of the global economic crisis by assisting the government (particularly SESA), local authorities, and the private sector. Component Two Goal: The GOAM is institutionally capable to help the most vulnerable individuals to mitigate the economic and social consequences of the global economic crisis and improve country’s workforce readiness and competitiveness Task 1. Foster greater attention to countering the effects of the economic downturn and catalyze new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps Illustrative Activities: • Develop policies and implement programs that bolster the social safety net given the negative effects in Armenia of the global economic downturn (job losses, increased informal economy, remittances flows slowed, social spending affected, new investments deferred, international trade decreased, and income inequalities expected to rise further) • Address the social and economic consequences emanating from job destruction and falling wages • Encourage social dialogue at enterprise, local, and national levels to overcome jobs crisis • Develop curricular, conduct short-term (2-4 months) trainings for selected key trades which are in demand in Armenia’s labor market, and provide subsequent job counseling and placement services to the graduates, in cooperation with private sector employers, other donors, VET institutions and local employment offices • Develop private sector online job search and employment counseling services provision to augment and complement the employment services offered by the public sector Task 2. Support State Employment Services Agency and its regional offices to implement active labor market measures and improve country’s workforce readiness and competitiveness Illustrative Activities: • Support SESA to create an easily accessible labor market information system that will provide information on job growth and demand projections, available employment opportunities, economic wage data, and training opportunities • Implement active labor market measures systemically integrated into SESA and its regional offices with performance-based monitoring and oversight functions to help job seekers and vulnerable groups gain the right skills for 21st century jobs and maintain a link with the labor market 50 • Help MLSA and SESA integrate national and local workforce development and placement strategies with job creation interventions and other development strategies to produce holistic solutions that will benefit Armenia’s population and economy • Explore the opportunities to use public-private partnerships in employment services delivery • Examine the workforce development for the labor-based technologies and related competitiveness challenges that Armenia faces in achieving its goal of becoming regionally and globally competitive in both traditional and new industries through the application of advanced technology, including but not limited to infrastructure, agriculture, textiles, light manufacturing, ICT-related industries, and alternative/green energy (these sectors are specifically mentioned to explore the potential for alternative energy sector growth/demand, and the others are areas in which Armenia could benefit in the longer-term) • Quantify (to the extent possible) the supply of new technology-enabled workers emerging from the various educational and preparatory institutions and the perceptions of how qualified such workers are likely to be • Identify and remove the labor market barriers and inefficiencies that impede job creation • Increase employment and economic opportunities for people with disabilities and strengthen vocational training programs to make them more responsive to labor market needs Expected results The Contractor’s performance shall be measured by its achievement of the following results by the end of the project: • Policies developed and law amended for wider coverage of unemployment benefits and insurance schemes against substantial losses of disposable income • SESA’s market share of notified vacancies increased to 25 per cent of total vacancies in the economy • Number of employers that recruited successfully through SESA branch offices increased by 25 percent (baseline TBD at beginning of project) • Self-service infrastructure for jobseekers and employers developed and implemented in at least 10 SESA regional offices in all marzes using the latest computer or ICT tools • Social dialogue and coordination mechanisms established between the private sector, local authorities, regional employment offices, and vocational training institutions at the marz level to address labor market gaps and promote local economic development Expected Impact Final outcome of all the activities under Component 2 will be a reformed State Employment Services Agency, along with its regional offices, as one of the key players regarding labor market mediation and workforce development. This institution, along with the private sector, will provide high quality employment services taking into account appropriate flexibility and responsiveness to ensure customer satisfaction among the majority of those who come in contact with these employment service providers, including employers and job seekers. These activities are intended to impact the business environment by improving the job mediation process, reducing unemployment, and improving the quality of the labor force. They are also intended to improve the effectiveness of job mediation and workforce development for the disadvantaged and other under-served groups. PALM’s results are intended to influence directly human capital development and better skills matching in Armenia. Ultimately, USAID’s technical assistance will impact Armenia’s ability to increase employee productivity, reduce employer costs, and allow enterprises to become more 51 profitable and competitive all in the interest of greater economic growth that will improve the standard of living for the citizens of Armenia. 52 ANNEX VI: PALM PROJECT PERFORMANCE MONITORING PLAN Pension and Labor Market Reform (PALM) project in Armenia Performance Monitoring Plan (Revised) October 18, 2012 INTRODUCTION This updated Performance Monitoring Plan (PMP) follows our proven Monitoring & Evaluation (M&E) methodology for USAID programs, which allows us to periodically refine, eliminate or add indicators to our PMP to better reflect project performance and the focus of our activities. In this revision of the PMP we have refined the indicators to incorporate our new understanding of PALM reporting requirements. The revised system has been streamlined to the extent practical to reduce the volume of data collection and reporting without compromising quality and consistency. RESULTS AND INDICATORS The revised PMP results and indicators are presented in Figure 1. Bolded results represent our contractual results and indicators. The table also includes a baseline value, and a set of yearly targets for each indicator. The baselines shown in Figure 1 are our reference points and present the value of the indicators as of the start of the PALM program (October 1, 2009). Yearly targets reflect our ambitious but realistic performance goals. While performance indicators can provide evidence of successful project results, they can also identify shortcomings in project activities and clarify and focus project objectives for management decision-making. Our PMP, therefore, will serve as an early warning system to indicate the need for making adjustments to project plans, scope of activities, counterpart accountability and other program elements. Regular and disciplined reporting and analysis of performance indicators will allow USAID and our management team to gauge progress throughout the project life cycle, identify problem areas for closer scrutiny and adjustment, and provide USAID with a valuable tool to support its financial sector stability activities in the PFS Beneficiary countries. PERIODIC REPORTING Data Collection and Analysis. Ongoing data collection will be the joint responsibility of Ms. Rosa Chiappe, the Chief of Party, and the local team. Ms. Chiappe will analyze and consolidate data into a PMP report submitted annually to USAID. Reporting. With indicator data collected at regular, defined intervals, the PMP team will be able to analyze actual results vs. indicator targets and demonstrate how activities conducted contributed to the Project’s results. 53 Figure 1 Results/Output Indicators Baseline Y1 Target Y1 Actual Y2 Target Y2 Actual Y3 Target Y3 Component 1: Pension Reform Implementation Result P.1 Pension System Administration modernized and business processes improved Number of business process maps developed 0 5 5 8 8 -- -- Concept Paper for Unified Income Tax and Personified Recordkeeping System developed n/a 1 0 1 1 -- -- Result P.2 A full cost-benefit analysis of the multi-pillar pension system developed Number of models developed to support policymakers in making informed decisions n/a 0 0 1 1 1 2 Number of simulations on pension system behavior over time and the fiscal implications of pension reform n/a 14 14 11 11 11 14 Result P.3 Significant missing and contradictory elements in pension reform and other laws identified and addressed State Pension (Pay-As￾You-Go System) Number of bylaws/regulations for which PALM provided support n/a -- -- 6 7 -- -- Funded Pension Number of bylaws/regulations for which PALM provided support n/a -- -- -- 26 4 10 Unified Income Tax Number of bylaws/regulations for which PALM provided support n/a -- -- 2 60 proposals for amendments made for all 3 laws -- -- Personified Recordkee ping of Income Tax and Funded Contributi on Number of bylaws/regulations for which PALM provided support n/a -- -- 4 2 6 Law on Investment Fund Number of bylaws/regulations for which PALM provided support n/a -- -- 4 1 10 54 Result P.4 Comprehensive assessment of the quality of administrative readiness to execute the pension account numbering, information transfers, and asset management conducted and submitted Data Exchange document for Unified Income Tax and Personified Recordkeeping System developed n/a 0 0 1 1 -- -- Connectivity Assessment for realization of data exchange for Unified Income Tax and Personified Recordkeeping System n/a 0 0 1 0.75 (not completed fully as MLSI cancelled the request) -- -- Result P.5 Integrated pension processes and MIS/IT operations system developed Functional Specification document for Unified Income Tax and Personified Recordkeeping System developed n/a 0 0 1 1 -- -- Result P.6 Public awareness and support for pension reform increased Number of surveys, public opinion polls, and focus groups supported n/a 3 0 3 1 -- -- Number of financial literacy events targeting the most handicapped literacy areas, based on the World Bank - funded nationwide financial literacy assessment n/a 1 1 5 12 12 18 Number of public awareness events conducted and supported, including seminars, workshops, round tables, consultations, etc. n/a 5 5 5 5 5 15 Result P.7 Capacity of pension reform stakeholder agencies involved in the pension reform process strengthened Number of training events conducted n/a 8 9 5 5 5 10 Number of Roundtables with stakeholders on legal issues related to pension system implementation n/a -- -- 10 20 20 20 Number of people trained Total 0 13 300 500 75 1800 Men 5 150 320 40 1100 Wome n 8 150 180 35 700 55 Result P.8 MLSI supported on disability pensions and financing of long-service and privileged pension benefits Recommendations developed on models for calculating disability pensions and financing long-service and privileged pension benefits n/a 0 0 2 2 1 1 Results/Output Indicators Baseline Y1 Target Y1 Actual Y2 Target Y2 Actual Y3 Target Y3 Component 2: Labor Market Intervention Result L.1 Policies developed and law amended for wider coverage of unemployment benefits and insurance schemes Employment Strategy designed in cooperation with counterparts n/a 0 0 1 2 -- -- Result L.2 SESA’s market share of notified vacancies increased to 25 per cent of total vacancies Number of vacancies disseminated through SESA 5872 10% increase over baseline 6649 15% increase over baseline 6194 25% increase over baseline 7659 Result L.3 Number of employers that recruited successfully through SESA branch offices increased by 25 Percent Number of employers that are using SESA services for recruitment 2200 2.5% increase over baseline 2615 10% increase over baseline 2527 25% increase over baseline 2364 Result L.4 Self-service infrastructure for jobseekers and employers developed and implemented Number of self-service booths operational 3 0 over baseline 0 6 6 -- -- Assessment of GORDZ system, including (1) connectivity, (2) reporting, search, and database inconsistencies, and (3) operation of the system at the self-service kiosks. Recommendations developed to address hardware and software issues n/a 1 1 1 1 -- -- GORDZ system hardware improved n/a -- -- 1 1 -- -- 56 GORDZ system in the self￾service kiosks provides self￾serving employment matching function 0 0 1 1 1 -- -- Result L.5 Social dialogue and coordination mechanisms established Number of job -clubs established 0 5 5 7 7 5 5 Number of nationwide Employment Surveys conducted n/a 2 2 2 2 2 1 Number of employment councils established (cumulative) 3 0 0 9 9 -- -- Result L.6 Improved capacity of MLSI, SESA and the Youth Center in development and implementation of programs on youth employment and employment of disabled people Review of international best practice in youth employment and employment of disabled people provided n/a -- -- 1 1 1 3 Concept paper on Your Vocational Orientation drafted n/a -- -- 1 1 -- -- International Study tour organized and conducted n/a 0 0 1 1 -- -- Result L.7 A comprehensive system of employment sector monitoring indicators in place A system of monitoring indicators designed per SESA needs; the system is tested and refined as required by counterparts n/a -- -- 1 1 -- -- Result L.8 Improved capacity of the Employment Centers to provide services Number of people trained in delivery of employment programs Total 50 150 149 10 19 Men 35 100 45 5 1 Women 15 50 73 5 16 Result L.9 Vocational guidance policy for youth in place Working Group for development of vocational guidance policy for youth formed and trained n/a 1 0 1 1 -- -- Vocational Guidance concept paper developed n/a 0 0 1 1 -- -- Number of public discussion held 0 0 0 3 3 -- -- 57 Result L.10 Improved capacity of the Youth Center to implement effective youth employment programs Module on training for young job-seekers designed and delivered n/a 0 0 1 0 -- -- 58 ANNEX VII: DISCLOSURE OF CONFLICTS OF INTEREST Name Joseph M. Anderson Title Senior Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pension and Labor Market Reform Project (PALM) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 6/4/13 59 Name Carl Ulbricht Title Local Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pension and Labor Market Reform Project (PALM) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Signature Carl Ulbricht Date 6/4/13 60 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 61