May 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Carl Ulbricht and John H. Sullivan through Social Impact Inc. Performance Evaluation of the Civil Society and Local Government Support (CSLGSP) Program Mid-term Evaluation Report Cover Photo: Students from the art school in Maralik perform auditorium newly renovated as a result of Counterpart’s Grassroots and Community Development Grants. Photographed by Tina Yesayan. PERFORMANCE EVALUATION OF THE CIVIL SOCIETY AND LOCAL GOVERNMENT SUPPORT (CSLGSP) PROGRAM Mid-term Evaluation Report May 17, 2013 Contract # AID-RAN-I-00-09-00019 Task Order # AID-111-TO-12-00002 Evaluation of Five USAID/Armenia Activities DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The authors would like to thank all those interviewed during the fieldwork for this evaluation, including USAID Mission and Counterpart staff members as well as community, civil society, government and donor representatives. They all without failing gave generously of their time to respond to questions and to provide materials and explanations relevant to the project. Without their contributions this report would not have been possible. We would also like to thank the Counterpart team as well as Kara Hakobyan for logistical support. CONTENTS Acronyms............................................................................................................................................................................. 6 Executive Summary ........................................................................................................................................................... 7 Evaluation Purpose & Evaluation Questions.............................................................................................................. 10 Project Background ......................................................................................................................................................... 10 Evaluation Methods & Limitations ............................................................................................................................... 14 Findings & Conclusions ................................................................................................................................................... 18 Evaluation Question 1 ................................................................................................................................................ 18 Findings ...................................................................................................................................................................... 18 Conclusions .............................................................................................................................................................. 21 Evaluation Question 2 ................................................................................................................................................ 23 Findings ...................................................................................................................................................................... 23 Conclusions .............................................................................................................................................................. 25 Findings ...................................................................................................................................................................... 26 Conclusions .............................................................................................................................................................. 27 Evaluation Question 4 ................................................................................................................................................ 28 Findings ...................................................................................................................................................................... 28 Conclusions .............................................................................................................................................................. 33 Evaluation Question 5 ................................................................................................................................................ 35 Findings ...................................................................................................................................................................... 35 Conclusions .............................................................................................................................................................. 35 General Conclusions ....................................................................................................................................................... 39 Recommendations ........................................................................................................................................................... 42 Annexes ............................................................................................................................................................................. 44 Annex I: Evaluation Statement of Work ................................................................................................................ 45 Annex II: Evaluation Methods and Limitations ..................................................................................................... 51 Annex III: Data Collection Instruments ................................................................................................................. 54 Annex IV: Sources of Information ........................................................................................................................... 57 Annex V: Disclosure Of Any Conflicts of Interest ............................................................................................. 58 6 ACRONYMS AED Academy for Educational Development CASP Civic Advocacy Support Program CoE Council of Europe COP Chief of Party CSLGSP Civil Society and Local Government Support Program CSO Civil Society Organization CSR Corporate Social Responsibility CWG Community Working Group FY Fiscal Year GIZ Gesellschaft fuer Internationale Zuzammmenarbeit ICA Inter-Community Association LAAD Legislative Agenda Advocacy Day LSG Local Self-Government MTA Ministry of Territorial Administration NGO Non-Governmental Organization NICRA Negotiated Indirect Cost Rate Agreement OSF-A Open Society Foundations - Armenia PMEP Project Monitoring and Evaluation Plan PPP Public-Private Partnership RFA Request for Application RTI Research Triangle International SOW Statement of Work USAID U.S. Agency for International Development Y/CAC Youth/Community Action Center 7 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this mid-term performance evaluation is to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial project designs are still valid in fostering the achievement of the original objectives. Findings from the mid-term evaluation will inform future work plans of the relevant projects, as well as designs of similar future activities. The evaluation findings will be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities, where applicable. The relevant Mission and project staff will develop plans for incorporation of relevant recommendations from the evaluations in their future work plans. This document is the Civil Society and Local Government Support Program (CSLGSP) performance evaluation report requested by the Mission. It is formatted around the five questions that were posed by the Mission in the scope of work (SOW) for the evaluation. The team’s findings and conclusions were achieved through the triangulation of information collected to answer the following evaluation questions:  How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia?  To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms?  To what extent has the program been successful in promoting decentralization?  How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)?  Do local governments/community councils and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? As CSLGSP has currently run two of its four years, the SOW poses the question: What can be improved for the remainder of the project given the resources available? Specific decisions to be determined by this evaluation may include the following:  Altering the balance of distribution of funds across the sectors covered by the project  Altering the modalities of interventions, including the grant-making process 8  Making adjustments to the management structure of the project  Decisions on the design of future projects or other USAID interventions PROJECT BACKGROUND As noted in the Request for Application, background to the project includes the following issues:  The domination of society by ruling elites and oligarchs, with a centralized and vertical power system that hinders civic engagement  The possibility of making progress in a number of areas such as decentralization, civic engagement in policy-making, public-private partnerships and community development  USAID’s history of engagement in local government (since 2000) and civil society (since 1994) development, albeit as a series of separate projects in each of the two spheres. The project’s three components (“Local Government and Civil Society Collaboration: Fostering Participatory Community Strategic Planning for Community Development and Improved Local Democracy”; “Fostering Civic Participation, Advocacy and Activism”; “Facilitating Decentralization and Local Fiscal Autonomy”) aim to strengthen local governance, build the capacity of civil society organizations and advance the reform agenda in the spheres of NGO governance, decentralization, local fiscal autonomy and civil society-government dialogue. EVALUATION DESIGN, METHODS AND LIMITATIONS The overall approach of the evaluation was qualitative analysis. Given the large number of beneficiaries (including 43 intervention communities where the project is working with the administrations and the community active groups, plus 388 grants given over the first two years of the project), it was not possible to interview a statistically representative sample of beneficiaries. Within the two weeks available for fieldwork, only a qualitative sample of the beneficiaries could be interviewed. In order to increase the number of interviewees, focus group discussions were held in five cases in order to get a range of opinions on common themes. Other stakeholders, such as central government and international donors were interviewed individually. Three comparison communities were visited, with community administrators interviewed. Interviews were supplemented by review of project documents as well as other material relevant to the sector. Limitations to the data collection included the inability to replicate the project’s baseline survey, as well as reliance on the project staff to select a balanced sample of beneficiary representatives for the focus groups. SPECIAL NOTE: Following the submission of the second draft of the CSLGSP report, the Mission, expressing some concerns, asked that a senior evaluation specialist for Social Impact review the report. In response to this request, Social Impact convened a panel of three evaluation experts, all of whom read the report and then went back to project documents. They keyed on the “change theory” that had been advanced for the project and questioned its validity, something that was not reflected in the five evaluation questions in the CSLGSP evaluation team’s scope of work, but nonetheless are deemed relevant. That omission was quickly seen by all parties as a limitation. As a result, the evaluation team has taken the further step of adding a question, providing findings and reaching a conclusion. The team believes that the Mission is owed this further inquiry. 9 FINDINGS, CONCLUSIONS & RECOMMENDATIONS The project design appropriately addresses the civil society and local governance issues in Armenia, working through the three components to strengthen cooperation between local government and civic active groups, enhance civil society and its role in the policy debate. However, implementation is imperfect due to the short-term nature of grants. This emphasis on small, short-term grants has also hindered the effectiveness of the project’s support to watchdog, advocacy and policy development mechanisms. Longer-term support is required in order to enable meaningful policy dialogue with government bodies to develop. Largely due to the political climate, efforts to promote decentralization have met with limited success. The project has enabled a coalition of NGOs to work on a strategic approach concept paper, and has supported inter-community associations as well as public-private partnerships. However, the enabling environment for these activities has not been developed further. The project staffing and structure are effective in supporting a large volume of work, although an additional post of deputy Chief of Party may enhance the project. Generally, indicator targets are likely to be met or exceeded, though the project monitoring and evaluation plan requires partial revision. Overall there is satisfaction by beneficiaries with the community development process, with both community administrations and civic action groups valuing Counterpart’s work in the 43 intervention communities. General Conclusions Briefly, these are:  The project is impressive with an effective staff, and its work is appreciated by beneficiaries  The large number of small grants impair effectiveness  Few inter-community associations have been supported by the project  Civic active groups and other grantees do not have many opportunities to share experiences  Sustainability of the various groups of beneficiaries is likely to be inconsistent Reflecting the Social Impact consensus view after the expert panel review, questions about validity of the “change theory” at the heart of the project indicate that success on this issue over the first two years of funding has been patchy at best. It is suggested that the Mission conduct its own review of the change theory to assure its effective implementation in the remaining months of CSLGSP. Recommendations  Adopt a more strategic approach to grants, being more realistic about the size and length of the grants required to achieve an objective  Increase the opportunities for peer sharing of experience  Increase support to inter-community associations  Revise the project monitoring and evaluation plan  Actively incorporating the USAID “Forward” policy, consider spinning off the project staff into an NGO 10 PROJECT BACKGROUND The Request for Application (RFA) was issued on June 18, 2010 and provides as a background to the proposed evaluation a number of relevant issues:  The domination of society by ruling elites and oligarchs, with a centralized and vertical power system that hinders civic engagement  The authorities’ stated goal to rebuild trust after the 2008 presidential elections and take advantage of the window of opportunity to make progress in a number of areas such as decentralization, civic engagement in policy-making, public-private partnerships and community development  USAID’s history of engagement in local government (since 2000) and civil society (since 1994) development, albeit as a series of separate projects in each of the two spheres. The present project in effect merges into one program three sectors that had previously been dealt with separately. Its precursors are Civic Advocacy Support Program (CASP), a civil society development project that had been implemented by Counterpart, a civic activism and community development project run by Academy for Educational Development (AED) that established the Youth/Community Action Centers (Y/CACs), and a local government development project that had been implemented by Research Triangle International (RTI). Overall project description The project was awarded to Counterpart International, Inc. on September 30, 2010, as an Associate Award, in which the USAID share amounted to $15.5m and the Counterpart Cost Share amounted to $3.1m. The program objective is described in the Project Monitoring and Implementation Plan (PMEP) as: Increase the level of informed and organized civic activism at the local and national levels, along with more participatory, decentralized, efficient and accountable local governance that leads to a more democratic society. The project consists of three main components and three sub-components, as follows:  Component 1: Local Government and Civil Society Collaboration: Fostering Participatory Community Strategic Planning for Community Development and Improved Local Democracy  Component 2: Fostering Civic Participation, Advocacy and Activism o Component 2.1: Fostering Civil Society Input in Policy Formulation, Implementation and Monitoring o Component 2.2: Fostering Grass Roots Civic Activism and Volunteerism through Mobilization and Networking o Component 2.3: Facilitating Informed Citizen Participation in Elections  Component 3: Facilitating Decentralization and Local Fiscal Autonomy A total of 43 communities were selected from an initial list of around 80, and the project has worked intensively with them over the two years, supporting the drafting and revision of development strategies, the creation and/or strengthening of Community Working Groups and Youth/Community Action 11 Centers, and the cooperation between community administrations and these volunteer groups as they undertake small projects. The split of funds among the various components is currently:  25% local self government (LSG) (USAID has stipulated this as a maximum)  35% community development  40% civil society It is planned to disburse a total of $7m in grants over four years, and in the first two years the project has disbursed a total of 346 grants or, if one also counts a number of sub-grants: 388. 12 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The evaluation of the Civil Society and Local Government Support Program (“CSLGSP”, also described as “the project”) is a mid-term one, the project having started two years previously, with two years to run until completion. As stated in the Statement of Work (SOW): The purposes of the mid-term evaluations are to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Findings from the mid-term evaluations must inform future work plans of the relevant projects, as well as designs of future similar activities. This evaluation of CSLGSP is conducted at a point where the project has sufficient history to evaluate its progress to date, while at the same time there is still an opportunity to make adjustments to the direction of future activities. As the SOW asks specifically in relation to CSLGSP: What can be improved for the remainder of the project given the resources available? Accordingly, it is expected that the findings will be used to:  Inform understanding of the sectors covered by the project, and of the project’s impact on those sectors  Provide information on the successes and failures of the project  Aid the process of influencing policy development Specific decisions that can be informed by this evaluation might include the following:  Altering the balance of distribution of funds across the sectors covered by the project  Altering the modalities of interventions, including the grant-making process  Making adjustments to the management structure of the project  Decisions on the design of future projects or other USAID interventions The main audience for this report is the USAID Yerevan Mission, but other audiences may include project management staff as well as relevant donor organizations and Non-Governmental Organizations (NGOs). EVALUATION QUESTIONS The evaluation questions as listed in the original statement of work are as follows:  How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? 13  To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms?  To what extent has the program been successful in promoting decentralization?  How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)?  Do local governments/community councils and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? 14 EVALUATION METHODS & LIMITATIONS Bearing in mind the nature of the evaluation questions and the overall purpose of the evaluation, a qualitative methodology was employed. The key sources were relevant program documentation, interviews with program staff and stakeholders (including grantees and other beneficiaries), other local/ international actors in the civil society and local government spheres, USAID representatives, and central government representatives. As the bulk of the interviews were undertaken over a two-week period, it was clearly not possible to interview all or even a majority of the beneficiaries, given that the program had awarded over 300 grants in its first two years. Therefore only a sample of beneficiaries could be interviewed. The evaluators could, in theory, have visited a number of intervention communities and also conducted separate interviews with a range of grantees, but this, it is submitted, would have been an inefficient use of the time available. Thus, in order to increase the number of interviewees, focus group discussions were held in several cases in order to get a range of opinions on common themes. This methodology was employed when interviewing civil society grantees, Y/CAC & Community Working Group (CWG) representatives, community mayors, Inter-Community Association (ICA) representatives and Local Self￾Government (LSG) sector NGOs. Thus, a total of five focus groups were held. Counterpart was requested to make the appointments for the focus groups, with clear instructions that the composition of each focus group should be a mixture of representatives from successful and unsuccessful communities/grantees. In the case of the CSO, mayor and Y/CAC / CWG focus groups, Counterpart was requested to contact ten people/organizations for each focus group, on the basis that ten was the maximum number of participants suitable for holding a focus group, although not all of those invited might turn up. The number of interviewees actually attending was as follows: six CSO grantees, eight community mayors, nine Y/CAC & CWG representatives, eight ICA representatives and five LSG sector NGO representatives. In the case of LSG sector NGOs, a total of six NGOs had received grants from Counterpart to work together on the decentralization strategic approach, and representatives of each of them were invited. It would not have been feasible to interview a similar number of beneficiaries individually and through visits to intervention communities during the two-week period. Other stakeholders, such as central government and international donors were interviewed individually. Three comparison communities were visited, with community administration interviewed, in order to facilitate a qualitative comparison between intervention and comparison communities. As indicated above, it was decided to hold focus groups rather than visit intervention communities, in order to gain information from as wide a range of communities as possible within the time available. For that reason, no visits were made to intervention communities. In contrast, it did not seem feasible to invite representatives of comparison communities to Yerevan for a focus group, as that might lead to unrealistic expectations that they would become beneficiaries of the project. Thus, it was decided that it would be better to visit a range of comparison communities, but the number was restricted by the time available, and the choice of the three (Ptghni, Aramus and Hrazdan) was based on the fact that they were all reasonably close to each other, being in the same region and no more than an hour’s drive from Yerevan. In addition, they represented a mixture of two small communities and one large town, with corresponding differences in terms of their needs and the size of their administrations. 15 In order to assess whether the project was on track for achieving results and benchmarks, the SOW requested that the evaluator “use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects.” However, at the time of data collection there had only been one survey undertaken by the project: a baseline survey of the intervention and comparison communities done shortly after commencement of the project and before any significant interventions had been carried out. The project has not undertaken any monitoring of the comparison communities since then. Thus, there is no recent data which would enable an assessment of the achievements of the project in the intervention communities by comparing with the comparison communities. Moreover, it was beyond the resources of the evaluation to attempt a survey of the communities similar in scope to the baseline survey. Thus, in lieu of a survey, during the evaluation three comparison communities were visited and relevant staff in the community administration interviewed. The question of undertaking a baseline survey was raised with the Mission staff in Yerevan at the start of the evaluation fieldwork, and it was agreed that it would be beyond the scope of the evaluation to attempt a quantitative survey. However, as a result the conclusions about the effectiveness of program implementation vis-à-vis the comparison communities are not based on quantitative data, and instead rely on a qualitative evaluation. In addition, this meant that there was no direct sampling of the views of ordinary citizens from the communities. Instead, the focus was on sampling community representatives. SPECIAL NOTE: Following the submission of the second draft of the CSLGSP report, the Mission, expressing some concerns, asked that a senior evaluation specialist for Social Impact review the report. In response to this request, Social Impact convened a panel of three evaluation experts, all of whom read the report and then went back to project documents. They keyed on the “change theory” that had been advanced for the project and questioned its validity, something that was not reflected in the five evaluation questions in the CSLGSP evaluation team’s scope of work but nonetheless relevant. That omission was quickly seen by all parties as a limitation. As a result, the evaluation team has taken the further step of adding a question, providing findings and reaching a conclusion. The team believes that the Mission is owed this further inquiry. 16 Table 1: Tabular Overview of Evaluation Methods and Limitations EVALUATION QUESTION EVALUATION METHOD DATA SOURCES NOTES ON SAMPLE SIZE LIMITATIONS How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? Key informant interviews; focus group discussions; document review Relevant USAID program reports; relevant stakeholders Interviewees included four USAID mission staff, five Counterpart employees, six CSO grantees, eight community mayors, 11 Y/CAC & CWG representatives, eight ICA representatives, five LSG sector NGOs, five donor representatives, one government representative Not possible to interview a statistically representative sample of grantees and communities in the time available; choice of participants for focus groups was guided by evaluation team, but made by Counterpart To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms? Key informant interviews; document review Relevant USAID program reports; relevant stakeholders To what extent has the program been successful in promoting decentralization? Key informant interviews; focus group discussion; document review Relevant USAID program reports; relevant stakeholders How effective is the project implementation: Is the project management structure and staffing appropriate for the effective implementation of the project? Key informant interviews; document review Relevant USAID program reports; relevant stakeholders How effective is the project implementation: How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? Key informant interviews; focus group discussions; document review Relevant USAID program reports; relevant stakeholders 17 How effective is the project implementation: Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)? Key informant interviews; document review Relevant USAID program reports, in particular the PMEP and the baseline survey; relevant stakeholders including project staff and selected communities Visits made to three comparison communities and community administration interviewed. There had been no further survey done since the initial baseline survey, and there were not sufficient resources to attempt an independent survey, therefore qualitative interviews with 3 comparison communities were undertaken, as well as interviews with representatives of intervention community administration and Y/CACs/CWGs Do local governments/community councils, and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? Key informant interviews; focus groups; document review Relevant USAID program reports; relevant stakeholders See response to first paragraph above See response to first paragraph above 18 FINDINGS & CONCLUSIONS EVALUATION QUESTION 1  How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? FINDINGS Civil Society needs and the project’s response A Policy Action Brief1 published in 2010 by Counterpart under the auspices of a previous USAID project reflects on the state and needs of civil society in Armenia, and lists a number of recommendations as to how civil society could be strengthened. These include:  Establishing a legislative framework that supports the long-term growth of CSOs  Expanding government-CSO collaboration  Transforming businesses’ support of CSOs into long-term strategic collaboration  Increasing the potential of volunteerism  Enhancing CSO networking and communications  Increasing CSO accountability and improving their internal governance systems  Turning the competition among CSOs into cooperation and coalition building Other major challenges were mentioned also by key informants in the donor community. One is the capacity of civil society organizations to undertake deep analysis of existing shortcomings. In other words, it is comparatively simple to observe imperfect systems or to monitor government programs, but it is harder to turn initial observations into valid conclusions and cogent arguments for reform. The capacity of Armenian NGOs to respond to this challenge is varying. The other challenge is enabling civil society-government dialogue. Recognizing the above issues, the project has been engaged in a wide range of activities with civil society organizations, including local and national advocacy, policy research, human rights issues and capacity building. The activities reflect the appropriate understanding at commencement of the project regarding the needs of CSOs and the way to promote their active involvement in the policy debate. This has included encouraging networking, as well as building coalitions to work on issues such as NGO legislation and decentralization strategy. However, as discussed under evaluation question 2 below, while the conceptual approach has been sound, implementation has had one drawback: the generally short￾term approach to grant-giving has constrained the ability to engage in long-term dialogue and capacity￾building efforts. 1 “Armenian Civil Society: from Transition to Consolidation”, Yerevan 2010 19 The struggle for LSG reform “If central government does not change, then in three years’ time we won’t see much development.” “We haven’t taken the initiative on legislative changes such as on borrowing and ICAs. Local government official says ‘Not now, I’ll tell you when.’” “I realize we need to go higher [i.e. seek a dialogue with top government officials] with reform issues but everyone is saying ‘Wait until after the [2013 Presidential] elections.’” ‐ Implementing Partner leadership In the fiscal year 2012, 227 grants were issued, of which 157 were for activities wholly rooted in individual communities or were awarded to CSOs which addressed issues in particular groups of communities or regions. Continuing the list of civil society needs, another important issue is that of active civil society participation in election processes. Ensuring transparency in elections in a nascent democracy is a hard task, where civil society activism has an important role to play. In Armenia there are a variety of CSOs with special interest in monitoring the election process. Increasingly, the challenge is in finding ways of benefitting from modern technology to enable quick, coordinated responses to perceived Election Day breaches. The project has played its role in coordinated donor efforts during the 2012 parliamentary elections to ensure transparency of the process, by issuing grants to 32 NGOs, covering a mixture of issues including awareness raising, active participation, observation and the use of modern technology to report breaches. Local Governance issues and the project’s response Key informants in the donor community recognize that the last decade or so has seen very little progress in the transfer of power and a corresponding increase in financial capacity from central government to local governance. Most community administrations lack sufficient financial means to provide quality services to the population, and the excessive fragmentation of administrative resources (there are 915 communities for a population of three million) exacerbates the problem. Despite the lack of substantive reform to date, the government acknowledges the issue and professes to be committed to a process of, on the one hand, progressive decentralization and strengthening of local fiscal autonomy and, on the other hand, consolidation of communities so that existing financial resources are better used. The deputy Minister of Territorial Administration acknowledged that reforms have to be accelerated in the future, while government policy documents such as the action plan developed with the Council of Europe and, more recently, the government’s 2012-2017 Programme, set out the overall direction of reforms. Inter-community associations (ICAs) are another way of combining resources to enhance service provision, and as such have been developing slowly in Armenia since the late 1990s. An ICA is an entity established by a group of communities, sometimes with a specific goal in mind, such as joint administration of local taxes. A total of 58 ICAs are currently registered in Armenia, but many of them are dormant, either because, after initial enthusiasm, participating communities have not been able to overcome the obstacles to working together, or because initial donor support has faded before sustainable cooperation models could take root. The lack of legislative regulation of this form of local government activity has also hindered the process. The project has focused on improving the capacity of local government. As well as training, there has been a focus on financial capacity, service delivery and accountability. The development of partnerships in the intervention communities between the administration on the one hand and civic active groups on 20 Contrasting Data from Three Control Communities Control Community 1 “15 years ago the Jinishian Foundation did a project here.” Village mayor when asked about recent donor projects Control Community 2 “We have village meetings three times a year. There are four committees, covering education, health, agriculture, culture and sport. We disburse pensions for war veterans and bursaries for excellent students, and subsidize winter heating for poorer residents.” Village mayor Control Community 3 “Council meetings are televised. The administration carries out surveys of the suburbs to assess development priorities and undertakes a consultation process. The mayor is available in his office every Friday for citizens’ enquiries. There are 17 local NGOs.” City administrative staff the other has been the main route to greater accountability. By focusing on 43 partner communities (as detailed below), the project has been able to work intensively with the beneficiaries. Community Development To implement this component of the project, some 43 communities were selected throughout Armenia for special attention, in addition to supporting a further 16 Y/CACs in other communities. Implementation of community development activities involved participation in a process of organization at the community level through a local decision-making body that would jointly agree on priority projects for the community that might be funded by USAID through the contractor. One of the evaluation team’s focus groups was made up of a cross-section of members of these local bodies. The team also learned details of the process from Counterpart staff, several of whom were regular attendees at the meetings of these local groups. The opening step was the signing of 43 Memoranda of Cooperation that were signed with ceremony at a Community Perspectives Public Dialogue Participatory Governance Conference held in Yerevan on June 3, 2011. One key informant suggested that the number was too large and that 10 to 15 communities would have allowed a more concentrated effort, and that numbers could have been enlarged upon as successes followed. However, the Counterpart response was that USAID wanted a substantial number of communities that would represent a cross-section and representation in all ten marzes (provinces). In addition the Mission wished to work in localities where Y/CACs were extant. The evaluators’ initial impression on being acquainted with the project documentation was that indeed 43 communities might be too many in order to make a significant impact in each one. However, no such evidence (such as dissatisfaction among community beneficiaries with the level of Counterpart involvement and support, or comments from other key informants) could be found to sustain that conclusion. Community development has long been an important programmatic intervention in the overall US foreign assistance portfolio of activities worldwide, preceding in time even the 1961 creation of the Agency for International Development. The “logframe” criteria for designing and implementing such programs contain some well recognized limitations, and has been established and utilized for a several decades. With that history in mind, the evaluation team asked a series of questions of both relevant focus groups and the contractor staff to elicit the process through which community development has been proceeding. We were able to receive a fairly accurate picture of how the community groups were formed, their meetings and the openness of same, the flow of discussion, decision-making practices, the monitoring done by outside resources, and the ability of the contractor to respond to requests. The second key element in community development is the provision of outside funding for the project or projects that have been identified through the participatory process 21 outlined above. Community awareness and action is met with new resources to realize the resulting prioritization of needs. This element very often involves construction and/or procurement processes that are far afield from usual local governance technical assistance requirements. Providing physical infrastructure, either “de novo” or in rehabilitation, requires special expertise. Counterpart, having recognized this need as the project grants are being implement, has recently hired an engineer and has plans for engaging others. It also is implementing a policy of discouraging purely construction-related projects through the community process and emphasizing projects that provide some measure of social capital (e.g. establishing daycare centers and playgrounds, rather than school buildings and trash dumpsters). While maintenance and sustainability issues will always be a cause for concern in community development, both beneficiaries and grantor indicated their knowledge and awareness of the issues. Visits to the administrations of 3 control communities revealed a range of views: one was rather passive, one was actively supporting citizens but in a rather paternalistic way; the third appeared to be practicing a genuinely more open type of governance with CSO participation, but in all three cases the attitude of the mayor played a key role. Two of the three communities were villages where there was little if any civic activism, and thus much depended on the ability and willingness of the mayor to initiate projects and, where necessary, attract donor funding. Thus, there tends to be a rather top-down process of project selection and implementation. In the larger community, Hrazdan, civil society was more active and the mayor was said to be willing to enter into dialogue. CONCLUSIONS Civil society needs are being addressed The project touches on almost all the issues that concern civil society development, including:  A working group has been formed to work on improving the NGO legislative framework  The wide-ranging grants program is supporting a number of areas such as advocacy, policy research, volunteerism and networking  Creating coalitions on issues such as NGO legislation and decentralization has encouraged CSOs to work together rather than compete for funds  Election grants have responded to individual initiatives, but also championed networking and use of ICT  The project recognizes that more work needs to be done in the future on NGO internal capacity building The project, through its work on policy and advocacy issues, has helped CSO-government dialogue. On one level, this is not such a challenge, in that it is comparatively easy to make initial contact with relevant state bodies and to engage in discussion with lower-level officials. But, the actual goal of influencing public policy is much more elusive and will require repeated efforts over a longer time-frame. In that respect, the short life-span of many of the grants issued does not support sustained efforts. Local governance issues are being addressed The project’s approach to local government development reflects the main issues affecting the sector. The need to improve the legislative framework is recognized in the project’s work on developing a strategic approach to decentralization, as well as its continuing advocacy on the issue of financial independence (e.g. public-private partnerships and the ability to borrow). Capacity building needs are recognized through the project’s training activities and more generally in its partnership with 43 22 communities. A number of ICAs have been assisted through the grants facility. More details specifically on the issue of decentralization are discussed under evaluation question 3. The project’s approach to community development successfully engages the stakeholders Component 1 is quite innovative in its combination of support to local government, civic activism and community development. In the past, donors engaged in community development have often experienced problems when it has been difficult to engage the whole community: they either work mainly with the community administration, (in which case the danger is that the chosen project is a pet concern of the mayor but not considered a priority by a majority of the community), or they convene a community meeting to assess priorities, running the risk of choosing a project which is not wholly endorsed by the mayor. The design of the project has attempted to counter these risks by developing long-term trilateral collaboration: project-community administration-local initiative group. Inevitably, though, as with any community development initiative, the common wish to focus on infrastructure projects leads to the danger that improvements are only temporary. Such projects achieve success and sustainability to the extent that they mobilize local people, not just those chosen for the community councils, but those who attend their meetings, contribute their views, and are in various ways engaged beyond their immediate family concerns in that they consider the welfare of the entire village or town. In the intervention communities the project is fostering a more horizontal model of collaboration, where the administration and civil society representatives are partners in the process of project identification and implementation. This contrasts with a more vertical form of administration-population relationship seen in the rural control communities. Therefore, although it is difficult to generalize, the evaluation team foresees that at the end of the project there will be a measurable difference between the intervention and control communities as regards the level of participatory governance, particularly if small, rural communities are compared. Overall, therefore, the project intervention has been quite relevant. The project design, in envisaging partnerships with 43 communities across the country, has proved to be realistic. The innovative combination of civil society and local government within one project has encouraged CSOs to be more rooted in their constituencies and has enabled more grass-roots development and CSO-local government dialogue as a result, as evidenced by the 157 grants issued in FY 2012 which focus on local and regional issues. However, as noted below, the practice of awarding short-term grants has lessened the potential impact of institutional development activities and advocacy where long-term CSO￾government dialogue could be beneficial. 23 Stakeholder views on Counterpart’s grants strategy “We are not able to meet Counterpart’s demands when we present large grant applications. But then Counterpart asks us to do small grants – too small to have an effect on the LSG sector.” ‐ Local NGO staff member “All the grants have been small, fragmented, but needed a lot of work.” ‐ Local NGO staff member “Small grants are good for regional NGOs.” ‐ Donor stakeholder “OSI issues a range of grants – many small grants, but when we focus on particular CSOs we do large, institutional support grants. But we also recognize that you have to support grassroots organizations.” ‐ Donor stakeholder administrator EVALUATION QUESTION 2  To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms? FINDINGS Large number of small grants In the first two years the project awarded 388 grants, 227 of which were awarded in fiscal year 2012 alone. According to a draft work plan for FY 2013 reviewed by the evaluation team, it is envisaged to award at least 232 grants in the current year (see the analysis set out in Table 2 below). At the same time, the project staff stated that, as regards the community development component, they are aiming to increase the average size and timeline and to combine two ideas in one grant, based on long term community plans. The project staff has stated that they are able to monitor the large number of grants, although they accept that the burden is heavy, since as many as two grant reports per day are submitted to the office. In addition, although the project has in place a cyclical grant review system, it is apparent from some of the reports prepared for 2011 that the project struggles to support grantees and is not always able to analyse the impact of grants. Since the project develops annual work plans and does not roll over grants from one fiscal year to the next, the result is that, to date, grants have tended to be for periods of six months or less, and very rarely nine months or more. Amounts awarded have tended to be less than $10,000, and often considerably less than that. One grantee working in the LSG sector claimed that this policy of awarding grants that are both small in amount and short in duration is quite unsuitable for the LSG sector, as it is impossible to resolve institutional issues in this manner. At the same time, the grantee recognized that awarding a large number of small grants may be appropriate for the development of smaller, regional CSOs. 24 Table 2: Draft Work Plan for Number of Grants Planned FY 2013 SECTOR NUMBER Participatory community planning 1+ Y/CACs and CWGs 150+ Mobilizing external resources for communities Not specified2 Institutionalize public policy input mechanisms Not specified Policy watchdog 9 Local advocacy initiatives 12 Human rights 8 Institutional development grants 10 Enabling legal environment for CSOs Not specified Fix my street 1 Volunteerism Not specified Grassroots advocacy and community development 41 Presidential election grants Not specified Monitoring and action plan for decentralization Not specified Legal environment for LSG Not specified Legal environment for LSG finance Not specified PPP Not specified ICAs Not specified Total 232+ Table 2 above shows that the project is planning to issue grants in 19 different categories, and a review of the grants awarded for 2011 and 2012 similarly shows a huge range of initiatives started, but often it is not clear how any degree of sustainability is to be ensured. Often, a grant will enable a certain activity to begin, but then there is no follow-up or support for the development of institutional legacy. For example, the grant may enable policy documents to be drafted or proposals to be developed, but then the grant finishes and it is not clear whether any dialogue will be continued to ensure that the relevant 2 In several sections of the Draft Work Plan for FY 2013, the number of grants to be issued under a particular sector has not been specified. In some cases, a sector may be included in the grants to be awarded under another sector. For example, any grants to be awarded under the sector “institutionalize public policy input mechanisms” will be part of the nine grants to be awarded under the “Policy watchdog” sector. However, in other cases where the number of grants has not been specified, if any grants are actually awarded then this will mean that the current total of 232 will increase. 25 ministry or government agency responds to the NGO’s initiative. Similarly, there are occasions where related projects might benefit from working together. For example, in 2011, two grants in the education sector covered similar issues:  A grant to “For Education and Science” NGO aimed to prepare policy changes that would result in greater transparency, efficiency and accountability in the state education budget  A grant to “Partnership and Training” NGO aimed to improve the efficiency of budgeting and expenditure plans in the school system NICRA rate formulation helped lead to large numbers of small grants At the start of the project the NICRA was set at 23% and 4% on grants. However the 4% rate applied only to grants under $10,000, while those over $10k got the full NICRA rate. Therefore there was a request from the USAID regional contracting officer in Tbilisi that a large proportion of small grants be issued. CONCLUSIONS Grants policy makes sustainability difficult to achieve While the grants component has funded a wide range of initiatives, the short length of these grants (generally around 6 months, some as short as 3 months) makes it unlikely that lasting impact can be attained. It is more likely that activities can lead to awareness raising and/or development of policy suggestions, but do not allow sufficient time for the relevant authorities to respond on policy issues or for a culture of CSO-government dialogue to be developed. In addition, some of the initiatives are purely regional with no provision for national roll-out, and there appears to be no attempt to group grantees together by sector (e.g. health, education) for the purpose of peer learning and joint initiatives. By contrast, the LAAD initiative, which is not grant based, may have a better chance of becoming sustainable. 26 Views on the “Strategic Approach” “This is not a strategy, which would be ‘what, when, who, how’. Here we only have ‘what’, so it’s more a ‘strategic approach.’” ‐ Local NGO staff member “The good aspect was that 5 NGOs were able to work together and form a common approach. The result was a consolidated document with each NGO contributing based on their expertise. The bad aspect is that USAID is not really focusing on the policy issues at Government of Armenia level.” ‐ Local NGO staff member “Right from the beginning we agreed that this was a “strategic approach”. It was the first time that the 5 NGOs had worked together and through a lot of discussions we created the document.” ‐ Local NGO staff member EVALUATION QUESTION 3  To what extent has the program been successful in promoting decentralization? FINDINGS LSG strategic approach imperfect but useful exercise On 4th August 2012, Counterpart submitted to MTA a document that has been described as a strategic approach to decentralization. The document was the product of intensive work by five NGOs which have between them many years of experience in the LSG sector. When interviewed in a focus group, the NGOs noted that one of the big advantages of the exercise was that it was the first time that they had worked together. With each focusing on core areas of expertise, a consolidated document was produced. However, some of the NGOs reported there could have been more detail regarding accountability, transparency, ICT and the provision of information, whilst others said that the section on decentralization of functions was not thorough enough. The MTA has been involved in the development of the strategy through a steering committee, but it has not been possible to get definitive commitments from the MTA as to how the strategic approach will be utilized. The MTA deputy minister is clear that the document is not a strategy as such – he prefers to call it a strategic approach – and says he will use aspects of it as and when appropriate. The NGOs that drafted it equally recognize it is not a strategy, as it only deals with ‘what’, but not ‘when’, ‘who’ and ‘how’. One NGO at least was very optimistic about the future, believing that the Prime Minister had undertaken to embark on reforms. The MTA deputy minister, in particular, noted that progress should be expected after the Presidential elections. Since the date of the field-based research, it is understood that the government has now accepted the strategy paper as a basis for decentralization discussion and is actively discussing consolidation options within the cabinet as well as with donors and partners including Counterpart. However, it the evaluators have not been able to verify this information independently. ICAs have received limited support from the project The legal framework for ICAs is very basic, with only one article in the Law on Local Self-Government. Not all of the existing ICAs have the same legal status, and there is uncertainty regarding their powers and mandate. Project staff note that ICAs have not taken the initiative as regards national policy issues, and the project itself likewise has not drafted any proposed legislative reform to enable ICAs to operate 27 Sectors that ICAs seek to coordinate “We do land and property tax, legal advice and internal audit advice. We would like to propose a project on rubbish collection.” ‐ Aparan ICA “We want to have experts to advise communities in the following areas: legal, spatial planning and procurement. Also we want to do rubbish collection and recycling.” ‐ Akhuryan ICA “We do legal advice and veterinary and agricultural support. We would like to do land and property tax.” ‐ Tumanyan ICA “We believe it is possible to coordinate the services of the civil acts registry office through an ICA, without legislative changes.” ‐ Sisian ICA with greater clarity. They are waiting for the green light from the Ministry of Territorial Administration (MTA). In interviews, project staff acknowledged that they could do more to assist ICAs, although there were varying views on whether they intended to be more active in this sector in the future. Institutional development grants may be available for ICAs in the future, though one staff member indicated that these would only be available for ICAs which were already active, while another staff member said that they would support currently dormant ICAs. Staff members argued that thus far minimal funding has been provided to ICAs because the MTA’s policy towards them was unclear: that ICAs might simply be an intermediary stage on the way to full consolidation of communities. However, the MTA deputy minister when interviewed clearly indicated that ICAs were regarded as part of the long-term solution. Capacity building has been undertaken The project has undertaken capacity building efforts for local government, including training, as well as support in developing municipal ordinances and financial planning processes. CONCLUSIONS Dialogue with the authorities has had limited success The project has identified many important areas and has had successes in capacity-building. However, many initiatives are dependent on policy reform (“strategic approach”, PPP, ICAs). The project has maintained dialogue with central government, but in the current political climate has understandably not had much success. Several stakeholders have commented that they expect to see reform after the Presidential elections; however, that is by no means guaranteed. The LSG “strategic approach” is too long to be adopted as a strategy. If the MTA has any document that could be called a decentralization strategy, then it is the draft action plan dated October 2010, developed with the Council of Europe. That, however, is a very general document with no clear outcomes. 28 Appreciation of CWG and Y/CAC representatives “Counterpart responds quickly to problems.” “The coordinator helped with the paperwork.” “We’re very happy with Counterpart’s help.” “The Counterpart training was useful.” “We’re very happy with the Counterpart coordinator.” EVALUATION QUESTION 4  How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks? FINDINGS Staff structure and coordinators are effective The project employs a large staff of almost 40, mainly local employees. In addition, other key staff, the regional coordinators, are employed through the three ISOs which have the mandate to oversee the community development work. Most of the coordinators were retained from the previous community development project, and thus, have considerable institutional memory and knowledge of the communities they work with. Rather unusually for Armenia, the coordinators operate a mobile office – this ensures that they are constantly in contact with the communities rather than operating from a fixed base in the marz. The COP related that this was at first a difficult model to establish; however, it seems to have worked well in terms of productivity. It has enabled the project to meet the challenge of developing partnerships with no less than 43 communities – a challenge which, as highlighted in the findings under evaluation question 1, had raised fears that project resources would be too widely spread. However, there is no evidence from beneficiaries that Counterpart has been inattentive in its relations with them. Rather the opposite: mayors, CWG and Y/CAC representatives have praised Counterpart’s assistance. Internal management meetings are held every week to coordinate the components’ work, and the regional coordinators meet every month in Yerevan. The project has in the past requested approval for the creation of the post of Deputy COP if the funds are available, and it is understood that this is still sought. The COP argues that there is a need for someone to handle the administrative burden, allowing the COP to focus on strategic issues and representing the project. Public-Private Partnerships (PPPs): work in progress PPPs are forms of cooperation between communities and private entities. The project has focused on encouraging corporate social responsibility (CSR), on working with other donors, and on enabling local government bodies to borrow in order to finance long-term development plans. 29 Local authority borrowing: a political conundrum “The legislation on borrowing needs to be clarified. It states that the community ‘needs to get approval from the authorized body,’ but does this apply just to loans from state bodies or also to loans from commercial banks? The problem is that the Ministry of Finance is against this. Component 2 has tried to open a dialogue. Probably 20 communities could access private finance – if allowed.” ‐ Implementing Partner staff member “Legally it [local authority borrowing] is possible but politically it is not.” ‐ Implementing Partner leadership The project’s efforts to encourage CSR, and through it to develop partnerships between businesses and communities, has included activities such as a survey of more than 100 businesses to assess their attitudes toward CSR, a database of companies engaged in CSR as well as seminars and following up with companies that have expressed interest. The legal right of a community to borrow is unclear. Some argue that legislation appears to prohibit borrowing from the State, but that leaves open the question of borrowing from private entities such as commercial banks. The Deputy Minister of Territorial Administration stated that technically it is permitted to borrow, although the legislation makes this difficult. However, the concern is that communities would have to offer collateral, thereby putting community assets at risk. In effect, it appears that communities are being discouraged by the caution expressed by central government. The project management stresses that one would not expect very small communities to risk their assets: this form of financing of community development would currently only be realistic in about 20 of the larger urban communities. While waiting for positive signals from central government, the project has been undertaking planning exercises to increase the capacity of selected communities, and looking at the possibility of borrowing from entities such as the Armenian Social Investment Fund. PMEP: some changes required; output targets generally being met The original PMEP has been supplemented by additional indicators earlier this year, some of which are still in the process of formulation. However the original indicators are still valid. Some of the targets are expressed in terms of progress on the baseline survey. The projected percentage increases were drafted before the results of the baseline survey became known. As a result, some of the targets appear to be unrealistic. Currently, the project is, on the whole, meeting or exceeding those indicators where targets have been set for the first two years of the project. Exceptions include outputs concerning advocacy campaigns (output indicators 2.1.1 and 2.1.2) where the targets for FY 2012 appear to be very high and have not been met. However, there are a number of indicators which arguably set targets which are either too high or too low:  Cases where the targets are too high include sectors where the project would have to finance a large number of grants. In the context of developing a more strategic approach to grant-making, these targets could be revised downwards.  Cases where the targets are too low are in two important sectors (CSO-legislature dialogue and ICAs) where the very modest targets could be raised. In Table 3, below, selected indicators from the PMEP have been commented on. It should be noted that in the PMEP, where the actual performance exceeds the target, the variance is, (confusingly), expressed with a negative number. 30 As regards project impact, the baseline indicators for this have not been subsequently measured, though it was understood at the time of the evaluation fieldwork that Counterpart was preparing to commission a mid-term survey. Consequently, as it was beyond the scope of the evaluation to replicate the baseline survey, it is not possible to comment with any certainty on the impact of the project. That said, some preliminary estimates are set out in the conclusions below. 31 Table 3: Analysis of Selected Indicators from the PMEP INDICATOR FY 2011 FY 2012 FY 2013 FY 2014 Target LOP Comments IMPACT INDICATOR II 1.2 Number of citizens among target population who view themselves as participants in the local governance Target 20% 20% The baseline has provided a surprisingly high starting point of 54.7%, making the projected increase of 20% possibly unrealistic Actual 54.7% Variance IMPACT INDICATOR II 1.3 Number of people among target population who believe the community strategic planning will positively impact their lives Target 92.1% 15% 15% As with the previous indicator, the baseline sets a very high starting point, and in this case it would be impossible to achieve a target of 107.1% Actual Variance OUTPUT INDICATOR OI 2.1.1 Number of CSO advocacy campaigns supported by USG Target 40 100 80 60 280 These targets are too high, there should be more Actual 56 66 emphasis on quality than quantity Variance -16 34 OUTPUT INDICATOR OI 2.1.2 Number of USG assisted Civil Society Organizations that engage in advocacy and watchdog functions Target 15 40 40 40 135 These targets are too high. Again, there should be Actual 17 30 more emphasis on quality. Variance -2 10 OUTPUT INDICATOR OI 2.1.3 Number of USG assisted Civil Society Organizations that participate in legislative proceedings and/or engage in advocacy with national legislature and its committees Target 4 4 4 4 16 These targets are too low, the project could be Actual 5 5 more ambitious Variance -1 -1 OUTPUT INDICATOR OI 2.1.4 Number of Civil Society Organizations using USG assistance to improve Internal Organizational Capacity Target 20 40 60 80 200 The targets are too ambitious and the numbers should be decreasing towards the end of the project, as the focus becomes more on selecting key CSOs Actual 57 145 Variance -37 -105 OUTPUT INDICATOR OI 2.1.5 Number of Civil Society Organizations using USG assistance to promote political Target 50 The target in FY 2012 reflects the fact that there was a parliamentary election. Since there is to be a Presidential election in FY 2013, should there not be a target there as well? Actual 7 53 Variance -7 -3 32 participation OUTPUT INDICATOR OI 3.3 Number of Inter-Community Unions providing shared services to member communities Target 8 1 1 1 3 The targets are too modest, as shown by the figure Actual 8 6 for actual assistance in FY 2012 Variance 0 -5 OUTPUT INDICATOR OI 3.6 Progress toward inclusion of a long-term National Strategic Plan for Decentralization in 2012 Government program Target 3 5 5 The target of 5 (i.e. formal adoption) is not feasible as the “strategic approach” cannot be adopted by Government as such. Actual 2 4 Variance 33 CONCLUSIONS Impressive volume of work achieved by the team The evaluators were left with a strong impression of an immense volume of work achieved by the Counterpart team (not least the large number of grants issued and monitored, as well as a full range of activities in each component). Despite the large numbers of communities, CWG and Y/CAC groups to be assisted, there has been considerable appreciation of Counterpart’s efforts. The project still aims to hire a deputy COP once the budget realignment is finalized. This may help the COP to focus more on the policy issues and perhaps achieve a greater degree of synergy between the various activities. Lack of policy reform is affecting PPP achievements In its efforts to support public-private partnerships, the project has explored many avenues and engaged in capacity building and innovative ways of bringing stakeholders together but, as mentioned above, policy constraints have led to little progress. Although it is accepted that the potential for further progress is limited, it remains a fact that the project is not managing to develop this sector in the way that was originally planned. Although ad hoc CSR initiatives may gradually gain momentum, the de facto inability of communities to borrow is a serious brake on this component. The project continues to look for a way round this impasse, but current circumstances suggest success will be very limited. PMEP impact targets could not be rigorously assessed: only preliminary estimates can be given It has not been possible within the scope of the evaluation to replicate the scale of the survey carried out in 2011. Therefore assessment of differences between the comparison and intervention communities is largely based on a focus group with intervention community representatives and visits to three comparison communities. As indicated in the conclusions section under evaluation question 1, a qualitative assessment of differences of approach observed in the intervention and control communities leads the evaluators to conclude that end-of-project survey data is likely to reveal a difference between intervention and control communities, as the former will have benefitted from Counterpart’s efforts to inculcate a more participatory form of governance. Some of the results of that survey set surprisingly high benchmarks so that the proposed progress over the project period may be difficult to achieve. In addition the PMEP requires extensive review, as some of the indicators are too modest whilst others are too ambitious. Further, the new 2012 indicators, which will complement the existing PMEP, have not yet been fully developed. That said, the project is expected to achieve and in some cases over-achieve realistically set output indicators. The project impact, while it cannot be measured at this stage, is likely to show some positive trends on the basis that successful completion of relevant outputs logically will lead to impact among the beneficiaries. However, at this stage, only a qualitative assessment can be given:  It seems likely that the project will achieve targets regarding implementation of community strategic plans (impact indicator 1.1), but the planned increases in participation (impact indicator 1.2) and perception of impact of community strategic planning (impact indicator 1.3) are unlikely to be achieved because the baseline figures were surprisingly high.  Given the project’s intensive work with civic action groups, it is quite possible that the target of increased civic participation will be achieved (impact indicator 2.1.1). However the targets of increasing the advocacy score of NGOs (impact indicator 2.1.2) and CSO policy impact (impact indicator 2.1.3) may be prejudiced by the short-term approach to grants, which lessens the 34 ability of CSOs to develop long-term relationships with state counterparts. In addition, the willingness of government to listen to CSO views is an independent variable.  The project activities regarding volunteerism may have a positive impact (impact indicators 2.2.1 and 2.2.2), though it is hard to predict the extent. The likelihood of achieving an improved regulatory framework on volunteering (impact indicator 2.2.3) has to be assessed as low, based on the lack of progress to date.  Greater transparency and accountability of local government (impact indicator 3.1) is very likely to be achieved on the basis of a comparison between intervention and control communities, however the exact degree of improvement is hard to predict. 35 EVALUATION QUESTION 5  Do local governments/community councils and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? FINDINGS Beneficiaries expressed satisfaction Overall, interviewees have expressed general satisfaction with the project. Time and again, in focus groups and individual interviews, stakeholders have expressed their thanks for the project’s support and commitment. Dissatisfaction has been muted, confined to some community groups which felt constrained by restrictions in the type of projects that could be supported, and by a national NGO in the LSG sector which argued that the short-term, small grants made continued development of the sector impossible. International donors have also expressed their view that fostering community active groups and ensuring participatory processes can only strengthen local democracy. In describing the processes of project selection and implementation, interviewees placed emphasis on the role of Counterpart in ensuring a participatory process. For example, when Sevan town council began to implement a project on street lighting, the Counterpart representative played an important role in the organization of a series of meetings in the different districts of the town, to ensure the population’s involvement in and support of the planned project. This can be contrasted with the attitudes expressed by interviewees in the three comparison communities visited. There, administration-citizen communication tends to be one-sided: the administration may periodically contact citizens either to ask their views on social issues or simply to inform them of planned or implemented projects, but rarely do the citizens take the initiative or play an equal part in project development and implementation. CONCLUSIONS Stakeholders supportive of participatory approach Stakeholders are largely supportive of Counterpart’s engagement. Feedback from the focus group discussions held with local government, community working group and Y/CAC representatives suggests that the joint work in the 43 intervention communities does serve their mutual interests. There is a difference of approach, in that citizens’ groups focus more on the process and on the outcomes, while council representatives focus more on the physical outputs. There are signs of a gradual change in mentality towards a better partnership with the community. The work of Counterpart in fostering a more democratic and participatory approach to project formulation and implementation contrasts with a more traditional approach still practiced in the comparison communities. It remains to be seen how permanent these changes will be in the intervention communities. 36 ADDED QUESTION, FINDINGS AND CONCLUSION Recognizing the value in elaborating on the project’s “change theory,” the evaluation team has taken the further step of adding a question, providing findings and reaching a conclusion. The team believes that the Mission is owed this further inquiry.  Has the validity of the CSLGSP “change theory” been borne out in the first two years of implementation? FINDINGS The CSLGSP project contains a theory of change. The program description that was attached to the Mission award letter to Counterpart states that its change theory is: “...Civic participation and good governance at the local level will trickle-up and complement democratic reform work being done at the national level.” This statement raises the further questions of “How are local actions supposed to trickle up?” and “What is this trickle up process intended to achieve regarding local government reform?” Because the project evaluation team did not have an evaluation question that focused on the validity of the change theory, it did not ask direct questions on that subject during the fieldwork. When the omission was pointed out strongly by the Social Impact senior evaluation panel, however, the team reviewed its interview and focus group notes and other documents to determine what evidence had been gathered about “trickle up.” The team found no references to the trickle up process in its interviews with Mission staff. The team found no such references in its key informant interviews with Counterpart staff. In reviewing notes from the five focus groups of Armenian beneficiaries there was considerable attention to the successes at the local level but no discussion of how these gains were being translated to the national level. Our discussion at the Ministry of Territorial Administration did not elicit any information that would even by inference indicate that the community development efforts in 43 communities were affecting the thought of national officials. For example, if there had been an attempt to integrate the work on developing community-level strategic plans into the relevant marz-level development plans, this would have been evidence of linking the micro- to the meso-level. However, the evaluators are not aware of any such attempt. A complementary concern for the evaluation team was the lack of any described “instrument” or activity in the program description for capturing the local gains and articulating them in ways to gain the attention of national officials. Nor does there appear to be a documented methodology for how the “trickled up” results would “complement democratic reform work being done at the national level.” It strikes the evaluators that, had there been a geographic focus to the work at local level – e.g. by concentrating on a particular marz or marzes – then it might have been possible to develop greater synergy between communities and build up a “groundswell” of change which would have made the chosen geographical area stand out from the surrounding marzes and thus be a clear advertisement for policy changes at the national level. Aware that the issue had been omitted during the initial fieldwork, the evaluators sent written questions to the former Chief of Party and interviewed the current Chief of Party. It became clear that there have been some successes in influencing the national reform agenda, as follows: 37  The LAAD (Legislative Agenda Advocacy Days) process involves discussing issues at grass-roots level, filtering to identify issues of national importance, and then, via NGOs, presenting those issues to the relevant standing committees of the National Assembly.  Cooperation between the Ministry of Territorial Administration and the Municipal Lawyers’ Association on municipal ordinance reforms  Other activities which attempted to influence the national agenda included the NGO working groups on decentralization strategy and on NGO legislation  Counterpart staff also noted the increasing readiness of local mayors to demand greater central government funding on specific issues such as drinking water repairs CONCLUSION Because it did not make the project “change theory” a focus of its field work, the team is not prepared to question its validity, but only to raise a concern about the project achieving substantive reforms in the months to come. Progress to date has been patchy. LAAD appears to be the sole process which truly consists of a multi-step trickle-up approach and is not narrowly focused on a specific issue. Other advocacy efforts such as the work on decentralization strategy and NGO legislation have taken a more direct, one-step approach. It is difficult to find examples of good governance at the local level which have led to a change at the national level. The (perceived) increased willingness of mayors to demand central government funds is to be lauded, but is nevertheless a very ad hoc approach rather than a systemic one leading to any change in national policy. Bearing in mind the above analysis, the Mission may want to review the theory to determine how it has been implemented up to the present and what other steps should be taken to insure that it is realized during remaining months through CSLGSP. On that element could hinge the assessment of success in any final evaluation. Since the team’s has given attention to this issue only “ex post facto,” we do not raise the foregoing to the level of a recommendation, but include it here as a strong suggestion. 38 CROSSCUTTING GENDER ISSUES The project is performing well in terms of gender issues according to figures for its work with beneficiaries in the first two years:  Out of 15,204 citizens trained, 56% were women  Out of 44,836 citizens mobilized at events, 56% were women These figures are plausible, in that the bulk of the project’s work is centered around community-based initiatives, where women are likely to be equal beneficiaries. Further, it is noted that eight of the 11 Y/CAC and CWG representatives attending the focus group were women, and they all played an active part in the discussion. However, a review of the list of grants awarded during FY 2012 shows that only eight out of 227 grants were to women’s rights NGOs and/or on themes exclusively concerning women’s rights. Also, local government is male-dominated with comparatively few female mayors, and the project’s 43 intervention communities reflect this bias. In the long term, the development of Y/CACs and CWGs could help the emergence of a new generation of female candidates for mayoral and other government posts. The project’s PMEP disaggregates relevant data by sex in order to capture information regarding the gender of those receiving training in local government issues. In addition, it is notable that a majority of the Counterpart project staff are female. In assessing the gender inputs and outputs of the CSLGSP, the Mission and contractor have satisfactorily disaggregated statistics and provided numbers that are meaningful for this mid-term evaluation. Our assumption is that this project was designed before the promulgation of USAID’s “Guiding Principles” on gender reflecting key features of USAID’s Policy Framework 2011-2015 and the parameters of the USAID Forward reform agenda. For the second half of the project, it will be useful for staff to review those principles to determine which might usefully be applied to CSLGSP. For example, one anecdotal indicator that impressed the evaluation team was a circumstance gained from our focus groups. As noted, a substantial number in each group were females who were participant/beneficiaries of the project. The evaluation team noted that they were not shy or reticent about expressing their opinions in front of the men, and in some cases, expressing disagreement with statements made by one or more of the male participants. It raised questions of whether female participation in CSLGSP had seemingly empowered them to speak out forcefully. One of the USAID Gender Guiding Principles calls for the following: “The Agency will measure performance in closing key gender gaps and empowering women and girls, learn from successes and failures and disseminate best practices on gender integration throughout the Agency.” This statement suggests that an effort might be made to craft a performance indicator or indicators that would attempt to capture some of the more difficult-to-quantify effects of CSLGSP, such as increasing the sense of empowerment women have as a result of their participation with CSLGSP inputs. For example, an indicator such as the number of women who are actively participating in the work of each local CD committee could be considered. This might done by review of minutes and attendance kept at the meetings, ensuring females are identified. Additional measures for guaranteeing this project meets the more recent USAID desire to go beyond numbers to assess elements of closing gender gaps may be explored via the inclusion of a gender-specific evaluation question in the CSLGSP end-of-project evaluation. 39 GENERAL CONCLUSIONS Overall, an impressive project Overall, there has been an impressive amount of effort expended in implementing this project. The large number and wide range of activities and the generally positive feedback from beneficiaries (in total the evaluators interviewed 36 beneficiaries and almost all of them were supportive of the project) is evidence of a committed leadership and staff, a successful organizational model and a good understanding of the issues. In particular, the support to community development and civic activism has been highly appreciated by beneficiaries and is evidence of the project’s ability to nurture the process carefully despite the comparatively large number of partner communities. Large number of small grants impairs effectiveness in terms of institutional sustainability The large number of small grants, as acknowledged by the project leadership, has been driven at least partly by instructions from the USAID regional contracting office, in order to avoid the higher NICRA that would apply to larger grants. However, now that the budget is in the process of being negotiated and the restriction of application of the NICRA no longer applies, it is advisable to adopt a more strategic approach. The size and length of grant should be based, as far as possible, on the objective to be achieved. Few ICAs have been supported by the project Within the local self-government component, the project has in the first two years issued grants to six ICAs. There are however a total of 58 ICAs. Although many of them are not currently active, nevertheless the project has only reached a handful of potential beneficiaries in this sector. Given that it is government policy to strengthen ICAs side-by-side with implementation of consolidation of communities, accordingly ICAs are not simply seen as an intermediate stage towards consolidation, but as a long-term solution to the problems of administrative inefficiency and poor quality services. Some ICAs have shown that it is possible to provide effective services for their communities, such as tax administration and solid waste collection, and others could benefit from learning about their success. Y/CACs and other grantees do not have many opportunities to share experiences The project has supported a wide range of Y/CACs and CSO grantees. The Y/CACs, although they naturally face very similar challenges, (how to implement construction projects, how to develop collaborative modes with the community administration, how to work towards sustainability, etc), tend to work in isolation from one another. The CSOs also have similarities, in that one could group them into sectors, such as health, education, etc. But again, they tend to work in isolation, often engaged in similar sectors and with similar objectives (such as advocacy, transparency), but with little chance to learn from each other. Therefore, in both cases there is considerable scope for peer learning, for sharing experiences and finding synergies. At the same time, there does not appear to be a concerted effort by the project to provide opportunities for peer learning. In one region a group of Y/CACs on their own initiative organized joint events, but this appears to have been an exception to the rule. Sustainability is likely to be inconsistent Conclusions as regards the likely sustainability of the various components of the project are as follows:  Y/CACs, CWGs and community development initiatives These informal associations, which have been supported by the project (and in some instances have a longer history, as they were initially set up under previous iterations of USAID’s community development activities), are likely to experience a degree of waning unless support continues in one 40 Sustainability “There is a need for institutional development and we have persuaded USAID to allow us to do this with 10 NGOs.” “Y/CAC sustainability is in the people, not in maintaining the clubs as such. Ex-members go on to do advocacy in other spheres.” “Advocacy NGOs the world over are not sustainable – they’re dependent on grants.” ‐ Implementing Partner leadership “The guarantee for sustainability is in the capacity building.” ‐ Implementing Partner staff member “We should be taught how to catch fish, not given fish.” ‐ Local NGO staff member “Give us fish and teach us to catch them.” ‐ Local village mayor, referring to assistance in contacting other donors “It’s not a question of finance, it’s a question of strategy – there’s a need for long-term advisors to assist with [NGO] capacity-building.” ‐ CSO grantee form or another. Community mayors have indicated that they have good relations with the associations and are likely to continue collaboration with them in future years. However, by its very nature enthusiasm can be hard to sustain, and as the youth members mature and move on to other spheres (including physically, as education and jobs may take them out of the community), it seems that the greatest sustainability – as pointed out by the COP – is in the human capital, not in the associations as such. That said, the project does not have an alumni database, so it cannot measure progress in human capital.  LSG Local self-government will of course continue after the project, but what are at stake here are the efforts to increase transparency and effectiveness. As noted above, the collaboration with local activist groups is likely to continue to some extent after the project finishes, but in the nature of things, no doubt there will be some relapse. Efforts to put communities on a more sustainable footing through fiscal decentralization and clarification of a political and legislative mandate for borrowing have failed so far due to the reluctance of central government to move forward on these issues. Therefore, the project has invested resources in capacity building to enable selected communities to borrow, should the opportunity arise.  CSO grantees As noted, the project has awarded an extremely large number of grants during the first two years, to a wide range of CSOs, some with a national coverage but many based in individual marzes or communities. The small grants offered can be likened to seed money which can encourage a brief “flowering” of the CSO round a particular topic. But longer-term interventions to encourage institution-building have been few and far between. Admittedly, it can be very hard to demand sustainability in areas such as advocacy, where, in the words of the COP, “advocacy NGOs the world over are not sustainable – they’re dependent on grants.” But overall, the lack of a longer-term vision for developing the CSO grantees through longer-term support and networking means that many will be left to sink or swim. Indicators are not always realistic or useful Some of the indicators are not realistic: they are either too high or too low. In some cases this is because the initial baseline survey gave surprisingly positive results in terms of citizen participation, and therefore to show substantial improvement on the already high baseline will be a struggle. In other 41 cases, the existing targets are very modest: arguably the project could set the bar higher for itself – e.g. the number of political initiatives in 2013, a Presidential election year. 42 RECOMMENDATIONS Adopt a more strategic approach to grants It is recommended that a more strategic approach be adopted as regards the objectives, size and time period of grants issued by the project. It is no longer necessary to ensure that a proportion of grants be under a pre-determined amount, and the fact that the project prepares annual work plans should not mean that a grant time period cannot span over two work plans. Therefore, the project should be realistic about the size and length of grant required to achieve an objective. Some processes outlined in grant proposals can take up to a year or even more, particularly if they are dependent on responses from and collaboration with third parties. Sustainability is more likely to be achieved where a grant project involves sustained activity and the development of relationships over a long period, as opposed to a brief burst of activity. Moreover, consideration should be given to narrowing the range of sectors covered by the grantees, in order to have a greater chance of achieving synergy between them. Increasing local ownership through cost-sharing is one means of enhancing synergy, while switching the focus to non-infrastructure (and therefore, non-temporary) projects is another. Increase the opportunities for peer sharing of experience The opportunities for peer sharing should be explored. Networking and sharing experience can lead to greater sustainability in the long run. The project should enable regular contact within its groups of beneficiaries. For example, Y/CAC representatives could meet periodically on a regional basis (e.g. each of the three ISOs could organize this within its respective geographical region). Equally, grantees in the same sector should be able to share experiences regarding their activities and objectives. For example, various grantees operating in the health (or education, disabilities, etc.) sector should be able to meet to discuss their advocacy/anti-corruption efforts and to learn from each other. Specifically, much could be gained by having NGOs work in tandem to prepare their policy proposals to the Ministry of Education. Increase support to ICAs Support to ICAs should be increased, to reflect the potential that they offer (enhanced efficiency and provision of services without the political issues that consolidation involves). Working with ICAs as a way of overcoming the inefficiencies of local government could show stakeholders that services can be improved with existing resources, and can help to facilitate the process of consolidation in the long run, as communities learn to work together. ICAs that are currently dormant should be enabled to attend networking events to learn from the more active ICAs, and replication of successful models (e.g. tax administration, solid waste collection) should be encouraged. Revise the PMEP The PMEP should be revised, not only by finalizing the 2012 indicators but also by reviewing the old indicators to make them more realistic and useful. Targets which are unrealistically high (apparently because of some surprisingly positive results from the baseline survey) should be revised downwards, whilst others which are too modest could be revised upwards. Ramifications of USAID “Forward” policy The assumption of the CSLGP evaluation team is that the Mission will want to continue supporting financially a program of grants to CSOs working in democracy and governance in Armenia. In any future iteration, however, the current USAID “Forward Policy” will be a consideration in how such a project is designed. The Forward Policy, which is in harmony with the Paris Declaration to which the U.S. 43 adheres, calls for more direct contributions to host country organizations. For a civil society project this may mean that the role currently played by the U.S. contractor -- in this case Counterpart -- may be eliminated or altered significantly as the implementing partner becomes one or more Armenian organizations. The Counterpart COP has emphasized to us the top quality professional team that has been built around the CSLGP project. Our discussions with Counterpart Armenia senior staff members confirm that view. Thus it occurs to us that steps could be taken by Counterpart in cooperation with the Mission to transform the local CSLGP staff into an NGO capable of being a direct conduit for future USAID grants-giving and other programs in the D&G field. There is precedent for such a move. In El Salvador in 2001, a development consulting firm called Development Associates at the end of a $24 million 5-year project assisted its highly professional local staff make the “jump” to becoming a non-profit organization. Called FUSADES, it was an entity to which USAID and other international donors could direct contracts with some confidence. That organization not only survived but prospered. Obviously the process for creating such a transformation varies greatly from country to country but may have potential worth exploring for Armenia. This does not eliminate the need for Counterpart participation. The support and assistance of that organization will be essential to helping to bring the new NGO into life and assist it into self-reliance. A key issue will be fostering an entrepreneurial spirit in the new organization whose staff members may not have vied in the marketplace in the past. In the El Salvador, for instance, Development Associates several times was a subcontractor to FUSADES, giving potential donor clients additional confidence about the quality of management. Counterpart might well play a similar role. If such a step is to be taken in a timely fashion, discussions among the parties should begin soon as the process from project staff to NGO is likely to be a lengthy one. 44 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK ANNEX II: EVALUATION METHODS AND LIMITATIONS ANNEX III: DATA COLLECTION INSTRUMENTS ANNEX IV: SOURCES OF INFORMATION ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST 45 ANNEX I: EVALUATION STATEMENT OF WORK Statement of Work Evaluation of USAID/Armenia Health Systems Strengthening Project, Civil Society and Local Government Support Program, Alternative Resources in Media Project, Assistance to the Energy Sector to Support Energy Security and Regional Integration Program, Pension and Labor Market Reform Project Summary: USAID/Armenia requires performance evaluations of the following activities: Health Systems Strengthening Project (HS-STAR), Civil Society and Local Government Support Program (CSLGSP), Alternative Resources in Media Project (ARM), Assistance to the Energy Sector to Support Energy Security and Regional Integration Program (ESRI), and Pension and Labor Market Reform Project (PALM). The purposes of this Task Order are to evaluate the success of these projects in their relevant technical areas and to assess the overall effectiveness of the projects in achieving set programmatic goals and USAID/Armenia’s strategic objectives. Three of the five planned evaluations are designed as mid￾term performance evaluations (HS-STAR, CSLGSP, and ARM), while the remaining two are designed as end-of-project performance evaluations (ESRI and PALM). The purposes of the mid-term evaluations are to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Findings from the mid-term evaluations must inform future work plans of the relevant projects, as well as designs of future similar activities. The purpose of the end-of-project evaluations is to assess the effectiveness of resources spent and to inform design and development of future strategies and projects. The evaluation findings must be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. The respective project AORs/CORs will develop plans for incorporation of relevant recommendations from the evaluations in their future work plans. Contractor Responsibilities and Projects: The evaluations should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, guided by the evaluation questions formulated for each individual activity. Each evaluation question must be answered empirically, relying on factual evidence, and must be addressed distinctly in the final reports. CSLGSP Sep 2010-Sep 2014, $15.5m This is a four-year activity in the second year of its implementation. The project aims to increase the level of informed and organized civic activism at the local and national levels, along with more participatory, decentralized, accountable governance that leads to a more democratic society. The project will reach this objective through three programmatic components: (1) local government and civil society collaboration, (2) fostering civic participation, advocacy and activism, and (3) facilitating decentralization and local fiscal autonomy. The project expands opportunities for the public to organize and advocate on behalf of their needs and concerns; it also increases local government accountability and capacity to be responsive to citizen interests. The program takes advantage of opportunities for reform at the local level by creating models of democratic governance through a highly integrated 46 community development approach to local self-governance, fueled by increased civic participation from Civil Society Organizations (CSOs) and businesses. Evaluation Questions: The evaluation questions for mid-term evaluations are designed to help the projects adjust and/or modify future implementation and work plans, as well as designs of future similar projects. The questions for end-of project evaluations will help assess the effectiveness of the resources spent and will help inform the design of future projects and strategies. The contractor must review and summarize the data collected from document review and from field work to answer the following evaluation questions: CSLGSP  How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia?  To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms?  To what extent has the program been successful in promoting decentralization?  How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)?  Do local governments/community councils, and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? USAID’ S Role in the Evaluation The USAID Mission in Armenia will: • provide relevant programmatic and budgetary information to the Contractor; • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor must: 47 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Identify data collection methodology to provide the best possible evidence to answer the evaluation questions, also considering feasibility issues. 3. Identify informants and stakeholders, samples and/or other relevant data sources. 4. Prepare a field work plan. 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports addressing comments by USAID and submit final reports to USAID/Armenia for acceptance. 8. Address implementing partner comments within one week as necessary after USAID/Armenia shares the final reports with implementing partners, and if partners choose to submit “Statements of Differences”. The proposed methodology should address the need for data collection from qualitative and quantitative sources, and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. All evaluation questions need to be answered empirically; therefore the data collection methods should be tailored to ensure that adequate evidence is collected to answer each of the questions in a definitive manner. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. Data should come from facts, rather than be based on anecdotal evidence. Conclusions should be based on findings received from multiple sources, and strengths and limitations of the methodology should be explicitly communicated. All people-level data should be disaggregated by sex to allow analysis of findings by sex. Baseline data for all projects is available from their monitoring data. A sample of indicators used for monitoring of each of the projects is provided in the Annex. Some of the baseline data sources include surveys, official statistics, automated information systems, and project records. In the case of CSLGSP baseline data was collected not only from participating 43 communities, but also from 15 control communities. Although not randomly assigned to treatment and control groups, communities nonetheless are comparable, and can therefore serve as comparisons for the achievements in citizen engagement in decision making and in community development projects. Deliverables The Contractor’s deliverables must include: 1. Written methodology plans (evaluation designs and work plans). The evaluation designs must include detailed evaluation design matrices (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final designs require COR approval. The work plans must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation teams. The methodologies will be shared with relevant stakeholders, including the implementing partners for their comments before finalizing. 2. Verbal debriefing The evaluation teams must meet with USAID/Armenia upon arrival. The teams must also provide oral briefings of findings and recommendations to the USAID/Armenia senior management and relevant technical teams prior to departure. 3. Draft Evaluation Reports 48 Prepare draft evaluation reports which must analyze data and summarize key findings, conclusions and recommendations. The Findings section must list all facts and evidence received from desk research and from field work in relation to each evaluation question. The Conclusions section must analyze the facts specified in the Findings section, and must discuss what worked, what did not work and why. The Recommendations section must address issues of what can be improved for future programming, based on Findings and the analysis provided in the Conclusions section. More specifically the Recommendations section must address the following: For HS-STAR – a) what can be changed for future implementation of the project, b) what recommendations can be made to ensure sustainability of project results; for CSLGSP - what can be improved for the remainder of the project given the resources available; for ARM a) what changes should be made in the approaches and directions of the current program, and b) what recommendations can be made for the design of future media programs; for ESRI - what recommendations can be made for future energy sector programming; for PALM - what recommendations can be made for design of future pension reform projects. The Evaluation Reports must at a minimum contain: 1) a 3-5 page Executive Summary summarizing key points (purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations); 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action-oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work; all evaluation tools; all sources of information properly identified and listed; any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team; disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Draft reports must be submitted to USAID/Armenia within three weeks after completing the fieldwork for each evaluation. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. Final Reports The Contractor must submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The reports shall follow USAID branding procedures. The reports must include an executive summary and not exceed 30 pages (excluding appendices). The final reports must meet the following quality standards (Please see the USAID Evaluation Policy): a) The reports must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports must address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings must assess outcomes and impact on males and females; f) Limitations to the evaluation shall be 49 disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. Evaluation Data The Contractor must submit Evaluation data to USAID/Armenia along with the final reports for record keeping and future use by the Mission. The data must be in an easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Schedule: The anticipated duration of all evaluations is not to exceed nine months. Staffing and Level of Effort: It is expected that the Contractor will provide five different teams of consultants to carry out the evaluation of the five activities described above. Consultants with diverse or cross-cutting skills may serve on more than one evaluation team. The mix of skills necessary for each evaluation team is specified below. Each evaluation team should consist of two experts, although other team compositions might also be considered. Inclusion of qualified local Armenian experts in the evaluation teams is highly encouraged. All evaluation team members must be required to provide a written disclosure of conflict of interest. Required Qualifications of the Personal For CSLGSP:  Substantial experience in public administration, international relations or related field with emphasis on civil society and local government  Experience working with civil society and/or local government projects, experience with civil society and local government project evaluations, or with program evaluations in the region is a plus Instructions on Preparation of Branding Implementation Plan and Marking Plan As part of the proposal submission, the Contractor will develop a Branding Implementation Plan (BIP) and a Marking Plan in accordance with the policies found at Automated Directive System (ADS) Chapter 320, revised on May 5, 2009, or any successor branding policy, and with the “USAID Graphics Standard Manual” that is available at www.usaid.gov/branding. Among other provisions, ADS 320 states that: 1. Contractors and subcontractors' corporate identities or logos must not be used on USAID￾funded program materials. 2. Marking is not required on contractor vehicles, offices, office supplies or other commodities used solely for administration of the USAID-funded program. 3. Marking is not permitted on any communications that are strictly administrative, rather than programmatic, in nature. USAID identity is also prohibited on contractor and recipient communications related to award administration, such as hiring/firing of staff or renting office space and/or equipment. 50 The Contractor shall also develop a Marking Plan for public communications, commodities, program materials, deliverables, and other items that visibly bear or will be marked with the USAID identity. The marking plan may include requests for exceptions to marking requirements, to be approved by the Contracting Officer. Contract deliverables to be marked with the USAID identity must follow design guidance for color, type, and layout in the Graphic Standards Manual (available at www.usaid.gov/branding) or any successor branding policy. With respect to this Task Order, the Contractor should develop a BIP and Marking Plan bearing in mind the following branding strategy: 1. Program Name: Evaluation of Five USAID/Armenia Projects. 2. Positioning: This task order is funded through the USAID/Armenia Mission. Materials and communications must be positioned as from the American People, using the USAID Identity. 3. Outreach to Beneficiaries and Host-Country Citizens: No special outreach efforts to beneficiaries and host-country citizens are planned under this Task Order. 4. Level of Visibility: The findings of the final evaluation report will be used by USAID in its implementation and further planning its activities. The report will be submitted to USAID’s Development Experience Clearinghouse for wider access. 5. Other Organizations to be Acknowledged: No other organizations are required to be acknowledged. 6. Specific branding issues: The only branding issue expected to arise as a result of implementing this Task Order is the proper use of graphics standards for all reports and other printed or electronically distributed information. Environmental Considerations The Evaluation Services IQC has an Initial Environmental Examination (IEE) approved on April 8, 2010. The IEE has determined that the activities described under the Evaluation Services IQCs qualify for a categorical exclusion per 22 CFR 216. Any activities found to be outside the scope of the approved Regulation 216 environmental documentation shall be halted until an amendment to the documentation is submitted and written approval is received from USAID. 51 ANNEX II: EVALUATION METHODS AND LIMITATIONS Methods The SOW for the evaluation provides five specific questions to be answered. Below is a description of the methodologies employed to answer these questions.  How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? To answer this question required an assessment of the current civil society and local governance situation and based on that an analysis of its developmental needs. This was then compared with the original project design and objectives and achievements over the period since project inception. Evidence was found from USAID and other stakeholder reports, backed up by interviews and focus group discussions with relevant stakeholders. Method: Key informant interviews; focus group discussions; document review Source: Relevant USAID program reports; relevant stakeholders  To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms? This question required the analysis to go beyond an assessment of the individual grants, and to assess whether sustainable mechanisms have been developed. Indications of success might be, e.g., the establishment of coalitions or regular civil society-government consultations. The evidence was obtained from interviews with project staff, selected grantees and independent experts, as well as reviewing relevant project documentation. Method: Key informant interviews; document review Source: Relevant USAID program reports; relevant stakeholders  To what extent has the program been successful in promoting decentralization? The project reports and interviews with local communities gave information on successful initiatives that have led to an increase in local government powers, whether in theory or in practice. Also, national policy initiatives were reviewed and local government and national government stakeholders were interviewed to assess if the national decentralization agenda has been influenced by the project. A focus group of local government actors was employed as a way of shedding light on the issue. In this respect, it was important to analyse the relationship of the project’s current work on decentralization strategy to the government agenda. Method: Key informant interviews; focus group discussion; document review Source: Relevant USAID program reports; relevant stakeholders  How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? This issue required a qualitative assessment of the management structure and staffing and a comparison with a) outputs achieved in the project’s three components, and b) the needs of the sectors as regards those component interventions. Interviews with project staff were held to reveal bottlenecks and other issues, while the document review established the outputs and needs as regards each of the components. Method: Key informant interviews; document review Source: Relevant USAID program reports; relevant stakeholders o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? 52 Evaluation of this issue involved an analysis of the project’s work with the 43 partner communities, interviewing project staff and a sample of the relevant communities, as well as reviewing project reports. Focus group discussions were held with a variety of local government stakeholders to enable the maximum number of views to be heard in the time available. Method: Key informant interviews; focus group discussions; document review Source: Relevant USAID program reports; relevant stakeholders o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)? The evaluation analyzed the project’s internal monitoring data to reach a conclusion on this, and sought clarification through interviews with staff members. The analysis included a review of the baseline data from comparison and intervention communities that was collected at the start of the project. It is understood that this data has not been updated since then, and it was not feasible for the evaluation team to attempt to undertake a further baseline survey given the tight timeframe and finite resources. Therefore, instead, a qualitative assessment was made of the current situation in the comparison and intervention communities, by engaging a sample of them and focusing on key issues posed in the baseline data. Method: Key informant interviews; key questions list; document review Source: Relevant USAID program reports; relevant stakeholders  Do local governments/community councils, and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? The evidence for this question was found in project reports and through interviews and focus groups. It is obviously a subjective issue, as the question addresses the perception of the beneficiaries and how they receive the assistance. Time constraints meant that, while a range of local government representatives could be interviewed, it was not possible to sample a meaningful number of citizens as such, so instead the focus was on sampling the views of citizens’ active groups. Method: Key informant interviews; focus groups; document review Source: Relevant USAID program reports; relevant stakeholders Limitations to Evaluation Design Limitations experienced during the fieldwork included the following:  In order to reach as many stakeholders as possible within the time-frame, five focus group discussions were held, with between five and ten participants in each group. On the positive side, this enabled a wider range of opinions to be held than would otherwise have been possible, but on the negative side, it meant that some of the issues were considered in less depth than was possible in individual interviews.  The assistance of the Counterpart team was sought in arranging the focus group discussions. They were asked to select a mix of representatives from each stakeholder group. For example, they were asked to select active and but also less active communities , so as to get a picture of both the successes and the relative failures of the project. A similar approach was adopted when selecting representatives from grantees. Whilst it is believed that Counterpart staff approached this task honestly, nevertheless there is a risk of a certain bias towards selecting the more active beneficiaries.  In one or two cases, not all of those invited to the focus groups actually participated. For example, no representative of the Communities Union of Armenia attended the focus group discussion on the local self-government strategic approach paper.  In order to sample comparison communities, it was decided to visit a sample of them rather than to invite their representatives to Yerevan, as the latter approach might give rise to unrealistic expectations that 53 they would become project beneficiaries. However, as a result, only three communities were visited, namely Ptghni, Aramus and Hrazdan in Kotayk marz. 54 ANNEX III: DATA COLLECTION INSTRUMENTS Set out below are the initial lists of questions, used as a guideline when interviewing the various stakeholder groups. Interview questions for beneficiaries: 1. What is the nature of your organization? What functions does your organization undertake? By what mandate? 2. What is your relationship with USAID? 3. What has been the nature of the assistance being provided? 4. What have been the results to date? 5. How satisfied are you with the experience with the USAID assistance? How would you rate your level of satisfaction on a scale of 1 – 10 (10 highest)? 6. Had you not been able to obtain this assistance, what other alternatives would have been available to you? 7. How effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? 8. Can you provide the evaluation team with information to help us understand the impact of the assistance on your activities? 9. How sustainable are the impacts of the assistance? Do you anticipate that your organization will continue with the same practices after the USAID assistance has finished? 10. Do you see the assistance as being relevant in light of the current civil society/local government environment in Armenia? 11. What, in your opinion, is the main constraint acting on USAID assistance in Armenia? 12. What, if any, kind of training have you received? What kind of training? Should this training be augmented? How? 13. In your view, how effective is the USAID assistance? Do you believe that it has achieved what it set out to do? 14. In your opinion, how could the assistance be improved? 15. What other types of assistance could be offered by USAID to the local government / civil society sector in Armenia? 16. Do you have a specific recommendation that the evaluation team could provide to USAID to help it improve future assistance? In light of your experience, what advice would you give USAID? Interview questions for the contractor and other stakeholders: 1. Can you please summarize your mandate for assistance to the Armenian civil society and local government sectors? 2. Are you involved in other assistance projects in Armenia other than those with USAID? What are they? 3. In your opinion, how appropriate is the USAID assistance? What are the strongest points of USAID assistance in these sectors? Are any improvements needed? 4. To what degree is cooperation achieved with other donor agencies operating in these sectors? 5. In your opinion, how successful is USAID assistance in these sectors? How well has it met the country’s needs? 6. How effective do you consider the work to be of implementing the assistance provided? What are your success stories and what are your failures? 7. How sustainable are the improvements brought about by the assistance? Will the changes continue once the assistance has ended? 8. Are there any areas in which provision of assistance might be more effective? 9. What, in your opinion, is the main constraint acting on USAID assistance in these sectors in Armenia? 10. Do you see the particular USAID assistance highly appropriate and relevant, in light of the current regulatory and legal environment in Armenia? Do you believe it fits well within the desired development strategy for Armenia? 11. Are the objectives of the assistance being accomplished? Please elaborate. 55 12. Would it help to make changes to the project design in order to deliver assistance more effectively or to respond better to current challenges? Please explain in detail. 13. Does your component of the project(s) have a human capital development component or an institutional strengthening component? If not, in your opinion, should there be? 14. Can you recommend any individuals, groups, or organizations in Armenia the evaluation team should be sure to contact? 15. On a scale of 1 – 10 (10 highest), how would you rate USAID assistance to the Armenian local government and civil society sectors? 16. Do you have a specific recommendation that the evaluation team could provide to USAID to help develop similar assistance in the future? In light of your experience, what advice would you give USAID? Interview questions for GOA: 1. What is the nature of your organization? What functions does your organization undertake? By what mandate? 2. What is your relationship with USAID? 3. What has been the nature of the assistance being provided? 4. What have been the results to date? 5. How satisfied are you with the experience with the USAID assistance? How would you rate your level of satisfaction on a scale of 1 – 10 (10 highest)? 6. Had you not been able to obtain this assistance, what other alternatives would have been available to you? 7. How effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? 8. Describe the current process of decentralization in Armenia and the GOA’s policy. 9. Has the USAID contractor’s work had an impact on GOA policy? Interview questions for USAID: 1. How does USAID/Armenia anticipate using this assessment? What are the special areas of concern? What should the assessment be sure to cover? 2. Can you briefly summarize USAID’s current policy and projects in the local government and civil society sectors? 3. Are there policies or other issues that currently represent an obstacle to the achievement of USAID’s objectives, or cause disagreement between USAID and stakeholders in these sectors, including GOA? 4. What are the respective roles and responsibilities of USAID/Armenia, USAID/ Washington, and the Armenia implementing agency in terms of project implementation? What were the respective roles in the design of the project? 5. Can the evaluation team obtain a timeline of the cost summary of the project-to-date? 6. How do the individual components fit with USAID’s development strategy for Armenia? 7. Can you please provide the evaluation team with the performance monitoring plan for each of the project’s components with the targets for these indicators, and the latest reports on how well the targets have been achieved? 8. Is it possible to obtain the original SOW for Counterpart International, as well as a timeline of modifications to the work over the life of the project? 9. Can you please provide the evaluation team with copies of the technical reports produced by the project? 10. In your view, how sustainable are the changes brought about by the projects? 11. How effective is the project’s contractor? How effective are the targeted recipients of the assistance? 12. What, in your opinion, are the main constraints in effectively carrying out the assistance? 56 13. In your opinion, does the project have a sufficient human capital development component or an institutional strengthening component? If not, should these support activities be augmented? 14. Who are the key organizations or people in Armenia that USAID feels that the evaluation team should meet while conducting the evaluation? 57 ANNEX IV: SOURCES OF INFORMATION Documents reviewed Request for Application Associate Award Quarterly reports Project Monitoring and Evaluation Plan and draft supplementary indicators Local Government Reform Strategic Approach Lists of grants awarded in FY 2011 and 2012 Cyclical grant review documents for 2011 Joint MTA and Council of Europe LSG reform action plan People interviewed USAID Yerevan Mission: Stephen Brager Anahit Martirosyan Mariam Gevorgyan Bella Markarian Project staff: Alex Sardar, Chief of Party Armen Ayunts Hayastan Stepanyan Lusine Hakobyan Artur Drampyan Five focus groups: A. CSO grantees Davit Grigoryan, Armenian Cross of Unity NGO Naira Arakelyan, Armavir Development Centre Vahan Tumasyan, Shirak Centre NGO Gohar Tonoyan, “HENK” NGO Artashes Torozyan, Partnership and Teaching NGO Artur Sakunts, Vanadzor Helsinki Citizens Assembly B. Community mayors Mayors from the following communities: Nor Geghi Akhtala Lukashin Maralik Lusadzor Vosketap C. Y/CACs and CWGs 58 Representatives of the following Y/CACs and CWGs: Akhtala Nor Geghi Y/CAC Maralik CWG Sevan CWG Urtsadzor CWG Dilijan Y/CAC Vosketap Y/CAC Vosketap CWG Lukashin CWG D. ICAs Representatives of the following ICAs: Melanya Pepanyan, Tumanyan ICA Artsruni Igityan, Akhuryan ICA Samvel Hovespyan, Berd Community Union Garnik Ghalumyan, Ijevan Community Union Armen Nersisyan, Sisian Community Union Hayk Sevsyan, Gavar (“Geghama”) ICA Davit Badoyan, Gavar (“Geghama”) ICA Askanaz Aslanyan, Aparan Community Union E. NGOs engaged in drafting the “Strategic Approach” for local government reform Vahan Movsisyan and Vahram Shahbazyan, Communities Finance Officers Association Artak Petrosyan, Municipal Councillors of Armenia Armine Tukhikyan, Urban Foundation Vahan Asatryan, International Centre for Human Development Grisha Khachatryan, Information Systems Development and Training Centre Donor stakeholders: GIZ: Hilke Ebert, Municipal and Economic Development Programme Director Council of Europe: Silvia Zehe, Head of Office British Embassy: Olya Ghazaryan, Grants Officer Open Society Foundations-Armenia: Larisa Minasyan, Executive Director Honorary Norwegian Consul: Tim Straight Armenian Government counterparts: MTA: Vache Terteryan, Deputy Minister Sites visited Counterpart’s Yerevan office Three comparison communities: Ptghni, Aramus and Hrazdan Other information sources Counterpart Armenia’s website: www.counterpart.am 59 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Joseph M. Anderson Title Senior Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pension and Labor Market Reform Project (PALM) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 6/4/13 60 Name Carl Ulbricht Title Local Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pension and Labor Market Reform Project (PALM) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Signature Date 6/4/13 61 USAID ARMENIA 1 AMERICAN AVENUE YEREVAN 0082, ARMENIA TELEPHONE: +37410 46 47 00 FAX: +37410 46 47 28 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523