MID-TERM PERFORMANCE EVALUATION FOR THE BLCD PROJECT EVALUATION REPORT DECEMBER 5, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Darcy Ashman, Kristie Evenson, and Naila Hashimova, Management Systems International. MID-TERM PERFORMANCE EVALUATION FOR THE BLCD PROJECT EVALUATION REPORT Management Systems International Corporate Offices 200 12th Street, South Arlington, VA 22202 USA Tel: + 1 703 979 7100 Contracted under Contract No. AID-OAA-I-10-00002 Task Order No. AID-112-TO-14-00014 MID-TERM PERFORMANCE EVALUATION FOR THE BUILDING LOCAL CAPACITY FOR DEVELOPMENT (BLCD) PROJECT DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report i CONTENTS Executive Summary ....................................................................................................................... 1 Introduction .................................................................................................................................... 1 II Methodology ............................................................................................................................... 2 III Findings ..................................................................................................................................... 4 Component 1: Strengthen the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas .................... 4 Component 2: Develop highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas............................. 10 Component 3: Improve legal and regulatory frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles ................................................................................................................................... 14 IV Recommendations................................................................................................................... 17 Annex 1 Tables and Figures ........................................................................................................ 20 Table 2: OCA Score Changes by Capacity Areas................................................ 20 Table 3: Completed Milestones ........................................................................... 22 Table 4: OCA Score Change and Milestones ...................................................... 23 Table 6: Sustainability Score Change .................................................................. 24 Annex 2: Summary of Legislative Changes in 2013 and 2014 ................................................. 25 Annex 3: The Evaluation Statement of Work ........................................................................... 27 Annex 4: Schedule of Evaluation ................................................................................................ 33 Annex 5: Evaluation design, methodology and questionnaires ............................................... 34 Annex 6. Interview respondents ................................................................................................. 44 Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 1 EXECUTIVE SUMMARY Evaluation purpose. The mid-term performance evaluation of the Building Local Capacity for Development (BLCD) Project determines the results of BLCD interventions as of Sept. 30, 2014; evaluates the progress made under each component based on established targets; and provides recommendations for adjustments and considerations for implementation of the rest of the project. The evaluation especially focuses on interventions and approaches that have proven to be feasible and sustainable in the toughening political context, and identify ones that provide a sound foundation for future programming. BLCD project background. Seeking to expand the space for civil society activity in meaningful ways, the ultimate objective of the Building Local Capacity for Development (BLCD) Project is to increase the organizational and technical capacity of civil society organizations (CSOs) to enable them to develop into mature and effective organizations able to fulfill their civil society public interest purposes and agendas so they may serve as relevant, effective and sustainable partners. Evaluation questions. The evaluation report addresses the following questions about each of the three main project components: Component 1: Strengthen the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas • How effective are BLCD Project capacity development interventions with partner CSOs? • Which interventions in local capacity development are most effective and why? • How far along the road to sustainability are BLCD Project partner CSOs? In what areas do they need most of the support? Component 2: Develop highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas • How effective was BLCD Project selection process for CSO resource centers? • What is the likelihood for sustainability of the selected resource centers and what support do they need to increase that likelihood? Component 3: Improve legal and regulatory environment frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles • How effective was BLCD Project in raising awareness among key stakeholders on deficiencies in current legislation affecting civil society? • Considering the changing political conditions, is there something the BLCD Project could do differently in the future to achieve additional results under this component? Evaluation methodology. The team used mixed-methods to analyze BLCD project achievements, lessons learned during the implementation efforts, and how best to refine programming for the continuation of the project. These methods included document review of BLCD reports, quantitative analysis of relevant indicator data, and 42 semi-structured interviews with five lead staff Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 2 of the project implementers (in headquarters and in Baku), 18 CSO partners, two regional CSO Resource Centers (CSRCs) and 10 potential CSO users in the regions, three representatives of the Council of State Support to NGOs, and four U.S. or international stakeholders. The team also consulted with relevant USAID staff in Washington DC and Baku. To avoid potential self-reporting bias in the performance data from BLCD reports, the team analyzed M&E data carefully, drawing attention in the report to areas where questions about the interpretation of some data remain. Visits to CSO partners included random checking of completed capacity development items. Further, the team identified key findings by triangulating data from the reports with qualitative themes derived from the interviews. The toughening climate for civil society affected the evaluation process as well as the BLCD project and its partners. Some project documents were more difficult to get in a timely way and the outlook among CSO partners worsened during the visit. Regional CSOs were feeling particularly under siege. However, in spite of these unavoidable events, the team gained access to sufficient project documentation and conducted interviews with the majority of project participants and a good cross￾section of stakeholders. Indeed, the first-hand experience helped the team to shape the evaluation report to address USAID’s interest in the evaluation of the project’s implementation and future relevance in precisely this toughening context. Major findings, lessons learned and conclusions. Overall, BLCD has made good progress towards its objectives and targeted results, despite operating in the headwinds of rapidly closing space for civil society. The report addresses project performance and key findings related to each component. Component 1: Most targets for CSO organizational and advocacy capacity development for 2013 and 2014 have been met or exceeded, although there was no evidence that the 2014 data showing increases in advocacy and diversified funding base were due to BLCD interventions. The most positive and telling increases are in the percentage of action plan items implemented, which correlates with interview data and increases in OCA scores in the nine remaining Group 1 CSOs – showing that the CSOs have improved their capacities in key targeted areas and are eager to continue. CSOs view BLCD interventions (OCA tool and action plans, training, technical assistance by staff and consultants, and grants) as a package rather than as individual elements. BLCD has won acceptance of the OCA tool and approach to capacity development through its professional and open relationships with CSOs. The trainings were mostly assessed to be useful, if a bit theoretical for many, who appreciate the inclusion of local trainers and especially the follow-up by BLCD staff and consultants. ‘Survivability,’ as one CSO put it, is probably a more relevant goal for CSO capacity development than ‘sustainability’ in the present context. Focusing on ‘survivability’ for the remaining years of BLCD would shift attention from the standardized OCA to assisting the key CSO partners to develop the capacity to keep their doors open and offices functioning, to adapt to the restrictions, identify new ways to access funds, strengthen informal information-sharing networks and work towards reopening the enabling environment for civil society. Component 2: BLCD’s process of selecting CSO resource centers has been partially effective. The centers exhibit a set of attributes needed for centers to function and provide services to users who Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 3 exhibit some level of satisfaction with their offerings. They also have some mid-term continuity perspective given their affiliation and ongoing support from the Council of State Support to NGOs. Each of the centers has a close affiliation with the NGO Support Council. This provides them a certain level of both political and financial support, particularly some guarantees of mid-term continuity. At the same time, the affiliation in some ways a priori limits the Centers’ ability to reach part of the target audience of users BLCD has prioritized; efforts to work with new established and small CSOs in more rural areas of the regions could be affected by the centers’ real and perceived levels of openness to all CSOs. This sensitivity has likely increased in the past year. BLCD support for CSO Resource centers can be linked to early gains in some areas of organizational development. Training inputs through the BLCD program have also enriched and added value to each center’s offerings. However, one third of the way through the grant, it is not clear if either center will significantly improve their functioning, interaction, and value to users beyond the grant efforts. The OCA tool and training offerings can be fine-tuned to more directly focus on each centers’ ability to engage in strategic planning, design, and servicing of their users. While this does not address the larger concerns of independent CSO user accessibility, it does provide some middle way to at least put in place better services and programming for many (if not all) of the CSOs in the regions. This, along with commitments from the NGO Council to continue funding the centers would bode well for their sustainability and relevance to smaller and less capable CSOs in the regions. Component 3: BLCD Component 3 lead and principal of MG Consulting, Mahammad Guluzade, is well-respected by the CSOs and stakeholders interviewed. BLCD has been very effective in working independently and together with the NGO Council to raise awareness about legislative changes and their effects on CSOs. BLCD has primarily addressed the changes adopted by the Parliament in December 2013 to the Law on Grants, Law on Non-governmental Organizations (‘Law on NGOs’), Law on Registration of Legal Entities and State Registry (‘Registration Law’) as well as to the Code on Administrative Offenses. The number of communication products, trainings for CSOs and policy dialogues produced by BLCD has far exceeded its targets, demonstrating that BLCD has been very responsive to the legislative changes in ways that assist CSOs to understand and cope with them. In particular, the BLCD trainings helped CSOs to increase their abilities to comply with Azerbaijani law and therefore more effectively pursue their statutory goals; learn to monitor developments with CSO legislation; access public information and interact with the government; and to solicit and account for funds. BLCD’s analysis of new legislation relating to donations and grants helped hundreds of CSOs to comply with the new rules and to avoid severe penalties resulting from related violations. BLCD sought to improve the regulatory framework for CSOs by proposing some modifications to the legislation. However, they were not accepted and the overall legislative context has become more restrictive. New changes to the Laws on NGOs and Grants were adopted on October 17, 2014 by the Parliament and signed by the President on November 15, 2014. As of this writing, the Cabinet of Ministers is developing procedural details of the new amendments and further information is not expected to be forthcoming until at least mid-January 2015. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 4 Recommendations for BLCD and USAID 1: Keep the BLCD doors open and continue implementing activities in all three components, including expansion to include more CSO partners and CSO Resource Centers. 2: Strengthen key elements in each component to be more effective in building organizational capacity in the current context. 2.1. In Component 1, consult with each CSO partner in Group 1 and 2 to reassess their action plans in light of its current strategy and situation. Revise individual action plans and capacity development interventions to strengthen strategic and survivability capacities. For Group 3 CSOs, begin each OCA assessment by selecting only the most relevant sub-areas to assess and strengthen. 2.2. Design future Component 1 training to be less theoretical and more practical in the local context. Involve selected Group 1 CSOs as co-trainers and continue follow up by BLCD staff and local consultants with CSOs to assist them to integrate and apply new skills, knowledge and systems. Include opportunities to work on actual capacity development activities in the trainings. 2.3. Offer additional training in policy research skills to key CSOs in DG and EG sectors. 2.4. Open BLCD capacity development interventions to a broad range of CSOs across the political spectrum. 2.5 In Component 2, make specific linkages between OCA subarea criteria and targets that encourage more strategy development and related learning. 2.6 Support the Resource Centers to improve their reach to smaller and less capable CSOs. 2.7. Expand Component 3 services to reach more CSOs, especially in the regions. 3: Senior USAID and USG officials reach out to GOAJ officials to create a more conducive partnership context for the BLCD and other USAID projects. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 1 INTRODUCTION The purpose of the mid-term performance evaluation of the Building Local Capacity for Development (BLCD) Project is to: 1) Determine the results of BLCD Project interventions from September 28, 2012 - September 28, 2014; 2) Evaluate the progress made under each component based on established targets; and 3) Provide recommendations on any adjustments and considerations for the implementation of the rest of the project. Management Systems International (MSI) was hired to conduct the evaluation. The MSI evaluation team included three civil society and evaluation specialists, Dr. Darcy Ashman (team lead), Dr. Kristie Evenson, and Ms. Naila Hashimova. A local logistician, Mr. Bahruz Babayev, also served as a translator for team members when necessary. The team began planning and document review on September 30, 2014, followed by an in-country visit from October 9 – October 31, which included both an in-brief and out-brief with USAID Azerbaijan. Comments made by USAID during the latter have been incorporated into this report. The evaluation team would like to give particular thanks to USAID for its helpful coordination and to the BLCD staff for their efforts to provide meeting time and documentation during the team’s visit. BLCD project background and development context. The BLCD evaluation Task Order (AID￾112-To-14-00014)1 provides the background on the project and the context for the midterm evaluation: “ NGO organizational capacity tends to be weak, with few… engaged in strategic planning or having clear reporting mechanisms that promote accountability to donors and partners. The capable, committed, and independent CSOs willing to engage in strengthening democracy are not strong enough and under-resourced.” USAID’s Country Development Cooperation Strategy (CDCS) for 2011-2016 establishes the strategic framework for its support to civil society in Intermediate Result 2.2 Civil society strengthened to increasingly engage in development and political processes, which contributes to Development Objective 2 Effective participation of diverse actors and institutions in the democratic development of Azerbaijan increased. USAID aims to develop a more participatory environment through strengthening non- governmental actors and organizations to have more influence with Government of Azerbaijan (GOAJ) and policy-makers on Azerbaijan’s development. Seeking to expand the space for civil society activity in meaningful ways, the ultimate objective of the Building Local Capacity for Development (BLCD) Project is to increase the organizational and technical capacity of civil society organizations (CSOs) to enable them to develop into mature and effective organizations able to fulfill their civil society public interest purposes and agendas so they may serve as relevant, effective and sustainable partners. BLCD aims to strengthen the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas; develop highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas; and improve legal and regulatory frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles. The mid-term evaluation will focus on interventions and approaches that have proven to be feasible and sustainable in the toughening political context, and identify ones that provide a sound 1 This BLCD project background statement is condensed from pages 3-7 of the Task Order. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 2 foundation for future programming. This report addresses the following key questions related to each of the BLCD project’s three main components: Component 1: Strengthen the organizational and technical capacities of key CSOs • How effective are BLCD Project capacity development interventions with partner CSOs? • Which interventions in local capacity development are most effective and why? • How far along the road to sustainability are BLCD Project partner CSOs? In what areas do they need most of the support? Component 2: Develop highly professional CSO Resource Center(s) • How effective was BLCD Project selection process for CSO resource centers? • What is the likelihood for sustainability of the selected resource centers and what support do they need to increase that likelihood? Component 3: Improve legal and regulatory environment • How effective was BLCD Project in raising awareness among key stakeholders on deficiencies in current legislation affecting civil society? • Considering the changing political conditions, is there something the BLCD Project could do differently in the future to achieve additional results in this component? II METHODOLOGY The team used mixed-methods to analyze BLCD project achievements, lessons learned during the implementation efforts, and how best to refine programming for the continuation of the project. Document review. The team reviewed BLCD reports and publications, including quarterly reports through June 2014, the first year annual and M&E reports, the second year M&E report, CSO partner organizational assessment and capacity development reports, and CSO resource center (CSRC) documents. The review included quantitative analysis of relevant indicators and organizational capacity scores to determine progress. Semi-structured interviews. Based on information in the reports and initial in-country consultations with USAID and interviews with BLCD, the evaluation team adapted evaluation sub￾questions to reflect key considerations and areas of investigation. They then drafted a set of interview templates for semi-structured interviews with CSO partners, CSRCs and CSO users, GOAJ and other international donors and stakeholders. As relevant, each interview included questions related to the three different components in order to map out basic trends in progress, feedback on how these processes were conducted, and forward looking reflections on addressing the current legal environment. The questions for the CSO partners and Resource Centers were field￾tested with Azeri NGO personnel not involved in the BLCD project. The questionnaires are included in Annex 5. Overall, the evaluation team conducted 38 interviews with BLCD project staff, CSO partners, CSO Resource Centers and potential CSO users, and key stakeholders in Azerbaijan between October 10 and 28, 2014. In Washington DC, the team lead conducted an additional 4 interviews in early October with USAID, Chemonics, and ICNL. The team interviewed 80% of BLCD CSO partners, including eight of 11 from Group 1 and eight of nine from Group 22 , most of whom were based in 2 See Annex 6 for list of CSOs and other stakeholders interviewed. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 3 Baku. The remaining four CSOs either did not return phone calls from the team or declined directly. Interviews were conducted in English or Azeri, the latter with the aid of a translator or by the local expert team member. Regional data collection. Both BLCD partner CSO Resource Centers were interviewed by a team member who traveled to Ganja, Shemkir, Sheki, Oguz, and Gabala. To assess CSO user perspectives, ten CSOs (five in each region) were randomly selected from a list provided by BLCD that had been compiled in June 2014 to conduct a user satisfaction survey. Most of those reached turned out to be relatively established CSOs. The team had intended to sample user perspectives during the visits to the Centers, but none were present. Similarly, the team had intended to interview smaller, less capacitated CSOs (as per the component objective), but learned that there was no existing database or list of such organizations. Notes from all interviews in Baku and the regions were analyzed and clustered together to identify common themes related to the evaluation questions in each of the three components. The team triangulated these themes with analysis from the reports and OCA reporting tools to identify key findings. This included a detailed analysis of the BLCD OCA matrices and mid-term outcomes data in order to verify improvements and determine whether there were any particularly useful trends or patterns to highlight for the report. Consistent with best practices of qualitative research, the key findings were written up in the report and supplemented with quotes from one or a few individuals that best illustrated the finding or added additional perspective to provide a fuller understanding of the situation3 . Other considerations/limitations. The toughening climate for civil society affected the evaluation process as well as the BLCD project and its partners. In the previous year, BLCD had decided not to implement some activities, e.g. advocacy interventions and grants to the second group of CSOs. The government or banks had blocked some of the first group of CSOs from accessing their BLCD (and other) grants. Several CSOs had either significantly scaled back their operations or closed their offices altogether. During the team’s in-country visit, the Parliament passed a new amendment further restricting civil society and BLCD was compelled to respond to a government inquiry. These events made getting some project documents in a timely way more difficult and seemed to worsen the outlook among CSO partners about their future, which probably led a few in Baku and the regions to decline interviews and colored the responses of those who did participate. Regional CSOs were feeling particularly under siege and unclear about their future. At least one interview was cancelled in Ganja and Gabala meetings with CSO users were circumscribed after the director of the Center requested that any future meetings take place as a larger group or in the Center. Despite these unavoidable events, by the end of the visit, the team had sufficient documentation and conducted all the interviews necessary to conduct the mid-term performance evaluation of BLCD. The evaluation report specifically addresses USAID’s interest in the assessment of the project’s implementation and future relevance in precisely this toughening context. 3 See testimony on qualitative research by Margery Austin Turner, The Urban Institute, submitted for the record to the Subcommittee on Human Resources Committee on Ways and Means, United States House of Representatives, July 17, 2013. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 4 III FINDINGS The key findings presented below address the main questions posed in the evaluation design, organized by component. For each component, the report includes: 1) a brief summary of the main activities implemented under the component, 2) specific findings related to each of the evaluation questions, and 3) conclusions about progress made in the component overall. Recommendations for each component and the project overall are provided in the final section of the evaluation report. Gender equality: USAID evaluation policy requires evaluations to assess project outcomes and impacts on males and females. BLCD was launched at about the same time as the 2012 USAID Gender Equality and Female Empowerment policy guidelines. BLCD produced an excellent plan to integrate gender into its technical approach, although the staff acknowledged that there had been little attention to gender-focused capacity development among the CSO partners, given the worsening environment. Indeed, the legislative and political events did not appear to discriminate by gender. Three of the 18 CSOs were women-headed, and two were women-focused. The other 15 CSOs were male-headed and all but one or two of the senior staff included in CSO interviews were male. Although a small percentage of the CSO beneficiaries, all three women leaders were especially thoughtful and proactive about their organization’s future. The only two CSOs to tell us that they intentionally worked ahead of schedule were women-headed. Component 1: Strengthen the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas Summary of main activities. To date, BLCD has strengthened the organizational capacities of 18 selected CSOs, primarily through a set of activities which include: conducting organizational assessments with the USAID Organizational Capacity Assessment (OCA) tool, assisting CSOs to use assessment results to prepare action plans for capacity building, providing comprehensive training in CSO management on topics related to the areas covered by the OCA, providing consultation to the CSOs by BLCD staff and local experts, and providing grants to CSOs to fund their individual capacity building plans. BLCD has implemented some activities to strengthen the technical advocacy capabilities of CSO partners, such as training by ICNL on advocacy, legislation monitoring and access to information, but the political environment, especially since July 2014, has curtailed the project’s implementation plans as well as uptake of actual advocacy and coalition building by CSOs. 1.1 How effective are BLCD capacity development interventions with partner CSOs? The team presumed BLCD interventions to be effective if they achieved their objectives for actual capacity development and satisfied those CSOs who participated in them4 . To measure capacity development, the team reviewed overall BLCD performance data, compared changes in OCA scores as evidence of capacity development (available for nine Group 1 CSOs), and interviewed 16 partner CSOs about their views on the effectiveness and sustainability of the interventions. 4 Herman and Renz (1999), “Theses on Nonprofit Organizational Effectiveness” Nonprofit and Voluntary Sector Quarterly, 28 (2), 107-126. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 5 Performance analysis. BLCD has been mostly successful in meeting or exceeding its targets, as shown in Table 1, below. In only one indicator, ‘number of CSOs receiving USG assistance engaged in building their organizational capacity’, BLCD missed its target of 12 CSOs by 3 for a total of 9 CSOs. The M&E report states that this was agreed to with USAID since 2 of the first group of 11 CSOs had effectively dropped out of the program5 . In the first year of the project, 2013, BLCD met or exceeded targets in the three key result area indicators in which it set targets. BLCD has surpassed its targets for CSOs having progressed in organizational capacity (reaching 9 as compared to 5 CSOs) and the percentage of action plan items implemented by target CSOs (19% as compared to 15% in 2013 and 20.44% as compared to 20% in 2014), even with 3 fewer CSOs engaged. This progress reflects solid performance in BLCD’s key focus of strengthening CSO organizational capacity, even in the midst of an increasingly difficult environment for CSOs. This finding was confirmed by the team’s analysis of OCA score changes and interviews with the CSO partners, as discussed below. Table 1: Component 1 Midterm Performance 6 Indicator Baseline Target 2013 Actual 2013 Target 2014 Actual 2014 PIR 1: Organizational and technical capacity of key CSOs, coalitions, and associations improved Number of CSOs having progressed in organizational capacity 0 0 0 5 9 KRA 1.1: Organizational capacity action planning and implementation improved Number of CSOs receiving USG assistance engaged in building their organizational capacity 0 10 11 12 9 Percentage of action plan items implemented by target CSOs 0% 15% 19% 20% 20.44% KRA 1.2: Good NGO governance standards adhered to by key CSOs Number of target CSOs with improved adherence to NGO good governance standard 0 0 0 3 6 KRA 1.3: Advocacy capacity of key CSOs and coalitions increased Number of Civil Society Organizations (CSOs) receiving USG assistance engaged in advocacy interventions 0 0 0 5 14 Number of key CSOs active in advocacy coalitions 0 0 0 3 4 KRA 1.4: Key CSO funding base diversified and increased Number of key CSOs with diversified funding base 1 1 1 2 5 5 The Azerbaijan Lawyer’s Confederation (ALC) grant was terminated and the Election Monitoring and Democracy Studies Center (EMDS) was destabilized when its Chairman was imprisoned. 6 Compiled from BLCD Monitoring and Evaluation Reports, Years 1 and 2. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 6 0 10 20 30 19 23 11 9 18 22 28 5 25 NOTE: Two organizations (MRI and AHD) changed the OCA category from "low" to "basic" by the mid-term assessment. However, an attribution question must be raised regarding the data provided for two indicators (number of key CSOs active in advocacy coalitions and number of key CSOs with diversified funding base): are the numbers reported due to BLCD interventions or to independent action by the CSOs? The intention behind these indicators is not clear from the documentation, although one would assume that they are meant to track progress due to the project interventions. Whereas the team’s interviews and BLCD project documentation confirms that the progress reported on the organizational capacity indicators can likely be attributed to interventions by the project, in contrast, none of the CSOs interviewed said that BLCD helped them to become active in advocacy coalitions (which may have been due to the closing space for advocacy) or to diversify their funding base. Further, several CSOs said they had participated in advocacy prior to BLCD and most of them had track records of receiving funding from diverse donors prior to participating in BLCD. Overall, the team found that the main BLCD capacity development interventions based on the OCA and related training, technical assistance, and grants have been very effective in enabling the CSOs to assess their organizational capacity, develop and implement plans to strengthen target areas. That said, the OCA tool has both strengths and weaknesses as applied in the Azerbaijani environment, as discussed below. OCA Scores and Milestone Progress. Comparing baseline OCA scores with the mid-term OCA scores should show additional evidence of capacity development, and indeed, the data for the nine Group 1 CSOs confirms progress, as Figure 1 below shows. The degree of score change ranged from five to 28 (both scores in health sector CSOs); the average score change was 18. More detailed data on OCA score changes by the nine CSOs in the specific capacity areas is shown in Table 2, Annex 1, p. 20. Two CSOs improved their category from ‘low’ to ‘basic’ (MRI and AHD). AZMA remained in the ‘low’ category and all of the others remained in the ‘basic’ category. Since there are only .49 points in the ‘low’ range as compared to 1 or more in the others, it is not surprising that the only two improvements in category moved from low to basic7 . Figure 1 Total Score Change CSO progress in submitting their milestones to demonstrate achievement of their action plans is shown in Table 3, Annex 1, p. 22. Three of the nine CSOs were on track in submitting milestones according to their action plans (TIA, AHD, and AHCA); of the other six CSOs, three had 14% delayed milestones (DL, L&D, and EDF) and three had 29% delayed milestones (WARD, MRI and AZMA). WARD informed us that they had discovered they needed to work on the milestones in a different order than planned, but that they expected to have fully completed their plan by the end date. No single sector or other variable (e.g. age or size) appears to be associated with capacity development effectiveness. 7 See BLCD OCA Baseline Report, 2013, for definitions and categories. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 7 Changes in Capacity Areas. BLCD and the CSOs sequenced the action plans to address weaker areas first, which included governance (priority), administration and human resource management (weaker). Hence, milestones had been completed and OCA score changes recorded in these areas. We were told by many of the CSOs that work was yet to be done in the financial management area, for example. Perhaps most telling of BLCD’s effectiveness is the evidence that CSOs appear to be working ahead of their plans to strengthen capacity in a range of areas. As Table 4, Annex 1, p. 23. shows, all of the CSOs showed improvements in mid-term OCA scores in areas NOT captured by milestones they had submitted. Some of the CSOs reported that they were working ahead because they were motivated to do so. There is no better evidence of organizational buy-in and ownership than this. CSO Partner Satisfaction. Most of the CSO partners we spoke with said that, overall, they are very satisfied with BLCD and have found the capacity development interventions useful. For many, BLCD is their first experience with a capacity development project that goes beyond training to enable them to make changes in their organizations. Given the uncertainty and challenges of the current context, most are very eager for BLCD to continue and asked the team to pass on the message to USAID not to leave the country, “otherwise we will be alone.” Virtually all of the CSOs described their new capacity as “documents” or “organizational policies and procedures” in areas which they had previously neglected, e.g. human resource management, administration, and governance. In the governance area, several described how BLCD had made them aware of the need for a succession plan and were glad to be putting one in place. Given the prevalence of founder-led CSOs in the country, this is a potentially significant contribution if it leads the CSOs to develop second tier leadership and delegate more responsibility, as at least one of the CSOs said they had done. Some are also looking forward to installing the “C.1” financial management software widely used in the country. More than half of the nine Group 1 CSOs said they had reacted negatively to the first OCA assessment, either disagreeing with their score or the categories/descriptions in the tool itself, seeing them as based on U.S. or international NGOs and “not relevant” to Azerbaijani law or context. However, all said they now better understood and accepted the tool and the BLCD approach. 1.2 Which interventions in local capacity development are most effective and why? The CSOs said that they could not rate the relative effectiveness of the main BLCD capacity development interventions (OCA/action plans; training; technical assistance; and grants) since they functioned as an integrated and complementary whole or ‘package’. They did, however, offer their analyses of the strengths and weaknesses of each; key themes are summarized in Table 5, below. The team was interested in the CSOs’ experience of the OCA process, since this is the core framework for BLCD capacity development and a relatively new USAID tool (although it was developed from similar tools used by many implementers for years). The BLCD OCA process of expert-led assessment allowed for a degree of participation and reflection by the organization. Yet this participatory aspect was somewhat circumscribed by a fairly standard action template. There was specification by organization, but for the most part the organizations were working on similar things during the same times, regardless of priorities. Organizations either worked ahead or changed around the order to make sense in their own organization. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 8 Table 5: Relative Effectiveness of Capacity Development Interventions Intervention Strengths Weaknesses OCA and Action Plans +Comprehensive, objective, specific +Seen as embodying international standards +Belief that donors (and GOAJ) will value results +Useful for growth phase of the organization, once it is established - Not adapted to Azerbaijan legal and civil society context - Not relevant to all CSOs -Overly standardized application Training +Comprehensive topics + Like inclusion of local experts and practical exercises +Legislative trainings “excellent” - Some too theoretical; not practical/local enough -Takes a lot time away from office Technical assistance by BLCD staff and consultants + Competent and professional +BLCD staff appreciated, “even in this context, Morana asks what they can do for us” +Local experts understand the context and are helpful to the CSOs -None mentioned Grants +Makes the organization commit + Covers external experts and some staff time +Valuable despite small amount -At first, seemed like a lot of paperwork, now okay As noted above, the political environment has thwarted implementation and uptake of advocacy and coalition building. Several CSOs mentioned that they had stopped previous types of networking for advocacy. One CSO in particular described how a group of CSOs met in early in 2014 to address the new regulations on NGOs, but stopped after it became obvious that the discussions were reported immediately to the government. Most of the CSOs said that, in addition to BLCD meetings, they kept up with their own informal networks to keep informed about relevant legislative and political events. 1.3 How far along the road to sustainability are BLCD partner CSOs? In what areas do they need most support? BLCD uses the OCA as a measure of sustainability, which attributes it to Level 4, e.g. “The CSO is fully functioning and sustainable, with diversified resource base and partnership relationships…. The CSO policies and procedures are in high quality, updated regularly and adhered consistently.” This is a questionable measure, considering international evidence of and professional experience with CSO sustainability, but it is used to address question 1.3 before presenting other alternatives for BLCD and USAID to consider. OCA-wide measure. As BLCD documents note, most of the CSOs were rated in the low (Level 1) or basic (Level 2) category overall. Neither BLCD nor the CSOs set targets of reaching Level 4. Instead, as noted above, they developed Action Plans to make concrete changes in specific areas, and it is these Action Plans that were used to set indicators and measure progress. It seems unlikely then, that the CSOs would achieve Level 4, for two reasons: 1) This was not set as an objective, and 2) It would be expecting too much change in too short a period of time. For example, one of the most enthusiastic CSOs said they aspired to reach a Level 3, ‘moderate’ capacity by the end of the project. Another suggested that Level 4 seemed like a very high level to attain, saying that he would like to go on a study tour to visit an actual Level 4 CSO. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 9 Common sub-areas associated with sustainability. A more conventional way to measure CSO sustainability would assess specific sub-areas shown to be associated with organizational longevity, as distinguished from effectiveness or high performance. Typical capacity areas found in the OCA would include financial viability and management, program/project management and governance/leadership. Donors want CSOs to have the capability to raise funds from other donors and local sources once their own grant is completed. This usually means that at a minimum, CSOs must demonstrate leadership, a track record of action in desired areas, and the ability to raise funds. Assessing OCA score changes among the nine Group 1 CSOs in these three sub-areas (see Table 6, Annex 1, p.24), the most capacity development has been achieved in governance (average score change of 4.33) as compared to financial management (average change of 1.3) and program management (average change of 2). Again, that is due to the sequencing of the BLCD capacity development interventions to target good governance in particular and the other weaker areas first. The most obvious answer to the question of the areas in which they need the most support is to follow through on the plans with each of the CSOs for building financial management and program management capacity, especially the item, “New Opportunity Development for Sustainability.” ‘Survivability’ as a route to sustainability. Given the severe threats to independent CSOs in the current environment, the sustainability question may be better framed as one of “survivability,” in the words of one CSO who put it particularly succinctly. Specifically, the most recent amendment has raised concerns about the extent to which international donors will be able to make grants to CSOs. If they are significantly restricted, CSOs will have to turn to other sources for funds, primarily the government. Despite the statements by the GOAJ that it will increase the scale of funding to CSOs, most of the BLCD partners say that such a move will limit their independence as well as the scale of their operations. Even those that work more closely with the government are uncertain and worried about the extent to which the prospective reduction of international funds will affect them. By all reports, donations are a relatively small source of funds and corporate social responsibility/philanthropy to CSOs is not well-established in the country. Besides the recent issue of international donor funding, several of the partner CSOs reported not being able to access even the grants that they had been awarded, due either to rejections by a ministry or being denied access to their own account by their bank. Since the basis for these rejections and denials is sometimes not clear or even accurate, an atmosphere of concern and uncertainty has been created. Therefore, focusing on ‘survivability’ for the remaining years of BLCD would shift attention from the standardized OCA or even traditional sustainability measure to assisting the key CSO partners to develop the capacity to keep their doors open and offices functioning (if even from living rooms), to adapt to the restrictions, identify new ways to access funds (such as adopting a business model as some have done), strengthen informal information-sharing networks and work towards reopening the enabling environment for civil society. The report returns to this theme in the recommendations. Conclusions: BLCD has made substantial progress in the organizational capacity development of its CSO partners. BLCD has won acceptance of the OCA tool and approach to capacity development through its professional and open relationships with CSOs in the context of severe threats to continued presence of foreign donors and independent civil society. However, the individual organizational action plans are relatively standardized rather than tailored, which may make them easier for the project to administer, but less relevant to the specific needs of the individual CSOs. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 10 For transferring content to actual practice, pairing international trainers and experts with local ones and building in practical application exercises to trainings are key to BLCD’s effectiveness with most CSOs, as is staff availability for individualized consultations. Although CSOs may have similar characteristics, each one is unique; successful organizational development work is ‘retail’ rather than ‘wholesale,’ as the respect and appreciation for the responsiveness of BLCD staff demonstrates. Component 2: Develop highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas Summary of main activities. BLCD’s support for the capacity development of NGO Resource Centers, accessible to regional and smaller NGOs has been underway for two years. However due to delays during year one, capacity development efforts have primarily been undertaken during the last 9 months. Based on an OCA assessment undertaken in November 2013, each organization signed a 24-month grant in January 2014 and since then they have been focused on addressing specific OCA areas while also delivering BLCD-initiated training activities for Resource Center users. 2.1 How effective was BLCD Project selection process for CSO Resource Centers? Performance analysis. Due to the delays described above, the most solid evidence the team found among the M&E data summarized in Table 7, below, is that BLCD has completed 10% of organizational and financial capacity targets by September 2014. Questions about the other numbers are noted in footnotes eight and nine, below. Table 7: Component 2 Midterm Performance Indicator Baseline Target 2013 Actual 2013 Target 2014 Actual 2014 PIR 2: Increased access to resources for smaller and less capable CSOs through resource centers Number of CSOs accessing resources through selected resources centers TBD TBD - TBD 45 KRA 2.1: Quality of service provision by CSO resource center(s) improved Percentage of user satisfaction with services delivered by CSO resource centers 75% TBD - ≥75% 89.23% KRA 2.2: Organizational and financial capacity of selected resource centers improved Percentage of target sustainability plan items implemented 0% 10% 0% 20% 10% KRA 2.3: Increased awareness of alternative resources for CSO support services Number of people accessing NGO portal (web visits) 0 0 0 200 0 Key findings. BLCD’s process of selecting CSO resource centers has been partially effective. The centers exhibit a set of attributes needed for centers to function and provide services to users 8 who exhibit some level of satisfaction with their offerings9 . They also have some mid-term continuity perspective given their affiliation and ongoing support from the GOAJ NGO Support Council. This 8 Actual number of users is unclear as the figure of 45 represents the number which participated in the June 2014 satisfaction survey. 9 The high satisfaction rate percentages noted in the baseline (75%) and for 2014 satisfaction survey results (89.235) should be considered in relation to satisfaction survey identified weaknesses as well as the difficulty in identifying and getting input from center users for the survey. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 11 is demonstrated through assessment and OCA reports as well as site visits, discussions with users of the centers, and the NGO Support Council. The process of identifying and selecting potential resource center partners was a combination of a mapping assessment, consultation efforts, and a call for proposals. BLCD undertook a mapping assessment of resources centers in December 2012 to better understand potential and context. They also were in close consultations with the NGO Support Council, USAID, and another USAID project, SEDA, which is supporting other existing resource centers affiliated with the NGO Support Council to identify most likely candidates. Based on this assessment and consultations, BLCD designed a proposal process to motivate potential resource center candidates. BLCD issued an open call for concept papers from interested centers in July 2013 and later a shortlisted call for full proposals of the two best candidates based on the specified criteria10. While it is somewhat unclear whether these criteria were ‘fitted’ to the two centers initially identified, within the criteria outlined, the selection of Shemkir and Gabala Resource Centers was reasonable. The selection of these centers presents an opportunity for BLCD, but also challenges. Each has a close affiliation with the NGO Support Council. This provides them a certain level of both political and financial support, particularly some guarantees of mid-term continuity. At the same time, the NGO Support Council affiliation in some ways a priori limits the Centers’ ability to reach part of the target audience of users BLCD has prioritized; efforts to work with newly established and small CSOs in more rural areas of the regions could be affected by the centers’ real and perceived levels of openness to all CSOs. This sensitivity has likely increased in the past year. Users in Shemkir suggested that while the center is relatively open, there are clear limits to this openness for unregistered CSOs and those working in areas that are considered politically sensitive11. The Gabala Resource Center noted that they work with initiative groups to assist with registration (often youth), but overall they are prohibited from working with unregistered CSOs. Users interviewed in this region were quite complimentary about the Center, so it is unclear whether CSOs also have this perception in Gabala region. Regardless, it appears that the degree to which each center can or does cooperate with other CSO or initiative groups comes down to individuals and personal connections. Partly in relation to this, the location of both centers provides some challenges. In both cases, Ganja town for Shemkir and Sheki town in relation to Gabala can be considered to be hubs of civil society and concentrations of activity rather than where the resource centers are located. The more established organizations in each of these cities cooperate with the resource centers, but they mostly do this in relation to hosting events rather than seeking specific assistance from the centers. Each city also has had or currently has a number of NGOs which provide de facto resource center type services, and most interviewed did not see a need to travel to the other city just to use computer or similar services of the center. Part of the selection logic was to engage with more rural and less experienced CSOs. Here the geographic concentration of active CSOs is less important than the ability of the centers to reach out 10 These include 3 main eligibility criteria: existence as a research center for at least 3 years; established base of operations outside of Baku; effectiveness in building good relations with local communities, political leadership and the NGO Support Council. (BLCD Capacity Assessment and Mapping Report of Two CSO Resource Centers in Azerbaijan, November 2013). 11 For example, one user described how the center tried to assist in registration, but only if the organization’s mandate would be changed. At the same time, such organizations have held trainings or meetings related to their work in the center, if they design and present the events in a particular way. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 12 to the more remote areas. This distance suggests that the centers themselves likely (and as is planned) need to have some mobile unit engagements and better utilize their representatives in these remote areas12. This means in practice, that at least currently much of the actual walk-in use of the centers appears to be more for individual community members where they are located – computer classes, English classes, etc., than for either established or more remotely located CSOs from each center’s region of responsibility. 2.2 What is the likelihood for sustainability of the selected resource centers and what support do they need to increase that likelihood? BLCD defines resource center sustainability13 as a combination of organizational and programming capacities and financial stability. The sustainability of the two resource centers is quite likely if considering some improvements in current organizational capacities and funding prospects and slightly enhanced programming. Sustainability perspectives are less clear if sustainability is considered in relation to making and sustaining more substantial improvements in these areas, particularly in the programming and quality of service provision to users. This is partially due to the relatively early stages of support for the centers – only 10% of capacity development OCA plans has been completed. It would be unfair to suggest that the centers should have significant improvements either in areas that they have not significantly covered in their action plans or where cumulative results would be more obvious after some time. Yet, already at this stage, the quality of the learning raises some questions. From the 10% completion, some areas show organizational development progress on a technical level. Both centers have met benchmarks for creation of succession plans, human resource plans etc. Both centers could identify areas of procedure/document enhancement and show these and ownership of these14. Yet, there has been little evidence of such progress on the strategic development level. BLCD early on identified weaknesses in both centers’ ability to identify and craft programming based on perceived user needs in the areas. BLCD has made efforts to infuse their grants with tools to do this, but results to date have been minimal. The resource center OCAs15 include 3 areas where more substantive guidance on programming development could be addressed. Namely program management, service delivery, and external relations subareas all include some components, which address the centers’ ability to identify, strategize, design, and to some extent carry out (or at least oversee) relevant and targeted programming for their users. 12 For example, Gabala Resource Center describes how it has 12 representatives – one in each district – to communicate on behalf of the center with CSOs in these regions. 13 BLCD uses the term ‘sustainability’ to denote two things: first, the name of the OCA plan for the CSO Resource Centers is a ‘Sustainability Plan.’ Within this plan one sub-area is ‘sustainability’ which includes programming, organizational and financial sustainability. We consider sustainability accordingly as a combination of organizational capacities (including financial stability) and programming capacities. 14 BLCD consultants work closely with staff on skill enhancement related to the OCA, and this mentoring has been appreciated. 15 The benchmarks when compared with the Sustainability Plans (the OCA Performance Based solution plans) do not provide an immediate logic for understanding how the structuring of OCA related development is taking place or the schedule for these as most are marked as ‘high’ priority with dates and schedule to be determined Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 13 While it is unclear actually how much has been done in any of these three subareas16, there is little evidence to date that sub-areas, in general, and their milestone targets have been designed or carried out in a way that maximizes the strategic learning opportunities. For example, both centers completed needs assessments in June 2014 as part of their benchmarks, but they are of a relatively poor quality. They identify the need to target programming on smaller and more remote CSOs and suggest general ways to do this; however, they provide few additional insights into how they will actually design and carry out more targeted programming. One quite obvious area, for example, that was expressed during user interviews was for enhanced legal services offerings (given the many changes in 2014); yet, neither center when interviewed indicated that they had responded to this with enhanced offerings (or had plans for enhanced offerings)17. Similarly neither center appears to have considered how different levels of CSOs might have specific needs and interests, despite the need for such differentiation noted earlier on in BLCD assessment reports18. It could be that both centers need to focus on newer CSOs, but then they need to be able to consider how and in which ways they can use more experienced CSOs and or can bring value for such users. Both centers seemed to think that having a set of trainings every so often would be sufficient, regardless if such trainings were repetitive for long-time users. Discussions with both centers underlined the gaps in this strategic development capacity (to date). They also raised the question of whether the centers and BLCD even have a shared understanding of what is expected. The gap between technical and strategic capacity development may also be partly due to the grant structures. The grants are structured to provide OCA related skill training to center staff, cover some core cost, and at the same time they are used to fund trainings for local CSOs. The general logic behind the BLCD intervention and grant structure appears to be that parallel inputs into the centers will be complementary; by partnering with centers to work on their own capacity development and also providing concrete ways for them to improve their service offerings (primarily trainings offered through the grant) then both objectives enhanced center capacities, and enhanced user services will be achieved. The results to date suggest that this logic might need to be re-examined; neither center appears to be using the training offerings as an opportunity for learning how to further develop their capacities for strategizing how to plan such training programming. Also neither center seems to have significantly changed their understandings or attitudes toward strategic programming development. It is unclear how either center will be able to further the larger strategic objectives for the centers without getting these basics. Specifically it raises the questions of how sustainable such improvements will be after the grants. BLCD inputs for trainings are now increasing centers’ offerings, and thereby ‘usage,’19 but it is 16 The benchmarks when compared with the Sustainability Plans (the OCA Performance Based solution plans) do not provide an immediate logic for understanding how the structuring of OCA related development is taking place or the schedule for these as most are marked as ‘high’ priority with dates and schedule to be determined 17 Each center has staff that can provide basic advice on registration and related issues, but does not have a legal expert either on staff (anymore) or regularly working with the centers’ users. 18 While the satisfaction survey suggests that a large majority is satisfied with needs assessment on subjects of trainings – 85.8%, the OCA assessment and field interviews suggest that this remains a concern. 19 This is also relevant to trends in user rates: Shemkir claims that user numbers went from 100+ to 250-300, given the BLCD training offerings (although they did not have specific data to confirm this). Gabala said that they have about the same user rates (250-300), however they offered relatively more activities (BLCD training offerings) to achieve the same number of users. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 14 unclear whether the centers will have the ability or the funds to replicate these in the future, or perhaps even more critically, to plan for the appropriate trainings needed by their users. Programming continuation/follow-on according to the current performance trajectories is therefore questionable. This also affects financial planning and sustainability. Both centers expressed concerns about financial assistance and the need to be more engaged. Both also explained how their income generating activities could or currently helps support the center’s financing, and there was interest to further explore financial diversification. Yet (and again this may be due to the stage of their grant), there was little evidence of much progress in thinking or skills beyond what was noted in the initial OCA baseline assessments. Overall then, if taking a comprehensive understanding of the objective to further strengthen the organizational capacity of resource centers and to improve the range and quality of services they provide to CSOs, the progress to date has been quite low. It is still early, and BLCD probably has time to amply the centers’ learning, but this needs to be done in a more explicit and calibrated way. Conclusion: BLCD support for CSO Resource centers can be linked to early gains in some areas of organizational development. Training inputs through the BLCD program have also enriched and added value to each center’s offerings. However, one third of the way through the grant, it is not clear if either center will significantly improve their functioning, interaction, and value to users beyond the grant efforts. The OCA tool and training offerings can be fine-tuned to more directly focus on each centers’ ability to engage in strategic planning, design, and servicing of their users. While this does not address the larger concerns of independent CSO user accessibility, it does provide some middle way to at least put in place better services and programming for many (if not all) of the CSOs in the regions. Component 3: Improve legal and regulatory frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles Summary of main activities. BLCD has conducted nine trainings for partner CSOs in Baku and an information session in Shemkir in January 2014 to inform regional CSOs on the recent legislative changes. BLCD also produced about 30 updates by the Component 3 lead, Mahammad Guluzade, on developments in CSO legislation which kept Azerbaijani stakeholders up to date with evolving CSO legislation. The project prepared six sets of recommendations to improve CSO legislation and submitted them to the Government. BLCD updated the Assessment of the Legal Framework for CSOs in the Republic of Azerbaijan which allows local and international stakeholders to learn about their rights and responsibilities under Azerbaijani CSO law and helps them to identify priorities for their advocacy efforts dedicated to improving the regulatory environment for CSOs. Several BLCD recipient CSOs and Mr. Guluzade are members of the Commission on Law and Monitoring which was established by the Council on State Support to NGOs under the President of the Republic of Azerbaijan. Working collaboratively with these state bodies, BLCD organized nine policy dialog events, including regional events in Gabala and Ganja to discuss changes to NGO legislation over the past two years. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 15 3.1 How effective was the BLCD project in raising awareness among key stakeholders on deficiencies in current legislation affecting civil society? Performance analysis. BLCD has met or exceeded its targets on all but one of its indicators for this component, as shown in Table 8, below. The number of communication products, trainings for CSOs and policy dialogues have far exceeded their targets, which demonstrates that BLCD has been very responsive to the legislative changes in ways that assist CSOs to understand and cope with them. BLCD has not been successful in seeing its proposed modifications approved, and in fact, the legislative context has worsened. Key findings. The team affirms that a healthy legal and regulatory enabling environment is an indispensable condition for CSOs’ success and sustainability. At a fundamental level, laws governing registration, taxation, activities, and funding sources affect CSOs’ ability to survive. Furthermore, the legal framework for CSOs can encourage or constrain their ability to advance reform agendas and promote a sustainable democratic culture. Table 8: Component 3 Midterm Performance Indicator Baseline Target 20123 Actual 2013 Target 2014 Actual 2014 PIR 3: Legal and regulatory frameworks to safeguard CSO enabling environment improved Number of positive modifications to enabling legislation/regulation for civil society Drafted 0 1 4 1 2 Approved 0 1 0 1 0 KRA 3.1: CSO and government of Azerbaijan representatives better informed on deficiencies in laws and regulations that affect civil society Number of communication products on legal issues disseminated to CSOs and GOAJ 0 2 13 2 9 KRA 3.2: Advocacy and monitoring capacity of CSOs and coalitions for legal reforms increased Number of trainings for CSOs to improve expertise & capacity to monitor, analyze, and publicize legal and regulatory reform issues 0 3 3 2 7 KRA 3.3: Increased interaction between government officials and representatives of CSOs in development and implementation of CSO related legislation Number of policy dialogue events/discussions in which government officials and representatives of USAID-supported CSOs participated 0 2 1 2 8 Two major sets of legislative changes have deeply affected CSOs (See detailed description in Annex 2). The BLCD interventions have primarily addressed the changes adopted by the Parliament in December 2013 to the Law on Grants, Law on Non-governmental Organizations (‘Law on NGOs’), Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 16 Law on Registration of Legal Entities and State Registry (‘Registration Law’) as well as to the Code on Administrative Offenses. The new Law was published and entered into force on February 3, 2014. The new Law introduced new obligations for organizations in regards to their registration as legal entities, their receipt and use of grants and reporting to the government, along with other obligations. In addition, the changes established high penalties for those who violate these new and previously existing obligations under the law. The second set of changes amends the Laws on NGOs and Grants. It was adopted on October 17, 2014 by the Parliament, and has yet to be fully understood and responded to by BLCD and the CSO community. The new amendments require foreign donors to obtain permission to give out grants to local NGOs, require NGOs to register donations and to sign a service contract to provide such services as sales of goods, provision of services, and fulfillment of works. NGOs shall submit a service contract for registration when such services are provided with foreign funding. According to the new amendments, the relevant executive authority shall define relevant rules for implementation of the new requirement on registration of donors. The President signed the new amendments on November 15, 2014 and they came into force from November 16, 2014. The Cabinet of Ministers is tasked with developing procedural details of the new amendments. This process is closed to NGOs. It is expected that the earliest any information will be forthcoming is mid-January 2015. All these amendments further restrict operations of CSOs in the country with many CSOs being unable to receive registration of their grants in the Ministry of Justice (MoJ), or obtain an updated certification of the registration of the organization. The MoJ issues rejection letters to all CSOs stating various reasons for rejection. There is a shared concern among CSOs that the process of grants registration will be delayed until the new amendments to the Law on NGOs come into force later this year. Given this context, the team heard nearly universal appreciation for BLCD’s awareness-raising efforts from the partner CSOs and other international donors. BLCD appears to have used every channel possible, including training, updates, dialogue, and ‘open door’ type availability to CSOs of the Component 3 lead and his fellow colleagues in MG Consulting. All the CSOs interviewed said that they found the information extremely useful and MG Consulting very approachable and helpful. In particular, the BLCD trainings helped CSOs to increase their abilities to comply with Azerbaijani law and therefore more effectively pursue their statutory goals; learn to monitor developments with CSO legislation; access public information and interact with the government; and to solicit and account for funds. BLCD’s analysis of new legislation relating to donations and grants helped hundreds of CSOs to comply with the new rules and to avoid severe penalties resulting from related violations. A small caveat to this acclaim rounds out this finding. Two respondents said that they would have liked more insight and influence on the legislation from Mr. Guluzade and his associates. 3.2 Considering the changing political conditions, is there something BLCD could do differently in the future to achieve additional results in this component? The team began the evaluation seeking to determine if alternative venues, communication channels, potential champions and opinion-leaders were available in the country for BLCD to consider tapping. However, as noted above, we discovered a project and an entire sector that appear to be fighting for their own survival. BLCD’s products and outreach in this component were excellent Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 17 and its collaborative relationships with the NGO Support Council were strategic as far as they went, but much stronger political currents appear to be at work in the country. IV RECOMMENDATIONS Based on the evidence-based findings presented above, the team offers several recommendations to BLCD and USAID to continue, improve, and expand the capacity development interventions that are proving to be effective and sustainable in the present context of Azerbaijan. Recommendation 1: Keep BLCD doors open and continue implementing activities in all three components, including expansion to include more CSO partners and CSO Resource Centers. Given the concerns about the potential negative impact on international donors of the Oct. 17th amendment and investigations of BLCD at the time of the evaluation, this recommendation needs to be stated explicitly. The evidence reviewed by the evaluation team indicates that project interventions are mostly achieving their objectives and satisfying their intended beneficiaries, especially in a very challenging environment. Although its advocacy, coalition building, and legislative influence interventions have not been successful to date, the project appears to have done what it could by providing training and regular updates, convening policy dialogues and making recommendations to the GOAJ. The core organizational capacity development interventions, including legislative updates and trainings with CSO partners and Resource Centers are valuable and relatively safe activities for BLCD to engage in with a broad group of CSOs for the next year or two, presuming the larger political and legislative context remains the same. BLCD may or may not be able to award grants, but even if not, the training and technical assistance is valuable and can be adapted to meet the capacity development objectives with CSO partners. The following recommendations address improvements in BLCD project components and possible action by USAID and the USG to improve the environment for BLCD’s work. Recommendation 2: Strengthen key elements in each component to be more effective in building CSO and CSO Resource Center capacity in the current context. While the BLCD overall methodology and technical expertise seem quite sound, the team encourages BLCD to adopt a more individualized approach with CSO partners and Resource Centers which would enable them to focus on the specific capacity areas most needed for their strategies and ‘survivability.’ Whereas the OCA tool is comprehensive and detailed, the BLCD OCA-driven capacity development process serves to focus attention on the tool and its presumed standards, rather than to help tailor assistance to address specific needs of particular organizations to meet present opportunities and challenges. Especially considering the threats to CSOs in the current environment, BLCD capacity development interventions should address CSOs most critical concerns and assist them to seek creative rather than pre-packaged solutions. 2.1. In Component 1, consult with each CSO partner in Group 1 and 2 to reassess their action plans in light of its current strategy and situation. Revise individual Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 18 action plans and capacity development interventions to strengthen strategic and survivability capacities. For Group 3 CSOs, begin each OCA assessment by selecting only the most relevant sub-areas to assess and strengthen. Key capacity areas would include, but not be limited to compliance with emerging laws and regulations affecting NGOs, options for financial viability, strategic planning (or coping) related to programming options, and networking to share information, provide mutual support and build coalitions when and where possible. 2.2. Design future Component 1 training to be less theoretical and more practical in the local context. Involve selected Group 1 CSOs as co-trainers and continue follow up by BLCD staff and local consultants with CSOs to assist them to integrate and apply new skills, knowledge and systems. Training topics could be revised based on results of the reviews of individual action plans. The external insights of international experts can be complemented by local knowledge and expertise of consultants and the CSOs themselves. Given the high ownership level expressed by several of the Group 1 CSOs, there seems to be a great opportunity for BLCD to develop local CSO management and capacity development expertise. If not possible to award grants, structure training to include actual capacity development. 2.3. Offer additional training in policy research skills to CSOs in DG and EG sectors. Although overt advocacy and coalition building are not advisable (or possible) in the current environment, working on research skills could be done in ways that will build the CSOs’ capacity for the short-term coping with the environment as well as more active advocacy in the longer term, when one hopes the environment will become more open again. Policy research and collaborative forms of advocacy in economic growth may not be as sensitive as in the democracy and governance sector. 2.4. Open BLCD capacity development interventions to a broad range of CSOs across the political spectrum. This would help to reduce perceptions of exclusivity and secrecy about USAID’s programming. See the last recommendation for more explanation. 2.5, Make specific linkages between OCA subarea criteria and targets that encourage more strategy development and related learning in Component 2. Work with the centers to improve strategic programming approaches and the centers’ ability to offer relevant and targeted services to users. This could mean that the parallel provision of BLCD supported trainings should be developed in a way where the centers have more responsibility over content and selection of trainings; namely they go from being hosts to partners in the training process in order to build their experience in managing and designing training planning processes beyond serving logistical and venue needs. M&E capacity should go beyond surveys at the end of trainings and consider how the centers can incorporate reflective learning into their strategy development. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 19 2.6 Support the Resource Centers reach out to smaller and less capable CSOs. BLCD might consider working with the centers to adapt some OCA components to create a tool/framework for organizational capacity development relevant for CSOs in these regions. BLCD could then assist the Centers to develop a plan for offering systematic capacity self-assessments and trainings in related areas for the CSOs. 2.7. Expand Component 3 services to reach more CSOs, especially in the regions. BLCD/MG Consulting expertise in this area is much needed by all CSOs to help them understand and comply with regulations and how the GOAJ is applying them. BLCD could disseminate its analysis by offering more sessions that include practical case-studies and ‘tips and tricks’ sharing among CSOs, especially in the regions. Recommendation 3: Senior USAID and USG officials reach out to GOAJ officials to create a more conducive partnership context for BLCD and other USAID projects. Several of the more senior Azerbaijani respondents asked us to pass on recommendations to USAID and the US government; presented here are the common themes that make sense to the team. These respondents encourage the US government to address the climate of misunderstanding and suspicion that they believe is a large part of the cause of the current restrictions on foreign donors and the CSOs they support. They argue that there is space for the US government to reach out to GOAJ counterparts, initiate constructive and cordial dialogue, and share information, especially about civil society programming. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 20 ANNEX 1 TABLES AND FIGURES TABLE 2: OCA SCORE CHANGES BY CAPACITY AREAS Key Capacity Areas* Gov. Admin. HRM Fin. Man. Org. Man. Prg. Man. PPM Total Score Total Avg. Score change Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. D&G 1 TIA Before 14 2.80 9 1.80 21 2.10 11 1.83 14 1.75 11 1.83 7 1.17 87 1.90 moderate basic basic basic basic basic low basic After 18 3.60 10 2.00 22 2.20 12 2.00 18 2.25 16 2.67 10 1.67 106 2.34 strong basic basic basic basic moderate basic basic Difference 4 0.80 1 0.20 1 0.10 1 0.17 4 0.50 5 0.83 3 0.50 19 0.44 2 WARD Before 11 2.20 5 1.00 13 1.30 9 1.50 22 2.75 15 2.50 7 1.17 82 1.77 basic low low basic moderate moderate low basic After 19 3.80 6 1.20 16 1.60 11 1.83 24 3.00 17 2.83 12 2.00 105 2.32 strong low basic basic moderate moderate basic basic Difference 8 1.60 1 0.20 3 0.30 2 0.33 2 0.25 2 0.33 5 0.83 23 0.55 3 MRI Before 11 2.20 5 1.00 10 1.00 9 1.50 13 1.63 11 1.83 7 1.17 66 1.48 basic low low basic basic basic low low After 14 2.80 6 1.20 12 1.20 9 1.50 15 1.88 13 2.17 8 1.33 77 1.73 moderate low low basic basic basic low basic ↑ Difference 3 0.60 1 0.20 2 0.20 0 0.00 2 0.25 2 0.33 1 0.17 11 0.25 4 DL Before 11 2.20 5 1.00 13 1.30 10 1.67 16 2.00 10 1.67 7 1.17 72 1.57 basic low low basic basic basic low basic After 16 3.20 6 1.20 15 1.50 10 1.67 16 2.00 10 1.67 8 1.33 81 1.80 moderate low basic basic basic basic low basic Difference 5 1.00 1 0.20 2 0.20 0 0.00 0 0.00 0 0.00 1 0.17 9 0.22 5 L&D Before 13 2.60 6 1.20 12 1.20 10 1.67 17 2.13 12 2.00 8 1.33 78 1.73 moderate low low basic basic basic low basic After 18 3.60 7 1.40 18 1.80 13 2.17 17 2.13 14 2.33 9 1.50 96 2.13 strong low basic basic basic basic basic basic Difference 5 1.00 1 0.20 6 0.60 3 0.50 0 0.00 2 0.33 1 0.17 18 0.40 Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 21 Key Capacity Areas* Gov. Admin. HRM Fin. Man. Org. Man. Prg. Man. PPM Total Score Total Avg. Score change Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. Score Avg. 6 AHD Before 10 2.00 7 1.40 15 1.50 8 1.33 12 1.50 9 1.50 7 1.17 68 1.49 basic low basic low basic basic low low After 14 2.80 12 2.40 21 2.10 9 1.50 15 1.88 11 1.83 8 1.33 90 1.98 moderate basic basic basic basic basic low basic ↑ Difference 4 0.80 5 1.00 6 0.60 1 0.17 3 0.38 2 0.33 1 0.17 22 0.49 7 AHCA Before 11 2.20 6 1.20 14 1.40 10 1.67 13 1.63 9 1.50 7 1.17 70 1.54 basic low low basic basic basic low basic After 15 3.00 10 2.00 23 2.30 13 2.17 18 2.25 10 1.67 9 1.50 98 2.13 moderate basic basic basic basic basic basic basic Difference 4 0.80 4 0.80 9 0.90 3 0.50 5 0.63 1 0.17 2 0.33 28 0.59 8 AZMA Before 11 2.20 5 1.00 10 1.00 6 1.00 9 1.13 7 1.17 6 1.00 54 1.21 basic low low low low low low low After 12 2.40 7 1.40 11 1.10 6 1.00 10 1.25 7 1.17 6 1.00 59 1.36 basic low low low low low low low Difference 1 0.20 2 0.40 1 0.10 0 0.00 1 0.13 0 0.00 0 0.00 5 0.12 EG 9 EDF Before 11 2.20 9 1.80 13 1.30 12 2.00 13 1.63 10 1.67 8 1.33 76 1.70 basic basic low basic basic basic low basic After 16 3.20 14 2.80 18 1.80 14 2.33 16 2.00 14 2.33 9 1.50 101 2.28 moderate moderate basic basic basic basic basic basic Difference 5 1.00 5 1.00 5 0.50 2 0.33 3 0.38 4 0.67 1 0.17 25 0.58 Average Increase in Scores 18 0.40 * Key capacity areas: Governance, Administration, Human Resource Management, Financial Management, Organizational Management, Program Management, and Project Performance Management NOTE: Arrows show change in category HEALTH Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 22 Table 3: Completed Milestones № Name of Organization Sector Completed Milestones by 30/09/2014 % Completed Milestones by 30/09/2014 Delayed Milestones by 30/09/2014 % Delayed Milestones by 30/09/2014 1 TIA D&G 3/7 43% 0/7 0% 2 WARD 2/7 29% 2/7 29% 3 MRI 2/7 29% 2/7 29% 4 DL 3/7 43% 1/7 14% 5 L&D 2/7 29% 1/7 14% 6 AHD HEALTH 4/7 57% 0/7 0% 7 AHCA 4/6 67% 0/7 0% 8 AZMA 1/7 14% 2/7 29% 9 EDF EG 3/7 43% 1/7 14% Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 23 Table 4: OCA Score Change and Milestones Name of Organization 1 2 3 4 5 6 7 8 9 Average TIA WARD MRI DL L&D AHD AHCA AZMA EDF Sector D&G HEALTH EG Baseline OCA Score 87 82 66 72 78 68 70 54 76 73 Mid-term OCA Score 106 105 77 81 96 90 98 59 101 90 Total Score Change 19 23 11 9 18 22 28 5 25 18 Value of OCA Areas in Total Score Change Governance 21% 35% 27% 56% 28% 18% 14% 20% 20% 27% Administration 5% 4% 9% 11% 6% 23% 14% 40% 20% 15% HR Management 5% 13% 18% 22% 33% 27% 32% 20% 20% 21% Financial Management 5% 9% 0% 0% 17% 5% 11% 0% 8% 6% Organizational Management 21% 9% 18% 0% 0% 14% 18% 20% 12% 12% Program Management 26% 9% 18% 0% 11% 9% 4% 0% 16% 10% Project Performance Management 16% 22% 9% 11% 6% 5% 7% 0% 4% 9% OCA Improvements by Milestones Completed 32% 39% 36% 67% 33% 68% 71% 40% 60% OCA Improvements by Milestones Not Completed 68% 61% 64% 33% 67% 32% 29% 60% 40% NOTE 1: Highlighted cells show the OCA areas, which constitute completed milestones. NOTE 2: Red color shows the OCA areas with the highest contribution towards the total score change in the mid-term OCA assessment. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 24 Table 6: Sustainability Score Change Key Capacity Areas Gov. Fin. Man. Prg. Man. Total Score Total Avg. Score Change Score Avg. Score Avg. Score Avg. 1 TIA D&G Before 14 2.80 11 1.83 11 1.83 36 2.16 After 18 3.60 12 2.00 16 2.67 46 2.76 Difference 4 0.80 1 0.17 5 0.83 10 0.60 2 WARD Before 11 2.20 9 1.50 15 2.50 35 2.07 After 19 3.80 11 1.83 17 2.83 47 2.82 Difference 8 1.60 2 0.33 2 0.33 12 0.76 3 MRI Before 11 2.20 9 1.50 11 1.83 31 1.84 After 14 2.80 9 1.50 13 2.17 36 2.16 Difference 3 0.60 0 0.00 2 0.33 5 0.31 4 DL Before 11 2.20 10 1.67 10 1.67 31 1.84 After 16 3.20 10 1.67 10 1.67 36 2.18 Difference 5 1.00 0 0.00 0 0.00 5 0.33 5 L&D Before 13 2.60 10 1.67 12 2.00 35 2.09 After 18 3.60 13 2.17 14 2.33 45 2.70 Difference 5 1.00 3 0.50 2 0.33 10 0.61 6 AHD HEALTH Before 10 2.00 8 1.33 9 1.50 27 1.61 After 14 2.80 9 1.50 11 1.83 34 2.04 Difference 4 0.80 1 0.17 2 0.33 7 0.43 7 AHCA Before 11 2.20 10 1.67 9 1.50 30 1.79 After 15 3.00 13 2.17 10 1.67 38 2.28 Difference 4 0.80 3 0.50 1 0.17 8 0.49 8 AZMA Before 11 2.20 6 1.00 7 1.17 24 1.46 After 12 2.40 6 1.00 7 1.17 25 1.52 Difference 1 0.20 0 0.00 0 0.00 1 0.07 9 EDF EG Before 11 2.20 12 2.00 10 1.67 33 1.96 After 16 3.20 14 2.33 14 2.33 44 2.62 Difference 5 1.00 2 0.33 4 0.67 11 0.67 Average Increase in Scores 8 0.47 NOTE: "New Opportunity Development for Sustainability" showed increase by 1 in the score for only one organization￾AHCA. All other organizations show no progress Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 25 ANNEX 2: SUMMARY OF LEGISLATIVE CHANGES IN 2013 AND 2014 “Overview of the changes to NGO legislation adopted on December 17, 2013 by the Parliament of the Republic of Azerbaijan”, prepared by Mahammad Guluzade and Natalia Bourjaily on February 19, 2014, lists the main changes introduced in December 2013: 1. Individual recipients of grants are now required to register grants with the Ministry of Justice (MoJ) in the same way as organizations. 2. Subgrants and other forms of assistance for the grant purposes shall be registered with MoJ the through the same process as original grants. In addition, NGOs shall register with the MoJ amendments to a grant agreement, including changes to its period, purpose, or amount. 3. NGOs are now required to inform the MoJ about changes to their factual address or changes to the number of their members. 4. Branches and representations of foreign NGOs (FNGOs) must inform the MoJ about term of the contract of their chief of party as well as his/her deputy, information about the deputy chief of party’s name, surname, citizenship, and place of residence. 5. Branches and representations of NGOs and FNGOs are required to inform the MoJ about the composition of the highest governing body and the term of service of members of the highest governing body. 6. NGOs and branches and representations of FNGOs are also required to inform the MoJ whether they spent entity’s property for statutory purposes after its dissolution. 7. Legal representatives of branches and representations of FNGOs operating in Azerbaijan need to have permanent residence in Azerbaijan and a document attesting to this fact must be submitted to the MoJ as part of the registration package. 8. All provisions of the NGO Law will apply to branches and representations of FNGOs (previously, only individual provisions specifically referencing branches and representations of FNGOs were applicable). 9. FNGOs can establish only one representation or branch in Azerbaijan. 10. The agreement that FNGOs must sign with the MoJ as part of their registration will now have an expiration date. 11. If the MoJ discovers that a NGO’s constituent documents are not in compliance with legislation then it may demand that the NGO adjusts its constituent documents within 30 days to make them compliant. 12. If a FNGO (i.e. their headquarters) merges with or joins another FNGO or splits into several organizations, then its branch or representation in Azerbaijan is terminated. 13. The provisions in the NGO Law regarding donations now apply to branches and representations of FNGOs. 14. NGOs now have a period of 30 days to rectify the deficiencies identified by the relevant state body. 15. The new Law sets additional grounds for suspension of a NGO’s activity: o If it creates obstacles for elimination of emergency situations; o If the NGO was penalized for failure to rectify the deficiencies identified by the ministry and did not rectify them; and o If the NGO violates the rights of its members. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 26 On 17 October 2014, the Parliament of Azerbaijan Republic adopted new amendments to the Laws on NGOs and Grants. According to new amendments: 1. Donors shall obtain a permission to give out grants to local NGOs; 2. Donations shall also be registered; 3. Local NGOs shall sign a service contract to provide such services as sales of goods, provision of services, and fulfillment of works. NGOs shall submit a service contract for registration to the relevant executive authority body when such services are provided with foreign funding. According to new draft amendments, local NGO, providing services without signing a contract or without registering it, will be held responsible under the Code of Administrative Offences. According to the new amendments relevant executive authority shall define relevant rules for implementation of the new requirement on registration of donors. The President signed the new amendments on November 15, 2014 and they came into force from November 16, 2014. The Cabinet of Ministers is tasked with developing procedural details of the new amendments. This process is closed to NGOs. It is expected that the earliest any information will be forthcoming is mid-January 2015. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 27 ANNEX 3: THE EVALUATION STATEMENT OF WORK SECTION C - DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION FOR THE BUILDING LOCAL CAPACITY FOR DEVELOPMENT (BLCD) PROJECT I. GENERAL This Statement of Work (SOW) is to provide USAID/Azerbaijan with a mid-term performance evaluation of the following Project: Project Title: Building Local Capacity for Development (BLCD) Contract Number: AID-112-C-12-00001 Period of Performance: September 28, 2012 - September 27, 2017 Total Estimated Cost: $5,899,823.00 Contractor: Chemonics International Inc. A. PURPOSE The purpose of the mid-term performance evaluation of the Building Local Capacity for Development (BLCD) Project is to: 1. Determine the results of BLCD Project interventions so far; 2. Evaluate the progress made by BLCD Project under each component based on established targets; and 3. Provide recommendations on any adjustments and considerations for the implementation of the rest of the project. 1. The evaluation must cover the BLCD) Project implementation period of September 28, 2012 - September 28, 2014. The primary audience for the BLCD Project mid-term performance evaluation is the USAID/Azerbaijan Mission. Other audiences include USAID/Washington (Bureau for Europe and Eurasia/Office of Democracy, Governance and Social Transition (EE/DGST), Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA), and Bureau for Policy, Planning, and Learning), Chemonics International Inc., other USAID implementing partners and beneficiaries of BLCD Project activities, such as civil society organizations (CSOs). The Mission may share the final evaluation report with other donors, other implementing partners, host Government counterparts and other United States Government (USG) agencies operating in Azerbaijan. The Mission will use the evaluation findings to guide BLCD Project’s further implementation. B. BACKGROUND The ultimate objective of the Building Local Capacity for Development (BLCD) Project is to increase the organizational and technical capacity of civil society organizations (CSOs) to enable them to develop into mature and effective organizations able to fulfill their civil society public interest purposes and agendas so they may serve as relevant, effective and sustainable partners. BLCD Project aims to strengthen the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas; develop highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas; and improve legal and regulatory frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 28 This evaluation will include specific aspects of three components of BLCD Project: Component 1 which relates to strengthening the organizational and technical capacities of key CSOs engaged in national, regional and community public interest agendas; Component 2 regarding developing highly professional CSO Resource Center(s), accessible to smaller and less capable CSOs working on public interest agendas; and Component 3 which addresses improving legal and regulatory frameworks to safeguard CSO enabling environments, sustainability and independent public interest agenda roles. The lack of meaningful competition is the most important challenge facing Azerbaijan’s democratic development. Azerbaijan is characterized by a strong executive branch in which most power is consolidated in the Presidency, which has placed significant limitations on CSOs, the media and freedom of association and assembly. Forces supporting meaningful reform in the country have been weakened or co-opted. According to 2013 USAID NGO Sustainability Index, NGO organizational capacity tends to be weak, with few NGOs engaged in strategic planning or having clear reporting mechanisms that promote accountability to donors and partners. While there are a number of capable, committed, and independent CSOs operating within the country who are willing to push the “democracy envelope,” they are not strong enough and under-resourced. The Mission funds supporting civil society development in the country steadily decline over the past several years. However, this particular area is of great importance to USAID worldwide, including Europe and Eurasia countries. It became one of the main pillars of the Administrator’s “USAID Forward” initiative endorsed in early 2011. The initiative set forth the strategy to promote sustainable development through high-impact partnerships and local solutions. In order to achieve long-term sustainable development, we have to support local institutions, private sector partners and civil society organizations that serve as engines of growth and progress for their own nations. The strategy played very well into the USAID/Azerbaijan Mission’s Country Development Cooperation Strategy (CDCS) for 2011-2016 where one of the three main Intermediate Results (IR 2.2 Civil society strengthened to increasingly engage in development and political processes) under the Development Objective 2 (Effective participation of diverse actors and institutions in the democratic development of Azerbaijan increased) is set to ensure a more participatory environment through strengthening non- governmental actors and organizations to have more influence with Government of Azerbaijan (GOAJ) and policy-makers on Azerbaijan’s development. Luckily, there is still space for civil society activity in Azerbaijan and USAID’s objective was to expand that space in meaningful ways. In such context and environment, in 2010 the Mission made decision to design Building Local Capacity for Development (BLCD) Project to increasing the breadth, depth and effectiveness of selected leading non-governmental actors and institutions while fostering dialogue and cooperation between Government and civil society. Through the BLCD Project USAID tries to strengthen the organizational and technical capacity of 30 selected non-governmental organizations and four (4) Resource Centers so that they can be more effective. Capacity development assistance is provided to those local implementing organizations and resource centers to address financial, organizational, technical, or performance measurement shortcomings to strengthen them to become relevant, effective and sustainable partners in Azerbaijan’s democratic development processes over the long run and be credible and capable partners to receive direct grants from other donors, including USAID. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 29 C. PROJECT’S RECENT ACHIEVEMENTS-PROGRESS TO-DATE: Under Component 1, the Contractor conducted initial Organizational Capacity Assessments (OCA) for each of the 20 (this might be different at the time of the evaluation) national CSOs selected by USAID. Using the results of the OCAs, baselines were determined and action plans and performance solution packages were developed for each of the partner CSOs. BLCD also started to support CSOs in the implementation of their action plans. The Contractor delivered three targeted training sessions for capacity development of the selected CSOs which covered areas like governance, financial and program management, human resources and other organizational areas critical to their effectiveness and sustainability. In addition, key CSOs were prepared to engage in productive dialogues with the Government, and to increase their capacity to formulate policy recommendations using research and evidence-based methodology. Under Component 2, the Contractor conducted an assessment and mapping of existing resource centers in the regions of Azerbaijan, which increased awareness on the regional operating environments and contributed to the development of a Work Plan to support the selected two (2) Resource Centers. Two grants were awarded to both resource centers to strengthen the organizational capacity of CSO resource centers and improve the range and quality of services they provide to local civil society organizations in their regions. Under Component 3, the Contractor provides high-quality and the most updated analysis of the legislative pieces concerning CSOs and strengthens CSOs’ capacity to defend their rights and advance legal reforms. The Contractor updated the Guide on Grant Registration both in English and Azerbaijani languages, incorporating the 2013 changes to the grant legislation. It also prepared a comprehensive analysis of possible implications of the changes to the activities and operations of local CSOs and foreign Non-Governmental Organizations (NGOs). D. PURPOSE OF THE EVALUATION AND ITS INTENDED USE An evaluation of the BLCD Project is needed at this time because the Award itself calls for a mid￾term evaluation of the project in its monitoring and evaluation plan. Further, the Office of Democracy and Governance and the wider USAID Mission want performance information from the BLCD Project evaluation to inform ongoing and related future programming. In particular, the Office of Democracy and Governance is interested to know: 1) which interventions/approaches have proven to be feasible/sustainable in the toughening political context; and 2) which interventions/approaches provide a sound foundation for future programming (continue, expand, modify, etc.). The information is needed in FY 2014 to ensure utilization in the design of related future projects/programs. The political dynamics surrounding the 2013 change in the NGO-related legislation will have a significant impact not just on the non-governmental institutions in Azerbaijan, but also on the ability of the BLCD Project to have an impact through working with those institutions. The Evaluation Team will need to formulate their findings in relation to those changes. Given that the answers to all evaluation questions are potentially affected by those changes, it will be important for the answers to indicate what was accomplished prior to and after the changes, including any important shift in dynamics. E. SUMMARY OF SPECIFIC TECHNICAL REQUIREMENTS Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 30 The Contractor shall ensure that the Evaluation Team will provide the following deliverables within the terms defined by the Award: 1. Draft evaluation design; 2. In-brief with USAID management to present the detailed evaluation design and Work Plan; 3. Conduct evaluation of BLCD Project on the ground in accordance with the USAID-approved evaluation design; 4. Out-brief with USAID management to present initial findings, conclusions and recommendations; and 5. Provide draft and final evaluation reports to USAID in accordance with the reporting guidelines. II. EVALUATION QUESTIONS AND METHODOLOGY The finalized evaluation design must be submitted to the designated Contracting Officer’s Representative (COR) five work days prior to the team’s arrival in-country. The evaluation design must outline in detail what methods the Contractor will use to get answers for each evaluation question. The evaluation design must include a detailed evaluation matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, a Work Plan, and a dissemination plan. This information together with the Mission’s comments will be discussed in detail during the in-brief meeting with USAID. The Work Plan must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the Evaluation Team. In order to evaluate the questions below related to BLCD Project performance, the Contractor must review pertinent background documents relating to BLCD Project and the broader civil society context in Azerbaijan, as well as conduct interviews in Washington, D.C. and field work in Azerbaijan for three weeks. Among other methodologies, the Contractor must collect information from primary sources such as Azerbaijani counterparts, the beneficiaries of the assistance; CSOs involved with BLCD Project; and other donors and implementing partners. For the purposes of answering the evaluation questions below, “institutionalized” means not only establishment of some mechanism, but regular utilization of the mechanism by the targeted institution without project support, or the institution’s expression of a strong commitment to do so. The Contractor must utilize these questions in the evaluation: Strengthen the organizational and technical capacities of key CSOs: How effective are BLCD Project capacity development interventions with partner CSOs? Which interventions in local capacity development are most effective and why? How far along the road to sustainability are BLCD Project partner CSOs? In what areas do they need most of the support? Develop highly professional CSO Resource Center(s). How effective was BLCD Project selection process for CSO resource centers? What is the likelihood for sustainability of the selected resource centers and what support do they need to increase that likelihood? Improve legal and regulatory environment. How effective was BLCD Project in raising awareness among key stakeholders on deficiencies in current legislation affecting civil society? Considering the changing political conditions, is there something the BLCD Project could do differently in the future to achieve additional results under this component? Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 31 III. TASKS A. Review of Key Documents: The Contractor must review key documents to develop a Work Plan prior to any field work. All available documentation describing BLCD Project activities carried out in Azerbaijan must be reviewed. Documents for review include but are not limited to those listed in the reference section. The Contractor must contact the designated Task Order Contracting Officer’s Representative (COR) for BLCD Project evaluation for access to relevant documents. B. In-briefing: Upon arrival in Azerbaijan, the Evaluation Team must provide an entrance briefing to the designated USAID officials, introduce the team, discuss logistics and scheduling, discuss submission of the Work Plan, and any other issues. USAID will assist with identification of the relevant stakeholders to meet and provide additional suggestions for interviews. C. Work Plan: The Work Plan must be in accordance with the USAID prepared timeline for all work to be concluded and the dates for submission of the draft and final reports. The Work Plan must include the following elements: i. Schedule of contacts and site visits (regions, beneficiaries and collaborators); ii. Arrangements for local logistics; iii. Schedule of briefings and submission of deliverables; and iv. Delineate the roles and responsibilities of the other members of the Evaluation Team to ensure coverage of all elements of the Statement of Work. USAID will provide comments within two days. D. Evaluation Design: The Contractor must prepare and submit to USAID for approval a final evaluation plan and schedule with the following major elements: Schedule, methodology for conducting the evaluation (data and information collection, field interviews, participant surveys), beneficiary groups to be contacted and regions to be visited. The Contractor must ensure that its findings and conclusions about the effectiveness of the BLCD Project activities are based on available data that is both accurate and reliable, and that information gathered is representative of and reasonably reflects results actually achieved. The Contractor must submit a final detailed evaluation design, which must consist of the following: i. List of topics and relevant questions, methods and data sources for data gathering; ii. A matrix of regions and beneficiaries to be contacted; iii. Data analyses for each question and presentation plan; iv. Data collection instruments; and v. Limitations of the evaluation design, if any. The Evaluation Team must share the evaluation design with the Implementing Contractor for comment, but, in the interest of objectivity and independence, the Implementing Contractor will not participate in the design, implementation, analysis, or presentation of the evaluation. E. Field Work: The Contractor must begin field work after finalization of the Work Plan and Evaluation Design and its approval by USAID. F. In-country USAID Debrief: The Contractor must provide an oral debriefing to USAID upon completion of the evaluation and prior to departing from Azerbaijan. Evaluation findings must include facts, evidence and data. Recommendations must be specific, concise and supported by evidence. Recommendations must be action-oriented and implementable. G. Evaluation Report: i. The Contractor must provide with a detailed Outline of the Evaluation Report, main findings and recommendations to USAID prior to departing Azerbaijan. ii. The Contractor must submit a draft report of its findings within seven (7) working days after departing Azerbaijan for review and comments. The first draft must address comments and recommendations made by USAID and stakeholders during the out-briefing. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 32 iii. The Contractor must submit the final evaluation report to USAID five days after receipt of comments. The length of the final report is not predetermined, but the report must be concise, well written, and comprehensive. Recommendations must be action-oriented, practical, and specific; define responsibilities and timelines for the action; and identify milestones and deliverables. Unresolved issues that highlight what remains to be done must also be included in the final report. The final report format must be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1” top/bottom and left/right. The body of the report must not exceed 20 pages, excluding the executive summary, table of contents, references and annexes. The final report must follow USAID branding and marking requirements. The final report must include an executive summary, introduction, the development context and the background of the project being evaluated, evaluation questions, explanation of evaluation methodology, the limitations of the evaluation, findings, conclusions and lessons learned so far, and recommendations on the directions and adjustments (if any is needed) for successful implementation of BLCD Project activities. The executive summary must summarize the purpose, background of the project being evaluated, evaluation questions, evaluation methodology, major findings, lessons learned, conclusions, and recommendations. The evaluation methodology must be explained in the report in detail. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annex to the report must include: 1. The Evaluation Statement of Work; 2. Schedule of Evaluation; 3. Evaluation design/methodology employed questionnaire and list of questions by topic etc.; 4. Names and contact information of key respondents, sites visited and other sources of Sources of information, properly identified and listed; 5. Information statements as appropriate regarding significant unresolved issues, difference of opinions (among members of the Evaluation Team, the implementing Contractor, beneficiary CSOs, Government counterpart(s) and other relevant stakeholders) and availability of data and its quality; and 6. The Evaluation Design. Disclosure of conflicts of interest forms of all evaluation team members, either attesting to a lack of conflict of interest or describing any existing conflicts of interest. V. REPORTING RELATIONSHIP USAID will provide overall direction to the Evaluation Team and identify key documents. The designated COR will be the primary contact for the Award. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 33 ANNEX 4: SCHEDULE OF EVALUATION Week Dates (dd/mm) Task Deliverable 1 30/09-02/10 Review relevant materials 1 06/10 Prepare draft evaluation design, work plan, preliminary schedule and dissemination plan; submit to USAID for review prior to departure X 1 01/10-07/10 Team planning meeting; conduct interviews in USAID/Washington 1 06/10 until complete Begin setting up interview dates and logistical arrangements 1 08/10-09/10 U.S. team members fly to Azerbaijan 2 10/10 USAID in-briefing; discuss USAID comments from review 2 14/10 Submit final work plan and schedule within three days of arrival X 2 16/10 Submit evaluation design within five days of arrival X 2 20/10 Submit final evaluation design, work plan within one day of receiving USAID comments X 2 20/10 Complete logistical arrangements for hotels and travel in regions 2 16/10-20/10 Translate data collection instruments, field test, revise as necessary 2-4 13/10-25/10 Conduct fieldwork, subject to USAID approval, and analyze data 4 28/10 Submit outline of final report, following USAID Evaluation Policy X 4 28/10 Out-briefing and presentation to USAID/Azerbaijan 4 31/10 U.S. team members depart 5-6 14/11 Further analyze data, complete first draft based on USAID criteria for quality evaluation reports; submit to USAID within 14 days of departure X 7 No later than 07/12 Submit final report to USAID within five days receipt of USAID comments X 8 14/12 Submit approved final evaluation report to DEC X Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 34 ANNEX 5: EVALUATION DESIGN, METHODOLOGY AND QUESTIONNAIRES Final evaluation design and methodology, approved October 10, 2014 Management Systems International (MSI) will conduct a mid-term performance evaluation that meets the high-quality criteria and standards of USAID’s Evaluation Policy, and answers USAID/Azerbaijan’s questions using evidence-based findings to develop a strong linkage between conclusions and recommendations. The evaluation report and debriefing will be structured to maximize learning about effective civil society organization (CSO) capacity development and how to improve it given the political context and recent changes in the NGO laws and regulations. This finalized draft is revised and expanded from the proposed evaluation design and work plan submitted by MSI in June 2014. Section A includes the evaluation design and Section B includes the work plan. Our preliminary review of BLCD documentation indicates that most of the CSO partners are based in Baku, so the team has deleted the issue of regional difference as a possible variable in effectiveness of capacity building, and emphasized instead size, age, sector and starting level of capacity. For the work plan, the regions to be visited will include the cities where the two CSO Resource Centers are located (Shemkir and Gabala), and possibly Sheki to visit the Uluchay Socio-Economic Innovation Center for the CSO partner survey. A. EVALUATION DESIGN 1. Overall Design MSI proposes a mixed-method design for this mid-term performance evaluation to support analysis of the Building Local Capacity for Development (BLCD) Project’s achievements, what expected results are occurring and how the project is valued at this point of implementation. Methods will include a short survey of assisted CSO partners, key informant interviews, documentation reviews and review and analysis of baseline, target and performance data. To strengthen the evidence for findings and their validity, the MSI team will collect data from multiple sources that can be used to answer each evaluation question and its sub-questions. The data collected from each source for a given question will be analyzed separately, then considered with the data analysis and findings from other sources used to answer the question to provide a more comprehensive understanding of findings. The team will also try to determine any convergence of findings from each data source. Comparing baseline and mid-term performance monitoring data, the team will determine progress to date. For analytical purposes, the design will also reference the BLCD logical framework, examining the relationship between the implementation of inputs to outputs and outcomes. 2. Evaluation questions, methods and data sources Overview of Data Collection and Analysis Methods The MSI team proposes several methods of data collection, including: a) short survey of CSO partners that have received capacity development interventions -(quantitative and qualitative questions); b) a review of targeted CSO partner policies, procedures, processes and related Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 35 documentation; c) semi-structured interviews with staff at CSO resource centers; d) a review of resource center procedures, processes and related documentation; e) a short survey of local CSOs that use or have used CSO resource centers; f) a review of the BLCD M&E plan, logframe, indicators, targets and performance data for before-and-after comparisons; g) key informant interviews with USAID/Azerbaijan, selected donor officials, selected Government of Azerbaijan (GOAJ) officials, BLCD staff and other key stakeholders identified by USAID; h) reviews of: BLCD work plans, quarterly and annual reports, capacity-building materials and methods; i) organizational capacity assessments (OCAs) per CSO, CSO action plans and performance solutions; and j) review of BLCD analysis of changes in legislation and implications for CSOs, materials and methods for raising awareness recent changes and activities to strengthen CSOs to defend their rights and advance legal reforms. Surveys of CSO partners and of local CSOs will collect sex-disaggregated data on the leader/director of each CSO to determine if there are important gender variables related to findings and analysis for the evaluation questions. Component One Questions, Methods and Data Sources Q.1. How effective are BLCD Project capacity development interventions with partner CSOs? Sub-questions:1) To what extent are capacity development interventions meeting the objectives established by the BLCD project? 2) Do changes in capacity vary by CSO characteristics such as size, age, sector or level of capacity according to the OCA baseline assessment? 3) How are changes in NGO legislation affecting CSO capacity development and their ability to carry out activities? 4) How close are CSO partners not currently USAID direct grantees to meeting USAID criteria for grant eligibility? The primary method the MSI team will use to answer this question is assessment of OCA baseline data on performance gaps for each CSO partner, the performance solutions selected to close identified gaps, and subsequent capacity measurements taken following BLCD interventions to increase CSO capacity. The team will use performance monitoring data from the BLCD M&E plan to compare each identified performance gap’s baseline capacity data to its mid-term performance and then to its targets. To explain variation in performance, the MSI team will examine the BLCD and OCA documentation for each CSO to understand the specific CSO characteristics and any other documented fundamental causes of identified performance gaps. The team will interview BLCD staff and consultants to understand their technical approach to CSO capacity development and their own views of effectiveness to date. We will also ask them whether changes in NGO laws have been a factor in the effectiveness of CSO partner capacity development and if so, to explain how and describe what changes are being implemented to increase CSO capacity in the current political context. Other methods to gain information about the effect of the legal and political context on CSO capacity development will include the CSO Partner Survey and interviews with key informants and stakeholders familiar with the CSO community and these CSOs in particular. If USAID is interested, the team will also assess the effectiveness of capacity development interventions by asking: “How close are the CSO partners not currently direct grantees to meeting USAID’s eligibility criteria to receive direct grants from USAID and from other donors?” To answer this, the MSI team will compare capacity scores of CSO partners with USAID eligibility criteria to determine how far CSO partners have to go before they are eligible. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 36 Q.2.Which interventions in local capacity development are most effective and why? Sub-questions: 1) What were the different modes of intervention to build capacity?2) What were the expected effects of each type of intervention? Did they occur? Why or why not? 3) Does effectiveness vary by type of intervention, type of capacity being developed, or other CSO characteristics noted in Question 1 above, e.g. size, age, sector, level of capacity, etc.? 4) Were the interventions customized based on CSO partner OCA baseline findings? OCA typically provides an effective means of assessing CSO capability, and is the first step in all capacity development interventions. USAID’s human and institutional capacity development (HICD) model provides a systematic process for identifying performance problems and developing action plans and performance solution packages. Effectiveness in building organizational capacity based on these assessments and plans ultimately depends on the types of interventions and how they are implemented. The team will examine the BLCD OCA tool and process, the types or mixes of technical assistance interventions and how they were implemented to develop capacity for each skill area. The team will use this information to create a matrix that aligns intervention methods according to capacity area. Then, to answer the question on which interventions for capacity development were most effective, the team will isolate capacity development interventions used in each skill area. For each CSO partner that has undergone BLCD capacity development, the MSI team will compare their baseline to targets and then to actual performance data to determine the extent to which its capacity has progressed in each skill area related to that topic. The team will construct a range of capacity improvement scores based on scores for all CSO partners, as well as the median and average scores for each skill area to determine which interventions in each topic (governance, financial management, advocacy, etc.) have been most effective. The team will then review and compare CSO scores by skill area to determine which intervention or mix of interventions yield higher capacity scores. As an additional data point for answering this question, the CSO partner survey will include questions on CSO opinions about the level of effectiveness of the set of interventions used in each skill area, which interventions they think were most effective, and why. The team will also interview BLCD staff with the same questions to obtain their perspective, as well as learn whether customized training was provided for certain subsets of CSO partners based on OCA findings and their related action plans and performance solution packages. Answering “why” is more complicated, and accordingly will be approached through the use of a short CSO partner survey (see Annex 2 for draft questionnaire) that will include questions such as which interventions they found most and least effective in each skill area, why, and what was useful about those interventions. Is more training or some other type of intervention needed to gain the skill? The team will also ask for CSO documentation, such as policy, procedures and processes, as evidence that they are improving on those skills or topic areas. Q.3 How far along the road to sustainability are BLCD Project partner CSOs?In what areas do they need the most support?Sub-questions:1) Have CSOs implemented policies, procedures, actions — especially in the areas of governance, program/project management, fundraising, financial management and constituency building — that would increase their sustainability? 2) Does sustainability of CSOs vary by region or be sector? 3) What is the potential effect of changes to NGO laws on CSO sustainability and what can be done to promote sustainability within that context? The answer to this question depends on the definition of sustainability used by USAID/Azerbaijan and BLCD and what sustainability measures are being used. MSI assumes that BLCD is applying a Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 37 sustainability measurement system to monitor CSO partner progress. If this assumption is wrong, the team will devise a basic sustainability index based on commonly agreed-upon elements such as fundraising, financial management, project management, constituency building, strategic planning, etc. The team will apply this index, using BLCD’s scoring for capacity development in the key areas they have implemented. The team will compare baseline data for each sustainability element measured against targets and actual performance data to judge how close CSO partners are to sustainability. This will allow the team to identify the areas with the most progress and those that have progressed less than anticipated; this will help the team answer “in what areas do they need most support?” Finally, the team will refer to BLCD’s analysis on the potential implications of CSO￾related legislative changes in interviewing BLCD staff on how this could affect partner sustainability, and what they are doing to increase sustainability in the current political environment. Component Two Questions, Methods and Data Sources Q.1. How effective was the BLCD Project selection process for CSO resource centers? Sub-questions: 1) What were the key considerations BLCD used for selecting CSO resource centers? 2) Does use of resource centers by smaller or less capacitated CSOs meet, exceed or fall below expectations? 3) Do the centers have the capacity required to serve as resource centers based on initial BLCD assessments? 4) What factors make the resource centers more accessible? The first step is to review project documentation of BLCD’s selection criteria and process, which the team will supplement with staff interviews. Considerations may have included factors such as a geographic location that provides access to numerous CSOs; current organizational capacity to provide services; experience of staff; and experience in providing CSO services. The MSI team will use several proxies for determining the effectiveness of the selection process. First, the team will assume that the more local CSOs used a resource center, the more effective the selection process was. The team will review data on center use from the BLCD performance monitoring database and the data reported by BLCD in its quarterly reports. The team will also visit each CSO resource center and examine its documented statistics on use to look for convergence with BLCD’s data. A second factor is the type and range of services a resource center is able to provide; a third is local CSO satisfaction with the resource center’s accessibility and services provided. The team will review the SIGMA generated satisfaction survey results for the two resource centers after which the team will collect data from CSO resource centers on services provided and staff qualifications to provide these services. Finally, the team will conduct brief, semi-structured interviews with a sample of local CSOs who are actual and potential users of the centers, drawn from those at the resource center during the site visit and others located in close range. The interview with users will include questions on satisfaction with the services provided by the resource centers and the quality of those services they have used. Interviews with non-users will attempt to learn whether they are aware of the centers’ services and, if so, why they have not used them. The analysis of these data will provide an additional data point on effectiveness that can be compared with data on use of the centers. Q.2.What is the likelihood for sustainability of the selected resource centers and what support do they need to increase that likelihood? Sub-questions: 1) What are the sources of revenue for each center? 2) Do local CSOs in the region use and value the resource center? 3) Do resource centers offer the range and type of services needed by local CSOs? To answer this question, the team will: 1) review BLCD performance data on sustainability measures used to track progress of the resource centers, and then compare the baseline to targets and current performance; 2) review resource center financial data on sources of current and projected revenues; Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 38 3)use the data the team collected under Component Two, Question 1 on resource center use by local CSOs; and 4) use data and analysis from the brief survey of local CSOs on resource center value and accessibility. The team assumes that data about center use, coupled with an assessment of revenue and revenue sources and the opinions of local CSO users, will provide valuable information to judge the likely sustainability of these centers. The analysis of these diverse data sources will also enable identification of the type of support needed to increase these resource centers’ likely sustainability after the close of the BLCD Project. Component Three Questions, Methods and Data Sources Q.1How effective was the BLCD Project in raising awareness among key stakeholders on deficiencies in current legislation affecting civil society? Sub-questions: 1) What has BLCD done to raise awareness in the capital and in the regions? 2) What is the response of stakeholders to information provided by BLCD? 3) Are there other venues or modalities that BLCD could use to further increase awareness that would help build a constituency for changes in the 2013 NGO legislation? The MSI team will first conduct a review of BLCD documentation and reports for a description of activities to increase awareness of deficiencies in legislation affecting CSOs. The team will augment this review with interviews of BLCD staff on the content of messages they used to raise awareness, the methods and venues for messaging, and how they targeted messages and information for different audiences. Next, the team will review any performance data collected on messaging to raise awareness and subsequent awareness changes among key stakeholders. The team will also conduct semi-structured interviews with selected policymakers and key opinion leaders, including those from the CSO community. Questions will pertain to respondents’ level of awareness of legislative deficiencies, their understanding of how this will impact the growth of the CSO community and its effectiveness, and their opinions on BLCD messaging and how it can better help build or support a constituency for legislative changes. Data from these sources will be analyzed separately to determine effectiveness from different perspectives. Variety in responses will be examined using factors such as type or category of respondent. This will help the team frame recommendations for tailored messaging for different audiences to build a constituency for CSO-supportive legislative changes. Q.2.Considering the changing political conditions, is there something the BLCD Project could do differently in the future to achieve additional results in this component? Sub-questions: What are the possible levers inside and outside government to promote a more positive and supportive political environment for CSOs? Has BLCD identified champions for CSOs within government that they can begin strategizing with to promote changes in the 2013 NGO legislation? Has BLCD identified prominent opinion leaders outside of government that are supportive of CSOs to work with? Has BLCD begun to provide support for CSOs to build alliances to advocate for their rights and for less restrictive legislation? Prior to arrival in the field, the MSI team will review BLCD’s analysis of current policies and legislation on NGOs and its analysis of the potential implications of current law on CSOs. In the field, the team will interview BLCD staff about their plans to support and promote changes in the current political environment for CSO activities and what they have done regarding the sub￾questions above. The team will also seek CSO champions inside and outside of government in the legal, civil society, academic and donor communities to obtain opinions and suggestions on how BLCD could promote positive changes in the political environment for CSOs. The team will Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 39 conduct a content analysis of interview responses and disaggregate the results of this analysis based on the affiliation or “community” of respondents. MSI’s team will use its strong expertise in civil society development in a range of countries with closing political contexts to build on the opinions and suggestions from respondents to recommend further actions. 3. Limitations of Evaluation Design A potential limitation of this evaluation design is its reliance on performance data collected by BLCD as a key data source for answering each question. At this time, the MSI team can only surmise what type of data BLCD collects for its sustainability measures (for example), or if the data collected have undergone data quality assessments. The principle means of overcoming this limitation will be to collect data from multiple sources for each evaluation question. Secondly, given that the first week requires redrafting the evaluation design, work plan, data collection instruments, selection of travel locations and schedules following discussions at the in-briefing, and then finalization and submission for approval following receipt of the mission’s comments, time will be scarce. The MSI team’s suggestion to overcome this potential problem is to work, with mission approval, toward finalizing the design, work plan and schedules so the team is able to begin field￾testing instruments and subsequent data collection during its first full work week in Baku. In consultation with USAID, the team will prepare a list of potential initial interviewees so that all or part of the team can begin the interviews even as the final design and work plan are awaiting final approval. Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 40 Questionnaire for CSO Partner Survey (Cohort 1) • Introduction of the evaluation team and purpose of the evaluation Name of Interviewee: Position: CSO Statistics: Name of Director, Male/Female Name of Organization, Location Sector Date of Establishment Size of budget: [-$50K, $51 - $150, $151+] Start date of BLCD support 1. What does your organization do? (Main purpose, constituencies, key activities) 2. Has your organization received capacity building support before? If yes, what kind? Are you currently receiving any other source of capacity building support? Explain. 3. The BLCD capacity building program started with the OCA tool and process. What was your experience with the BLCD use of this approach? How effective was the OCA in helping you to assess your organization’s capacities? 4. Considering the various types of capacity development support your organization has received from BLCD, which have been most useful? Why? Training workshop Advice and mentoring - staff Advice and mentoring - consultants Grant Which have been least useful? Why? Training workshop Advice and mentoring - staff Advice and mentoring - consultants Grant 5. Will you please give an example of how you have used the capacity development you have received? 6. Is there something you haven’t been able to use? 7. Is the capacity building support contributing to your sustainability? If so how Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 41 8. What current or proposed legislation or legislative changes will most affect your organization’s ability to fulfill its purpose and operate freely? 9. How do you get information on current legislation that concerns CSOs? What sources of information are most useful? 10. How do you use the information concerning the legislative and regulatory environment for civil society? Are you a part of any networks, formal or informal? 11. What else do you think BLCD could do to influence the legal environment for CSOs? 12. What has been the overall benefit of BLCD support? What advice would you give to BLCD to better assist your organization to develop its capacity? Questionnaire for CSO Partner Survey (Cohort 2) • Introduction of the evaluation team and purpose of the evaluation Name of Interviewee: Position: CSO Statistics: Name of Director, Male/Female; Name of Organization, Location; Sector; Date Established Size of budget: [-$50K, $51 - $150, $151+]; Start date with BLCD 1. What does your organization do? (Main purpose, constituencies, key activities) 2. Has your organization received capacity building support before BLCD? If yes, what kind? Are you currently receiving any other source of capacity building support? Explain. 3. The BLCD capacity building program started with the OCA tool and process. What was your experience of it? What was most useful? How could it be improved? 4. Have you made other organizational capacity changes since BLCD startedt? Why? How have they affected the organization? 5. Will the capacity building support contributing to your sustainability? If so how? 6. What legislation or legislative changes most affect your organization’s ability to fulfill its purpose and operate freely? 7. How do you get information on current legislation that concerns CSOs? What sources of information are most useful? Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 42 8. How do you use the information provided by BLCD concerning the legislative and regulatory environment for civil society? Are you a part of any coalitions, formal or informal? 9. What else do you think BLCD could do to influence the legal environment for CSOs? 10. What has been the overall benefit of BLCD support so far? Assuming BLCD goes forward, how could BLCD better assist your organization to develop its capacity? Questionnaire for CSO RESOURCE CENTERS 1. Tell about your resource center and its main services. 2. Could you give an overview of your cooperation with BLCD? 3. Why did you apply for the BLCD program? 4. Can you tell about your experience with the OCA process/action plan? What was useful? Less useful? 5. Who are your intended users? (types of orgs – young, date established, sectors (if applicable) geographic range, etc). And how did you expect this use to change? 6. User rates data? Do you have records/data that you could share with us? 7. Are you satisfied with current user levels? What were they? Have they changed from the start of the grant? 8. What are your challenges in reaching CSOs, particularly smaller ones? 9. And in meeting their needs? 10. What are your expectations for the center in the coming year? 11. How do you does the new legislation affect the use of your center? (role of ICNL) 12. Can you describe how you envision the sustainability of your center? What do you need to make it sustainable? (New services, sources of revenue) Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 43 Questionnaire for Local CSO RESOURCE Center Users CSO Statistics: Name of Organization, Location Central Purpose (Mission) Date of Establishment Number of staff/gender; staff positions Date when CSO began using CSO resource center Capacity development support from CSO resource center received to date Previous capacity development support 1. Tell me about your organization and your mission. 2. Have you used any of the services at the CSO resource center? 3. If yes, what services have you used? (ask if attend ICNL meeting) 4. Which of those services have been most useful to your organization? 5. Have you made any organizational changes or changes in practices since using the center? If so, please describe these changes. (Example, manner of external relations, other) 6. Which have been less useful? Are there types of services offered at the center that you haven’t used? 7. If so, what are these? What is the reason you haven’t used them? 8. Are there any other types of services you need to strengthen the capacity of your organization that are not available from the center? 9. How accessible is the resource center to you? Is it easily reached? Open when you need it? 10. Do you feel welcomed there? 11. Overall, what has been the value of the center to your organization? (Are there things you are able to do now as a result of the center?) 12. What advice would you give to the Center in its mission to better support local CSO development? Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 44 ANNEX 6. INTERVIEW RESPONDENTS Category Name, Position, Organization or Department USAID Washington • Faye Haselkorn, Civil Society Specialist, Europe and Eurasia Bureau • Larry Garber, Office of Policy, Planning and Learning BLCD Implementers • Jennifer Burdett, Chemonics Washington • Natalia Bourjaily, ICNL Washington • Morana Smodlaka, Chief of Party, Chemonics Azerbaijan • Bahar Arabova, Deputy Chief of Party, Chemonics Azerbaijan • Rena Abdullayeva, CSO Resource Centre Specialist, Chemonics Azerbaijan • Gunay Ismayilova, Capacity Development Specialist, Chemonics Azerbaijan • Mahammad Guluzade, Director, MG Consulting, Azerbaijan CSO Partners (Groups 1 and 2) • Shahla Ismayil, Director, Women’s Association for Rational Development • Elvin Yusifli, Project Director, Transparency International Azerbaijan • Hafiz Hasanov, Chairman, Law and Development • Sabit Baghirov, President, Entrepreneurship Development Foundation • Samir Kazimli, Acting Chairman, Election Monitoring and Democracy Studies Centre • Mahammad Mammadzade, Project Coordinator, Election Monitoring and Democracy Studies Centre • Mirali Huseynov, Chairman, Democracy Learning • Matlab Verdiyev, Chairman, Assistance to Health Care Development • Nurlan Aliyev, Head of the Public Union, Azerbaijan Health Communication Association • Fuad Dargahli, Head of Education, Azerbaijan Children’s Union • Alimammad Nuriyev, President, ‘Constitution’ Research Foundation • Galib Bayramov, Executive Director, Economic Research Centre • Azer Mehtiyev, Director, Support for Economic Initiatives Public Union • Mayis Safarov, Chairman, ‘Uluchay’ Socio-Economic Innovation Centre • Lyudmila Khalilova, Chairwoman, Women for Development of Municipalities • Solmaz Aliyeva, Director, Young Accountants Union • Afig Malikov, Chairman, “Education” Public Support Association of Youth of Azerbaijan Public Union Mid-Term Performance Evaluation For The BLCD Project: Evaluation Report 45 Category Name, Position, Organization or Department Civil Society Resource Centers and CSO Users Shemkir • Fuad Badalov, Director, Shemkir Regional NGO Resource and Training Social Union • Asaf Ahmadov, Director, ‘Demos’ Public Union • Irshad Abbasov, Chairman, ‘Eco-Renesans’ Public Union • Sevinc Sersan, Director, Ganja office of Transparency Azerbaijan • Razala Abbasova, Coordinator, United Aid for Azerbaijan’s Community￾Based Rehabilitation Centre in Civil Society Resource Centers and CSO Users Gabala • Anar Maharramov, Deputy Chairman, Gabala Regional Youth Resource Centre Public Union • Ilgar Ilyasli, Chairman, "Yaddash" Enlightener Youth Public Union • Mayis Safarov, Chairman, ‘Uluchay’ Socio-Economic Innovation Centre • Arzu Valibayova, Chairwoman, ‘Creative Youth’ PU • Javidan Hasanov, Chairman/Founder, ‘Towards Development’ Social Research Public Union • Daghistan Heydarov, Chairman, ‘Shans’ Support to Regions Public Union Government of Azerbaijan • Farasat Gurbanov, Executive Director, the Council of State Support to NGOs under the Auspices of the President of the Republic of Azerbaijan • Alimammad Nuriyev, Chairman of the Commission, the State Commission on Law and Monitoring under the Council of State Support to NGOs under the Auspices of the President of the Republic of Azerbaijan • Mahammad Guluzade, Member of the Commission, the State Commission on Law and Monitoring under the Council of State Support to NGOs under the Auspices of the President of the Republic of Azerbaijan Other US Implementers • Ilgar Aghasiyev, Chief of Party, COUNTERPART INTERNATIONAL • Timothy Madigan, Chief of Party, East West Management Institute Other international donors/embassies • Andrew Harvey, Second Secretary Political and HM Consul, British Embassy • Mariam Hajismayilova, Programme Manager, European Union Delegation to Azerbaijan Republic