|RUSSIA PERFORMANCE EVALUATION USAID/RUSSIA HUMAN RIGHTS PROJECT FINAL EVALUATION REPORT September 27, 2012 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared under the Russia Monitoring & Evaluation Project (RMEP) by International Business &Technical Consultants, Inc. (IBTCI).    USAID/Russia Human Rights Project   “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” FINAL EVALUATION REPORT SEPTEMBER 27, 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive   Suite 220   Vienna, VA 22182   USA Contracted under RAN‐I‐00‐09‐00016, Order No. AID‐118‐TO‐11‐00004 Russia Monitoring and Evaluation Project Evaluation team:   Alexey Kuzmin (Team Leader) Natalia Kosheleva Vladimir Balakirev Viacheslav Bakhmin DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report i Table of Contents Figures and Tables ........................................................................................................................................ ii Acronyms and Abbreviations ...................................................................................................................... iv EXECUTIVE SUMMARY .......................................................................................................................... 1 1. INTRODUCTION ................................................................................................................................ 5 1.1. Description of the Project Evaluated ................................................................................ 5 1.2. Purpose of the Evaluation ................................................................................................ 8 1.3. Evaluation Questions ........................................................................................................ 8 1.4. Collaboration with the local partners and stakeholders ................................................... 9 1.5. Methodology .................................................................................................................... 9 2. ACHIEVEMENT OF THE PROJECT GOALS AND OBJECTIVES ............................................... 14 2.1. Project Design ................................................................................................................ 14 2.2. Networking among human rights and other civil society organizations (goal 4) .......... 16 2.3. Information collection and dissemination (goal 3) ........................................................ 19 2.4. Strengthening skills and capacity of civil society organizations (goal 2) ...................... 19 2.5. Promoting citizen awareness about human, social and civil rights (goal 1) .................. 19 2.6. Facilitating dialogue between HR groups and the government (goal 5’) ...................... 20 2.7. Creation of institutions that allow citizens to assert their rights (goal 5’’) .................... 21 3. EFFECTIVENESS OF THE PROJECT INTERVENTIONS ............................................................ 23 3.1. Involving Mass Media in Mainstreaming Human Rights Activities .............................. 23 3.2. Trainings for NGOs ........................................................................................................ 25 3.3. Networking activities ..................................................................................................... 26 3.4. Web-based project activities .......................................................................................... 27 3.5. Activities implemented by the project grantees ............................................................. 27 4. USE OF PUBLIC OPINION SURVEYS IN MEASURING PROJECT RESULTS ......................... 34 4.1. Key characteristics of the MSI survey ........................................................................... 34 4.2. Mismatch of the margin of error with the scale of expected changes ............................ 34 4.3. Mismatch of the survey sample with the population affected by the project ................. 35 4.4. Mismatch of the survey sites with the project sites ........................................................ 36 5. PERCEPTION OF HUMAN RIGHTS GROUPS AMONG CITIZENS ........................................... 38 6. FACTORS THAT LED CITIZENS TO DEFEND THEIR RIGHTS ................................................ 39 7. UNINTENDED CONSEQUENCES AND EFFECTS OF THE PROJECT ...................................... 40 Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report ii 8. LESSONS LEARNED ........................................................................................................................ 41 8.1. Mainstreaming in the HR Assessment Report ............................................................... 41 8.2. Mainstreaming in the USAID RFA ................................................................................ 42 8.3. Possible interpretations of USAID RFA ........................................................................ 42 8.4. Mainstreaming in the MSI proposal ............................................................................... 44 8.5. Mainstreaming in the real life of the HR project ........................................................... 46 9. MAIN CONCLUSIONS AND RECOMMENDATIONS .................................................................. 49 9.1. Main conclusions ............................................................................................................ 49 9.2. Main recommendations .................................................................................................. 50 Annex 1. Evaluation Scope of Work (Notification of Performance Evaluation) ........................................ 52 Annex 2. Evaluation Design and Work plan ............................................................................................... 60 Annex 3. List of organizations consulted .................................................................................................... 74 Annex 4. List of documents studied ............................................................................................................ 76 Annex 5. List of the project grantees .......................................................................................................... 77 Annex 6. Types of interventions used by the project grantees by regions .................................................. 78 Annex 7. Target groups of the project grantees and the areas of rights they were focused on ................... 81 Annex 8. Project sites and survey sites in Voronezh oblast ........................................................................ 82 Annex 9. Developmental evaluation ........................................................................................................... 82 Annex 10. Social marketing campaigns implemented by .......................................................... 83 Annex 11. Disclosures of Conflict of Interest ............................................................................................. 84 Figures and Tables Figures Figure 1. Project Results Framework 5 Figure 2. Logic of Human Rights Project and Human Rights Program 15 Figure 3. Interactions between Project Participants before and during the project 16 Figure 4. Interactions between Project Participants – Voronezh 17 Figure 5. Interactions between Project Participants – Nizhni Novgorod 17 Figure 6. Interactions between Project Participants - Novosibirsk 18 Figure 7. Interactions between Project Participants - Perm 18 Figure 8. Number of visitors of the MSI project web-site 27 Figure 9. Frequency of use of various categories of activities by the grantees 28 Figure 10. Number of rights-related links in Runet by cities 32 Figure 11. Number of grantee event participants (cumulatively) 36 Figure 12. Illustrative map of survey and project sites in Voronezh oblast 37 Figure 13. Survey results: to whom appeal a violation? (2011) 38 Figure 14. Model of mainstreaming human rights in Russia proposed in the HR 41 Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report iii Assessment Tables Table 1. Project Indicators as per Project Monitoring Plan 6 Table 2. Advantages and disadvantages of the data collection methods 12 Table 3. Project indicators measured through survey 35 Table 4. Indicators proposed by MSI to measure progress towards IR1.1 and IR1.2 45 Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report iv Acronyms and Abbreviations CSO Civil Society Organization FPS Perm Federal Penitentiary GOR Government of Russia HR Human Rights HRG Human Rights Groups IBTCI International Business & Technical Consultants, Inc. MSI Management Systems International NGO Non-governmental organization RFA Request for Application ROL Rule of Law USAID United States Agency for International Development Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 1 EXECUTIVE SUMMARY This report presents the findings, conclusions and recommendations of the formative evaluation of the USAID/Russia Human Rights Project1 “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” implemented by Management Systems International (MSI). International Business and Technical Consultants, Inc. (IBTCI) is responsible for implementing this evaluation as per the terms of the Russia Monitoring and Evaluation Project (RMEP) and based on the Notification of Performance Evaluation #22 sent to IBTCI on May 15, 2012. The project aims to increase the knowledge of Russian citizens about their human, social, and civil rights and to create conditions that enable them to effectively protect those rights. This means that the overall project goal is to improve the observance of fundamental human rights as defined in the Mission Rule of Law (ROL) and Human Rights (HR) Project Results Framework. The project’s objectives are to: 1. Promote citizen awareness about human, social and civil rights, as well as mechanisms to effectively defend them; 2. Strengthen the skills and capacity of civil society organizations to educate and reach out to citizens; 3. Collect and broadly disseminate information about Russian citizens who successfully defended their rights; 4. Facilitate networking among human rights and other civil society organizations; and 5. Facilitate constructive dialogue between human rights groups and the government to develop and strengthen institutions that allow citizens to assert their rights. The purpose of this evaluation was to assess activity progress to-date in order to allow the team to make any necessary adjustments or amplify particularly successful interventions. The main evaluation questions were: - What is MSI’s progress in achieving the goals/objectives as defined in the Project Documents (Project Description, Work Plan, and M&E Plan)? Which objectives are particularly lagging or progressing and why? - What is the effectiveness of the project’s interventions as defined by the Project Description, Work Plan and M&E plan? Which ones are the most/least successful and beneficial to the target audiences and why? - How effective was the methodology of MSI’s public opinion surveys “Russian citizens and their rights: Priorities, attitudes and experiences” in measuring project results? - What is perception among citizens of human rights groups at this stage of the project? How effective are HR advocacy groups in the eyes of citizens? What are the factors that affect citizens’ perception both in positive and negative ways? - What have been the principal factors that led citizens to defend their rights, and how do those vary by region? - What have been the unintended consequences and effects of the project thus far?                                                              1 When referring to the subject of the evaluation as the “I’ve Got Rights project,” or “project” by itself throughout the document, project means a set of interventions/activities executed primarily through one USAID/Russia-funded implementing mechanism awarded to Management Systems International (MSI). 2 The Notification is included as Annex 1. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 2 - What are the key lessons learned that could affect future programming and potential regional dissemination? - Are there any key adjustments that can be made to the project now in order to ensure that goals are met? If so, what are they? How can they be introduced at this phase? The evaluation was conducted by a team of three evaluation consultants and a Russian human rights expert. Evaluation data were collected through document review, direct observations at the project sites, and semi-structured in-depth interviews with MSI staff, project partners, regional coordinators, grantees3 , project beneficiaries, journalists and human rights experts. When possible, local government representatives were interviewed as well. The Evaluation Team visited all four regions targeted by the Project (Nizhni Novgorod, Novosibirsk, Perm and Voronezh oblasts) and collected data in Moscow. Key Evaluation Findings 1. USAID internal process of preparing RFA was rather comprehensive; internal understanding of the mainstreaming concept was rather clear, plus there were numerous meetings with NGOs representatives, where there was a mutual understanding of a mainstreaming concept. At the same time USAID RFA did not include a crystal clear definition of ‘mainstreaming human rights’ concept. 2. Evaluation revealed that there were at least three possible ways of interpreting the USAID RFA as calling for: (1) a PR Campaign in the selected region(s); (2) Complex program with a focus on the human rights area the majority of population is most interested in; and (3) Small scale projects aimed at mainstreaming HR in various areas and creating models for the future use. The MSI project proposal combined the 2nd and 3rd approaches. It resulted in an implicit contradiction between the intent to implement mainstreaming HR approaches at a smaller scale in several selected domains and the suggested way to measure the project progress by the overall growth of the citizens’ awareness, motivation and capability to protect their rights in the four regions. Indicators used by MSI to measure project’s progress do not consider the notion of selected domains and referred instead to the general public – that is, the adult population of the four regions. 3. The project was not designed to considerably affect the human rights awareness of the population or citizens’ capacity to defend their own rights in the four oblasts and naturally did not succeed in that. 4. The project succeeded in promoting awareness of relatively narrow target groups in all four regions about specific human rights that varied depending on the groups’ interests. A number of effective approaches to mainstreaming human rights in various domains were implemented in the course of the project, but the existing experience is not conceptualized yet in the form of mainstreaming models. There is enough time before the end of the project to elaborate those models.                                                              3 The term ‘grantees’ hereinafter means ‘recipients of donations’ Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 3 5. The project succeeded in helping vulnerable and marginal groups of people and in some cases literally saved people’s lives. 6. The project succeeded in facilitating networking among grantees. The number of connections between participating NGOs increased considerably in all four regions in the course of the project. 7. The project has a very informative and well-designed website that can be maintained and used after the project is over. 8. The use of surveys in measuring project results was based on the assumption that the project would affect HR awareness, the capacity to defend their rights and willingness to do that among regular citizens in the four regions, while in reality the project was implemented so that it affected only selected narrow target groups in certain areas of human rights. Hence, the survey turned out to be irrelevant to the measuring of actual project results: the measurement approach did not fit the project design. Public opinion surveys were conducted by the two lead Russian professional survey groups to assess citizen awareness and understanding of the current state of human, social, and civil rights in the four regions of Russia. Surveys provided valuable information related to human rights in the four project regions and beyond. This information was disseminated and discussed by the project participants and other interested parties, which contributed to the better understanding of the state of human rights and priority areas for mainstreaming human rights. 9. The project under evaluation was innovative. Innovative projects are different from standard ones. One of the unique features of innovative initiatives is that it may not be clear form the very beginning if the project or program will work as expected and how it should work in general. Innovative programs should be highly flexible and adaptive. Traditional evaluation approaches to evaluation of innovative programs (such as mid￾term and final evaluations) may not be as effective as needed. Main Recommendations Recommendation 1. Pay special attention to communicating innovative concepts4 . In the future make sure innovative concepts, such as ‘mainstreaming human rights’, are properly communicated to the applicants and implementing partners. To transfer innovative concepts to the implementing partners use comprehensive communication approaches that include various channels such as expert ‘round tables’ (with participation of external experts), group discussions and documentation exchange. Double check that the key concepts are interpreted by the implementing partners properly. Recommendation 2. Use most relevant evaluation approaches to measure progress and results of innovative initiatives5 . Evaluation of innovative projects should differ from the traditional one. To make the evaluation most useful consider contemporary approaches to evaluation of innovations such as                                                              4 To USAID 5 To USAID Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 4 developmental evaluation that applies complexity concepts to enhance innovation and support evaluation use6 . Incorporate evaluation into the program as an on-going process. Recommendation 3. Conceptualize project experience in the form of mainstreaming models, mainstreaming criteria, and mainstreaming competencies7 . Reflect on the existing experience and together with the local partners explicitly describe effective models of HR mainstreaming. Describe possible strategies of mainstreaming HR in various domains. Based on the successful models provide definition(s) of mainstreaming HR and criteria for assessing mainstreaming HR effectiveness. Based on the existing experience describe competencies NGOs should have to mainstream human rights in Russia. One of the key tasks here will be finding an equivalent of the term ‘mainstreaming’ in Russian. Recommendation 4. Consider limitations of the possible use of the surveys in measuring project results8 . Do not interpret the results of the third survey in the terms of the project outcomes. In the future pay special attention to the relevance of the measurement approaches to the particular project/program design. Disseminate and use the results of the third public opinion survey planned under this project as a foundation for in-depth discussion with HRGs, CSOs and other interested parties. Recommendation 5. Consider creation of a Virtual Resource Center on mainstreaming HR in the MSI exit strategy9 . Find a successor among Russian NGOs (preferably – among project grantees and regional coordinators) who can further maintain the project website and/or incorporate its content into some other website. Consider an idea of creation of a virtual resource center10 specialized in mainstreaming human rights. This resource center should be established in collaboration with the local partners (possibly, regional coordinators) and further maintained by them.                                                                 6 See Annex 9 for more details. 7 To MSI 8 To MSI 9 To MSI 10 Virtual resource center (VRC) is a website that provides useful cases, articles, links, etc. on a particular topic. VRCs are often designed to support learning process and make self-education most convenient and effective. In fact, the existing project website can be maintained and positioned as a VRC in the future. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 5 1. INTRODUCTION 1.1. Description of the Project Evaluated The Human Rights Project “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” is part of the USAID/Russia Rule of Law (ROL) and Human Rights (HR) portfolio and is pursuant to the ROL and HR design document approved in June 2008. The “I’ve Got Rights” Project was developed by MSI in response to Request for Applications (RFA) No. USAID-Russia-DI-09-0001-RFA –Mainstreaming of Human, Social, and Civil Rights Project in Russia issued in February 2009. The “I’ve Got Rights” project is scheduled to operate from August 2009 to August 2013 in the following four regions of Russia: Perm, Novosibirsk, Nizhniy Novgorod, and Voronezh Fig. 1 depicts the project results framework as included in the RFA. The results framework presents the following project logic: if citizens are willing to defend their rights, are aware of their rights, and have the tools and capacity to defend their rights, they will assert their rights through actions such as filing court cases, appealing to administrative bodies, turning to local journalists and officials, and launching community actions. Further, if citizens assert their rights, then the observance of fundamental human rights will improve. The results framework also assumes that if the observance of human rights improves, then citizens will be more willing to defend their rights. Figure 1. Project Results Framework Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 6 The “I’ve Got Rights” Project aims to increase the knowledge of Russian citizens about their human, social, and civil rights and to create conditions that enable them to effectively protect those rights. This means that the overall project goal is to improve the observance of fundamental human rights as defined in the Mission ROL and HR Project Results Framework. The project’s objectives are to:  Promote citizen awareness about human, social and civil rights, as well as mechanisms to effectively defend them;  Strengthen the skills and capacity of civil society organizations to educate and reach out to citizens;  Collect and broadly disseminate information about Russian citizens who successfully defended their rights;  Facilitate networking among human rights and other civil society organizations; and  Facilitate constructive dialogue between human rights groups and the government to develop and strengthen institutions that allow citizens to assert their rights. The Project Monitoring Plan (Table 1) consists of indicators linked to the expected project impact. Table 1. Project Indicators as per Project Monitoring Plan Progress achieved to date Impact Indicators Frequency of measurement Baseline Life of Project Target As of end of Y2 (August 31, 2011) As of February 29, 2012 As of end of Y311 (August 2012) RESULT 1: OBSERVANCE OF FUNDAMENTAL HUMAN RIGHTS IMPROVED Community satisfaction with HR protection Disaggregated by gender and region Annual starting from Y3 15.3% 16.2% N/A 15.1% IR 1: Citizens Assert Their Fundamental Rights (highest level direct result of project’s activities) Number of HR court cases filed in target regions by project-supported legal services groups Disaggregated by gender and region Semi-annual 15 80 496 929 TBD IR 1.1 Citizens are willing to defend their rights % of citizens polled who say they are willing to defend their rights Disaggregated by gender and region Annual starting from Y3 51.6% 54.7% N/A 57.4% IR 1.2 Citizens have an awareness of their rights % of citizens polled who exhibit Annual 20.4% 21.2% N/A 12.8%                                                              11 Data in this column is still not fully available and will be submitted to USAID by MSI in the end of September, 2012 with the semi-annual report. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 7 awareness of their rights Disaggregated by gender and region starting from Y3 IR 1.3 Citizens have the tools and capacity to defend their rights Number of HRGs’ and CSOs’ representatives with improved knowledge and skills Semi-annual 0 160 340 484 TBD Percent of survey respondents who understand how to defend their rights Disaggregated by gender and region Annual starting from Y3 18.9% 19.7% N/A 11.3% USAID Operational Plan Indicators Number of domestic human rights NGOs receiving US Government support through the project Annual 0 113 121 TBD Number of people receiving legal consultations (from NGOs supported by the project) (disaggregated by gender and region) Annual 0 4000 8,808 TBD Critical Assumption 1: Responsible government authorities act to fulfill human rights obligation Percent of HR cases brought by legal services groups (supported by the project) to appropriate government offices that are resolved Annual 0 N/A 17% TBD Critical Assumption 2: Legal framework for the protection of fundamental human rights exists. Number of advocacy campaigns (to reform laws and procedures) supported by the US Government Annual 0 N/A 33 TBD Responsibilities for project management are divided between MSI offices in Washington, DC, and Moscow, Russia. The Washington office is responsible, for example, for managing public opinion surveys in the project regions and overall reporting to USAID. The Moscow office is responsible for managing the project activities in the regions. The “I’ve Got Rights” Project has four regional coordinators – one in each of the four abovementioned project regions – that are responsible to support and coordinate the project activities in their regions. The project has also established a network of human rights NGOs in the target regions and provides financial support through small grants to projects that are consistent with the overall goal of the “I’ve Got Rights” Project. In addition to this small grants component,12 the project provides training to human rights NGOs, organizes annual competitions for journalists from the four project regions, and develops print and on-line information materials. By April 2012 the “I’ve Got Rights” Project supported 61 projects that were implemented by civil society organizations to increase public awareness about human, civil, and social rights; provide legal assistance to citizens whose rights were violated; and advocate for reforms to better observe and defend citizen rights including rights of youth and children, the disabled, workers, women, etc. Outputs of the project at this stage include the following:                                                              12 The term ‘grant’ hereinafter means ‘donation’ Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 8  more than 13,000 citizens whose rights were violated received legal assistance;  929 citizens’ rights cases were defended in court and 3,000 cases were forwarded to other governmental agencies; and  more than 20,500 people participated in 517 seminars, trainings, round tables and other events. Other outputs include:  108 brochures and flyers about human rights were developed and distributed and 16 analytical reports prepared;  1,014 people participated in 58 discussions of human rights clubs organized regularly by regional partners of the project;  237 materials were published in print media, 1,389 on the Internet, 89 stories and programs were on TV, and 157 on the radio;  the project website expanded to include 229 success stories and about 1,540 reference materials;  484 representatives of human rights and other NGOs took part in workshops, organized by the project;  handbook ”Human Rights NGOs’ Technologies” was published;  four interregional networking meetings for project partners were conducted; and  more than 200 journalists participated in the annual Journalists Contest conducted in 2010 and 2011. 1.2. Purpose of the Evaluation13 This is a formative evaluation of the USAID/Russia Human Rights Project14 “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” implemented by Management Systems International (MSI). International Business and Technical Consultants, Inc. (IBTCI) is responsible for implementing this evaluation as per the terms of the Russia Monitoring and Evaluation Project (RMEP) and based on the Notification of Performance Evaluation #215 sent to IBTCI on May 15, 2012. The purpose of this evaluation is to assess activity progress to-date in terms of the degree to which MSI’s activities have successfully empowered and mobilized citizens to assert their rights. It will also allow USAID and the project team to make any necessary adjustments or amplify particularly successful interventions, as well as to inform the design of the USAID/Russia future programming in the field of human rights. 1.3. Evaluation Questions  What is MSI’s progress in achieving the goals/objectives as defined in the Project Documents (Project Description, Work Plan, and M&E Plan)? Which objectives are particularly lagging or progressing and why?                                                              13A detailed description of evaluation design and work plan is included into the Annex 2. 14When referring to the subject of the evaluation as the “MSI project,” the “I’ve Got Rights project,” or “project” by itself throughout the document, project means a set of interventions/activities executed primarily through one USAID/Russia-funded implementing mechanism awarded to Management Systems International (MSI). 15 The Notification is included as Annex 1. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 9  What is the effectiveness of the project’s interventions as defined by the Project Description, Work Plan and M&E plan? Which ones are the most/least successful and beneficial to the target audiences and why?  How effective was the methodology of MSI’s public opinion surveys “Russian citizens and their rights: Priorities, attitudes and experiences” in measuring project results?  What is perception among citizens of human rights groups at this stage of the project? How effective are HR advocacy groups in the eyes of citizens? What are the factors that affect citizens’ perception both in positive and negative ways?  What have been the principal factors that led citizens to defend their rights, and how do those vary by region?  What have been the unintended consequences and effects of the project thus far?  What are the key lessons learned that could affect future programming and potential regional dissemination?  Are there any key adjustments that can be made to the project now in order to ensure that goals are met? If so, what are they? How can they be introduced at this phase? 1.4. Collaboration with the local partners and stakeholders The key stakeholders were USAID-Russia and MSI (Moscow and DC offices). From the outset this evaluation was conducted in a collaborative manner. USAID Mission staff and the implementing partner, together with the Evaluation Team, discussed evaluation methods, sampling, and the work plan. The evaluators consulted with all three parties on interpretation of the key evaluation findings and recommendations. 1.5. Methodology 1.5.1. Overall approach The evaluation questions were purely empirical and answering them required an in-depth comprehensive study of the project and its context. The evaluation used a mixed method approach – qualitative and quantitative data were collected and analysed concurrently. Initially the team planned to work with the data collected by two project surveys (baseline and mid-term), but as the analysis of the project activities and their results and of the survey methodology revealed that the survey was unable to detect the project results (discussed further in Chapter 4), the analysis of the survey data was dropped. In the course of data collection and analysis, the evaluators remained open to all kinds of information and ideas and used the ‘progressive focusing’ approach that allowed new issues and themes to emerge and change in relative prominence. A rigorous inductive analysis was used to come to conclusions and recommendations. In summary, this evaluation had the following characteristics: - The evaluation questions were purely empirical. - The evaluation used a mixed method approach. - The evaluation design was relatively flexible to accommodate emerging issues and themes. 1.5.2. Sources of Information The evaluation team used the following sources of information: Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 10  Project documentation: RFA, “I’ve Got Rights” project description, M&E plan, work plans and progress reports to USAID; descriptions and progress reports of the small projects implemented by partners in the project regions.  Baseline and mid-term surveys (“Russian Citizens and their Rights: Priorities, Attitudes and Experiences”).  USAID/Moscow staff working on HR issues.  The “I’ve Got Rights” Project staff in Moscow and Washington, DC.  Project partners in four regions.  People who used services of the project partners or participated in their events in the regions.  Journalists who participated in the competitions organized by the project.  Human rights ombudsman in the project regions and human rights experts. 1.5.3. Sampling strategy Evaluators interviewed all staff of the “I’ve Got Rights” Project staff in Moscow and representatives of all partners in the regions. To study the experiences of the project beneficiaries (i.e., people who used the services of the project partners or participated in their events as well as participants in the journalist competition), the evaluators used the purposive sampling strategy based on the identification of information-rich cases – the sources from which they could learn most about the activities. Interviews with beneficiaries were conducted in all four project regions. 1.5.4. Data Collection Methods Document Review16 The Evaluation Team reviewed the documents provided by USAID and the “I’ve Got Rights” Project staff (RFA, SOW, Work Plans, M&E Plans, reports, M&E data). The Evaluation Team also explored online publications, media publications and videos produced in the course of the project. Direct observation The trip to Voronezh was scheduled to overlap with the Human Rights Week organized by the “I’ve Got Rights” Project and a networking meeting of the project partners. One of the members of the Evaluation Team attended and observed most of the events that took place during the Human Rights Week as well as the networking meeting in Voronezh. Semi-structured in-depth individual and group interviews17 The Evaluation Team used semi-structured in-depth interviews to collect information from the “I’ve Got Rights” Project staff, representatives of project partners, journalists and regular citizens who were involved in project activities. As experiences of the respondents were heterogeneous, the evaluation team used a mixed approach to developing the interview instrumentation based on a combination of a) interview guides and, b) informal conversational approaches.                                                              16List of documents studied is included as Annex 4. 17 List of interview respondents is included in Annex 2. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 11 The interview guide approach requires that interview topics and issues are specified in advance, while the interviewer can decide on the sequence and wording of questions in the course of the interview. The strength of this approach is that the interview guide increases the comprehensiveness of the data and makes data collection more systematic. The weakness of this approach is that important and salient topics may be inadvertently omitted. When the informal conversational interview approach is used, questions emerge from the immediate context and are asked in the natural course of the conversation. There is no predetermination of question topics or wording. The strength of this approach comes from its ability to increase the salience and relevance of questions during an interview and those that emerge from observations. These can be matched to individuals and circumstances. The weaknesses of this approach are that it is less systematic and comprehensive and different information is collected from different people resulting in organizational and analytical challenges. Mixing these two approaches allows for the minimization of their respective weaknesses while benefiting from the strengths. Project Reach Mapping Using the data from small project reports and interviews with the project partners, the evaluation team mapped the locations where small project activities were implemented (for one of the project regions – Voronezh oblast).The resulting list of locations in Voronezh was then compared with the lists of locations included in the relevant MSI surveys (from survey reports). Social Network Analysis (Spider-webbing) To assess whether the project partners have established new working links between each other, in the course of interviews all partners were asked to look through the list of organizations that implemented small projects and mark those that they worked with before and in the course of the “I’ve Got Rights” Project. These data were then processed and mapped using NodeXL software. 1.5.5. Data analysis Four distinct processes were involved in making sense out of evaluation findings18. 1. Description and analysis: Describing and analysing findings involves organizing raw data into a form that reveals basic patterns. The factual findings as revealed in actual data will be presented in a user-friendly fashion. 2. Interpretation: What do the results mean? What is the significance of the findings? Why did the findings turn out this way? What are possible explanations of the results? Interpretations go beyond the data to add context, determine meaning, and tease out substantive significance based on deduction or inference. 3. Judgment: Values are added to analysis and interpretations. Determining merit or worth means resolving to what extent and in what ways the results are positive or negative. What is good or bad, desirable or undesirable, in the outcomes?                                                              18Based on Patton, M. Q. (1997).Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage. (page 307) Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 12 4. Recommendations: The final step adds action to analysis, interpretation and judgment. What should be done? What are the action implications of the findings? Only recommendations that follow from and are grounded in the data ought to be formulated. 1.5.6. Protection of human subjects It is important to ensure that evaluation participants are protected. The evaluation team made arrangements for all evaluation activities with the heads of participating entities. Evaluators explained the purpose and tasks of the study to them and to all interviewees. Verbal permissions from organizations and individuals to collect and use information were obtained. In particular, all the interviewees agreed to have their names included into the list of people interviewed. They also were assured that their opinions will be used for data analysis and report writing purposes anonymously. Informal rules of joint work were negotiated. When asked, evaluators always allowed people to see the interview questions in advance. 1.5.7. Briefing and Dissemination of Evaluation Findings The evaluation team presented the preliminary evaluation findings to the USAID/Moscow Human Rights Design Team in the form of a PowerPoint presentation. Feedback received during and after the presentation was carefully considered in the process of developing this draft report. 1.5.8. Challenges and limitations of the evaluation methodology Table 2 presents advantages and disadvantages of the main methods used in the course of this evaluation. The majority of the data collected in the course of this evaluation was qualitative. Qualitative data allows for a comprehensive and multifaceted understanding of people’s experiences with the project, but its collection is both labor-intensive and time-consuming. The Evaluation Team interviewed representatives of all project partners, but the number of beneficiaries who were involved in this evaluation was relatively small. Beneficiaries were selected through purposeful sampling process, so the data generated through these interviews do not allow generalization. Still the data collected in the course of interviews with beneficiaries allow for extrapolation,19 which is sufficient for the purposes of this evaluation. Table 2. Advantages and disadvantages of the data collection methods Method Advantages Disadvantages Observation - Collect data where and when an event or activity is occurring - Does not rely on people’s willingness to provide information - Directly see what people do rather than relying on what they say they do - Susceptible to observer bias - Hawthorne effect – people usually perform better when they know they are being observed - Does not increase understanding of why people behave the way they do Document Review - Relatively inexpensive - Good source of background information - Information may be inapplicable, disorganized, unavailable or out of date - Could be biased because of selective survival of information - Information may be incomplete or                                                              19 Extrapolation here is understood as an inference about the future (or about some hypothetical situation) based on known facts and observations. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 13 - Unobtrusive - Provides a “behind the scenes” look at a program that may not be directly observable - May bring up issues not noted by other means inaccurate - Collection, review, and analysis of numerous documents can be time consuming Interviews - Useful for gaining insight and context into a topic - Allows respondents to describe what is important to them - Useful for gathering quotes and stories - Susceptible to interview bias - Time consuming and expensive compared to other data collection methods - May seem intrusive to the respondent Focus Groups (Group Interviews) - Quick and relatively easy to set up - Group dynamics can provide useful information that individual data collection does not provide - Is useful in gaining insight into a topic that may be more difficult to gather information through other data collection methods - Susceptible to facilitator bias - Discussion can be dominated or sidetracked by a few individuals - Data analysis is time consuming and needs to be well planned in advance - Does not provide valid information at the individual level - The information is not representative of other groups Another challenge of the qualitative methodology is that an evaluator is an instrument of research and inevitably brings his/her perceptions and values to the study. To minimize the possible distortion of the findings, evaluators regularly discussed within the team what values they were bringing to this study. The team members carefully monitored each other’s neutrality about the evaluation. 1.5.9. Disclosure of conflicts of interest All evaluation team members provided a signed statement attesting to a lack of conflict of interest relative to the project being evaluated; this statement was signed prior to beginning work on this evaluation. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 14 2. ACHIEVEMENT OF THE PROJECT GOALS AND OBJECTIVES 2.1.Project Design The USAID/Russia Human Rights Project has the following goals: 1. Promote citizen awareness about human, social and civil rights, as well as mechanisms to effectively defend them; 2. Strengthen the skills and capacity of civil society organizations to educate and reach out to citizens; 3. Collect and broadly disseminate information about Russian citizens who successfully defended their rights; 4. Facilitate networking among human rights and other civil society organizations; and 5. Facilitate constructive dialogue between human rights groups and the government to develop and strengthen institutions that allow citizens to assert their rights. Project activities were planned and implemented in accordance with these goals. MSI also developed and implemented M&E plans to monitor the project’s progress towards its goals. Before discussing MSI’s progress in achieving the project goals, we propose to look at how the five project goals are inter-related and how they fit the bigger picture – the overall HR program logic. This will help us establish a proper context for the following analysis. Figure 2 is based on the RFA developed by USAID and shows the hierarchy of all the expected results (goals) and the key activities implemented to achieve them. The HR program goal, intermediate results and sub-intermediate results are colored peach; the project goals (hereinafter called ‘goals’) are colored blue; the project activities are colored grey. The five project goals are inter-related. Networking among HR and other CSOs (goal 4) contributes to building their capacity to educate and reach out to citizens (goal 2) and to collection and dissemination of information (goal 3). The latter contributes to the citizens’ willingness to defend their rights (IR1) and helps promote citizens’ HR awareness (goal 1), which also motivates citizens to defend their rights (IR1). Strengthened capacity of HR groups and CSOs (goal 2) helps them promote citizen HR awareness (goal 1). To make the project logic more explicit we divided goal 5 into two parts: facilitation of constructive dialogue between HR groups and the government leads to development of institutions that allow citizens to assert their rights, which in turn contributes to increasing government ability to fulfill its HR obligations. To analyze achievement of the project goals we will start from the bottom of Figure 1 and go up. Russia Monitoring and Evaluation Project – HR Project Evaluation Draft Report 15 Figure 2. Logic of Human Rights Project and Human Rights Program    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 16 2.2.Networking among human rights and other civil society organizations (goal 4) Networking could be characterized by the number of connections between the organizations. In order to explore how networking worked between project participants, we used the social network analysis technique. All the grantees’ leaders were asked to indicate which of the project participants from all four regions they had interactions with (a) before the project and (b) during the project. Figures 3 through 7 summarize their answers. In order to make the diagrams readable, numbers are used instead of full names of organizations. A list of the project grantees with their respective numbers is included as Annex 5. Interactions are defined as any mutual or reciprocal actions ranging from distanced consultation to joint projects. For example, simply knowing each other or sitting together at a conference were not considered interactions. Figure 3 demonstrates a considerable overall increase in the number of interactions between the project participants during the project. Figure3.Interactions between Project Participants before and during the project Figures 3-6 illustrate the development of social networks among project participants by cities. The graphs are developed in NodeXL using the Harel-Koren Fast Multiscale algorithm, which is one of NodeXL’s two force-directed algorithms. Force-directed algorithms are designed to make all the lines (a.k.a. “edges”) about the same length and to minimize line crossings, which can make for a more aesthetically pleasing and readable graph. Organizations that form a vertical line on the left side of the figure 4 deal with issues in Voronezh, Nizhni Novgorod and Perm. Interestingly, they were in touch with each other before the project but were not connected to any other project participants. During the project, an organization located in Voronezh (36) established a connection with one more organization from Nizhni Novgorod (2) and developed links with two organizations in Voronezh, one of which is the regional project coordinator (30). The latter was already in touch with the majority of project participants in Voronezh (before the project), and dramatically    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 17 increased the number of its connections with the project participants from the other cities during the project. Figure 4.Interactions between Project Participants – Voronezh There are two interesting features of network development in Nizhni Novgorod (figure 5). One organization (6) looks isolated before the project and has only one connection during the project. This organization, is a very strong NGO well-known in Russia. is very well connected with other NGOs but does not cooperate with NGOs working on topics not related to The project did not change that. once received a request for information from one of the other project partners. Figure 5. Interactions between Project Participants – Nizhni Novgorod    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 18 The second interesting feature of situation in Nizhni Novgorod is that one organization that considerably increased the number of its connections was not the project coordinator (1). It was a strong organization that is very active at networking and has a lot to share with the others. Figure 6.Interactions between Project Participants - Novosibirsk The most outstanding feature of Novosibirsk graph (figure 6) is the increased number of connections beyond Novosibirsk of the regional coordinator (11). It could be explained by the nature of their internet-based project which is very interactive and is open for any number of human rights experts from all over the country (more details on this project are in the next chapter). Figure 7.Interactions between Project Participants - Perm Another organization that increased the number of its links (from two to seven) was an association dealing with They are very successful in    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 19 asserting those rights and were contacted by many organizations from the other regions during the project. The case of Perm differs from the previous three by the number of connections between the project participants before the project (figure 7). The project coordinator (21) was already very well connected with most project participants before the start of the project. This can be explained by the fact that Perm based NGOs are very well networked. They know each other, have joint projects, and have coordinated their activities for a long time. That is why the growth of their network was not as impressive as in the three other cities: it was already strong before the project started. 2.3.Information collection and dissemination (goal 3) The project team made a considerable effort to collect information about project events and activities of the project partners. All major activities were filmed and later edited to make a video. Collected information was made public via the project web site, regular new bulletins, and social media. Project partners also disseminated information through their channels. The project produced over one hundred books and leaflets.MSI activities in this area are discussed in the next chapter. 2.4.Strengthening skills and capacity of civil society organizations (goal 2) Representatives of the project partners were very positive about the content and quality of training they received in the course of the project: “There were some unique things like the seminar on fundraising by One won’t be able to learn this elsewhere.” Most people said that material they learned was useful. At the same time very few people said that they were actually using what they learned. To some extent this can be explained by the fact that new technologies promoted by the project, e.g. the use of social media, require a considerable investment of time and energy. Given that people at NGOs are already stretched, trying and practicing new things can be very difficult. There were several instances when one project partner learned from another. For example, adopted the idea and methodology for doing schools of self-advocacy developed by and is actively using it. Several representatives of a less established NGOs said that working with the MSI staff on their project proposals helped them to build their project planning skills. “It felt like they took you by the hand and led. Their questions led me to get a deeper understanding of the problem,” noted one of the respondents. Effectiveness of these activities is further discussed in the chapter 3. 2.5.Promoting citizen awareness about human, social and civil rights (goal 1) The project started with a thorough needs assessment and selection of region specific HR domains in which to work. Each of the NGOs supported by the project worked in its respective domain with its target group in a particular area of human rights (see Annex 7). This was fully in line with those NGO’s missions and made sense under existing circumstances (modest project    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 20 budgets and limited duration of the projects). None of the participating NGOs’ activities were aimed at the entire population of their respective regions20. For some NGOs, such as or the key priority was not to raise awareness of their clients but to defend their client’s rights (in some cases literally save their lives). Hence, the project affected awareness of certain target audiences in certain areas related to human rights. All the activities implemented by the project grantees modelled what can be done to promote citizen awareness about human rights in certain domains and how. Lessons learned were analysed and disseminated among the interested parties in the project regions and beyond. Although this information is available online and via other channels, regular citizens are not the ones who will actively use it. The interested parties are NGOs or CSOs working in similar areas and people whose rights were violated in a similar way. Public opinion polls conducted by the project did not show any considerable changes in the human rights awareness of the general public in the four regions. Those studies were not designed to measure changes in the domains where project interventions were implemented. (See more discussion on this in Chapter 4.) 2.6.Facilitating dialogue between HR groups and the government (goal 5’) According to the MSI project proposal, the project intent in this area was “to fully integrate local government representatives in the project’s implementation.” In particular, the project was going to work with such governmental institutions as “regional Civil Society Institutions and Human Rights Councils, Councils on Cooperation with Non-Governmental Organizations, Public Chambers, Public Councils under law enforcement institutions (interior authorities, prosecutor offices, drug inspections, penitentiary, and other authorities), Observation Commissions monitoring human rights observance of the imprisoned and detained, and other institutions depending on issue.” At an early stage of the project, dialogues with the governments in all four regions started with meetings with representatives of the regional administrations and other governmental agencies to establish a common foundation for cooperation. The project representatives - when possible - consulted with the local authorities on the human rights priority areas in their regions and ideas for approaches to deal with these issues. From the very beginning it became clear that an American organization21 was not well positioned to participate in or to facilitate a dialogue between HR groups and the government due to the                                                              20 What would be a rights activity that is aimed at the entire population? The first surveys demonstrated that there were some human rights issues that were considered critical by the majority of population. Based on the survey findings one focus area such as housing, health care or education could be selected to cover the entire population. A comprehensive strategy should be developed to raise awareness of citizens in that area, equip and empower them to protect their rights. Systematic coordinated efforts of NGOs, community activists, professional associations, etc. could lead to the expected changes at the regional level. Some of the topical areas chosen by the project participants could potentially be used as a basis for such an intervention. Social marketing campaigns implemented by in 2003 in Ryazan, Perm, and Rostov regions could be used as illustrative examples of such an approach. See Annex 10 for more details.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 21 sensitivity of the human rights issues (for any government in any country) and cooling of relations between Russia and the US. Under such circumstances MSI made a decision to let its NGO partners in the regions develop and maintain dialogue with the government entities. The project’s role then became to provide informational and/or technical support if and when needed. Most NGOs supported by the project are dealing with various government entities. In conversations with us they explained that dialogue with government is an essential part of their work since many problems faced by their constituents can be resolved only with government participation. There is a broad range of NGO-government collaboration ‘modes.’ Organizations such as or have very close relations with the government and often receive financial support from it. Other disability organizations such as are more distanced from the government although they are often contacted by government entities as experts on disability issues in general and on the rights of disabled people in particular. Human rights groups such as known for their uncompromising work to defend the rights of their clients, are definitely even more distanced from the government, but are considered by the government as important stakeholders and subject experts. Even representatives of the n Voronezh, who position themselves more like a watchdog organization, maintain dialogue with the local ombudsman and some other government entities. Dialogue between project participants and government entities in many cases resulted in some notable results that will be discussed in the following subsection. 2.7. Creation of institutions that allow citizens to assert their rights (goal 5’’) An institution here is understood as any structure or mechanism of social order and cooperation governing the behavior of a set of individuals within a given human community. Institutions are identified with a social purpose, transcending individual human lives and intention by mediating the rules that govern cooperative human behaviour. There were several cases when the project grantees made contributions to the development of institutions that allowed citizens to assert their rights (the list is illustrative): - in Novosibirsk created conditions for people to vote - n Perm made a considerable contribution to the creation of new systems that allow better healthcare and educational services to - Youth n Perm in collaboration with various entities contributed to the implementation of the institute of in the Perm Krai. Conclusions: 1. Goal 1. The project succeeded in promoting awareness of relatively narrow target groups about specific human rights that varied depending on the groups’ interests.                                                                                                                                                                                                 21 Although this is beyond evaluation scope of work, the evaluation team would like to stress that an American organization with well-established management systems was able to effectively administer a large number of grants and brought some utility in the project ability to do large scale re-granting.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 22 The project was not designed to affect the human rights awareness of the general public in the four oblasts and naturally did not succeed in that. 2. Goal 2. Project implemented a series of skill-building activities for participating NGOs. Overall, less experienced NGOs benefited from those activities to a greater extent than their more experienced colleagues. 3. Goal 3. Project collected and systematically disseminated information about Russian citizens who successfully defended their rights via different channels. It is hard to make any judgement regarding how broad this dissemination was since the term ‘broad dissemination’ was not defined in the project documents. Dissemination was not aimed at the general public in the four oblasts. 4. Goal 4. Project succeeded in facilitating networking among grantees. The number of connections between participating NGOs considerably increased in all four regions during the course of the project. 5. Goal 5. All NGOs supported by the project have to be in touch with some government entities in order to achieve their goals. The level of collaboration varies. In several cases project grantees succeeded in establishing institutions that will allow certain categories of citizens to assert their rights. MSI was not well positioned to facilitate NGO-government dialogue and rightly delegated this responsibility to its regional coordinators and grantees.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 23 3. EFFECTIVENESS OF THE PROJECT INTERVENTIONS This chapter provides information on the effectiveness of the key project activities implemented by MSI and the project grantees. 3.1.Involving Mass Media in Mainstreaming Human Rights Activities The project planned to use three approaches to involve media in educating citizens on their rights and their protection: - Providing financial support to mass media organizations to mainstream human rights, - Promoting human rights NGOs’ cooperation with mass media, and - Conducting an annual journalist contest for best broadcasts and publications on human rights. 3.1.1. Financial support to mass media The project has provided financial support to two media partners: the TV programme in Novosibirsk and the TV company n the city of Nizhny Novgorod region. The TV programme is a independent investigative program Since , has bought air time on TV channels in Novosibirsk and has covered its production expense by selling time to advertisers. The program audience is about 100,000. In 2010, participated in the journalist contest organised by the project. As a winner of the regional round, the producer of the program came to Moscow for the final round and one of the jury members introduced to the MSI management team. The first project grant to supported its regular production, while the second extended the scope of work: eam makes extended versions of its reports on that aired in Novosibirsk by adding information from other regions for the use of the project. Before working with the project, the production team of never used grants as a source of financial support of their activities, so the project alerted them to this option of raising money. Recently the program also received a grant from the received project support to produce the talk show which There is no data on the audience of the talk show (the population of the city of is about people). 3.1.2 Promoting cooperation between media and human rights NGOs Since June 2010 the Project grantee organizations participated in 246 TV and radio shows and generated 237 publications in print media.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 24 Representatives of both media partners who received MSI grants regularly participate in networking meeting which gives them an opportunity to establish contacts with NGOs. As a result media partners involve NGO partners when doing stories. For example, worked with the when journalists were working on the story about bad land that was given to families entitled to free land lots. Due to their participation in the project, media partners have expanded their contacts with NGOs. “In the past we also communicated with NGOs, but not very often. Now we do it in a more systemic manner,” said one of the media partners. 3.1.3 Seminars for journalists In 2010 the project organized two-day workshops titled Workshops were attended by more than 70 print, radio and TV journalists. The participants were trained on Russian and international norms of human rights defence, methods of journalistic investigation, and information verification. According to participants whom the evaluation team interviewed, the quality of these workshops was good but methods of journalistic investigation were irrelevant to journalists of local media working under serious time and budget constraints. During the Human Rights Week in Voronezh, the organized a master￾class with correspondent of magazine, on covering issues of human rights, civic activism and nationalism. The master-class was attended by 3 local journalists from 2 media outlets. There were also several participants from local human rights NGOs. 3.1.4 Annual journalist contest The project has already conducted two annual competitions. Both included regional rounds and a final attended by the winners of the regional rounds. In 2010 regional rounds were open only to journalists from the project regions. In 2011 the contest expanded its reach by inviting journalists from the neighbouring regions. Several respondents from organizations involved in organization of the contest mentioned that launching a new journalist contest was not such a good idea. There are already many similar contests for journalists, e.g. Artem Borovik Award for investigative journalism and Golden Tamburin Award. And for some journalists preparing materials for submission to the contest seemed too burdensome. “Competitions are not particularly significant for me. I don’t have energy and time to participate,” said one of the project’s contest winners. As a result the project’s partners organizing the competition had to use their contacts with journalists to get them to submit materials for the competition. It is important to mention that the annual journalist contests organized by the MSI project were designed to involve regional journalists and were more open for regional media representatives than some national contests. Despite this initial reluctance to submit materials, participants of the final in Moscow highly appreciated the format of open discussion of contestant materials by a jury made of leading Russian journalists: “Journalists from regional media rarely have an opportunity to get their materials reviewed by nationally-known professionals and    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 25 get their comments and advice,” said one of the contest organizers. “An opportunity to talk with getting professional opinion about your work is the main value of participating,” noted one of the journalists. Still, the data collected in the course of the evaluation does not suggest that this professional review had much impact on the quality of work of the journalists who received it. Several winners of the 2010 contest participated in 2011. And, according to one of the members of the jury, the quality of the materials was not any better. The evaluation did not find any data that after the contest its participants and even winners started to write more on the issues related to human rights. One of the winners even had to totally abandon the human rights beat because of the change in the editorial policy. 3.2.Trainings for NGOs Trainings for NGOs were conducted under the ‘Mainstreaming Human Rights Training School.’ Based on the project’s work and contacts with representatives of CSOs in the regions, MSI developed a curriculum to address “the best ways to mainstream human rights, using the best communication and outreach approaches, including examples and best practices from international experience.”22 It was expected that the training program would include topics such as: - Strategic communications and public outreach skills, especially targeting key citizen constituencies and framing human rights issues as important to everyday life, to effectively communicate with the public and other targeted groups through key messages, earn the public's trust and support, improve public understanding of the value of human rights, and build public confidence in protecting their rights. - Effective programming by assessing the interests, needs, beliefs and attitudes of key constituent groups, defining, developing and implementing strategic objectives, identifying the target audiences, evaluating impacts, and revising strategies accordingly. - Advocacy skills, including advocacy and lobbying, negotiation, and coalition-building - Financial and organizational sustainability - The status of current human rights issues from a legal and practical implementation perspective.23 In accordance with the plan, MSI recruited several lead Russian trainers who conducted workshops on public relations and effective informational campaigns, working with mass media, application of civil rights in NGO activities, technologies of human rights advocacy and lobbying, and new developments in human rights legislation for civil activists and journalists. After publication of the manual “Technologies for Human Rights Organizations,” the project organized several master classes to introduce the new publication and promote some of the ideas presented in it. According to our information, feedback on some of the trainings from the majority of participants was very positive. For example, participants in all four regions appreciated training on fundraising from individual donors and training on the use of internet-based approaches in the                                                              22MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA.MSI Quarterly Report: December 1, 2009 – February 28, 2010 23 Ibid.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 26 human rights field. These trainings took place at the September interregional meeting in Voronezh. There was a certain level of disagreement between the training participants about the effectiveness of some of the other trainings – more experienced people were less satisfied with the training than their less experienced colleagues. There were several comments on the lack of regional context in the trainings conducted by people coming from the other regions of Russia. Considering the difficulty of meeting the needs of all the participants in such a diverse group, overall the training program was well organized and useful. Several questions for further discussion24 could be: - Why was the training program called ‘Mainstreaming Human Rights Training School?’ - Was there a concept of mainstreaming HR that became a theoretical basis for the curriculum? - Was there a list of competencies needed to successfully mainstream human rights developed? - Were there any trainings or sessions conducted on mainstreaming human rights? There might be potential for future development in this area. 3.3.Networking activities Networking activities took place on two levels – inside the regions and between them. Regular networking meetings of the project partners were the principal vehicle for promoting the networking and cooperation between project partners. The Project also used other vehicles and incentives to promote cooperation between its participants including facilitation of exchange of information among grantees (weekly news, exchange of the ‘products’ developed by grantees, discussion clubs, etc.), as well as incentivizing cooperation through the grants program. Results of the social network analysis indicate that this project component was effective. In the regions, project coordinators organized regional networking activities – meeting and group discussions – where they invited both the MSI project partners and other local NGOs and activists. For example, in Voronezh, according to the regional coordinator, the total number of people who attended those meeting since the beginning of the MSI project was about 150, and about 15-20 people came on a regular basis. These meetings were open to anyone, and there are examples when people came out of curiosity, got interested in the human rights issues, and became activists.                                                              24 This evaluation was not focused on the training component of the MSI project. We propose these questions for consideration by MSI and further discussion. It may lead to some changes in the training program or help formulate lessons learned for the future projects.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 27 3.4. Web-based project activities The project has a full-fledged and frequently-updated website that offers both text and video information about all project partners and activities, materials on various rights, and success stories. Figure 8 shows the number of visitors to the site in June and July 2012. The number of visitors usually stays in the range of several dozen, and most people come during the week days which may indicate that most visitors come from office computers. Figure 8. Number of visitors of the MSI project web-site Source: http://rating.openstat.ru/site/2118436 The project is active on social media. As of July 31, 2012 the project has 160 followers on Twitter and 1271 friends on Facebook. 3.5. Activities implemented by the project grantees This section discusses effectiveness of various types of interventions implemented by the project grantees. Fourteen types of activities were identified by the evaluation team on the basis of data collected. Figure 9 shows how often various types of interventions were used by the project grantees.25 A ‘typical’ activity of a project grantee would include: - Consultations (individual, group, in person, telephone, on-line, via e-mail) - Educational activities (both traditional and not-so traditional, e.g. street information meetings) - Development and dissemination of informational and educational materials (print, audio and video)                                                              25 More detailed information on this is included into Annex 6    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 28 - Use of on-line instruments (sites, on-line media, social media) - Use of traditional media (publication of self-produced materials and working with journalists) Figure 9. Frequency of use of various categories of activities by the grantees In some cases grantees would also use: - Case support and representation (in court and other institutions) - Open public events (including conferences and festivals) - Expert discussions and think tanks - Monitoring of HR observance ( and making proposals to authorities on necessary improvements) - Contributing to development of norms and regulations (drafting, discussion, expert review) - Serving on various bodies established by authorities (public councils, commissions, working groups, etc.) In a few cases, two more types of interventions were used: - Facilitation of citizen initiative groups (outreach) - HR-related studies (including dissemination of their results) One activity included a unique intervention related to the special needs of its beneficiaries (blind people): - Adjustment of physical infrastructure necessary to promote the observance of HR    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 29 3.5.1. Consultations (individual, group, in person, telephone, on-line, via e-mail) Consultations were essential in the work of grantees. In one case, the availability of expert advice proved very important for people willing to defend their rights. The variety of communication channels in this case made the service more friendly and allowed people with special needs or people in remote locations get and stay in touch with the consultants. 3.5.2. Case support and representation (in court and other institutions) Case support is typical for ‘traditional’ human rights work but is considered by some experts whom we interviewed somewhat alien for mainstreaming human rights. Those experts believe that mainstreaming presumes that people for the most part defend their rights themselves. The latter was often impossible according to beneficiaries we had a chance to talk to. It will be fair to say that in some cases for some people there will be a need for case support and representation. This is particularly true for the vulnerable and marginal social groups. 3.5.3. Educational activities (both traditional and not-so traditional, e.g. street information meetings) Short training programs proved very effective in informing participants on various human rights issues. Training allows a relatively large group of people to be properly informed in a short period of time. It is important to mention that this refers not only to the people whose rights are at risk or are violated. In many cases, training could be used to inform those responsible for observance of human rights – i.e. civil servants. Many project grantees are recognized as experts in their respective areas and are invited to train government staff. Such training can result in improvement of government practices, which happened for example in Perm region, received numerous requests for and conducted numerous workshops on s s, as well as improvement of They have also conducted several assessments of municipal management services. As a result of the trainings, amendments to administrative regulations proposed by were adopted. 3.5.4. Facilitation of citizen initiative groups (outreach) While the three previous types of interventions are rather universal and can be used in various domains, facilitation of citizen initiative groups is relevant only under particular circumstances: there should be a group of citizens united by a common problem willing to act and open for collaboration with an external facilitator. Skilful facilitation26 could help the group better organize its work and achieve results in a limited period of time. The center in Nizhni Novgorod effectively facilitated activities of one of the local communities to 3.5.5. Open public events (including conferences and festivals) Traditional open public events such as conferences were used by many grantees. When organized properly they allow space for a structured experience exchange and even some discussion. Conferences are ‘official’ events, hence they open opportunities to invite government representatives to participate and make presentations. The conference setting usually is not ideal for a dialogue but is very good for information exchange and networking.                                                              26 Note: in this case facilitation skills become a part of mainstreaming competencies.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 30 3.5.6. Expert discussions and think tanks Expert discussions, often called ‘round-table’ discussions, are designed for a dialogue between interested parties in an informal manner. Often such discussions are an important (and even necessary) intermediate step between informal conversations and decision-making. A good example of such an evolution was a roundtable to discuss the need for an Ombudsman in Novosibirsk organized in the end of 2010 by After a series of follow-up discussions at various levels, the process was inherited by the official entities currently working on establishing an Ombudsman position in the region27. Roundtables often create a space for a dialogue between NGOs and government. For example, the had several ‘roundtables’ with the main office of the Perm Federal Penitentiary (FPS) on the Human Rights Problems in the Institutions of the FPS in Perm. FPS staff emphasized the need to create opportunities to attract a new cadre that would be able to ensure the protection of the human rights in prisons. 3.5.7. HR-related studies (including dissemination of their results) In order to properly implement such interventions, the organization has to have research skills28 or involve professional groups, which requires additional funding. Hence, studies were implemented only by a few grantees in their respective regions. On the other hand, all the grantees participated in discussing and disseminating results of the surveys commissioned by MSI. Rigorous studies provide evidence that could be used as a basis for developing strategies and making important decisions. Such studies can contribute to the development of a culture of evidence-based discussions and decision making. 3.5.8. Monitoring of HR observance (and making proposals to authorities on necessary improvements) This is one of the ‘traditional’ activities of human rights groups and is sometimes referred to as a ‘watchdog’ function. In many cases it helps identify violations of human rights but presumes a certain level of confrontation with the government entities, which does not necessarily help mainstream human rights. For example, – organized an award ceremony “Nano-Bureaucrat and Innovative Administrative Those awards were presented to public officials for the illegal actions in 2010. During the year, HR activists collected thousands of evasive responses provided by different government institutions to citizen inquiries and complaints. Several bureaucrats were nominated to these awards and two of which were announced as the winners. Government entities were not enthusiastic about the ceremony and none of them participated in it. 3.5.9. Contributing to development of norms and regulations (drafting, discussion, expert review) The majority of the project grantees consider this intervention to be one of the most crucial in mainstreaming human rights since it helps establish the institutions needed to guarantee observance of the rights of their constituents. NGO leaders think that collaboration with                                                              27 In the three other regions Ombudsmen were informed about the project but the level of their involvement varied from very active and supportive in Perm to somewhat neutral in Nizhni Novgorod. 28Another component of mainstreaming competencies?    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 31 government entities in the capacity of subject experts is one of the most effective ways of accomplishing their mission. For example, prepared a proposal, for the City’s 2011-2013 development program. The proposal was aimed at creating a free informational environment f people. The proposal was approved by a decree of the Novosibirsk mayor’s office. 3.5.10. Serving on various bodies established by authorities (public councils, commissions, working groups, etc.) Many representatives of the project grantees serve on various committees established by the government. This allows their regular participation in the decision making process. This approach is unacceptable for traditional human rights groups that tend to keep distance from the authorities. 3.5.11. Development and dissemination of informational and educational materials (print, audio and video) This was the most popular intervention among the project grantees. That was natural since awareness-raising presumes information dissemination. There still is a huge demand for printed materials since not all the project target audiences are active internet-users. For example, all the copies of a pocket-size book published by in Nizhni Novgorod were gone in a very short period of time. Citizen guides for governmental services written in plain language developed and published by in Perm are also very popular. 3.5.12. Use of on-line instruments (sites, on-line media, social media) Information on human rights is available in the Russian sector of Internet (Runet), but the coverage of different HR areas in Runet is not even (see Figure 10). Obviously, there are considerably more links related to the consumers’ rights or the rights of children. Since Internet access will definitely further develop in Russia, the potential Internet audience for HR related content will grow and there is a huge potential for increasing the amount of Russian content related to various HR categories in Runet. For instance, the number of links related to the rights of draftees and associated with the four project regions varies from 1,200 to 2,020, which is quite small compared to the links related to consumers’ rights (between 54,000 and 89,000 links).    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 32 Figure 10. Number of rights-related links in Runet by cities (Google search results29) Grantees’ experience confirms that even their presence on the Internet in any form allows potential clients to find them. One of our interviewees said, “I have got a problem with my entered description of my problem into the Google search line and found the organization I needed in my city.” This would be more typical for the larger cities and young and middle-aged people. Elderly people in smaller cities are less likely to do the same. Grantees that have websites regularly upload the most important informational materials and publications. This is a more interactive use of websites since the users can download materials for their own use. Many grantees created special HR sections on their existing websites or even launched new websites: - (Nizhny Novgorod) launched a new section on their website devoted to issues of legal protection of Center (Nizhny Novgorod) created a section on their website with information for independent protection of rights: - Voronezh) launched a website with legal information on and online consultations rights: What is probably the most interactive internet-based activity is implemented by the Their website ( was created to let                                                              29 Originally in Russian, conducted on August 27, 2012    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 33 people who face problems related to share their problems with the virtual community and experts and receive advice/solutions. Good solutions that could be useful for many people are published in a separate section of the website. In most cases, project grantees are far less capable in terms of Internet uses than the project implementer, given its larger financial and technical resources. The website could be useful after the project is over. 3.5.13. Use of traditional media (publication of self-produced materials and working with journalists) Most grantees worked with traditional media to communicate their messages to people, and this was strongly supported by MSI. Use of mainstream media such as federal TV channels was indeed problematic, and in most cases grantees used local or regional newspapers, radio and/or TV programs. Grantees reported a considerable increase of phone calls and clients’ visits following TV programs or publications. One of the challenges NGOs faced was that even local media is rather restricted in the terms of what particular rights to talk about. Editorial policy depends on the policy of the GOR and on what could be interesting to the majority of the media audience. Hence, consumers’ rights, rights of disabled, children’s rights or rights of patients that are less politically charged are more likely to be covered than the others. Greater involvement of the mainstream media was not initially envisioned by the project as unrealistic under existing circumstances (political situation and budget constraints). 3.5.14. Infrastructural adjustment necessary to promote the observance of HR Only one activity implemented by the project grantees belongs to this category: the project, which protected electoral rights developed a number of educational materials and informational resources on the elections, social security law and other topics. Nevertheless, we decided to keep this as a separate category because several grantees that work on ssues could potentially use similar interventions. As a result of the project mentioned above, people in Novosibirsk were be able to vote t became a precedent for Russia and was appreciated by a formal letter of the Central Election Committee.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 34 4. USE OF PUBLIC OPINION SURVEYS IN MEASURING PROJECT RESULTS In accordance with the evaluation scope of work, this section discusses the effectiveness of MSI’s public opinion surveys in measuring project results. 4.1.Key characteristics of the MSI survey MSI commissioned two surveys: one in 2009-10 to collect baseline data and the second one in 2010-11 to measure the project’s progress towards its goals. These surveys were conducted in partnership with two Russian professional survey groups – 30 The surveys explored the following areas: - Priority human rights issues - Citizen knowledge of human rights - Belief and satisfaction in how well rights are protected - Why rights are violated - Citizen readiness to defend their rights - Citizen experiences with human rights violations - Appeals of rights violations - Citizen attitudes toward government and human rights group effectiveness - Citizens’ capability to protect their rights The survey used “a probabilistic stratified cluster (territorial) multistage sampling design, representing the region’s population between the ages of 18-70 (based on 2008 Rosstat data). The first stage was carried out inside each region, where an economical-geographical regionalization process (territorial clustering) took place. The second stage was conducted within these territorial clusters, where a random selection of populated settlements in accordance with the main categories was conducted according to the number of inhabitants. In the third stage, a random selection of residential areas within these populated settlements occurred. During the fourth stage, a random selection of households was conducted based on a random routing algorithm. No more than one person from each household was polled and this person was within the age quota of 18-70 years old. The control quotas were the distribution of the regions’ population according to gender, age, and type of settlement (rural or urban).”31 The sample size “from each region was approximately 2400 respondents to ensure a margin of error that is +/- 2% with a 95% accuracy rate. This allows for sufficient sample size for independent analysis of each region as well as cross-tabulations with demographic variables.”32 4.2.Mismatch of the margin of error with the scale of expected changes The surveys were used as a data collection method to measure several indicators related to the project results (see Table 3). The first survey showed that:33                                                              30 31MSI (2011). Russian citizens and their rights: priorities, attitudes and experiences (Nizhny Novgorod, Novosibirsk, Perm, and Voronezh regions of Russia). Survey Report. 32 Ibid.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 35 - Citizens’ readiness to defend their rights ranges from 40,8% in Perm region to 62,4% in Nizhni Novgorod , - citizens’ awareness of their rights ranges from 13,5% in Novosibirsk region to 25,9% in Perm region, - citizens’ knowledge of how to defend their rights ranges from 15,9% in in Novosibirsk region to 25,9% in Perm region. The expected 2-3% growth of the above mentioned values (Table 3) could result in the absolute growth of 0,3% through 1,8%. Hence, MSI survey with the margin of error of +/- 2% is not sensitive enough to measure the expected changes. Table 3. Project indicators measured through survey Expected results Indicators Progress Benchmarks for Life of Award Result 1: Observance of fundamental human rights improved Community satisfaction with HR protection Yr 3 - 3 % improvement over previous year Yr 4 - 3 % improvement over previous year IR 1.1 Citizens are willing to defend their rights % of citizens polled who say they are willing to defend their rights Yr 3 - 2 % improvement over previous year Yr 4 - 2 % improvement over previous year IR 1.2 Citizens have an awareness of their rights % of citizens polled who exhibit awareness of their rights Yr 3 - 2 % improvement over previous year Yr 4 - 2 % improvement over previous year IR 1.3 Citizens have the tools and capacity to defend their rights Percent of survey respondents who understand how to defend their rights Yr 3 - 2 % improvement over previous year Yr 4 - 2 % improvement over previous year 4.3.Mismatch of the survey sample with the population affected by the project As it has been already mentioned earlier in this report, project activities were rather fragmented: they were aimed at relatively narrow social groups and addressed particular areas of human rights in each case. Figure 11 shows the growth of this number since August 2010. The total number of grantee event participants by February 2012 was 20,503.34 While this absolute number is rather impressive, it constituted only about 0,25% of the entire population of the four regions35.                                                                                                                                                                                                 33MSI (2009). "Russian citizens and their rights: priorities, attitudes and experiences (Nizhny Novgorod, Novosibirsk, Perm, and Voronezh regions of Russia). Survey Report." 34MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA, Semi-Annual Report: September 1, 2011 – February 29, 2012, p.15 35 Of course, the total number of direct and indirect beneficiaries of the projects supported by MSI was somewhat larger. Unfortunately, it was not available for evaluators. Hence, the above mentioned number was used just to make an estimate.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 36 Figure 11. Number of grantee event participants (cumulatively) The expected change could most likely occur in the groups directly affected by the project. But there was a very low probability of inclusion of the direct project beneficiaries into a random sample since the number of such people was very small as compared to the population of the four regions. Hence, the survey sample included people who were not affected by the project and whose HR awareness and willingness to protect their rights depended on factors other than the project. In addition, some groups of project beneficiaries could not be included into the survey sample due to their specific characteristics: - the key beneficiaries of several activities implemented by the project grantees in all four regions were under the age of 18 or older than 70 and could not be included into the sample by default since the age quota for respondents was 18-70 years old; - people with disabilities (e.g. clients in Nizhni Novgorod) could not be included into the survey sample due to their special communication needs; - who benefited from the activities implemented by the region and in Voronezh could not be reached by the survey as it required special permissions; and - many clients of the could hardly be included into the survey as many of them tend to live in remote places, and avoid contacts with people whom they do not know. 4.4.Mismatch of the survey sites with the project sites In all four regions, locations of project activities were purposefully selected by the implementing NGOs. In most cases activities took place in the oblast centers and in a few other major cities. Sites for collecting survey data were selected randomly to provide sample representativeness. Hence, surveys in most cases collected data in the places where no project activities were implemented. Figure 12 is an illustration of how this worked in Voronezh oblast36. Project activities took place in 15 cities while survey data collection took place in 28 settlements. Only 5 out of 28 survey sites matched project sites.                                                              36 A table with the exact names of the settlements is included as Annex 8    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 37 Figure 12. Illustrative map of survey and project sites in Voronezh oblast Conclusions: 1) The survey was conducted by the two lead Russian professional survey groups in accordance with their task to assess citizen awareness and understanding of the current state of human, social, and civil rights in the four regions of Russia. 2) The survey was based on the assumption that the project will affect HR awareness, capacity to defend rights and willingness to do that among regular citizens in the four regions, while in reality the project was implemented so that it affected only selected narrow target groups in certain areas of human rights. 3) The survey was irrelevant to measure the actual project results: the measurement approach did not fit the project design. Recommendations37 : Disseminate and use the results of the third survey as a foundation for in-depth discussion with HRGs, CSOs and other interested parties. Do not interpret the results of the third survey in terms of the project outcomes. In the future, pay special attention to the relevance of the measurement approaches to the particular project/program design: change should be measured where it actually occurs. For the project under evaluation a number of surveys could be designed in accordance with the intended effects of each small project supported by MSI. Survey may not be the only possible approach to measuring effects.                                                              37 To MSI    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 38 5. PERCEPTION OF HUMAN RIGHTS GROUPS AMONG CITIZENS Perception of human rights groups among citizens depends not only on how well those groups perform but also on what citizens know about those groups. The majority of the citizens are not well informed about human rights groups. Since they use ‘mainstream’ sources of information, namely federal TV channels, their perception of the human rights groups and NGOs in general depends to a great extent on the ‘official point of view’ of the GOR communicated by the mainstream media channels. Hence, the current perception of HR NGOs receiving foreign (especially American) money ranges from neutral (many people simply do not care) to negative (‘foreign agents’). The perception of human rights groups dealing with children or disabled people is traditionally better than average. The survey conducted by the project confirms that HRGs are the least likely entity to whom regular citizens will appeal in the case of violation of their rights (from 2,4% of respondents in Novosibirsk to 6,3% of respondents in Nizhni Novgorod) (Figure 13). Figure 13. Survey results: to whom appeal a violation? (2011) The constituents and clients of the human right groups, people whom they helped as well as their relatives and friends, are much better informed about the HRGs that an average citizen. Their perception is based on the real life experience and is positive. Of course, the number of such people is relatively small as compared to the population of the four regions and many of them belong to the marginal social groups.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 39 6. FACTORS THAT LED CITIZENS TO DEFEND THEIR RIGHTS This chapter is based on the interviews with representatives of the human rights groups, CSOs, human rights experts, journalists and people who decided to defend their rights. There are three key factors that lead citizens to defend their rights:  obvious threat of severe violation of the rights of the person, his/her family, relatives or close friends that could put at risk property, health or life Example38: a teenager suffering from a serious chronic disease such as diabetes receives a notification that he will be recruited to the army in the nearest future. His mother immediately starts to act.  severe violation of the rights of the person, his/her family, relatives or close friends, Example: while a single elderly lady was in the hospital her apartment was sold to people who were not aware of her. She is back from the hospital and has no place to live.  information on rights person was not previously aware of. Example: Novosibirsk oblast has a regional law that allows families with three or more children to have benefits from the government until their older child is 23 in the case s/he is a student. The federal law sets up a limit of 18. When women learned about the regional law they asserted their rights they were not aware of. Additional factors that affect citizens’ readiness to defend their rights: - Individual psychological characteristics: some people are warriors by their nature, some are not. - Availability of time: protection of rights requires time. Often people decide not to get involved with defending their rights because violation is not serious enough to spend their time dealing with it. - Availability of consultations/technical support upon request. - Information on success stories. - Small victories in defending his/her rights in the past. - Availability of relatives, friends, and colleagues ready to jointly work on protection of his/her rights. These factors do not vary by region.                                                                 38 Examples in this table emerged from the data collected in the course of this evaluation,    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 40 7. UNINTENDED CONSEQUENCES AND EFFECTS OF THE PROJECT The evaluation team did not identify any significant unintended consequences and effects of the project. Many positive things that happened during the project as a result of it, such as participants’ considerable contribution to the establishment of the ombudsman institute in Novosibirsk region or letter of appreciation received by the project grantee from the Central Election Committee of Russia, were not planned initially. But they were in the area of intended rather than unintended consequences of the project. One issue could be mentioned here as an unintended effect of the project related to the project management systems. There is a general consensus among the project participants that the reporting requirements (both financial and programmatic) are too complicated. All the grantees whom we interviewed said that in various forms. Several interviewees even shared their decision not to participate in the next grant round since the reporting is “too much of a headache as compared to the amount of money potentially available.” Such a consensus among such varying organizations, some of which are mature and others with much less experienced in administrative issues, is very unique and needs to be taken into consideration. MSI has a proven record of developing and implementing administrative systems that meet all the USAID requirements, and the assessment of management systems was beyond the evaluation SOW. Nevertheless, we recommend MSI consider opportunities for simplifying reporting requirements and/or better explaining to grantees - in a friendly manner - why they are so complicated. Our superficial conversations with MSI staff in Moscow on these issues demonstrated that there might be some opportunities for improvements in this area. Conclusions: 1) The project did not have any significant unintended consequences or effects. 2) General consensus among all the grantees on the extreme complexity of the reporting requirements could be considered an unintended effect of the project in the administrative area. Recommendation39: consider opportunities for simplifying reporting requirements and/or better explaining to grantees why they have to be so complicated.                                                              39 To MSI    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 41 8. LESSONS LEARNED This chapter discusses what one can learn about mainstreaming human rights from the project experience. 8.1.Mainstreaming in the HR Assessment Report There is not a single definition of mainstreaming human rights but there clearly are several common themes involved when the mainstreaming concept is discussed by different authors. Those themes (USAID Human Rights Assessment Report, 2008) are: - deliberate infusion of human rights principles into the activities and work of government entities at all levels, - raising awareness of human rights among the masses of common people, so that through ubiquity and familiarity, the concept of observing human rights becomes normative in society, - deliberate infusion of human rights principles into various spheres of society such as education, healthcare, culture, business, etc. Raising awareness of human rights among the ordinary people is indeed closely related to infusing human rights into the mainstream sources of information such as the major TV channels. Figure 14. Model of mainstreaming human rights in Russia proposed in the HR Assessment40                                                              40 Adopted from USAID Human Rights Assessment Report, 2008    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 42 8.2. Mainstreaming in the USAID RFA The USAID RFA (2009) described the goal of this project as follows: “to ‘mainstream’ the concept of human rights, and correspondingly, the feasibility of the protection of human rights by ordinary citizens… Ultimately, the long-term objective of the project is that citizens will, in the majority of cases, be able to assert their rights without special assistance; in essence, citizens shall be the agents of change.” The RFA (2009) also clarified that “in addition to focusing on issue areas of broad public interest and support, mainstreaming in this project means bringing in local governments and other official entities in constructive roles.” The RFA (2009) included a clear statement regarding existing limitations and limited expectations: “Because of resource limitations, it is not expected that USAID’s Mainstreaming of Human, Social, and Civil Rights project can reach all actors in the area of human rights protection. At the same time it is possible for applicants to devise models, mechanisms, frameworks and procedures that can be used with targeted actors or in targeted locations, serving as positive examples for future adaptation and replication.” Hence, applicants were required to take “a needs-based approach in selecting areas of concentration and methodology.” USAID was explicit about the project priorities: “The second and third lower-level results (IR1.2 and IR 1.3), awareness and capacity, are where the majority of project activities and outputs are anticipated.” “Responsible Government authorities act to fulfil human rights obligations” in USAID RFA is both an Intermediate Result and a “critical assumption” which is likely beyond the manageable interest of the applicant to achieve, although attention to stakeholder buy-in is part of the required approach.” In fact, citizens’ HR awareness and their activity became synonyms of HR mainstreaming in the context of this particular RFA. USAID also proposed to select areas of concentration by identifying the burning issues with a large community impact, which implies modelling mainstreaming HR at the community level. Hence, the approach proposed by USAID could be entitled “Modelling HR mainstreaming at the community level in Russia: raising citizens’ awareness and building their capacity to protect their rights.” 8.3.Possible interpretations of USAID RFA USAID internal process of preparing RFA was rather comprehensive; internal understanding of the mainstreaming concept was rather clear, plus there were numerous meetings with NGOs representatives, where there was a mutual understanding of a mainstreaming concept. Nevertheless, the proposed concept was rather innovative - at least for Russian HR NGOs - and allowed various interpretations. This is not a criticism of RFA, innovativeness is a core characteristic of its nature. Prior to the project human rights issues were actually at the margins of life of Russian society, were not incorporated in the activities of the Russian government in a systematic manner and were not recognized as really important by Russian people. Hence, the mainstreaming concept proposed in USAID RFA was rather new to most Russian HR NGOs.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 43 Our conversations with HR experts, MSI staff and project partners in Moscow and in the regions showed that there could be at least three interpretations of the approach presented in the USAID RFA: 1) “PR Campaign” In order to empower citizens to assert their rights money should be invested into a region￾wide PR campaign (in each of the regions). Project implementer will need to buy space (newspapers) and time (TV, radio) in the mainstream sources of information to communicate HR-related messages to the general public. This will raise HR awareness of regular citizens. Human rights experts the evaluation team consulted with think that such approach will allow achievement of the desired results in the short term but will not be sustainable. Another negative consequence of this approach could be creation of a wrong attitude among representatives of the mass media: ‘publications on human rights issues are well paid for.’ The latter may cause serious challenges for public campaigns in the future. 2) “Complex program with a focus on the human rights area the majority of population is most interested in41” The project implementer should assess the interests of the general public in the HR￾related areas and choose one or two focus areas that prove to be of the most importance to the majority of population in one or more project regions. Then a complex program should be designed for each region to raise citizens’ awareness in the selected area of human rights, building citizens’ capacity in that area and motivating citizens to act in order to improve situation. This approach will require serious design work in each of the participating regions and careful selection of potential partners among NGOs, CSOs, HRGs and other entities that can contribute to the project activities in the selected area of human rights. The fact that this area of HR will be important for the majority of population will allow collaboration with the mainstream sources of information (media) at no cost or at low cost and with the local authorities. 3) “Small scale projects aimed at mainstreaming HR in various areas and creating models for the future use” Select domains where, on the one hand, there is a demand to protect human rights and, on the other hand, there are local CSOs and/or HRGs that can or already are working on that issue. Provide financial, informational and technical support to those groups to model mainstreaming human rights in their selected domains. Assess effectiveness of mainstreaming in each case, select and describe the most effective models, disseminate information beyond the region for possible use under similar circumstances by similar groups and individuals. This approach can not result in a massive change in the HR awareness of the general public. Certain changes could be found only in the domains where the project interventions will be implemented. This approach requires serious analytical work to assess and describe the models of mainstreaming. If RFA is interpreted this way, it has an implicit contradiction: applicants develop effective models of mainstreaming human rights (at the community level or in their respective domains), but measure progress by the overall growth of the citizens’                                                              41 Another possible title for this approach would be “Use of various approaches to mainstreaming, while focusing on one issue that appeals to the broad public” in a coordinated manner.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 44 awareness, motivation and capability to protect their rights, rather than by assessing effectiveness of each model and considering the number of effective models of mainstreaming. It is really hard to compare the three above mentioned approaches in the terms of their “effectiveness” without deciding what exactly is meant by effectiveness. In principle, the first approach will most likely allow to increase HR awareness of the regular citizens in a short term. It will be less effective in enabling citizens to protect their rights. The second approach may considerably increase HR awareness of the regular citizens in particular area and enable citizens to protect their rights. The third approach may allow developing and piloting a variety of mainstreaming models in various domains for different target groups. 8.4.Mainstreaming in the MSI proposal Our analysis revealed that the MSI proposal combined the 2nd and 3rd interpretations. The MSI proposal put an emphasis on enlightenment and empowerment of ordinary citizens and groups impacted by discrimination: “The critical element that sets this program apart from previous efforts is the concept of “mainstreaming.” It is recognized that supporting organizational capacity building of human rights groups and other NGOs is not sufficient by itself to achieve improved human rights observance and action. Ordinary citizens and those groups impacted by discrimination and intolerance need to be enlightened and empowered. This needs to be the job of Russian human rights groups and interested NGOs, but new strategies and approaches must be employed. This is where MSI’s proposal provides a new engine for change.” The MSI approach included: - Strategic targeting of four pilot regions based on key criteria. - A strategic communications model that helps human rights and other civil society groups target particular constituencies as well as the broader population with human rights messages. - A wide range of advocacy and public education activities that will capture the attention of the broad public and … will be a training ground for citizens to test how they can make constructive demands and act collectively to defend and strengthen their rights. - Demonstrating the value of such activities is essential to dampen the public’s cynicism. - Buy-in by regional authorities is also critical. - Monitoring and feedback methods. - Strategic alliances with the Moscow Helsinki Group, as well as the Youth Human Rights Movement and the Association of Regional Human Rights Organizations and cooperation with regional civil society organizations. Interestingly, MSI included work with regional authorities although it was not required by USAID. They even included an explicit goal (goal 5) to “facilitate constructive dialogue between human rights groups and the government to develop and strengthen institutions that allow citizens to assert their rights42.” In general USAID RFA’s description and parameters were sufficiently explained to enable MSI to develop the project approach accordingly taking into account the country and regional context.                                                              42 This goal was not actually achieved though    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 45 Although, the overall approach proposed in USAID RFA was considered and operationalized in the MSI proposal, the notion of modeling HR mainstreaming (developing, implementing and evaluating HR mainstreaming models in certain domains/communities) is not present in the MSI proposal at all. MSI aims at achieving certain effects at the regional level in the four selected regions (see Table 4). The implicit contradiction here is: in order to achieve effects at the regional level the project needs to have very broad coverage (see interpretation 2 of the RFA); but in order to implement new creative approaches to mainstreaming HR in various domains the project has to support a number of small-scale activities aimed at different relatively narrow target audiences43 (interpretation 3). Of course, interpretation 2 also requires creativity. It is an extremely challenging and innovative approach targeting the majority of population (ideally) or its considerable part. Hence, all the activities under this approach should be well coordinated, harmonized and contributing to one common overall goal. Interpretation 3 does not require such coordination and harmonization: each activity should be targeted at a particular audience and aimed at achieving results meaningful for that audience. Given the budget limitations and relatively short project duration, both approaches (2 and 3) cannot be implemented at the same time. Table 4. Indicators proposed by MSI to measure progress towards IR1.1 and IR1.2                                                              43 By relatively narrow target audiences we mean groups of intended direct beneficiaries of the small projects supported by MSI in the 4 regions. Each small project had a modest budget under $15,000 and had to be implemented in 12 months or less. Under such limitations none of those projects could pretend to considerably affect general population of their respective regions. It is important here to make clear distinction between potential availability of services to the general population of the region (such as a website open for anyone) and actual impact on the general population (such as changing awareness of certain aspects of human rights).    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 46 8.5.Mainstreaming in the real life of the HR project Activities implemented by the project and its partners in some cases successfully implemented mainstreaming human rights in the selected domains such as communities or vulnerable groups. The following projects are illustrative examples of such activities:  in Perm was established by parents who have such kids. This NGO was extremely instrumental in identifying the existing problems particularly related to the areas of education and healthcare, exploring the root causes of those problems and developing strategy for solving the problems. Their goal was to incorporate special approaches to into the everyday work of government entities at the regional level. They succeeded in promoting that idea. Hence, in this case a self-organized group of people with active support from professional communities (educators and healthcare professionals) and in collaboration with the regional government entities achieved the following: (a) deliberate infusion of a particular set of human rights principles related to the into the work of government entities at the regional level, (b) raising awareness of human rights among the regional community of parents who have professionals who serve in the region and government officials involved in this work so that the concept of observing parents’ rights and rights of is becoming normative in the region, (c) deliberate infusion of a particular set of human rights principles related to the into education and healthcare at the regional level.  has a long history of work with institutions in the region and beyond. In the course of the project under evaluation the Center conducted “round tables”, training sessions and consultations for the management and staff of the institutions in the Perm region on how to implement international human right standards to guarantee observance rights. In this case a professional NGO helped government entities infuse a particular set of human rights principles into their work. This is fully in line with one of the most common definitions of mainstreaming HR44. Once again, in this case we see narrow target group of beneficiaries, particular set of human rights, and clearly defined group of institutions involved.  Voronezh implemented a project aimed at protection of rights of people They identified cases when such rights were violated and successfully defended by citizens, described those cases and published a book that included cases with coordinates of people who can serve as consultants on each case. Expectation is that citizens who face problems similar to the ones described in the book will use the book as a ‘manual’ on how to protect their rights and will contact                                                              44 Interestingly, some HR experts would not consider this as an example of mainstreaming HR. For instance, Mr. Bakhmin who was part of the team that conducted this evaluation argues that mainstreaming is “an approach that results in HR awareness, acceptance and observance in various areas of social life, especially in those areas that are related to the everyday activities of regular citizens. Two important features of mainstreaming approach are: (1) it empowers and encourages common people to defend their rights and advocate their interests themselves, and (2) it inclines authorities to recognise human rights and observe them.” This project does not fit Bakhmin’s criteria of mainstreaming as its direct beneficiaries are not involved in defending their rights.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 47 volunteer consultants if needed. This project developed a model of involving more and more people in protecting their rights and created conditions for mainstreaming human rights at least in the communities covered by project and potentially beyond those communities. In some cases MSI grantees implemented activities that could potentially be considered as essential elements of mainstreaming human rights but under certain conditions. The following project is an illustrative example of such activities:  has implemented a project One of the project goals was to promote the citizens’ right to have about the work of the government. Access to information is an essential condition of HR mainstreaming. Legal assistance can be one of the mainstreaming tools when it’s included into a broader context and coordinated with other activities. It should empower people to protect their rights on their own. Professional legal assistance as such cannot be considered mainstreaming. In this project most activities were implemented by NGO staff (see the project title), while regular citizens remained in a consumer position. would be a more relevant title and approach for a mainstreaming project. In some cases project partners in the regions implemented activities that were aimed at defending people’s rights and even saving their lives but could hardly be considered mainstreaming. The following projects are illustrative examples of such activities:  protects the rights of who live and work in the region. Specialists of the in many cases literally save the lives of their clients, but it has nothing to do with mainstreaming human rights.  is a very well-known and highly respected NGO with a clear mission to provide high quality professional legal services to at the national and supranational levels. does an excellent job in the HR area that is not related to mainstreaming HR. Public opinion polls conducted in the four regions in a very professional manner showed no considerable changes in the citizens’ HR awareness or willingness to defend their rights. The public opinion polls were designed to measure parameters that could not change due to the project interventions and did not measure smaller scale changes that did occur.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 48 The project has rich material on various approaches to mainstreaming HR, and there is enough time until the end of the project to reflect on the existing experience and explicitly describe those approaches as models. It will be extremely useful if the project team together with its partners and grantees could contribute to defining the concept of mainstreaming HR in Russia, proposing mainstreaming criteria and developing recommendations on the possible future strategies in that area. Conclusions: 1. USAID RFA did not include a crystal clear definition of mainstreaming HR. There were several possible ways of interpreting it. 2. Evaluation data and the discussion of the draft of this report demonstrated that there is still no mutually agreed understanding among the parties involved and HR experts of what mainstreaming HR is and what the criteria of mainstreaming HR are. 3. MSI proposal included an implicit contradiction between the intent to implement mainstreaming approaches to a wide spectrum of rights at a smaller scale in number selected domains and the suggested way to measure the project progress by the overall growth of the citizens’ awareness, motivation and capability to protect their rights in the four regions. 4. This contradiction resulted in emergence of a gap between the actual project results and what the project measured as the results. 5. The project succeeded in supporting and implementing a number of successful approaches45 to mainstreaming human rights46 in various domains, but the existing experience is not conceptualized yet. 6. Some (minority) of the activities supported by the project (grants) can hardly be considered mainstreaming. Recommendations47 : Reflect on the existing experience and together with the local partners explicitly describe effective models of HR mainstreaming. Describe possible strategies of mainstreaming HR in various domains. Based on the successful models provide definition(s) of mainstreaming HR and criteria for assessing mainstreaming HR effectiveness. Based on the existing experience describe competencies NGOs should have to mainstream human rights in Russia. This might become an important contribution to the international body of knowledge in the HR area.                                                              45 This statement is based (a) on the analysis of project documentation (grantees’ reports in particular) and (b) on the results of the interviews. 46 By ‘mainstreaming’ here we mean deliberate infusion of human rights principles into the activities and work of government entities in certain domains; raising awareness of human rights among the target audiences in those domains; deliberate infusion of human rights principles into the work of respective professional organizations. 47 To MSI    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 49 9. MAIN CONCLUSIONS AND RECOMMENDATIONS This chapter includes the most important conclusions and recommendations related to the project in general. More specific conclusions and recommendations were included where relevant into each chapter of this report. 9.1. Main conclusions USAID internal process of preparing RFA was rather comprehensive; internal understanding of the mainstreaming concept was rather clear, plus there were numerous meetings with NGOs representatives, where there was a mutual understanding of a mainstreaming concept. At the same time USAID RFA did not include a crystal clear definition of ‘mainstreaming human rights’ concept. There were several possible ways of interpreting USAID RFA such as48 calling for: 1) PR Campaign. 2) Complex program with a focus on what the majority of population is most interested in. 3) Small scale projects aimed at mainstreaming HR in various areas and creating models for the future use. The MSI project proposal combined the 2nd and 3rd approaches. It resulted in an implicit contradiction between the intent to implement mainstreaming HR approaches at a smaller scale in several selected domains and the suggested way to measure the project progress by the overall growth of the citizens’ awareness, motivation and capability to protect their rights in the four regions. Indicators used by MSI to measure project’s progress do not consider the notion of selected domains and refer to the general public – adult population of the four regions. The project succeeded in promoting awareness of relatively narrow target groups in all four regions about specific human rights that varied depending on the groups’ interests. A number of effective approaches to mainstreaming human rights in various domains were implemented in the course of the project, but the existing experience is not conceptualized yet in the form of mainstreaming models. There is enough time before the end of the project to do that. The project succeeded in helping vulnerable and marginal groups of people and in some cases literally saved people’s lives. The project was not designed to affect the human rights awareness of the regular citizens’ or their capacity to defend their own rights in the four oblasts and naturally did not succeed in that. The project succeeded in facilitating networking among grantees. The number of connections between participating NGOs considerably increased in all four regions in the course of the project. The project has a very informative and well-designed website that can be maintained and used after the project is over. Use of the surveys in measuring project results was based on the assumption that the project will affect HR awareness, capacity to defend their rights and willingness to do that among regular citizens in the four regions, while in reality the project was implemented so that it affected only selected narrow target groups in certain areas of human rights. Hence, the surveys turned out to                                                              48 See Chapter 8 of this report for more details    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 50 be irrelevant to measure the actual project results: the measurement approach did not fit the project design. Public opinion surveys were conducted by the two lead Russian professional survey groups in accordance with their task to assess citizen awareness and understanding of the current state of human, social, and civil rights in the four regions of Russia. The surveys provided interesting information related to human rights in the four project regions and beyond. This information was disseminated and discussed by the project participants and other interested parties, which contributed to the better understanding of the state of human rights and priority areas for mainstreaming human rights. The project under evaluation was innovative. Innovations are different from standard projects and programs. One of the unique features of innovative initiatives is that it may not be clear form the very beginning if the project or program will work as expected and how it should work in general. Innovative programs should be highly flexible and adaptive. According to Michael Quinn Patton, “complex situations challenge traditional evaluation practices. Complexity can be defined as situations in which how to achieve desired results are not known (high uncertainty), key stakeholders disagree about what to do and how to do it and many factors are interacting in a dynamic environment that undermine efforts at control, making predictions and static models problematic49.” Traditional evaluation approaches to evaluation of innovative programs (such as mid-term and final evaluations) may not be as effective as needed. 9.2.Main recommendations Recommendation 1. Pay special attention to communicating innovative concepts50. In the future make sure those innovative concepts, such as ‘mainstreaming human rights’, are properly communicated to the applicants and implementing partners. To transfer innovative concepts to the implementing partners use comprehensive communication approaches that include various channels such as expert ‘round tables’ (with participation of external experts), group discussions and documentation exchange. Double check that the key concepts are interpreted by the implementing partners properly. Recommendation 2. Use most relevant evaluation approaches to measure progress and results of innovative initiatives. 51 Evaluation of innovative projects should differ from the traditional one. To make the evaluation most useful consider contemporary approaches to evaluation of innovations such as developmental evaluation that applies complexity concepts to enhance innovation and support evaluation use52. Incorporate evaluation into the program as an on-going process. Recommendation 3. Conceptualize project experience in the form of mainstreaming models, mainstreaming criteria, and mainstreaming competencies53. Reflect on the existing experience and together with the local partners explicitly describe effective models of HR mainstreaming. Describe possible strategies of mainstreaming HR in                                                              49 http://aea365.org/blog/?p=1298 50 To USAID 51 To USAID 52 See Annex 9 for more details. 53 To MSI    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 51 various domains. Based on the successful models provide definition(s) of mainstreaming HR and criteria for assessing mainstreaming HR effectiveness. Based on the existing experience describe competencies NGOs should have to mainstream human rights in Russia. One of the key tasks here will be finding an equivalent of the term ‘mainstreaming’ in Russian. Recommendation 4. Consider limitations of the possible use of the surveys in measuring project results54. Do not interpret the results of the third survey in the terms of the project outcomes. In the future pay special attention to the relevance of the measurement approaches to the particular project/program design. Disseminate and use the results of the third public opinion survey planned under this project as a foundation for in-depth discussion with HRGs, CSOs and other interested parties. Recommendation 5. Consider creation of a Virtual Resource Center on mainstreaming HR in the MSI exit strategy55. Find a successor among Russian NGOs (preferably – among project grantees and regional coordinators) who can further maintain the project website and/or incorporate its content into some other website. Consider an idea of creation of a virtual resource center56 specialized in mainstreaming human rights. This resource center should be established in collaboration with the local partners (possibly, regional coordinators) and further maintained by them.                                                              54 To MSI 55 To MSI 56 Virtual resource center (VRC) is a website that provides useful cases, articles, links, etc. on a particular topic. VRCs are often designed to support learning process and make self-education most convenient and effective. In fact, the existing project website can be maintained and positioned as a VRC in the future.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 52 Annex 1. Evaluation Scope of Work (Notification of Performance Evaluation) Performance Evaluation Notification #2 USAID/Russia Human Rights project USAID Project/Activity to be Evaluated: Human Rights project(I’ve got Rights: Mainstreaming Human, Social, and Civil rights activity)   Initial and Final Funding Years: FY 2009 – FY 2013 Type Evaluation:               __X__    Formative             _____     Summative                  _____     Impact (Post‐Facto) Purpose and Intended Uses of the Evaluation: The research purpose is to assess activity progress to‐ date in order to allow the team to make any necessary adjustments or amplify particularly successful interventions.   Key Stakeholders: Director of the Office of Democratic Initiatives, Rule of Law and Human Rights Team, Human Rights Design Team, Russian Human Rights community, International and Russian Donors Revision:  New Date:  May 14, 2012 Design team:                                    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 53 Brief Description of Project and its Intended Results: The central intent of the Mission’s Human Rights (HR) project is to strengthen monitoring of and advocacy for human rights in Russia.  To reach this goal, the Results Framework of the project posits that there are two key intermediate results that must first be achieved.  These are: - Citizens must assert their fundamental rights; and, - Authorities must fulfill their obligation to enforce and protect these rights. There are multiple implementers within the overall Rule of Law (ROL) and HR portfolio in the Mission and this HR project is pursuant to the ROL and HR design document approved in June 2008.  The “Mainstreaming of Human, Social and Civil Rights in Russia” activity, awarded to Management Systems International (MSI) following a Request for Applications, is the Mission’s largest human rights activity in terms of financial resources.  The project design proceeded from the assessment that a disconnect existed between the human rights community and the rest of Russian society, and that a majority of Russian citizens discounted the relevance of human rights in their daily lives.  In the course of achieving the result of improving the observance of human rights, the activity was intended to help “mainstream” the concept of human rights among a broader public. The design of the activity imposed responsibility on the implementer for the first intermediate result‐‐that citizens assert their rights.  The continued existence of a legal framework requiring authorities to enforce and protect rights was a critical assumption of the project’s Results Framework.  However, the implementer was not responsible for ensuring that authorities fulfill their obligation to enforce and protect citizens’ rights. This formative evaluation is intended to identify and quantify the degree to which the interventions of the MSI activity have successfully empowered and mobilized citizens to assert their rights.  The results of the study will enable the Mission and its partners to make any adjustments to the remainder of the project and overall program.  The Mission focused this evaluation on the MSI activity both because it is the largest human rights activity in the Mission’s portfolio; and because of the important issues it raises regarding whether and how demand for rights enforcement can be increased among the public. Per the new Agency Policy for Evaluations issued on January 13, 2011 “each Mission is required to conduct at least one performance evaluation of each large project it implements.” For these purposes, a “large project” is one that equals or exceeds in dollar value the average project size for the operating unit.  The HR activity and project fall into this category.  Therefore, USAID/Russia tasks International Business & Technical Consultants, Inc. (IBTCI) with this evaluation.  IBTCI maintains the Mission’s overall independent evaluation services contract.    As a foundation to form an integral analysis, the Mission provides IBTCI the Project Results Framework and key project indicators in Annex 1.  While this is not an exhaustive list of all the project indicators, these key indicators provide the framework from which to start a work plan for the evaluation.  IBTCI will find the full list of indicators in the HR Project Design, MSI Work Plans, MSI Monitoring and Evaluation (M&E) plans, MSI Program Semi‐Annual Reports, and MSI Program Monitoring Surveys.  The key indicators to use as the foundation for analysis are:    Community satisfaction with human rights protection (percent of citizens polled who believe that their rights are protected).  Number of human rights court cases filed in target regions by project‐supported legal services groups.  Percent of citizens polled who say they are willing to defend their rights.  Percent of citizens polled who exhibit awareness of their rights.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 54  Number of human rights groups and civil society organization representatives with improved knowledge and skills.  Percent of survey respondents (among those whose rights were violated) who understand how to defend their rights.  Number of domestic human rights NGOs receiving US Government support through the project.  Percent of human rights cases brought by legal services groups to appropriate government offices that are resolved.  Number of advocacy campaigns supported by the US Government. Description of the activity implemented under Human Rights project: I’ve got Rights: Mainstreaming Human, Social and Civil Rights Activity Dates: August 2009 – August 2013 Implementing Partner: MSI Regions: Perm, Novosibirsk, Nizhniy Novgorod, Voronezh Activity Purpose: This activity aims to increase the knowledge of Russian citizens about their human, social, and civil rights and to create conditions that enable them to effectively protect those rights. Activity Description: The major objectives of the activity are to: •  Promote citizen awareness about human, social and civil rights, as well as mechanisms to effectively defend them; •  Strengthen the skills and capacity of civil society organizations to educate and reach out to citizens; •  Collect and broadly disseminate information about Russian citizens who successfully defended their rights; •  Facilitate networking among human rights and other civil society organizations; •  Facilitate constructive dialogue between human rights groups and the government to develop and                 strengthen institutions that allow citizens to assert their rights. Contact Information:   600 Water Street, SW, Washington, DC 20024   Tel: (202) 484‐7170 Fax: (202) 488‐0754 www.msiworldwide.com http://vprave.org/ Moscow Office: 7 Radio Str., Building 1, Suite 404 Tel: (495) 640‐1479 Information about regional partner NGOs can be found in program reports and other materials that will be available for the evaluation team.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 55 Existing Performance Information Sources: The following performance information sources will be available for the evaluation team: Human Rights Project Design, MSI Work Plans, MSI M&E plans, MSI Program Semi‐Annual Reports, MSI Program Monitoring Surveys, and any others that the team may deem essential for this evaluation.    Evaluation Questions: The evaluation team members will provide answers to the core evaluation questions below through quantitative and qualitative research methods.  The questions that IBTCI will address are presented below in priority order. Other questions may be developed by IBTCI to provide context or a more robust evaluation. Core evaluation questions:   1. How effective was the methodology from MSI’s 2009 and 2011 public opinion survey “Russian citizens and their rights: Priorities, attitudes and experiences” in explaining deviations from expected results?    Review and assess the methodology for the surveys.  Provide recommendations if the methodology needs to be adjusted for the final survey. 2. What is MSI’s progress in achieving the goals/objectives as defined in the Program Description, Work Plan, and M&E Plan, including but not limited to program indicators, Activity Objectives and Expected Interim Results? What is the evidence (documentary, key informant interviews, other data) that supports a deviation from expected results?  Which objectives are particularly lagging/progressing and why? 3. What is the effectiveness of the activity’s interventions as defined by the Program Description, Work Plan and M&E plan?  Which ones are the most successful and beneficial to the target audiences? 4. How effective are HR advocacy groups in the eyes of citizens?  If there is a negative perception among citizens of human rights groups, what can be recommended to counteract negative citizen perception toward human rights groups? Is there a positive perception?  If yes, what event or piece of information helped form that perception? 5. Based on analysis of the MSI survey and other qualitative data, what were the principal factors that led citizens to defend their rights, and how do those vary by region?   Lessons Learned : 6. What are the key lessons learned that could affect future programming and potential regional dissemination?  What were the unintended consequences and effects of the activity? 7. Are there any key adjustments that need to be made to the project now in order to ensure that goals are met? If so, what are they?  How can they be introduced at this phase? Evaluation Methods: The evaluation will use a combination of data collection and analysis methods which must include project‐ related document review, secondary source data review, media scans, baseline and actual program monitoring data review, MSI program survey report reviews, and site visits. The evaluation team will plan site visits to all of the four project regions – Permskiykray, Novosibirskaya oblast, Nizhenovgorodskaya oblast, Voronezhskaya oblast.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 56 IBTCI will use a mix of various research methods as appropriate; including, but not limited to: briefing sessions; focus groups; in‐depth interviews with grantees, stakeholders and beneficiaries; program results review, monitoring survey review; social network analysis (spider‐webbing); analysis of mapping coverage of HR group outreach; analysis of data collected through MSI baseline and follow‐on public opinion surveys; and other evaluation methods as necessary.  The evaluators will propose other evaluation methods.  The IBTCI Contracting Officer’s Representative (COR) will review and approve these methodologies before field work starts.  The design plan will indicate the advantages and disadvantages of each method utilized by the evaluation questions.  The design plan will note where cost factors may prohibit a method with strong advantages, or otherwise indicate data limitations to the chosen method(s). Deliverables:   1. Evaluation Design and Work Plan: Draft due 30 days from this written notification.   The Contractor will develop a written Evaluation Design and Work Plan for the evaluation defined herein and submit it to the IBTCI COR at USAID/Russia for approval. The Design will include, but is not be limited to: • Methodology: Strengths and limitations of IBTCI proposed methodology.  IBTCI will also provide an alternative methodology reflecting a scenario where in‐country travel is restricted or limited. • Data plan: Sources, instruments and procedures for collecting, monitoring, storing, transmitting, and reporting information; as well as documentation of data quality (validity, reliability, timeliness, precision, integrity). • All evaluation questions: Include all questions to be addressed and which individuals and/or groups that IBTCI plans to interview. • Data Analysis Plan   • Local partners and stakeholders • Briefings and Dissemination plan The Work Plan will include, but is not be limited to: • Evaluation Team composition: Titles, names, level of effort, roles, responsibilities, qualifications, and identification of subcontractors. The work plan will include language: “The evaluation team members are considered Key Personnel for this evaluation and that no Key Personnel will be removed from the team nor will the level of effort, roles, or responsibilities be changed without prior written approval of IBTCI COR”. • Schedule: including information collection, interviews, site visits, travel/logistics, and the evaluation team should consider the possibility of simultaneous data collection by sub‐teams, given the geographic scope of this evaluation. • Preliminary logistics: Information including criteria and site selections; planned meetings with host country government officials at the regional and local levels; planned meetings with USAID/Russia contractors and grantees implementing Human Rights activities in‐country; planned meetings with important human rights NGOs at selected sites; and meetings with USAID program staff to update progress of the evaluation (to be finalized in concert with implementation partner team and subject to the availability of stakeholders).  IBTCI will also provide an alternative methodology reflecting a scenario where in‐country travel is restricted or limited. • Stakeholder review mechanism: for the Evaluation Briefing and Draft Evaluation Report. • Deliverables schedule • Budget The Design should be shared with key stakeholders including implementing partners to receive their feedback and comments. The Design and Work Plan requires written approval from IBTCI COR at USAID/Russia prior to any international travel.  Any changes to the Evaluation Design and Work Plan must be approved in advance in writing by IBTCI COR at USAID/Russia followed by formal change/revision to the document.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 57 2. Discussion of Preliminary Draft Evaluation Report: the team will present key preliminary findings to the USAID/Russia evaluation design team who will provide comments prior to the final Mission de‐briefing.  IBTCI will present and review with USAID/Russia a proposed outline of the Final Evaluation Report.    3. Draft Evaluation Report: due 15 working days following the briefing on preliminary key findings. The Contractor will incorporate comments from USAID/Russia and key stakeholders made during the discussion and submit a written Draft Evaluation Report for the evaluation to USAID/Russia for approval. The draft report will include all sections of the Final Evaluation Report (see below) except that the annexes may merely be referenced and need not be included; the electronic file of data may merely provide the definition of readable format and structure of organization, and; the submission of the final report to the Development Experience Clearinghouse may simply be stated.   The evaluation report should represent a well‐researched and well‐organized effort to objectively evaluate what worked in the project, what did not and why. Evaluation findings will assess outcomes and impact on males and females.   USAID/Russia will provide comments/recommendations to the Draft Evaluation Report within 15 working days after receipt of the document.   4. Evaluation Briefings: due at the start and at the end of the evaluation field work.  IBTCI will present major findings through a Power Point presentation to USAID/Russia, and/other stakeholders, and HR partners while in‐country for the evaluation defined herein.  The team will consider USAID and partner comments and revise the draft report accordingly. 5. Final Report: due within 15 working days from receipt of written comments by USAID/Russia on the Draft Evaluation Report. The Contractor will submit a Final Evaluation Report for the evaluation to IBTCI COR at USAID/Russia for approval. Each Final Evaluation Report will be submitted as 5 hard copies and 1 electronic copy, with all annexes. The report will adhere to the following format: • Cover Sheet: with a descriptive title that includes the Activity Description and Implementing Partner being evaluated, author's name(s) and title(s), contract number, Contractor's name, date of publication, revision number, and any other report requirements as per the Contractor's statement of work and contract terms and conditions. • Table of Contents: including list of figures and tables. • Executive Summary: will state the purpose, background, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned. The Executive Summary will be appropriate for broad dissemination, and will be translated into Russian language with the Russian text included in the document. The English version of the Executive Summary will not exceed 3 pages. • Body of the report: will include thorough descriptions of the justification for the evaluation; clarify the theory of change, a description of the evaluation design and limitations; the environment in which the team operated; the methodology used including the adherence to data quality standards (ADS 203 and 578) and limitations, a data quality assessment, the data analysis elements, and statistical significance (as applicable given the methods); as well as the major findings, conclusions, and recommendations against the Partner SOW, Work Plan, M&E Plan objectives and deliverables; as well as lessons learned that will be useful for future activities. The body of the report will not exceed 30 pages (more detailed analysis should be included as separate annexes that will buttress the more abbreviated analysis mandated by this page limit and should be referenced in the main body of the evaluation report. • Glossary: of terms and acronyms. • Annexes: a separate annex will include, but is not limited to: scope document, the Evaluation Design, Evaluation Work Plans, lists of persons interviewed (name, title, company and date of    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 58 meeting), source materials, tools used, background supplemental materials useful for a fuller understanding of the report, a bibliography of significant documents used to consult, and any site visit photos. Development Experience Clearinghouse: due 30 calendar days after USAID/Russia approval of the Final Evaluation Report. The Contractor will make publically available the Final Evaluation Report for each evaluation activity through the Development Experience Clearinghouse and confirm this in writing to USAID/Russia. All deliverables will be submitted in writing in the English language.  The Executive Summary will be provided in both Russian and English. Evaluation Team Composition: Evaluation team members are Key Personnel as identified in and governed by the Evaluation Work Plan for the evaluation defined herein. The Contractor will define the evaluation team in the Work Plan, such description will include, but is not limited to: • team member title • name • roles and responsibilities to include, but not limited to, the design and management of the evaluation, and evaluation deliverables; • qualifications and experience to include, but not limited to: USAID programs; Human Rights program design, implementation, management; Rule of Law program; M&E; Russian democracy, government, civil society, economy, reforms, and language. The evaluation team may be accompanied and reinforced by USAID/Russia and/or USAID/Washington staff as available during the evaluation, including site visits in Russia (although those costs will not be the responsibility of the evaluation contractor). All evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated; this statement will be signed prior to beginning work on this evaluation.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 59 Procedures and Logistics: 1.Responsibilities of USAID/Russia: USAID/Russia Program and Project Development (PPD) office will oversee and approve all aspects of the evaluation.  Formal USAID/Russia direction and approvals will be communicated in writing via IBTCI COR.  A member of USAID PPD office may accompany team members on any or all visits in‐country.   USAID/Russia will be responsible for obtaining country clearances prior to travel subsequent to a request from the contractor with the pertinent information.  USAID/Russia may provide limited assistance if needed in scheduling appointments for team members with host government officials, USAID/Russia contractors and cooperating agencies and other organizations doing work in this area in Russia related to this SOW, although the team may reschedule or propose additional meetings as needed.  All meetings scheduled by team members relating to this SOW should be cleared through USAID/Russia. USAID/Russia Key Technical Program Contacts in the Office of Democratic Initiatives: 2.  Responsibilities of the Contractor:  The contractor will be responsible for all pre‐trip travel arrangements and logistics including visas, insurance, etc., and in‐country meetings and site visits. The contractor will be responsible for coordination, issuance and dissemination of the draft and final report. 3. In-Country Meetings: While in Russia, the team will meet with the following individuals and groups as appropriate:  In‐Brief and Exit Brief with USAID/Russia Front Office.  USAID program staff on a regular basis to update on the progress of the assessment as requested by USAID.  US Mission/Russia interagency officials working in human rights areas.  Host country government officials at the regional and local levels, as appropriate.  USAID/Russia contractor and grantees implementing “I’ve Got Rights” activities in‐country. 4. Schedule: The following is the proposed schedule for the team’s visit: one week of preliminary document review prior to field work; two weeks of data collection and field work; one week of data analysis; one week of preparation briefing on preliminary findings; and three weeks of final report preparation. The team will be expected to work six-day workweeks, including travel days. USAID/Russia requests that work commence in May 2012.       Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 60 Annex 2. Evaluation Design and Work plan EVALUATION DESIGN Purpose and Intended Uses of the Evaluation This is a formative evaluation of the USAID/Russia Human Rights Project57 “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” implemented by Management Systems International (MSI). International Business and Technical Consultants, Inc. (IBTCI) is responsible for implementing this evaluation as per the terms of the Russia Monitoring and Evaluation Project (RMEP) and based on the Notification of Performance Evaluation #258 sent to IBTCI on May 15, 2012. The purpose of this evaluation is to assess activity progress to-date in terms of the degree to which MSI’s activities have successfully empowered and mobilized citizens to assert their rights. It will also allow USAID and the project team to make any necessary adjustments or amplify particularly successful interventions, as well as to inform the design of the USAID/Russia future programming in the field of human rights. The evaluation will be conducted in accordance with the USAID Evaluation Policy adopted in January 2011. Key Stakeholders Key stakeholders of this evaluation include the USAID/Russia Director of the Office of Democratic Initiatives, the USAID/Russia Rule of Law and Human Rights Team, the USAID/Russia Human Rights Design Team, the “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” project team, the Russian human rights community, and international and Russian donors. Description of the Project to Be Evaluated The Human Rights Project “I’ve Got Rights: Mainstreaming Human, Social, and Civil Rights” is part of the USAID/Russia Rule of Law (ROL) and Human Rights (HR) portfolio and is pursuant to the ROL and HR design document approved in June 2008. The “I’ve Got Rights” Project was developed by MSI in response to Request for Applications (RFA) No. USAID-Russia-DI-09-0001-RFA –Mainstreaming of Human, Social, and Civil Rights Project in Russia issued in February 2009. The “I’ve Got Rights” project is scheduled to operate from August 2009 to August 2013 in the following four regions of Russia: Perm, Novosibirsk, Nizhniy Novgorod, and Voronezh                                                              57When referring to the subject of the evaluation as the “MSI project,” the “I’ve Got Rights project,” or “project” by itself throughout the document, project means a set of interventions/activities executed primarily through one USAID/Russia-funded implementing mechanism awarded to Management Systems International (MSI). 58 The Notification is included as Annex 1.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 61 Fig. 1 depicts the project results framework as included in the RFA. The results framework presents the following project logic: if citizens are willing to defend their rights, are aware of their rights, and have the tools and capacity to defend their rights, they will assert their rights through actions such as filing court cases, appealing to administrative bodies, turning to local journalists and officials, and launching community actions. Further, if citizens assert their rights, then the observance of fundamental human rights will improve. The results framework also assumes that if the observance of human rights improves, then citizens will be more willing to defend their rights. Figure 1. Project Results Framework The “I’ve Got Rights” Project aims to increase the knowledge of Russian citizens about their human, social, and civil rights and to create conditions that enable them to effectively protect those rights. This means that the overall MSI project goal is to improve the observance of fundamental human rights as defined in the Mission ROL and HR Project Results Framework. The project’s objectives are to:  Promote citizen awareness about human, social and civil rights, as well as mechanisms to effectively defend them;    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 62  Strengthen the skills and capacity of civil society organizations to educate and reach out to citizens;  Collect and broadly disseminate information about Russian citizens who successfully defended their rights;  Facilitate networking among human rights and other civil society organizations; and  Facilitate constructive dialogue between human rights groups and the government to develop and strengthen institutions that allow citizens to assert their rights. The Project Monitoring Plan (Table 1) consists of indicators linked to the expected project results. Table 1. Project Indicators as per Project Monitoring Plan Impact Indicators RESULT 1: OBSERVANCE OF FUNDAMENTAL HUMAN RIGHTS IMPROVED Community satisfaction with HR protection Disaggregated by gender and region IR 1: Citizens Assert Their Fundamental Rights (highest level direct result of project’s activities) Number of HR court cases filed in target regions by project-supported legal services groups Disaggregated by gender and region IR 1.1 Citizens are willing to defend their rights % of citizens polled who say they are willing to defend their rights Disaggregated by gender and region IR 1.2 Citizens have an awareness of their rights % of citizens polled who exhibit awareness of their rights Disaggregated by gender and region IR 1.3 Citizens have the tools and capacity to defend their rights Number of HRGs’ and CSOs’ representatives with improved knowledge and skills Percent of survey respondents who understand how to defend their rights Disaggregated by gender and region USAID Operational Plan Indicators Number of domestic human rights NGOs receiving US Government support through the project Number of people receiving legal consultations (from NGOs supported by the project) (disaggregated by gender and region) Critical Assumption 1: Responsible government authorities act to fulfill human rights obligation Percent of HR cases brought by legal services groups (supported by the project) to appropriate government offices that are resolved Critical Assumption 2: Legal framework for the protection of fundamental human rights exists. Number of advocacy campaigns (to reform laws and procedures) supported by the US Government Responsibilities for project management are divided between MSI offices in Washington, DC, and Moscow, Russia. The Washington office is responsible, for example, for managing public opinion surveys in the project regions and overall reporting to USAID. The Moscow office is responsible for managing the project activities in the regions. The “I’ve Got Rights” Project has four regional coordinators – one in each of the four abovementioned project regions – that are responsible to support and coordinate the project activities in their regions. The project has also established a network of human rights NGOs in the target regions and provides financial support through small grants to projects that are consistent with the overall goal of the “I’ve Got Rights” Project. In addition to this small grants    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 63 component,59 the project provides training to human rights NGOs, organizes annual competitions for journalists from the four project regions, and develops print and on-line information materials. By April 2012 the “I’ve Got Rights” Project supported 61 projects that were implemented by civil society organizations to increase public awareness about human, civil, and social rights; provide legal assistance to citizens whose rights were violated; and advocate for reforms to better observe and defend citizen rights including rights of youth and children, the disabled, workers, women, etc. Outputs of the project at this stage include the following:  more than 13,000 citizens whose rights were violated received legal assistance;  929 citizens’ rights cases were defended in court and 3000 cases were forwarded to other governmental agencies; and  more than 20,500 people participated in 517 seminars, trainings, round tables and other events. Other outputs include:  108 brochures and flyers about human rights were developed and distributed and 16 analytical reports prepared;  1,014 people participated in 58 discussions of human rights clubs organized regularly by regional partners of the project;  237 materials were published in print media, 1,389 – in the Internet, 89 stories and programs were on TV, 157 – on radio;  the project website expanded to include 229 success stories and about 1,540 reference materials;  484 representatives of human rights and other NGOs took part in workshops, organized by the project;  handbook «Human Rights NGOs’ Technologies» was published;  four interregional networking meetings for project partners were conducted; and  more than 200 journalists participated in the annual Journalists Contest conducted in 2010 and 2011. Evaluation Questions The evaluation will provide answers to the following questions60: 8. What is MSI’s progress in achieving the goals/objectives as defined in the Project Documents (Project Description, Work Plan, and M&E Plan)? Which objectives are particularly lagging or progressing and why?                                                              59Due to specific Russian legislation, the project calls these small grants “donations.” 60These evaluation questions have been modified by the evaluation team vis-à-vis the questions included in the notification. They were presented to USAID/Russia in a pre-design briefing on June 5, 2012 and were given verbal acceptance by USAID.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 64 9. What is the effectiveness of the project’s interventions as defined by the Project Description, Work Plan and M&E plan? Which ones are the most/least successful and beneficial to the target audiences and why? 10. How effective was the methodology of MSI’s public opinion surveys “Russian citizens and their rights: Priorities, attitudes and experiences” in measuring project results? 11. What is perception among citizens of human rights groups at this stage of the project? How effective are HR advocacy groups in the eyes of citizens? What are the factors that affect citizens’ perception both in positive and negative ways? 12. What have been the principal factors that led citizens to defend their rights, and how do those vary by region? 13. What have been the unintended consequences and effects of the project thus far? 14. What are the key lessons learned that could affect future programming and potential regional dissemination? 15. Are there any key adjustments that can be made to the project now in order to ensure that goals are met? If so, what are they? How can they be introduced at this phase? Data Collection Plan and Methodology Overall approach The evaluation questions are not related to an existing project hypothesis, they are purely empirical. Hence, answering them will provide new knowledge about the project and its context. In the course of this evaluation the evaluators will remain open to any information and use rigorous inductive analysis to come to conclusions and recommendations and the ‘progressive focusing’ approach, which means that new issues may emerge in the course of evaluation and some will assume a greater importance than others. Sources of Information The evaluation team will use the following sources of information:  Project documentation: RFA, project description, M&E plan, work plans and progress reports to USAID; concepts, descriptions, progress reports and evaluation data sheets for small projects implemented by partners in project regions.  Baseline and mid-term surveys (“Russian Citizens and their Rights: Priorities, Attitudes and Experiences”) conducted by the project, reports, data collection instruments (if available) and raw data (if available).  USAID staff working on HR issues.  The “I’ve Got Rights” Project staff in Moscow and Washington, DC.  Project partners in four regions.  People who used services of project partners or participated in their events in the regions.  Journalists who participated in the competitions organized by the project.  Human rights ombudsman in the project regions and human rights experts. Data Collection Methods Document Review    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 65 The evaluation team will review available project documentation (see above Sources of Information). Project Reach Mapping (in conjunction with Document Review) Starting with the list of locations in each region included in the MSI surveys (from survey reports), the evaluation team will map what location as well as age groups were reached by each of the small projects funded by the “I’ve Got Rights” Project as well as the number of small project participants. Social Network Analysis (Spider-webbing) A preliminary assessment of project documentation indicates that project partners have been required to report on working relationships they establish with other NGOs and government agencies while implementing their “I’ve Got Rights” activities. As such, the evaluation team will track and record this information during the document review and then develop a map of the network of organizations that have been involved in the implementation of the project activities. The evaluation team will validate the recorded data with the project partners during the in-depth interview process (see below). Direct observation The evaluation team plans to attend and observe Human Rights Week events from June 25 to 30 and a networking meeting on June 29 and 30in Voronezh. Short interviews with participants of the Human Rights Week in Voronezh The evaluation team will conduct short interviews with people attending Human Rights Week events in Voronezh. The purpose of these interviews will be to obtain information about their attitudes on human rights protection and human rights NGOs. Semi-structured in-depth individual and group interviews61 The Evaluation Team will use semi-structured in-depth interviews to collect information from the “I’ve Got Rights” Project staff, representatives of project partners, journalists and regular citizens who were involved in project activities. As experiences of the respondents will be heterogeneous, the evaluation team will use a mixed approach to developing the interview instrumentation based on a combination of a) interview guides and, b) informal conversational approaches. The interview guide approach requires that interview topics and issues are specified in advance, while the interviewer can decide on the sequence and wording of questions in the course of the interview. The strength of this approach is that the interview guide increases the                                                              61See Annex 2 for an illustrative list of key informants.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 66 comprehensiveness of the data and makes data collection more systematic. The weakness of this approach is that important and salient topics may be inadvertently omitted. When the informal conversational interview approach is used, questions emerge from the immediate context and are asked in the natural course of the conversation. There is no predetermination of question topics or wording. The strength of this approach comes from its ability to increase the salience and relevance of questions during an interview and those that emerge from observations. These can be matched to individuals and circumstances. The weaknesses of this approach are that it is less systematic and comprehensive and different information is collected from different people resulting in organizational and analytical challenges. Mixing these two approaches will allow us to minimize their respective weaknesses, while benefiting from the strengths. The length of each semi-structured interview will be about 1-1.5 hours. Most interviews will be with individuals, though in some cases group interviews could be used as well. Table 2. Advantages and disadvantages of the data collection methods Method Advantages Disadvantages Observation - Collect data where and when an event or activity is occurring - Does not rely on people’s willingness to provide information - Directly see what people do rather than relying on what they say they do - Susceptible to observer bias - Hawthorne effect – people usually perform better when they know they are being observed - Does not increase understanding of why people behave the way they do Document Review - Relatively inexpensive - Good source of background information - Unobtrusive - Provides a “behind the scenes” look at a program that may not be directly observable - May bring up issues not noted by other means - Information may be inapplicable, disorganized, unavailable or out of date - Could be biased because of selective survival of information - Information may be incomplete or inaccurate - Can be time consuming to collect, review, and analyze many documents Interviews - Useful for gaining insight and context into a topic - Allows respondents to describe what is important to them - Useful for gathering quotes and stories - Susceptible to interview bias - Time consuming and expensive compared to other data collection methods - May seem intrusive to the respondent Focus Groups (Group Interviews) - Quick and relatively easy to set up - Group dynamics can provide useful information that individual data collection does not provide - Is useful in gaining insight into a topic that may be more difficult to gather information through - Susceptible to facilitator bias - Discussion can be dominated or sidetracked by a few individuals - Data analysis is time consuming and needs to be well planned in advance - Does not provide valid information at the individual level - The information is not representative of other groups    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 67 other data collection methods Sampling Given the complex nature of the activities being evaluated, as well as the above mentioned characteristics of the study, the evaluators propose to adopt the purposeful sampling strategy based on the identification of information-rich cases – the sources from which we can learn most about the activities. The information-rich cases that will be selected for in-depth exploration represent a broad variety of approaches and activities used by the project in different environments. One of the best descriptions of purposeful sampling is provided by Michael Patton:62 The logic and power of probability sampling depends on selecting a truly random and statistically representative sample that will permit confident generalization from the sample to a larger population. The purpose is generalization. The logic and power of purposeful sampling lies in selecting information-rich cases for study in-depth. Information-rich cases are those from which one can learn a great deal about issues of central importance to the purpose of the research, thus the term purposeful sampling. For example, if the purpose of an evaluation is to increase the effectiveness of a program in reaching lower￾socioeconomic groups, one may learn a great deal more by focusing in depth on understanding the needs, interests, and incentives of a small number of carefully selected poor families than by gathering standardized information from a large, statistically representative sample of the whole program. The purpose of purposeful sampling is to select information-rich cases whose study will illuminate the questions under study. Hence, the size of a purposefully selected sample is not as important as the criteria used for the selection of cases. If we carefully select the right cases, we shall get the information we need to answer the evaluation questions effectively and fully. We shall illustrate this approach by discussing how we are going to answer evaluation question #4: “What is the perception among citizens of human rights groups at this stage of the project? How effective are HR advocacy groups in the eyes of citizens? What are the factors that affect citizens’ perception both in positive and negative ways?” Due to the existing restrictions discussed above, the team cannot use a probability sampling approach. Thus, we need to thoughtfully and purposefully select sources that can inform us about the perception of HR groups by citizens. Those sources will include (at least): - surveys conducted by MSI, - publications in media, - existing research results related to the evaluation question, and - expert opinions. In each region we shall talk to people knowledgeable about the subjects of interest: HR NGO leaders and activists, ombudsmen, journalists specialized in HR area. In addition, we shall use a “snowball” sampling approach and find independent HR experts in Moscow and in the regions who will share their opinions with us. MSI key staff will also be one of the important sources of                                                              62Patton, M. (1990).Qualitative evaluation and research methods (p. 169).Beverly Hills, CA: Sage.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 68 information for us. With the data being collected from various sources, by using various methods, this will help us verify our findings answer the evaluation question. Table 2 shows key categories of respondents and the evaluation sampling approach in relation to each of these categories. Table 2. Respondent categories and sample sizes Respondent category Sample MSI Moscow office project staff All Washington office project staff Key members involved in the project implementation in Russia Regional project coordinators All four Project partners in four regions and in Moscow All – both past and current People who used legal consultations to protect their rights 3 persons per region Journalists – participants of competition organised by the project 3 persons per region Human rights ombudsmen in the project regions (working with children and adults) All who will be available and willing to meet evaluators (7-8 people max) Human right experts TBD in the course of evaluation In summary, we will rely on existing studies (some of which were based on the random sampling approach) and on the opinions of people who are considered subject experts. The number of people we are going to talk is not a relevant factor for the evaluation, but will likely consist of several dozen people from the four regions. Data analysis Four distinct processes will be involved in making sense out of evaluation findings63. 1. Description and analysis: Describing and analyzing findings involves organizing raw data into a form that reveals basic patterns. The factual findings as revealed in actual data will be presented in a user-friendly fashion. 2. Interpretation: What do the results mean? What’s the significance of the findings? Why did the findings turn out this way? What are possible explanations of the results? Interpretations go beyond the data to add context, determine meaning, and tease out substantive significance based on deduction or inference. 3. Judgment: Values are added to analysis and interpretations. Determining merit or worth means resolving to what extent and in what ways the results are positive or negative. What is good or bad, desirable or undesirable, in the outcomes? 4. Recommendations: The final step adds action to analysis, interpretation and judgment. What should be done? What are the action implications of the findings? Only recommendations that follow from and are grounded in the data ought to be formulated.                                                              63Based on Patton, M. Q. (1997).Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage. (page 307)    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 69 Protection of human subjects It is important to ensure that evaluation participants are protected. The evaluation team will make arrangements for all evaluation activities with the heads of participating entities. Evaluators will explain the purpose and tasks of the study to them and to all interviewees. Verbal permissions from organizations and individuals to collect and use information will be obtained. Informal rules of joint work will be negotiated. When asked, evaluators will always allow people to see the interview questions in advance. Disclosure of conflict of interests All evaluation team members will provide a signed statement attesting to a lack of conflict of interest relative to the project being evaluated; this statement will be signed prior to beginning work on this evaluation. Briefing and Dissemination of Evaluation Findings This section describes overall approaches and key activities for communicating evaluation findings to the key stakeholders and disseminating evaluation results. A more detailed description of the time-bound evaluation milestones is included in the Evaluation Work plan. The evaluation team will present draft key evaluation findings to the USAID/Russia Human Rights Design Teaming the form of a PowerPoint presentation. Feedback received during and after the presentation will be considered in the draft report. A draft evaluation report will be prepared in English and sent to USAID/Russia for review and commentary. With USAID/Russia’s concurrence, it can also be shared with “I’ve Got Rights” Project staff for review. The draft evaluation report will be formally presented by the evaluation team to a broader audience at USAID/Russia and the representatives of the “I’ve Got Rights” Project. The evaluation team understands that the Final Report will be made public via the USAID website and the executive summary will be translated into Russian. Dissemination of evaluation results will be done with careful consideration of the local context and after consultations with the key stakeholders. Challenges and limitations Qualitative data allows a comprehensive and multifaceted understanding of people’s experiences with the program. However, collecting such in-depth and detailed data is time consuming so evaluators are compelled to restrict the number of people who they can interview. Another challenge of the qualitative methodology is that an evaluator is an instrument of research and inevitably brings his/her perceptions and values to the study. To minimize the possible distortion of the findings evaluators will be constantly discussing within the team what values they bring to this study. The team members will monitor each other’s neutrality throughout the evaluation.    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 70 EVALUATION WORK PLAN Evaluation Team The Evaluation team members are considered Key Personnel for this evaluation. No Key Personnel shall be removed from the Team nor shall their level of effort, roles or responsibilities be changed without prior written approval of USAID/Russia. See Annex 3 for the complete CVs of the evaluation team. Dr.Alexey Kuzmin, Team Leader Dr. Kuzmin will lead the Evaluation Team. He will be involved in document review, project reach mapping, data collection in Moscow, Nizhny Novgorod and Navosibirsk, data analysis and writing the report. He will be responsible for briefing USAID/Russia on evaluation progress and presenting evaluation findings. Dr. Kuzmin has extensive experience as evaluation team leader and is skilled working with international teams. He has conducted over 100 evaluations of projects and programs in Russia, CIS, and Central and Eastern Europe and his clients include USAID, UN agencies, international development agencies, private foundations, NGOs, businesses, and government entities. Dr. Kuzmin is co-founder and director general of Process Consulting Company, a Moscow-based evaluation consulting firm. He combines a deep knowledge of evaluation theory with extensive evaluation experience at the country, multi-country, and regional level. Alexey was a Team Leader for Human Rights Assessment conducted for the USAID/Russia in 2008. His key specializations include: Program and Project Evaluation, Organizational Assessment (Diagnostics), organizational change design and implementation, Customer Oriented Interactive Training Design, and Project Management. A Russian based in Moscow, Dr. Kuzmin holds a PhD in Organizational Behavior and Development. Natalia Kosheleva, Evaluation Consultant Ms. Kosheleva will be involved in document review, project reach mapping, data collection in Moscow and Voronezh, data analysis and writing the report. Ms. Kosheleva, based in Moscow, has served as team member and team leader on numerous evaluation projects for a wide variety of donor agencies, including USAID, UN agencies, and the Russian Red Cross. She has consulted for Process consulting since 2006. She was a member of the team who conducted Human Rights Assessment for the USAID/Russia in 2008. She holds MAs in Public Administration and Physical Geography and is a member of the American Evaluation Association. Her post-graduate study includes focus group moderator training, mixed methods evaluation, and interviewing techniques and content analysis for program evaluation. Vladimir Balakirev, Evaluation Consultant    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 71 Mr. Balakirev will be involved in the document review, project reach mapping, data collection in Perm, and data analysis. He will also contribute to writing the report. Mr. Balakirev, based in Moscow, has served as team member and team leader on numerous evaluation projects for a wide variety of donor agencies, including USAID. He has rich experience of work with human rights organizations in the capacity of consultant, evaluator and facilitator. Mr. Balakirev was a member of the team who conducted Human Rights Assessment for the USAID/Russia in 2008. ViacheslavBakhmin, Human Rights Expert Mr. Bakhmin will contribute to the evaluation design, help identify key informants both in Moscow and in the regions, and participate in data analysis and report writing. Mr. Bakhmin will also share his expert opinion on the project related issues. Mr. Bakhmin is currently the C.S. Mott Foundation Consultant in Russia and Manager of the Swiss Human Rights Program for Russia. Previously, from 1995 to 2003, he was the Executive Director of the Humanities Programs of the Open Society Institute in.Mr. Bakhmin was the Director of the Department of Humanitarian Cooperation, Human Rights and International Cultural Cooperation in the Ministry of Foreign Affairs of the Russian Federation from 1991 to 1995.He studied as an economic-engineer, and in his early career, worked as a programmer. Mr. Bakhmin was a prisoner of conscience from 1980 to 1984 for his defense of human rights. He has been active in the human rights movement in his country for many years. From 1992 to 1995, he was the Deputy Chair of the Russian Delegation to the UN Human Rights Commission. Mr. Bakhmin is a member of several boards and expert committees related to the promotion of human rights and civil society. Dr. Alice Willard, Survey Method and Social Network Analysis Expert Dr. Willard will review the survey methodology and support the social network analysis. Dr. Willard, IBTCI’s RMEP Senior Technical Advisor, is a seasoned M&E expert with more than 24 years of experience in managing, designing, and implementing evaluations and surveys for USAID, as well as in developing monitoring and evaluation plans and indicators, and in the training of staff. Her recent positions include Senior Monitoring and Evaluation Officer for International Relief and Development and M&E Manager for the American Red Cross. Dr. Willard has hands-on experience implementing M&E projects designing M&E systems, conducting workshop sessions, carrying out mid-term evaluations and impact assessments, and providing technical assistance on M&E and has updated an evaluation guide for USAID/PVC. Dr. Willard has worked across a wide range of sectors including health, relief/humanitarian assistance, democracy and governance, environment, etc. Table 3. Responsibilities of the team members Team members Kuzmin Kosheleva Balakirev Bakhmin Willard Tasks Designing evaluation tools + + + + Document review + + +    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 72 Project Reach Mapping + + + Survey Methods Review + Social Network Analysis (spider￾webbing) + Direct Observation and short interviews with citizens during Voronezh Human Rights Week and at a networking meeting in Voronezh + Direct Observation and survey at a networking meeting in Voronezh + Interviews with MSI staff in Washington (via Skype) + + Interviews in Moscow + + Interviews in Voronezh + Interviews in Nizhny Novgorod + Interviews in Novosibirsk + Interviews in Perm + Data analysis + + + + Report writing + + + + Evaluation Schedule This is a tentative evaluation schedule. The evaluation team will communicate with the respondents in the regions to finalize actual travel dates and the interview schedule. Deliverables schedule    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 73  Key evaluation findings will be presented to USAID on July13, 2012 in a PowerPoint presentation.  Draft Report will be submitted on July 31, 2012  Final report will be submitted within 3 working weeks after receiving comments on the draft report from USAID and implementing partners (tentatively – by September20, 2012) Stakeholder review mechanism and collaboration with the key stakeholders The evaluation team will work in close collaboration with the key stakeholders (USAID and the “I’ve Got Rights” Project staff) from the very beginning. The key stakeholders already have been involved in the preparation of the Evaluation Design and Evaluation Work Plan. If USAID concurs, the draft evaluation reports will be sent to the “I’ve Got Rights” Project staff for review and commentary and their comments will be discussed and considered by the evaluation team. Russia Monitoring and Evaluation Project - HR Project Evaluation Final Report Annex 3. List of organizations consulted Mo Novosibirsk Perm 74    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 75 Protection Voronezh    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 76 Annex 4. List of documents studied MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA. Year 1 Implementation Plan: August 31, 2009 – August 30, 2010 MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA. Year 2 Implementation Plan: August 31, 2010 – August 30, 2011 MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA. Year 3 Implementation Plan: August 31, 2011 – August 30, 2012 Mainstreaming of human, social and Civil Rights in Russia. Award Monitoring Plan. October 29, 2009 (revised on February 26, 2010, September 29, 2010, and November24, 2010) Russian citizens and their rights: Priorities, attitudes and experiences (Nizhniy Novgorod, Novosibirsk, Perm, and Voronezh regions of Russia). Baseline Survey Report. 2010 Russian citizens and their rights: Priorities, attitudes and experiences (Nizhniy Novgorod, Novosibirsk, Perm, and Voronezh regions of Russia). Survey Report. 2011 MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA. Quarterly Report: August 31, 2009 – November 30, 2009 MAINSTREAMING OF HUMAN, SOCIAL AND CIVIL RIGHTS IN RUSSIA. Quarterly Report: December 1, 2009 – February 28, 2010 Mainstreaming of human, social and Civil Rights in Russia. Semi-annual report. March 1, 2010 – August 31, 2010 Mainstreaming of human, social and Civil Rights in Russia. Semi-annual report. September 1, 2010 – February 29, 2011 Mainstreaming of human, social and Civil Rights in Russia. Semi-annual report. March 1, 2011 – August 31, 2011 Mainstreaming of human, social and Civil Rights in Russia. Semi-annual report. September 1, 2011 – February 29, 2012    Russia Monitoring and Evaluation Project - HR Project Evaluation Final Report 77 Annex 5. List of the project grantees '" Organizations = 'Q = Q!    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 78 Annex 6. Types of interventions used by the project grantees by regions Nizhni Novgorod Novosibirsk    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 79 Perm       Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 80 Voronezh    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 81 Annex 7. Target groups of the project grantees and the areas of rights they were focused on Nizhny Novgorod Novosibirsk Perm Voronezh    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 82 Annex 8. Project sites and survey sites in Voronezh oblast Annex 9. Developmental evaluation Developmental evaluation applies complexity concepts to enhance innovation and support evaluation use. The concept has been created and described by Michael Quinn Patton. “Innovations are different from standard projects and programs. Innovators are often different from people implementing typical programs. Innovators are in a hurry, value rapid, real time feedback, have a high tolerance for ambiguity, embrace uncertainty, learn quickly, and adapt rapidly to changed conditions. They’re not always sure where they’re heading, so they resist being boxed in by concrete, pre-set targets. They’re propelled into action more by vision than by clear, specific and measurable outcomes. They want an evaluation approach attuned to their fast pace and innovative spirit. They are at home in complex dynamic systems. Such systems characterize the world in which they live and work. Thus, they want an evaluation approach attuned to complexity. Complex situations challenge traditional evaluation practices. Complexity can be defined as situations in which how to achieve desired results is not known (high uncertainty), key stakeholders disagree about what to do and how to do it, and many factors are interacting in a dynamic environment that undermine efforts at control, making predictions and static models problematic. Complexity concepts include nonlinearity (small actions can produce large reactions), emergence (patterns emerge from self-organization among interacting agents), and dynamic adaptations (interacting elements and agents respond and adapt to each other). Developmental evaluation aims to meet the needs of social innovators by applying complexity concepts to enhance innovation and use. Developmental evaluation focuses on what is being developed through innovative engagement64.” Publications on developmental evaluation: Developmental evaluation: Applying complexity concepts to enhance innovation and use by Michael Quinn Patton (Guilford Press, 2010). A developmental evaluation primer. Jamie Gamble. (2008). Montréal: The J.W. McConnell Family Foundation. DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozois, Marc Langlois and Natasha Blanchet-Cohen. Montréal: The J.W. McConnell Family Foundation. AEA Annual Conference professional development workshop on Developmental Evaluation, with Michael Quinn Patton, November 8-9, San Antonio.                                                              64 http://aea365.org/blog/?p=1298    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 83 Annex 10. Social marketing campaigns Social marketing campaigns were implemented by in 2003 in Ryazan, Perm, and Rostov regions. “Steps were taken in each of the above regions to urge the population to seek that some or other human rights are observed. Two campaigns were conducted in Perm: one to protect rights and one to protect rights. Legal instruction for was organised in three Rostov region cities. A campaign against ran in Ryazan, aiming at all cohorts in the city. In keeping with the main principles of social marketing, each campaign was preceded by a study of the "demand": political views and aesthetic preferences of the population. The nation’s leading sociological carried out public opinion polls. Focus-groups were also held among the cohorts supposed to be the target audience for the planned activities. Once a campaign was over, new public opinion polls were conducted based on which we were able to assess its performance. Surveys confirmed that the use of PR and advertising technology in human rights advocacy turned out to be a success. Thus, in Ryazan the number of those who think grew 39% to 54% after the campaign, while the number of those who believe the shrank 13% to 6%. Over 9,000 people attended the photo exhibition pictures were shown. (Cf.: Two to three thousand took part in the most numerous meeting recently.) Growing numbers of people willing to become became one of the most significant results of in Perm. Their number more than doubled as the campaign was running in the autumn of 2003. More than a half (57%) of the Novocherkassk, Rostov region, residents surveyed remembers what slogans accompanied Seventy-eight per cent of young people learnt about f established as part of our project65.”                                                                 Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 84 Annex 11. Disclosures of Conflict of Interest    Russia Monitoring and Evaluation Project – HR Project Evaluation Final Report 85