1 PERFORMANCE EVALUATION OF MARGINALIZED POPULATIONS SUPPORT ACTIVITY Final Report September, 2019 This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of authors and do not necessarily reflect the views of USAID or the United States Government. 2 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BIH) PERFORMANCE EVALUATION OF MARGINALZIED POPULATIONS SUPPORT ACTIVITY Final Report September, 2019 Prepared under the USAID’s Bosnia and Herzegovina Monitoring and Evaluation Support Activity (MEASURE-BiH) Submitted to: USAID/Bosnia and Herzegovina, September 2019 3 CONTENTS ABSTRACT ................................................................................................................................... 4 ACKNOWLEDGEMENTS ........................................................................................................... 5 LIST OF EXHIBITS....................................................................................................................... 6 ACRONYMS.................................................................................................................................. 7 EXECUTIVE SUMMARY.............................................................................................................. 8 I. BACKGROUND INFORMATION ON THE ACTIVITY................................................. 11 II. EVALUATION PURPOSE, QUESTIONS, METHODS, AND LIMITATIONS ............. 16 Evaluation Purpose......................................................................................................................................................16 Evaluation Questions..................................................................................................................................................16 Methodology ................................................................................................................................................................17 LIMITATIONS.........................................................................................................................................................18 III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS........................................ 19 EVALUATION QUESTION 1: ACHIEVEMENT OF CONTRACTUAL OBLIGATIONS........................19 MEL PLAN TARGETS ...........................................................................................................................................19 OTHER CONTRACTUAL OBLIGATIONS ...................................................................................................21 EVALUATION QUESTION 2: RESULTS ACHIEVED THROUGH CAPACITY BUILDING INTERVENTIONS ......................................................................................................................................................25 EVALUATION QUESTION 3: IMPROVEMENTS IN MARGINALIZED GROUPS’ REPRESENTATION IN POLITICAL AND CIVIC ISSUES.......................................................................................................................32 RECOMMENDATIONS .......................................................................................................................................40 IV. ANNEXES ............................................................................................................................ 42 ANNEX I: EVALUATION AND ASSESSMENT STATEMENT OF WORK.....................................................42 ANNEX II: DATA COLLECTION INSTRUMENTS...............................................................................................53 INTERVIEW GUIDES.................................................................................................................................................53 INTERVIEW GUIDE FOR USAID/BIH .............................................................................................................54 INTERVIEW GUIDE FOR THE IMPLEMENTING PARTNER....................................................................55 INTERVIEW GUIDE FOR GOVERNMENT INSTITUTIONS AND INTERNATIONAL ORGANIZATIONS ...............................................................................................................................................56 FOCUS GROUP/INTERVIEW GUIDE FOR BENEFICIARIES ....................................................................60 ONLINE SURVEY QUESTIONNAIRES ................................................................................................................61 ANNEX III: LIST OF DOCUMENTS REVIEWED...................................................................................................99 ANNEX IV: DISCLOSURE OF CONFLICTS OF INTEREST............................................................................ 100 ANNEX V. MEASURE-BIH RESPONSES TO COMMENTS ON DRAFT MARGINALIZED POPULATIONS SUPPORT ACTIVITY PERFORMANCE EVALUATION REPORT .................................. 105 4 ABSTRACT This performance evaluation is commissioned by the U.S. Agency for International Development in Bosnia and Herzegovina (USAID/BiH) to examine the USAID/BiH Marginalized Populations Support Activity (PPMG). The PPMG’s purpose is to improve representation of marginalized populations in political and civic issues by providing financial and capacity building assistance to local organizations supporting marginalized populations. The Activity defines marginalized populations as youth, women, Roma, persons with disabilities (PWDs), and lesbian/gay/bisexual/transgender/intersex (LGBTI) individuals. This performance evaluation addresses three evaluation questions: 1. PPMG’s progress toward targets; 2. PPMG’s achievements in terms of capacity building of local organizations, 3. PPMG’s results in terms of improving representation of local organizations and marginalized populations in political and civic issues. The evaluation team employed a mixed-methods approach. We combined information obtained through a review of the Activity documentation; from 38 key informant interviews; four focus groups; and online surveys of PPMG’s grantees, trainees, and government representatives. Our evaluation indicates that the PPMG may not reach all its contractual targets and that there are areas for improvements when it comes to monitoring of the Activity’s progress. There are indications that the grantee organizations improved capacities with the PPMG’s assistance, but there is no evidence that similar results were achieved among non-grantees or government representatives. The service centers providing services for persons with disabilities and their families showed the greatest improvements in terms of capacity building and representation of marginalized groups in political and civic issues. These organizations have been intensively and continuously supported by the Activity, indicating that longer-term and extensive support is needed to achieve such results. The PPMG supported marginalized women through the Women Entrepreneurship Program. Although in some cases the Program improved the financial situation of women, it did not address women’s perceptions about gender roles in society. Representatives of local organizations find that connecting the organizations focusing on similar issues has been an important effect of the program. Moreover, owing to the Program, several local organizations built capacity and became mentors to smaller organizations, assisting them in building their organizational capacities. Local organizations made limited contributions in terms of policy changes related to marginalized populations, public perception, and visibility of these groups. However, most stakeholders noted that policy changes and changes in public perception are long-term processes. The main recommendations for potential new interventions targeting marginalized groups include changes in the following areas: improving the monitoring, evaluation, and learning (MEL) systems; continuing the support to service centers; providing long-term and extensive assistance to local organizations; continuing to build capacities of government officials; building an advocacy platform; provision of technical assistance and awareness raising interventions within the Women Entrepreneurship Program; coordination with other donors. 5 ACKNOWLEDGEMENTS USAID/Bosnia and Herzegovina (BiH) commissioned IMPAQ International within the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct the performance evaluation of the USAID/BiH’s Marginalized Populations Support Activity (PPMG). The PPMG is a $4.99 million Activity implemented by the Institute for the Youth Development KULT with a period of performance of February 22, 2015 through in February 2022. The evaluation team members for this evaluation include Salminka Vizin (Team Lead and MEASURE-BiH Senior Research Analyst), Nikolina Obradovic (Team Co-lead, social protection expert, and MEASURE￾BiH Research Fellow), Bergin Kulenovic (evaluation team member and grant management expert), Boris Badza (evaluation team member and MEASURE-BiH GIS Specialist), and Amina Smajovic (evaluation team member and MEASURE-BiH Analyst). The authors wish to thank all those who contributed to the process of developing this evaluation report. At USAID/BiH, the following people were instrumental in designing the evaluation: Elma Bukvic Jusic, MEASURE-BiH Contracting Officer’s Representative (COR); Elvira Challenger, PPMG COR; Kent Larson, Program and Project Office Director. We would also like to thank the representatives of the PPMG team, international and local organizations, government institutions, and marginalized populations for their collaboration during data collection. Any errors in this report are the sole responsibility of the authors. 6 LIST OF EXHIBITS Exhibit 1. Training Topics with Short Description, Activity Award....................................................................14 Exhibit 2. Activity Logical Framework, with Annual and Overall Targets .........................................................15 Exhibit 3. Number of KIIs/Focus Groups and Key Informants.............................................................................17 Exhibit 4. Online Surveys response rate....................................................................................................................18 Exhibit 5. Activity Indicators, with Targets and Actuals for FY 2015, 2016, 2017, and 2018 andLife of Activity Targets................................................................................................................................................................23 Exhibit 6. Allocation of program funds, by type of assistance (left-hand side) and by type of marginalization (right).....................................................................................................................................................24 Exhibit 7. Grantees' ratings of pre-award requirements, OCA Tool, and one-on-one assistance..............26 Exhibit 8. Satisfaction with trainings among PPMG grantees ................................................................................27 Exhibit 9. Grantees’ ratings of influence of trainings on their skill improvement (average rating of assessments on the scale from 1 to 4, 1 being “Not at all” and 4 being “Extremely effective”)..................27 Exhibit 10. OCA results for service centers, annual program statements, and other grants.......................28 Exhibit 11. Satisfaction with trainings among PPMG trainees...............................................................................29 Exhibit 12. Participants’ ratings of influence of trainings on their skill improvement .....................................29 Exhibit 13. Government representatives’ ratings of PPMG trainings..................................................................30 Exhibit 14. PPMG Interventions with local organizations (grantees), geographic distribution .....................36 Exhibit 15. PPMG municipalities supported through Women Entrepreneurship Program, geographic distribution........................................................................................................................................................................37 7 ACRONYMS CCSP Center for Civil Society Promotion COR Contracting Officer’s Representative CS Canton Sarajevo CSOs Civil Society Organizations DQA Data Quality Assessment FGD Focus Group Discussion IP Implementing partner KI Key informant KII Key informant interview LGBTI Lesbian, gay, bisexual, transgender, intersex LoA Life of activity M&E Monitoring and evaluation MEL Monitoring, evaluation and learning NGOs Non-governmental organizations OCA Organizational Capacity Assessment PIRS Performance indicator reference sheet PPMG Marginalized Populations Support Program PwD People with Disabilities RfA Requests for Application SC Service center SMEs Small and medium enterprises SOC Sarajevo Open Centre TO Task order USAID U.S. Agency for International Development USAID/BiH U.S. Agency for International Development in Bosnia and Herzegovina USG U.S. government 8 EXECUTIVE SUMMARY PURPOSE AND EVALUATION QUESTIONS USAID Mission to BiH commissioned IMPAQ International to conduct the performance evaluation of the USAID/BiH’s Marginalized Populations Support Activity (PPMG). The performance evaluation is focused on progress toward targets and expected results. Specifically, the evaluation addresses three questions pertaining to the achievement of contractual targets, results of capacity building interventions, and representation of local organizations and marginalized populations in political and civic issues. The evaluation employs rigorous methods to capture high-quality data and produce credible findings, conclusions, and recommendations. The evaluation design included a wide range of stakeholders to inform the findings, conclusions and recommendations. These stakeholders include: the USAID/BiH Activity COR and other officials who participated in the Activity design; the PPMG Implementing Partner (IP) and sub￾contractor; relevant international and government institutions; PPMG grantees and trainees; PPMG experts/trainers; representatives of marginalized populations; and representatives of local organizations that have not received any support from the program. The primary purpose of the evaluation is to inform USAID/BIH’s management decisions related to the ongoing PPMG Activity and any potential future interventions aiming to assist marginalized populations. The evaluation report will also be made available to other stakeholders to inform their interventions supporting local organizations and marginalized populations in BiH. EVALUATION METHODS AND LIMITATIONS The evaluation team used a mixed-method design for this performance evaluation. The data sources included relevant Activity documentation (see Annex III), secondary data sources (see Annex III), key informant interviews (KIIs), focus group discussions (FGDs), and online surveys. Whenever possible, the evaluation team triangulated data from different data sources and stakeholders to ensure validity of findings, conclusions, and recommendations. Different sources of bias (response bias, recall bias, social desirability bias), incomplete monitoring data, and limited response rate for online surveys represent potential limitations for this performance evaluation. BACKGROUND The Marginalized Populations Support Activity (PPMG) is a $4.99 million Activity funded by USAID/BiH and implemented by the Institute for Youth Development KULT. The Activity started in February 2015 and ends in February 2022. The Activity contributes to Project Purpose 1.2 – Increased citizen participation in governance – under Development Objective 1: “more functional and accountable institutions and actors that meet citizens’ needs.” The primary objective of the PPMG Activity is “to rapidly respond to the changing socio-political situation of marginalized populations and tense political situation in BiH.” The Activity target groups involve youth, women, people with disabilities (PWDs), the Roma, and lesbian/gay/bisexual/transgender/intersex (LGBTI) individuals. The PPMG aims to increase 9 underrepresented citizens’ participation in governance by providing financial, technical, and in-kind assistance to organizations focused on these marginalized groups. EVALUATION FINDINGS AND CONCLUSIONS Contractual obligations USAID/BiH monitors PPMG’s progress through five performance indicators. The Activity has already exceeded MEL targets for three of these indicators related to the number of: marginalized persons assisted by the Activity, advocacy interventions implemented to improve the position of marginalized populations, and human rights organizations trained or supported by the Activity. However, it is inconclusive whether the reported numbers are accurate given data quality issues (i.e. double counting, lack of MEL data for grantees’ interventions, definitions of indicators not being shared with grantees). The Activity has not yet met the target for the number of organizations receiving at least 12 hours of training and counselling on the OCA tool. However, it is likely this target will be met if the USAID continues awarding grants to local organizations in the remaining years of implementation. Activity progress has been slow in terms of the number of government representatives who participate in trainings, as USAID has not requested any trainings for this group since 2016. The Activity will not reach the target for this indicator if the trainings are discontinued. PPMG implements high quality procedures in grant award and management, with considerable support from the sub-contractor’s staff (NGO and Business Audit). The Activity implemented some elements of the Award differently than it was envisaged (one-on-one assistance, support for marginalized women, networking sessions). The PPMG’s support was strongly focused on PWDs and marginalized women and considerably less on other marginalized groups. Results of the Capacity Building Interventions The PPMG provided technical assistance to several types of beneficiaries: local organizations that have received the PPMG’s financial assistance (grantees), other local organizations, and government institutions (youth officials). The PPMG provided extensive capacity building technical assistance to grantee organizations. This assistance included individualized assistance in the implementation of pre-award survey recommendations, implementation of Organizational Capacity Assessment (OCA) tool1 , and continuous training and mentoring. As envisaged in the Activity design, the intensity of assistance provided to non￾grantee organizations and government officials was considerably lower. The results indicate that the PPMG assistance was most effective in building the capacity of grantee organizations, particularly of service centers who have been continuously trained and supported for longer periods (compared to other types of grantees). Non-grantee beneficiaries and government officials felt that the assistance provided by the PPMG motivated them to change their organizational policies and work performance. The PPMG did not build capacities of any women’s organizations, but opted to support women by co-financing women-led business with selected local self-government units. 1 An instrument designed for non-government organizations to assess their capacities and define their capacity building priorities. For more information, see https://www.usaid.gov/sites/default/files/documents/1864/OCA%20Tool%20Generic%20Version%20Facilitators%20Copy.pdf. 10 Improvements in Local Organizations’ Representation in Political and Civic Issues Service centers achieved the greatest success in terms of improving local organizations’ and marginalized groups’ representation in political and civic issues. However, the service centers’ form of registration is not recognized by the RS regulation on certification of social service providers. Grantee organizations made some contributions in terms of policy-making related to marginalized population, although in most cases the policies for which they advocated have not yet been adopted. Also, grantees achieved limited results in terms of improving their and marginalized populations’ interactions with the general public and media to affect public perception. However, the majority of stakeholders view changes in social policy and public perceptions as a long-term process that exceeds the Activity’s lifespan. There is no evidence that local organizations or government institutions that received only technical assistance improved their representation in political and civic issues. The evaluation team was also not able to find evidence that women’s entrepreneurship program improved women’s perceptions about gender roles in society. Because LGBTI rights remain a taboo subject among the public in BiH, progress in this area has lagged behind other areas. Networking of local organizations was identified as being an important unintended effect of PPMG trainings. Additionally, some organizations supported by the program considerably improved their organizational capacities and assumed roles as mentors of smaller organizations. RECOMMENDATIONS The evaluation team suggests that the PPMG should consider conducting a thorough review and revision of the Activity MEL Plan, specify indicator definitions, and revise annual targets for the following years. It is recommended that the PPMG continues providing extensive capacity building assistance to grantees, but expands capacity building interventions of highly motivated small organizations. All trainings should be tailored to the needs and development level of the recipients. PPMG should consider continuing working with government officials to raise their awareness about marginalized populations and to assist them in providing better-targeted support to local organizations. The Activity should motivate grantees’ to improve their interactions with the authorities and to invest more efforts in promotion of positive stories. The Program should consider providing technical assistance and awareness-raising about women rights to beneficiaries of the Women Entrepreneurship Program. In addition, providing long-term and intensive support to a limited number of local organizations may be more beneficial than providing project-based support to a large number of beneficiaries. USAID/BiH should continue supporting and expanding service centers based on the already established model, and facilitate the mentorship role of the existing service centers for newly established ones. For future interventions implemented through TOs, USAID should consider developing a general MEL Plan without targets and adopting individual MEL Plans with specified targets for each TO. 11 I. BACKGROUND INFORMATION ON THE ACTIVITY The Marginalized Populations Support Activity (PPMG) is a $4.99 million USAID/BiH-funded Activity implemented by the Institute for Youth Development KULT. The Activity started in February 2015 and ends in February 2022. This Activity contributes to Project Purpose 1.2 – Increased citizen participation in governance – under Development Objective 1: “more functional and accountable institutions and actors that meet citizens’ needs.” The primary objective of the PPMG Activity is “to rapidly respond to the changing socio-political situation of marginalized populations and tense political situation in BiH.” The Activity design envisaged assistance to the following marginalized groups: youth, women, people with disabilities (PWDs), the Roma, and lesbian/gay/bisexual/transgender/intersex (LGBTI) individuals. The PPMG aims to increase underrepresented citizens’ participation in governance by providing technical, material, and financial assistance to organizations focused on these marginalized groups. The PPMG’s tasks are organized under three components: I. Grants award and management II. Human and organizational capacity development III. In-kind logistical support The PPMG’s contract with USAID/BiH is specific in the sense that it is the first Indefinite Delivery/Indefinite Quantity Contract (IDIQ) signed with a local organization in BiH. This type of contract envisages that the implementer would provide an indefinite quantity of services during a fixed period of time. The main purpose of such contract was to make sure that the USAID can rapidly respond to the needs of marginalized populations as they are identified. The services to be provided by the contractor are defined through task orders (TOs). During the course of the Activity, PPMG received 10 TOs from USAID/BiH – first one in 2015 and the last one in in 2018. The first TO, the base TO, determines general Activity tasks. All other TOs refer to specific grants to be awarded and managed within the contract. GRANTS AWARD AND MANAGEMENT Component 1 aims to provide funding opportunities to local organizations focused on improving the status of marginalized populations. Originally, it was envisaged that the PPMG would award and manage three types of grants to such local organizations: unsolicited grants (awarded based on review of unsolicited proposals received by USAID/BiH), directed grants (allocated through USAID’s internal selection process), and annual program statements (awards for initiatives related to specific areas connected to USAID/BiH Development Objectives). The PPMG is responsible for solicitation, negotiation, grant award and administration, and monitoring of grant implementation. The award envisaged that the PPMG would review all unsolicited proposals on behalf of the Mission and provide feedback and recommendations to the USAID regarding whether the proposals should be funded 12 or not based on criteria established by USAID/BiH and the Activity. To be recommended for funding, proposals must support the PPMG objectives and demonstrate strong impact, innovation, and facilitate ongoing USAID interventions. When USAID/BiH awards a directed grant to an organization, the PPMG is responsible for conducting a pre-award survey to assess the organization’s capacities to carry out the grant. The pre-award survey includes review of the organizations’ legal structures, financial management, internal control systems, procurement systems, human resources, program performance management, and organizational sustainability. The assessment is conducted through a desk review of the grantee’s documentation, interviews, file audits, and other procedures. When the pre-award survey findings require improvements in organizational policies and procedures, the PPMG’s responsibility is to provide the organization with the necessary guidance and support in organizational development. The second type of grant, those awarded through the annual program statements, are allocated based on requests for applications (RfAs). The PPMG is responsible for issuing RfAs, reviewing and evaluating submitted proposals, and submitting their recommendations to the USAID, who makes the final decisions about any grant award. The PPMG was required to develop a grants manual based on the USAID’s grants manual template. It was envisaged that the manual would provide rules for the contractor regarding solicitation, evaluation, award, and grant monitoring. PPMG is responsible for ensuring that all grant management procedures comply with the USAID’s rules and regulations. In addition, all grant awards need to be approved by the Activity’s Contracting Officer Representative’s (COR). The PPMG was also responsible for providing logistical support in cases where USAID decides to provide in-kind grants to organizations (Component 3). This type of assistance was envisaged to be allocated in cases of unanticipated events since the program was designed after the 2014 floods in BiH. CAPACITY BUILDING TECHNICAL ASSISTANCE The PPMG Award envisaged extensive capacity building interventions for local organizations. According to the Award, the PPMG is responsible for conducting Organizational Capacity Assessments (OCA) of all grantees using the OCA Tool, a self-assessment questionnaire that explores organizational capacities in seven broad domains: 1. governance 2. administration 3. human resources 4. financial management 5. organizational management 6. program management, and 7. project performance management The purpose of the OCA tool is to assist grantees in defining their capacity development priorities. The PPMG is obligated to use the OCA Tool three times over the life of each grant to monitor grantees’ progress and to assist them in developing and implementing their individual Capacity Building Plans. Task 13 Order (TO) #1 envisages two versions of the OCA Tools: a simplified version for small and medium-sized organizations receiving up to $10,000 and a more extensive version for medium and large organizations receiving higher grant amounts. The Activity award also foresees individualized capacity building support for grantees and selected USAID implementing partners. This type of assistance includes one-on-one coaching, mentoring, and training. The purpose of the one-on-one assistance is to provide organizations with direct support to implement pre￾award and OCA recommendations. In terms of coaching, the design envisages developing a one-to-one coaching plan for each grantee, and a three-day intervention with in-depth individual assistance and oversight in order to improve organizations’ administration and financial management. The PPMG design envisages delivering four coaching sessions to each organization. One-on-one assistance through mentorship includes providing support to local organizations as requested via email, the project website, by phone, by Skype, by Facebook, during non-formal portion of the training, and through a support line. Approximately twelve consultations are envisaged with each organization to assist them in developing their own policies. In addition, TO #1 envisages that less developed organizations would take study visits to more developed organizations to learn about examples of good practices employed by these organizations. The PPMG design includes delivering formal classroom trainings to grantee organizations, other local organizations, and selected government officials collaborating with the USAID to improve their performance and management capacity. For the first four years of implementation, PPMG planned to deliver 13 trainings, 5 for grantees and 8 for other local organizations and government officials. According to the design, trainings would have a theoretical portion followed by a problem-solving exercise (simulation), reflection, and homework assignment. All PPMG grantee organizations have been obligated to attend trainings on five topics2 organized by the PPMG. Exhibit 1. presents the training topics and a brief description of the training content. The PPMG is responsible for overseeing design and delivery of training programs, training needs assessments, and training material; making logistical arrangements; evaluating training impact; conducting trainings of trainers when applicable; and administering post-training tests. The design requires integrating gender equality and female empowerment into all training curricula and developing guidelines for addressing gender as a cross-cutting issue. To implement this, prominent CSOs promoting gender equality would hold brief trainings on gender equality during training sessions on other topics (listed below). 2 Legal and organizational structures, Strategic planning, Administration and Human Resources, Project Cycle Management, Financial Management. 14 Exhibit 1. Training Topics with Short Description, Activity Award TRAINING TOPICS SHORT DESCRIPTION OF TRAINING CONTENT Legal and organizational structures Legal registration at different government levels; compliance with tax and labor laws; organizational vision and mission; board of directors and development of procedures and oversight over boards Administration and human resources Developing and implementing high quality organizational policies and procedures, administrative systems, updating job descriptions, recruitment policies, keeping personnel files, developing personnel manuals Project performance management Best practices in terms of strategic planning, communication strategies, project design and implementation; budgeting, fund raising, monitoring and evaluation; evidence-based decision making; learning; improving procedures and activities; improving management’s response to shocks Organizational sustainability How to tackle changes in leadership, including budgeting, fundraising, utilization of constituents, and revenue diversification Financial management Accounting systems, using accounting journals and charts of accounts, financial record keeping, use of vouchers, keeping files for financial audits, financial and management reporting, management and board meeting minutes Internal control systems Developing strong organizational control environments; methods for risk assessments in relation to financial reporting; implementation of control procedures in relation to documentation, management of cash on hand and in bank accounts, management of personnel, timekeeping, salaries and benefits, procurement procedures and procedures related to use of commodities; developing transparent accounting systems; developing a monitoring system Procurement systems Developing procurement plans, full and open competition, written procedures for procurement, keeping procurement files, policies and procedures for management of fixed assets Monitoring and evaluation Developing results, output and outcome indicators, data collection and analysis, impact evaluation The design foresees that sustainability would be integrated throughout the grants and that the program would facilitate collaboration between local organizations and government institutions, and between different local organizations working on similar issues. Five networking sessions among CSOs and five meetings between CSOs and government partners are planned to facilitate collaboration. The Activity design also envisages establishing a monitoring system to track progress of pre-award survey3 , an examination that informs pre-award risk assessment. The design also envisages post-award survey implementation to review whether pre-award survey recommendations have been implemented. The PPMG design also includes five one-day roundtable events per year with grantees, USAID partners, and representatives of government institutions to exchange knowledge and explore opportunities for cooperation. The PPMG plans to conduct monitoring visits to all grantee organizations. The PPMG design also envisages a clear visibility strategy for the PPMG. This includes launching a website with a restricted access component for project beneficiaries and a public domain component accessible to a wider audience, a discussion forum, a database of grantees and training participants, and the development of promotional materials. 3 https://www.usaid.gov/sites/default/files/documents/1868/303.pdf 15 The PPMG is tracking five indicators to measure the Activity’s progress. These indicators and their annual targets are presented in the table below (Exhibit 2). Exhibit 2. Activity Logical Framework, with Annual and Overall Targets Level of result Narrative Summary Indicators Targets Life of Activity Targets FY 2015 FY 2016 FY 2017 FY 2018 Activity Purpose 1.2.3 Individuals and CSOs representing underrepresented groups are constructively engaged in civic/political issues Number of underrepresented people assisted by a USG￾funded intervention providing services to beneficiaries 300 600 3,000 3,000 8,900 Activity Sub￾purpose 1 Activities of groups of local organizations that advocate for the rights and dignity of underrepresented groups are supported (Corresponds to USAID 1.2.3.1 CSOs representing and led by underrepresented groups advocate for policy changes) Number of advocacy initiatives focused on improving the status of youth, women, the Roma, the disabled, or LGBTI in the public and political life in BiH 0 5 60 60 185 Activity Outcome/ Output 1.1. The capacity of local NGOs, BiH institutions and USAID implementing partners is built. Number of human rights organizations trained and supported 6 30 100 80 296 Number of organizations representing marginalized populations that receive minimum 12 hours of training and counseling on OCA tool 0 8 8 8 32 Number of representatives of government institutions in BiH trained to increase their organizational capacities 15 25 25 25 115 16 II. EVALUATION PURPOSE, QUESTIONS, METHODS, AND LIMITATIONS Evaluation Purpose USAID/BiH commissioned IMPAQ International (IMPAQ) to conduct the performance evaluation of the PPMG Activity. The PPMG Activity is a $4.99 million Activity implemented by the Institute for Youth Development KULT. The Activity started in February 2015 and ends in February 2022. This performance evaluation explores the Activity’s progress toward contractual targets and expected results. The evaluation employs rigorous methods and design to capture high-quality data and produce credible findings, conclusions, and recommendations. The evaluation design includes data collection from various stakeholders to inform the evaluation findings, conclusions and recommendations. These stakeholders include the USAID/BiH Activity Contract Officer’s Representative (COR) and other officials who participated in the Activity design, the PPMG implementing partner (IP) and sub-contractor, relevant international and governmental institutions, PPMG grantees, trainees, experts/trainers, representatives of marginalized populations, and local organizations that have not received any support from the program. The primary audience for this evaluation is USAID/BiH, who will use the evaluation findings, conclusions, and recommendations to inform management decisions for the ongoing PPMG Activity and any potential future interventions aiming to assist marginalized populations. In addition, the results will assist the Activity IP in improving CSO capacity building practices and their representation in political and civic issues. Finally, relevant government institutions, civil society, and other stakeholders can use the evaluation findings to design or modify their own interventions targeting marginalized populations in BiH. Evaluation Questions The evaluation questions below address the PPMG progress toward contractual targets and the Activity results in building the capacity of local organizations and improved representation of these organizations and marginalized groups in political and civic issues. 1. What progress has PPMG achieved to date in reaching contract targets? 2. To what extent has the organizational capacity of PPMG-supported organizations (both grantees and trainees) been improved? What is some of the evidence that this has happened? 3. To what extent has the representation in civic and political issues of PPMG-supported organizations been improved? Are there indications showing that supported local organizations are interacting more often and more effectively with public authorities and/or the public at large? 17 Methodology The evaluation team employed a mixed method design for this performance evaluation. The team used the following data sources for the evaluation:  Desk review of relevant documentation and data. The evaluation team reviewed relevant Activity documentation. For the full list of documents reviewed, see Annex III.  Key Informant Interviews and Focus Groups. The evaluation design included key informant interviews with nine categories of stakeholders and focus groups with representatives of marginalized populations. Overall, the evaluation team conducted 38 key informant interviews (KIIs) and four focus group discussions (FGDs). FGDs were conducted with representatives of youth, women, PWDs, and Roma. In some cases, these persons were beneficiaries of the PPMG grantees’ interventions (PWDs, youth, women). In other cases, these persons were beneficiaries of regular grantee interventions, not necessarily those related to PPMG (Roma and LGBTI). Exhibit 3. presents number of KIIs/FGDs per category of stakeholders and number of key informants included.  Online surveys. In addition to qualitative data collection, MEASURE-BiH also conducted an online survey of PPMG grantees, trainees, and government leaders who participated in PPMG interventions. Email contacts for grantees, trainees, and government leaders have been provided by the IP either in the form of a compiled list (for grantees) or as scanned training attendance sheets (for trainees). The objective of the surveys was to gather additional information on beneficiaries’ perceptions about grant award and management and human and organizational capacity development. Participation in the survey was voluntary and anonymous. For more information about the data collection instruments see Annex II. The sample size of the surveyed grantees, trainees, and government leaders and surveys’ response rates are illustrated in Exhibit 4. Exhibit 3. Number of KIIs/Focus Groups and Key Informants Key Informants Number of KIIs/FGDs Number of KIs USAID/BiH 1 3 IP 2 54 International organizations 4 8 Representatives of government institutions 4 9 Experts/trainers 1 1 Grantees 10 23 Trainees, local organizations 5 6 Trainees , government leaders 5 9 Representatives of marginalized populations 4 27 Non-grantees, local organizations 6 7 TOTAL 42 98 4 Out of five interviewees, three were experts included in grant award and management, implementation of OCA, one-on-one assistance, and delivering trainings to local organizations. 18 Exhibit 4. Online Surveys response rate Type Sample # of respondents Response rate PPMG grantees 37 24 65% PPMG trainees 202 82 41% Government leaders 58 23 40% Whenever possible, the evaluation team triangulated data from different data sources and stakeholders to address the evaluation questions from multiple perspectives and to provide relevant and valid findings. Limitations Limitation 1: Social desirability bias PPMG grantees and trainees may overstate the improvements their organizations experienced due to the program and positive effects of their interventions on political and civic issues. Moreover, government representatives may overstate improvements in the status of marginalized groups. To ensure the validity of the findings, the evaluation team conducted a thorough review of Activity documentation and other secondary data sources. The team included marginalized populations and representatives of local organizations which were not included in the program as key informants, since it is expected that they are less motivated to give socially desirable responses. Limitation 2: Recall bias Some of PPMG’s trainees were involved in only a few interventions or in interventions that were implemented at the beginning of the Program implementation. Therefore, they may not recall all the details of their engagement in the program. The evaluation team reminded such beneficiaries about interventions in which they participated. However, in some cases their inputs were limited as they could not recall any details about the interventions. Limitations 3: Limited sample size of sub-groups of PPMG beneficiaries and limited response rate for online surveys PPMG provided support to various groups of interest, including five sub-groups of marginalized populations, grantees-local organizations, grantees-government institutions, trainees-local organizations, trainees-government officials. The program may have had different effects on these subgroups of beneficiaries. To assess this, the evaluation team included small samples of these sub-groups as key informants and conducted a thorough review of Activity documentation and online surveys to overcome this obstacle. PPMG does not keep records of grantees’ monitoring data. This was the reason that the evaluation team could not randomly select grantees’ beneficiaries or conduct an online survey of beneficiaries without grantees’ assistance. The evaluation team tried to include the beneficiaries by asking grantee organizations to distribute the survey link to their beneficiaries. However, the response rate was very low and this data has not been included in the analysis. 19 III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1: ACHIEVEMENT OF CONTRACTUAL OBLIGATIONS What progress has PPMG achieved to date in reaching contract targets? MEL Plan Targets Finding 1. Actuals reported by PPMG deviate from targets for all indicators. According to their MEL Plan, PPMG reports to USAID/BiH on five performance indicators as presented in Exhibit 5. As presented in the Exhibit 5, PPMG already exceeded targets for three performance indicators related to number of underrepresented people they assisted, number of advocacy initiatives they implemented, and number of human rights organizations they trained. For instance, the Activity envisaged assisting 8,900 persons over the LoA, and they reported assisting 14,138 underrepresented individuals in four years of implementation. The LoA target for the indicator tracking advocacy initiatives is 185 and the Activity reported conducting 201 (+16) advocacy initiatives. When it comes to human rights organizations, the LoA target for this indicator is 296 and the Activity reported assisting 426 human rights organizations in the four years of implementation. But, unlike for the first three indicators, the Activity has not reached their targets when it comes to number of organizations that receive 12 hours of support in OCA implementation and number of government representatives trained. In the last four years, the Activity provided this training and counselling to 25 organizations (all grantees), while the LoA target was 32. Similarly, the number of government representatives trained to increase their organizational capacities was substantially slower than envisaged. According to the Activity MEL Plan, the Activity should train 115 government representatives over the LoA, and the Activity reported training 42 representatives in the first four years of implementation. As previously described in the background section, PPMG has the ID/IQ contract with the Mission. The contract itself does not involve any numerical targets. Numerical targets have been stated in TOs issued during implementation. Overall, 10 TOs have been issued to the Activity from 2015 to 2018. However, PPMG developed their MEL Plan at the beginning of the Activity implementation, before receiving any TOs. Since the Activity could not have known the scope of the TOs that would be issued in the future, they defined their targets arbitrarily. This may be one of the reasons why their actuals considerably deviate from targets. Regardless of the targets, based on KIIs and the Activity documentation, the numbers (actuals) reported by the Activity may not correspond to the actual number of people, advocacy initiatives, and organizations supported. According to the definition in the PIRS and interviews with the IP, when counting the number of underrepresented people assisted, the Activity also counts training participants in addition to the beneficiaries reported by the grantees under this indicator. However, it is not clear how many training participants belong to marginalized categories as defined by the Program. This is particularly of concern in relation to marginalized youth and women, who are not clearly defined in the PIRS. When it comes to advocacy initiatives, the Activity’s PIRS provides some instructions on counting an advocacy initiative. 20 However, this definition was not shared with the grantees; none of the interviewed grantees could remember if the Activity explained the definition or how to count advocacy interventions to them or their staff. With regard to number of human rights organizations, as explained by the IP, the Activity did not count only the organizations they trained and/or supported with grants toward this indicator. They also counted organizations participating in grantees’ interventions. The Activity did not report unique or precise numbers of underrepresented people they assisted, advocacy initiatives carried out, or human rights organizations trained. During the Data Quality Assessment (DQA) site visit conducted in September 2017, the IP clarified that, when tracking the number of underrepresented people, some participants in trainings were counted multiple times if they attended multiple activities, within and across reporting periods. The IP also clarified that the same advocacy initiative was counted multiple times if advocating for adoption at different government levels. The Activity also counted organizations that attended different trainings multiple times (however, this has been noted in their PIRS)5 , as well as the organizations participating in grantees’ interventions, and those supported in conducting advocacy initiatives. The latter is not in line with the USAID’s Standard Foreign Assistance Indicator (SFAI) definition. The PPMG has been informed about conclusions and recommendations related to their data quality issues after the DQA site visit in 2017. However, the Activity has not modified their MEL Plan to date. According to the IP, the number of local organizations that received 12 hours of training and counselling on OCA was lower than envisaged, as several allocated grants were in-kind grants and entities receiving such assistance did not receive capacity building assistance. Women’s organizations were not supported by the Program; women were supported by awarding grants to local governments who then distributed assets for female-led start-ups. These sub-grantees have not participated in capacity building interventions, as capacity building was designed for CSOs, not for businesses. Moreover, some grantees were awarded considerably larger grants than originally envisaged, decreasing the overall number of grantees and therefore the number of organizations that received capacity building support for OCA implementation. The IP noted that, in the coming years, they plan to award only two new grants for establishing two new service centers (most likely in Zavidovici and Trebinje). Similarly, when it comes to the number of government officials trained, the IP noted that the USAID requested that only three trainings be organized for government representatives. The trainings were organized in 2015 and 2016 and two of these trainings were merged and included the same audience (youth officials). The IP noted that the training component was discontinued and that no further trainings are anticipated for the life of the Activity. PPMG keeps records and contact information of their grantees in a simple and user-friendly .doc file. However, they do not have a database of their trainees, but keep the trainees’ records in the form of scanned attendance sheets. It was observed that the Activity staff also signs the attendance sheets. Moreover, PPMG does not possess the lists of participants in grantee interventions, advocacy initiatives, or organizations that participated in grantee interventions, but relies on the numbers reported in grantees’ quarterly reports when calculating these indicators. Hence, the numbers that the PPMG reports cannot be verified on site. As explained by PPMG, this is in order to protect the privacy of marginalized populations. 5 It is important to note that all grantees had to attend five obligatory trainings. 21 Finding 2. There is no clear evidence that the PPMG regularly verifies grantees’ performance indicator data. According to the IP, the Activity staff verifies MEL data during quarterly monitoring visits to grantees by reviewing their documentation. However, there is no evidence of such verification in the monitoring reports for IPs other than service centers. Monitoring reports for service centers clearly describe that the IP verified service centers’ data sources for their performance indicators. For other grantees, monitoring reports focus only on fulfilment of pre-award requirements and do not mention monitoring data related to grantees’ performance indicators. Finding 3. The PPMG is facing problems with timely reporting on their performance indicators. The PPMG tracks five indicators to measure progress in meeting LoA targets. At the time of data collection for this performance evaluation, the PPMG did not enter the actuals for the third year of implementation (FY2017) nor the last two quarterly reports (Q2 FY2018 and or Q3 FY2018) into the BiHPERFORM system. In addition, two quarterly reports for FY 2016 are missing from the system. According to the IP, since the Activity collects data from many grantees, a single grantee’s delay in delivering data delays the entire report. Other Contractual Obligations Finding 4. Based on documentation review and KIIs, the PPMG employs high-quality procedures in grant award and management. The PPMG solicits, negotiates, awards, and administers grants on behalf of the USAID. To ensure high quality implementation of grant award and management, the IP contracted an audit company, NGO and Business Audit, whose staff has extensive experience in conducting NGO and project audits. The sub-contractor’s staff plays an essential role in carrying out grant administration tasks. According to the IP, three of four core PPMG team members are employees of the sub-contractor company. This includes the grant manager, financial manager, and organizational development expert. Problems in grant management were reported as isolated cases (2 KIs). Examples included issues with certification of a service center6 and a problematic clause in a grant award. During the four years of implementation, two grant agreements were cancelled. During the first year of implementation, the PPMG developed the grant manual, which was approved by USAID/BiH. The manual describes the grant award and management procedures for three types of grants: unsolicited proposals, directed grants, and annual program statements. According to KIs (with grantees, USAID, and PPMG) and available documentation (quarterly reports, grant awards, pre-award survey results), the PPMG conducted pre-award surveys with all grantees who received these three types of grants7 and developed recommendations grantees needed to fulfill before receiving the grant and/or during grant implementation. According to KIIs and Activity documentation, the PPMG discontinued reviewing unsolicited proposals in May 2017, when TO #1 amendment was issued. The PPMG reviewed unsolicited proposals on behalf of USAID/BiH for about two years and submitted monthly reports with funding recommendations. USAID 6 A KI, representative of a government institution, stated that a service center has not been properly certified for providing services. 7 Pre-award survey had not been conducted with organizations that received in-kind grants. 22 and the IP explained that the discontinuation of the task was an internal decision of the USAID Mission, which decided to continue reviewing this type of proposals internally. Six out of ten grantees felt the reporting requirements were very complex and took too much time. According to the grant manual, grantees were required to submit monthly work plans, monthly program reports, monthly financial plans, monthly financial reports, quarterly program reports and final program reports. Two grantees noted that these procedures took too much of the time they needed to implement their interventions successfully. Finding 5. Some elements of the Award have not been implemented as originally envisaged. For capacity building interventions, one-on-one coaching was not implemented as envisaged. TO #1 envisaged developing a one-to-one coaching plan for each grantee, which includes a three-day intervention (a stay in the field) with a focus on improving organizations’ administration and financial management. Four coaching measures were planned for each organization. However, according to the IP and Activity documentation, coaching was implemented during monitoring and study visits and consultations by email and phone were the dominant type of one-on-one assistance. According to the KIIs and Activity documentation, PPMG employed a different model to support marginalized women compared to the way they worked with other marginalized groups. Local organizations advocating for the rights or providing services to other marginalized groups would receive financial and technical support (capacity building). However, in working with marginalized women, PPMG partnered with local governments and co-financed women’s start-ups instead of women’s organizations. TO #1 (base TO) envisaged implementing 10 networking sessions per year, five among CSOs working on similar issues and five between CSOs and government representatives to improve their collaboration. The number of networking sessions that the Activity implemented cannot be derived from the Activity quarterly reports which don’t mention this task. Upon the evaluation team’s request, implementer provided a list of 13 events they identified as networking by character. Based on their brief description, seven of these events can be considered networking sessions as defined in the base TO (gathering together local organizations, government institutions, or local organizations and government institutions), while there is insufficient information to make such determination for the rest. The IP explained that CSOs showed no interest in networking sessions. However, according to some KIIs with grantees, facilitating networking among organizations focused on similar issues was a much appreciated unintended effect of trainings for local organizations and government institutions. 23 Exhibit 5. Activity Indicators, with Targets and Actuals for FY 2015, 2016, 2017, and 2018 and Life of Activity Targets Level of result Narrative summary Indicators Actuals (Targets) Life of Activity targets Deviation from target FY 2015 FY 2016 FY 2017 FY 2018 Activity purpose 1.2.3 Individuals and CSOs representing underrepresented groups are constructively engaged in civic/political issues 1. Number of underrepresented people assisted by a USG-funded intervention providing services to beneficiaries 495 (300) 5,245 (600) 4,580 (3,000) 3,818 (3000) 8,900 +5,238 Activity sub￾purpose 1 Activities of groups of local organizations that advocate for the rights and dignity of underrepresented groups are supported (Corresponds to USAID 1.2.3.1 CSOs representing and led by underrepresented groups advocate for policy changes) 2. Number of advocacy initiatives focused on improving the status of youth, women, the Roma, the disabled, or LGBTI in the public and political life in BiH 0 (0) 4 (5) 98 (60) 99 (60) 185 +16 Activity outcome/ output 1.1. The capacity of local NGOs, BiH institutions and USAID implementing partners is built. 3. Number of human rights organizations trained and supported 27 (6) 166 (30) 107 (100) 126 (80) 296 +130 4. Number of organizations representing marginalized populations that receive minimum 12 hours of training and counseling on OCA tool 0 (0) 14 (8) 6 (8) 5 (8) 32 -7 5. Number of representatives of government institutions in BiH trained to increase their organizational capacities 12 (15) 0 (25) 30 (25) 0 (25) 115 -73 Source: BiHPERFORM and USAID/BiH 24 Finding 6. Financial support was most intensively provided to organizations working with persons with disabilities and their families. Although the PPMG design envisaged provision of support to persons belonging to five marginalized groups (youth, women, persons with disabilities, Roma, and LGBTI), according to the Activity documentation, approximately 50 percent of total funding has been spent on interventions that aim to assist persons with disabilities and their families, and about 30 percent has been spent on the women’s entrepreneurship program. One-fifth of funds were allocated for interventions focusing on youth, Roma, and LGBTI (see Exhibit 6 for more detail). Exhibit 6. Allocation of program funds, by type of assistance (left-hand side) and by type of marginalization (right) Source: www.ppmg.ba Conclusions PPMG is facing challenges in reporting on MEL indicators, including problems with definitions of indicators, timeliness of reporting, deviation from targets for all indicators, and lack of adequate raw MEL data. The Activity over-performed on three indicators and recorded slow progress on two. Issues with reaching targets may partially be due to the Activity’s contract type (ID/IQ). On those indicators where targets have yet to be met, the Activity will probably not be able to reach the targets defined in the MEL Plan, given that the Activity will award only two new grants in the following years and that the trainings will be discontinued. The PPMG performance of grant award and management tasks has been of high quality. The PPMG conducted pre-award surveys with all potential grantees and followed through to ensure implementation of recommendations. They successfully managed and monitored grant implementation. The Activity has not implemented several elements of the TOs as planned or to the extent planned (one-on-one coaching, support for marginalized women, networking sessions for local organizations and government institutions). In addition, the PPMG has strongly focused on providing support to local organizations working with persons with disabilities and the women’s entrepreneurship program. 25 EVALUATION QUESTION 2: RESULTS ACHIEVED THROUGH CAPACITY BUILDING INTERVENTIONS To what extent has the organizational capacity of PPMG-supported organizations (both grantees and trainees) been improved? What is some of the evidence that this has happened? Finding 7: According to KIIs and the online survey, the PPMG provided intensive capacity building and valuable guidance to grantee organizations8 through support in implementation of pre-award survey recommendations, the OCA Tool, and trainings. Before awarding a grant to an organization, the PPMG conducted pre￾award surveys with potential grantees to assess their organizational capacity and the financial risks of awarding a grant. Based on the pre-award survey results and KIIs, all local organizations needed to meet certain conditions to receive a grant award. 9 Subsequently, all grant recipients10 had to fulfill additional requirements in the first six months of grant implementation. In addition to these requirements, all grantees were required to conduct a self-assessment using the Organizational Capacity Assessment (OCA) Tool and develop and implement their capacity building plans with PPMG support. Implementation of pre-award and OCA recommendations provide a learning opportunity for grantee organizations. Although all grantees described pre-award and OCA processes as complex and sometimes “painful” or “stressful,” 11 five grantee organizations stated that the PPMG was very open and available to provide them with individualized, one-on-one assistance whenever needed (most often by phone or email, but also through quarterly monitoring visits to grantees). Six of seven interviewed grantee organizations stated that the pre-award survey and OCA assessment had been useful to their organizations. An online survey of grantee organizations confirmed positive views about the pre-award survey and OCA Tool (see Exhibit 7.). All grantee organizations interviewed changed their existing organizational policies and/or adopted new ones to meet the PPMG 8 For the purposes of this evaluation, we interviewed 10 grantees, seven local organizations and three local government institutions. 9 According to two KIs, such requirements discourage small local organizations from applying for USAID-funded grants. 10 According to the Activity documentation, the only exception is Internews, an international non-profit organization. 11 Most organizations failed to comply with all requirements within six months and in such cases the PPMG defined new deadlines. A couple of organizations failed to comply with the new deadlines and their funds were temporarily suspended until the requirements were met. Based on the analysis of grant awards, the number of recommendations a local organization had to meet ranged from 3 to 14; on average, local organizations were required to implement 9 recommendations. Recommendations most often required changes in regulations related to accounting, financial operations, procurement, and organizations’ statutes. They provided us with tremendous support. […] I am not sure what would have happened if we did not have somebody to guide us. They led us and I am not sure we would have had 1% of what we have if it was not for them. – a grantee It was very important to me that I knew I could ask them about anything. Usually, with big donors there is no room for such questions. […] What was fascinating to me is that Kult, or the PPMG, every time had a clear and specific answer and they were so open, so you could ask them questions anytime, day or night. That was really great. – a grantee It helped to establish the system in line with best practices available, and now we have it all. When applying for financial assistance we have all policies, the whole system... Everything we do, we do it by these rules. We developed a framework and it really helps us make sure that everything we do is in compliance with applicable regulations. – a grantee 26 requirements. PPMG provided assistance in this process. Several grantee organizations (4 out of 7 KIs12) found organizational policy development useful and stated that they continued to use these policies when appropriate. As noted by the IP, one of the most important accomplishments in terms of aligning their policies with international standards was that all grantee organizations introduced segregation of duties into their accounting policies. Exhibit 7. Grantees' ratings of pre-award requirements, OCA Tool, and one-on-one assistance Source: MEASURE-BiH (2019), Online survey of PPMG grantee organizations All grantee organizations attended obligatory trainings covering five topics aimed at building their organizational capacity. All grantee organizations described these trainings as interactive, practical, and useful, and teachers as highly competent. Additionally, online survey results indicated that grantee organizations felt that their skills improved due to the PPMG capacity building assistance (Exhibit 8). As presented in Exhibit 9, grantees particularly noticed skill improvement with regard to administrative procedures, monitoring and evaluation, and financial management and reporting. Grantees believed that they experienced more limited skill improvement in human resource management, advocacy, and fund raising. The OCA Tool requires that organizations assess their capacities in ten different areas13 on a scale from 1 to 4, with 1 being the lowest and 4 the highest level of development. Based on the review of OCA results for 18 grantee organizations that implemented the OCA assessment twice, all organizations but one14 increased their capacity during the course of the implementation of their intervention. On average, organizations scored 2.74 on their first assessment and 3.03 on the second assessment (see Exhibit 10). 12 Only grantees – local organizations answered this question. 13 The areas include: strategic management; relationships with members and target groups; project management; capacity for advocacy and monitoring of public policy implementation; providing services and care for beneficiaries; collaboration, partnership, and networking; management and organizational structure; human resources; finances and administration; media and public relations. 14 OCA results indicated that this organization’s capacities had been high to begin with, and the organization showed improvements after the third OCA assessment. 27 Exhibit 8. Satisfaction with trainings among PPMG grantees Source: MEASURE-BiH (2019), Online survey of PPMG grantee organizations Exhibit 9. Grantees’ ratings of influence of trainings on their skill improvement (average rating of assessments on the scale from 1 to 4, 1 being “Not at all” and 4 being “Extremely effective”) Source: MEASURE-BiH (2019), Online survey of PPMG grantee organizations Finding 8. According to OCA results, the service centers showed higher capacity improvements compared to other grantees. The changes in OCA results were highest for the service centers (0.45 increase on the 1-4 scale), followed by recipients of directed grants (0.35 increase). The lowest change in OCA was among organizations that implemented counter-trafficking and deinstitutionalization initiatives (0.18 increase). 28 Exhibit 10. OCA results for service centers, annual program statements, and other grants Number of organizations OCA – 1 st assessment, average rating OCA - 2 nd assessment, average rating Change Service centers 4 2.60 3.05 0.45 Annual program statements 9 2.62 2.98 0.35 Other grants 5 2.87 3.06 0.18 TOTAL 18 2.74 3.03 0.29 Finding 9: Representatives of local organizations and government institutions who only participated in trainings had positive perceptions about trainings (10 KIIs) and sometimes reported changes in their policies and/or work performance due to the PPMG capacity building. Local organizations that did not receive PPMG grants applied for trainings through public calls for proposals. The intensity of capacity building assistance provided to these organizations varied from attendance at a single training to participation in all trainings. According to KIs, all training participants found them to be interactive, practical, and useful. An online survey of training participants shows the same results, with the majority of participants being satisfied or very satisfied with all aspects of trainings (see Exhibit 11). According to all KIIs and most online survey respondents, experts who delivered the trainings had adequate expertise and teaching skills. Although some trainees participated in just one or two trainings, some of these local organizations changed their organizational policies due to trainings (e.g. revised their strategy or statutes), even though they were not required to do so. However, three out of five trainees stated that trainings were not tailored to their organizations’ development level, i.e., the trainings were too advanced. In addition, one trainee noted that the allocated time was insufficient to learn about the training topics. As presented in Exhibit 12, most local organizations felt that the skills of their staff improved in all areas covered by PPMG trainings. These trainees found the trainings on project management, compliance with legal procedures, and administrative procedures to be most effective. The trainings were perceived to have the least impact on human resource management, advocacy, and fund raising skills. I attended a training on project writing. At the time, I was a new employee in the association, I knew nothing about project writing. After the training, everything was clear to me. […] I am very satisfied with how practical it was and how dedicated the trainer was to each participant. It was important to him that we understand, and he knew how to present it so we would. Most of us were beginners who never had an opportunity to write projects and he really achieved that we understood. - a trainee, representative of a local organization 29 Exhibit 11. Satisfaction with trainings among PPMG trainees Source: MEASURE-BiH (2019), Online survey of PPMG trainees Exhibit 12. Participants’ ratings of influence of trainings on their skill improvement (Average rating of assessments on the scale from 1 to 4, 1 being “Not at all” and 4 being “Extremely effective”) Source: MEASURE-BiH (2019), Online survey of PPMG trainees 30 The PPMG organized three trainings for government representatives – youth officials. Two out of five government leaders did not remember the trainings in which they participated. This may be because the trainings were delivered in 2015 and 2016. When reminded about the training topics, facilitators, and locations where the trainings took place, these key informants shared positive perceptions about trainings and the experts who delivered them. KIs from government institutions felt that the PPMG capacity building effort helped bring about improvements in their work. One government official noted that, although he acquired new knowledge in the training, he did not have an opportunity to apply his knowledge in his institution because his position in the institutional hierarchy was too low. Exhibit 13. Government representatives’ ratings of PPMG trainings Source (2019): MEASURE-BiH, Online survey of government representatives who participated in PPMG trainings Finding 10. According to the KIIs and Activity documentation, among all training topics, gender equality sessions were the least useful to trainees. According to the IP and Activity documentation, the PPMG implemented 41 gender equality sessions during the first four years of implementation. The trainees were taught about the Law on Gender Equality in 90-minute sessions incorporated in other training events. An expert noted that the time envisaged for these sessions was insufficient to learn about the Law on Gender Equality and that the same participants attended multiple trainings, so for some trainees, gender equality sessions were repetitive. According to evaluation sheets that the PPMG distributed to trainees after each training, gender equality sessions were repeatedly perceived as the least useful among the grantees and trainees. According to participants’ comments, these sessions were not well integrated into the overall training topics. I recommended to the youth from the Youth Council to attend Kult’s trainings. And now, we have young persons educated to work with youth, to be youth officials. They are peer educators who will share their knowledge. – a trainee, representative of a local government I initiated development of the strategy for youth in our municipality, because the old one had expired, and then I worked on the new one with a local non￾government organization. We developed the whole strategy with Kult’s support, not direct support, but advisory support. – a trainee, representative of a local government 31 Finding 11: No capacity building assistance was aimed at women’s organizations. The PPMG employed a different model in assisting marginalized women when compared to other marginalized categories. Specifically, the PPMG partnered with local governments to finance female-led start-ups. However, according to KIIs, this type of support was limited to financial assistance. FGD participants, women who started their companies with this assistance, noted that they lacked technical support throughout the process. The IP agreed with these statements, but pointed out that they lacked the capacity to provide such assistance. The Program did not directly support any women’s organizations. Finding 12. Multiple development organizations offer similar trainings available to PPMG grantees and other local CSOs. One international organization mentioned delivering trainings to local organizations and covering topics similar to PPMG; four PPMG grantees participated in these sessions. A couple of grantees also mentioned attending trainings on similar topics. Moreover, four people who participated in the survey of grantees stated that they attended similar trainings organized by the BBRZ Gruppe Austria, Czech Development Agency, and Center for Civil Society Promotion. Conclusions The results indicate that the PPMG assistance was effective in building the capacity of local organizations. The strongest effects were noted among grantee organizations, particularly service centers, which received the most extensive capacity building support (pre-award survey, several rounds of OCA, trainings, one-on-one assistance). These organizations developed all organizational policies required by the law and the donor community. Although perceived as complex and time-consuming, the PPMG’s assistance in implementation of pre-award recommendations and the OCA Tool was useful and appreciated by grantee organizations. Although they did not receive high-intensity continuous training as grantee organizations, non-grantee beneficiary local organizations felt that PPMG capacity building motivated them to start thinking about changing their organizational policies. Similarly, government officials felt that their skills have been enhanced due to PPMG trainings. The perceived quality of gender equality sessions was lower compared to other parts of the trainings; apparently they have not been well integrated within the overall training topics. Moreover, the overall trainings were repetitive for some organizations given that several other organizations work on building organizational capacities of local organizations in similar domains. 32 EVALUATION QUESTION 3: IMPROVEMENTS IN MARGINALIZED GROUPS’ REPRESENTATION IN POLITICAL AND CIVIC ISSUES To what extent has the representation in civic and political issues of PPMG-supported organizations been improved? Are there indications showing that supported local organizations are interacting more often and more effectively with public authorities and/or the public at large? Finding 13: Service centers “Dajte nam sansu” are positive examples of local organizations that considerably improved the representation of persons with disabilities and their families in political and civic issues as a result of PPMG assistance. USAID/BiH provided financial support to the organization “Dajte nam sansu” after its establishment in 2014. The organization set up the first service center in BiH to provide holistic services to families of persons with disabilities. 15 Initially, the organization operated with USAID’s funding only. Today, the organization receives 60 percent of its funding from other sources (including funds from cantonal and local governments within the canton, and private sector) and has started to apply for other donors’ funding. The number of families benefiting from their services has increased over the years, as they successfully built trust with families in need. For example, the service centers in Sarajevo served 175 families in 2015, and by April 2019 the number of families who used the service center services increased to 581. Representatives of the Sarajevo service centers are educating teachers, students, and other organizations about people with disabilities and their rights and needs. The organization initiated development of the standards for the operation of service centers, worked on their development, and delivered about 14 trainings to other local organizations interested in starting service centers. In addition, the service centers have initiated a number of policy changes with regard to persons with disabilities and their rights, including policies on inclusive education and teaching assistants and changes to the FBiH Law on Social Protection. The service centers established relationships with local and mainstream media, including Klix.ba, BHRT, and RTRS. The PPMG established six service centers in five years, 16 four in FBiH and two in RS. Based on the analysis of the service center progress reports and KIIs, all service centers have shown improvements in their interactions with the government and increased government support for their interventions, although the first service center in Sarajevo is at the forefront in this regard. 15 The services include short stays in the service center, psychological and legal aid for family members, emergency assistance, transport, educational and recreational activities. 16 Service centers were introduced in three cantons in the FBiH (Sarajevo, Tuzla and Zenica) and the City of Banja Luka and Foča Municipality in the RS. However, the service center in Tuzla was closed during the course of the implementation because of the local organization’s noncompliance with USAID requirements. The PPMG is planning to find a new partner to run the service center in the Tuzla Canton. For me, short stays are the most important service. And they are always calling, telling us that we can come and exercise, do yoga, talk to a psychologist. They are always there. In our lives, everything is the same, every day is the same. And then they come with a change. Their spirit is amazing. I admire them. – a focus group participant 33 Finding 14: The service centers model developed by the PPMG is not aligned with the RS bylaw that regulates certification of social services. 17 According to a government official, the service center operating in Banja Luka is not properly registered to provide the services it is currently providing. Since the Ministry of Health and Social Protection is responsible for certification of social service providers in the RS, the local grant recipient “Zvjezdice” in Banja Luka submitted their application for certification in 2018. However, their application was rejected on the grounds that the request was non-compliant with requirements specified in the bylaw. According to a KI, the implementing partner failed to inform the competent ministry of activities implemented in the RS, and the conference organized in May 2019 in Banja Luka was the first encounter between the responsible civil servants from the RS Ministry of Health and Social Protection and the representatives of “Dajte nam šansu – Zvjezdice” from Banja Luka. Despite this, the service center in Banja Luka “Zvjezdice” continues to receive demands from citizens/beneficiaries from Banja Luka and is supported by the City of Banja Luka in provision of these services. Finding 15: Apart from the service centers, most grantee organizations implemented interventions that to some extent involved government and public institutions. According to KIIs, all grantees believed that their interactions with the government improved as a result of the PPMG assistance. Based on the analysis of their progress reports and final reports of 18 grantee organizations, 14 organizations had some interaction with government and/or public institutions. A couple of grantee organizations directly targeted representatives of government and public institutions in their interventions (Sarajevo Open Center-SOC and Otaharin) and considerably increased their interaction with these institutions. Out of 12 other grantee organizations covered in the analysis, four implemented advocacy activities with cantonal or local level institutions, two organizations collaborated with cantonal level institutions, two collaborated with institutions at the municipal level, and three mentioned cooperation with public institutions (i.e. schools, social protection institutions, and hospitals). In all, only four grantee organizations did not include government or public institutions in their interventions, and only one included a higher-level government. Finding 16: There is evidence that grantee organizations made some contribution to policy making. However, few of these policy documents have been adopted to date. According to our analysis of KIIs and grantees’ progress reports, only two of six organizations that initiated some policy changes to improve the status of marginalized groups had some partial success. The most significant initiatives came from the association “Dajte nam šansu,” whose members are the main proponents of the need to amend the FBiH Framework Law on Social Protection. The amendments to the law were officially entered into parliamentary procedure by a SDP member of the FBiH House of Representatives and were endorsed by a vast majority of votes in the FBiH House of Representatives in January 2019. The amendments envisage the introduction of a special professional status of a career parent or other family member providing long-term (24 hour) care to a family member with a disability. Benefits would include a financial allowance, as well as a pension and health insurance to be funded from the budget. This initiative needs to be endorsed by the FBiH House of Peoples before it can become a legislative proposal. The “Dajte nam šansu – Zvjezdice” from Banja Luka are behind the initiative to update the entity bylaws on certification of service centers. 17 The Rulebook on Conditions for Establishment of Institutions of Social Protection and Social Protection Activities (Službeni glasnik of Republika Srpska 90/17). 34 In the Tuzla Canton, the grantee organization (Zemlja djece u BiH) advocated for health insurance of children with disabilities who cannot attend schools because of their disability, which resulted in a resolution that the cantonal authorities would finance medical expenses of these children. Some of these children were without health insurance because the Cantonal Law on Health Insurance stipulated regular school attendance for school age children as an eligibility criterion for health insurance. Other examples of grantees’ contributions to policy development include the Action Plan for Equality of LGBTI and Women (Sarajevo Open Center), the local action plan for LGBTI rights (Helsinki Committee for Human Rights Bijeljina), and a local action plan for the rights of persons with intellectual disability (Sunce Mostar). Finding 17: Most PPMG grantee organizations interact with the public, but in a limited manner. Considerable work has been done by the “Dajte nam sansu” service center in Sarajevo. The service center staff invested substantive efforts to inform the public and families of persons with disabilities about the center and notify them that, in addition to children with disabilities, their family members also need and deserve support. The center also included several companies in corporate volunteering, and the volunteers obtained first-hand experience of working with service center beneficiaries and their families, and could better understand obstacles which they face. The Center involved all schools in Sarajevo Canton in inclusive workshops for children without disabilities, their parents, and teachers, and brought children without disabilities as volunteers into the service center. Other service centers also interact with the public, but to a lesser extent. Other grantees also contributed to improved public perception of marginalized groups. Based on the documentation review, 14 out of 20 grantees had some interaction with the public during the course of implementation. However, these were small-scale interactions with the local public. For example, the Sarajevo Open Center put considerable effort into raising awareness about LGBTI issues, but mainly through public campaigns disseminated through their own websites (soc.ba and lgbti.ba)18. In addition, Narko-Ne worked on raising awareness about child begging in six cities across BiH. Finding 18. PPMG grantees’ interventions received local media coverage, however only some organizations reported coverage by the mainstream media. Based on grantees’ progress reports, nine out of 18 organizations (excluding four service centers) reported that their interventions were covered by the mainstream media, including Klix.ba, nezavisne.com, avaz.ba, BHRT, FTV, or RTRS. The service centers, SOC, and the Association for Advancement of Education and Support for Children with and without Developmental Delays EDUS reported the most extensive coverage in the mainstream media. Some interventions that achieved important results, such as Proreha, which helped find employment for 12 persons with disabilities, and SUMERO, whose intervention resulted in deinstitutionalization of 60 persons with disabilities, did not receive extensive media coverage. According to the IP, grantee organizations lacked capacities to reach out to the mainstream media. The IP supported grantee organizations by helping draft and disseminate media materials to relevant media outlets. According to the Activity documentation on press-clipping, the coverage in the mainstream media was highest at the start of the project (for the second half of 2015), and kept declining during the project. Finding 19: There is no evidence that local non-grantee organizations or government institutions that attended PPMG trainings improved their representation in political or civic issues. PPMG 18 During development of the final report, the first Bosnian Pride was organized in Sarajevo. Sarajevo Open Center’s staff was highly involved in the organization of this public protest. 35 provided capacity building technical assistance to local organizations at different development stages; some were very small, others were larger and better known in their communities. The intensity of capacity building varied between these organizations. Some organizations attended a single training, while others were exposed to the entire set of trainings. These organizations have not received intensive one-on-one support nor financial support. According to five KIIs with local organizations, the PPMG trainings have not resulted in improvement of their representation in government or political issues. However, the PPMG assistance provided them with the direction they needed to strengthen their organizational capacity. The government leaders who participated in trainings for youth officials felt that their interactions with youth improved as a result of the PPMG intervention. Out of five government leaders interviewed, three confirmed that PPMG interventions helped them improve their interaction with youth and perform their duties as youth officials. One of these KIs stated that she even obtained her job as youth official due to knowledge she gained in PPMG trainings. Finding 20: Eighteen local self-government units in BiH financially supported women-led start-ups due to the PPMG intervention. Although this intervention improved financial situation of some women, there is no evidence that these interventions improved participants’ perceptions about women’s roles in society or their representation in political and civic issues. The PPMG supported marginalized women in BiH by investing assets to support entrepreneurship among marginalized women. The PPMG advertised the Women Entrepreneurship Program in all municipalities in BiH. Municipalities decided on the size of their investment19 and the PPMG matched that amount. According to the IP, representatives of 79 local government units applied to invest in start-ups for marginalized women and 18 municipalities received PPMG funding. The first group of businesses were established in 2016, with nine municipalities supporting 96 businesses. Without the PPMG, it is likely that these funds would have been allocated for other purposes, as municipalities did not previously have such programs for women (according to the IP). Six municipalities that participated in the first round of the program responded to the online survey. The results showed that these six municipalities supported 69 businesses. Of this number, 44 businesses, or 63 percent, are still operational. The analysis showed that most grants were awarded to support unemployed women (28 out of 44), and that other categories of marginalized women (i.e. victims of violence, mothers of children with disabilities) were less represented. Although women’s interaction with local governments improved during the grant implementation, there is no evidence that this type of assistance had long-term effects on political or civic participation of women. Further, there is no evidence that it improved perceptions about women’s traditional role in the society, or about women’s rights. More specifically, the conclusions of the focus group discussion with women who started their businesses with PPMG support suggest that women’s attitudes toward gender roles have not changed and that women have not become aware of their rights. Out of six participants, only one 19 The condition was that the investment could not be lower than 20,000 BAM and no higher than 100,000 BAM. I don’t need to do some things to prove that I am equal to him, things I do not feel good doing. For instance, if something is hard in a physical sense, I do not want to prove myself. If he says that something is not for me, and I do it anyway, a problem always appears, although he let me do it. He says “you can’t, you may not”, and I do it, and it always proves that he was right in the end… I respect rights of a man and rights of a woman, and I am not asking for equality. – a focus group participant 36 woman seems to have started a business that she actually runs on her own. Other women seem to have used the grants to support their family businesses led by men (four FGD participants) or to obtain their first job (one FGD participant). Finding 21. According to the PPMG website, most organizations financially supported by the Program have been registered in the Canton Sarajevo. The PPMG’s support to organizations in other regions was more limited. According to the PPMG website, the Program supported 15 organizations in the Canton Sarajevo (15 interventions), followed by four in the Tuzla Canton and four in the eastern RS. The Program supported eight organizations from the RS, three in Herzegovina, and no organizations from the Brcko District (see Exhibit 14). When it comes to the Women’s Entrepreneurship Program, the participating municipalities were spread out across BiH. However, several small cantons, i.e., Posavina and Bosnian Podrinje, received no support within this program (see Exhibit 15). Exhibit 14. PPMG Interventions with local organizations (grantees), geographic distribution Source: www.ppmg.ba 37 Exhibit 15. PPMG municipalities supported through Women Entrepreneurship Program, geographic distribution Source: www.ppmg.ba Finding 22: LGBTI rights are still a taboo topic among the public and in government institutions and the PPMG seems to have had the most limited success in this area. PPMG disbursed three grants to LGBTI organizations20 . The organizations promoting LGBTI rights invested considerable efforts to improve their representation in political and civic issues during the PPMG, by improving their collaboration with government and citizens. However, their efforts had limited reach, as they focused on three geographic regions (Sarajevo, Tuzla and Prijedor) and they have rarely been covered by mainstream media. Representatives of government institutions interviewed were reluctant to talk about LGBTI rights and none of those interviewed stated that their institutions provided support to the organizations working in this field. Also, of all local organizations we spoke with, a local organization focusing on LGBTI rights is the only one interviewed that receives no funding from government sources. The number of organizations representing LGBTI rights in BiH is still small; according to a KI, there are only three registered organizations in the country that are focused primarily on the LGBTI population. However, KIs and focus group participants noticed changes in their position in the last five years, pointing out that, five years ago, holding the Pride event would not even have crossed their minds, but currently they are preparing for the Pride in September 2019. However, the representatives of this organization 20 Two grants were given to the same organization. 38 noted that the Pride may not happen if they assess that security of participants cannot be guaranteed21. It is important to note that multiple donors support these organizations, so the results can be only partially contributed to the PPMG assistance. Finding 23: Most stakeholders report that changes in the social policy domain and public perceptions take longer than the period of time the grant covered. The changes in social policy depend on a myriad factors, e.g., competence of responsible civil servants, budgetary constraints, political dynamics and change of political officials in office, as well as discriminatory attitudes towards certain groups of population. The BiH system of social protection is complex and highly fragmented, with different and sometimes overlapping competencies devolved to different levels of government. As the PPMG primarily supports grassroots organizations working directly with beneficiaries and oftentimes they provided a short-term support, in the BiH context, the impact of these initiatives on policy and public perceptions can take longer than the time covered by the grants. Finding 24: Networking with other similar organizations is an important outcome of trainings for local organizations, as reported by several KIs. Three KIs noted that, due to PPMG trainings, they established collaboration with other local organizations and had the opportunity to exchange their experiences and examples of good practices. Moreover, a government representative noted a similar effect among the representatives of local governments. The importance of networking among local organizations was also mentioned in PPMG’s surveys conducted after each training. Finding 25: Some local organizations supported by PPMG became mentors to other organizations as a result of extensive capacity building. For example, the Sarajevo Open Center has two sub-grantees to which they provided assistance in strengthening organizational capacity. Similarly, the first established service center initiated the development of standards for operation of the service centers. This organization provided mentorship to organizations establishing new service centers, as well as to several other local organizations interested in such interventions. Moreover, the Mostar Rock School was supported by the Norwegian Embassy in assisting the Srebrenica Wave, a local organization from Srebrenica, in building their capacity. According to the PPMG documentation, the Mostar Rock School uses the methodology and experiences acquired through the PPMG in their capacity building work with this organization. Conclusions The service centers established by the PPMG are positive examples of local organizations that considerably improved their capacities and representation of persons with disabilities and their families in political and civic issues. Apart from the service centers, grantees supported by the program made limited contributions in terms of participation in policy-making and interactions with the general public. Some grantee organizations financially 21 As mentioned above, during the development of the final evaluation report, the first Bosnian Pride was organized. The preparations were followed by negative political rhetoric, few cases of violence, and two protests against the event were organized during the same weekend when the event took place. However, the police was very professional in providing the security of the participants and no problems occurred during the event. The main problem is that it was a short-term intervention. You cannot make a change with (BAM) 15,000 in a year, there is no way. So, I am wondering what we have gained from this, what have the children or institutions gained, except a couple of brochures we brought them and told them it was a donation? We had started really well and just when we got used to one another, the project ended. – a representative of a grantee organization 39 supported by the PPMG became mentors to other organizations owing to intensive capacity building they had received. There is no evidence that local organizations that received only technical assistance improved their representation in political and civic issues, or that the Women’s Entrepreneurship Program improved women’s role in society. Moreover, the PPMG achieved limited results in terms of improving public perception of marginalized populations. LGBTI rights are still a taboo subject in the public discourse and the least progress has been achieved in this sphere, despite considerable efforts. However, most stakeholders recognized that changes in social policy and public perceptions take a longer time than the grant period allowed. 40 RECOMMENDATIONS The following recommendations have been developed based on the evaluation findings and conclusions. USAID should consider employing the following actions for the ongoing and future interventions supporting marginalized groups:  In terms of PPMG MEL Plan, consider conducting a thorough review and revision of the Activity MEL Plan. This would include the following: specifying indicator definitions; revising annual targets for the remaining two years of the Activity (particularly for the indicators referring to number of organizations that receive 12 hours or more of OCA consultations and number of government leaders trained, especially if no more grants are to be awarded or if the trainings are to be discontinued); developing an anonymized database with information on grantees, trainees, and grantees’ beneficiaries; developing simplified MEL Plans for grantees and training them to measure performance indicators; simplifying administrative requirements for grantee organizations.  For future interventions implemented through TOs, consider developing a general MEL Plan including all obligatory elements except targets and adopt individual MEL Plans with specified targets for each TO.  Continue providing extensive capacity building assistance to grantees but consider expanding capacity building interventions (besides formal classroom trainings) of selected (highly motivated) small organizations and monitor their progress. This could include implementation of the OCA Tool, obligatory participation in all trainings, and one-on-one assistance. Intensive capacity building assistance to small organizations could be delivered through mentoring by organizations that already considerably improved their capacities through the program, as a requirement for continuing funding. Human resource management, advocacy, fundraising, project proposals, and income diversifications are some of the training topics which, in their own opinion, could benefit all organizations. All trainings should be tailored to the needs and development level of the recipients. For important and/or complex topics, separate trainings could be delivered to basic, intermediate and advanced beneficiaries. The concept of gender equality sessions should be revised for better integration into other training topics. The new training concept could be developed through collaboration of technical experts and gender specialists. Gender equality sessions should be interactive and practical.  Continue supporting and expanding service centers based on the already established model. Facilitate the mentorship role of the existing service centers.  Continue working with government officials to raise their awareness about marginalized populations and to assist them in providing better-targeted support to local organizations. Facilitate grantees’ efforts to improve their interactions with the authorities by incorporating such components in the interventions they are implementing.  Integrate intensive, high-level awareness-raising campaigns into all interventions supported through the program. Facilitate collaboration and networking of local organizations working on same issues to implement such campaigns and influence the general public. Build better relationships with the mainstream media and promote examples of good practices to raise awareness and combat prejudice about marginalized populations.  Consider providing technical assistance to women supported through the Women Entrepreneurship Program. Require the partnering local governments to provide such support and monitor their performance. Implement awareness-raising interventions among these women on women’s rights, gender stereotypes, and prejudice. 41  When implementing interventions, maintain regular flow of information and regular communication with competent government institutions at all levels.  Coordinate with other donors when selecting organizations for capacity building.  Instead of short-term support for more organizations, provide long-term and intensive support to a limited number of local organizations to maximize the effectiveness of interventions. 42 IV. ANNEXES ANNEX I: EVALUATION AND ASSESSMENT STATEMENT OF WORK INTRODUCTION AND BACKGROUND INFORMATION INTRODUCTION The Marginalized Populations Support Activity (PPMG) is a USAID/BiH-funded Activity implemented by the Institute for Youth Development KULT. This Activity contributes to Project Purpose 1.2 – Increased citizen participation in governance – under Development Objective 1: “more functional and accountable institutions and actors that meet citizens’ needs.” IMPAQ International (IMPAQ) has been commissioned by USAID/BiH within the Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct the performance evaluation of the PPMG Activity and a brief assessment of the status of marginalized populations in BiH. The performance evaluation component will analyze the Activity’s design, progress toward expected results, and implementation. The brief assessment will focus on the effectiveness of local organizations assisting marginalized populations at raising funds from different stakeholders and potential changes in status of marginalized populations in the last five years. The evaluation and assessment will employ rigorous methods and design to capture high-quality data and produce credible findings, conclusions, and recommendations. Exhibit 1 presents the Activity details. Exhibit 1. Basic Information on Marginalized Population Support Activity Activity Name Marginalized Populations Support Activity (PPMG) Contractor Institute for Youth Development KULT Contract # AID-168-I-15-00001 Total Estimated Cost $4,999,000.00 Life of Activity February 22, 2015 to February 22, 2022 Target Groups Youth, women, Roma, people with disabilities, sexual minorities CDCS Intermediate Result IR 1.2 Increased citizen participation in governance BACKGROUND INFORMATION ABOUT MARGINALIZED POPULATIONS SUPPORT ACTIVITY According to the PPMG Statement of Work, the primary objective of the Activity is “to rapidly respond to the changing socio-political situation of marginalized populations and tense political situation in BiH.” The Activity design envisaged assistance to youth, women, people with disabilities, the Roma, and LGBT individuals. PPMG aims to increase underrepresented citizens’ participation in governance by providing technical, material, and financial assistance to organizations and institutions that work with these marginalized groups. 43 PPMG’s tasks are organized under three components: I. Grants award and management II. Human and organizational capacity development III. In-kind logistical support Component 1: Grants Award and Management Component 1 aims to provide funding opportunities to local organizations that advocate for the rights of underrepresented groups. On behalf of USAID/BiH, PPMG awards and manages three types of grants to such local organizations: unsolicited grants (awarded based on review of unsolicited proposals received by USAID/BiH), directed grants (allocated through USAID’s internal selection process), and annual program statements (awards for initiatives related to specific areas connected to USAID/BiH Development Objectives). PPMG is responsible for conducting pre-award surveys with all grantees and for developing a grants manual. In addition, PPMG is obligated to ensure that all grant management procedures comply with USAID rules and regulations, and must obtain Contracting Officer Representative’s (COR) approval for all grant awards. Component 2: Human and Organizational Capacity Development The second component aims to build the capacity of PPMG grantees through technical assistance in program and financial management. This includes trainings, mentorship, and one-on-one coaching. Non-governmental organizations (NGOs), associations, civic society organizations (CSOs), BiH institutions, and small and medium enterprises (SMEs) are eligible for technical assistance. PPMG is expected to achieve the following results:  Allocate grants to at least 40 organizations, associations, institutions, or SMEs (approximately 8 per year)  Provide capacity building technical assistance to at least 40 organizations, associations, institutions, or SMEs (approximately 8 per year)  Provide capacity building technical assistance to 75 current USAID local partners to improve their performance and management capacity (approximately 15 per year)  Build expertise of at least 75 (approximately 15 per year) BiH government leaders working in collaboration with USAID 44 Component 3: In-kind Logistical Support The third component is focused on providing humanitarian and/or material assistance (e.g. food items, clothing, vehicles, etc.) to improve the well-being of marginalized groups in cases of unanticipated events. PPMG is also responsible for providing logistical support in such cases. PLANNED AND ACHIEVED RESULTS USAID/BiH issued 10 task orders (TOs) to PPMG implementers. Under TO #1, the Activity aimed to build the capacity of PPMG grantees, USAID local partners, and government leaders, and assist USAID/BiH in reviewing unsolicited proposals. The remaining TOs cover specific grants awards, management, and monitoring. Exhibit 2 presents a summary of tasks under each TO (TOs highlighted in grey have been completed). Exhibit 2. Summarized Tasks by Each TO Task Order Tasks #1  Conduct organizational capacity assessments of eight sub-grantees, develop Capacity Development Plans for each organization, and support their implementation through one-on-one trainings, coaching and mentoring (deliver 5 trainings for each sub-grantee - 15 training days each; 15 coaching and mentoring sessions; establish support line for sub-grantees; organize three study visits; establish online education platform, develop 8 M&E Plans), and hold roundtable events dedicated to monitoring and evaluation  Build capacity of at least 15 USAID local partners and government leaders through eight 3-day trainings  Implement 13 gender equality and female empowerment training sessions and 10 networking meetings  Finalize the Grants Manual  Review USAID/BiH unsolicited proposals (8 to 10 per week, 450 per year; canceled in 2017 after TO modification)  Visibility activities #2  Award small grants to seven local organizations (this list was modified, some organizations were replaced with new ones, in total there are now eight organizations after the modification)  Manage and monitor grant implementation #3  Award small grants to four local organizations to establish service centers for families of persons with disabilities in four local communities (modified to open additional two service centers)  Manage and monitor grant implementation #4  Administer two RFAs related to deinstitutionalization of persons with disabilities and human trafficking, review proposals, and provide recommendations to USAID  Award a small grant to Local Democracy Foundation to provide equipment, training, and scholarships to 20 to 30 female victims of violence (modified: grants reallocated for entrepreneurship assistance to women victims of violence)  Manage and monitor grant implementation #5  Award grants to organizations for interventions related to deinstitutionalization of persons with disabilities and human trafficking  Modified to reallocate unspent grants to other organizations that support marginalized populations  Manage and monitor grant implementation #6  Acquisition of first aid equipment for 12 schools in BiH  Deliver equipment to U.S. Embassy in BiH #7  Grant award to Mostar Rock School for their core activities; expected to organize 120 session bands (including students belonging to different ethnicities), 12 program concerts, and 3 major concerts.  Manage and monitor grant implementation #8  Grant award to Association EDUS to 1) develop methodology for early detection and intervention for children with developmental disorders, 2) develop a protocol for autism diagnosis; 3) verify the effectiveness of the early intervention system for children with developmental disorders; three studies conducted  Manage and monitor grant implementation #9  Grant award to Sarajevo Open Center to assist LGBTI population in BiH by organizing information sessions with influencers and public figures, strengthening local groups working with LGBTI rights in Tuzla and Prijedor (sub- 45 grants), and organizing an internship program for volunteers to raise a new generation of LGBTI activists  Manage and monitor grant implementation #10  Grant award for Entrepreneurship Assistance to Unemployed Marginalized Women  Award two research grants to an organization to design methodology for and conduct survey on prevalence of violence against children in BiH  In-kind grant to “Education and Rehabilitation center for Children, Youth, and Adults with Developmental Disabilities” in Ljubuski for methodical and didactic equipment for rehabilitation exercises  Manage and monitor grant implementation As envisaged in the contract, PPMG has awarded and managed three different types of grants to local organizations on behalf of USAID/BiH: unsolicited grants, directed grants, and annual program statements. During the first year of implementation, PPMG developed a grants manual and guidelines for financial reporting for grantees. The grants manual included questionnaires to guide pre-award assessment procedures. Throughout Activity implementation, PPMG has reviewed unsolicited proposals received by USAID/BiH and continued to award and manage other grant types in close cooperation with the Mission. For each grantee, PPMG has conducted organizational capacity assessments, helped develop a capacity building plan, and supported its implementation. One-on-one capacity building support has been provided to grantees through working meetings, phone calls, emails, and monitoring visits. From May 9, 2017, due to a TO 1 modification, PPMG discontinued unsolicited grant reviews, but continued other duties related to grant management and monitoring. According to available documentation, PPMG has awarded and managed more than 50 grants to 34 local organizations and 18 local communities, with a particular emphasis on supporting persons with disabilities. The Activity has supported the opening of six service centers for families of persons with disabilities in BiH to-date. These centers provide a range of services to persons with disabilities, including short-term stay, creative and occupational workshops, outdoor activities, individual and group treatment, and psychological support and legal services to their family members. Within the annual program statements grants, PPMG supported initiatives in deinstitutionalization of persons with disabilities. Finally, PPMG supported the development of a methodology for early detection and intervention for children with developmental disorders. PPMG also supported initiatives related to anti-trafficking in persons with the objective of reducing the incidence of human trafficking through preventative measures, such as educational activities and improved access to social assistance for potential victims. PPMG also supports the LGBTI population in BiH by strengthening the capacities of CSOs and informal groups that work on LGBTI issues. The Activity motivates these groups to design and implement local initiatives, thereby building an institutional support network for LGBTI people. PPMG also allocated funds to 18 local communities within the Fund for Supporting Development of Businesses Owned by Marginalized Women. With regard to youth, PPMG supports activities of Mostar Rock School. They also supported several small-scale interventions like the National Geographic Photo Camp and awarded in-kind grants to youth organizations and schools. PPMG awarded several in-kind grants, including the donation of sports equipment to sports clubs, therapeutic and sports equipment to persons with disabilities, didactic and sports equipment to schools, and purchasing first aid equipment for 12 schools in BiH. During Activity implementation, PPMG held various trainings aimed at building the capacity of local organizations, government institutions, and USAID partner organizations that promote the rights of marginalized populations. 46 As envisaged in the Activity design, the trainings covered: legal and organizational structures; administration and human resources; financial management; public advocacy; project cycle management and writing project proposals; strategic planning; monitoring and evaluation, youth entrepreneurship; administrative procedures in municipalities and governments; and teamwork and leadership in youth. PPMG was required to hold 15 sessions on gender equality, which were incorporated in the aforementioned trainings. After each training, PPMG surveyed the participants to assess their knowledge of the topic and perceptions of the training. The results continuously showed that the trainings improved participants’ understanding of the topics and that the trainings were positively received by the participants. PPMG is tracking five indicators to measure progress in meeting Life of Activity targets (see Exhibit 3). To date, PPMG has not entered their actuals for the fourth year of implementation (FY2018). Moreover, PPMG has not submitted two quarterly reports for FY 2016, or the last quarterly report for Q2 FY 2018 to the BiHPERFORM system. Exhibit 3. Activity Indicators, with Targets and Actuals for FY 2015, 2016, and 2017, and Life of Activity Targets Level of result Narrative Summary Indicators Actuals (Targets) Life of Activity Targets FY 2015 FY 2016 FY 2017 FY 2018 Activity Purpose 1.2.3 Individuals and CSOs representing underrepresented groups are constructively engaged in civic/political issues Number of underrepresented people assisted by a USG-funded intervention providing services to beneficiaries 495 (300) 5,245 (600) 4,580 (3,000) 3,818 (3000) 8,900 Activity Sub￾purpose 1 Activities of groups of local organizations that advocate for the rights and dignity of underrepresented groups are supported (Corresponds to USAID 1.2.3.1 CSOs representing and led by underrepresented groups advocate for policy changes) Number of advocacy initiatives focused on improving the status of youth, women, the Roma, the disabled, or LGBTI in the public and political life in BiH 0 (0) 4 (5) 98 (60) 99 (60) 185 Activity Outcome/ Output 1.1. The capacity of local NGOs, BiH institutions and USAID implementing partners is built. Number of human rights organizations trained and supported 27 (6) 166 (30) 107 (100) 126 (80) 296 Number of organizations representing marginalized populations that receive minimum 12 hours of training and counseling on OCA tool 0 (0) 14 (8) 6 (8) 5 (8) 32 Number of representatives of government institutions in BiH trained to increase their organizational capacities 12 (15) 0 (25) 30 (25) 0 (25) 115 47 EVALUATION AND ASSESSMENT: PURPOSE, QUESTIONS, DESIGN AND LIMITATIONS The main purpose of this performance evaluation and brief assessment is to measure the Activity’s progress to￾date and to help USAID/BiH improve design and the efficiency of future interventions. This performance evaluation and brief assessment will examine attitudes and experiences from various stakeholder groups to inform the evaluation and assessment findings, conclusions, and recommendations. This includes USAID/BiH officials who designed the intervention; international organizations and government institutions working with marginalized groups; NGOs/CSOs focused on assisting marginalized groups, including PPMG grantees and non-grantees; and representatives of marginalized populations receiving PPMG assistance. The evaluation and assessment findings, conclusions, and recommendations will help the Mission inform future programming that targets assistance to marginalized groups. In addition, the results will assist the Activity Implementing Partner (IP) improve capacity building practices. Finally, relevant government institutions, civil society, and other stakeholders can leverage the evaluation and assessment findings to design their own interventions to assist marginalized populations in BiH. EVALUATION AND ASSESSMENT QUESTIONS The PPMG performance evaluation will answer the following questions and a sub-question: 1. What progress has PPMG achieved to date in reaching contract targets? 2. To what extent has the organizational capacity of PPMG-supported organizations (both grantees and trainees) been improved? What is some of the evidence that this has happened? 3. To what extent has the representation in civic and political issues of PPMG-supported organizations been improved? 3.1. Are there indications showing that supported local organizations are interacting more often and more effectively with public authorities and/or the public at large? The brief assessment will answer the following questions: 1. How effective are local organizations working on marginalized populations issues at raising funds? Which government entities, international donors, or other philanthropic organizations have provided funding to those organizations? 2. Has the status of marginalized populations improved over the last five years? 48 DESIGN AND METHODOLOGY This performance evaluation and brief assessment will adopt a mixed method approach. The evaluation/assessment team (hereinafter, the research team) will use the following data sources: 1. Activity documentation, including the award, task orders and modifications, work plans, quarterly reports, MEL documentation, Activity records and databases, and other PPMG documents and deliverables. 2. Secondary data sources on marginalized groups, including policy documents, research papers, and other documents developed by government institutions, international and donor organizations, local organizations, and other organizations or institutions, including documents and data produced by MEASURE-BiH (e.g. National Survey of Citizens Perceptions, National Youth Survey). 3. Key informant interviews with USAID/BiH and PPMG implementing partner (IP), other USAID’s IPs, relevant international and donor organizations, government institutions, PPMG grantees and their beneficiaries, and other local organizations. The full list of key informants will be presented in the Evaluation and Assessment Work Plan and subject to USAID/BiH comments. 4. Focus groups with PPMG beneficiaries (representatives of marginalized populations who received the assistance provided by PPMG grantees). Draft focus group guide will be included in the Evaluation and Assessment Work Plan and subject to USAID/BiH’s comments. 5. Online survey of PPMG grantees and, if possible, mini survey of non-grantees (representatives of relevant local non-government and civil society organizations that work with marginalized populations who either received or did not receive PPMG assistance). A draft survey questionnaire will be included in the Evaluation and Assessment Work Plan and subject to USAID/BiH comments. 6. An online survey of PPMG beneficiaries (representatives of marginalized populations who received the assistance provided by PPMG grantees). A draft survey questionnaire will be included in the Evaluation and Assessment Work Plan and subject to USAID/BiH comments. The research team will have access to relevant PPMG documentation, including award documentation, work plans, quarterly reports, annual reports, MEL documentation, databases, and other relevant documents and deliverables. Exhibit 4 presents the Evaluation Matrix, which details the methodological approach used to answer each evaluation question. Specifically, under the Evaluation Question 1, the research team will review PPMG’s progress toward the following: number of underrepresented people assisted by a USG-funded intervention providing services to beneficiaries; number of advocacy initiatives focused on improving the status of youth, women, Roma, disabled, or LGBTI in the public and political life in BiH; number of human rights organizations trained and supported; number of organizations representing marginalized populations that received a minimum of 12 hours of training and counseling on OCA tool; number of representatives of government institutions in BiH trained to increase their organizational capacities; and other contractual obligations. The research team will assess PPMG’s 49 performance when it comes to grants award and management and organizational capacity building. The research team will use Activity documentation, key informant interviews, and online survey data to answer this question. The research team will explore the extent to which the organizational capacity of PPMG grantees and trainees has improved because of PPMG capacity building interventions (Evaluation Question 2). The research team will assess perceptions of different stakeholders (USAID, IP, grantees, trainees) about quality, relevance, and usefulness of PPMG interventions, resultant changes in grantees’ and trainees’ capacities, and evidence of such changes (i.e. new organizational policies and procedures, management practices, monitoring and evaluation systems, etc.). The findings will be formulated based on review of the Activity and grantees’ documentation, key informant interviews, and online survey data. Exhibit 4: Evaluation Matrix The research team will explore the results achieved by PPMG in terms of improving local organizations’ representation in civic and political issues (Evaluation Question 3) by investigating perceptions of wide range of stakeholders (including grantees, trainees, beneficiaries, non-grantees, and government and international EVALUATION QUESTIONS DATA SOURCES AND DATA COLLECTION METHODS RESEARCH DESIGN What progress has PPMG achieved to date in reaching contract targets? Activity documentation (MEL Plan and progress reports)/Documents review Key informants/Key informant interviews, online survey Mixed methods To what extent has the organizational capacity of PPMG-supported organizations (both grantees and trainees) been improved? What is some of the evidence that this has happened? Activity documentation (MEL Plan and progress reports) and secondary data sources/Documents review Key informants/Key informant interviews and/or focus groups, online survey Mixed methods To what extent has the representation in civic and political issues of PPMG￾supported organizations been improved? Are there indications showing that supported local organizations are interacting more often and more effectively with public authorities and/or the public at large? Activity documentation (MEL Plan and progress reports) and secondary data sources/Documents review Key informants/Key informant interviews and/or focus groups, online survey Mixed methods 50 stakeholders) about improvements in this area in the last five years. The team will review potential changes in local organizations’ interactions with government authorities, participation in policy development and advocacy for the rights of marginalized populations. The findings will be formulated based on the Activity documentation, key informant interviews, online survey data, and secondary data. To answer the assessment questions, the research team will assess the effectiveness of local organizations focused on marginalized groups at raising funds by investigating perceptions of different stakeholders about experiences and challenges facing these organizations’ fund raising activities and by exploring their major providers of funding (Assessment Question 1). Moreover, the team will assess improvements in the status of marginalized populations over the last five years in terms of new policies, facilitated implementation of formerly adopted policies, new services, and other developments leading to improvement of their status (Assessment Question 2). The assessment findings will be formulated based on desk review of secondary data, key informant interviews, focus groups, and online surveys. The evaluation and assessment findings, conclusions, and recommendations will be triangulated across multiple data sources. The research team will review the Activity’s documentation; record, transcribe, and code the key informant interviews; analyze and (whenever possible) compare the survey data of grantees and their beneficiaries, as well as for non-grantees; review the secondary data and compare them against the primary data. Exhibit 5: Assessment Matrix ASSESSMENT QUESTIONS DATA SOURCES AND DATA COLLECTION METHODS RESEARCH DESIGN How effective are local organizations working on marginalized populations issues at raising funds? Which government entities, international donors, or other philanthropic organizations have provided funding to those organizations? Secondary data/Desk review Key informants/Key informant interviews, focus groups, online survey Mixed methods Has the status of marginalized populations improved over the last five years? Secondary data/Desk review Key informants/Key informant interviews, focus groups, online survey Mixed methods 51 EVALUATION AND ASSESSMENT LIMITATIONS Potential limitations of this evaluation and assessment include:  Social desirability bias, including overstatement of positive effects by beneficiaries and of negative effects by non-beneficiaries: the research team will triangulate data across multiple data sources to formulate credible results.  Limitations related to the online survey methodology. o Low response rates for online surveys: the research team will send regular reminders to potential survey respondents. o Limited contact data on grantees’ beneficiaries (grantees may resist sharing contact information of their beneficiaries to protect their privacy): if such cases occur, the research team will ask the grantees to distribute the survey link to their beneficiaries. EVALUATION AND ASSESSMENT IMPLEMENTATION DELIVERABLES AND SCHEDULE All deliverables will be submitted electronically and in English. The deliverables will include: 1. Detailed evaluation and assessment work plan and data collection instrument(s) The evaluation and assessment work plan will include: (1) a detailed evaluation and assessment design matrix (including the key questions, methods, and data sources used to address each question and the data analysis plan for each question); (2) draft data collection instruments (interview guides and questionnaires) (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation/assessment design; (5) a dissemination plan; (6) the anticipated schedule and logistical arrangements; and (7) a list of the members of the evaluation/assessment team, delineated by roles and responsibilities. 2. Presentation of preliminary findings A presentation of preliminary findings to USAID/BiH will include a summary of preliminary findings and recommendations to USAID/BiH. 3. Draft evaluation and assessment report The draft evaluation and assessment report will be consistent with the USAID Evaluation Report Requirements provided in ADS REFERENCE 201MAH (https://www.usaid.gov/ads/policy/200/201mah) and take into account criteria to ensure the quality of the evaluation report specified in ADS REFERENCE 201MAA (https://www.usaid.gov/ads/policy/200/201maa). Once the initial draft report is submitted, USAID/BiH will have 15 working days to review and comment on the initial draft and submit the consolidated comments to the research team. The research team will 52 address the consolidated comments and submit a revised final draft report within 10 days of receiving USAID/BiH comments. 4. Final evaluation and assessment report The research team will take no more than 10 calendar days to respond/incorporate the final comments from USAID/BiH. The research team lead will then submit the final report. The final evaluation and assessment report will be between 50 and 75 pages, excluding any annexes. The tentative schedule is as follows: Tentative Dates Tasks and Deliverables June 14, 2019 Submit draft Work Plan to USAID/BiH June 17 to June 21, 2019 Logistical preparation, scheduling KIIs interviews, online survey preparation, piloting data collection instruments June 21 to July 18, 2019 Data collection through KIIs and online survey Interview transcription Initial data analysis Review of Activity documentation Review of secondary data July 18, 2019 Recommendations’ discussion workshop with USAID/BiH July 18 to August 6, 2019 Continue and finalize transcribing interviews Continue and finalize data analysis Report drafting August 6, 2019 Presentation to USAID/BiH to discuss the preliminary findings and recommendations August 9, 2019 Submit Draft Evaluation Report to USAID TENTATIVE TEAM COMPOSITION AND KEY QUALIFICATIONS It is expected that the evaluation and assessment team will include four team members. The tentative team composition and team members’ key qualifications are as follows: Position Key Qualifications Team Lead Team and project management skills; technical expertise in evaluation methodologies Subject Matter Experts (two experts) Subject matter expertise in human rights, social inclusion, and grants management Research Analyst (RA) Data collection and data analysis skills Research Assistant Organizational skills and ability to ensure smooth process of data collection and processing The team composition and level of effort will be finalized in the Evaluation and Assessment Work Plan. 53 ANNEX II: DATA COLLECTION INSTRUMENTS INTERVIEW GUIDES The interview guides are intended to serve as semi-structured guides for conversations with key stakeholders of PPMG. Do not read the questions or probes word for word. Instead, adapt the wording to match the phrasing used by the respondent and ask only those questions which have not been already addressed by the interviewees during earlier part of the interview. Take notes on key terms or phrases used by the respondents that may be helpful in coding the interview data. Ask for clarification and definitions as needed. Familiarize yourself with the interview protocol guides in advance of your meeting. The questions in bold are the questions you will prioritize if the respondent’s time is limited. Be respectful of the respondent’s time and keep the interview to the agreed length of time. Follow up by phone or email for more information as needed. In addition:  Take notes during the discussion. To ensure we accurately report what is discussed during the interview, we will record this session as well.  As necessary, tailor all questions to fit the individual stakeholders’ relationship with PPMG.  Keep the discussion under sixty minutes.  The research team will ensure that the information shared through these interviews remain strictly confidential. 54 Interview Guide for USAID/BIH Describe the context at the time when you decided to design the Activity.  What were the major problems facing marginalized groups in BiH at that time?  Were there any other USAID/BiH’s interventions supporting marginalized groups before 2015? If yes, please describe the interventions, results, and lessons learned. Describe the Activity’s design process.  Describe the Activity’s development hypothesis/theory of change, expected results, and implementation mechanisms (combination of grants and capacity building technical assistance)  Why have you selected youth, women, persons with disabilities, Roma, and LGBTI as a target group? Why has PPMG spent most grants on persons with disabilities, followed by women? How have you envisaged the organizational capacity building interventions, and how were they implemented?  Why have you decided to build organizational capacities among local organizations and government leaders, and what results has PPMG achieved in this regard?  To what extent have the program and financial management capacities of local organizations, BiH institutions, and USAID’s partners been improved? How do you know this?  What skills do the trained staff have now that they did not have before? What organizational and/or management policies and procedures have been established due to PPMG capacity building that have not been established before?  Please describe the examples of good practices employed by PPMG, and areas for improvement? To what extent have PPMG’s interventions improved the local organizations’ participation in civic in political issues? Please elaborate.  How has their interaction with government authorities and public changed?  How have their advocacy initiatives changed?  Have you noticed any unintended effects during the Activity’s implementation?  How has PPMG ensured local ownership and sustainability of their results? How has PPMG been reporting on their results?  Were there any challenges with regard to reporting actuals for the Activity performance indicators?  Why did PPMG over-perform on some, and underperform on other performance indicators? To your knowledge, is PPMG going to achieve their contractual obligations during the Life of the Activity (extended)?  Why was the Activity’s implementation extended until February 2022?  Has PPMG made any changes in their MEL Plan after the DQA conducted in 2017? Are you aware of any other donor or government institution’s activities focused on assisting marginalized groups in BiH? How has PPMG coordinated their work with other stakeholders? What are the major problems facing marginalized groups in BiH today? 55  Where do marginalized populations, particularly youth, women, Roma, LGBTI, and persons with disabilities, stand as a priority for the governments? Are some of these groups more marginalized than others?  What is the public perception of these marginalized groups?  Has their situation improved in the last five years? If yes, how?  In your opinion, what issues do local organizations working on marginalized populations’ problems face with regard to sources of funding and fund raising? In your opinion, what is required for the marginalized populations to prosper in BiH? If you had an opportunity to design the Activity again, what would you have done differently? Interview Guide for the Implementing Partner Describe the context at the time when the Activity started.  What were the major problems facing marginalized groups in BiH at that time, particularly youth, women, Roma, LGBTI, and persons with disabilities? Describe the Activity’s design.  development hypothesis/theory of change and expected  implementation mechanisms How have you implemented the following elements of the Activity design? Please describe the examples of good practices, and areas for improvement?  Grants award, management, and monitoring (unsolicited grants, annual program statements, in-kind grants)  One-on-one capacity building support to grantees  Capacity building of USAID/BiH partners and government leaders  Did you have any experience with grants management and capacity building before PPMG?  How did you handle different contractual obligations and various TO modifications?  Why has the implementation of the Activity been extended until February 2022? Please describe your experiences with monitoring and evaluation of your own activity?  What challenges did you face when reporting actuals for the Activity performance indicators?  To your knowledge, are you going to achieve your contractual obligations during the Life of the Activity (extended)? Why did you over-perform on some, and underperform on other performance indicators?  Have you made any changes in the Activity MEL Plan after DQA conducted in 2017? To what extent have the program and financial management capacities of local organizations, BiH institutions, and USAID’s partners been improved? How do you know this?  What skills do the trained staff have now that they did not have before?  What organizational and/or management policies and procedures have been established due to PPMG capacity building that have not been established before? 56 To what extent have PPMG’s interventions improved the local organizations’ participation in civic in political issues? Please elaborate.  How has their interaction with government authorities and public changed?  How have their advocacy initiatives changed?  How has the PPMG ensured local ownership and sustainability of their results?  Have you noticed any unintended effects during the Activity’s implementation? What are the major problems facing marginalized populations in BiH today?  Where do the marginalized populations, particularly youth, women, Roma, LGBTI, and persons with disabilities, stand as a priority for the governments? Are some of these groups more marginalized than others?  What is the public perception of these marginalized groups?  Has their situation changed in the last 5 years? Are you aware of any other government or donor interventions related to providing assistance to marginalized groups?  Were these interventions coordinated with PPMG interventions?  Have you received funds or any other support from another donor, or from a government institution?  What are your organization’s sources of funds? What is the share of donor versus government funding?  What challenges does your organization face in terms of fund raising?  What challenge do other local organizations face in this regard? If you had an opportunity to participate in the design or implementation of the future Activity aimed at providing assistance to marginalized groups, what would you have done differently? In your opinion, what is required so that the marginalized populations can advance in BiH? Interview Guide for Government Institutions and International Organizations What are the major problems facing marginalized populations in BiH today?  Which social groups are the most marginalized in BiH society, particularly youth, women, Roma, persons with disabilities, and LGBTI?  Has the status of these groups changed in the last five years? If yes, how?  Where do these marginalized populations stand as a priority for the government?  What is the public perception of these marginalized groups? Please describe any interventions your institution/organization implemented to improve conditions for marginalized populations:  Marginalized groups involved  Implementation mechanisms used to improve their status Are you aware of any other government or donor interventions related to providing assistance to marginalized groups?  How do different donors and implementers coordinate their interventions? 57  What challenges do local organizations assisting marginalized groups face in terms of fund raising? What are their main sources of funding? Have you heard of PPMG, USAID/BiH-funded Activity supporting marginalized groups? PPMG employed a combination of financial assistance (grants) and capacity building technical assistance to strengthen the capacities of organizations that engage with marginalized groups, with the objective to improve representation of these organizations in civic and political issues of particular concern for the marginalized populations in BiH.  Are you aware of any results achieved by this Activity? (service centers, grants for women entrepreneurships, improving employment opportunities for Roma, advocacy activities for LGBTI, activities for youth)  Do you think that grants and capacity building of local organizations and government leaders employed by PPMG have helped improve local organizations representation in civic and political issues? Please elaborate. FOR TRAINEES FROM GOVERNMENT INSTITUTIONS How would you describe the PPMG capacity building trainings? Describe good practices and areas for improvement. What is your opinion about …?  Expertise and pedagogical skills of trainers  Quality of training  Relevance/usefulness of training To what extent have the organizational capacities of your institution changed?  What skills does the trained staff have now that they did not have before?  What organizational and/or management policies and procedures have been established due to PPMG capacity building that have not been established before? To what extent have PPMG’s interventions improved your institutions’ stance toward marginalized populations? Please elaborate.  How has your interaction with civil society and public changed? How have your advocacy initiatives changed? In your opinion, what is required for the marginalized populations to prosper in BiH? What are your recommendations for future interventions aiming to increase underrepresented persons’ participation in political and civic activities? 58 Interview Guide for Local Organizations On which marginalized population has your organization focused? What was the position of this group five year ago? What are the major problems facing marginalized populations in BiH today?  Where do marginalized populations, particularly youth, women, Roma, LGBTI, and persons with disabilities, stand as a priority for the governments? Are some of these groups more marginalized than others?  What is the public perception of these marginalized groups?  Has their situation changed in the last 5 years? Are you aware of any other government or donor interventions related to providing assistance to marginalized groups?  Have you received funds or any other support from another donor, or from a government institution?  Were these interventions coordinated with PPMG interventions?  What are your organization’s sources of funds? What is the share of donor versus government funding?  What challenges does your organization face in terms of fund raising? If you had an opportunity to participate in the design and implementation of a future Activity aimed at assisting marginalized groups, what would you have done (differently)? In your opinion, what is required for the marginalized populations to prosper in BiH? FOR GRANTEES: Describe your experience with PPMG:  How did you first hear about PPMG and why did you decide to participate?  In your understanding, what are the development hypothesis/theory of change and expected results of the PPMG Activity? Describe the intervention you implemented with PPMG assistance.  developmental hypothesis/theory of change and expected results  methodology  results How have you been reporting to PPMG on your results and achievements?  Please describe the procedures established for reporting on performance indicators.  On which performance indicators have you been reporting?  Did you face any challenges with regard to reporting? How would you describe the PPMG’s performance when it comes to… ?  OCA Tool  one-on-one technical assistance and mentorship 59  grant management and monitoring  describe good practices and areas for improvement! FOR GRANTEES AND TRAINEES How would you describe the PPMG capacity building trainings? Describe good practices and areas for improvement. What is your opinion about …?  Expertise and pedagogical skills of trainers  Quality of training  Relevance/usefulness of training To what extent have the organizational capacities of your organization or institution changed?  What skills does the trained staff have now that they did not have before?  What organizational and/or management policies and procedures have been established due to PPMG capacity building that have not been established before? To what extent have PPMG’s interventions improved your organizations’ participation in civic in political issues? Please elaborate.  How has your interaction with government authorities/civil society and public changed?  How have your advocacy initiatives changed?  Are the results you achieved sustainable? FOR NON-GRANTEES: Have you ever heard of PPMG, a project funded by USAID/BiH and implemented by KULT? PPMG employed a combination of financial assistance (grants) and capacity building technical assistance to strengthen the capacities of organizations engaging with marginalized groups, with the objective to improve representation of these organization in civic and political issues facing marginalized populations in BiH.  Are you aware of any results achieved by this Activity? (service centers, grants for women entrepreneurships, improving employment opportunities for Roma, advocacy activities for LGBTI, activities for youth)  Do you think that grants and capacity building of local organizations and government leaders employed by PPMG have helped improve underrepresented groups’ participation in civic and political issues? Please elaborate.  Would such assistance be useful for your organization? Describe the interventions you implemented to improve the situation of marginalized populations.  developmental hypothesis/theory of change and expected results  methodology  results 60 Focus Group/Interview Guide for Beneficiaries Please describe the situation and problems facing (name the marginalized category) in BiH?  Where does this group stand as a priority for the government?  What is your opinion about NGOs and CSOs providing assistance to this group?  How does the public perceive this group? Have there been any changes in the status of this group in the last five years?  To your knowledge, have any new policies been developed and/or implemented in this period related to the status of this group?  Has the government support for your group improved in the last five years?  Has the public perception of your group improved in the last five years?  Has the CSO support for your group improved in the last five years?  Overall, has the situation of your group improved in the last five years? If applicable: Please describe the intervention in which you participated.  What was the objective of this intervention?  How was it implemented?  What were the short-term and long-term results of the intervention?  What is your level of satisfaction with this intervention? Why?  Was this intervention useful for this group? Has it improved the situation of your group? If yes, how?  Was this intervention in line with your group’s needs?  Please describe examples of good practices, and areas for improvement of this intervention? If you had an opportunity to participate in the design and/or implementation of an intervention aimed at providing assistance to this group, what would you have done differently? What is required so that this group can advance in BiH? 61 ONLINE SURVEY QUESTIONNAIRES Marginalized Groups Support Program Survey of grant beneficiaries This survey is conducted within the framework of the Impact Assessment of the Marginalized Groups Support Program (MGSP) and the assessment of the situation of marginalized groups in BiH by USAID/BIH’s MEASURE-BiH, the BiH Monitoring and Evaluation Support Project (for more information on MEASURE-BiH, visit www.measurebih.com). The purpose of this survey is to identify good practices and areas for improvement in the implementation of the Marginalized Groups Support Program in BiH, with the aim of improving existing and future programs designed to support marginalized groups and local organizations fighting for the rights of these groups. Your opinion is extremely important to us. Therefore, we would appreciate if you take the time to complete this survey, even if you have already participated in the interviews within the framework of this evaluation. It takes about 10 minutes to complete the survey. All responses you provide in this survey will held fully confidential and anonymous. Please respond to the survey questions in an open, honest and timely manner. Survey of grant beneficiaries - Marginalized Groups Support Program Which of the following groups does your organization focus on in its work? 1. Do you focus on promoting YOUTH status? Yes No 2. QC4. In your opinion, has the status of youth improved over the last 5 years in terms of…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems facing this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 62 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 3. Do you focus on advancing women's status? Yes No 63 4. QC5. In your opinion, has the status of women improved in relation to the past 5 years in terms of…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 5. Do you focus on enhancing the status of PEOPLE WITH DIFFICULTIES AND THEIR FAMILIES? Yes No 64 6. QC6. In your opinion, has the status of persons with disabilities improved OVER THE PAST 5 YEARS IN TERMS OF…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 65 7. Do you focus on improving the status of the Roma? Yes No 8. QC7. In your opinion, has the status of Roma improved in the last 5 years in terms of…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 9. Do you focus on improving the status of the LGBTI population (lesbian, gay, bisexual, transgender and intersex)? Yes No 66 10. QC8. In your opinion, has the status of lesbian, gay, bisexual, transgender and intersex persons improved over the past 5 years in terms of…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 11. QG3. What are the main problems these groups face? 67 12. QC8a. Apart from young people, women, people with disabilities, Roma and members of the LGBTI population, are there other marginalized groups that need special support? 13. QF9. Have you received funding from the following sources over the past five years? 14. QG10. The year your organization was registered? 15. QG11. Number of persons in paid employment? 16. QF12. What are the primary challenges facing civil society organizations in BiH when it comes to fundraising? Funds of ministries and/or government institutions in BiH at any level 68 16. QF13. What funding sources does your organization use? 17. QF14. What proportion of the funds does your organization have from each of these sources? (The sum of these percentages should be 100 %.) 19. To what extent do you agree with the following statements? Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Local organizations that are forced to adjust their programs to donor strategies to obtain funding ☐ ☐ ☐ ☐ ☐ Most local organizations do not have enough resources to hire the optimal number of workers ☐ ☐ ☐ ☐ ☐ Many organizations that used to be donors are now applying for donor funding and as a result local organizations have reduced access to funding ☐ ☐ ☐ ☐ ☐ Company revenue achieved by my organization 69 Donors who are funding local organizations are demanding high requirements for the funds they donate ☐ ☐ ☐ ☐ ☐ Donor organizations often finance short-term projects that cannot achieve the desired results ☐ ☐ ☐ ☐ ☐ Long-term programs need to be implemented to achieve changes in the status of marginalized groups ☐ ☐ ☐ ☐ ☐ Local authorities finance local organizations but invest minimal resources ☐ ☐ ☐ ☐ ☐ Local authorities mostly support veterans and/or sports associations ☐ ☐ ☐ ☐ ☐ Constant changes of politicians in positions in governmental institutions slow down changes in the status of marginalized groups ☐ ☐ ☐ ☐ ☐ 20. QG15. Did you receive a grant from USAID's Marginalized Groups Support Program (PPMG), which implements KULT? Yes, a grant in cash Yes, a grant in equipment 21. QIK26. Would your organization be interested in participating in training on any of these topics? Not at all Slightly Moderately Considerably Compliance with legal requirements ☐ ☐ ☐ ☐ Administration ☐ ☐ ☐ ☐ 70 Project management ☐ ☐ ☐ ☐ Fundraising ☐ ☐ ☐ ☐ Monitoring and Evaluation ☐ ☐ ☐ ☐ Human resources ☐ ☐ ☐ ☐ Financial management and reporting ☐ ☐ ☐ ☐ Advocacy ☐ ☐ ☐ ☐ Writing project proposals ☐ ☐ ☐ ☐ Marketing and promotion ☐ ☐ ☐ ☐ 22. QG16. What is your level of satisfaction with each of the items listed? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Eligibility requirements for your organization to qualify for a grant (e.g. changes to the organization's statutes and/or rules and regulations) ☐ ☐ ☐ ☐ ☐ Grant management by PPMG ☐ ☐ ☐ ☐ ☐ PPMG staff professionalism ☐ ☐ ☐ ☐ ☐ 71 Implementation of the OCA tool ☐ ☐ ☐ ☐ ☐ Usefulness of OCA tools for your organization ☐ ☐ ☐ ☐ ☐ Providing individual assistance to your organization by PPMG during the implementation of the OCA grant tool ☐ ☐ ☐ ☐ ☐ Reporting requirements and complexity ☐ ☐ ☐ ☐ ☐ Support and technical assistance provided by PPMG during the implementation of the grants ☐ ☐ ☐ ☐ ☐ 23. QG17. Did you receive any of the following guidance or training when it comes to monitoring your grant? Yes No Precise instructions on how to count project participants ☐ ☐ Precise instructions on how to avoid a double counting when reporting on number of project participants ☐ ☐ Precise instructions on what needs and what should not be counted as advocacy interventions ☐ ☐ 72 24. Have you had a three-day consultation with PPMG dedicated solely to your organization, that is, to improving the administration and financial management of your organization? Yes No 25. QT18. Did you or someone from your organization or institution participate in any of the PPMG trainings? Yes No 26. Please answer the following questions: QT18a. Number of staff from your organization who attended the training (approximately if you don't remember) QT18b. Number of training days, including all staff (approximately if you don’t remember) 27. To what extent do you agree with the following statements: Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree The reporting procedures under the PPMG program were too extensive and complex ☐ ☐ ☐ ☐ ☐ At trainings organized by PPMG we partnered with other organizations that are dealing with similar issues ☐ ☐ ☐ ☐ ☐ 73 27. QT19. Please rate the following training elements: Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Quality of training in general ☐ ☐ ☐ ☐ ☐ Level of knowledge among experts who held training ☐ ☐ ☐ ☐ ☐ Teaching skills ☐ ☐ ☐ ☐ ☐ Stimulating active participation among participants in training ☐ ☐ ☐ ☐ ☐ Opportunities to practice new skills acquired during training ☐ ☐ ☐ ☐ ☐ Suitability of training topics for your organization ☐ ☐ ☐ ☐ ☐ Usefulness of gender equality sessions ☐ ☐ ☐ ☐ ☐ 28. Did you receive homework after PPMG training? Yes No 74 29. QCB20. How much has your organizational capacity improved THANKS to PPMG TRAININGS in each of the following areas? Not at all Slightly Moderately Considerably Compliance with legal requirements ☐ ☐ ☐ ☐ Administration ☐ ☐ ☐ ☐ Project management ☐ ☐ ☐ ☐ Fundraising ☐ ☐ ☐ ☐ Monitoring and Evaluation ☐ ☐ ☐ ☐ Human resources ☐ ☐ ☐ ☐ Financial management and reporting ☐ ☐ ☐ ☐ Advocacy ☐ ☐ ☐ ☐ Writing project proposals ☐ ☐ ☐ ☐ Marketing and promotion ☐ ☐ ☐ ☐ 75 30. Have you ever gone through a similar capacity building process before? 31. QCB21. Has your organization developed new or modified existing organizational policies or procedures THANKS FOR PPMG TECHNICAL ASSISTANCE TO CAPACITY BUILDING? Yes No 32. QCB21a. Who made the decision to develop/change organizational policies/procedures/regulations? The decision was made by our management/staff PPMG advised us to develop specific organizational policies PPMG required from us to develop specific organizational policies 33. QCB21b. How much has the PPMG helped your organization develop organizational policies/procedures/regulations? Not at all Somewhat Moderately Considerably 34. QCB22. How helpful were these policies for your organization? Not at all Somewhat Moderately Considerably 76 35. QCB23. How often do you use these policies and procedures? Never Often Rarely All the time Sometimes 36. QR25. To what extent do you agree with the claim that PPMG capacity building has improved your organization's representation in civil and political matters? Strongly disagree Agree Disagree Strongly agree Neither agree nor disagree 37. QR24. Has your organization experienced any of these changes ACKNOWLEDGING PPMG'S FINANCIAL ASSISTANCE OR CAPACITY BUILDING ASSISTANCE? Not at all Slightly Moderately Considerably Increased involvement in development of rights’ policies for marginalized groups ☐ ☐ ☐ ☐ Increased frequency engaging in advocacy for rights marginalized groups ☐ ☐ ☐ ☐ Increased quality advocacy intervention for rights of marginalized groups ☐ ☐ ☐ ☐ Increased interaction with governmental institutions ☐ ☐ ☐ ☐ Increased funding by government institution on any level because of skills acquired through this program ☐ ☐ ☐ ☐ Increased funds from other international donors other than USAID because of the skills acquired through this program ☐ ☐ ☐ ☐ Increased cooperation with other NGOs that deal with similar issues ☐ ☐ ☐ ☐ 77 38. What advocacy initiatives has your organization initiated as a result of the PPMG program? 39. Additional comments and observations: 78 Marginalized Groups Support Program Survey for government servants/-participants for the Marginalized Groups Support Program This survey is being conducted by USAID/BiH MEASURE-BiH, the BiH Monitoring and Evaluation Support Project. The above mentioned survey was conducted as part of an evaluation of the USAID/BiH-funded Marginalized Groups Support Programs (PPMG), implemented by the Institute for Youth Development ”Kult”. You received this survey because, as a representative of a government institution, you participated in one of the training courses organized under this program (e.g., monitoring and evaluation, teamwork and leadership, or administrative regulations). In addition, the purpose of this survey is to contribute to the assessment of the situation of marginalized groups and the difficulties that local organizations face when it comes to funding currently being implemented by USAID/BiH’s MEASURE-BiH Activity. We kindly ask you to take the time to complete this survey. All information you share will remain confidential and anonymous, and will be used solely for research purposes. By participating in this survey, you will help USAID design its activities to help marginalized groups and strengthen the capacity of local organizations. The survey will take 5 minutes. Please answer the questions in an open, and frank manner. 1. Does your institution support young people (e.g. does it work on projects or initiatives aimed at improving the status of this group)? Yes No 2. QC4. In your opinion, has the status of Youth improved over the last 5 years in terms of…? Not at all Slightly Moderately Considerably Increased awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 79 Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 3. Do you focus on advancing women's status? Yes No 4. QC5. In your opinion, has the status of women improved in relation to the past 5 years in terms of…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group faces ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 6. Does your institution support persons with disabilities (e.g. work on projects or initiatives aimed at improving the status of this group)? Yes No 80 6. QC6. In your opinion, has the status of persons with disabilities improved OVER THE PAST 5 YEARS IN TERMS OF…? Not at all Slightly Moderately Considerably 1. Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 2. New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 3. Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ 4. Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ 5. Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 7. Does your institution support the Roma national minority (e.g. work on projects or initiatives aimed at improving the status of this group)? Yes No 8. QC7. In your opinion, has the status of Roma improved in the last 5 years in terms of…? Not at all Slightly Moderately Considerably 81 Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this grou ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 9. Does your institution support LGBTI (lesbian, gay, bisexual, transgender and intersex) persons, for example, working on projects or initiatives aimed at improving the status of this group? Yes No 18. QC8. In your opinion, has the status of lesbian, gay, bisexual, transgender and intersex persons improved over the past 5 years in terms of…? Not at all Slightly Moderately Considerably Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ 82 Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 19. QG3. What are the main problems these groups face? 12. QC8a. Apart from young people, women, people with disabilities, Roma and members of the LGBTI population, are there other marginalized groups that need special support? Please specify. 13. QF9. Have you worked with any of the following international organizations on interventions focused on marginalized groups in the past five years? 83 14. QT18. Did you or anyone from your institution participate in any training organized by PPMG? Yes No If your answer to the previous question is YES, please answer the following questions: 15. QT18b. Number of training days, including all staff (approximately if you do not remember): 16. QT19. Please rate the following training elements: Not at all On a small scale Moderately On a large scale Did not attend Quality of training in general ☐ ☐ ☐ ☐ ☐ Level of knowledge among experts who held training ☐ ☐ ☐ ☐ ☐ Teaching skills ☐ ☐ ☐ ☐ ☐ Stimulating active participation among participants in training ☐ ☐ ☐ ☐ ☐ Opportunities to practice new skills acquired during training ☐ ☐ ☐ ☐ ☐ 84 Suitability of training topics for your organization ☐ ☐ ☐ ☐ ☐ Usefulness of gender equality sessions ☐ ☐ ☐ ☐ ☐ 17. QCB20x. How much/up to which degree /has your capacity improved THANKS to PPMG TRAININGS in each of the following areas? Not at all On a small scale Moderately On a large scale Didn’t attend Teamwork and leadership during the work with young people ☐ ☐ ☐ ☐ ☐ Administrative procedures in municipalities and authorities ☐ ☐ ☐ ☐ ☐ Monitoring and evaluation ☐ ☐ ☐ ☐ ☐ 18. QGLG. Has your institution received a grant for women's businesses under the Marginalized Groups Support Program?? Yes No 19. QFE26. How many businesses run by women have you supported through PPMG and your own funds? 20. QFE26a. How many women whose businesses are supported belong to each of these groups? 85 21. QFE27. How many businesses run by women are still active? 22. QFE28. Would you agree that, according to the criteria required by the PPMG, all categories of women who are eligible for grants (women victims of war, victims of violence, women with disabilities, mothers of children with disabilities, young unemployed women) had or have the same opportunity to receive grant? Strongly disagree Somewhat disagree Neutral Somewhat agree /agree Strongly Agree 86 23. What is your level of satisfaction with each of the following items? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Grant management by PPMG ☐ ☐ ☐ ☐ ☐ PPMG staff professionalism ☐ ☐ ☐ ☐ ☐ Providing individual assistance to your institution by PPMG during grant implementation ☐ ☐ ☐ ☐ ☐ Requirements and complexity for PPMG’s reports on grant implementation ☐ ☐ ☐ ☐ ☐ Support and technical assistance provided by PPMG during the implementation of the grants ☐ ☐ ☐ ☐ ☐ 24. Have you received any of the following instructions or training regarding the monitoring of your grant? Yes No Precise instructions on how to count project participants ☐ ☐ Precise instructions on how to avoid double counting when reporting on the number of project participants ☐ ☐ 25. Additional comments and observations: 87 Marginalized Groups Support Program Participant Survey This survey is conducted within the framework of the Impact Assessment of the Marginalized Groups Support Program (MGSP) and the assessment of the situation of marginalized groups in BiH by USAID/BIH’s MEASURE-BiH, the BiH Monitoring and Evaluation Support Project (for more information on MEASURE-BiH, visit www.measurebih.com). The purpose of this survey is to identify good practices and areas for improvement in the implementation of the Marginalized Groups Support Program in BiH, with the aim of improving existing and future programs designed to support marginalized groups and local organizations advocating for the rights of these groups. Your opinion is extremely important to us. Therefore, we would appreciate if you take the time to complete this survey, even if you have already participated in the interviews within the framework of this evaluation and assessment. It takes about 10 minutes to complete the survey. All responses you provide in this survey will held fully confidential and anonymous. Please respond to the survey questions in an open, frank and timely manner. Which of the following groups does your organization focus on in its work? 1. Do you focus on promoting YOUTH status? Yes No 2. QC4. In your opinion, has the status of youth improved over the last 5 years in terms of…? Not at all Slightly Moderately Considerably Increased the awareness and understanding of the authorities about the problems facing this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these ☐ ☐ ☐ ☐ 88 groups or increased quality of support from civil society organizations 3. Do you focus on advancing women's status? Yes No 4.QC5. In your opinion, has the status of women improved in relation to the past 5 years in terms of…? Not at all Slightly Moderately Considerably Increased awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 89 6. Do you focus on enhancing the status of PEOPLE WITH DIFFICULTIES AND THEIR FAMILIES? Yes No 7.QC6. In your opinion, has the status of persons with disabilities improved OVER THE PAST 5 YEARS IN TERMS OF…? Not at all Slightly Moderately Considerably Increased awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 10. Do you focus on improving the status of the Roma? Yes No 90 11. QC7. In your opinion, has the status of Roma improved in the last 5 years in terms of…? Not at all Slightly Moderately Considerably Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 12. Do you focus on improving the status of the LGBTI population (lesbian, gay, bisexual, transgender and intersex)? Yes No 20. QC8. In your opinion, has the status of lesbian, gay, bisexual, transgender and intersex persons improved over the past 5 years in terms of…? Not at all Slightly Moderately Considerably Increased the awareness and understanding of the authorities about the problems faced by this group ☐ ☐ ☐ ☐ 91 New government policies that protect the rights of this group ☐ ☐ ☐ ☐ Increased awareness and understanding among citizens ☐ ☐ ☐ ☐ Increased media coverage of the problems this group is facing ☐ ☐ ☐ ☐ Increased number of civil society organizations supporting these groups or increased quality of support from civil society organizations ☐ ☐ ☐ ☐ 11. QG3. What are the main problems these groups face? 12. QC8a. Apart from young people, women, people with disabilities, Roma and members of the LGBTI population, are there other marginalized groups that need special support? 13. QF9. Have you received funding from the following sources over the past five years? 92 The following questions are related to your organization and its sources of funding. 14. QG10. The year your organization was registered? 15. QG11. Number of persons in paid employment? 16. QF12. What are the primary challenges facing civil society organizations in BiH when it comes to fundraising? 17. QF13. What funding sources does your organization use? Funds of ministries and/or government institutions in BiH at any level 93 18. QF14. What proportion of the funds does your organization have from each of these sources? (The sum of these percentages should be 100 %.) 19. To what extent do you agree with the following statements? Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Local organizations that are forced to adjust their programs to donor strategies to obtain funding ☐ ☐ ☐ ☐ ☐ Most local organizations do not have enough resources to hire the optimal number of workers ☐ ☐ ☐ ☐ ☐ Many organizations that used to be donors are now applying for donor funding and as a result local organizations have reduced access to funding ☐ ☐ ☐ ☐ ☐ Donors who are funding local organizations are demanding high requirements for the funds they donate ☐ ☐ ☐ ☐ ☐ Company revenue achieved by my organization 94 Donor organizations often finance short-term projects that cannot achieve the desired results ☐ ☐ ☐ ☐ ☐ Long-term programs need to be implemented to achieve changes in the status of marginalized groups ☐ ☐ ☐ ☐ ☐ Local authorities finance local organizations but invest minimal resources ☐ ☐ ☐ ☐ ☐ Local authorities mostly support veterans and/or sports associations ☐ ☐ ☐ ☐ ☐ Constant changes of politicians in positions in governmental institutions slow down changes in the status of marginalized groups ☐ ☐ ☐ ☐ ☐ 20. QT18. Did you or someone from your organization or institution participate in any of the PPMG trainings? Yes No 21. Please answer the following questions: QT18a. Number of staff from your organization who attended the training (approximately if you don't remember) QT18b. Number of training days, including all staff (approximately if you don’t remember) 22. QT19. Please rate the following training elements: Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Quality of training in general ☐ ☐ ☐ ☐ ☐ 95 Level of knowledge among experts who held training ☐ ☐ ☐ ☐ ☐ Teaching skills ☐ ☐ ☐ ☐ ☐ Stimulating active participation among participants in training ☐ ☐ ☐ ☐ ☐ Opportunities to practice new skills acquired during training ☐ ☐ ☐ ☐ ☐ Suitability of training topics for your organization ☐ ☐ ☐ ☐ ☐ Usefulness of gender equality sessions ☐ ☐ ☐ ☐ ☐ 23. Did you receive homework after PPMG training? Yes No 24. QCB20. How much has your organizational capacity improved THANKS to PPMG TRAININGS in each of the following areas? Not at all Slightly Moderately Considerably Compliance with legal requirements ☐ ☐ ☐ ☐ Administration ☐ ☐ ☐ ☐ Project management ☐ ☐ ☐ ☐ Fundraising ☐ ☐ ☐ ☐ Monitoring and Evaluation ☐ ☐ ☐ ☐ 96 Human resources ☐ ☐ ☐ ☐ Financial management and reporting ☐ ☐ ☐ ☐ Advocacy ☐ ☐ ☐ ☐ Writing project proposals ☐ ☐ ☐ ☐ Marketing and promotion ☐ ☐ ☐ ☐ 25. To what extent do you agree with the following statements: Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree The reporting procedures under the PPMG program were too extensive and complex ☐ ☐ ☐ ☐ ☐ At trainings organized by PPMG we partnered with other organizations that are dealing with similar issues ☐ ☐ ☐ ☐ ☐ The trainings contributed to better networking and cooperation of our organization with other organization working on similar issues ☐ ☐ ☐ ☐ ☐ 26. Have you ever gone through a similar capacity building process before? 97 27. QCB21. Has your organization developed new or modified existing organizational policies or procedures THANKS FOR PPMG TECHNICAL ASSISTANCE TO CAPACITY BUILDING? Yes No 28. QCB21a. Who made the decision to develop/change organizational policies/procedures/regulations? The decision was made by our management/staff PPMG advised us to develop specific organizational policies PPMG required from us to develop specific organizational policies 29. QCB21b. How much has the PPMG helped your organization develop organizational policies/procedures/regulations? Not at all Somewhat Moderately Considerably 30. QCB22. How helpful were these policies for your organization? Not at all Somewhat Moderately Considerably 31. QCB23. How often do you use these policies and procedures? Never Often Rarely All the time Sometimes 32. QR25. To what extent do you agree with the claim that PPMG capacity building has improved your organization's representation in civil and political matters? Strongly disagree Agree Disagree Strongly agree Neither agree nor disagree 98 33. QR24. Has your organization experienced any of these changes ACKNOWLEDGING PPMG'S FINANCIAL ASSISTANCE OR CAPACITY BUILDING ASSISTANCE? Not at all Slightly Moderately Considerably Increased involvement in development of rights’ policies for marginalized groups ☐ ☐ ☐ ☐ Increased frequency engaging in advocacy for rights marginalized groups ☐ ☐ ☐ ☐ Increased quality advocacy intervention for rights of marginalized groups ☐ ☐ ☐ ☐ Increased interaction with governmental institutions ☐ ☐ ☐ ☐ Increased funding by government institution on any level because of skills acquired through this program ☐ ☐ ☐ ☐ Increased funds from other international donors other than USAID because of the skills acquired through this program ☐ ☐ ☐ ☐ Increased cooperation with other NGOs that deal with similar issues ☐ ☐ ☐ ☐ 34. What advocacy initiatives has your organization initiated as a result of the PPMG program? 35. Additional comments and observations: 99 ANNEX III: LIST OF DOCUMENTS REVIEWED 1. Activity Award AID-168-I-15-00001 2. The Activity’s Task Orders #1 to #10, modifications and amendments 3. USAID/PPMG – Quarterly Reports 1 to 15 4. USAID/PPMG –Work Plans for 2015, 2016, 2017, 2019, and 2019 5. USAID/PPMG Monitoring and Evaluation Plan 6. USAID/PPMG’s BiHPERFORM entries (tracking table, quarterly reports) 7. USAID/PPMG 2017 DQA results 8. USAID/PPMG Pre-award survey reports 9. USAID/PPMG Grant Awards 10. USAID/PPMG Grant Manual 11. USAID/PPMG Grantees’ Progress Reports 12. List of all PPMG grant applicants, grantees, experts/trainers 13. Attendance Sheets from PPMG trainings 14. USAID/PPMG Training Agendas 15. USAID/PPMG Surveys of training participants 16. USAID/PPMG Capacity Development Plans of grantees 17. USAID/PPMG Organizational Capacity Assessments of grantees 18. USAID/PPMG Organizational Capacity Assessments results of grantees 19. USAID/PPMG Final Reports of grantees 20. USAID/PPMG Request for application – examples 21. USAID/PPMG Monitoring Visits Reports 22. USAID/PPMG’s Press-clipping Reports 100 ANNEX IV: DISCLOSURE OF CONFLICTS OF INTEREST Name Salminka Vizin Title Senior Research Analyst/C2 Lead Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member (Methods Lead) Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/BiH Marginalized Populations Support Activity (PPMG), implemented by the Institute for Youth Development KULT, Award No. AID-168-I-15-00001 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 1, 2019 101 Name Nikolina Obradovic Title External Expert Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/BiH Marginalized Populations Support Activity (PPMG), implemented by the Institute for Youth Development KULT, Award No. AID-168-I-15-00001 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 4, 2019 102 Name Bergin Kulenovic Title External Expert Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/BiH Marginalized Populations Support Activity (PPMG), implemented by the Institute for Youth Development KULT, Award No. AID-168-I-15-00001 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 4, 2019 103 Name Boris Badza Title GIS Specialist Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/BiH Marginalized Populations Support Activity (PPMG), implemented by the Institute for Youth Development KULT, Award No. AID-168-I-15-00001 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 4, 2019 104 Name Amina Smajovic Title Analyst Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/BiH Marginalized Populations Support Activity (PPMG), implemented by the Institute for Youth Development KULT, Award No. AID-168-I-15-00001 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 4, 2019 105 ANNEX V. MEASURE-BIH RESPONSES TO COMMENTS ON DRAFT MARGINALIZED POPULATIONS SUPPORT ACTIVITY PERFORMANCE EVALUATION REPORT COMMENT: by PPMG (general comment) The USAID/PPMG's general comment is that more positive examples should have been outlined for the purpose of collecting experiences for potential future similar programs (e.g. 50% of local government co￾funding for the Marginalized Women Fund, support to organizations to establish legal ways of functioning, funding obtained from other donors following the capacity building, quarterly monitoring of the OCA process, changes to local regulations that change the daily lives of marginalized people, etc.). The Report gives the impression that the USAID/PPMG was responsible for initiating and implementing advocacy initiatives. The task of the Activity was to provide technical, advisory and financial support in the implementation of the grants. We believe that the Report contains important recommendations and findings, primarily for the creation of a future similar program, but also for the immediate improvement of the USAID/PPMG. For the purposes of the evaluation, the PPMG made available nearly 1000 documents. We regret that the response rate to the questionnaires was low; we believe that the PPMG may have prompted the beneficiaries to fill the questionnaires in, but this could have possibly affected the survey results. We would like to complement the fair collaboration with the MEASURE Team on conducting the evaluation. RESPONSE: by MEASURE-BiH Evaluation findings were formulated to be comprehensive and relevant for the evaluation questions. The evaluation team highlighted a number of positive examples when answering the evaluation questions, although not as separate findings but as parts of more comprehensive findings:  50% of local government co-funding for the Marginalized Women Fund – mentioned in finding 20  support to organizations to establish legal ways of functioning – see finding 7, where the evaluation team added a note related to provision of support through quarterly monitoring visits, in addition to using emails and phone  funding obtained from other donors following capacity building – mentioned in finding 13 of the evaluation report  quarterly monitoring of the OCA process – added to findings 2 and 7  changes to local regulations that change the daily lives of marginalized people – mentioned in finding 16. In terms of advocacy, one of the PPMG’s performance indicators refer to the number of advocacy interventions supported, so this is one of the areas of the Activity’s performance. It is important to note that evaluations are expected to compare the Activity’s design with implementation, and that evaluation findings do not refer solely to the IP’s performance, but to USAID/PPMG as a whole. 106 FINDING The intensity of assistance provided to non-grantee organizations and government officials was considerably lower. COMMENT: by PPMG Technical assistance to non-grantee organizations and government officials was not the focus of the training activity nor our task. In situations when any non-grantee organization and government officials asked for any kind of assistance, the PPMG team addressed the defined issues. RESPONSE: by MEASURE-BiH The evaluation team revised the sentence to: As envisaged in the Activity design, the intensity of assistance provided to non-grantee organizations and government officials was considerably lower. FINDING Finding 1. PPMG reported assisting a considerably higher number of underrepresented people compared to the life of activity (LoA) targets. However, based on KIIs and the Activity documentation, the number reported may not correspond to the actual number of underrepresented persons assisted. The Activity envisaged assisting 8,900 persons over the LoA. The Activity considerably exceeded this target, as they reported assisting 14,138 underrepresented individuals in four years of implementation. According to the definition in the PIRS and the KII with the IP, in addition to the beneficiaries reported by the grantees, the Activity also counts training participants under this indicator. However, it is not clear how many training participants belong to marginalized categories as defined by the Program. This is particularly of concern in relation to marginalized youth and women, who are not clearly defined in the PIRS. During the Data Quality Assessment (DQA) site visit conducted in September 2017, the IP clarified that some participants were counted multiple times if they attended multiple activities, within and across reporting periods. Finally, the PPMG does not possess the lists of participants in grantees’ interventions, but uses the numbers reported in grantees’ quarterly reports when calculating this indicator. Hence, the numbers that the PPMG reports cannot be verified on site. Based on the DQA site visit in 2017 it was recommended that the PPMG should address these data quality issues. However, this has not been done to date. COMMENT: by PPMG During the DQA, we had disagreements regarding the collection of personal information about individuals from third parties (grant beneficiaries). And then we emphasized that some organizations and individuals agreed to submit data while other organizations refused to do so because they did not want to share the beneficiaries' data with third parties. They have a legal basis for this attitude, which relates to the protection of personal data, which includes contact information (name, surname, address, phone, mail). During the DQA, we emphasized that this was a problem and that we could not implement it because we did not control the data collection process. Beneficiaries provide personal information to organizations 107 from which they use services and not to us as a third party. Grant beneficiaries do not agree to pass on information from the beneficiaries of their services without the beneficiaries' explicit agreement. RESPONSE: by MEASURE-BiH While noting the complexity of data privacy issues, the evaluation team finds that local organizations receiving USG assistance should properly document and share information about their beneficiaries with the donor for the purposes of transparent monitoring and evaluation, and that willingness to contribute to transparent monitoring of USG interventions should be a precondition for receiving financial assistance. FINDING Finding 2. The PPMG already exceeded the LoA target with regard to the number of advocacy initiatives focused on improving the status of marginalized populations in public and political life in BiH. However, based on KIIs and the Activity documentation, the reported number may not correspond to the actual number of advocacy initiatives since grantees are unaware of the definition of an advocacy initiative as defined in the PIRS. The LoA target for this indicator is 185 and the Activity reported conducting 201 (+16) advocacy initiatives in the first four years of implementation. The Activity’s PIRS provides some instructions on counting an advocacy initiative. However, according to the IP and the grantees, this definition was not shared with the grantees. In the DQA conducted in 2017, the IP clarified that the same advocacy initiative can be counted multiple times if advocating for adoption at different government levels. None of the interviewed grantees could remember if the Activity explained the definition or how to count advocacy interventions to them or their staff. Finally, the PPMG does not keep records of grantees’ initiatives, but use the numbers reported in grantees’ quarterly reports when calculating this indicator. The PPMG has been informed about conclusions and recommendations related to their data quality issues. However, the Activity has not modified their MEL Plan. COMMENT: by PPMG Within the quarterly report form, there is a definition of the 3 indicators for which we receive data from the grant beneficiaries. They were briefed on the detailed definitions at the briefings after the contract was signed (part of the presentation of the reporting and monitoring process of the grant). We need to take into consideration the fact that the USAID/PPMG works with grass root organizations which do not possess internal capacities for dealing, more proficiently, with monitoring. An enormous challenge would be to expect from the organizations to fill in a PIRS. As a replacement for a PIRS, the USAID/PPMG has created a separate section in the project application and in the quarterly reports for planning and reporting on the PPMG's indicators, in which the definitions are provided, so it does not seem that we have not shared the definition. During the analysis of quarterly reports, we often called the grantees to provide clarifications for the numbers they had indicated in order to prevent double counting. 108 RESPONSE: by MEASURE-BiH The evaluation team did not make any changes to this finding. The evaluation team finds that definitions in quarterly reports do not offer sufficient details to ensure that grantees count toward performance indicators the same way as the IP. Since the IP is responsible for this data and reports on this data, the IP needs to ensure that all actors whose data contribute to the indicators use the same definitions. FINDING Finding 3. The PPMG already exceeded the LoA target in terms of the number of human rights organizations trained and supported, although the KIIs and Activity documentation suggested that the grantees and the IP did not use the PIRS definition consistently when calculating this indicator. The LoA target for this indicator is 296 and the Activity reported assisting 426 human rights organizations in the four years of implementation. However, as explained by the IP, the Activity did not count only the organizations they trained and/or supported with grants toward this indicator. They also counted organizations participating in their grantees’ interventions. The Activity also counted organizations that attended different trainings multiple times, as well as the organizations supported in conducting advocacy initiatives. The latter is not in line with the USAID’s Standard Foreign Assistance Indicator (SFAI) definition. The intensity of exposure to trainings varied considerably for organizations counted under this indicator as grantees are obliged to attend five trainings, while non￾grantee organizations participated in selected trainings only. The Activity keeps the data on their own training participants in the form of scanned participant sign-in sheets. Additionally, the Activity did not keep detailed records that included data provided by grantees, relying instead on grantees’ reports. These data quality issues were identified during the DQA in 2017, however the MEL Plan has not been revised. COMMENT: by PPMG The counting of the organizations was the subject of discussion about double-counting issues when we spoke with the Measure, both before and during the DQA. Back then we explained to them that we count an organization that receives different services (more training) each time the organization uses a USAID￾supported service. The number of organizations participating in activities of the grant beneficiaries is also in line with the definition in the PIRS and it is clarified that those organizations are the beneficiaries of the services (supported). Nowhere in our PIRS does it say that we only count organizations once when they use the first service and no longer, but we count them every time they use different services supported by the USAID. RESPONSE: by MEASURE-BiH The evaluation team addressed this comment by adding the portion of the sentence in the brackets: The Activity also counted organizations that attended different trainings multiple times (however, this has been noted in their PIRS), as well as the organizations participating in grantees’ interventions, and those supported in conducting advocacy initiatives. 109 FINDING Finding 5. The number of government representatives trained to increase their organizational capacities was substantially slower than envisaged. According to the Activity MEL Plan, the Activity should train 115 government representatives over the LoA. However, the Activity reported training 42 representatives in the first four years of implementation. COMMENT: by PPMG As it is the case with other indicators, the targets indicated here are also indicative because here we are referring to the IDIQ contract in which no predefined activities have been set to determine the exact targets. During the Activity, the need for educating government officials was not perceived. RESPONSE: by MEASURE-BiH The evaluation team added a paragraph about the ID/IQ contract to the background section and explained in Finding 1 that the Activity could not have known the targets at the time when the MEL Plan was developed (please note that findings 1 to 5 have been integrated into a single finding in the final evaluation report). FINDING Finding 6. There is no clear evidence that the PPMG regularly verifies grantees’ MEL data. According to the IP, the Activity staff verifies MEL data during monitoring visits to grantees. However, other than visits to services centers, there is no evidence of such verification in the monitoring reports for other IPs. For other grantees, monitoring reports focus only on fulfilment of pre-award requirements. COMMENT: by PPMG After putting into practice the verification of indicators during our monitoring visits, we also performed checks on other grant beneficiaries, not just SCs. This group includes: MoRS, Srce puno osmjeha, both LGBTI grants (SOC and Helsinki Committee). Grants from the ODI and Anti-TIP series were not in this procedure, but at times we also checked the attendees' lists of events for visits, as we checked the registers. RESPONSE: by MEASURE-BiH Whenever possible, the evaluation team formulated the evaluation findings by triangulating different data sources. The evaluation report clearly states that, according to the IP, data has been verified, but also that there are no clear indications of this in monitoring reports (except in the reports on service centers). Therefore, the evaluation team did not make any changes to this finding. 110 FINDING Finding 8. Based on documentation review and KIIs, the PPMG employs high-quality procedures in grant award and management. --- Six out of ten grantees felt the reporting requirements were very complex and took too much time. According to the grant manual, grantees were required to submit monthly work plans, monthly program reports, monthly financial plans, monthly financial reports, quarterly program reports and final program reports. Two grantees noted that these procedures took too much of the time they needed to implement their interventions successfully. COMMENT: by PPMG The Grants Manual was approved by the USAID and their request was such an intense reporting process. The PPMG insisted on the introduction and constant enforcement of local legislation and the USG procedures. Some organizations resisted this because they had been accustomed by other donors not to be monitored on a monthly basis and that the organizational development was not important. The PPMG was created for the purpose to strengthen organizations' capacities they will be using following the USG support. RESPONSE: by MEASURE-BiH This finding is based on local organizations’ feedback and they are not intended for the IP only, but also for the USAID. FINDING Finding 9. Some elements of the Award have not been implemented as originally envisaged. --- According to the KIIs and Activity documentation, the PPMG employed a different model to support marginalized women compared to the way they worked with other marginalized groups. Local organizations advocating for the rights or providing services to other marginalized groups would receive financial and technical support (training). However, in working with marginalized women, the PPMG partnered with local governments and co-financed women’s start-ups instead of women’s organizations. COMMENT: by PPMG The Model for supporting women's businesses was based on a partnership with the local community that provides 50% of the funding, not with local NGOs dealing with women's issues. 111 The fact that we managed to motivate local governments to co-fund the Fund with at least 50% is worth to be mentioned. Local governments first provided their 50%, and then the USAID/PPMG funded the other 50%. Our grantees were local governments. Our intention was to empower local governments to adopt all necessary procedures to introduce the Fund as regular support, not just through the USAID, and to prompt them to keep the Fund even following the cooperation with the USAID / PPMG. It was not even intended for the PPMG to work individually with each beneficiary, but we were at disposal to all beneficiaries whenever they reached out to us. The Report lacks positive findings about co-funding from local budgets, the introduction of the Fund into local authorities as a policy level change, sensitization of local authorities on marginalized women related issues. RESPONSE: by MEASURE-BiH This finding notes the fact that PPMG does not work with women organizations. The report clearly states that the practice of matching funds with local governments was introduced, and that the governments probably would not support women start-ups if it were not for the PPMG. The evaluation team did not make any changes to this finding. FINDING Finding 9. Some elements of the Award have not been implemented as originally envisaged. For capacity building interventions, one-on-one coaching was not implemented as envisaged. TO #1 envisaged developing a one-to-one coaching plan for each grantee, which includes a three-day intervention (a stay in the field) with a focus on improving organizations’ administration and financial management. Four coaching measures were planned for each organization. However, according to the IP and Activity documentation, coaching was implemented during monitoring and study visits and consultations by email and phone were the dominant type of one-on-one assistance. COMMENT: by PPMG The PPMG staff regularly performed (planned and sudden) monitoring visits to each grantee on a quarterly basis. These visits contributed more to strengthening the grantees' capacity rather than staying in the field for three days. The period between two visits was an opportunity to detect the ongoing problems in the process of the project implementation and gave an opportunity for making proper interventions. RESPONSE: by MEASURE-BiH The evaluation team compared the difference between the planned approach and implementation, and noted the difference. We believe there is a reason why the design envisaged stays in the field. We could not find anything in the IP’s progress reports on reasons why the approach had been altered, and our intention was to note that this happened. 112 FINDING Finding 10. Although the PPMG design envisaged provision of support to persons belonging to five marginalized groups (youth, women, persons with disabilities, Roma, and LGBTI), financial support was most intensively provided to organizations working with persons with disabilities and their families. According to the Activity documentation, approximately 50 percent of total funding has been spent on interventions that aim to assist persons with disabilities and their families, and about 30 percent has been spent on the women’s entrepreneurship program. One-fifth of funds were allocated for interventions focusing on youth, Roma, and LGBTI (see Exhibit 6 for more detail). COMMENT: by PPMG USAID identifies target groups and amounts within the Task Order, which is outside our area of the Activity’s responsibility. RESPONSE: by MEASURE-BiH The finding does not state that the IP makes decisions about grant allocation. Also, this has not been stated as a negative finding, but as an objective analysis of what was done in order to show on which groups the Activity has focused and to inform USAID accordingly, so they can consider if they wanted to continue supporting these particular groups, add other beneficiary categories or shift support to other groups. FINDING Finding 11: According to KIIs and the online survey, the PPMG provided intensive capacity building and valuable guidance to grantee organizations through support in implementation of pre-award survey recommendations, the OCA Tool, and trainings. COMMENT: by PPMG All of the grantees have aligned their statutes and organizational structures in such a way as to ensure the division of management and executive functions, and they thus have aligned these with international standards. Prior to the PPMG intervention, in most of the organizations there was no division between the management and the executive functions, whereas now all grantees that have gone through the PPMG capacity building process have their statutes and an organizational structure with a clear division between the management and the executive functions. We find this to be important to emphasize. RESPONSE: by MEASURE-BiH As suggested, the evaluation team emphasized separation of duties in organizations’ accounting policies as a major accomplishment of PPMG by adding the following sentence to this finding: As noted by the IP, one of the most important accomplishments in terms of aligning their policies with international standards was that all grantee organizations introduced segregation of duties into their accounting policies. 113 FINDING Finding 22. The PPMG grantees’ interventions have been covered by the media, however only some organizations reported mainstream media coverage. Based on grantees’ progress reports, nine out of 18 organizations (excluding four service centers) reported that their interventions were covered by the mainstream media, including Klix.ba, nezavisne.com, avaz.ba, BHRT, FTV, or RTRS. The service centers, SOC, and the Association for Advancement of Education and Support for Children with and without Developmental Delays EDUS reported the most extensive coverage in the mainstream media. Some interventions that achieved important results did not receive extensive media coverage. Some examples include Proreha, the organization whose intervention helped 12 persons with disabilities to find employment, and SUMERO, whose intervention resulted in deinstitutionalization of 60 persons with disabilities. COMMENT: by PPMG Local organizations do not have enough capacity to reach out to the mainstream media, but their news was published by local media. This is why the USAID/PPMG made positive stories and promoted them in the mainstream media. The PPMG has detailed monitoring of the entire media coverage. In addition, the PPMG has been very actively promoting videos about individual grantees' results, in social media. RESPONSE: by MEASURE-BiH The evaluation team notes that, in order to reduce prejudice and improve social inclusion of marginalized populations, both local and mainstream media should be engaged. This finding was initially based on grantees’ performance reports and PPMG’s quarterly reports. Other documentation on this has not been at evaluation team’s disposal during data collection; only some Activity progress reports mention visibility. Based on press-clipping report that the IP delivered to the evaluation team after submitting their comments on the draft evaluation report, the original finding (finding 18 in the final report) was reformulated to read: Finding 18. PPMG grantees’ interventions received local media coverage, however only some organizations reported coverage by the mainstream media. Based on grantees’ progress reports, nine out of 18 organizations (excluding four service centers) reported that their interventions were covered by the mainstream media, including Klix.ba, nezavisne.com, avaz.ba, BHRT, FTV, or RTRS. The service centers, SOC, and the Association for Advancement of Education and Support for Children with and without Developmental Delays EDUS reported the most extensive coverage in the mainstream media. Some interventions that achieved important results, such as Proreha, which helped find employment for 12 persons with disabilities, and SUMERO, whose intervention resulted in deinstitutionalization of 60 persons with disabilities, did not receive extensive media coverage. According to the IP, grantee organizations lacked capacities to reach out to the mainstream media. The IP supported grantee organizations by helping draft and disseminate media materials to relevant media outlets. 114 According to the Activity documentation on press-clipping, the coverage in the mainstream media was highest at the start of the project (for the second half of 2015), and kept declining during the project. FINDING Finding 24: Eighteen local self-government units in BiH financially supported women-led start-ups due to the PPMG intervention. However, there is no evidence that these interventions improved participants’ perceptions about women’s roles in society or their representation in political and civic issues. The PPMG supported marginalized women in BiH by investing assets to support entrepreneurship among marginalized women. The PPMG advertised the Women Entrepreneurship Program in all municipalities in BiH. Municipalities decided on the size of their investment22 and the PPMG matched that amount. According to the IP, representatives of 39 local government units applied to invest in start-ups for marginalized women and 18 municipalities received PPMG funding. The first group of businesses were established in 2016, with nine municipalities supporting 96 businesses. Without the PPMG, it is likely that these funds would have been allocated for other purposes, as municipalities did not previously have such programs for women (according to the IP). Six municipalities that participated in the first round of the program responded to the online survey. The results showed that these six municipalities supported 69 businesses. Of this number, 44 businesses, or 63 percent, are still operational. The analysis showed that most grants were awarded to support unemployed women (28 out of 44), and that other categories of marginalized women (i.e. victims of violence, mothers of children with disabilities) were less represented. Although women’s interaction with local governments improved during the grant implementation, there is no evidence that this type of assistance had long-term effects on political or civic participation of women. Further, there is no evidence that it improved perceptions about women’s traditional role in the society, or about women’s rights. More specifically, the conclusions of the focus group discussion with women who started their businesses with PPMG support suggest that women’s attitudes toward gender roles have not changed and that women have not become aware of their rights. Out of six participants, only one woman seems to have started a business that she actually runs on her own. Other women seem to have used the grants to support their family businesses led by men (four FGD participants) or to obtain their first job (one FGD participant). COMMENT: by PPMG The primary objective of the Fund was the financial independence of marginalized women and not the improved participants' perceptions of women's roles in society or their representation in political and civic issues. We had two calls for women's businesses, the total number of applications from municipalities to both calls was 79, not 39 (on the first call 39 local communities applied and 40 on the second call). Data on 63% of still active businesses is very good and has not been adequately highlighted. This is an extremely high percentage for start-ups and this has not been emphasized. 22 The condition was that the investment could not be lower than 20,000 BAM and no higher than 100,000 BAM. 115 It is necessary to make a comparison with the successes made by other local and international projects in the same field. RESPONSE: by MEASURE-BiH Based on the data collected, the evaluation team found that, in addition to financial independence of marginalized women, efforts should be made to improve women’s awareness about their rights and their role in the society. The finding on the number of applications has been corrected. The Activity purpose, according to the LogFrame, was to improve political and civic representation of marginalized groups, including women. FINDING Finding 25. According to the PPMG website, most organizations financially supported by the Program have been registered in the Canton Sarajevo. The PPMG’s support to organizations in other regions was more limited. According to the PPMG website, the Program supported 15 organizations in the Canton Sarajevo (15 interventions), followed by four in the Tuzla Canton and four in the eastern RS. The Program supported eight organizations from the RS, three in Herzegovina, and no organizations from the Brcko District (see Exhibit 14). When it comes to the Women’s Entrepreneurship Program, the participating municipalities were spread out across BiH. However, several small cantons, i.e., Posavina and Bosnian Podrinje, received no support within this program (see Exhibit 15). COMMENT: by PPMG USAID dictates through the TOs the locations where grants will be made (except for RFAs). This finding is correct, but it is not right to be presented in a way that would burden the PPMG or portray the work in a negative light. The in-kind support to the Plivački klub and to the Vatrogasno društvo also cannot be equally matched to the same level, in which case we have financially insignificant support in relation to the funds for other grants throughout the entire BiH. RESPONSE: by MEASURE-BiH The aim of the evaluation finding has been to show how the funds have been distributed geographically. It is neither a positive nor a negative finding. FINDING Finding 26: LGBTI rights are still a taboo topic among the public and in government institutions and the PPMG seems to have had the most limited success in this area. The organizations promoting LGBTI rights invested considerable efforts to improve their representation in political and civic issues during the PPMG, by improving their collaboration with government and citizens. 116 However, their efforts had limited reach, as they focused on three geographic regions (Sarajevo, Tuzla and Prijedor) and they have rarely been covered by mainstream media. Representatives of government institutions interviewed were reluctant to talk about LGBTI rights and none of those interviewed stated that their institutions provided support to the organizations working in this field. Also, of all local organizations we spoke with, a local organization focusing on LGBTI rights is the only one interviewed that receives no funding from government sources. The number of organizations representing LGBTI rights in BiH is still small; according to a KI, there are only three registered organizations in the country that are focused primarily on the LGBTI population. However, KIs and focus group participants noticed changes in their position in the last five years, pointing out that, five years ago, holding the Pride event would not even have crossed their minds, but currently they are preparing for the Pride in September 2019. However, the representatives of this organization noted that the Pride may not happen if they assess that security of participants cannot be guaranteed. It is important to note that multiple donors support these organizations, so the results can be only partially contributed to the PPMG assistance. COMMENT: by PPMG If we take a look at the Task Orders, the USAID/PPMG's focus was not on the LGBT population. Despite this, organizations provided with a grant have been strengthened to be more prepared to take major steps to promote the rights of the LGBT population, including Pride 2019. RESPONSE: by MEASURE-BiH The evaluation added a sentence to this finding stating that only three grants had been distributed to LGBTI organizations. 117 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) Fra Anđela Zvizdovića 1 UNITIC Tower B, Floor 21 71000 Sarajevo Bosnia and Herzegovina Phone: + (387) 033 941 676 contacts@measurebih.com www.measurbeih.com