Photo credit: Konbit Ayiti EVALUATION REPORT MID-TERM PERFORMANCE EVALUATION OF THE Konbit PROGRAM Haiti Evaluation and Survey Services Project April 15, 2020 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. EVALUATION REPORT MID-TERM PERFORMANCE EVALUATION OF THE Konbit PROGRAM April 2020 Evaluation Mechanism Number: AID-521-A-15-00009 Haiti Evaluation and Survey Services for USAID/Haiti DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT Social Impact’s Haiti Evaluation and Survey Services project conducted an independent mid-term evaluation of the United States Agency for International Development (USAID)/Haiti’s Capacity Building Mechanism (Konbit) Activity implemented by Papyrus S.A. Konbit was designed to increase the number of local development partners influencing and achieving development results in Haiti. The evaluation report aims to inform USAID/Haiti about possible mid-course corrections and Konbit’s future program orientation. Konbit provided a range of capacity development (CD) services, primarily targeting 23 USAID local awardees and nine other Haitian entities aiming to become awardees and/or deliver CD services to others. Three awardees have achieved significant improvements in internal capacity, and at least four have new or improved core systems and procedures. However, half of all targeted organizations did not actively engage with Konbit for reasons including reluctance or inability of awardees and USAID technical office staff to focus efforts on CD and poor USAID-Konbit communications. Thus, three years of training and technical assistance (TA) from Papyrus and international partners has been absorbed by about ten organizations that participated consistently, while most others attended sporadically. Although participant feedback is generally positive and capacity assessments indicate some progress, Konbit monitoring has yet to capture the effects of training and TA on organizations. CD grants show promise in helping to overcome capacity gaps, but slow grantmaking processes have limited their impact. Moreover, Konbit has taken few steps towards strengthening and using the expertise of Haitian CD service providers, which could undermine its success as a capacity building hub. TABLE OF CONTENTS Abstract Abbreviations and Acronyms Executive Summary i I. Introduction 1 Activity Background 1 Theory of Change 1 Implementers 1 Components 2 Evaluation Purpose and Audience 4 Evaluation Questions 4 II. Evaluation Met holodogy 5 Sa mpling 5 Data Collection M ethods 5 Analysis 6 Limitations and Mitigation Strategies 7 III. Findings and Conclusions 8 EQ 1 9 EQ 2 16 EQ 3 23 EQ 4 26 IV. Recommendations 28 For USAID 28 For Konbit 29 ANNEXES Annex A: Evaluation Statement of Work 31 Annex B: Evaluation Team Members 38 Annex C: Konbit Theory of Change 40 Annex D: Konbit Results Framework 41 Annex E: List of Awardees And Core CDSPS 42 Annex F: List of Persons Interviewed 43 Annex G: Evaluation Design Matrix 45 Annex H: Documents Reviewed 47 Annex I: Data Collection Protocols 49 Annex J: OCA Scoring Highlights 65 Annex K: Disclosure of Conflict of Interest 66 TABLE OF TABLES Table 1: Konbit Consortium Roles 1 Table 2: Number of Interview Participants by Stakeholder Group 6 Table 3: Awardees' Engagement in Konbit vs. Capacity Improvement 12 Table 4: Number of System and Procedure Improvements Reported by Awardees 13 Table 5: Awardee and Core CDSP Engagement in Konbit vs. Capacity Improvement 17 Table 6: Personnel Roles and Responsibilities 38 Table 7: List of Awardees and Core CDSPs 42 Table 8: List of Persons Interviewed 43 Table 9 . Evaluation Design Matrix 45 TABLE OF FIGURES Figure 1: Konbit Theory of Change....................................................................................................................... 40 Figure 2: Konbit Results Framework .................................................................................................................... 41 Figure 3: OCA Scoring Highlights.......................................................................................................................... 65 ABBREVIATIONS AND ACRONYMS ADS Automated Directive System AGCRDDM Association de Gestion du Centre Rural de Développement Durable de Montrouis ANACAPH Association Nationale des Caisses Populaires Haïtiennes ANIMH Association Nationale des Institutions de Microfinance d'Haïti AOR Agreement Officer’s Representative APS Annual Program Statement CAP Capacity Action Plan CD Capacity Development CDSP Capacity Development Service Provider CEDEL Centre d’Entrepreneurship et de Leadership en Haiti CEGEC Cabinet d’Etudes de Gestion, d’Economie et de Comptabilité CFET Centre de Formation et d’Encadrement Technique CMP Centre Universitaire de Management et de Productivité COP Chief of Party COR Contracting Officer’s Representative CPFO Centre de Promotion des Femmes Ouvrières CSL Centre Spécialisé de Leadership DAI Development Alternatives, Inc. (now known as DAI Global) DIN Development Innovation Network DSLF Danielle St. Lot Foundation ECA Experts Conseils et Associés EQ Evaluation Question EQUI® Evaluation Quality, Use, and Impact ESS Evaluation and Survey Services ET Evaluation Team FAA Fixed Amount Award FACSDIS Femmes En Action Contre La Stigmatisation Et La Discrimination Sexuelle FGD Focus group discussion FHADIMAC Fondation Haïtienne de Diabète et de Maladies Cardio-Vasculaires FONHDAD Fondation Haïtienne de Développement Agricole Durable FONHEP Fondation Haïtienne de l'Enseignement Privé FONKOZE Fondasyon Kole Zepòl FPDI Fondation Paradis des Indiens FY Financial Year GI Group interview GHESKIO Groupe Haïtien d'Etude du Sarcoma da Kaposi at des Infections Opportunistas HQ Headquarters IDEH Initiative pour un Développement Equitable en Haïti IDEJEN Initiative pour le Développement des Jeunes IFOS Institut de Formation du Sud IR Intermediate Result J/P HRO Jenkins-Penn Haitian Relief Organization JSI John Snow International KCS Konbit pour la Compétitivité Stratégique KII Key Informant Interview KOFAVIV Komisyon Fanm Viktim pou Viktim LEVE USAID/Local Enterprise and Value Chain Enhancement Project M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning N.p. Not published OCA Organizational Capacity Assessment OFM Office of Financial Management PA Program Assistant PAGS PAGS Cabinet d’Experts Comptables PD Project Director PM Project Manager RFA Request for Applications SEFIS Société d'Etudes et de Formation en Information Stratégique SGP Small Grants Program SHAA Société Haïtienne d'Aide aux Aveugles SI Social Impact SKA Steve Kroll and Associates SOW Scope of Work TA Technical Assistance TL Team Leader USA United States of America USAID United States Agency for International Development USG United States government i| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested Social Impact’s (SI) Evaluation and Survey Services (ESS) project to design and conduct an independent mid-term evaluation of the Capacity Development (CD) Mechanism Activity, known as Konbit (see Annex A: Evaluation Statement of Work [SOW]). The purpose of this report is to present the evaluation team’s (ET) main findings, conclusions, and recommendations. EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this mid-term performance evaluation is to inform USAID/Haiti about possible mid￾course corrections and Konbit’s future program orientation. Lessons learned and recommendations provided through the evaluation will also guide future programming in areas such as organizational governance, leadership, financial management, project management, human resources, monitoring and evaluation (M&E), and other aspects of organizational development. The primary audience for the evaluation is USAID/Haiti. Secondary audiences include Papyrus and its sub-partners as Konbit implementers, and other counterparts to be identified by USAID/Haiti. This report answers the Evaluation Questions (EQs) below: 1. To what extent have local USAID/Haiti awardees working with Konbit acquired and implemented systems and procedures to manage USAID funds effectively and responsibly? The response includes: where firms are at in the process currently, noting significant successes and failures; what remains to be done so that the systems and procedures are in place to ensure sustainability of new/improved systems and procedures; and any potential obstacles to organizations’ ability to independently and appropriately manage USAID funds. 2. How are organizations applying the knowledge and tools gained through Konbit (such as Organizational Capacity Assessment [OCA], Helpline, Development Innovation Network (DIN), and M&E)? 3. To what extent are systems and procedures to implement the small grants program in place and sufficient to ensure effectiveness? 4. To what extent and in what ways are project activities affecting gender equality and women’s empowerment? PROJECT BACKGROUND USAID/Haiti recognizes the need for a more intensive and expansive CD approach when partnering directly with local organizations. The Mission and its Local Solutions team envisions Konbit as a means to reinforce Haitian Capacity Development Service Providers (CDSPs), who, in turn, will strengthen local institutions in various areas of internal capacity. In October 2015, USAID awarded the Konbit contract to Haitian firm Papyrus S.A. and its three sub-partners with an original ceiling price of $4,650,000 (increased to $7,495,166 in 2016). Konbit’s stated goal is to increase the number of development partners influencing and achieving significant and sustainable development results in Haiti, who are accountable to their constituents, and can effectively compete for and manage resources. EVALUATION DESIGN, METHODS, AND LIMITATIONS The ET used a qualitative evaluation design consisting of document review, analysis of Konbit M&E data, key informant interviews (KIIs), and group interviews (GIs). The team used purposive sampling to select informants who epitomized diverse stakeholder groups including representatives of USAID, the implementer team, targeted organizations including USAID awardees, other donors, and other USAID ii| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV implementers. During three weeks of in-country data collection, the ET interviewed 67 individuals from those groups, then used a range of methods to analyze the primary data and cross-reference to data from the document review. The ET developed mitigation strategies to overcome potential recall, selection, response, and gender bias risks in data collection and analysis. To increase the validity of findings, the team asked similar questions to all informants and triangulated responses. The lack of Konbit monitoring data related to the effects of their interventions on targeted organizations also imposed a limitation on the evaluation, especially in relation to EQs 1 and 2. FINDINGS AND CONCLUSIONS EQ 1: TO WHAT EXTENT HAVE LOCAL USAID/HAITI AWARDEES WORKING WITH KONBIT ACQUIRED AND IMPLEMENTED SYSTEMS AND PROCEDURES TO EFFECTIVELY AND RESPONSIBLY MANAGE USAID FUNDS? The evaluation focused on 23 targeted awardees for this question. Of those, 12 have not yet engaged in any significant way with Konbit, due to factors such as low levels of understanding, commitment, and interest among awardees and USAID technical office staff, and uncertainty about future USAID funding. USAID and Konbit personnel also mentioned that Konbit’s simultaneous timing with many new local awards was a disadvantage, and the discontinuation of Annual Program Statements severely limited the flow of new awardees to be targeted by Konbit. Communications between USAID and Konbit were problematic, with minimal information-sharing about what Konbit could offer and what awardees needed to become stronger implementers, which meant that most awardees’ Agreement Officer’s Representatives (AORs) and the Office of Financial Management (OFM) were not involved in identifying or addressing awardees’ CD needs in conjunction with Konbit. Capacity was highly variable among awardees in the three areas prioritized for this EQ (financial management, project management, and M&E), and the degree of capacity change has been very inconsistent. Five awardees have done more than one OCA, and all showed some improvement on overall scores. Considering all the evidence, six awardees appear ready to effectively handle direct USAID funding, but for most, no recent measurement is available. Successes among awardees working with Konbit include three awardees with significant improvements in capacity, and four to five with new or improved core systems and procedures in place. On the other hand, the minimal uptake of CD offerings by many awardees could hamper their ability to successfully execute current and future awards. A key challenge for Konbit was the attitude of some USAID/Haiti staff towards Local Solutions, which resulted in reluctance by technical office staff to cooperate in providing the support needed by local awardees. Low leadership engagement and/or weak governance structures have constrained improvement and sustainability of systems and procedures in several organizations, despite significant investments in their capacity. After governance, the most commonly mentioned constraint was awardees’ financial instability. If funding flows are interrupted, organizational capacity is immediately reduced, which was cited as the primary obstacle to sustainability of improved systems and procedures. EQ 2: HOW ARE ORGANIZATIONS APPLYING THE KNOWLEDGE AND TOOLS (SUCH AS OCA, HELPLINE, DIN, AND M&E) GAINED THROUGH KONBIT? This EQ encompasses a wider range of organizations, including awardees, Core CDSPs (organizations selected by Papyrus to be groomed for future USAID funding and/or CD service provision), and others. Konbit has not closely monitored the effects of training, technical assistance (TA) and other CD inputs on targeted organizations, which limited the ET’s ability to evaluate the application of knowledge and iii| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV tools imparted by Konbit in the operations and systems of those entities. Therefore, EQ 2 findings place more emphasis on participation and utilization of CD services. In terms of engagement in Konbit, the “Management Track” organizations aiming for future USAID funding have participated far more than other Core CDSPs—who largely lost interest when Konbit did not offer opportunities to deliver CD services to other organizations. In fact, a cluster of about ten Core CDSPs and awardees have participated consistently in most training, TA, and roundtables, while most others have attended sporadically. Konbit implementers have provided nearly all CD services so far, although the project design specified that local partners were supposed to benefit from “expertise transfer” from Konbit’s international partners and increasingly deliver these services. This could undermine the success of Konbit as a capacity-building hub. The majority of OCA-assessed organizations and Konbit staff expressed satisfaction with the OCA process as a diagnostic tool to identify weak points and form a plan to address them. Besides the awardees mentioned under EQ 1, two CDSPs completed follow-up OCAs and both achieved moderate improvements. However, the OCA has a range of limitations, including questionable efficacy in monitoring change over time and variation depending on who is facilitating the assessment. Also, Konbit has not closely monitored the resulting Capacity Action Plans (CAPs), so it is unclear to what extent organizations are addressing identified weaknesses. Konbit has involved 95 organizations in training workshops, and participants were generally satisfied with their quality and indicated a positive effect on their organizations. However, Konbit has not yet monitored specific effects of training and TA on participating organizations, as noted above, and those organizations raised concerns about training not being followed up with specific activities to strengthen uptake and application of learning at an organization level. Overall demand for the Experts Konbit advice service has been relatively low, with project data showing 130 instances of use to date, though recent innovations, such as Speed-linking have incited more interest. Eight awardees and CDSPs interviewed said the advice had been useful or had a positive effect. The majority of targeted organizations said that Konbit TA and coaching had been highly useful and expressed a desire for more tailored TA. With respect to Helpline, the remote advice service by Konbit’s international partners, the vast majority of advice has been dedicated to Ayiti Nexus, which was seen as useful for that organization. Other CDSPs have made very limited use of this opportunity (only available to Management Track CDSPs). Konbit has organized 59 diverse networking events, ranging from media tours to networking power breakfasts. Most events have targeted awardees and core CDSPs repeatedly, while recurrent participation by others has been low. Awardees and CDSPs generally agreed that the presentation quality at roundtables had been good, while they had mixed views on the usefulness of these events. The majority of informants consider the DIN platform to be a good idea with strong potential to facilitate information sharing and networking. Although more than 1000 organizations have registered, few users are returning to the site after registering. Most informants indicated that they seldom use the platform, mentioning technical limitations including English language and awkward user interface. Konbit staff said that certain features, expected when the host was chosen, have not been delivered. EQ 3: TO WHAT EXTENT ARE SYSTEMS AND PROCEDURES TO IMPLEMENT THE SMALL GRANTS PROGRAM IN PLACE AND SUFFICIENT TO ENSURE EFFECTIVENESS? The Konbit Grants Manual contains a very detailed description of the grants process, including instructions for staff complemented by extensive formats and examples. Konbit documents and interviews with staff and grantees indicate the manual has been implemented as written. No grants have been terminated early and two grants have been closed out satisfactorily. USAID personnel have observed no compliance issues or other concerns with grantmaking rules. iv| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV To date, the project has funded only 11 grants (nine CD and two innovation) out of the life of project target of 47 grants. These amount to approximately $750,000 of a total grants fund of $2,845,000. The time for issuance of both CD and innovation grants has averaged 7.2 months from the time of starting an application to first funds advance, generating frustration among applicants. The grant negotiation processes have been difficult, as most applicants did not clearly understand Fixed Amount Award processes nor how to budget appropriately. Therefore, they needed extensive handholding and time to finalize terms, but the majority of CD grantees agreed that the process had built their capacity. The non-competitive CD grants average about $50,000 for up to one year. Grantee representatives and Konbit staff stated that these grants were helping to address capacity gaps that would otherwise be difficult to overcome. While most CD grants are still being implemented, recent OCA results show that the two awardees whose grants are closed have improved in grant-related capacity areas, which underlines the contribution of these grants to Konbit objectives. Overall, the Konbit systems and procedures for CD grants appear to be functioning slowly, but smoothly and in compliance with USAID rules and the Grants Manual. The challenge for Konbit is to expedite the prerequisite OCA and grants processing while continuing to ensure quality control and compliance. With respect to innovation grants, the competitive selection process appears to have been fair and transparent. Konbit has funded two grants of approximately $150,000 each for one year, and a third grant is pending. A high attrition rate due to technical errors in concept notes and proposals resulted in few proposals reaching the final review stage, and no proposals were funded from the second solicitation round. Konbit staff explained that stringent application of the rules was intended as a CD measure because international donors are often strict on technical requirements. How innovation grants contribute to project objectives is not explicitly stated in contract documentation, and various people at Konbit and USAID understood the purpose of these grants in different ways. Given the insubstantial connection to USAID priority sectors and funding opportunities, it is unclear how these grants will have a tangible impact in identifying potential new partners or fundable ideas for USAID. EQ 4: TO WHAT EXTENT AND IN WHAT WAYS ARE PROJECT ACTIVITIES AFFECTING GENDER EQUALITY AND WOMEN’S EMPOWERMENT? The Konbit Cooperative Agreement indicates that gender equality and female empowerment should be integrated in project goals and activities to the greatest degree possible, though no specific targets or objectives were defined. Various factors compromised the project’s attention to gender, according to Konbit staff, largely related to insufficient staffing dedicated to the subject. Konbit staff and documents indicate the project has had no real strategy for gender integration. Both innovation grants funded to date included a brief gender strategy in their proposals and have a substantial emphasis on women’s empowerment. However, at least five Konbit implementer staff agreed that the project had rarely integrated gender into its activities. All interviewed targeted organizations indicated that they had either not participated in an activity on gender equality or had participated in only one activity, and the majority had received no support with gender integration in their organizations. Male and female participation in training activities is balanced, but Konbit is tracking no other gender￾related data. Indicators and the Konbit M&E database do not disaggregate by sex because they are tracking on an organization level. At the time of data collection, the part-time advisor was developing a gender work plan and a gender￾focused checklist to complement the OCA tool. This is promising, but the consultant’s level of effort is very limited and this initiative is coming late in the project. Overall, there is no evidence of measurable results by Konbit related to gender equality or women’s empowerment at this point, though two innovation grants may have some positive impact. v| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV RECOMMENDATIONS FOR USAID 1. USAID should consider relocating the Local Solutions team/Konbit management to either the Program Office or Front Office, or otherwise provide higher-level leadership and strategic guidance for this initiative. 2. USAID should identify which organizations and sectors are the priority for future local awards and subawards, to the extent possible, so Konbit can achieve better synergy with Mission-wide strategies. 3. USAID should improve communications and coordination among Konbit, technical offices, OFM and awardees, via a systematic process for sharing information and planning capacity support. FOR KONBIT 1. Konbit should agree with USAID on a targeting strategy for all future Konbit assistance, prioritizing current and anticipated recipients of USAID resources for Konbit training and TA. 2. Konbit should re-engage with and expand the group of CD service provider organizations, providing and promoting opportunities to skilled Haitian professionals to deliver paid services. Transferring expertise to local service providers should now be the focus of international partners. 3. Konbit should prioritize one-on-one (or small group) hands-on technical assistance to strengthen capacity, tailored to tackle targeted organizations’ priorities. 4. Konbit should develop more efficient, frequent, and accurate tools for monitoring progress on CD, including close monitoring of the full scope of CAP implementation. 5. Konbit should focus more on helping selected organizations achieve greater financial sustainability and funding diversification. 6. If resources are limited, Konbit should favor CD grants due to their direct contribution to USAID priorities and the significant unmet needs and take steps to expedite grants processing. 7. USAID and Konbit should clarify or modify the purpose of the innovation grants component before issuing more grants of that type. 8. Konbit should rework the DIN web platform to optimize usability, including migration to a new host site, if necessary. 9. Konbit should focus future activities related to gender integration and equality on a select group of awardees and core CDSPs. 1| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV I. INTRODUCTION The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) project design and conduct an independent mid-term evaluation of the Capacity Development (CD) Mechanism, known as Konbit. Prior to conducting the fieldwork for this evaluation, the evaluation team (ET) presented its initial findings based on a document review in an inception report. The ET, whose members are described in Annex B: Evaluation Team Members, then presented its evaluation design, a detailed report approved by USAID/Haiti prior to starting fieldwork. The purpose of this report is to present the main evaluation findings, conclusions, and recommendations. ACTIVITY BACKGROUND Through its USAID Forward initiative and Local Solutions Framework, USAID prioritizes the strengthening of local actors—and the systems in which they operate—as a critical tool in implementing more efficient and effective funding with more sustainable and impactful development results. USAID has found that many Haitian organizations lack the technical and organizational capacity to effectively and responsibly manage United States Government resources. Thus, USAID/Haiti recognizes the need for a more intensive and expansive CD approach when partnering directly with local organizations. In October 2015, USAID awarded the Konbit contract to Haitian firm Papyrus S.A. and its three sub￾partners—Ayiti Nexus, Steve Kroll and Associates (SKA), and Tetra Tech—with an original ceiling price of $4,650,000 (increased to $7,495,166 in 2016). The Mission and its Local Solutions team envision Konbit as a means to reinforce Haitian Capacity Development Service Providers (CDSPs), who, in turn, will strengthen local institutions in various areas of internal capacity. THEORY OF CHANGE The Konbit Theory of Change (Annex C) posits that by providing comprehensive CD services using local CDSPs, establishing an Innovation Network, and setting up a monitoring and evaluation (M&E) platform, partner organizations will improve their capacity, and local organizations will share and adopt lessons learned, be exposed to innovation, and work collaboratively. These inputs, outputs, and outcomes will ultimately achieve Konbit’s goal of increasing the number of development partners influencing and achieving significant and sustainable development results in Haiti, who are accountable to their constituents, and can effectively compete for and manage resources. In turn, that will contribute to a stable and economically viable Haiti. The results framework in Annex D illustrates the links among the sub-results, intermediate results (IR) areas and Konbit’s overall goal. IMPLEMENTERS The Konbit Activity is led by Papyrus S.A. in conjunction with one Haitian partner and two companies based in the United States of America (USA). Consortium members’ expected roles are set out below. Table 1: Konbit Consortium Roles CORE MANAGEMENT TEAM Papyrus S.A. (prime) • Provides overall leadership, responsible for direct assistance to USAID local awardees, coordinates the Development Innovation Network (DIN), and manages small grants. Ayiti Nexus (Sub) • Responsible for M&E, gender, communications, technical assistance (TA), training, and sub-office logistics in Cap-Haitian. 2| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV • CORE SUPPORT TEAM SKA (Sub years 1-3) • Provides individualized and group TA and training on contract mechanisms, compliance, finance & cost accounting, and USAID regulations to CDSPs. Tetra Tech (Sub years 1- 3) • Individualized and group CD to CDSPs via Organizational Capacity Assessments (OCAs), TA, training, advisory services, and related tools. COMPONENTS The five-year Konbit Activity, having just finished its third year of implementation, was designed to offer a range of approaches and activities to establish a “Capacity Development Hub” by pursuing the following four results areas: 1) set up and coordinate the provision of local CD services; 2) create a DIN; 3) develop and implement monitoring, evaluation, and learning (MEL) activities that accurately gauge project progress and promote improved results; and 4) provide funding to reinforce service providers’ capacities and to support innovative activities. IR 1: ESTABLISH AND COORDINATE COMPREHENSIVE CD SERVICES Konbit aims to build local organizational capacity by: a) providing assistance to up to 37 USAID/Haiti local awardees to enable them to meet the terms of their awards and deliver intended development results; b) strengthening the proficiencies of about 100 CDSPs, engaging some to respond to USAID local awardee training needs, and upgrading six to ten to potentially serve as USAID programming implementers; and c) supporting Papyrus and Ayiti Nexus to hone their respective capabilities to implement major donor awards. A group of ten “Core CDSPs” was pre-selected in the Papyrus proposal as key beneficiaries; one was already a USAID awardee (Group Croissance), one later became an awardee (Ayiti Nexus), and one more awardee, Centre de Formation et d’Encadrement Technique (CFET), joined as a Core CDSP in the first quarter, bringing the total to 11.1 The proposal and subsequent documentation do not identify the remaining CDSPs that make up the target of 100. Five of 11 Core CDSPs opted for Konbit’s “Management Track” to prepare themselves for future USAID funding. These have engaged much more than other CDSPs; four have shown the highest overall participation in Konbit training.2 Six Core CDSPs chose the “Service Track,” as they were most interested in responding to the CD needs of other organizations in collaboration with Konbit. This report will use the term “targeted organizations” to refer collectively to the USAID local awardees and Core CDSPs. Annex E provides a full listing of the awardees and Core CDSPs primarily targeted by Konbit to date. The following are Konbit’s principal types of CD interventions to date: OCAs are in-depth self-assessments covering seven categories of organizational capacity, which are intended not only to measure capacity, but also build capacity as the leaders and key stakeholders in the organization work with a trained facilitator to analyze their own strengths and weaknesses. More detail on the OCA tool is presented on page 10. Group training workshops have addressed a range of topics including: Writing a Grant Proposal and Budget, USAID Rules and Regulations for Contracts, M&E, Financial Management, and Internal Control Systems. 1 In Year 3, one more organization joined Konbit as a Core CDSP on the Management Track: Fondation Haitienne de Diabète et de Maladies Cardio-Vasculaires (FHADIMAC). 2 Centre d’Entrepreneurship et de Leadership en Haiti (CEDEL), Cabinet d’Etudes de Gestion, d’Economie et de Comptabilité (CEGEC), CFET and Group Croissance have each attended more than ten training events. 3| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Technical assistance has been provided to selected organizations to address internal challenges that require a tailored or intensive intervention. Areas of intensive coaching or TA have included M&E plan development and contract negotiation, among others. Experts Konbit is a service that sets up one-on-one coaching and advice sessions with Konbit staff and/or other qualified individuals for any interested organization, via the project website. Advice topics have ranged from System for Award Management and Data Universal Numbering System registration and reactivation, to M&E assistance, coaching on OCAs, and project reporting. Helpline is offered by Konbit’s international partners Tetra Tech and SKA. They provide high-level one￾on-one (usually remote) consultative services on an as-needed basis to Management Track CDSPs only. IR 2: LAUNCH A DEVELOPMENT INNOVATION NETWORK Konbit organized and supported networking events for a wide range of local and international stakeholders and established a dedicated interactive website, to encourage collaboration and sharing of best practices, innovations, and industry standards in both CD and technical areas. Konbit launched the DIN in Year 1 and has organized 60 diverse networking events of varying sizes and audiences. In June 2017, Konbit created a web-based platform to serve as the DIN virtual hub (name later changed to Konbit pour la Compétitivité Stratégique [KCS]). The platform is a microsite hosted by the Global Innovation Exchange, which uses two web addresses: http://kcshaiti.ht and http://konbit.ht. An entity called the KCS Association emerged as a group of service providers designed to ensure sustainability of Konbit’s tenets. Largely consisting of Core CDSPs, it plans to manage this platform after Konbit ends. IR 3: DEVELOP AND IMPLEMENT MEL ACTIVITIES Konbit intended to measure its progress and achievements, utilize its M&E processes and results to create learning opportunities for program participants, and employ an interactive platform for sharing key lessons learned. Konbit began by monitoring nine indicators in Year 1, then added two new outcome indicators in Year 2 in agreement with USAID. In September 2017, Konbit contracted an internal evaluation by The Kaizen Company to inform critical reflection and maximize impact in the Activity’s second half. Beyond the bounds of Activity M&E, Konbit continues to encourage local partners to document their lessons learned through a template developed in Year 1. In Year 2, Konbit started implementing targeted CD activities to strengthen the M&E systems of four targeted organizations. To address cross￾cutting M&E issues and better manage performance, Konbit developed an M&E training curriculum tailored to the needs of USAID awardees and other targeted organizations. IR 4: IMPLEMENT SMALL GRANTS PROGRAM Konbit launched the Small Grants Program (SGP) in 2016 after modification of the Cooperative Agreement, which anticipated grants as a possible add-on activity. This component aims to engage local organizations through fixed-amount awards of two kinds: 1) non-competitive CD grants dedicated to funding TA for awardees and Management Track CDSPs to help fulfill OCA-identified priorities; 2) competitively awarded innovation grants for pilot projects that propose innovative solutions to development challenges in Haiti. The modification allocated up to $1,645,000 for up to 47 CD grants and $1,200,000 for an unspecified number of innovation grants. As of October 2018, the grants status is as follows: CD Grants. Seven awardees and two Management Track CDSPs have received grants ranging from $7,000 to $80,000 each, totaling just under $450,000. Two grants have been closed. 4| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Innovation Grants. Based on the first Request for Applications (RFA), Konbit funded two grants for a combined value of $292,000, while one more grant will be funded shortly. The second RFA did not generate any proposals meeting the minimum score requirements. EVALUATION PURPOSE AND AUDIENCE The purpose of this mid-term performance evaluation is to inform USAID/Haiti regarding possible mid￾course corrections and Konbit’s future program orientation. Lessons learned and recommendations provided through the evaluation will also guide future programming in areas such as organizational governance, leadership, financial management, project management, human resources, M&E, and other aspects of organizational development. Additional details can be found in the evaluation SOW (Annex A: Evaluation Statement of Work). The evaluation’s primary audience is USAID/Haiti. Secondary audiences include Papyrus and its sub-partners as Konbit implementers, and other counterparts to be identified by USAID/Haiti. EVALUATION QUESTIONS This report answers the evaluation questions (EQs) below. EQ1. To what extent have local USAID/Haiti awardees working with Konbit acquired and implemented systems and procedures to effectively and responsibly manage USAID funds? In answering this question, the ET assessed: a) where firms are at in the process, noting significant successes and failures; b) what remains to be done so that the systems and procedures are in place to ensure sustainability of new/improved systems and procedures; and c) any potential obstacles to organizations’ ability to independently and appropriately manage USAID funds. The ET particularly examined M&E and Learning, Finance, and Management, focusing exclusively on awardees that have worked with Konbit for the scope of this question, based on a list of 23 relevant awardees selected by USAID/Haiti. EQ2. How are organizations applying the knowledge and tools gained through Konbit (such as OCA, Helpline, DIN, and M&E)? For this question, the ET studied a wider set of organizations including but not limited to USAID awardees and CDSPs. Analysis included numbers of users, frequency and nature of utilization, perceptions of utility, and, where available, evidence of application of learning/tools. EQ3. To what extent are systems and procedures to implement the small grants program in place and sufficient to ensure effectiveness? In responding to this question, The ET examined the systems and procedures developed and used by Konbit to solicit, award, and oversee CD and innovation grants. This analysis considered the adequacy of those systems and procedures for promoting compliance and contribution of grants to the Konbit goal and objectives. EQ4. To what extent and in what ways are project activities affecting gender equality and women’s empowerment? To answer this question, the ET focused on identifying Konbit’s strategies and efforts to promote gender equality and women’s empowerment, primarily looking at targeting of activities, content of tools, training and other major interventions, beneficiary tracking, and reporting by Konbit. 5| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV II. EVALUATION METHODOLOGY SAMPLING The ET collected data on the perspectives and experiences of key stakeholders involved in the Konbit Activity using a purposive sampling technique. This approach enabled the team to prioritize interviewees with the most information, balanced by interviewee diversity for larger populations. The ET identified stakeholders involved in Activity oversight and implementation as well as those receiving Konbit services or who were otherwise cognizant of its programming. Annex F: List of Persons Interviewed contains a list of interviewed persons. The main stakeholder categories were the following: 1. USAID Konbit Contracting Officer’s Representatives (CORs), Agreement Officer’s Representatives (AORs) managing selected awardees, and representatives of the Democracy and Governance (DG) Office, Office of Financial Management (OFM), Policy Coordination and Program Support Office, and senior Mission management (Front Office). 2. Management personnel of prime and subcontracted Konbit implementers. 3. Konbit Activity full-time personnel and consultants. 4. USAID awardee representatives. 5. Core CDSP representatives. 6. Innovation grant applicants, recipients, and review committee members. 7. Other donors and USAID-funded implementers. Purposive sampling was supplemented by snowball sampling to include additional informants as they were identified during data collection. This was especially pertinent for USAID personnel, as consultations with the Mission revealed that diverse offices and individuals had valuable viewpoints related to Konbit due to its cross-cutting nature. The final sampling enabled the ET to compare different key stakeholder groups’ perspectives and to cross-check and supplement Konbit reporting data. DATA COLLECTION METHODS The ET used a qualitative evaluation design, consisting of document review, analysis of Konbit M&E data, key informant interviews (KIIs) and group interviews (GIs). The evaluation design matrix in Annex G: Evaluation Design Matrix shows how the ET used each source and method to answer the EQs. The ET reviewed Activity documents and produced an inception report in August 2018. The team conducted interviews and preliminary data analysis during fieldwork from September 24 to October 22, 2018. DOCUMENT REVIEW The ET reviewed Konbit and USAID documents to understand the project design and implementation, extract findings relevant to the EQs, and inform data collection protocol development. The main documents and categories of documents reviewed are listed below, while Annex H: Documents Reviewed contains a detailed listing. • Konbit annual and quarterly reports, and annual work plans. • Konbit M&E plans and databases. • Cooperative Agreement and modifications. • Konbit training materials and OCA tools, checklists, and reports. • Konbit Grants manual, RFAs, and grant-related documentation. • Konbit internal mid-term evaluation report. • USAID Guidance on Fixed Amount Awards. 6| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV • USAID Gender Equality and Female Empowerment Policy. • USAID/Haiti Strategic Framework 2018-2020. KEY INFORMANT INTERVIEWS The ET conducted interviews with a total of 67 key stakeholders, either in person in Port-au-Prince or remotely by Skype or telephone. Table 2 below outlines the number of KIIs and GIs by stakeholder group and Annex F: List of Persons Interviewed presents a full list of interviewees. The ET prepared interview protocols tailored to each respondent type, which they piloted and then adjusted during the first week of data collection. Annex I: Data Collection Protocols contains the final versions of all protocols. The ET conducted all KIIs and GIs in French or English, using notetakers to assist in capturing data whenever possible. Table 2: Number of Interview Participants by Stakeholder Group STAKEHOLDER GROUP PARTICIPANTS USAID (Konbit AORs, Front office, DG Office, OFM, 6 AORs) 14 Implementers/Konbit team3 15 USAID awardees (7 KIIs, 2 GIs covering 18 awardees in total) 20 Core CDSPs 7 Innovation grant stakeholders 8 Other donors/international programs 3 Total KII and GI participants Male participants Female participants 67 37 30 ANALYSIS ET members took detailed notes of all KIIs and GIs, then cleaned and shared electronic summaries on a rolling basis throughout fieldwork. Team members conducted internal debriefs regularly throughout fieldwork to discuss progress, review evidence collected to help answer the EQs, and identify any data gaps or discrepancies. The ET then analyzed all interview data using an Excel-based matrix organized by EQ. The ET used this comprehensive matrix while reviewing interview data to articulate important themes captured by interview notes (e.g. “Financial systems set up or enhanced”, or “Some awardees were not interested in Konbit or capacity development”). The matrix was used to identify which respondents made statements consistent with each theme, then to tally the numbers of informants supporting each theme. This process enabled the ET to spot themes repeated by multiple respondents, which contributed to analysis and development of findings. The ET then employed triangulation approaches when consistent statements existed across multiple respondents to aid in developing the findings and conclusions. The matrix allowed the ET to identify which interviewed individuals or groups made statements consistent with a certain theme, to consider key characteristics of those respondents that might affect their responses (e.g. employer, role in relation to the project), and to assess the extent to which the repeated theme cut across different respondents and different categories of respondent. Where marked differences in opinion emerged across 3 Three of the individuals included in this figure were leaders or consultants of Ayiti Nexus, which was a subcontracted partner of Konbit, as well as an awardee and Core CDSP. 7| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV respondents or categories, the ET conducted further analysis of possible reasons for those differences, in particular considering the respondents’ degree and type of involvement with Konbit. The ET also compared themes from the matrix with reviewed documents containing information on the same themes, to identify whether there was consistency and thus corroboration between interview data and documentary data, or whether there were differences that needed to be further analyzed before making findings and conclusions. Triangulation enabled the ET to gain a deeper understanding of the perspectives of diverse respondents and to cross-check findings emerging from different data collection methods and sources to identify similarities and differences. The matrix also included metadata such as respondent type and interview method (GI or KII). This enabled the ET to look for trends within and across sub-groups, including trends by sex. The matrix: a) ensured a systematic and thorough response to each EQ; b) considered gender and social dimensions; c) identified gaps where additional clarification or analysis was necessary; and d) served as a basis for developing the evaluation report. The ET employed several specific data analysis methods to identify key findings from the collected data, as well as to draw conclusions and develop recommendations for the remaining period of the Activity: 1. Content Analysis: The ET reviewed and analyzed all KII and GI data to identify and highlight notable examples of success (or lack of success) that contributed to (or inhibited) achievement of Konbit objectives. 2. Trend Analysis: The ET examined Konbit M&E data over time to identify trends in performance, insofar as they were relevant to the EQs. 3. Gap Analysis: The ET examined which aspects of Konbit have fallen short of anticipated performance, and the likely factors contributing to these gaps. 4. Comparative Analysis: The ET compared different stakeholder groups perspectives to assess either convergence or divergence. The ET identified findings that arose from interviews and developed a matrix that cross-checked findings by respondent type, interview type, and sex. 5. Gender Analysis: The ET disaggregated all data collected through KIIs and GIs by sex and analyzed these data for comparative effects on both male and female beneficiaries to illustrate any significant differences. LIMITATIONS AND MITIGATION STRATEGIES BIAS The ET recognized several bias risks in data collection and analysis and developed mitigation strategies to overcome these: 1. Recall Bias: Recall bias is a common challenge in evaluations, especially if informants are no longer actively engaged in the Activity. Konbit recipients could fail to remember accurately or could respond to questions with answers that blended their experiences into a composite memory if, for example, they had received CD support from multiple sources. The ET probed respondents’ answers to verify recollections, seeking specific examples where relevant, and cross-checked with data from other informants and documentary evidence when possible. 2. Response Bias: Response bias is the risk that key informants may be motivated to provide responses considered socially desirable or influential in obtaining donor support. For example, an awardee may provide positive remarks about Konbit because they would like their organization to receive support in the future. The ET mitigated this risk by explaining the objectives and potential benefits associated with the evaluation, and assuring informants that it would keep interview content 8| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV confidential and omit personally identifiable information from the report (see informed consent protocol in Annex I: Data Collection Protocols). Additionally, the ET used the matrix-based triangulation processes described above to support analysis and comparison of responses from different informants, considering their respective involvement with Konbit when analyzing their responses. 3. Selection Bias: Selection bias is an inherent risk when implementers help to identify or facilitate contact with Activity recipients. There was a risk that the ET would be oriented to stakeholders with primarily positive views of the Activity. To mitigate this risk, the ET identified interviewees using lists provided by Konbit and USAID, used purposive sampling methods, and arranged meetings directly without assistance from the implementers. The ET made specific efforts to ensure a balanced and inclusive selection process, for example by inviting all Konbit awardee organizations to either individual or group interviews, rather than including only a sample of awardees, and by interviewing a wide cross-section of USAID/Haiti current and former staff. 4. Gender Bias: Gender bias is a risk because most individuals have a subconscious sense of appropriate roles and behavior for women and men. The ET reviewed gender-sensitive approaches during the inception period by taking USAID’s Gender 101 training, discussing possible gender preconceptions that could affect this evaluation, and reviewing how to minimize these during data collection and analysis. OTHER LIMITATIONS AND MITIGATION STRATEGIES The safety and security situation in Haiti during fieldwork was affected by several instances of political unrest in the capital, some of which were violent. The ET abided by SI’s safety and security protocols, received frequent updates on the security situation from the SI team in Haiti, and was ultimately able to avoid being affected by the unrest. Konbit monitoring systems capture limited information about capacity change and application of learning at an organizational level; project indicators do not address this issue other than by counting how many organizations show an increase in OCA scores. As described further in EQ 2: How are organizations applying the knowledge and tools (such as OCA, Helpline, DIN, and M&E) gained through Konbit? Findings, the OCA has proven problematic in terms of measuring change, which is a constraint on the utility of such indicators. These shortcomings have limited the utility of Konbit monitoring data in answering both EQ 1 and EQ 2, necessitating a focus on utilization and perceptions of utility. The ET has made use of OCA scores despite their limitations, paying careful attention to the need for triangulation with other sources and methods to support findings and conclusions. Although the ET planned three focus group discussions, they were cancelled due to poor response from invitees. Instead, the ET conducted interviews with two participants who showed up for the first discussion. Although limited direct observation was planned, it was not feasible due to scheduling constraints. Several stakeholders alluded to perceived conflicts of interest within Papyrus, USAID and other USAID￾funded programming, related to personal/professional connections and key personnel interests. Such conflicts may have influenced some informants’ responses or their willingness to be interviewed. When aware of potential conflicts of interest, the ET considered them when interpreting responses and emphasized triangulation of data from those interviews. III. FINDINGS AND CONCLUSIONS This section synthesizes the evaluation’s main findings and conclusions, which are presented by EQ. 9| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ 1: TO WHAT EXTENT HAVE LOCAL USAID/HAITI AWARDEES WORKING WITH KONBIT ACQUIRED AND IMPLEMENTED SYSTEMS AND PROCEDURES TO EFFECTIVELY AND RESPONSIBLY MANAGE USAID FUNDS? Konbit has supported the CD of 24 USAID/Haiti awardees to date, according to a USAID list. The ET agreed with USAID that Papyrus as prime implementer of Konbit should be excluded, leaving 23 direct awardees as the focus of this question. Annex E provides a full listing of those awardees. The ET paid particular attention to awardee capacity, systems, and procedures related to financial management, project management, and MEL, as instructed by the evaluation SOW. FINDINGS Background on Awardees’ Participation Low engagement by approximately half of the awardees The response to this question requires a preliminary discussion of the extent to which awardees have been involved in or exposed to Konbit to date. Project records (corroborated by interviews) show that only about half of the 23 awardees examined have regularly engaged in Konbit activities. Konbit and USAID staff explained that participation in Konbit was entirely optional for awardees, while document review indicated some awards had short periods of overlap with Konbit’s implementation period and 15 awardees were receiving CD support from other USAID contractors. At least four were sub-recipients of prime implementers or awardees rather than direct awardees themselves.4 Because 12 organizations on the list (refer to Table 3 below for details) have not engaged in any significant way with Konbit, the ET can make little comment on their working with Konbit on systems and procedures to effectively manage USAID funding. Workloads and strategic shifts affected participation levels USAID and Konbit personnel repeatedly observed that Konbit award timing was disadvantageous as many new local awards were getting underway simultaneously, meaning awardees and technical offices were busy with those awards’ startup processes—often their first experience with USAID direct funding. Informants from awardees, Konbit, and USAID indicated that this contributed to many awardees giving a low priority to the CD support on offer from Konbit, although there were exceptions, such as Fondation Haïtienne pour le Développement de l’Agriculture Durable (FONHDAD) and Fondasyon Kole Zepòl (FONKOZE), which actively engaged with Konbit from the outset. “Based on what I have seen, I don’t know whether these two entities (the awardee and Konbit) knew that there was supposed to be collaboration between them.” - USAID AOR The majority of USAID and Konbit personnel reported that certain AORs with award oversight responsibilities made minimal effort to link awardees with Konbit or otherwise encourage their awardees’ organizational CD. In fact, some AORs mockingly refer to Local Solutions as “Local Problems” due to the heavy workload and perceived higher risk involved in managing these relatively small awards. The level of enthusiasm for and promotion of Local Solutions within USAID globally and in Haiti has declined since 2015, according to the majority of interviewed USAID staff, who described this as contributing to some AORs’ low interest level in Konbit and CD of local implementers. 4 Association Nationale des Caisses Populaires Haïtiennes (ANACAPH), Association Nationale des Institutions de Microfinance d'Haïti (ANIMH), Femmes En Action Contre La Stigmatisation Et La Discrimination Sexuelle (FACSDIS), and Komisyon Fanm Viktim pou Viktim (KOFAVIV). 10| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV USAID designed Konbit, in large part, to support local awardees funded via Annual Program Statements (APS), which USAID expected to generate up to 37 awards across multiple sectors. However, the mission’s discontinuation of APS in 2016 severely limited the flow of new awardees that Konbit could target, and according to Konbit staff, also undermined existing awardees’ interest in developing their capacity as they perceived a diminished prospect of future USAID funding. Communications gaps weakened engagement of awardees Interviewees observed that communications between USAID technical office staff and Konbit were deficient in both directions. As one example, an AOR was not aware until the ET interviewed him that the awardee he was managing had received a CD grant from Konbit five months previously. Interviewed AORs consistently stated that their focus was on technical delivery, with one mentioning that “capacity development is not my jurisdiction”, so they paid little attention to communications from or about Konbit. With one exception, interviewed AORs were not involved in identifying awardees’ CD needs nor were they regularly informed of what support Konbit was providing. AORs and other USAID staff reported that they primarily received information about Konbit via occasional presentations organized at the Mission, although Konbit staff described a weekly highlights bulletin that was sent to certain USAID personnel, and Konbit copied relevant AORs on awardee OCA reports shared with the Mission. “Synergy is lacking between USAID and Konbit.” - Awardee representative Konbit and USAID personnel indicated that there was virtually no direct contact or information sharing between Konbit and OFM, although they were both aware, in a general sense, that some awardees were being assessed and/or supported by both Konbit and OFM. Records from OFM show that 13 awardees on the evaluation list were subject to assessment and provision of financial management support services since Konbit began. (See text box below for information on the assessment tools used by Konbit and USAID.) Informants from both OFM and Konbit expressed the view that it would have been advantageous to have more information about identification of capacity needs and actions taken by each of them respectively. OCA and Other USAID Tools The OCA is a structured tool for a facilitated self-assessment of an organization's capacity followed by action planning for capacity improvements. The self-assessment approach aims to increase ownership, and the format helps the organization reflect on its processes and functions and score itself against benchmarks. Although various adaptations have evolved over time, the OCA tool currently shared on USAID’s Learning Lab website assesses capability in seven capacity areas: 1. Governance and Legal Structure 2. Financial Management and Internal Control Systems 3. Administration and Procurement Systems 4. Human Resource Systems 5. Program Management 6. Project Performance Management (including M&E) 7. Organizational Management and Sustainability Each category has between five and eight sub-categories related to distinct elements of capacity. If not relevant in a certain situation, sub-categories can be omitted from the process. 11| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV USAID’s Non-U.S. Organization Pre-Award Survey (NUPAS), an assessment generally administered by USAID staff as part of the selection process prior to contracting a local organization, has considerable overlap with the OCA, including 19 of 40 sub-categories in the current USAID OCA tool, cutting across six of the seven areas listed above. USAID now publishes versions of the OCA guide that specifically denote the overlapping areas to facilitate complementarity of assessments of the same organization over time. USAID/Haiti also conducts less intensive reviews of potential/current recipients of Fixed Amount Award (FAA) funding, using a “Fixed Amount Award Entity Eligibility Checklist.” Generally led by OFM and focused on financial management, the assessment team examines some aspects of legal structure, human resources management, and administration. Both NUPAS and FAA reviews can lead to special conditions being imposed on funded organizations at different points in time. Current Capacity and Awardees’ Progress Capacity is highly variable among the group of awardees and the degree of capacity change since awards were issued has also been inconsistent, according to interviews with awardees, AORs, OFM, and the Konbit team. Among the 23 awardees, ten completed baseline OCAs under Konbit, and of them, five did at least one follow-up OCA; all five showed some degree of improvement on overall score.5 The increase in score was substantial for two awardees, moderate for two, and incremental for one. Those results can be summarized as follows:6 • Ayiti Nexus has made notable improvements in each category of the OCA since 2016. • FONKOZE scores declined for financial management, while registering notable improvements on program management7 and M&E. Their overall score showed a solid increase since 2015. • CFET baseline scores were high in 2017. They have slightly improved in five of seven categories, though a decline was noted in financial management. • FONHDAD showed modest advances in administration and financial management, but no improvement in program management or M&E. Their overall score increased slightly since 2016. • Group Croissance increased its overall score marginally from the 2017 baseline, a result of incremental improvement in program management. Ayiti Nexus, FONKOZE, and CFET have reached approximately 80 percent of the total possible score on their most recent OCAs, while all other assessed awardees scored below 70 percent.8 Annex J presents OCA scoring highlights (overall scores and three priority subjects of financial management, program management, and M&E) for all awardees as well as Core CDSPs that have undergone at least one OCA with Konbit. 5 OCA data and analysis in this section should be read in the context of the limitations noted above in the Evaluation Methodology section, and the OCA challenges described under EQ 2. 6 The seven OCA categories of capacity are listed above in the text box “OCA and Other USAID Tools”. 7 For purposes of OCA analysis, the ET has selected “program management” as the OCA category that corresponds most closely to project management, which was specified as a key capacity area by the evaluation SOW. 8 Konbit management advised the ET that several recent OCA reports were under review at the time of data collection, and that some scores might be amended as a result. The ET has used the scores and reports most recently provided by Konbit as the basis for analysis. 12| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Table 3 summarizes the data the ET gathered from all sources (including interviews with awardees, USAID staff, Konbit staff and trainers, as well as OCA reports, Konbit participation data, and OFM review data) to depict the extent to which each awardee engaged in Konbit support activities compared to the degree of change in organizational capacity since Konbit began in late 2015. Group Croissance, Ayiti Nexus, CFET are denoted in bold font, indicating they are Core Management Track CDSPs as well as awardees. Table 3: Awardees' Engagement in Konbit vs. Capacity Improvement9 SIGNIFICANT CAPACITY IMPROVEMENT MODERATE CAPACITY IMPROVEMENT MINIMAL OR UNKNOWN CAPACITY IMPROVEMENT HIGH ENGAGEMENT • Ayiti Nexus • FONKOZE • CFET • Group Croissance MODERATE ENGAGEMENT • FPDI • FONHDAD • SHAA • Build Change • IDEH • J/P HRO • SEFIS MINIMAL ENGAGEMENT • AGCRDDM • Agritech • ANACAPH • ANIMH • CPFO/The Edge • Carbon Roots • FACSDIS • FONHEP • GHESKIO • IDEJEN • KOFAVIV • Service Chretien As shown by the table, only three awardees demonstrated significant improvements in capacity during Konbit. Four others moderately improved after engaging at least moderately with Konbit. Considering the evidence from interviews, OCAs, and financial review data from OFM, with a focus on project management, financial management and M&E capacities, six awardees currently appear ready to effectively handle direct USAID funding: FONKOZE, CFET, FPDI, Group Croissance, FONHDAD, and 9 Acronyms used in this table that appear in the report for the first time: AGCRDDM (Association de Gestion du Centre Rural de Développement Durable de Montrouis); CPFO (Centre de Promotion des Femmes Ouvrières); FONHEP (Fondation Haïtienne de l'Enseignement Privé), FPDI (Fondation Paradis des Indiens); GHESKIO (Groupe Haïtien d'Etude du Sarcoma da Kaposi at des Infections Opportunistas); IDEH (Initiative pour un Développement Equitable en Haïti); IDEJEN (Initiative pour le Développement des Jeunes); J/P HRO (Jenkins-Penn Haitian Relief Organization); SEFIS (Société d'Etudes et de Formation en Information Stratégique); SHAA (Société Haïtienne d'Aide aux Aveugles) 13| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANACAPH. Excepting ANACAPH, all have received a high or moderate level of assistance from Konbit, though no attribution to Konbit support is possible. However, the data from OFM and other sources show that even those six have capacity gaps that might necessitate special conditions on awards and/or additional support to ensure consistent compliance with USAID requirements, especially regarding financial management. Successes with Awardees Konbit reports describe the establishment by various awardees of new or improved systems and procedures for M&E, human resource management, procurement, and financial management, among others. Table 4 below presents the number of interviewed awardees that reported having benefited from Konbit support to establish new or improved systems and procedures. Advances in M&E systems following Konbit assessments and technical assistance were particularly notable for FPDI and FONKOZE, according to OCAs and USAID staff. Table 4: Number of System and Procedure Improvements Reported by Awardees TYPE OF NEW/IMPROVED SYSTEM OR PROCEDURE NUMBER OF INTERVIEWED AWARDEES REPORTING CHANGE Manuals on procedures 4 Financial systems 5 M&E Plans or systems 3 Other systems/procedures 3 In terms of specific awardees achieving notable progress, project data and staff reported that Ayiti Nexus has taken full advantage of Konbit offerings as core CDSP and implementing partner, and their three OCAs to date showed substantial improvement across the board. They began implementing their first USAID direct award in early 2017, though the ET did not receive any information related to a NUPAS or FAA review by USAID. FONKOZE is also better able to meet USAID requirements relative to three years ago, according to USAID informants and reports as well as their OCA scores. Interviews with John Snow International (JSI) and Konbit indicated that complementary support by both these implementers as well as strong leadership commitment were key factors in improving capacity, which was confirmed by FONKOZE staff. Simultaneous support by USAID’s OFM and Konbit to different functions of awardee FPDI, under the guidance of a highly engaged and supportive AOR, succeeded in boosting that organization’s capacity in M&E and financial management, and contributed to new donor funding, according to representatives of the awardee, USAID, and Konbit. A smooth collaborative relationship exists between Konbit and Development Alternatives, Inc (DAI), implementer since 2017 of the “Finance Inclusive” project; this was indicated by staff of both implementers. Two awardees (ANACAPH and ANIMH) are project partners, referred by DAI to Konbit for OCAs on USAID’s recommendation. Afterwards, DAI and Konbit agreed on how each would support further CD, though that division was not explained in the Capacity Action Plans (CAPs) on file for either organization. DAI recently referred another local partner to Konbit to replicate that successful collaboration. Konbit M&E data show that none of these three entities has so far regularly engaged in Konbit training and TA offerings (perhaps because DAI is meeting those needs), but ANIMH is presently in the pipeline for a CD grant. 14| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Failures and Challenges Working with Awardees The challenge that Konbit and USAID staff highlighted most frequently was the minimal engagement with Konbit of 12 of 23 awardees included in this evaluation, discussed at the beginning of this EQ. Factors cited by informants for the low engagement levels were AORs’ lack of interest (13 mentions); awardees’ lack of interest (ten mentions); and awardees targeted with CD by another implementer (seven mentions). In at least four cases (GHESKIO, J/P HRO, SEFIS, and Service Chretien), informants said that participation in Konbit was affected by the fact that JSI, implementer of a major health sector project since 2014, was already working with those local organizations with a significant focus on CD, including use of OCAs. Misunderstandings and poor information exchange with JSI during Year 1 contributed to lack of interest in Konbit’s assistance among those awardees and their AORs. Since JSI ended its CD component one year ago, Konbit has recorded an uptick in engagement by these organizations. “Konbit was intended to be the only CD provider of USAID Awardees and CDSPs. There was misunderstanding, even in the mission… and resistance among some implementers.” - USAID staff member For several awardees, low leadership engagement and/or weak governance structures constrained improvement and sustainability of their capacity, according to 11 interviews with implementing partners and USAID. Informants described how some awardee staff sent to training and other activities were not able to make decisions or influence changes in their organizations, and how leaders did not prioritize certain CAP items, which undermined efforts to move forward on those issues. Three interviewed Konbit implementer staff favored scaling back support to certain organizations that consistently failed to integrate the considerable assistance provided, but Konbit senior management expressed reluctance to do so. In the same vein, the internal midterm evaluation report suggested a targeting approach that would focus on organizations that would maximize the return on investment, rather than investing further in organizations that had been unable to absorb Konbit assistance. Sustainability of Systems and Procedures The governance and leadership issues mentioned above as challenges to CD are also a major factor casting doubt on the sustainability of new and improved systems and procedures, according to a majority of USAID and Konbit informants. These informants repeatedly expressed concern about the “one-man show” phenomenon and lack of succession planning which affect some of the core group targeted with most Konbit support. Although OCA reports and training records show that Konbit has recognized and tried to ameliorate these issues, interviews with trainers and other implementer staff described limited ability to make progress except in isolated cases. After governance, the most commonly mentioned constraint on sustainability of systems and procedures was awardees’ financial instability. Those without a diversified funding base, especially non￾profit awardees, are vulnerable to severe funding fluctuations that can cause them to cut back staffing and operations drastically. As Konbit and USAID informants described, these situations result in reduced organizational capacity because although improved systems and new manuals may still theoretically exist, in practical terms they have fallen into disuse. Informants pointed out that for-profit awardees tend to have a more stable/diversified funding base, and thus are more likely to sustain capacity improvements and systems in the absence of USAID/donor funding. “Systems will be sustainable if the flow of funds continues… it is great to put books in order, but manuals and procedures do not help if there is no staff or activity.” - Konbit staff member 15| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Organizations Konbit assessed had an average baseline score of 2 (basic capacity) for the sub-category of New Opportunity Development for Sustainability, which is the most closely related to financial sustainability in the OCA tool used by the project. Only one awardee, FONKOZE, recorded an increased OCA score in this area. Although OCA report narratives and CAPs have paid some attention to this issue, interventions documented by Konbit and targeted organizations have not yet emphasized financial sustainability and resource mobilization strategies. Potential Obstacles to Handling USAID Funding Given that 12 awardees eligible for Konbit support did not actively engage with the project for a variety of reasons, the capacity of those organizations may not have grown since receiving their awards unless they received support from other sources. Konbit was not able to provide baseline or current data on their capacity, and OFM data on most awardees were gathered at least two years ago, rendering it of limited use for this analysis. As previously mentioned, weak governance and financial instability pose significant obstacles to past and present USAID awardees’ ability to effectively handle USAID funding. Another challenge highlighted by informants was the attitude of USAID/Haiti staff, both American and Haitian, towards Local Solutions. As described above, some technical office staff have shown reluctance to provide the additional guidance and monitoring often needed by local awardees to achieve compliance and show a lack of interest in CD, as they tend to focus on ensuring that project results are delivered in a satisfactory way. This conveyed to awardees that organizational strengthening was a low priority for USAID. In addition, Konbit and USAID staff reported that some awardees’ attitudes have been problematic; some do not feel they need any CD, especially if they already have USAID funding and believe they will be funded again without improving capacity. In interviews, awardees tended to rate their own capacity in project management, financial management, and M&E as higher than what is perceived by their AORs and what was captured in their OCAs. This over-confidence in some cases has led to leadership not taking seriously certain weaknesses, which could ultimately compromise the organization’s performance. CONCLUSIONS Engagement with Konbit has been highly variable across the 23 USAID awardees, which has naturally limited Konbit’s ability to identify and work on capacity gaps and help improve systems and procedures of the organizations that were not substantively involved. The minimal uptake of CD offerings by half of the targeted awardees could hamper their ability to successfully execute current and future awards. Factors undermining participation include low levels of understanding, commitment, and interest among awardees, AORs, and other technical office staff (with some notable exceptions), uncertainty about future USAID funding flows, and lack of financial resources for CD. Weak coordination between Konbit and other USAID implementers working with awardees was another early constraint, but more recent experience shows that open information sharing and clear division of roles can lead to positive results. Current capacity levels in the three areas prioritized for this EQ (financial management, project management, and M&E) are highly variable, based on the limited available evidence at hand. Where available, OCAs indicate some progress after the initial assessments and relatively high overall scores for three awardees, while also pointing to significant remaining capacity gaps. Seven awardees have made at least moderate improvements after participating in Konbit interventions. While six awardees presently appear able to effectively manage USAID funding, that ability may not be sustainable and they may still need support to meet all contractual requirements. For most awardees, no recent measurement is available from either Konbit or OFM. In fact, valuable information on awardee capacity strengths and weaknesses has been obtained and held by the OFM, AORs, and Konbit (and other USAID implementers in some cases), with no system for information 16| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV sharing, consultation or coordination of activities aimed at building capacity, which could have increased synergies and boosted time and resource savings. Konbit has made a major investment in OCA as “the tool” for capacity needs analysis, which has been mostly effective. However, continued reliance on OCA is subject to widespread concerns about its efficacy as a means of monitoring capacity change, the time and effort required to perform OCAs, variations depending on the facilitator, and its adaptability to different types of organization. “There have to be incentives for organizations to keep procedures in use and maintain capacity after Konbit, or they will go back to a zero starting point.” - USAID staff member Financial sustainability directly affects sustainability of organizational systems and procedures, which may be lost or severely diminished with a decline in funding and may not be easily recoverable with a new influx of resources. This is a major risk factor for awardees, especially those without their own means of revenue generation or who may not receive renewed USAID funding to continue their work. Weak governance and succession planning are cross-cutting challenges with a significant effect on both effectiveness and sustainability of any improvements in systems and procedures, and, in fact, of the organization itself in some cases. EQ 2: HOW ARE ORGANIZATIONS APPLYING THE KNOWLEDGE AND TOOLS (SUCH AS OCA, HELPLINE, DIN, AND M&E) GAINED THROUGH KONBIT? This EQ considers a wider range of organizations, which includes the 23 awardees, but expands the focus to include the Core CDSPs selected by Konbit, as well as others engaged by key project interventions. In areas where the awardees have already been covered under EQ 1, this section will focus on additional findings related to non-awardee organizations. FINDINGS Limited Data on Application of Learning As noted in the methodology limitations, Konbit has not carried out the kind of close follow-up and monitoring the organizational level that would identify the effects of training and TA on participating organizations, in terms of changes in knowledge, skills, practices, policies or other aspects of internal operations. This makes it difficult for this evaluation to assess what learning has been retained and shared by participants, and to what extent learning or tools are applied by them or their organizations after training. This finding is consistent with the internal mid-term evaluation in October 2017, which responded as follows to a question similar to EQ 2: “Are organizations taking steps to integrate Konbit capacity building outputs into their operations, including M&E support (e.g., manuals, templates, processes), as a result of Konbit? Maybe, based on anecdotal evidence, but these steps and the resulting outputs are not being captured by the Konbit program, creating a missed opportunity. … there does not appear to be any follow-up on how these outputs are actually integrated into the organizations, how they are implemented, and if they increase organizational performance.”10 Utilization and Delivery of Konbit Support Similar to EQ 1, a response to this question demands a review of the extent to which Konbit has generally engaged targeted organizations to date before delving into participation in and application of specific types of support. It is notable that Core CDSP engagement in Konbit has been marked by 10 Mid-term Evaluation of the Konbit Cooperative Agreement: Building Local Capacity in Haiti, The Kaizen Company, October 2017, at page 20 17| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV significant differences; the five “Management Track” CDSPs have engaged far more than other CDSPs. Four of them have shown the highest overall participation in Konbit training, with each attending more than ten events (CEDEL, CEGEC, CFET, and Group Croissance).11 Among the six “Service Track” CDSPs, Institut de Formation du Sud (IFOS) and Experts Conseils et Associés (ECA) were moderately active while three have barely participated in Konbit activities [PAGS Cabinet d’Experts Comptables (PAGS), Centre Universitaire de Management et de Productivité (CMP), and Centre Spécialisé de Leadership (CSL)].12 Konbit staff and the latter three organizations reported that they largely lost interest in the project when the expected opportunities to deliver paid CD services and other expertise to other organizations did not emerge. On the same point, Papyrus publicly posted a lesson learned on the motivation and expectations of targeted organizations, which noted that an initial consultative process could have reduced or avoided misunderstandings around how Core CDSPs could benefit from Konbit. Overall, Konbit M&E data show that a small cluster of about ten Core CDSPs and awardees have participated consistently in most training and TA activities, as well as many roundtables, while other organizations have attended on an insubstantial or sporadic basis. The pattern of engagement compared to evidence of capacity improvement is represented for both awardees and Core CDSPs in Table 5 below, as a complement to Table 2 under EQ 1 (which only included awardees). Table 5: Awardee and Core CDSP Engagement in Konbit vs. Capacity Improvement SIGNIFICANT CAPACITY IMPROVEMENT MODERATE CAPACITY IMPROVEMENT MINIMAL OR UNKNOWN CAPACITY IMPROVEMENT HIGH ENGAGEMENT • Two awardees (One also CDSP) • Two awardees (both also CDSPs) • Two CDSPs (CEDEL, CEGEC) MODERATE ENGAGEMENT • One awardee • Two awardees • Four awardees • Two CDSPs (ECA, IFOS) MINIMAL ENGAGEMENT • Four awardees • Eight awardees • Four original CDSPs CMP, CSL, Fondation Espoir, PAGS) plus FHADIMAC as new CDSP According to project M&E data, Konbit implementers, including international partners SKA and Tetra Tech, have provided most CD services so far; only one of 21 training workshops was delivered by a 11 Fondation Espoir was also listed in the proposal as a Core CDSP and chose the Service Track. They engaged actively with Konbit in 2016, before losing their primary project funding and effectively ceasing to participate. 12 Papyrus Lesson Learned and Best Practices Documentation Form, completed by Nathalie Guerrier on behalf of Konbit, dated October 7, 2016. 18| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV local organization. However, Konbit core documents show that local partners were supposed to increasingly deliver CD services to awardees and others, rather than Konbit being the primary vehicle for service delivery. The Konbit Cooperative Agreement indicates: “The Mission and its Local Solutions team envision tapping and reinforcing existing Haitian capacity development service providers, who in turn will strengthen local institutions…”13 It goes on to say: “Over the course of their three year involvement, it is anticipated that these international partners will transfer knowledge and technical expertise to emerging local CDSPs, the most highly qualified of which will in turn provide these technical services to their brethren over the remainder of the program.”14 The Year 1 Annual Report is consistent with this, stating that: “The core group of CDSPs were written into the Konbit proposal for two reasons. The first was to ensure that organizational reinforcement would be planned not just for the recipients of USAID funding, but also to strengthen the capacity of the firms that would eventually provide these services themselves: in essence, focusing on the supply of services as well as the demand.”15 “Konbit has not gone far enough with the concept of Local Solutions.” - Core CDSP/Awardee representative The “expertise transfer” aspect of the project is not yet in evidence, although subcontracts with the international partners expired at the end of Year 3 (during data collection for the evaluation). Konbit reports mention only two short-term TA contracts with local service providers and one training contract. The Year 3 Konbit work plan indicated that training capacity would be developed among members of the KCS Association (largely made up of Management Track CDSPs), but subsequent progress reports do not mention any such training. Konbit staff said that they are negotiating with both SKA and Tetra Tech to allow continuation for up to five more months, including an OCA facilitator training in November 2018. The internal mid-term evaluation report Konbit commissioned in 2017 commented specifically on this issue as follows: “Trainings, especially by Shannon Meehan from SKA, were a highlight for many of the participants interviewed. However, this raises questions about longer-term sustainability and how the training program can be sustained by Konbit through Year 5 and by the KCS after Konbit ends, due to expensive rates commanded by international firms. Konbit has not yet moved forward with a specific plan for training sustainability, including a clear means of transferring this capability to local firms, capturing it through technology for scaling to larger numbers of organizations, or another means of transition for longer-term sustainability.”16 Utilization and Utility of Specific Types of Support OCAs All Konbit-led OCA processes used the seven categories listed in the text box on pages 9 and 10, although the sub-categories varied slightly due to use of different OCA versions over time. The output was an OCA report with detailed narrative findings for each category, accompanied by a scoresheet to summarize the agreed-on capacity level in each sub-category, ranging from 1 (low capacity) to 4 (strong capacity). Most Konbit-supported OCA reports included a year-long CAP in table format, listing the steps identified to address key capacity gaps and assigning a priority level or timeline for each. These plans usually allocated responsibility to staff of the organization, indicating where outside support was expected from sources including Konbit. 13 Cooperative Agreement No. AID-521-C-15-000019 between USAID and Papyrus S.A, 2015, at page 22. 14 Id, at page 25. 15 Konbit Year 1 Annual Report, at page 15. 16 Mid-term Evaluation of the Konbit Cooperative Agreement: Building Local Capacity in Haiti, The Kaizen Company, October 2017, at page 11. 19| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV The majority of Konbit staff, as well as interviewed organizations that have done the OCA, expressed satisfaction with the process as a diagnostic tool to identify areas of weakness and form the basis of a plan to address them. On the other hand, at least three targeted organizations said they were reluctant to undergo an OCA either because they had done it previously with another implementer or because they would not have time or resources to carry out the remedial actions identified. Additional OCA limitations observed by informants are highlighted in the text box on page 18. Per Konbit records, 15 CDSPs and awardees have done an OCA with facilitation by the Konbit implementers (either Papyrus or Tetra Tech); seven of them have done more than one assessment. In all, scores from 25 OCAs are recorded in the project M&E database. Besides the awardees, four other organizations have done a baseline OCA, and two Management Track CDSPs have completed follow-up OCAs. Those two, CEDEL and CEGEC, both achieved moderate improvements in their overall scores. Although each organization completing an OCA is expected to undertake implementation of the resulting CAP, Konbit records do not contain information on progress on those plans. Konbit staff said that they only regularly monitor action on CAPs insofar as they relate to CD grants. Konbit has not required OCA recipients to report on actions taken to implement CAPs on their own or with other sources of support, although Konbit staff make occasional enquiries. Targeted organizations mentioned this fact, with one awardee commenting that: “The OCA made a set of recommendations on management, and we have dealt with 98 percent of them, but Konbit has not made any follow-up with us.” OCA Challenges and Limitations Interviewees and document reviews indicated a range of concerns related to the OCA. These are key factors that affected the utilization and utility of the OCA by targeted organizations: • OCA efficacy as a method of monitoring an organization’s capacity for change over time is doubtful. • OCA results can vary greatly depending on the person or organization facilitating the process. • Some organizations felt reluctant to have another local organization (Papyrus) as OCA facilitator, especially if they could be viewed as competitors. • “OCA fatigue” affected organizations asked to undergo a second or third OCA, due to the significant time investment and pressure of other priorities. • The OCA was designed for non-profit organizations and is not fully suitable for businesses, notably in the areas of project management and governance that may not be as relevant. • The scoring of capacity can become too much of a focus for the organization and facilitator, as compared to the discussion and action planning parts of the process. • The OCA is a time-bound tool, so the results can become immediately outdated with the start or termination of major funding or an internal transformation. After the OCA, some organizations did not have the will or ability to systematically pursue implementation of their CAPs to address the identified gaps. 20| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV “On the OCA - A higher score does not equal better. We must look beyond the scores. It is a perfectly good tool, but the quality depends on who is doing it.” - Konbit staff member Training Konbit records describe 21 training workshops that attracted participation by 404 individuals (223 unique participants) from 95 organizations. Total participant numbers have declined each year, with only five workshops organized in Year 3 for 54 unique participants. Konbit staff indicated the reduction was due to a more focused approach after Year 1 and limited budget for training. The majority of interviewed organizations were satisfied with the quality of group training workshops, which was consistent with post-training survey data. Overall ratings of 4 or 5 out of 5 were reported across the ten trainings for which Konbit provided records.17 Knowledge testing was introduced recently for the new comprehensive M&E course, all 8 participants who took pre- and post-tests increased their scores from an average 65 percent beforehand to 92 percent after the workshop. The majority of interviewed organizations said that training had a positive effect on their organization. However, training and TA providers and the majority of participating organizations indicated the subject matter and/or level of training was, in some cases, not sufficiently adapted to suit the needs of target groups and individual participants. Participants sent to training by awardees and core CDSPs were not always the appropriate individuals to share learning and bring about change in their organizations, according to Konbit personnel. As well, lack of training follow-up was cited as a weakness by four targeted organizations. In fact, the first time that the SKA trainer had done individual follow-up visits to trainees was during data collection, which was highly appreciated by those interviewed. Konbit tried to commission an assessment of the impact of their training and TA, which has not been pursued due to budgetary constraints. Financial sustainability and resource mobilization have received some training attention, in the form of three workshops on proposal development. Five KIIs highlighted this concern and OCAs consistently point out this weakness among targeted organizations; the majority of those assessed scored either 1 or 2 (low or basic capacity) in baseline OCAs on “New Opportunity Development for Sustainability.” However, three of seven organizations with repeated OCAs have increased their score in this category from 2 to 3 (moderate capacity) out of a possible 4 (including one awardee, as mentioned above). Experts Konbit Overall demand for this advice service has been relatively low, with project data showing 130 instances of use by 70 organizations to date (excluding Speed-linking).18 Recent reports indicate that increased Konbit outreach has generated higher levels of interest. The new Speed-linking strategy attracted a large number of applicants and high participation levels, with 30 organizations attending three events for brief advice sessions with a range of experts. The most common requests for assistance have related to registration processes (primarily those required to qualify for USAID funding), M&E, contracts, marketing, and gender. Konbit M&E data show that four CDSPs have taken advantage of this service extensively (more than four times each). On the other hand, nearly half (15) of 32 awardees and core CDSPs have never used Experts Konbit. 17 Konbit discontinued compilation of this data after ten training workshops. 18 Speed-linking is a new Konbit tactic for offering expert advice based on the concept of speed dating: selected organizations gather in one place, and each has the opportunity to spend up to 20 minutes with each expert on hand, who represent a range of different CD topics. This was introduced in the third quarter of Year 3. 21| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Eight awardees and CDSPs interviewed (individually or in groups) said that Experts Konbit advice had been useful or had a positive effect, and none indicated that it had not been useful. Konbit staff have delivered most advice to date, with personnel of two Management Track CDSPs also utilized as experts. Most volunteer experts on the roster developed by Konbit have not yet been asked by Konbit to provide any advice; two informants said they were on the list but had not been contacted. Helpline Six organizations have used this service to date according to Konbit records. The vast majority of international partner time has been dedicated to Ayiti Nexus, amounting to 60 percent of recorded SKA time on this service and 98 percent of Tetra Tech time. Interviews indicated this advice was useful for Ayiti Nexus, especially for starting their new USAID award. Other CDSPs have made very limited use of this opportunity. Intensive TA/Coaching M&E has been the focus of intensive individual coaching so far, and has led to tangible results for awardees such as FPDI and FONKOZE, as described under EQ 1. Activity reports also mention activities to strengthen the M&E systems of Group Croissance and IDEH, and to assist Ayiti Nexus in setting up M&E systems for the USAID-funded Konbat Vyolans project. Other TA has included short￾term technical assistance to Ayiti Nexus to strengthen administrative and financial guidelines, provided by CEGEC with oversight from SKA; and advice to CFET on coaching techniques, management of a consortium, and partnership agreements. Interviews with targeted organizations indicated demand for more individualized capacity building tailored to their specific needs; the majority said that Konbit￾provided TA and coaching so far had been highly useful. “Where awardees need more help is in application of the support based on very clear objectives. They need to know how to use these tools for their organization.” - USAID AOR Networking Events Konbit has organized 59 diverse networking events, ranging from media tours to networking power breakfasts. They included eight roundtables for awardees, and 18 roundtables for CDSPs. In all, 256 distinct organizations had attended as of the end of Year 3, excluding the “media tour” series of events in 2018.19 Konbit data indicate that most events targeted awardees and core CDSPs repeatedly, while the level of recurrent participation by others is low. Only 50 organizations have attended at least three networking events, according to Konbit M&E indicator data, which represents 36 percent of the life of project target of 137. “The roundtables are useful because the format is not too heavy; the subject matter has a lot of value.” - Awardee representative Awardees and CDSPs generally agreed that the quality of presentations at roundtables has been good, while they had mixed views on usefulness of these events. Innovation grant applicants, who are relatively new to Konbit, found the events especially valuable for making new contacts; some awardees and CDSPs also mentioned this aspect as useful. USAID and Konbit staff indicated that attendance has been 19 The media tour was a series of events in different locations aimed at raising awareness of and interest in Konbit and its services, with less emphasis on networking and many one-off participants. Therefore, the ET considered that the figures without the media tour were more relevant in the context of this EQ. 22| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV very low at some events, while others attracted significant participation. Project M&E data confirm this: for example, participation at awardee roundtables ranged from a low of five to a high of 17 awardees. DIN Platform (konbit.ht) The majority of informants across all categories consider the DIN platform to be a good idea with strong potential to facilitate information sharing and networking. The latest Konbit data show that more than 1,000 organizations have registered on the platform, including 500+ required to sign up when included in the Konbit CDSP mapping. Konbit provided statistics showing that very few users are returning to the site after registering, and 50 percent of those registered have not visited the site for at least a year. Entrances to the site have averaged about five per day for the past two quarters, while unique pageviews were about 15 per day in that period, according to Google analytics data provided by Konbit.20 The majority of targeted organizations, as well as various informants from USAID and other international entities, indicated that they seldom use the platform, even if they are registered as members. “I visited the platform one time. Now I do not know how to get back on it.” - Core CDSP representative Only two targeted organizations said that the platform was highly useful, with the remainder giving it a medium or low rating for utility. All interviewees asked about the platform mentioned its technical limitations, ranging from the predominant use of English language, awkward user interface, difficulties with registration and uploading information, and inability to automatically “push” info to users based on their recorded areas of interest. Konbit staff said that certain features expected when they chose the host site (Global Innovation Exchange) have not been delivered. The internal midterm evaluation report pointed out that Konbit had little control over the platform itself and the order of priority of upgrades for the platform, especially in terms of its functionality. Konbit staff and reports indicated that motivating targeted organizations to share lessons learned on the platform has been problematic, citing competitive attitudes as well as cultural norms. Although Konbit staff said that the absence of ongoing fees for website hosting would boost platform sustainability, the majority of targeted organizations expressed doubts about its sustainability, given that human resources to ensure quality and functionality would still be required. CONCLUSIONS During its first three years, Konbit has invested intensively in a small cohort of about ten organizations that have intensively engaged in its activities and services, largely by a process of self-selection among the awardees and Core CDSPs, while others have participated on a sporadic or short-term basis. The focus on such a limited group represents a risk, especially as there is no guarantee of these organizations’ sustainability or future funding by USAID; they may nor may not respond to mission priorities in coming years. Papyrus and its international partners have been delivering almost all CD services, and have not yet taken serious steps to transfer expertise and utilize local organizations as CD providers, although that was a core element of the project design and the internal evaluation remarked on this issue one year ago. This constitutes a neglected opportunity to build the capacity of skilled and prominent Haitian service providers, including but not limited to those initially proposed as Core 20 Entrances are the number of times someone has visited any part of the site; pageviews are the total views of any page on the site; unique pageviews aggregate the pageviews generated by the same user during the same session, thus eliminating multiple views or reloads of the same page. More information is available at https://support.google.com/analytics/answer/1257084?hl=en. 23| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV CDSPs, which could undermine the success of Konbit as a capacity-building hub if not addressed in the remaining two years. Konbit and assessed organizations appreciate the OCA as a means of identifying needs for internal strengthening, though evidence of application of OCA findings and implementation of resultant action plans is less than optimal. Participant satisfaction with group training is relatively high, which reflects positively on the project and contracted trainers. TA services including Experts Konbit and Helpline have also garnered appreciative feedback and contributed to advances for certain organizations. Networking events have had mixed reviews and variable participation. The DIN platform has aroused great interest and was frequently mentioned by informants as a key potential success of Konbit, which are promising signs. However, low utilization rates and perceived utility raise serious questions about its current benefits for users and sustainability. Given the technical difficulties reported with the site, other host sites may be more fit for purpose. EQ 3: TO WHAT EXTENT ARE SYSTEMS AND PROCEDURES TO IMPLEMENT THE SMALL GRANTS PROGRAM IN PLACE AND SUFFICIENT TO ENSURE EFFECTIVENESS? FINDINGS Purpose of Konbit Grants The Konbit Cooperative Agreement anticipated the grants program in the following terms: “If activated, it is anticipated that this small grants program would reflect the overarching capacity development goals of the Konbit Program whereby applicants would be eligible to receive capacity assessment, planning, and strengthening support to ensure they are qualified to manage USAID funds.”21 The 2016 Modification to the Cooperative Agreement that established the SGP describes the purpose of the grants as follows: “The small grants component for local civil society organizations will support: 1) capacity building for local partner beneficiary organizations; and 2) pilot innovation projects developed under the Development Innovation Network.”22 As described in the Introduction section above, USAID approved a total of $2,845,000 to fund these two types of grants. Konbit has funded grants valued at approximately $750,000 to date, with several more in the pipeline. The ET has analyzed below the Konbit grant systems and potential contribution of each type of grant to Konbit objectives in this context. General Findings on Grants Systems and Procedures The Grants Manual drafted by Konbit in November 2016 contains a very detailed description of the grants process including: SGP objectives, code of conduct, eligibility, Konbit team roles, and detailed description of five phases: 1) design; 2) competition; 3) pre-award; 3) award; 4) post-award Implementation; and 5) grant closeout. It provides extensive information on U.S. Government statutory requirements governing small grants assistance. The 80-page manual provides instructions for Konbit staff at different stages of the process complemented by extensive formats and examples. The Manual is directed at and used by Konbit staff, who also developed a two-page guidelines document based on the Manual to provide information on the process to CD grant applicants. Konbit grant documents and interviews with staff and grantees indicate the manual has been implemented as written thus far, with no changes or avoidance of provisions reported or observed. USAID personnel said they had observed no compliance issues and had no concerns in terms of Konbit 21 Cooperative Agreement No. AID-521-C-15-000019 between USAID and Papyrus S.A, 2015, at page 24. 22 Modification #2 to Cooperative Agreement No. AID-521-A-15-00009 between USAID and Papyrus S.A, signed November 7, 2016, at page 1 of Attachment No. 1. 24| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV following grantmaking and related rules. No grants have been terminated early for any reason, two grants have been closed out satisfactorily, and two no-cost extensions have been issued thus far, per Konbit staff and records. Pace of grantmaking. The time for issuance of grants of both types has averaged 7.2 months from the time of starting the application to first advance of funds, based on Konbit’s grants process timeline. To date, the project has funded 11 grants (nine CD and two innovation grants) out of the LOP target of 47 grants. In accordance with the modification authorizing the SGP, Konbit has issued all grants using FAA agreements that link payments to predetermined milestones.23 Both Konbit staff and applicants reported that grant negotiation processes had been difficult at times, as most applicants did not clearly understand FAA processes and milestones nor how to budget appropriately. Therefore, they needed extensive handholding and time to finalize terms and reach the point of grant agreement signature. Konbit staff also referred to a “learning curve” for Papyrus as a first-time grant maker. The internal mid-term evaluation in September 2017 observed that grants were moving slowly at that time for multiple reasons. It reported that both Konbit staff and applicants viewed CD grants as imposing significant administrative burdens on grantees before any funds are distributed. Processing times were heavier and longer than expected, for what applicants described as relatively small amounts of funding. The mid-term evaluation report noted a perception among applicant organizations that Papyrus was doing this to be cautious and reduce risk, rather than to build local capacity. The ET did not observe any notable acceleration of the pace of grantmaking following the mid-term evaluation. Konbit staff described the grant development process as part and parcel of their CD approach that aims to build applicant skills along the way, which contributed to delays. The majority of CD grantees agreed that the process had built their capacity, though innovation grant applicants did not concur and were more critical of the grants timeline. Both Konbit staff and grantees expressed frustration that the processes had taken longer than expected. CD Grants The CD grants are described only in general terms in the Cooperative Agreement modification, which specifies that each grant must be based on a previous OCA or another similar assessment, and grant activities will be based on the resulting action plan. These non-competitive grants average about $50,000 for a maximum of one year, according to Konbit project documents, though amounts have ranged from approximately $7,000 to $80,000. The Grants Manual and other documentation reviewed by the ET did not define any criteria to determine the maximum amount available for each organization. The funded activities all relate to internal capacity but are relatively diverse, including support to strategic planning, a market study, a communications plan, accounting personnel, management software, and procedures manuals. Recent OCA results show that two awardees have improved in grant-related capacity areas after completing their grants—FONKOZE in communications and stakeholder involvement, and FONHDAD in strategic planning. Most CD grants are still being implemented, so final results are not yet available. Grants have been regularly monitored via reports and site visits according to Konbit documents and grants management staff. Grantee representatives and Konbit staff stated that these grants were helping to address capacity gaps that would otherwise be difficult to overcome, as they required significant resources. Expectations were high among CD grant applicants according to Konbit staff; some expected to access funds for their 23 KONBIT is using FAAs, which require that detailed costing and justification of all anticipated expenses be done in advance of an award being made. USAID guidelines on these awards point out that: “An organization with limited or no experience in receiving United States government (USG) grants, as a recipient of a fixed amount award, will often need the assistance of USAID at all stages, including pre-award and post-award, for compliance with USAID requirements and successful completion of the grant.” Fixed Amount Awards to Non-Governmental Organizations: An Additional Help Document for Automated Directive System (ADS) Chapter 303. USAID, 2014. at page 3. 25| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV entire CAP and/or for other pressing needs, which led to frustration and misunderstandings in some cases. The time elapsed from grant application to signing the grant agreement has ranged from three to nine months for CD grants, according to Konbit data. In one case, 12 months elapsed before funds were advanced. Having an OCA is a prerequisite, as that process informs selection and design of grant activities. The time required for the OCA adds to the grants processing time, if that organization has no recent OCA to inform development of the grant. Innovation Grants The Cooperative Agreement Modification describes the purpose of these grants as follows: “While many small grants will be awarded to improve organizational capacity…Konbit recognizes that building strong local institutions requires self-reliance, a strong sense of ownership over the development process, and innovative solutions to achieve sustainability. Konbit will promote innovative ideas for sustainability and periodically issue RFAs for pilot innovation projects developed under the DIN that clearly demonstrate a unique, innovative or proprietary capability and/or represent appropriate use of grant funds to support program objectives.”24 The link to Konbit’s overarching objectives and theory of change is not explicitly stated in the Modification nor in any other documents reviewed by the ET. Konbit has issued two RFAs to date, both using similar wording to the above and indicating that Konbit was looking for “innovative solutions to key development challenges in Haiti,” with little additional guidance on what type of initiatives or organizations would be prioritized. During data collection, different people interviewed at Konbit and USAID described the purpose of these grants in different ways, underlining the lack of clarity. Informants variously said they were important to attract promising new members or potential partners to USAID, Konbit, the DIN platform, and/or KCS Association. Two USAID staff and one grant review committee member expressed doubts about the contribution of these grants to Konbit objectives, considering the significant cost. The internal mid-term evaluation of Konbit, completed during the design stage of these grants, concluded that they should be more clearly defined with an end goal in mind to ensure that project objectives would be met. Advertising of the RFAs was seen as adequate by informants, who commented that the “normal methods” for projects in Haiti were used, and that people seemed to be aware of the solicitations. The ET noted that the newspaper advertisements and RFA were in English only in the first round, which was changed in the second round to both English and French. Applications were accepted in both languages for both rounds. The first RFA gave a preference to Cap Haitien-related applicants or projects, while the second specified a preference for organizations/activities outside Port-au-Prince. Konbit has funded two grants to date of approximately $150,000 each for a duration of one year, after application processes of six to eight months. By coincidence, both projects aim to promote the use of liquefied propane gas in northern Haiti as a safer and cleaner cooking fuel than charcoal; one is promoting clean cook stoves for schools, while the other will engage the diaspora in funding gas stoves for vulnerable families. A third grant has been signed but is waiting to receive its first advance. A fourth applicant from the first round became frustrated by the process and withdrew at an advanced stage of negotiations, though discussions with Konbit are ongoing. According to the project’s log sheet for applications, Konbit rejected many concept notes at the first screening stage [17 of 59 submitted (29 percent) for the first RFA, 26 of 49 (53 percent) for the second], for reasons such as exceeding the maximum number of pages, incorrect font, lack of logos, and lack of signatures. The high attrition rate due to technical errors continued at the proposal stage, which 24 Modification #2 to Cooperative Agreement AID-521-A-15-00009 between USAID and Papyrus S.A, signed November 7, 2016, at page 2 of Attachment No. 1. 26| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV meant that the review committees evaluated very few proposals: nine in the first round, and four in the second. Project records show that Konbit rejection emails offered applicants the chance to discuss the reasons for not being selected. Konbit staff explained that the strict application of RFA specifications was intended as a CD measure because international donors are often strict on compliance with technical requirements. Two applicants interviewed by the ET had been refused due to technical errors in their concept papers, and were highly dissatisfied with the process and unhappy with Konbit as a result. Other informants outside of Konbit expressed concern that rules had been applied too rigidly without achieving CD. CONCLUSIONS Overall, the Konbit systems and procedures for CD grants appear to be functioning smoothly, especially once grants are awarded, and in compliance with USAID rules and the Konbit Grants Manual. No major problems in procedures have been observed, except the long delays between application and funding of CD Grants, which were largely due to challenges of the unfamiliar FAA process and some internal Konbit delays. Although evidence is limited by most grants being in progress, the CD Grants appear to be enabling a limited cohort of targeted organizations to tackle capacity challenges for which they needed additional resources. As such, the CD grants are contributing to Konbit overall objectives. The challenge for Konbit in the remaining two years will be to expedite the prerequisite OCA and grants processing and expand access to these resources while continuing to ensure quality control and compliance. The innovation grants have also been subject to long processes of selection and negotiation, which means that the funded projects are in their early stages and no conclusions can be drawn in relation to their implementation and monitoring. The selection process appears to have been managed in a fair and transparent fashion, without any compliance concerns. However, strict enforcement of application requirements, slow processing times, and lack of awards under the second RFA have had some negative effect on perceptions of Konbit and the range of proposals ultimately reviewed, which could ultimately be counterproductive to attracting new organizations and ideas for furthering the development of Haiti. The connection between innovation grants and project objectives is not clearly documented and is subject to differing interpretations, which has constrained the ET’s ability to comment on their effectiveness as an element of Konbit. Nevertheless, given the insubstantial connection to USAID priority sectors or funding opportunities, it is unclear how three grants with a combined cost of nearly $450,000 will have a tangible impact in terms of identifying new potential partners or fundable ideas for USAID. EQ 4: TO WHAT EXTENT AND IN WHAT WAYS ARE PROJECT ACTIVITIES AFFECTING GENDER EQUALITY AND WOMEN’S EMPOWERMENT? FINDINGS Expectations in Relation to Gender Equality and Women’s mpowerment The Konbit Cooperative Agreement contains several statements about how the project was expected to address gender equality and female empowerment. The Gender Statement provides that: “In accordance with USAID policies, activities will address gender issues as appropriate and promote gender equality as a goal of program activities.” The Program Description in the same document indicates that activities to address the three major Konbit objectives should be pursued to enhance gender equality and female empowerment to the greatest possible degree. However, there are no specific targets for women’s participation nor specific objectives related to gender equality or women’s empowerment. 27| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV The Agreement also contains a text box entitled “Konbit’s Spotlight on Gender”: The composition of the Konbit Program’s proposed management team is reflective of its gender-centric approach in all phases of its design… A trio of highly experienced professional women governs both the prime and local subcontracting companies... the female Chief of Party (COP) and Deputy COP will lead the project and receive full-time support from three female Results Team Leaders. This will be proactively enhanced by project policies that ensure institutional practices to avoid gender biases and promote equal status in the access to and control over project assets. Staffing and Strategy on Gender Various factors compromised the project’s attention to gender, according to Konbit staff. Initially, the staff member with dual responsibility for both gender and communications became intensively involved in DIN support, and as a result, devoted very little time to gender-related issues. After that individual’s departure, difficulties in recruiting a suitable consultant on gender mainstreaming led to a staffing gap, according to Konbit reports. Konbit finally engaged a part-time Gender Advisor in the third quarter of Year 3, with a contract for three days per month. Although she has been proactive, Konbit staff indicated concern about the consultant’s ability to properly support the team and targeted organizations with such a limited level of effort. Konbit staff and documents indicate there was no real strategy for gender integration from the start of the project. The Cooperative Agreement allocated “Strategic guidance on gender” to sub-contractor Ayiti Nexus, but Konbit documents and implementers reported that the Ayiti Nexus contribution has largely been confined to recruiting staff or consultants with gender in their mandate. One Konbit staff remarked that having Ayiti Nexus (rather than Papyrus) responsible for gender within Konbit made it more difficult to incorporate gender considerations in their work. At least five Konbit implementer staff agreed that the project had rarely integrated gender in its activities and that staffing on gender had been insufficient to ensure adequate attention to the subject. In fact, the Year 3 Konbit work plan emphasized the need for stronger gender mainstreaming practices. At the time of data collection, the part-time advisor was developing a gender work plan. The draft work plan reviewed by the ET proposes a Gender Mission Statement and a Gender Vision: “Konbit believes that local development partners should be models for gender inclusion and the empowerment of women in Haiti.” The plan proposes two key interventions: an educational campaign and an assessment tool for local partners to identify gaps and improve their gender mainstreaming. Specific Activities and Tools Related to Gender The Cooperative Agreement specified that the DIN would develop regular highly participative networking events around CD topics, with a steady focus on gender equity. The Konbit M&E database shows only one DIN activity specifically dealing with gender issues—a workshop entitled “Gender Sensitive M&E” in Cap-Haitian in August 2017. Konbit’s database lists one other activity focused on gender equality or integration, which was a one-day gender awareness training in Port-au-Prince. The new M&E Curriculum devotes one slide to gender-sensitivity and makes several references to gender disaggregation of data. Otherwise, the training materials provided by Konbit scarcely mention gender. All interviewed targeted organizations indicated they had participated in either no activities or only one activity (training or networking event) on the subject of gender equality, and the majority said they had received no particular support with gender integration in their organizations. “It was very unclear for me in the beginning, who is going to develop and implement any kind of gender strategy? We are just starting to work on a gender tool to complement the OCA, this is positive. We need concrete guidance and steps to follow. That has been lacking.” - Konbit staff member 28| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV The OCA package used by Konbit shows that out of 46 sub-categories of capacity, one is called “culture and gender” (within program management); there is no other reference to gender or women’s empowerment in the scoresheet. Scores for assessed organizations on the OCA element of culture and gender ranged widely from 1 to 4 (top score), with the most recent scores averaging 2.9. Three targeted organizations with repeated OCAs showed an increase in this area, three others remained static, and one declined. To ensure this area receives greater attention in future, Konbit reports indicate that the Gender Advisor is developing a gender self-assessment checklist to complement the OCA. With respect to innovation grants, the Grants Manual includes the following as a cross-cutting consideration: “Does the idea incorporate gender, youth, minorities, and environmental concerns (when applicable)?” These criteria were worth 20 points out of 100, and the full proposal format required a gender strategy. Both innovation grants funded to date included a brief gender strategy in their proposals and have a substantial emphasis on women’s empowerment. Monitoring Gender and Women’s Empowerment The Cooperative Agreement specifies that Konbit should disaggregate data by gender, age, ethnicity, and other factors, and that the M&E Plan must explain how gender analysis will be incorporated into the M&E methodology. Konbit progress reporting does not disaggregate participation by sex, but Konbit has reported a breakdown by sex on USAID’s Trainet database. The data indicate 51 percent female participation in training, rising from 46 percent in Year 1 to 60 percent in Year 3. The two M&E Plans reviewed by the ET barely mention gender or sex. Indicators and the Konbit M&E database do not disaggregate by sex because they are tracking by organization rather than by individual; they also do not disaggregate by women-owned or gender-focused organizations. CONCLUSIONS Konbit has given very low priority to gender equality and women’s empowerment, based on the totality of evidence. Male and female participation in training activities is balanced, but Konbit is tracking no other gender-related data and has lacked a strategy to ensure appropriate consideration and integration of these issues in the project. The new gender advisor and work plan are promising signs that Konbit aims to put more emphasis on these subjects in its final two years. However, the level of effort for the consultant is very limited and this initiative is coming late in the project. Overall, there is no evidence of measurable results by Konbit related to gender equality or women’s empowerment at this point, though two of the innovation grants may have some positive impact. IV. RECOMMENDATIONS FOR USAID 1. As a cross-cutting initiative that needs to liaise with all USAID programs and technical offices, as well as OFM and Office of Acquisitions and Assistance, USAID should consider relocating the Local Solutions team/Konbit management to either the Program Office or Front Office, or otherwise providing higher-level leadership and strategic guidance for this initiative. This would help to ensure a stronger impetus for promoting and supporting Local Solutions across the Mission, and a comprehensive approach that considers all the ramifications of directly funding local organizations. 2. USAID should identify which organizations and sectors are the priority for future local awards and subawards, to the extent possible, so Konbit can achieve better synergy with Mission-wide strategies during the next two years. This could be part of a broader Local Solutions strategy for the Mission, now that most APS awards have ended. In parallel, as local organizations are 29| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV included in new Mission programming as direct or indirect recipients, USAID should link them with Konbit in a systematic way and prioritize inclusion of a budget line item for CD in those awards and subawards (possibly requiring an OCA or similar process), to underline the importance of organizational capacity and provide resources for purchase of CD services on the local market, if available. 3. USAID should improve communications and coordination among Konbit, technical offices (especially AORs), OFM, and awardees by establishing a systematic process for sharing information and planning capacity support on a regular basis. In particular, USAID and Konbit should ensure that technical offices and eligible organizations are fully and periodically informed of Konbit offerings for awardees and potential awardees, especially after key staff turnover. Konbit and OFM should take steps to strengthen complementarity among the assessments conducted with local awardees at different points in time, to ensure valuable information is fully utilized and overlaps minimized. FOR KONBIT 1. Konbit should liaise closely with USAID to agree on a targeting strategy for all Konbit assistance to inform their Year 4 and Year 5 work plans, ensuring that current and anticipated recipients of USAID resources (direct or indirect) are prioritized for Konbit training and TA to enable effective implementation of their awards. Konbit should favor this approach as compared to training a large number of new targets from a broad cross-section of organizations without a clear connection to USAID programming. Indicator targets should be reviewed to reflect this strategy. 2. Konbit should re-engage with and expand the Service Track group of CDSPs, providing and promoting opportunities to skilled Haitian professionals to deliver paid services within their areas of expertise. If skill levels are insufficient in certain areas, Konbit should support mentoring and co-delivery of services with Konbit international partners to build capacity in a hands-on and practical fashion. Transferring expertise to local service providers should be the primary focus of international partners going forward, as these Haitian entities should by the end of Konbit constitute a reliable source of core professional advisory services to meet the fundamental needs of USAID programming partners. 3. Konbit should prioritize the provision of one-on-one (or small group) hands-on technical assistance to strengthen capacity, tailored to address the priorities of targeted organizations based on OCAs and other consultations. These coaching activities could follow up and complement group training, promote adoption of learning, and potentially serve as one means of assessing progress in capacity. 4. Konbit should identify or develop more efficient, frequent, and accurate tools for monitoring progress on CD, to reduce the risk of reliance on a single method. At a minimum, this should include close regular monitoring of the full scope of CAP implementation in collaboration with assessed organizations, potentially complemented by periodic “partial OCAs” focused on specific areas where interventions have been made. Ultimately, USAID and Konbit should identify ways to assess cumulative impact on organizations receiving significant CD support over time, including effects on overall performance of those entities, which could include Most Significant Change or Outcome Mapping methods. 5. Konbit should devote more support to helping selected awardees and other priority organizations to achieve greater financial sustainability and diversification of funding sources, especially those largely dependent on one or two donors. Strategies could include pilot income generation projects for non-profits, and improvement of technical or marketing skills to increase competitiveness of both non-profits and for-profits. While some needs could be met with training or TA, Konbit should also consider CD grants to support development of revenue-generating activities by targeted 30| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV organizations to enhance financial sustainability, which by extension will make their organizational capacity more sustainable and thereby engender longer-term return on Konbit investments. 6. If resources are limited for the SGP, Konbit should favor CD grants due to their direct contribution to project objectives and USAID priorities as well as the significant unmet needs for capacity support among targeted organizations. To maximize reach of CD grants in the remaining two years, Konbit should take steps to expedite negotiation and processing of grants to minimize frustrations of grantees, promote timely response to capacity needs, and ensure enough time to implement and close out grants. 7. USAID and Konbit should clarify or modify the purpose of the innovation grants component to strengthen its connection to project objectives before issuing more grants of that type. Strategies to tighten the focus of those grants could include requiring applicants to propose partnerships or consortia and to focus on USAID priority sectors. If more grants are offered, Konbit should ensure that potentially strong grant applications are not ruled out based on technical errors, for example by deducting points for technical flaws or by using templates that limit the possibility of such errors. This would serve Konbit’s CD priorities, allow for review of a wider range of proposals, and be less likely to alienate applicants. 8. Konbit should rework the DIN web platform to optimize usability, including migration to a new host site, if necessary. French language functionality and improved user interface would enhance its appeal to a wider base of users, encourage more frequent usage, and boost its ultimate sustainability.25 9. Konbit should focus future activities related to gender integration and equality on a selected group of awardees and core CDSPs, with a view to achieving measurable impact in terms of awareness and good practices in those organizations in the remaining two years. 25 Guidance from sources such as Principles for Digital Development may be helpful to Konbit in considering next steps. See https://digitalprinciples.org/ for more information. 31| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX A: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Mid-Term Evaluation Capacity Development mechanism / KONBIT PURPOSE OF THE EVALUATION The purpose of this mid-term performance evaluation is to inform USAID/Haiti about possible mid-course corrections and future program orientation of the KONBIT program, whose objective is to improve USAID/Haiti’s local partners capacity to manage funding. Lessons learned and recommendations provided through the evaluation report will also guide future programming in areas such as organizational governance, leadership, financial management, project management, human resources, monitoring and evaluation, and other aspects of organizational development. SUMMARY INFORMATION Strategy/Project/Activity Name Capacity Development Mechanism (KONBIT) Implementer Papyrus (sub partners: Ayiti Nexus, Steve Kroll & Associates, TetraTech) Cooperative Agreement/Contract # AID-521-A-15-00009 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $7,646,479 Life of Strategy, Project, or Activity October 2015–September 2020 Active Geographic Regions All geographic regions Development Objective(s) (DOs) Increased participation of Haitian civil society and private sector in the implementation of development programs. USAID Office USAID/Haiti Democracy and Governance Office BACKGROUND Through its USAID Forward initiative and Local Solutions Framework, the Agency prioritizes the strengthening of local actors—and the systems in which they operate—as a critical tool in implementing more efficient and effective funding with more sustainable and impactful development results. Drawing on the USAID Local Solutions framework, USAID/Haiti requested the establishment of a “Capacity Development Hub” to: 1) set up and coordinate the provision of local capacity development services; 2) create a Development Innovation Network (DIN); and 3) develop and implement monitoring, evaluation, and learning activities that accurately gauge project progress and promote improved program results. In response, local Haitian prime partner Papyrus S.A. and its three sub-partners—Ayiti Nexus, Steve Kroll and Associates, and TetraTech—offer a range of innovative approaches and thoughtfully-designed activities 32| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV to address the three major objectives under the KONBIT Program. The activity began in October 2015 and is in its second year of implementation. A. Description of the Problem, Development Hypothesis, and Theory of Change USAID/Haiti recognizes the need for a more intensive and expansive capacity building approach when partnering directly with local organizations. Based on market research, USAID has found that many local Haitian organizations lack the technical and organizational capacity to effectively and responsibly manage U.S. Government resources. Recognizing that USAID/Haiti does not currently possess the internal workforce to fully direct these capacity building efforts and manage the administrative burden required to support multiple new awards to local organizations, a third-party organization was needed to lead this effort in the Northern, St. Marc, and Port-au-Prince corridors. This was expanded to all geographic regions following Hurricane Matthew as part of a post-hurricane response strategy. The third party is responsible for achieving four main objectives: 1. Establish and coordinate comprehensive capacity development services; 2. Launch a development innovation network; 3. Develop and implement monitoring and evaluation learning activities; and 4. Implement a small grants program. KONBIT has developed a Theory of Change (TOC): If capacity of local actors is built, a development Innovation Network is established, a Monitoring, Evaluation and Learning activities are implemented then local development partners will influence and achieve significant and sustainable results, be accountable to their constituency, be able to effectively compete for and manage resources. While some assumptions and preconditions may have changed over time due to evolving contextual circumstances, expected results will not be modified. KONBIT´s TOC is complex and overarching; nevertheless, testing all its assumptions would require resources beyond the capabilities of the project. B. Summary Strategy/Project/Activity/Intervention to be evaluated First, the KONBIT Program intended to build local institutional capacity at three levels by: a) providing assistance to up to 37 USAID/Haiti local awardees to enable them to meet the terms of their grants and deliver intended development results; b) strengthening the proficiencies of about 100 CDSPs, a number of which will be engaged to respond to the management and technical training needs of USAID local grant recipients, and a select six to ten of which will be upgraded to potentially serve as direct implementers of USAID programming in the future; and c) supporting the program’s core management team—Papyrus and Ayiti Nexus—to hone their respective capabilities to implement major donor awards. Second, the KONBIT Program intended to mobilize regular networking events for a wide range of local and international stakeholders—and establish a dedicated interactive website—to encourage collaboration and sharing of best practices, innovations, and industry standards in both capacity development and technical areas. The Development Innovation Network will also facilitate local access to USAID solicitations and assist local firms to more capably respond to USAID requirements. Third, the KONBIT Program intended to measure its progress and achievements, utilize its M&E processes and results to create learning opportunities for program participants, and employ an interactive platform, DIN/Knowledge Management & Learning, for sharing key lessons learned. 33| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan A Service Log Database was to be used internally by KONBIT technical experts to record all capacity development inputs extended to organizations. For remote support services provided by Steve Kroll & Associates and Tetra Tech, information was to be gathered through invoices submitted by international experts, compiled and then entered onto the Service Log Database. The following documents will be available to the evaluators: • Local partners’ advocacy plans • Database training attendance lists • OCA Database, Small Grants Program reports • DIN events database and DIN website • Funding source survey, financial records, MOU and contracts • Posts on the DIN/KM&L Platform • Event attendance sheets, trip reports and support logs • M&E OCA Database; M&E OCA scores EVALUATION QUESTIONS Evaluation Title/Questions 1. To what extent have local USAID/Haiti awardees working with KONBIT acquired and implemented systems and procedures to effectively and responsibly manage USAID funds? In answering this question, the evaluation team should assess: a) where firms are at in the process currently, noting significant successes and failures, b) what remains to be done so that the systems and procedures are in place to ensure sustainability, and c) identify any potential obstacles to organization’s ability to independently and appropriately manage USAID funds. The team should particularly examine: MEL, Finance, and Management. 2. How are organizations applying the knowledge and tools (such as OCA, help line, DIN, M&E) gained through KONBIT? 3. To what extent are systems and procedures to implement the small grants program in place and sufficient to ensure effectiveness? 4. To what extent and in what ways are project activities affecting gender equality and women’s empowerment? EVALUATION DESIGN AND METHODOLOGY It is expected that the evaluation team proposes an overall research design to address the evaluation questions and a plan for collecting and analyzing data. Nevertheless, the evaluation design should be based solely on a combination of qualitative techniques to address the evaluation questions. Key informant interviews, focus group interviews, in-depth review of projects reports and Organization Capacity Assessment (OCA) are among the techniques that the evaluation team should consider when addressing the evaluation questions. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Upon receipt of this Activity Request, Social Impact (SI) shall submit within two weeks a draft work plan to the Contracting Officer’s Representative (COR). The work plan will 34| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, delineated by roles and responsibilities with their level of effort; (3) the identification of other required personnel and relevant local subcontractors, their LOE, roles and responsibilities and qualifications; and (4) the deliverable schedule. 2. Inception Report: The evaluation team will have two weeks to review the project documents and produce an inception report or background report that addresses what the evaluation team has learned based on program documents provided to them. The inception report should inform the design of the evaluation. Therefore, during this phase period, USAID/Haiti should consider the possibility of revising evaluation questions based on evaluation team input. Any revisions to the questions in the SOW should be documented in writing in the evaluation report. 3. Evaluation Design: Within four weeks of approval of the work plan, SI must submit to Contracting Officer’s Representative (COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID/Haiti will take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 business days. 4. In-briefing: Prior undertaking field work, the evaluation team will have an in-briefing with the USAID/Governance Team and the Evaluation and Survey Services (ESS) COR to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. 5. Evaluation Briefing/Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations to USAID within 20 business days after the conclusion of fieldwork. 6. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers having a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID/Haiti will have 10 working business days in which to review and comment on the initial draft, after which point the ESS COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit revised final draft report 5 business days hence, and again USAID/Haiti will review and send comments on this final draft report within 5 business days of its submission. 7. Final Evaluation Report: The evaluation team will be asked to take no more than 15 business days to respond/incorporate the final comments from the Governance Office. The evaluation team leader will then submit the final report to the COR. All project data and records (FGD and KII summary reports) shall be submitted in full and should be in electronic form in easily readable format, organized, 35| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITION The Evaluation Team shall be comprised of two Key Personnel positions: (i) a Team Leader, and ii) a specialist in Governance and Capacity Building. The Team Leader (TL) is ultimately responsible for the overall management of the evaluation team, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the final evaluation report in conformity with this Statement of Work. The TL must be an experienced evaluation expert, with a documented track record of 10 years of experience in the field of evaluation. S/he should have a strong background in the governance and capacity building field. S/he should be fluent in French and English. S/he should have at least a Master’s Degree in Economics, Statistics, Political Science, or Public Administration. The Capacity Building Specialist will help the TL in the overall management of the evaluation team and the final products, in conformity with this Statement of Work. S/he should be familiar with the administration and Institutional Capacity. S/he must possess excellent writing and interpersonal skills and must be familiar with USAID programs, objectives, and reporting requirements. S/he should have experience in designing and implementing FGDs. Fluency in French is required. English and Haitian Creole are highly desirable, as is significant prior work experience in Haiti. A Bachelor's degree in Political Science, Public Administration, or a related field is required to ensure that all areas of technical expertise required for the evaluation are effectively covered. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. EVALUATION SCHEDULE Timing (Anticipated Weeks or Duration) Scheduled Activities 6 Weeks Preparation of the work plan, inception report and evaluation design 3 Weeks USAID review of the work plan, inception report and evaluation design 1 Week Submission of the revised evaluation design, inception report and work plan; in-briefing 3 Weeks Data collection 2 Weeks Data analysis and Evaluation Briefing 2 Weeks Draft Report writing 2 Weeks USAID review of Draft Report 2 Weeks Incorporate USAID comments and prepare Final Report FINAL REPORT FORMAT 36| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.26 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. 26 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 37| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 38| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX B: EVALUATION TEAM MEMBERS EVALUATION TEAM Evaluation Team Leader, Ms. Melanie Reimer, is an international development professional with over 20 years of experience in M&E and the civil society sector. She has led multiple evaluations of USAID-funded civil society projects in numerous countries including Ecuador, Myanmar, Georgia, Pakistan, Cambodia, Sri Lanka, Rwanda, and Zambia. Ms. Reimer has sound experience conducting Civil Society Sector Assessments and developing Performance Monitoring Plans. In Haiti, she designed and delivered participatory trainings on M&E. Ms. Reimer holds a Bachelor of Laws from the University of Manitoba. She is fluent in French and English. Sector Specialist, Mr. Ralph Denize, brings over 20 years of professional experience designing, leading, and implementing multi-million-dollar projects in private sector enterprise, financial inclusion, workforce development and capacity building in Haiti, Canada, and Jamaica. Mr. Denize led the Multilateral Investment Fund’s local capacity strengthening strategy, managing a 50 million dollar portfolio of over 30 projects in the fields of financial inclusion, smart cities, and knowledge economy, among others. He has direct experience working for USAID as a Private Enterprise Specialist and intimate knowledge of the challenges faced by Haitian organizations in establishing autonomy and executing capacity-building exercises. Mr. Denize holds a Master of Science in Engineering Management from Northeastern University and is fluent in English, French and Haitian Creole. SI HQ AND ESS STAFF SI HQ and ESS staff managed the evaluation and provided quality assurance. The Project Director (PD) oversaw and advised the ET on deliverable quality. The Project Manager (PM) was responsible for the evaluation timeline, financial management, and trouble-shooting. PD and PM reviewed all major evaluation deliverables against SI’s EQUI® checklists. The Project Assistant (PA) handled administrative tasks related to the successful completion of the evaluation. The ESS COP provided technical and managerial oversight. She served as a point of contact with USAID and will participate in all meetings with USAID. Table 6: Personnel Roles and Responsibilities EVALUATION TEAM ROLES RESPONSIBILITIES Team Leader (TL) Ms. Melanie Reimer • Manage the entire evaluation, oversee ET members, and produce timely, high-quality deliverables. • Lead the development of all deliverables by delegating assignments to team members, then reviewing and consolidating their inputs. • Deliver briefings and presentations to USAID and other stakeholders. Sector Specialist Mr. Ralph Denize • Contribute significantly to the desk review, development of the design, data analysis, and report writing. • Conduct data collection and collaborate with notetakers when split into sub-teams. • Provide subject matter expertise in development of the evaluation design, data collection tools and data analysis. • Assist the TL to prepare and deliver briefings and presentations. Mr. Reginald Page & Mr. Mackenson Desrosiers Note-takers • Coordinate in-country logistics, such as arranging interviews. • Support the ET in conducting data collection, primarily though taking detailed notes. • Assist with coding and data analysis. 39| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV SI HQ AND ESS FIELD OFFICE STAFF ROLES RESPONSIBILITIES PD Mr. Samir Panjwani • Review and provide feedback on Inception Report, Evaluation Design, data collection tools, draft, and final reports to ensure quality, focusing on evaluation logic. • Provide high level guidance on methodology, major deliverables’ quality, and any problems encountered during the evaluation. • Oversee TL and ESS management staff. • Conduct in-depth review of deliverables using EQUI® quality assurance check points. PM Ms. Mariana Davila • Train the ET on SI’s procedures and EQUI® quality standards. • Review all final deliverables to ensure quality standard. • Manage the budget by tracking level of effort usage and other spending. PA Mr. Justin Archangel • Provide administrative and logistical support. • Copyedit and format deliverables. COP Ms. Jennifer Mandel • USAID Point of Contact. • Ensure USAID satisfaction via routine check-ins. 40| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX C: KONBIT THEORY OF CHANGE Figure 1: Konbit Theory of Change 41| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX D: KONBIT RESULTS FRAMEWORK Figure 2: Konbit Results Framework 42| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX E: LIST OF AWARDEES AND CORE CDSPS Table 7: List of Awardees and Core CDSPs ORGANIZATION NAME* CAPACITY MECHANISM OFFICE/APS APS AWARDEE CORE CDSP MGMT TRACK AGCRDDM Chanje Lavi Plante EGAD Y AGRITECH Konbit EGAD Y ANACAPH Konbit - plus DAI EGAD Y ANIMH Konbit - plus DAI EGAD Y Ayiti Nexus Konbit DG Y Y Y Build Change PVO EGAD Y Carbon Roots USAID/Local Enterprise and Value Chain Enhancement Project (LEVE) Y CFET Konbit/KONEKTE EGAD Y Y Y CPFO Konbit DG Y FACSDIS Konbit plus Konbat Vyolans DG Y FONHDAD Chanje Lavi Plante EGAD Y FONHEP N/A Yunus Social Business Y FONKOZE JSI Health Y FPDI Konbit Nutrition Y GHESKIO JSI Health Y Group Croissance Konbit DG Y Y Y IDEH Konbit DG Y IDEJEN LEVE EGAD Y J/P HRO JSI Health Y KOFAVIV Konbat Vyolans DG Y SEFIS JSI Health Y Service Chretien JSI Health Y SHAA Konbit Education Y CEDEL Konbit Y Y CEGEC Konbit Y Y CMP Konbit Y CSL Konbit Y ECA Konbit Y FHADIMAC Konbit new new Fondation Espoir Konbit Y IFOS Konbit Y new PAGS Konbit Y *Name in bold font indicates USAID local awardees 43| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX F: LIST OF PERSONS INTERVIEWED Table 8: List of Persons Interviewed USAID ORGANIZATION TITLE/POSITION USAID Mission Director USAID Deputy Mission Director USAID Former Konbit AOR USAID Konbit AOR USAID Mission Monitoring and Evaluation Officer, Program Office USAID Head of Office of Financial Management and Deputy Office Chief USAID Acting DG Office Chief USAID AOR of Agritech USAID AOR of CFET USAID AOR of FPDI USAID AOR of Groupe Croissance USAID AOR of J/P HRO USAID AOR of SHAA IMPLEMENTING PARTNERS ORGANIZATION TITLE/POSITION Papyrus Chief Executive Officer (CEO) of Papyrus Ayiti Nexus Managing Partner Ayiti Nexus Managing Partner Tetra Tech Associate Tetra Tech Grants specialist SKA Training Consultant Konbit COP Konbit IR4 Lead Konbit IR1 Lead Konbit IR2 Lead Konbit IR3 Lead Ayiti Nexus/Konbit Gender Advisor Ayiti Nexus/Konbit Former Communications and Gender Officer Ayiti Nexus/Konbit M&E consultant DONORS/INTERNATIONAL ORGANIZATION TITLE/POSITION Embassy of Canada Head of Trade JSI Capacity Building Advisor DAI COP Finance Inclusive AWARDEES ORGANIZATION TITLE/POSITION Agritech CEO 44| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV CFET Executive Director FPDI PM Group Croissance President/General Director J/P HRO Communications Manager and Community Project Manager SHAA National Coordinator, M&E Officer, and Project Manager CDSPS ORGANIZATION TITLE/POSITION CEGEC Associate Director CEDEL Executive Director FHADIMAC Executive Director Fondation Espoir Manager ECA Accountant-Manager CMP Executive Director INNOVATION GRANTS STAKEHOLDERS ORGANIZATION TITLE/POSITION Switch Executive Director and Communications Manager Danielle St. Lot Foundation (DSLF) Executive Director Gestion Etudes Comptabilité Audit (GECA) Finance Director FormEtic Executive Director Développement des activités de santé en Haïti (DASH) Co-director Fejeg Consult Director USAIDUSAID AWARDEES - GROUP INTERVIEW #1 ORGANIZATION TITLE/POSITION ANACAPH Communication Manager FACDIS Executive Director FACDIS Administrative Assistant FONHEP Co-director Service Chrétien Executive Director IDEH President FONKOZE Director FONHDAD Accountant USAID AWARDEES - GROUP INTERVIEW #2 ORGANIZATION TITLE/POSITION ANIMH Administrator AGCRDDM Technical Director Build Change Project Manager IDEJEN Liaison Officer 45| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX G: EVALUATION DESIGN MATRIX Table 9 . Evaluation Design Matrix DATA SOURCES SAMPLE QUESTIONS ANALYSIS PLAN DATA COLLECTION METHODS EVALUATION QUESTION • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis • To what extent has Konbit support met awardee needs? • What effect has been noted on OCA scores of awardees, and why? • To what extent has Konbit affected awardee ability to effectively manage USAID funds? • What exactly has changed? • What have been the obstacles to improvement? • How sustainable are the improved systems? • How has Konbit support complemented assistance from other sources? • Which kinds of Konbit support have been most and least useful? • USAID/Haiti personnel • Konbit implementer staff • Awardee representatives • Core CDSP representatives • Other CDSP representatives • Donors and other international programs • Activity documentation • USAID documentation • Document Review • KIIs and GIs 1. To what extent have local USAID/Haiti awardees working with Konbit acquired and implemented systems and procedures to effectively and responsibly manage USAID funds? • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis • To what extent has Konbit support met core CDSP needs? • What effect has been noted on OCA scores of CDSPs, and why? • To what extent has Konbit affected core CDSP ability to effectively manage USAID funds? • What exactly has changed? • Which kinds of Konbit support have been most and least useful to CDSPs in general? • To what extent have CDSPs adopted or used Konbit learning or tools to improve internal capacity? • USAID/Haiti personnel • Konbit implementer staff • Awardee representatives • Core CDSP representatives • Other CDSP representatives • Donors and other international programs • Activity documentation • USAID documentation • Document Review • KIIs and GIs 2. How are organizations applying the knowledge and tools (such as OCA, Helpline, DIN, and M&E) gained through Konbit? 46| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV DATA SOURCES SAMPLE QUESTIONS ANALYSIS PLAN DATA COLLECTION METHODS EVALUATION QUESTION • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis • How would you describe the Konbit procedures and tools for selection and management of grants? • What has been your experience so far in obtaining and implementing a CD grant or innovation grant? • What challenges have arisen in implementation of the grants system? (adapted for applicants, grantees and administrators) • How well as the system promoted compliance and supported Konbit objectives? • To what extent have the issues identified by the internal evaluation been addressed? • USAID/Haiti personnel • Konbit implementer staff • Awardees applying for grants • Innovation grant stakeholders • Donors and other international programs • Activity documentation • Document Review • KIIs and GIs 3. To what extent are systems and procedures to implement the small grants program in place and sufficient to ensure effectiveness? • Content analysis • Trend analysis • Gap analysis • Comparative analysis • Gender analysis • To what extent have Konbit approaches and activities integrated gender equality and women’s empowerment? • In what ways has Konbit worked to include and empower both men and women? • Have the effects of Konbit been the same or different for different gender groups? • To what extent has Konbit helped awardees better mainstream gender? • To what extent has Konbit helped CDSPs better mainstream gender? • What have been the challenges to increased gender mainstreaming by Konbit and its targeted organizations? • USAID/Haiti personnel • Konbit implementer staff • Awardee representatives • Core CDSP representatives • Other CDSP representatives • Activity documentation • Document Review • KIIs and GIs 4. To what extent and in what ways are project activities affecting gender equality and women’s empowerment? 47| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX H: DOCUMENTS REVIEWED Activity Progress Reports and Plans USAID/Haiti. Konbit Year I Annual Report. Not published (N.p).: USAID, 2016. USAID/Haiti. Konbit Year 2 Annual Report. N.p.: USAID, 2017. USAID/Haiti. Konbit Quarterly Report: Quarter 1 Financial Year (FY)1. N.p.: USAID, 2016. USAID/Haiti. Konbit Quarterly Report: Quarter 2 FY1. N.p.: USAID, 2016. USAID/Haiti. Konbit Quarterly Report: Quarter 3 FY1. N.p.: USAID, 2016. USAID/Haiti. Konbit Quarterly Report: Quarter 1 FY2. N.p.: USAID, 2017. USAID/Haiti. Konbit Quarterly Report: Quarter 2 FY2. N.p.: USAID, 2017. USAID/Haiti. Konbit Quarterly Report: Quarter 3 FY2. N.p.: USAID, 2017. USAID/Haiti. Konbit Quarterly Report: Quarter 1 FY3. N.p.: USAID, 2018. USAID/Haiti. Konbit Quarterly Report: Quarter 2 FY3. N.p.: USAID, 2018. USAID/Haiti. Konbit Draft Quarterly Report: Quarter 3 FY3. N.p.: USAID, 2018. USAID/Haiti. Konbit Weekly Highlights July-Sept 2018. N.p.: USAID, 2018 USAID/Haiti. Konbit 2015-2016 Project Monitoring & Evaluation Plan. USAID, 2015. USAID/Haiti. Konbit FY3 Annual Monitoring & Evaluation Plan. USAID, 2017. USAID/Haiti. USAID/Haiti. Konbit Annual Work Plan FY 2015-2016. USAID, 2015. USAID/Haiti. Konbit Annual Work Plan FY 2016-2017. USAID, 2016. USAID/Haiti. Konbit Annual Work Plan FY 2017-2018. USAID, 2017. Activity Agreements & Modifications USAID/Haiti. Request for Applications Funding Opportunity No. USAID-HAITI- RFA-521-15- 000002 Konbit. N.p: USAID, 2014. Papyrus S.A. Konbit Technical Proposal. N.p.: Papyrus S. A., 2015. USAID/Haiti. AID-521-C-15-000019 Konbit. Cooperative Agreement. N.p: USAID, 2015. USAID/Haiti. AID-521-C-15-000019 Konbit. Cooperative Agreement Modifications 1 to 10. N.p.: USAID, 2015 to 2018. Papyrus S.A. Sub-Contractor Agreement #1 with Steve Kroll and Associates, including Task Orders 1 to 10. N.p.: Papyrus S.A,. 2016-2018. Papyrus S.A. Standard Grant Agreement with Ayiti Nexus, N.p.: Papyrus S.A., 2016. Papyrus S.A. Standard Grant Agreement with Tetra Tech, N.p.: Papyrus S.A., 2015. Other Activity Documents Papyrus S.A. Konbit Partners Final List. N.p.: Papyrus S. A., Undated. Papyrus S.A. Konbit CDSP Mapping Database. N.p.: Papyrus S. A., 2017. Papyrus S.A. Konbit Updated Organization Chart. N.p.: Papyrus S.A., 2017. Papyrus S.A. Konbit Project Description, Small Grants Program. N.p.: Papyrus S. A., 2016. Papyrus S.A. Konbit Grants Manual. N.p.: Papyrus S. A., 2016. Papyrus S.A. Guideline Process for Capacity Development Grants. Papyrus S. A., Undated. Papyrus S.A. Fixed Amount Award Agreements with Grantees. N.p.: Papyrus S. A., 2017-2018. Papyrus S.A. Grantee Performance Reports. N.p.: Papyrus S. A., 2017-18. Papyrus S.A. Grant Negotiation Memoranda. N.p.: Papyrus S. A., 2017-18. Papyrus S.A. Grantee Site Visit Reports. N.p.: Papyrus S. A., 2017-18. Papyrus S.A. Innovative Grants RFAs. N.p.: Papyrus S. A., 2017-2018. Papyrus S.A. List of Grant Amounts. N.p.: Papyrus S.A., 2018. Papyrus S.A. CDSP Survey Form. N.p.: Papyrus S. A., Undated. Papyrus S.A. DIN Performance Metrics. N.p.: Papyrus S. A., 2018. Papyrus S.A. DIN List of Registered Organizations. N.p.: Papyrus S. A., 2018. Papyrus S.A. DIN Database of Individual Users. N.p.: Papyrus S. A., 2018. 48| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Papyrus S.A. Experts Konbit, List of Experts. N.p.: Papyrus S. A., 2018. Papyrus S.A. Experts Konbit, Speed-linking Reports. N.p.: Papyrus S. A., 2018. Papyrus S.A. Konbit Training Materials, various. N.p.: Papyrus S.A., Undated. Papyrus S.A. Konbit Monitoring, Evaluation and Learning Training Curriculum. N.p.: Papyrus S.A., 2017. Papyrus S.A. Lessons Learned and Best Practices Documentation Forms & Surveys. N.p.: Papyrus S.A., 2018. Papyrus S.A. Konbit Training Gender Breakdown data. N.p.: Papyrus S.A., 2018. Papyrus S.A. Konbit Event Feedback data. N.p.: Papyrus S.A., 2018. Papyrus S.A. Request for Quotation to assess Konbit training effectiveness. N.p.: Papyrus S.A., 2018. Papyrus S.A. M&E Datasets including Helpline logs. N.p.: Papyrus S. A., 2018. Papyrus S.A. M&E Training Pre and Post-Test Summary of Results. N.p.: Papyrus S.A., 2018. Statutes of Konbit pour la Competitivité Stratégique Association. Undated USAID/Haiti. Summaries of Awardee FAA Entity Eligibility Checklist Limited Reviews and NUPAS. USAID, 2018. USAID/Haiti. Méthodologie d’Utilisation du OCA. N.p. USAID, 2015. USAID/Haiti. Konbit Organizational Capacity Assessment Tools and Training Package. N.p. USAID, 2016. USAID/Haiti. Konbit Organizational Capacity Assessment Reports. Np. USAID, 2016. USAID/Haiti. Mid-term Evaluation of the Konbit Cooperative Agreement. The Kaizen Company. 2017. Other Documents and Resources Cummings, S., Ahmed, U., Koopman, M., Magarua, C., and Zielinski, C. Checklist for the Development of Knowledge Hubs/Portals/Websites for International Development. KM4Dev, 2018. United States Department of State. U.S. Government Assistance to Haiti: 2010 to 2015. USDOS, 2014. USAID. Organizational Capacity Assessment Non-Facilitator’s Guide Version With NUPAS Items, USAID, Revised: July 27, 2015. USAID. Fixed Amount Awards to Non-Governmental Organizations: An Additional Help Document for ADS Chapter 303. USAID, 2014. USAID. Non-Governmental Organizations Pre-Award Survey Guidelines and Support: An Additional Help Document for ADS Chapter 303. USAID, 2012. USAID. Gender Equality and Female Empowerment Policy. USAID, 2012. USAID/Haiti. Gender Assessment, Volume 1. Gender Assessment Report. USAID, 2016. USAID/Haiti. Gender Assessment, Volume II. Gender Action Plan. USAID, 2016. USAID/Haiti. Women and Gender Fact Sheet. USAID, 2017. USAID/Haiti. Haiti Democracy, Human Rights, and Governance (DRG) Assessment Final Report. USAID, 2016. USAID/Haiti. Local Solutions: Building Up Haitian Organizations Fact Sheet. USAID, 2017. USAID/Haiti. Private Sector Landscape Assessment. USAID, 2017 USAID/Haiti. Strategic Framework 2018-2020. Undated. DIN website Konbit.ht (https://konbitayiti.globalinnovationexchange.org/) Principles for Digital Development website (https://digitalprinciples.org/) 49| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX I: DATA COLLECTION PROTOCOLS INFORMED CONSENT AGREEMENT - KII Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the USAID/Haiti project known as Konbit. Konbit began in 2015 and will continue until 2020; it is implemented by a consortium led by Papyrus S.A., with the overall goal of strengthening the ability of Haitian organizations to effectively manage funding from USAID and other international donors, including through development of a sustainable cadre of Haitian service providers. You have been asked to participate today so that we can learn more about the support your organization received from Konbit/your involvement in the implementation of Konbit/your experience related to Konbit’s work. We are speaking with about 50 individuals who participated in the program. We would like your honest impressions, opinions and thoughts about various issues related to this Activity’s implementation and outcomes. We are independent consultants who have no affiliation with those who implemented Konbit. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities implemented under Konbit. The interview will take about 1 hour of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Rather the report will include only a composite of all of the answers received by all of the individuals we interview. Although we may use quotes, none of the individuals interviewed will be named in the report. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future capacity building programs in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 50| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV INFORMED CONSENT AGREEMENT – GROUP INTERVIEW Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the USAID/Haiti project known as Konbit. Konbit began in 2015 and will continue until 2020; it is implemented by a consortium led by Papyrus S.A., with the overall goal of strengthening the ability of Haitian organizations to effectively manage funding from USAID and other international donors, including through development of a sustainable cadre of Haitian service providers. You have been invited to participate in this discussion as a representative of an organization funded by USAID and eligible for support from Konbit. We would like your opinions and thoughts about Konbit in general, and more specifically the Konbit activities in which you participated. We are independent consultants who have no affiliation with Konbit. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities of the Konbit program. The discussion will take up to 2 hours of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Although we may use quotes, no individuals will be named in the report. However, as this is a group setting, to preserve confidentiality, we ask you not to share anything we discuss here today with anyone outside of this group. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs in Haiti. Voluntary Participation: Participation in this discussion is completely voluntary. You do not have to agree to be in this study. You are free to leave the discussion at any time or to decline to answer any question which you do not wish to answer. If you decline to participate, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the discussion as outlined above and understand that I can withdraw from the discussion at any time and for any reason. I agree to participate in the discussion (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ PROTOCOL FOR KIIS WITH KEY KONBIT STAFF A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 51| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Questions 1. What have been KONBIT’s greatest achievements to date? (all EQs) Probe: What achievements have been experienced with organizations targeted by Konbit? Please highlight factors that have contributed to achievements. 2. What are the most significant internal or external obstacles that have hindered progress towards KONBIT objectives? (all EQs) Probe: What challenges have been experienced with targeted organizations? What KONBIT strategies and interventions have been less successful than expected? Have any internal organizational factors affected progress? (Ex. within consortium, with USAID, with other USAID project implementers) 3. How would you describe the current capacity of targeted organizations to implement systems and procedures to effectively manage USAID funds? Has KONBIT had a measurable effect on capacity in these areas? If so, how? If not, why not? Was the impact different for awardees and the core CDSPs? (EQ 1) Probe: On a scale of 1 to 5 (5 = able to manage USAID funding in full compliance without difficulties, and 1 = very unlikely to be able to implement in compliance with USAID requirements) how would you rate these organizations at present? (Focusing on M&E, finance, and project management) Note: The ET will present a checklist of awardee names, and ask the informant to indicate the current level of capacity of each awardee (insofar as they are aware), and indicate whether each awardee has either improved, stayed the same, or deteriorated during the period of KONBIT support. Response highlights will then be discussed with the informant. 4. To what extent will any improvements and new systems be sustainable after KONBIT no longer works with the targeted organizations? (EQ 1) Probe: Which organizations are most likely to sustain improvements in capacity and systems, and why? Which types of improvements and systems are more likely to be sustained, and why? What factors may constrain sustainability? 5. Which kinds of support offered by KONBIT have been most useful to targeted organizations (awardees, CDSPs, others)? Which specific types of knowledge and tools have directly helped them to develop their organizational capacity? (EQ 2) 52| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Probe: How do you know that organizations have integrated or adopted certain learning, methods, models, or tools in their systems or their work? Mention types of support as required, and use table to capture this. Tool or Learning Activity How much used by organizations? (1=not at all, 5=used all opportunities) How useful was it to those orgs? (1=not at all, 5= extremely) What effect did this have on their capacity? How likely is that effect to be sustained? (1=already gone, 5=very likely) Training Experts Konbit M&E support (TA) OCA/CAP Helpline (only Mgmt Track) Devt Innovation Network Other (optional) 6. How would you describe the key procedures and tools used by KONBIT for selection and management of grants? What challenges have arisen in the implementation of the grants system? (EQ 3) Use the table below to gather feedback (scoring can be completed by oral questions or passing paper to informant, then discussing the scores), making sure to discuss the reasons for very high or low scores. Aspects of CD Grants Procedure What is your appreciation of the grants process? (1=Very Poor, 5=Excellent) 1 2 3 4 5 Comment Amount of funds on offer Clarity of requirements and procedures Konbit support in application process Konbit negotiation of grant Timing of process Identification of capacity needs via OCA Support after award, inc. monitoring visits 53| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Reporting requirements Overall satisfaction with Konbit grant procedures 7. To what extent have KONBIT activities and approaches integrated gender equality and women’s empowerment? In what ways has KONBIT worked to include and empower both men and women? (EQ 4) Probe: Please provide specific examples and/or reasons for your response (could include targeting of activities, content of tools, training and other major interventions, beneficiary tracking, and reporting by KONBIT.) Have you observed any effect of these strategies and efforts on USAID awardees? What have been the challenges to integration or gender mainstreaming? 8. To what extent has KONBIT coordinated or collaborated with other USAID or US Government funded programs? With other donor-funded programs or government initiatives? (Cross-cutting, EQ 1) Probe: To what extent did KONBIT fill unmet needs and/or complement other assistance programs? Were there any examples of overlap or inconsistencies? 9. What lessons have been learned in the implementation of the KONBIT? What, if anything, could be done to strengthen KONBIT’s approaches and impact? (all EQs) PROTOCOL FOR KIIS WITH INNOVATION GRANT RECIPIENT/APPLICANT A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Questions 1. How and when did you learn about Konbit? What do you see as its objectives? (EQ 2, 3) 2. Has your organization attended any Konbit-supported activities? Which ones? How much did you participate? (EQ 2) (ask them to fill the feedback tool below, then discuss.) 54| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Probe: Are you involved in the Development Innovation Network? In what way? 3. On a scale of 1 to 5, how would you rate the usefulness of those KONBIT activities or interventions? (EQ 2) Probe: Why were certain activities more or less useful? Tool or Learning Activity How much used by your organization? (1=not at all, 5=used all opportunities) How useful was it to your org? (1=not at all, 5= extremely) What effect did this have on your capacity? (mention by subject of training) How likely is that effect to be sustained? (1=already gone, 5=very likely) Training Experts Konbit M&E support (TA) OCA/CAP Helpline (only Mgmt Track) Devt Innovation Network Other (optional) 4. Did you learn or receive anything through those activities or interventions, which you later tried to adopt in your organization and its work? (EQ 2) Probe: What are some examples? If so, what happened as a result? If not, what were the obstacles? 5. How would you describe the application and selection process for the Innovation Grants? (EQ 3) a) How long did the process last, from start to finish? b) (For those not receiving grant) What were the reasons that you did not ultimately get a grant? What were the challenges on your side or on Konbit side? c) What is your perception of the following aspects of the grants procedure? (scoring can be completed by oral questions or passing paper to informant, then discussing the scores) Aspects of Innovation Grants Procedure What is your appreciation of the grants process? (1=Very Poor, 5=Excellent) 1 2 3 4 5 Comment Reasonableness of criteria for eligibility Dissemination of RFA and related information Amount of funds on offer Clarity of purpose of the innovation grants 55| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV Clarity of requirements and procedures Konbit support in application process Timing of process Konbit negotiation of grant Implementation only (grant recipients) Support and demands during implementation, inc. monitoring visits Reporting requirements Overall satisfaction with Konbit grant procedures Comparison to other grant programs in Haiti 6. To what extent have KONBIT activities and approaches integrated or promoted gender equality and women’s empowerment? (EQ 4) Probe: How were these concepts integrated in the grants competition, including applications, scoring, etc.? What more could be done by KONBIT in these areas? 7. How are you integrating (or how did you propose to integrate) gender equality and women’s empowerment in your innovation grant? (EQ 3, 4) 8. How would you describe the overall objectives of Konbit? To what extent do these Innovation Grants contribute to those objectives? 9. What, if anything, could be done to strengthen KONBIT’s approaches and impact? (all EQs) PROTOCOL FOR KIIS WITH DONORS/USAID PROGRAMS A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: 56| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Questions 1. Please describe the role/mission of your organization/program, and your main activities in Haiti in the last three years. 2. Please describe your involvement (if any) with KONBIT. Probe: Have you or your colleagues participated in any activities i.e. training, meeting, workshop, or events organized by KONBIT? How have you interacted with KONBIT personnel during your work in Haiti? Are you aware of the Development Innovation Network or KONBIT grantmaking activities? 3. What is your perception of the work done by KONBIT? (EQ 1, EQ 2) Probe: To what extent is the capacity development support provided by KONBIT relevant to the needs of Haitian organizations (for profit and non-profit)? Which activities appear to be most appropriate to meeting those needs? 4. Which organizations do you know, that have engaged actively with Konbit? Were those organizations also being supported by your project or other sources? Have you observed any change in the capacity of those organizations in project management, financial management, or M&E? What has contributed to those changes? What did you do to build capacity of those organizations, and what did Konbit do? 5. Have you observed any change in the ability of KONBIT targeted organizations to compete for donor funding, or to meet USAID or other donor requirements to qualify for that funding? (EQ 1, EQ 2) Probe: If there has been a change, to what extent has KONBIT contributed to it? To what extent have other initiatives contributed? 6. To what extent did KONBIT coordinate with or complement the work of your organization? What, if anything, could be improved? (EQ 1, EQ 2) 7. Are you aware of the grants that Konbit has been awarding (either CD or innovation)? If so, what is your view of the procedures and tools used by KONBIT for selection and management of grants? Are you aware of any challenges in implementation of the grants? (EQ 3) 8. What do you see as the main challenges for KONBIT to achieve its objectives? How could those challenges be overcome or alleviated as KONBIT continues? 57| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV PROTOCOL FOR KIIS WITH USAID/HAITI STAFF A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Questions (adapted and prioritized according to specific role of person) 1. What is your view of the objectives of Konbit? To what extent has it been able to progress towards those objectives? 2. What have been KONBIT’s key achievements to date? (all EQs) Probe: What results have been achieved with organizations targeted by Konbit? What factors that have contributed to achievements? 3. What are the most significant internal or external obstacles that have hindered progress towards KONBIT objectives? (all EQs) Probe: What challenges have been experienced with organizations targeted by Konbit? What KONBIT strategies and interventions have been less successful than expected? What was the effect of the Annual Program Statement being suspended? Have any internal organizational factors affected progress? (Ex. within consortium, with USAID, with other USAID project implementers) 4. (For informants with general knowledge of Konbit, across multiple awardees. Otherwise focus on the awardee they know.) How would you describe the current capacity of targeted organizations to implement systems and procedures to effectively manage USAID funds? Have you noticed a change in capacity in project management, financial management, and M&E in the past three years? If so, what has been the change? To what extent did Konbit contribute to that change? If not, why not? (EQ 1-2) Probe: On a scale of 1 to 5 (5 = able to manage USAID funding in full compliance without difficulties, and 1 = very unlikely to be able to implement in compliance with USAID requirements) how would you rate the awardees at present? (Focusing on M&E, finance, and project management) Note: The ET will present a checklist of awardee names, and ask the informant to indicate the current level of capacity of each awardee (insofar as they are aware), and indicate whether each awardee has either improved, stayed the same, or deteriorated during the period of KONBIT support. Response highlights will then be discussed with the informant. 58| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 5. To what extent will any improvements and new systems be sustainable after KONBIT no longer works with the targeted organizations? (EQ 1) Probe: Which organizations are most likely to sustain improvements in capacity and systems, and why? Which types of improvements and systems are more likely to be sustained, and why? What factors may be a constraint on sustainability? 6. Which kinds of support offered by KONBIT have been used most by targeted organizations? Which specific types of knowledge and tools have directly helped them to develop their organizational capacity? (EQ 2) Probe: How do you know that organizations have integrated or adopted certain learning, methods, models, or tools in their systems or their work? Use table to capture this. Tool or Learning Activity How much used by organizations? (1=not at all, 5=used all opportunities) How useful was it to those orgs? (1=not at all, 5= extremely) What effect did this have on their capacity? How likely is that effect to be sustained? (1=already gone, 5=very likely) Training Experts Konbit M&E support (TA) OCA/CAP Helpline (MT CDSPs only) Devt Innovation Network Other (optional) 7. How would you describe the key procedures and tools used by KONBIT for selection and management of grants? What challenges have arisen in the implementation of the grants system? (EQ 3) Probe: How well is the system working for the two types of grants (capacity development and innovation)? How effective has the system been in promoting compliance? How effective has the system been ensuring that grants contribute to the KONBIT goal and objectives? 8. To what extent have KONBIT activities and approaches integrated gender equality and women’s empowerment? (EQ 4) Probe: Please provide specific examples and/or reasons for your response. Have you observed any effect of these strategies and efforts on targeted organizations? What have been the challenges to integration or gender mainstreaming? 9. How would you describe the relations or links between KONBIT and other USAID or US Government funded programs? (Cross-cutting, EQ 1) Probe: To what extent did KONBIT fill unmet needs and/or complement other assistance programs? Were there any examples of overlap or inconsistencies? 59| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 10. What lessons have been learned in the implementation of the KONBIT? What, if anything, could be done to strengthen KONBIT’s approaches and impact? (all EQs) PROTOCOL FOR KIIS WITH USAID AWARDEES/CORE CDSPS A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Questions 1. Please provide an overview of the capacity development support you received from KONBIT, and events in which your organization participated. (EQ 1, EQ 2) (refer to list from Konbit database, and confirm with informant) 2. To what extent has the support by KONBIT met the needs of your organization? What factors have contributed to and/or limited KONBIT from meeting those needs? (EQ 1) Probe: Were you satisfied with the quality of KONBIT services? What is your view of the OCA process? Did Konbit support reflect the needs identified in the OCA and CAP? How much progress have you made on the CAP? Has your OCA score improved? Why or why not? 3. Did your organization apply to Konbit for an innovation grant earlier in 2018? (EQ3) If not, why not? Was the RFA adequately disseminated? If yes: a) What is your view of the application and selection process for those grants? b) Was the RFA adequately disseminated? c) Were the requirements clear to applicants? d) Do you have any concerns about how the process was handled? e) Did you get reasons for the refusal of your application? 4. Has your organization applied for or received a capacity development grant from Konbit? (EQ3) If not, why not? If yes: d) How long did the process last, from start to finish? 60| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV e) What is your perception of the following aspects of the CD grants procedure? (scoring can be completed by oral questions or passing paper to informant, then discussing the scores) Aspects of Innovation Grants Procedure What is your appreciation of the grants process? (1=Very Poor, 5=Excellent) 1 2 3 4 5 Comment Reasonableness of criteria for eligibility Amount of funds on offer Clarity of requirements and procedures Konbit support in application process Konbit negotiation of grant Timing of process Identification of capacity needs via OCA Support after award, inc. monitoring visits Reporting requirements Overall satisfaction with Konbit grant procedures 5. To what extent has KONBIT affected your organization’s ability to effectively manage USAID funds? Did KONBIT assist you in developing or implementing systems or procedures to better meet USAID requirements? (EQ 1) Probe: How has KONBIT assisted your organization, especially in areas of M&E, finance, and project management? What exactly has changed? Can you show us an example of a new system or tool being used? 6. In which areas, if any, were improvements not achieved? What are the most significant internal or external obstacles that hindered progress? (EQ 1) Probe: What KONBIT strategies and interventions have been less effective than expected? What factors within your own organization or its environment may affect your ability to effectively manage USAID funding in future? 7. Did you receive other assistance to improve the capacity of your organization, during the same time as KONBIT? For example, support from other donors/USAID partners? If so, to what extent has KONBIT filled gaps and/or complemented that assistance? (EQ 1) Probe: Did the improvements you mentioned earlier flow from KONBIT support or other sources of support, or a combination? 61| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 8. To what extent do you think that improved systems and procedures within your organization will be sustainable? What factors contribute to and/or limit sustainability? (EQ 1) Probe: Which types of improvements and systems are more likely to be sustained, and why? What more could be done to shore up sustainability? 9. How useful has KONBIT support and activities been to your organization? (EQ 2) Use table below to question the informant or have them complete the relevant rows. Tool or Learning Activity How much used by your organization? (1=not at all, 5=used all opportunities) How useful was it to your org? (1=not at all, 5= extremely) What effect did this have on your capacity? (mention by subject of training) How likely is that effect to be sustained? (1=already gone, 5=very likely) Training Experts Konbit M&E support (TA) OCA/CAP Helpline (only Mgmt Track) Devt Innovation Network Other (optional) 10. To what extent have KONBIT activities and approaches integrated gender equality and women’s empowerment? In what ways has KONBIT worked to include and empower both men and women? (EQ 4) Probe: Please provide specific examples and/or reasons for your response (could include targeting of activities, content of tools, training and other major interventions, beneficiary tracking, and reporting by KONBIT.) What have been the challenges to integration or gender mainstreaming? 11. More specifically, to what extent has KONBIT contributed to building awareness and capacity to integrate gender considerations into your organization and its work? (EQ 4) 12. What, if anything, could be done to strengthen KONBIT’s approaches and impact? (all EQs) PROTOCOL FOR GROUP INTERVIEW WITH USAID AWARDEES A. Demographic Info: 1. Date: 2. Location: 3. Primary Facilitator: 62| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 4. Primary Notetaker: Participant List: Name Institution Sex 1. 2. 3. 4. 5. 6. B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask organizer(s) if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. Interview Guide 1. Introductions of the participants. 2. Please provide an overview of the capacity development support you received from KONBIT, and events in which your organization participated. (EQ 1, EQ 2) (Time allowed for participants to complete the table below, which was discussed then collected afterwards for analysis.) Discussion points • Point out general pattern of ratings for each one, and unusually high or low ratings for use and utility – probe into reasons. What was motivation for participating? What were expectations? Tool or Learning Activity How much used by your organization? (1=not at all, 5=used all opportunities) How useful was it to your organization? (1=not at all, 5=extremely) What effect did this have on your organization’s capacity? (especially Financial Management, M&E and Project Management) How likely is that effect to be sustained? (1=not at all, 5=very likely) Please explain why. Training Experts Konbit M&E support (TA) OCA/CAP Networking Events Devt Innovation Network (Konbit.ht) Other _________ 63| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV • If a service was not used much – why not? Were they offered/aware of that service or support? If so, how? Were they aware or invited and decided not to engage? Why not? • If something was used a lot but not very useful nor having effect on capacity or systems – what went wrong? What could be done better? Were there problems with quality of activity? With relevance? Other constraints? • Different effects noted – what are some examples of changes in capacity/systems/ways of working? Or why was there no change in capacity? Is there potential for impact as Konbit continues, or as you continue to operate? How? • General pattern of sustainability, and unusually high or low ratings for sustainability – probe into reasons, challenges, how that could be improved. Which types of improvements and systems are more likely to be sustained, and why? 3. How many of your organizations have had a NUPAS (pre-award survey) or an FAA Review by the Office of Financial Management of USAID? (EQ 1-2) a. How many reviews in the past three years? b. Were you given support by the OFM to ensure that you could meet USAID requirements or conditions? What kind of support? 4. How many of your organizations have done an OCA, supported by OFM or Konbit or another entity? (EQ 1-2) a. What is your view of the OCA process? To what extent did it identify the priority needs of your organization? Was it suitable for your size and type of organization… NGO, business, etc.? b. Was the process similar to the FAA review or NUPAS? If not, how was it different? Were the findings similar? c. Did Konbit support help respond to the needs identified in the OCA or the other review? d. How much progress have you made on the CAP? On your own and with Konbit support? 5. To what extent has KONBIT affected your organization’s ability to effectively manage USAID funds (or other donors)? Did KONBIT assist you in developing or implementing systems or procedures to better meet USAID requirements? (EQ 1) 6. In which areas, if any, were improvements not achieved? What are the most significant internal or external obstacles that hindered progress? (EQ 1) Probe: What factors within your own organization or its environment may affect your ability to effectively manage USAID funding in future? 7. Did you receive other assistance to improve the capacity of your organization, during the same time as KONBIT? If so, to what extent has KONBIT filled gaps and/or complemented that assistance? (EQ 1) Probe: Did the improvements you mentioned earlier flow from KONBIT support or other sources of support, or a combination? 64| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 8. To what extent have KONBIT activities and approaches integrated gender equality and women’s empowerment? (EQ 4) Probe: Please provide specific examples and/or reasons (could include targeting of activities, content of tools, training and other major interventions, beneficiary tracking, and reporting by KONBIT.) 9. Has KONBIT contributed to building awareness and capacity to integrate gender considerations into your organization and its work? (EQ 4) 10. What, if anything, could be done to strengthen KONBIT’s approaches and impact? How could KONBIT help your organization in the next two years? (all EQs 65| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX J: OCA SCORING HIGHLIGHTS This table represents the scores for certain categories of the OCA, for all organizations that have undergone at least one OCA with Konbit. Awardees are listed first, and Core CDSPs afterwards in red font. For those organizations that had more than one OCA, both initial and followup scores are included. Figure 3: OCA Scoring Highlights Organization Year of Initial OCA Year of Followup OCA* Initial overall score (%) Followup overall score (%) Initial score Fin Mgmt Followup score Fin Mgmt Initial score Prog Mgmt Followup score Prog Mgmt Initial score M&E Followup score M&E AGCRDDM 2016 n/a 61 n/a 2 n/a 3 n/a 3 n/a AGRITECH 2016 n/a 58 n/a 1 n/a 2.6 n/a 2 n/a ANACAPH 2018 n/a 67 n/a 2.8 n/a 3.2 n/a 1 n/a ANIMH 2017 n/a 50 n/a 2.4 n/a 2.5 n/a 1 n/a Ayiti Nexus 2016 2018 66 79 2.1 2.8 3 3.5 3 3 CFET 2017 2018 74 79 3.4 3.4 2.2 2.4 2 2 FONHDAD 2016 2018 62 67 1.8 2.2 3 2.4 3 3 FONKOZE 2015 2018 62 81 2.6 2.6 2.4 3.6 2 3 FPDI 2016 n/a 49 n/a 1.8 n/a 1.6 n/a 1 n/a Group Croissance 2017 2018 63 64 3.2 3.2 2.4 2.4 2 2 IDEH 2017 n/a 58 n/a 2.2 n/a 2.7 n/a 2 n/a CEDEL 2017 2018 52 58 1.7 2.4 2.2 2.5 1 1 CEGEC 2016 2018 64 72 2.4 2.8 2.3 2.8 3 3 FHADIMAC 2018 n/a 55 n/a 2 n/a 2.6 n/a 3 n/a IFOS 2018 n/a 64 n/a 2.8 n/a 2.7 n/a 1 n/a *Ayiti Nexus and CEGEC have each had two followups; the first and last OCA scores are used in this table. OCA Scoring Highlights 66| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX K: DISCLOSURE OF CONFLICT OF INTEREST 67| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV 68| KONBIT MID-TERM PERFORMANCE EVALUATION USAID.GOV U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523