24 OCTOBER 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Democracy International, Inc. RWANDA PERFORMANCE EVALUATION OF HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT IN RWANDA PROJECT Submitted to: USAID/Rwanda AID-696-O-16-00016 Prepared by: Melanie Reimer, Team Leader Wilson Kumwenda, Organizational Development Expert, The Kaizen Company Aime Tsinda, Local Evaluation Specialist Contractor: Democracy International, Inc. 7600 Wisconsin Avenue, Suite 1010 Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com PERFORMANCE EVALUATION OF HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT IN RWANDA PROJECT FINAL REPORT 24 OCTOBER 2016 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT TABLE OF CONTENTS EXECUTIVE SUMMARY .................................................................................................................................... I EVALUATION PURPOSE AND QUESTIONS ............................................................................................. 1 EVALUATION PURPOSE ..................................................................................................................................... 1 EVALUATION QUESTIONS.................................................................................................................................. 1 PROJECT BACKGROUND.............................................................................................................................. 2 THE HICD APPROACH......................................................................................................................................... 2 HICD IN THE RWANDA CONTEXT ....................................................................................................................... 4 PROJECT DESCRIPTION...................................................................................................................................... 4 EVALUATION METHODS AND LIMITATIONS ......................................................................................... 8 DOCUMENT REVIEW ......................................................................................................................................... 9 KEY INFORMANT INTERVIEWS........................................................................................................................... 9 FOCUS GROUP DISCUSSIONS........................................................................................................................... 10 DATA ANALYSIS ............................................................................................................................................... 11 LIMITATIONS ................................................................................................................................................... 11 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .................................................................. 12 QUESTION 1 – EFFECTS ON PERFORMANCE .................................................................................................... 12 QUESTION 2 – FACTORS AFFECTING CAPACITY DEVELOPMENT...................................................................... 21 QUESTION 3 – CSO AND GOVERNMENT NEEDS............................................................................................... 26 QUESTION 4 – CAPACITY AND NEEDS OF RSPI AND MEMBERS ........................................................................ 37 QUESTION 5 – PROJECT IMPLEMENTATION AND MANAGEMENT................................................................... 43 ANNEX A: STATEMENT OF WORK .......................................................................................................... 55 ANNEX B: EVALUATION WORK PLAN................................................................................................... 64 ANNEX C: DATA COLLECTION TOOLS ................................................................................................ 75 ANNEX D: LIST OF PERSONS INTERVIEWED ........................................................................................ 97 ANNEX E: DOCUMENTS REVIEWED......................................................................................................100 ANNEX F: DATA SYNTHESIS TABLE FORMAT ....................................................................................102 ANNEX G: PMP INDICATOR DATA TABLE ..........................................................................................103 ANNEX H: CONFLICT OF INTEREST STATEMENTS ..........................................................................105 ANNEX I: PROJECT SCOPE OF WORK EXCERPT...............................................................................110 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT ACKNOWLEDGEMENTS This evaluation would not have been possible without the contributions of the current and former staff of USAID/Rwanda and the key implementers of the HICD/R Project, whether in Kigali, Washington DC, or on assignment elsewhere in the world. In addition to patiently and candidly answering many questions and sharing numerous documents, they helped to link us with the right people to ensure we had a balanced overall perspective of the Project and its context, and provided valuable feedback during the analysis and report-writing stages. Of course, many other informants in Rwanda, from civil society, government and other sectors, gave their time freely to participate in in-person interviews and focus group discussions. The participation of each and every individual added value to the evaluation, which we hope will ultimately contribute to future initiatives focused on performance improvement and related subjects, in Rwanda and beyond. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT ACRONYMS ADRA Adventist Development and Relief Agency AEE African Evangelical Enterprise CCOAIB Conseil de Concertation des Organisations d’Appui aux Initiatives de Base COMPRAC Community of Practice COP Chief of Party CPT Certified Performance Technologist CSO Civil Society Organization DAI Development Alternatives Incorporated DI Democracy International DG Democracy and Governance EDPRS Economic Development Poverty Reduction Strategy FGD Focus Group Discussion FORECAST Focus On Results: Enhancing Capacity Across Sectors In Transition FXB François-Xavier Bagnoud GOR Government of Rwanda HICD Human and Institutional Capacity Development HICD/R Human and Institutional Capacity Development/Rwanda HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HPT Human Performance Technology IPAR Institute of Policy Analysis and Research ISPI International Society for Performance Improvement IT Information technology KII Key Informant Interviews KNO Kelly Novak Opportunities NAR Never Again Rwanda M&E Monitoring and Evaluation MINAGRI Ministry of Agriculture and Animal Resources MINALOC Ministry of Local Government MINECOFIN Ministry of Finance and Economic Planning MoU Memorandum of Understanding NAEB National Agriculture Export Board NCBS National Capacity Building Secretariat NGO Non-governmental organization NISR National Institute of Statistics Rwanda NWC National Women’s Council OAG Office of Auditor-General for State Finances OCI Organizational Capacity Index PDIA Problem Driven Iterative Adaptation PMP Performance Monitoring Plan PSP Performance Solutions Package PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT RAB Rwanda Agriculture Board RFP Request for Proposals RMI Rwanda Management Institute RSPI Rwanda Society for Performance Improvement SFH Society for Family Health SOW Scope of Work TOR Terms of Reference U.S. United States USAID United States Agency for International Development YWCA Young Women’s Christian Association PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT i EXECUTIVE SUMMARY Project Background Human and Institutional Capacity Development (HICD) is a model of structured and integrated processes designed by USAID to support systemic improved performance in its partner institutions, including ministries, civil society organizations (CSOs), and other host-country organizations. The HICD approach advocates a holistic focus on organizational and individual performance, based on comprehensive assessments followed by a variety of interventions to improve performance in priority areas, which are described in a Performance Solutions Package (PSP) for each targeted institution. USAID has produced a handbook that describes the HICD approach in detail, and provides detailed guidance on its utilization. The HICD/R Project is a five-year US$14.85 million initiative, launched in April 2012 with the goal of increasing overall performance of Government of Rwanda (GOR) institutions and CSOs in a manner that fosters sustainability and improves Rwanda's overall development outcomes. It is managed within the Democracy and Governance (DG) Office but is cross-sectoral, also receiving funding from the Economic Growth, Education and Health offices of USAID/Rwanda. The implementing partner is Development Alternatives Incorporated (DAI). Project implementation is organized around three components: 1. Government Institutions: HICD-based assessments and performance improvement support to three selected GOR counterparts: the Ministry of Agriculture and Animal Resources (MINAGRI), National Capacity Building Secretariat (NCBS) and National Women’s Council. 2. Civil Society Organizations: HICD-based assessments and support to selected CSOs. The Project focused assistance on six CSOs between 2012 and 2014; one CSO was added in 2014, and three more joined the program in 2015, for a total of 10 CSOs working in different sectors. 3. Rwanda Society for Performance Improvement (RSPI): Development of a community of practice for those interested in the field of performance improvement. Originally aimed at consultants, the group broadened its scope and with Project support became the RSPI. HICD/R has provided training, technical assistance, and event sponsorship to RSPI and its members, including intensive training and coaching for members applying for the Certified Performance Technologist (CPT) designation. Evaluation Purpose and Methods This performance evaluation has been carried out during the final year of the Project. The purpose is to assess progress by the Project towards its objectives, and document the effectiveness of Project approaches and responsiveness to overall capacity building needs in Rwanda. The findings will be used by USAID/Rwanda and the implementer to strengthen implementation over the remaining life of the Project, and will help USAID/Rwanda determine what steps to take to sustain the Project’s achievements. The Evaluation Team (“the Team”) used a mixed-method approach that relied on both quantitative data (primarily Project monitoring data) and qualitative data, primarily collected from Project and other documents, and through key informant interviews and focus group discussions. The Team conducted 50 key informant interviews in Kigali and remotely, and four focus group discussions with people from civil society, government, USAID and RSPI. In all, 75 informants were directly consulted by the Team. The most important limitation was posed by the gaps in the monitoring data gathered by the implementer. Key Conclusions and Recommendations PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT ii Question 1: To what extent have the Project’s specific interventions improved the management and technical performance of the targeted Rwandan institutions and CSOs? Overall, the management capacity of targeted institutions has been enhanced with Project assistance, through various mechanisms and in diverse areas of activity. Knowledge testing of participants before and after training and the adoption of more than 450 PSP recommendations by the 13 targeted institutions both indicate advances in capacity, as do results of the annual impact perception survey carried out with those institutions. Some changes flowing from the recommendations have been fully implemented by targeted institutions with Project support, but various new skills and systems are not yet firmly entrenched. Also, nearly half of all PSP recommendations have not yet been implemented, indicating that multiple areas of need continue to exist (although this data is still being updated by the Project). Specifically with respect to targeted CSOs, progress has clearly been made towards establishment of an impressive list of policies, systems and strategies; change is less visible when considering the performance improvement that was intended to result from these investments in capacity, although there is some evidence of impact on that level. With respect to government, the Project helped develop certain capacities at human and institutional levels in both MINAGRI and NCBS. At MINAGRI, the best results were in planning and budgeting, although it was difficult to clearly attribute performance improvement to the Project. Significant challenges were posed by quality and management issues related to several consultants embedded in the Ministry. In the case of NCBS, the evaluators found changes in systems and ways of working that hold significant promise, but reforms are still fragile and subject to staff turnover, and certain tools have fallen into disuse. Clearly, it is a long term process to translate learning (on individual level) and introduction of new systems, plans and tools (on institutional level) into performance change that can be observed and measured. In spite of a reasonable amount of data on capacity change, it was difficult to comprehensively answer Question 1 because the Project Performance Monitoring Plan was very limited in its measurement of performance improvement, and neither the implementer nor the targeted institutions took responsibility for establishment and monitoring of indicators to measure performance at the level of each institution. As well, due to a heavy focus on management performance, the Project had little observable effect on technical capacity or technical performance of the targeted institutions. Question 1 Recommendations:  DAI should carry out an immediate re-assessment of key areas of each targeted institution, to see what changes in capacity and performance have occurred and been maintained, including examination of both management and technical performance. USAID technical office staff should participate to help identify both changes and ongoing needs within their partners, especially in relation to technical performance.  USAID and DAI should devote remaining Project resources to continued assistance to ensure sustainability of outcomes in each institution, rather than attempting to target new institutions.  In future, USAID and implementers should ensure that targeted institutions are fully aware of results monitoring as an obligation in return for performance improvement support.  USAID and implementers should use caution in the future engagement of embedded consultants, whose utility can easily be undermined if not carefully controlled. Question 2: What key factors may have hindered or aided the achievement of capacity building activities within CSOs vs. government institutions? The evaluation found considerable differences in working with government and civil society to develop capacity, notably in levels of motivation to engage in the Project and embrace the HICD approach to improve performance. First and foremost, cooperation with MINAGRI (and GOR more generally) was undermined to a great extent by the U.S. government’s decision not to proceed with direct budget PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT iii support of the GOR, which coincided with the launch of this Project and hindered buy-in at senior levels. As well, government personnel expectations of incentives to engage in training posed challenges in attracting committed participants. However, there were more points in common between CSOs and government than differences. Shared challenges included the generally low absorptive capacity of line personnel and constraints on staffing numbers, which underscored the need for long-term systemic support to ensure change is sustainable. Informants from both MINAGRI and CSOs commented that the Project approach to their needs felt somewhat supply-driven, especially with respect to identification of priority areas for support by the Project and promotion of certain solutions. It is likely that this perception among CSOs was partly based on the fact that Project technical assistance alone had not proven sufficient to fully resolve their diverse performance issues. On the other hand, both government and CSOs valued the participatory approaches of the Project and the high quality of most technical assistance consultants, although staff turnover in the DAI team did pose some challenges. The Team’s recommendations related to these conclusions are captured under Questions 1 and 5. Question 3: To what extent are Rwandan institutions and CSOs still in need of capacity development support that should be addressed in a future Project in order to leverage and sustain USAID/Rwanda investments?1 The evaluation found a high level of ongoing needs among CSOs; the key areas of capacity gaps are advocacy, resource mobilization, monitoring and evaluation (M&E), networking and collaboration. Funding shortfalls are significant, especially for organizations that wish to do more than service delivery, and/or are working in thematic or geographic areas that are not clearly in line with government priorities. There are increasing indications that external support to a stronger voice for civil society may not be condoned by the current government, which presents special challenges for effectively supporting the sector. The most significant capacity gaps in government are at the decentralized level, and by extension in the Ministry of Local Government. At the central level, the cross-cutting priority in GOR according to external observers is better harmonization and coordination, within and among institutions. Monitoring, linking data collection to policy, and knowledge management were also highlighted by donors, while GOR sectoral working groups prioritized a diverse set of more technical needs related to service delivery. The notable overlap between the needs identified by the GOR and external informants was M&E. The inherent focus of HICD on periodic assessment of performance and close monitoring of results appears to be consistent with the Imihigo system of government performance contracts, and in that sense well suited to the context. Due to high staff turnover in both government and CSOs, an approach that pays close attention to systems, tools and the institution itself is important as a counter-balance to investment in individual capacities. However, other models of organizational development could also add value; in particular, HICD could be used in combination with other methods or adapted to ensure that targeted CSOs have enough absorptive and operational capacity to benefit from assistance, and to incorporate a greater emphasis on the operating environment and inter-organizational connections. As for the GOR, although this Project’s experience has been mixed, the open-ended approach of HICD (with its reliance on a wide-ranging initial assessment of the entire targeted institution to identify performance gaps) may not be a particularly good fit. There was general agreement that the leadership 1 USAID/Rwanda agreed to limit the scope of enquiry for this question, considering the timeframe for the evaluation. The Team thus focused on unmet needs for capacity development of selected government institutions and CSOs, to the extent those could be determined from a defined number of sources, then looked at whether HICD approaches may be appropriate for meeting those needs. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT iv of a government entity must have a clear idea of the problem to be solved by a donor-funded project in order to gain traction. Question 3 Recommendations:  USAID should engage with various actors to inform future activities to develop CSO capacity, seeking collaboration whenever possible and exploring ways to package CSO assistance that can be effective in the current political environment, including close integration with technical programming in non-DG areas.  USAID should consult and research further with other donors and implementers, staff of technical offices, and the GOR to inform targeting and methods for future efforts to develop capacity of government institutions.  USAID should consider additional investment in HICD approaches in Rwanda but also be open to other models for performance improvement; a flexible approach to HICD should be adopted, tailoring it to each organization, the priorities for improvement, and resources available. Question 4: What capacity development support does the RSPI need to sustainably continue HICD interventions beyond the life of the Project, both with USAID/Rwanda financial support in the near term, and without USAID/Rwanda financial support in the long term? How long will it take to build such capacity? Relying on Project support, the RSPI has helped to create awareness of the HICD approach in Rwanda, and there are high expectations and some positive energy around performance improvement, notably after the recent conference in Kigali. However, RSPI is young and has been run by DAI staff until now, so the genuine commitment of its members and leaders remains in doubt. RSPI may well not survive long after the Project phases out; its fate will be in the hands of its members, who must themselves determine whether the organization has enough value to invest their own time and resources in its development. Even though it may not remain active, RSPI did play a role in the Project as a platform for sharing information about HICD, and connections made among professionals interested in performance improvement may continue to bear fruit. The Project provided a number of professional development initiatives for COMPRAC/RSPI members, aimed at increasing technical knowledge and strengthening communities of practice. However, the level of capacity in performance improvement methodologies has remained relatively low. Considering the resources required to achieve the certification of 15 persons, the return on investment in CPT seems doubtful—although the value of an international certification in attracting the interest of learners cannot be denied. The RSPI members came from a low starting point and still need training, coaching and on￾the-job experience. Although the initial experience with RSPI members working alongside international consultants as paid interns was frustrating, it appears that a very important strategy for professional capacity development was dropped very quickly by the implementer. Question 4 Recommendations:  DAI should continue to provide modest funding for activities of RSPI, including regular awareness￾raising, networking and training events. However, USAID and DAI should not provide funding for an office or staff for RSPI, as that would perpetuate the dependent relationship.  DAI should work with the executive to convene a meeting to explore and analyze strategies by which RSPI can be sustained, and support an RSPI assembly before the end of 2016, where the future can be discussed and the strategic plan finalized. Prior to that event, DAI and USAID should communicate to all RSPI members their intentions regarding support following the end of the Project.  DAI and USAID should provide more opportunities for motivated RSPI members to deepen their experience through twinning or mentoring arrangements with experts. USAID should support involvement of selected members in such arrangements related to other projects in Rwanda. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT v  DAI should sponsor more forums and events where RSPI members and others interested in performance improvement can exchange experiences and learn from others in a non-competitive and collegial environment. Question 5: What lessons learned about Project management and implementation can inform project management for the remainder of the Project life and future potential follow-on designs? In spite of considerable efforts, multiple factors combined to undermine the cross-sectoral nature of the Project, which contributed to lack of understanding of the model and its results within USAID/Rwanda. While trying to maintain the “DG flavor” of the Project after the DG Office budget was cut, it proved difficult to accommodate the interests of technical offices and sustain their engagement, and ultimately results were diluted by these mixed motives. The implementer’s high staff turnover made it difficult to solidify relations with counterparts in government and civil society, and to develop essential skills and knowledge within the team, in spite of ongoing mentoring and other support by senior DAI staff and consultants. The starting point for relevant capacities was low, given the lack of any previous HICD activity in Rwanda. Frequent changes of key government personnel (and to a lesser extent, CSO staff) also posed a significant barrier to sustainable performance improvement. The Project’s international consultants were largely recognized as qualified experts and good communicators. However, their unfamiliarity with the context and the languages most widely spoken in Rwanda were obstacles, continuity of support was compromised, and associated costs were high. With respect to material support for targeted institutions, grants under contract and direct financial support were not included in the Project design documents or contract, although support with some types of “commodities” was mentioned. The only such support provided was some software. The lack of material and financial assistance for CSOs constrained their progress in certain areas of performance where technical assistance alone was simply not sufficient to meet needs, and may have affected motivation or the priority given by CSOs to Project-related activities. It was also found that longer duration support at lower intensity can be more valuable when using holistic HICD approaches, especially for smaller organizations. This is especially important to deal with unexpected institutional changes, to ensure that reforms survive and momentum continues. DAI appears to have made a management decision not to enter into more than one Memorandum of Understanding (MoU) with partners, although the duration and scope of the initial MoUs was not sufficient to cover the actual collaboration. The lack of clearly defined obligations affected the commitment of CSOs and may have contributed to gaps in monitoring. The Evaluation Team concurs with one key Project consultant: “Multiple MoUs are an administrative burden and often time￾consuming, but I still think provide good value.” The Project faced various challenges in development and use of robust monitoring tools, as well as selection of suitable indicators and setting of realistic targets; these difficulties have also been observed with similar programming in other countries. The most useful data about an HICD project is often derived from monitoring change in targeted institutions, but there was no shared understanding of who would monitor those results and this important component of the HICD approach was neglected. Even though several M&E experts assisted institutions in this area, the resulting M&E plans for CSOs do not have indicators on management performance that could measure progress towards direct outcomes expected by this Project. The decision not to monitor these indicators appears to be inconsistent with the terms of the Scope of Work for the Project. Question 5 Recommendations: Current Project:  The DG office should liaise regularly with other contributing offices in USAID to update them on the Project and explore options for assistance to targeted institutions after the end of the Project. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT vi  DAI should ensure that indicators related to management performance improvement are included in M&E systems of targeted institutions, and analyze whether Project support to management performance has had an effect on technical performance. Future programming:  USAID should base decisions about future HICD interventions on an in-depth analysis of the priorities of the mission, the evolving context, and the budgets and personnel available to different offices in the mission. Relevant staff from technical offices should be involved in performance assessments, to ensure that chosen priorities are consistent with overarching objectives in that sector.  USAID should design project staffing with due consideration for the relatively low level of expertise in HICD and organizational development in Rwanda; inclusion of long term experts on staff may be advisable. When possible, consultants and key personnel should speak French and/or Kinyarwanda as well as English, and budgets should allow for translation of key documents to Kinyarwanda.  USAID and implementers should maintain a more flexible approach to provision of material or financial support to targeted CSOs, to be able to respond to situations where progress toward improved performance requires resources that are not otherwise available to those organizations As USAID considers future capacity development interventions, financial support and the duration of support should be taken into consideration.  USAID and implementers should prioritize strong M&E systems, tools and follow-through as a core element of any performance improvement initiative. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 1 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this performance evaluation is to assess progress that the Human and Institutional Capacity Development in Rwanda (HICD/R) Project has made to date in achieving its objectives and document the effectiveness of the Project’s approaches and responsiveness to overall capacity building needs in Rwanda. The evaluation aims to help USAID/Rwanda identify key technical and management systems strengths, weaknesses, opportunities, and threats in the implementation of the HICD/R Project (hereafter, “the Project”). The findings will be used by USAID/Rwanda and the implementer to develop and support activities to strengthen the Project’s implementation in order to improve performance over the remaining life of the Project, and help USAID/Rwanda determine what steps to take to sustain the Project’s achievements. More specifically, this evaluation expected to assist USAID/Rwanda in identifying those aspects of the Project that have been particularly effective or ineffective and provide guidance on changes that can be made to improve its sustainability. The evaluation assessed Project implementation from April 2012 to June 2016.2 The evaluation makes conclusions based on findings, identifies opportunities and challenges, and formulates recommendations for improvements across existing program activities. The primary audience for this evaluation is the USAID/Rwanda Democracy & Governance Office. Other users include the Health, Economic Growth and Education offices which have all contributed funds to implement the Project in ways that benefit their portfolios. The report may also be of interest to a broader audience of USAID stakeholders as they continue to roll out the HICD approach and learn from global experience of its implementation. EVALUATION QUESTIONS The Evaluation Team (hereafter, “the Team”) was asked to research a specific set of evaluation questions, within the framework of the overall purpose set out above. The questions posed in the Scope of Work by USAID/Rwanda were defined and their scope clarified by the Team in consultation with USAID personnel, in advance of data collection. The evaluation questions and corresponding definitions are set out in Table 1 below. Table 1: Definition of Evaluation Questions Evaluation Question Definition and Scope 1. To what extent have the Project’s specific interventions improved the management and technical performance of the targeted Rwandan institutions and CSOs? Evaluation will focus on changes that have occurred during the period of participation in the Project, in both management and technical areas that were identified and targeted with significant support by the Project. The specific focus of this question will be on performance of the institutions and CSOs, as compared to capacity, which is addressed by Question 3. Targeted institutions are: Ministry of Agriculture and Animal Husbandry (MINAGRI), National Capacity Building Secretariat (NCBS) and 10 CSOs. 2. What key factors may have hindered or The evaluation will consider both internal factors (arising within 2 The period of evaluation originally ended in April 2016, but was later extended to June 2016 by contract modification to ensure a more complete coverage of project implementation. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 2 aided the achievement of capacity building activities within CSOs vs. government institutions? the targeted institution, the Project itself or USAID, including those related to use of the HICD methodology) and external factors (political, security, economic, etc.) The evaluation will seek to differentiate factors that affected capacity building of CSOs as compared to government institutions. 3. To what extent are Rwandan institutions and CSOs still in need of capacity development support that should be addressed in a future Project in order to leverage and sustain USAID/Rwanda investments? Evaluation will make a limited investigation of the extent to which selected government institutions and CSOs have needs for capacity development that are not being met, and whether in general the HICD methodology may be appropriate for meeting those needs. The selected government institutions are: Ministry of Justice, Supreme Court of Rwanda, Ministry of Local Government, Ministry of Education, Ministry of Health, Office of the Auditor General of State Finances, and Rwanda Bio Medical Center. The type of CSOs to be studied are those defined by the European Union 2013 Civil Society Mapping as “second level” and “third level” organizations. This question will not analyze the capacity needs of institutions already targeted by the Project (as agreed with USAID/Rwanda), except to the extent that certain CSOs may be part of the categories of CSO being studied. 4. What capacity development support does the RSPI need to sustainably continue HICD interventions beyond the life of the Project, both with USAID/Rwanda financial support in the near term, and without USAID/Rwanda financial support in the long term? How long will it take to build such capacity? Evaluation will examine the needs of the Rwanda Society for Performance Improvement (RSPI) as an organization in the following areas, which have been identified as the most relevant for the purposes of this evaluation: governance, financial sustainability, member services, and marketing/communications. The Team will also assess whether sufficient capacity exists among RSPI members to continue to competently provide HICD interventions after the Project. In both cases, the evaluation will consider the role of USAID/Rwanda in supporting development of capacity in the short and longer term. We will also consider how long it may take to achieve that level of capacity. 5. What lessons learned about Project management and implementation can inform project management for the remainder of the Project life and future potential follow-on designs? "Project management and implementation" are defined to include lessons related to the Project's human resources, external relations and coordination, selection of targeted institutions, and other relevant issues. PROJECT BACKGROUND THE HICD APPROACH HICD is a model of structured and integrated processes designed by USAID to support systemic improved performance in USAID Partner institutions, including government ministries, civil society organizations, and other entities. This is done by identifying root causes of performance gaps and addressing those gaps through a wide array of interventions. The model is designed to assist USAID partners with practical methodologies and tools to clarify and fulfill their mandates and provide the highest level of service to constituents and stakeholders. The model draws on methods and tools from the field of Human Performance Technology (HPT), which focuses on three levels of performance variables that ultimately determine the performance of organizations and individuals: the broader organization, critical work processes, and the individual job/performer. This evidence-based model, PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 3 depicted graphically in Figure 1 below, focuses on creating measurable performance improvement through a seven-step process: 1. Define optimal or desired performance with stakeholders 2. Define actual performance with stakeholders 3. Define performance gap in measurable terms 4. Identify root causes of performance gaps 5. Develop a set of recommendations—called a Performance Solutions Package (PSP)—to address root causes of gaps 6. Implement PSP recommendations 7. Measure performance change USAID has produced a handbook that describes the HICD approach in detail, and provides detailed guidance on its utilization, including links to tools and examples.3 The approaches of the HICD/R Project were based on that handbook, as specified by the contract between USAID and the implementer. As described in that Handbook, “The ultimate goal of HICD is to help USAID’s partners improve performance in critical areas leading to measurable results in achieving the organization’s goals and objectives. In undertaking HICD initiatives, USAID Missions will strengthen their partner organizations’ abilities to more effectively perform for their constituents and stakeholders and will increase the effectiveness of ongoing technical assistance provided by the United States Government and other International Donors.”4 Figure 1. HICD Performance Improvement Model5 3 Human and Institutional Capacity Development Handbook, October 2010, available at: pdf.usaid.gov/pdf_docs/PNADT442.pdf. For additional information, the HICD Policy Paper, A Mandatory Reference to ADS 201, can be found through the ADS site at: http://www.usaid.gov/policy/ads/200/201maf.pdf 4 HICD Handbook, page 7. 5 Source: HICD Handbook, page 8. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 4 HICD IN THE RWANDA CONTEXT6 The Government of Rwanda’s (GOR) commitment to effective capacity development and its reforms to promote capacity improvement were seen as an opportunity for USAID/Rwanda to collaborate with the government to ensure that public administration reform is sustainable and self-improving. The HICD model complements the GOR’s current reform activities and capacity building tools.7 Using the model, a thorough and comprehensive assessment of the performance of targeted institutions is conducted so that performance gaps can be identified and appropriately addressed through a variety of targeted interventions. Ongoing monitoring and evaluation aims to ensure that solutions can be modified to changing needs or conditions and that interventions incorporate best practices. One often hears that Rwanda is a “country in a hurry,” making up for lost time and opportunity. At the same time, the country’s institutions are still relatively new and successive reforms have left many ministries with a dearth of staff and a lack of understanding of the ministry’s vision and mission. New personnel may not receive sufficient training or orientation to understand their roles and responsibilities. The most competent people may be called to fill the role of several staff members, whose positions have not yet been hired, or to do additional ad hoc tasks on any given day. The frequent turnover of staff and ministry leadership has been a consistent feature of the Rwandan bureaucracy, exacerbating these problems. On the positive side, many personnel are young, highly committed, and willing to work long hours. Among some staff, however, the long hours and enormous responsibilities are taking a toll. Overall, Rwanda presents a unique environment for an HICD program. The government is very receptive to institutional strengthening and capacity development, as highlighted in Rwandan strategy documents and plans. There is general understanding by GOR officials of the meaning of performance improvement, something often missing in other countries and contexts. Capacity development is also a cross-cutting priority for USAID/Rwanda, with all technical teams dedicating resources to enhancing the capacity of targeted government institutions and civil society organizations. PROJECT DESCRIPTION The HICD/R Project is a five-year initiative, which began in April 2012 and is due to end in April 2017. The purpose of this US$14.85 million contract is defined in the Scope of Work as: to strengthen institutional and human capacity in targeted GOR institutions, as well as in targeted civil society organizations, that work closely with or support the objectives of USAlD/Rwanda. The goal of the Project is to increase overall performance of GOR institutions and Rwandan CSOs in a manner that fosters sustainability and improves Rwanda's overall development outcomes. This is USAID/Rwanda’s first HICD project. It is managed within the Democracy and Governance Office, but is cross-sectoral, receiving funding from the USAID/Rwanda Democracy and Governance, Economic Growth, Education and Health offices. The implementing partner is Development Alternatives Incorporated (DAI), which is responsible for management and execution of all aspects of HICD/R. The HICD/R subcontractor, Kelly Novak Opportunities (KNO), is responsible for providing technical assistance in conducting HICD assessments and in developing PSPs.8 Project implementation is organized around three components:9 6 Adapted from HICD/R Project Scope of Work, 2012. Page 14. 7 The HICD model closely resembles the new capacity building framework being developed by the Public Sector Capacity Building Secretariat. While there may be differences between the two approaches, the HICD model will closely follow government initiatives in this area. 8 The KNO subcontract with DAI had a maximum value of US$1.5 million. 9 Source: Scope of Work for HICD/R Evaluation, page C-1. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 5 1. Government Institutions: Providing focused, HICD-based assessments and PSPs to selected Government of Rwanda (GOR) counterparts and assisting in PSP implementation. During year three, the Project concluded two years of assistance to MINAGRI and completed a performance analysis of the NCBS10. Year 4 focused on implementing a PSP for NCBS, adding the National Women’s Council as a targeted institution, and also planned to institutionalize a performance improvement curriculum developed with the Rwanda Management Institute (RMI). 2. Civil Society Organizations: Providing focused, HICD-based assessments and PSPs to selected CSO counterparts and then assisting in PSP implementation. The Project concluded assistance to the six CSOs in the first group of organizations selected to participate, namely: Ingabo, Imbaraga, Conseil de Concertation des Organisations d’appui aux Initiatives de Base (CCOAIB), Francois Xavier Bagnoud (FXB), Caritas, and African Evangelical Enterprise (AEE). Society for Family Health was targeted from 2014 onwards, and three additional CSOs (Never Again Rwanda, Adventist Development and Relief Agency, and Young Women’s Christian Association) were added to the program in 2015. Year 4 focused on implementing PSPs for the CSOs targeted since 2014, as well as some follow-up support to the first round CSOs. 3. Rwanda Society for Performance Improvement (RSPI): Developing a community of practice for those interested in learning from and contributing to the body of performance improvement knowledge in Rwanda. Originally called COMPRAC and aimed only at consultants, the component evolved to target professional managers from any sector and academics, and transitioned with Project support to become the RSPI, a chapter of the International Society for Performance Improvement (ISPI). HICD/R provides training, technical assistance, and event sponsorship to RSPI and its members. The Project has provided HICD training, paid internships, and coaching for members applying for the Certified Performance Technologist (CPT) designation awarded by the ISPI.11 According to its Scope of Work, the Project focuses on enabling targeted government institutions and CSOs to achieve the following objectives as the building blocks for effective organizations and institutions: • Have a clear sense of purpose and vision; • Align their organizational structures and efforts with their vision and purpose; • Perform their work effectively and efficiently; • Motivate personnel by ensuring that they have the resources and support necessary to do their jobs well; • Coordinate with other organizations and institutions to avoid duplication and ensure that they work collaboratively when seeking to achieve the same goal and objectives; • Strengthen internal and external communications; and • Function as “learning organizations” that are more resilient and able to adapt to changing circumstances and challenges. A full listing of targeted institutions is provided in Table 2 below. 10 Previously the Public Sector Capacity Building Secretariat 11 The ISPI is the professional association promoting the application of Human Performance Technology in all spheres of action. The performance-based certification of CPT ensures uniform standards and codes of ethics in practice. USAID recognizes this credential and emphasizes use of these experts to lead assessment teams and HICD-focused projects. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 6 Table 2: HICD/R Targeted Institutions Institution Name Primary Sector of Activity Start of HICD/R Support Brief Description of Institution Government Institutions12 1. Ministry of Agriculture and Animal Resources (MINAGRI) Agriculture 2012 The Ministry’s mission is to develop and manage suitable programs of transformation and modernization of agriculture and livestock to ensure food security and to contribute to the national economy. The Ministry oversees the development and implementation of policy of to increase animal and crop production, modernize farming, reduce poverty, ensure food security and have surplus for the market. The Rwanda Agriculture Board and National Agricultural Export Board are affiliated government agencies. 2. National Capacity Building Secretariat (NCBS) Various 2014 The NCBS was established to coordinate capacity building efforts in Rwanda across the public, private, and civil society sectors, in alignment with national priorities outlined in the Economic Development Poverty Reduction Strategy (EDPRS II) and Vision 2020. NCBS is focusing its coordination efforts through sector representatives to achieve its expanded mission (prior to 2013, it targeted only the public sector). It operates under the auspices of the Ministry of Economic Planning and Finance. 3. National Women’s Council (NWC) Women 2016 Established as an autonomous institution in 2003 on the base of a pre-existing structure, the NWC is mandated to be a forum for advocacy and mobilization on issues affecting women in order to build their capacity and ensure their participation in the national development and the development of women in particular. Civil Society Organizations 4. Ingabo Agriculture 2012 Ingabo is a farmers’ organization operating in four districts in Rwanda. Its goal is to promote professional farming and farmer welfare through contributions to the value chain and protection of farmer interests. Founded in 1992 and registered in 2005 as a trade union, the organization is currently registered as NGO, reflecting their mandate as well as local NGO law. Ingabo has not received any significant funding support from USAID. 5. Imbaraga Agriculture 2012 Imbaraga is a local NGO representing more than 25,000 Rwandan subsistence farmers since 1992. Imbaraga’s mission is to increase farmer professionalism through capacity building for productivity, competitiveness and protection of farmers’ social economic rights. Imbaraga has not received any significant funding from USAID, although it benefited from technical assistance under the LAND project, begun in 2012. 12 Note: The Rwanda Governance Board was identified as a potential targeted institution by the Project, but USAID decided not to proceed with the collaboration shortly after the performance assessment began. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 7 Institution Name Primary Sector of Activity Start of HICD/R Support Brief Description of Institution 6. African Evangelical Enterprise (AEE) Health 2013 AEE is a faith-based NGO working on improving livelihoods of vulnerable populations since 1987. AEE was founded by the international entity known as African Enterprise, a Christian, interdenominational organization whose mission is to “evangelize the cities of Africa through word and deed.” AEE￾Rwanda defines their target population as “vulnerable groups, including orphans, widows, people with chronic diseases, victims of natural calamities, prisoners, sex workers, etc.” AEE began to receive direct funding from USAID in 2012, with a grant of US$8 million for their work with orphans and vulnerable children. 7. Caritas Rwanda Health 2013 Caritas is a non-profit faith-based organization providing services to vulnerable communities across all 30 districts of Rwanda. Established in 1963, Caritas Rwanda’s mission is to build a better world by serving the poor and oppressed and giving voice to the socially marginalized. They have a particular focus on healthcare. Caritas began to receive direct funding from USAID in 2012, with a grant of US$14 million to support work on nutrition and HIV/AIDS mitigation. 8. François Xavier Bagnoud (FXB) Rwanda Health 2013 FXB Rwanda is a non-governmental organization that was founded in 2012 as the Rwandan arm of FXB International. The overall mission is to fight poverty and AIDS and support the world’s orphans and vulnerable children by advocating for their needs and basic rights and providing direct support to families and communities. FXB began to receive direct funding from USAID in 2012, with a grant of US$7 million for an integrated livelihoods project. 9. Conseil de Concertation des Organisations d’Appui aux Initiatives de Base (CCOAIB) Agriculture and poverty reduction 2013 CCOAIB is an umbrella organization of 40 Rwandan CSOs that all promote sustainable development, with a focus on poverty reduction and agricultural development. Established in 1987, it envisions full citizen participation in development, and champions this by making government policy and budget documents accessible to the public, and by representing member organizations’ interests to government officials at the national and district level. CCOAIB has not received any significant funding from USAID. 10. Society for Family Health Rwanda (SFH) Health 2014 SFH Rwanda is a local NGO that formerly operated as a branch of Population Services International (PSI). It operates in all regions of the country and its mission is “using evidence based social marketing to empower Rwandans to choose healthier lives”. Since its formation in 2012, the organization’s mandate to deliver services and materials for HIV/AIDS prevention, malaria control, family planning, maternal health promotion, and proper nutrition. SFH began to receive direct funding from USAID in 2014, with a grant of US$9.3 million for the Rwanda Social Marketing Program. 11. Never Again Rwanda Peace-building, youth 2014 NAR was founded in 2002 to address societal divisions among young Rwandans in the aftermath of the 1994 genocide. It PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 8 Institution Name Primary Sector of Activity Start of HICD/R Support Brief Description of Institution (NAR) engagement, education works to promote citizen engagement and advocacy, and is an acknowledged leader in peace and reconciliation interventions targeting youth. NAR also seeks to develop the critical thinking capacity of youth as a sustainable means of peace building and social-economic development. NAR was granted US$350,000 in USAID funding in 2014, to work with youth on human rights issues. 12. Young Women’s Christian Association (YWCA) Empowerment of girls and young women, including education 2015 YWCA Rwanda is a local non-governmental organization established in February 1995, in response to the crisis concerning the growing number of widows and orphans in the wake of the Rwanda Genocide against Tutsi in 1994. YWCA Rwanda is an affiliate of World YWCA, a global network of women working to deliver equitable social and economic change in over 120 countries. YWCA has not received any significant funding to date from USAID. 13. Adventist Development and Relief Agency in Rwanda (ADRA) Various, including education 2016 ADRA was established in Rwanda in 1978 under the auspices of the Seventh-Day Adventist Welfare Service Inc. The faith￾based organization has worked to empower individuals and communities through development and relief services emphasizing education, healthcare, economic empowerment and food security programs. It is currently seeking registration as a local NGO. ADRA has not received any significant funding to date from USAID. EVALUATION METHODS AND LIMITATIONS In designing and implementing this evaluation, the Evaluation Team has followed the guidelines outlined in USAID’s Request for Task Order Proposal and accompanying Scope of Work (Annex A). The evaluation design and approach by Democracy International (DI) reflect principles outlined in USAID’s 2011 Evaluation Policy. The evaluation was conducted by a team of three experts: Team Leader, Ms. Melanie Reimer, an international expert with significant experience in civil society programming and in conducting similar evaluations; Mr. Wilson Kumwenda, a specialist in organizational development; and Dr. Aime Tsinda, a Rwandan researcher and evaluation specialist. For the evaluation, DI has drawn on the recognized technical expertise of subcontractor The Kaizen Company, a leading provider of HICD programming support to USAID. The Subject Matter Expert on Organizational Development position has been staffed through Kaizen, and played a key role in all stages of the evaluation. The Team has used a mixed-method approach that relied on both quantitative data (primarily Project￾level monitoring data) and qualitative data (primarily collected from Project documents and through key informant interviews and focus group discussions). Those methods are described in more detail below. Data collection was conducted over the course of 15 business days in August 2016. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 9 Selection of informants was purposive, and informed by suggestions from USAID and DAI, participant lists, desk research, and recommendations by other stakeholders. The final decision on informants was in all cases made by the Team, and whenever feasible informants were contacted directly. To contribute to the collection of reliable information, the Team assured all informants of confidentiality and was careful to protect that confidentiality. Interviewees were always offered the option of having their names omitted from the final list annexed to the report, but none chose that option. The selection of informants primarily considered the type and duration of their interaction with the Project, the type of organization and affiliation of informant (CSO, government, implementer, academic, etc.), and their role in that organization. The Team aimed for a reasonable balance of those factors among informants, to ensure that a wide range of voices were heard. The Team conducted all in-person data collection in Kigali, which is the location of almost all direct Project beneficiaries and activities, as well as the location of USAID/Rwanda and implementer offices. While data collection outside Kigali was considered, it was ultimately decided in coordination with USAID/Rwanda that the Team’s time would be better spent in Kigali. The methods used allow the Team to generate an in-depth, comprehensive description and understanding of USAID and DAI’s intervention as a whole and in its context. By triangulating findings and soliciting views from persons most knowledgeable about the Project and the context, the Team was able to maximize the accuracy of its findings and suitability of its recommendations. The Evaluation Work Plan (Annex B) describes the evaluation methodology in greater detail, including an evaluation matrix listing sources and methods used for each evaluation question. DOCUMENT REVIEW The Team reviewed a comprehensive set of documents related to the Project, HICD generally and the Rwandan context, most provided by USAID and DAI. The initial desk research prior to fieldwork allowed the Team to gather a foundation of relevant data about the Project’s goals, plans, activities, and results, which formed the base for the Team’s understanding of the Project, including relevant context and institutions. Key documents reviewed at that stage by the Team included the Statement of Work from the USAID/Rwanda contract with DAI; the Project’s annual work plans and Performance Monitoring Plans; the Project’s quarterly and annual reports; the USAID/Rwanda Country Development Cooperation Strategy (CDCS),13 the HICD Handbook, and performance solution packages for all targeted institutions. The Team identified and sourced additional documents from USAID and DAI, as well as from other key informants and through independent research by the Team, during and after fieldwork. Those documents included various sectoral capacity building plans provided by NCBS, examples of policies and plans developed by CSOs with Project support, the RSPI constitution and draft strategic plan, reports of the USAID CSO Sustainability Index, and research on Rwandan civil society from other sources. A complete list of the documents and websites reviewed for the evaluation can be found in Annex E. KEY INFORMANT INTERVIEWS The Team conducted 50 key informant interviews (KII) that involved 54 diverse informants (40 of them male, and 14 female), in person in Kigali and remotely with individuals in other countries.14 The Team used these in-depth interviews to collect data from USAID officials, current and former DAI staff, KNO 13 Available at: https://www.usaid.gov/sites/default/files/documents/1860/Rwanda-CDCS-ApprovedPublic￾Version.pdf 14 Although the work plan indicated a target of 55 KIIs, unavailability of informants on Saturdays resulted in a slight reduction of the target. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 10 consultants, key personnel of targeted institutions, current and former RSPI leaders, and staff of other donor organizations, among others. The interviewed informants are listed by name and organizational affiliation in Annex D, and a breakdown by affiliation of informant is presented below in Table 3. Table 3 and Figure 2. Number of KIIs, by Affiliation of Informant Informant Affiliation # of KIIs Targeted CSO 10 Targeted government 8 USAID 6 Implementer (DAI) 6 Other government 5 Other donor 3 KNO 2 RSPI 4 Other 6 Total 50 The Team conducted all interviews in English, based on question guides developed to suit the particularities of different categories of informants and respond to the core evaluation questions. Those guides can be found in Annex C. The interviews used a semi-structured approach, allowing some degree of deviation from the questionnaire to allow prioritization among the questions and promote a free￾flowing conversation. Selected interview guides were tested on the first and second days of data collection, with at least two Team members participating in each interview in order to allow careful observation of informant reactions and ways that the questions could be improved.15 No significant revisions were made to the draft guides after piloting. FOCUS GROUP DISCUSSIONS The Team conducted four focus group discussions with four diverse clusters of people from civil society, government, USAID and RSPI.16 These discussions aimed to gather qualitative information from people who had a very specific type of interaction with the Project, as outlined in Table 4 below. Each group was therefore organized around a homogenous identity, and composed of people with similar key characteristics. People who had participated in interviews were purposely excluded from these discussions, to avoid over-burdening of informants and maximize the range of viewpoints canvassed by the evaluation. In all, 21 people participated, including 4 females and 17 males. All but one group had a mix of males and females. 15 Of the 50 KIIs, 17 were conducted by one Team member, 24 by two members, and 9 by three members (primarily in the piloting period). As for FGDs, three were done by all three Team members, and one was done by two Team members. 16 A fifth focus group was planned and scheduled with MINAGRI staff, but the absence of the Director of Planning made it difficult to contact the six targeted participants on a timely basis, and only one person was available. The Team conducted a KII with that individual, and a later KII with another invitee, in order to obtain information about their experience with the Project and its consultants. Chart Title Targeted CSO Targeted government USAID Implementer (DAI) Other government Other donor KNO RSPI Other PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 11 Table 4. Focus Group Composition Participant Affiliation Profile of Participants # of participants Targeted CSO Mid-level managers and staff who worked directly with the Project technical assistance providers 9 Targeted government Cluster specialists from NCBS capacity building unit, with responsibility for the social, governance and economic clusters 3 USAID Staff of technical offices, other than DG, that contributed to funding of the Project 3 RSPI RSPI current or past members that obtained the CPT designation with support of the Project 6 Total 21 The focus group discussions were based on a prepared guide that contained a limited number of open￾ended questions designed to prompt candid responses as part of a flexible conversation. Each lasted about 1.5 hours. Two FGDs were held entirely in English, while French and Kinyarwanda translation was provided on request to certain participants in the other two FGDs. The questions used to guide each discussion are included in Annex C. DATA ANALYSIS The Team relied primarily on content analysis to synthesize information from the document review and analyze the key data emerging from KIIs and FGDs. Performance monitoring data gathered by the Project was analyzed using descriptive statistics and comparative tables. As described in the Evaluation Work Plan, the Team used a Data Synthesis Table to facilitate analysis of KII and FGD data. During the first days of data collection, each evaluation question was broken down into appropriate thematic areas to reflect the range of relevant data being generated while responding to the separate components of each question. Each thematic area was allocated a column of the Data Synthesis Table. The notetaker assigned for each data collection activity was required to record the key KII data beside each informant’s name in that Table, in the most appropriate column, within two days of the activity being completed. A similar process was followed for FGD data. This method allowed for data to be comparable across interlocutors, and easy to review and code. The structure of that Table can be seen in Annex F. LIMITATIONS As with all evaluations, there were certain limitations that affected data collection and analysis. For each limitation, the Team took specific steps to minimize effects on the evaluation and thereby maximize the reliability of findings and conclusions. The key limitations encountered and mitigating approaches were as follows. 1. The timeframe and size of team for this evaluation required a relatively narrow approach focused on a limited number of informants, and the short period in country constrained the use of “snowball sampling” to identify informants based on suggestions from those interviewed. The same factors precluded the collection of new quantitative data. Nevertheless, the Team was able to compare data from a wide range of different primary and secondary sources, as described above, and quantitative data generated by the Project served as a supplementary data source. Telephone and email follow-ups were used to expand the range of sources and amount of data collected, as well as to verify and triangulate information as it was emerging. 2. A design that uses purposive sampling of informants is inherently subject to selection bias, because it cannot use randomization or sampling that is representative of an entire population. The Team PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 12 worked with USAID and DAI to develop the interviewee list, ensuring that it included individuals best positioned to provide substantive feedback in a balanced, unbiased manner; final decisions on informants were always made by the Team. The Team triangulated the data collected as much as possible, and used particular caution with data from sources who may have perceived some benefit or had some personal reason for either praising or criticizing the Project, implementer or USAID. 3. Evaluations are subject to various cognitive biases, including response bias. This type of bias includes the tendency of respondents to answer questions in the way they think the interviewer wants them to or according to what is socially desirable, rather than according to their beliefs. This was mitigated by designing data collection tools to avoid leading questions, and maintaining neutral reactions to responses during the interviews. The Team also stressed the independent nature of the evaluation, provided confidentiality guarantees, and conducted interviews in private settings. The Team confirmed findings by probing similar topics with various stakeholders to see whether findings converged and by triangulating responses with information from document review. 4. The Project’s monitoring systems and tools were found to have serious weaknesses, especially with respect to measuring change at the level of targeted institutions, and therefore their usefulness in terms of assessing results was constrained in some areas. These challenges affected Question 1 in particular, and are described in detail under Question 5 as an implementation issue. 5. USAID/Rwanda requested that the Team not contact certain former DAI staff members, whom they considered might be biased against the Project. The Team respected those wishes. None of those staff members was seen by the Team as a high priority informant, thus the likely effect on the evaluation was minimal. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS QUESTION 1 – EFFECTS ON PERFORMANCE To what extent have the Project’s specific interventions improved the management and technical performance of targeted Rwandan institutions and CSOs? FINDINGS A. General Although changes in capacity of the targeted institutions and its personnel were not the primary focus of this question, the Evaluation Team gathered some information on that subject through interviews and document review, given that improved human capacity is a factor that can contribute to positive change in management and technical performance of institutions. Pre and post-test scores for training workshops show a significant level of immediate change in knowledge of participants, as staff of the targeted institutions, especially in the first three years. In years 1-3, the average increase was 35 percentage points (pretest average of 40 percent and post-test of 75 percent), while in Year 4 testing has shown an average increase of 17 percentage points (pre-test average of 59 percent and post-test of 76 percent). Figure 3. Average scores on pre and post-tests administered to training participants PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 13 It was not clear from the data why the pretest scores were much higher in the last year, though this may be a function of different organizations with a higher baseline level of knowledge participating, as the focus of training activities shifted away from the initial targeted institutions. The implementer also monitored the “number of PSP recommendations implemented by targeted institutions”, as one of its PMP indicators.17 According to the PMP, data was gathered via reports from consultants supporting implementation of recommendations (primarily related to provision of training or establishment of systems or policies) and reports from partner institutions in relation to the use, maintenance and sustainability of the established systems. The latest Project data analyzed by the Team shows 476 recommendations implemented out of the total of 857 recommendations made in PSPs, meaning that the current average implementation rate across all institutions is 56%. At the end of Year 3, the Project reported 409 recommendations completed with targeted institutions as of September 2015 (six first round CSOs, plus SFH and MINAGRI). All 67 recommendations completed in Year 4 relate to the four CSOs and two government institutions that joined the Project in 2014 or later, because the earlier targeted institutions were no longer being monitored by the implementer. The percentage of recommendations implemented for the first round CSOs ranges from 52% to 66% of all PSP recommendations for that organization, and averages 57%. The PSPs for those CSOs had relatively high numbers of recommendations (no less than 82), because the consultants leading those assessment teams used an approach that made a discrete recommendation out of every step along the way to performance improvement. Later assessments, usually led by DAI staff, favored the identification of a lesser number of recommendations, each being broader in scope.18 As well, first round assessments (and SFH in 2014) tended to examine more key areas, while later the assessment process was shortened by focusing on 4-6 key areas of performance that were identified in initial discussions with the targeted institutions’ leaders. CSOs selected for support from 2014 onwards (SFH and second round) have an average implementation rate of 62%, slightly higher than the rate for the first round. There is significant variation 17 In analyzing this indicator, it should be kept in mind that many recommendations are activities either led by the Project—such as training or coaching on a certain subject—or undertaken by the targeted institution with significant Project support—such as development of a new strategic plan or establishment of a new M&E database. 18 For example, a first round CSO assessment broke down the development of a business plan into 12-15 steps, each constituting a separate recommendation, while later assessments tended to have 4-5 distinct recommendations related to formulation of a business development plan. 0 20 40 60 80 100 Y1 Y2 Y3 Y4 Average Pre and Post-test scores, by year Pre-test Post-test PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 14 in implementation rates for government institutions: from 100% for the newest partner, NWC, which only had 10 recommendations, to only 36% (24 of 66 recommendations) for MINAGRI. Table 5. PSP Recommendations Implemented to Date Category Targeted Institution Recommendations Implemented Total PSP Recommendations Ratio of implementation First round CSOs AEE 67 129 52% Imbaraga 72 116 62% Ingabo 57 106 54% FXB 54 94 57% CCOAIB 63 96 66% Caritas 42 82 51% SFH SFH 21 32 66% Second round CSOs YWCA 15 28 54% NAR 12 17 71% ADRA 17 30 57% Govt bodies NCBS 22 51 43% MINAGRI 24 66 36% NWC 10 10 100% All Total 476 857 56% These figures as well as a review of the recommendations themselves indicate that targeted institutions have initiated significant change processes with the support of the Project, in their efforts to reduce performance gaps identified by the assessments. They also show that many identified needs still appear to be unmet, even for first round institutions that were assisted from the early days of the Project. A detailed analysis of implemented versus outstanding recommendations by the Team was not possible, due to time constraints (once data was received), and the fact that DAI did not have updated information on the earlier targeted institutions, which may have implemented some recommendations on their own. Based on discussions with the Evaluation Team, the implementer is presently re-visiting those institutions to assess the current situation with respect to PSP recommendations, some of which may have been implemented in Year 4 without the knowledge or direct support of the Project, since monitoring had been discontinued. It is significant to note that most interviewed representatives of the targeted institutions described serious challenges in the use of increased capacity and new systems in some areas. Obstacles most commonly mentioned were human resource and financial constraints, competing priorities, a tendency to continue old ways, changes in leadership, etc. These challenges are further analyzed under Question 2. As well, five CSO and six government informants (plus participants of one FGD) said their difficulties in translating capacity changes into performance improvement was partly due to an insufficient level of follow-up by DAI and its consultants, to check on implementation of new tools and systems established with Project support, and to help troubleshoot as needed. DAI staff said that they made themselves available to provide ongoing support and periodically visited the targeted institutions, even after “graduation” of the first six CSOs in 2015 and the end of formal engagement with MINAGRI. However, the Project focus had at that point shifted to the newer targeted institutions, based on review of reports and work plans. In some cases, international consultants provided remote assistance to targeted institutions after their visits, although that was limited by budgetary constraints and the difficulties inherent in long-distance communications on complex issues. As for actual management and technical performance of targeted institutions, the Project monitoring data provides some evidence of improvement. An “impact perception” survey was used to try to capture and quantify the perceptions of key staff working at targeted institutions across the Project, PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 15 specifically focusing on the impact and sustainability of the PSP recommendations implemented within each entity. The results of the impact perception survey shown in Table 6 below were highly positive from both government and CSOs in Years 2 and 3, and variation among institutions was minimal. Data for Year 4 is not yet available. Table 6. Results of Impact Perception Survey, average scores Year of Survey19 Impact Sustainability Entities surveyed # respondents Year 2 (Sept 2014) 90% 95% 6 CSOs plus MINAGRI 22 Year 3 (Sept 2015) 93% 94% 7 CSOs, no Government20 24 These data indicate that respondents considered the benefits of the Project to be significant, and could be interpreted as a positive indication of performance change influenced by the Project’s interventions. Comments recorded by the DAI team upon collecting the survey feedback were also uniformly positive about the effects of the Project in both 2014 and 2015. However, the Evaluation Team found that the survey methodology might have compromised the objectivity of responses due to lack of confidentiality, as further analyzed under Question 5. In particular, it was observed that survey responses on sustainability were not necessarily consistent with what evaluators were hearing at the end of Year 4 from many CSOs and government respondents, who had significant reservations about the durability of the Project’s effects on their entities. With respect to other indicators tracked by the Project PMP, there is little concrete data to support conclusions related to performance change. As discussed under Question 5, there were also issues with the suitability and clarity of some indicators as well as target-setting challenges that affect the utility of PMP data in evaluating Project results. No thorough re-assessments have been carried out with targeted institutions, although USAID staff encouraged the implementer to do so. The findings under Question 5 also describe gaps in Project monitoring systems, which failed to systematically select and track indicators that would measure change in performance of each partner. The absence of data on that level means that evidence of performance improvement is mostly anecdotal and not systematically documented or quantified. In addition, it was observed by the evaluators that the vast majority of PSP recommendations related to management performance, i.e. the administrative, financial and other “back office” functions that enable an entity to operate, such as financial management, planning, monitoring, fundraising, and human resource management. Very few recommendations targeted technical performance, i.e. the delivery of services or other activities aimed at achieving the institution’s objectives, although some capacity gaps of that nature were identified in various assessments. According to key DAI staff, this tendency was primarily a function of the targeted institutions’ priorities; CSOs in particular were keen to focus on important functions that their donors were seldom eager to support, with funding or otherwise. B. CSOs The 10 targeted civil society organizations have benefited from a range of interventions, starting with introductory workshops on HICD and participatory performance assessments to identify gaps and 19 The survey was not implemented in Year 1. 20 NCBS and MINAGRI were also reportedly surveyed, but that data were not included in the tabulation of scores provided by Project. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 16 priorities of each CSO. To assist in implementation of recommendations, the Project organized training on topics including advocacy, human resource management and M&E, which often included representatives of more than one CSO that had prioritized the same area. Training was generally followed up by some amount of tailored technical advice and/or coaching at the organization level. As well, the Project procured at least one software package for most CSOs to assist in responding to needs in M&E or financial management, and also covered costs of website hosting. Informants in the implementer and USAID uniformly expressed the view that capacity improvement following Project interventions had been more evident in CSOs than in government. They and the targeted CSOs highlighted the development of new or revamped strategies and systems, as well as moderately improved staff skills, in areas such as M&E, fundraising/business development and advocacy. Those three subject areas were supported by the Project in nine of the 10 CSOs. Strategic planning was also noted by CSOs as an area of advancement through the Project’s assistance, with at least six CSOs developing new strategic plans. The implementer used a tool called the Organizational Capacity Index (OCI) to assess its effects on targeted CSO capacity. Targeted CSOs have demonstrated significant progress towards fulfilling this checklist, as the Project assisted them to establish a wide range of policies and guidelines. The first six targeted CSOs registered an average of 37 items out of 59 at the time of baseline in 2013, and one year later averaged 57 – meaning that they had met all but two items on the list. While some change may have arisen from factors beyond the Project, the similarities between many OCI criteria and the PSP recommendations suggest a high level of attribution. The index does not actually measure the degree of utilization or effectiveness of the items on the list, therefore it could not be used by the Evaluation Team to assess change in CSO performance. (The efficacy of the OCI as a monitoring tool is discussed further under Question 5.) The Project PMP uses an indicator that aims to measure “Effective service delivery to members” by CSOs, as one way of assessing performance improvement. The exact indicator is: percentage change in HICD/R-supported CSO membership receiving direct services from their CSO. The Project data indicates that these numbers were on average up by 26% in Year 2 and 122% in Year 3 – the latter being far beyond expectations. Significant gains were reported by four of the six targeted CSOs from Year 2 to 3, although in two of those cases Project documents describe these numbers were “unverifiable”. The Project was not able to report verifiable data for one other CSO, while in the case of CCOAIB, the indicator was not applicable since they are an umbrella organization with CSOs as its members. Even if these figures could be verified, changes in this indicator would be difficult to attribute to the Project, due to the many factors that can influence the number of members reached by services of a CSO. Primary among those factors would be donor funding, which increased for at least four targeted CSOs since they became involved in the Project. As one DAI staff observed in relation to the recorded increases in donor funding of the CSOs: “It could be better capacity, or it could be just having all those policies in place that donors like.” For these reasons, although it is highly suggestive of better fundraising skills or strategies, the Team does not find this indicator data to be strong evidence of improvement in performance. (Question 5 provides more analysis of the indicator.) Based on review of Project reports and interviews, the following are some specific examples of performance change among the targeted CSOs, including factors that have affected sustainability of change.  AEE is now better able to track its beneficiaries, report to donors and develop advocacy positions, thanks to support to create M&E database customized to meet AEE needs and in￾depth training on its usage and modification. At least five of the targeted CSOs received significant assistance with the development or improvement of monitoring systems and establishment of online databases. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 17  In Caritas, a Project consultant improved organizational network security by reconfiguring systems, drafting maintenance and usage policies to maintain infrastructure and data. IT also proved to be challenging for CCOAIB, so the Project assisted them to develop relevant IT policies in order to secure data and coordinate systems. A leader of that organization is quoted in the Year 3 report as saying: “The training and coaching in IT (information technology) was an eye-opening experience…Now we back up our data and it is more protected”. More recently, one CSO observed limitations in this same area, saying: “We now have an IT strategic plan, but we do not have resources to implement it. Moving a theory to practice is not so easy.”  Several CSOs pointed to increased success in fundraising, as mentioned above. For example, Imbaraga reported having attracted two new donors following Project support with business development, while YWCA said they had raised over US$2 million including two grants from USAID. As well, SFH was able to secure a sizeable grant from a new private sector donor. Assistance in this subject area was universally valued by the CSOs.  One result widely reported as a success story was the new monitoring system that enabled farmer members of Ingabo to keep and report records of profitability of their crops. The beneficiary farmers in the pilot district, Kamonyi, were visited by several USAID delegations to observe this positive development, which was not only providing the farmers with useful data but also informing Ingabo advocacy for policy change on crop priorities (to a large extent determined by the government for each area). Although Ingabo hoped to build on this success by rolling the system out in other districts, staff of the CSO said that implementation had been stalled for the past year, primarily due to internal conflicts (including staff demanding incentives to support collection and entry of data).21 The organization is still struggling with governance and conflict issues, in spite of Project assistance in those areas.  According to the Year 3 annual report, which was corroborated by the CSO staff, CCOAIB’s evidence-based lobbying persuaded the Ministry of Local Government to adopt five out of seven research findings in a report entitled “Citizen Participation in management process of performance contracts at local administrative levels.” More generally, Project personnel reported that some targeted CSOs are now better able to participate when invited by the government to meetings of sector working groups, including through use of data to support their views.  On special request from USAID, the Project helped several CSOs receiving direct funding from USAID to address issues identified in 2013 financial audits, which was noted as having a positive effect on their subsequent financial management and reporting to USAID. It should be noted that the tendency for informants from targeted institutions was to mention policies or documents that had been developed, which ones were being used, and in some cases capacities of staff, but they had more difficulty when the evaluators asked for Project impacts on their performance, and probed for tangible changes in the way they are working. This positive but somewhat vague comment by a CSO leader was typical of the responses: "I believe that overall performance was improved. Our image has changed completely. We used not have strategic plan but now we have it. The reputation was improved and we can share it with our partners…" In other words, the emphasis by most informants (both in CSOs and government) was on changes to policies and plans and tools, with the links to change in performance remaining less clear. Several CSOs also raised concerns about being able to sustain changes, especially given their near-total dependence on donor funding, which was often tied to specific activities and thus not available to finance development of the organization as such. 21 Also mentioned by the Project’s perception survey report, September 2015: M&E database for INGABO was developed and successfully completed. However, the database has been stagnant for 6 months or more due to lack of financial capacities to fund M&E activities and governance issues that affected data flow processes. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 18 C. Government The Project used different approaches and had very distinct experiences with MINAGRI and NCBS, the two government entities on which this evaluation question is focused (due to the newness of the collaboration with NWC, the third targeted government body). In both cases, informants and reports provided evidence that the Project had helped to develop capacities at human and institutional levels, in several areas of operation. The results of pre and post-tests suggest increased human capacity in areas in which the Project provided training. However, unlike with the CSOs, there were no specific tools or indicators used by the implementer to measure improvement in performance, other than the impact perception survey. MINAGRI was a pre-selected target of the Project, based on U.S Government plans to provide direct budget support to that Ministry. (Question 2 describes the challenges that arose from the decision not to proceed with those plans.) The primary method used to assist MINAGRI was provision of four consultants, who were embedded as coaches for various periods. The coaches and other support by the Project included the Rwanda Agriculture Board (RAB) and the National Agriculture Export Board (NAEB), and there was some incidental engagement with other agencies. The areas of focus and total duration of coaching support (according to Project documents) were as follows.  Planning and Budgeting – 11 months + 7 months (respectively for two consultants)  Procurement – 6 months  Policy Formulation – 12 months Informants from USAID and the implementer explained that these priorities were not necessarily based on the Project’s assessment of MINAGRI; the first two areas were selected primarily based on USAID priorities, while the third area of policy formulation was later suggested by the Minister and accepted by USAID and DAI. The evidence of impact of these interventions on capacity and management performance is mixed, although several participants in training and coaching activities were highly appreciative of these consultants. This was especially notable with respect to planning and budgeting, where Ministry staff said that they now had models and tools that helped in improved planning. A senior official said that “there were less questions coming back from MINECOFIN on their plans than before,” though the Team was not able to confirm this with MINECOFIN (the Ministry of Finance and Economic Planning). The same informant noted that the Project had helped them develop a timeline that was enabling them to meet their planning deadlines each year. The Project provided trainings and supported the use of Computerized General Equilibrium Modeling, which was highly valued by participants, especially those sent to the U.S. for advanced training. However, several informants observed that utilization of their new skills was later undermined by expiry of the relevant software (which was being provided by another international agency when the training was taking place). Interestingly, this difficulty was reportedly not communicated to the implementer or USAID until it was revealed by the evaluation. Positive effects of Project assistance were also reported in relation to procurement. Informants pointed to procurement guidelines, a checklist for design of Terms of Reference (TOR), and other related job aids and tools developed with the help of the consultant. Procurement staff valued the consultant’s contribution, especially in the challenging area of contract management, while acknowledging that use of the procurement manual was not widespread within the Ministry. Other Ministry informants indicated minimal impact on procurement functions, primarily because capacity was already high in procurement, which was corroborated by USAID sources. The Project’s work with MINAGRI was highly dependent on efficacy of the embedded consultants, who delivered the vast majority of technical assistance after the assessment was completed. As it turned out, the significant investment in these consultants was only considered by informants to be successful in one of four cases. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 19 1. The primary consultant in Planning and Budgeting was universally acclaimed by informants as “outstanding” and “excellent”. The second consultant provided “economic advisory services” of a non-specific nature. He was specifically requested by the Ministry and allocated to tasks that were not relevant to the Project. 2. The Procurement consultant was seen as competent, but with little value to add where low staffing levels were more of a problem than gaps in human or institutional capacity. It was also mentioned that the Ministry did not consider this area as a priority for assistance. 3. The Policy Formulation consultant was described as ineffective by various informants, who mentioned that he was not able to explain important concepts to people he was mentoring. NCBS has been well aware of the HICD/R Project since the outset, due to its role as the key government counterpart for the Project. In 2014, its leadership requested assistance from the Project to help improve its performance as the agency that coordinates much of the capacity building directed at government institutions (with a broadened mandate that includes private sector and civil society, since the same year). In consultation with DAI staff and consultants in late 2014, NCBS identified four key areas for Project support:  Monitoring and Evaluation  Knowledge Management  Communications  Quality Assurance The work in the first two key areas is more advanced, which led informants from NCBS and the implementer to focus most comments there. With respect to M&E, informants at NCBS noted that significant results flowed from Project assistance (primarily training and coaching), including a major effort to reduce the number of indicators they were tracking. One informant commented: “We had never prioritized before.” Staff also highlighted the introduction of an M&E annual plan and the clear definition of indicators for the first time, as key outcomes. Coaching and technical assistance in the realm of knowledge management was received with great enthusiasm by NCBS personnel, many of whom the consultant engaged in cross-cutting teams that were responsible for certain aspects of knowledge management. One NCBS staff observed that: “An internal community of practice has evolved where staff share information, experiences and best practices through Whatsapp and email groups.” Others praised the “knowledge cafés” that met every two weeks to share knowledge among all NCBS teams, and the toastmasters club established with Project support. Both mechanisms were reportedly still operational, although staff said the Whatsapp group had become inactive and the Taxonomy shared drive was not being used regularly. Several personnel expressed concerns about institutionalization of Project-supported changes, with one commenting that: “NCBS got very good skills but no mechanism for continuity.” Another responded more optimistically to a direct question about sustainability by saying “I am not sure what will happen, if there are staffing changes. But at least there are manuals that could help to keep going.” CONCLUSIONS Overall, the management capacity of targeted institutions has been enhanced with the Project’s assistance, through various mechanisms and in diverse areas of activity. Testing of training participants and the adoption of more than 450 PSP recommendations by the 13 entities both indicate advances, as do the results of the impact perception survey. However, it is a long term process to consolidate and translate such learning (on individual level) and introduction of new systems, plans and tools (on institutional level) into performance change that can be observed and measured. Also, many PSP recommendations have not yet been implemented, indicating that multiple areas of need continue to exist (although this data is still being updated by the Project). Some changes are reported as being PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 20 implemented by targeted institutions, but various new skills and tools/policies/systems are not yet firmly entrenched in CSOs or government entities. As one DAI staff summed it up: "Having a tool is one thing, but bringing change by using that tool is another thing." PMP data and feedback from informants provide evidence of some progress towards increased capacity and performance in certain areas, for all targeted institutions. However, it was difficult for the evaluators to comprehensively answer Question 1 in the absence of institution-level monitoring of related indicators on management performance. (As well, due to the above-described focus on management performance and a lack of relevant monitoring data, the Project had little observable effect on technical capacity or performance of the targeted institutions.) Monitoring tools were not ideal for measuring change at Project level, and setting targets for HICD-type projects is difficult due to the open-ended nature of the approach, which relies on a wide-ranging initial assessment of the entire targeted institution to identify performance gaps, as compared to targeting previously identified issues. The combination of these constraints makes it hard to draw conclusions from the fact that the Project met and often exceeded its PMP targets. (Question 5 delves further into these monitoring issues). A key Project staff member observed: “We helped them in establishing systems but this is not always enough to make them perform better. It is very hard to attribute any significant achievements and change clearly to HICD/R Project.” The Evaluation Team has reached a very similar conclusion. Specifically with respect to targeted CSOs, progress has clearly been made towards establishment of a range of policies, systems and strategies, especially with organizations with a longer involvement in the Project. Both the OCI and the number of recommendations implemented support this conclusion, as does primary data collected by the evaluation. Progress is less visible when considering the performance improvement that was intended to result from the investments in capacity of CSOs, although there is evidence of certain impacts in all of the organizations. On being asked specifically about organizational change, one CSO staff summed it up as follows: “I would say 70% improvement. Through the project, we came up with new things – there is a process, like M&E system, strategies. Very helpful tools, but other 30% is for moving forward. How will it change the way we do our work? That is the part that is still lacking.” With respect to government, based on the above findings, it can be concluded that the Project helped to develop certain capacities at human and institutional levels in both MINAGRI and NCBS. At MINAGRI, the best results were achieved in area of planning and budgeting, although it was difficult to attribute performance improvement to the Project due to support received from other government and donor agencies during the same period. In other areas of intervention, although there was limited sign of change at institutional level, the Project helped build individual capacities and left behind tools that are being used to some extent. Significant challenges to achieving results were posed by quality and management issues related to several embedded consultants, in addition to other factors described under Question 2. In the case of NCBS, there is evidence of changes in systems and ways of working that hold significant promise, but the reforms are still fragile and subject to staff turnover, and certain tools have fallen into disuse. The situation is succinctly described by one consultant who worked with NCBS since 2015: “It’s too early to judge if performance has improved.” Collaboration is ongoing, particularly in the areas of quality assurance and communications, and commitment of the organization appears to be high. RECOMMENDATIONS Note: Additional recommendations closely related to the above conclusions are captured under Question 5, as they are overlapping but more relevant to that area of enquiry. Current Project PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 21 1. DAI should carry out an immediate re-assessment of key areas of each targeted institution, to see what changes in capacity and performance have occurred and been maintained, and identify potential areas for troubleshooting. Those re-assessments should include examination of change in both management and technical performance, to capture the effects of improved management functions on the entity’s ability to deliver on its organizational objectives. Relevant USAID technical office staff should be engaged in this process, especially with respect to technical performance of their partner organizations, to facilitate identification of changes that may have occurred and ongoing needs that could be addressed by this Project and/or through technical office support. (For future projects, periodic re-assessments should be an obligatory element of monitoring.) 2. USAID and DAI should devote remaining Project resources to continued assistance to ensure sustainability of outcomes in each institution, rather than attempting to target new institutions (or new key areas) when future HICD programming is not guaranteed. Future programming 3. USAID and implementers should ensure that targeted institutions are fully aware of results monitoring as an obligation in return for performance improvement support, recognizing that technical assistance interventions are also investing significant resources in their organizations, and that there is a need to be accountable for those investments. Tools such as Performance Scorecards should be incorporated to facilitate monitoring, as suggested by the USAID HICD Handbook. 4. USAID and implementers should use caution in the engagement of embedded consultants. The method can be valuable in certain circumstances, but the justification should be strong, and implementers need to be very careful with vetting processes and contracting mechanisms. Experts need not only technical knowledge but also teaching/mentoring, interpersonal and communications skills. The implementer needs to have a strong say in recruitment and retention decisions, and monitor closely to ensure that consultants are truly contributing to project objectives and defined solutions. QUESTION 2 – FACTORS AFFECTING CAPACITY DEVELOPMENT What key factors may have hindered or aided the achievement of capacity building activities within CSOs vs. government institutions? FINDINGS A. Factors Facilitating Capacity Development CSOs The Team heard broad consensus among informants that: “Once the CSO leadership was on board, the middle managers would remain dedicated to the HICD process.” This was seen as an important factor in working effectively with civil society, and definitely distinct from the experience with government. The targeted CSOs in general were motivated to improve their performance and to become more professional—which one informant observed was because their survival as organizations depends on it (due to reliance on donor funding). At least four of the 10 targeted CSOs were simultaneously receiving significant funding from USAID while several others were candidates for major grants, which may have made their motivation levels even higher. It was repeatedly observed that working with CSOs was much easier than with government, especially in terms receptivity to the HICD model. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 22 Other facilitating factors mentioned by informants, which coincided with government factors, included the participatory nature of the approach and the sense of ownership that was inculcated. The international consultants selected to work with CSOs were considered to be of high quality, and several CSOs also commented on the strong support received from DAI staff. While coaching at organization level was greatly appreciated, CSOs also valued the Project training workshops that included other targeted CSOs, which contributed to sharing of knowledge and experiences amongst CSOs. MINAGRI In spite of significant obstacles in getting the Ministry on board with the Project (described below), the implementer team was persistent in its efforts to work with MINAGRI, and ultimately they managed to get some buy-in at senior levels of the Ministry. Even the new Minister became enthused about Project assistance to policy formulation, which helped pave the way for a new area of intervention. There was strong commitment to the HICD approach by the Director of Planning, whose encouragement of other staff was essential to the progress achieved in the area of planning and budgeting. Although staff turnover was repeatedly noted as a challenge, some informants also pointed out the positive side - after restructuring and widespread layoffs, the remaining personnel at the Ministry were more responsive and receptive to the Project, which they saw as an avenue to make a good impression on their supervisors. As described under Question 1, the Project consultant in budgeting and planning was very well respected by all parties, for both technical and interpersonal skills. His ability to see the “big picture” in the agriculture sector was useful in aligning Project support with the wider government agenda, and he was instrumental in engaging other important government bodies that play a key role in MINAGRI planning and budgeting. NCBS The Project’s collaboration with NCBS was very different, starting from the fact that Project assistance was based on a specific request by the NCBS Executive Secretary—as compared to having been decided at much higher levels for MINAGRI. That request was based on a pre-existing relationship with NCBS, which was the Project’s main government counterpart agency from the outset; as such, the senior staff already had a base of knowledge about HICD approaches. Informants described the existence of a core technical team who was willing and able to learn, and grateful for the technical assistance provided without any expectations of financial inputs. NCBS is much smaller than a Ministry, so it was easier for the Project to assess actual and desired performance in areas identified together with the NCBS leadership, and to establish relations with a wide cross-section of staff. NCBS personnel universally respected the international consultants provided by DAI and KNO as technically skilled in relevant subjects, although cross-cultural differences were also observed. Government in General For both entities, informants valued the participatory nature of the HICD approach, which encouraged the personnel of these institutions to “own the results”, including the documents produced with Project assistance. B. Factors Hindering Capacity Development CSOs One key hindrance mentioned by various stakeholders (not only CSOs) was that the Project did not provide financial or material resources to help CSOs implement what they had developed through Project technical assistance. This was seen as limiting the impact of Project investments, which could (and had) in some cases become “another plan gathering dust on the shelf” because there were no funds PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 23 or people available to take the plan or tool to the implementation stage, where it would ultimately have an effect on organizational effectiveness. One specific gap was the lack of specialized personnel. For example, one CSO explained the need for a resource mobilization officer to take forward its new business development plan, and others mentioned that their M&E personnel were hired with donor funds and devoted to particular projects, and had neither the time nor the mandate to contribute to organization-level monitoring. The duration of visits and time lapse between consultants’ trips was noted by a number of CSOs as a constraint, with some mentioning a shortage of regular communication between visits to liaise on issues arising from implementation. High levels of staff turnover in DAI were also cited by some informants as a challenge to maintaining continuity, as described further under Question 5. In a related issue, the intensive nature of interaction with the consultants over a short period posed difficulties at times, because CSOs had to juggle other commitments and some staff required more time to process and absorb new information and ideas. On the other hand, some DAI personnel attributed the slow progress in some cases to the lack of a “reading culture” in Rwanda,22 which meant that resources and documents intended to complement training or lay foundations for future coaching were not utilized. Several CSO representatives saw the initial process of performance assessment as heavily influenced by the views and background of the Project consultants and staff who led the assessments and supported implementation of PSP recommendations. This “supply-driven” orientation was felt especially in the choice of priority recommendations to be supported by the Project; some CSOs commented that their priorities were not the same as those chosen by the implementer team. According to several CSOs, certain tools and templates used by consultants to address needs (such as a communications strategy or human resources policy) were perceived as “standardized solutions” that were not tailored to their specific situation. One CSO described their impression as follows: “The approach was one of “guided democracy”, or working with someone to get them where you want them to go…we participated, but felt the approaches were pushing us towards certain solutions.” Often, these remarks by CSOs were closely linked to the lack of resources to fully implement the policies and tools that had been developed, because the Project was only providing technical assistance (as further discussed under Question 5). This perception may also have been influenced by the timing of some technical assistance, which in some cases was organized by DAI to respond to needs shared by several CSOs, in order to get the maximum benefit from international consultant visits. (As noted under Question 1, there were many needs with which the Project was not able to assist, presumably due to time and/or budgetary constraints.) Various informants also mentioned that the predominant use of the English language by international consultants and other Project personnel, including in training activities and documents developed with Project assistance, was not conducive to full understanding and absorption by their staff. (See Question 5 for additional analysis.) Some informants noted that CSOs had difficulty in attracting and keeping well￾qualified personnel, which contributed to weaknesses observed in critical and analytical thinking of CSO junior staff. Finally, international personnel of the Project observed the tendency of Rwandans to avoid asking questions in front of others, and to admit lack of understanding or competence. For some CSOs, this was an obstacle to progress. MINAGRI USAID had identified Rwanda’s agriculture sector as the initial focus of the Project, taking into account the overall importance of that sector and the planned initiation of direct, government-to-government (or “G2G”) funding. Since 2010, discussions were ongoing about a Sector Program Assistance package whereby USAID funds would go directly into the GOR’s agriculture sector budget, to be deployed using 22 This observation is corroborated by this article, which also explores some possible reasons: http://www.newtimes.co.rw/section/article/2015-06-10/189582/ PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 24 host-country systems. Due to “geopolitical factors” related to Rwanda’s role in regional conflicts, it was decided not to proceed with this government-to-government funding mechanism, at almost the same time as this Project was being launched. Naturally, the loss of this expected financial support negatively affected the interest of MINAGRI (and other government) officials in the HICD/R initiative. As explained by one DAI staff: “When proposed budget support was withdrawn, great enthusiasm immediately changed to huge frustration within MINAGRI.” Informants described the significant efforts of the implementer just to get permission of Ministry leadership to do an assessment of MINAGRI, which had already been subject to prolonged scrutiny in preparation for government-to-government assistance. Realizing over time that USAID and DAI were eager to appease them, the Ministry made demands that were not always in line with PSP recommendations or Project objectives. This is demonstrated by the situation of several embedded consultants, kept on at the Ministry’s insistence in spite of poor results. (See Question 5 for details.) High staff turnover and low staffing levels at the Ministry were also mentioned by several informants as problematic for the establishment of relations and sustainable development of capacity. For example, one respondent said that he had not managed to apply the knowledge from training because he was transferred to another department soon afterwards. There was evidence that several targeted staff had moved or been moved to other positions or ministries following Project support. NCBS The key factors mentioned by informants as specific challenges for NCBS were: 1) staff had many competing priorities, which made it difficult to fully participate in and commit to Project-led initiatives; 2) insufficient ongoing engagement by Project consultants (not embedded in this case) to sustain what was learnt in the various key areas of intervention. Government in General The international consultants were seen by informants to have a very high level of expertise – which was a positive factor, but posed difficulties for those consultants who were not skilled in mentoring staff with much less knowledge. In addition to technical gaps, consultants from outside the region also faced huge cultural differences and lack of awareness of the specific Rwandan context. Government informants observed a limited level of follow-up by the implementer, which they felt was necessary to ensure that participants were applying what they had learnt and implementing new tools and procedures. Another challenge repeatedly mentioned was the significantly lower staff abilities and motivation in government bodies beyond the most senior levels, which made it difficult for those personnel to absorb the technical assistance being provided. As noted above for CSOs, the limited culture of reading in Rwanda also presented challenges for the presentation and entrenchment of new tools and practices in the targeted institutions. C. Comparison of Government and CSOs On analyzing the above findings, the Team identified the following points of similarity and differences between targeted institutions in government and civil society. Table 7. Comparison of key facilitating factors and hindrances Key facilitating factors affecting both CSOs and government The Project’s participatory approach was generally appreciated, as it encouraged participants to “own the outcomes” of technical assistance. The Project generally provided highly capable consultants (with some exceptions for MINAGRI). PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 25 Key facilitating factors affecting only CSOs Key facilitating factors affecting only government Once the leadership was on board, the rest of the staff would remain dedicated to the process. Buy-in from Director of Planning (and eventual support from Minister of MINAGRI) was achieved. CSOs and their staff were generally motivated to improve performance and to become more professional. HICD assistance was specifically requested by the leadership of NCBS, and their level of enthusiasm for the Project was generally high. Joint workshops were helpful in sharing knowledge and experiences amongst CSOs. Key hindrances affecting both CSOs and government Initial assessment and identification of priority PSP recommendations for Project support at times felt more supply driven (by the objectives of the Project or its consultants) than demand driven. This was mentioned by several CSOs as well as MINAGRI. Low absorptive capacity and lack of reading culture made it difficult to build capacity using international consultants who came for short periods of intensive support. Some consultants were less able to gear down their high level of knowledge for the target audience or fully understand the local context. Leaders and staff of targeted institutions need more and longer follow-up assistance to help with application of learning, plans and policies, including remote and in-country coaching. Staffing levels of targeted institutions are often insufficient to perform heavy workload to a high standard. It took considerable time for staff of targeted institutions at all levels to understand the HICD approach, which varied significantly from the type of donor project support they were accustomed to seeing. Frequent changes in staffing of DAI team created some confusion and had an effect on continuity of support. Key hindrances affecting only CSOs Key hindrances affecting only government Some personnel had limited critical and analytical thinking skills. Significantly less capacity and motivation among personnel below the top levels. People are interested in personal advancement through Masters degrees and overseas study. People were often reluctant to ask questions in public and engage in learning in a group setting. (Although not mentioned, probably applies to government as well.) High staff turnover undermined the sustainability of Project investments, especially in terms of training inputs. (Although not mentioned, probably applies to CSOs as well.) Lack of material support to complement Project assistance and ensure implementation was sustained. The collapse of government-to-government funding plans resulted in top MINAGRI officials losing interest in the Project. Relations took a long time to re-establish. Use of English language posed a barrier to effective understanding of international consultants and documents. (Also mentioned by one informant as a problem for NWC.) Several consultants were maintained on the insistence of MINAGRI, even though not contributing to objectives. CONCLUSIONS The above analysis clearly shows that there were considerable differences in working with government and civil society to develop capacity, notably in terms of levels of motivation to engage in the Project and embrace the HICD approach to improve performance. Only with government representatives did the evaluators hear about issues of allowances and the need to hold training outside Kigali to ensure PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 26 interest was maintained. It appears that expectations have been created within government for donor￾funded projects to provide incentives to training participants, which posed challenges to the Project in attracting committed participants. The Project’s cooperation with MINAGRI (and GOR more generally) was undermined to a great extent by the decision of the U.S. government not to proceed with direct budget support to the GOR, as noted under Question 1. However, there were more points in common than differences, overall. Shared challenges included the generally low absorptive capacity of line personnel and staffing constraints, which both contributed to the need for long-term support to ensure change is sustainable. It is noteworthy that informants from both MINAGRI and CSOs commented on the Project approach being somewhat supply-driven, especially with respect to identification of priorities for support by the Project. This perception among CSOs was probably influenced by frustration with their inability to fully implement some of their new systems, tools and strategies that the Project, primarily due to limited human and financial resources. In other words, the “solutions” provided by the Project were limited in scope and duration by the available technical assistance, which made some CSOs feel that their real needs had not been fully met. On the other hand, both government and CSOs valued the generally participatory approaches used by the Project, and the high quality of most of the consultants that provided technical assistance. RECOMMENDATIONS The Team’s recommendations related to factors that could be influenced in future programming are captured under Question 5, as they relate directly to lessons learned on implementation and management of the Project, and under Question 1 in the case of embedded consultants. QUESTION 3 – CSO AND GOVERNMENT NEEDS Evaluation Question 3: To what extent are Rwandan institutions and CSOs still in need of capacity development support that should be addressed in a future project in order to leverage and sustain USAID/Rwanda investments? PARAMETERS As described in the evaluation matrix in the Methodology section, the Team agreed with USAID/Rwanda to limit the scope of this question, considering the timeframe for the evaluation. We agreed that the Team would focus on unmet needs for capacity development of selected government institutions and CSOs, then look at whether HICD approaches may be appropriate for meeting those needs. The Team focused on CSOs defined by the European Union’s 2013 civil society mapping as “second level” and “third level” organizations, leaving out the grassroots level or community-based groups.23 The government institutions selected by USAID are: Ministry of Justice, Supreme Court of Rwanda, Ministry of Local Government, Ministry of Education, Ministry of Health, Office of the Auditor General of State Finances, and Rwanda Bio Medical Center. In terms of data sources, for CSO needs they were: NCBS, three donor agencies, DG Office of USAID/Rwanda, targeted CSOs, Rwanda Civil Society Platform (RCSP) and Rwanda Governance Board. Several documentary sources are mentioned below. For needs of government, the sources were: NCBS, 23 Second level organisations include NGOs, and in general those CSOs that are not the direct emanation of a “community” or in which a sharp distinction exists between the organisations’ members and the organisations’ “beneficiaries”, while third level organisations include formal and informal networks, umbrella organisations, “collectifs d’organisations”, platforms and coalitions sector or local based. Mapping of the Civil Society and Project Identification of a Support Program to the Civil Society in Rwanda, Prepared by Gianfrancesco Costantini, Stefano Verdecchia and Fidèle Rutayisire, December 2013. Available at https://eeas.europa.eu/delegations/rwanda/documents/press_corner/mapping-of-the-civil-society-and-project￾identification_en.pdf PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 27 three donor agencies, the DG Office of USAID/Rwanda, and group discussions with staff of USAID technical offices. The only available documentation for government was from NCBS. The Team was asked by USAID not to make direct contact with any of the listed institutions. USAID personnel who were available to the evaluation have limited direct contact with those institutions, so awareness of capacity needs was limited. No relevant assessment reports were available related to these institutions, from USAID or other donors. FINDINGS A. Civil society needs Based on review of a range of studies and reports, and consultation with at least 15 informants, the evaluation found that capacity development needs in the sector are still substantial, especially beyond the “elite level” of 15-20 national level organizations that are most supported by donors. Informants say that there is a significant difference in organizational capacity and funding levels between those working in fields approved by the government (most often mentioned were education, health and agriculture), and other organizations. Human rights organizations are especially vulnerable to obstacles posed by the government, either overtly or indirectly, which affect their ability to effectively staff and finance their operations. The 2014 CSO Sustainability Index for Sub-Saharan Africa24 also evidences the wide range of capacity challenges that face civil society in Rwanda (a draft of the 2015 report viewed by the evaluators confirms these findings). The following are key points from that report:  Rwanda’s rating for 2014 was 4.6 on scale of 1 to 7 (1 being the best), down from 4.5 in the previous year. A further decline is likely in 2015. Financial viability was very low at only 5.6.  Overall, CSO sustainability in Rwanda deteriorated in 2014. The organizational and financial capacity of CSOs was weakened by administrative restrictions and requirements, the take-over of some organizations by pro-government leadership, and CSOs’ inability to mobilize funds from internal and external donors.  Well-established CSOs involved in service delivery find it relatively easy to access funds, since their work is in line with donor and government priorities. Organizations involved in human rights, civic and electoral education, justice, and prison monitoring encounter more difficulties in acquiring funds from development partners.  The government generally views CSOs as policy implementers and seeks to limit their ability to shape the political and legal framework. As a result, advocacy capacity in the sector is weak, and CSOs often choose to pursue advocacy in less risky areas, such as agriculture and health.  CSO coalitions are weak to non-existent in Rwanda, mainly because of funding constraints. Mutual trust among CSOs is lacking, and conflicts of interest arise when coalitions compete against their own members for funding. The ongoing needs for assistance have been recognized by several international donors, whose planned civil society support projects are under negotiation with the government at the time of this evaluation. Those plans include both financial inputs and capacity development. In terms of specific areas of need, informants of this evaluation have consistently stressed fundraising, M&E, advocacy (and research to inform advocacy), financial management, and strategic planning as cross￾cutting challenges that affect many CSOs. These capacities also consistently emerged as weak when the Project did assessments of targeted CSOs, even though most of them would be considered as relatively elite or well-established organizations in the Rwanda context. 24 Available at https://www.usaid.gov/africa-civil-society PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 28 Certain informants also mentioned issues such as lack of independence (from donors and government), shortage of qualified staff, and weak governance systems. Many networks exist, but often do not really support their members; some are seen as mechanisms of control by government. Lack of fluency in English language was also mentioned as a constraint by informants working with civil society, as well as two targeted CSOs. While the new generation in Rwanda tends to speak less French than previously (it was dropped as a language of instruction in schools in 2008), many people are still not fully conversant in English, resulting in a situation in which Kinyarwanda is the only language of full fluency. The European Union did thorough research on the needs of Rwandan civil society in 2013-14, which led to publication of the mapping report cited above. That study describes a wide range of challenges facing CSOs in Rwanda; some highlighted capacity-related issues are: • Most CSOs mainly interact with donors and public authorities as a service provider or as a beneficiary, and consequently have a high degree of dependency. • Each CSO seems to be alone – effective collaboration among CSOs is very limited. • Influence on policies is limited. CSOs need better analytic capacities, advocacy capacities and linkages with their constituencies. A project recently initiated by DFID has also identified advocacy and collaboration as important needs in the sector. The expected results of that project include: increased civil society capacity to conduct research and engage constructively and effectively with government, and increased collaboration and joint working across civil society organizations on issues of common interest.25 Discussions with NCBS staff confirmed that capacity within civil society is at a low level. Since 2014, when its mandate was expanded to encompass civil society, the NCBS has been working with a Capacity Building Thematic Working Group led by the RCSP, which develops an annual needs assessment for the sector. The 2016-17 plan was shared with the Team, and indicated the following priorities for CSO members of the RCSP:  Advocacy and policy analysis  English language  Resource mobilization  Monitoring and evaluation B. Government needs According to informants, the level of need for capacity development is variable across government, with certain ministries and agencies being considerably stronger than others, but capacity needs are generally much more significant at decentralized levels. Harmonization and coordination among ministries and agencies has been repeatedly identified as a cross-cutting area of difficulty within the government. Informants also pointed to problems of coordination between the center and districts. In particular, MINALOC and the district governments face many competing demands that they are hard pressed to meet. M&E was mentioned by various informants as another key area where the government is lacking both skills and resources; they emphasized the importance of strong M&E to ensure accurate tracking of Imihigo contracts. Several informants noted the need to build knowledge management systems in government institutions, with one saying: “The government has lots of data, but it is not accessible.” Linking data to policy was another area prioritized by informants. For example, a USAID staff member 25 Multi-Donor Civil Society Support Programme 2015-2021, DFID Rwanda Business Case, July 2014, available at https://devtracker.dfid.gov.uk/projects/GB-1-203927/documents/ PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 29 mentioned that education quality has been the most significant issue in that sector for years, but the data being collected by government is not shedding light on that problem. With respect to the specific government institutions that were the focus of this evaluation question, NCBS proved to be the best source of information. The data in Table 8 below is based exclusively on sector-level capacity building plans provided by NCBS, except where otherwise noted. With respect to the priorities identified in those plans and listed below, the proviso mentioned by a NCBS staff member should be kept in mind. That person observed that these sectors are often only able to get one or two of their capacity needs met per year by NCBS, so although some of these institutions’ leaders may well recognize deficiencies in their management and systems, they tend to prioritize more technical needs with a direct impact on service delivery. Table 8. Capacity Needs of Government Institutions Institution Priority Capacity Needs JUSTICE SECTOR Judiciary (includes Supreme Court and Ministry of Justice)  Limited skills and knowledge of Judges on new laws and guidelines, and changes made to the laws. Some of the Laws include; damages and compensation, sentencing, judgement writing, land and property law, procurement law, new penal code and new family code.  Limited skills of judges in managing Terrorism and Cross Border crimes.  Limited skills of Registrars in legal practice and case management  Staff have not been trained in the new Electronic Case Management System procured by the Justice Sector in 2014/2015.  Limited skills of staff in M&E to enhance performance management within the Judiciary  Limited skills in Registrar and Front Desk Managers in customer care, office management and strategic management to enhance service delivery in Judiciary. EDUCATION SECTOR Ministry of Education  The Ministry of Education has reviewed the competence-based curriculum for Pre￾Primary, Primary and Secondary Education, the challenge remains to deliver training to teachers at all levels.  Lack of feasibility study regarding implementation of training levy in TVET (Technical & Vocational Education & Training)  Lack of strong Education Management Information System.  Limited number of trained teachers in special needs Education Rwanda Education Board  Teachers have not been trained on the Students Career Planning Management Information System (MIS) which is aimed at minimizing mismatches between choice of career and student’s talents and skills. Note: USAID staff said that the Board needs support with training of teachers by distance learning, and with linking research and data to policy. Higher Education Council  Limited skills to analyze and utilize the new Rwanda National Qualification Framework  Limited skills and competencies amongst personnel to audit, evaluate and monitor Higher Learning Institutions  Limited skills of personnel in policy formulation/design, policy analysis and monitoring and evaluation.  More skills required in legal affairs especially as related to Higher Learning Institutions in Rwanda and outside Rwanda.  More skills required in Management of Assets Acquisition, inventory and management PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 30 Institution Priority Capacity Needs  Limited skills in Business language, communication, management of assets, and records management. University of Rwanda  Lack of skills among professors in teaching and learning in Higher Education  Limited skills of researchers in proposals writing, fundraising and projects management  Lack of skills in monitoring and evaluation among planners  Lack of Capacity Building Plan for University HEALTH SECTOR Ministry of Health  Insufficient qualified health professionals in health facilities  Limited implementation and monitoring of Continued Professional Development Programs amongst health professionals  There is no Qualified staff In Dialysis specialty Note: USAID staff observed that there were significant management gaps within the Ministry, which are particularly acute at the district level, including hospital management. University Central Hospital of Kigali (CHUK)  There is need for increased skills in maintenance of equipment  Lack of adequate skills in Pediatric and neonatal services  Lack of adequate skills in Nephrological care (kidney disease) OTHER SECTORS (not based on NCBS data) Office of Auditor￾General for State Finances (OAG) The Finance and Economics Sector of the government does its own needs assessment, which meant that NCBS had no information on this entity. It was however mentioned by two donor representatives as a promising partner for capacity development. Other donors are providing financial assistance, and some capacity support, but implementation capacity was described as still low. Ministry of Local Government (MINALOC) According to NCBS, this Ministry has its own capacity building plan, to which the Team did not have access. The Ministry has huge responsibilities in multiple sectors, since it helps in implementation on the ground of a wide range of government services. One informant observed that “It does not help to fix problems in centralized ministry, if the workers who carry out activities are under another ministry (Local Government).” The Ministry has fairly good capacity at central level, according to informants, but is stretched very thin. At district level, capacity is highly variable. The Local Government Institute is a new body under the auspices of MINALOC, which could well be in need of capacity development support. C. Suitability of HICD methodology PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 31 The HICD approach is broadly appreciated by the institutions and individuals in Rwanda that have been exposed to the approach, including targeted CSOs and government bodies, as well as others closely engaged by the Project such as Rwanda Management Institute and members of COMPRAC/RSPI. The approach is highly adaptable to different entities, as noted by the USAID HICD Handbook; this was also found to be the case in this Project. Several informants remarked that the focus of HICD on results monitoring and assessment of performance was highly consistent with the Imihigo system, which made it eminently suitable for the context. (See text box for information about Imihigo.26) NCBS staff noted that the HICD approach is quite similar to methods used by NCBS and the government more generally, as it not only looks at individuals but also focuses on institutions and systems. CSO informants underlined the fact that HICD comes as a package of solutions, which they found effective. In the words of one informant: “HICD looks at the whole organization and understands the whole organization.” An experienced consultant noted: "The HICD Model is simple, digestive, logical and follows the anatomy of the organization", although he also observed that each model has its strengths and weaknesses. He preferred to use HICD in combination with the “problem driven iterative adaptation” (PDIA) model, for more flexibility. Several USAID staffers remarked that HICD was not the only capacity development model that USAID should be using, since other approaches were also able to assess needs accurately and might be more suitable or efficient in certain sectors or projects. Informants from both within and outside the Project commented that HICD was hard to “sell” to stakeholders in Rwanda at first. HICD still has a low profile, even within USAID; there are few projects explicitly based on the approach to date. Because the model was unfamiliar and did not resemble a typical donor-funded project, significant time and energy was required to first explain the approach and then persuade stakeholders in Rwanda of its value. In fact, some senior personnel of targeted institutions (CSO and government) still seem to be baffled by the Project’s open-ended approach, which lacks the predefined targets, activities or even sectors to which 26 Sources for information in text box on Imihigo are: The Promise of Imihigo: Decentralized Service Delivery in Rwanda, 2006-2010, by Daniel Scher, available at http://successfulsocieties.princeton.edu/publications/promise￾imihigo-decentralized-service-delivery-rwanda-2006-2010, with updated information from http://rwandapedia.rw/explore/imihigo Imihigo, in brief In 2000, in an effort to improve local service delivery, the Rwandan government began a program of decentralization. Under the new arrangement, mayors were responsible for implementing development programs. To promote accountability at local level, the government in 2006 launched an innovative system known as the Imihigo process. Imihigo had its roots in a pre-colonial Rwandan cultural practice whereby leaders or warriors would publicly vow to achieve certain goals and face public humiliation if they failed. By 2010, Imihigo had come to refer to a combined performance-management system, planning tool, multi-layered oversight mechanism, and way of using social and traditional pressure to push mayors to greater levels of achievement. The government has since moved to introduce Imihigo processes at other levels of government. The civil service's Imihigo system has incorporated specific measures for planning, monitoring and evaluation, and accountability. Every member of the civil service is required to complete and sign a performance contract, known as Imihigo, pursuant to a Prime Minister’s Order. The Imihigo consists of the employee’s main responsibilities, their targets as well as their individual competencies. Individual performance contracts are based on the action plan of each institution, and a public servant’s Imihigo performance is evaluated annually. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 32 they are accustomed. The process of educating and getting key leaders on board with HICD is critical in working with the government, because according to informants in government, USAID and donors, the buy-in of both top and mid-level officials is fundamental to making any progress. This is not easily achieved, because many leaders may not be inclined to open up their operations in their entirety for the identification of weak points in all areas by external parties, which is the foundation of the HICD approach. As part of its efforts to advance HICD and performance improvement in Rwanda, the implementer entered into a collaboration with the Rwanda Management Institute (RMI); this represented an unanticipated opportunity to work more closely with the government body mandated to provide capacity building support to a wide range of GOR institutions. DAI staff and consultants provided intensive training to at least seven RMI staff, to develop their knowledge of HICD and their abilities to train others in the approach. With Project support, a new Performance Improvement Curriculum was designed, including both training and hands-on learning opportunities for both senior executives and line managers. After an orientation for 29 top officials of 14 selected GOR institutions, the Project and RMI co-facilitated training for 16 mid-level managers from those institutions. The Project reported that those 16 went on to use performance improvement approaches to address concrete issues affecting their respective institutions. While it was hoped that RMI would continue to use the detailed curriculum to offer similar training to others in government, it was found that RMI has not yet done so. Different informants put forth divergent reasons, ranging from internal factors such as the death of a key trainer at RMI and a major reshuffle of staff, to the lack of requests for the training from government entities. There is conflicting evidence about the level of interest in HICD within RMI’s leadership, which has left the future of this promising cooperative effort in doubt. A full assessment in line with HICD principles (examining all aspects of an institution’s functioning as a “total performance system”, according to the HICD Handbook) is very intensive and time-consuming for both the assessment team and the targeted institution. Although it takes considerable less time with smaller entities, informants from the implementer and USAID were concerned about the cost-benefit ratio, especially for assessments led by international consultants – which were necessary in the first stages of the Project. Since the assessments are so wide-ranging, the team must have not only strong research skills but also a well-developed understanding of a diverse set of topics affecting organizational performance, such as financial management and human resources, as well as technical knowledge of the relevant sector. The model puts considerable emphasis on the assessment process, to ensure proper identification of needs and root causes. Informants said that many GOR institutions (including MINAGRI) had been analyzed extensively at the insistence of various donors, which could make them less than enthusiastic about the prospect of more assessment. Because of this, and due to USAID priorities at the time, the MINAGRI assessment was focused on specific functions rather than examining the entire ministry. As well, in order to streamline processes and reduce costs, in Year 3 the implementer decided to modify the HICD approach by limiting the scope of assessments. Rather than investigating every aspect of the institution, certain areas of focus were agreed with the CSO and government institutions from the start, based on pre-identified interests or on previous analyses. By comparison, the Evaluation Team looked at the experience in Georgia, which is a pioneer of HICD within USAID programming. Under the FORECAST Program27 (2007 to 2011), “full HICD” initiatives (examining all aspects of organizational performance) were implemented with nine individually selected 27 Focus On Results: Enhancing Capacity Across Sectors In Transition Program PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 33 organizations, while also carrying out numerous stand-alone training and related interventions that also used HICD tools. Based on the evaluation of FORECAST, the design of the follow-on HICD Plus project concentrated on quality and duration rather than quantity of activities. As a result, HICD Plus used the full HICD treatment with only three partners between 2011 and 2015; most activities consisted of more targeted technical assistance that used selected HICD methods. The approach in Rwanda was quite distinct, in that all 13 Project interventions were based on a comprehensive assessment and package of support to meet multiple needs in accordance with the original design. As described above, many CSOs are under-developed and need to improve their capacity in multiple areas. The holistic approach of HICD was therefore considered useful and appropriate by CSO informants, including in FGD discussion. They commented that most donors are reluctant to invest in the capacity of people working in “back office functions” such as human resources, accounting, and M&E, so HICD is able to fill a gap with support in those areas. The CSOs generally felt that the HICD approach had helped them to gain skills and knowledge, which they were ready to apply. However, that stage of application was often constrained by the twin factors of limited funding and limited human resources, as described under Question 1. CONCLUSIONS Civil Society There is a high level of ongoing need among the analyzed levels of CSOs, although certain serious issues are beyond the realm of capacity development, such as challenges with independence and registration. The key areas of capacity gaps are advocacy, resource mobilization, M&E, networking and collaboration. Working on these could also contribute to greater independence of targeted civil society organizations. Funding shortfalls are significant beyond the elite CSOs, and especially acute for organizations that wish to do more than service delivery, and/or are working in thematic or geographic areas that are not clearly aligned with government priorities. Even with increased organizational capacity, lack of funds and government restrictions will continue to impose severe limitations on the quality and quantity of CSO activity. Faced with weak or mistrusted formal networks, CSOs are increasingly “going it alone” and letting ties lapse with like-minded organizations. Coordination and information sharing are declining as a result. There is considerable donor interest in the civil society sector, although delays in getting government approvals are affecting the launch of planned projects. These are among the signs that external support to a stronger voice for civil society may not be condoned by the current government. Government The most significant capacity gaps in government are at the decentralized level, and by extension in MINALOC. At the central level, the cross-cutting priority according to external observers is better harmonization and coordination, within and among institutions. Monitoring, linking data collection to policy, and knowledge management are also important needs observed by multiple informants working closely with different government entities. On the other hand, sectoral working groups prioritized a diverse set of more technical needs related to service delivery; these differences in prioritization underline the need for further consultations and assessment before any firm conclusions can be reached on government capacity development needs. The notable overlap between the NCBS-provided data on sector needs and the external informants was M&E, which emerged as a priority in several sector lists. HICD Suitability A foothold has been established in Rwanda in terms of awareness among certain stakeholders in civil society and government, and a group of professionals who have a base of knowledge of the approach, PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 34 which could be built upon in a follow-on intervention. The RMI could potentially serve as an important channel for spreading knowledge about performance improvement approaches within key government institutions, if sufficient demand can be generated. However, progress has been gradual, and much remains to be done for a critical mass to be achieved. Generally, the HICD focus on periodic assessment of performance at multiple levels and close monitoring of results appears to be consistent with the Imihigo system, and therefore well suited to the current Rwandan context. Due to high staff turnover in both government and CSOs, an approach that pays close attention to systems, tools and the institution itself is important as a counter-balance to investment in individual capacities. Implementing HICD as a standalone project has the advantage of ensuring a sustained focus on capacity development, but also has its challenges, as Question 5 analyzes in more depth. The flexibility of HICD programming can be especially appropriate in an unpredictable political context, since it can readily respond to opportunities and threats as they emerge. However, other models of organizational development and capacity development (such as Problem-Driven Iterative Adaptation) could also add value in the Rwanda context and should not be excluded from consideration.28 In particular, it is important to include the operating environment and inter-organizational connections as part of the capacity development process, although these elements are not emphasized by the HICD Handbook. This is consistent with a recent USAID report on Local Capacity Development, which recommended to: • Increase the focus on systems-level capacity by focusing on the connections between actors, not just capacities at the individual or organizational levels, and increase the focus on enhancing the enabling environment through political economy analyses and other means. • Strengthen social capital and networking and examine resources hubs and spheres of influence that can connect actors at the local and regional levels, possibly using organizational networking analysis.29 With respect to CSOs in particular, HICD approaches are certainly relevant, given the multiple interlocking capacity gaps of many CSOs. HICD could potentially be used in combination with other methods or adapted to ensure that CSOs have enough absorptive and operational capacity to benefit from assistance. The open-ended approach to assessment of needs does not seem to be problematic with CSOs in Rwanda, since they tend to openly share information and are accustomed to external review processes. A so-called “full HICD” intervention that identifies multiple key areas for change is less suitable for smaller, less established and/or poorly resourced CSOs, which are likely to struggle with ongoing implementation of sweeping changes to their organizations. As pointed out by the HICD Handbook, it is important to target institutions that can be sustainable and have a good probability of thriving after HICD support.30 As for the Rwandan government, although this Project’s experience has been mixed, there is evidence that the open-ended approach of HICD is not a particularly good fit. There was general agreement that the key people at all levels of the institution must have a clear idea of the problem that is to be solved by a donor-funded project, and how it fits into their existing priorities and objectives (including Imihigo), in order for the project to gain traction. Achieving buy-in from senior officials in targeted institutions 28 Other widely accepted models that could be a source of ideas to enrich future programming in Rwanda are the Capacity Assessment Framework of UNDP, and the Capacity Development and Results Framework promoted by the World Bank. Another approach to consider is the so-called “Capacity 2.0” with a focus on soft skills, as outlined in a new USAID report, entitled “Strengthening Local Capacity in Southeast Asia: Approaches and Experiences” (see next footnote). 29 Strengthening Local Capacity in Southeast Asia: Approaches and Experiences, RTI International for USAID, June 2016, at page 10. 30 HICD Handbook, page 11. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 35 will continue to be difficult unless specific areas of assistance can be identified and agreed upon from the outset; those initial areas of focus could potentially be augmented as the relationship grows and results of the approach are demonstrated. As called for in the contract between USAID/Rwanda and DAI, the full HICD treatment was applied across the board in this Project (albeit slightly scaled back in Years 3 and 4), but that may not always be the best approach. For example, the Georgia HICD Plus project has demonstrated that considerable success can be achieved with a mixed approach, where only a few chosen institutions are assessed comprehensively and supported in multiple key areas of need, while capacity gaps in other entities are met with more specific technical assistance. It is possible that the Rwanda Project was too ambitious in its scope, including the number of institutions targeted with full HICD (well beyond the original target of eight institutions), as well as the sheer number of PSP recommendations tackled with the initially targeted institutions; achieving measurable impact proved challenging, especially in a context where HICD was completely unknown and the starting point of human and institutional capacities was probably lower than in Georgia. It is important to ensure that “standard solutions” and checklists of policies and strategy documents do not become the drivers of the change process. There was some sign of this happening with CSOs targeted by the current Project, including through use of the OCI tool for CSO capacity measurement, as discussed under Questions 1 and 5. There is a risk with both CSOs and government of what has been called “isomorphic mimicry”, a term recently coined to describe the situation when “…governments and organizations pretend to reform by changing what policies or organizations look like rather than what they actually do. The flow of development resources and legitimacy without demonstrated improvements in performance, however, undermines the impetus for effective action to build state capability or improve performance.”31 The authors define one of the dangers of this phenomenon as “premature load bearing”: the organization is asked to change too much, too quickly, and can in fact be weakened as a result. RECOMMENDATIONS Civil Society (applicable to civil society programming and HICD programming that targets CSOs) 1. USAID should carry out further consultations with the DG Office and other technical offices, as well as other donors and implementers (including the European Union in relation to their draft Civil Society Roadmap), to inform any future activities aiming to develop capacity of civil society in Rwanda. USAID and other donors interested in promoting democratic governance in Rwanda should share information, work together to persuade government of the value of these interventions, and be persistent in attempts to engage with government to overcome obstacles. 2. USAID should evaluate the need to package CSO assistance in creative ways, since a low profile may be more effective in the current political context. Instead of major standalone civil society support programs, USAID should continue to explore ways of incorporating civil society strengthening and citizen participation elements in “technical” programming that aligns with government priorities in health, education, economic and other sectors. 3. If a priority objective is to strengthen civil society and civic engagement in Rwanda, USAID should provide more support to organizations outside Kigali and outside the sphere of typical donor funding. HICD may be highly suitable for assisting CSOs that are “on the cusp” (as one 31 Andrews, Pritchett and Woolcock, Escaping Capability Traps through Problem-Driven Iterative Adaptation (PDIA) - Working Paper 299, Center for Global Development, 2012, abstract page. Available at http://www.cgdev.org/publication/escaping-capability-traps-through-problem-driven-iterative-adaptation-pdia￾working-paper PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 36 informant described it) of being able to manage direct funding for major projects, where a modest investment could help them move to the next level of operations and effectiveness. 4. USAID and implementers should prioritize (or continue to prioritize) the following topics and approaches in working with CSOs in Rwanda:  Topics: M&E, especially harmonization of indicators and streamlining of processes; income generation strategies, including advocacy for tax breaks for CSOs; specific capacities for engaging in governance issues, such as policy analysis, evidence based advocacy, local development planning, and policy dialogue. Support to improvement of technical performance should be prioritized for high-functioning CSO partners implementing activities supported by USAID technical offices.  Approaches: support to joint initiatives involving different CSOs in the same geographical or thematic area; training for clusters of organizations to promote opportunities to network; individual coaching on implementation to follow up on training workshops; regular check-ins and supplementary assistance (as needed) to obtain and sustain results. Government Recognizing that this evaluation was only able to conduct preliminary investigations into the capacity needs of the seven selected institutions: 5. USAID should carry out further consultations with other donors and their implementers to inform the targeting and methods for future activities aiming to develop capacity of government institutions in Rwanda. This is important to ensure a coordinated approach and benefit from valuable lessons learned by other agencies. 6. The DG Office should conduct additional discussions with USAID staff of relevant technical offices in the mission, to verify how they may be contributing to addressing capacity needs of the seven institutions, either through current or planned programming, and to seek more inputs in relation to targeting and methods. 7. USAID should conduct further research with NCBS (and potentially with the Sector Working Groups that prepare requests for capacity-building support for NCBS), as well as with other government bodies that organize or coordinate capacity building in sectors that are not covered by NCBS, to verify and clarify the preliminary data obtained by this evaluation about the seven institutions and their affiliated agencies. Exploration of the needs for improved M&E capacity would be especially interesting, as would consideration of the Office of the Auditor General and Local Government Institute as potential future partners. HICD Suitability 8. USAID should consider additional investment in HICD approaches in Rwanda in order to sustain the investment already made, but not necessarily exclude the use of other models for performance improvement. In particular, future interventions should ensure that full consideration is given to the environment of targeted institutions and their connections with other entities; this is especially important for civil society support, given the evidence of weak links among CSOs in the current situation. 9. Future USAID programming could take one of several forms, including: another performance improvement project without predetermined sectors or targets, a performance improvement project with some predetermined sectors and targets, or an approach that aims to incorporate HICD and complementary methods in the work of different USAID offices to meet performance improvement needs of partners of various projects. The last-mentioned option could involve either internal (within USAID) or external contracting of organizational development experts to PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 37 work closely with technical office staff and key implementing partners to deepen their understanding of the HICD approach and other relevant models, and to support them in using those methods effectively with their key Rwandan partner institutions. 10. USAID and implementers should take a flexible approach to HICD, tailoring the model to the situation of each organization, the priorities for organizational development, and resources available to that organization (from sources including the HICD project and any USAID office or technical assistance implementer). It is not always useful or necessary to do the full assessment and comprehensive gamut of solutions with all organizations, especially considering the costs and time required. More short-term and focused technical assistance can be effective and efficient in targeting specific areas of need, if accurately identified. 11. USAID/Rwanda should carefully examine the lengthy experience of USAID/Georgia in relation to the management of HICD programming, including in relation to the pros and cons of using the Program Office to manage HICD as a cross-cutting activity, and the balance between “full HICD” and other support to capacity development. An exchange visit between Rwanda and Georgia may be useful in this respect. 12. USAID and implementers should adapt the HICD approach for work with the Rwandan government, where it is more likely to be welcomed and successful if focused on pre-identified areas of need that fit into established priorities of the overall government, and of the institution in question. An emphasis on system-wide management of human capital may be particularly relevant in addressing issues of high staff turnover. 13. With respect to future programming, USAID should consider the pros and cons of collaborating with various Rwandan entities with mandates highly relevant to performance improvement. Given the recently broadened scopes of work of both the NCBS and RMI, one or both could prove to be strong partners in the promotion of HICD and performance improvement more generally. 14. USAID and implementers should anticipate the need for multi-year support for major organizational changes, as further underlined under Question 5. Performance improvement should be supported in sequential stages for specific intervention areas that clearly define outcomes/deliverables, each based on a separate MoU (also consistent with the FORECAST Program evaluation in Georgia).32 This can serve to reinforce the buy-in of all parties, and allow USAID and the implementer an “exit point” if collaboration is not working out. QUESTION 4 – CAPACITY AND NEEDS OF RSPI AND MEMBERS What capacity development support does the RSPI need to sustainably continue HICD interventions beyond the life of the Project, both with USAID/Rwanda financial support in the near term, and without USAID/Rwanda financial support in the long term? How long will it take to build such capacity? 32 See FORECAST evaluation report, page 15: “an MOU could be signed at the point of initial partner organization selection, prior to the performance assessment, in mutual agreement to the performance solutions package, prior to any performance solution of high fiscal value, or to address subsequent cycles of performance improvement assistance. As stated in an earlier section, establishing change management as a responsibility of the host country partner organization could help to establish buy-in and commitment from multiple levels of the organization early on in the activity… An MOU should also be executed when leadership changes occur within the partner organization.” PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 38 Based on guidance from USAID/Rwanda, the Team has approached this question as composed of two related but distinct sub-questions: the first (Part 4.1) focuses on the RSPI as an organization, while the second (4.2) looks at the members of RSPI as professionals and HICD practitioners. In the course of researching these questions, the Team spoke to 14 current and former members of RSPI from government, private consultancies and international development, via one FGD and eight KIIs. 4.1. What are the main capacity gaps of the RSPI, and what type of USAID support could best help to meet those gaps? How long will it take RSPI to achieve sustainability? FINDINGS A. Evolution of RSPI COMPRAC was founded by the Project in early 2013 as a community of practice of 60 Rwandan professional consultants with diverse backgrounds in finance, organizational development, monitoring and evaluation, human resources development, policy analysis, and strategic management. Based on the Project design, these consultants were expected to play a key role in implementing solutions to address performance gaps in HICD/R targeted institutions and in establishing the field of performance improvement in Rwanda. The implementer supported the consultants to meet regularly to network and share information regarding professional and market development, and trained them on HICD methods and performance improvement more generally. Training and coaching were also provided to help members work towards the CPT designation. COMPRAC was originally envisaged as an informal body, and had no legal registration. At one stage, the membership was reported to be as high as 88 individuals. Several informants close to the Project mentioned that competition among members was undermining group cohesion and contributing to internal conflicts. Therefore, the implementers worked with a core group of members to analyze options for its future, and it was decided in 2014 to create a more formal body as a chapter of the International Society for Performance Improvement. A senior Project staff said that formation of RSPI was seen as a way of “salvaging the situation” by giving more structure to the community of practice and extending it beyond the ranks of consultants. RSPI was officially registered in July 2014 as a not-for-profit organization. The structure includes an executive committee and a general assembly. According to its draft strategic plan (dated March 2016), the mission is to further advance application of knowledge and skills through collection, development, and dissemination of information about instructional technology and human performance technology. Its constitution states that membership is open to any individual professionally interested in the purpose of the organization who visibly supports its mission, goals and objectives and participates in the organization’s activities. The target audiences for members include: • Staff of public institutions • Employees of private sector • Staff and volunteers from civil society and non-profit organizations • Consultants for capacity building • Professors and students of academic institutions • Experts from professional bodies An individual may become a member of RSPI upon receipt of a completed membership application, full payment of membership dues, and active participation in activities. The RSPI executive provided a list of 55 current members, showing that 49% of members are independent consultants, 7% are from academia, and the balance are almost equally divided among staff of private sector bodies, government, and international development agencies. The graph below provides details. It should be noted that the PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 39 list from RSPI did not include several members of the current executive (according to a list provided by DAI), so it may not be complete. Figure 4. Composition of RSPI membership, 2016 B. Capacity Gaps of RSPI Several informants familiar with the organization expressed concern about both organizational management skills and motivation within the executive and other core members. While some placed more confidence in the incoming president for 2017, the view shared by DAI staff and consultants is that the leaders see themselves as a board of directors with only oversight responsibilities, as compared to active implementers of the RSPI strategy and activities. One individual who worked closely with the executive observed that: “They do not want to put their feet in the mud.” This was consistent with recommendations put forward by various RSPI members, when asked about the organization’s needs, that USAID (or other donors) provide funding for an office, equipment, and at least one staff for the organization. One informant close to RSPI considered that the organization had not yet managed to attract “respected thought leaders” to its ranks, which was posing challenges to its credibility and efficacy. A key RSPI member considered that the organization needed a much wider base of support to be effective, including government policy makers, Imihigo chiefs and academia. RSPI only recently opened its first bank account, and the evaluators were not able to determine the level of financial management experience within the executive. RSPI has no track record of managing funds, since DAI directly covered all expenses related to its activities and operations. RSPI developed a draft strategic plan in a process facilitated by Project consultants in early 2016, with a view to planning the organization’s transition to independence. The strategic plan has outlined a number of potential activities and strategies for the future, however, it had not yet been finalized as of early September 2016. Notably, no budget projections or sources of funding had yet been worked out. Executive members interviewed by the Team were not able to clearly explain the reasons for delay, although they mentioned plans to meet soon to finalize the plan. Sustainability To date, RSPI has been reliant on the Project to cover all of its expenses, and it has no assets. Its only source of independent income is membership fees of 20,000 Rwandan francs (approximately US$25) per person per annum. DAI staff and RSPI leaders said that fees were deliberately set low to attract new members who might not be willing to risk a high investment in an untested organization. Even so, only 11 members have paid their annual dues for 2016 so far. As one informant remarked, since the 49% 13% 7% 15%16% CONSULTANCY GOVERNMENT ACADEMIA INTER DEV PRIVATE SEC RSPI MEMBERS BY SECTOR PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 40 members can generally be characterized as professionals with a high level of education and income potential, the nominal fees should not be an obstacle. Informants from USAID, DAI and RSPI agreed that RSPI is still in an infant stage as an organization, and several observed a high likelihood that it may become inactive soon after the Project ends. RSPI members and leaders have become used to getting everything for free from the Project, and so far have contributed little as volunteers, according to DAI staff. Key members were exposed on various occasions to the volunteer-based ISPI model, including at an international conference in the U.S. In fact, the draft strategic plan reflects that approach by saying: “As a volunteer organization, the chapter leadership will take direct responsibility and action for all programs, communications and resource mobilization efforts.” Nevertheless, conversations with various informants show that leaders still have expectations for funding of operations from USAID. Research by the Team on civil society in Rwanda shows that few CSOs make use of volunteer labor to meet their human resource needs; volunteerism is not well developed in Rwanda. DAI allowed and even encouraged its Project personnel to become members of RSPI and even to serve on the executive, which may have contributed to dependency. Various informants noted that the future of RSPI would depend to a great extent on its ability to actively organize interesting events and expand its networks and membership base. Options mentioned include organization of fee-based training on performance improvement, assistance with CPT applications of other consultants, and continuation of the well-received knowledge café. To quote the draft strategic plan: “Without a good reason to come to meetings and events, there is a lack of motivation for professionals to join RSPI. As well, events need to attract a large enough audience in order to generate revenues to defray costs and to keep our member fees affordable.” CONCLUSIONS Relying on Project support, RSPI has helped to create awareness of the HICD approach in Rwanda, and there are high expectations and some positive energy around promotion of performance improvement more generally across institutions. As such, there is potential impetus for the organization to retain current members and mobilize new interest—especially on the heels of the recent conference in Kigali, which attracted great interest. However, RSPI is still a young and untested organization, essentially run by the HICD/R staff until now. The failure of most members to pay their annual dues may be evidence of lack of interest, low confidence in the executive, or simply lack of effort to collect the dues—or a combination thereof. RSPI has been assisted to develop a strategic plan, which describes the group’s ideas for the initial development of the organization and can be seen as a positive step. However, the leadership has (so far) failed to finalize the strategic plan when left by the Project to take the process forward, which raises additional doubt about genuine levels of commitment to the organization’s survival. RSPI may well not survive long after the Project phases out; its fate will be in the hands of its members, who must themselves determine whether the organization has enough value to invest their own time and resources in its development. Since the evaluators are not recommending anything more than basic financial and technical assistance between now and the end of the Project, it is not possible to predict how long it may take for the organization to become stable and self-sufficient. Even though it may not remain active, the organization did play a role in the Project as a platform for sharing information about HICD and other performance improvement approaches, and many connections made among professionals interested in performance improvement may continue to bear fruit. Activities that were well received could potentially be continued by other entities, such as NCBS or RMI, or another civil society group with a related mandate. Not all will be lost if RSPI does not thrive. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 41 RECOMMENDATIONS 1. DAI should continue to provide modest funding for activities of RSPI until the end of the Project, such as regular awareness-raising, networking and training events, to leverage the investments made already in the organization and in promotion of performance improvement approaches in Rwanda. However, activities should be identified and organized by RSPI leaders, rather than the implementer. 2. USAID and DAI should not provide funding for creation or staffing of an office for RSPI. Such funding would only perpetuate the dependent relationship that already exists, and encourage the executive to continue their passive role in the organization’s activities. 3. As soon as possible, DAI should work with the executive to convene a meeting to further explore and analyze strategies by which core functions of the RSPI can be sustained after the Project ends. These should include ways of attracting new members and continuing to offer activities and services that will encourage retention of members. 4. DAI should support an RSPI assembly before the end of 2016, where the future of RSPI can be discussed and the strategic plan finalized. Prior to that event (as soon as possible), DAI and USAID should communicate to all RSPI members their intentions regarding support following the end of the Project. 4.2. Does sufficient capacity exist among RSPI members to provide HICD interventions after the Project ends, and what type of USAID support for those members would be most suitable? FINDINGS A. Training and Certification As noted above, the Project provided considerable training opportunities to members of COMPRAC (and later RSPI). The two main areas of emphasis were HICD as a performance improvement approach, and the standards that any Certified Performance Technologist must demonstrate in their work. There are 10 standards that must be met by a CPT applicant, and workshops focused on explaining those standards and how to apply them in the implementation of performance improvement methods. Although not all participants completed the entire workshop series (10 sessions of 2-3 hours each), Project records show a high level of attendance for the first round. One DAI staff described how participants “had their minds opened on performance improvement. The certification was not the actual objective for us, it was a vehicle to attract people to the training.” Following the workshops, extensive coaching was also provided to help people navigate the arduous certification process, which requires in￾depth analysis and documentation of each candidate’s experience in using CPT standards. The two rounds of training and coaching ultimately assisted 15 candidates to become certified. This achievement was proclaimed as a major Project achievement, and the CPTs with whom the evaluators spoke were universally grateful for the support and proud of their achievement. Although not able to point to tangible benefits flowing from the certification (for example, additional contracts), several mentioned that they were using the principles in various aspects of their work. Other informants close to the process expressed doubts about the value of the CPT designation in the Rwandan context, where the designation is virtually unknown. When asked about the ability of RSPI members, including the CPTs, to competently carry out HICD￾based assessments or related consultancies, informants within the implementer and RSPI had reservations. Several of them mentioned that only two or three members would be able to lead a small- PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 42 scale HICD assessment, and assessment of a large entity would be a huge challenge. One international consultant remarked that most members still have limited analytical skills and lack the very broad base of knowledge about organizational structures and systems that is needed to lead a full-blown HICD assessment. Several informants observed that there have been limited opportunities for application of the skills and knowledge that CPT training participants have gained, which has lessened the impact. B. Internships The Year 2 Project Workplan laid out specific plans for at least 12 internships for Rwandan consultants involved in COMPRAC, and the first internships began in August 2013. The Year 2 annual report states that two consultants/CPTs were contracted to work alongside international consultants and DAI staff to conduct performance assessments. According to the report: “The organizations successfully completed their systems, and the interns now have certificates from HICD/R verifying their mastery of HICD principles.” On the other hand, Project personnel said that the contracted members demonstrated a low level of capacity for analysis of the CSOs, and added very little value to the assessments. One international consultant noted that the interns did not take the initiative to review documents or contribute to planning, and showed little commitment beyond attending meetings. At the same time, the interns expected to be paid at rates of US$300-400 per day, and were not inclined to work at low rates in exchange for the experience to be gained—which had been the Project’s expectation. The PMP indicator related to internships was dropped at DAI’s request by November 2013, suggesting that the Project discontinued this activity immediately after the initial experience. There is no mention of internships in the Year 3 report, and no further internships were mentioned by informants. CONCLUSIONS A number of professional development initiatives were provided to COMPRAC/RSPI members aimed at increasing technical knowledge and strengthening communities of practice. Those initiatives included presentations, workshops, coaching through CPT applications, international conferences and launching of RSPI as a platform for their professional development. However, the level of capacity in performance improvement methodologies, as far as could be determined by this evaluation, remains relatively low. Considering the level of funding and human resources required to achieve the certification of 15 persons, the return on investment in CPT seems doubtful—although in the Rwandan context, the value of any international certification as a “carrot” to attract and maintain the interest of learners cannot be denied. In fact, after achieving its target of 15 CPTs, the Project decided not to invest further in the designation. (It is interesting to note that in Georgia, arguably the flagship of HICD within USAID circles thus far, there are only two CPTs: the Georgian Chief of Party (COP) of the current project, and a U.S. citizen resident in the country. The Georgia projects opted not to prioritize the certification in their strategy, according to the COP.) The CPTs and other RSPI members came from a low starting point, and are still in need of opportunities for further training, coaching and on-the-job experience to learn how to apply skills. Although the initial Project experience with interns was by all accounts frustrating, it appears that a very important strategy for professional capacity development was dropped very quickly. Other options could have been explored and tried, especially given the serious capacity gaps that reportedly emerged from the first internships—which underscored the need for practical experience, even for stronger consultants. It is possible that engagement of consultants who were not CPTs would have been more fruitful, as the designation itself may have boosted expectations and lowered motivation of the selected interns. The Evaluation Team concurs with the view of a Project consultant: “There is need for commitment on the part of those being mentored to apply new and better ways of doing things”, in order for such methods to be effective. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 43 RECOMMENDATIONS  DAI should seek to provide more opportunities for interested RSPI members to apply new skills and deepen their experience by organizing twinning arrangements or mentoring by international consultants. As the need for consultant inputs may be minimal in the remaining Project period, USAID should support the involvement of selected RSPI members in twinning or mentoring arrangements related to other projects in Rwanda. This should be presented as a nominally paid learning opportunity and participants selected based on motivation, even if technical skills do not appear to be high.  DAI should sponsor more forums and events where RSPI members and others interested in performance improvement can exchange experiences and learn from others in a non￾competitive and collegial environment (as mentioned above under RSPI support). QUESTION 5 – PROJECT IMPLEMENTATION AND MANAGEMENT What lessons learned about project management and implementation can inform project management for the remainder of the Project life and future potential follow-on designs? FINDINGS A. Focus of Project The Project was designed and managed by the DG Office of USAID/Rwanda. It was described by staff of that office as a cross-sectoral initiative that aimed to target priority institutions in various sectors of interest to the mission, with a cross-cutting objective of addressing issues related to democracy and governance. During the final stages of Project design, the DG Office budget was cut significantly, at which time the Education, Health and Economic Growth offices (hereafter the “technical offices”) all agreed to contribute to the Project. That was reportedly in part owing to encouragement from the Mission Director, and initially there was considerable positive energy around the collaborative effort. The Contracting Officer’s Representative (COR) of the Project used various tactics to raise awareness in all offices of the HICD approach, and to keep the technical offices informed and engaged in Project decisions and implementation as they evolved. This proved to be a challenging task, for a variety of reasons mentioned by USAID staff: a zero baseline understanding of HICD, large and busy technical teams with competing priorities, and in some cases, a perception that close ties to a DG activity could present a risk to their core programming, due to political sensitivities in the GOR. Although the Mission Director was an early champion for the Project and the cross-sectoral approach, in early 2013 he became unable to play a supporting role due to a conflict of interest.33 The initial CSO selection process was somewhat confusing and complex due to the need to balance programmatic priorities within USAID while preserving a DG orientation, weak understanding and buy￾in among technical offices, and an implementer team that was not experienced in HICD and thus relying on subcontractor consultants from outside Rwanda to get the Project off the ground. The outcome was a set of six targeted CSOs: three health-focused organizations that had just been granted significant direct funding from the Health Office, and three agriculture-oriented organizations that were minimally engaged with the mission. (See Table 2 for description of the CSOs.) One more health-focused CSO was targeted from 2014. Only in late 2014 did the Project take on three organizations involved in education programming. 33 Near the end of Year 1, the Mission Director’s partner was hired as the second Chief of Party of the HICD/R Project, and a strict firewall approach was adopted to mitigate the resulting conflict of interest. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 44 These factors all contributed to a decline of interest and engagement within USAID, exacerbated by difficulties in describing and sharing the tangible results of the Project—largely due to the longer-term nature of the HICD approach, slow first year of implementation, and gaps in monitoring. The Evaluation Team found little evidence of efforts to link HICD/R interventions with the substantial investments in those same CSOs by the Health Office. For example, there was no mention of meetings or exchanges between the Project implementer and the Health Office team to discuss how inputs from one could leverage the other’s investment, or how gaps in one type of support could be met by another project. The Team also noted evidence of difficulties in identifying suitable and interested GOR targeted institutions, after MINAGRI. It is not clear to what extent USAID technical offices were consulted with respect to government targeting. Informants within the mission said that MINALOC was under consideration at one point, but USAID decided against it. The Rwanda Governance Board was actually selected for targeting and the assessment process started – at which point USAID decided to discontinue the relationship, for reasons not known to the Team. The selection of NCBS and NWC was based on requests coming from those entities themselves; neither is linked with any specific technical office of USAID. B. Human resources High staff turnover in the implementer’s team, especially but not only in Year 1, was remarked upon by various informants within and outside the Project. This was especially problematic because the Project was heavily dependent for HICD expertise on internationally-based consultants, whose efforts to build capacity of implementer personnel through mentoring were undermined by frequent changes in the staff. The level of turnover can be summed up as follows.  37 individuals served on the DAI staff between April 2012 and June 2016, including three COPs and four Deputy COPs.  Considering that 12 persons were active DAI staff at the time of evaluation, which was within the last nine months of the project, this translates roughly into three complete rotations (total 37 divided by current team of 12), although the staff composition did fluctuate over time.  Only four personnel hired in 2012 were still on the DAI team at the time of data collection in August 2016. Turnover of officials at all levels in government institutions was also problematic, according to numerous informants within those institutions and the implementer. Changes at senior levels made it necessary to re-introduce the Project to ensure buy-in, and made it harder to gain traction with line workers directly targeted with activities. As well, training participants were regularly moved into different jobs or left for other opportunities, including after many months of mentoring or training in the US. This made it difficult to perceive a translation of capacity into performance change of the targeted institution, although certain benefits may be accruing to another entity as the participant uses acquired skills and knowledge in their new role. Inadequate staffing levels in MINAGRI were noted in the PSP assessment, which described a “mismatch between the Ministry’s ambitious work plan and available personnel.” This may have contributed to difficulties with embedded consultants described under Question 1; in more than one case, senior Ministry officials insisted on maintaining consultants who were not performing well or not adding value in the view of the implementer. It is likely that these consultants were being used to supplement low staffing levels. HICD consultants contracted by DAI, including through KNO as subcontractor, were mostly of good quality according to informants, and several were very highly esteemed by targeted institutions. The implementer said that regular feedback from those working with consultants was solicited to monitor their performance. In fact, several informants commented that consultants were “too expert”, and they struggled to transmit their knowledge and advice in a way that could readily be absorbed by the PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 45 recipients of training or coaching. There were some quality issues with consultants (from KNO and elsewhere), as well as problems posed by lack of continuity, although in-country DAI staff worked to provide ongoing assistance. One informant mentioned that consultants “lose a large part of the spectrum of communications unless they speak both English and French”, which was consistent with the observations of the evaluators. Project staff and consultants also referred to the “front end loading” of the Project as a factor in stretching the available human resources. This was in reference to the fact that six CSOs were selected during Year 1, while the plan was for only one (per SOW for Project): “For the first year of implementation, the contractor shall work with the Ministry of Agriculture and Animal Resources, and one civil society organization to be determined in close consultation with USAID/Rwanda…. For years two, three and four of implementation, two new HICD initiatives shall be launched each year... The fifth year of implementation shall focus on consolidating and reinforcing ongoing and recently completed initiatives.” The decision to launch so many HICD interventions at once with heavy reliance on external consultants was also reported as problematic from a budgetary perspective. A major modification of the original budget was required after Year 1 to ensure sufficient human and financial resources for subsequent periods. An informant close to the Project described the situation in colorful terms: “Like in medicine – there is no point in spending a lot on diagnosis, if you have no money left for the cure!” C. Material support The only kind of material support offered by the Project to targeted institutions was the purchase of certain software and payment for website hosting for CSOs. Otherwise, all inputs were in the form of technical assistance and related investments, including travel as needed for training, meeting costs, etc. The possibility of “commodities” was mentioned in the Project Request for Proposals (RFP) and in the contract between USAID/Rwanda and DAI, but no grant fund was in the design, and the only examples of commodity support in the contract were laptops, software and publications. Several DAI staff said they understood that material support was not “part of the package” and only to be provided when absolutely necessary. On the other hand, key USAID staff thought the Project could have been more flexible with this type of assistance, in order to provide holistic solutions to CSO needs. The importance of material support for targeted CSOs was underlined by many informants – including Project personnel, CSO representatives, and some USAID personnel. The lack of such tangible support posed challenges for CSOs and government, who lacked sufficient human or other resources to devote to ongoing implementation of certain recommendations and actions that naturally flowed from them (ex. execution of advocacy plans, management of M&E database, etc.). As one informant put it: “we also need help in hiring someone who can work with us in some specific areas such as M&E”, in order for the learning to be translated into organizational change. The issue was also recognized in at least two PSPs for targeted CSOs, for example: “There are 27 specific recommendations described in the report within the six areas of emphasis. IMBARAGA will require funding and technical assistance from USAID or other international donors for most of these interventions.”34 D. Duration of Support Some informants felt that the duration of cooperation with each targeted institution was sufficient to allow for absorption of the changes being made, and no concerns were raised about this point in relation to government partners of the Project. However, most involved in the civil society component (as service providers or beneficiaries) said that the timing of support was a constraint. Several CSOs from the first round expressed the wish that Project assistance had been spread out over a longer 34 Imbaraga Performance Solutions Package, 2013, page 1. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 46 period of time, saying that they felt rushed in certain areas. Those CSOs were formally “graduated” from the Project in late 2015, meaning that cooperation lasted at least two years. One of the key Project consultants said that there is often pressure to engage with CSO clients during a shorter time period (6-18 months), but many CSOs cannot absorb this type of strengthening in multiple key areas so quickly. While there is no “standard duration”, the HICD expert suggested that three years would be a suitable period of engagement for the type of PSPs developed in this Project. CSOs would not receive intensive assistance during that entire period, but rather would benefit from ongoing coaching and advice from staff or other experts as they progressively roll out reforms within the organization. The evaluation of the FORECAST Program in Georgia also supported a long-term approach, recommending that: “The HICD component of the follow-on cross-cutting program should be designed to be more selective and focus on longer-term, more in-depth HICD assistance, even if doing so results in the selection of fewer institutions. This will allow for greater engagement through more focused collaboration, and more targeted assistance.”35 As a result, the ensuing RFP for HICD Plus urged the implementer to concentrate on quality and duration rather than quantity of activities. The Feed the Future HICD Strategy Review 2015 corroborated that view in strong terms by saying: “Virtually everyone with solid knowledge of the subject, every review and every report notes that to be effective in HICD, and especially in institutional strengthening, support must be built over a long time and on a relationship of trust.”36 On a related point with respect to graduation of targeted institutions, the FORECAST evaluation stated that: “Careful consideration should be given before withdrawing HICD support from those host country partner organizations that remain active, productive partners with USAID in improving their performance, so as not to diminish or reverse any performance gains already achieved.”37 E. Documentation of commitments The targeted CSOs expressed and demonstrated high levels of motivation to engage in the Project, as confirmed by various informants. During a focus group of first round CSO representatives and several interviews, CSOs said they had entertained doubts about the Project’s firm commitment to supporting them, since the MoUs signed by USAID and DAI between 2012 and 2014 with those seven CSOs were of very short duration (four months on average) and only covered the assessment stage of the process. Informants from USAID and DAI said that it was originally intended to enter into new MoUs for the implementation stage, once the nature of intervention was more clearly defined based on each assessment. The use of multiple MoUs to reflect the various stages of assistance is neither clearly contemplated nor ruled out by the HICD Handbook, but is consistent with recommendations by evaluations of HICD projects in Georgia (as described further under Question 3). According to the Project SOW, the use of subsequent MoUs was contemplated but optional: “Based on the performance solutions package, a second Memorandum of Understanding may be signed between the targeted institution and USAID to guide implementation of the performance solutions. The Memorandum of Understanding will outline, for example, who has responsibility for specific solutions. A targeted institution may, for instance, have responsibility for hiring staff, while the contractor may assist the institution in ensuring that staff members have the motivation, resources, information, knowledge, and skills to do their work.” 35 Evaluation of the USAID/Georgia FORECAST Program, page 20. Available at kdid.org/sites/kdid/files/resource/files/Evaluation%20of%20the%20USAIDGeorgia%20FORECAST%20Program.pdf 36 Feed the Future Human and Institutional Capacity Development (HICD) Strategy Review, May 2015, page 3. Available at http://pdf.usaid.gov/pdf_docs/PBAAE119.pdf 37 Evaluation of the USAID/Georgia FORECAST Program, page 20. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 47 Discussions with USAID and DAI generated various reasons for the limited use of MoUs with CSOs; ranging from the cumbersome process of getting new MoUs executed by targeted institutions and USAID, to DAI’s concern that MoUs could “create expectations” or “cause problems”, and in any case “would not have made any difference”. The time it took to get MoUs signed by all parties were variable, ranging from two weeks to two months. (Note: USAID/Rwanda staff consulted by the Team do not consider MoUs to be legally binding, but certain DAI staff thought that they were binding. As well, USAID advised that DAI could have entered into MoUs without USAID’s signature; it is not clear whether DAI senior staff contemplated this option.) It is important to note that Performance Solution Packages agreed with each institution following the assessments describe much of what the Project was promising to support, and some obligations of the partner are implied there. However, there is no document (formal or informal) in these first seven cases that clearly sets out each party’s commitments. MoUs designed for CSOs and government bodies in 2015 and 2016 have longer durations that are more reflective of the probable period of engagement, and use slightly different wording, but still do not define the obligations of either party following the assessment stage.38 F. Results Monitoring Project Level Monitoring USAID and DAI staff highlighted the difficulties faced in first coming up with overall indicators to monitor the Project (11 of 13 indicators were eliminated or substantially revised after the first year of implementation), and secondly in setting realistic targets for each year. As they pointed out, and the Team confirmed through Project reports and PSPs, the “key areas” selected for support and ensuing activities were somewhat different for each institution—although there were common themes among CSOs. Neither the types of intervention in each case nor even the organizations to be targeted were defined at the outset of the Project (or even at the start of each year), but rather were determined at various points in time. As noted under Question 2, the Project faced a range of challenges that contributed to a slow start, and did not meet many indicator targets for Year 1. For Year 2, a major revision of indicators and targets took place, and from then on the Project often encountered the opposite problem – the actual results were regularly exceeding the annual targets. (The Team’s compilation of PMP indicator targets and data to date is contained in Annex F to this report.) USAID and DAI staff described how they worked to address this by periodically adjusting targets, which did to some extent alleviate the problem. Nevertheless, challenges in this area hindered this evaluation in relying on the PMP indicators as truly indicative of Project achievements. Targets for several indicators, including number of days of coaching and number of PSP recommendations implemented, were consistently outstripped by more than a 20% margin, while others proved extremely unpredictable, such as the number of COMPRAC members attending HICD-related events and the percentage increase in CSO members receiving services. At least six out of 12 PMP indicators presently being tracked measure outputs rather than outcomes, and thus do not provide strong evidence of change in capacity or performance. In analyzing the outcome indicators, the evaluators observed a number of weaknesses. For example:  Number of PSP recommendations implemented. This has been difficult to predict as the number of recommendations in each PSP varies widely. The indicator includes recommendations 38 MINAGRI declined to sign an MoU with the project; the relationship challenges underlying that refusal have been described in Question 2. There has never been an MoU with the RSPI, although it has been registered since mid￾2014. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 48 implemented with and without Project support, which introduces doubt as to how well the data reflect Project results. An indicator based on percentage (number of recommendations implemented/total recommendations) and specifically linked to Project assistance could have been easier to predict and more useful in assessing results.  Percentage change in number of CSO members receiving direct services. Although this was claimed as a key indicator of impact on CSO performance, it is difficult to attribute changes in the reach of CSO services to Project activities. This is due to the dependency of CSOs on donors and the myriad variables affecting success in CSO fundraising over time. Based on comments by CSO informants and knowledge of donor processes for selection of grantees, it is probably safe to assume that the establishment of various written policies and plans in each CSO was helpful in convincing potential donors of the credibility and management capacity of these organizations— which in turn may have contributed to approval of applications for funding. The Year 3 target was exceeded by 713%, which illustrates the difficulty in predicting change in this indicator. Two outcome indicators relied on monitoring tools that had the advantage of being relatively simple and easy to administer on an annual basis, but also had several disadvantages. The Organizational Capacity Index or OCI was essentially a checklist of documents (only three of 59 items were not documents of some kind), which the implementer personnel would verify with each CSOs, once per year. The list included highly relevant items such as strategic plan and staff job descriptions, alongside others that may not be relevant or particularly important for some organizations, such as copy of invitation for conference and sample newsletter. The list had no weighting of these items to indicate relative importance nor scale to show progress towards a higher level of capacity. The targeted CSOs demonstrated significant progress towards fulfilling this checklist, as described under Question 1. The index does not measure the degree of utilization or effectiveness of the items on the list, therefore it could not be used to assess change in performance of the CSOs. The impact perception survey is described in the PMP as “an annual survey of representatives of partner institutions to quantify perceptions of the effectiveness and impact of the PSP recommendations that have been implemented. The survey will rate interventions based on outcomes and impact on a one-to￾five basis, with five indicating strong agreement and/or impact.” Originally designed for only government institutions participating in the Project, the tool was ultimately used with all targeted institutions. The survey was explained and completed forms collected in person by DAI staff, although the responses were not viewed or read out during those meetings. In both 2014 and 2015, data collection from some institutions was done by the COP and Deputy COP, while others were done by the M&E Specialist. The survey generated consistently high marks for both sustainability and impact, as described under Question 1, which may have reflected the informants’ views at the time. However, as external evaluators, the Team has not been hearing such positive reactions from targeted institutions – particularly those engaged in the early stages and who are no longer primary targets of the Project. It should also be borne in mind that only one individual from MINAGRI and one from NCBS participated in the survey, although three to four were usually surveyed from each CSO. While the Team did not pose the same questions to the same individuals, the lower level of extremely positive views of the Project raises doubt about the validity of the survey data. The evaluation did not explore how the targeted institutions viewed this survey process, but one possible explanation is that their responses to the survey were influenced to some extent by response bias. The Project conducted evaluation of formal training by using feedback forms completed by participants, and through systematic pre and post testing of knowledge. These were commendable efforts to evaluate the effectiveness of training by the Project – covering the first two levels of the well-known Kirkpatrick PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 49 Levels of Training Evaluation.39 While the tools mentioned above (OCI and perception survey) represented attempts to gather information about the third and fourth levels, “utilization/behavior change” and “results” or benefits for the organization or entity, their scope and reliability was limited. Targeted Institution Level Monitoring The Project’s monitoring of outcomes at the level of targeted institutions was severely limited. The initial PSPs prepared by KNO consultants for the Project included annexes with long lists of illustrative indicators that were intended as a starting point in each institution for establishment of systems to track change flowing from implementation of PSP recommendations. Revisions of the Project PMP in Year 2 significantly cut back the role of the implementer in such monitoring. The PMPs for Year 2 and subsequent years includes the following wording: “It should be noted that HICD/R has two sets of indicators: Level I Indicators: These include output, outcome, and impact indicators that are used to measure the performance of the HICD/R project. These indicators are general enough to apply to multiple HICD/R partner institutions, and are designed to inform the decisions of project management and USAID. Level II Indicators: These indicators include specific impact/performance-change indicators that have been recommended to partner institutions and approved Performance Solution Plan (PSPs). These indicators are specific to the institution for which the PSP was written. It should be noted that HICD/R will attempt to record progress against level II indicators through quarterly interviews with partner institution representatives in order to provide project management and USAID with a wider barometer of performance change, but the project is not equipped to directly verify document Level II indicators given its current resources. This Monitoring and Evaluation Plan discusses Level 1 Program Indicators only. As already observed under Question 1, the responsibility of tracking Project-related change at the level of targeted institutions was never clearly allocated or prioritized. In the view of a key DAI staff member: “It was the responsibility of the partners to track, but over time as their needs evolved, they prioritized indicators as part of their M&E support. Most partners have limited budgets and are required by their donors to track technical more than operational performance.” Interviewed CSOs thought that tracking of indicators related to the Project was not their responsibility. One DAI staff member said that the Project staffing was not sufficient for them to take on this level of monitoring (there was one person dedicated to M&E), which is also implied by the PMP wording above. However, others noted that the DAI staff in general was not busy much of the time, especially when international consultants were not on the ground. While the Project has invested and continues to invest significant time and resources in M&E capacity development through mentoring of most partners (all CSOs and some government), the new or updated systems have not incorporated indicators or tools that could generate data about change in most areas targeted by the Project, which were focused on management performance (ex. staff retention rates, number of issues found by audits, percentage of strategic plan or business development plan being implemented). In a situation where implementer staff and consultants were apparently not promoting the inclusion of such indicators in these systems, it is unsurprising that the institutions claim to be unaware of any role in collecting monitoring data for the Project. They have also reportedly not been asked by the implementer for any data on technical performance that may be captured by those 39 Information available at http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 50 monitoring systems. As another sign of the low priority given to this level of monitoring, the inclusion of illustrative indicators in PSPs was discontinued by the implementer. Based on the evaluators’ understanding of HICD approaches, including reference to the HICD Handbook, high priority would normally be given to this kind of monitoring, to ensure that the impact of support provided to organizational development is measured, and that any issues can be identified and addressed. This understanding is corroborated by the Project’s Scope of Work (see excerpt in text box below --emphasis is ours), which emphasizes the importance attached to this element by the Project design. “Step 5” of the HICD process is one of the deliverables of the Project, and thus a fundamental element of its implementation. Annex I contains the full text of the SOW, including this step. CONCLUSIONS In spite of considerable efforts within the mission, multiple factors beyond the Project’s control combined to undermine its cross-sectoral nature, which contributed to lack of understanding of the model and its results in USAID/Rwanda. While trying to maintain the DG orientation of the Project, it proved difficult to accommodate the interests of USAID technical offices, notably at the CSO selection stage. In the end, the effects of the Project were diluted by these “mixed motives”; DG objectives were advanced in terms of advocacy and generally more robust CSOs, but it is unclear whether any difference was made in achieving the objectives of those CSOs in technical sectors. For almost three years of implementation, only the Health Office had a significant stake in the Project’s work with CSOs. When immediate results relevant to their objectives were not demonstrated, the engagement of the technical offices dropped off sharply. Due to complications with government targeting, the Project had a heavier focus on CSOs than anticipated. On the positive side, HICD approaches created an entry point to working on certain DG￾related priorities such as civil society capacity and advocacy, allowing USAID to work slightly “under the radar” on issues that are politically sensitive in Rwanda. The absence of a stable corps of staff for the implementer was a factor that made it difficult to solidify relations with counterparts in government and civil society, and to develop essential skills and Step 5: Operationalize Performance Monitoring System A hallmark of HICD is its results-oriented nature. In order to gauge the performance gains of partner institutions, it is important to establish measurable performance goals for each area of performance targeted through HICD initiatives. The HICD contractor shall assist host country partner organizations with establishing baseline performance data along with an appropriate performance monitoring and management system that is tailored to the context of each organization… The performance data being collected should be directly linked to the desired and actual performance as defined in the performance assessment…While this information will be mainly used by host country partner organizations for performance and strategic management, it should also be accessible to the Mission for its reporting and management needs. While the internal performance monitoring and management system is being established or strengthened at host country partner organizations, the contractor shall conduct periodic performance re-assessments in order to gather data on the indicators established in the initial performance assessment. The timing of such re-assessments shall be approved by the COTR… Once the internal performance monitoring and management system is operational, host country partner organizations should be able to transmit relevant performance data and information directly to the Mission and other stakeholders. The contractor shall ensure that the commitment to gathering this data, and access to it, is explicitly stated upfront in the initial Memorandum of Understanding with each host country partner organization. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 51 knowledge in core areas within the team, in spite of ongoing mentoring and other support by senior DAI staff and consultants. The starting point for relevant capacities was low across the board, given the lack of any previous HICD activity in Rwanda. Frequent changes of key government personnel (and to a lesser extent, CSO staff) also posed a significant barrier to sustainable performance improvement at both human and institutional levels. The international consultants engaged by DAI and KNO (including one of four embedded at MINAGRI) were largely recognized as having a high level of expertise and being good communicators. However, as is often the case with short term consultants, their unfamiliarity with the context and the languages most widely spoken in Rwanda were obstacles to effectiveness, the continuity of support was compromised, and the costs associated with the extensive external support needed by the DAI team were high. The current language situation in Rwanda is extremely unusual, as they transition towards an English-based education system. This poses unique challenges in terms of optimizing communications, both oral and written, especially when consultants are not fluent in either French or Kinyarwanda. With respect to material support, it is unclear whether there was a meeting of minds between DAI and USAID on this point. The lack of material and financial assistance for CSOs has limited their progress towards performance improvement in certain areas, especially where staffing was constrained. Motivation may also have been negatively affected for those CSOs not receiving significant USAID funding. In general, technical assistance that is not accompanied by material support can only achieve meaningful results and change for a small subset of CSOs in such a context, such as those with significant funding that is flexible enough to complement training, launch new systems, etc. Longer duration periods of support at lower intensity can be more valuable when using holistic HICD approaches, especially for smaller organizations, as discussed in Question 3 in relation to HICD Suitability. Ongoing assistance is widely agreed as extremely important in HICD, and is especially important to deal with institutional changes that were not anticipated, to ensure that reforms survive and momentum continues. DAI appears to have made a management decision not to enter into more than one MoU with targeted institutions, although it was realized that the duration and scope of the initial MoUs was not sufficient to cover the actual collaboration, and the contract contemplated follow-up MoUs. It is possible that the lack of clearly defined obligations based on a current agreement affected the commitment level of CSOs in the first round, and it may have contributed to the misunderstanding about monitoring responsibilities, as described above. The Evaluation Team concurs with the conclusion of a key Project consultant: “Multiple MoUs are an administrative burden and often time-consuming, but I still think provide good value.” The nature of the Project—based on a fairly open-ended methodology of identifying targeted institutions over time, identifying and prioritizing needs, and then identifying and facilitating tailored solutions—does not easily lend itself to development of robust monitoring tools that capture the outputs and outcomes of the Project as a whole. This difficulty is not specific to this Project; evidence from other HICD programming and experts also indicates ongoing challenges in identifying solid indicators at project level for this type of intervention. The challenges start with selection of suitable indicators, especially when one aims to go beyond simple outputs, and continue with the process of setting realistic targets for each period of implementation. DAI and USAID worked together to counter these difficulties, revamping both indicators and targets repeatedly, with limited success. Several indicators suffered from weaknesses in their construction, as described above, while others relied on tools that were not optimal for measuring meaningful outcomes in an objective manner. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 52 In fact, the more useful and interesting data for an HICD project is derived from monitoring of change in the targeted institutions, as illustrated in the HICD Plus final report from Georgia.40 The importance of this type of monitoring is also underlined by the USAID HICD Handbook. However, in this case, there was no shared understanding of who would monitor the results of the Project’s support on each institution. Consequently, this important component of the HICD approach was prioritized by nobody and ultimately, not done. This is especially hard to comprehend, given that the implementer engaged several M&E experts to assist institutions with monitoring plans and systems—and yet the resulting sets of M&E plans for CSOs do not incorporate indicators on management performance that could provide meaningful data on progress towards direct outcomes expected by this Project. The M&E Plans do contain plenty of information on technical performance, some of which would also be relevant to the Project, given that strong technical performance is often a reflection of strong management performance. However, that data appears not to have been considered as pertinent by the implementer. After a careful review of the Project’s Scope of Work, it appears to the evaluators that the decision not to monitor the so-called Level 2 Indicators may have run contrary to the terms of the contract for the HICD/R Project. This is an important lesson learned for the future management of potential follow-on HICD/R projects. RECOMMENDATIONS Current Project 1. The DG office, with the support of mission leadership, should liaise regularly with other contributing offices in USAID to update them on activities and outcomes of the Project, and to explore options for additional assistance (as needed) to targeted institutions after the end of the Project, as relevant to each office. The implementer and institutions should be engaged in this process as appropriate. (This is also relevant for future projects.) 2. For institutions where the Project is still supporting development of M&E systems, DAI should work with targeted institutions to ensure that selected indicators related to management performance improvement are included. While this data may not be available to the implementer or USAID during the Project period, it should be useful to the management of these institutions and may incentivize them to carry on with reforms begun under the Project. 3. DAI should attempt to gather more data from targeted institutions with respect to changes in their technical performance during and following Project support, and analyze to what extent those may be linked to improvements in management performance. (This has been attempted in relation to the PMP indicator of number of CSO members being reached by services, but difficulties arose with reliability of data.) Future programming 4. USAID should base decisions about the structure and management of any future HICD programming on an in-depth analysis of the priorities of the mission as a whole and of the various technical offices, the evolving context (especially as it relates to DG work), and the budgets and personnel available to different offices in the mission. As noted in the Question 3 recommendations, the lessons learned in Georgia may be very useful in this respect, including their experience of management by the Program Office—which may be suitable if a cross￾sectoral arrangement is contemplated. 5. In any future HICD-based project, USAID and implementers should involve relevant staff from the technical offices of the mission in the assessment (and re-assessment) of performance of 40 Available at http://www.chemonics.com/OurWork/OurProjects/Documents/Georgia_HICDPlus_Final_Report.pdf PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 53 targeted institutions. This would not only contribute to the technical expertise of assessment teams, but also strengthen links between the HICD intervention and the technical offices and help ensure that the chosen priorities are consistent with overarching objectives in that sector. This is particularly important in situations where one or more technical offices is working directly with a targeted institution, and may be aware of specific areas of need (especially in relation to technical performance), but is also relevant in other circumstances. 6. USAID should design future project staffing with due consideration for the relatively low level of expertise in HICD and organizational development generally in Rwanda. The inclusion of one or more long term experts may be advisable in addition to the COP, who has heavy management responsibilities. Such expert(s) should be on the ground continuously for at least one year, depending on the project staffing structure. (Note: The cost of having one expert resident in Rwanda for one year is roughly equal to the cost of a short-term consultant visiting four times per year, for 3-4 weeks each time.) 7. USAID and implementers should, whenever possible, try to recruit consultants and key personnel with either French or Kinyarwanda plus English language proficiency, for at least the next five years, to mitigate communication gaps as Rwanda continues its transition to an English￾based society. As well, projects should make allowances in budgets and schedules for interpretation of training and coaching, and the translation of key documents from English to Kinyarwanda. 8. USAID and implementers of HICD projects should maintain a flexible approach to provision of material or financial support to CSOs targeted with capacity development, to be able to respond to situations where genuine progress toward improved performance requires resources that are not otherwise available. That assistance could include equipment or short-term personnel, but in all cases should be directly necessary for achievement or follow-through of the capacity development objectives agreed with the CSO. This kind of complementary support may also help to minimize perceptions that solutions are to some extent driven by the technical assistance resources of the project, which emerged under Question 2. As USAID considers future capacity development interventions, financial support and the duration of support should be taken into consideration. 9. USAID and implementers should prioritize strong M&E systems, tools and follow-through as a fundamental element of HICD programming, and in fact of any performance improvement initiative. Such monitoring should aim to include analysis of technical performance as a higher level indicator of management performance—even if assistance to a particular institution is focused on management functions. This enhanced monitoring may entail increased staffing levels and/or specialized training, as this type of project is unusually demanding in terms of M&E. Implementers should use enhanced instruments for monitoring outcomes, including stronger CSO capacity assessment tools, such as the USAID Organizational Capacity Assessment,41 and exploration of anonymous methods of surveying targeted institutions on a periodic basis to reduce chance of bias. 41 Available at https://usaidlearninglab.org/library/organizational-capacity-assessment PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 54 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 55 ANNEX A: STATEMENT OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK C.1 PURPOSE The purpose of this mid-term performance evaluation is to assess progress that the Human and Institutional Capacity Development in Rwanda (HICD/R) project has made to date in achieving its objectives and document the effectiveness of the project’s approaches and responsiveness to overall capacity building needs in Rwanda. The evaluation will help USAID/Rwanda identify key technical and management systems strengths, weaknesses, opportunities, and threats in the implementation of the HICD/R project. The findings will be used by USAID/Rwanda and the HICD/R project implementer to develop and support activities to strengthen the project’s implementation in order to improve performance over the remaining life of the project, and help USAID/Rwanda determine what steps to take in order to sustain the achievements of the project. The primary audience for this evaluation is the USAID/Rwanda Democracy & Governance Office. Other users include the Health, Economic Growth and Education offices who have all contributed funds to implement the project in ways that benefit their portfolios. Other key audiences include the Program Office and Front Office, in the context of their collaboration to implement the HICD/R program. C.2 BACKGROUND Rwanda’s successes in healthcare and economic growth are already remarkable, but for them to be sustainable over the long term, Rwanda’s local institutions have to be high performing. In May 2012, USAID/Rwanda initiated the five-year Human and Institutional Capacity Development in Rwanda (HICD/R) project. The goal of the project is to improve the overall performance of targeted Rwandan institutions in a manner that supports USAID/Rwanda’s development outcomes and fosters the long￾term social, economic, and environmental sustainability of the country. This is to be achieved by helping the Rwandan government and civil society organizations (CSOs) to be effective learning institutions and for CSOs to develop the capacity to manage direct U.S. Government support. Building this capacity is a key aspect of the USAID Local Solutions agenda to transition toward the use of host-country systems to provide assistance. The project provides technical assistance, material support, and knowledge and skills transfer services to build the capacity of targeted organizations and reinforce ongoing reforms and development activities being carried out by the GOR and civil society. This is USAID/Rwanda’s first HICD project. It is cross-sectoral, receiving funding from the USAID/Rwanda Democracy and Governance, Economic Growth, Education and Health offices. It is managed within the Democracy and Governance office. The project's implementation approach is organized around the following three components: 1. Government Institutions: Provides focused, HICD-based assessments and Performance Solutions Packages (PSPs) to selected Government of Rwanda (GOR) counterparts and assists in PSP implementation. During year three, the project concluded two years of assistance to the Ministry of Agriculture and Animal Resources (MINAGRI) and completed a performance analysis of the National Capacity Building Secretariat (NCBS)42. Year four is focused largely on implementing a 42 Previously the Public Sector Capacity Building Secretariat PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 56 Performance Solutions Package (PSP) for NCBS and also institutionalizing a performance improvement curriculum within the Rwanda Management Institute (RMI). 2. Civil Society Organizations (CSOs): Provides focused, HICD-based assessments and PSPs to selected CSO counterparts and then assists in PSP implementation. The project concluded assistance to the six CSOs in the first group of organizations selected to participate (namely: Ingabo, Imbaraga, CCOAIB, FXB, CARITAS, and the African Evangelical Enterprise) and completed performance analyses of four additional CSOs (namely: Never Again Rwanda, Adventists Development and Relief Agency, YWCA, and Society for Family Health) added to the program in FY2015. Year four will focus largely on implementing the PSPs for the new CSOs. 3. Rwanda Society for Performance Improvement (RSPI): Develops a community of practice for professional managers, consultants and academics interested in learning from and contributing to the body of performance improvement knowledge in Rwanda. The RSPI is a chapter of the International Society for Performance Improvement (ISPI). HICD/R provides training, technical assistance, and event sponsorship to RSPI and its members. The HICD Approach The HICD framework is based on the following model that provides a systematic process for analyzing and improving performance: HICD is a model of structured and integrated processes, designed by USAID to identify root causes of performance gaps in USAID Partners (e.g., ministries, civil society organizations, and other host￾country organizations) and address those gaps through a wide array of interventions. The model is designed to assist USAID partners with practical methodologies and tools to clarify and fulfill their mandates, and to provide the highest level of service to constituents and stakeholders. 1. Institutional Analysis: Once the target institution is identified and a Memorandum of Understanding is signed, an institutional analysis is conducted to examine the system in which the organization functions. This includes assessing its vision, mission, goals, strategies, and culture. The analysis will solicit the perspectives of management, clients, communities, employees, and other stakeholders. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 57 2. Identification of Desired Performance: Based on observations and interviews of staff, clients, and other stakeholders, as well as a review of records and other documents, the analysis identifies the desired performance of the institution. Desired performance takes into account international or national standards. The description of desired performance creates a manageable set of objectives for the process to follow. 3. Description and Measurement of Performance Gap: In addition to describing desired performance, the analysis describes and measures, where possible, the performance gap, which is the difference between the desired and actual performance of the institution. It is important to develop measures and collect baseline data for performance whenever possible. This will allow for later measurements to assess relative progress, as well as to inform internal performance management and capacity building at the selected local partner organizations. 4. Root Causes of Performance Gaps: The analysis then identifies the root causes of these gaps. Root causes are linked to the performance factors that affect institutions and people in doing their work: information, resources, incentives, knowledge/skills, capacity, and motives. Linking the root causes of performance gaps to these performance factors helps HICD practitioners generate solutions that address the root causes. Finally, solutions are developed to address the root cause of performance gaps and the related performance factors. These solutions are ranked and selected according to cost, benefit, priority, feasibility, or other criteria. 5. Implement Performance Solutions (Interventions): Using the information and recommendations from the analysis, the partner institution, with support from stakeholders and the Contractor, implements the selected performance solutions. It is critical during this stage of the process to maintain an environment of transparency and manage the change process by consistently communicating the intended results of the HICD initiative to staff and stakeholders. 6. Monitor and Evaluate Performance: The partner organization, in consultation with its stakeholders, keeps the solutions on track and evaluates performance on an ongoing basis to remeasure the performance gap, assess the effect of the solutions, and use the feedback to recalibrate efforts to improve performance. The same performance objectives and measures that were examined during the performance analysis stage should be monitored by the partner organization. An internalized performance monitoring and management system will support local partners in their efforts to continually improve performance independent of external assistance. Project Objectives HICD/R focuses on enabling targeted government institutions and civil society organizations to achieve the following objectives that constitute the building blocks for effective organizations and institutions: 1. Determine a clear purpose and vision 2. Align organizational structures and efforts with their vision and purpose 3. Perform work effectively and efficiently 4. Motivate staff by ensuring that they have the resources and support necessary to do their jobs well 5. Coordinate with other organizations and institutions to avoid duplication and ensure that they work collaboratively when seeking the same goals and objectives 6. Strengthen internal and external communications 7. Function as “learning organizations” that are more resilient and able to adapt to changing circumstances and challenges PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 58 Expected Results 1. Institutions develop strategic visions that further their respective development agendas 2. Institutions understand and incorporate best practices in organizational development theory, helping them to become learning organizations and incorporate systems thinking and other tools into their institutional repertoire 3. Information and communication technologies are strengthened to increase efficiency and effectiveness 4. Professional staff skills are improved to increase capacity to perform on the job 5. Communication and information flow, incentives, and motivation are increased to address perceived shortcomings in interagency cooperation 6. Targeted institutions develop clear plans for achieving desired staffing levels and competencies to enable them to better perform critical functions 7. New or improved data collection methods are developed for performance monitoring Implementing Partners The HICD/R implementing partner, Development Alternatives Incorporated (DAI), is responsible for management and execution of all aspects of HICD/R. The HICD/R subcontractor, Kelly Novak Opportunities (KNO), provides technical assistance in conducting HICD assessments and in developing PSPs. C.3 GENERAL PROGRAM PARAMETERS Results and Goals This evaluation is expected to assist USAID/Rwanda in identifying those aspects of the HICD project that have been particularly effective or ineffective and provide guidance on changes that can be made to the project to improve its sustainability. A successful evaluation will support improved management for the remainder of the current project and provide lessons learned for future potential follow-on work. HICD/R supports the USAID/Rwanda Country Development Cooperation Strategy (2015-2019) Development Objective (DO) 2 (Improved conditions for durable peace and development through strengthened democratic processes) through direct contributions to the underlying Intermediate Result (IR) 2.1 (Increased civic engagement and consultation and decision-making at all levels) by developing the capacity of CSOs to engage in evidence-based advocacy. Additionally, the project directly supports Sub-IR 2.1.1 (Improved performance and engagement by CSOs and GOR entities) in its holistic approach to performance improvement for selected partner institutions. Cross-cutting Pillar 1 (Gender-sensitive approach to development, with a particular focus on women, children, and vulnerable populations), and Cross-cutting Pillar 2 (Institutional capacity building) are also supported by the project through Component 1 and 2 activities. HICD/R seeks opportunities to provide training, skills and Goal: Accelerating Rwanda's progress to middle income status and better quality of life through sustained growth and reduction of poverty DO2: Improved conditions for durable peace and development through strengthened democratic processes IR 2.1: Increased civic engagement and consultation in decision￾making at all levels Sub-IR 2.1.1: Improved performance and engagement by CSOs and GOR entities PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 59 knowledge to current or potential women leaders in Rwanda. Additionally, HICD/R assistance to health and education CSOs directly supports youth and vulnerable populations. This five-year project has 12 indicators (including two Feed the Future standard indicators) to report on: 1. Number of partner institution representatives receiving HICD/R-supported training 2. Number of days of coaching and mentorship provided by HICD/R to partner institution representatives 3. Average variance between pre- and post-training test scores (indexed) 4. Number of civil society organizations receiving HICD/R (USG) assistance engaged in advocacy interventions (2.4.1-9) 5. Number of food security private enterprises (for profit), producers organizations, water users associations, women’s groups, trade and business associations, and community based organizations (CBOs) receiving USG assistance 6. Number of HICD/R performance solution package recommendations implemented 7. Score in percent of combined key areas of organizational capacity among USG direct and indirect local implementing partners 8. Number of COMPRAC members earning certified performance technologists (CPT) designation 9. Government institution impact perception scores 10. Percentage change in HICD/R-supported CSO membership receiving direct services from their CSO 11. Percentage increase in COMPRAC member attendance and participation in HICD-related professional forums and meetings 12. Percentage change in the number of RSPI members that report using performance improvement approaches in their work Key Evaluation Questions The Performance Evaluation will respond to the following questions and provide recommendations, where appropriate. These questions have been formulated based on the specific objectives outlined above. The evaluation will assess project implementation from April 2012 to April 2016. The evaluation will make conclusions based on findings, identify opportunities and challenges, and formulate recommendations for improvements across existing program activities. The key evaluation questions are: 1. To what extent have the project’s specific interventions improved the management and technical performance of targeted Rwandan institutions and CSOs? 2. What key factors may have hindered or aided the achievement of capacity building activities within CSOs vs. government institutions? 3. To what extent are Rwandan institutions and CSOs still in need of capacity development support that should be addressed in a future project in order to leverage and sustain USAID/Rwanda investments? 4. What capacity development support does the Rwanda Society for Performance Improvement (RSPI) need to sustainably continue HICD interventions beyond the life of the project, both with USAID/Rwanda financial support in the near term, and without USAID/Rwanda financial support in the long term? How long will it take to build such capacity? PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 60 5. What lessons learned about project management and implementation can inform project management for the remainder of the project life and future potential follow-on designs? Approximately 100 individuals from five stakeholder groups are familiar with the HICD activity and can provide information to help answer the evaluation questions. These include Government institutions, CSOs, RSPI, beneficiaries, DAI, and USAID staff. C.4 TECHNICAL REQUIREMENTS Evaluation Design and Methodology The Contractor evaluation team will use available quantitative data and collect qualitative data to conduct the evaluation. The evaluation methodology will include, but not be limited to: 1. Review of key program documents. USAID/Rwanda will provide a detailed document review matrix that lists each relevant document: the Statement of Work from the USAID/Rwanda contract with DAI (Attachment 5); the HICD/R project’s annual work plans, M&E Plan, M&E Indicators and Results (Attachment 3); statements of work from sub contracts, and progress reports; USAID/Rwanda Country Development Cooperation Strategy (CDCS) which is available at: https://www.usaid.gov/sites/default/files/documents/1860/Rwanda-CDCS-ApprovedPublic￾Version.pdf, and; assessments and performance solution packages for all supported institutions and CSOs, among others. 2. Key informant interviews with a cross-section of HICD/R project staff, sub-partner organization staff, national and local government officials, CSOs, and other program stakeholders. Key informant interviewees should include both men and women. USAID/Rwanda will provide a list of contacts that the Contractor should consider for key informant interviews, but the Contractor should be prepared to expand this list. A comprehensive list of the roles of all key informants will be included as an annex to the final evaluation report. The sample selected should be robust and capture diverse stakeholder perspectives in order to ensure objectivity of evaluation findings. 3. Approximately four to five focus group discussions with a cross-section of male and female GOR officials and program beneficiaries of the HICD project. A comprehensive list of all focus groups with details such as gender and geography will be included as an annex to the Final Evaluation Report. 4. Review of the data generated by the implementing partner for ongoing project monitoring and evaluation (M&E). The HICD/R project will provide these documents. 5. Review of M&E data and reports being tracked through the AIDTracker Plus reporting system and the HICD/R project M&E Plan. Relevant indicators and indicator data from USAID/Rwanda’s annual Performance Plan and Report (PPR) and Performance Management Plan (PMP) can be found in Attachment 2. The HICD/R project indicator data from its M&E Plan can be found in Attachment 3. C.5 SERVICES/TASKS REQUIRED The Contractor will provide the following deliverables. Upon award, USAID will provide key program documents that should serve as the foundation for the evaluation. All written documentation for submission by the Contractor to USAID/Rwanda must be in English. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 61 1. Work plan: The draft work plan will include the evaluation design, methodology, and draft data collection instruments that will be pilot tested. Specifically, the work plan will include: a. The schedule of anticipated activities and logistical arrangements for the overall evaluation study plan, data collection activities, and analyses. b. The participant selection criteria and/or sampling plan. c. A detailed evaluation design matrix that links the evaluation questions in the SOW to data sources, methods, and the data analysis plan. (Attachment 4) d. Detailed plan for data collection, entry, and analysis. e. Plan for synthesis (from analysis to findings) and report completion for all data collection efforts. f. Draft survey questionnaires and other data collection instruments, or their main features, along with a detailed plan for how data collection instruments will be developed. g. Methods for selection and training of data collectors, data entry personnel, and data analysts. USAID offices and relevant stakeholders will provide comments within eight working days through the COR. The Contractor will revise and submit a final work plan one week before travel to Rwanda. 2. In-briefing: Within one working day of arrival in Rwanda, the Contractor evaluation team will participate in an in-briefing with primary stakeholders in USAID/Rwanda to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. 3. Data Collection and Interim Meetings: During a two week data collection period, the Contractor evaluation team will periodically brief USAID stakeholders on the progress of data collection, challenges, and impact on the timeline. If desired or necessary, weekly briefings by phone can be arranged. 4. Draft performance evaluation report: The Contractor will report on and conduct a preliminary findings workshop on key findings, initial conclusions and recommendations within five days of completing the data collection. 5. Final Performance Evaluation Report: Within ten business days of receipt of comments from USAID/Rwanda, the Contractor will submit its final Performance Evaluation Report. This Performance Evaluation must comply with the USAID evaluation policy requirements, which are available at the following link: http://www.usaid.gov/sites/default/files/documents/2151/USAIDEvaluationPolicy.pdf C.6 PERSONNEL The Contractor evaluation team will have demonstrated capability to implement successful evaluations. Required personnel include: a Team Leader, a Subject Matter Expert on Organizational Development, a Local or Regional Evaluation Expert, and a local Logistics Coordinator. The following skills and experience will be represented on the evaluation team: • Demonstrated experience in conducting qualitative research • Evidence of strong group facilitation skills • Organizational Development sector expertise and experience PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 62 • Kinyarwanda language skills • Team development and management experience • Logistics experience • Communication skills (including report writing and presenting) USAID/Rwanda staff may participate in the evaluation in a limited capacity to support the evaluation team. This may include witnessing the data collection. a) Team Leader The Team Leader’s primary role is to oversee and coordinate all evaluation tasks related to this evaluation, guide other team members and make sure that all deliverables are of high quality and delivered on time. The Team Leader is ultimately responsible for ensuring the production and completion of a quality report, in conformance with this statement of work and USAID’s evaluation standards. b) Subject Matter Expert on Organizational Development The Subject Matter Expert on Organizational Development is responsible for supporting the Team Leader and Evaluation Expert. Her/his input is critical in the development of Focus Group and Key Informant Interview questions, interpretation of results, and recommendations based upon her/his expertise in organizational development. c) Local or Regional Evaluation Expert The Evaluation Expert is responsible for providing local context and advice to the evaluation team and coordinating technical analyses related to her/his area of expertise. She/he is involved in the planning, qualitative data collection, analysis, and drafting of the report, as well as other relevant tasks as determined by the Team Leader. d) Logistics Coordinator The Logistics Coordinator is responsible for assisting the team in qualitative data collection coordinating and scheduling meetings, focus groups, key informant interviews, transportation and other administrative services as required. C.7 SCHEDULES AND LOGISTICS The total duration of the evaluation will be 11 weeks. The team will spend two weeks in Rwanda collecting data and participating in an in-briefing and out-briefing with highlights of findings. The team will coordinate logistical arrangements with the USAID/Rwanda Office of Democracy and Governance staff. The Mission will assist the team by providing key references and contacts. C.8 FINAL REPORT FORMAT The evaluation final report must not exceed 50 pages, excluding cover page, table of contents, and attachments, and must be written in English. It should include an executive summary; introduction; background on the local context and the projects being evaluated; the main evaluation questions; the methodology; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described in the Preparing Evaluation Reports How-To Note.43The report will be formatted according to the evaluation report template.44 43 Available online at: http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports 44 Available online here: http://usaidlearninglab.org/library/evaluation-report-template PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 63 The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology will be explained in the report in detail. Limitations to the evaluation will be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, etc.) and how the limitations were mitigated. The annexes to the report will include: • The evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by USAID/Rwanda, implementers, and/or members of the evaluation team; • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • Sources of information, properly identified and listed; and • Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts. In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. C9. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 45 • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the SOW. • The evaluation report will include the SOW as an annex. • The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation. • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information, with gender and geographic location, need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. END OF SECTION C 45 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 64 ANNEX B: EVALUATION WORK PLAN PURPOSE AND APPROACH Democracy International (DI) hereby presents its plan for conducting the interim performance evaluation of USAID/Rwanda’s Human and Institutional Capacity Development (HICD) Project, implemented by prime contractor Development Alternatives Incorporated (DAI) and subcontractor Kelly Novak Opportunities (KNO). The HICD/Rwanda (HICD/R) project was designed to enable targeted government institutions and civil society organizations to achieve objectives that constitute the building blocks for effective organizations and institutions. The purpose of this performance evaluation is to assess progress that the Human and Institutional Capacity Development in Rwanda project has made to date in achieving its objectives and document the effectiveness of the project’s approaches and responsiveness to overall capacity building needs in Rwanda. The evaluation will provide a detailed examination of the progress towards meeting the overarching objectives of the Project, as described in the Scope of Work. The timeline for the evaluation is presented in Annex A. The evaluation will also help USAID/Rwanda identify technical and management systems strengths, weaknesses, opportunities and threats in the implementation of the HICD/R project. Finally, the evaluation team (“the Team”) will develop conclusions and recommendations that will be used by USAID/Rwanda and the HICD/R project implementer inform project implementation over the remaining life of the project, and be used by USAID/Rwanda to inform decisions on what steps to take in order to sustain project achievements and further support project objectives after the project ends. Evaluation Approach DI’s evaluation design and approach reflect principles outlined in USAID’s 2011 Evaluation Policy. According to that document, “importance and relevance will be achieved by explicitly linking evaluation questions to specific future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders.” By analyzing project effectiveness and challenges to date, DI will provide USAID with objective information to inform evidence-based decision-making on future programming. DI will work closely with USAID and other key stakeholders to ensure that the evaluation addresses the most salient questions of project performance. The evaluation design, including methodology and main limitations, proposed questions for key informant interviews and focus groups, selection methods, and data analysis plan, is outlined below. For all aspects of the evaluation, DI will draw on the recognized technical expertise of subcontractor the Kaizen Company, a leading provider of HICD programming support to USAID. Kaizen has two core roles in the implementation of this evaluation. First, the Subject Matter Expert on Organizational Development position has been staffed through Kaizen. Second, they will provide serve as a resource on issues related to HICD in the drafting and finalizing of the evaluation report. To conduct this evaluation, DI will use an approach it has honed through its extensive experience evaluating capacity-building, governance, civil society, and other relevant programs. DI will use a mixed-method approach that relies on both quantitative data (primarily Mission- and project-level monitoring data) and qualitative data (primarily collected from project documents and through key informant interviews and focus group discussions). This design will allow the evaluation team to generate an in-depth, comprehensive description and understanding of USAID and DAI’s PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 65 intervention as a whole and in its context. This approach actively promotes diverse ways of thinking about issues related to the project, including how and why HICD/R has been successful or faced challenges in identifying and addressing performance gaps within targeted Government of Rwanda (GOR) institutions, civil society organizations (CSOs), and the Rwanda Society for Performance Improvement (RSPI). By triangulating findings using multiple data collection methods and soliciting views from persons most knowledgeable about the project—a critical characteristic of a non￾experimental design—DI will describe project outcomes more accurately and with a greater degree of nuance. Our approach is summarized in the evaluation design matrix below. Evaluation Matrix DI will apply specific data sources, methodology, and data analysis to respond to each of the core Evaluation Questions, to ensure a rigorous and streamlined process. The Evaluation Matrix is an effective vehicle for capturing that process in a concise and logical manner, and will serve as a key guiding document for the evaluation. The Matrix below builds on Table 2 in the evaluation Scope of Work and reflects DI’s learning from the document review phase and discussions among the Team on best approaches to the evaluation. Each Evaluation Question posed by USAID has been further elaborated and defined to indicate clearly how each question will be approached by the Team. Evaluation Question Definition Data Sources and Collection Methods Data Analysis Methods 1. To what extent have the project’s specific interventions improved the management and technical performance of the targeted Rwandan institutions and CSOs? Evaluation will focus on changes that have occurred during the period of participation in the project, in both management and technical areas that were identified and targeted with significant support by the project. The specific focus of this question will be on performance of the institutions and CSOs, as compared to capacity, which is addressed by Question 3. Targeted institutions are: Ministry of Agriculture and Animal Husbandry (MINAGRI), National Capacity Building Secretariat (NCBS) and 10 CSOs. Document Review, including Mission/Project monitoring and evaluation (M&E) Data, progress reports, and monitoring data from targeted institutions to the extent available Using content analysis of narrative documents, as well as descriptive statistics, frequency distributions, and comparative tables for quantitative data, we will be able to assess the extent of change in performance of targeted institutions and CSOs on parameters tracked by the project, including identification of areas where more and less improvement has occurred. Key Informant Interviews with USAID, DAI, other donors, experts, GOR officials, and key staff of targeted institutions (both GOR institutions and CSOs) Using content analysis and thematic coding of responses, we will be able to triangulate the Mission and Project M&E data with key stakeholder perceptions of the project’s results, and elicit information about changes that may not be tracked by project tools. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 66 Focus groups with staff of targeted institutions, and potentially with service recipients/members of selected institutions Using content analysis and thematic coding of responses, we will be able to assess whether changes related to HICD/R activities have had an impact on both management and technical performance, and gain some understanding of any trickle-down effect on selected service recipients or members. 2. What key factors may have hindered or aided the achievement of capacity building activities within CSOs vs. government institutions? The evaluation will consider both internal factors (arising within the targeted institution, the project itself or USAID, including those related to use of the HICD methodology) and external factors (political, security, economic, etc.) The evaluation will seek to differentiate factors that affected capacity building of CSOs as compared to government institutions. Document review, including project progress reports, M&E data, as well as reports on the broader project context Using content analysis of document, we will be able to make a preliminary identification of key challenges and facilitating factors that have affected the project's capacity building work, and look for linkages between those factors and results shown in indicator data. Key Informant Interviews with key staff from targeted GOR institutions and CSOs, USAID and U.S. Embassy staff, RSPI members, DAI, GOR officials and a limited number of CSO leaders from non￾targeted institutions, academics, and journalists Using content analysis and thematic coding of responses, we will be able to: (1) identify factors that facilitated or obstructed achievement of capacity building objectives; (2) study the effect of those factors on specific institutions; (3) explore how those factors could be either minimized or maximized in future. Focus groups with staff of targeted institutions, and potentially with service recipients/members of selected institutions 3. To what extent are Rwandan institutions and CSOs still in need of capacity development support that should be addressed in a future project in order to leverage and sustain USAID/Rwanda Evaluation will make a limited investigation of the extent to which selected government institutions and CSOs have needs for capacity development that are not being met, and whether in general the HICD methodology may Document Review of 1) reports and other analyses produced by the NCBS and USAID in relation to the selected government institutions, and 2) of reports produced by the European Union and USAID in relation to Using content analysis of documents, we will aim to gauge the extent of capacity gaps in the selected institutions and organizations, and identify areas or functions with the greatest need for capacity development support. Depending on the available PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 67 investments? be appropriate for meeting those needs. The selected government institutions are: Ministry of Justice, Supreme Court of Rwanda, Ministry of Local Government, Ministry of Education, Ministry of Health, Office of the Auditor General of State Finances, and Rwanda Bio Medical Center. The type of CSOs to be studied are those defined by the European Union 2013 Civil Society Mapping as “second level” and “third level” organizations. This question will not analyze the capacity needs of institutions already targeted by the project, except to the extent that certain CSOs may be part of the above-noted levels. capacity of civil society, to the extent that such documents are available to the evaluation team. data, it may be possible to analyze for each government institution, but for CSOs the analysis will be generalized. Key Informant Interviews with 3-4 key staff of NCBS, with the Democracy and Governance office of USAID/Rwanda, and with up to 3 other donor agencies. Using content analysis and thematic coding of responses, we will aim to gauge the extent of capacity gaps in the selected institutions and organizations, and identify areas with relatively greater need for capacity development support, comparing this data with that generated by document review. We will also solicit views as to the suitability of future interventions targeting those institutions and organizations using HICD methodology. Depending on the available information, it may be possible to analyze for each selected government institution, but for CSOs the analysis will be generalized. Focus group with selected staff of NCBS, to be identified, and a brief group discussion with up to 10 of the most relevant USAID/Rwanda staff. 4. What capacity development support does the RSPI need to sustainably continue HICD interventions beyond the life of the project, both with USAID/Rwanda financial support in the near term, and without USAID/Rwanda financial support in the Evaluation will examine the needs of RSPI as an organization in the following areas, which have been identified as the most relevant for the purposes of this evaluation: governance, financial sustainability, member services, and marketing/communicatio ns. The team will also assess whether sufficient Document Review of project documents related to RSPI and community of practice, and internal RSPI documents to the extent available Using content analysis of documents, we will be able to ascertain the capacity building support that has already been provided (and is planned for the remainder of the project) to RSPI, the structure and membership of RSPI, current services, funding situation, and progress made to date towards sustainability. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 68 long term? How long will it take to build such capacity? capacity exists among RSPI members to continue to competently provide HICD interventions after the project. In both cases, the evaluation will consider the role of USAID/Rwanda in supporting development of capacity in the short and longer term. We will also consider how long it may take to achieve that level of capacity. Key Informant Interviews with RSPI leaders and members (plus a focus group with selected members), International Society for Performance Improvement (ISPI) representatives, GOR and CSO representatives, donors, experts, USAID/Rwanda and DAI staff Using content analysis and thematic coding of responses, we will be able to identify the main capacity gaps of the RSPI, and the type of USAID support that could help to meet those gaps in the near term. Combined with other information and based on experience of informants, we will be able to estimate how long it may take RSPI and its members to achieve the desired level of sustainability, although that will constrained by the multitude of factors that may affect that estimate. 5. What lessons learned about project management and implementation can inform project management for the remainder of the project life and future potential follow-on designs? "Project management and implementation" are defined to include lessons related to the project's human resources, external relations and coordination, selection of targeted institutions, and other relevant issues. Document review, including project progress reports, M&E data Using content analysis of narrative documents, as well as descriptive statistics, frequency distributions, and comparative tables for quantitative data, we will be able to identify and analyze any lessons learned that have been documented by the project, and also gather ideas for enquiring into other lessons learned by USAID or implementer staff. Key Informant Interviews with USAID and DAI staff, key staff of targeted institutions, other donors, GOR and CSO representatives from selected non￾targeted institutions Using content analysis and thematic coding of responses, we will be able to identify lessons from different perspectives, and elicit recommendations for the improvement of the management and implementation of the project as well as future potential follow-on projects. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 69 METHODOLOGY Overview The evaluation will utilize several evaluation methods to collect data quickly yet systematically, relying primarily on document review, key informant interviews, and focus group discussions. By using multiple complementary evaluation methods and triangulating results (as described in more detail below), the Team will strengthen the validity of its findings and provide reliable answers to the evaluation questions as included in the Scope of Work. Selection of informants will be purposive, and will be informed by suggestions from USAID, participant lists from project activities, desk research, and recommendations by experts. Although project implementers such as DAI and KNO may be asked for suggestions and contact information to facilitate the process of arranging meetings, the final decision on informants will in all cases be made by the DI Team, and whenever feasible the Team will contact informants directly to request appointments. The selection will take into account such factors as type of support received from the project (training, mentoring, funding, information, etc.), location, gender, role in the organization, and type of organization and affiliation of informant (CSO, government, media, academic, etc.). The Team will aim for a reasonable balance of all those factors among interview informants, to ensure that a wide range of voices are heard and that no relevant group is left out. The Team plans to conduct most in￾person data collection in Kigali, which is the location of the majority of direct project beneficiaries and activities, as well as the location of USAID/Rwanda and DAI offices. Locations for data collection outside Kigali, primarily for focus group discussions, will be determined and finalized in close coordination with USAID/Rwanda. Data collection will take place over the course of approximately two weeks, between August 17 and 30, 2016. Detailed notes will be kept of all data collection activities. At least every second day during data collection, the three team members primarily tasked with data collection (Team Leader, Subject Matter Expert and Evaluation Specialist) will meet to share highlights of data collected, discuss implications of that data for the evaluation findings and for ongoing data collection activities, discuss any challenges encountered and ways of mitigating them, and decide on any adjustments needed to tools, approaches or the informant list. To contribute to the collection of honest viewpoints and reliable information, the Team will assure informants of their anonymity. Although an informant list will be maintained by the Team, interviewees will have the option of having their names omitted from the final list annexed to the report. Sound data governance – including clear data ownership, access control, and backup and archiving processes – will ensure data quality, accuracy, and security. Phase I: Preparation and Initial Desk Review Before arriving in country, the Team Leader, Subject Matter Expert and Evaluation Expert are conducting a comprehensive document review, primarily based on information provided by USAID/Rwanda and DAI (see Annex D for documents reviewed to date) but supplemented by the Team’s own research and enquiries. The document review will allow the Team to gather a foundation of relevant data about the project’s goals, plans, activities, results and M&E efforts. The information thus gathered will enhance the Team’s understanding of the project, including relevant context and institutions (both national and international), along with the range of programming approaches, opportunities, and constraints that have emerged during the project’s implementation. The review has helped the Team to refine its evaluation design and work plan based on a more thorough understanding of the project’s activities. Over the course of the evaluation, DI will analyze a variety of qualitative data, including the HICD/Rwanda statement of work, annual work plans, statements of work from subcontracts, PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 70 progress reports, the USAID/Rwanda Country Development Cooperation Strategy (CDCS), and all assessments and performance solution packages. DI will also analyze quantitative data, such as those included in the project’s M&E plan, indicators, and results, in NCBS reports, as well as in the Mission’s Performance Management Plan (PMP) and annual Performance Plan and Report (PPR). As additional documents come to the attention of the Team during data collection, they will be reviewed and added to a master list of documents reviewed. Phase II: Fieldwork and Data Collection In-Brief and Consultations with the Mission Upon arrival in Kigali, the Team will participate in an in-brief meeting with the Mission to develop a more nuanced and detailed understanding of USAID’s goals for the evaluation as well as to clarify the evaluation activities as needed. During the in-brief, the Team will discuss the evaluation agenda with USAID, including any ideas for additional or alternate informants, and the regions and sites to be visited by the Team. If changes are agreed at this meeting, the Team will revise the work plan accordingly. The final version will set forth, in accordance with decisions made at the in-brief, the agreed evaluation methods and activities, initial list of targeted interviewees, evaluation time line, and selected interview and focus group guides. DI will use three principal tools for primary data collection. For individual Key Informant Interviews, we will use a Key Informant Interview Protocol to guide a discussion that investigates the evaluation questions outlined in the statement of work, which will be tailored for each specific type of informant (e.g. GOR official, CSO member). Illustrative questions for several categories of key informant are contained in Annex C to this work plan. DI’s team will employ a semi-structured approach, allowing some degree of deviation from the questionnaire. This will allow the interlocutor to have a partial role in determining the path of the conversation, and enable a free-flowing conversation in which the interviewee will be encouraged to share their views openly. The interview protocols will be tested by the Team on the first and second days of data collection, during which at least two team members will participate in each interview in order to allow careful observation of informant reactions and ways that the questions could be improved (in terms of subject matter, phrasing and ordering). Following each pilot interview, the participating team members will discuss the experience and potential revisions to the protocols; those revisions will be agreed by the entire team after the second day of data collection. For Focus Group Discussions, DI will prepare a Focus Group Discussion Protocol, again appropriately tailored for each particular group. A draft protocol and question guide for a proposed focus group discussion is contained in Annex C. As with the key informant interviews, our focus groups will follow a protocol containing open-ended questions designed to prompt candid responses as part of a flexible conversation. Finally, in order to synthesize evaluation findings into a coherent whole, DI will employ a Data Synthesis Table, based on the format in Annex E, which helps to ensure that all members of the evaluation team record key interview data in a way that is comparable across interlocutor, and is easy to review and code, as appropriate. During the first days of data collection, each evaluation question will be broken down into appropriate thematic areas that capture the range of relevant data being generated through interviews and focus groups. Key Informant Interviews The evaluation team will conduct approximately 55 in-depth interviews with a wide range of key informants (more if time permits and there is a need to expand the scope of informants).46 Each 46 The sample size is a function of the time frame available for data collection and the size of the evaluation team. Although more time would allow for a larger sample, based on analysis of project documents so far, we PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 71 individual interview will last between 45 minutes and 1 hour. These interviews will enable the Team to gather detailed inputs on the topics covered by the evaluation questions, based on the informant’s unique perspective, and also elicit specific recommendations for ongoing and future programming. Key informant interviews will be conducted, whenever possible, by two team members – one acting as interviewer, and the other as note-taker. This will enhance data collection by allowing for the interviewer to focus on interacting with the subject and covering the essential subject matter, while the note-taker concentrates on capturing as much as possible of the conversation. The note-taker will also assist as a timekeeper for interviews, and serve as a sounding board after interviews so that the interviewer can verify their understanding of certain responses, and the two team members can share their impressions of the interview and the interviewee. Given time constraints on data collection, some interviews will be conducted by a single team member. The Team will take care to select such interviews in order to minimize the effect on reliability of data; for example, the Team Leader may conduct certain interviews with international actors alone, given her long experience in interacting with such persons, and the Subject Matter Expert may conduct some interviews with government officials alone, especially when specifically related to organizational development of a government institution. In consultation with USAID, DI will select interviewees who can provide substantive feedback in relation to the project and its context, and who reflect diverse views. A draft list of potential interviewees can be found in Annex B. The Team will interview informants with broad knowledge and understanding as well as individuals who have detailed information on a specific topic related to one or more evaluation questions. DI will focus on interviews with key stakeholders, including: USAID and U.S. Embassy officials; current and former HICD/R staff; project beneficiaries (who directly participated in project interventions) from targeted government and civil society bodies; government and civil society leaders who did not directly participate in the project (but who are aware of the project - in order to elicit views of a limited number of project “outsiders” to triangulate with other sources; other donor organizations and international institutions; experts in civil society and governance issues in Rwanda and the region; journalists; government officials from relevant agencies; and other stakeholders that may be identified as the evaluation moves forward. In the event that the Team is not able to interview a particular constituency or key actor, it will identify alternate sources as proxies who are able to provide similar information. These interviews will be conducted in person whenever possible, although in a few cases the Team may be limited to telephone or e-mail communications. It is anticipated that most key informants will be sufficiently fluent in English to be interviewed in that language, however, on request the team will ensure that either Kinyarwanda or French is used by the interviewer. If conducted in a language other than English, the interview notes will be translated into English and transcribed on the same day to ensure freshness of memory. Focus Group Discussions The evaluation team plans to convene discussion with at least five focus group discussions (FGDs). Each discussion will last approximately 1.5 hours. These facilitated discussions will allow for exchange and discussion of experiences and viewpoints, and enable the Team to probe for lessons learned and recommendations for future activities and support through the HICD/R project. The focus group discussions will in all cases be conducted by at least two team members – one acting as facilitator, and the other as note-taker. As above, this will allow the interviewer to focus on believe that this number of interviews and five FGDs will allow for collecting a reasonable range of perspectives from key project stakeholders, sufficient to enable a response to the evaluation questions. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 72 interacting with the group and covering the essential subject matter, while the note-taker captures as much as possible of the conversation. Each focus group will be organized around a homogenous identity, with each group being composed of people with similar key characteristics—for example, a single group will include only members of RSPI, while another would target only staff of MINAGRI. Respondent selection will be done purposively, mindful of the need to canvass diverse viewpoints, the differences between local- and national-level institutions, and the need to incorporate the perspectives of both men and women. When planning focus groups, DI will begin with consideration of groups/persons suggested by USAID and DAI as the planning process moves forward, but will also explore potential participants with stakeholders such as specific CSOs or GOR institutions once in country. This will be particularly important for identification of indirect beneficiaries or service recipients of targeted institutions, to ensure that the participants are genuinely able to assist in answering the evaluation questions. At present, the Team has identified the following informants for focus group discussions in Kigali: mid-level staff of targeted CSOs (to complement interviews with leaders and senior managers); and members of RSPI (to complement interviews with senior leadership/founders). Outside of Kigali, if time permits, we are aiming to conduct focus groups with locally based staff of targeted CSOs, and possibly with indirect beneficiaries of those institutions. When discussing detailed technical topics, focus groups are ideally made up of around eight participants to allow a more nuanced discussion. Therefore, the Team proposes to aim for groups of approximately six to ten participants each. Discussion guides will be adjusted to focus on the evaluation questions of most relevance to that particular group. DI will ensure the guides reflect the context of the HICD/R project, and fully account for gender considerations relevant to the project. As appropriate, considering the composition of the group members, the Team will hold discussions with men and women separately, in order to ensure that participants feel free to express their views. DI will assign evaluation team members to focus group discussions so that at least one native Kinyarwanda speaker is present at all discussions. Where appropriate based on the composition of participants, that person will lead the facilitation of the discussion in Kinyarwanda or another language that will facilitate full participation. Immediately after the FGDs, the notes will be transcribed and translated into English. A native speaker will also be actively engaged in focus groups conducted primarily in English, to ensure any language barriers are overcome and all cultural nuances are taken into consideration. If participants agree, the Team will consider recording the discussions, in order to ensure that all viewpoints are accurately captured and taken into account. Phase III: Data Analysis and Development of Recommendations This mixed-methods data-collection strategy will utilize an analytical approach that examines data from each tool individually and then combines information gleaned from that varied process into a coherent set of findings and recommendations. DI accordingly proposes a data-analysis plan that simultaneously allows for discrete analysis of data from individual methods and aggregate analysis of all data collected during the evaluation process. Data collected during the document review process, in addition to guiding the Team in work planning, will help to provide context for the evaluation and inform its answers to all evaluation questions, as described in the evaluation matrix. The team will rely primarily on content analysis to synthesize information from the document review. Performance monitoring data (at project level and targeted institution level) will be analyzed using descriptive statistics, frequency distributions, and comparative tables, where appropriate. Data collected through key informant interviews and focus groups will be analyzed using content analysis to analyze the notes and transcripts of interviews and focus groups. Team personnel will PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 73 rely on their extensive technical expertise and experience evaluating organizational development programming to draw qualitative conclusions based on this information. The team will also create a coding system prior to data collection that allows for categorization and comparison of qualitative responses. This coding will allow the team to quantify data gathered through interviews, which will in turn allow more structured and objective analysis of interview data than would otherwise be possible. The findings from our analysis of the various data sources will be organized and presented in the evaluation report, and provide the basis for our conclusions and recommendations as envisioned in the USAID Evaluation Policy. LIMITATIONS AND MINIMIZING BIAS Like all performance evaluations, the HICD/Rwanda evaluation is subject to several limitations. The team will take active steps to minimize bias and generate systematic, verifiable, credible information. The following are some specific limitations anticipated by the Team as this time: 1. The timeframe and size of team for this evaluation requires a relatively narrow approach that focuses on a limited number of informants, and the short period in country will constrain the potential use of “snowball sampling” to identify informants during the data collection process based on suggestions from those interviewed. The Team will work with USAID to carefully select the interviewee list to ensure that it includes individuals who are best positioned to provide substantive feedback in a balanced, unbiased manner. The same constraints will preclude the collection of new quantitative data. Nevertheless, the Team can compare data collected using one method to data collected using other methods, and compare data from different sources. Existing quantitative data generated by the project and targeted institutions will serve as a valuable supplementary data source. 2. A design that uses purposive sampling of informants is inherently subject to selection bias, because it cannot use randomization or representative sampling to draw conclusions representative of an entire population. DI’s team will utilize social science techniques (as described above under Data Analysis) to triangulate the data collected, draw lessons based on input from those most knowledgeable about the project’s interventions, and correct, as much as possible, for sources of selection bias. 3. Evaluations are subject to cognitive biases, including response and recall bias. DI will mitigate response bias—a type of cognitive bias where respondents answer questions in the way they think the interviewer wants them to rather than according to their beliefs—by designing the data-collection instruments to elicit open, honest, and unbiased participation and avoid leading questions. The team will confirm findings by probing similar topics with various stakeholders to see whether findings converge and triangulating results with information from document review and site visits. To overcome recall bias—respondents’ inability to accurately remember past details—we will compare themes and patterns that emerge in key informant interviews, focus group responses, and DAI’s HICD/Rwanda monitoring data to triangulate findings. DELIVERABLES As part of the evaluation, DI will provide the following deliverables to USAID: Work Plan: DI is submitting the present draft work plan for the USAID/Rwanda Mission to review and provide comments during the preparation phase and before the Team Leader and Subject Matter Expert on Organizational Development travel to Rwanda. DI will submit a revised work plan based on comments and feedback from USAID/Rwanda. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 74 In-briefing: Upon arrival in country, the Team will conduct an in-brief with the Mission to finalize the evaluation design and work plan. Data Collection and Interim Meetings: The Team will periodically brief USAID stakeholders on the progress of data collection, challenges, and any issues that may affect the timeline. Pre-Departure Preliminary Findings Workshop: Before departing the country after data collection, the Team will conduct an outbrief meeting with the Mission and present a detailed outline of main findings, as well as preliminary conclusions and recommendations. Draft Performance Evaluation Report: Following the Team’s departure, DI will submit the draft evaluation report to USAID/Rwanda. As per the evaluation timeline, DI will receive feedback from the Mission and revise the report accordingly. Final Performance Evaluation Report: DI will submit the final evaluation report after receiving comments, along with all relevant evaluation records. The final report will meet all criteria for quality evaluation reports as laid out in the USAID Evaluation Policy and ADS 203. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 75 ANNEX C: DATA COLLECTION TOOLS A. USAID STAFF INTERVIEW QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What is your role related to the project, and how long have you done that? n/a 2. What do you consider to be the most important achievements of the project since 2012 (or since you became involved)? (Probe for specifics and reasons.) 1 3. What have been the major challenges or obstacles to achieving the objectives of the project? (Probe for internal factors, such as implementation difficulties, staffing, methodology, selection of targeted institutions, as well as external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) • Are there specific areas in which the project has not been able to make as much progress as expected? Why? • Is the project on track to meet its indicator targets? Do the PMP indicators adequately measure the results? If not, what else should be measured? 2,5 Support to government 4. Do you believe that the project has helped to build the capacity of the Ministry of Agriculture in public financial management and policy formulation? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components of MINAGRI that have improved or not, specific capacities that have been improved or not.) 1,2 5. If YES to question 4 - Has the improved capacity of MINAGRI helped it to perform better (be more effective or efficient in its work)? Probe in relation to areas in which project focused its support: public financial management and policy formulation. • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 6. What methods used by the project were most effective? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (i.e. had the least effect on the institution’s capacity or performance) (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 76 Interview questions Most relevant EQ # 7. What factors have helped the project in building capacity in MINAGRI? 2 8. What challenges has the project faced in building capacity in MINAGRI? 2 9. Do you think that the Project has contributed to lasting improvement of performance by MINAGRI? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 10. The project has more recently started to work with the NCBS. What progress has been made so far in building capacity in that institution? 1 11. What factors have helped or hindered progress with the NCBS? 2 12. Are the same approaches and methods being used by the project with NCBS as with MINAGRI? • Which methods have been most effective with NCBS, and least effective? (i.e. had the most positive /least effect on the institution’s capacity or performance) 1,5 13. Have improvements in capacity so far helped NCBS to be more effective in its work? (Probe about changes in communications, knowledge management, M&E, quality assurance) • Why or why not? 1,2 QUESTIONS RELATED TO EQ3 AND PRIORITY INSTITUTIONS A. What do you see as the main capacity needs of (select according to informant): • Min of Justice • Supreme Court of Rwanda • Ministry of Local Government, • Office of the Auditor General of State Finances • Ministry of Education, • Ministry of Health, • Rwanda Bio Medical Center. B. Would HICD be appropriate for addressing those needs? Why or why not? Support to Rwanda Society for Performance Improvement 14. What interaction (if any) have you had or work have you done with the Rwanda Society for Performance Improvement (RSPI)? n/a 15. What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications.) • How has support from HICD/R affected the RSPI in those areas? 4 16. What have been the challenges so far in establishing and supporting the RSPI? 4,5 17. What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? • What specific kind of support would be most useful? 4,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 77 Interview questions Most relevant EQ # Support to Civil Society 18. Do you believe that the project has helped to build the capacity of the targeted CSOs? • If yes, how? If not for some CSOs, why not? (Probe for examples and reasons, specific CSOs or parts of CSOs that have improved or not, specific capacities that have been improved or not.) 1,2 19. If YES to question 18 - Has the improved capacity of those organizations helped them to perform better (be more effective or efficient in their work)? • If yes, in what specific ways or areas of work? • If not, why not? • How much variation is there among the targeted CSOs? What are the reasons for that variation? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 20. What methods used by the project were most effective in that situation? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 21. What factors have helped the project in building capacity of that organization? 2 22. What challenges has the project faced in building capacity of that organization? 2 23. Do you think that the Project has contributed to lasting improvement of performance of that organization? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 QUESTIONS RELATED TO EQ3 AND CIVIL SOCIETY NEEDS A. What do you see as the main capacity needs of civil society organizations (2nd and 3rd level)? B. Would HICD be appropriate for addressing those needs? Why or why not? General Questions – challenges, coordination, future, gender issues (primarily for management staff) 24. Did the Project consistently coordinate with other stakeholders, including other US agencies, Rwandan and international bodies that are involved in the sector? • Did the project complement the work of others, or was there overlap or confusion? 5 25. How would you describe the project’s relationship with the Government of Rwanda? • Have government officials (elected and other) been supportive of the project? 2,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 78 Interview questions Most relevant EQ # (Probe for examples of support and cooperation, and any obstacles.) 26. How would you describe the project’s relationship with Rwandan civil society? • Have civil society leaders and networks been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 27. Has the Project targeted the appropriate organizations/entities in its work? • What were the main differences between working (and using HICD) with government and with civil society?? 2,5 28. The Project has provided significant amounts of training and technical assistance. How was learning of participants measured, for training and for TA? • Did the project have sufficient information about the effect of training and TA? • How could the project improve training and TA in the future? 5 29. What should be the priorities for future capacity development support by USAID? • Would another freestanding/separate initiative based on HICD approaches be appropriate for either government or civil society? • Why or why not? What needs to change for it be effective? (probe about USAID systemic constraints, funding cycles, lack of awareness in USAID or beneficiaries, need to include “hard assets” in the package of support, etc.) • Is HICD suitable for advancing the USAID Forward and Local Solutions agenda? For paving the way for direct funding of the government? (N.B. recent European Union commitment to budget support of MINAGRI) 3 30. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 31. How has the project taken gender issues into account in its approaches and activities? • What specific project strategies or activities were used to identify and respond to the different needs of men and women? • How effective were those measures? 2,5 32. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 33. Would you like to make any final recommendations for the project or for USAID to consider? 3,4,5 B. IMPLEMENTER STAFF INTERVIEW QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What is your role in the project, and how long have you held that position? n/a 2. What do you consider to be the most important achievements of the project since 2012 (or since you joined the project)? (Probe for specifics and reasons.) 1 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 79 Interview questions Most relevant EQ # 3. What have been the major challenges or obstacles to achieving the objectives of the project? (Probe for internal factors, such as implementation difficulties, staffing, methodology, selection of targeted institutions, as well as external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) • Are there specific areas in which the project has not been able to make as much progress as expected? Why? • Is the project on track to meet its indicator targets? Do the PMP indicators adequately measure the results? If not, what else should be measured? 2,5 Support to government (focus here for staff working on government and select questions to focus on the particular institution with which they are most familiar) 4. Do you believe that the project has helped to build the capacity of the Ministry of Agriculture in public financial management and policy formulation? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components of MINAGRI that have improved or not, specific capacities that have been improved or not.) 1,2 5. If YES to question 4 - Has the improved capacity of MINAGRI helped it to perform better (be more effective or efficient in its work)? Probe in relation to areas in which project focused its support: public financial management and policy formulation. • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 6. What methods used by the project were most effective? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (i.e. had the least effect on the institution’s capacity or performance) (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 7. What factors have helped the project in building capacity in MINAGRI? 2 8. What challenges has the project faced in building capacity in MINAGRI? 2 9. Do you think that the Project has contributed to lasting improvement of performance by MINAGRI? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 10. The project has more recently started to work with the NCBS. What progress has been made so far in building capacity in that institution? 1 11. What factors have helped or hindered progress with the NCBS? 2 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 80 Interview questions Most relevant EQ # 12. Are the same approaches and methods being used by the project with NCBS as with MINAGRI? • Which methods have been most effective with NCBS, and least effective? (i.e. had the most positive /least effect on the institution’s capacity or performance) 1,5 13. Have improvements in capacity so far helped NCBS to be more effective in its work? (Probe about changes in communications, knowledge management, M&E, quality assurance) • Why or why not? 1,2 Support to Rwanda Society for Performance Improvement (for relevant staff) 14. What interaction (if any) have you had or work have you done with the Rwanda Society for Performance Improvement (RSPI)? n/a 15. What do you think about the transition of COMPRAC to the RSPI? (probe re decision on what form the organization would take going forward, why RSPI was the chosen path, etc.) 4,5 16. What do you see as the priority needs of the RSPI as an organization, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re governance, human resources, financial sustainability, member services, and external relations/ communications.) 4 17. How do you see the future of RSPI, after the project ends? 4 18. What role should USAID play in further supporting the RSPI as an organization, especially in terms of boosting its sustainability? • What specific kind of support would be most useful? In the next year? And after the end of project? 4,5 19. What are the priority needs of the members of the RSPI, in terms of capacity development? How many of the members have the competency to deliver quality HICD-related services to potential clients in Rwanda? 4 20. What should be the role of USAID in developing capacity of the RSPI members to deliver HICD services in future? 4 21. What specific kind of support would be most useful? In the next year? And after the end of project? 4 Support to Civil Society (focus here for staff working on civil society, and customize questions to focus on the particular CSOs with which they have been working) 22. Do you believe that the project has helped to build the capacity of ___ organization? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/staff that have improved or not, specific capacities that have been improved or not.) 1,2 23. If YES to question 18 - Has the improved capacity of that organization helped it to perform better (be more effective or efficient in its work)? • If yes, in what specific ways or areas of work? • If not, why not? 1,2 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 81 Interview questions Most relevant EQ # (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 24. What methods used by the project were most effective in that situation? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 25. What factors have helped the project in building capacity of that organization? 2 26. What challenges has the project faced in building capacity of that organization? 2 27. Do you think that the Project has contributed to lasting improvement of performance of that organization? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 General Questions – challenges, coordination, future, gender issues (primarily for management staff) 28. Did the Project consistently coordinate with other stakeholders, including other US agencies, Rwandan and international bodies that are involved in the sector? • Did the project complement the work of others, or was there overlap or confusion? 5 29. How would you describe the project’s relationship with the Government of Rwanda? • Have government officials (elected and other) been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 30. How would you describe the project’s relationship with Rwandan civil society? • Have civil society leaders and networks been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 31. Has the Project targeted the appropriate organizations/entities in its work? • What were the main differences between working (and using HICD) with government and with civil society?? 2,5 32. The Project has provided significant amounts of training and technical assistance. How was learning of participants measured, for training and for TA? • Did the project have sufficient information about the effect of training and TA? • How could the project improve training and TA in the future? 5 33. What should be the priorities for future capacity development support by USAID? • Would another initiative based on HICD approaches be appropriate for either government or civil society? • Why or why not? 3 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 82 Interview questions Most relevant EQ # 34. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 35. How has the project taken gender issues into account in its approaches and activities? • What specific project strategies or activities were used to identify and respond to the different needs of men and women? • How effective were those measures? 2,5 36. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 37. Would you like to make any final recommendations for the project or for USAID to consider? 3,4,5 C. TARGETED GOVERNMENT INTERVIEW QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What is your role in this institution, and how long have you held this position? n/a 2. What kind of interaction have you (or your institution) had with the HICD project? (Probe for specific activities they have observed, participated in, been informed about officially or otherwise.) n/a 3. How would you describe the fundamental approach or strategy of the project? (Probe for how much the informant knows about HICD and its approach to capacity development, to put rest of interview in context.) n/a Support to government (adapt as needed from MINAGRI to NCBS) (may first need to explain specific support provided by project, if informant not familiar) 4. Do you believe that the project has helped to build the capacity of the Ministry of Agriculture in public financial management? in policy formulation? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components of MINAGRI that have improved or not, specific capacities that have been improved or not.) 1,2 5. If YES to question 4 - Has the improved capacity of MINAGRI helped it to perform better (be more effective or efficient in its work)? Probe in relation to areas in which project focused its support: public financial management and policy formulation. • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 83 Interview questions Most relevant EQ # 6. What methods used by the project were most effective? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 7. What factors have helped the project in building capacity in MINAGRI? 2 8. What challenges has the project faced in building capacity in MINAGRI? 2 9. Do you think that the Project has contributed to lasting improvement of performance by MINAGRI? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 Support to Rwanda Society for Performance Improvement 10. What interaction (if any) have you had with the Rwanda Society for Performance Improvement (RSPI)? Are you a member or do you know its members? (may need to explain what it is, originally community of practice of performance improvement trainers and consultants) n/a 11. What is your general impression of the RSPI? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications.) • How has support from HICD/R affected the RSPI in those areas? 4 12. IF THE INFORMANT IS A MEMBER: What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re same areas.) 4 13. IF THE INFORMANT IS A MEMBER: What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? • What specific kind of support would be most useful? 4,5 Support to Civil Society 14. What kind of support has the project provided to civil society, besides the RSPI? (Probe to find out which targeted CSOs they are most aware of. If none, then move on to next section. If some, then focus on those for next questions.) n/a 15. Do you believe that the project has helped to build capacity of those CSOs? • If yes, how? If not, why not? (Probe for examples and reasons, specific CSOs that have improved or not, specific capacities that have been improved or not.) 1,2 16. What challenges has the project faced in building capacity in civil society? 2 General Questions – challenges, future, gender issues 17. What have been the major challenges or obstacles for the project generally? (Probe for internal factors, such as implementation difficulties, staffing, methodology, 2,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 84 Interview questions Most relevant EQ # selection of targeted institutions, as well as external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) 18. In general, do you think that the project’s approach and activities (based on HICD) were relevant to the situation in Rwanda since 2012? • Why or why not? 3 19. Beyond your own institution, what do you think should be the priorities for future capacity development support in Rwanda by USAID? • Would another initiative based on HICD approaches be appropriate, for the Rwandan government? For Rwandan civil society? • Why or why not? 3 20. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 21. How has the project taken gender issues into account in its approaches and activities? • Are you aware of any specific project strategies or activities that recognized the different needs of men and women? • If yes, do you think those measures were appropriate? 2,5 22. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 23. Are there any final recommendations that you would like the project or USAID to consider? 3,4,5 D. TARGETED CSO INTERVIEW QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What is your role in this organization/institution, and how long have you held this position? n/a 2. What kind of interaction have you (or your organization/institution) had with the HICD project funded by USAID and implemented by DAI since 2012? (Probe for specific activities they have observed, participated in, been informed about officially or otherwise.) n/a 3. How would you describe the fundamental approach or strategy of the project? (Probe for how much the informant knows about HICD and its approach to capacity development, to put rest of interview in context.) n/a Support to CSOs 4. Do you believe that the project has helped to build the capacity of your organization? • If yes, how? • If not, why not? (Probe for examples and reasons, specific units/components that have improved 1,2 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 85 Interview questions Most relevant EQ # or not, specific capacities that have been improved or not.) 5. If YES to question 4 - Has the improved capacity of your organization helped it to perform better (be more effective or efficient in its work)? • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 6. What methods used by the project were most effective? (i.e. had the most positive effect on the organization’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 7. What factors have helped the project in building capacity in your organization? 2 8. What challenges has the project faced in building capacity in your organization? 2 9. Do you think that the Project has contributed to lasting improvement of performance by your organization? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 Support to Rwanda Society for Performance Improvement 10. What interaction (if any) have you had with the Rwanda Society for Performance Improvement (RSPI)? (may need to explain what it is, originally community of practice of performance improvement trainers and consultants) n/a 11. What is your general impression of the RSPI? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications with government and CSOs, etc.) • How has support from HICD/R affected the RSPI in those areas? 4 12. What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re same areas.) 4 13. What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? • What specific kind of support would be most useful? 4,5 Support to Government 14. What kind of support has the project provided to government? (Probe to find out which targeted institutions they are aware of. If none, then move on to next section. If some, then focus on those for next questions.) n/a PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 86 Interview questions Most relevant EQ # 15. Do you believe that the project has helped to build capacity of that institution? • If yes, how? If not, why not? (Probe for examples and reasons, specific capacities or functions that have been improved or not.) 1,2 16. What challenges has the project faced in building capacity in government generally? • What challenges has it faced with that institution specifically? 2 General Questions – challenges, future, gender issues 17. What have been the major challenges or obstacles for the project generally? (Probe for internal factors, such as implementation difficulties, staffing, methodology, selection of targeted institutions, as well as external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) 2,5 18. How would you describe the project’s relationship with the Government of Rwanda? • Have government officials (elected and other) been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 19. How would you describe the project’s relationship with Rwandan civil society? • Have civil society leaders and networks been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 20. In general, do you think that the project’s approach and activities (based on HICD) were relevant to the situation in Rwanda since 2012? • Why or why not? 3 21. What do you think should be the priorities for future capacity development support by USAID? • Would another initiative based on HICD approaches be appropriate for either government or civil society? • Why or why not? 3 22. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 23. How has the project taken gender issues into account in its approaches and activities? • Are you aware of any specific project strategies or activities that recognized the different needs of men and women? • If yes, do you think those measures were appropriate? 2,5 24. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 25. Are there any final recommendations that you would like the project or USAID to consider? 3,4,5 E. NON-TARGETED GOVERNMENT INTERVIEW QUESTIONS Interview questions Most relevant EQ # PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 87 Interview questions Most relevant EQ # General introductory questions 1. What is your role in this institution, and how long have you held this position? n/a 2. What kind of interaction have you (or your institution) had with the HICD/R project? (Probe for specific activities they have observed, participated in, been informed about officially or otherwise.) n/a 3. How would you describe the fundamental approach or strategy of the project? (Probe for how much the informant knows about HICD and its approach to capacity development, to put rest of interview in context.) n/a Support to government (depending on what project beneficiaries they are most aware of, adapt as needed for MINAGRI or NCBS or activities through Rwanda Management Institute) (may first need to explain specific support provided by project, if informant not familiar) 4. Do you believe that the project has helped to build the capacity of the ___________? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components that have improved or not, specific capacities that have been improved or not.) 1,2 5. If YES to question 4 - Has the improved capacity of ________ helped it to perform better (be more effective or efficient in its work)? • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 6. What methods used by the project were most effective? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 7. What factors have helped the project in working with _________? 2 8. What challenges has the project faced in working with _________? 2 9. Do you think that the Project has contributed to lasting improvement of performance by MINAGRI? • What are the challenges to sustainability of changes? • What can be done (by the project or others) to shore up sustainability? 1,3 Support to Rwanda Society for Performance Improvement 10. What interaction (if any) have you had with the Rwanda Society for Performance Improvement (RSPI)? Are you a member or do you know its members? (may need to explain what it is, originally community of practice of performance improvement trainers and consultants) n/a PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 88 Interview questions Most relevant EQ # 11. What is your general impression of the RSPI? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications.) • How has support from HICD/R affected the RSPI in those areas? 4 12. IF THE INFORMANT IS A MEMBER: What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re same areas.) 4 13. IF THE INFORMANT IS A MEMBER: What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? • What specific kind of support would be most useful? 4,5 Support to Civil Society 14. What kind of support has the project provided to civil society, besides the RSPI? (Probe to find out which targeted CSOs they are most aware of. If none, then move on to next section. If some, then focus on those for next questions.) n/a 15. Do you believe that the project has helped to build capacity of those CSOs? • If yes, how? If not, why not? (Probe for examples and reasons, specific CSOs that have improved or not, specific capacities that have been improved or not.) 1,2 16. What challenges has the project faced in building capacity in civil society? 2 General Questions – challenges, future, gender issues 17. What have been the major challenges or obstacles for the project generally? (Focus on external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) 2,5 18. In general, do you think that the project’s approach and activities (based on HICD) were relevant to the situation in Rwanda since 2012? • Why or why not? 3 19. What do you think should be the priorities for future capacity development support in Rwanda by USAID? • Would another initiative based on HICD approaches be appropriate, for the Rwandan government? For Rwandan civil society? • Why or why not? 3 20. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 21. How has the project taken gender issues into account in its approaches and activities? • Are you aware of any specific project strategies or activities that recognized the different needs of men and women? • If yes, do you think those measures were appropriate? 2,5 22. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 89 Interview questions Most relevant EQ # 23. Are there any final recommendations that you would like the project or USAID to consider? 3,4,5 F. SHORT TERM CONSULTANT INTERVIEW QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What role have you played in the project, with which targeted institutions? n/a 2. What have been the major challenges or obstacles to achieving the objectives of the project, from your point of view? (Probe for internal factors, such as implementation difficulties, staffing, methodology, selection of targeted institutions, as well as external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) 2,5 Support to government (focus here for consultants working on government and select questions to focus on the particular institution with which they are most familiar) 3. Do you believe that the project has helped to build the capacity of ________? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components of that institution that have improved or not, specific capacities that have been improved or not.) 1,2 4. If YES to question 3 - Has the improved capacity of that institution helped it to perform better (be more effective or efficient in its work)? Probe in relation to areas in which project focused its support. • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 5. What methods used by the project were most effective? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (i.e. had the least effect on the institution’s capacity or performance) (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 6. What factors have helped the project in building capacity in that institution? 2 7. What challenges has the project faced in building capacity in that institution? 2 8. Do you think that the Project has contributed to lasting improvement of performance by that institution? 1,3 Support to Rwanda Society for Performance Improvement (for relevant consultants) 9. What interaction (if any) have you had with the Rwanda Society for Performance Improvement (RSPI)? Are you a member of RSPI (or of ISPI)? n/a 10. What do you think about the transition of COMPRAC to the RSPI? (probe re 4 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 90 Interview questions Most relevant EQ # decision on what form the organization would take going forward, why RSPI was the chosen path, etc.) 11. What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications.) • How has support from HICD/R affected the RSPI in those areas? 4 12. How do you see the future of RSPI, after the project ends? 4 13. What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? What specific kind of support would be most useful? 4,5 14. What are the priority needs of the members of the RSPI, in terms of capacity development? How many of the members have the competency to deliver quality HICD-related services to potential clients in Rwanda? 4 15. What should be the role of USAID in developing capacity of the RSPI members to deliver HICD services in future? 4 16. What specific kind of support would be most useful? In the next year? And after the end of project? 4 Support to Civil Society (focus here for consultants working on civil society, and customize questions to focus on the particular CSOs with which they have been working) 17. Do you believe that the project has helped to build the capacity of ___ organization? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/staff that have improved or not, specific capacities that have been improved or not.) 1,2 18. If YES to question 13 - Has the improved capacity of that organization helped it to perform better (be more effective or efficient in its work)? • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 19. What methods used by the project were most effective in that situation? (i.e. had the most positive effect on the institution’s capacity or performance) • Which seemed to be less effective? (Probe re in-depth initial assessments, training workshops, mentoring/coaching, development of guidelines, monitoring of progress, etc.) 1,5 20. What factors have helped the project in building capacity of that organization? 2 21. What challenges has the project faced in building capacity of that organization? 2 22. Do you think that the Project has contributed to lasting improvement of performance of that organization? 1,3 General Questions – challenges, coordination, future, gender issues (primarily for management staff) 5 23. How would you describe the project’s relationship with the Government of Rwanda? • Have government officials (elected and other) been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 2,5 24. How would you describe the project’s relationship with Rwandan civil society? 2,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 91 Interview questions Most relevant EQ # • Have civil society leaders and networks been supportive of the project? (Probe for examples of support and cooperation, and any obstacles.) 25. Has the Project targeted the appropriate organizations/entities in its work? • What were the main differences between working (and using HICD) with government and with civil society?? 2,5 26. The Project has provided significant amounts of training and technical assistance. How was learning of participants measured, for training and for TA? • Did the project have sufficient information about the effect of training and TA? • How could the project improve training and TA in the future? 5 27. In general, what should be the priorities for future capacity development support by USAID? • Would another initiative based on HICD approaches be appropriate for either government or civil society? • Why or why not? 3 28. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? 2 29. How has the project taken gender issues into account in its approaches and activities? • What specific project strategies or activities were used to identify and respond to the different needs of men and women? • How effective were those measures? 2,5 30. Are there ways that project management and implementation could be improved for the remainder of the project, and for future USAID projects? 5 31. Would you like to make any final recommendations for the project or for USAID to consider? 3,4,5 G. DONOR AND EXPERT QUESTIONS Interview questions Most relevant EQ # General introductory questions 1. What is your role in this institution, and how long have you held this position? n/a 2. What kind of interaction have you (or your institution) had with the HICD/R project? (Probe for specific activities they have observed, participated in, been informed about officially or otherwise.) n/a 3. How would you describe the fundamental approach or strategy of the project? (Probe for how much the informant knows about HICD and its approach to capacity development, to put rest of interview in context.) • How suitable do you think that approach (HICD) has been for the Rwanda context? n/a Support to government (may first need to explain specific support provided by project, if informant not familiar) (depending on what project beneficiaries they are aware of, adapt 1,2 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 92 Interview questions Most relevant EQ # questions as needed for MINAGRI or NCBS or activities through Rwanda Management Institute). If not aware of any of those three, then skip to next section. 4. Do you believe that the project has helped to build the capacity of the ___________? • If yes, how? If not, why not? (Probe for examples and reasons, specific units/components that have improved or not, specific capacities that have been improved or not.) 5. If YES to question 4 - Has the improved capacity of ________ helped it to perform better (be more effective or efficient in its work)? • If yes, in what specific ways or areas of work? • If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions for those gaps, did other unaddressed problems or factors prevent the achievement of better performance?) 1,2 EQ3 RELATED 6. Do you work with any of these institutions? Ministry of Justice, Supreme Court of Rwanda, Ministry of Local Government, Office of the Auditor General of State Finances, Ministry of Education (including REB), Ministry of Health, Rwanda Bio Medical Center 3 7. For those with which you are familiar - how would you describe the level of need for capacity development (both human and institutional)? (in case of difficulty, propose a scale of 1 to 5) 3 8. Are the main problems able to be resolved with donor assistance? (Are they “changeable” within scope of international assistance?) 3 9. Is there a need for USAID support to meet those needs, or are they being met (or would be better met) through other means? 3 10. Would HICD method/approach be suitable for supporting capacity development in other government institutions? 3 Support to Rwanda Society for Performance Improvement 11. What interaction (if any) have you had with the Rwanda Society for Performance Improvement (RSPI)? (may need to explain what it is, originally community of practice of performance improvement trainers and consultants) n/a 12. What is your general impression of the RSPI? (Probe re governance, human resources, financial sustainability, member services, information management, and external relations/ communications.) • How has support from HICD/R affected the RSPI in those areas? 4 13. What do you see as the main needs of the RSPI, in order to become effective (in achieving its stated objectives as an organization) and sustainable? (Probe re same areas.) 4 14. What role could USAID play in further supporting the RSPI, especially in terms of boosting its sustainability? 4,5 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 93 Interview questions Most relevant EQ # • What specific kind of support would be most useful? Support to Civil Society (may first need to explain specific support provided by project, if informant not familiar) 15. Are you aware of the support the project has provided to civil society organizations? Do you work with any of the targeted organizations? (Probe to find out which of 10 targeted CSOs they are most aware of. If none, then move on to next section. If some, then focus on those for next questions.) n/a 16. Do you believe that the project has helped to build capacity of those CSOs? • If yes, how? If not, why not? (Probe for examples and reasons, specific CSOs that have improved or not, specific capacities that have been improved or not.) 1,2 17. In general, what are the main capacity needs of CSOs in Rwanda? (NGO type, 2nd and 3rd level) 3 18. Is there a need for USAID support to meet those needs, or are they being met (or would be better met) through other means? Or are they not possible for donor intervention to change? 3 General Questions – challenges, future, gender issues 19. What have been the major challenges or obstacles for the project generally? (Focus on external factors (such as coordination, politics, economics, security) that may have limited effectiveness.) 2,5 20. In general, do you think that the project’s approach and activities (based on HICD) were relevant to the situation in Rwanda since 2012? • Why or why not? 3 21. What do you think should be the priorities for future capacity development support in Rwanda by USAID? • Would another initiative based on HICD approaches be appropriate, for the Rwandan government? For Rwandan civil society? For RSPI? • Why or why not? 3 22. Are there any final recommendations that you would like the project or USAID to consider? 3,4,5 H. RSPI MEMBER FOCUS GROUP QUESTIONS 1. How did you first get involved with the HICD/R project and specifically with the community of practice that they supported? (Note: this is mostly a warmup question) 2. What do you think you have personally gained from that involvement? (probe for whether benefits have been similar for all consultants/RSPI members, or different, and how project helped them to get CPT) 3. Has the CPT certification been helpful to you as a professional? If so, how? If not, why not? Would you encourage others to get certified? 4. What support do RSPI members (including yourselves) need to further develop their capacity for PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 94 working in performance improvement, including HICD methods? (probe re training, coaching, twinning, financial assistance, motivation, etc.) 5. With respect to RSPI as an organization now - what do you see as its primary purpose, and what can it offer to its members? 6. How do you see the future of RSPI? (probe on leadership, sustainability options, etc.) How do you think it can survive as an organization? 7. What are the main needs of the RSPI at this time? 8. In general, what should be the priorities for future support to performance improvement in Rwanda by USAID? • Would another initiative based on HICD approaches be appropriate? Why or why not? 9. Are there any other recommendations or ideas you would like the project or USAID to consider? I. NCBS STAFF FOCUS GROUP QUESTIONS 1. What kind of interaction have you personally had with the HICD/R project? 2. Do you believe that the project has helped to build the capacity of the NCBS? • If yes, how? If not, why not? (Probe for examples and reasons, human and institutional, specific capacities that have improved.) M&E, knowledge management, quality assurance were three main areas 3. Has the improved capacity of NCBS changed the way it works, or helped it to be more effective? • If yes, what has changed? If not, why not? (Probe: did the project target the correct or highest priority capacity gaps, did it choose the right solutions, were there other obstacles…) GOVERNMENT INSTITUTIONS 4. Are you familiar with the capacity building needs of any of these state institutions? Ministry of Justice, Supreme Court of Rwanda, Ministry of Local Government, Office of the Auditor General of State Finances, Ministry of Education (including REB), Ministry of Health, Rwanda Bio Medical Center A) How are those needs identified or verified?? 5. For those with which you are familiar - how would you describe the level of need for capacity building (both human and institutional)? (in case of difficulty, propose a scale of 1 to 5) 6. Would HICD method/approach be suitable for supporting capacity development in other government institutions? • Why or why not? What should be changed to make it suitable? CIVIL SOCIETY 7. In general, what are the main capacity needs of CSOs in Rwanda? (Probe re NGOs and networks operating at national or sub-national level) 8. What do you think should be the priorities for future support to capacity development of civil society by USAID? • Do you think HICD approaches are appropriate for civil society? Why or why not? 9. Are there any other recommendations you would like the project or USAID to consider? J. USAID STAFF FOCUS GROUP QUESTIONS 1. What is your understanding of the HICD approach? (Probe for key elements.) PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 95 • What was done to inform USAID staff in technical offices (and beyond) about HICD approach and the project? • Have you heard about HICD programming in Georgia? Or elsewhere? 2. What do you think the HICD/R project has achieved so far? (Probe - Especially in relation to the CSOs with which your offices are working.) • What challenges have you observed, from your technical office perspective? 3. How do you see the current level of need for capacity development in the government institutions with which you work most closely? • Probe about the following institutions, if not already covered. i. Ministry of Education (esp. REB), ii. Ministry of Health, and iii. Rwanda Bio Medical Center 4. Is the HICD approach suitable for addressing those needs in your government counterparts? • If yes, why? If not, why not? What needs to change for it be effective? • How else could USAID help develop capacity in those institutions? 5. Is the HICD approach suitable for addressing needs in civil society? • If yes, why? If not, why not? What needs to change for it be effective? 6. What would you like to see done differently, in a new intervention by USAID to promote performance improvement in Rwanda? (Probe for cross-cutting vs. standalone project, run by Program Office or by DG, options for “full HICD” treatment or partial, need to include “hard assets” in the package of support, etc.) 7. What other recommendations do you have, for this project or future USAID programming? K. CSO STAFF FOCUS GROUP QUESTIONS 1. Your organizations have been collaborating since (2013-14) with the HICD/R project. What kinds of project activities have you personally participated in? (probe re assessment process, training workshops, hands-on coaching, etc.) 2. How would you describe the HICD approach or methodology? 3. Has the project helped to bring about real change in your organizations? • If yes, how? (What changed and how did that happen?) How do you know the change was caused by the project? • If not, why not? In which areas was there less improvement? (Probe for examples and reasons, technical skills vs. management capacity, capacity change vs change in actual ways of working and results) 4. What challenges have your organizations faced in working with the project and achieving the expected results? Probe – if involved in the assessment process, did they ever discuss material needs, or was the process focused on needs for technical assistance, training, development of policies and procedures and strategies? 5. What are the challenges in Rwanda for developing the capacity of men and women in an equitable manner? • How did the project recognize those challenges, and how has the project team taken gender issues into account in its approaches and activities? 6. What do you think should be the priorities for future support to capacity development of civil society by USAID? • Would another initiative based on HICD approaches be appropriate for civil society? Why or why not? PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 96 • What could be done differently to have better results? 7. Are there any other recommendations you would like the project or USAID to consider? PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 97 ANNEX D: LIST OF PERSONS INTERVIEWED Type of Informant Organization Name and position USAID and U.S. Embassy USAID/Rwanda Joseph Rurangwa, COR USAID/Rwanda Gloria Jean Garland, DG Officer USAID/Rwanda Andrew Fourney, M&E Fellow USAID/Rwanda Adriana Hayes, Program Office Director USAID/Rwanda Marcia Musisi-Nkambwe, Mission Director USAID Emily Krunic, former DG Officer U.S. Embassy in Rwanda Jimmy Aguirre, Chief Political and Economy Officer HICD/R Project staff DAI Leslie Gonzales, current COP, former Deputy COP DAI John Palmucci, former COP DAI Albert Nzamukwereka, Deputy COP, former CSO Program Manager DAI Ivan Mbaraga, Civil Society Program Officer DAI Francisca Mujawase, M&E Advisor DAI Eric Tuyishime, COMPRAC Program Officer Short term consultants KNO Steven Kelly, KNO principal, former Director ISPI, trainer and coach KNO Dr. Ingrid Guerra-Lopez, Technical Coach Ministry of Agriculture and Animal Resources and affiliated agencies MINAGRI Raphael Rurangwa, Former Director of Planning MINAGRI Alfred Kamana, Procurement Officer Rwanda Agriculture Board Aimable Gatete, M&E Officer National Agriculture Export Board Eric Kabayiza Former Director of Planning National Capacity Building Secretariat NCBS Antonia Mutoro, Executive Secretary NCBS Judith Katabarwa, Division Manager, Strategic Operations & Quality Assurance NCBS Gerard Murasira and Pierre Kalinda, M&E Officers PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 98 Type of Informant Organization Name and position Other Government of Rwanda institutions Rwanda Management Institute Wellars Gasamagera, Director General Rwanda Management Institute William Twagirumukiza, Trainer Ministry of Finance and Economic Planning Obald Hakizimana, Economist, Macroeconomic Division National Institute of Statistics Rwanda Herve Kagimbura, Statistician Rwanda Governance Board Edward Kalisa, Head of Corporate Services and Projects National Women’s Council Alice Umuhire, Resource Mobilization Officer Rwanda Society for Performance Improvement RSPI Jean Marie Vianney Makuza, President RSPI Jules M. Sebahizi, Former President RSPI Judith Katabarwa, President-elect (listed above for NCBS) RSPI Godfrey Gakwandi, Vice-President Finance Africa Development Consultants (ADC) Rebecca Ruzibuka, RSPI member, consultant intern International organizations International Society for Performance Improvement Klaus Wittkuhn, President European Union Emanuel-Jose Amaral, Head of Governance and Economics Section United Nations Development Programme Stijn Houben, Programme Analyst UK Department for International Development Mark Montgomery, Senior Governance Advisor Independent experts Institute of Policy Analysis and Research (IPAR) Alfred Bizoza HICD 2020 and HICD-Plus Projects, Georgia Roman Tsutskiridze, COP Targeted civil society organizations Conseil de Concertation des Organisations d’appui aux Initiatives de Base Jean Claude Ngendandumwe, Executive Secretary Ingabo Gilbert Usanase, Executive Secretary Caritas Fr. Emmanuel Rubagumya, Deputy General Secretary African Evangelical Enterprise Isaac Nyarwaya, Chief Operating Officer African Evangelical Enterprise Charlotte Usanase, M&E Officer PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 99 Type of Informant Organization Name and position Francois-Xavier Bagnoud Emmanuel Habyarimana, Executive Director and Frank Sebaziga Imbaraga Juvenal Musine, Program Manager Never Again Rwanda Eric Mahoro, Program Director Young Women's Christian Association Prudentienne Uzamukunda, Executive Secretary, Kyambadde Robirah, and Ruhumuriza Theojore Society for Family Health Manasseh Wandera Gihana, Executive Director Other civil society actors Rwanda Civil Society Platform Edouard Munyamaliza, President Transparency International Rwanda Appolinaire Mupiganyi, Executive Secretary PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 100 ANNEX E: DOCUMENTS REVIEWED HICD/R Project documents • Scope of Work of HICD/R Project • Scope of Work of KNO as subcontractor • Annual reports (Years 1-3) • Performance Monitoring Plans (Years 1-4) • Quarterly reports (Years 1-4) • Annual Work Plans (Years 1-4) • Organigrams and staffing lists (Years 1-4) • Performance Solutions Packages for all targeted institutions • Pre and post test results, data and samples of those tests • Organizational Capacity Index tool and data • Impact Perception Survey – annual survey reports, raw data and sample surveys • List of PSP recommendations, indicating implementation by each institution • Memoranda of Understanding between USAID, DAI and targeted institutions • Sample of tools used to guide assessments of targeted institutions • Reports delivered by all short term consultants • Desk Review Report on Proposed Civil Society Organizations to be HICD Rwanda Partner Candidates • Documents related to COMPRAC and RSPI establishment and activities, including RSPI constitution, draft strategic plan, lists of members, newsletters, meeting notes, election results, and conference information • Sample of institutionalization plans developed with targeted institutions USAID reports and information • Request for Proposals for HICD/R Project • USAID/Rwanda Country Development Cooperation Strategy, 2015-19 • Human and Institutional Capacity Development Handbook, 2010 • Human and Institutional Capacity Development Policy Paper, 2009 • Feed the Future HICD Strategy Review, May 2015 • Evaluation of the USAID/Georgia FORECAST Program, 2011 • HICD-Plus (Georgia) Final Report, 2015 • Human and Institutional Capacity Development (HICDpro) Request for Proposals, 2012 • 2014 CSO Sustainability Index for Sub-Saharan Africa (and draft Rwanda section for 2015 Index) • USAID/Rwanda Gender Assessment, 2011 Other documents and reports • Mapping of the Civil Society and Project Identification of a Support Program to the Civil Society in Rwanda, Prepared by Costantini, Verdecchia and Rutayisire, December 2013 (and related comments on the report by Rwanda Governance Board) • Multi-Donor Civil Society Support Programme 2015-2021, DFID Rwanda Business Case, July 2014 • Escaping Capability Traps through Problem-Driven Iterative Adaptation (PDIA) by Andrews, Pritchett and Woolcock - Working Paper 299, Center for Global Development, 2012 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 101 • Building a Legacy of Performance Improvement in Rwanda, by Steven J. Kelly and John Palmucci, in Performance Improvement, vol. 53, no. 8, September 2014 • Sectoral capacity building plans, provided by NCBS • NCBS Capacity Building Toolkit • Rwanda 2015 Human Rights Report, U.S. Department of State • Rwanda Governance Scorecard 2014, by Rwanda Governance Board • Rwanda Civil Society Development Barometer, Transparency International, December 2015 • The Promise of Imihigo: Decentralized Service Delivery in Rwanda, 2006-2010, by Daniel Scher • National report submitted to Human Rights Council by Rwanda (United Nations Universal Periodic Review), October 2015 • Strategic Plan for the Transformation of Agriculture in Rwanda Phase III, 2013 • Rwanda Citizen Report Card Survey, 2015, Rwanda Governance Board • Sample of plans and policies developed by CSOs with project support • Strengthening Local Capacity in Southeast Asia: Approaches and Experiences, RTI International for USAID, June 2016 Additional websites viewed • USAID/Rwanda https://www.usaid.gov/rwanda • DAI Project page http://dai.com/our-work/projects/rwanda%E2%80%94human-and￾institutional-capacity-development-project-hicd • RSPI http://www.rspi.rw/ • ISPI http://www.ispi.org/ • Rwanda Governance Board http://www.rgb.rw/index.php?id=2 • Rwanda Management Institute http://rmi.rw/ • The Kirkpatrick Model http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel • Sites of selected targeted institutions PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 102 ANNEX F: DATA SYNTHESIS TABLE FORMAT Note: For each informant interviewed by the Evaluation Team, one row of this table would be completed with the key information emerging from that KII as it related to each of the topics defined in the top rows of the table. A similar table was used to summarize and analyze FGD data. # Name Organiza tion Male Female Note￾taker Type of Informant Govt CSOs govt CSOs govt CSOs 1 John Doe USAID 1 W USAID Identification monitoring of performance indicators # of participants challenges facilitating factors EQ 1 - project effects on technical and management performance EQ 2 - challenges and facilitating factors for capacity development govt (7 institutions) CSOs suitability of HICD priority needs USAID support and duration priority needs USAID support and duration for current project future project Useful Quotes Team comments on data quality RSPI as organization RSPI members and reliability EQ 3 - unmet capacity needs of non-targeted institutions EQ 4 - capacity gaps of RSPI, options for USAID support EQ 5 - project implementation and management, lessons learned Other comments by informant PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 103 ANNEX G: PMP INDICATOR DATA TABLE PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 104 Indicator name Y1 Target Y1 Actual* Y2 Target Y2 Actual Variance Y3 Target Y3 Actual Variance Y4 Target** Y4 Actual (3 quarters) Variance Y5 Target LOP target LOP actual 1.1 Number of partner institution representatives receiving HICD/R supported training 260 500 611 18% 350 355 1% 200 201 0% 100 1150 1427 1.2 Number of days of coaching and mentorship provided by HICD/R to partner institutions representatives 77 700 906 23% 400 550 27% 300 205 -46% 300 1700 1738 1.4 Average variance between pre and post-training test scores measuring improved professional knowledge acquired by the trainees during the training session 37.8% 40% 38.1% -5% 30% 39.5% 24% 30% 16.7% -80% 30% 30% 33% 1.5 Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions 5 0 6 6 0% 10 10 0% 3 4 25% 3 13 10 1.6 Number of food security private enterprises (for profit), producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) receiving USG assistance (RiA) (WOG) 3 3 0% 0 3 3 2.1 Number of performance solution package recommendations implemented by partner institutions 15 80 225 64% 80 169 53% 60 67 10% TBD TBD 476 2.2 Score in percent, of combined key areas of organization capacity amongst USG direct and indirect local implementing partners (S) 4.5.1 (CBLD 5): (4.5.1-27) 63% 68% 80.6% 16% 68% 83% 18% TBD n/a n/a TBD TBD 76% 2.3 Number of supported Community of Practice members that become Certified Performance Technologists (CPT) 0 0 10 9 -11% 5 0 0 n/a n/a 0 15 15 3.1 Partner institution impact perception scores on the effectiveness of PSP recommendations implemented 70 89.8 22% 75 91.5 18% 75 n/a n/a 75 75 90.7 3.2 Percentage change in HICD/R supported CSO members receiving direct services from their CSO 25% 26% 4% 15% 122% 88% TBD n/a n/a TBD TBD 74% 3.4 Percentage change in COMPRAC member attendance and participation in HICD-related professional forums and meetings 60% 100% 40% 40% -66% -161% 30% n/a n/a 20% 38% 17% 3.5 Percentage change in the number of RSPI members that report using performance improvement approaches in their work 40% n/a n/a 30% 35% n/a indicator did not exist indicator did not exist indicator did not exist *Actuals reported here for Year I represent May 2012-September 2013. Numbers reported here differ from those in Year I annual report which only covered May 2012- June 2013. HICD/R adjusted its Project Year to align with USAID fiscal year starting from Year II. **Figures used are from Year 4 PMP, which in some cases varied from the Year 3 Annual Report indicator did not exist indicator did not exist indicator did not exist No longer applicable due to nature of new CSOs base year PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 105 ANNEX H: CONFLICT OF INTEREST STATEMENTS PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 106 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 107 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 108 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 109 PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 110 ANNEX I: PROJECT SCOPE OF WORK EXCERPT The text below is an excerpt from the Scope of Work of the Project, which is also contained in the contract between USAID and DAI as implementer. It shows the extent of the performance monitoring that was expected as a deliverable of the Project. “As outlined in the Background section, the contractor shall use the process for implementing the HICD model in targeted institutions. These five steps described below constitute project deliverables. Step 1: Confirm Participation of Targeted Institutions… Step 2: Conduct Performance Assessments… Step 3: Develop the Performance Solutions Package… Step 4: Implement Performance Solutions… Step 5: Operationalize Performance Monitoring System A hallmark of HICD is its results-oriented nature. In order to gauge the performance gains of partner institutions, it is important to establish measurable performance goals for each area of performance targeted through HICD initiatives. The HICD contractor shall assist host country partner organizations with establishing baseline performance data along with an appropriate performance monitoring and management system that is tailored to the context of each organization. Ideal performance statements will be developed based on organizational priorities and strategies and compared to actual performance providing critical information from which management can analyze the performance of the organization. Institutionalizing a performance monitoring system enables a host country partner organization to regularly monitor its own performance in critical areas that are leading indicators of the organization’s overall performance success. The contractor shall assist host country partner organizations in developing a comprehensive measurement tool that will establish performance measures at every level of performance, individual worker level, process level, and organizational level, with indicators at each level rolling up to the next highest level. The performance data being collected should be directly linked to the desired and actual performance as defined in the performance assessment. This will allow for performance to be continually monitored, thus providing a strong, data-driven basis for making management decisions. The contractor shall assist host country partner organizations with developing the processes to analyze data on a regular basis and ensure that appropriate corrective action is taken when needed. These processes for data collection and analysis should be as simple as possible, integrated into daily functions, and not add any unnecessary bureaucratic requirements. A reliable, working performance monitoring and management system is one of the keys to making the performance of an organization sustainable. While this information will be mainly used by host country partner organizations for performance and strategic management, it should also be accessible to the Mission for its reporting and management needs. The performance monitoring and management system should allow for strategic course corrections at the macro-level as well as the re-alignment of activities being implemented throughout all levels of the selected host country partner organizations. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 111 While the internal performance monitoring and management system is being established or strengthened at host country partner organizations, the contractor shall conduct periodic performance re-assessments in order to gather data on the indicators established in the initial performance assessment. The timing of such re-assessments shall be approved by the COTR. The information from these re-assessments should be used to analyze progress against the baseline, suggest revisions to the performance solutions package, and inform the Mission’s reporting of results. Once the internal performance monitoring and management system is operational, host country partner organizations should be able to transmit relevant performance data and information directly to the Mission and other stakeholders. The contractor shall ensure that the commitment to gathering this data, and access to it, is explicitly stated upfront in the initial Memorandum of Understanding with each host country partner organization. The contractor shall also conduct a performance re-assessment to confirm any performance data coming from host country partner organizations’ internal performance monitoring and management systems, as requested by the COTR. In consultation with the Mission, the Contractor shall also use the information gathered from the host country partner organization’s performance monitoring and management system to provide input for the Mission’s Performance Monitoring Plan, utilizing indicators tied to the performance of host country partner organizations to show results of HICD initiatives.” PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 112 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov