MIDTERM EVALUATION AGILE AND HARMONIZED ASSISTANCE FOR DEVOLVED INSTITUTIONS (AHADI) FINAL REPORT This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), a Tetra Tech Company. AGILE AND HARMONIZED ASSISTANCE FOR DEVOLVED INSTITUTIONS (AHADI) MIDTERM EVALUATION FINAL REPORT July 5, 2018 USAID/Kenya and East Africa IDIQ No. AID-623-1-12-00001 Award No: AID-615-TO-17-00008 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii | AHADI MIDTERM EVALUATION REPORT USAID.GOV CONTENTS LIST OF ACRONYMS...........................................................................................IV EXECUTIVE SUMMARY........................................................................................1 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS............................5 EVALUATION QUESTION 1 ...............................................................................................................................5 EVALUATION QUESTION 2 .............................................................................................................................10 EVALUATION QUESTIONS 3 AND 3(A)......................................................................................................14 EVALUATION QUESTION 4 .............................................................................................................................19 EVALUATION QUESTION 5 .............................................................................................................................21 RECOMMENDATIONS........................................................................................25 ANNEX 1. EVALUATION METHODS AND LIMITATIONS.........................28 ANNEX 2. GENDER CONSIDERATIONS ........................................................31 ANNEX 3. LIST OF DOCUMENTS REVIEWED..............................................32 USAID.GOV AHADI MIDTERM EVALUATION REPORT | iv LIST OF ACRONYMS ADP Annual Development Plan AHADI Agile and Harmonized Assistance for Devolved Institutions ASAL Arid and Semi-Arid Lands CAF County Assemblies Forum CCI County Capacity Index CEB County Education Board CEC County Executive Committee CG County Government CID Centre for International Development CIDP County Integrated Development Plan CLO County Liaison Office CO County Officer CO Contracting Officer COG Council of Governors COP Chief of Party COR Contracting Officer Representative CPSB County Public Service Board CPST Centre for Parliamentary Studies and Training CRA Commission on Revenue Allocation DCOP Deputy Chief of Party DDWG Donor Devolution Working Group DFID Department for International Development DSWG Donor Sector Working Group EDE Ending Drought Emergencies EDY Education and Youth Office ENRM Environment, Natural Resources Management GOK Government of Kenya HO Home Office HPN Health, Population and Nutrition IBEC Intergovernmental Budget and Economic Council IDS Institute for Development Studies IFMIS Integrated Financial Management System v | AHADI MIDTERM EVALUATION REPORT USAID.GOV IGRTC Intergovernmental Relations Technical Committee KLRC Kenya Law Reform Commission KSG Kenya School of Government KSP Kenya Support Project LDI Local Development International M&E Monitoring and Evaluation MCA Member of County Assembly MERL Monitoring, Evaluation, Research and Learning MERLO Monitoring, Evaluation, Research and Learning Officer MODP Ministry of Devolution and Planning MOEST Ministry of Education Science and Technology NCCK National Council of Churches in Kenya NDMA National Drought Management Authority OEG Office of Economic Growth OGE Office of Government Ethics PFM Public Finance Management PEM Public Expenditure Management PPOA Public Procurement Oversight Authority PREG Partnership for Resilience and Economic Growth RTI Research Triangle Institute REC Regional Coordinator SHARED Stakeholder Approach to Risk and Evidence Based Decision-Making SOCATT Society of Clerks-at-the-Table SUNY State University of New York TISA The Institute for Social Accountability UNDP United Nations Development Program USAID United States Agency for International Development USAID/KEA USAID/Kenya and East Africa USG United States Government UI Urban Institute WB World Bank WASH Water, Sanitation and Hygiene 1 | AHADI MIDTERM EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY BACKGROUND AND PROJECT PURPOSE Kenya’s Vision 2030 and its 2010 Constitution provide the foundation for transformational changes in the nation’s governance and economy. These seminal documents and policies that followed set Kenya inexorably on the path of decentralization by creating a two-tier governance system with 47 newly created county governments under one national government. The United States Agency for International Development’s (USAID’s) Country Development Cooperation Strategy (CDCS) supports this endeavor and accordingly positions devolution at the heart of its assistance framework, principally through formation of the five-year, $49.4 million Agile and Harmonized Assistance for Devolved Institutions (AHADI) Project.1 The project’s purpose is to enhance capacity to implement devolution by piloting innovative tools, knowledge￾sharing mechanisms and capacity-building activities that benefit both devolved and national-level institutions in partnership with Kenyan institutions. Thus, the project’s mandate is to: ● Strengthen the capacity of counties through training, mentoring and technical assistance for key leaders – building skills and accountability in governance and service delivery; ● Build the capacity of civil society to more effectively represent citizen interests and needs to county governments; and ● Support institutions and structures that will facilitate the devolution process and strengthen cooperation and ties between counties and national government. EVALUATION PURPOSE This is a midterm evaluation of the AHADI project. The evaluation is designed to facilitate learning and utilization by USAID/KEA, AHADI implementers and other stakeholders. It identifies opportunities to strengthen the existing AHADI project and recommendations for future USAID devolution programming in Kenya. The evaluation also identifies lessons that may be useful for USAID decentralization/ devolution programming globally. AHADI has an important opportunity to anchor the recommendations with newly elected county and national government leadership. EVALUATION QUESTIONS Five broad questions guided this evaluation, resulting in conclusions regarding current activities and providing recommendations going forward: 1. Evaluation Question 1 (EQ1): How appropriate is the AHADI design? 1 The AHADI contract was awarded to the SUNY/Center for International Development in mid-December 2013, with an official start date of January 1, 2014 (Award Number: Award No: AID-615-C-14-00002). The current end date extends to December 2018 (five years). The period covered by this evaluation is January 14 to September 2017. SUNY’s implementing partners are the Urban Institute (UI) and Local Development International (LDI). USAID.GOV AHADI MIDTERM EVALUATION REPORT | 2 2. Evaluation Question 2 (EQ2): What factors have contributed to and hindered implementation of AHADI, and how well has AHADI addressed implementation challenges? 3. Evaluation Question 3/3a (EQ3): How does the scope of AHADI (including number and selection of counties and types of interventions) affect project results? EQ3a: How effective are AHADI’s interventions in building capacity of key partner institutions at the national and county levels, and citizen engagement in the devolution process? 4. Evaluation Question 4 (EQ4): How effective is AHADI’s cross-sectoral approach and funding? 5. Evaluation Question 5 (EQ5): To what extent are SUNY and its subcontractors effectively organized and deployed to implement AHADI? DESIGN, METHODS AND LIMITATIONS This qualitative evaluation utilized both primary and secondary data. The team relied on desk review of relevant documents (see Annex 6), and key informant and group interviews. USAID requested visits to six of AHADI’s 22 counties, so selection criteria were applied and accordingly interviews were conducted in Bomet, Mombasa, Nairobi, Tharaka-Nithi, Turkana and Vihiga counties. 2 National institutions, implementing partners, other donors, AHADI staff and USAID representatives also participated in interviews. In all, the evaluation conducted 55 interviews, encompassing 122 respondents (some in groups). Data analysis utilized content analysis of qualitative data, pattern analysis by means of a coding process to identify and name key themes, and constant comparative analysis, which involved comparing and triangulating results from all data sources to identify the main evaluation findings that respond to evaluation questions. As with any assessment or evaluation, various limitations occurred, such as selection biases and availability of key informants. The team sought to mitigate these limitations by drawing from a respondent pool that was as wide as possible and using by multiple data sources, data collection and analysis methodologies to triangulate responses so that no single piece of biased data skewed the analysis. In line with Automated Directives System (ADS) 205.3.6.2, the evaluation design reflected sex-disaggregated and gender-specific data considerations. None of the evaluation questions or interview guides specifically explored gender. Also, the respondent selection was based on institution, position and knowledge of the AHADI project. While the evaluation did not identify differences in results and benefits of the project along gender lines, the majority of AHADI beneficiaries interviewed for the evaluation were male. Data trends presented in the report generally reflect the institutional affiliation of respondents rather than their gender. Annex 4, however, outlines steps AHADI has taken to address gender considerations going forward. AUDIENCES The key audience for this evaluation is the USAID/KEA Democracy, Governance and Conflict (DGC) Office and the Center for International Development of the State University of New York (SUNY/CID), which will use the evaluation to inform project implementation. Other possible audiences for the report include: ● Government of Kenya partners: The Ministry of Devolution and Planning (MODP, now MOA), the National Treasury, Council of Governors (COG), Senate, the Intergovernmental Relations 2 The county selection criteria considered the balance of rural and urban counties, cosmopolitan and homogeneous populations, geographical and political coverage, key thematic areas and sectoral support provided by AHADI and security considerations. 3 | AHADI MIDTERM EVALUATION REPORT USAID.GOV Technical Committee (IGRTC), County Assembly Forum, Kenya School of Government (KSG), Center of Parliamentary Studies and Training and the Commission on Revenue Allocation (CRA). ● Non-Governmental External Audiences: Including the University of Nairobi, the Institute for Development Studies (IDS) and selected civil society organizations. ● USAID Audiences: USAID Front Office, Development Objective 1 (DO1) team members and USAID’s Strategic Planning and Analysis Office. ● Other Donors: Primarily the United Nations Development Program (UNDP), the World Bank (WB) and the United Kingdom’s Department for International Development (DFID). CROSS-SECTORAL APPROACH AND FUNDING Though AHADI is primarily focused on improving the functioning of Kenya’s newly devolved government system, it also has an impact on other sectors, especially as counties take on significant responsibilities for service delivery. Kenya’s devolution process affects every sector in which USAID/KEA works, namely health, environment and natural resources; agriculture; education and youth development; and, of course, governance. Thus, AHADI is meant to accommodate funding and technical input from other USAID/KEA sectors. To date, AHADI has received various streams of funding, including specific earmarks with their own thematic requirements, defined outcomes, priorities and indicators. In addition, AHADI has received funding of approximately $6.26 million from the United Kingdom’s Department for International Development (DFID) for certain aspects of its work.3 GENDER In line with Automated Directives System (ADS) 205.3.6.2, the evaluation design reflected sex-disaggregated and gender-specific data considerations. None of the evaluation questions or interview guides specifically explored gender. Also, the respondent selection was based on institution, position and knowledge of the AHADI project. While the evaluation did not identify differences in results and benefits of the project along gender lines, the majority of AHADI beneficiaries interviewed for the evaluation were male. Data trends presented in the report generally reflect the institutional affiliation of the respondent rather than their gender. Annex 4, however, outlines steps AHADI has taken to address gender considerations going forward. KEY FINDINGS AND CONCLUSIONS QUESTION 1: DESIGN RELEVANCE ● The AHADI design was (and is) relevant. National and county-level beneficiaries affirmed the relevance of AHADI assistance, in particular, the choice to work at both national and county levels, and to support both policy and practical skills development. ● The AHADI design was flexible, allowing assistance to be demand-driven. 3 To date, cross-sector funding has been received by: 1. OEG - Feed the Future (Resilience), $2.975 million; 2. OEG – WASH, $1.062 million; 3. EDY Basic Education, $7 million; 4. Climate Change (Both USAID and DFID), $630,000; 5. Good Governance, $5.1 million. General DG and Local Governance, $13.633 million; 5.2. DFID Good Governance, $2.284 million; 5.3. Anti￾Corruption, $350,000; 5.4. Political Processes, $300,000; and 5.5. Extractives Industry Transparency Initiative, $800,000. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 4 ● The purpose of the cross-sectoral approach in support of devolution was not well understood or enforced Mission-wide. Devolution/ governance tended to be seen as a sector rather than as a driving philosophy as envisioned in the CDCS. QUESTION 2: IMPLEMENTATION FACTORS ● AHADI built positive relationships that facilitated implementation. AHADI invested time and effort in securing stakeholder buy-in, resulting in new allies and close working relationships. ● The grant mechanism has been a major strain on AHADI and USAID. At the time of the evaluation, AHADI was reportedly managing 105 grants. ● Restricted access to leadership of key partners hinders implementation. The 2015 “communication guidance” memo has limited AHADI’s access to key decision-makers (e.g., governors, assembly speakers) and limited shared learning by the Devolution Sector Working Group (DSWG). QUESTION 3: SCOPE AND INTERVENTIONS ● The AHADI scope was overly ambitious. AHADI works with multiple actors in 22 counties, 11 national institutions and several USAID sector offices. ● County beneficiaries appreciate the public expenditure management (PEM) approach. Respondents related that the PEM intervention has been effective, complements other donor efforts and should be continued. QUESTION 4: CROSS-SECTORAL APPROACH ● The cross-sectoral approach realized positive outcomes when stakeholders shared interests, were committed to coordination and managed resources well. AHADI has demonstrated a lot of flexibility and goodwill to make cross-sectoral coordination succeed. ● The cross-sectoral approach is not well understood, embraced or adequately driven by the Mission as a whole. Respondents described a lack of unifying vision for the approach. QUESTION 5: STRUCTURE ● The current personnel structure limits the ability of the project to effectively deliver on its mandate. The project is understaffed, lacking ground presence in counties and sufficient in-house technical expertise to manage and deliver interventions of consistent quality. KEY RECOMMENDATIONS The items below are a representative sample and do not constitute the entire list of recommendations. ● Grant AHADI a no-cost extension. The evaluation team strongly recommends giving the AHADI project a no-cost extension so that it has ample time to solidify promising interventions. (e.g., PEM). AHADI should also consider increasing assistance in three related areas during this extension: 5 | AHADI MIDTERM EVALUATION REPORT USAID.GOV 1. Support to county assemblies so that both legislative actions and oversight can improve. 2. Support to improve inter-governmental relations 3. Civic engagement (both public participation and civic education) ● Test the new citizen engagement strategy and document learning. The new citizen engagement strategy that AHADI has developed should be tested in the time remaining, with a view to capturing best practices. This learning should be well-documented and used to inform future programming. ● Focus on governance and reduce cross-sectoral activities. A future program should focus on supporting efficient governance and reducing cross-sectoral involvements. USAID should also clearly communicate its vision to support devolution to the entire Mission and implementing partners, and assign a senior staff person (preferably from the Front Office) to champion its devolution assistance strategy. ● Rescind or revise the “communication guidance.” USAID should reassess the communication guidance issued to AHADI in 2015 to determine whether it still serves the intended purpose. Related to this, USAID should allow AHADI leadership to regularly attend Devolution Sector Working Group (DSWG) meetings to better share sector learning. ● Design a future devolution project. The design and scope should be carefully constructed to ensure that it does not attempt to fulfill too many objectives. One key area of support should be to facilitate national institutions to transition from a “command and control” paradigm to “enabling” counties to fulfill devolved functions. ● Review the grants mechanism for future programming. The assistance delivery mechanism needs to be re-examined in a future project design. Eliminating or reducing the use of grants could enable more agile and responsive programming. ● For future programming, increase in-house technical staff and renegotiate the use of local subcontractors. The implementing partner for future devolution support should reduce reliance on individual consultants in favor of permanent staff. Also, USAID should consider renegotiating with the Government of Kenya (GOK) on the use of local subcontractors. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Findings and conclusions are organized by the five evaluation questions. Recommendations are arranged in one section, presented at the end. EVALUATION QUESTION 1 How appropriate is the AHADI project design? This evaluation question examined the appropriateness of the AHADI design in relation to the project’s ability to effectively and pragmatically contribute to implementation of devolution in Kenya and to demonstrate the USAID.GOV AHADI MIDTERM EVALUATION REPORT | 6 validity of the theory of change upon which it is premised.4 The team assessed whether the design remains relevant in the current political landscape and identified recommendations for improving the design in response to both the current political situation and project learning to date. It is well understood that internationally, devolution stratagems (more often referred to as “decentralization”) sometimes take decades to attain enduring change in government operations; economic impacts are likewise a long process. Thus, this evaluation was not expected to draw final conclusions on the theory but was able to make observations regarding the efficacy of the strategies and processes being pursued. FINDINGS This evaluation question obtained the perspective of stakeholders most familiar with the AHADI project design: staff from AHADI and SUNY/CID, USAID and other donors, implementing partners and, especially, the Ministry of Devolution and Planning because of its familiarity with design features. The evaluation team deemed it useful to set the theory of change as the context in which AHADI evolved, rather than as a verifiable achievement at this early stage in Kenya’s devolution journey. While many respondents were not intimately conversant with the stated theory of change per se, the team nonetheless derived its estimation of the theory’s validity from several reliable perspectives. The team explored themes such as AHADI’s operations in relation to the complexity of devolution, the relevance of targets of intervention, the adoption of a “demand-driven”5 approach, and what came to be labeled the “lean Secretariat” staffing model. General perceptions of AHADI’s design as relevant and valued: Respondents at both national and local levels were nearly unanimous in affirming the value of AHADI’s efforts to respond to and assist their efforts to meet the challenges of devolution. Likewise, implementing partners valued key elements of the AHADI design, such as flexibility, responsiveness, collaborative decision-making and the shift toward more mentoring and technical assistance for public expenditure management (PEM). At the national level, a key design element was the mandate to strengthen intergovernmental relations and performance among agencies having obligations to support devolution.6 Many of these agencies had overlapping responsibilities and potential for institutional conflict was high. While other decentralization support programs may target only one or a few major national institutions, AHADI was tasked with and encouraged to respond to all devolution-related institutions. This element of the AHADI design construct did not impose constraints on which institutions were eligible to receive assistance. This occasioned both positive and negative outcomes. On the positive side, AHADI was able to work with a wide variety of institutions without favoring a few. On the negative side, this increased AHADI’s already demanding workload. 4 USAID/KEA articulated its theory of change as: “If the devolution process is effectively implemented, then Kenya’s governance and economy will be sustainably transformed.” For greater elaboration, see the entire content of the 2014–2018 CDCS’s discussion of Development Objective 1 (DO1), pp. 27–43, and more specifically the articulation of the underlying hypothesis on p. 29. 5 We note some confusion regarding the term “demand-driven.” Internationally, it is typically used to point to strategies that systematically elicit, define and are responsive to a beneficiary’s felt needs, rather than pre-defining or pre-packaging interventions. It is fundamentally different from an “open checkbook” style because it seeks to narrow, focus and refine an array of interventions, but not simply react to random requests from beneficiaries. 6 Agencies include Treasury, the Commission for Revenue Allocation (CRA), the Ministry of Devolution and Planning (MODP), the Council of Governors (COG), the Secretary to Parliament, the Kenya School of Government (KSG), the County Assemblies Forum (CAF), the Society of Clerks-at-the-Table (SOCATT), the Intergovernmental Technical Relations Committee (IGRTC) and the Kenya Law Reform Commission (KLRC). In addition, AHADI networked with non-USAID donors which also interacted with one or more of the aforementioned national agencies (i.e., chiefly UNDP, World Bank and DFD). 7 | AHADI MIDTERM EVALUATION REPORT USAID.GOV A key observation among respondents in various USAID sectors, AHADI Nairobi and SUNY/CID leadership, as well as several government institutions, is that devolution is more complex than initially anticipated. Respondents remarked on the need in the future to redesign the assistance approach of AHADI (or its successor) in light of learning gained about operations in the complex political environment. Several respondents acknowledged that Kenya’s current factional political environment also increased the project’s complexity. While respondents were not requested to suggest future design elements, it is evident from discussions responding to other questions (especially Q2 and Q3) that continued focus on PEM, support to assemblies and county planning would be high on the list. Approach to County Engagement: Several respondents noted that in the early stages of the project, the AHADI team did not have a clearly defined logic for how to engage counties. Thus, the default course of action was initially to use what amounted to an “open checkbook” approach by responding to numerous needs expressed by beneficiaries. As AHADI matured, its design fostered adaptive management that notably permitted it to become a more focused effort by concentrating on critical assistance needs common to most of the 22 counties. The evaluation found that evolution to a more strategic, demand-driven approach was effective in responding to beneficiary’s most vital priorities, while streamlining assistance into areas that would create higher impact. The project avoided using a “supply-side” approach common in many decentralization efforts, in which it might have relied on pre-designed, planned and packaged interventions. Opportune Timing: AHADI’s original start date arrived as Kenya’s groundbreaking devolution experiment was in its early stage and rapidly beginning to dominate both local and national government operations, priorities and intergovernmental relationships. This backdrop provided an opportunity for AHADI to align activities and test options to address USAID’s theory of change. As such, AHADI’s design was premised on and advanced in a manner that encouraged a “devolution perspective” in USAID’s sector programming. It also challenged all sector programs to embrace the programming implications of Kenya’s commitment to devolved governance. The design encouraged AHADI support to national agencies to tackle their mandate to enable, rather than control, newly created local government bodies. Perhaps most importantly, the design called for practical interventions at the county level to demonstrate types of assistance that are most effective in advancing devolution. Uniqueness: Several respondents noted that the design was seen to be relatively unique among donor projects in both the breadth of support and its emphasis on multiple, practical demonstrations of good governance as a key means to improve governance, service delivery and economic progress. Comments illustrating this finding include: “AHADI’s program design is very different from others. It was the best. AHADI works along with county officers even after workshops, so capacity building doesn’t end with workshops” and “The AHADI program, in my view, has been effective because of the design. The program is designed to be participatory, end-user-focused and output-oriented.” As a measure of relevance, after an initial period of learning, AHADI responded assertively to the shift of policy and responsibilities apportioned to county governments and national support institutions as put in place by the Kenya Constitution of 2010. Two sections of the constitution set a firm legal basis for the devolution policy that AHADI was designed to support. Chapter 11 (Devolved Government) assigns various functions and powers to county governments. AHADI’s technical interventions, such as improving legislative skills, assisting development of county integrated development plans (CIDPs), improving specific staff skills and assisting the executive branch to develop policies reflecting the national devolution framework, were one means by which selected counties took up those responsibilities. Chapter 12 (Public Finance) provides for the allocation of a portion of national revenues to county governments so counties have at least some measure of resources needed to carry out new responsibilities. AHADI grasped that counties urgently needed help to USAID.GOV AHADI MIDTERM EVALUATION REPORT | 8 manage those resources in a sensible, systematic manner coupled with well-developed CIDPs. Thus, the project’s greater emphasis on public expenditure management significantly contributes to the vision of shared public financial resources set forth in the constitution. National-Level Assistance Challenges: At the national level, the project design inherently presented a challenging balancing act to the AHADI organization: on the one hand, it needed to equalize support to remain an impartial resource; on the other, it was advantageous to concentrate on agencies that demonstrated motivation for improvement or emerged to play the most active roles in the national devolution structure. It is therefore significant that, while some relatively minor reservations were articulated, every national agency interviewed expressed appreciation for AHADI’s assistance and none reported a grievance of favoritism. Three quotations illustrate these views. One major national organization stated: “AHADI support for institutional capacity building has to a large extent helped us focus on the desires of the constitution.” Another respondent said: “The intergovernmental design has been fantastic from AHADI – it leads to a more coherent devolution process. That the AHADI project design was able to work simultaneously in the intergovernmental space and at the local level has been fantastic. They have been able to build capacity of MODP and work with counties simultaneously.” Lastly, “Part of the support the CoG received is in identifying the gaps and filling these capacity gaps in order to enhance our service delivery. AHADI built our capacity in order to provide better services. … Through AHADI support we have managed to put policies in place.” In other ways, AHADI’s design positioned the project to support Kenya’s demanding journey toward an effective polity premised on devolved governance. Because Kenya’s devolution effort is so well-grounded in enabling legislation and policies supported by a robust institutional framework, the ability of AHADI to respond opportunistically is likely to have contributed to positive results.7 The evaluation team noted that both elected officials and administrative personnel had strong knowledge of and adherence to the spirit and legal responsibilities in these documents, and their appreciation of AHADI’s efforts to assist effective execution of these responsibilities. For instance, county assembly and executive officials alike referenced legal requirements to facilitate public participation, communication and civic education.8 Thus, the design’s inclusion of support for public participation and citizen engagement is prescient because of the importance placed on this by both national- and county-level personnel.9 It is, however, the case that AHADI did not fully engage this component. The primary reason why AHADI did not focus on public participation during the first half of the project at first was that GOK requested that AHADI phase support to public participation. This was so the project could first allow for building capacity of government institutions (supply side) and then strengthen the demand side during a second phase so that new county governments would have some capacity to respond to citizen demands. Having developed a new citizen engagement strategy in partnership with the SUNY/CID home office, the project is currently set to engage the public participation component going forward. 7 Compared to international experience, Kenya has built an impressive groundwork of policy and law supporting devolution. In addition to the 2010 Constitution of Kenya, the government’s attention to framing devolved governance is evident in major policy and strategy documents, including the County Governments Act No. 17 of 2012 (Revised 2016), MODP’s Policy on Devolved System of Government 2016, Kenya Vision 2030, the Government of Kenya Second Medium-Term Plan 2013–2017 and the Council of County Governors Strategic Plans for 2014–2017 and 2017–2022. Annual meetings of the Council of Governors were organized, well documented and often articulated key challenges as devolution progressed. 8 See Parts VIII, IX and X of the (Revised) County Governments Act No. 17 of 2016 9 The government is developing national public participation guidelines under AHADI’s facilitation. This should be completed in early 2018. This will bring together the policies of all 47 counties. An act is expected to pass. 9 | AHADI MIDTERM EVALUATION REPORT USAID.GOV While the team can report widespread affirmation of the project, it was not without issues. Two key design issues can be highlighted: Liabilities posed by the cross-sectoral design: The cross-sector approach was designed to accommodate financial and technical support from other sector actors involved in devolution activities. Key among these were USAID/KEA and the U.K.’s Department for International Development. EQ4 will present a more thorough analysis of the pluses and minuses of the cross-sectoral approach, but from a design perspective, the approach was not as effectively integrated into project activities as intended. Key liabilities included conflicting goals between AHADI’s mandate and that of sector actors, mediocre internal agency coordination coupled with no clear understanding or buy-in from other sector offices and a more demanding management load than anticipated. The “lean secretariat” model: The original design of the project intended to accomplish project activities primarily through local cooperating partners, thus permitting AHADI management to employ what has been described as a “lean secretariat” model. When negotiations with the Government of Kenya precluded use of local partners to deliver interventions, the underlying design assumptions and management implications were not correspondingly revisited, leading to what is perceived to be the core problem in the design, as currently applied. By necessity, AHADI had to shift to achieving objectives by administering all of the anticipated $15 million in grants under contract (GUCs) on its own, rather than through subcontractors. Thus, project implementation must occur through a voluminous number of mostly in-kind grants.10 Managing so many in￾kind grants is a formidable task, even with AHADI’s competent grants administration team. The impact of this essential change to AHADI’s mode of operations cannot be overstated. This issue arises in one form or another for all evaluation questions. CONCLUSIONS In assessing AHADI’s design relevance, it became clear that the complexity of devolution warranted considerable flexibility and responsiveness which the original design accommodated. As it progressed, AHADI learned from its missteps and paid attention to the feedback of beneficiaries (a hallmark of adaptive management). The AHADI team thus made important choices to shift from providing assistance on a somewhat random basis to become more strategically focused. The modification from using local subcontractors to administering grants and relying on consultants, however, has affected, and continues to adversely affect, the project’s agility. With this in mind, the evaluation team has found that AHADI has adopted a learning and adaptive approach to implementation, and concludes the following: Complexity of Devolution: The evaluation recognizes that a key learning of AHADI is that devolution is far more complex than originally thought. USAID’s CDCS certainly recognized devolution’s implications and admirably positioned support to devolution at the center of its assistance strategies. That said, high-impact, well-coordinated assistance delivery requires major and sustained buy-in from agencies and sectors that have diverse agendas, not the least because Kenya’s government – and services – will increasingly lean toward decentralized operations. Flexible Project Design: As the project matured, the flexible design of the project enabled AHADI to make some necessary adjustments to the way assistance is provided. Three areas stand out. First, CIDPs enable county leadership to plan and ground policies of devolved governance. AHADI’s assistance to the executive branch to develop CIDPs improved the quality and practicality of these key devices. Second, the relatively 10 The evaluation team’s research reveals that as of the end of 2017, AHADI had 105 grant initiatives; only 10 percent of these were closed out while a good number are still operational or in the process of closure. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 10 recent shift away from training and toward mentoring is a sound means to strengthen county staff skills with hands-on assistance. Third, AHADI’s new emphasis – working on parts of public expenditure management identified for support by beneficiaries and ideas for increasing citizen engagement and public participation efforts going forward – promises to bring greater focus and relevance to project interventions. EVALUATION QUESTION 2 What factors have contributed to and hindered the implementation of AHADI, and how well has AHADI addressed implementation challenges? This question sought to examine broad factors that contributed to or limited the implementation of AHADI, and how well the project has been able to address both internal and external implementation challenges. A sub-question also explored AHADI’s responsiveness, flexibility and adaptability to social-political changes and broader learning that occurred. FINDINGS Q2 findings are arranged to first delineate various “limiting factors,” followed by a discussion of “contributing factors” to include how AHADI successfully responded. LIMITING FACTORS The interviews yielded both internal and external limiting factors. The evaluation team defines internal factors as those that relate to USAID, SUNY headquarters and the AHADI project office in Nairobi. External factors largely relate to relationships with and among government institutions. Start-up challenges: As the findings section of EQ1 showed, the evaluation noted the highly dynamic and politicized environment AHADI was operating in. Although the evaluation found that most beneficiaries were ready to accept and capitalize on assistance, one of the key challenges during this time was the delayed commitment from certain national political leadership to grant necessary approval for project commencement. Interviews with respondents highlighted that this period was used to focus on relationship￾building, to overcome a general sense of suspicion and mistrust on the part of MODP in particular, as well as help stakeholders have a better understanding of project objectives. However, as the project moved forward, some new relationship challenges arose from AHADI’s inability to rapidly respond to county requests for specific assistance (drawn from “wish lists” that were sometimes beyond the project mandate). In some instances, the time taken to identify qualified, skilled consultants further exacerbated tensions between the project and county actors. Strained inter-governmental relations: AHADI also found itself encountering inter-governmental rivalries, misalignment of priorities and lack of cooperation. For instance, a respondent mentioned: “The counties have not been able to align their sectors to those of national government; hence, they experience challenges in their inter-government relations agreeing on sector goals.” Respondents said some institutions were prone to overstepping mandates and were seen as encroaching on the domain of other national institutions. From prior experience, the evaluation team can also assert that such rivalries and disagreements are not uncommon when political and administrative systems are being remodeled by devolution. Both AHADI staff and national government respondents noted that AHADI was sometimes drawn into conflicts between government institutions amid disagreements as to which agency should shoulder a specific 11 | AHADI MIDTERM EVALUATION REPORT USAID.GOV task. The AHADI team reported that it was occasionally called upon to intervene in these discussions, which required good negotiation practice to preserve the project’s impartiality. According to respondents in one county, there was a relatively limited understanding of separation of powers between county assemblies and county executives. The project was called upon to deftly help clarify matters. Staffing constraints: AHADI staff, USAID, other donors and county beneficiaries reported staff constraints as an impeding factor. This problem was introduced in EQ1. To briefly summarize under this heading, the evaluation team respondents referenced the strains produced by the “lean secretariat model” and the failure to revisit it after negotiations between the GOK and USAID limited use of local subcontractors. Respondents observed that AHADI then had to rely on an ill-defined “consultancy model” that, among other things, compelled the lean staff to identify and oversee numerous individual consultants and simultaneously issue more grants than anticipated. Respondents indicated that the consultancy model could lack continuity because consultants may change and may have a lower level of commitment to the project than full-time staff would. Quality supervision of implemented activities also suffered and overstretched the four regional coordinators, such that it could sometimes take several months before county personnel would see AHADI personnel. Respondents also expressed concern with the high turnover in personnel (both internal staff and consultants). Uneven consultant performance: Responses drawn from AHADI staff, USAID and UNDP suggest that consultants sometimes had insufficient aptitude and low commitment. Some regional coordinators noted this problem, as it fell to them to coordinate with counties and manage expectations when consultants failed to deliver quality assistance. Decentralization experience: A donor expressed the view that SUNY appeared to lack decentralization programming experience. This observation is logically confirmed by the unsystematic “wish-list” startup, regardless of whether this was a function of the previously described staffing challenges or SUNY/CID not transferring the sufficient strategic and tactical decentralization know-how it may possess. One respondent aptly noted, “Devolution is new to all of us.” Evaluators also heard from respondents who mentioned that AHADI was unable to fully utilize its foreign-based subcontractors enough to remedy gaps in its local staff’s experience. In hindsight, several AHADI respondents offered that a full-time, on-staff, experienced expatriate providing daily counsel might have helped alleviate this problem. Adding an experienced expatriate may still be useful but may not be realistic in the time remaining. Bureaucracy: The AHADI team and some donors noted that some implementation problems may be, at least in part, due to overly time-consuming internal bureaucratic procedures. This was evident in discussion with SUNY headquarters, which affirmed that AHADI is not truly agile and the administrative structures need to be streamlined and made simpler. Beneficiary respondents who had previously interacted with SUNY’s Parliamentary Support Project (PSP) contrasted AHADI as being much more bureaucratic and apparently frequently referred decisions to its home office and USAID for approval. The evaluation team did not sufficiently explore this problem to determine its origin or, more importantly, the reasons the problem was not being handled, even though it was clearly recognized as a constraint to agility. Interviewees also spoke about slow approvals processes. The evaluation team’s inquiry revealed that budget modification approvals seem to have taken a long time. For instance, the follow-up analysis found that “Mod 4” took three months between submission in October 2015 and approval in February 2016. More recently, a modification request submitted in June 2017 was in abeyance awaiting a new work plan approval. SUNY also described frustration with work plan approvals. According to SUNY, the 2016 work plan went through 13 revisions and was not entirely approved until the following year, in February 2017. Furthermore, in both 2016 USAID.GOV AHADI MIDTERM EVALUATION REPORT | 12 and 2017, USAID cross-sector teams did not meet any established deadlines for work plan approval. The evaluators did not follow up on these findings with USAID, and therefore cannot explain the delays. Grants management procedures: A review of the AHADI grant mechanism document detailed a process that was well advanced in concept, but exceptionally complex and seemingly lacking flexibility. According to respondents, the AHADI Procurement and Grants teams were stretched. Operational staff were sometimes either partially assigned to, or “pulled into,” administrative operations instead of program work. This seems largely due to the demands of administering grants; however, the team could not verify how much time technical staff took away from program operations. In an interview with the evaluators, SUNY headquarters staff acknowledged, “There is large administrative staff to prepare and process grants.” The evaluators found that the grants portion of the $49.5 million budget is $16.55 million. The bulk of funds disbursed thus far have been put into in-kind grants that have resulted in a correspondingly hefty management load. Discussions with the AHADI grants team indicated that at the time of the evaluation, they were managing 105 grants. Restricted visibility: Another key concern for several respondents has been the restricted operating room allowed to AHADI by USAID. Interviews made reference to the AHADI Communication Guidance memo that USAID issued in January 2015, which provides direction about SUNY’s communication with the GOK and other parties and USAID’s role in these communications. The guidance appears to restrict AHADI’s contact with partner institutions to technical personnel in these institutions. AHADI contact with senior echelons of government above a certain rank, including governors and assembly speakers – the county’s key decision-makers, is not formally permitted. Interviews with donors also supported the view that AHADI leadership should be included in the Devolution Sector Working Group (DSWG) to exchange technical views and help inform discussions on devolution. Similar sentiments were expressed by MODP, opining that AHADI’s continued absence at essential technical meetings is an impediment to devolution sector coordination. Respondents readily admitted that while the DSWG meets infrequently, AHADI project staff have valuable insights from their field activities and should have a seat at the table. (Please see further discussion on visibility under EQ5.) Coordination within USAID: Several respondent groups raised concerns regarding USAID’s in-house coordination for devolution support. The evaluators heard, in their words, that USAID sector teams were “not on board” with implementing AHADI, with one respondent noting, “It took us a long time to figure out what we could do for sectors and what we could not do.” Several respondents within the Mission believed that USAID did not effectively provide in-house coordination for the Agency’s devolution strategy. Aside from that, AHADI pointed to a lack of clear-cut expectations and direction from USAID regarding the strategy. A USAID respondent noted, “The different priorities in offices and different reporting ways complicated AHADI’s life.” There were also respondent observations that the USAID contracting office seemed to change staff rather frequently and seemed understaffed. FACILITATING FACTORS Building positive relationships: Respondents described how AHADI invested time and effort in securing stakeholder buy-in, resulting in new allies and close working relationships. Several respondents reported that AHADI has successfully leveraged connections with government and national institutions. The Intergovernmental Relations Technical Committee (IGRTC), the Commission on Revenue Allocation (CRA) and the Kenyan Law Reform Commission (KLRC) affirmed “cordial,” “productive” and “trusting” relationships with AHADI. 13 | AHADI MIDTERM EVALUATION REPORT USAID.GOV Several beneficiaries also lauded AHADI for encouraging their input during implementation (e.g. “AHADI listened to us”). Respondents at the national and county levels mentioned this in describing AHADI’s inviting them to either generate criteria for selection of consultants or help plan upcoming activities. UNDP noted that AHADI is well known on the ground and UNDP sometimes refers counties to AHADI for support. A UNDP respondent noted, “County partners talk about the value of AHADI.” Indeed, a county assembly speaker told the evaluation team, “There are so many people training outside, but AHADI is the best.” There was also goodwill from beneficiaries to acquire appropriate capacities and methodologies and translate those into new ideas. As a Bomet County official noted: “AHADI has contributed to the change in mindset of county staff toward taking charge of policy development and delivery of services.” Adaptive management: Interviews with AHADI project staff pointed to a slow start early in implementation after getting approval to commence activities. (See discussion under EQ1.) One of the main reasons cited was the challenge in identifying county priorities with which the project could align. This made it difficult to develop a set of relevant interventions that were county-specific, so the project initially reacted to “wishlist” requests from counties. AHADI staff described the situation: “In the beginning, counties just wanted everything. And no one else was really helping them. The demand was enormous. So, we began to help focus governance work in certain sectors.” In time, AHADI began working more systematically along thematic lines such as planning (e.g., support to CIDPs) and financial management, as linked to the PEM cycle. AHADI and USAID respondents indicated that the project was able to find a good stride in roughly the past 18 months. Responsiveness: Respondents pointed to AHADI’s ability to respond to beneficiary needs and challenges, appreciating the patience with which AHADI team sought their buy-in. Several respondents, including KSG, IGRTC and the Senate, praised the AHADI team for being politically tactful and strategic in delivering support to both national and county governments. Project reporting shows that in response to the 2017 election cycle, when county government programming was constrained by upper levels, AHADI sensibly focused on technocrats and staff at director levels and below to retain built capacity. Flexibility: Respondents had mixed views about AHADI flexibility. Those who thought AHADI was flexible cited its ability to conduct its business responsively, rather than sticking to a strict prescribed format of doing things. However, respondents contended that AHADI lacked flexibility. Many respondents who mentioned this largely attributed it to the grants mechanism, described in detail above. Earlier sections of the report also touch on this dichotomy, i.e., project flexibility and responsiveness in trying to address stakeholder needs on one hand and internal structural limitations on the other. Learning: Respondents expressed the view that learning by AHADI has steadily improved. For example, AHADI has learned about county planning and increasingly concentrated efforts around the CIDP as a linchpin in budgeting processes. Respondents, including AHADI, also mentioned the adoption of the “cluster” approach for training county personnel, thus extending support across counties and encouraging cross-county sharing. CONCLUSIONS The evaluation team concludes that despite AHADI’s difficult start, it was able to build resourceful relationships, accumulate political capital and focus assistance to be relevant and strategic. This is due in large part to the AHADI team’s rapid learning curve and willingness to adjust tactics as needed. AHADI is now valued and sought after by national and county entities seeking technical assistance. Undoubtedly, the internal project course correction improved programmatic focus and now AHADI is offering more demand-driven support than it was at the beginning. The utility of the AHADI support to beneficiaries has, in general, been a USAID.GOV AHADI MIDTERM EVALUATION REPORT | 14 good fit. Kenya’s devolution framework contained multiple provisions that were ambitious and untested, hence difficult to predict; however, amid many uncertainties, AHADI managed to be opportunistic and willing to experiment with different intervention models. AHADI’s learning curve has been equally impressive and well-informed. The project has also worked hard to build positive relationships with stakeholders. On the downside, factors impeding implementation, especially staffing issues, remain a serious shortcoming. The seeming inability to redesign after cancellation of the local subcontractor model contributed to implementation delays. A clear impediment was the underutilization of international and local subcontractors to deliver support alongside poor performance of the consulting model. The evaluation team would conclude that this is a drawback of AHADI having to shift from utilizing local subcontractors and/or permanent staff to a consultant model in which quality could not always be assured. The grant mechanism as structured is cumbersome and is apparently slowing down implementation. Given USAID and SUNY oversight requirements, this problem may not be amenable to correction until the entire assistance model is revisited. With regard to restrictions on upper-level contacts via the “communication guidance,” the evaluation team recognizes there may be or may have been sound political reasons for its imposition but would simply note it is unaware of similar restrictions on decentralization programming globally and should be recognized as an impediment to more agile operations. EVALUATION QUESTIONS 3 AND 3(A) How does the scope of AHADI (including number and selection of counties and types of interventions) affect project results? How effective are AHADI’s interventions in building capacity of key partner institutions at the national and county levels and citizen engagement in the devolution process? For this evaluation’s purpose, “scope” (as distinct from “design” or “approach”) refers to the extent of technical interventions and geographic reach undertaken by AHADI. 11 This evaluation question contained two components, each with two sub-questions. The main question regarded AHADI’s “scope” in general, primarily related to the effect of providing assistance to 22 counties, and the efficacy of the interventions that AHADI choose to pursue. Question 3A concerned the effectiveness of AHADI’s chosen interventions on institutional capacity building, and on contributions to citizen engagement. FINDINGS All respondents addressed this line of questioning, with both county officials and national institutions expressing generally positive appraisals of AHADI’s assistance. In interviews, the team found that while respondents were positive, they also expressed useful suggestions as to how the program was constrained and where improvements were needed. While much of the ensuing discussion describes issues with the scope of operations, it is important to recognize that nearly all comments were put forth in a positive light. Thus, 11 This evaluation was not asked to verify whether AHADI counties were doing better than non-AHADI counties. This would be impossible without baseline data. Even with a baseline, reliable comparisons would pose complications. County capacity assessments were not tools that yielded baseline data per se and were not perceived to produce either comparative or reliably actionable data. 15 | AHADI MIDTERM EVALUATION REPORT USAID.GOV AHADI successfully addressed an exceptionally wide range of requests from partners as well as innovating solutions where needed. FEATURES AND CHALLENGES OF AHADI’S SCOPE Geographic scope: Perhaps the most challenging decision emerging from prolonged USAID-GOK negotiations is that AHADI was compelled to implement activities in 22 counties. However, no changes were made to the design, staffing and grants approach to reflect this decision. Interviews revealed that that the 22-county directive came about for understandable reasons. USAID initially intended for AHADI to cover a smaller number of counties. The GOK sought wider coverage because many counties at that point were not receiving adequate assistance. Respondents noted that AHADI’s resources were sizable compared to other donors. Thus, few other donors offered the potential for wide-ranging field presence and technical capacity potentially available through AHADI. In addition, they noted that devolution was new and virtually no institution had a handle on how to assist its implementation at the local level. In addition to serving 22 counties, AHADI provided various levels of assistance for about 10 national government institutions and coordinated with multiple USAID sector offices in pursuit of the cross-sectoral vision. One respondent aptly described the breadth of AHADI’s responsibilities, noting, “The work was not just with counties, but actually with multiple actors representing county assemblies, the county executive and CSOs.” In this sense, AHADI was not just working with 22 counties, but with perhaps 50 or more distinct beneficiaries at the county level. One donor stated categorically: “There were too many counties. That is the main problem. AHADI had too many partners to deal with and this led to the reality that the project was spread too thin.” To put AHADI’s situation in context, the team would note that projects supporting devolution in other countries may have worked with many more local government units (LGUs). When the volume of LGUs is large (in some cases exceeding 100), projects typically take a more “cookie-cutter” approach, using pre￾packaged events delivered by local subcontractors who are trained in common methodologies to address a set of key needs, such as strategic planning, budgeting, public participation techniques, etc. In this model, the task of project staff is to oversee regional and local subcontractor’s delivery of project modules. AHADI’s mandate was to be more hands-on, but it was unable to use local subcontractors as originally intended. So, while the scope expanded, key design elements (especially the reliance on the grants mechanism) were not realigned to reflect the new conditions. Closely related were responses regarding limiting or reducing the scope (here “scope” is understood to embrace both geographic spread and numbers/types of technical interventions). Respondents in at least 20 interviews discussed limitations of the scope or stated that in the future it would be more useful to reduce the scope. In both instances, responses were understood by the evaluators to focus on how to improve the current project and give better definition to the scope in future efforts to assist devolution, rather than critiques of current operations. Examples of responses illustrate the tenor of discussion concerning this issue, such as: ● “Each county has a huge mandate. A small number of counties with a focus on a few sectors which can cross-pollinate would be more ideal – counties learn best from other counties.” ● “There is a need to revisit the number of counties. If AHADI and/or future projects are to have meaningful impact in the form of PFM and other support to county governments, working in 22 counties is unrealistic.” USAID.GOV AHADI MIDTERM EVALUATION REPORT | 16 ● “The scope is too demanding, especially the administrative component of the program. A lot of time is spent on dealing with administrative issues and approvals other than actual implementation of the program.” Initially unfocused “checkbook” approach: Upon mobilization, AHADI’s challenge was how to rapidly mobilize to provide assistance to many clients with many, and sometimes, conflicting demands. Three factors came into play as soon as AHADI began implementation. First, the AHADI staff and Kenyan counterparts had little experience with strategies useful to support devolution. While solid expertise was available from the Local Development Institute and the Urban Institute, neither organization had experienced international personnel permanently on staff (several respondents noted that there was – and to some degree remains – a need for a full-time, in-house expatriate well versed in devolution support strategies). Second, as AHADI engaged beneficiaries, there was a strong tendency to try to respond to any requests for assistance. This has been described by the project team as the “open checkbook” approach. With the late start and absent experience and a defined strategic path, AHADI did not want to delay engagement. So initially (in 2105 and 2016), AHADI’s response was ad hoc, with no clear direction, but a lot of push for quick execution of county￾level activities. As the project matured and gained a clearer picture of needs, AHADI shifted toward having a slower pace with better focus guided by planning frameworks provided at the county level making implementation activities more responsive to county needs and better aligned to county planning and budgeting cycles (which helped make AHADI be more appealing to county governments). Challenges of the technical scope: Comments were mixed concerning the scope of technical assistance. Most perceived AHADI’s technical assistance in a positive light, but this was tempered by a variety of concerns associated with the project’s ability to maintain technical consistency, given an expansive geographic mandate and the need to rely on consultants for technical interventions. Other respondents questioned AHADI’s ability to provide consistent, quality technical assistance. The most common problems noted were: inadequate staffing and consequent reliance on consultants, slow approvals and a grants mechanism that hindered, rather than enabled, agile and quality technical assistance (see discussion under Q1 and Q2). Inadequate staffing: As frequently mentioned, AHAID’s staff was clearly overstretched, primarily as a function of the large scope coupled with the prohibition on using local subcontractors without commensurate design changes. This led to gaps in the team’s ability to maintain consistency and quality. Several county respondents and implementing partners mentioned the inconsistent nature of AHADI technical support. The development of quality CIDPs, for instance, was linked to a given consultant’s personal approach. AHADI leadership of course recognizes this problem: “The AHADI program is seriously understaffed (technical and operational) and this continues to be a problem. This has never been addressed. (see discussion under Q5 on recent budget amendment request). The consultant model has not been effective. Consultants may have a low level of commitment to the program, can be difficult to manage and are spread thin (usually serving multiple counties). County staff are also aware of the transient nature of the consultant engagement.” EFFECTIVENESS OF AHADI’S INTERVENTIONS County capacity tools: AHADI employed two tools intended to help determine which interventions to prioritize in each of the 22 counties. The first tool was the county capacity assessment (CCA), a county preparedness study that AHADI conducted. This later became an annual exercise. The second tool was the County Capacity Index (CCI), designed to measure county capacities in 19 areas (e.g., human resource recruitment to fill devolved positions, CIDPs linked to budget, financial management, public participation, procurement management, etc.). The tool numerically ranks the 19 factors on a 0 to 5 scale of 17 | AHADI MIDTERM EVALUATION REPORT USAID.GOV accomplishment. It is not clear how extensively this tool was actually applied to decisions regarding interventions. The first CCI was finalized in April 2017 (during the electoral campaign) and respondents noted that it was difficult to vet them publicly as anticipated and make it useful to counties given the turnaround time. The evaluation team did a quick desk review of county capacity assessments accomplished in the six counties visited by the team. Each provided some useful information. However, overall the assessment tool was of uncertain value for making actionable technical assistance decisions, especially in contrast to a demand￾driven strategy, which sets local priorities ahead of ranked deficiencies derived from a theoretical model. 12 Public finance/expenditure management (PFM/PEM): AHADI identified support to PFM/PEM in its first work plan and continued to prioritize it in subsequent plans. Respondents overwhelmingly related that the PFM/PEM intervention has been effective, complements other donor efforts in this area and should be expanded/deepened to emphasize a public expenditure management strategy. Several respondents encouraged a focus on the “whole-cycle approach” (as opposed to the narrower focus suggested by public finance management). The PEM methodology takes an integrated approach to the cycle of planning, budgeting, implementation, monitoring and evaluation and, finally, project/expenditure review. Respondents also supported AHADI’s current emphasis on making PEM a key intervention for the “mentoring” approach. It was clear that respondents at both the national and local levels agree that if devolution is to succeed, PEM is at the core and counties need sustained support and a deeper understanding of how to translate budgets into programs. Counties face continued challenges such as procurement oversight, internal audits, aligning CIDPs with budget frameworks and sector development planning, and involving the public. Virtually all stakeholders affirmed AHADI as a reliable partner in these endeavors and hoped the project continues its support. AHADI is now pursuing a strategy to shadow the public expenditure cycle of the national government and build capacity to ensure county governments are proficient, align expenditure with CIDPs (part of program-based budgeting). This strategy is certainly supported by respondents’ commentary. Civic engagement: In the case of citizen engagement, AHADI reported relatively less engagement. There was an intentional decision to put that activity on a lower priority for the first few years while the project tackled the many other technical assistance demands. A few respondents knowledgeable about the topic suggested that in the future, USAID might consider a separate civic engagement project instead of folding the task into the technical assistance activities. By comparison globally, missions with decentralization support agendas have taken both approaches (i.e., integrating public participation into a decentralization project or allocating the task to a separate project). A cross-section of respondents recommended that in addition to public participation events, the project consider assisting ways to boost civic education about devolution and citizen responsibilities in a devolved system. County officials reported that citizens often expect a sitting allowance or other incentive to participate in government outreach activities or that citizens misunderstand the authorities of local government under devolution and pressure them to address issues beyond their scope. County assembly and executive branch officials interviewed expressed strong support for more public participation and outreach activities. Some county assemblies are mounting their own participation events using local talent or assembly staff. Several executive offices set citizen engagement as a high priority and inquired if AHADI could assist in that effort. 12 Three main problems with the county capacity assessment were identified in the team’s interviews: (a) wide variance in how data is collected across counties, so data is not very comparable, (b) the instrument itself may not cover relevant domains to comprehensively measure capacities and (c) data is not be comparable across years because county bureaucrats change and won’t know the situation from previous years, so they will likely just be reporting opinions. From the evaluation team’s comparative experience, numeric rank indexing activities such as the county capacity index are conceptually appealing, but seldom of great use in defining and focusing technical assistance because they tend to be, as in AHADI’s case, rather broad and theoretical. AHADI staff noted that counties need much more detailed types of analysis for county planning. World Bank reportedly also has an instrument/ index but has not shared it. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 18 Three counties indicated that new communication technologies could increase outreach to citizens (e.g., social media, e-governance portals). Several institutional respondents also highlighted the need to share public participation experiences and methodologies across institutions and counties. Beneficiary Satisfaction and Learning: Responses to EQ2 pointed to the readiness of beneficiaries to respond positively and enthusiastically to offers of assistance. This readiness set the stage for AHADI’s interventions. The evaluation team heard examples of beneficiary’s learning new and useful skills around county planning and budgeting. County assembly staff likewise expressed both appreciation and a desire for more assistance in learning legislative management skills. This is an area of SUNY comparative advantage as the organization has recognized experience in legislative strengthening programing in Kenya. County respondents considered the induction program for county executives (partnering with the Kenyan School of Government) effective. AHADI supported KSG with its induction program by developing a manual for policy planning, financial management, public procurement, etiquette and protocol, public participation for county executive committees (CECS), County Secretaries and County Public Service Boards. CONCLUSIONS Based on the team’s interviews, review of relevant documents and application of each member’s expertise, the team reached several broad conclusions. First, the expanded mandate to work in 22 counties was overly ambitious, but politically inescapable. It would clearly have been more advantageous to focus in-depth on fewer counties. That said, AHADI performed well and as it learned, focused on useful interventions such as PEM, CIDP development, and improving assembly and executive skills through a variety of training events. It is remarkable the amount of work that was accomplished given major obstacles posed by the geographic spread, lean staffing model, grants management requirements and complex technical demands arising from so many project beneficiaries. Second, AHADI’s interventions by and large developed relevant capacities at both national and local levels. After an initial period of relatively random assistance to counties, AHADI learned, adapted to and began to focus on building capacities in key areas that were identified as a beneficiary’s priority (e.g., PEM and CIDP development in counties, support for induction training conducted by KSG). Capacity building at the national level was less an outcome of direct training than the byproduct of the project proactively helping various agencies enable devolution. Third, AHADI was not able to aggressively address public participation due to concerns from the government that AHADI first focus on strengthening county administration before moving to eliciting citizen voice. As noted in the findings, it is AHADI’s intent to ramp up the public participation component going forward, and has a newly articulated strategy to guide them. Last, the problems associated with in-house staffing and quality oversight of external consultants may have been better addressed if a budget realignment had occurred to expand the size and technical skills of the AHADI team, accompanied by a renewed effort to engage local subcontractors as originally intended. Both actions come with inherent challenges. In the first case, even an expanded in-house technical team could not have delivered on all demands of the project and would thus have to rely to some extent on external consultants; improved quality control would be the key aim. In the second case, finding sufficient local subcontractors with the requisite skills applicable to devolution may have been difficult. Nonetheless, both actions would help improve delivery of AHADI’s technical assistance. 19 | AHADI MIDTERM EVALUATION REPORT USAID.GOV EVALUATION QUESTION 4 How effective is AHADI’s cross-sectoral approach and funding? In answering this question, the evaluation team assessed whether and how the cross-sectoral approach contributed to effective implementation of devolution by examining what worked well and what didn’t. The team analyzed respondent views to ascertain factors internal to USAID, those internal to AHADI and those arising from contractual relations between USAID and AHADI. The question explores elements that can be built upon, redesigned or discarded. FINDINGS The AHADI statement of work13 introduced the concept of a cross-sector approach and funding. It recognized that although AHADI is primarily a democracy and governance program focused on improving the governance of Kenya’s devolved system, it also has impact on other sectors, especially related to the devolved system of service delivery. It requires cognizance that Kenya’s devolution process affects every sector in which USAID/Kenya works – namely, health, environment and natural resources, agriculture, education and youth development. Cross-sectoral programming is a new endeavor for USAID/KEA. OUTCOMES Donors, implementers and beneficiaries acknowledged that a cross-sectoral approach is sensible because the new governance framework cuts across nearly all sectors; going forward, it will become increasingly relevant to devolved operations as counties work to define their (sector) priorities through CIDPs and annual plans. The evaluation team did note AHADI’s reporting that the cross-sectoral approach has helped focus county assistance on relevant issues, such as tourism in Lamu and agriculture in counties located in the central regions of Kenya. Some respondents indicated that it is easier to give support when providing interventions through sectors as entry points. In Vihiga, for example, it was observed that support was more easily handled when broken down into sectors: economic planning, health, environment, etc. A donor also said: “The cross￾sectoral concept is brilliant. It is the way to go. Governance itself should not be a sector.” The evaluation team notes that in discussing the cross-sectoral approach, beneficiaries did not necessarily understand that this was a deliberate technique used by AHADI to provide support to counties. AHADI coordinated with other USAID implementing partners in the Partnership for Resilience and Economic Growth (PREG), particularly with the Catholic Relief Services (CRS) Resilience for Arid Lands Partnership in Development (RAPID) program on its objective for strengthening governance.14 For instance, in Turkana, AHADI supported the county government in drafting their water and sanitation policy and bill (2016). This included input received from the Ministry of Water and Natural Resources and, because water is a crosscutting issue, also involved the Ministry of Agriculture, Pastoralism and Fisheries and the Ministry of Health. The cross-sectoral approach realized positive outcomes when participants shared interests, were committed to coordination, and resources were well-managed. AHADI’s comparison of the two cross-sector approaches is illustrative. As reported, AHADI achieved solid success from EDY engagement as opposed to the OEG partnership. Cross-sector sharing was accomplished between AHADI and the Tusome project, funded through USAID’s EDY office and operated by Research Triangle Institute (RTI) in cooperation with the Ministry of Education. The EDY office was clear on the focus it desired and gave clear instructions to AHADI. 13 Refer to USAID/Kenya and East Africa: Task Order #AID-615-70-17-00008, Sec. C-Statement of Work, Sub-Section D, and Pg. 5. 14 Ref CRS-RAPID Objective 1. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 20 EDY introduced AHADI to partners at the Ministry of Education and ensured that MOE understood AHADIs role in delivering the support discussed. EDY was very responsive to AHADI; their grant proposals were approved in record time and whenever AHADI encountered a challenge with MOE, EDY took initiative to engage with the ministry and get solutions to problems. Upon receiving approval, EDY allowed AHADI to implement the activity with no interruption and even took time to participate in field activities. The program assisted the MOE to deliver a huge portfolio of training to county boards of management, ensuring that the training was delivered by ministry staff who had been trained by AHADI, and enhanced ministry systems in the process. EDY was highly supportive of AHADI and did not create any obstacles to the implementation process. A quote from one respondent might summarize this cross-sectoral endeavor: “Coordination between partners was very good, as partners knew what they wanted, where to work and what people to work with, thus facilitating speedy implementation as the work was very clear. In this process, AHADI got useful information that helped design interventions. The biggest successes realized in education came in translating national policy to the county level. On the other hand, the OEG office appeared to have little clarity on what they wanted AHADI to focus on, gave conflicting instructions, and frequently changed goals. They did not appear to have well-grounded relationships with parent ministries and expected AHADI to build these relationships. OEG required that AHADI engage multiple partners before implementing activities, although these partners were not required to engage with AHADI before rolling out their own work. CHALLENGES While some respondents identified benefits of the cross-sectoral approach, others found implementation of the approach problematic. The AHADI project was intended accommodate all of USAID/KEA’s target sectors and include funding from several sources.15 In reality, this meant that the contractor had to coordinate with multiple offices, meet a variety of administrative requirements, including contributing to additional targets, managing results monitoring and reporting, and undertaking more complex accounting. Simply put, the addition of cross-sector responsibilities required considerable administrative effort for an already complex pilot project. Considerable staff time was involved as AHADI reached out to other USAID implementing partners to share work plans and identify areas of synergy. The evaluation team noted that respondents regarded this as tending to be one-way initially, as it was not a requirement for partners to share organizational information with AHADI. The sharing of its work plans also proved cumbersome to AHADI. Key respondents reported that this was due to the fact that there was no clarity at the beginning as to how the project would coordinate with other IPs and coordinate with USAID sector offices. AHADI staff and the USAID Health Office reported that there was no close working relationship for health. The USAID Health team was not among the eight sectors that added money to AHADI. Even though Health put no funding into AHADI, an expectation remained that AHADI would coordinate with the Health team. This relationship is instructive of potential conflicts arising from governance versus sector objectives. Health office success is in part measured by the amount of increase in the county health proportional budget. Conversely, AHADI’s objective was to help counties prioritize among all sector needs and assign funds accordingly. Health funding may or may not be a priority in a given county, and AHADI’s responsibility was to help the county to prioritize, even if that meant that other priorities went ahead of health funding. This caused some 15 AHADI Statement of Work AID 615-C-14-00002 21 | AHADI MIDTERM EVALUATION REPORT USAID.GOV tension between health IPs and AHADI, reportedly because AHADI does not have subject matter experts for health and the health team did not see an opportunity for AHADI to add technical value. USAID and AHADI staff described programming as cross-sectoral in funding, but not in approach. There was a lack of a unifying vision, shared purpose and coordination for the cross-sectoral approach within USAID and between USAID and AHADI. Comments from AHADI and USAID sector offices both contended that the purpose of the cross-sectoral approach in support of devolution was not well understood and that without a more senior-level champion inside the Mission (possibly from Programs or Front Office (personnel), it was difficult for the DO1 team alone to keep devolution and cross-sector coordination at the forefront of Mission activities. One USAID respondent said, “The reason for this is because in our minds, devolution was simple. But in reality, it is very complex.” Another Mission officer suggested. “There should be some form of ‘Devolution 101’ because colleagues don’t really understand devolution. They only know their sectors.” Devolution was seen as a sector (governance) and not really a driving philosophy or context as envisioned in the CDCS. These findings align with broader USAID research and learning on cross-sectoral DRG integration, including: the importance of Mission leadership, the need for a commonly understood purpose for integration and the value of a dedicated integration advisor.16 CONCLUSIONS The team came to several broad conclusions. The cross-sectoral approach has had concrete “wins” when stakeholders share interests, are committed to coordination and have managed resources well. That said, the cross-sector approach has also been occasionally problematic for both the AHADI team and USAID sector offices. The approach has been a heavy responsibility for AHADI in particular. AHADI has demonstrated a lot of flexibility and goodwill to make coordination succeed, but the overall approach is still not well understood, embraced or adequately driven by the Mission as a whole. Cross-sector funding is a reality, but effective cross-sectoral programming is not. Overall, the complexities of synchronization and the multiple administrative demands on management have hampered the pace and agility required for fully successful interventions using this strategy. This conclusion does not mean that the cross-sector idea should necessarily be abandoned going forward (especially if another project is considered). However, it does mean that the Mission and the AHADI team will need to seriously review first, the managerial demands imposed on an already stressed implementing team and second, the need to reinvest and better define the devolution vision in relation to practical field level operations. As an agency, USAID has captured important learning about cross-sectoral DRG integration that could be helpful for future USAID/ KEA strategic planning and programming. EVALUATION QUESTION 5 To what extent are SUNY and its subcontractors effectively organized and deployed to implement AHADI? Evaluation Question 5 focuses on stakeholder recommendations for improving the AHADI structure. These recommendations fall into three broad categories: personnel structure, implementation mechanism (grants), 16 DRG Cross-Sectoral Programming Support Project: DRG Integration Case Study Synthesis, March 2016. Prepared by Social Impact Inc. and RTI International. https://usaidlearninglab.org/sites/default/files/resource/files/case_study_synthesis_report_- _light_version_7.14.16.pdf USAID.GOV AHADI MIDTERM EVALUATION REPORT | 22 and communications. These recommendations do not necessarily align with the evaluation team recommendations, which are presented separately at the conclusion of this document. FINDINGS Personnel structure: Project reporting indicates that AHADI is understaffed. AHADI has worked to augment its technical, operations, knowledge management and monitoring and evaluation capacity,17 but gaps remain. In January 2018, AHADI submitted a budget modification request to USAID to address some of its staffing needs. Its proposed organizational chart, which is annexed to the pending budget modification, intends to add staff and shows needed positions that are vacant in all departments and at all levels.18 Filling these positions will be especially important if a no-cost extension is granted. All respondent groups interviewed for this evaluation recommended updates to the personnel structure. Enhance Ground Presence: AHADI currently has a ground presence in 12 of 22 counties where the project operates. The 12 include seven county liaison officers, each responsible for managing one county, and five regional program officers who manage the remaining 15 counties. All 12 individuals are consultants. All respondent groups, including numerous county-level beneficiaries, recommended that AHADI enhance its ground presence in the counties. They noted that this would facilitate support to the counties, enhance project responsiveness and information flow, allow for necessary monitoring and evaluation, increase results, and ease coordination with other implementers. Donors and county beneficiaries recommended that AHADI deploy technical advisors to counties (akin to the U.N. model) but recognized that this may not be feasible. Both USAID and AHADI noted that enhancing county ground presence is a significant cost consideration but vital for effective programming. AHADI does not plan to expand its ground presence to all 22 counties for the remainder of the current project. One interim strategy being used is for regionally-based personnel to support county clusters and link them to the project’s Monitoring, Evaluation and Learning (MEL) agenda to assess impact and results of AHADI initiatives.19 Increase full-time staff / decrease consultants: USAID, other donors, AHADI staff and project beneficiaries all emphasized the need for AHADI to increase full-time staff and technical experts and decrease the reliance on consultants. They described this as important for the quality and consistency of capacity building support, and beneficiary rapport and trust. Beneficiaries and donors noted the value of imbedding carefully vetted long￾term experts (as opposed to shorter-term consultants) in partner institutions to serve as mentors. They added that this is increasingly important as devolution matures and partner needs become more nuanced. AHADI has plans to increase mentoring efforts. As outlined in the Year 4 Annual Report, AHADI plans to scale up in-county mentoring for the remainder of the project and is seeking a full complement of technical experts for Public Expenditure Management, sector issues and quality control.20 One key staff position to be filled is a full-time senior expatriate decentralization expert. AHADI noted that a resident expatriate decentralization expert with comparative international expertise would help promote 17 See, e.g., AHADI Annual Work Plan Year 3, p. 11. 18 This includes new positions that AHADI has requested to increase its staffing levels: a second governance and devolution lead for one of the county clusters, some additional regional and county program officers, additional program assistants, a new operations director and staff for human resources and administration, procurement and logistics, finance, grants, monitoring and evaluation and communications. 19 AHADI Annual Progress Report October 2016-September 2017, p 12. 20 AHADI Annual Report Year 4, p10 23 | AHADI MIDTERM EVALUATION REPORT USAID.GOV project consistency, learning and adaptation. AHADI subcontractor Local Development International (LDI) currently provides this comparative expertise via key experts, and one local staff person, who is an internationally experienced Kenyan. While LDI’s international experts are highly valued, they work intermittently (one spends six months of the year in Kenya) and are not residents of Kenya. This evaluation did not include a focused review of the efficiencies of project administration. IMPLEMENTATION MECHANISM Reduce use of grants mechanism: Grants are the primary implementation mechanism for AHADI. The project budget includes approximately $156.55 million for grants. The purpose of the grant fund is “to ensure that local organizations and governments have access to funds to carry out devolution-related efforts, and that the contractor has the flexibility to respond accordingly to needs as these arise.”21 Initially, the process of negotiating and securing approvals for grants under contract (GUC) took four to six months.22 In Year 2, a new grants mechanism (Government of Kenya Initiatives, or GOKI) was introduced, enabling AHADI to work with government entities for the first time and streamline negotiations with counties (counties essentially agree to the list of activities, but do not need to sign a traditional grant agreement.) The evaluation team recognizes GOKI as a creative and effective solution for supporting Kenyan initiatives and institutions. However, GUC and GOKI both require significant administrative time and management resources. The administrative and accounting processes are the same and both require USAID approvals for changes. AHADI noted the strain that the grants mechanism has been on the AHADI procurement and grants teams as well as the program teams (who review grant applications). Some respondents reported that the involvement of technical staff in administrative work has diverted attention from their important technical work. Grants administration also requires considerable SUNY home office support. The organizational chart includes a senior financial manager (75 percent), project manager (60 percent), senior associate for contract management (50 percent) and a procurement specialist (35 percent).23 Also included are positions for senior technical advisor, a reporting specialist and an M&E advisor. The January 2018 budget modification request reportedly includes more funds allocated for directly funded activities. Directly funded activities involve concurrence from the contracting officer’s representative (COR) but not the lengthy approval process required for grants. COMMUNICATIONS/ VISIBILITY Allow AHADI greater engagement with governmental and quasi-governmental entities: As described under EQs 1 and 2, AHADI’s access to governmental and quasi-governmental entities is restricted. In 2015, USAID issued communication guidance limiting AHADI’s contact with senior leadership of national and county partner entities, including cabinet secretaries, principal secretaries, committee and commission chairs and governors. The evaluation team understands that the guidance was issued due to the political and cross-sectoral nature of the project, and possibly with consideration to heightened tensions in U.S.-Kenya relations and project start-up challenges. The restriction extends to AHADI’s participation in the Devolution Sector Working Group (DSWG), the national body chaired by the Ministry of Devolution and Planning that monitors and supports implementation of the Medium-Term Plan of Vision 2030, Kenya’s national long-term 21 AHADI Scope of Work, p45 22 AHADI Annual Report Year 2, p9 23 AHADI organizational chart as annexed to proposed budget modification, January 2018 USAID.GOV AHADI MIDTERM EVALUATION REPORT | 24 development policy.24 Both the Medium-Term Plan 2 (2013–2017) and the Medium-Term Plan 3 (2018-2022) prioritize devolution.25 The Devolution Sector Working Group includes a high-level steering committee and a technical working group. AHADI beneficiaries and non-USAID donors expressed a strong desire for AHADI to participate in DSWG technical discussions to coordinate and communicate their technical work with other programs. For instance, one respondent noted: “Partners in the donor community don’t benefit from the expertise within USAID and AHADI. Having someone attend from USAID did not necessarily improve the quality of technical discussions. The AHADI team is very capable, knowledgeable and respected. USAID should have given AHADI more latitude.” Similarly, MODP reported that AHADI’s absence from DSWG means that their mandate has not been well-communicated or understood. Another interviewee suggested that USAID should treat AHADI as “ambassadors” on technical matters for the project, suggesting that the Mission should encourage a more visible technical role for AHADI, rather than requesting that the implementing partner defer to the Mission to communicate technical matters. AHADI indicated that the restrictions have affected project visibility, mentioning, for example, that governors are often unaware of AHADI and USAID support. Interviewees emphasized that support from leadership of partner institutions is important for project momentum and success. It was not evident to the evaluation team that USAID has the time to engage the senior leadership of partner entities or inform them of project progress. Increase use of subcontractors and information sharing: AHADI has proposed budget increases for LDI and Urban Institute (UI) to be able to draw more extensively on their expertise for the remainder of the current project. Project reporting shows that LDI has provided vital technical leadership for key initiatives such as PEM (which LDI helped to conceptualize and launch), monitoring and evaluation, sector policy development and gender-based budgeting guidelines; and that UI is helping to integrate the knowledge and learning agenda into AHADI and develop substantive knowledge products on devolution.26 In addition to capitalizing on the expertise of subcontractors, respondents recommended increased information sharing and communication with stakeholders. For instance, beneficiaries requested more transparency about the type of support the project can or cannot provide (e.g. participant allowances for attending AHADI activities) and clearer communication of the terms of engagement (e.g. processing times and reporting requirements for grants and clear explanations in case of delays or changes to the support). One interviewee stated: “The whole bureaucracy made us feel we had to really beg, so we couldn’t complete everything. Changes in AHADI [staff] meant we had to start over from the beginning. There was little to no clear communication to explain why there were delays [in the support]. [AHADI] would just say they were working on it and it just died.” CONCLUSIONS Despite the praise for AHADI personnel identified in earlier sections of this report, the current personnel structure limits the ability of the project to effectively deliver on its mandate. This includes a lack of full-time expatriate staff with devolution experience, staff (including a senior decentralization expert with comparative international expertise), lack of full-time staff (a budget modification to increase staff is pending), an over￾reliance on short-term consultants and insufficient ground presence in counties. A successful mentoring approach will require long-term experts who can develop relationships and trust with beneficiaries. 24 http://www.vision2030.go.ke/about-vision-2030/ 25 http://www.devolutionplanning.go.ke/?publication=second-medium-term-plan-2013-2017; http://www.mtp3.go.ke/ 26 See, e.g. AHADI Annual Report Year 3, pp. 7-8 and p. 40 and AHADI Annual Report Year 4, p. 10. 25 | AHADI MIDTERM EVALUATION REPORT USAID.GOV The grants mechanism has been unwieldy from an administrative perspective. The Government of Kenya Initiative (GOKI) was a useful, creative solution for providing grants to government entities, but more efficient, flexible implementation mechanisms are needed. Eliminating or reducing the use of the grants mechanism could enable more agile and responsive programming. Restrictions on AHADI access to partner entities has hindered project implementation, profile and quality of technical discussions and shared learning on devolution. It is also not clear that the restrictions have served the intended purpose. Project beneficiaries and other implementers and donors working in the devolution space desire improved communication and greater information sharing from AHADI. RECOMMENDATIONS Globally, decentralized governance has been extensively supported by USAID for good reason. It opens political processes to greater public participation and transparent oversight, and in the long term can accelerate economic performance if properly implemented. Changes attributable to decentralization strategies are achieved only over the very long term; it is an evolving process rather than a concise target. As noted in this report, Kenya’s devolution policy is impressively complex and warrants sustained support. The following recommendations may move the devolution process forward. The evaluation team’s recommendations are: ● AHADI has a broad scope for technical interventions, many of which are now positioned to more meaningfully contribute to Kenya’s devolution objectives if given time to mature. The evaluation team thus strongly recommends that the AHADI project be given a no-cost extension to give it ample time to solidify successful interventions (especially public expenditure management). ● Related to the above, the project currently benefits from the flexibility to bring in different types of expatriate STTA expertise that can contribute comparative experience and practical strategies. It may also endeavor to incorporate long-term, on-staff expertise in an advisory capacity if possible within current budget realities. Such expertise might focus on public participation methods as a structured element of devolution or a local government finance expert (emphasizing experience in interventions to address issues arising from the Auditor General’s reports). Such expertise is widely available due to the extent of USAID’s history of supporting decentralization around the world. Kenya’s efforts are impressive but could benefit from the practical experience gained across the globe. The evaluation team recognizes that this recommendation may not be feasible in the time remaining for the current project. Experienced expertise should similarly be included in a future project. ● In addition to providing a no-cost extension, USAID should consider a future devolution support project to follow AHADI. It will be important to design it with more focus, based on learning from the AHADI project, which has already accumulated considerable knowledge regarding best practices and county-level needs. Overloading the project with too many expectations and diverse objectives should be avoided. If an extension is granted, it is recommended to give priority to continued assistance around PEM, management support to both executive and legislative branches and public participation activities. In the latter case, designing, training and integrating solid public participation methods into county operations should be paramount in that bringing the public voice into the governing process is at the center of Kenya’s devolution vision. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 26 ● The evaluation team encourages AHADI and USAID/ KEA to urgently seek solutions to structural impediments to the project, principally the heavy reliance on short-term consultants and the use of grants as the primary implementation mechanism. These are important design and costs issues. ● AHADI should increase efforts to help national government entities more aggressively develop long￾term, permanent staff and capabilities to support county government through training, mentoring and sharing of best practices. AHADI will be especially helpful to the degree that it can assist national government to shift to an enabling and mentoring role (as opposed to a directive role). ● It will take time to effectively scale up mentorship efforts for the project, as indicated in the evaluation findings. The current work plan year (2017–2018) is an opportunity to test the viability and success of embedding mentors in counties. AHADI and USAID/KEA should aim to capture key learnings from this effort (including value for money) to inform future devolution support efforts. As part of good adaptive management, the project may scale up sharing among partners, showcasing best practices and or practical mentorship models that have been proven to work well under the project. A more assertive sharing program should be advocated, especially as inter-governmental sharing of successes driven by local partners (as opposed to expert-driven training) has proven to be effective in other decentralization support projects. ● It is not recommended that AHADI continue to use or pursue modifications to the County Capacity Index tool. It does not appear to have added significant value and, as noted in the Findings section, is not entirely compatible with a demand-driven approach, which is more flexible, opportunistic and less prescriptive. ● The project delivery mechanism needs to be thought through when undertaking future project design. Eliminating or reducing use of the grants mechanism could enable more agile and responsive programming. It will be important to identify viable alternatives that consider available resources and prioritize the quality of delivery. Related to this, USAID/ KEA could consider working with AHADI on a review to determine the administrative time, costs and administrative encumbrances associated with grant administration, to verify that this is the mechanism that should be retained for the future. ● The evaluation team recommends that USAID/ KEA reassess the communication guidance issued to AHADI in 2015 to determine whether it is still needed and serves the intended purpose. In the short term, removing the guidance could facilitate implementation, strengthen key relationships and contribute shared learning among key players in the devolution space. Related to this recommendation, USAID should allow AHADI leadership to regularly attend DSWG meetings to better share project learning. ● This evaluation did not include a focused review of project administration. This could be a valuable supplemental exercise undertaken internally to examine some key areas. We do not suggest an elaborate undertaking. The project is already proposing a new organizational structure (per the proposed budget modification request). It may also be useful to examine ways to reduce the time involved to process work plans, issue and close out grants and process grant modifications, as these have considerable impact on agility. (This review would require active participation by USAID.) Issues around staff retention should also be explored to determine if there are ways the project could retain skilled staff. SUNY will need to begin a closeout process by July 1, 2018. Thus, the administrative review suggested here is desirable, but the decision to extend should not depend on completion of this exercise. 27 | AHADI MIDTERM EVALUATION REPORT USAID.GOV ● Devolution was and will remain a challenging strategy for Kenya. As USAID’s CDCS states: “business as usual is not an option.” In view of this, and the foundational base that is needed for devolution to work, AHADI should, to the degree possible, focus and deepen key interventions supporting more efficient governance at the county government level (i.e., PEM, legislative strengthening, planning, etc.). Cross-sectoral involvements are less important at this juncture. The evaluation team recognizes that funding levels of the current contract incorporate cross-sectoral funding, so some continuance of cross-sector work is recognized as obligatory.27 ● In terms of scope, respondents identified key areas that AHADI (to the extent possible in the remaining project period) and future USAID devolution support programming should consider. The evaluation recommends attention to three areas: First, more support to county assemblies so that both legislative actions and oversight can be improved. In our county interviews, we found considerable demand for this type of assistance. Second, there should be more support to improve inter-governmental relations. This view applied to assembly-executive relationships, which are sometimes contentious and to local-national relationships, where the issue is how to assure sustainability by building the capacity of relevant national institutions to maintain and perfect their assistance to the devolution process (e.g., better training, improving internal staff capabilities, collaborating with county leadership to develop more effective policies, etc.). Third, greater attention to increasing civic engagement (in terms of both public participation and civic education). ● Experience from other decentralization projects suggests that a solid, sustainable strategy to integrate public participation into governance programming is advisable. It is also a very demanding and time-consuming task. As designed and staffed, AHADI was probably wise to limit its citizen engagement efforts. The project now has a new citizen engagement strategy which should be tested in the time remaining, with a view to capturing best practices. ● There is need to clarify the cross-sectoral strategy, as DO1 can work only in well-circumscribed areas. The evaluation team would recommend that USAID also make a renewed effort to communicate its devolution support vision to the entire Mission and perhaps assign a senior staff person to champion devolution and governance strategy. The Mission should draw on USAID-wide learning about cross-sectoral DRG integration, including the experiences and lessons contained in the 2016 DRG Integration Case Study Synthesis report. 27 The evaluation team would note that it was beyond the scope of this evaluation to determine if the cross-sectoral approach yielded better results than an approach that was singular. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 28 ANNEX 1. EVALUATION METHODS AND LIMITATIONS The AHADI mid-term evaluation was conducted in two phases (Phase 1 was October 30-November 23, 2017 and Phase 2 December 8, 2017- April 23, 2018). USAID/KEA approved the final evaluation design and instruments on December 15, 2017. Given the nature of the evaluation questions and the stakeholders involved, qualitative data collection and analysis methods were used. Qualitative methods are appropriate because they facilitate the acquisition of rich detail about the project and triangulation across multiple data sources to inform the report conclusions and recommendations. The evaluation team consisted of team leader Katherine Vittum, co-team leader Judy Oduma, decentralization specialist Kenneth Ellison and senior researcher Kenneth Odary. KSP provided management support for the evaluation. Short-term consultants Susan Opiyo, Edgar Omungu and Elizabeth Kamwaro provided vital note￾taking, scheduling and administrative support. DATA COLLECTION The evaluation team gathered data from both primary and secondary sources. The team refined the data collection methodology in collaboration with USAID/KEA as part of Inception Report development. Below is a summary of data collection methods and sources used. For the desk review, the evaluation team reviewed all activity documentation provided by USAID and AHADI, as well as materials from evaluation team members’ own libraries and additional resources obtained during the evaluation. Annex 6 contains a list of the key documents reviewed. Findings from the desk review informed the strategy and lines of inquiry for primary data collection (interviews). The list of stakeholders to be interviewed was finalized in consultation with USAID/KEA and SUNY. The list was revised to include the most appropriate and relevant informants at the national and county levels, based on the stakeholders who have knowledge of AHADI. Primary data collection was conducted November 17-23 and December 18-20, 2017 and January 4-19, 2018. The sample included six of the 22 counties where AHADI operates: Bomet, Mombasa, Turkana, Vihiga, Tharaka-Nithi, and Nairobi. The evaluation team recorded 55 interviews with 122 respondents (67 men and 55 women). There were six respondent groups: 1. USAID (Development Objectives 1, 2 and 3) 2. Other Donors 3. Implementing Partners (AHADI and USAID Development Objective 2 and Development Objective 3 Implementing Partners) 4. National Government/ Institutions 5. County Government 6. Civil Society Organizations The 55 interviews included: six with USAID, three with other donors, 16 with implementing partners (14 with AHADI and two with other USAID implementers), ten with national government/institutions, 18 with county government and two with civil society. 29 | AHADI MIDTERM EVALUATION REPORT USAID.GOV The four-member evaluation team conducted the initial interviews jointly. Thereafter, the team split into two sub-teams (each with one Kenyan and one expat) for efficiency. DATA ANALYSIS The inception report contains details on the data sources and methods used answer each evaluation question. Table 4 of the annex describes how primary and secondary data were used to answer each evaluation question. In summary, Evaluation Question 1 was largely answered through interviews with AHADI, USAID and the Ministry of Devolution and Planning;; answers to Evaluation Question 2 and Evaluation Question 3 drew on project reporting documents and interviews with all respondent groups; Evaluation Question 4 answers are based on key project design documents and interviews with donors, implementers and select county-level beneficiaries; and Evaluation Question 5 answers are based on an analysis of project documents and interviews with AHADI, donors and national and county level beneficiaries. The data analysis methods included content analysis, comparison and mixed-methods data integration. Preliminary data analysis began during data collection to ensure that the evaluation team was capturing the information necessary to fully address the evaluation questions. The team prepared detailed notes for each interview, using a standard reporting format. The interview notes formed the basis for data analysis. The team held regular debriefings to identify emerging themes from the data. Following data collection, the team developed written definitions for each theme and these definitions were used to code each set of interview notes. For quality control, approximately one-third of the notes were independently coded by two evaluation team members and then compared for consensus. As needed, the team leader weighed in to make final coding decisions and to refine the theme definitions. Each team member was then responsible for independently coding the remainder of the notes for a particular evaluation question (or maximum two evaluation questions). With coding completed, the team analyzed the interview data using content and pattern analysis to compare the perspectives of various respondent groups. The team developed a tally sheet (frequency table) to capture the themes that emerged in each interview and underlying evidence from the interview notes. The team looked for areas of convergence and investigated divergent and unique perspectives to assess whether they merited consideration in the analysis. The team then integrated information from the document review and compared and triangulated evidence to identify the key findings for each evaluation question. A USAID/KEA representative participated in a data analysis session with the team. The team walked USAID through each of the steps undertaken for data analysis and the representative coded a portion of interview notes with the evaluation team. The evaluation team notes that perspectives of various respondent groups cannot be weighted equally, considering the differing roles and levels of engagement that respondents had with AHADI. The team analyzed the primary data considering each source’s experience with AHADI and the interview questions they answered. As such, the report focuses on substantive significance rather than on numeric data. On January 30, a validation workshop was held with USAID and SUNY. The evaluation team presented the key findings from the initial data analysis and discussed the findings with participants. SUNY and USAID provided valuable feedback, which is reflected in the final evaluation report. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 30 LIMITATIONS As with any evaluation, there were limitations. Three weeks after the evaluation began and field data collection was underway, the evaluation was suspended for a period of two weeks and redesigned. The new evaluation design introduced a central question on the relevance of the AHADI project design and overall had a clearer focus on learning and utilization, which the evaluation team endorses. Changes to the evaluation timeline meant that the evaluation team was not able to conduct new interviews in the two counties where data collection had already been completed (Mombasa and Bomet), but the team does not believe that this impacted the analysis that was possible. The timeline changes also meant that the evaluation team leader was only available through the beginning of the report writing stage. The other team members, under the guidance of the decentralization specialist, completed the evaluation report. One evaluation team member was replaced. KSP was able to hire a replacement with sectoral knowledge and USAID/KEA evaluation experience. There were also changes to KSP management and support staff for the evaluation. KSP supported the team with data collection when needed. The evaluation team experienced minimal scheduling challenges. AHADI, USAID and the co-team leader for the evaluation were instrumental in securing interviews. The evaluation team also extended data collection to accommodate respondent availability. The team was unable to meet with the Society of Clerks at the Table (SOCATT), the Centre for Parliamentary Studies and Training (CPST), the USAID Development Objective 3 Team Lead, or the USAID Implementing Partner for Education and Youth in Mombasa. Additionally, the team was only able to meet with governors in two of the six counties visited. This challenge was anticipated. The team pursued interviews with governors as a matter of protocol but did not expect them to offer many insights on the project. The evaluation team made every effort to schedule these interviews, but several of the governors were not in their counties at the time of the evaluation team visits. The team does not feel that this negatively impacted the quality of the data collected. For this evaluation, the most effective approach for combating limitations was to use multiple layers of triangulation. By combining information from multiple sources, the unavailability of some target respondents or the existence of any specific piece of biased respondent data did not unduly skew the analysis. GENDER The evaluation design reflected sex-disaggregated and gender-specific data considerations (see Annex 2, Inceptions Report, Table 5 which was reviewed by the USAID/KEA gender advisor.) The evaluation did not identify differences in results and benefits of the project along gender lines. However, none of the evaluation questions or interview guides specifically explored gender. Also, the majority of AHADI beneficiaries interviewed for the evaluation were male. Respondents were selected based on institution, position and knowledge of the AHADI project. Data trends presented in the report generally reflect the institutional affiliation of the respondent rather than their gender. 31 | AHADI MIDTERM EVALUATION REPORT USAID.GOV ANNEX 2. GENDER CONSIDERATIONS AHADI in its Year 3 Annual Report incorporate in all its activities gender sensitive and inclusive implementation and monitoring. In order to ensure effective inclusivity, AHADI also developed guidelines, procedures and protocols to guide programming and an institutional policy to guide operational and administrative compliance on gender issues. These tools: An assessment report on AHADI programming’s gender sensitivity and inclusivity, AHADI’S institutional policy for gender inclusivity and gender strategy, a county model gender strategy & a template and guideline for mainstreaming gender County operations, guided AHADI for proper interpretation of gender mainstreaming. In the FY 2017 work plan, for instance, AHADI undertook specified activities in a bid to fully comply with the USAID gender policy including supporting target counties to adopt and implement the proposed county model gender strategy; Supporting target counties in mainstreaming gender in development planning such as, revision of CIDPs and sectoral planning; and Training-of- Trainers targeting AHADI consultants & trainers and county leadership. Additionally, AHADI (utilizing expertise from sub-contractor LDI) worked with UN Women, Treasury, and Parliament to develop county gender-based planning and budgeting guidelines. The program supported civic education activities that empower women to participate in governance, and to build the capacity of county officials to incorporate women’s empowerment into policy development and planning processes (CIDPs and sector plans). In terms of reporting, AHADI also put in place systems and processes for data collection, analysis and submission of gender-aggregated data from all programme activities. The AR for 2016/17 indicates that AHADI supported capacity development and public participation forums which on average saw 31% participation rates among women, attributing this to county and national institutions recruiting women to substantive positions and management as well as AHADI working closely with county and national institutions to ensure that women are included in capacity development initiatives. USAID.GOV AHADI MIDTERM EVALUATION REPORT | 32 ANNEX 3: LIST OF DOCUMENTS REVIEWED USAID DOCUMENTS USAID/Kenya Country Cooperative Development Plan, 2014-2018 USAID/Kenya Cross-Sectoral Integration Case Study Synthesis, March 2016 AHADI DOCUMENTS AHADI Annual Reports (APR), Years 1-3 • Year 1 APR, October 2014 • Year 2 APR, September 30, 2014 • Year 3 APR, October 30, 2016 AHADI Annual Progress Report, 10/2016 – 09/2017 AHADI Quarterly Reports (sampled from years 2 and 3) AHADI Annual Work Plans (AWP), Years 2015, 2016, 2017 • AWP 1 10/1/2014 – 12/31/2015 • AWP 2 10/01/2015 – 09/31/2016 • AWP 3 10/01/2016 – 09/30/2017 AHADI Monitoring and Evaluation Plan (revised), 01/07/2015 AHADI Citizen Engagement Strategy and Interventions for the Next Year (2018) DO1 Consultation Reports, August 2017 • National Level • County Level • CSOs • Sector Notes AHADI Grants Manual (revised), May 2017 AHADI Government of Kenya Grants Initiative, Implementation Manual, May 2016 AHADI List of Grants 2015-2017 (sampled) County Capacity Assessments for Bomet, Mombasa, Nairobi, Tharaka-Nithi, Turkana, Vihiga Final Summary Report “Devolution and County Capacity in Kenya, An Assessment of 22 Counties, Institute of Development Studies, March 2017 Management Sciences International (MSI) Technical Proposal (revised) for AHADI Mid-Term Evaluation, September 21, 2017 33 | AHADI MIDTERM EVALUATION REPORT USAID.GOV GOVERNMENT OF KENYA DOCUMENTS Constitution of Kenya, 2010 County Governments Act (No. 17), 2012 GoK County Governments Act (No. 17 of 2012), Revised Edition 2016 Government of Kenya 2030 Vision (Popular Version), 2017 Government of Kenya County Public Participation Guidelines (MoDP), January 2016 Government of Kenya Guide to the Legislative Process, Kenya Law Reform Council, Government of Kenya National Capacity Building Framework & Medium-Term Interventions, January 2016 Government of Kenya National Capacity Building Framework and Medium-Term Interventions, 12/15 Government of Kenya Policy on Devolved System of Governance (MoDP), Oct. 2016 Government of Kenya Second Medium-Term Plan, 2013-2017 Guide to the Legislative Process, KLRC, 2015 Handbook on County Planning, Budgeting and Social Accountability, Institute of Economic Affairs, 2014 Reports of Devolution Conferences, Years 2014, 2015 and 2016 COUNCIL OF GOVERNORS DOCUMENTS Council of Governors Strategic Plans • CoG Strategic Plan, 2014-2017 • CoG Strategic Plan, 2017-2020 CoG Reports from Devolution Conferences • 1st Devolution Conference, April 2014 • 2nd Devolution Conference, April 2015 3rd Devolution