DECEMBER 2015 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by Katherine Vittum, John Lis, Martin Ocholla, Carolyne Njihia and Management Systems International. Parliamentary Strengthening Program: Final Performance Evaluation USAID/Kenya Office of Democracy, Governance and Conflict Parliamentary Strengthening Program: Final Performance Evaluation USAID/Kenya Office of Democracy, Governance and Conflict December 2015 Contracted under Task Order # AID-615-TO-15-00017 Final Performance Evaluation for USAID/PSP: Award Number: Phase 1 and 2: Indefinite Quantity Contract (IQC) #DFD-I￾00-04-04-00128-00; Phase 3: CA. No AID-623-A-10-00019 Cover Photo Promoting peaceful 2013 elections through concerts. The national flag is displayed during the singing of the national anthem at the opening of the Mombasa Jomo Kenyatta beach peace concert, which was organized by Transparency International. Well-known artists entertained thousands of fans while promoting peace during the 2013 general elections. They also advised their fans to vote wisely by electing leaders that would serve them with integrity. Transparency International Kenya partners with USAID/Kenya to promote peace during and after the coming 2013 general elections. Photo credit: USAID/Siegfried Modoloa DISCLAIMER This study is made possible by the support of the American people through USAID. The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. CONTENTS Acknowledgments.............................................................................................................iii Acronyms...........................................................................................................................iv Glossary of Terms .............................................................................................................vi Executive Summary.......................................................................................................... 1 Evaluation Purpose and Evaluation Questions.............................................................................. 1 Project Background............................................................................................................................. 1 Evaluation Methods and Limitations................................................................................................ 1 Findings and Conclusions................................................................................................................... 1 Recommendations............................................................................................................................... 4 Evaluation Purpose & Evaluation Questions .................................................................. 5 Evaluation Purpose .............................................................................................................................. 5 Evaluation Questions.......................................................................................................................... 5 Activity Background.......................................................................................................... 6 Development Problem ....................................................................................................................... 6 USAID Activity Strategy and Tasks................................................................................................. 6 Evaluation Methods & Limitations .................................................................................. 7 Data Collection Methods and Sources........................................................................................... 7 Data Analysis Methods....................................................................................................................... 8 Limitations ............................................................................................................................................. 8 Findings & Conclusions ..................................................................................................... 9 Question 2............................................................................................................................................. 9 Question 3........................................................................................................................................... 15 Question 1........................................................................................................................................... 22 Question 4........................................................................................................................................... 28 Question 5........................................................................................................................................... 33 Recommendations .......................................................................................................... 35 Annex 1. Evaluation Statement of Work ..................................................................... 36 2.1 Evaluation Purpose ..................................................................................................................... 41 2.2 Audience and Intended Use ..................................................................................................... 41 2.3 Evaluation Questions................................................................................................................. 41 3. Evaluation Design and Methodology ......................................................................... 43 3.1 Evaluation Design........................................................................................................................ 43 3.2 Data Collection and Analysis Methods.................................................................................. 43 3.3 Gender.......................................................................................................................................... 52 3.4 Methodological Strengths and Limitations............................................................................ 53 4. Evaluation Products .................................................................................................... 54 4.1 Deliverables.................................................................................................................................. 54 4.2 Reporting Guidelines.................................................................................................................. 56 5. Team Composition ..................................................................................................... 57 6. Evaluation Management ............................................................................................. 59 6.1 Logistics......................................................................................................................................... 59 6.2 Scheduling ..................................................................................................................................... 59 6.3 Budget............................................................................................................................................ 59 Annex 2. Final Workplan................................................................................................ 60 Annex 3. Data Collection Instruments ......................................................................... 62 Annex 4. Data Collection Limitations: Interviews....................................................... 68 Annex 5. Analytical Approach to Answering the Evaluation Questions ................... 69 Annex 6. Additional Findings on Women..................................................................... 70 Annex 7. List of Interviewees......................................................................................... 72 Annex 8. List of Documents Reviewed ......................................................................... 73 Annex 9. Disclosure of Any Conflicts of Interest ......................................................... 75 Annex 10: Abridged Team Bios..................................................................................... 78 Parliamentary Strengthening Program: Final Performance Evaluation iii ACKNOWLEDGMENTS The authors wish to thank the current and former members of Parliament (MP) and parliamentary staff; State University of New York (SUNY), USAID and Department for International Development (DFID) of the United Kingdom officials; and outside experts who contributed their time to meet with the evaluation team. The authors also wish to thank Zephaniah Aura, Sheila Karani, Betty Mugo and Jaynnie Mulle of USAID/Kenya for their cooperation with this evaluation. Parliamentary Strengthening Program: Final Performance Evaluation iv ACRONYMS AHADI Agile and Harmonized Assistance for Devolved Institutions ADS Automated Directives System AOR Agreement Officer’s Representative CAF County Assemblies Forum CDF Constituencies Development Fund CDCS Country Development Cooperation Strategy COP Chief of Party COR Contracting Officer’s Representative CPAIC County Public Accounts and Investments Committee CPST Center for Parliamentary Studies and Training CSO Civil Society Organization DFID Department for International Development DGC Office of Democracy, Governance and Conflict DRG Office of Democracy, Human Rights and Governance DO Development Objective FIDA International Federation of Women Lawyers (Kenya) FSN Foreign Service Nationals FY Fiscal Year GD Group Discussion ICJ International Commission of Jurists IQC Indefinite Quantity Contract IT Information Technology KEWOPA Kenya Women Parliamentarians Association KII Key Informant Interview KNA Kenya National Assembly KYPA Kenya Young Parliamentarians Association MCA Member of the County Assembly MP Member of Parliament MSI Management Systems International NA National Assembly PAC Public Accounts Committee Parliamentary Strengthening Program: Final Performance Evaluation v PBO Parliamentary Budget Office PIC Public Investments Committee PIN Parliamentary Initiatives Network PMP Performance Monitoring Plan PSC Parliamentary Service Commission PSP Parliamentary Strengthening Program SOW Statement of Work SUNY/CID State University of New York Center for International Development SUNY State University of New York TPM Team Planning Meeting USAID United States Agency for International Development Parliamentary Strengthening Program: Final Performance Evaluation vi GLOSSARY OF TERMS A performance evaluation focuses on the following descriptive and normative questions: • What a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); • How it is being implemented; • How it is perceived and valued; • Whether expected results are occurring; and • Other questions that are pertinent to program design, management and operational decision￾making. Performance evaluations often incorporate before-and-after comparisons, but generally lack a rigorously defined counterfactual. A theory of change describes the hypotheses through which activities will be transformed into results.1 It is analogous to a USAID development hypothesis or project hypothesis.2 The development hypothesis identifies causal linkages between USAID actions and the intended strategic objective (highest-level result). External validity is the degree to which findings, conclusions and recommendations produced by an evaluation are applicable to other settings and contexts. The results framework is a graphical representation of the development hypothesis and includes the country development cooperation strategy’s (CDCS) goal, development objectives (DO), intermediate results (IR), sub-IRs and performance indicators. It includes any critical assumptions that must hold for the development hypothesis to lead to the relevant outcome. Typically, it is presented in graphic form and supplemented by narrative.3 Findings are empirical facts collected during the evaluation. Conclusions are interpretations and judgments based on the findings. Recommendations are proposed actions for management and are based on the conclusions. Lessons learned are the conclusions extracted from reviewing a development program or activity by participants, managers, customers or evaluators with implications for effectively addressing similar issues/problems in another setting. 1 USAID Project Starter, http://usaidprojectstarter.org/content/learning-pathway-1-integrating-evaluation-lessons-design. 2 USAID Technical Note: Developing Results Frameworks, July 2013. 3 USAID Automated Directives System (ADS) Glossary of Terms, Mar 30, 2014. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 1 EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions This is a final performance evaluation of the Kenya Parliamentary Strengthening Program (PSP), which operated from August 2000 to May 2015 (Award Number: Phase 1 and 2: IQC #DFD-I-00-04-04-00128- 00; Phase 3: CA. No AID-623-A-10-00019). The evaluation is intended to help USAID understand the extent of the project’s successful implementation and how well it maintained its relevance to the Kenyan Parliament. The evaluation also analyzes the difference that the 15-year activity duration made in parliamentary support. USAID/Kenya will use this evaluation to inform current and future programming in the democracy, human rights and governance sector, including the current Agile and Harmonized Assistance for Devolved Institutions (AHADI) activity. USAID/Washington and other USAID Missions may also benefit from the activity’s broader lessons learned. The evaluation addressed five questions covering the final five-year phase of the activity, its long-term adaptability, achievements over its 15-year period, its sustainability of institutional development and lessons learned. Project Background USAID/Kenya launched PSP in 2000 and the activity ended in May 2015. The Center for International Development of the State University of New York (SUNY/CID) at Albany implemented PSP, which aimed to support the development of an independent Parliament in Kenya by providing training and technical assistance to the development of the committee system and parliamentary support agencies, as well as promoting parliamentary engagement with the public and civil society. The activity took place during a historic period in Kenyan politics, including the 2007–2008 post-election crisis, passage of the 2010 Constitution and the 2013 transition to a bicameral legislature. Evaluation Methods and Limitations This evaluation uses qualitative data collection and analysis methods and triangulation across perspectives to assess emerging trends and themes. Primary data collection took place in Nairobi over four weeks, from September 23 to October 22, 2015, following a desk review of activity documents. The evaluation team conducted several interviews in Washington, DC, from October 14-22. Limitations included difficulty reaching some key informants and the lack of some critical documentation and information. Findings and Conclusions The evaluation questions are addressed in the following order: 2, 3, 1, 4 and 5. The USAID/Kenya Office of Democracy, Governance and Conflict (DGC) team agreed that this order provides the most logical flow of information. The original numbering of the questions is retained. Question 2: How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? PSP relied on internal and external assessments in its activity design, which the activity adapted throughout the 15 years, often in response to developments in Kenyan politics and Parliament itself. External developments included the 2002 elections; the 2007 elections, post-election violence and subsequent national accord; implementation of the 2010 Constitution; and the 2013 elections marking the return of a bicameral legislature. The design of PSP incorporated Kenyan views from MPs, parliamentary staff, SUNY/Kenya staff, and USAID/Kenya staff. The Mission’s development hypothesis was not clearly communicated to SUNY officials, who evinced an understanding of their own development hypothesis and key assumptions. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 2 In conclusion, PSP adapted effectively to changing circumstances. PSP staff used political openings as an opportunity to increase its pace of programming. The activity also responded to political developments in Kenya and played an important role beyond institutional development, such as working with the Speaker’s office to develop jurisprudence in managing interests of a coalition government in Parliament. The flexibility of USAID and PSP enabled the activity to add new elements that responded to developments like the new Constitution and the transition to a bicameral legislature, enabling PSP to stay relevant and help the Mission support democracy in Kenya. Question 3: To what extent and in what ways did PSP’s 15-year engagement with the Kenyan Parliament change the project’s ability to achieve its objectives? PSP was initially awarded as a three-year activity at a time that parliamentary development in Kenya faced political hurdles. Decisions to extend and renew the activity provided an opportunity for longer engagement on the original activity components and for the addition of supplementary components. The Parliament’s leadership at the time of PSP’s commencement was resistant to reform. Consequently, the activity had few achievements before the 2002 elections, other than building alliances with reformers in Parliament. Those elections brought a change of power that enabled the activity to gain traction and SUNY officials cited the rise of reformers to power as a major factor in approval of a second phase. Thanks to the 15-year activity, 10 components from Phase 1 of PSP demonstrated achievements relative to those cited in the 2004 evaluation. These components include: • Parliamentary service commission • Standing orders • Committees • Internships • Member and staff training • Public outreach • Civil society • Research • Budget • Legislative drafting The 15-year engagement enabled PSP to pursue additional objectives and activities such as a background role in supporting Parliament to resolve the post-election crisis in 2007–2008. The reconciliation process led to the drafting of the 2010 Constitution, which led to the approval of Phase 3 to support constitutional implementation and the transition to a bicameral legislature. USAID and SUNY officials agreed that due to political dynamics, the activity could not have accomplished its objectives in the initial three-year period and that it required 15 years to accomplish all that it did. PSP facilitated improvement in the effective functioning of Parliament through its support to functions such as lawmaking, oversight, administrative operations, committee effectiveness, public outreach and engagement. PSP supported Parliament to be more open to Kenyan citizens and helped develop parliamentary independence, professionalism and legitimacy over the course of 15 years. PSP in Phase 3 intended to undertake activities regarding ethics, but those interventions gained little traction. Continuous funding over the life of the activity enabled SUNY to avoid closing out the activity, which often meant the loss of experienced staff. The staff of SUNY/Kenya successfully built trust; and this was cited as a key factor in PSP’s success. MPs and parliamentary staff generally expressed satisfaction with PSP (of 30 parliamentary respondents, 28 expressed satisfaction). More than two-thirds of parliamentary respondents cited PSP’s support on the budget and work with committees as the activity’s most effective contributions. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 3 In conclusion, the net result of PSP programming was to strengthen the independence of the Parliament. From an institution that was administered as a branch of the president’s office, 4 the Kenyan Parliament has developed into a vibrant, professional, independent branch of government. The Parliament makes the laws of the country and oversees their implementation, while ensuring that the views of the people are represented in their government. While some of this progress could have been accomplished in less than 15 years, the full body of achievement required the entire time. Question 1: To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives during the last five years of programming? PSP was a nonpartisan, professional activity that adapted quickly to the new political and governance environment created by the 2010 Constitution. In Phase 3 (2010–2015), PSP recorded achievements under all four objectives: lawmaking, oversight, transparency/accountability and devolved funding. The vast majority of members (87 percent) said PSP provided useful support to lawmaking, but gave mixed feedback on the quality of legislation. 5 Civil society organizations (CSO) provided policy analysis to Parliament in support of lawmaking, though Parliament’s relationship with CSOs was strained at the end of the activity. Members and staff said Parliament is now able to effectively exercise its oversight mandate, though the current Parliament is less willing to do so than its predecessor. They highlighted PSP’s significant role in strengthening Parliament’s capacity to analyze and monitor the implementation of the national budget, as well as government programs and the Constitution; support to the Parliamentary Budget Office was cited as a significant factor. PSP support for live broadcasting of plenary proceedings and modernization of information technology (IT) services increased the openness and efficiency of Parliament’s work; however, ethics proved to be an area where it was difficult to gain much traction, given the lack of demand or interest from the Parliament. PSP capacity building via induction workshops and study tours helped staff of the new Senate understand their role in protecting devolution and providing oversight of county government. Factors that contributed to this success included the 2010 Constitution, reformers in parliamentary leadership, high-quality local PSP staff and flexibility by USAID. Limiting factors included the large number of new members in 2013, a high demand for PSP assistance, lack of programming for the Research Department and nepotism in the intern selection process in Phase 3 (after Parliament took over this selection process from PSP). SUNY’s focus on the committee system and capacity building for staff were effective approaches and respondents across all categories generally expressed satisfaction with PSP. Members and staff appreciated the internship program, seminars to build capacity and examine issues, quality of PSP experts and the value of study tours for inspiring reforms. Coordination between USAID and DFID was smooth and productive. In conclusion, PSP adapted quickly and effectively to support Parliament’s role under the new Constitution, and SUNY was a trusted, neutral player during this sensitive period of transition. PSP’s continued focus on the committee structure and staff capacity were effective approaches for advancing Parliament’s reform agenda in Phase 3 and its achievements in lawmaking and oversight were clearer than for transparency/accountability and devolved funding. Question 4: To what extent are the institutional developments and improvements that PSP supported likely to be sustainable beyond its 15 years of programming? The Phase 3 activity description contains a plan for sustainability and the transfer of PSP programming 4 Joel D. Barkan, Strengthening the Kenya National Assembly, (Washington: Management Systems International, June 1999), p. 2. 5 June 2015 PSP Exit Survey of MPs and Staff, p.12. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 4 responsibilities to Parliament in the final five years, including integrating the clerk’s office directly into the planning and execution of PSP activities and a sequenced handover of the internship program and staff training to Parliament. Many legislators and staff were unaware of the closure of PSP, despite SUNY officials having communicated this to the clerks and speakers of the two chambers. The Center for Parliamentary Studies and Training was legally constituted in 2011 to provide training to members and staff, but two-thirds of respondents stated it has not been effective in that role. While the Parliamentary Service Commission (PSC) has a plan for a post-PSP internship program, the program has not begun because of the need to train a large number of permanent staff, who were hired when the second chamber was created in 2013. The sustainability of the Senate is not in question; it is enshrined in the Constitution and the committee system is strong and established, serving as the primary tool for lawmaking and oversight. Many committees in both houses now have strategic plans and work plans, hold meetings guided by specific agendas, make recommendations that are timely and actionable and are able to hold the executive accountable. The PSC is established as the body through which Parliament administers its staff and budget. The budget committee has a track record of public engagement, and the Parliamentary Budget Office (PBO) is a strong, well-staffed office with top-notch leadership. While Parliament views research as critical to legislative work, few examples showed the research department providing high-quality information. In conclusion, while SUNY built plans for sustainability into the activity design, the execution of the sustainability plan was lacking. Despite the uncertainty about the activity’s end, PSP did help parliament build sustainable practices and institutions. The Center for Parliamentary Studies and Training (CPST) is in place as the mechanism for continued training of members and staff. However, CPST is not well established in Parliament; a perception among some is that training will suffer with the end of PSP. Public outreach and inclusion of public input into the work of Parliament remain a weakness. Question 5: What lessons can be learned from PSP’s 15 years of parliamentary strengthening work? The success of PSP, due in part to its 15-year engagement, holds several lessons for legislative strengthening activities worldwide, including the following: • Collaborative partnership with Parliament is important; • Having an institutional orientation is vital; • Focus on capacity building of institutions and staff; • Institutional development facilitates other objectives; • Committees are key to improving lawmaking and oversight; • Institutional strengthening is long term; • Success requires reformers in Parliament; • Parliamentary strengthening needs program flexibility; • Political activities require sensitivity; • Local chief(s) of party (COP) and staff can build trust and sensitivity; • Communicating a clear end date is necessary; and • Joint funding of a single activity can be more efficient than many donors funding their own activities. Recommendations Based on the PSP experience, USAID legislative strengthening programming should include: • Collaborative institutional capacity-building approaches; • Capacity building for both members and staff; • Committee strengthening; Kenya Parliamentary Strengthening Program: Final Performance Evaluation 5 • Long-term, flexible structure; and • A clear exit strategy and communication of end date. While USAID/Kenya is not planning a full-scale, national-level parliamentary strengthening activity, several of these recommendations may be relevant to its current and future programs, including AHADI. EVALUATION PURPOSE AND EVALUATION QUESTIONS Evaluation Purpose The purpose of this evaluation of the PSP is twofold. First, USAID would like to understand to what extent the activity successfully implemented and maintained its relevance to the Kenyan Parliament through an extended period of transformation and democratization. Second, USAID would like to know what difference 15 years of funding made to the sector. USAID hopes to use this evaluation to learn how much, if at all, its investment contributed to Parliament’s development. USAID/Washington and other USAID Missions may also be interested in the broader lessons learned from this evaluation, particularly as they relate to the 15-year investment in parliamentary strengthening. The key audiences for this evaluation are USAID/Kenya Mission’s Program Office and the DO I DGC technical team. They will use this evaluation to inform future programming in this sector. The DO technical team will use the evaluation to inform the design of possible new activities and/or work plan revisions within the AHADI activity, which aims to support devolved government in Kenya, as well as other USAID-supported programs. USAID/Kenya will share the final report with Kenya’s Parliament, specifically the PSC and the speaker.6 The Mission anticipates this audience will use the evaluation results to inform its planning for Parliament’s development. Findings will also be disseminated through preparation of a one-page fact sheet by Management Systems International (MSI) and USAID/Kenya-arranged stakeholder workshops. Evaluation Questions7 Evaluation Question 1: To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives during the last five years of programming? Evaluation Question 2: How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? Evaluation Question 3: To what extent and in what ways did PSP’s 15-year engagement with the Kenyan Parliament change the project’s ability to achieve its objectives? Evaluation Question 4: To what extent are the institutional developments and improvements that PSP supported likely to be sustainable beyond its 15 years of programming? Evaluation Question 5: What lessons can be learned from PSP’s 15 years of parliamentary strengthening work? 6 The evaluation team understands this to mean the speakers of the National Assembly and Senate. 7 Minor adjustments to the evaluation questions in the statement of work (SOW) were made following consultations with USAID/Kenya. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 6 ACTIVITY BACKGROUND Development Problem Although initially established as a bicameral legislature immediately following independence in 1963, parliament lost power in part through a reversion to a unicameral body during Kenya’s long period of authoritarian, single-party rule. Parliament established departmental committees in 1997; they became fully functional in 2001.8 Parliament began increasing its structural autonomy in 1999 and in 2000, the PSC was established. The PSC is responsible for internal budget and management.9 The post-2007 election violence resulted in an expansion of Parliament’s role and responsibilities for reconciling the country and enacting legislation. The 2010 Constitution established a bicameral legislature and county-level assemblies.10 To support the Kenyan Parliament in its reform efforts, USAID/Kenya launched PSP in 2000, following the passage of the constitutional amendment that increased the National Assembly’s structural autonomy. USAID/Kenya based its PSP partnership with the Kenya National Assembly (KNA)—implemented by SUNY/CID—on Parliament’s 12-year strategic plan, which is understood as the blueprint for modernizing, reforming and strengthening Parliament. Thus, the KNA has led the reform process supported by USAID/Kenya.11 USAID Activity Strategy and Tasks To support Parliament in its reform efforts, USAID/Kenya launched PSP (Award Number: Phase 1 and 2: IQC #DFD-I-00-04-04-00128-00; Phase 3: CA. No AID-623-A-10-00019). USAID/Kenya based its partnership with the KNA on Parliament’s 12-year strategic plan.12 Target Areas and Groups PSP targeted Parliament and—beginning with Phase 3—the 47 newly created county assemblies. Intended Results The overall focus over the 15 years was to facilitate Parliament’s reform agenda. Phase 1 (2000–2004) supported Parliament to organize and carry out its initial plan for reform. Phase 2 aimed to improve the balance of power among the institutions of governance. Phase 3 aimed to consolidate the reforms achieved in Phases 1 and 2 and support Parliament’s expanded role under the new Constitution. The evaluation scope of work in Annex I provide a detailed activity background and elaboration of the intended results. 8 USAID/Kenya PSP Original Task Order 2005. SUNY/Kenya (2011). PSP Baseline Survey: Members of Parliament and Staff Report. USAID and DFID. 9 SUNY/Kenya (2011). PSP Baseline Survey: Members of Parliament and Staff Report. SUNY/Kenya (2010). PSP 2005–2010 Final Report. Mathooko, Bonnie M. (2014). “Adapting Research and Library Services in Parliament to the Changed Constitutional Framework in Kenya.” Paper presented at 80th IFLA General Conference and Assembly, France, IFLA Section on Library and Research Services for Parliaments’ 30th Pre￾Conference Parliamentary Libraries: Past and Future, Session: “Research and Library Services Working Together to Meet Client Needs.” 10 Kenya Parliament Magazine, April 2013. 11 SUNY/Kenya (2011). PSP Baseline Survey: Members of Parliament and Staff Report. USAID and DFID. 12 Ibid. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 7 EVALUATION METHODS AND LIMITATIONS The PSP final performance evaluation assesses the relevance, effectiveness and sustainability of PSP to inform the design of a related future activity. Using the methodology framework described in the PSP evaluation SOW (see Annex 1) as a base, the evaluation team refined the methodology during its team planning meeting (TPM) held September 14–19, 2015. Based on feedback from USAID provided during a September 21 meeting (and in writing), the team revised the work plan and instruments, which USAID approved on September 22. The approved documents are in Annex 2 and 3. Data collection was scheduled for September 23 through October 16 and it was extended to October 22 to maximize the potential for interviews. Data analysis took place October 17–28, followed by a validation meeting with USAID/Kenya’s DO1 team on October 29 and a formal presentation to the Mission on November 3. Report drafting took place November 3–10, under the direction of the team leader. The evaluation team consisted of Team Leader Katherine Vittum, International Sector Expert John Lis and Local Sector Experts Marjorie Walla and Martin Ocholla. Parliamentary Liaison (and former PSP staff member) Elsie Opiyo assisted the team with identifying and scheduling individuals for interviews, as well as compiling a complete list of PSP documents for review. When Marjorie Walla left the team October 9, MSI Evaluation Specialist Carolyne Njihia stepped in to assist, particularly with data analysis. Given the nature of the evaluation questions and the stakeholders involved, this evaluation uses qualitative data collection and analysis methods. Qualitative methodologies are the most appropriate because they facilitate acquisition of rich detail regarding how and why things happened or did not and triangulation across perspectives to assess emerging trends and themes. They allow exploration of the processes behind the results, which is what interests USAID in this evaluation. Data Collection Methods and Sources The evaluation SOW includes detailed descriptions of data collection, analysis and sampling methods by evaluation question. While the evaluation team refined this methodology, it did not make major changes. A summary of methods used and data sources follows. The specific collection methodologies used include secondary data review, key informant interviews (KII) and group discussions (GD). For the desk review, the evaluation team reviewed all activity documentation provided by MSI, USAID and SUNY/CID, as well as additional resources obtained online, from stakeholders and from evaluation team members' own libraries. Key PSP documents reviewed include the following: activity awards and modifications; performance monitoring plans, indicators and available performance data; annual, quarterly and final reports; PSP evaluations, surveys and technical reports; and work plans. Annex 8 contains a full list of reviewed documents reviewed. The findings from the initial document review identified information needed to completely answer each evaluation question and guide discussions about sources for that information. Those discussions helped shape both the list of individuals to be interviewed and the semi-structured interview guides. To determine those individuals the evaluation team should conduct KIIs and GDs, the team began with the list of organizations and titles included in the SOW and consulted with both the USAID officers commissioning the evaluation and the parliamentary liaison to identify, locate and schedule meetings with interviewees. The international sector expert, who previously worked with Kenya’s Parliament both in Kenya and in the U.S., played a central role in this process. During the four weeks of data collection, the team conducted 45 KIIs and five GDs with 65 total respondents (43 women and 22 men). A full list of interviewees is in Annex 7. Although the SOW anticipated collecting data from a purposive sample of the 47 new county assemblies, Kenya Parliamentary Strengthening Program: Final Performance Evaluation 8 the evaluation schedule and budget did not allow travel to the counties. The evaluation team attempted to schedule a GD with the County Assemblies Forum (CAF) in Nairobi, but only one member was available. Unfortunately, this individual had limited knowledge of PSP, restricting the amount of quality information the team could obtain. The SOW also called for observation of a random selection of committee meetings. During the TPM, the evaluation team determined this method would not likely yield as much useful information as the planned KIIs and GDs. The team observed two committee sittings. Data Analysis Methods Data analysis methods included content analysis, comparison and mixed-methods data integration. The evaluation team integrated information from the desk review, KIIs and GDs by evaluation question to facilitate comparisons to identify common trends and themes related to PSP’s relevance, effectiveness and sustainability and successes and challenges, as well as suggestions for possible new programming. This preliminary data analysis began during the fieldwork to ensure that the team was capturing the information necessary to fully address the evaluation questions. During data collection, the team held weekly debriefings to identify common themes among interviews by evaluation question. The team refined these themes as it progressed through data collection. After completing data collection, the team reviewed all interview notes to finalize the list of key themes, which it used to code each set of interview notes. With coding completed, the team analyzed the interview data using parallel analysis, comparing the perspectives of various stakeholder groups to determine the extent to which and how effectively PSP achieved its objectives. While the team looked for areas of convergence, it also investigated divergent and unique perspectives to assess whether or not they merited consideration in the analysis. For the content analysis of secondary sources reviewed, the team used a process developed by MSI called “Answering Questions with Background Documents” and organized findings by theme and sub-themes for each evaluation question. The evaluation team then compared and triangulated key pieces of evidence from the various interviews and documents to identify the main evaluation findings that responded to each question. In some cases, the team used secondary sources to identify PSP activities, results and definitions and drew upon findings from the interviews to flesh out the complete answers to the evaluation questions. For example, to answer part of Evaluation Question 4, the team reviewed PSP documents to determine the extent to which plans for sustainability were articulated and used interview data to determine PSP's contribution to establishing training systems. Annex 5 contains a description of the analytical approach used to answer each evaluation question. The individuals interviewed by the evaluation team ranged widely in their positions and their experience with the program. More senior, longer-tenured sources had a deeper and broader understanding of the program than junior/newer sources. Consequently, evaluators did not quantify most responses and instead analyzed the information collected in light of the extent of each source’s experience with PSP. In other words, the report focuses on substantive significance rather than on percentages or precise numbers, as these could be misleading. Limitations The evaluation team encountered a number of data collection limitations that threatened the rigor of the original evaluation design and used creative solutions to ensure that answers to the evaluation questions would be as robust as possible. Specifically, the team had difficulty reaching some key informants, lacked some critical documentation and information to answer some of the evaluation questions and lost some team member support. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 9 Scheduling interviews with MPs proved especially challenging. Parliament was not in session during the first week of data collection and staff were occupied with Parliament Week, when the legislature opens to the public for a series of events. Several MPs and other key informants were in New York that week for the United Nations General Assembly. In weekly reports to USAID, the evaluation team identified target informants with whom the team was unable to meet. This included current and former high level parliamentary officials, a former USAID Agreement Officer’s Representative (AOR); some CSOs that had benefitted from PSP; and the CAF. From the beginning of the evaluation, the team lacked a number of key PSP documents and worked throughout the preparation and data collection periods to obtain them. As detailed in emails and weekly reports to the Mission, this included development hypothesis language, award documents, various reports, performance monitoring plan (PMP) and indicator protocols. Several had to be obtained from SUNY, which requested a copy of the finalized evaluation SOW before providing documents. After the Mission shared that SOW with SUNY, the evaluation team sent a list of requested documents and questions to SUNY on October 7. SUNY responded October 15 and 16, which left the evaluation team limited time to review the information and incorporate it into ongoing data collection and analysis. In addition, SUNY’s seven-year document retention policy meant that some older PSP documents were unavailable. During the second week of data collection, the evaluation team confirmed doubts that initially rose during the TPM about its ability to answer Evaluation Question 2 rigorously and fully; this was due to the lack of a clearly articulated development hypothesis for PSP in activity documents. While Mission-level strategies included development hypotheses, no PSP-specific development hypotheses could be found, except what can be gleaned from a detailed description of the strategic objective under which PSP fell in USAID’s 2008 Development Assistance Grant Agreement.13 Documents provided by USAID on October 30 clarified the development hypotheses for the Project Appraisal Document (2012) and DO1 overall (2014). Finally, the evaluation suffered when team member Marjorie Walla resigned from the evaluation on October 9, about halfway through the data collection period. The rest of the team completed data collection, dividing it up as needed, to cover the remaining interviews. MSI’s Monitoring and Evaluation Specialist Carolyne Njihia stepped in to help; she joined the team full time for the data analysis period. Annex 4 lists additional limitations that resulted from key informant unavailability. FINDINGS AND CONCLUSIONS The evaluation questions are addressed in the following order: 2, 3, 1, 4 and 5. The USAID/Kenya DGC team agreed on this order as the most logical flow of information. The original numbering is retained. Question 2 How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? Findings This question examines the extent to which the activity’s development hypothesis and design retained their relevance by changing over time to keep pace with changes in the Kenyan context; assessments and analyses that shaped the development hypothesis and program design; the extent to which changes in the 13 Development Assistance Grant Agreement 615-006, Annex 1: Amplified Description. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 10 development hypothesis and activity design incorporated Kenyan views; and the extent to which the assumptions that shaped the development hypothesis are valid for 2015 and beyond. The question examines whether the development hypothesis was clearly defined in program documents and how extensively key players understood and adapted to the changing development hypothesis. Assessments and Analyses The initial USAID agreement with SUNY to provide support to the Kenyan Parliament was signed in August 2000 for a three-year period. This agreement followed an external assessment of the Parliament in June 1999 by Joel Barkan, a University of Iowa professor working under contract for MSI. Joel Barkan concluded that the 1997 elections had brought a group of 55 to 75 reformist MPs into office; they were seeking to pass a constitutional amendment that would establish parliamentary independence by delinking Parliament from the office of the president. In response, Joel Barkan recommended that USAID “mount a modest program to strengthen the KNA over the next two to three years.”14 In March 2001, SUNY published an assessment that formed the basis of the original program framework. The assessment stated three goals to (1) assess the needs and capabilities of the KNA; (2) understand the political environment within which the Parliament functions; and (3) develop several specific recommendations for project activities. The 2001 SUNY assessment also followed the publication in November 2000 of Parliament’s strategic plan for 2000–2012, known as the “Blueprint.” As a result, the SUNY authors stated that they endeavored “to build this assessment around Parliament’s plan”; this theme would guide PSP work plans throughout the program. In 2004, USAID directly hired Joel Barkan and Njuguna Ng’ethe, a Kenyan professor at the University of Nairobi, to conduct an evaluation of PSP. Their evaluation was largely positive and it praised SUNY for establishing trust with the legislature amid “changing political conditions in Kenya,” a reference to the 2002 elections that ended the Daniel arap Moi regime. The report cited the responsiveness and adaptability of SUNY and recommended extending the program through 2007 for the life of the Ninth Parliament and emphasized that “under no circumstances should USAID/Kenya consider switching the implementing contractor in mid-stream.”15 Among the key programming recommendations were that PSP target support to the committee system, staff training and development of a parliamentary budget office. SUNY produced final reports for Phase 1 of the program in 2005 and for Phase 2 in 2010. Those internal reports reviewed the previous phase and based recommendations for the upcoming phase on its analysis of past programming; for example, the 2005 report recommended support for the committee system and staff training, and the 2010 report recommended support for implementing legislation for the new Constitution. Further, SUNY conducted a 2011 baseline survey and 2015 exit survey with MPs and staff. Program Design Several examples show the design of PSP changing throughout the 15 years of the program, often following developments in Kenyan politics and Parliament itself. Among notable external developments were the 2002 elections; 2007 elections that marred by post-election violence and resolved by the subsequent national accord; the 2010 Constitution; and the 2013 elections and return of a bicameral legislature. (See timeline in Figure 1 below.) 14 Barkan (1999), p. 2. 15 Joel D. Barkan and Njuguna Ng’ethe, An Evaluation of USAID/Kenya’s Program to Strengthen the Kenya National Assembly (Nairobi: USAID, August 2004), p. 34. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 11 FIGURE 1: TIMELINE OF PSP ACTIVITY INFLECTION POINTS AND HISTORICAL EVENTS (2000-2015) Kenya Parliamentary Strengthening Program: Final Performance Evaluation 12 The first award in August 2000 set PSP’s initial objectives: (1) support the institutional base for reform (the PSC); (2) provide development assistance to help Parliament achieve the objectives of the PSC’s strategic plan; and (3) assist civil society in understanding and relating to the Parliament. Objective 2 included support for the committee system, new directorates, MPs and staff. The program focused initial efforts on building trust among leadership and senior staff, many of whom were resistant to change. “There was a lot of suspicion at first,” a former SUNY COP said, expressing a view corroborated by MPs and senior staff. Internally, reformist members of Parliament worked to build support for reform, according to SUNY and parliamentary sources. The 2002 elections enabled SUNY to increase the pace of its activities. “We built trust and relationships. When we had the opportunity to have a more intensive relationship, we were able to react,” another former COP recalled. USAID extended the program’s end date by 18 months, ultimately ending Phase 1 in February 2005. While Phase 1 program objectives did not change, PSP added activities under those objectives. Former SUNY officials said that due to increased receptivity in Parliament, PSP increased its support to committees and researchers, initiated a pilot project on oversight and commenced support for a new PBO, public relations office and revised standing orders. The objectives for Phase 2, which began in March 2005, reflected these developments, according to the Phase 2 award document. “Work with committees,” originally a sub-objective, was elevated alongside “support to the PSC” as a primary objective. “Support to staff” became a Phase 2 objective, with development of the PBO as an element. In addition, PSP activities included support for other U.S. Government goals, such as health programs. This broader role would come to the fore following the December 2007 elections. Allegations of vote rigging in the presidential poll led to two months of widespread communal violence across Kenya that ended with a February 2008 power-sharing agreement, formalized in March when Parliament passed the National Accord and Reconciliation Act, which created a new office of prime minister. “SUNY was of great assistance to us,” said a former clerk. “They were able to help us bring the two antagonistic groups together through a neutral speaker and a neutral clerk. When [former United Nations Secretary General] Kofi Annan wanted someone neutral, he turned to the speaker. Behind the scenes, [then-U.S. Secretary of State] Condoleezza Rice consulted with the speaker through the efforts of SUNY.” In response to the national accord, USAID in July 2008 modified its agreement with SUNY to add as an objective assistance to the KNA to deal with challenges of the post-election crisis and the National Accord, according to the modification document. In September 2009, USAID again modified the agreement to add assistance to the Parliamentary Office of the Prime Minister and to the Office of the Speaker as program tasks, according to the modification document. The political reform process in Kenya turned to the drafting of the new Constitution, devolving power to 47 new counties and re-establishing the Senate as a second chamber to which the Constitution assigned a focus on the relationship between the counties and the national government. USAID and SUNY officials said the constitutional process was a driving force in the agreement for Phase 3 of PSP, which was signed in April 2010. According to the award document, Phase 3 focused on supporting Parliament to pass the implementing legislation for the new Constitution and continued support to committees and the PBO and other technical departments. It also added an objective to oversee implementation of the Constituencies Development Fund (CDF) that each MP oversaw in his/her constituency. With the first elections under the new Constitution imminent, the program design was again adapted. A February 2013 modification shifted the program focus to support for the changes that would accompany the shift to a bicameral Parliament. The modification also recognized that the CDF component was moot after the High Court of Kenya at Nairobi ruled that CDFs did not comply with the new Constitution. In its place, the modification added an objective to enhance Parliament’s oversight and responsiveness regarding devolved funds, shifting the focus to the county level, according to the modification document. “We owe a lot to PSP for the transition,” a senior senate official said. “From the period that it was apparent that we Kenya Parliamentary Strengthening Program: Final Performance Evaluation 13 would become bicameral, there were a lot of programs that SUNY supported.” Kenyan Views From its initial program design through the life of the activity, participants in this evaluation noted that the design of PSP incorporated Kenyan views. This feedback came not only from MPs and staff, but also from PSP’s Kenyan staff and Kenyan Foreign Service nationals working at USAID/Kenya. The SUNY assessment took place after Parliament published its own strategic plan and the SUNY authors said, “[we] have done our best to build this assessment around Parliament’s plan.” While transitioning to the second phase, SUNY staff consulted with parliamentary leadership on PSP’s design and incorporated those views. Participants in this evaluation, particularly senior parliamentary leaders, described this responsiveness to Parliament’s vision for the program as a hallmark of PSP over the 15 years. “They came up to me and asked me candidly, ‘What would you like us to do for you?’” a former speaker said. A former senior clerk also said, “It was not SUNY who was triggering our programs; it was us.” USAID officials corroborated these views. “PSP was owned by Parliament,” one said. A colleague noted, “PSP was really demand driven. They would respond to issues and requests of Parliament.” The Kenyan influence on program design became more apparent in 2006, when Fred Matiang’i became the COP. Members and staff credited him and PSP’s other Kenyan staff with having an excellent understanding of the local political environment. “That was the best decision; when local staff came in and managed it,” a senior clerk said. “They understood the local politics and the institution. I think that’s why the program succeeded in the long run.” Kenyan views were also incorporated into the program at USAID, whose Kenyan Foreign Service nationals (FSN) were praised for their understanding of the political environment and commitment to the program. A former SUNY COP described a Kenyan Contracting Officer’s Representative (COR) as one of the […] “architects of the program,” Development Hypothesis The development hypothesis for PSP was not clearly defined in program documents. Contemporary development hypotheses for USAID governance objectives, including PSP, can be discerned from USAID/Kenya strategies over the years, but these were not explicitly stated at the time. PSP began under USAID/Kenya’s Strategic Objective 6: “sustainable reforms and accountable governance strengthened to improve the balance of power among the institutions of governance.”16 In reviewing IR 6.2, under which PSP falls, one can construct a development hypothesis: if PSP strengthens the parliamentary committee system, increases parliament’s awareness of alternative practices and available resources and creates linkage mechanisms between parliament and civil society that provide parliament with CSO-generated information and more accurate citizen opinion, then parliament will be increasingly independent from the executive branch of government. However, such a development hypothesis was never explicitly written, nor was it communicated to PSP and it does not appear (even in the form in which it is in the integrated plan) in any PSP document. A mission document from Phase 2 contains an amplified description of Strategic Objective 6, part of which clearly describes PSP when compared with PSP documents. A development hypothesis can easily be 16 USAID/Kenya, Integrated Strategic Plan 2001-2005 (Nairobi: USAID, 2000), p. 45. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 14 discerned from this description of USAID's strategy for strengthening Kenya's parliament.17 The Phase 2 development hypothesis for PSP, from activities to results, would therefore read as follows: • If PSP organizes action-oriented workshops for the PSC and assists the parliament to incorporate the experiences of other countries, then the PSC will determine its priorities for immediate and long-term structural reforms and develop an action plan for addressing those priorities. • If PSP strengthens the linkages among the legislature, civil society and the executive; engages the national assembly in supporting PSC leadership, committees and staff; and trains parliamentary staff that provides support to committees, then parliament will implement its action plan. • If Parliament implements its action plan, then the Kenyan parliament's effectiveness will improve to more effectively represent the will of the Kenyan people, participate in legislative and policy reforms and oversee and hold accountable the executive branch of government. Following promulgation of the new Constitution, which devolves power to the counties, both the DO and the development hypothesis changed in the most recent country strategy. DO1 is “devolution effectively implemented,” and the development hypothesis is: “if the devolution process is effectively implemented, then Kenya’s governance and economy will be sustainably transformed.”18 SUNY officials stated that the USAID/Kenya development hypotheses were not clearly communicated to them, and a development hypothesis was not explicitly stated in activity documents. “We didn’t think in those days about a development hypothesis,” one program manager said of the initial program. Nevertheless, SUNY officials articulated variations on a development hypothesis that if parliamentary becomes more independent, then the monopoly on power by the executive will be broken and parliament will hold the executive to account. Key Assumptions The PSP Phase 3 award document does not list a development hypothesis or key assumptions. However, the scope of work was designed for two scenarios: 1) if the 2010 Constitution passed and 2) if the Constitution did not pass. Because the Constitution has passed, this assumption is irrelevant. Conclusions PSP adapted effectively to changing circumstances affecting its interventions over the course of 15 years. PSP staff used political openings, such as greater receptivity to reform in the Ninth Parliament, as opportunities to increase the pace of programming. PSP was also able to react to political developments in Kenya and to play an important role beyond institutional development, such as facilitating negotiations to end the 2007-2008 post-election crisis. The flexibility of both USAID and PSP enabled the activity to add new elements that responded to political developments, such as assistance to the newly created Parliamentary Office of the Prime Minister and support for legislation to implement the new Constitution; and developments in Parliament, particularly the transition to a bicameral legislature. This adaptability enabled the activity to stay relevant for 15 years and provide the Mission with the ability to support the development of democracy in Kenya. The PSP activity design in Phases 1 and 2 was based on several assessments and analyses of the Parliament by outside experts, SUNY staff and Parliament itself. PSP’s own final reports provided analysis of past programming and recommendations for future work that guided the design of Phases 2 and 3. 17 Strategic Objective 6 DAGA 615-006 (amended), July 2008, Annex I: Amplified Description, sub-heading "Strengthening Parliament," pp. 5-6. 18 USAID/Kenya, CDCS 2014-18, Nairobi: USAID, 2014, p. 28. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 15 Among the analyses used in the program design was the Parliament’s own strategic plan, which formed the basis of the 2001 SUNY assessment. This responsiveness to Parliament’s own initiatives and incorporation of Kenyan views was a strength of the program and ensured that its activities had strong support among parliamentary members and staff. The decision to have an all-Kenyan staff, including the COP, at SUNY/Kenya helped the activity be sensitive to the local political environment, particularly important at sensitive moments such as the post-election crisis and constitutional implementation. The decision by USAID to have Kenyan FSNs oversee the activity ensured that local views were incorporated at all stages of activity design. The Mission’s formal development hypothesis was not communicated to PSP staff, but they understood the thinking behind the program and their own development hypothesis that contributed to the program. Phase 3 considerations about whether the 2010 Constitution would pass are irrelevant. Question 3 To what extent and in what ways did PSP's 15-year engagement with the Kenyan Parliament change the project's ability to achieve its objectives? Question 3 assesses whether or not continuous, but ever-adapting funding made a difference in the development of a strong, fully functional Parliament that operates transparently and democratically. It explores whether and how achievements were realized as a result of PSP’s 15-year engagement, focusing on those that might not have been accomplished in a shorter program. It examines how PSP facilitated improvement in key parliamentary functions and assesses how the democratic performance, political maturity and effective functioning of the Kenyan Parliament changed as a result of PSP assistance. It also assesses how satisfied MPs and parliamentary staff were with the program and gauges areas they think U.S. assistance was most effective. The full question description is in the SOW for the evaluation. Findings The length of PSP is unusual compared to the standards of legislative strengthening at USAID. PSP was initially awarded as a three-year activity at a time when parliamentary development in Kenya faced political hurdles. PSP staff said they decided to focus their initial efforts on building relationships with reformers in Parliament who were looking to take advantage of the new institutional landscape. As a result, they acknowledged that the program had few achievements before the December 2002 elections, other than building alliances that would soon prove useful with reformist members and clerks in Parliament. The importance of this approach was cited in the 2004 evaluation: “The thriving reform coalition in Parliament is perhaps the most lasting impact of the SUNY project.”19 The 2002 elections brought a change of power that significantly altered the landscape for PSP and enabled the program to gain traction in Parliament. “When we had the opportunity to have a more intensive relationship [after the 2002 elections], we were able to react,” a former COP said. USAID officials noted that the decision to extend the program in 2010 was driven by the role that Parliament would play in implementing the new Constitution. One USAID/Kenya official said, “it is a big ‘maybe’ if PSP would have been extended without constitutional reform.” USAID and SUNY officials agreed that the program could not have accomplished its objectives in the initial three-year period and that the program required 15 years to accomplish all that it did. “I think 15 years was appropriate and proper. It added to the success of PSP,” a USAID/Kenya official said. “It needed the full 15 years to accomplish what it did,” a former COP said. 19 Barkan and Ng’ethe, p. 29. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 16 This view was mirrored in Parliament. “Fifteen years was required to institutionalize,” one senior adviser said. “In five years, I don’t know what they would have achieved. It would have been zero.” Original Components In reviewing the activities of PSP in Phase 1, it was evident that 10 components remained part of the program through all 15 years. In all 10 cases, one can identify achievements that were realized due to the 15-year program, relative to the achievements cited in the 2004 evaluation. Parliamentary Service Commission Support to the PSC was an objective of the original program. SUNY’s 2001 assessment noted that the PSC, created at the end of 1999, had the responsibility for the budget and management of Parliament. However, it had not developed the technical capacity to manage the budget. Progress was initially slow; one senior adviser said, “When I joined Parliament in 2003, the institution had no capacity to effectively and competently manage its affairs.” The 2004 evaluation found that the main achievement of PSP with the PSC was its support for strategic plans for both the Parliament and the PSC itself. Concurrently, the evaluation credits PSP with providing “an institutional focal point for [reformers] within Parliament, thereby legitimizing the reform process.” SUNY’s 2010 final report notes that the program in Phase 2 supported the PSC to double the number of professional staff and “built capacity to support committees in their legislative and oversight functions.” In addition, PSP supported the PSC in revising the strategic plan for 2008-2018. In Phase 3, PSP supported the PSC to develop the institutional structure for the new Senate that was created in the 2010 Constitution. Parliamentary staff noted that while direct assistance to the PSC was de-emphasized in Phase 3, the PSC continued to serve as the main point of contact for PSP. Committees Although they were created in 1997, committees became active in the Ninth Parliament in 2003, according to PSP documents. Strengthening committees in their legislative and oversight roles became a primary feature for the remainder of the program. According to the 2005 final report, Phase 1 work with committees focused on capacity building—like strategic planning—and workshops and study tours for members and staff. The 2004 assessment found a “noticeable impact of the program to date” on committees; it recommended that the program be extended through 2007. “The committee system is beginning to function as it does in established legislatures in established democracies,” it said.20 By Phase 3, support to committees had become “80 to 90 percent” of PSP’s work, according to a senior SUNY/Kenya staffer. In 2015, committees are an integral part of both the KNA and the Senate. Their meetings are announced and open to the public, while committee staff and a researcher from the research department support members. An evaluator who attended two committee meetings found members engaged in oversight and scrutiny of legislation; they were supported by background materials that PSP staff provided them and a researcher who was prepared to answer their questions. SUNY reported that ministers expect oversight, and an evaluator observed a committee making amendments to a bill, which were incorporated into the final legislation that passed the following week. 20 Barkan and Ng’ethe, p. iii. “We became the go-to institution for capacity building. … Toward the end, they referred to SUNY as, ‘the other department of Parliament.’” — Senior SUNY/ Kenya Official Kenya Parliamentary Strengthening Program: Final Performance Evaluation 17 MP/Staff Training Support to MPs and the parliamentary service were sub-objectives in Phase 1. The 1999 assessment recommended specialized training for members and staff. The 2004 evaluation cited a significant impact from study tours, particularly a 2001 staff visit to the U.S. to examine service provision and a 2003 Liaison Committee visit to Ottawa, Canada, and Albany, NY to learn about committee operations. “The training of MPs has had a noticeable impact, and again SUNY can claim some credit for this. For example, more and more members are now prepared to offer private bills,” the evaluation found.21 It also noted that staff needed new skills to respond to increased demand from members and the public. By the end of Phase 2 in 2010, PSP had conducted 158 workshops and seminars, training more than 3,000 MPs and 1,200 staff.22 By 2015, the total had reached 255 training sessions serving 6,568 participants.23 “We could not have had this exposure in five years,” a senior KNA clerk said about the capacity building for members and staff. “This time allowed SUNY to understand the client.” Civil Society Work with civil society was PSP’s original third objective. The 1999 assessment called for USAID to “nurture linkages between civil society and the National Assembly,” and PSP regularly invited CSOs to participate as resource persons in parliamentary workshops. The 2004 evaluation found that “SUNY’s interaction with CSOs has arguably increased access to Parliament by more CSO actors than before.” The length of PSP enabled greater interaction with civil society, including the development of the Parliamentary Initiatives Network (PIN), a CSO network that PSP convened to provide policy inputs to Parliament. A CSO official said that PSP helped build the confidence of CSOs through the PIN and helped CSOs establish relationships with Parliament. The increased interaction was noted mostly with regard to the budget. In Phase 2, SUNY reported that CSOs had quadrupled the number of budget issues raised with Parliament, and the percentage of those issues reflected in the final budget increased tenfold. In 2015, SUNY reported that “the number of formal interactions between committees and outside actors increased by nearly 250 percent.”24 Standing Orders The 2001 assessment identified that revising the standing orders was necessary to improve lawmaking and increase transparency. PSP worked with reformers to try to revise them before the end of the Eighth Parliament in 2002 because standing orders that win approval at the end of one Parliament take effect in the next. This effort proved unsuccessful, delaying the revision for another five years. SUNY and parliamentary sources described a process of incremental progress in the Ninth Parliament. That process led to success on the second attempt in 2007. In Phase 3, the promulgation of a new Constitution and the establishment of the Senate required another revision of the standing orders, which SUNY supported. The 2015 final report noted that “PSP provided support in developing the new standing orders in the form of technical expertise, reflection retreats and printing services that were necessary to bring the new rules to fruition.”25 Furthermore, PSP printed several hundred copies of the new Senate rules and had them available for the inaugural Senate session in 2013. Parliamentary staff said this proved to be essential because the official government printer had failed to complete publication in time. 21 Barkan and Ng’ethe, p. 30. 22 2010 Final Report, p. 5. 23 Kenya PSP Final Report, Albany: SUNY Center for International Development, August 2015, p. 2. 24 2015 Final Report, p. 2. 25 2015 Final Report, p. 8. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 18 Internships The 2001 assessment found that PSP was interested in establishing an internship program. The first interns were selected in April 2002 to begin work that June as part of a pilot program. By the end of Phase 3, 160 interns had worked in Parliament.26 One senior parliamentary official estimated that approximately 70 percent of PSP interns were subsequently recruited to work in Parliament and said many others joined country assemblies. “The success of the internship program would not have been evaluated in a span of five years. Many interns were [later] employed in Parliament,” said one departmental deputy director in the National Assembly, noting that in 2013 his department received four interns to augment its staff, which was facing increased demands: “I was using them as permanent staff.” Research Both the 1999 and 2001 assessments identified assistance to the parliamentary library and establishment of a proposed research center as possible areas of PSP involvement; however, DFID and the European Union initially took on this task. As a result, the 2004 evaluation noted that PSP had not engaged in these areas. The 2010 final report states that with direct funding from DFID, PSP in Phase 2 began assistance to the research department. Parliamentary staff reported that this increased around 2007. In 2010, SUNY reported that research department staff was “more highly qualified and trained” than before. Interns were assigned to support the research department, which then had also defined operating procedures and staff structures. A former department director cited exchange programs for research department staff and attachments to research services in the U.S. and United Kingdom as particularly useful for him and his colleagues. Support declined in Phase 3, as discussed below. Budget Support for budget analysis and oversight is a prime example of what PSP accomplished in a 15-year program. Parliament’s 2000 strategic plan called for establishing a budget office and the 2001 SUNY assessment cited a “need for professional budget staff to assist Parliament to play its budget-making and oversight roles more effectively.” In Phase 1, the most notable achievement in this area was the establishment of pre-budget and post-budget workshops each year, bringing together MPs, senior executive branch officials, civil society and the private sector. According to the 2004 evaluation, “debate on the budget is showing increasing sophistication, whereas before there was little or none.” With PSP support, Parliament in 2008 passed a Fiscal Management Act that established a Budget Committee in Parliament and a PBO to support it. The initial budget workshops were institutionalized under the PBO. “It was in the original strategic plan, but it didn’t happen [until] many years later,” a former SUNY COP said. A former senior parliamentary official said, “If not for that intervention, we might not have the budget systems we have today.” By 2015, SUNY reported that 92 percent of members rated the PBO as “good or very good.”27 In Phase 3, PSP helped the PBO develop a macro-economic model that underpinned an independent economic forecast and an analysis of the executive budget. Public Outreach Parliament’s 2000 strategic plan called for “establishing outreach of Parliament” at a time when committee meetings were closed to the public. The 2004 evaluation found that PSP had supported publication of introductory booklets about Parliament for both adults and children and the production of a magazine about Parliament. 26 2015 Final Report, p. 4. 27 2015 Final Report, p. 2. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 19 Longer engagement enabled public outreach activities in subsequent phases. In Phase 2, PSP supported reform of parliamentary procedures to enable live broadcasting of plenary sessions and revisions to the standing orders that provide for open committee meetings and hearings, as well as helping create a media center. In Phase 3, PSP procured an electronic voting system, making members’ votes publicly available. Legislative Drafting The 2001 SUNY assessment noted that members wishing “to introduce private members’ bills must bear the cost of developing and drafting them—there are no parliamentary resources available.” Parliament’s 2000 strategic plan called for creation of a Directorate of Legal Services to provide drafting and legal support to MPs. By 2004, PSP had just started a pilot project on implementation of private member bills. By the end of Phase 2, PSP had supported Parliament to develop the structure and mandate of the Legal Counsel’s Office. Phase 3 provided additional capacity building to the office and strengthened Parliament’s legislative drafting capability. The Senate director of legal services credited PSP with helping to train the counsels in the department, including training at the international legal drafting center at Tulane University. New Components In addition to the original components, the 15-year engagement enabled PSP to pursue objectives and activities that had not been envisioned when the program began. Among the more notable achievements was PSP’s role in supporting Parliament during and after the post-election crisis in 2007-2008. USAID added support to Parliament in resolving the crisis as a program objective; and assistance to the Parliamentary Office of the Prime Minister and to the Office of the Speaker were added as tasks. The reconciliation process led to the drafting of a new Constitution, which was approved in a 2010 referendum. The expected results under the Phase 3 objectives included the passage of transition legislation to implement the new Constitution and improved capability to monitor implementation of the Constitution. Finally, in 2013, PSP added an objective to improve the Kenyan Parliament’s responsiveness and oversight regarding devolved funding and assisted Parliament with the transition to bicameralism. Parliamentary Functions PSP facilitated improvement in the effective functioning of Parliament through its support to functions such as lawmaking, oversight, administrative operations, committee effectiveness, public outreach and engagement. Activities in the various components discussed above were the primary means through which PSP facilitated this improvement. PSP facilitated improvement in lawmaking through its support to parliamentary committees; capacity building for members and staff through policy forums and skill-based workshops, particularly focusing on legislative analysis and drafting; revisions to the standing orders that require committee consideration of bills and open committee meetings to the public; and development of the Legal Department. Of special note is PSP capacity building for women MPs and support to them in drafting and advancing bills like the Matrimonial Property Act and Protection Against Domestic Violence Act, Kenya Women Parliamentarians Association (KEWOPA) staff said. PSP made a substantial contribution to oversight, according to interviewees from Parliament, SUNY and international donors. First, PSP supported the development of standing committees to oversee government ministries, which created an oversight structure. In addition, PSP supported research and analysis, which informed the oversight process. Of particular note is PSP support to budget oversight, which was strengthened by creating and developing the Budget Committee and PBO. Democratic Performance Assessing the democratic performance of a legislature is a difficult task. For the purposes of this report, evaluators sought to examine how Kenya’s Parliament sought to reflect the will of the country’s people by Kenya Parliamentary Strengthening Program: Final Performance Evaluation 20 being open and responsive to the public. All respondent groups agreed that PSP supported Parliament to be more open to input from Kenyan citizens. One senior senate clerk said, “When I came to Parliament, it was shrouded in mystery. No one wanted to come to Parliament. You would be chased away at the gate. No one wanted anything to do with Parliament.” A parliamentary colleague said, “Parliament is more democratic. It has to sell itself to the people.” Parliamentary sources cited workshops organized by PSP as a key mechanism for including public input into the legislative process, by bringing together committee members with outside experts to discuss proposed legislation. PSP’s development of the PIN, a CSO network that PSP convened to provide policy inputs to Parliament, facilitated incorporating outside expertise into Parliamentary legislation. However, Parliament has yet to determine how best to incorporate public input into the legislative process. Political Maturity Parliament, donors, SUNY and outside sources stated that PSP contributed to parliamentary independence, professionalism and legitimacy over the course of 15 years. “Parliament has become more independent,” a senate clerk said. “Parliament does not look over its shoulder. Parliament does not serve at the beck and call of other institutions. Parliament is strong.” Donors and outside experts said PSP improved the quality of debate in Parliament through provision of research and analysis that resulted in better-informed debates. However, donors, experts, a senior parliamentary adviser and a committee chairperson stated that MPs and committees in the Tenth Parliament (2008-2013) were stronger and more effective than those in the current Eleventh Parliament were. One senior adviser cited the Tenth Parliament as the “apex” of PSP and as the time when “we matured.” According to parliamentary and SUNY respondents, members of the Tenth Parliament understood their role; members and staff were highly skilled and competent; debates were lively; and Parliament held the government accountable and made reforms. The Eleventh Parliament has had to incorporate a large number of new members because of the expansion of the National Assembly, the establishment of a new Senate and the shift of ministers out of parliament. Parliamentary and SUNY interviewees said the Eleventh Parliament is less reformist as a result. All respondent groups alleged that there is continued corruption in Parliament; these allegations were supported by a December 2015 audit report by the national Ethics and Anti-Corruption Commission. 28 PSP in Phase 3 intended to undertake program activities regarding ethics, but this programming gained little traction. “[A]s the SUNY program operates on a ‘demand-driven’ approach, the demand from Parliament has been very low either in ethics legislation or in the formation of an ethics committee,” the 2015 Final Report states,29 noting that leadership began discussions about a code of conduct in May 2015, at the end of PSP, only after a series of scandals. “The ethics component was too late,” said a SUNY/Kenya staffer, adding that the established relationship made it difficult to introduce a new, potentially discomfiting element to the activity. The PSP final report notes that parliamentary leadership in May 2015 did discuss creating a code of conduct and/or strengthening the Powers and Privileges Committee, but PSP closed before much work could be done. There are some aspects of ethics and integrity in the revised standing orders such as rules that MPs must disclose personal interests when speaking on any matter and shall not vote on any matter in which the member has a pecuniary interest. 28 For example, see Nancy Agutu and Albert Nyakundi , “Corruption in Parliament -- Former MPs on Payroll, EACC Audit Shows,” The Star (Nairobi), Dec. 4, 2015, and Wambugu Kanyi, “DPP Asked To Charge MPs with Corruption,” The Star (Nairobi), Dec. 1, 2015. 29 2015 Final Report, p. 21. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 21 Continuous Funding While PSP lasted for 15 years, it was not designed or funded as a 15-year program. According to award and modification documents, the original three-year activity was extended on three occasions, ultimately lasting four-and-a-half years. Phase 2 was initially a four-and-a-half-year activity that was extended to last five years. Phase 3 began as a four-year activity that ended up being a five-year activity after two extensions. While the activity length was the product of three agreements and several modifications, funding was continuous, according to award and modification documents. As a result, SUNY never had to close out the program, which can often bring about the loss of experienced staff. As noted, SUNY/Kenya staff gained trust over the years, and many sources cited this as a key factor in the PSP’s success. Because PSP was continuously funded, the activity was established and had a trusted, experienced staff in place during the 2007-2008 post-election crisis. As a result, PSP was able to support the Mission and the Embassy as they sought to engage Parliament in resolving the crisis and in negotiating a new Constitution, USAID and parliamentary officials said. Having an existing program also gave the U.S. Government a quick entry point to address other political and development goals, USAID and SUNY officials acknowledged. Parliamentary Satisfaction MPs and parliamentary staff generally expressed satisfaction with PSP, either by explicitly stating that they were satisfied or by providing positive responses when asked their overall view of the program. Of 30 parliamentary respondents, 28 expressed satisfaction, while one was partially satisfied and one was unsatisfied (see Figure 2). In responding to an open-ended question about which aspects of U.S. assistance were most effective, more than two-thirds of parliamentary respondents cited PSP’s work on the budget and with committees; 22 cited the budget support; and 21 cited committees. Other components that were cited as most effective were the internship program (18), training and capacity building (17) and revisions to the standing orders (16) (see Figure 3 on the next page). Conclusions PSP’s 15-year engagement enabled the activity to achieve its initial objectives. The initial years of the program were devoted to small projects and building alliances with reformers in Parliament, developing trust among MPs and staff that would impact in later years. If the activity had ended after the initial three￾year phase, it would have had few results. By extending the activity, USAID enabled PSP to build on the foundation of trust that it had developed to achieve results. The activity began after the 1997 election brought an estimated 55 to 75 reformist MPs to office, but they were not yet in a position to implement their goals. Fortunately, it remained in place through the Ninth (2003-2008) and Tenth (2008-2013) Parliaments, when reformers were in a position to push through changes that resulted in an independent, effective Parliament that serves as a co-equal branch of government in a system of checks and balances. 93% 3% 3% Level of Parliamentary Satisfaction Satisfied (n=28) Partly Satisfied (n=1) Not Satisfied (n=1) FIGURE 2: LEVEL OF PARLIAMENTARY SATISFACTION Kenya Parliamentary Strengthening Program: Final Performance Evaluation 22 The extended engagement also enabled SUNY, in consultation with USAID, to set new objectives for PSP in response to political and parliamentary developments over the 15 years. Most notable were the response to the 2007-2008 post-election crisis, support for the 2010 Constitution and its implementing legislation, and support for the bicameral Parliament that took office after the 2013 election. While it is conceivable that new assistance activity could have been developed in response to these events, having PSP in place enabled a rapid response that Kenyans credit with helping to end the crisis and with contributing to the development of the current political system and parliamentary structure. PSP helped Parliament improve its lawmaking and oversight functions and develop its internal administration, as well as “open up” to the public. The improvement in parliamentary functions can be attributed to the successful implementation of the key components of PSP. First among these was PSP support to the PSC, which helped Parliament develop its own administrative structure for its staff and budget. This, in turn, supported the development of the committee system, which is at the core of lawmaking and oversight. Additional components, such as budget, research, standing orders and legislative drafting, played an important role in these areas. PSP’s initial work in public outreach and with civil society assisted Parliament in opening to the public. The net result of PSP programming was to strengthen the independence of Parliament. From an institution that was administered as a branch of the president’s office, the Kenyan Parliament has developed into a vibrant, professional, independent branch of government. The Kenyan Parliament makes the laws of the country, oversees their implementation and ensures that the views of the people are represented in their government. While some of this progress could have been accomplished in less than 15 years, the full body of achievement required the full length of time. Question 1 To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives during the last five years of programming? Question 1 focuses on Phase 3. It examines the effectiveness of the adaptive approach used; the extent to which PSP achieved its objectives; and Parliament’s impression of and level of satisfaction with the program. It identifies factors that contributed to the success or failure of the program, areas in which PSP’s approach was most effective, and strengths and weaknesses of programming and the mechanisms used. The full question description is contained in the SOW for the evaluation. Findings Adaptive Approach, Non-Partisanship PSP adapted quickly to the new political and governance environment created by the 2010 Constitution, 22 21 18 17 16 14 14 11 9 4 0 5 10 15 20 25 Parliamentary Responses: Most Effective U.S. Assistance, 2000-2015 total n=30 for each response FIGURE 3: PARLIAMENTARY RESPONSES: MOST EFFECTIVE U.S. ASSISTANCE, 2000-2015 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 23 according to parliamentary, USAID and SUNY sources. USAID characterized this period as the climax of PSP; Parliament was at the center of the reforms and it was a critical phase of Kenyan history. Much of the effort was aimed at making the Senate operational and preparing Parliament for the 2013 elections. Senate staff described the support as forward looking and proactive, preparing the ground for the Senate to succeed. Staff highlighted PSP’s assistance for the new standing orders even before the Senate was fully established and work with the new Senate committees and departments to develop their work plans. USAID and current and former staff of Parliament described SUNY as nonpartisan and professional. USAID noted SUNY’s success in insulating itself from politics within parliament by working closely with the parliamentary committees, the Office of the Speaker, the clerks and various departments as they were evolving. The then clerk of the KNA described SUNY as a neutral arbiter in organizing meetings with key members to build consensus for transitional legislation following promulgation of the 2010 Constitution. Officials from SUNY articulated its institutional ethic of being responsive to demands and nonpartisan. No respondents reported that SUNY was partisan or biased. Achievements A review of PSP reports and interviews found achievements under all four objectives. SUNY designed a series of indices to measure improved quality in lawmaking, oversight and representation in Phase 3. Lawmaking The PSP 2015 exit survey shows that 87 percent of legislators reported that PSP was useful in supporting Parliament’s lawmaking functions,30 a finding confirmed by a majority of parliamentary respondents, as well as interviewees in other categories. Common examples centered on high-quality support for the scrutiny of bills, induction sessions for new members and planning sessions for committees. Data on the quality of legislation is somewhat mixed. In Phase 3, SUNY developed a quality of legislation index based on seven benchmarks: (1) constitutionality; (2) objects of the bill; (3) legislative due process and debate; (4) legislative due process and stakeholder consultation; (5) status upon review by other authorities; (6) feasibility/enforceability; and (7) clarity of drafting. The PSP final report shows a steady, upward trend in quality in Phase 3,31 despite the 70 percent turnover in members, creation of a new Senate and increased demands on Parliament arising from the new Constitution. Similarly, senior parliamentary staff stated in interviews that the quality of legislation improved, including the quality of bills brought by women MPs who received PSP support. However, a key expert for the program said the panel of experts that reviews the legislation has been unable to establish whether improvement has taken place. PSP reporting contains several positive examples of PSP-funded CSOs providing policy analysis, training and tools for Parliament in Phase 3 (e.g., on transition legislation and budget oversight). It also mentions that senior parliamentary officials cited instances where Parliament sought critiques and policy alternatives from CSOs. However, respondents in and out of Parliament cited tension and mistrust between Parliament and some CSOs that engage in advocacy, which undermines their ability to work together. Oversight MPs, parliamentary staff, USAID and SUNY reported that PSP capacity-building efforts with committee clerks, researchers and legal officers have led to Parliament now being able to effectively exercise its oversight mandate, including making informed inquiries to hold government accountable for expenditures. MPs, parliamentary staff, USAID and SUNY also highlighted PSP’s Phase 3 role in strengthening Parliament’s 30 PSP Exit Survey Findings: Members of Parliament and Staff, June 2015, p.44. 31 PSP Final Report, Aug. 31, 2015, p.26. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 24 capacity to analyze and monitor implementation of the national budget, as well as government programs and the Constitution. Much of this effort centered on enhancing the systems and tools of the PBO and support to key committees, including Budget, the Public Accounts Committee (PAC) and the Public Investments Committee (PIC). The PBO noted that PSP’s support for a macroeconomic forecasting model helped empower the office and improve its work in the past two years. The 2015 exit survey found that 92 percent of legislators rated PBO oversight and analytical capacity as good or very good.32 Despite these advances in capacity attributed to PSP’s efforts, the 2015 exit survey and a small but diverse group of journalists, civil society and experts noted a lack of will by the current Parliament to exercise oversight. For instance, one committee chair reported that the current legislature blankly accepts reports from the auditor general without reviewing them or making inputs, and does not monitor the spending of the executive branch, indicating a breakdown in the system of checks and balances and revealing opportunities for corruption. The exit survey shows a stark difference in the views of legislators and staff on this issue, with more than 90 percent of MPs reporting improved oversight capacity. However, only 14 percent of staff reported improved efforts by MPs in this regard, while 28 percent reported that efforts have declined compared to 2011.33 The evaluation team cannot confidently determine what this means, but it may relate to the changes introduced by the new Constitution and the degree of ambiguity regarding the oversight roles of the two houses. It may also relate to the overwhelming majority that the government has in Parliament or the perceived decline in the quality from the Tenth Parliament to the Eleventh. SUNY also noted that the timing of the survey may have played a role, since it was administered near the time that several scandals were breaking in Parliament and allegations of corruption arose among key oversight committees. Transparency and Accountability Public participation is a key element of the Constitution and is reflected in the revised standing orders supported by PSP. Legislators, parliamentary staff and journalists covering Parliament reported that PSP support for live broadcasting of plenary proceedings and modernization of IT services increased the openness and efficiency of Parliament’s work in Phase 3. A KNA committee chair and a Senate committee clerk described the live coverage of plenary proceedings as a major strength of PSP that brought Parliament closer to the public and allowed citizens to have direct, objective access to leaders. However, another committee chair stated that the live coverage has had a counter effect, making debate “very artificial” as leaders care more about being seen on television than they do about the topic of discussion. PSP introduced new programming aimed at reducing impunity for elected representatives, officials and public institutions in Phase 3. SUNY staff described ethics as an area where it was difficult to gain much traction, given the lack of demand or interest from Parliament. A former COP said the program may have had more momentum if its focus broadened beyond ethics legislation. However, a 2015 case study by a professor at SUNY/CID notes that opportunities for targeted assistance are opening up. Similarly, a Senate committee clerk said senators understand better how to deal with issues of ethics and conflict of interest after participating in PSP capacity building. He said senators know the tools to use in engaging the counties on issues of performance, transparency and accountability. Devolved Funding PSP introduced programming to support Parliament’s responsiveness and oversight regarding devolved funding in Phase 3. This objective initially was to provide parliamentary oversight for CDFs, but shifted its focus when the CDF structure was found to be in conflict with the new Constitution, according to the 32 PSP Exit Survey Findings: Members of Parliament and Staff, June 2015, p.47. 33 Ibid, p.33. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 25 modification document. Activities under the devolved funding objective included assistance to the country assemblies on devolved governance beginning in 2013. Senate staff reported that PSP capacity building via induction workshops and study tours helped them understand their role in protecting devolution and providing oversight of county assemblies. A senior staffer noted that the Senate is passing on the knowledge and capacity it gained through PSP support such as the functions of various committees to the counties. Similarly, another senior parliamentary staff member added that the counties have borrowed heavily from the new bicameral standing orders that PSP helped develop. PSP’s engagement with devolved funding meant to lay the groundwork for AHADI and other USAID-funded programs to offer more comprehensive and targeted support for devolved governance. The PSP exit survey concludes a need for specialized training for both staff and legislators at the national level on oversight of devolved funds in particular, noting that the majority of legislators do not believe that adequate systems are in place to monitor these funds.34 Contributing and Limiting Factors for Achievement Contributing Factors The 2010 Constitution contributed to program achievement by increasing Parliament’s budget-making authority and requiring public outreach and inclusion in the work of Parliament, according to parliamentary interviewees. They also noted that the one-third quota for women MPs has increased the number of women in Parliament, including in leadership positions. Parliament and SUNY both underscored the importance of reformers inside Parliament holding leadership positions. Parliament, USAID, civil society and SUNY respondents cited the quality and capacity of PSP/Kenya staff in Phase 3, stating that having locals who understood the local dynamics and protocols allowed buy-in and ownership of the PSP by Parliament. In addition, SUNY staff and one MP noted that Embassy and USAID support, flexibility and contextual knowledge contributed to program achievement in Phase 3. Limiting Factors Bicameralism introduced a large freshman class of legislators in 2013, as Parliament expanded from 224 members to 418. This included turnover of approximately 70 percent in MPs and 40 percent growth in staff, according to election results and senior administrators. PSP reporting and parliamentary interviews showed conflict between the two chambers and wrangling over mandates began almost immediately; consequently, SUNY reported that it was unsuccessful in holding a joint leadership retreat for the two houses. Several outside sources and a couple of members said that the Eleventh Parliament is not as strong as the Tenth in terms of a reform agenda, behavior and holding government accountable. They attributed some of this to the challenges of a new governance system and the major coalition in Parliament. Parliamentary staff and SUNY reported a strain on PSP resources created by the rapid growth of members and staff and corresponding requests for assistance. SUNY described “the flood of requests” as satisfying, but noted the need to vet the services provided because resources could not keep pace with demand. Some staff in the research department expressed dissatisfaction about the level of PSP support provided in Phase 3, stating that PSP did not reach out to the department. SUNY and a senior parliamentary staff member stated that the department lacked the same level of leadership or drive as other departments; PSP responded to requests for support, which the research department did not make. In Phase 3, after Parliament took over this selection process from PSP, two committee clerks and one committee chair stated that MPs influenced the selection of interns. These respondents said nepotism had 34 PSP Exit Survey Findings: Members of Parliament and Staff, June 2015, p.52. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 26 become the norm. Selection based on political connection affected the standards and quality of the activity. Effective Approaches More than half of parliamentary respondents, as well as USAID, outside experts and SUNY, described the focus on developing a committee system as an effective approach for professionalizing Parliament and facilitating its work. In Phase 3, PSP support focused on nine key committees. Parliamentary respondents reported that committees that worked with PSP are proactive and have strategic plans, are able to drive an agenda and are sought out for advice on their area of expertise. They said other committees are more reactionary. Outside experts who worked with PSP noted that committees helped entrench the principles of financial and political independence, since committees are bipartisan and it is difficult to bribe all of them. They described the committees as centers for good legislative debate and believed that the ability of PSP to bring committee members together to have a dialogue is also an achievement. One respondent also noted that PSP’s support to female MPs and staff through KEWOPA equipped women with the leadership skills that enabled them to chair some committees and actively participate in committees that were previously male-dominated. These favorable views of the committee approach align with the PSP 2015 exit survey, in which parliamentary staff reported that their best experience in Parliament was working in committees, while legislators regarded committee work as their most important priority.35 Second, all respondent groups reported that the emphasis on building staff capacity was effective and appropriate. Respondents consistently mentioned that trained staff tend to build parliamentary careers, while the majority of legislators turn over with each new Parliament. A former senior clerk noted that PSP achievements are evident in the quality of the staff, including interns who have been absorbed into the institution. A current departmental director believed that parliamentary staff involved in PSP were prepared for the transition from the Tenth Parliament to the Eleventh and bicameralism. Programming and Mechanism Strengths and Weaknesses Respondents across all categories generally expressed satisfaction with PSP, particularly the internship program and workshops and seminars to build capacity and examine issues as the strongest areas of engagement. The following table summarizes respondent views. FIGURE 4: RESPONDENT VIEWS ABOUT PROGRAMMING STRENGTHS AND WEAKNESSES Strengths (n=50) Weaknesses (n=50) • Internship program (24) • Capacity building/training/ seminars/ workshops/ retreats (22) • Study tours/ exchanges (14) • Induction/ orientation programs (13) • PSP technical experts (12) • Support for new standing orders (11) • Support for work planning/ strategic planning (10) • Support to research department (8) • Support to legal department (8) • Media center (4) • Cost of residential workshops in locations outside Nairobi (4) • SUNY did not follow USAID branding requirements (3)** • SUNY as a gatekeeper to Parliament (2) • Support to research department (2) • PSP technical experts (1) • Electronic voting system 35 PSP Exit Survey Findings: Members of Parliament and Staff, June 2015, Executive Summary. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 27 Strengths (n=50) Weaknesses (n=50) • Stakeholder consultations/public outreach (4) • Macro-economic forecasting model (3) • Pre- and post-budget workshops (3) • USAID-allowed light branding (3)* • Parliamentary Study Group (2) implementation challenges (1) * Parliament, SUNY ** USAID Consultants, Resources and Networks Parliamentary staff, the CPST and SUNY emphasized the high quality of experts that PSP fielded to support the work of Parliament. In Phase 3, experts facilitated induction and capacity-building workshops, helped analyze the large volume of transitional legislation and supported commissions outside of Parliament that were involved in implementing the Constitution. Parliamentary staff described this support as client driven and said PSP experts were engaging and sought the perspectives of each member to improve the quality of bills. One respondent noted that members would have overlooked a lot of things in the bills without this support, particularly given the short, 15- to 30-day window for review. Another said, “There was not one single day when we were disappointed with the expertise brought in.” This group also mentioned PSP’s targeted expert support to female MPs to prepare and present private member bills during this period, adding that women gained courage and learned how to strategize on legislation through this experience. Several respondents also indicated that Parliament does not have the same access to resource persons or flexible systems in place to mobilize external experts that PSP had, noting that Parliament has to follow a lengthy procurement processes to identify experts. Study Tours Parliament, USAID and SUNY highlighted study tours as important for peer-to-peer learning and exposure to best practices. In Phase 3, tour topics included bicameral and presidential systems, research capacity and public outreach. Respondents stated how Parliament has applied knowledge and skills gained from the tours. For example, the director of CPST reported that a 2010 study tour to the U.S. by the clerk of the KNA and head of the research department helped drive the establishment of the CPST. Likewise, PSC leadership reported that benchmarking visits and study tours led the PSC to start holding open debates with citizens in 2014 and the director general of joint parliamentary services said that a lot of what is in place today in Parliament is based on study tour attachments. Interestingly, the 2015 exit survey found that parliamentary staff ranked study tours lower in terms of “best experiences working with the legislature” than they did a few years ago.36 SUNY staff said, “this might point to an overall maturing of the institution where the ‘perks,’ as they were once perceived, are not as desirable and that they see themselves as professionals who perform a necessary and important service to the legislators and the institution.”37 Coordination with DFID In 2013, DFID co-funded PSP. DFID focused on the areas of lawmaking and devolved funding, as well support to the research department and PBO. SUNY, DFID and USAID described the coordination on PSP as smooth and productive. DFID officials said USAID had a big comparative advantage in legislative strengthening that made the partnership attractive to DFID. 36 PSP Exit Survey Findings: Members of Parliament and Staff, June 2015, p.14. 37 Ibid. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 28 Conclusions PSP adapted quickly and effectively to support Parliament’s substantially expanded role under the new Constitution. SUNY proved to be a trusted, neutral player during this sensitive period of transition, promptly sensitizing the Senate to its role in the new, bicameral legislature and effectively building the consensus needed to advance transitional legislation. The program made clear progress toward the achievement of all objectives in Phase 3, despite the strain on activity resources created by the rapid growth of members and staff. There were also corresponding requests for assistance and factors beyond PSP’s control, including the typical growing pains that accompany significant institutional change such as the re-introduction of bicameralism in this case. Achievements in lawmaking and oversight were clearer than for transparency/accountability, which lacked buy in from Parliament, and devolved funding, which PSP had little time to address. PSP’s continued focus on the committee structure and staff capacity were effective approaches for advancing Parliament’s reform agenda in Phase 3. The committee system is entrenched: committees— especially those supported by PSP—are the center of legislative activity and are prioritized by both legislators and parliamentary staff. Parliamentary staff are staying in their positions and are central to the strength, viability and reputation of the institution. The study tours inspired Parliament to introduce key reforms. A combination of program and non-program factors contributed to achievements in Phase 3. The Constitution provided the framework for reform. The mix of people involved in Phase 3—particularly in the Tenth Parliament—was a significant factor for success and all respondent groups were satisfied with PSP. The program had strong consultants, resources, networks and a high degree of flexibility and the capacity to mobilize expertise that Parliament lacks. USAID and DFID coordinated well on PSP. DFID benefitted from USAID’s comparative advantage in legislative strengthening and Parliament gained access to access to new resources and ideas. Question 4 To what extent are the institutional developments and improvements that PSP supported likely to be sustainable over its entire 15 years of programming? Findings Question 4 examines the extent to which the long-term investment by PSP has resulted in sustainable institutions in the Kenyan Parliament. To address this question, the evaluation team was guided by USAID’s definition of sustainability.38 This question considers the people, systems and tools that Parliament has in place, and how PSP contributed to these changes. Planning for and Existence of Systems General Planning PSP exposed activity participants to best practices in legislative strengthening. The PSP 2010 program description contains a plan for sustainability and the transfer of PSP responsibilities in Phase 3.39 This 38 “In the context of USAID-funded programs and projects, the continuation of benefits after major assistance has been completed.” USAID ADS Glossary, April 30, 2014. 39 PSP III 2010 Award Program Description, pp. 33-34. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 29 includes integrating the clerk’s office directly into the planning and execution of program activities and a sequenced handover of the internship/junior fellowship program and staff training to Parliament. A PSP COP noted that in the last three years of PSP, a particular emphasis was on engaging Parliament in program planning, and the director general of Joint Parliamentary Services said, “the plan was to help Parliament take over all of the services provided by SUNY.” While the plan is explicit, the evaluation team did not find clear evidence of how it was carried out. Some confusion surrounded PSP’s ending. SUNY said it was finally clear in February 2015 that the program would end by May. SUNY officials communicated this to Parliament through the clerks and speakers, but felt the message may have been too ambiguous to other members and staff. Specifically, it was difficult to explain that PSP (as an activity) was closing, but that SUNY (as an institution) was continuing through AHADI. As a case in point, the PAC chair in the KNA told the evaluation team that he was no longer seeing much of SUNY in Parliament and thought the program might have stopped, but was not sure. A number of USAID officials reported that neither SUNY nor Parliament took the end of PSP seriously enough, noting that Parliament is well resourced yet did not have a transition plan in place. CPST The CPST was conceived in 2008 and legally constituted in 201140 to provide training to MPs, staff and “others.” The director of CPST reported that the center will update its mandate to explicitly include training for county assemblies, noting that this was not part of the original vision for the CPST since the counties are a creation of the 2010 Constitution. The center was expected to start functioning in 2012, but did not begin in earnest until after the 2013 elections. It has become a main support system for county assemblies, informing the evaluation team that approximately 60 percent of its current efforts focus on counties, where it is conducting weekly programs. PSP collaborated with CPST and the CAF to conduct induction workshops for all county assemblies following the 2013 elections, according to the 2015 final report. PSP supported the center in developing course content and a facilitator’s guide for six training manuals and trained a pool of trainers to deliver training at the national and county level. SUNY reported that the CPST is working to finalize accreditation of these trainers. The evaluation team did not have an opportunity to speak with county participants in CPST training sessions. A member of the county assembly (MCA) for Nairobi, who is a member of the CAF executive committee, knew of the center, but said he was not aware of its activities or its impact on counties. Of the 30 parliamentary respondents and SUNY staff, 20 shared the view that the CPST has not been effective in providing trainings and capacity building to Parliament. Parliamentary respondents noted that the CPST has remained largely “invisible” and is struggling to find its footing. They said the center lacks in￾house capacity and the success of its training programs would depend highly on the ability to mobilize external consultants, which is complicated by stringent procurement rules and government bureaucracy. Some members and staff believe training for Parliament will suffer as a result of PSP’s exit. The CPST director reported that it has the funding it needs, saying, “parliament gives us what we ask for; you ask based on what you can absorb.” She noted that the center is careful when approaching donors to specify what the benefit will be. The director said that AHADI’s approach has been “a bit discouraging,” noting its much stricter bureaucratic procedures than PSP’s, which she did not think was suitable for a political program because of the delays it creates. A senior parliamentary staff member involved in setting up the CPST said that while the operational framework and infrastructure for the center are in place, 40 CPST operates through Legal Notice No. 95 of July 22, 2011, which vests the management of the center in a board that is responsible for its policy direction. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 30 SUNY support should have continued a bit longer (e.g., to identify resource persons). The CPST confirmed that its experts—mostly university professors—are external. Interns/ Junior Fellows Parliamentary respondents expressed a strong desire to continue an internship program and said the PSC has an internship policy modeled on PSP.41 Parliament has continued the “pupilage program,” whereby a half-dozen law school interns work each year in the KNA and Senate legal service and the Litigation and Compliance Directorate of the Parliamentary Joint Services. (Senate officials reported that the first group of Senate interns was retained in September 2015.) The director general for Joint Parliamentary Services and the research director noted that Parliament has held off on the larger internship program for now because of the need to train approximately 240 permanent staff members who have been recruited since 2013. For instance, the PBO director noted a conscious decision not to take on new interns until they integrate their 15 new staff into the office. In addition, Senate staff described two-week “attachment” programs that the Senate is coordinating to bring county assembly staff to Nairobi for capacity building and mentoring. The PBO said it is also considering an internship model for county assemblies. Parliamentary respondents noted that the internship program will be on a smaller scale than PSP and will focus on support to individual directorates, rather than offering interns broad exposure to financial management, governance issues and policymaking. Some respondents cited concerns about Parliament’s ability to coordinate and administer the program. For example, a senior Senate clerk said no framework exists to recruit interns and recent problems have developed in paying them. In Phase 3 of the program, two committee clerks and one committee chair said MPs began influencing the selection of interns. These respondents felt that nepotism has become the norm and that selection on the basis of political connection has affected the previously rigorous standards of the program. Institutionalization of Practices, Functions and Roles The Senate is now an element of the legislative architecture, with its mandate, functions and rules of business enshrined in the Constitution and standing orders, which PSP supported. Parliament’s work is presently governed by the PSC’s strategic plan (2008-2018), which the program helped revise in 2000 and again in 2012. Other sections of this report describe the contributions that PSP made to the new bicameral parliamentary system and devolved system of government. The USAID-funded AHADI activity and other donor efforts are aimed at supporting the next phase of this development, according to donors. More than half of the respondents from Parliament, USAID, DFID and SUNY, as well as expert consultants to the program, indicated that PSP support for the institutionalization of the committee system has enhanced the overall capacity of Parliament. The bulk of legislative work now happens in committees, whereas it used to originate with the executive branch. Key informants felt that PSP support—through capacity-building workshops, international study tours, training for technical staff and facilitation of linkages with local experts who provided technical support and mentorship—is manifested in the committees. Many committees in both houses now have strategic plans and work plans, hold meetings guided by specific agendas, make recommendations that are actionable and time-bound and are able to hold the executive branch accountable for its actions. Informants pointed out that this is a stark difference from the early years. The clerk of the KNA said in those days, Parliament had no knowledge of how committees were supposed to run and used to “look for business.” A former COP echoed this, saying members of committees initially sat around and said, “What do you want to talk about?” Parliamentary respondents felt that this is still a period of transition. For 41 One official said this is a draft policy. The evaluation team did not see it. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 31 example, while the research department is now enshrined in the Constitution and nearly 30 researchers are assigned to committees, a committee clerk commented that Parliament’s ability to review bills has “collapsed” a bit without SUNY, noting Parliament’s difficulty in mobilizing external experts and members’ lack of time and competing interests, such as representation. SUNY highlighted the value of residential seminars, where committee members and experts would travel to a location outside of Nairobi for a weekend program focusing on a particular issue such as draft legislation. SUNY, USAID and the committee chair who spoke on this issue were not certain whether Parliament would retain the model. Both SUNY and the committee chair believed that Parliament could do it, but noted that it would be a slower process because of Parliament’s official procedures. The chair also noted that Parliament has not succeeded in engaging outside experts, so it would need to rely on government facilitators. KEWOPA staff noted PSP support to developing gender-responsive budgeting guidelines, which is defined by the United Nations as “government planning, programming and budgeting that contributes to the advancement of gender equality and the fulfillment of women’s rights.” They reported that this is a key tool that is now being provided to the county assemblies and used by groups such as the International Federation of Women Lawyers (FIDA) Kenya, the International Republican Institute and the National Gender and Equality Commission Kenya to enable women legislators to engage in the budget-making process. KEWOPA reported that it has limited resources to track how the tool is being used. The PBO described its macro-economic model, which enables it to generate its own economic forecasts as the basis for budget, revenue and spending projections, as “another legacy of PSP.” The director noted that the model allows her office to develop budget options that contribute to debate about where to put the money and allows for much more accurate forecasting than other available models. The Constitution mandates new levels of public engagement.42 PSP reporting shows that the KNA and Senate have both taken up the constitutional provision, but it remains a weak area for Parliament. For example, 56 percent of legislators who participated in the 2015 exit survey stated that “the level of public participation in the committee work [the main sphere of legislative deliberation] was poor or very poor.”43 This finding was echoed in parliamentary interviews. For example, a committee clerk and senior PSC official noted that the pubic still treats Parliament as a mysterious place, and said efforts such as Parliament Week and regional interactive sessions with citizens have not gained much traction. The director of the Senate Speaker’s Office and a committee chair reported that Parliament has been too conservative and too generic in sharing information, doing the bare minimum, and said it should explore social media options to promote public conversations. Ability of Systems to Withstand Changes in Personnel PSP’s main support was for, with and around the PSC. By all stakeholder accounts, the PSC is well established; it is independent of the executive branch, has a long-term strategic plan, has a calendar of programs, controls its own budget, has a competitive staff recruitment process and procures its own equipment. Parliamentary respondents, outside experts and SUNY frequently commented that parliamentary staff now consider working in Parliament a prestigious and worthwhile career, and that this is an important factor for high staff retention. These respondents expressed confidence that the positive changes will continue. The speaker of the Tenth Parliament and current clerk of the KNA noted that Parliament has a healthy budget and just needs to exercise fiscal discipline and requires targeted external technical support to effectively implement its legislative agenda. 42 Committees are required to open policy deliberations to the public and ensure consultation between Parliament and external actors. 43 PSP Final Report, Aug. 31, 2015 , p.16. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 32 Parliament, USAID, DFID and SUNY and expert consultants highlighted the enhanced capacity of Parliament because of the institutionalization of the committee system. They said the budget committee has a clear structure, strategy and work plan. They said it has conducted annual public hearings and consultative forums since 2011 in which many CSOs presented reports on the budget.44 SUNY reporting indicates that new members joined the financial committees in the Eleventh Parliament and the budget committee has struggled with having to review both the national and county budget estimates.45 Parliament, expert consultants, USAID, DFID and SUNY identified the PBO as a strong, well-staffed office with excellent leadership. Respondents noted PBO’s role in parliamentary budget making, which is enshrined in the Constitution, and in institutionalizing tools and practices of budget making such as the macro-economic forecasting model and pre- and post-budget workshops first introduced by PSP. A committee chair noted that PBO is helping to build the capacities of committees to carry out their mandates to oversee ministerial budgets. Several informants said changes in PBO personnel would not affect the sustainability of the office, noting the emphasis that the office has placed on staff recruitment and mentorship. One respondent said, “[the director] makes sure that juniors can run without her.” The research department is part of the Directorate of Information and Research Services. Plans are underway to upgrade the department to an independent directorate, where it would be entitled to its own budget. Staff of the research department are now embedded in every committee as part of the committee secretariat, which parliamentary respondents reported as a positive development. More than half of parliamentary respondents identified research as a critical part of effective legislative work. However, the evaluation team heard few examples of the research department providing high-quality information to the committees. One committee chair said the quality of research is “wanting in terms of veracity of facts” and that everyone who needs research does it themselves. As reported, some staff in the research department expressed dissatisfaction about the level of support provided in Phase 3, while SUNY staff said they responded to requests for support, which did not come from the department. Sustainability of Support to CSOs PSP reporting shows that its small grantees, including PIN, actively engaged Parliament through forums, written documentation and testimony to committees. PIN also developed a strategy for strategic engagement with the Eleventh Parliament, though USAID noted that PIN still relied on PSP to convene meetings with Parliament up to the end of the program. A senior parliamentary official said that Parliament and the committees sometimes approach policy-oriented CSOs for support, including the Institute for Economic Affairs on budget matters. They also reach out to Parliament and the International Commission of Jurists (ICJ), Kenya Chapter on legislative drafting, analysis of bills and constitutional issues. However, parliamentary respondents, a PIN member, DFID and SUNY reported that tension and mistrust exist between Parliament and some CSOs that engage in advocacy, threatening their ability to work together. Respondents said this has to do with the adversarial approach of some CSOs. They did not explicitly name any PIN members, but suggested that the general environment for CSO-Parliament collaboration is difficult. As reported under Evaluation Question 1, DFID noted that the situation has worsened since 2013, when CSOs cooperated with the International Criminal Court on investigations into the 2007-2008 post-election violence in the country. Conclusions While SUNY built plans for sustainability into the program design, the execution of the sustainability plan was not well communicated. As a result, some members and staff were not prepared for the end of PSP 44 PSP Kenya Final Report 2015, p.39. 45 PSP Kenya Final Report 2015, p.18. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 33 and continued to expect PSP support. Despite the uncertainty about the program’s end, PSP did help Parliament build sustainable practices and institutions. The PSC is an established body for administering Parliament’s staff and budget. The committee system is entrenched, and committees regularly consider legislation and conduct oversight in accord with international practices. Support agencies like the PBO and the research department are established institutions in Parliament; their staff and budgets are part of the structure of Parliament and their existence does not depend on current personnel. The research department did not benefit from PSP support as much as other departments did, but they integrated their researchers into committee operations. Their plans to make the department an independent directorate will improve its prospects for sustainability. Parliament has a mechanism in place for continued training of members and staff, but the CPST is not well established in Parliament; training will suffer with the end of PSP unless the CPST moves quickly to fill the gap. The greatest issue is the inability of a parliamentary entity to quickly and effectively procure the needed outside expertise because of the government’s poor procurement processes. Similarly, Parliament needs to move quickly to implement its plan for reviving the internship program, to retain momentum and institutional memory. Public outreach and inclusion of public input into the work of Parliament remain a weakness, despite a constitutional requirement. While Parliament is undisputedly more open than it was before PSP, there has been difficulty in finding an effective mechanism for incorporating public input. PSP played an effective role in bringing experts from civil society into Parliament, but the long-term viability of this cooperation is in jeopardy due to deteriorated relations between Parliament and civil society. Question 5 What lessons can be learned from PSP’s 15 years of parliamentary strengthening work? The success of PSP, due in part to its 15-year engagement, holds several lessons for legislative strengthening programs in Kenya and worldwide. The lessons here are derived from evaluators’ analysis of the findings and conclusions for the first four evaluation questions and from interviewees’ suggestions of lessons learned from the program. Collaborative partnership with Parliament is important. A recurrent theme in interviews was the importance of the collaborative relationship between SUNY and Parliament. Parliamentary interviewees regarded SUNY as a partner—as the “other department” of Parliament—not as an outside entity imposing its agenda on Parliament. They saw PSP as responsive to Parliament’s needs; members and staff of Parliament saw themselves as driving the reform agenda, with PSP playing an important supporting role. Having an institutional orientation is important. PSP had an institutional relationship with Parliament, and it gained trust from all corners of Parliament because it was viewed as being for Parliament, not for any faction or interest within Parliament. PSP was also effective because members and staff saw its agenda as the same as Parliament’s own agenda. While PSP on occasion turned its efforts to committees that were pivotal to other USAID development goals such as the Health Committee, it focused on developing the capacity of those committees, rather than a policy agenda. Focus on capacity building of institutions and staff. To gain the trust and support of MPs, PSP necessarily engaged them throughout the program, particularly in residential workshops over a long weekend or on study visits to other legislatures, from which members could derive lessons for their own Parliament. However, the significant turnover of MPs at each election demonstrated that such capacity building was not lasting. Instead, it was programming aimed at developing the key institutions of Parliament, such as the committees, PBO and legal department, which proved most sustainable. Furthermore, staff training has had a lasting effect on the administration of Parliament, particularly as junior staff that benefited from early SUNY support have moved into the most senior leadership positions. When competent staff was needed to lead the administration of the new Senate, a trained cadre of parliamentary staff was already available to fill those positions. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 34 Institutional development facilitates other objectives. A study commissioned by the Center of Excellence on Office of Democracy, Human Rights and Governance (DRG) at USAID found that legislative strengthening programs that have a successful institutional development component are, on average, twice as likely to be successful as programs lacking such a component.46 Furthermore, a successful focus on institutional development correlates with successful efforts to improve oversight, lawmaking, budgeting, research and committees. Both findings held true for PSP, where the initial focus of the program was on supporting the PSC as it developed into an independent parliamentary administration and implemented its own strategic plan for Parliament. Committees are key to improving lawmaking and oversight. The social impact study for the DRG Center at USAID also found that committee strengthening is critical for legislative strengthening programs to improve legislation and oversight. Efforts to improve lawmaking were nearly twice as likely to be successful in programs that strengthened committees than in programs that did not have an effective committee support component. Similarly, programs were three times more likely to improve parliamentary oversight if they had a successful committee component. These two points were confirmed by the Kenya PSP, where a focus on committees was a core objective over the initial years of the program, facilitating achievement of later objectives to improve lawmaking and oversight. Institutional strengthening is long term. Building successful parliamentary institutions requires a time commitment of more than one term. As the experience in Kenya demonstrated, member turnover can be significant at elections and a program needs to introduce itself to new members at the start of each term. The 15-year engagement enabled the program to work with four different Parliaments, providing time for the institutional dimension of the program to take hold and result in lasting institutional reform. Success requires reformers in Parliament. Changing Parliament is not possible unless voters have elected MPs who seek to strengthen the independence and effectiveness of the institution. For outside parliamentary strengthening to be successful, the program’s efforts must complement the inside work of Parliament members and senior staff. PSP successfully identified reformers in its initial years, positioning itself to support their efforts when they were in a position to advance reforms after the 2002 elections. Parliamentary strengthening needs program flexibility. Parliamentary strengthening programs need a collaborative relationship with Parliament to succeed; and this requires that it be flexible in responding to the needs that Parliament identifies, as well as to needs that may arise over the course of a program. PSP demonstrated this flexibility, designing activities in response to parliamentary requests and working with USAID to identify new objectives and tasks in response to political and parliamentary developments. Political programs require sensitivity. Parliament is a political institution; MPs have political goals and constraints that may affect their ability to advance institutional reforms. Legislative strengthening programs must be sensitive to the “political ecology” of Parliament and learn the incentives and interests of MPs. A local COP and staff can build trust and sensitivity. Several members and staff cited the hiring of a Kenyan COP in 2006 as a key factor in PSP gaining the full trust of Parliament, where some individuals previously had suspicions about the true intent of a U.S. program. Over its last decade, SUNY/Kenya had an all-Kenyan staff, ensuring that the program reflected local views throughout. A clear end date must be communicated. While SUNY officials stated they had an exit strategy and communicated it to parliamentary leadership, many members and staff were unaware that PSP was ending in May 2015. In fact, some interviewees said they were unaware that the program had ended. The exit strategy and end date should be communicated to all beneficiaries so they can work to ensure 46 John Lis and Gabrielle Plotkin, Legislative Strengthening Evaluations and Their Implications for Future Programs, Arlington: Social Impact, September 2015, p. 14. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 35 sustainability of key program elements. Joint funding of one program can be more efficient than multiple donor programs. Both USAID and DFID officials cited PSP as a prime example of cooperation between the two agencies in legislative strengthening, providing for greater efficiency and coordination than separate programs. A USAID official noted that a similar USAID-DFID partnership has been established with other legislative strengthening programs in Kyrgyzstan and Malawi. This model merits examination in other international programs. RECOMMENDATIONS PSP's experience suggests a number of recommendations for USAID as a whole to consider when funding legislative strengthening interventions. While USAID/Kenya is not planning a new full-scale national-level parliamentary strengthening activity, the recommendations can be applied to USAID’s work with the country assemblies and targeted work with the senate under AHADI and other related programs. USAID legislative strengthening programming should be designed using collaborative institutional capacity-building approach. PSP illustrates how support for the legislature as an institution can strengthen a nascent democratic system. PSP in Kenya developed a collaborative partnership because it was seen as supporting parliament. Parliament saw the PSP agenda as being the same as its own agenda. USAID’s legislative strengthening programming should include institutional support to legislatures that aims to develop an independent body that serves as a co-equal branch of government in a system of checks and balances. This applies to national and sub-national programming alike. USAID should ensure that legislative strengthening activities focus on capacity building of institutions and staff alike. Because members turn over at each election, programming aimed at developing the key institutions of the legislature such as committees and support agencies proves to be sustainable. Further, staff training had a lasting effect on the administration of the Kenyan Parliament. Institutional development facilitates other objectives, including efforts to improve oversight, lawmaking, budgeting, research and committees. USAID should urge implementers to focus on building strong institutions and staff at the national and sub-national levels. USAID should ensure that legislative strengthening activities strengthen committees. The experience of PSP corresponded with many legislative strengthening activities globally in that its efforts to strengthen Kenyan parliamentary committees were vital to improving lawmaking and oversight. Activities that have improved lawmaking and oversight as objectives should include a committee-strengthening component. USAID should view legislative strengthening as a long-term, flexible endeavor. Building successful parliamentary institutions requires a time commitment of more than one legislative term. As the experience in Kenya demonstrated, member turnover can be significant at elections. As such, a program needs to introduce itself to new members at the start of each term. Building trust that enables reform takes time, and activities need to be flexible in responding to the needs that a legislature identifies, as well as to needs that may arise over the course of an activity. USAID should avoid rigid, short-term activities and adopt a long-term, adaptable perspective for national and sub-national activities. USAID should ensure that implementers have a clear exit strategy and communicate a clear end date. An exit strategy for a legislative strengthening activity should be communicated to all beneficiaries well in advance of a project’s scheduled end date so they can work to ensure sustainability of important activity elements after the project ends. USAID should pursue joint funding of programs with other donors. In light of the successful collaboration between USAID and DFID in jointly funding PSP, USAID should consider joint funding of common development objectives with other foreign aid agencies. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 36 ANNEX 1. EVALUATION STATEMENT OF WORK PURPOSE 1. Background Information 1.1 Identifying Information 1. Program: Democracy, Rights and Governance 2. Project: Parliamentary Strengthening Program (PSP) 3. Award Number: CA. No AID-623-A-10-00019 4. Award Dates: 2000 to 2015 5. Period to be Evaluated: 2000 to 2015 6. Funding: $6.78 million (for the last 5 years of the project) 7. Implementing Organization: State University of New York, Kenya (SUNY/Kenya) 8. Contracting Officer’s Representative (COR): Zeph Aura 1.2 Development Context 1.2.1 Problem or Opportunity Addressed Although initially established as a bicameral legislature immediately following independence in 1963, parliament lost power, in part through a reversion to a unicameral body during Kenya’s long period of authoritarian, single-party rule. Starting in 1999, with the passage of a constitutional amendment, the Kenyan parliament began increasing its structural autonomy from the government’s executive branch, toward the end of re-establishing democratic rule based on separation of powers. One key piece of legislation in this process that was passed in 2000 established the Parliamentary Service Commission (PSC), which was formally constituted in 2002 and is responsible for the Kenyan National Assembly (KNA) budget and management. This bill formally separated the legislative and executive branches of the government47. In 1997, Parliament added to its committee structure by establishing a range of departmental committees that work alongside the pre-existing, but reinvigorated Public Accounts and Public Investments Committees (PAC and PIC), which are in charge of legislative oversight. These committees review legislation relevant to their counterpart ministries. Despite this critical change, the committees were not fully functional until 2001 due to insufficient staff knowledgeable about policy and legislative review, meetings rooms, and MP support.48 Another critical opportunity for reform occurred following the post-2007 election violence, which resulted in an expansion in Parliament’s role and responsibilities for reconciling the country and enacting legislation that was key to constitutional and institutional reforms, including promulgation of a new constitution, which was approved by referendum on August 4, 2010. One key component of the new 47 SUNY/Kenya (2011). Parliamentary Strengthening Program Baseline Survey: Members of Parliament and Staff Report. SUNY/Kenya (2010). Parliamentary Strengthening Program 2005-2010 Final Report. Mathooko, Bonnie M. (2014). “Adapting Research and Library Services in parliament to the Changed Constitutional Framework in Kenya.” Paper presented at 80th IFLA GENERALCONFERENCE AND ASSEMBLY, FRANCE, IFLA Section on Library and Research Services for parliaments' 30th Pre- Conference Parliamentary Libraries: Past and Future, Session: “Research and Library Services Working Together to Meet Client Needs.” 48 USAID/Kenya Parliamentary Strengthening Program (PSP) Original Task Order 2005. SUNY/Kenya (2011). Parliamentary Strengthening Program Baseline Survey: Members of Parliament and Staff Report. US Agency for International Development and UK Department for International Development. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 37 constitution was the establishment of a bicameral legislature including both the Kenyan National Assembly and the Senate. Another critical element was governmental devolution with the establishment of county level assemblies.49 To support the Kenyan Parliament in these reform efforts, the United States Agency for International Development (USAID)/Kenya launched the Parliamentary Strengthening Program (PSP) in 2000, following the passage of the constitutional amendment that increased the National Assembly’s structural autonomy. USAID/Kenya based its PSP partnership with the KNA, which has been implemented by The State University of New York/Center for International Development (SUNY/CID), on Parliament’s 12-year Strategic Plan, which is understood as the blueprint for modernizing, reforming and strengthening Parliament. Thus, the KNA has led the reform process supported by USAID/Kenya.50 1.2.2 Target Areas and Groups Although the KNA is the primary target group for this activity, clearly the project benefits all of Kenya. Moreover, in Phase III, the activity specifically targeted the 47 newly created county assemblies. 1.3 Intended Results USAID’s development hypothesis behind PSP is that enhancing the legislative, oversight and representative capacities of the Kenyan parliament will facilitate the development of a legislature that is democratic, transparent and accountable to citizens.515. To achieve this, PSP focuses on strengthening these three key parliamentary functions, which have remained at the heart of PSP’s activities throughout its 15-year history. The specific goals and objectives of each funding phase have changed slightly as they have adapted to Parliament’s goals, objectives and needs. In Phase I (2000-2004) SUNY activities were expected to: 1. Support Parliament in organizing its reform efforts; 2. Assist Parliament in carrying out its plan for reform; and 3. Conduct activities that strengthen the representation, legislative and oversight functions of Parliament. Building on this the Phase II (2005-2010) strategic objective was: SO 6: Sustainable Reforms and Accountable Governance Strengthened to Improve the Balance of Power among the Institutions of Governance. Achievement of this was supported by the following results: Result 1: More Effective National Assembly Result 2: Stronger Parliamentary Service Commission and Committees 49 Kenya Parliament Magazine April 2013. 50 SUNY/Kenya (2011). Parliamentary Strengthening Program Baseline Survey: Members of Parliament and Staff Report. US Agency for International Development and UK Department for International Development. 51 USAID/Kenya (2010). PSP Award AID-623-A-10-000019, Project Description. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 38 Result 3: Improved Capacity of Parliamentary Staff to Conduct Research and Analysis Result 4: Increased Specialized Budget Expertise within Parliament Result 5: Improved Systems of Oversight of Government Expenditure Result 6: Parliament responds effectively to challenges presented by the 2007-2008 post-election crisis52 Finally, the Phase III (2010-2015) PSP goal was to consolidate the reforms achieved in the previous two phases by ensuring that the Kenyan Parliament performs its representation, oversight and lawmaking functions in a more democratic, effective, and transparent manner. This goal is supported by three objectives:53 Objective 1: To improve effectiveness of the Kenyan Parliament in its lawmaking function Objective 2: To improve effectiveness of the Kenyan Parliament in its oversight function Objective 3: To enhance institutional transparency and accountability 1.4 Approach and Implementation Over a fifteen year period (2000-2015), through PSP, USAID/Kenya has supported the on-going development of the Kenyan Parliament by (1) providing support in organizing its reform efforts; (2) assisting in carrying out its plan for reform, and (3) conducting activities that strengthen Parliament’s representation, legislative and oversight functions. Although specific award activities have evolved over time to keep pace to Parliament’s 12-year plan and changes in Kenya’s democratization process, they have also consistently remained focused on establishing the systems and structures necessary to institutionalizing these changes. Likewise, the project’s approach has remained consistently focused on facilitating the KNA’s own institutional development agenda.54 To accomplish this, PSP focuses on providing technical assistance including training staff and Members of Parliament (MPs), providing legal, policy and sectoral/subject-matter experts, and assisting committee development. In addition, PSP facilitates interaction and engagement between Parliament and civil society organizations on key policy issues. In Phase I (2000-2004) award activities focused on training Parliamentary Committee members and staff, arranging study tours for MPs, connecting MPs to experts who could provide policy related information, linking Parliament with various interest groups, and addressing the need for a Parliament￾wide management information system. In addition, the activity facilitated linkages among the Executive, Parliament and Civil Society Organizations (CSOs) to ensure that the planning and delivery of PSP’s assistance to Parliament was collaborative and based on relevant policy discussions and information sharing.55 Specifically, the main purpose of Phase I was to increase the KNA’s ability to56: 1. Generate sound legislation that would have a positive impact on the economic development and democratization processes in Kenya; 2. Act as an arena for citizens' input into public policy and legislative formation and articulate the concerns of ordinary citizens about important local and national issues; 3. More effectively oversee and monitor the policies and actions of the Executive branch, particularly regarding fiscal management; 52 SUNY/Kenya (2010). USAID Parliamentary Strengthening Program In Kenya Final Report. 53 USAID/Kenya (2010). PSP Award AID-623-A-10-000019, Project Description. 54 Ibid. 55 PSP 2005 Original Task Order. 56 Ibid. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 39 4. Assess its own priorities for immediate and longer term strengthening of Parliament and craft a legislative development plan; 5. Benefit from the legislative experiences of other countries; and 6. Establish links with Civil Society Organizations (CSOs) and other organizations committed to strengthening Parliament. Phase II (2005-2010) was extended for a total of five years, ending on March 31, 2010, through a series of modifications and no-cost extensions, that resulted in a total cost of US$6.78 million. During Phase II the activity maintained the same general approach by supporting committees, increasing Parliament’s capacity for oversight generally, and fiscal oversight specifically, and connecting Parliament more directly to the Kenyan people. In Phase II some key award activities included57: 1. Establishing the Parliamentary Budget Office (PBO); 2. Launching the House Live Broadcast; 3. Training all Committee staff to improve procedures and reporting; 4. Providing expert legislative and policy analysis; 5. Linking Parliament to CSOs through the PIN Network; 6. Supporting revision and implementation of Parliament’s standing orders; 7. Supporting revision and implementation of the KNA Strategic Plan; and 8. Providing a new member orientation program including training on new rules and strategic plan. In total during Phase II, PSP supported over 158 workshops and seminars for over 4,000 MPs and staff, funded and guided 101 interns who supported committees, and produced over 200 reports and manuals related to parliamentary work and institutional reform.58 PSP Phase III (2010-2015), which was extended through a series of eight modifications and no-cost extensions for a total of five years until May 31, 2015. Most of the modifications only provided incremental funding; however, the second modification also specifically identified new tasks associated with supporting County Assembly development. Overall PSP Phase III sought to consolidate the changes brought about in the first two activity phases and also responds to the most recent changes in Kenyan governance, promulgation of the new constitution, which established a bicameral legislature and county assemblies to support devolution. To that end, PSP focused on59: 1. Tailoring legislative/legal and policy analysis to pass constitutionally mandated enabling legislation; 2. Supporting new legislative procedure rules, including supporting the PSC in revising the House Standing Orders; 3. Supporting comparative studies regarding development of bicameral structures and the PSC in developing new rules and administrative structures for support staff and a bicameral parliament; 4. Continuing to strengthen KNA capacity to carry out its regular oversight, legislative and representative functions including technical and mentoring support to committees, issue caucuses, the PSC, and the Parliamentary Training Institute (PTI);60 and 5. Providing orientation to and basic skills for the KNA, Senate, and all 47 County Assemblies. 1.5 Existing Data and Missing Information 57 SUNY/Kenya (2010). USAID Parliamentary Strengthening Program In Kenya Final Report. 58 SUNY/Kenya (2010). USAID Parliamentary Strengthening Program In Kenya Final Report. 59 SUNY/Kenya (2010-2014). PSP Project Annual Reports. 60 USAID/Kenya (2010). PSP Project Award AID-623-A-10-000019-fully executed. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 40 1.5.1 Existing Data 1. PSP 2005 Original Award 2. PSP 2010 Award AID-623-A-10-000019-fully executed 3. PSP (2010-2015) Modifications 1-8 4. Nakamura-Johnson Rising Legislative Assertiveness in Uganda and Kenya 5. PSP 2009-2015 Annual Work Plans 6. PSP PMPs 2010 & 2013-2015 7. PSP Phase III (2010-2015) Indicator Protocols 1-15 8. PSP Complete Survey Report 2011 9. PSP 2005-2010 Final Report 10. PSP 2008-2009 Annual Report 11. PSP 2010, 2011, 2012, 2013 Annual Reports 12. PSP 2009 Q4 Report July-Sept 2009 13. PSP FY2010 Q1 & Q4 Reports 14. PSP FY2011 Q1-Q4 Reports 15. PSP FY2012 Q1-Q4 Reports 16. PSP FY2013 Q1-Q4 Reports 17. PSP FY2014 Q1-Q4 Reports 18. PSP FY2015 Q1-Q2 Reports 1.5.2 Missing Information61 Phase I Documents and Data 1. Original Award including project description and modifications 2. Work plans 3. PMPs 4. Quarterly and Annual reports 5. Final evaluation 6. Any available data (e.g. monitoring data, trainee data, etc.) Phase II Documents and Data 1. Modification documents – The PSP Phase II final report indicated that there had been a number of modifications, but we do not have specific information about them. 2. Work plans - all 3. PMPs - all 4. Quarterly and Annual reports – all except: PSP 2005 MAR-2010 MAR FINAL REPORT PSP 2008 OCT-2009 SEPT Annual Report PSP 2009 Q4 Report July-Sept 2009 PSP 2010 Q1 Report Oct-Dec 2009 5. Any available data (e.g., monitoring data, trainee data, etc.) Phase III Documents and Data 1. PMPs – FY2010-FY2011, FY2011-FY2012, and FY2012-FY2013 2. PSP 2010 Q3 Report APR-JUN 2010 3. PSP 2011 Q3 Report APR-JUN 2011 4. PSP 2013 Q3 Report APR-JUN 2013 5. Any available data (e.g. monitoring data, trainee data, etc.) 61 MSI will need to receive all missing documents and information prior to the start of the evaluation. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 41 6. List of PSP Grantees and activity descriptions SECTION C – STATEMENT OF WORK C.1 BACKGROUND 2. Evaluation Rationale 2.1 Evaluation Purpose The purpose of the PSP evaluation is two-fold. First, USAID would like to understand to what extent the project was successfully implemented and maintained its relevance to the Kenyan Parliament throughout an extended period of transformation and democratization. The evaluation will facilitate USAID’s decision￾making about the design of the next phase of parliamentary strengthening programming. This will not necessarily be a stand-alone activity, such as PSP, and therefore will not necessarily be a “follow-on” activity. Specifically, USAID would like to understand what approaches and models worked best at the national level so they can assess their potential utility in supporting development of County Assemblies. In addition, USAID would like to know what lessons can be learned from PSP that might be applied in the next phase of funding in this sector. Second, USAID would like to know what difference 15-years of funding made to the sector. Despite arguments regarding the benefits of long-term development investments, it is rare for such an extended to investment to be made in any given activity. USAID would like to use this evaluation to learn to what extent, if any, it made in Parliament’s development. This information will inform future decisions about the efficacy of a similar long-term investment approach. 2.2 Audience and Intended Use The key audience for this evaluation is the USAID/Kenya front office and the DO1 technical team specifically, will use this evaluation to inform future programming. The evaluation will be used by the project technical team to inform the design of possible new activities and/or work plan revisions within the AHADI project under this DO. USAID/Washington Democracy and Governance office may also be interested in the broader lesson learned from this evaluation, particularly as they relate to the 15-year investment in parliamentary strengthening. USAID will share the final report with Kenya’s Parliament, specifically the PSC and Speaker. They anticipate this audience using the evaluation results to inform their future planning for Parliament’s development. Findings will also be disseminated through preparation of a one-page factsheet by MSI and USAID arranged stakeholder workshops to USAID/Washington and with other stakeholders. 2.3 Evaluation Questions There are two sets of evaluation questions. The first two only address Phase III, while the last four address PSP’s entire 15-year history. Below are the six evaluation questions, with the evaluation team's interpretation of each. Evaluation Question 1: To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives during the last five years of programming? The evaluation will specifically examine the effectiveness of the adaptive approach the program used, particularly with reference to tactics or approaches to remain non-partisan while working with a highly political institution. Initial analysis will focus on review of PSP monitoring data, which will be triangulated to a limited degree with KIIs/GIs to examine the extent to which PSP achieved its overall objectives. Subsequently, the question will assess the extent to which project activities responded to the dynamics within Parliament, as well as Parliament's (MPs, staff, etc.) impression of, and level of satisfaction with, PSP. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 42 The evaluation will also identify factors that contributed to the success or failure of the program to transform the Kenyan Parliament. In examining the above issues, the evaluation will identify areas in which PSP's approach was most effective, as well as strengths and weaknesses of programming to date and the mechanisms used to achieve results. For example, how effectively did PSP employ international consultants and study tours, as well as their broader institutional resources and networks? How well did PSP coordinate and interact with other international organizations and donors working on parliamentary strengthening? Evaluation Question 2: How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? Explanation: The analysis will examine the extent to which the project’s development hypothesis and design retained their relevance by changing over time to keep pace with changes in the Kenyan context over the course of the project. In addition, this question will examine any assessments and analyses that shaped the development hypothesis and ultimate program design, and will explore whether and to what extent changes in the development hypothesis and activity design incorporated Kenyan views. Analysis will include the extent to which the assumptions that shaped the development hypothesis continue to be valid for 2015 and beyond. The question will examine whether the development hypothesis was clearly defined in program documents and to what extent key players understood and adapted to the changing development hypothesis. Evaluation Question 3: To what extent and in what ways did PSP’s 15-year engagement with the Kenyan Parliament change the project’s ability to achieve its objectives? Explanation: This question will assess what difference continuous, but ever adapting, funding made in the development of a strong, fully functional parliament that operates transparently and democratically. This question will explore whether and how achievements were realized as a result of PSP’s 15-year engagement. It will focus particularly on identifying those that might not have been accomplished in a shorter program. In addition, it will examine how PSP facilitated improvement in parliamentary functions such as lawmaking, oversight, administrative operations, committee effectiveness, public outreach and engagement. Further, it will assess how the democratic performance, political maturity, and effective functioning of the Kenyan Parliament changed as a result of PSP assistance. Finally, to the extent possible, it will assess how satisfied MPs and parliamentary staff are with the program and in which areas they think U.S. assistance was most effective. Evaluation Question 4: To what extent are the institutional developments and improvements that PSP supported likely to be sustainable over its entire 15 years of programming? Explanation: This question will examine the extent to which this long-term investment in parliamentary strengthening is likely to be sustainable in the future. The evaluation will examine the degree to which PSP has planned for, established systems to ensure and is achieving the sustainability of its interventions, especially training. Specifically, the evaluation will assess the extent to which Parliament’s new or revised practices, functions and roles have been institutionalized. Focusing particularly on the Parliamentary Service Commission, Budget Committee, Parliamentary Budget Office, and Parliamentary Research Service, it will explore whether and how the program built systems that can withstand changes in personnel. This question will also examine the sustainability of support to CSOs, particularly PIN members who received funding through PSP to work with parliament. Evaluation Question 5: What lessons can be learned from PSP's 15 years of parliamentary strengthening work? Explanation: This question will take a broad look at PSP as a 15-year initiative and extract overall lessons learned about long-term engagement with the Kenyan Parliament specifically, and development Kenya Parliamentary Strengthening Program: Final Performance Evaluation 43 more generally. The analysis will identify the most important lessons regarding changes in the performance of the Kenyan Parliament. Strengths and weaknesses of this long-term approach to parliamentary strengthening will be a key component of the answer to this question. In particular, to the extent possible, this question will examine DFID’s contribution to PSP and identify lessons on the effectiveness of joint partnership/funding. 3. Evaluation Design and Methodology 3.1 Evaluation Design This is a final performance evaluation aimed at assessing the relevance, effectiveness and sustainability of PSP to inform the design of a related future activity. The evaluation will focus on key committees, caucuses and directorates/offices within Parliament as well as the 47 new County Assemblies, with sample counties purposively selected for inclusion such that, to the extent possible, all major geographic regions of the country are covered. The evaluation questions will be answered using the framework described below and contained in Annex III (Getting to Answers) as a guide to data collection and analysis. The evaluation team will review and refine the methodology as part of the work plan development during the Team Planning Meeting (TPM). 3.2 Data Collection and Analysis Methods Given the nature of the evaluation questions and the stakeholders involved this evaluation uses qualitative data collection and analysis methods. Qualitative methodologies are the most appropriate because they facilitate acquisition of rich detail regarding how and why things happened or did not and triangulation across perspective to assess emerging trends and themes. They allow exploration of the processes behind the results, which is what interests USAID in this evaluation. The specific methodologies that will be used include review and analysis of secondary sources, key informant interviews, and group discussions with MPs, Parliamentary staff, County Assembly members and staff and other relevant stakeholders. For the desk review, the evaluation team will review all activity documentation as provided by MSI/USAID. The evaluators will also be responsible for using other relevant online and print resources to inform answers to evaluation questions. Key secondary sources include, but are not limited to, the following: • Evaluation reports from Phases I and II • Activity Descriptions in Award Documents • Activity modifications • Activity work plans • Quarterly and Annual reports Key Informant Interviews (KIIs) will be conducted with members of the PSC, particularly the Speaker, who chairs the Commission. In addition, the team will conduct KIIs with the leadership and key members of KNA and Senate committees, caucuses and directorates/offices with which PSP worked most closely. Group discussions (GDs) will be held with relevant staff from the same KNA and Senate committees, causes, and directorates/offices. At least three KIIs will be conducted with key members of each committee, caucus, and directorate/office. A preliminary and illustrative list of KIIs and GDs can be found in Table 1. The final interview list may include additional relevant interviewees identified by the evaluators and/or USAID during the detailed work-planning and/or fieldwork phases. Also, depending on interviewee availability, some interviews may not be feasible. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 44 Table 2. Illustrative List of KIIs and GDs KIIs Group Discussions PSC - The Hon. Justin Muturi, Speaker – Chairman, The Hon. Beth Mugo, M.P. - Vice Chair, and 2-4 additional members. Group discussions will be held with key staff for each of the listed committees, caucuses and directorates. Group discussions will also be held with the Steering Committee of the Counties Assemblies Forum, Women County MPs, and County Assembly MPs. Each discussion will have between 3-5 participants. Key Members of the following KNA committees including the Chair, Vice Chair, Clerk and 2-4 additional members: • Budget and Appropriations • Public Accounts • Public Investments • Procedures and House Rules • Education • Health • Local Authorities and Funds Accounts Committee (LAFAC) Key Members of the following Senate committees including the Chair, Vice Chair, clerk, and 2-4 additional members: • Committee on Devolved Government • Health • Labour and Social Welfare Key Members of the following Directorates and other key offices including the Director: • Director General: Mr. Clement M. Nyandiere • Finance & Accounting: Mr. Samuel Obudo. • Information & Research Services: Mr. Paul Ngetich • Parliamentary Budget Office • Centre for Parliamentary Studies & Training Key Members of the following Caucuses and other key stakeholder groups including the Chair, Vice Chair, clerk and 2-4 additional members: • Kenya Women Parliamentary Association Kenya Parliamentary Strengthening Program: Final Performance Evaluation 45 • Kenya Youth Parliamentarians Association • Parliamentary Initiatives Network • Constituencies Development Fund • Other key CSOs UKAID activity manager USAID relevant staff Key staff from implementing partners, particularly former COPs. These may be phone/Skype interviews. MSI will make every endeavor to reach as many key personnel, both former and current, from the various groups listed above, including the LAFAC, which is no longer a functional committee. However, given that the project is now closed and the turnover in both elected officials and staff, this may be challenging. Where the team cannot meet with the identified people, they will make every attempt to meet with an appropriate substitute, however in some instances one may not exist. Information from the desk review, interviews, and group discussions will be integrated using a question by method matrix to facilitate comparisons to identify common trends and themes related to PSP’s relevance, effectiveness and sustainability, successes and challenges as well as suggestions for possible new programming. This analysis (preliminary data analysis) will begin during the fieldwork so the team can assure that they are capturing the information necessary to fully address the evaluation questions. The document review, group discussions and interviews will allow researchers to examine the knowledge, attitudes and perceptions of participants, to identify divergent and convergent trends in their perceptions regarding the issues addressed in the evaluation questions. Table 2 below shows how the data collection and analysis methods map to each question. Table 3: Data Collection Methods and Analysis Methods for the Three Evaluation Questions Evaluation Question 1) To what extent and how effectively did PSP achieve its objectives? Data Collection Methods Secondary Data Committee Observation Key Informant Interviews Group Discussions X X X X Secondary data from activity routine data collection will yield information on PSPs achievements. This data was selected as a source because it is available, relevant, and representative of the activity and provides key information on the activity’s accomplishments. The evaluation team will assess the indicators used to measure improvements in the democratic performance, political maturity and effective functioning of the Kenyan Parliament as a basis for assessing how effectively PSP achieved its results. Understanding these indicators is also critical to the team’s understanding of how PSP defined effectiveness. These secondary data sources documenting the activities implemented to support parliamentary strengthening will also be reviewed to guide development of interview tools aimed at evaluating the processes through which the activity outputs and outcomes were achieved. Finally, to the extent possible within the limits of time and availability, the team will also review media reports about Parliament’s activities relevant to PSP’s work. This will provide an additional window on the extent to Kenya Parliamentary Strengthening Program: Final Performance Evaluation 46 which PSP has achieved its objectives from an “outsider” perspective, which will balance PSP’s own reporting and Parliament’s perspective. KIIs will be conducted with key stakeholders including USAID/Kenya, PSP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, other donors and implementers, and SUNY/Albany staff. Members and staff of Parliament worked directly with PSP through a combination of activities. They have had critical experiences with PSP as primary targets of its interventions. They will, therefore, have relevant information and insights on any changes in parliamentary operations with respect to lawmaking, oversight and representation. Both USAID and the IP had the initial concept of what the activity intended to achieve through parliamentary strengthening. Both, but especially SUNY/Kenya, were also involved in implementation and regularly monitoring the activity’s performance. They also are, therefore, critical informants about the activity’s accomplishments including any challenges, achievements, successes, etc. KIIs with the leadership of both KEWOPA and KYPA will provide critical insight on the extent to which PSP facilitated increased support to women and young parliamentarians. In addition, they will provide a unique perspective on the broader issues addressed by this question regarding PSP’s effectiveness. Similarly, KIIs with leaders of the Senate Committee on Devolved Government will provide insight on PSP’s efforts to strengthen linkages between national and sub-national governance. The KIIs will yield narrative content, which will provide in-depth insight to identify key trends and themes. The information generated through these KIIs will be triangulated with other sources to arrive at the conclusions and recommendations addressing this research question. GDs with parliamentary staff who have worked on committees, caucuses, and directorates/offices are knowledgeable and comfortable enough with each other so they carry on a conversation guided by the evaluation team. These GDs will allow for deep insight into the knowledge and perceptions of staff about the changes in Parliamentary operations resulting from the project, but at a lower cost in terms of both time and financial resources than the KIIs. Likewise group discussions with staff related to the KEWOPA and KYPA will provide a critical counter point to the KIIs on the extent to which PSP facilitated support for women and youth within Parliament. GDS with the Steering Committee of the Counties Assemblies Forum, Women County MPs, and County Assembly MPs will provide a similar counter perspective on the issues related to PSP’s efforts to strengthen national and sub-national governance linkages. Finally, GDs with CSOs, journalists and academics who regularly interact with parliament in their work will provide another set of perspectives to triangulate against those of members of Parliament and staff. These “outsider” perspectives are critical for balance in the evaluation. As with the KIIs, the GDs will yield rich textual data through which key trends and themes can be identified. This information will be triangulated with other sources, particularly the KII data to assess key points of convergence and divergence in perspective among the various participants. Finally, to the extent possible within the limits of time and access granted by Parliament, team members will observe a random selection of committees in action. Using a pre-designed protocol, they will be non-participant observers, taking notes on how the committee conducts its business. This will provide further insight to the effectiveness of PSP’s achievements. Data Analysis Methods Content Analysis Comparison Mixed Method data integration X X X Qualitative data analysis begins with note writing. Each interview will be written up so that it can be shared among team members so that everyone has as complete a picture as possible of all the information obtained by the team. This is important as the evaluation team will work as two separate teams in collecting data. In other words, two team members will be present at each interview so that two interviews can be conducted simultaneously. In addition, to note writing, the team will hold Kenya Parliamentary Strengthening Program: Final Performance Evaluation 47 debriefings at the end of each data collection day during which they will begin to identify common themes that will be used later for coding the collected data during the formal data analysis process. After the data collection is completed, beginning with the list of themes identified during the fieldwork, the team will conduct an open coding process to identify key themes and issues that emerged in the interviews. Once the list of themes has been generated, working independently to enhance researcher triangulation, each team of two will then code the interviews collected by the other team. This process entails identifying what themes and issues emerged in each interview. Once both team members have completed their independent coding, they will compare coding and discuss and come to consensus about how to deal with any differences in perspective. Through this process the team will develop a solid understanding of the various perspectives that emerged among different stakeholder groups with respect to what extent and how effectively PSP achieved its objectives. Of particular importance is identifying the key themes and issues that emerged so as to ensure that the evaluation findings do not rest on the perspectives of one or two people. Nonetheless, divergent and unique perspectives will be investigated to assess whether or not they merit consideration in the analysis. If so, the team will specify why. The team will use a similar coding and analysis process for the content analysis of all secondary sources including any media reports identified. Once the key themes and issues that emerged in the various sets of interviews, documents, and media reports are identified, the team will then analyze these data. Parallel analysis, wherein the perspectives of various stakeholder groups are compared, will be used to analyze the evidence from interviews. These data will also be compared with data from the content analysis based on the document and media report review. In this analytical approach, each type of data for an activity is analyzed in parallel, and then across data type. Put another way, this approach facilitates both within case (each stakeholder group and secondary data source) and between case comparisons. In this way, key pieces of evidence from the various interviews and documents are compared and triangulated to identify the main evaluation findings that respond to the question. Evaluation Question 2) To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives? Data Collection Methods Secondary Data Key Informant Interviews Group Discussions X X X Work plans and quarterly, annual, and evaluation reports will also provide insight into the approach PSP used to achieve its results. Any changes and adaptations in approach will be particularly important to understand, particularly why they were made. This information is critical to understanding how PSP set out to achieve its objectives, which is fundamental to assessing to what extent and how effectively PSP’s adaptive approach facilitated achieving its objectives. These secondary data sources documenting how PSP implemented its interventions in support of parliamentary strengthening will also be reviewed to guide development of interview tools aimed at evaluating the effectiveness of PSP’s approach. KIIs will be conducted with key stakeholders including USAID/Kenya, PSP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, other donors and implementers, and SUNY/Albany staff. Members and staff of Parliament worked directly with PSP through a combination of activities. They have had critical experiences with PSP as primary targets of its interventions. They will, therefore, have relevant information and insights on the approach used and the extent to which SUNY/Kenya was perceived as non-partisan. These KIIs are also critical to Kenya Parliamentary Strengthening Program: Final Performance Evaluation 48 understanding the extent to which MPs and parliamentary staff are satisfied with the PSP approach. Moreover, they can provide insight on the factors that may have contributed to the success and/or failure of the program’s approach to transforming the Kenyan Parliament. These participants will also be asked about strengths and weaknesses in the PSP approach. Of particular interest will be the mechanism used in the approach (e.g. training, international and local experts, study tours, etc.). Likewise, these KIIs will investigate how well PSP coordinated and interacted with other donors and international organizations. Both USAID and the IP had the initial concept of what the activity intended to achieve as well as ideas about how best to accomplish them. They also are, therefore, critical informants about the activity’s implementation process including any modifications to the initial design, challenges, achievements, successes, etc. While this question does not ask specifically about the perspectives of KEWOPA and KYPA members, KIIs with their leadership and staff will provide critical insight on how the PSP approach facilitated increased support women and young parliamentarians. The KIIs will yield narrative content, which will provide in-depth insight to identify key trends and themes. The information generated through these KIIs will be triangulated with other sources to arrive at the conclusions and recommendations addressing this research question. As with question 1, GDs with parliamentary staff will enable deeper insight into their knowledge and perceptions about the PSP approach. Likewise group discussions with staff related to the KEWOPA and KYPA will provide a critical counter point to the KIIs. Finally, GDs with CSOs, journalists and academics who regularly interact with parliament in their work will provide another set of perspectives to triangulate against those of members of Parliament and staff. These “outsider” perspectives are critical for off-setting any bias. As with the KIIs, the GDs will yield rich textual data through which key trends and themes can be identified. This information will be triangulated with other sources, particularly the KII data to assess key points of convergence and divergence in perspective among the various participants. Data Analysis Methods Content Analysis Comparison Mixed Method data integration X X X Data analysis for this question will follow the same procedures detailed above. Evaluation Question 3) How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? Data Collection Methods Secondary Data Key Informant Interviews Group Discussions X X X Work plans and quarterly, annual, and especially evaluation reports from all three PSP phases will provide critical insight into how PSP adapted to the changing political circumstances in Kenya. Of particular importance to answering this question, are the three original project design documents and any modifications that addressed changes in approach or scope of work (SOW). This information is critical to understanding how PSP evolved over time and how effectively its adaptations addressed the changing circumstances. The document will also be critical to examining whether the development hypothesis was clearly defined in program documents. The team will also review these secondary data sources documenting the original design and modifications to each PSP Phase to guide development of interview tools aimed at evaluating the effectiveness of PSP’s adaptations. KIIs will be conducted with key stakeholders including USAID/Kenya, PSP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, other donors and implementers, and SUNY/Albany staff. Evaluators will also interview former MPs involved in PSP Phases I and II, USAID and SUNY staff who worked with the program in the past, either in person or by telephone. USAID and SUNY/Kenya staff who were involved in the project in all its iterations are especially critical to answering this question because they can speak to how PSP’s development hypothesis and design changed over time. In addition, these KIIs will provide information about any assessments and analyses Kenya Parliamentary Strengthening Program: Final Performance Evaluation 49 that shaped the development hypothesis and ultimate program design. Finally, USAID and SUNY/Kenya will provide insight about to what extent key players understood and adapted to the changing development hypothesis. While Members and staff of Parliament may not be able to speak directly to changes in PSP’s development hypothesis, they worked directly with PSP in each of its iterations and therefore have firsthand knowledge of its design and implementation. This experience and knowledge of how PSP’s design evolved over time enables them to effectively address questions about the extent to which PSP kept pace with changes in the Kenyan political context. Moreover, they will provide information on whether and to what extent changes in the development hypothesis and activity design incorporated Kenyan views. The KIIs will yield narrative content, which will provide in-depth insight to identify key trends and themes. The information generated through these KIIs will be triangulated with other sources to arrive at the conclusions and recommendations addressing this research question. GDs with parliamentary staff will enable deeper insight into their knowledge and understanding of the PSP development hypothesis and the extent to which it adapted to changing Kenyan political circumstances. While this question does not specifically ask for perspectives on women, GDS with KEWOPA leadership and staff will provide critical insight regarding the extent to which the most recent PSP development hypothesis accounted for recent changes in the Kenyan constitution with respect to the promotion of female empowerment and inclusion of women. Finally, GDs with CSOs, journalists and academics who regularly interact with parliament in their work will provide another set of perspectives to triangulate against those of members of Parliament and staff. These “outsider” perspectives are critical for off-setting any bias. As with the KIIs, the GDs will yield rich textual data through which key trends and themes can be identified. This information will be triangulated with other sources, particularly the KII data to assess key points of convergence and divergence in perspective among the various participants. Data Analysis Methods Content Analysis Comparison Mixed Method Data Integration X X X Data analysis for this question will follow the same procedures detailed above. Analysis will particularly aim to identify the assumptions that shaped the PSP development hypothesis and then assess the extent to which they continue to be valid for 2015 and beyond. Evaluation Question 4) To what extent and in what ways did PSP’s 15-year engagement with the Kenyan Parliament change the project’s ability to achieve its objectives? Data Collection Methods Key Informant Interviews Group Discussions X X KIIs will be conducted with key stakeholders including USAID/Kenya, PSP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, other donors and implementers, and SUNY/Albany staff. KIIs with MPs and parliamentary staff involved in PSP Phases I and II, and USAID and SUNY staff who worked with the program in the past will be particularly important respondents for addressing this question because they will have a longer term historical perspective. Nonetheless MPs, parliamentary staff, USAID/Kenya and SUNY/Kenya staff involved in the project more recently will also be important because they should be able to speak to how and to what extent Phase III built on the other two. These interviews will assess what difference continuous, but ever adapting, funding made in the development of a strong, fully functional parliament that operates transparently and democratically. Further, they will explore whether and how achievements were realized as a result of PSP’s 15-year engagement. They will focus particularly on identifying those that might not have been accomplished in a Kenya Parliamentary Strengthening Program: Final Performance Evaluation 50 shorter program. In addition, they will examine how over its 15-year history PSP facilitated improvement in parliamentary functions such as lawmaking, oversight, administrative operations, committee effectiveness, public outreach and engagement. Further, KIIs will provide insight on how the democratic performance, political maturity, and effective functioning of the Kenyan Parliament changed as a result of PSP assistance. The KIIs will yield narrative content, which will provide in-depth insight to identify key trends and themes. The information generated through these KIIs will be triangulated with other sources to arrive at the conclusions and recommendations addressing this research question. GDs with parliamentary staff will enable deeper insight into their perceptions about the difference PSP’s 15-year engagement made in the development of the Kenyan Parliament. Given that staff may be less likely to turnover regularly than MPs, they are more likely to have a long term perspective. In addition, the GDs will also ask how satisfied parliamentary staff were with the program and in which areas they think PSP was most effective. Finally, GDs with CSOs, journalists and academics who regularly interact with parliament in their work will provide another set of perspectives to triangulate against those of members of Parliament and staff. These “outsider” perspectives are critical for off￾setting any bias. As with the KIIs, the GDs will yield rich textual data through which key trends and themes can be identified. This information will be triangulated with other sources, particularly the KII data to assess key points of convergence and divergence in perspective among the various participants. Data Analysis Methods Content Analysis Comparison Mixed Method Data Integration X X X Data analysis for this question will follow the same procedures detailed above. Analysis will aim to clearly identify how exactly USAID’s 15-year investment in Kenyan Parliamentary reform facilitated the reform process. However, it is important to note that given qualitative approach it will only be possible to establish some correlation and not causality between PSP’s work over the 15- year period and changes in the Kenyan government. Evaluation Question 5) To what extent are the institutional developments and improvements that PSP supported likely to be sustainable? Data Collection Methods Secondary Data Key Informant Interviews Group Discussions X X X Routinely collected monitoring data over PSP’s 15-year will yield critical information on the extent to which PSPs achievements are sustainable. The evaluation team will assess the indicators used to measure improvements in the democratic performance, political maturity and effective functioning of the Kenyan Parliament as a basis for assessing to what extent PSP’s results are likely to be sustainable. Secondary sources documenting the activities implemented over PSP’s 15-year history to support parliamentary strengthening will also be reviewed to guide development of interview tools aimed at the sustainability of PSP’s interventions. In addition, quarterly and annual reports from all three PSP phases will be examined comparatively against relevant Parliamentary reports (e.g. PBO quarterly report) to assess the extent to which PSP interventions are manifesting in Parliament’s on-going activities. KIIs will be conducted with key stakeholders including USAID/Kenya, PSP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, other donors and implementers, and SUNY/Albany staff. Evaluators will also interview former MPs involved in PSP Phases I and II, USAID and SUNY staff who worked with the program in the past, either in person or by telephone. USAID and SUNY/Kenya staff who were involved in the project in all its iterations are especially critical to answering this question because they can speak to how PSP’s interventions over the past 15 years have manifested in specific changes within Parliament. These informants will specifically be asked about the degree to which PSP has planned for, established systems to ensure and achieved the sustainability of its interventions, especially training. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 51 These KIIs with MPs and parliamentary staff will examine the extent to which this long-term investment in parliamentary strengthening has yielded clear sustainable transformations in Parliament’s approach to lawmaking, oversight and representation. Specifically, they will be asked to speak about how exactly Parliament’s new or revised practices, functions and roles have been institutionalized. KIIs for this question will focus particularly on the Parliamentary Service Commission, Budget Committee, Parliamentary Budget Office, and Parliamentary Research Service and will explore whether and how PSP built systems that can withstand changes in personnel. They will also be asked about they are using the training they have received in their daily work. The KIIs will yield narrative content, which will provide in-depth insight to identify key trends and themes. The information generated through these KIIs will be triangulated with other sources to arrive at the conclusions and recommendations addressing this research question. GDs with parliamentary staff working for the committees and offices mentioned above will also be asked about how they are using their training and the systems that have been established to ensure institutionalization of the new and revised practices, functions and roles such that these changes can withstand personnel turnover. Finally, as with the other questions, GDs with CSOs, journalists and academics who regularly interact with parliament in their work will provide another set of perspectives to triangulate against those of members of Parliament and staff. These “outsider” perspectives are critical for off-setting any bias. As with the KIIs, the GDs will yield rich textual data through which key trends and themes can be identified. This information will be triangulated with other sources, particularly the KII data to assess key points of convergence and divergence in perspective among the various participants. Data Analysis Methods Content Analysis Comparison Mixed Method Data Integration X X X Data analysis for this question will follow the same procedures detailed above. Analysis will aim to clearly identify the specific ways in which the changes brought about through PSP’s interventions have been institutionalized and systematized such that they can withstand changes in both personnel and administrations. In other words, the analysis to specifically assess the extent to which PSP’s interventions are sufficiently durable to withstand changes in political party ensuring the sustainability of Kenya’s democratic reforms. Evaluation Question 6) What lessons can be learned from PSP's 15 years of parliamentary strengthening work? Data Collection Methods Key Informant Interviews Group Discussions X X The response to this question will draw on the data collected to address the other five questions that comprise this evaluation. In addition, the evaluation team will specifically ask all KIIs and GDs about lessons learned, the activity’s strengths and weaknesses in design and implementation and their suggestions for future programming. These data will be combined with all the other information gathered to form the basis upon which to identify the lessons learned. Data Analysis Methods Content Analysis Comparison Mixed Method Data Integration X X X Data analysis for this question will follow the same procedures detailed above. Analysis will aim to clearly identify specific and recurring lessons that emerge from all the information gathered for this evaluation and the analyses done for the other five questions. That data will be triangulated with analysis of questions specifically addressing the stakeholder’s views of lessons learned, suggestions for future programming and PSP’s strengths and weaknesses. Only those lessons that recur most frequently and that provide a clear direction for future programming will be reported. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 52 3.3 Gender As per Automated Directives System (ADS) 203.3.1.5, gender will be considered when looking at all questions. Records on applicants and selected scholars will be disaggregated by sex, and survey results also disaggregated by sex. All evaluation questions will include data disaggregated by sex, and examine the gender differential in access as well as any differential in results and/or benefits. Table 4: Evaluation Questions, Gender Data and Differentials62 Evaluation Questions Sex Disaggregate d Data Gender Specific/Differential Effects: Access and Participation Gender Specific/Differential Effects: Results and Benefits To what extent and how effectively did PSP achieve its objectives?  Yes training data only o PSP's specific support to KEWOPA as well as more broadly to staffers, committees and members of parliament to facilitate women’s participation in Parliament. o Percentage of women who participated in trainings To what extent and how effectively did PSP’s adaptive approach facilitate achieving its objectives?  Yes training data only o PSP's specific support to KEWOPA as well as more broadly to staffers, committees and members of parliament to facilitate women’s participation in Parliament. o Percentage of women who participated in trainings How effectively did PSP adapt to changing circumstances affecting its interventions over the course of 15 years? N/A o Extent to which PSP’s development hypothesis accounted for recent changes in the Kenyan constitution with respect to the promotion of female empowerment and inclusion of women. N/A To what extent and in what ways did PSP’s 15- N/A N/A N/A 62 ADS 203.3.1.5 says, "Identify all evaluation questions for which gender-disaggregated data are expected; also identify questions for which an examination of gender specific or gender differential effects are expected." Kenya Parliamentary Strengthening Program: Final Performance Evaluation 53 year engagement with the Kenyan Parliament change the project’s ability to achieve its objectives? To what extent are the institutional developments and improvements that PSP supported likely to be sustainable? N/A o Extent to which PSP's efforts over 15 years have facilitated sustainable women’s participation in Parliament. What lessons can be learned from PSP’s 15 years of parliamentary strengthening work? N/A o Specific lessons about increasing women’s participation in parliament. In addition to examining gender issues across all the evaluation questions, the evaluation team will also assess the extent to which PSP promoted the inclusion of youth and youth issues as they pertain to all the evaluation questions. 3.4 Methodological Strengths and Limitations As with any assessment or evaluation, there are biases and other limitations that must be addressed through methodological or analytical methods. An evaluation of parliament-related activities by multiple implementers, with multiple sources of funding, which may impact a future activity design is subject to many of these issues: First, recall bias may be present, such as parliament staff responding to team questions with answers related to one parliament-related program or another, or those of another donor. A similar problem is that participants in multiple training activities may be blending their experiences into a composite memory or response (e.g., staff received training on several topics both before and during the evaluation period and subsequently do not distinguish between them as separate activities in their responses). This will be less of a problem in this evaluation because the team will be soliciting responses for both the 2011-2015 period and for the overall duration of the program. However, given the extended period being evaluated, respondents may have general problems recalling activity interventions and their results. To mitigate this, the evaluation team will make every effort to take respondents to the specific time period by citing major events that happened at the time and by making the questions as specific as possible regarding the interventions under discussion. These approaches are generally effective in increasing the quality of recall data. Second, response bias is a common problem for program evaluations and assessments, particularly for highly technical DRG subsectors such as parliamentary strengthening. For example, MPs may give the interviewer positive remarks about an activity like exchange trips because s/he would like to go on more such trips in the future. The team fully expects that key points of contact, trainees, and partner organizations may understand that this assessment will shape future project opportunities. Therefore, the evaluation design includes as wide a range of respondents as may reasonably have valuable information to contribute on the issues under investigation. By have a wide and diverse respondent pool, the possibilities of response bias will be mitigated. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 54 Third, selection bias in the form of contacts provided by the implementers can mean that the team only hears from people with positive experiences. Again, this is particularly a problem for parliamentary strengthening. Again, a wide and diverse respondent pool including both MPs and staff will offset this. In addition, the team will obtain information from non-project sources, particularly USAID, about key individuals to include. The most effective approach to combating any and all of these biases is to use data multiple sources to triangulate on an assessment issue. By combining information found in documents or interviews from multiple sources, any one piece of biased data would not skew the analysis. Another approach that pertains specifically to interviews is the inclusion of key informants from organizations that do not directly benefit from the assessed program(s), and the use of questions about specific examples of knowledge use. For this evaluation, “outsider” perspectives in the form of journalists, CSOs, etc. will provide an addition counterbalance. The team’s biggest concern is about the availability of contacts. Interviews with MPs and senior staff may be difficult to schedule because of existing demands on their time or the need to accommodate last-minute scheduling changes. If parliament is not in session during the time period of the assessment, MPs may return to their home districts, which would preclude in-person interviews in Nairobi. In contrast, if Parliament is in session, it may be difficult to schedule meetings with Parliamentary staff. MSI, therefore, has scheduled the data collection period when it can bridge both when parliament is in session and when it is not. 4. Evaluation Products 4.1 Deliverables The following dates are illustrative as they are based on a start date of September 14, 2015. Task Order approval on or before August 31, 2015 is important for a timely and successful implementation. Deliverable Responsible Party Date Initial meeting with USAID to discuss expectations, review evaluation questions, and answer any specific questions. MSI/DRG/COR September 15, 2015 Work plan submitted to USAID, including detailed methodologies for each evaluation question and precisely what elements of each project are to be examined. MSI September 18, 2015 Meeting with USAID on Work Plan where agreement is reached and provisional approval provided (perhaps with articulated changes). MSI/DRG /COR September 21, 2015 USAID approves Work Plan DRG September 22, 2015 Dates for key informant interviews with USAID/DRG staff. MSI/DRG September 23, 2015 Mid-point check in MSI October 2015 A half-day (morning) validation workshop with all partners and USAID MSI/DRG October 29, 2015 Weekly reports at the end of each week of data collection MSI Weekly for 3 weeks of fieldwork; Sept 25, 2, 9 October, 2015 Presentation of findings to USAID MSI 2 November, 2015 Draft Report submitted MSI 23 November, 2015 Comments from USAID and IP on Draft Report USAID 30 November, 2015 Final Report MSI December 14, 2015 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 55 USAID approval of final report and notification to MSI if statements of differences are expected USAID 4 January, 2015 Statement of Differences sent to MSI if applicable 18 January, 2016 MSI will incorporate any statements of differences if applicable MSI 25 January, 2016 USAID approval for DEC submission USAID 1 February, 2016 One-page fact sheet about the evaluation MSI 8 February, 2016 Approval of one-page fact sheet USAID 15 February, 2016 A detailed breakdown of the process is listed below: Week 1 Desk Review In order to initiate data collection, the evaluation team will review all the documents from their home base. These initial findings will be presented to MSI during the Team Planning Meeting and will be used to inform tool development. The evaluation team/USAID/partners are also expected to provide feedback on the tools during this period. Week 2 Team Planning Meeting (TPM) The TPM will be held in MSI offices once the evaluation team is in country. It is expected that the team will have the initial meeting with USAID (Day 2 of Week 2) to discuss expectations, review evaluation questions, and answer any specific questions. The outcomes of the team planning include: • Presentation of the initial findings of the document review by evaluation question (MSI-only); • Clear understanding of TOC model for the evaluation; • Clarification of team members' roles and responsibilities; • Establishment a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion; • Review of the final evaluation questions; • Review and finalization of the assignment timeline and share with USAID; • Development of data collection and analysis methods, instruments, tools, and guidelines; • Review and clarification of any logistical and administrative procedures for the assignment; • Development of a preliminary draft outline of the team's report; and • Assignment of drafting responsibilities for the final report. Week 3 Workplan and Methodology During the TPM, the team will prepare a detailed work plan, which will include the methodologies (evaluation design, tools) and operational workplan to be used in the evaluation. This will be submitted to USAID on Day 5 of Week 2 (COB). The team will meet with USAID on Day 1 of Week 3 for the Work Plan Review Meeting, to discuss the methodology and get approval prior to implementation. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 56 Weeks 3 through 6 Data Collection and Updates on Progress: MSI will present weekly reports by email to USAID starting at the end of the first week of data collection and continuing through the end of week 6, the end of data collection and the beginning of analysis. The report will discuss ongoing activities during the course of the evaluation describing the process, any issues encountered, and relevant emerging findings. Week 7-8 Data Analysis, Validation Meeting and Presentation Prep: A half-day meeting (morning) with all partners and USAID to validate and discuss findings, answer/clarify any data gaps; and discuss feasibility of potential recommendations. Week 9-11 Presentation with USAID/DRG and Partners and report writing/editing: The evaluation team will present the major findings of the evaluation to USAID in a PowerPoint presentation. The presentation will follow a similar structure to the final report and present major findings, conclusions, and recommendations. USAID will have an opportunity to comment and provide input/feedback as part of the presentation. These comments will be incorporated into the draft report, as appropriate. Week 12-13 Draft Evaluation Report and USAID comments: The written report should clearly describe findings, conclusions, and recommendations, fully supported by triangulated evidence. USAID will provide comments on the draft report within two weeks of submission. Week 14-15 Responding to USAID comments Week 16-18 Final Evaluation Report and USAID approval: The team will submit the final report that incorporates the team responses to Mission comments and suggestions. The format will adhere to the standard reporting guidelines listed in 4.2. USAID has one week thereafter for approval. If there are some outstanding questions, MSI will attempt to answer/incorporate them into the report as appropriate. Otherwise, USAID can consider a Statement of Differences. 4.2 Reporting Guidelines The evaluation report will adhere to USAID Evaluation Policy (including Appendix 1). The format for the evaluation report shall be as follows. The report should be a maximum of 30 pages not including the cover page, table of contents, executive summary, acronyms list, and glossary of terms or annexes. The report format should be in English and restricted to Microsoft products. In accordance with USAID's Evaluation Report Template, it should use USAID fonts: Gill Sans or Gill Sans MT (bold for headlines, subheads and highlighted text; regular or light for body text; italic for captions), or Garamond or Ariel if Gill Sans is not available. An electronic copy in MS Word shall be submitted. If the report contains any potentially procurement sensitive information, a second version of the report excluding this information shall be submitted (also electronically, in English). Below represents a guideline for the report structure: a. Table of Contents (1 pg.); b. Executive Summary—concisely state the most salient findings and recommendations (3-4 pg.); c. Evaluation Purpose and Evaluation Questions—purpose, audience, and synopsis of task (1 pg.); Kenya Parliamentary Strengthening Program: Final Performance Evaluation 57 d. Project Background—brief overview of development problem, USAID project strategy and activities implemented to address the problem, and purpose of the evaluation (1-3 pg.); e. Evaluation Design, Methods, Limitations—describe evaluation methods, including constraints and gaps (1-3 pg.); f. Findings/Conclusions/Recommendations—for each evaluation question (15-25 pp); g. Annexes that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. These include references to bibliographical documentation, meetings, interviews and group discussions. 5. Team Composition The evaluation team will be composed of four evaluators, two local and two expat. The team lead is an evaluator who will have ultimate responsibility for the report and will guide the team throughout the evaluation process. Research demonstrates that the quality of evaluation reports is significantly enhanced if an evaluation team lead is primarily an evaluator and secondarily possesses sector expertise. The expat sector expert will provide critical DG expertise generally and on parliamentary strengthening specifically. The team will also include two local sector experts who have experience in Kenya’s DG sector, particularly parliamentary strengthening. This team will do all the data collection themselves working in two teams of two (1 expat and 1 local). This will facilitate conducting two interviews simultaneously so that more data can be collected. In addition, given that PSP is now ended and therefore not available to provide assistance in making contacts and setting up meetings, MSI will hire a former PSP staff member who knows Parliament very well and has all the necessary connections to facilitate setting up meetings. She will only work as liaison in setting up meetings and will not participate in any way in the evaluation data collection and analysis. CVs for personnel can be found in Annex VI. The following qualifications are sought for the evaluators: International Evaluation Team Leader • Advanced degree in Political Science, International Development, Governance, and/or related social science field • Expert knowledge and at least 7year of experience in the Democracy and Governance sector, preferably in parliamentary strengthening • Expert knowledge and at least 7 years of experience in conducting (preferably leading) external performance evaluations • Prior Kenyan and/or East African regional experience a plus • Prior USAID evaluation experience; evaluation design experience, including the selection of appropriate data collection methods on a question specific basis and development of a detailed data analysis plan; basic social science research skills including basic statistics, strong data visualization skills consistent with USAID expectations; familiarity with and ability to apply a range of data collection methodologies covered by USAID TIPS. Field experience that includes at least some, but not necessarily all of the following: o Qualitative research including design of a tools, specifically development of structured and semi-structured interview and/or observation instruments o Qualitative research experience, particularly in conducting structured and semi-structured individual and group interviews and/or focus groups, including the creation of written instruments and transcripts for same; content analysis and other techniques for coding and transforming group/open-ended data into analyzed information; qualitative data analysis software (MAXQDA or alternative) and/or case study documentation experience; transformation of qualitative data into a quantitative form that can be merged with other quantitative data in a mixed methods analysis. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 58 • Superior writing ability, including evidence of an ability to structure evaluation reports in-line with USAID expectations and in a way that logically and transparently lays out empirical findings, conclusions, and recommendations. International Sector Expert • Advanced degree in Political Science, International Development, Governance, and/or related social science field • Expert knowledge and at least 10 years of experience in the Democracy and Governance sector, preferably in parliamentary strengthening • Some knowledge and experience in conducting external performance evaluations preferred • Prior Kenyan and/or East African regional experience a plus • Prior USAID evaluation experience. • Superior writing ability, including evidence of an ability to structure evaluation reports in-line with USAID expectations and in a way that logically and transparently lays out empirical findings, conclusions, and recommendations. Local Sector Expert • Advanced degree in Political Science, International Development, Governance, and/or related social science field • Expert knowledge and at least 10 years of experience in the Democracy and Governance sector, preferably in parliamentary strengthening • Expert knowledge and at least 7 years of experience in conducting external performance evaluations • Prior USAID evaluation experience; evaluation design experience, including the selection of appropriate data collection methods on a question specific basis and development of a detailed data analysis plan; basic social science research skills including basic statistics, strong data visualization skills consistent with USAID expectations; familiarity with and ability to apply a range of data collection methodologies covered by USAID TIPS. Field experience that includes at least some, but not necessarily all of the following: o Qualitative research including design of a tools, specifically development of structured and semi-structured interview and/or observation instruments o Qualitative research experience, particularly in conducting structured and semi-structured individual and group interviews and/or focus groups, including the creation of written instruments and transcripts for same; content analysis and other techniques for coding and transforming group/open-ended data into analyzed information; qualitative data analysis software (MAXQDA or alternative) and/or case study documentation experience; transformation of qualitative data into a quantitative form that can be merged with other quantitative data in a mixed methods analysis. • Superior writing ability, including evidence of an ability to structure evaluation reports in-line with USAID expectations and in a way that logically and transparently lays out empirical findings, conclusions, and recommendations. Parliamentary Liaison • Advanced degree in Political Science, International Development, Governance, and/or related social science field • Expert knowledge and at least 5 years of experience in the Democracy and Governance sector, preferably in parliamentary strengthening in Kenya • Expert knowledge and at least 5 years of experience in working with the Kenyan Parliament. • Superior communication skills and solid network within Parliament. All team members will provide a written disclosure of conflicts of interest. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 59 In addition, technical oversight and review will be provided by MSI Technical Director Ms. Katharine Hoffman. Ms. Hoffman will review and provide feedback to the evaluation team during the evaluation, including during the development of methodology and the draft report. 6. Evaluation Management 6.1 Logistics USAID/Kenya will provide input through an initial in-briefing to the evaluation team, identify key documents, and assist in introducing the evaluation team to the implementing partner and key MPs and staff. It will also be available for consultations with the evaluation team during the evaluation process regarding information sources and technical issues. MSI will assist in arranging meetings with key stakeholders identified prior to the initiation of field work. The evaluation team will be responsible for arranging other meetings as identified during the course of the evaluation. USAID/Kenya is requested to advise MSI if they would like to participate in any of meetings with the Government of Kenya. MSI is responsible for arranging vehicle rental and drivers as needed for site visits around Nairobi and in the field. MSI will also provide hotel arrangements, office space, internet access, printing, and photocopying. It will also make all payments to vendors directly after team members arrive in country. 6.2 Scheduling The period of performance for this evaluation is 31 August, 2015 to 8 February, 2016. Following a one week desk review, which the team will conduct from their home stations, the team will gather in Nairobi. They will spend one and a half weeks preparing for the fieldwork, which will take three weeks. This will be followed by three weeks of data analysis and week of report writing. Once the team has submitted the draft report to report to MSI, it will undergo technical review and revision, followed by editing and formatting, prior to being submitted to USAID. These final steps prior to submission will take two weeks. 6.3 Budget The proposed task order budget is attached as a separate document. MSI requests that the payment terms of the task order follow IDIQ terms, by which payment shall be calculated by dividing the price of the order by the number of months in the period of performance. Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 2. FINAL WORKPLAN Day Date Location Activity September Mon-Fri Sept 7-11 Home Stations Desk Review Mon-Sat Sept 14-19 (Sept 16: initial meeting w/ USAID re expectations, review of evaluation questions) Nairobi Team Planning Meeting Fri Sept 18 Nairobi Submit Draft Interview & Group Discussion Tool to USAID Fri Sept 18 Nairobi Submit Methodology and Work Plan to USAID Mon Sept 21 Nairobi Meeting with USAID on Methodology and Work Plan where agreements is reached and provisional approval provided Tues Sept 22 Nairobi USAID Approval of Methodology and Work Plan Wed-Fri Sept 23-25 Nairobi Group Discussions, KIIs Sat Sept 26 Nairobi Preliminary data analysis Mon Sept 28 Nairobi Group Discussions, KIIs Evaluation team weekly report due Tues-Wed Sept 29-30 Nairobi Group Discussions, KIIs October Thurs-Fri Oct 1-2 Group Discussions, KIIs Sat Oct 3 Nairobi Preliminary Data Analysis Mon Oct 5 Nairobi Evaluation team weekly report due Mon-Fri Oct 5-9 (Technical Specialist John Lis departs Oct 8) Nairobi Group Discussion, KIIs Sat Oct 10 Nairobi Preliminary Data Analysis Mon Oct 12 Nairobi Evaluation team weekly report due Mon-Tue Oct 12-13 Nairobi Group Discussion, KIIs Wed-Sat Oct14-17 Nairobi Data Analysis Mon-Sat Oct 19-24 Nairobi Data Analysis Mon-Wed Oct 26-28 Nairobi FCR Thurs Oct 29 Nairobi Validation Meeting w/ SUNY and USAID Fri-Sat Oct 30-31 Nairobi Presentation Prep Kenya Parliamentary Strengthening Program: Final Performance Evaluation Day Date Location Activity November Mon Nov 2 Nairobi Presentation at USAID Tues-Sat Nov 3-7 (Team Leader Kathy Vittum departs Nov 7 night) Nairobi Report Writing Mon-Wed Nov 9-11 Nairobi MSI Report Review Thur-Fri Nov 12-15 Home Stations Team Revises Report Mon-Fri Nov 16-19 Home Office MSI HQ Report Editing Thurs Nov 19 Nairobi Draft due to USAID December Thurs- Dec 3 Nairobi USAID & IP Comments Thurs Dec 17 Nairobi Final Report Submitted to USAID January Thurs Jan 7 Nairobi USAID Approves Final Report Thurs Jan 21 Nairobi Statement of Differences to MSI, if applicable Thurs Jan 28 Nairobi MSI Incorporate Statements of Differences, if applicable February Thurs Feb 4 Nairobi USAID approves report for DEC submission Thurs Feb 11 Nairobi One-page Fact Sheet Submitted Thurs Feb 18 Nairobi One-page Fact Sheet Approved Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 3. DATA COLLECTION INSTRUMENTS GROUP DISCUSSION USAID # Interview Question 1 What do you consider the main successes and shortcomings of the Kenya Parliamentary Strengthening Program? (EQ1,2) 2 How did the program’s long-term engagement with Parliament affect the achievement of its objectives? (EQ3) 2a Which, if any, of these objectives likely would not have been achieved in a shorter program? (EQ3) 3 How well did PSP respond to (EQ1, EQ2): 3a Dynamics within Parliament? 3b Broader changes in the Kenyan political context over the course of the program? (e.g., general political environment, between levels of government, popular pressures on the government, etc.) 4 In retrospect, are there any changes that you would have made to the program design or objectives to make the program more relevant to the dynamic Kenyan context? (EQ2) 5 What factors did USAID consider in supporting Phase II of the PSP? (EQ3) 5a To what extent did PSP achieve the results expected from adding this phase? 6 What factors did USAID consider in supporting Phase III of the PSP? (EQ3) 6a To what extent did PSP achieve the results expected from adding this phase? 7 How did PSP select the committees that received PSP assistance in Phase 3? (EQ1) 8 How would you describe the partnership between DFID and USAID? (EQ1,5) Who initiated it and why? (Use the sub-questions as prompts) 8a Who initiated the partnership and why? 8b How satisfied are you with the coordination? Explain. In retrospect, how could coordination have been better? 8c What difference did the joint partnership make to the achievement of PSP’s objectives? 8d What lessons for other jointly funded programs can be learned from the USAID/DFID PSP partnership? 9 What lessons for other jointly funded programs can be learned from the USAID/DFID PSP partnership? 10 What are some of the indicators on the sustainability of systems or institutional changes in Parliament that were introduced with the program’s assistance? (EQ4) 10a Which of these do you think Parliament will be successful at? 11 What were the key lessons for USAID/Kenya from this project? (EQ5) Kenya Parliamentary Strengthening Program: Final Performance Evaluation GROUP DISCUSSION Outside Experts (CSOs/ academics/journalists/private sector) # Interview Question 1 What have been the most significant changes in parliament within the past 15 years (prompts: e.g. with respect to performance, maturity and functioning; support for women and youth within parliament)? (EQ1) 1a What explains these changes? (EQ1) 2 What have been the most significant changes in parliament within the past 5 years (prompts: e.g. with respect to performance, maturity and functioning; support for women and youth within parliament)? (EQ1) 2a What explains these changes? (EQ1) 3 What evidence do you see that Parliament is taking steps to ensure that improvements in systems, practices, functions and roles can withstand changes in personnel? (EQ4) 3a Which of these do you think Parliament will be successful at and why? (EQ4) 4 Are you familiar with the Kenya Parliamentary Strengthening Program (PSP), which was implemented by SUNY? (Y/N) 4a If yes, how would you describe the purpose of the program? (EQ2) 5 In your opinion, what were PSP’s key achievements and shortcomings? (EQ3) 5a Journalists and CSOs only: In what ways do you think that PSP helped parliament become more democratic, mature and effective? 5b What factors contributed to PSP’s ability to achieve these results? 6 How did PSP respond to Parliament’s evolving needs? (EQ2) 6a How did PSP respond to the changing Kenyan political context more generally? 7 CSOs only: How did PSP address the ability of CSOs to work with parliament? (EQ2,4) 8 Academics and journalists only: What type of technical support do you think CSOs are able to provide to parliament today?? (EQ4) 9 CSOs only: How prepared is your organization currently to provide technical support to parliament? What are some examples? 10 How has the participation of women in parliament changed in the past 15 years? 10a To what do you attribute these changes? 10 b What, if any effect did PSP have on these changes? 11 Did the project assist parliament to address issues of concern to women? How? (cross-cutting) 12 CSOs only: What are some key lessons from this project about supporting CSO engagement with parliament? (EQ5) 13 What do you think needs to be done to strengthen County Assemblies’ functions, operations, capacities, etc.? (EQ5) Kenya Parliamentary Strengthening Program: Final Performance Evaluation KEY INFORMANT INTERVIEW SUNY/ CID # Interview Question 1 What have been the most significant changes in parliament within the past 15 years (prompts: e.g. with respect to performance, maturity and functioning)? (EQ3) 1a What factors have contributed to these changes? (EQ3) 2 What have been the most significant changes in parliament within the past 5 years (prompts: e.g. with respect to democratic performance, political maturity and effective functioning; support for women and youth within parliament)? (EQ1) 2a What factors have contributed to these changes (EQ1) 2b Are these changes sustainable? Can you give examples? (EQ4) 3 How do you think PSP was perceived by parliamentary leadership? (Prompts: e.g. in terms of project relevance to Kenyan needs, SUNY technical capacity, SUNY neutrality) (EQ1, 2) Could you provide some examples? 4 How did PSP respond to: 4a Dynamics within Parliament? (EQ1, EQ2) 4b Broader changes in the Kenyan political context over the course of the project? (e.g., general political environment, between levels of government, popular pressures on the government, etc.) (EQ1, EQ2) 5 What were PSP’s most significant achievements and challenges? (EQ1, EQ2) 6 What factors affected PSP’s ability to achieve results? (EQ 1,3) (Prompts: long-term engagement with parliament; incremental, as opposed to long term, funding; political and other external factors; Kenyan cultural norms about the roles of men and women) 6a Which results, if any, could have been achieved within a shorter timeframe? 7 For respondents involved with PSP during Phase 1: Please describe PSP’s initial (development hypothesis/theory of change/results framework). [Note to interviewer: use the appropriate term based upon when the respondent was involved with the project. If you are unsure, use “development hypothesis” and clarify if the respondent does not understand.] (EQ2) 7a For respondents involved with PSP at the design stage: What assessments, analyses, research or consultations did PSP conduct to inform creation of the development hypothesis and project design? (EQ2) 8 Please describe any changes made to the development hypothesis during the course of the project. [Note to interviewer: use the appropriate term based upon when the respondent was involved with the project. If you are unsure, use “development hypothesis” and clarify if the respondent does not understand.] (EQ2) 8a Based upon what information were those changes made? (EQ2) 8b How consistent were these changes with changes in the project’s operating environment? (EQ2) 8c How did the evolving development hypothesis affect your activities? (EQ2) 9 In retrospect, are there any changes that you would have made to the project design and objectives to make the project more relevant to the dynamic Kenyan context? (EQ2) 10 Were there any gaps or problems with the PSP design, and how did the program address them? 11 Are you familiar with PSP’s critical assumptions? (Note to interviewer: if not, list them.) 11a To what extent do you think these assumptions are still valid? (EQ2) 12 What did PSP do to promote sustainability of its interventions? (Prompt: if the respondent does not mention training, ask specifically about it.) Kenya Parliamentary Strengthening Program: Final Performance Evaluation 13 What evidence do you see that Parliament is taking steps to sustain systems or institutional changes introduced with the project’s assistance? (EQ4) 13a Which of these do you think Parliament will be successful at? (EQ4) 13b To what extent do you think the systems PSP built for the Parliamentary Service Commission, Budget Committee, Parliamentary Budget Office and Parliamentary Research Service can withstand personnel changes? (EQ4) 14 What kind of technical support are CSOs able to provide to parliament today? (EQ4) 15 How has the inclusion and representation of women in parliament changed in the past 15 years? (cross-cutting) 15a To what do you attribute these changes? 15b How did PSP contribute to these changes? 16 Did the project assist parliament to address issues of concern to women? How? (cross-cutting) 17 What were the successes and challenges of partnership between DFID and PSP? (EQ1, 5) 17a Could you give some examples of what did/ didn’t work well? (EQ1,5) 18 Based upon your experience with PSP, what are some key lessons about engaging with parliament and legislative strengthening in Kenya from this project? (EQ5) 19 What do you think needs to be done to strengthen County Assemblies’ functions, operations, capacities, etc. (EQ5) Kenya Parliamentary Strengthening Program: Final Performance Evaluation KEY INFORMANT INTERVIEW Parliament (MPs/former MPs and staff/ former staff/CPST) # Interview Question 1 Overall, what is your opinion of the Kenya Parliamentary Strengthening Program (PSP)? (Prompts: e.g. in terms of project relevance to Kenyan needs, SUNY technical capacity, SUNY neutrality) (EQ1, 2) 2 In your opinion, what were the most and least effective areas of PSP assistance? Can you provide examples? (EQ1, 3) 3 In your opinion, how well did PSP respond to parliament’s needs over time? (EQ2) 3a How well did PSP respond to the changing Kenyan political context more generally (Prompts: 2007-08 crisis; new constitution, bicameralism)? (EQ2) 4 What have been the most significant changes in parliament within the past 15 years (prompts: e.g. with respect to democratic performance, political maturity and effective functioning; support for women and youth within parliament)? (EQ3) 4a What factors contributed to these changes? (EQ3) 5 What have been the most significant changes in parliament within the past 5 years (prompts: e.g. with respect to democratic performance, political maturity and effective functioning; support for women and youth within parliament)? (EQ1) 5a What factors contributed to these changes? (EQ1) 6 How did PSP contribute to changes in parliamentary functions such as lawmaking, oversight, administrative operations, committee effectiveness, public outreach and engagement? (EQ1,3) 6a What other factors affected changes in these functions? 6b To what extent do you think these changes have been institutionalized? (Probe for examples, specifics) 7 Are there any ways in which PSP has helped parliament become more democratic, politically mature and effective? (EQ1,3) 7a What other factors have affected these changes? 8 Phase 3 participants only: In the last 5 years, how well did PSP: (EQ1) 8a Assist parliament to draft and pass constitutionally mandated enabling legislation? 8b Assist parliament to draft and adopt the revised Standing Orders? 8c Assist parliament to develop new rules and administrative structures for support staff and a bicameral parliament? 8d Assist parliament to carry out its regular oversight, legislative and representative functions? 8e Develop basic skills for the KNA, Senate, and all 47 County Assemblies? 9 Phase 3 participants only: What other factors have affected parliament’s ability to do these things? (EQ1) 10 How satisfied are you with the activities of the PSP program with which you were involved?(EQ1, 3) 10a Could you explain your answer? 11 Phase 1-3 participants only: Did PSP need 15 years to achieve its objectives? (EQ3) (interviewer prompt: have list of objectives on hand, incl. those that were constant over 15 years) 11a Which, if any objectives could have been achieved in less time? (EQ3) 11b Which require more time? (EQ3) Kenya Parliamentary Strengthening Program: Final Performance Evaluation 12 PSP helped parliament develop several key institutions. How will parliament support and fund these institutions? (Parliamentary Service Commission, Budget Committee, Parliamentary Budget Office, and Parliamentary Research Service.) (EQ4) 12a What gaps are there in parliament’s support for these institutions? (EQ4) 12b Is CPST able to adequately provide training for MPs and parliamentary staff? (EQ4) 12c How likely is the junior fellowship program to continue without PSP support? (EQ4) 13 How has the experience of women in parliament changed in the past 15 years? (cross-cutting) 13a To what do you attribute these changes? 13b What, if any, effect did PSP have on these changes? 13c To what extent did Kenyan cultural norms about the roles of men and women affect the project’s ability to address the participation of women in parliament? 14 Have you been involved in any other international support programs to parliament? (Y/N) 14a If yes, how does PSP compare to those? (EQ5) 15 From your knowledge, what are some key lessons from this project about parliamentary strengthening in Kenya? (EQ5) 16 Senate and County Assemblies’ Forum only: What do you think needs to be done to strengthen County Assemblies’ functions, operations, capacities, etc.? (EQ5) Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 4. DATA COLLECTION LIMITATIONS: INTERVIEWS The evaluation team began by preparing a list of all potential key informants and group discussants, totaling 110 people. From there, the team - with input from USAID/ Kenya - reduced the list on the basis of the viability and relevance of the proposed interviewees and time limitations. The evaluation sector expert and the team scheduler (a former PSP/ Kenya staff member with excellent contacts and rapport with program participants) led this process. USAID provided a general letter of introduction for the team and personalized letters to some key informants. The team faced limitations in reaching out to some of the proposed interviewees. One way that the team compensated for this was to extend the data collection period by one week to maximize the potential for interviews. The team requested USAID support as needed along the way. In total 45 KIIs and give GDs were carried out. 65 respondents participated in the evaluation (43 women, 22 men). The evaluation team was unable to meet with several key informants. The team provided USAID with a list of these individuals. The names have been omitted from this report for the purposes of confidentiality. Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 5. ANALYTICAL APPROACH TO ANSWERING THE EVALUATION QUESTIONS EQ1: The analysis draws considerably on the Phase 3 indicator protocols, 2015 PSP final report, and 2015 exit survey, in particular for issues of achievement. The analysis of both primary and secondary data focuses on what is unique to Phase 3; evidentiary findings for phases 1 and 2 are covered elsewhere in the report. EQ2: The analysis draws on USAID documents that shaped PSP, program documents that chronicle key inflection points over 15 years, internal and external assessment reports, and interviews with key stakeholders who were involved in all the three phases of PSP. EQ3: The response draws on documents to chronicle overall program achievements, starting with initial assessments and continuing through the 2004 evaluation, 2010 final report and 2015 final report. Findings from documents are supported by interviews of long-term stakeholders. EQ4: The analysis draws on documents covering the end of Phase 1 through Phase 3 to identify planning and programming aimed at sustainability. This includes the 2004 evaluation report and subsequent award documents, end of phase’s reports, 2011 baseline survey, and 2015 exit survey. Reference was also made to the definition of sustainability per the 2009 USAID Glossary of Evaluation Terms. This information triangulated with key interviews. EQ5: The 15-year engagement holds several lessons for legislative strengthening programs in Kenya and worldwide. The lessons learned are derived from: 1. Evaluators’ analysis of the findings and conclusions for the first four evaluation questions; 2. Interviewees’ suggestions of lessons learned from the program. Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 6. ADDITIONAL FINDINGS ON WOMEN PSP partnered with and provided support to KEWOPA, a membership association for women in Parliament. According to Parliament, the goal of PSP’s support toward the caucus was to improve the lawmaking and oversight capacity of women legislators, most importantly through training, access to experts, and integration of research into their legislative work. PSP also provided small grants to the caucus. The support provided to KEWOPA was perceived to be particularly beneficial in building the capacity and confidence of women parliamentarians to participate in matters of parliament, as indicated by respondents across three specific categories: Parliament, SUNY and KEWOPA. These benefits can be categorized in three core areas: advanced lawmaking capacity, gender responsive budgeting and operational support. Advanced Lawmaking Capacity One of the recurring themes of PSP support to KEWOPA was the improved legislative capacity of women parliamentarians. Respondents from Parliament, SUNY and KEWOPA indicated that PSP was able to advance women’s role in lawmaking and advocacy. With PSP support, women legislators became better equipped to initiate private member bills. One parliamentary staff member noted that women’s contributions on the floor have improved, while another noted that women’s capacity to lead parliamentary committees and carry out parliamentary business was attributable to PSP support. Examples of women legislators occupying roles such as deputy speaker, deputy chairperson of committees including health and education, and committee chair positions were provided. One SUNY associate observed that PSP played a role in supporting the growth of women in parliament, which was supported by other views from Parliament on the progression of KEWOPA from a weak to a very strong caucus. Importantly, women members played a critical role in advancing the women’s agenda leading up to the ratification of the new Constitution. The changing role of women in Parliament from administrative players to substantive players was a highlight, as was the increased confidence of women to participate more actively in lawmaking. According to a former KEWOPA chairperson interviewed, PSP provided training on critical issues identified by KEWOPA members, most notably climate change and natural resource conservation, issues which affect the day to day lives of women across the country. With PSP support, KEWOPA also contributed to the development of the following legislation touching on FGM, domestic and sexual violence, moving them from bills to acts: Sexual Offenses Act, Victims’ Protection Act, Matrimonial Property Act, and Protection Against Domestic Violence Act. Gender Responsive Budgeting Guidelines With PSP support, KEWOPA also developed gender responsive budgeting guidelines to strengthen women’s participation in the budget making process. The guidelines have been operationalized for use in a number of institutions including FIDA Kenya and the Gender and Equality Commission. Further, they have been customized for use in the county assemblies, where they are being drawn upon to engage women legislators at county level. Operational Support PSP further supported KEWOPA in developing an operational structure, including strategic planning, policy development, staffing and setting up of a secretariat. This administrative support was important for the development of the caucus. Kenya Parliamentary Strengthening Program: Final Performance Evaluation Other Key Issues Other key issues that were perceived to be of concern were that women still experienced intimidation on the house floor, despite their strong debate skills, an area that was highlighted by one Parliamentarian. It was further noted by two Parliamentarians that a large proportion of women in Parliament are not elected, but are instead nominated. This posed a challenge as women are selectively recruited into Parliament. On a related note, three respondents (journalists, a parliamentarian and the executive member of the County Assemblies Forum) observed that the performance of women legislators in Parliament was limited, with others noting that KEWOPA’s performance and leadership was also weak. It was also noted as a concern that Parliament lacks a gender unit or gender policy. However, an interview with KEWOPA staff revealed that the caucus has already approached the speaker of the national assembly to facilitate a gender assessment of Parliament. This is in response to the current gaps, specifically the lack of a gender oversight unit as well as staffing to support this unit. Currently, KEWOPA is supported by a small secretariat and only a few technical staff. While KEWOPA has successfully managed to make important contributions with respect to gender and is well-recognized for doing so, KEWOPA staff interviewed noted that the caucus requires additional support to strengthen its contributions toward gender responsive legislation. Kenya Parliamentary Strengthening Program: Final Performance Evaluation ANNEX 7. LIST OF INTERVIEWEES The full list of interviewees and their contact information has been removed to maintain their confidentiality. Kenya Parliamentary Strengthening Program: Final Performance Evaluation 73 ANNEX 8. LIST OF DOCUMENTS REVIEWED The evaluation team reviewed PSP project documents provided by USAID/Kenya and SUNY/CID, as well as several external reports and other information. The main documents consulted are listed below. Indefinite Quantity Contracts PSP I IQC PSP II IQC Award Documents and Modifications PSP I Original Award PSP II Original Award PSP II Pre Award Authorization PSP III Award AID-623-A-10-000019-fully executed PSP I Modifications 1-5 PSP II Modifications 1-8 PSP III Modifications 1-8 Work Plans September 2000-March 2002 March 2001-March 2002 2004-2005 2007-2008 2008-2009 2009-2010 2011-2012 2013-2014 2014-2015 PMPs and Other Indicator Data PSP I PMPs (2002 and 2003) PSP II PMP PSP III PMP PSP III Performance Progress Report FY 2011 PSP III Performance Progress Report FY 2012 PSP III Indicator Protocols Project Reports (Quarterly, Annual and Final) Quarterly Reports PSP II FY 2005 Q4 Report PSP II FY 2006 Q4 Report PSP II FY 2007 Q2 Report PSP II FY 2007 Q4 Report PSP II FY 2008 Q2 Report PSP II FY 2008 Q3 Report PSP III FY2010 Q1 & Q4 Reports PSP III FY2011 Q1-Q4 Reports Kenya Parliamentary Strengthening Program: Final Performance Evaluation 74 PSP III FY2012 Q1-Q4 Reports PSP III FY2013 Q1-Q4 Reports PSP III FY2014 Q1-Q4 Reports PSP III FY2015 Q1-Q2 Reports Annual Reports PSP III 2010-2013 SUNY Evaluations National Assembly of Kenya Assessment and Program Options, Bob Nakamura, Paul Bagyenda, John Johnson and Jim Ketterer, 2001 PSP III Baseline Survey 2011 PSP III Exit Survey 2015 Creating and Implementing a Kenyan Model of Parliamentary Development, PSP Case Study, 2015 Final Reports PSP I Final Report 2004 PSP II Final Report 2009 PSP III Final Report 2015 External Assessments and Reports Kenya’s Democratic Transition, A Strategy for USAID Continued Support: An Assessment of Kenyan Public Institutions, Judith Geist, Joel Barkan, Jesse McCorry, 1999 Rising Legislative Assertiveness in Uganda and Kenya, Bob Nakamura and John Johnson, 2003 PSP Phase 1 External Evaluation Report, 2004 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 75 ANNEX 9. DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Martin Ocholla Title Field Associate Organization Management SystemsInternational Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) PSP FINAL EVALUATION ( Project No.720700.36-500-03-15) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) N/A I have real or potential conflicts ofinterest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of theUSAID operating unit managing the project(s) being evaluatedor the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significantthough indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome ofthe evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterationsof the project. 4. Current or previous work experience orseeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor withthe implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could biasthe evaluation. Between 2008-2010, I worked with Pact Kenya in the Kenyan Civil Society Strengthening Programme (KCSSP) as a Capacity Building Services Officer (Democracy & Governance). My role entailed providing technical supportto civil society organizations implementing a range of reform oriented and governance based programmes. In the course of my duty and as part of the contractual agreement between USAID and Pact Kenya, I interacted with programme staff of SUNY Kenya as the institution implemented “Legislative Strengthening Program tothe National Assembly of Kenya,” a project funded by USAID through PactKenya. I also had similar interaction with the Socio-Economic Rights Foundation that was implementing a USAID supported project titled “Parliamentary Support Programme” In spite of all these, I do not see any real or potential conflict of interest in my role as a Field Associate with MSI or its local partners since my engagement with the two institutions were largely indirect as I was notthe technical person detailed to closely work with thethem. I would also like to state that I am able to maintain objectivity and high level professionalism in exercisingmy role as the Field Associate with MSI as we conduct the final evaluation of the parliamentary support program. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/09/2015 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 76 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 77 Kenya Parliamentary Strengthening Program: Final Performance Evaluation 78 ANNEX 10: ABRIDGED TEAM BIOS Katherine Vittum Katherine Vittum has 15 years of experience as a manager and adviser for democracy and governance programs in politically sensitive environments. She has expertise in the design, implementation and evaluation of multi-stakeholder programs. Ms. Vittum has participated in numerous evaluations of USAID-funded electoral and political process reform programs, both as a technical expert and as team leader. She has worked in nearly 20 countries throughout Africa, Asia and Europe, and has advised on numerous other programs. She has authored and contributed to several publications on electoral and constitutional issues. John Lis John Lis is a legislative strengthening consultant with nearly two decades of experience in the U.S. Congress and international parliamentary bodies. As an independent consultant, he has conducted evaluations and assessments of democracy and governance programs in Iraq, Lebanon, Kyrgyzstan, Bangladesh and Burma. From 2003 to 2013, he was a Professional Staff Member for the House Committee on Foreign Affairs, including eight years as Staff Director of the House Democracy Partnership, the peer-to-peer legislative strengthening initiative of the U.S. House of Representatives. In that role, he supported the efforts of 17 newly democratic countries to develop independent, effective parliaments and advised two chairmen, Rep. David Dreier (R-California) and Rep. David Price (D-North Carolina). In 2003 and 2004, he served as the Senior Policy Advisor for Transatlantic Relations for Rep. Doug Bereuter (R-Nebraska), Chairman of the Subcommittee on Europe and President of the NATO Parliamentary Assembly. Mr. Lis worked in Brussels from 1999 through 2002 as Director of the Defense and Security Committee of the NATO Parliamentary Assembly, the legislative oversight body of the North Atlantic Treaty Organization. A former journalist, he has worked as the international affairs analyst at the Congressional Budget Office and as a legislative fellow for the Senate Foreign Relations Committee, on the staff of Sen. Joseph Biden (D-Delaware). He holds a bachelor’s degree in history from Stanford University, a master’s degree in international affairs from Columbia University, and the Certificate of the Institute on East Central Europe at Columbia. Martin Ocholla Martin Ocholla is an experienced development practitioner with vast experience the field of conflict, democracy & governance, advocacy, institutional strengthening, public policy analysis, and peace-building within the East Africa region. Mr. Ocholla has worked for a variety of development partners in Kenya, including International Rescue Committee and PACT Kenya. He holds a Master’s degree in Political Science and Public Administration from the University of Nairobi. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523