May 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by John Lis, Taiabur Rahman, AKM Saifullah, and Naim Mostofa for Social Impact, Inc. Final Performance Evaluation of the Promoting Democratic Institutions and Practices Project Cover Photo: The Jatiyo Sangsad Bhaban (National Parliament House) of Bangladesh Photo Credit: John Lis FINAL PERFORMANCE EVALUATION OF THE PROMOTING DEMOCRATIC INSTITUTIONS AND PRACTICES PROJECT EVALUATION OF LEGISLATIVE STRENGTHENING SUPPORT TO DEVELOP THE PARLIAMENT OF BANGLADESH May 14, 2015 USAID Contract AID-OAA-I-10-00003 Task Order AID-388-TO-12-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. PRODIP Final Evaluation ii CONTENTS Acronyms............................................................................................................................................................................ iv Executive Summary ........................................................................................................................................................... v Introduction .................................................................................................................................................................... v The Development Problem and USAID’s Response ............................................................................................ v Purpose of the Evaluation ........................................................................................................................................... v Evaluation Methodology ............................................................................................................................................. vi Findings and Conclusions........................................................................................................................................... vi Recommendations ..................................................................................................................................................... viii Lessons Learned........................................................................................................................................................... ix Introduction ........................................................................................................................................................................ 1 The Development Problem and USAID’s Response................................................................................................. 3 The Development Problem........................................................................................................................................ 3 USAID’s Response........................................................................................................................................................ 3 Purpose of the Evaluation................................................................................................................................................ 5 Evaluation Purpose........................................................................................................................................................ 5 Evaluation Questions .................................................................................................................................................... 5 Evaluation Methodology................................................................................................................................................... 7 Data Collection Methods............................................................................................................................................ 7 Limitations....................................................................................................................................................................... 8 Findings and Conclusions............................................................................................................................................... 10 Findings for Question 1............................................................................................................................................. 10 Conclusions for Question 1 ..................................................................................................................................... 14 Findings for Question 2............................................................................................................................................. 15 Conclusions for Question 2 ..................................................................................................................................... 18 Findings for Question 3............................................................................................................................................. 18 Conclusions for Question 3 ..................................................................................................................................... 21 Findings for Question 4............................................................................................................................................. 21 Conclusions for Question 4 ..................................................................................................................................... 23 Findings for Question 5............................................................................................................................................. 23 Conclusions for Question 5 ..................................................................................................................................... 24 Findings for Question 6............................................................................................................................................. 24 Conclusions for Question 6 ..................................................................................................................................... 25 Findings for Question 7............................................................................................................................................. 25 Conclusions for Question 7 ..................................................................................................................................... 26 PRODIP Final Evaluation iii Recommendations........................................................................................................................................................... 28 Immediate Recommendations.................................................................................................................................. 28 Future Recommendations......................................................................................................................................... 28 Lessons Learned............................................................................................................................................................... 30 Annexes ............................................................................................................................................................................. 32 Annex I: Evaluation Statement of Work................................................................................................................ 33 Annex II: Documents Reviewed.............................................................................................................................. 43 Annex III: Persons Interviewed................................................................................................................................ 45 Annex IV: Evidence Matrix........................................................................................................................................ 48 Annex V: Data Collection Protocols...................................................................................................................... 50 Annex VI: Telephone Mini-Survey Protocol......................................................................................................... 58 Annex VII: Disclosure of Conflicts of Interest..................................................................................................... 65 Tables Table 1: PRODIP Results and Activities as of Year 5 ............................................................................................... 4 Table 2: Key Informant Statistics by Data Collection Method ............................................................................... 7 Table 3: Telephone Mini-Survey Respondent Profile................................................................................................ 8 Table 4: Committee Effectiveness and Processes.................................................................................................... 11 Table 5: Committee Oversight and Public Hearings............................................................................................... 12 Table 6: PRODIP Performance Indicators................................................................................................................. 13 Table 7: PRODIP Committee Activities..................................................................................................................... 16 Table 8: BAMU Functioning .......................................................................................................................................... 20 Table 9: Usefulness of PRODIP Training ................................................................................................................... 22 PRODIP Final Evaluation iv ACRONYMS AL Awami League BAMU Budget Analysis and Monitoring Unit BDGPE Bangladesh Democracy and Governance Program Evaluations BNP Bangladesh Nationalist Party CSO Civil Society Organization DFID Department for International Development DRG Democracy, Human Rights, and Governance JS Jatiyo Sangsad (National Parliament) M&E Monitoring and Evaluation MP Member of Parliament PM Prime Minister PRODIP Promoting Democratic Institutions and Practices PROGATI Promoting Governance, Accountability, Transparency, and Integrity RIG Regional Inspector General RSS Rural Social Service SI Social Impact, Inc. SOW Statement of Work SUNY/CID State University of New York/Center for International Development TAF The Asia Foundation ToT Training of Trainers UNDP United Nations Development Programme USAID United States Agency for International Development USD United States Dollar USG United States Government PRODIP Final Evaluation v EXECUTIVE SUMMARY INTRODUCTION The first democratic elections in Bangladesh were held in 1991, and free elections were held again in 1996 and 2001; during this period, the Awami League (AL) and the Bangladesh Nationalist Party (BNP) alternated in power. In January 2007, a caretaker government, with military backing, declared emergency rule and postponed elections until December 2008. AL won the 2008 elections and formed the government. Domestic and international observers called the December 2008 elections fair, transparent, and credible. Following those elections, in April 2010, the United States Agency for International Development (USAID) awarded a cooperative agreement to The Asia Foundation (TAF) to implement a five-year legislative function and processes improvement project known as Promoting Democratic Institutions and Practices (PRODIP), with a total cost of USD 23,210,783. Part of the funding came from a contribution by the United Kingdom’s Department for International Development (DFID). The scope of the project was twice reduced in 2013 and 2014, with the overall budget being reduced to USD 14,268,933. This final performance evaluation examines the final two years of this project, from April 2013 to April 2015. After taking office in 2008, AL used its overwhelming majority to pass a set of constitutional amendments that led BNP to boycott the January 2014 elections. The resulting Tenth Parliament has no BNP representatives, and 153 MPs were elected unopposed. Moreover, the Prime Minister (PM) in Bangladesh is unusually powerful. There are weak institutional checks on the power of the executive; and parliament’s role in legislation, budgeting, and oversight has become marginal. At present there is no true opposition in parliament, and there is little incentive for Members of Parliament (MPs) to challenge the government. Committees generally only make non-binding recommendations on bills to their respective ministry. The rules call for committees to meet in closed session; holding a public hearing requires permission of the Speaker. Committees have no role in the budget process. The Parliamentary Secretariat is not independent of the executive. Most of the senior staff are detailed from other civil service cadres to work in parliament. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE The transition back to elected government in 2008 was seen as an opening for democratic development and a catalyst to reform political practices and institutions of governance—an opportunity for USAID and other donors to help Bangladesh improve the country’s democratic governance. The PRODIP project complemented and expanded upon activities conducted through an existing USAID project, Promoting Governance, Accountability, Transparency, and Integrity (PROGATI), which ran from 2007 to 2012 and included establishment of a parliamentary budget office and support to the Public Accounts Committee of parliament, which reviews the work of the country’s auditor general. That parliamentary budget office, called the Budget Analysis and Monitoring Unit (BAMU), became part of PRODIP when PROGATI ended. PRODIP’s two objectives were (1) An Effective and Open Parliament and (2) A More Constructive and Sustainable Role for Civil Society in Democratic Governance. TAF implemented Component 2, and the State University of New York Center for International Development (SUNY/CID) served as a subcontractor to implement Component 1. PURPOSE OF THE EVALUATION This external, final performance evaluation reviews the progress made toward achieving the outcomes of those activities that were retained by PRODIP under Objective 1. Specifically, the evaluation: evaluates PRODIP’s performance by assessing results against Objective 1 stated targets and indicators; PRODIP Final Evaluation vi assesses the efficacy and results of the PRODIP implementation approaches and management structure; evaluates to what degree TAF took into consideration the recommendations of the previous evaluations of PRODIP in its design and implementation of Year 4 and Year 5 project activities; and identifies lessons learned for future programming in the field of legislative strengthening. EVALUATION METHODOLOGY The evaluation team used a variety of targeted methods to address the evaluation questions and objectives. Methods included review of background documents, key informant interviews, group interviews, and a telephone mini-survey of parliamentary staff. Parallel analysis was used to analyze the evidence from document review, interviews, and the telephone mini-survey. Limitations included recall bias, response bias, and selection bias. To combat bias, evaluators used multiple sources of data to triangulate on an evaluation issue and included key informants who did not directly benefit from the project. The team’s biggest challenge was the availability of MPs and parliamentary staff, and ongoing blockades and general strikes interfered with the team’s ability to meet with key informants. FINDINGS AND CONCLUSIONS Question 1: To what extent has PRODIP been successful in achieving the program objectives as set out in the Agreement, Modifications to Assistance, Year 4 and Year 5 Work Plans, and other program documents? PRODIP had modest, limited success in achieving the objectives. Evaluators found a limited number of activities that achieved their intended results, such as certain committee hearings leading to modifications in legislation; however, some other activities did not prove successful. For example, an improvement in staff knowledge and skills did not consistently translate into improved parliamentary capacity. PRODIP did not achieve Objective 1, An Effective and Open Parliament. Parliament is less effective and less open than two years ago. PRODIP succeeded in achieving greater openness in the Ninth Parliament, but this progress was reversed in the Tenth Parliament. Donors and implementers stated that PRODIP may have been too large and dispersed to achieve this objective. PRODIP improved lawmaking skills in limited instances. PRODIP’s overall success in improving oversight was limited. PRODIP met few of its performance indicators for Years 4 and 5. Modifying the PRODIP cooperative agreement to eliminate the civil society component and reduce the scope of PRODIP activities in parliament proved to be the correct decision. Even with a tighter focus, PRODIP enjoyed only marginal success in achieving its objectives over the last two years of the project. The political developments of the past two years, combined with the institutional shortcomings of the parliament’s own structure, prevented lasting achievement by PRODIP. PRODIP missed an opportunity to develop the research section of the parliamentary library. While personnel and organizational tension between the two implementers contributed to PRODIP not achieving its results, USAID nevertheless may want to rethink the strategy of combining components like legislative strengthening and civil society support into a single project. Question 2: What has been the impact of PRODIP on the functioning of the Parliament’s committee system? What have been the major obstacles to this activity? How has PRODIP sought to overcome them? The most notable impact of PRODIP on the functioning of the committee system was the introduction of public hearings in the Ninth Parliament. Unfortunately, public hearings were not incorporated into the routine of parliamentary committees, and only one hearing was held in the first 14 months of the Tenth Parliament. There is no provision for public hearings in the rules of procedure and no separate budgets for committees. PRODIP advocated for changes in the rules to provide for public hearings and committee budgets, but no action was taken by parliament. PRODIP’s other attempts to strengthen the PRODIP Final Evaluation vii committee system showed very modest improvements to its operations. PRODIP’s efforts to develop public hearings had a greater impact on the committee system than any other initiative. Unfortunately, political and parliamentary factors conspired to undermine the public hearing experiment. Holding shorter, more frequent hearings in the parliament complex might have routinized the practice and made the organizational challenges less daunting. Question 3: What have been the successes and challenges in those activities of PRODIP in support of the Budget Analysis and Monitoring Unit (BAMU) of the Secretariat? PRODIP’s support for the BAMU—an institution established by USAID’s PROGATI project—was a success. BAMU’s most visible and appreciated contribution was the Help Desk, a booth outside the parliamentary chamber that operated when parliament debated the annual budget. PRODIP has also helped BAMU to produce a variety of written products to assist MPs in understanding the budget process and key issues. A significant challenge is that BAMU staff are part-time, non-experts with primary responsibilities elsewhere in parliament. The greatest challenge to the sustainability of BAMU has been that it is outside of the parliamentary structure. The Parliamentary Service Commission is reviewing a plan to incorporate BAMU into the Secretariat, which is expected to be approved in 2015. The increased awareness and demand for BAMU, and the unit’s pending institutionalization, are likely to be the most lasting legacy of PRODIP. BAMU functions like the budget division of a legislative reference service rather than a parliamentary budget office, but this is well-suited to the needs of the Bangladeshi parliament, given its limited budget role. Question 4: What have been the changes made in PRODIP’s training strategy for staff of the Parliamentary Secretariat in response to the recommendations of previous evaluations? How effective have these changes been in developing the capacity of the Secretariat? While noting that the Speaker and Senior Secretary ultimately decided which staff would attend training activities, previous evaluations concluded that PRODIP training was not always aimed at the appropriate staff, and they recommended that PRODIP ensure that appropriate staff received training with appropriate content during Year 4 and Year 5. PRODIP did select more appropriate staff for training and provided training at an appropriate level. The greatest complaint from staff was that training events were too short. In addition, some training did not have a parliamentary component or was not linked to the participants’ parliamentary duties. Question 5: Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? Previous evaluations recommended that PRODIP disaggregate trainee data, that appropriate documentation be maintained to support results data, and that spot testing be conducted during field visits to verify data. PRODIP did improve its performance management system in response to those recommendations. In particular, PRODIP centralized supporting documentation from its two components and instituted spot checks of its field events. However, the training database itself was not finalized until February 2015, and the performance management system remained far from ideal. This was a missed opportunity for using monitoring data to inform programming. Question 6: To what extent have PRODIP management and stakeholders incorporated knowledge gained through monitoring and evaluation (M&E) into project management? External mid-term reviews were critical of PRODIP’s failure to incorporate knowledge gained through M&E into project management. The mid-term performance evaluation conducted by Social Impact, Inc. PRODIP Final Evaluation viii (SI) recommended that PRODIP gather information that would be of better management use and of interest to stakeholders, including an annual review of indicators for stakeholders. PRODIP management and stakeholders did not incorporate knowledge gained through their M&E processes into project management. As a result, PRODIP was not able to respond to opportunities and challenges that arose. Question 7: How sustainable are the results of PRODIP’s support to the Parliamentary Secretariat beyond USAID support? What measures has PRODIP taken in Year 4 and Year 5 to ensure sustainability? A few key contributions of PRODIP to parliament are sustainable after the project ends in April 2015, but many of the project’s initiatives are likely to end. There is a sustainability plan for BAMU, and it appears likely that the unit will be continued by the parliament. Some other PRODIP initiatives are sustainable, such as the documentation software for the library. PRODIP reference documents, including 14 major publications, are also a sustainable contribution. PRODIP support for public hearings provided a basis for them to become sustainable, but the political challenges to holding public hearings are unlikely to be overcome. PRODIP drafted sustainability plans for BAMU, committee work plans, and committee budgets. PRODIP also developed a Training of Trainers (ToT) series of events. The BAMU plan is expected to be adopted. The committee plans will not be adopted, and the ToT attendees will probably not be used to train other staff. A lack of political support represents the greatest threat to the sustainability of PRODIP initiatives. There is no internal movement for parliamentary development or reform, and the current political crisis compounds this threat to sustainability. RECOMMENDATIONS Recommendation 1: USAID should not fund a new legislative strengthening project until free, fair, and inclusive elections are held in Bangladesh. Recommendation 2: USAID staff should remain engaged with senior parliamentary staff to provide advice and encouragement to make PRODIP initiatives, particularly BAMU, sustainable after the project ends. Recommendation 3: If USAID decides to consider a new legislative strengthening project, the agency should first of all undertake a critical analysis of whether future political events represent a true democratic transition. They may represent a revolution, initial democratic elections, or end of military rule. International experience shows that some events that seemed like democratic transitions may, in retrospect, have little effect on political culture, such as the 2008 elections in Bangladesh. Recommendation 4: USAID should lean against developing a legislative strengthening project for the Bangladeshi parliament unless there is clear, compelling evidence that a true democratic transition has taken place. Ideally this transition would see a core group of MPs with the commitment and political will to reform parliament and develop it as a counterweight to executive power. It is important to remember that free, fair, and inclusive elections do not always represent a democratic transition. If changes do not occur in the political structure and culture, improvements in parliamentary operation and capacity of both members and staff may do little to make the political system more democratic or to increase parliament’s role as an independent counterweight to executive power. Recommendation 5: If USAID in the future determines that the circumstances are right for a new parliamentary strengthening project, it should design the project with input from the Parliament Secretariat and from any parliamentary bodies to be targeted for assistance. Any new project should be limited in scope and tightly focused. Such a project could include: a strategic plan for Secretariat reform; PRODIP Final Evaluation ix development of a Bangladesh Institute of Parliamentary Studies; institutionalization of public hearings in committees; development of best international practices in committee oversight; and development of a committee role in reviewing and amending the national budget. LESSONS LEARNED • Free, fair, and credible elections do not always represent a democratic transition. • Buy-in from leaders of the parliamentary institution is essential. • Smaller, targeted parliamentary strengthening projects can be more effective than large projects. • Downsizing a project scope can be an appropriate response when the political environment changes. • Take into account the political context and the powers of the particular parliament. • Work for changes to rules of procedure, if needed. • Building institutions like a research department is more sustainable than paying for research papers, software, or events. • Training must be targeted to parliamentary audiences. • Implementers should incorporate M&E into regular project review. PRODIP Final Evaluation 1 INTRODUCTION Bangladesh has been independent since 1971, when a devastating war gained the nation its freedom from Pakistan. Following years of military dictatorship, the first democratic election in Bangladesh was held in 1991. Subsequently, elections were held in 1996 and 2001. For the past quarter-century, power has been contested between the Awami League (AL) led by Prime Minister Sheikh Hasina, and the Bangladesh Nationalist Party (BNP) led by Khaleda Zia. From 1991 to 2006, the two parties alternated in power. The leaders, who have both served multiple terms as Prime Minister (PM), are bitter political rivals. When in opposition, they have led boycotts of parliament, street demonstrations, and strikes that have immobilized the government and impeded economic activity. Political violence between supporters of the two parties has also plagued the country. In October 2006, the government resigned in advance of elections, as required by law, and was replaced by a caretaker government that was to administer elections three months later. In January 2007, the caretaker government, with military backing, declared emergency rule and postponed elections until December 2008. Both of the former PMs and then-leaders of their respective parties—AL and BNP— were arrested on corruption charges by the caretaker government; they were among 170 political figures arrested. Both were released on bail in 2008 and went on to contest the elections. AL won the December 2008 elections with 230 of 299 seats in the Ninth Parliament and formed the government under Bangladesh’s parliamentary system. Domestic and international observers called the elections fair, transparent, and credible. At the time, there was hope that those elections could represent a clean break from the corruption, boycotts, and political violence of the past if the two leaders were chastened by their jailing and by popular support for the caretaker government’s anti￾corruption drive. However, the minority BNP promptly ordered a boycott of parliament in February 2009 to protest parliamentary seating arrangements and criminal charges pending against Khaleda Zia and her sons, a pattern of behavior that continued until the end of the Ninth Parliament in January 2014. For its part, AL used its overwhelming majority to pass a set of constitutional amendments, including one that abolished the institution of the caretaker government during the election period. One European diplomat described the actions as “a coup by installments.”1 As a result, BNP boycotted the January 2014 elections, leading The Economist to declare, “The election in Bangladesh was a farce. Because of the boycott, the incumbent AL had won a parliamentary majority even before polling stations opened.”2 The resulting Tenth Parliament has no BNP representatives, and 153 MPs were elected unopposed. Bangladesh is a multiparty parliamentary democracy. It is modeled on the British Westminster system. It is a unitary state, the parliament is unicameral, and the electoral system is first past the post. The Westminster system features a strong PM and a government that comes out of the parliament, but the PM in Bangladesh is unusually powerful. The Bangladeshi parliament is subservient to the PM. She is the head of party, head of the government, and leader of the House, and there is no term limit. No institution exists to hold the PM accountable. At the same time, parliament’s role in legislation, budgeting, and oversight has become marginal. “Parliament is not a counterweight because parliament is controlled by the executive,” a donor country official told the evaluation team. 1 “Bangladesh: The Campaign Trail,” The Economist, December 21, 2013. 2 “The Trouble with Democracy,” The Economist, January 11, 2014. PRODIP Final Evaluation 2 Article 70 of the constitution prevents Members of Parliament (MPs) from voting against their parties despite their being elected individually from single-member districts, and there is no internal party democracy. “Within the parties, there is no questioning of the leaders,” a Western diplomat told the evaluation team. The two party leaders owe their positions to family ties: Prime Minister Sheikh Hasina is the daughter of the assassinated independence leader, and the leader of the extra-parliamentary opposition is the widow of an assassinated president. The political discourse is consumed by their personal hatred of each other, an animosity driven by a belief that the other party is responsible for the death of her loved one. As a result, politics have a winner-takes-all mentality. In the Tenth Parliament, there is no true opposition; the nominal opposition party is a junior partner in the government, and 153 MPs were elected uncontested. BNP has taken to the street to bring its demands home through protests and violence, calling for new parliamentary elections under a caretaker interim government. The current impasse led one American analyst to write: “But if the political rivalry is exacerbated by the personal dramas of the two women, the ongoing crisis is also attributable to a deeper malaise in the nation, due to the authoritarian drift of the current government that has led the country’s politics into an impasse. A series of constitutional amendments have effectively suppressed the institution of the caretaker government and ended the separation of powers between the executive, legislature, and judiciary, constraining political and public freedom throughout the country. Simultaneously, repression of the press and media has increased.”3 There is little incentive for MPs, all of whom come from government parties, to challenge that government. MPs who want to be re-nominated will seek to remain in the good graces of their parties. “Their relationship with party bosses is most important,” one academic said. Evaluators interviewed two MPs who lost committee chairmanships because they were too active and irritated ministers with assertive questioning and effective scrutiny of proposed legislation. “I don’t think parliament does the oversight role too well because no one dares to question the prime minister,” a Western diplomat said. Institutionally, the parliamentary committee system in Bangladesh is modeled on the Westminster system but lacks several features that serve to check government power. While committees in theory can choose their members and chair, in practice the final decision falls to the PM. While committees can consider legislative bills referred to them by the House, they generally only make non-binding recommendations to the ministry, which can be ignored. Nearly all bills originate with the government; one senior staffer stated that only nine private-member bills have become law in the country’s 44-year history; three of those were in the Ninth Parliament. Committees have no role in the budget process. The rules of parliament call for committees to meet in closed session; holding a public hearing requires the express permission of the Speaker. The government may decline to produce a document on the grounds that its disclosure would be prejudicial to the safety or interest of the state, and ministers can ignore a request to appear before a committee. Ministers are ex-officio members of the committees that exercise jurisdiction over them. The Parliamentary Secretariat is not independent of the executive. Most of the senior staff are “deputed” (i.e., detailed) from the civil service to work in parliament for two- or three-year rotations. Permanent staff stated that 20 of the top 23 staff and 57 of the 145 Class I staff in parliament are detailees. Committees are understaffed, and committee secretaries serve multiple committees. 3 Frederic Grare,”Is Bangladesh’s Crisis About to Get Much Worse?” The Diplomat, March 2015. PRODIP Final Evaluation 3 THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE THE DEVELOPMENT PROBLEM Since independence, Bangladesh has held democratic elections sporadically, including three successive peaceful transfers of power from 1991 to 2006. Yet, the country’s development as a democracy has been interlaced with military rule, debilitating political polarization, ineffective institutions of governance, and endemic corruption. Partisan deadlock over electoral issues resulted in the cancellation of the January 2007 parliamentary elections and self-imposed rule by a military-backed caretaker government. Following a two-year state of emergency enforced by the caretaker government, free and fair parliamentary elections were held on December 29, 2008. The transition back to elected government was seen at the time as an opening for democratic development and a catalyst to reform political practices and institutions of governance. This was viewed as an opportunity for the United States to help Bangladesh improve the country’s democratic governance. At the time of the 2008 elections, political development in Bangladesh had not kept pace with economic expansion in the previous decade. Despite continued economic gains, government institutions and political parties (particularly AL and BNP) continued to operate in a patron-client relationship with the people of Bangladesh. This relationship and a winner-take-all approach to governing after securing electoral victory stalled Bangladesh’s political and democratic advances. The functioning of the parliament (Jatiyo Sangsad) suffered under this system, as ruling-coalition MPs took little interest in their legislative, oversight, or public policy responsibilities and opposition MPs simply refused to participate in plenary sessions. As a result, Bangladeshis had little space or interest in pressing their elected leaders for changes to the status quo, which only heightened the disconnect between citizens and their government. USAID’S RESPONSE In April 2010, the United States Agency for International Development (USAID) in Bangladesh awarded a cooperative agreement to The Asia Foundation (TAF) to implement a five-year legislative function and processes improvement project known as Promoting Democratic Institutions and Practices (PRODIP), with a total cost of USD 23,210,783. PRODIP was jointly funded by USAID/Bangladesh and the British Department for International Development (DFID). PRODIP’s two objectives were (1) An Effective and Open Parliament and (2) A More Constructive and Sustainable Role for Civil Society in Democratic Governance. Under these objectives, PRODIP targeted the following results: (1) Improved parliamentary lawmaking skills and oversight capacity; (2) A stronger parliamentary committee system; (3) Increased public awareness of—and citizen participation in—parliamentary functions and processes; (4) Strengthened institutional capacity of civil society organizations (CSOs) in areas of public and democratic governance; (5) More informed public policymaking and monitoring of policy implementation at the national and sub-national level; and, (6) Increased citizen voice for transparent and accountable governance. PRODIP Final Evaluation 4 PRODIP’s strategic approach to achieving these results was to simultaneously assist both parliament (supply-side – Component 1) and CSOs (demand-side – Component 2) in creating an open and effective parliament and to foster a more constructive and sustainable role for civil society in democratic governance. On the supply side of the governance equation, PRODIP assisted parliament to improve its lawmaking and oversight capability, strengthen its committees, increase public awareness of its functions, and create opportunities for increased public input into the development of national policy. On the demand side, PRODIP assisted civil society by providing technical assistance, training, and institutional support to CSOs to help shape public policy and provide a vehicle for public input. The prime contractor, TAF, implemented Component 2, and the State University of New York Center for International Development (SUNY/CID) served as a subcontractor to implement Component 1. An existing USAID project, Promoting Governance, Accountability, Transparency, and Integrity (PROGATI), ran from 2007 to 2012 and included establishment of a parliamentary budget office and support to the Public Accounts Committee of parliament, which reviews the work of the country’s auditor general. That parliamentary budget office, the Budget Analysis and Monitoring Unit (BAMU), became part of PRODIP when PROGATI ended. PRODIP underwent two rounds of modifications to its activities; this coincided with a reduction in the overall budget from USD 23,210,783 to USD 14,268,933, owing in part to reductions in availability of budget resources. The first round of modifications was in response to a series of evaluations by DFID (2012), the USAID Regional Inspector General (RIG) (2013), and USAID/Bangladesh (2013)—the latter conducted by Social Impact, Inc. (SI) through the Bangladesh Democracy and Governance Program Evaluations (BDGPE) project. The second round of modifications was in response to the changed political conditions resulting from the flawed 2014 parliamentary elections, which called into question some of the premises of the project with respect to the role of the parliament and its relations with the executive. As a result of these modifications to the cooperative agreement, all of the planned results and activities under Objective 2, A More Constructive and Sustainable Role for Civil Society in Democratic Governance, were scaled back in the first round (Year 4 Work Plan), then eliminated entirely in the second round (Year 5 Work Plan). Due to the cutbacks, SUNY/CID was dropped as a subcontractor in February 2014, and TAF took over implementation of Component 1 on legislative strengthening. Under Objective 1, there were also significant cutbacks in Years 4 and 5 of PRODIP, leaving TAF accountable for the following activities through April 2015: Table 1: PRODIP Results and Activities as of Year 5 Result 1: Improved Parliamentary Lawmaking Skills and Oversight Capacity Activity 1.2: Provide Research, Report Writing, and Drafting Training for Parliamentary Staff Activity 1.3: Support Policy Research Activity 1.4: Establish Research and Documentation Services in Parliament Activity 1.11: Support the orientation of new members of the Tenth Parliament Activity 1.12: Support for BAMU – Provide Financial Analysis Services to Parliament Result 2: A Stronger Parliamentary Committee System Activity 2.2: Develop Committee Work Plans Activity 2.3: Engage Target Committees to Improve Committee Work Processes Activity 2.4: Engage Target Committees to Improve Oversight Functions Activity 2.6: Produce Committee Activity Briefs Result 3: Increased Public Awareness and Participation in Parliamentary Functions and Processes Activity 3.4: Support Public Hearings PRODIP Final Evaluation 5 PURPOSE OF THE EVALUATION EVALUATION PURPOSE This external, final performance evaluation reviews the progress made toward achieving the outcomes of activities that were retained by PRODIP under Results 1, 2, and 3 of Objective 1. The evaluation specifically reviews the activity period from April 2013 to April 2015. The first three years of the project (April 2010 to April 2013) were evaluated in the mid-term audits and evaluations described in the previous section. Specifically, this evaluation: • Evaluates PRODIP’s performance by assessing results against stated targets and indicators set out for Objective 1 in the Agreement, Modifications of Assistance, Year 4 and Year 5 Work Plans, and other documents; • Assesses the efficacy and results of the PRODIP implementation approaches and management structure in meeting Objective 1; • Evaluates to what degree TAF took into consideration recommendations of previous evaluations in the design and implementation of PRODIP activities during Years 4 and 5; and • Identifies lessons learned for future programming in the field of legislative strengthening in Bangladesh by USAID, DFID, and other donors. The audience for this evaluation is USAID/Bangladesh, USAID/Washington, leaders of USAID Forward, TAF, and USAID partners like DFID, the United Nations Development Programme (UNDP), and other bilateral and multi-lateral donors to Bangladesh. EVALUATION QUESTIONS A. Results 1. To what extent has PRODIP been successful in achieving the program objectives as set out in the Agreement, Modifications to Assistance, Year 4 and Year 5 Work Plans, and other program documents? 2. What has been the impact of PRODIP on the functioning of the Parliament’s committee system? What have been the major obstacles to this activity? How has PRODIP sought to overcome them? 3. What have been the successes and challenges in those activities of PRODIP in support of the Budget Analysis and Monitoring Unit (BAMU) of the Secretariat? 4. What have been the changes made in PRODIP’s training strategy for staff of the Parliamentary Secretariat in response to the recommendations of previous evaluations? How effective have these changes been in developing the capacity of the Secretariat? B. Management and Administration 5. Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? 6. To what extent have PRODIP management and stakeholders incorporated knowledge gained through monitoring and evaluation (M&E) into project management? PRODIP Final Evaluation 6 C. Sustainability 7. How sustainable are the results of PRODIP’s support to the Parliamentary Secretariat beyond USAID support? What measures has PRODIP taken in Year 4 and Year 5 to ensure sustainability? PRODIP Final Evaluation 7 EVALUATION METHODOLOGY DATA COLLECTION METHODS Data collection took place from February 3 to March 10, 2015. The evaluation team used a variety of targeted methods to address the evaluation questions and objectives by eliciting rich information and triangulating emerging trends and themes. Methods included: • Review of background documents, including the technical proposal from the PRODIP award; the Modifications to Assistance; all PRODIP Quarterly and Annual Performance Reports from Year 4 and Year 5; PRODIP Work Plans and M&E Plan; DFID’s annual reviews that outline PRODIP’s successes and ongoing challenges; the RIG report (2012); the mid-term evaluation conducted by SI (2013); reports from PRODIP-sponsored surveys and analytical research; and lists of PRODIP points of contact. See Annex II for documents reviewed. • Key informant interviews with USAID/Bangladesh and USAID/Washington staff, PRODIP staff, parliamentary leaders and staff, members of the Parliament Secretariat, representatives of other donor agencies, senior TAF staff, civil society representatives, journalists, and academics. Interviews were conducted in the United States from February 9 to 18, 2015 and in Dhaka from February 23 to March 6, 2015. See Annex III for a comprehensive list of 54 persons interviewed. • Telephone mini-survey conducted from March 4 to 10, 2015 with 30 parliamentary staff who participated in PRODIP activities. Table 2: Key Informant Statistics by Data Collection Method Respondent Group Key Informant Interviews Telephone Mini￾Survey Subtotal Total MPs/former MPs Males 6 6 9 Females 3 3 Parliament staff Males 13 27 40 46 Females 3 3 6 External actors Males 11 11 14 Females 3 3 USG and DFID Males 3 3 5 Females 2 2 PRODIP/TAF Males 6 6 10 Females 4 4 Total 54 30 Male: 66 Female: 84 18 The team finalized its semi-structured interview protocols after in-depth discussions with USAID personnel and TAF staff. Each protocol differed depending on the key informant’s role and “causal distance” from activities, as well as the extent of the key informant’s involvement in parliament-related activities and the time available for interviewing; the questions addressed not just knowledge and general perceptions, but more importantly probed for specific examples of attitude and behavior change. Group interviews were conducted with the same protocols. The telephone mini-survey protocol was based PRODIP Final Evaluation 8 closely on related interview protocol questions. See Annex V for data collection protocols and Annex VI for the telephone mini-survey protocol. The team used parallel analysis to analyze the evidence from interviews, document review, and the telephone mini-survey. In this analytical approach, each type of data for an activity is analyzed in parallel, and then across data type. For example, the team developed preliminary findings by first analyzing interviews with trainees and with supervisors or other leadership of those trainees; developed complementary findings from the telephone mini-survey responses; developed additional findings from the key documents and other secondary materials; and finally, analyzed the findings across all of the types of data to develop definitive, activity-level findings. See Annex IV for an evidence matrix that depicts the relationship between evaluation questions, data collection methods, and respondent groups. Table 3: Telephone Mini-Survey Respondent Profile Frequency Percent Sex Male 27 90 Female 3 10 Employment Status Permanent parliamentary staff 18 60 Detailed from executive branch 12 40 Education Masters 26 86.67 Bachelor 2 6.67 Higher Secondary Certificate (HSC) 1 3.33 N/A 1 3.33 Number of Years Working in Parliament 1-5 14 47 6 - 10 3 10 11 - 15 0 0 16 - 20 6 20 21 - 25 4 13 26> 3 10 Committee staff Yes 22 73 No 8 27 BAMU staff Yes 5 17 No 25 83 LIMITATIONS As with any evaluation, there are biases and other limitations that must be addressed through methodological or analytical methods, such as using multiple sources of data to triangulate on an issue, which are discussed in more detail below. An evaluation of parliament-related activities conducted by multiple implementers, with multiple sources of funding, is subject to many of these issues: • First, recall bias was present, such as parliament staff responding to questions with answers related to PRODIP activities outside the evaluation period or to activities funded by another donor. A similar problem is that some participants in multiple training activities blended their experiences into a composite memory or response, e.g., staff received training on several topics both before and during the evaluation period and subsequently did not distinguish them as separate activities. PRODIP Final Evaluation 9 • Second, response bias is a common problem for evaluations, particularly in highly technical democracy, human rights, and governance (DRG) subsectors such as parliamentary strengthening. For example, MPs may give the interviewer positive remarks about an activity like support for the budget unit because s/he would like such support to continue in the future. Many key informants expressed an explicit desire for donor support to continue. • Third, selection bias in the form of contacts provided by TAF meant that the team heard from MPs and staff who were active with the project and the implementer; again, this is particularly a problem for parliamentary strengthening. This was partly responsible for the low number of female key informants; for example, of the 10 committees prioritized for PRODIP assistance, none was chaired by a woman in the Ninth Parliament, and just 21 of the 119 individual staffers trained in Years 4 and 5 were women. The most effective approach to combating bias is to use multiple sources of data to triangulate on an issue. By combining information found in documents or interviews from multiple sources, any one piece of biased data does not skew the analysis. Other approaches that pertain specifically to interviews are the inclusion of key informants from organizations that do not directly benefit from the project, such as academics and journalists, and the use of questions about specific examples of knowledge use. The telephone mini-survey provided an additional set of data against which to check findings from documents and interviews. The team’s biggest obstacle was the availability of contacts. Interviews with MPs or senior staff were difficult to schedule because of existing demands on their time or the need to accommodate last-minute scheduling changes. In addition, elections occurred during the evaluation period, so many key informants are no longer MPs or parliamentary staff and were difficult to locate, particularly those from the opposition party that is not represented in the Tenth Parliament. The parliamentary administration was less cooperative than expected and did not issue the evaluation team leader an entry pass to the parliament building until the final day of fieldwork, which resulted in the cancelation of several interviews that had been scheduled inside the parliament building. Finally, the ongoing blockades and general strikes in Dhaka interfered with the evaluation team’s efforts to meet with key informants; specifically, a security alert issued by the U.S. Embassy led the evaluation team to cancel interviews scheduled for March 4, 2015. Despite these limitations, the evaluation team was able to reach a sufficient number of key informants to draw valid conclusions from the interviews. PRODIP Final Evaluation 10 FINDINGS AND CONCLUSIONS FINDINGS FOR QUESTION 1 To what extent has PRODIP been successful in achieving the program objectives as set out in the Agreement, Modifications to Assistance, Year 4 and Year 5 Work Plans, and other program documents? The evaluation team found that PRODIP had modest, limited success in achieving its objectives. Some of the specific PRODIP activities achieved their intended results; however, some other activities did not prove successful. In some cases, an improvement in staff knowledge and skills did not translate into improved parliamentary capacity due to constraints identified in the Introduction. An Effective and Open Parliament The primary objective for PRODIP is Objective 1, An Effective and Open Parliament. PRODIP did not achieve this objective. At the time of the evaluation in March 2015, parliament was less effective and less open than it had been two years previously. In large part, this was due to the political and structural factors identified in the Introduction—such as the strong PM, overwhelming political party control, erosion of separation of powers, and lack of a true opposition—which marginalized parliament and made it less relevant and less effective. With the main opposition political party shut out of parliament and with active MPs being shunted aside, parliament in 2015 was less effective than two years prior. PRODIP succeeded in achieving greater openness in the Ninth Parliament, but this progress was reversed in the Tenth Parliament. Seven committees organized 28 public hearings in the Ninth Parliament, most of these outside Dhaka; during the evaluation period, four hearings were held, all in Spring 2013. These events opened parliamentary proceedings to a wider audience and enabled ordinary Bangladeshis to have their voices heard by their representatives. Unfortunately, in the first 14 months of the Tenth Parliament’s tenure, only one public hearing has been held, and there are no plans to organize another despite dogged efforts by PRODIP staff. One donor official reflected, “I don’t think we broke the culture of closing parliament.” Donors and implementers stated that PRODIP may have been too large and unfocused to achieve this objective, noting that through February 2014 there was a second component that supported civil society, as well as parliamentary outreach initiatives like youth engagement, constituency offices, and a visitors center. “PRODIP was trying to do too many things at once,” a donor official said. “I think there was too much money; the problem wasn’t that there wasn’t enough. There was just too much going on. We should have been more strategic.” “I wonder if PRODIP might have been too much,” said one TAF official. “The design was: ‘We want you to do all of these activities.’ It should have been more targeted.” A colleague added, “PRODIP was probably the largest legislative strengthening program. We had millions of activities. We tried to do everything.” “The idea was to put out a large slate of activities and hone it down after seeing where we had success,” a former PRODIP official said. “I think to some degree it was unwieldy.” Other objectives In looking at other stated objectives of the project, the Year 4 Work Plan contained several objectives for committees and for BAMU. These were: Committees: • A stronger committee wing • Institutionalization of procedures such as work planning and committee briefs PRODIP Final Evaluation 11 • A sustainability strategy BAMU: • Increased capacity for BAMU staff • Increased demand for BAMU services • Institutionalization of Help Desk The evaluation team found that two objectives for committees were not achieved. While some procedures were implemented by some of the 10 committees targeted by PRODIP, these were not institutionalized, and the sustainability strategy proposed by PRODIP was not adopted by the Secretariat. The evaluation team did find that the committee wing is stronger; staffers have a better understanding of their role, and they have learned specific skills, particularly in organizing public hearings. While the staff is stronger, though, they have not had an opportunity to use their skills because committees are not holding public hearings in the Tenth Parliament. Of the 22 committee staff who participated in the telephone mini-survey, 59 percent stated that PRODIP training helped make their committee more effective. Only 27 percent, though, responded that PRODIP helped improve the work processes of their committee. Table 4: Committee Effectiveness and Processes Q 7: Has PRODIP improved the functioning of parliament’s committee system? Frequency Percent Yes 13 59 No 6 27 Do not know 3 14 Total 22 100 Q 9: Has PRODIP helped your committee to improve its work processes? Frequency Percent Yes 6 27 No 12 55 Do not know 4 18 Total 22 100 The evaluation team found that the objectives for BAMU have been mostly achieved. Demand for BAMU services, as measured by the number of MPs seeking assistance from the unit, increased between 2012 and 2014. Members who use the service stated that the capacity of their staff has increased as a result of PRODIP training. Plans to institutionalize the Help Desk are moving forward and are expected to be approved by the Parliamentary Service Commission in 2015. Findings related to BAMU are discussed in more detail under Evaluation Question 3. Improved Parliamentary Lawmaking Skills and Oversight Capacity At the results level, evaluators found only isolated cases where PRODIP improved lawmaking skills or oversight capacity. Most notably, PRODIP support did enable the Committee on Labor and Employment to hold a May 2013 hearing in the wake of the April 2013 Rana Plaza garment-factory collapse that killed more than 1,100 workers and injured more than 2,500. The hearing heard from 75 witnesses, including survivors, labor unions, international organizations, and foreign ambassadors. A follow-on hearing in June 2013 to consider a new labor law heard from 150 witnesses over three days. PRODIP Final Evaluation 12 Following the hearings, the then-chairman identified 82 modifications that the committee made to the bill that the government submitted, including nine entirely new sections, all but three of which were accepted by the government. “I was successful in doing this thanks to public hearings,” the chairman said. In all, the committee held four hearings on various topics in the Ninth Parliament. However, for his efforts in using his committee to improve life for ordinary Bangladeshis, the chairman was stripped of his position in the Tenth Parliament. “He was admonished by the prime minister for doing it,” a former PRODIP official said. “The prime minister didn’t like the parliament questioning anything the government did.” The sole hearing in the Tenth Parliament was held by the Committee on Social Welfare on the topic of the “Rural Social Service (RSS) Program: Experience and Future Remedies” at the parliament Members’ Club on November 26, 2014. The hearing was attended by committee members and staff, ministry officials, CSO representatives, and RSS beneficiaries. The committee received 12 actionable suggestions from the hearing focused on the selection of borrowers; the maximum limit of RSS loans; service charges; selection of locations; training; government and civil society collaboration; insurance; use of information technology; and monitoring. According to the chairman, the ministry increased the RSS benefit by 30 percent in response to a recommendation made by the committee as a result of the hearing. “Through the public hearing, the MPs heard from the people,” the chairman said. “The people became more aware of what benefits they could get from the program, and they could tell MPs and officials their problems.” The Commerce Committee published a report detailing recommendations that the committee had made to the ministry over the previous five years and analyzing which were implemented, which were not, and the reasons. The committee chairman, appointed to the position in March 2014, was replaced in September 2014. The chairman told evaluators that he was forced to resign after the minister objected to what he saw as the chairman’s overzealous oversight. Despite these individual examples, PRODIP’s overall success in improving oversight was limited. Of 22 committee staff surveyed by telephone, none believed that PRODIP improved their committee’s oversight capacity. Eighteen of the 22 said PRODIP did not improve oversight, while four responded that they didn’t know. These data may be skewed, though, by the fact that only one of those 22 respondents had received PRODIP assistance to organize a public hearing; the other committee staffers received training on other topics not related to oversight. Table 5: Committee Oversight and Public Hearings Q 10: Has PRODIP helped your committee to improve its oversight functions? Frequency Percent Yes 0 0 No 18 82 Do not know 4 18 Total 22 100 Q 12: Has PRODIP supported your committee in organizing and conducting public hearings? Frequency Percent Yes 1 4.5 No 20 91 Do not know 1 4.5 Total 22 100 PRODIP Final Evaluation 13 Activities Likewise, PRODIP activities showed only isolated successes. Training was generally well-received by beneficiaries, who could point to specific skills that they learned. However, in many instances staffers were unable to use the training that they received. In particular, research training focused on general research methodology, rather than research and writing for MPs, and staff were not able to use what they learned in their work for MPs. PRODIP did contract with outside researchers to provide committees with policy briefs to inform public hearings. While these were solid analytical reports that provided committees with a firm policy foundation for hearings, the reliance on outside research meant that parliament’s own research section remained underdeveloped and underutilized. “We sometimes have a lack of information and statistics to answer the government. We should have our own people to feed information to MPs,” said a former opposition MP, who added that the parliament’s own research section “is not well-known” to MPs. The research section is headed by a detailed civil servant, not a career researcher. PRODIP’s main activity to help establish research and documentation services was the procurement of software that enabled the parliamentary library to store and catalog plenary debates from the Eighth and Ninth Parliaments, as well as training for staff on using the software. This purchase did enable the library to more efficiently find speeches for MPs. PRODIP also helped the library establish an electronic news clipping service for MPs. TAF provided the evaluation team with statistics on library usage in 2014 that were compiled by the library; these showed that MPs used the library 197 times from January through May 2014 (39.4 times per month) and 296 times from June through December 2014 (42.3 times per month). Statistics were not available for the Ninth Parliament (January 2009-January 2014); a long-time librarian said that demand for the library’s services increased slightly in the Ninth Parliament, but “the Tenth Parliament is worse” in terms of the number of MPs using the library. In light of the controversial 2014 elections and the subsequent decision by USAID to halt all contact with MPs, the proposed orientation program for new members was scrapped. Instead, PRODIP developed an MP Handbook that it planned to distribute to the Secretariat to provide to MPs. At the time of the evaluation, the handbook had not been printed, so evaluators were unable to review it. Performance Indicators PRODIP met few of its performance indicators for Years 4 and 5. PRODIP exceeded its targets in the one area in which it exercised the greatest control: number of staff trained. Most of the other indicators were subject to the political dysfunction discussed above, as well as other factors beyond PRODIP’s control. The Ninth Parliament wound down in late 2013 amid campaigning and a growing political standoff. After the January 2014 elections, PRODIP was instructed to cease working with MPs, who themselves were less willing to undertake oversight activities like public hearings or to request research to scrutinize the government’s legislative proposals. Of the 119 individual staffers who received training in Years 4 and 5, 21 were women. Some staff attended multiple training events. Table 6: PRODIP Performance Indicators Year 4 (April 2013 – March 2014) Indicator Target Output Number of public hearings held 9 4 Number of research reports or policy briefs prepared 53 50 PRODIP Final Evaluation 14 Number of bills subject to technical analysis and review 4 2 Number of oversight actions taken 10 4 Number of staff trained 360 404 Year 5 (April 2014 – December 2014) Indicator Target Output Number of public hearings held 2 1 Number of research reports or policy briefs prepared 9 8 Number of bills subject to technical analysis and review 0 0 Number of oversight actions taken 2 1 Number of staff trained 180 192 CONCLUSIONS FOR QUESTION 1 PRODIP enjoyed only marginal success in achieving its objectives over the last two years of the project. Instances of more effective lawmaking and oversight were few. The training and support provided to BAMU did improve the capacity of the staff and the quality of budget information provided to MPs, particularly through the Help Desk. The introduction of public hearings by select parliamentary committees did increase openness in the Ninth Parliament; however, this achievement has been reversed in the Tenth Parliament. PRODIP missed an opportunity to develop the research section of the parliamentary library, which was supposed to have been a key activity of the project. While PRODIP did procure high-quality outside research for committees that held public hearings, it did not develop a central research unit that could have become a sustainable resource for the parliament. Development of a parliamentary research unit is usually a core component of legislative strengthening projects; such an activity was listed as part of PRODIP, but it never came to fruition. The political developments of the past two years, combined with the institutional shortcomings of the parliament’s own structure, prevented lasting achievement by PRODIP. For example, one of the greatest successes of the project was the series of public hearings held by the Committee on Labor and Employment in the wake of the Rana Plaza factory collapse, which led to dozens of modifications to the country’s new labor law. Unfortunately, the current political climate does not reward politicians who conduct aggressive oversight or who question the policy of the government, and the chairman was punished for his efforts with the loss of his chair. While PRODIP programming was imperfect (as will be discussed in more detail below), it is unlikely that a more effective project could have achieved any lasting success in a deteriorating political environment. The decision to modify the cooperative agreement to reduce the range of PRODIP activities proved to be the correct decision. The mid-term audits and evaluations identified difficulties in implementing such an expansive project, and an austere budget environment necessitated a refocusing of the project onto its core objective. Furthermore, PRODIP was conceived at a time of optimism about the trajectory of Bangladeshi democracy. The 2008 elections were seen as a restoration of civilian, democratic rule after two years under a military-backed, unelected caretaker government. The expansive initial design of the PRODIP project came in response to a perceived democratic transition, but by 2013 it was apparent that little had changed in the Bangladeshi political system. The flawed 2014 elections demonstrated that Bangladesh was not moving toward a stronger democracy, and the Mission made the right decision to PRODIP Final Evaluation 15 drastically scale back the project in a deteriorating political environment. USAID may want to rethink the strategy of combining components like legislative strengthening and civil society support into a single project. Mid-term evaluations and audits found that the two components of PRODIP did not work together effectively, and donor officials and implementers told this evaluation team that PRODIP was too large and unfocused. While personnel and organizational tension between the two implementers contributed to PRODIP not achieving its results, and the political environment in Bangladesh made it difficult for any parliamentary strengthening project to succeed completely, a more targeted project solely dedicated to supporting parliament might have had a better chance of achieving greater success. FINDINGS FOR QUESTION 2 What has been the impact of PRODIP on the functioning of the Parliament’s committee system? What have been the major obstacles to this activity? How has PRODIP sought to overcome them? Impact of public hearings The most notable impact of PRODIP on the functioning of the committee system was the introduction of public hearings. Seven committees held 28 hearings in the Ninth Parliament. While most of these were held prior to the evaluation period (an election campaign limited parliamentary activity during Year 4 of PRODIP), there were four hearings held in Spring 2013. Most notable of these were the hearings held by the Committee on Labor and Employment in the wake of the Rana Plaza garment factory collapse and consideration of the new labor law, which was amended to incorporate ideas generated during the hearings. As noted in the previous section, the then-chairman identified 82 modifications that the committee made to the bill that the government submitted, including nine entirely new sections. All but three of the committee’s 82 proposed modifications were accepted by the government. Committee staff who were trained by PRODIP mostly reported that they now have the skills to organize public hearings without assistance from PRODIP, though a couple of staff questioned their ability to organize logistics for field hearings outside Dhaka. MPs were split on whether they believed staff could organize hearings without PRODIP assistance. “We can conduct hearings by ourselves without outside support,” one committee secretary said. “What is lacking is initiative.” Obstacles to public hearings Unfortunately, public hearings were not incorporated into the routine of parliamentary committees, and only one hearing was held in the first 14 months of the Tenth Parliament, with no indication that any additional hearings will take place after the project ends. Major obstacles include: • The Rules of Procedure state that committee meetings are closed to the public, which requires committees to seek the permission of the Speaker to hold a hearing that is open to the public; • Political disincentives for committee chairs, all of whom come from government parties, to hold the government to account; • Sacking of active committee chairs who had organized hearings, amended legislation, and conducted rigorous oversight in the past; • Lack of individual committee budgets to pay the cost of public hearings. Most notably, committees are culturally expected to provide refreshments to those who attend hearings. According to a sample of hearing costs provided by TAF to the evaluation team, each public hearing costs between USD 1,000 and USD 5,000. The lowest cost (USD 1,008) was for a one-day hearing held PRODIP Final Evaluation 16 near Dhaka. A three-day hearing in Dhaka for Dhaka residents cost USD 2,547, similar to the cost of a one-day hearing in Dhaka where out-of-town residents were brought to the capital. The most expensive was a one-day hearing held in Jessore, a six-hour drive west of Dhaka, which cost USD 5,021. A PRODIP staffer said that one committee chairman in the Tenth Parliament had approached the project about organizing a public hearing. “The committee wants to hold a hearing, but the chairman says that the political situation, with the hartals (general strikes), doesn’t allow for a public hearing because people cannot come to the parliament.” Efforts to overcome obstacles to public hearings PRODIP took several steps to overcome the obstacles to public hearings. Through 2013, PRODIP advocated for changes in the Rules of Procedure. Among the rules changes advocated by PRODIP were a provision for public hearings and committee budgets. A proposal was submitted to the Speaker in October 2013 but not acted upon, and this activity was dropped from PRODIP’s mandate after the January 2014 elections when USAID directed PRODIP to end contacts with MPs. In May 2014, PRODIP submitted a proposal for committee budgets to the Secretariat; the Senior Secretary told the evaluation team that the proposal would not be adopted. MPs interviewed by the evaluation team were split on how great an obstacle this posed to organization of hearings, with some maintaining that the Speaker could authorize spending for hearings when she authorized their organization. Evaluators found the lack of political will among committee chairmen to hold hearings to be a greater obstacle than the financial cost of holding public hearings. Impact of other activities PRODIP’s other attempts to strengthen the committee system showed very modest improvements to its operations. Some committee staff who were trained on committee procedure told evaluators that the training gave them a better understanding of how committees operate and context for performing their jobs, though specific examples were hard to come by. Of 22 committee staff who participated in the telephone mini-survey, 27 percent stated that PRODIP improved their committee’s work process and that PRODIP helped them better prepare minutes of the committee’s meetings. Table 7: PRODIP Committee Activities Q 8: Has PRODIP helped your committee to develop a committee work plan? Frequency Percent Yes 4 18 No 15 68 Do not know 3 14 Total 22 100 Q 9: Has PRODIP helped your committee to improve its work processes?4 Frequency Percent Yes 6 27 No 12 55 Do not know 4 18 Total 22 100 4 Work processes include activities such as report writing, internal policies and procedures, and budgeting. PRODIP Final Evaluation 17 Q 10: Has PRODIP helped your committee to improve its oversight functions? Frequency Percent Yes 0 0 No 18 82 Do not know 4 18 Total 22 100 Q 11: Has PRODIP helped your committee to produce Committee Activity Briefs? Frequency Percent Yes 6 27 No 14 64 Do not know 2 9 Total 22 100 The committee work plans introduced by PRODIP are simple outlines of one to two pages, listing one or two activities per month for a six- or twelve-month period, but they do lend a basic coherence to committee activities and schedule follow-up for committee business. Such planning is routine for parliamentary committees elsewhere. The main obstacle to their widespread adoption is that they are not required for committees. In May 2014, PRODIP submitted a proposal to the Secretariat to institutionalize committee work plans, but the Senior Secretary told evaluators that this proposal would not be adopted. Nevertheless, some chairmen said they found the work plans useful for their oversight role and planned to make them routine. Another PRODIP initiative to improve the functioning of committees was to develop templates for committees to use in their reports, in order to standardize the structure of reports on committee proceedings, committee annual reports, and subcommittee reports. The three report templates proposed by PRODIP have the potential to make committee reports more accessible and useful for MPs; however, they have not been adopted by the Secretariat, ostensibly because the Secretariat is working to reconcile committee template proposals from PRODIP and a completed UNDP project. A meeting was held in March 2014 with Secretariat officials and officials of PRODIP, the UNDP project, and a World Bank Institute project (which developed a similar template for financial oversight committees), with the goal of finalizing the templates for the Speaker’s approval. No further action has been taken, and the two other projects have since closed. PRODIP sought to improve continuity from the Ninth to the Tenth Parliaments by helping seven committees produce committee activity briefs, detailing the work of the panel in the Ninth Parliament, as a reference for new committee chairs and members in the Tenth Parliament. Each brief includes legislative and oversight actions taken, experts engaged, and testimony and reports received by committees, as well as initiatives that remained pending. The briefing books also recorded recommendations presented by the committees to their respective ministries, enabling new committee chairs and members to better monitor and track ministerial actions in response to their recommendations. The briefs are comprehensive and appear to be a useful resource for MPs and staff, but the evaluation team found that MPs and at least one new chair were unfamiliar with the briefs and had not used them in their work. PRODIP attempted to quantify its effectiveness in supporting committees through a Parliamentary Standing Committee Effectiveness Evaluation Index, which rated committees on their meeting organization, research and information, reporting, transparency, lawmaking, oversight, and public outreach according to explicit criteria. This index was computed during the Ninth Parliament, from December 2011 through September 2013. Focusing on the period of this evaluation, PRODIP reported PRODIP Final Evaluation 18 that 12 of 13 committees that received support had improved their effectiveness from March to September 2013, most notably in public outreach. Out of a total score of 35, the average committee score rose from 23 in March 2013 to 29 in September 2013. The average committee score for public outreach rose from 3 to 5, the maximum. TAF stated that it was not feasible to revive the index for the Tenth Parliament because committees were not set up for several months, the modified project scope ceased to work with MPs, and work with committee staff was reduced. CONCLUSIONS FOR QUESTION 2 PRODIP’s efforts to develop public hearings as a tool for committee legislative and oversight activities had a greater impact on the committee system than any other initiative. In a few cases, hearings had demonstrable impact in improving legislation and legislative oversight of the executive. International experience has demonstrated that public hearings can be the most impactful activity that a committee can undertake and enable a committee both to understand the policy implications of proposed legislation and oversee the implementation of laws already enacted. The experience of public hearings in the Ninth Parliament demonstrated significant potential for these activities to become an integral part of Bangladeshi committee activities and to increase the role of committees in legislation and oversight. Unfortunately, the political and parliamentary factors identified above conspired to undermine the public hearing experiment and the successful work that PRODIP undertook in this area. There is no incentive for a committee chair in the Tenth Parliament to hold a hearing that might embarrass a minister from the same party or to recommend changes to legislation that represents government policy, and there is no true opposition to ask the difficult questions that could hold government to account. Committee chairs have been denied reappointment or been pressured to resign as a consequence of amending legislation or asking tough questions. Regardless of the quality of PRODIP training and support, the political environment is proving insurmountable for public hearings to become routine in Bangladesh. Because parliament’s Rules of Procedure call for committee meetings to be closed, PRODIP sought to organize public hearings outside the capital or lasting several days. While these events raised the visibility of parliament and enabled participation by many Bangladeshis from outlying districts, they entailed a cost and a degree of difficulty that may not have been sustainable without external support, even if there were political will for committees to hold hearings. Holding shorter, more frequent hearings in the parliament complex might have routinized the practice and made the organizational challenges less daunting. As mentioned under Evaluation Question 1, PRODIP provided committees with outside research in support of hearings by commissioning and paying academic researchers to write reports. While the quality of these products was generally high, PRODIP missed an opportunity to develop parliament’s own research capacity. As a result, even if there was political will to hold hearings, committees would have difficulty generating the analytic reports needed to provide policy background for MPs at hearings. FINDINGS FOR QUESTION 3 What have been the successes and challenges in those activities of PRODIP in support of the Budget Analysis and Monitoring Unit (BAMU) of the Secretariat? Successes Along with its work on public hearings, PRODIP’s support for the BAMU was the contribution cited most often by MPs and staff when asked about the project. BAMU’s most visible and appreciated contribution was the Help Desk, a booth outside the parliamentary chamber that operated during the month of June, when parliament debated the annual budget that takes effect at the start of the fiscal year on July 1. First operated in June 2012, the Help Desk saw demand treble in two years, with 147 MPs PRODIP Final Evaluation 19 (115 male, 32 female) seeking assistance in June 2014, up from 50 in 2012 and 97 in 2013. This represented 42 percent of MPs. Those MPs asked a total of 225 questions of BAMU in June 2014, up from 155 questions in 2013. All MPs interviewed by the evaluation team cited the Help Desk as the PRODIP contribution they valued most. The main use of the Help Desk by MPs was in drafting their budget speeches, the 10-minute floor statements each MP delivers in a plenary session every June. Under the Bangladeshi budget process, there is no committee consideration of the budget, and parliament cannot change line items in the budget without the consent of the government. Because parliamentary consideration occurs so late in the process, actual changes in the budget are rare. If they have an effect, budget speeches tend to influence the government as it drafts the budget for the following year and the out-years. Parliament’s role is to make recommendations, not amendments. While parliament plays a limited role in the budget process, MPs and staff who are involved in that process uniformly stated that the BAMU Help Desk has enabled MPs to deliver more fact-based, relevant budget speeches than previously, when the statements tended to be based on political platitudes instead of facts and figures. For example, a deputy speaker told evaluators, “The [budget] speeches of the honorable MPs have improved; this is why I support BAMU.” One MP noted that BAMU quickly provided her with data comparing per capita income under the AL government to that of its BNP predecessor. “BAMU is important. It helps us get information and raise our voice. We are thinking about the welfare of society, especially women, children, and low-caste people. This organization helps us collect information on budget and finance,” she said. “Without this, I couldn’t speak properly.” Previously, in the 2013 budget cycle, the same MP said she had requested information on taxes and used it to request a reduction of tax on aluminum pots that was accepted by the government. Such information is particularly useful to opposition MPs, who need independent research to challenge the assertions of the government. “We were the opposition party. We used this data against the government. It made our arguments stronger,” one former opposition MP told the evaluation team. PRODIP has also helped BAMU to produce a variety of written products to assist MPs in understanding the budget process and key issues. Among these are compendiums on the national budget and the national development plan, which serve as references to MPs and staff on the budget process and development planning; mid-term analyses of the budget; budget notes on key sectors; paper clippings on budget analysis; and a booklet analyzing the overall budget called “How About This Year’s Budget?” The documents are well done and are useful for MPs and staff, as well as for students, researchers, civil society activists, and development practitioners. The bilingual development compendium provides an overview of development planning and related processes for use by MPs and parliament staff. The compendium will serve as a reference guide for current and future staff to enable them to provide support on the formulation of development budgets and the preparation of development projects, including M&E, after the end of the PRODIP project. The bilingual budget compendium provides an overview of the legal framework of budget preparation, detailed process of budget preparation, and budget glossary. Current and prospective MPs can use this compendium in delivering budget speeches and reviewing the annual budget. This compendium can be used in the orientation program of newly-elected MPs. “How About This Year’s Budget?” is a brief account of the annual budgets of fiscal years 2013-2014 and 2014-2015. It presents information and analysis about different aspects of the annual budget. It includes macroeconomic analysis, description of sources of incomes and sector-wise allocation, tax collection, public borrowing, and foreign assistance. The booklet, which was distributed to MPs at the BAMU Help PRODIP Final Evaluation 20 Desk, is simple, and the language is Bangla. MPs, politicians, civil society activists, students, researchers, and development practitioners will find it useful. The seven Budget Analysis and Monitoring Notes include analyses of implementation of the government’s fiscal year 2013-2014 Annual Development Plan and the government’s response to climate change, a mid-year evaluation of implementation of the fiscal year 2013-2014 budget, and analyses of budget trends from 2009 to 2014 for the ministries of agriculture, local government, education, and health and family planning. All these publications are useful for MPs to contribute to the budgetary analysis and oversight of the budget implementation. BAMU staff interviewed by the evaluation team reported that, as a result of PRODIP training, they are able to find data from the Ministry of Finance when it is requested by MPs and they can provide MPs with short reports. They said, however, that they are not qualified enough to provide independent budget analysis to MPs or to write the publications that had been produced by analysts detailed to BAMU from the Ministry of Finance with funding from PRODIP. Telephone mini-survey respondents included five BAMU staffers. Of these, three stated that PRODIP assisted in expediting the work of BAMU. Table 8: BAMU Functioning Q 8: Has PRODIP improved the functioning of BAMU? Frequency Percent Yes 3 60 No 1 20 Do not know 1 20 Total 5 100 Challenges A significant challenge for BAMU is that the staff of the unit work for BAMU on a part-time basis, and all have primary responsibilities elsewhere in parliament. These seven individuals are not trained economists or budget experts. PRODIP did fund three budget experts from the civil service who were detailed to BAMU, but that practice has ended. One BAMU staffer said those budget analysts wrote the analytical reports for BAMU. The greatest challenge to the sustainability of BAMU has been that the unit is located outside of the parliamentary structure. In May 2014, PRODIP submitted a proposal to the Secretariat to incorporate BAMU into the structure of the Secretariat. The Senior Secretary told the evaluation team that the Parliamentary Service Commission, the governing body of the Secretariat, is expected later in 2015 to approve a modified plan to incorporate BAMU into the Finance and Information Services Wing of the Secretariat. In Year 5, PRODIP was supposed to establish a webpage for BAMU on the parliamentary website, but this did not occur because the Secretariat stated in November 2014 that its website did not have sufficient server capacity to host the BAMU webpage. The World Bank Institute’s Strengthening Parliamentary Oversight project had designed the BAMU website and provided training to BAMU staff; PRODIP provided content and was supposed to transition the BAMU website to the parliamentary website by the end of 2014. In April 2013, USAID directed PRODIP to eliminate information technology assistance to parliament in response to cuts in the PRODIP budget. UNDP had primary responsibility for assisting parliament in upgrading its online presence through its legislative strengthening project, which closed in June 2014. According to TAF, “PRODIP proposed an alternative plan of including the BAMU webpage on the server for the financial oversight committee webpages; however, the Parliament Secretariat did not agree because BAMU is not a committee. Instead the Parliament Secretariat indicated PRODIP Final Evaluation 21 that it would increase the capacity of its main server to accommodate the inclusion of the BAMU webpage.” This increase in capacity had not occurred at the time of the evaluation. CONCLUSIONS FOR QUESTION 3 BAMU, particularly its Help Desk, is one of the most visible activities of the PRODIP project, and it is likely to be the most lasting legacy. The decisions on its future will be finalized by parliament after PRODIP ends; the most important will be to incorporate BAMU into the Secretariat and to hire a permanent, full-time staff of trained economists and budget analysts. BAMU is not a true parliamentary budget office along the lines of the U.S. Congressional Budget Office or even smaller budget offices in countries like Afghanistan or Kenya. Because of the limited role of the Bangladeshi parliament in the budget process, BAMU functions more like the budget division of a legislative reference service, undertaking basic research and information gathering on budget issues, rather than economic and budgetary analysis. This, however, is well-suited to the needs of the Bangladeshi parliament, given its budget role. Because of this function, thought could have been given to incorporating BAMU within an expanded Research and Education Section of the parliamentary library. This could have expanded the reference services available to parliament and allowed for greater cross-cutting support across all issues on which parliament needs basic research and information. In this light, the budget Help Desk, which has been valued by MPs, could have been expanded into a year-round reference Help Desk that provides background on any issue before parliament, rather than being limited to providing budget information one month per year. FINDINGS FOR QUESTION 4 What have been the changes made in PRODIP’s training strategy for staff of the Parliamentary Secretariat in response to the recommendations of previous evaluations? How effective have these changes been in developing the capacity of the Secretariat? Previous recommendations As noted above, PRODIP was the subject of three mid-term reviews: a DFID review in 2012, a mid￾term evaluation conducted by SI in March 2013, and a USAID RIG audit conducted in June 2013. While acknowledging that the Speaker and Senior Secretary ultimately decided which staff would attend training activities, both the SI evaluation and the RIG audit concluded that training was not always aimed at the appropriate staff (particularly committee secretaries and the personal secretaries of chairmen), and both recommended that PRODIP ensure that appropriate staff receive training with appropriate content. The SI evaluation and the RIG audit both noted the differences between the committee secretaries, who are generally permanent parliamentary staff with responsibility for tasks like writing reports and drafting amendments, and personal secretaries, who are generally civil servants detailed to parliament for short￾term rotations who serve as advisors to the committee chairmen. The reports differed somewhat on the implications of these distinctions for PRODIP. The SI evaluation recommended that the permanent committee secretaries represent the best training investment for PRODIP; the RIG audit faulted PRODIP for not including more personal secretaries in public hearing training. Changes made The evaluation team found that PRODIP did select more appropriate staff for training, particularly committee staff. For example, in a February 2015 training on committee budgeting and work planning for 36 staff, 30 worked either in the committee support branch, as a committee secretary, or as a PRODIP Final Evaluation 22 personal secretary to a chairman. A review of the PRODIP training database found that training was targeted appropriately and that selected staff participated in multiple activities in a given training sequence. For example, of 22 permanent staff who attended the February 2015 training, 13 had participated in earlier public hearing training, 19 had participated in committee procedure training, and eight had participated in committee reporting training. Of those 22 permanent staff, 20 had participated in at least one earlier training. While training was more targeted at permanent staff, as recommended in the mid-term evaluation, some detailed staff continued to be trained at the request of the Secretariat, which ultimately decided which staff would participate in a given training event. For example, 18 personal secretaries received training; with one exception, this was limited to committee budgeting and work planning, public hearings, and committee procedures. “In Year 5, our approach was quality, not quantity,” one PRODIP official said. “We tried to find those staff who would have a long career in parliament,” rather than detailed staff who would rotate out. Effectiveness Parliamentary staff interviewed by the evaluation team generally reported that the training was conducted at a level they found useful given their pre-existing knowledge and, for the most part, they reported that the skill level and responsibilities of their trainee class were relatively uniform. Staff praised the ability of PRODIP instructors to explain advanced concepts in a way that made them understandable. The telephone mini-survey of 30 staff who attended PRODIP training corroborated the interview findings. Of those surveyed, 76 percent agreed or totally agreed that their skills improved through PRODIP training, 80 percent learned a new skill, and 67 percent said training was at an appropriate level. When asked if they were able to use the skills learned, 53 percent said yes. The sample included 18 permanent parliamentary staff and 12 staff detailed from the executive branch. Three are women. Table 9: Usefulness of PRODIP Training Q 2: On a scale of 1 to 5, where “1” means “not useful at all” and “5” means “very useful,” please click the button that corresponds with how useful the PRODIP training was in helping you acquire information relevant to the performance of your responsibilities? Frequency Percent Not at all 0 0 Little agreement 1 3 Somewhat agreed 5 17 Agreed 22 73 Totally Agreed 1 3 Do not know 1 3 Total 30 100 Q 3: Did the training provide you a particular tool/skill/piece of information you did not have before? Frequency Percent Yes 24 80 No 5 17 Do not know 1 3 Total 30 100 PRODIP Final Evaluation 23 Q 4: What was the level of the training relative to your experience with the material? Frequency Percent Too basic 9 30 Appropriate 20 67 Too advanced 0 0 Don't know 1 3 Total 30 100 Q 5: Have you used the tool/skill/piece of information (referenced in Q 3) in performing your responsibilities? Frequency Percent Yes 16 53 No 13 43 Do not know 1 3 Total 30 100 The greatest complaint from staff was that training events were too short. Of 12 staff who spoke with the evaluation team about their personal experience, seven stated that the training should be longer. This was particularly true of the two Training of Trainers (ToT) events, which were each only two days. In addition, evaluators reviewing training materials found that some training did not have a parliamentary component or was not linked to the participants’ parliamentary duties. For example, one research training had parliamentary staff develop 75-page academic papers rather than shorter reports aimed at MPs, while one training for BAMU staff focused on macroeconomic policy. “We learned, but we didn’t have a chance to apply it,” said one BAMU staffer who attended that session. “Some were useful, and some were irrelevant to our work,” said one legislative support staffer who attended the research training. “There was not enough skill or knowledge on legislative research. It focused on research methodology … and the duration was not adequate.” CONCLUSIONS FOR QUESTION 4 PRODIP did make appropriate changes to its training strategy in response to recommendations from previous reviews. The sequence of training offered was coherent and targeted appropriate staff. However, training was too short to be effective, particularly the ToT events that were intended to make PRODIP training sustainable after the project ends. In addition, while staff reported that they learned new skills, the reduced pace of parliamentary activity in the Tenth Parliament means that some staff have not had the opportunity to use their training. PRODIP training could have been more effective if it had kept in mind the parliamentary dimension of staff duties. For example, rather than training on macroeconomic theory, staff might have benefited from learning how to analyze government budget documents. Research training should have focused on shorter, applied reports that MPs would read and use. In particular, researchers would have benefited from assistance in developing products that would be useful for MPs, such as the six-page reports and targeted memos that the U.S. Congressional Research Service uses to communicate information and analysis to Members of Congress. FINDINGS FOR QUESTION 5 Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? The previous reviews were critical of PRODIP’s management system during its first three years of operation. The SI mid-term evaluation recommended that PRODIP disaggregate trainee data for its own PRODIP Final Evaluation 24 use and for that of stakeholders. The RIG audit was critical of lack of documentation in support of performance data; the audit recommended that appropriate documentation be maintained to support results data and that spot testing be conducted during field visits to verify data. Those earlier reviews criticized the lack of integration between the parliamentary component, implemented by SUNY/CID until February 2014, and the civil society component, implemented by TAF until that component was terminated in February 2014. This evaluation team found that PRODIP did improve its performance management system in response to those recommendations. In particular, PRODIP centralized its supporting documentation from its two components, which had previously maintained separate repositories. PRODIP also instituted spot checks of its field events, which mostly were organized by the civil society component. PRODIP also worked to align its M&E indicators with U.S. Government Standard Foreign Assistance Indicators, added two new indicators to measure synergy between the parliamentary and civil society components, and simplified some spreadsheets. All staff and civil society partners were trained to use the new system and file monthly reports. The evaluation team found that PRODIP did maintain appropriate documentation to support reported performance results, such as attendance rosters for each training event. However, the training database itself was not finalized until late February 2015, long after it would have been useful for PRODIP activities. PRODIP planned to turn over the database to the Secretariat as a reference for future training. CONCLUSIONS FOR QUESTION 5 While PRODIP’s performance management system improved in response to recommendations from previous evaluations, it remained far from ideal. For example, while the trainee database can be sorted by sex, this information was not included in quarterly performance management reports. PRODIP’s training database was completed after the project completed its training activities, rather than being maintained and updated throughout the life of the project. As a result, monitoring data maintained in the database could not be used to inform project decision-making in real-time. FINDINGS FOR QUESTION 6 To what extent have PRODIP management and stakeholders incorporated knowledge gained through M&E into project management? The mid-term reviews were critical of PRODIP’s failure to incorporate knowledge gained through M&E into project management. The SI mid-term evaluation recommended that PRODIP gather information that would be of better management use and of interest to stakeholders, including an annual review of indicators for stakeholders. The DFID evaluation recommended establishment of a knowledge bank that would be easily accessible to beneficiaries. This evaluation team found no evidence that PRODIP management and stakeholders incorporated knowledge gained through M&E into project management. The former director of Research, Monitoring and Evaluation at PRODIP, a part-time position, told evaluators, “We shared findings and lessons learned in different meetings. To be honest, the implementing team hardly deviated from the work plan. We gave them input, but I didn’t see much change in program activities.” The former Chief of Party stated that PRODIP would have put M&E data to use in project management in Year 5; however, the project was downsized significantly at that point, and his own position was eliminated. A PRODIP official who remained until the end of the project stated that cuts in staffing ultimately reduced PRODIP from 17 to four staff, so M&E tasks like updating the database took a backseat to more immediate needs like organizing training activities. PRODIP Final Evaluation 25 CONCLUSIONS FOR QUESTION 6 PRODIP did not use M&E knowledge to modify and improve project management and design. As a result, PRODIP was not able to respond to opportunities and challenges that arose in between the annual drafting of its work plans. FINDINGS FOR QUESTION 7 How sustainable are the results of PRODIP’s support to the Parliamentary Secretariat beyond USAID support? What measures has PRODIP taken in Year 4 and Year 5 to ensure sustainability? Sustainability A few key contributions of PRODIP to parliament are sustainable after the project ends in April 2015, but many of the project’s initiatives are likely to end without support from USAID or other donors, with no effort by parliament to sustain them. Both UNDP and the World Bank Institute have closed their parliamentary strengthening projects, and DFID has no concrete plans for future support to parliament. “We have a hard time calling this a success,” one donor official said. “It is a question of sustainability.” There is no parliamentary development or reform strategy, and the evaluation team found no internal support for parliamentary reform without donor assistance. “There is a lack of political will for reform in parliament,” one donor official said. “There is no coherent agenda for parliamentary development. From inside, there is no interest at all in reforming parliament. The question of who can drive reforms was never answered.” There is a sustainability plan for BAMU, and it appears likely that the unit will be continued by the parliament. The plan has been modified somewhat from the original PRODIP design; as approved by the Senior Secretary and the Speaker, it calls for BAMU to have a staff of 22, including 13 support staff, rather than the five staff envisioned by PRODIP. Furthermore, BAMU reference documents like the Budget Compendium and Development Compendium are lasting resources for MPs and staff. One deputy secretary who has worked closely with BAMU estimated that BAMU will require external support for the next two years, and he stated that parliament would attempt to hire outside consultants from a local economics association. Some other PRODIP initiatives are sustainable after USAID support ends. In the library, the software and training for archiving parliamentary debates will continue to be used; however, electronic cataloging software for which PRODIP was to have provided training has not yet been procured by parliament. With regard to research, PRODIP’s commissioning of outside research is unlikely to be funded by parliament. While the Research and Education Section is growing, with a sixth staffer being hired (compared to three staff when PRODIP began), the project did not help to develop the section into a true parliamentary research service. PRODIP stopped all support to MPs in January 2014 at the direction of USAID; however, PRODIP did move forward with an MP Handbook that is being provided to the Secretariat. This document was to be printed in March 2015 and should be a lasting resource for members. With regard to committee activities, PRODIP support for public hearings provided a basis for them to become sustainable. The public hearing manual provides detailed accounts of the process of holding public hearings in a systematic way, including the planning, conduct, questioning, and follow-up, as well as practical experience from public hearings held in Bangladesh. Staff told evaluators that PRODIP has provided them with the necessary knowledge to organize hearings without PRODIP support. The political challenges to future public hearings are the greatest obstacle. There is no internal demand in parliament to hold hearings, there are disincentives for chairmen to call hearings, the Rules of Procedure PRODIP Final Evaluation 26 do not permit hearings as a matter of course, and the Senior Secretary stated that there will not be committee budgets to enable them to pay for hearings. Other committee initiatives could be sustainable. Staff trained by PRODIP are capable of drafting semi￾annual committee work plans, but these are not required and the Senior Secretary said that there is no plan by parliament to require that committees submit them. The committee activity briefs detailing the activities of committees in the Ninth Parliament are completed, but there is no evidence that parliament will undertake a similar initiative at the end of the Tenth Parliament. Measures taken PRODIP has undertaken several measures to ensure the sustainability of its initiatives. With regard to BAMU, PRODIP in May 2014 submitted a sustainability proposal to the Secretariat, which itself revised the proposal in September 2014. The proposal has been approved by the Senior Secretary and the Speaker, and final approval from the Parliamentary Service Commission is expected in 2015, in time for the Help Desk to operate for consideration of the 2015-2016 budget. PRODIP likewise in May 2014 submitted a sustainability plan for committee work plans and budgets. The Senior Secretary told evaluators that this plan will not be adopted. PRODIP shifted its training in November and December 2014 to a ToT model, with an eye toward having participants serve as trainers for future sessions after PRODIP ends. Two courses were offered on bill analysis and on research methodology, and each of these courses lasted only two days. The ToT on bill analysis was a total of eight hours of instruction, and participants told the evaluation team that the time was too short. According to the PRODIP report, “The objective of the training was to form a trainer group within the Parliament Secretariat staff on the issue of technical content of Bills and procedure for bill drafting and analysis.” It appears that this goal will not be reached. The Senior Secretary told evaluators that parliament does not plan to continue PRODIP training and does not plan to use as trainers staff who completed the PRODIP ToT course. “You cannot make trainers in two days,” he said. “Without at least 20 sessions, ToT is useless.” PRODIP resource documents represent another measure to ensure sustainability. These include 14 major publications like the Budget Compendium, Development Compendium, MP Handbook, and public hearing manual, as well as committee report templates. CONCLUSIONS FOR QUESTION 7 A lack of political support represents the greatest threat to the sustainability of PRODIP initiatives. There is no internal movement for parliamentary development or reform, and the current political crisis compounds this threat to sustainability. There are no incentives for leadership, MPs, or the Secretariat to make major changes in how parliament operates. As a result, initiatives like public hearings that could have been sustainable with political support are likely to end. While some residual knowledge will remain among permanent staff and in PRODIP publications, it is unlikely that parliament will be able to organize hearings on its own if much time passes without such experience. BAMU appears to have political support in parliament and commitment from leadership. While PRODIP’s original proposal of five staff is more realistic than the Secretariat’s 22-strong unit, parliament has taken ownership of this entity and demonstrated a commitment to its continued operation. The proposal for committee budgets and work plans is likely to meet a different fate, with no plans to mandate these. While staff of targeted committees can draft work plans, the practice will not become widespread absent a parliamentary requirement. PRODIP Final Evaluation 27 PRODIP missed an opportunity to develop a sustainable research capacity in parliament. With its emphasis on paying for outside research, PRODIP failed to develop the Research and Education Section into a true parliamentary research service, and parliament does not appear willing to pay itself for outside research. PRODIP Final Evaluation 28 RECOMMENDATIONS IMMEDIATE RECOMMENDATIONS Recommendation 1: USAID should not fund a new legislative strengthening project until free, fair, and inclusive elections are held. The current parliament is not representative, with the primary opposition party excluded, and it has shown no interest in serving as a counterweight to an executive that is consolidating power and marginalizing all opposition. Furthermore, lack of internal will to reform parliament means that a legislative strengthening project would fail to achieve traction. Recommendation 2: USAID/Bangladesh staff should remain engaged with senior staff of the parliament after PRODIP ends, in order to provide advice and encouragement to make PRODIP initiatives sustainable, particularly the effort to incorporate BAMU into the Secretariat. PRODIP is scheduled to end in April 2015, approval of the BAMU plan is expected later in 2015, and the BAMU Help Desk is scheduled to operate that year. Beyond this, no further engagement is warranted. FUTURE RECOMMENDATIONS While concurring that the political situation in 2015 is not conducive to a new legislative strengthening project, USAID/Bangladesh officials asked the evaluation team to develop recommendations that could be useful to the Mission in the future, should the political environment improve and consideration of a new legislative strengthening project be justified. The evaluation team was asked to draft this section for the time after free, fair, and inclusive elections take place and a representative parliament is seated. Overall considerations Recommendation 3: USAID officials in the next post-election period should first of all undertake a critical analysis of what the elections represent. Legislative strengthening projects tend to be successful when one can identify a clear transition to a democratic system. This may be a revolution, initial democratic elections, end of military rule, or independence. International experience shows that some events that seemed like democratic transitions may, in retrospect, have little effect on political culture. The December 2008 elections in Bangladesh represented such a case. At the time, the free and fair elections were seen by many as a break from the military-backed caretaker government and a return to civilian democratic rule. In retrospect, the 2008 elections simply represented a return to the past following a two-year pause, not a real democratic transition. Countries that “muddle along” in their democratic development tend not to be good candidates for a legislative strengthening project, unless there is a clear initiative by the parliament itself to reform and develop institutionally. Legislative strengthening projects work best when there are MPs who already demonstrate a commitment to institutional reforms, ideally in an organized fashion. Successful projects also require MPs to be receptive to training, not just to the political trappings of a relationship with the U.S. Government or the possibility of study visits abroad. Projects can potentially succeed even if leadership is ambivalent toward reforms, but there must be a critical mass of reform-minded MPs willing to advance reforms even without the full backing of the leadership. Even then, USAID/Bangladesh should ask whether a stronger parliament—one better able to analyze and amend legislation, conduct oversight hearings, and represent its constituents—will have an impact on the political system. Even after the 2008 elections, it became evident that parliament did not have the power to rein in the executive. Recommendation 4: The evaluation team recommends that future USAID/Bangladesh officials lean against developing a legislative strengthening project for the Bangladeshi parliament unless there is clear, PRODIP Final Evaluation 29 compelling evidence that a true democratic transition has taken place. Ideally this transition would see a core group of MPs with the commitment and political will to reform parliament and develop it as a counterweight to executive power. It is important to remember that free, fair, and inclusive elections do not always represent a democratic transition. Political considerations As noted in the findings, the executive branch in Bangladesh exercises outsized influence, even for a Westminster system. The following structural features limit parliamentary independence and are arguments against investing in a legislative strengthening project: • Ministers remain MPs; • The PM is the leader of the House and not subject to parliamentary oversight; • Ministers sit on the parliamentary committees that oversee their ministries; • MPs are not permitted to vote against their parties in parliament; • Parliamentary candidates must be nominated by their parties; • Parliament cannot amend the budget submitted by the government; • The Auditor General reports to the Ministry of Finance, not parliament. If some of these elements of the political structure in Bangladesh are changed, a project could be successful. If not, improvements in parliamentary operation and capacity may do little to make the political system more democratic or to increase parliament’s role as an independent counterweight to executive power. Potential Project Elements Recommendation 5: If USAID/Bangladesh determines that the circumstances are right for a new parliamentary strengthening project, it should design the project with input from the Parliament Secretariat and from any parliamentary bodies to be supported. Any new project should be limited in scope and tightly focused. Project elements could include: • A strategic plan for reform of the Secretariat, including establishment of a Parliamentary Service, independent of the executive civil service, from which all parliamentary staff will be drawn. • Development of the Bangladesh Institute of Parliamentary Studies, which was authorized by an act of parliament in 2001, but never implemented. If parliament were to approve funding and staffing, the Bangladesh Institute of Parliamentary Studies could be developed as parliament’s research service, with BAMU incorporated as its budget analysis unit. • Institutionalization of public hearings in committees. This should include amendment of the Rules of Procedure to explicitly permit committees to hold open meetings. • Development of best international practices in committee oversight, including amendment of the Rules of Procedure to compel executive branch officials to appear before committees and to compel ministries to share requested documents. This component should also work to improve the relationship between the Auditor General and the key fiscal oversight committees and seek to change the Rules of Procedure to require that opposition MPs chair those committees. • Development of a committee role in reviewing and amending the national budget. PRODIP Final Evaluation 30 LESSONS LEARNED • Free, fair, and credible elections do not always represent a democratic transition. After two years of rule by the military-backed caretaker government, the December 2008 elections were welcomed by the international community as a return to civilian, democratic rule in Bangladesh. USAID, DFID, UNDP, and the World Bank Institute all moved to establish legislative strengthening projects to support the democratically-elected parliament. In retrospect, the 2007-2008 period merely represented a pause in the imperfect course of Bangladeshi democracy, and the 2008 elections did not represent change. • Buy-in from leaders of the parliamentary institution is essential. A legislative strengthening project cannot change a parliament if the parliament does not want to change. PRODIP undertook an ambitious development agenda, but parliament itself had no development plan or reform agenda, and there was no reform caucus or parliamentary service commission composed of MPs committed to change. As a result, changes needed for PRODIP initiatives to become sustainable, such as amending the Rules of Procedure to permit public hearings as a matter of course, lacked for advocates and ultimately were not enacted. • Smaller, targeted parliamentary strengthening projects can be more effective than large projects. The three mid-term reviews of PRODIP all highlighted the problems encountered in trying to coordinate a parliamentary strengthening component with a civil society support component in the same project. Both donors and implementers reflected that the project was probably too big, with too much funding and too many activities. Driven by budget constraints and a deteriorating political environment, USAID ultimately eliminated the civil society component of PRODIP and pared down its parliamentary activities. A smaller project with an exclusively parliamentary focus may have proven more effective. • Downsizing a project scope can be an appropriate response when the political environment changes. The flawed January 2014 elections provided conclusive evidence that the 2008 elections did not represent a democratic transition in Bangladesh and that the country’s democratic development was regressing. The decision to reduce the scope of PRODIP under these circumstances was correct. USAID should be open to downsizing or even closing DRG projects when political circumstances change for the worse. • Projects must take into account the political context and the powers of the particular parliament. Before seeking to develop a given capacity or body within a parliament, it is necessary to assess how that capacity or body can improve parliament’s operation. For example, the parliament of Bangladesh cannot amend the government’s budget, and the budget role of MPs is mostly limited to a budget speech and making recommendations. In light of this, a full-service parliamentary budget office is not needed because parliament does not write or amend the budget. A reference service that can provide MPs with information needed for their speeches and recommendations is a more appropriate body. • Work for changes to rules of procedure, if needed. PRODIP worked to establish public hearings in Bangladeshi parliamentary committees, but the parliamentary Rules of Procedure do not permit such hearings as a matter of course. In order for hearings to be sustainable, USAID should have permitted PRODIP to work for the necessary rules change. PRODIP Final Evaluation 31 • Building institutions like a research department is more sustainable than paying for research papers, software, or events. In order to support committees, particularly in conducting public hearings, PRODIP contracted with outside researchers to provide them with analytic reports. While the research was generally of good quality, this practice did nothing to develop a sustainable research capacity in parliament. Resources are better spent on developing parliament’s own research capability so as to leave behind a lasting capability after the project ends. • Training must be targeted to parliamentary audiences. MPs do not have time to read lengthy academic research papers. In order to be useful for MPs, research must be timely, relevant, practical, and concise. Training for parliamentary staff should always focus on which skills staff need to learn and how they can use those skills to serve MPs. • Implementers should incorporate M&E into regular project review. For a project to be coherent, it is important to have a long-term plan and to break that down into annual work plans. However, circumstances can change more rapidly than the annual planning cycle, and an implementer who is locked into a work plan may end up directing resources into dead-end activities while missing new opportunities for programming. M&E should be part of a constant feedback loop, enabling implementers to identify which activities might merit increased resources and which activities should be considered for termination. PRODIP Final Evaluation 32 ANNEXES PRODIP Final Evaluation 33 ANNEX I: EVALUATION STATEMENT OF WORK Statement of Work for the Promoting Democratic Institutions and Practices (PRODIP) program Final Performance Evaluation USAID/Bangladesh Office of Democracy and Governance Program Identification Data Program Title: Promoting Democratic Institutions and Practices (PRODIP) Program Number: 388-A-00-10-00092-00 Program Dates: April 28, 2010 – April 27, 2015 Program Funding: $14,268,933 (reduced from $23,210,783) Implementing Organization: The Asia Foundation (TAF) and State University of New York/Center for International Development Assistance Officer Representative (AOR): Billy Woodward I. BACKGROUND Since independence, Bangladesh has held democratic elections sporadically, including three successive peaceful transfers of power from 1991 to 2006. Yet, the country’s development as a democracy has been interlaced with military rule, debilitating political polarization, ineffective institutions of governance, and endemic corruption. Partisan deadlock over electoral issues resulted in the cancellation of the January 2007 parliamentary elections and the self-imposed rule by the military-backed caretaker government. Following the two-year state of emergency enforced by the caretaker government, a new chapter in the country’s history opened for democratic rule with the success of the free and fair parliamentary elections held on December 29, 2008. The difficult but successful transition back to elected government brought about new opportunities for democratic development and a new sense of urgency in the country to reform political practices and institutions of governance. This new chapter also presented the United States with an historic opportunity to help Bangladesh improve the country’s governance, foster economic development, and deny space to terrorists. Political improvements in Bangladesh have not kept pace with economic expansion over the past decade. Despite continued economic gains, including economic growth averaging 5.66% over the past 10 years (compared to the world average of 3.49% for the same period), government institutions and political parties (particularly the Awami League and the Bangladesh Nationalist Party) continue to operate in a patron-client relationship with the people of Bangladesh. This relationship and a winner-take-all approach to governing after securing electoral victory has stalled Bangladesh’s political and democratic advances. The Jatiyo Sangsad (Parliament) has especially suffered under this system, as ruling-coalition Members of Parliament (MPs) take little interest in their legislative, oversight, or public policy responsibilities and “loyal opposition” MPs simply refuse to participate in government. As a result, Bangladeshis have traditionally had little space (or interest) in pressing their elected leaders for changes to the status quo, which only heightens the disconnect citizens have with their government. In April 2010, USAID awarded a Cooperative Agreement to The Asia Foundation (TAF) to implement a five-year legislative function and processes improvement program known as Promoting Democratic Institutions and Processes (PRODIP) with a total estimated cost of $23,210,783. PRODIP was jointly PRODIP Final Evaluation 34 funded with United Kingdom aid from the Department of International Development (DfID). PRODIP’s two objectives were (1) An Effective and Open Parliament and (2) A More Constructive and Sustainable Role for Civil Society in Democratic Governance. Under these objectives, PRODIP targeted the following results: (1) Improved parliamentary lawmaking skills and oversight capacity; (2) A stronger parliamentary committee system; (3) Increased public awareness of—and citizen participation in—parliamentary functions and processes; (4) Strengthened institutional capacity of Civil Society Organizations (CSOs) in areas of public and democratic governance; (5) More informed public policy-making and monitoring of policy implementation at the national and sub-national level; and, (6) Increased citizen “voice” for transparent and accountable governance. PRODIP’s strategic approach to achieving these results was to simultaneously assist both parliament (supply-side – Component 1) and Civil Society Organizations (demand-side – Component 2) in creating an open and effective Parliament and to foster a more constructive and sustainable role for civil society in democratic governance. On the supply side of the governance equation, PRODIP has assisted Parliament to improve its lawmaking and oversight capability, strengthen its committees, increase public awareness of its functions, and create opportunities for increased public input into the development of national policy. On the demand side, PRODIP has assisted civil society to assist Parliament to become more effective, transparent, and participatory by building capacity in the areas of legislation, oversight, and representation by providing technical assistance, training, and institutional support to CSOs to help shape public policy and provide a vehicle for public input. PRODIP has undergone two rounds of modifications to the program activities; this coincided with a reduction in the overall budget from $23,210,783 to $14,268,933, owing in part to reductions in availability of budget resources. The first round of modifications was in response to a series of evaluations by DfID (2012), the USAID Regional Inspector General (RIG) (2013), and USAID/Bangladesh (2013)—the latter conducted by Social Impact, Inc. (SI) through the Bangladesh Democracy and Governance Program Evaluations (BDGPE) project; the second round of modifications was in response to the changed political conditions resulting from the 2014 Parliamentary elections, which called into question some of the premises of the program with respect to the role of the Parliament and its relations with the Executive. As a result of these modifications to the Cooperative Agreement, all of the planned results and activities under Objective 2, A More Constructive and Sustainable Role for Civil Society in Democratic Governance, were scaled back in the first round (Year 4 Work Plan), then eliminated entirely in the second round (Year 5 Work Plan). Under Objective 1, there were also significant cutbacks in Years 4 and 5 of PRODIP, leaving the following activities accountable: Result 1: Improved Parliamentary Lawmaking Skills and Oversight Capacity • Activity 1.2: Provide Research, Report Writing, and Drafting Training for Parliamentary Staff • Activity 1.3: Support Policy Research • Activity 1.4: Establish Research and Documentation Services in Parliament • Activity 1.11: Support the orientation of new members of the Tenth Parliament • Activity 1.12: Support for BAMU – Provide Financial Analysis Services to Parliament PRODIP Final Evaluation 35 Result 2: A Stronger Parliamentary Committee System • Activity 2.2: Develop Committee Work Plans • Activity 2.3: Engage Target Committees to Improve Committee Work Processes • Activity 2.4: Engage Target Committees to Improve Oversight Functions • Activity 2.6: Produce Committee Activity Briefs Result 3: Increased Public Awareness and Participation in Parliamentary Functions and Processes • Activity 3.4: Support Public Hearings II. OBJECTIVES OF THE EVALUATION This external, final performance evaluation will review the progress made toward achieving the results/outcomes of those activities that were retained by PRODIP under Results 1, 2 and 3 of Objective 1. Specifically, the evaluation will: • Evaluate PRODIP’s performance by assessing results against Objective 1 stated targets and indicators set out in the Agreement, Modifications of Assistance, Year 4 and Year 5 Work Plans, and other documents; • Assess the efficacy and results of the PRODIP implementation approaches and management structure in meeting Objective 1; • Evaluate to what degree the implementing partner in the design of and implementation of program activities took into consideration the recommendations of the previous evaluations of PRODIP; and • Identify lessons learned for future programming in the field of legislative strengthening in Bangladesh by USAID, DfID, and other donors. The audience for this evaluation is USAID/Bangladesh, USAID/Washington, leaders of USAID Forward, TAF, and existing USAID partners such as DfID, UNDP, and other bi-laterals and multi-lateral donors to Bangladesh. III. EVALUATION QUESTIONS This Scope of Work (SOW) is for a final evaluation of PRODIP’s last two years of implementation (April 2013 to April 2015). The evaluation should review, analyze, and evaluate the PRODIP program along the following criteria, and, where applicable, identify opportunities and recommendations for improvement. In answering these questions, the evaluation team should assess the performance of both USAID and its implementing partner(s). A. Results 1. To what extent has PRODIP been successful in achieving the program objectives as set out in the Agreement, Modifications to Assistance, Year 4 and Year 5 Work Plans and other program documents? 2. What has been the impact of PRODIP on the functioning of the Parliament’s committee system? What have been the major obstacles to this activity? How has PRODIP sought to overcome them? 3. What have been the successes and challenges in those activities of PRODIP in support of the Budget Analysis and Monitoring Unit (BAMU) of the Secretariat? PRODIP Final Evaluation 36 4. What have been the changes made in PRODIP’s training strategy for staff of the Parliamentary Secretariat in response to the recommendations of previous evaluations? How effective have these changes been in developing the capacity of the Secretariat? B. Management and Administration 5. Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? 6. To what extent have PRODIP management and stakeholders incorporated knowledge gained through M&E into project management? C. Sustainability 7. How sustainable are the results of PRODIP’s support to the Parliamentary Secretariat beyond USAID support? What measures has PRODIP taken in Year 4 and Year 5 to ensure sustainability? IV. PROPOSED EVALUATION METHODOLOGY The detailed methodology of this final evaluation will be designed by the evaluation team in the work plan; this will include presentation of a data collection matrix that will explicitly link evaluation questions to particular data collection approaches and data sources. It is suggested that the evaluation team use a mixed methods approach utilizing both qualitative and quantitative analysis. Some illustrative methods (to be finalized by the evaluation team in coordination with the Mission), include: • Review of background documents, including, for example: the technical proposal from the PRODIP award; the Modifications to Assistance; all PRODIP Quarterly and Annual Performance Reports from Year 4 and Year 5; PRODIP Work Plans and M&E Plan; DfID’s annual reviews that outline PRODIP’s successes and on-going challenges; the RIG report (2012); the mid-term evaluation conducted by SI (2013); reports from PRODIP-sponsored surveys and analytical research; and lists of PRODIP grantees and subcontractors with activity descriptions, levels of PRODIP resources, and points of contact. • Key Informant Interviews with USAID/Bangladesh, USAID/Washington, PRODIP staff, leaders and staff of Parliament, the Parliament Speaker, the Parliament Secretariat, CSOs, citizens’ organizations and other donors, senior TAF staff. • Focus group discussions with CSOs, journalists, public intellectuals, etc., informed about PRODIP. The evaluation team should review the proposed methodology and alternative methods in light of the evaluation questions, timeframe, budget, data collection requirements, quality of existing data sources, and potential biases. The team will build on the proposed methodology and provide more specific details on the evaluation methodology in the evaluation work plan (see Deliverables below). The evaluation will focus on generating practical recommendations for future USAID programming, and the evaluation methodology will be finalized through review and discussion between USAID/Bangladesh and the evaluation team. The methodology narrative should discuss the merits and limitations of the evaluation methodology. The evaluation team will design appropriate tools for collecting data from various sources. Data collection protocols will be prepared for interviews or discussions with each type of target group. The tools will be shared with USAID at the in-brief and as part of the evaluation report. PRODIP Final Evaluation 37 The information collected should be evidence-based and will be analyzed by the evaluation team to establish credible answers to the questions, provide major trends and issues, draw conclusions, make recommendations, and identify lessons learned. USAID requires that evaluations explore issues of gender; thus, the evaluation should examine gender issues within the context of the evaluation of PRODIP activities. Specifically, what was the gender balance on PRODIP sponsored workshops, trainings, and other forums? Methodological limitations and challenges for this evaluation are expected to include: • Ensuring that samples of interview sources are sufficiently representative to support evaluation findings; • Taking systematic actions to counter any biases in (a) reporting by data collection sources and (b) interpretations of collected data by the evaluation team; and • Ensuring that the evaluation team is able to measure “actual” results, which will only be possible if data beyond respondent perceptions can be gathered and analyzed – for example, changed activities of Parliamentary committees, or research services provided by the Secretariat to MPs or committees. The work plan should discuss the merits and limitations of the final evaluation methodology. The evaluation team will be required to perform tasks in Dhaka, Bangladesh. V. EXISTING SOURCES OF INFORMATION USAID/Bangladesh DG Office will provide documents for the desk review (see Annex A for a full list of documents) and contact information for relevant interviewees. In-country staff of SI’s BDGPE project also has established a working relationship with PRODIP staff, to support sharing of information for the evaluation. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation. VI. Deliverables All deliverables are internal to USAID and the evaluation team unless otherwise instructed by USAID. Evaluation deliverables include: Evaluation Team Planning Meeting(s) – essential in organizing the team’s efforts. During the meeting(s), the team should review and discuss the SOW in its entirety, clarify team members’ roles and responsibilities, work plan, develop data collection methods and instruments, review and clarify any logistical and administrative procedures for the assignment, and prepare for the in-brief with USAID/Bangladesh. Work Plan – Detailed draft work plan (including task timeline, methodology outlining approach to be used in answering each evaluation question, team responsibilities, and data analysis plan): Within 5 working days after commencement of the evaluation. In-brief Meeting – In-brief with USAID/Bangladesh: Within 2 working days of international team members’ arrival in Bangladesh. Evaluation Design Matrix – A table that lists each evaluation question and the corresponding information sought, information sources, data collection sources, data analysis methods, and limitations. PRODIP Final Evaluation 38 The matrix should be finalized and shared with USAID/Bangladesh before evaluation fieldwork starts. It should also be included as an annex in the evaluation report. Data Collection Instruments – Development and submission of data collection instruments to USAID/Bangladesh during the design phase prior to the commencement of the evaluation fieldwork. The completed evaluation should also include the data collection tools, instruments, and list of people interviewed as an annex in the evaluation report. Weekly Updates – The evaluation Team Leader (or his/her delegate) will brief the BDGPE COR on progress with the evaluation on a weekly basis, in person or by electronic communication. Any delays or complications must be quickly communicated to USAID/Bangladesh as early as possible to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities for the evaluation should also be discussed with USAID/Bangladesh. Debriefing with USAID – PowerPoint presentation of initial findings, conclusions, and preliminary recommendations to USAID/Bangladesh before the international team members depart from Bangladesh. Debriefing with Partners – The team will present the major findings from the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from the country. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider partner comments and incorporate them appropriately in drafting the evaluation report. Draft Evaluation Report – The evaluation team will analyze all data collected during the evaluation to prepare a draft Performance Evaluation Report and submit the report within 15 working days after the departure of international team members from Bangladesh. The draft report must be of a high quality with well-constructed sentences, and no grammatical errors or typos. The report should answer ALL the evaluation questions and the structure of the report should make it clear how the evaluation questions were answered. The draft report must meet the criteria set forth under the final report section below. USAID will provide comments on the draft report within 10 working days of submission. The Evaluation Team will in turn revise the draft report into a final Performance Evaluation Report, fully reflecting USAID comments and suggestions. Final Report: The evaluation team will submit a final Performance Evaluation Report that incorporates Mission comments and suggestions no later than 10 working days after USAID/Bangladesh provides written comments on the draft Performance Evaluation Report. The format of the final report is provided below. The final report must meet the following criteria to ensure its quality:  The evaluation report must represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why.  Evaluation reports shall address all evaluation questions included in the SOW.  The evaluation report should include the SOW as an annex. All modifications to the SOW— whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the technical officer. PRODIP Final Evaluation 39  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included as annexes in the final report.  Evaluation should include tables, graphs, and maps as necessary.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. The format of the final performance evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, recommendations, and lessons learned. Where appropriate, the evaluation should utilize tables and graphs to link with data and other relevant information. The report should include, in the annex, any “Statement of Differences” by any team member or by USAID on any of the findings or recommendations. The report should not exceed 30 pages, excluding annexes. The report will be submitted in English, electronically. The report will be disseminated within USAID. Upon instruction from USAID, SI will submit (also electronically, in English) this report excluding any potentially procurement-sensitive information to Development Experience Clearinghouse (DEC) for dissemination among implementing partners, stakeholders, and the general public. The DEC submission must be within three months of USAID’s approval of the final report. All quantitative data, if gathered, must be (1) provided in an electronic file in easily readable format; (2) organized and fully documented for use by those not fully familiar with the project or the evaluation; (3) owned by USAID and made available to the public barring rare exceptions. A CD with all the data could be provided to the COR. VII. TEAM COMPOSITION AND QUALIFICATIONS USAID/Bangladesh wants to engage the services of a team of one local and one international consultant to work on the evaluation team. The evaluation team will require: 1. An international Senior Evaluation Specialist (Team Leader) with experience in evaluating legislative strengthening programs in developing countries. The team leader will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to USAID/Bangladesh. At least eight (8) years of experience in evaluation management is required. Experience in conducting evaluations or assessments of legislative function and processes improvement programs in developing countries are required. Ability to produce a high quality evaluation report in English is essential. PRODIP Final Evaluation 40 2. A Bangladesh-based Parliamentary Sector Specialist. Team member experience should include graduate level economics, social science, law, and/or public finance training. At least six (6) years of experience in democracy and governance programs and some experience managing or implementing legislative function and processes improvement programs is required. In-depth knowledge of issues relating to Bangladeshi legislative function and process improvement programs, improving governmental institutional capacity, and fostering civil society oversight of parliament in developing and/or transitional economies and democracies is required. Familiarity with legislative strengthening “best practices” methods and programming is essential. Some experience in conducting evaluations or assessments is expected. Ability to conduct interviews and discussions and write well in English is essential. Overall the team will need expertise in USAID practices and expectations in program evaluation; program design and analysis; quantitative and qualitative data collection and analysis; survey design and analysis; program issues, innovations and challenges in building parliamentary capacity; and USAID practices and requirements in program performance measurement. VIII. CONFLICT OF INTEREST All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. USAID/Bangladesh will provide the conflict of interest forms. IX. SCHEDULING AND LOGISTICS Funding and Logistical Support The proposed evaluation will be funded and implemented through the BDGPE project. SI will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate consultants. SI support includes arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. SI will make all logistical arrangements (including vehicles for travel around Dhaka) and should not expect any logistical support from the Mission. The team should also make its own arrangements regarding space for team meetings and equipment support for producing the report. Evaluation Schedule The evaluation is planned to follow the schedule and Level of Effort breakdown given below. Task/ Deliverable Proposed Dates Team Leader Local Parliamentar y Expert Review background documents & preparation work (offshore) 2/2/2015- 2/19/2015 5 2 Travel to Bangladesh by expat team member 2/20-2/22/2015 2 Team Planning Meeting hosted by BDGPE 2/22/2015 1 1 Meet with PRODIP/TAF staff 2/23/2015 .5 .5 Final work plan due to USAID (OOB Dhaka time) 2/24/2015 - In-brief with USAID/Bangladesh 2/24/2015 .5 .5 PRODIP Final Evaluation 41 Data collection 2/23-3/6/2015 10 10 Analysis and product drafting in-country Evaluation Team submits annotated report outline and draft presentation for USAID/Bangladesh DG Team review; data collection continues after submission 3/4/2015 - USAID provides comments (as needed) on report outline and draft presentation 3/5/2015 - Presentation and debrief with DG Team and USAID/Bangladesh 3/8/2015 .5 .5 Debrief meetings with key stakeholders, including GOB if necessary 3/8/2015 .5 .5 Expat Team members depart Bangladesh 3/9-3/10 2 Produce draft report 3/11-3/31 10 5 SI delivers draft report 3/31 USAID and partners review draft and provide comments 4/1-4/14/2015 - Team revises draft report and submits to BDGPE 4/15-4/28 3 1 SI delivers final report 4/28 TOTAL 35 21 X. REPORTING REQUIREMENTS The total report, excluding references and annexes, should not be more than 30 pages. The following content (and suggested length) should be included in the report: Table of Contents Acronyms Executive Summary – concisely state the project purpose and background, key evaluation questions, methods, most salient findings and recommendations (2-3 pp.); 1. Introduction – country context, including a summary of any relevant history, demography, socio-economic status, etc. (1 pp.); 2. The Development Problem and USAID’s Response – brief overview of the development problem and USAID’s strategic response, including design and implementation of the PRODIP program and any previous USAID activities implemented in response to the problem, (2-3 pp.); 3. Purpose of the Evaluation – purpose, audience, and synopsis of task (1 pp.); 4. Evaluation Methodology – describe evaluation methods, including strengths, constraints, and gaps (1 pp.); 5. Findings and Conclusions – describe and analyze findings for each evaluation question using graphs, figures, and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcomes. Conclusions should be credible and should be supported by the findings (12-15 pp.); 6. Recommendations – prioritized for each evaluation question; should be separate from conclusions and be supported by clearly defined set of findings and conclusions. Include PRODIP Final Evaluation 42 recommendations for future project implementation or relevant program designs and synergies with other USAID projects and other donor interventions as appropriate (3-4 pp). 7. Lessons Learned – provide a brief of key technical and/or administrative lessons on what has worked, not worked, and why for future project or relevant program designs (2-3 pp.); 8. Annexes – to include SOW, documents reviewed, bibliographical documentation, evaluation methods, data generated from the evaluation, tools used, interview lists, meetings, focus group discussions, surveys, and tables. The Evaluation Design Matrix must be presented as an annex to the report. Annexes should be succinct, pertinent, and readable. Should also include if necessary, a statement of differences regarding significant unresolved difference of opinion by funders, implementers, or members of the evaluation team on any of the findings or recommendations. The report format should be restricted to Microsoft products and 11-point type font should be used throughout the body of the report, with page margins one inch top/bottom and left/right. Annex A Documents for review will include, but are not limited to the following: 1. Program Description of PRODIP project. 2. USAID Modifications of Assistance 3. RIG Report on PRODIP 4. DfID Annual Review Reports on PRODIP 5. Mid-Term Evaluation of PRODIP 6. Performance Management Plan 7. Performance Reports 8. Bangladesh Democracy and Governance Assessment, Social Impact, 2014 9. PRODIP’s Survey reports 10. Relevant GOB documents (e.g. relevant acts and policies) 11. USAID/Bangladesh Strategy, Annual Reports, Operational Plan, annual Performance Monitoring Plan, DQA reports 12. Relevant USAID Forward documents 13. Other documents relevant to Legislative Strengthening in Bangladesh PRODIP Final Evaluation 43 ANNEX II: DOCUMENTS REVIEWED PRODIP Project Documents: USAID/Bangladesh. FINAL REPORT FOR THE LEGISLATIVE COMPONENT PROMOTING DEMOCRATIC INSTITUTIONS AND PRACTICES IN BANGLADESH (PRODIP). By Susan Crowdy. Dhaka: PRODIP, 2014. Print. USAID/Bangladesh. MODIFICATION OF ASSISTANCE, CA NO.388-A-00-10-00092-00 MODIFICATION NO. 14 PAGE 2 OF 26. N.p.: n.p., n.d. Print. USAID/Bangladesh. PROGRAM DESCRIPTION, PROMOTING DEMOCRATIC INSTITUTIONS AND PRACTICES (PRODIP) 388-A-00-10-00092-00. N.p.: n.p., n.d. Print. PRODIP Contracts: USAID/Bangladesh. REPORT FROM JULY 2012 CONSULTATION—NEXT STEPS. By George Crawford. Dhaka: PRODIP, 2012. Print. USAID/Bangladesh. STRENGTHENING THE RESEARCH & LIBRARY SERVICES FOR THE NATIONAL PARLIAMENT Jatiya Sangsad OF BANGLADESH. By DON SCHNEIDER. Dhaka: PRODIP, 2012. Print. PRODIP Work Plan: USAID/Bangladesh. YEAR 4 ANNUAL WORK PLAN. Dhaka: PRODIP, 2013. Print. USAID/Bangladesh. YEAR 5 ANNUAL WORK PLAN. Dhaka: PRODIP, 2014. Print. PRODIP Quarterly Reports: USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER ONE OF YEAR FOUR. N.p.: PRODIP, 2013. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER TWO OF YEAR FOUR. Dhaka: PRODIP, 2013. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER THREE OF YEAR FOUR. Dhaka: PRODIP, 2013. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER FOUR OF YEAR FOUR. Dhaka: PRODIP, 2013. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER ONE OF YEAR FIVE. Dhaka: PRODIP, 2014. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER TWO OF YEAR FIVE. Dhaka: PRODIP, 2014. Print. USAID/Bangladesh. PERFORMANCE MONITORING REPORT, QUARTER THREE OF YEAR FIVE. Dhaka: PRODIP, 2014. Print. PRODIP Final Evaluation 44 PRODIP Advocacy and Research: Anam, Mujibul, Sayema Khatun, Mohammad Nasir Uddin, and Siddiqur Rahman. AN ANTHROPOLOGICAL INVESTIGATION OF CITIZEN AND CIVIL SOCIETY ORGANIZATION PARTICIPATION IN THE FORMULATION AND IMPLEMENTATION OF EDUCATION POLICY IN BANGLADESH. Rep. Dhaka: Department of Anthropology Jahangirnagar U, n.d. Print. Barkat, Abul. WOMEN IN PARLIAMENT PROCESS AND EXTENT OF PARTICIPATION, AND SCOPE FOR STRENGTHENING THEIR ROLE. Rep. Dhaka: HDRC, 2012. Print. Dasgupta, Swapan Kumar and Shan-e-Jahan Anwar Zahid. GOVERNANCE OF PUBLIC SERVICE DELIVERY IN BANGLADESH: ROLE OF MEMBERS OF PARLIAMENT. Rep. Kotbari, Comilla: Bangladesh Academy for Rural Development (BARD), 2013. Print. DEVELOPMENT OF COASTAL AGRICULTURE AND LIVESTOCK: POLICY ADVOCACY WITH HONORABLE MEMBERS OF PARLIAMENT. Rep. Dhaka: Bangladesh Centre for Advanced Studies, 2012. Print. Hassanullah, Mohammad, and Noor Md. Rahmatullah. EFFECTIVENESS OF FOOD GRAIN PROCUREMENT SYSTEM OF BANGLADESH: TRANSPARENCY AND ACCOUNTABILITY OF ALLIED FIELD INSTITUTIONS. Rep. Dhaka: Agricultural Advisory Society (AAS) In Collaboration with Department of Agricultural Statistics Sher-e-Bangla Agricultural U (SAU), 2012. Print. Hussain, Akhter and Mobasser Monem. SOCIAL SAFETY NET PROGRAMS IN BANGLADESH: DYNAMICS AND AGENDA FOR POLICY REFORM. Rep. Dhaka: Department of Public Administration U of Dhaka, 2012. Print. Mannan, M. A., and Rumana Huque. PUBLIC SERVICE DELIVERY SYSTEMS IN BANGLADESH: GOVERNANCE ISSUES IN THE HEALTH SECTOR. Rep. Dhaka: Bangladesh Institute of Development Studies (BIDS), 2013. Print. O’Connor, Niamh "LAUNCH ‘BUDGET INFORMATION HELP DESK’ INSIDE THE PARLIAMENT BUILDING DURING BUDGET SESSION 2013-14." Letter to Md. Mahfuzur Rahman, Secretary, Bangladesh Parliament Secretariat. 20 May 2013. MS. Bangladesh, Dhaka, Bangladesh. USAID Documents: USAID/Bangladesh. AUDIT OF USAID/BANGLADESH’S PROMOTING DEMOCRATIC INSTITUTIONS AND PRACTICES PROGRAM. MANILA, PHILIPPINES: OFFICE OF INSPECTOR GENERAL, USAID, 2013. Print. USAID/Bangladesh. BANGLADESH COUNTRY DEVELOPMENT COOPERATION STRATEGY FY 2011 – FY 2016. N.p.: n.p., 2011. Print. USAID/Bangladesh. BANGLADESH DEMOCRACY AND GOVERNANCE ASSESSMENT. 2009. USAID/Bangladesh. MID-TERM PERFORMANCE EVALUATION OF DEMOCRATIC INSTITUTIONS AND PRACTICES (PRODIP) PROJECT. Dhaka: Social Impact, 2013. Print. Secondary Research and Documents: Rahman, Taiabur. PARLIAMENTARY CONTROL AND GOVERNMENT ACCOUNTABILITY IN SOUTH ASIA: A COMPARATIVE ANALYSIS OF BANGLADESH, INDIA AND SRI LANKA. London: Routledge, 2008. Print. PRODIP Final Evaluation 45 ANNEX III: PERSONS INTERVIEWED Telephone Key Informant Interviews outside Bangladesh Name Sex Position Affiliation Gordon Hein Male Vice President for Programs TAF Jeff Vanness Male Democracy and Governance Advisor USAID Niamh O’Connor Female Former PRODIP Deputy Chief of Party SUNY/CID Susan Crowdy Female Ex-PRODIP Program Director SUNY/CID Russell Pepe Male Former PRODIP Chief of Party TAF Group Key Informant Interviews in Dhaka, Bangladesh Name Sex Position Affiliation Anonymous 3 Male 1 Female Journalists (4) Media Anonymous 3 Male 1 Female Academicians (4) Dhaka University Key Informant Interviews in Dhaka, Bangladesh Name Sex Position Affiliation Hasan Mazumdar Male Country Representative TAF Sara L. Taylor Female Deputy Country Representative TAF Manash Mitra Male Parliamentary Program Director PRODIP Kazi Maruful Islam Male Former Director Research and Monitoring and Evaluation PRODIP Nurul Islam Male Program Manager PRODIP Sumana Sultana Mahmud Female Former Senior Program Manager PRODIP PRODIP Final Evaluation 46 Key Informant Interviews in Dhaka, Bangladesh Name Sex Position Affiliation Md. Fazle Rabbi Miah Male Deputy Speaker Bangladesh Parliament Md. Israfil Alam Male Member of Parliament Bangladesh Parliament Md. Habibe Millat Male Member of Parliament Bangladesh Parliament Mir Showkat Ali Badsha Male Member of Parliament, Chair￾Standing Committee on Ministry of Fisheries and Livestock Bangladesh Parliament Md Mozammel Hossain Male Member of Parliament, Chair￾Standing Committee on Ministry of Social Welfare Bangladesh Parliament Advocate Fazilatun Nasa Bappy Female Member of Parliament (Women seat) Bangladesh Parliament Professor Md. Ali Ashraf Male Member of Parliament (former Deputy Speaker) Bangladesh Parliament Rasheda Begum Hira Female Former Member of Parliament (9th Parliament) Bangladesh Parliament Nilufar Chowdhury Moni Female Former Member of Parliament (9th Parliament) Bangladesh Parliament Md. Ashraful Moqbul Male Senior Secretary Bangladesh Parliament Secretariat Md. Abu Al Helal Male Deputy Secretary, Inter Parliamentary Affairs Bangladesh Parliament Secretariat Md. Munim Hassan Male Deputy Secretary (Committee Support) Bangladesh Parliament Secretariat Md. Kamrul Islam Male Director, Debate Editing and Publication Bangladesh Parliament Secretariat Mr. Md. Ohiduzzaman Male Senior Assistant Secretary Bangladesh Parliament Secretariat Md. Nazmul Hoque Male Deputy Director, Training Bangladesh Parliament Secretariat Md. Faisal Morshed Male Senior Committee Officer, Committee Support Bangladesh Parliament Secretariat Maleka Parvin Female Deputy Director (Technical Support) Bangladesh Parliament Secretariat PRODIP Final Evaluation 47 Key Informant Interviews in Dhaka, Bangladesh Name Sex Position Affiliation Abu Hena Mostafa Zaman Male Former Personal Secretary to Chairman of Standing Committee on Labor (9th Parliament) Bangladesh Civil Service M. M. Fozlur Rahman Male Legislative Draftsman, Legislative Support Bangladesh Parliament Secretariat Naimul Azam Khan Male Deputy Director (Research) Bangladesh Parliament Secretariat Zeb-un-Nessa Female Librarian, Finance and Public Relation Bangladesh Parliament Secretariat Farhana Begum Female Committee Officer, Committee Support Bangladesh Parliament Secretariat Wahiduzzaman Male Personal Secretary to Honorable Chairman, Social Welfare Bangladesh Parliament Secretariat Rokonuddin Bhuyan Male Personal Secretary to Chairman, Fisheries Committee Bangladesh Parliament Secretariat Md. Ali Akbar Male Research and Education Officer Bangladesh Parliament Secretariat Mahfuz Kabir Male Senior Researcher Unnayan Shamunnay Helen Barnes Female Governance Advisor DFID Pushpinder Dhillon Male Counselor for Political and Economic Affairs U.S. Embassy, Dhaka Alistair Legge Male Chief of Party International Foundation for Electoral Systems Sikjia Passilinna Female Deputy Chief of Party International Foundation for Electoral Systems Noel Boxer Male Democratic Participation and Reform Democracy International Rishi Datta Male Chief of Party National Democratic Institute Saiful Karim Male Senior Advisor National Democratic Institute Billy Woodward Male Authorized Organization Representative, PRODIP USAID Rezwana Farha Female Political Process Advisor USAID PRODIP Final Evaluation 48 ANNEX IV: EVIDENCE MATRIX Evaluation Question Data Collection Methodology Questions for Data Collection Instrument Respondent Group 1. To what extent has PRODIP been successful in achieving the program objectives as set out in the Agreement, Modifications to Assistance, Year 4 and Year 5 Work Plans, and other program documents? 1.Key informant interviews 2.Review of program documents 1. Did you/your staff improve your research, report writing, or drafting skills thanks to PRODIP? 2. Has the policy research available to parliament improved thanks to PRODIP? 3. Did the orientation program for the 10th Parliament help you understand the role of an MP? 4. Have parliamentary lawmaking skills and oversight capacity improved thanks to PRODIP? 1.Beneficiaries 2.Outside parliamentary observers 3. PRODIP staff 2. What has been the impact of PRODIP on the functioning of the Parliament’s committee system? What have been the major obstacles to this activity? How has PRODIP sought to overcome them? 1.Key informant interviews 2.Review of program documents 3.Telephone Mini-Survey 1. Did your committee develop a work plan? 2. Did your committee improve its work processes thanks to PRODIP? 3. Did your committee improve oversight thanks to PRODIP? 4. Did your committee produce an activity brief? 5. Did PRODIP assist your committee to hold public hearings? 6. Has the functioning of parliament’s committee system improved? 1.Beneficiaries 2.Outside parliamentary observers 3. PRODIP staff 4. U.S. Embassy staff 3. What have been the successes and challenges in those activities of PRODIP in support of the Budget Analysis and Monitoring Unit (BAMU) of the Secretariat? 1.Key informant interviews 2.Review of program documents 3. Telephone Mini-Survey 1. Did BAMU improve the quality of its financial analysis services thanks to PRODIP? 2. Has BAMU assisted you/your committee with budgetary analysis, and was this assistance helpful? 3. Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis? 1.Beneficiaries 2.Outside parliamentary observers 3. PRODIP staff 4. What have been the changes made in PRODIP’s training strategy for staff of the 1.Key informant interviews 2. Review of program 1. How has PRODIP changed its training strategy in response to recommendations of previous evaluations? 1.PRODIP staff 2.Beneficiaries PRODIP Final Evaluation 49 Evaluation Question Data Collection Methodology Questions for Data Collection Instrument Respondent Group Parliamentary Secretariat in response to the recommendations of previous evaluations? How effective have these changes been in developing the capacity of the Secretariat? documents 2. Have changes in PRODIP’s training strategy helped to develop the capacity of the Secretariat? 5. Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? 1.Key informant interviews 2. Review of program documents 1. Has PRODIP’s performance management system improved in response to recommendations from previous evaluations? How? 1. PRODIP staff 6. To what extent have PRODIP management and stakeholders incorporated knowledge gained through M&E into project management? 1.Key informant interviews 2. Review of program documents 1. To what extent have PRODIP management and stakeholders incorporated knowledge gained through M&E into project management? 1. PRODIP staff 7. How sustainable are the results of PRODIP’s support to the Parliamentary Secretariat beyond USAID support? What measures has PRODIP taken in Year 4 and Year 5 to ensure sustainability? 1.Key informant interviews 2. Review of program documents 1. Has parliament funded research and documentation services? 2. Has BAMU been incorporated into the Secretariat, with adequate funding for its activities? 3. Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? 4. Do other donors plan follow￾on work in the areas in which PRODIP has provided support? 1. PRODIP staff 2. Beneficiaries 3. Other donors PRODIP Final Evaluation 50 ANNEX V: DATA COLLECTION PROTOCOLS Respondent Group: USG and other donors  O verall, what has been your experience with PRO DIP?  W hat has been the impact of PRODIP in the parliament?  Have members and staff been receptive to the current programs?  W hat is the current political climate in Bangladesh?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Has the policy research available to parliament improved thanks to PRO DIP?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRODIP?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  How has PRO DIP changed its training strategy in response to recommendations of previous evaluations?  Have changes in PRO DIP’s training strategy helped to develop the capacity of the Secretariat?  Has PRO DIP’s performance management system improved in response to recommendations from previous evaluations? How?  To what extent have PRO DIP management and stakeholders incorporated knowledge gained through M&E into project management?  Has parliament funded research and documentation services?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? PRODIP Final Evaluation 51 Respondent Group: PRODIP and TAF staff  W hat has been the impact of PRO DIP in the parliament?  Have members and staff been receptive to the current programs?  W hat is the current political climate in Bangladesh?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Has the policy research available to parliament improved thanks to PRODIP?  Did the orientation program for the 10th Parliament help MPs understand their role?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRO DIP?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  How has PRO DIP changed its training strategy in response to recommendations of previous evaluations?  Have changes in PRO DIP’s training strategy helped to develop the Secretariat’s capacity?  Has PRO DIP’s performance management system improved in response to recommendations from previous evaluations? How?  To what extent have PRODIP management and stakeholders incorporated knowledge gained through M&E into project management?  Has parliament funded research and documentation services?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? Other donors and implementers  O verall, what has been your experience with parliament?  W hat has been the impact of PRO DIP in the parliament?  Have members and staff been receptive to your legislative strengthening programs? PRODIP Final Evaluation 52  W hat is the current political climate in Bangladesh?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Has the policy research available to parliament improved thanks to PRODIP?  Did the orientation program for the 10th Parliament help MPs understand their role?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRO DIP?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  Has parliament funded research and documentation services?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes?  Do other donors plan follow-on work in the areas in which PRODIP has provided support? Respondent Group: MPs and former MPs  O verall, what has been your experience with the PRO DIP program?  W hat has been the impact of PRO DIP in the parliament?  Has PRO DIP helped parliament improve:  Its ability to draft legislation?  Committees’ ability to analyze and modify bills?  Parliamentary oversight of the government?  The internal functioning of the Secretariat?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability? PRODIP Final Evaluation 53  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Have members and staff been receptive to the current programs?  Did you/your staff improve your research, report writing, or drafting skills thanks to PRODIP?  Has the policy research available to parliament improved thanks to PRO DIP?  Did the orientation program for the 10th Parliament help you understand the role of an MP?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRODIP?  Did your committee develop a work plan?  Did your committee improve its work processes thanks to PRO DIP?  Did your committee improve oversight thanks to PRO DIP?  Did your committee produce an activity brief?  Did PRO DIP assist your committee to hold public hearings?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has BAMU assisted you/your committee with budgetary analysis? Was this assistance helpful?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  Have changes in PRO DIP’s training strategy since April 2013 helped to develop the capacity of the Secretariat?  Has parliament funded research and documentation services?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? Respondent Group: Parliament Secretariat Staff  O verall, what has been your experience with PRO DIP?  Has PRO DIP helped parliament improve:  Its ability to draft legislation?  Committees’ ability to analyze and modify bills? PRODIP Final Evaluation 54  Parliamentary oversight of the government?  The internal functioning of the Secretariat?  Does BAMU help parliament to analyze, modify and oversee the budget?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Have staff from your office attended PRO DIP training? If so, what was their experience?  Have members and staff been receptive to the current programs?  Did you/your staff improve your research, report writing, or drafting skills thanks to PRODIP?  Has the policy research available to parliament improved thanks to PRO DIP?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRO DIP?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  Have changes in PRO DIP’s training strategy since April 2013 helped to develop the capacity of the Secretariat?  Has parliament funded research and documentation services?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities? Respondent Group: Parliament Secretariat staff: Standing Committee Staff  Briefly describe the work of Standing Committee staff.  Does your committee draft its own legislation? W ho does the actual drafting?  How does your committee analyze and modify legislation?  W hat tools does your committee use to conduct oversight?  How frequently do you hold public hearings?  O verall, what has been your experience with PRO DIP?  Has PRO DIP helped your committee improve: PRODIP Final Evaluation 55  Its ability to draft legislation?  Its ability to analyze and modify bills?  Oversight of the ministry?  Does BAMU help your committee to analyze, modify and oversee the budget of the ministry over which you have jurisdiction?  Does your committee have adequate access to independent research and analysis?  Does your committee have adequate budget analysis capability?  Does your committee adequately perform oversight?  Does your committee receive adequate support in drafting legislation?  W hat has been the impact of PRO DIP in the parliament?  Have members and staff been receptive to the current programs?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Did you/your staff improve your research, report writing, or drafting skills thanks to PRO DIP?  Has the policy research available to parliament improved thanks to PRO DIP?  Did your committee develop a work plan?  Did your committee improve its work processes thanks to PRO DIP?  Did your committee improve oversight thanks to PRO DIP?  Did your committee produce an activity brief?  Did PRO DIP assist your committee to hold public hearings?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRO DIP?  Has BAMU assisted you/your committee with budgetary analysis? W as this assistance helpful?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis? PRODIP Final Evaluation 56  Have changes in PRO DIP’s training strategy since April 2013 helped to develop the capacity of the Secretariat?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? Respondent Group: Parliament Secretariat Staff: Budget Analysis and Monitoring Unit (BAMU) Staff  Please describe how BAMU analyzes, modifies and oversees implementation of the national budget.  O verall, what has been your experience with PRO DIP?  Did PRO DIP help BAMU to analyze, modify and oversee the budget?  Have members and staff been receptive to the current programs?  Does parliament have adequate budget analysis capability?  Did BAMU improve the quality of its financial analysis services thanks to PRODIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  Have changes in PRO DIP’s training strategy since April 2013 helped to develop the capacity of BAMU?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities? Respondent Group: Outside Experts/Other Program Implementers  W hat is the current political climate in Bangladesh?  How do Bangladeshi citizens view the parliament?  How does parliament engage civil society in the legislation and oversight processes?  How does parliament engage civil society in the budget process?  In what areas does parliament need to improve its performance?  In the past two years, has parliament improved:  Its ability to draft legislation?  Committees’ ability to analyze and modify bills?  Parliamentary oversight of the government? PRODIP Final Evaluation 57  The internal functioning of the Secretariat?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  W hat has been the impact of PRO DIP in the parliament?  Does the parliament have adequate access to independent research and analysis?  Does parliament have adequate budget analysis capability?  Does parliament adequately perform oversight?  Do Members receive adequate support in drafting legislation?  Has the policy research available to parliament improved thanks to PRO DIP?  Have parliamentary lawmaking skills and oversight capacity improved thanks to PRO DIP?  Has the functioning of parliament’s committee system improved?  Did BAMU improve the quality of its financial analysis services thanks to PRODIP?  Has parliament conducted more effective budget scrutiny thanks to BAMU’s analysis?  Has parliament funded research and documentation services?  Has BAMU been incorporated into the Secretariat, with adequate funding for its activities?  Have committees integrated into their operations public hearings, work plans, activity briefs and other improvements in oversight and work processes? PRODIP Final Evaluation 58 ANNEX VI: TELEPHONE MINI-SURVEY PROTOCOL Social Impact is conducting a mini-survey of parliamentary staff who participated in a training /seminar organized by the PRODIP project between April 2013 and today. PRODIP is a parliamentary support project funded by USAID and DFID and implemented by The Asia Foundation. Your responses are confidential, and your name will not be used in the evaluation report. Name of respondent Position Number of years working in Parliament Date of filling the questionnaire 1. When and for how many days did you participate in the PRODIP training? (If you participated in multiple training activities, please answer based on the most recent activity. Month: |___|___| Year: |___|___|___|___| Number of days: |___|___| 2. On a scale of 1 to 5, where “1” means “not useful at all” and “5” means “very useful,” please click the button that corresponds with how useful the PRODIP training was in helping you acquire information relevant to the performance of your responsibilities? Not useful at all Not useful Average Useful Very useful Don’t know 1 2 3 4 5 99 3. Did the training provide you a particular tool/skill/piece of information you did not have before? 1 Yes 2 No 99 Don’t know If yes, in a few words, please explain what the tool/skill/piece of information was? 4. What was the level of the training relative to your experience with the material? 1 Too basic for me 2 Appropriate for my experience 3 Too advanced for me 99 Don’t know 5. Have you used the tool/skill/piece of information in performing your responsibilities? 1 Yes 2 No 99 Don’t know If yes, please explain briefly PRODIP Final Evaluation 59 6. Do you work for the parliamentary committee service or directly for the chairman of a committee? (If no, please skip to question 13.) 1 Yes 2 No 99 Don’t know 7. Has PRODIP improved the functioning of parliament’s committee system? 1 Yes 2 No 99 Don’t know 8. Has PRODIP helped your committee to develop a committee work plan? 1 Yes 2 No 99 Don’t know If yes, please explain briefly how PRODIP helped your committee develop a work plan: 9. Has PRODIP helped your committee to improve its work processes? 1 Yes 2 No 99 Don’t know If yes, please explain briefly how PRODIP helped your committee improve its work processes: 10. Has PRODIP helped your committee to improve its oversight functions? 1 Yes 2 No 99 Don’t know If yes, please explain briefly how PRODIP helped your committee improve its oversight functions: PRODIP Final Evaluation 60 11. Has PRODIP helped your committee to produce Committee Activity Briefs? 1 Yes 2 No 99 Don’t know If yes, please explain briefly how PRODIP helped your committee to produce activity briefs: 12. Has PRODIP supported your committee in organizing and conducting public hearings? 1 Yes 2 No 99 Don’t know If yes, please explain briefly how PRODIP helped your committee organize and conduct public hearings: 13. Do you work for the Budget Analysis and Monitoring Unit? If yes, please answer question 13. (If no, you are finished.) 1 Yes 2 No 99 Don’t know 14. Has PRODIP improved the functioning of the Budget Analysis and Monitoring Unit? 1 Yes 2 No 99 Don’t know Please explain briefly ====OO==== PRODIP Final Evaluation 61 Telephone Mini-Survey Protocol (Bangla Version) PRODIP Final Evaluation 62 PRODIP Final Evaluation 63 PRODIP Final Evaluation 64 PRODIP Final Evaluation 65 ANNEX VII: DISCLOSURE OF CONFLICTS OF INTEREST PRODIP Final Evaluation 66 PRODIP Final Evaluation 67 PRODIP Final Evaluation 68 PRODIP Final Evaluation 69 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523