CEPPS III - Support to the Political Process in Lebanon Evaluation Report December 30, 2011 CEPPS III - Support to the Political Process in Lebanon Page 2 Evaluation Report Contents Acronyms .................................................................................................................................... 3 1. Introduction ............................................................................................................................ 4 2. Evaluation Methodology ......................................................................................................... 4 2.1 Evaluation Limitations ...................................................................................................... 5 3. Electoral Context .................................................................................................................... 6 4. Program Background .............................................................................................................. 7 5. Evaluation by objective: Enhance capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration ........ 8 5. 1 Activities and outputs under this objective ...................................................................... 8 5. 2 Performance evaluation and findings............................................................................. 10 6. Evaluation by objective: Facilitate opportunities for the effective training of electoral officials and stakeholders .......................................................................................................... 12 6. 1 Activities and outputs under this objective .................................................................... 12 6. 2 Performance evaluation and findings............................................................................. 13 7. Evaluation by objective: Enhance the capacity of the election administration and reform of the electoral process. ................................................................................................................. 15 7.1 Activities and outputs under this objective ..................................................................... 15 7.2 Performance evaluation and findings .............................................................................. 16 8. M&E/Program Procedures .................................................................................................... 16 9. Lessons Learned/Recommendations ..................................................................................... 18 10. Conclusion .......................................................................................................................... 21 Annex 1: List of People Met ..................................................................................................... 22 Appendix 2: Timeline ............................................................................................................... 23 Appendix 3: Scope of Work ..................................................................................................... 24 CEPPS III - Support to the Political Process in Lebanon Page 3 Evaluation Report Acronyms BRIDGE Building Resources in Democracy, Governance and Elections CCER Civil Campaign for Election Reform CEPPS Consortium for Electoral and Political Party Strengthening DG Director General DGPAR Director General for Political Affairs and Refugees DGPS Director General for Personal Status GEO Global Election Organization IEC Independent Electoral Commission IFES International Foundation for Electoral Systems LADE Lebanese Association for Democratic Elections NDI National Democratic Institute M&E Monitoring and Evaluation MP Member of Parliament MoIM Ministry of Interior and Municipalities PIRS Performance Indicator Reference Sheets PMP Performance Management Plan SCEC Supervisory Commission on Electoral Campaigns UNDP United Nations Development Program USAID U.S. Agency for International Development CEPPS III - Support to the Political Process in Lebanon Page 4 Evaluation Report 1. Introduction This evaluation of the CEPPS III - Support to the Political Process in Lebanon documents the results achieved by the program, captures some of the learning, and provides suggestions for improvement of future activities. This could be used for subsequent democracy and governance programs in Lebanon, as well as for capacity-building programs of other government entities. It is based on an analysis of how the program worked towards its objectives and how these could have been improved. The program was launched during a critical period in the development of Lebanon’s electoral system. The two elections that occurred following the introduction of a 2008 law were better than expected but fell short of international standards. CEPPS III was launched with the aim of addressing some of these shortcomings, many of which were highly contentious and politically charged, before the next 2013 Parliamentary election. Throughout the program’s implementation, the government partner was subject to the oscillations of Lebanese politics. The government coalition fell within the weeks that the program began, leaving the program with a ministry partner that lacked a clear mandate. The program continued to provide support amid this uncertainty. Despite room for improvement in the program’s strategic design, the program made a significant contribution to the electoral process in Lebanon. Among other accomplishments, it supported its government partner on key reforms and improved the knowledge of government officials on key electoral issues. 2. Evaluation Methodology IFES contracted a consultant to conduct an end-of-project evaluation of its activities in Lebanon. The purpose of the evaluation is to identify lessons learned for future IFES work in Lebanon and globally and to assess IFES’ work from the inception of the CEPPS III agreement in January 2011, through the program’s modification, to the closure of the program at end of the 2011 calendar year. The report loosely follows a prescribed organization, detailing the project’s activities under each of the three programmatic objectives. In keeping with USAID’s new evaluation guidance, this report is a performance evaluation that focuses on normative questions of what the particular project or program has achieved. The report avoids conclusive findings of program impact, which are only possible with a rigorously defined counterfactual that does not CEPPS III - Support to the Political Process in Lebanon Page 5 Evaluation Report exist on this program.1 The report’s discussion of each objective is loosely structured around the OECD-DAC criteria of relevance, efficiency, effectiveness, impact, and M&E.2 These criteria were the basis for many questions posed to respondents. The evaluation was conducted over a ten-day period by Ethan Arnheim, who was contracted by IFES for the evaluation. Five days were spent in Beirut, from December 26 to December 30, 2011 and a draft of the report was required by the end of the calendar year. A centerpiece of the evaluation was an extensive desk review of all project background documents including quarterly reports, training materials, workshop reviews, and other materials produced by IFES. Field research was conducted through in-person, semi-structured interviews with key participants in IFES’ Lebanon office and other Beirut-based officials and trainees. Broadly, the evaluation seeks to apply a theory-based approach. A theory-based approach carefully examines the logic model, or set of assumptions for how the program intends to achieve higher level results. Central to the theory-based approach is the program’s results framework, which was reviewed in detail and forms a substantial portion of the evaluation. The semi-structured interviews seek to identify what evidence there is to support each piece of the logic model. The value of a theory-based approach is that it is useful for program learning. By identifying which steps in a logic model can be improved and which ones are supported by evidence, subsequent programming can be grounded in sound design and assumptions that have already been tested. 2.1 Evaluation Limitations A primary challenge facing this evaluation was the limited resources available for the evaluation, both in terms of time and staffing. One might say that this brings new meaning to the concept of a rapid appraisal. Ideally, an evaluation would be conducted by a team of no fewer than two evaluators, who could divide the responsibilities of reviewing the literature, identifying stakeholders for interview, preparing materials, and researching the context. Then this preparation would be followed by adequate time to speak with the panoply of interviewees. Because an extension was not granted and the program closed at the end of the calendar year, the evaluation was scheduled quickly, and the ideal amount of time for preparation was not available. The timing was made more difficult by the absence of some key interviewees due to the Christmas and New Year’s holidays. 1 USAID. 2011. USAID Evaluation Policy: Learning From Experience. Online at http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf 2 OECD-DAC. 2010. Evaluating Development Co-operation: Summary of Key Norms and Standards. Online at http://www.oecd.org/dataoecd/12/56/41612905.pdf CEPPS III - Support to the Political Process in Lebanon Page 6 Evaluation Report Sound evaluation practice seeks to limit the potential sources of evaluator biases. Ideally, the entity conducting the evaluation is not the same organization that has implemented the program under evaluation. While the evaluator had no institutional affiliation, he was contracted through IFES. At the same time, this did not substantively impact the findings, as there was no direction about the substance and content of the evaluation findings, nor limitations on meetings. It is important to keep in mind that the real test of an evaluation is whether another evaluator, reviewing the same data and conducting the same interviews, would reach similar conclusions. This report maintains that this is the case. 3. Electoral Context Lebanon has one of the most complex, politically charged electoral landscapes in the world. An electoral law passed in September 2008 sought to balance sectarian interests. The result of an earlier compromise that defused heated tensions between rival groups, this law provided the basis for the 2009 parliamentary elections and 2010 local elections. Although the two most recent elections were considered an improvement over previous votes, significant electoral issues remained unaddressed. Changes to proportional representation, the use of pre-printed ballots, and creation of an Independent Electoral Commission (IEC) were identified as requirements for reforming the electoral process for the next parliamentary vote, (required by June 2013) in order to meet international standards. Under the 2008 law, the Ministry of the Interior and Municipalities (MoIM) is the primary body responsible for administering elections in Lebanon. The Supervisory Commission on Electoral Campaigns (SCEC) is a semi-independent regulatory body that falls under the authority of the MoIM and is charged with oversight of elections. In 2011, Lebanon’s history of political volatility continued. In mid-January, the resignation of 11 ministers led to the collapse of the cabinet headed by Prime Minister Saad Hariri. The MoIM, headed by Ziyad Baroud, did not have the same prerogatives when operating as a caretaker ministry. Despite his status, Ziyad Baroud was considered a remarkable appointment for a country that does not typically assign Cabinet officials based on expertise. A member of the Boutros Commission that previously played a formative role in the creation of the CEPPS III - Support to the Political Process in Lebanon Page 7 Evaluation Report 2008 election law, Baroud was previously a leading civil society activist and head of the Lebanese Association for Democratic Elections (LADE). After a new government was formed in June headed by Prime Minister Najib Mikati, MoIM leadership changed from Baroud to Gen. Marwan Charbel. Before his departure, Minister Baroud presented a draft of a new electoral law. The makeup of the new Mikati government, which included cabinet members from Hezbollah, caused the international donor community to reevaluate its relationships and support. The ministerial change coincided with the retirement of the MoIM’s Director General of Political Affairs and Refugees, the Ministry position that manages elections, and his succession by Elias El-Khoury. 4. Program Background In August 2010, USAID/Lebanon awarded grant 268-A-00-10-00004-00 to NDI and IFES, members of the Consortium for Electoral Political Process Strengthening (CEPPS) consortium active in for Support to the Political Process in Lebanon. This round of funding was the third cycle, or CEPPS III, and aimed to build on progress achieved in earlier phases. The CEPPS partners designed coordinated strategies, with IFES primarily working with the MoIM. NDI’s activities focused on increasing the political participation of Lebanese women and strengthening civil society’s advocacy for electoral reform. The division of responsibilities aimed to build on each partner’s experience and background in working with the government and civil society. IFES’ activities, which received $1,128,148 in funding, centered around providing immediate support to the MOIM to implement reforms critical to the conduct of the June 2013 elections. The main areas of assistance identified in the program proposal were: • Support for MOIM-led electoral reform initiatives • Facilitating the establishment of an independent electoral commission (IEC) • Development of training capacity for electoral officials • Engagement in national, regional, and international training opportunities • A review of key electoral procedures IFES’ activities were grouped into three assistance objectives. These were: CEPPS III - Support to the Political Process in Lebanon Page 8 Evaluation Report • Enhance capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration • Facilitate opportunities for the effective training of electoral officials and stakeholders • Enhance the capacity of the election administration and reform of the electoral process On June 28, a modification was approved permitting the program to expand its partnerships to include civil society organizations. Program documents explain that these assistance objectives are intended to achieve the short term goal “to improve the capacity of the election administration in Lebanon through the development of its skills, knowledge, transparency and professionalism.” This short-term goal will give electoral administrators and stakeholders “the tools to undertake the long-term goal of strengthening the effective management of parliamentary and municipal elections in Lebanon to ensure they are credible and meet international standards.” Later project documents merge these two goals to state more broadly that “the objective of this program is to assist Lebanon in consolidating the role of democratic institutions and their responsiveness to the needs of citizens, by enabling the successful conduct of the 2013 parliamentary elections via an effective and credible election administration.” IFES identified four indicators for its work, two of which are from State Department’s Foreign Assistance indicators (F:) • Number of electoral administration procedures and systems strengthened with USG assistance • Number of reports, briefings and/or advisory notes issued to Lebanese authorities on electoral reform issues • Number of election officials and stakeholders trained with USG assistance • Number of model guidance documents on electoral procedures and draft regulations issued to Lebanese authorities 5. Evaluation by objective: Enhance capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration 5. 1 Activities and outputs under this objective CEPPS III - Support to the Political Process in Lebanon Page 9 Evaluation Report Most of IFES’ work in CEPPS III fell under the two activities in this objective, support for MoIM-led electoral reform initiatives and facilitating the establishment of an IEC. These activities were intended to support the development of an electoral law that was in accordance with international standards. Assistance under this objective primarily took the form of policy papers, tools, and advice that informed MoIM’s efforts to draft a new election law. The 13 outputs counted by IFES as part of the indicator, “Number of electoral administration procedures and systems strengthened with USG assistance,” were provided under this objective. Of these products, all but four related to the development of the new electoral law. The 11 outputs for the indicator, “Number of reports, briefings, and/or advisory notes issued to Lebanese authorities and stakeholders on electoral reform issues,” were also under this objective. Deliverables related to the development, usage and results of a computerized electoral systems simulator are counted four times between these two indicators. The electoral systems simulator was contracted to a political science professor and a software developer consultant with expertise in Lebanese election procedures and seat allocation scenarios. This advanced software is operated through a simple interface that allows the user to adjust the key variables that may be affected by an election law. The program can simulate the consequences of adopting proportional representation using closed lists, open lists, different numbers of preferential votes for candidates, block votes, and the system proposed by the Boutros Commission. In addition to providing insight across the country, the software program allows for disaggregation on regional and district bases. The software also supports a quantitative comparison of “fairness measures” that is based on international standards and motivated by fair representation, participation of minorities, and political efficiency. In addition to the election simulation software, IFES provided other products to MoIM as MoIM considered election reform options. IFES provided technical advice comparing the current version with those proposed by MoIM and CCER and with analyses and examples of electoral reform laws undertaken in other countries. Redistricting maps complemented the computer simulations. Other support included regular meetings with MoIM officials, both at the senior level and in collaboration between MoIM, and with working level staff. IFES provided in these meetings a number of briefing papers, research analysis, and presentations. The second component of this objective sought to facilitate the creation of an IEC. To this aim, IFES provided a briefing on the legal framework for an IEC and briefing on the staffing tables for an IEC. CEPPS III - Support to the Political Process in Lebanon Page 10 Evaluation Report 5. 2 Performance evaluation and findings IFES support to enhance the capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration was highly relevant. By several accounts, MoIM simply did not have the resources to conduct the electoral analysis needed to model the subtleties of electoral reform. These reforms were critical to the development of electoral reform legislation that was required for the 2013 vote to meet international standards of fairness. IFES’ support provided valued input on many aspects of the new law. There is evidence that IFES worked efficiently. IFES staff developed a close relationship with MoIM leadership and could anticipate MoIM needs and provide appropriate assistance. This partnership continued after the change in Ministry and DG leadership. One interviewee explained that a consequence of MoIM’s limited resources is its difficulty in securing the right expertise. The development of the computer simulation and technical feedback on electoral law briefings, were quickly developed by anticipating MoIM’s needs. Other products, such as comparison between electoral laws, were IFES’ initiative. In other words, the support was both demand and supply driven. This balance was possible because of the close working relationship between MoIM and IFES staff, a synergy that existed through most of the program. The change in the MoIM’s DG and Minister required new relationship building. IFES was quick to support the technically sound proposals of the new leadership, providing informal guidance on briefings delivered by the DG about electoral reform. Efficiency could have been increased if there were greater donor and stakeholder coordination. While IFES’ electoral simulation was lauded by all respondents, NDI supported the creation of a similar electoral simulation through its civil society partner, CCER. IFES staff reported that the IFES version had enhanced capability, namely the ability manipulate a wide range of capability. Improved donor and stakeholder coordination may have avoided duplicate efforts. IFES’ work with civil society, which complemented its efforts at electoral reform, was impossible until a modification to its scope of work was approved. An analysis of the impact of this objective, to enhance the capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration, is more challenging. The results framework identified several intermediate results that would contribute to this higher-level outcome. Among the CEPPS III - Support to the Political Process in Lebanon Page 11 Evaluation Report anticipated outcomes are a more efficient and transparent election administration and improved administration of electoral events. As is discussed in greater detail in the section on M&E, IFES should not have expected this ambitious series of intermediate outcomes to follow from its technical support around the electoral law. A more reasonable outcome and a qualitative indicator of program effectiveness is IFES’ impact on the electoral reform process led by MoIM. A key milestone in the process was an introduction of a new electoral reform bill by MoIM. The simulator provided MoIM leadership the ability to model the implications of different versions of the law. For example, MoIM was considering how to create an open list but still ensure seats for women. The simulator showed the scenario of an open list with one seat designated to another party. This was used in the final language of the legislation. Although MoIM would have introduced proportional representation without IFES support, the simulator and other analysis helped MoIM understand the implications of a new system and consider all the different options. A June 16, 2011 article in El Nashra that described the new law mentioned the electoral simulator, saying that it “proves the feasibility” of proportional representation. The law proposed by MoIM under its new leadership did not create an IEC, the development of which IFES had planned to support. IFES assistance still served to influence the MoIM on this issue, however. The DG explained that he envisioned the proposed SCEC as a “SCEC plus” which will have many of the responsibilities of an IEC though located within MoIM. In the absence of other outcomes, it is not reasonable to evaluate the program based on whether the electoral law containing the reforms IFES encouraged was enacted. There are simply too many external factors influencing the passage of the electoral law to measure IFES’ effectiveness on whether there is a new electoral law. Acceptance of proportional representation among MPs will be difficult if the new voting arrangement puts their reelection in jeopardy. IFES staff also explain that involvement on contentious issues like proportional representation entailed more than just electoral reform but were political reforms as well and especially sensitive. IFES could not be seen to push any single proposal too directly lest it be seen as a US- or donor-led initiative. Major obstacles to the implementation of the electoral law remain. It is uncertain if MoIM has the capacity to manage other elements of electoral reform, such as pre￾printed ballots. If the new electoral law is passed, MoIM must manage redistricting and the move to a PR system should these be included. Although there is evidence that MoIM’s capacity has improved thanks to IFES support, there is simply not enough time CEPPS III - Support to the Political Process in Lebanon Page 12 Evaluation Report elapsed from the end of the program to estimate the degree to which MoIM is better prepared to administer the 2013 election. 6. Evaluation by objective: Facilitate opportunities for the effective training of electoral officials and stakeholders 6. 1 Activities and outputs under this objective Under this objective, IFES provided training in Lebanon using the Building Resources in Democracy, Governance and Elections (BRIDGE) curriculum and developed international trainings opportunities and study tours for senior MoIM staff and stakeholders. BRIDGE is an internationally recognized professional development program with a particular focus on electoral processes and is designed to be used as a tool within a broader capacity development framework. Under CEPPS III, IFES organized a total of four BRIDGE trainings tailored to provide a comprehensive understanding of the electoral systems and election management design in Lebanon. A total of 55 unique participants attended either one or both of the workshops. Thanks to previous BRIDGE training offered by IFES, seven of the participants became fully accredited BRIDGE workshop facilitators, including four MoIM staff. As a modular professional development program, the BRIDGE training was customized to address the needs of the participants. The technical workshops, which centered on two topics, election management design and electoral systems, covered the main electoral systems, seat-allocation formulas, quotas, and reserved seats, as well as on the electoral systems in Lebanon and current proposals for reform. According to IFES reporting, the main purpose of the Electoral Systems module was to discuss issues related to various electoral system options and their impact on potential electoral reform in Lebanon ahead of the 2013 parliamentary elections. The aim of the Electoral Management Design workshops was to introduce other models of electoral management bodies and discuss issues related to election management professionalism. The second element of IFES work under this objective was to provide international training opportunities. IFES encouraged the participation of MoIM staff in an exchange of best practices for elections through their participation in the annual Global Election Organization (GEO) conference that was held in Botswana in March 2011. The GEO conference, which is recognized as the only global gathering of election management CEPPS III - Support to the Political Process in Lebanon Page 13 Evaluation Report bodies, held its meeting under the title “Credible Elections for Democracy” and supported an exchange of best practices for elections. Preparations were made for Minister Baroud’s attendance, but they were canceled at the last minute due to political developments. In November 2011, IFES arranged a study tour to Scotland for DG Khoury. Other participants included a senior CCER official who also served on the board of LADE and a key civil society activist. This tour was planned because the Scottish experience with the design and development of ballot papers and counting procedures were particularly relevant to Lebanese electoral reform and efforts to develop pre-printed ballots. 6. 2 Performance evaluation and findings IFES wholly fulfilled its objective to facilitate opportunities for the effective training of electoral officials and stakeholders. In addition to arranging these opportunities, by creating a qualified cadre of trained professionals within MoIM, IFES enabled MoIM to facilitate additional opportunities for its staff in the future and a possible degree of sustainability. The BRIDGE trainings were highly relevant. Since Lebanon has one of the most complex electoral systems in the world, senior MoIM staff have devoted most of their attention to understanding the intricacies of this system. Yet meaningful election reform requires an appreciation of what can be borrowed from other electoral systems, such as proportional representation. According to one veteran MoIM civil servant, the BRIDGE training was an “eye opening” opportunity that led him to reconsider longstanding assumptions about facets of electoral administration that he had previously considered immutable. Training feedback forms, collected from all participants, were laudatory and echoed sentiments expressed by the MoIM civil service official. Among the 20 MoIM participants, all rated the training a four or five (out of five) in terms of goals achieved, training styles, training skills and knowledge, and overall workshop quality. According to the civil society representatives, the training was effective because it did not advocate one electoral system over another but provided the background for determining which is the most appropriate. All those consulted over the course of the evaluation agreed that it underscored that a proportional representation system was most suitable to Lebanon. Their appreciation for these points made them more effective advocates of electoral reform, civil society participants said. The interactive quality of the training appealed to civil society participants, who felt that it gave them more experience speaking on these issues. CEPPS III - Support to the Political Process in Lebanon Page 14 Evaluation Report IFES increased the effectiveness of the training support by structuring the training to build on previous training provided in CEPPS II. Of the six MoIM participants, the four who previously participated in CEPPS II “train the facilitators” workshop received full accreditation as BRIDGE. The four staff are members of the MoIM office of Directorate General for Political Affairs and Refugees (DGPAR), which is responsible for election day logistics. By strategically structuring the training, IFES has created a nucleus of expertise within MoIM. Although these staff asked for additional training, they said that if tasked they could serve as a training unit within MoIM. The international study trip to Scotland had concrete effects in helping MoIM advance electoral reform. DG Khoury explained that he had considered instituting electronic vote counting, but he abandoned the idea after seeing the challenges of implementation in Scotland, specifically the 18 months of preparation that were required. The conversations between DG Khoury and Scottish officials covered the mechanics of ballot counting both in electronic and paper format, delving into details such as the software required to link a candidate’s name with the logo for his political party and whether to count at the polling station level. The study trip also cemented the relationship between IFES and MoIM, a relationship that before had seemed uncertain given the change in MoIM leadership and donor concerns over the new government makeup. While the BRIDGE trainings were administered efficiently and at a cost typical of this support, the efficiency of international professional development could have been improved with better targeting and planning. IFES spent resources in terms of time to arrange for the participation of Minister Baroud at the GEO in Botswana. Identifying alternate representatives, such as the DG from MoIM or senior civil society activists who can participate could help ensure that the program fully capitalizes on opportunities for international exchange. An additional efficiency could have been gained if the BRIDGE training were offered earlier. The BRIDGE training did not occur until in the final two months of the program, only after a modification was issued permitting work with these groups. Modifying the program to include civil society organizations was an important change and could contribute to the program’s long-term effectiveness. MoIM is severely under resourced in terms of human capital. Although training civil society activists when the program aimed to build the capacity of MoIM to implement electoral reform may seem like a tortuous rationale, one respondent explained that this approach is ideal for achieving long-term, sustainable impact. Because MoIM lacks adequate human capital, CEPPS III - Support to the Political Process in Lebanon Page 15 Evaluation Report it regularly recruits staff from the civil society community. By training a range of stakeholders, IFES created the conditions for long-term institution building. This lays the foundation for MoIM to eventually draw on the necessary expertise from civil society and operate self-sufficiently without international donor support. 7. Evaluation by objective: Enhance the capacity of the election administration and reform of the electoral process. 7.1 Activities and outputs under this objective IFES aimed to enhance the capacity of the election administration and reform of the electoral process through the review of key electoral procedures. This activity was expected to achieve two intermediate outcomes: first, improving MoIM’s knowledge of electoral operation systems suitable to Lebanon and, second, enhancing MoIM’s understanding of issues to consider in preparation for future elections. The program identified an ambitious set of deliverables to achieve these outcomes. These included a comprehensive review of the key electoral procedures, a roundtable on drafting regulations and procedures, a final review report with recommendations to improve the conduct of the 2013 elections, model guidance documents for various electoral procedures, and ongoing advice to the MoIM on the development of improved electoral procedures. IFES did not report any progress toward these objectives in the first two quarters, citing political developments. Given the uncertain mandate given to a caretaker ministry, more active partnership with MoIM at this time would have been difficult. Once the program submitted a modification on June 28 changing the SOW to include work with civil society organizations, IFES made more progress toward this objective. In the fourth quarter of the program, IFES completed its first activities toward this objective. On December 16, 2011, IFES held a roundtable for electoral stakeholders to review regulations and procedures related to electoral access. The roundtable focused on a review of the implementation of regulations mandating disability access to elections. In addition to reviewing whether there was adequate access for persons with disabilities, participants discussed questions such as criteria for choosing locations for polling stations and measures to ensure a calmer election day. The program identified one activity completed under this objective, a roundtable for civil society on the reform of polling procedures. Among respondents interviewed over CEPPS III - Support to the Political Process in Lebanon Page 16 Evaluation Report the course of the evaluation, feedback regarding about the activity was uniformly positive. There were requests for more, similar roundtables, and a concern that shutting down IFES’ programming would diminish the possibility of engagement between civil society activists, government officials, and IFES experts. 7.2 Performance evaluation and findings In evaluating the degree to which IFES achieved this objective, it is worth first asking whether this objective was organized appropriately within the context of the program. Many of the results that were anticipated under this objective are similar, if not identical to what IFES has proposed in other aspects of CEPPS III. For example, the program’s first objective, enhance the capacity of the Lebanese authorities and election stakeholders to deliver more professional, efficient, and reliable electoral administration, identifies “improvement of the MoIM's and other stakeholders' knowledge of different electoral systems and electoral reforms suitable to Lebanon” as one of the results from its work to support to electoral reform initiatives. This is nearly identical to one of the results anticipated from its review of key electoral procedures, which is articulated as “an improvement of the MoIM’s knowledge of electoral operation systems suitable to Lebanon.” IFES did fulfill many of the deliverables expected under this objective. However, because of the way the program was organized through its PMP and other program planning documents, the accomplishments that could be counted toward this objective are ascribed toward other objectives. The program’s efforts to provide comparisons of the different electoral reform laws under consideration could comfortably fit under this objective, but the program decided to designate it as contributing to the first objective. In discussing a rationale with IFES staff, a distinction was drawn between “procedures,” and “reforms.” Reforms are legal changes mandated by law, such a redistricting, pre-printed ballot voting, or voting on a single election day. Procedures are less significant, often logistical issues, such as ensuring handicapped access or adequate marking of polling station. However, this distinction is not explained in any of the program documents. This ambiguity underscores the need for a more effective program design that avoids similar, redundant, or even identical program outcomes. 8. M&E/Program Procedures IFES’ work on CEPPS III centers around three different objectives. As explained in earlier sections, these objectives have significant overlap and ambiguity about their measurement. IFES identified several indicators across the program as a whole, which are not specifically tied to any of the objectives. The vagueness of the objectives makes it CEPPS III - Support to the Political Process in Lebanon Page 17 Evaluation Report difficult to know what outputs contributed to which objective. IFES staff mentioned uncertainty about how to count outputs from different activities and were concerned about double counting, although they were careful to make sure that none occurred. On many programs, including those in the democracy and governance sector, the typical logical framework hierarchy delineates inputs, outputs (sometimes referred to as activities), outcomes (sometimes referred to as purpose or intermediate results), impact (sometimes referred to as objectives), and goals. A program’s theory of change can be developed by articulating this hierarchy through if  then statements. For example, if the program provides funding for a training (input)  20 people will be trained (output)  increased understanding of electoral principles (outcome)  improved ability to implement electoral reform (impact)  successful conduct of elections in accordance with international standards. In CEPPS III, the terminology used is activities, objectives, results and deliverables. While the use of goals does not appear in the results framework, elsewhere in project documentation long-term and short-term goals are mentioned. The program’s theory of change, or the steps through which its outputs achieve higher￾level outcomes, is not immediately clear from results framework. The most likely hierarchy is that IFES’ deliverables and activities are both outputs, with activities comprising multiple outcomes. Results are intermediate outcomes. Objectives are what others might term, impacts. A central weakness in the results framework is that many of the results seem not to precede the objective but to follow from it. For example, CEPPS III’s first objective is to “enhance capacity of the Lebanese authorities to deliver more professional, efficient, and reliable electoral administration.” Among the results, or intermediate outcomes, are “More efficient and transparent election administration” and “Improved administration of electoral events.” It is in the reverse order to suggest that improved administration of electoral events will cause enhanced capacity of Lebanese authorities. Rather, enhanced capacity of Lebanese authorities will lead to improved administration of electoral events and more efficient and transparent election administration. Another concern about the results framework is that it is unwieldy in its repetition. Many of the results are repeated across different outcomes. This could contribute to the ambiguity and overlap between the first and third objectives of the program. According to IFES’ staff, the results framework was not a meaningful representation of the program’s activities and theory of change. It was not used to direct the program CEPPS III - Support to the Political Process in Lebanon Page 18 Evaluation Report during program implementation and does not reflect the real outcomes the program anticipates. The program instead referred to a work plan that details the main activities on the project in a format similar to a Gantt chart. Part of the challenge in constructing the framework is that it would not be realistic to arrange a meeting with senior MoIM officials to discuss a results framework. This is a more reasonable expectation for an organization partnering with civil society. Apart from issues about the program’s theory of change and its results hierarchy, the program adequately monitored its outputs. Though no measure of the program’s effectiveness, quantitative tracking used four different indicators. Indicator Target Actual Number of electoral administration procedures and systems strengthened with USG assistance 8 13 Number of reports, briefings and/or advisory notes issued to Lebanese authorities on electoral reform issues 7 11 Number of election officials and stakeholders trained with USG assistance 20 81 Number of model guidance documents on electoral procedures and draft regulations issued to Lebanese authorities 7 8 The program staff maintained files containing every item used to count these indicators. The program also developed a Performance Management Plan (PMP), a document that is used frequently by USAID partners as a tool to manage the collection of performance data. Performance Indicator Reference Sheets (PIRS) comprise much of the PMP. The CEPPS III PIRS carefully defines each indicator, discusses relevance to the program objectives, and considers issues of data quality. The PMP shows a high level of attention to monitoring program achievements at the output level. Yet similar to the results framework, the PMP also overstates its activities in gauging higher-level outcome and impacts. The PMP explains that after each training “As part of project monitoring and evaluation, IFES will assess the impact of program results on the status of women and men.” The program did collect review forms from training participants and disaggregate their feedback by gender. It also made efforts to include women in training activities. Yet this was not considered a program objective, and there is no higher￾level analysis of how the program had any meaningful gender-related outcomes, nor systems set up to measure this change. 9. Lessons Learned/Recommendations Diversify partnerships CEPPS III - Support to the Political Process in Lebanon Page 19 Evaluation Report Before the program received approval for a modification to its scope of work, it was not permitted to work with the civil society activists under CEPPS III. This had serious implications for the program, as the BRIDGE trainings and civil society roundtables were limited to the final quarter of the program. The rationale for this limitation lay in the different scopes of work CEPPS designated to other consortium members. For several reasons, working with civil society organizations to advance electoral reform is critical. First, civil society will be the long-term advocates on these issues after IFES leaves. Many of the civil society activists expressed an interest in receiving additional BRIDGE training, which could help spread this curriculum without IFES leadership. Second, MoIM will frequently draw on civil society expertise when hiring consultants and staff. By strengthening civil society’s capacity for electoral reform, MoIM is assisted in the long run. Third, civil society advocacy is less subject to concerns about whether the government is a suitable partner. When the donor community was wary of partnering with the Mikati cabinet that included Hezbollah, civil society could continue to push for electoral reform. Improve M&E by focusing on realistic outcomes As described in detail in the section on program procedures and M&E, the program did not develop a results hierarchy that realistically described how project outputs would lead to program outcomes and impacts. IFES staff recognize that the results hierarchy was simply a mixed list of many wildly aspirational outcomes. This posed a serious evaluation challenge, as it was not possible to examine whether the theory of change was plausible. IFES’ ability to learn from this program and construct a program that builds on its strengths is diminished. In the future, IFES needs to develop a simple chain of how the program envisions outputs will lead to outcomes. In the absence of a realistic results chain, IFES is vulnerable to unreasonable expectations by donors, beneficiaries, and other partners. The program is liable to be evaluated on whether or not electoral reform is enacted, which is clearly outside the program’s control or manageable interests. Cultivating and maintaining relationships are critical Much of the program’s effectiveness comes from the close relationships that were cultivated between IFES staff and MoIM officials. Without these partnerships, it would have been impossible to establish a balance between supply and demand driven capacity building. IFES provided some deliverables on its own initiative, while others were produced at MoIM request, yet all were valued to some degree. The maintenance of this partnership was particularly impressive after the departure of Minister Baroud. CEPPS III - Support to the Political Process in Lebanon Page 20 Evaluation Report The study tours were important in this regard, as they provided an opportunity over several days for senior MoIM officials to familiarize themselves with IFES and its expertise. Through carefully managing its relationship with MoIM, IFES balanced its interest in supporting electoral reform with the understanding that the appearance of too much involvement by a foreign organization could undermine support for the measures. Increase coordination between donors and other stakeholders The program’s efficiency could have been improved if the boundaries between NDI and IFES were more strategically defined and there were increased coordination between both partners. Although there was a general understanding of each organization’s mandate, with NDI working on civil society and women’s issues and IFES focusing on support to MoIM, the assistance resulted in redundant deliverables. NDI’s funded an electoral simulator through CCER, though several interviewees explained that it did not have the same functionality as the IFES supported version. The structure of the CEPPS partnership also meant that IFES could not start its work with civil society, including the BRIDGE training and procedural roundtables, until after a modification was issued midway through the program. Coordination between IFES and the UNDP, which provided additional staffing support to MoIM, operated more smoothly and appeared complementary. While CEPPS III was underway, IFES was simultaneously implementing U.S. government funded programs on behalf of the Department of State’s Middle East Partnership Initiative (MEPI) and DRL. IFES staff explained that there these programs were also complementary and helped the program diversity its partnerships. Disseminate deliverables more widely The analysis, reports, and tools prepared as part of the project were developed to improve the capacity of the MoIM. But the insights, especially around the election law, are valuable to anyone interested in Lebanese politics. This is not limited to civil society, which received access to some of the products, but includes journalists, academics, political parties, and the international community. While the electoral simulator was recoded to allow its use as an open source product, it is not widely accessible. Expanding distribution could increase the overall public understanding on these complex issues. Though this entails an element of awareness raising regarding election issues which was outside the scope of IFES’ work, some small efforts in this direction would increase the long-term impact of its contributions. IFES staff explained that the reason to restrict products to a limited distribution is that these resources could also be CEPPS III - Support to the Political Process in Lebanon Page 21 Evaluation Report used to question MoIM’s positions, such as their support for proportional representation. Seek additional opportunities for sustainability The program’s BRIDGE training, one of the most valued parts of its work, created a set of certified trainers within MoIM. This group could serve as an entity for internal training within MoIM, which does not currently exist. The MoIM staff are willing to assume this responsibility but have not received the mandate to do so. Such a group could continue to build the capacity of MoIM without donor support. IFES could help formalize this process. Though outside the scope of IFES’ work on CEPPS III, it is something to consider on future programs. 10. Conclusion When the CEPPS III program closes on December 31, 2011 MoIM’s task of preparing for the 2013 elections will be more difficult. When an electoral reform law is ultimately passed, its implementation will be a challenge for a government agency that has been historically understaffed and under resourced. The activities provided under CEPPS III have improved MoIM’s ability to face this challenge, but significant hurdles remain. Given pressing democracy and governance challenges facing other parts of the Arab world, one respondent expressed concern that the U.S. may leave Lebanon “blowing in the wind.” Other donors, such as the UNDP or the European Union, could fill some of the gaps left by IFES’ absence. What is irreplaceable is the personal relationships that have been built between IFES and MoIM officials. USAID will not have the same influence on these issues that it did while CEPPS was active. CEPPS III - Support to the Political Process in Lebanon Page 22 Evaluation Report Annex 1: List of People Met Name Title Organization Ahmad Abdallah Voter Registration MoIM Samih Ayoub Head of the Election Affairs Department MoIM Ghassan Bitar Office Manager IFES Richard Chambers Chief of Party IFES Elie Haddad Administrative Officer IFES Micheline Haddad Voter Registration MoIM Riad Kaake Finance Manager LADE Susan Kemp Program Officer, MENA IFES Elias El-Khoury Director General MoIM - DGPAR Rindala Mikhael Program Assistant CCER Makram Ouaiss Board Member LADE Marie-Louise Ramy Program Officer IFES Ahmad Rawas Head of the Election Affairs Department MoIM Nada Abu Saleh Head of PR MoIM Said Sanadiki Election Administration Specialist IFES Mirna Shidawi Program Manager IFES CEPPS III - Support to the Political Process in Lebanon Page 23 Evaluation Report Appendix 2: Timeline Program Activity June 28: Program modification issued expanding SOW to include work with civil December 16: Program holds roundtable for civil society December 26: Program evaluation begins January 12: Unity government collapses after ministers resign over tribunal January 28: Analysis of IEC staffing structure and requirements presented to MoIM January 19: Briefing on electoral districting provided to MoIM March 7-9 Planned trip for MP Baroud to GEO Conference in Botswana, later cancelled November 10-14: Study trip to Scotland June 26: Marwan Charbel succeeds Ziyad Baroud as MoiM Minister June:MoIM and CCER introduce electoral reform bills June 13: PM Mikati introduces new cabinet October: MoIM introduces new draft law developed by Charbel June: Electoral systems simulator presented to MOIM October 5-14: BRIDGE training presented in Beirut Political Developments November 29 – December 8: 2nd round of BRIDGE training presented in Beirut CEPPS III - Support to the Political Process in Lebanon Page 24 Evaluation Report Appendix 3: Scope of Work Under the leadership of the Chief of Party in Lebanon, the consultant will be expected to fulfill the following responsibilities: • Review the performance management plan (PMP) and results framework for IFES-Lebanon’s USAID-funded program; • Meet with IFES’ donors, partners and staff, as necessary, to gain an understanding of the activities and results of IFES’ program; • Review the tools that IFES is currently utilizing to collect qualitative and quantitative data; • Review project documents, reports and other project materials for evaluation purposes; • Develop a narrative report of approximately 15 pages in English on program results and impact that follows the provided template.