Political Party Development Program Performance Evaluation Report June 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP) by Katherine S. Vittum, Nicholas J. Demeter, and Atiq Ur Rehman. POLITICAL PARTY DEVELOPMENT PROGRAM PERFORMANCE EVALUATION REPORT June 2014 Contracted under No. GS-23F-8012H and Order No. AID-391-M-11-00001 Monitoring and Evaluation Program (MEP) DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. i CONTENTS Program Summary .................................................................................................................................................................................. 5 Program Background .............................................................................................................................................................................. 6 Evaluation Purpose and Evaluation Questions ...............................................................................................................................18 Evaluation Methods and Limitations .................................................................................................................................................20 Findings and Conclusions for Evaluation Question 2 ..................................................................................................................23 Findings and Conclusions for Evaluation Question 1 ..................................................................................................................47 Main Conclusions and Recommendations ......................................................................................................................................51 Annexes ...................................................................................................................................................................................................54 Annex I: Evaluation Statement of Work ....................................................................................................................................55 Annex II: Data Collection Instruments.......................................................................................................................................89 Annex III: Sources of Information ................................................................................................................................................93 Annex IV: NDI Statement of Differences on the Evaluation of USAID/Pakistan’s Political Party Development Project................................................................................................................................................................................................. 95 Annex V: MEP Response to NDI Statement of Differences on the Evaluation of USAID/Pakistan’s Political Party Development Project ................................................................................................................................................................... 101 Tables and Figures Table 1: Project Summary ..................................................................................................................................................................... 5 Table 2: Results Framework for PPDP ............................................................................................................................................14 Table 3: Political Parties Participating in PPDP ..............................................................................................................................16 Table 4: Status of Activities by Party................................................................................................................................................17 Table 5: Field activities for Focus Group Discussions, Group Interviews and Key Informant Interviews .....................21 Figure 1: Political Party Development Program – Theory of Intervention .............................................................................. 7 Figure 2: The Original Policy Development Cycle ......................................................................................................................... 8 Figure 3: The Revised Policy Development Cycle as Presented in the SOW for the PPDP Evaluation .......................... 9 Figure 4: Views of Party Leadership Involvement in PPDP ........................................................................................................25 Figure 5: Method of PWG Selection ................................................................................................................................................28 Figure 6: Training Modules Attended by Research Unit Members ..........................................................................................33 Figure 7: PWG and RU Composition by Member Rank .............................................................................................................34 Figure 8: Review, Revision, and Incorporation of Policy Drafts................................................................................................38 ii ACKNOWLEDGMENTS The Monitoring and Evaluation Program (MEP) would like to thank Katherine S. Vittum (short-term Democracy and Governance Specialist and Evaluation Team Leader), Nicholas J. Demeter (short-term Governance Specialist) and Atiq-ur-Rehman (short-term Organizational Development Specialist) for undertaking desk reviews and field work for the mid-term evaluation and completing the evaluation report. Jennifer Mandel (Senior Evaluation Advisor, MEP) guided MEP staff and the evaluation team in the data collection design and analysis of qualitative data and provided training for conducting focus group discussions. Tariq Husain (Director, Evaluation Unit, MEP) managed the evaluation and also conducted focus group discussions with Asma Kiran (Relationship Manager, MEP) and Ghazanfar Ali Khan Hoti (Senior Evaluation Specialist, MEP) before the arrival of the evaluation team. Fatima Abbas (Research and Evaluation Officer, MEP), Asma Kiran and Zameer Haider (Evaluation and Information Officer) participated in the peer review process and helped with substantive and editorial improvements. Douglas Krieger (MSI Technical Director for MEP) provided detailed comments in the final steps of report writing to improve the evaluation logic, clarity and presentation of the report. MEP expresses special thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:  Political party officials at the national, provincial and district levels;  The country team of National Democratic Institute (NDI), including the Senior Resident Country Director/Chief of Party, the Deputy Chief of Party, the M&E Manager and other staff members;  The Resident Director of the International Republican Institute (IRI) and the Director of the Institute of Public Opinion Research (IPOR);  The team of Strengthening Participatory Organization (SPO); and  USAID officials responsible for the Political Party Development Program. iii ACRONYMS ADS COR Automated Directives System Contracting Officer’s Representative CSO DEC ECP Civil Society Organization Development Experience Clearinghouse Election Commission of Pakistan ESOMAR FGD FY HDR European Society for Opinion and Market Research Focus Group Discussion Fiscal Year Human Development Report IIW Issue Identification Workshop IPOR IR Institute for Public Opinion Research Intermediate Result IRI International Republican Institute KI Key Informant KII Key Informant Interview KP Khyber Pakhtunkhwa MEP MDG Monitoring and Evaluation Program Millennium Development Goal MNA Member of National Assembly MPA MSI Member of Provincial Assembly Management Systems International NDI National Democratic Institute PDC PDI Policy Development Cycle Policy Development Index PPDP PSLM Political Party Development Program Pakistan Social and Living Standards Measurement Survey PWG Policy Working Group RTA RU Research Training Academy Research Unit SOW Statement of Work SPO TA TFP ToT USAID Strengthening Participatory Organization Technical Assistance Training Fellowship Program Training of Trainers United States Agency for International Development 1 EXECUTIVE SUMMARY The Political Party Development Program (PPDP) works with the political parties of Pakistan to help them engage their members and leaders in policy development and selected additional aspects of political participation1. With a budget of US$21.5 million, the program2 operates under a cooperative agreement and is implemented by the National Democratic Institute (NDI) and its two partners, the International Republican Institute (IRI), which partners with the Pakistan-based Institute for Public Opinion Research (IPOR), and Strengthening Participatory Organization (SPO), a Pakistani non￾governmental organization. The overall goal of PPDP is to “enhance the ability of political parties to contribute to democratic policy-making and governance processes at the national and local levels in a more open, research-driven and representative manner and thereby generate greater public trust and confidence in their ability to govern and effect positive policy change.”3 The framework for the program is the Policy Development Cycle (PDC), a sequence of 12 phases designed to engage a diverse array of party members from the grassroots to party leadership in policy development. The PDC emphasizes research-based policy development, with each of its phases designed to build on the previous phase. PPDP implements 15 activities (the 12 phases of the PDC and three supplementary activities), which it refers to as 1) Leadership commitment to the PDC, 2) Issue Identification Workshops (IIWs), 3) Formation of Policy Working Group (PWG), 4) Party Member Surveys, 5) Formation of Research Units, Research Training Academy (RTA) Curriculum, 6) Public Opinion Research, 7) Collaboration with Civil Society Organizations (CSOs), Think Tanks and Experts, 8) Policy Drafting, 9) Leadership Review of Policy Drafts, 10) Policy Conferences, 11) Communication Training (central and district level), 12) Training Fellowship Program (TFP), 13) Multiparty Roundtables, 14) IPOR Capacity Development, and 15) Technical Training for Political Parties. These activities4 contribute to five objectives and 11 intermediate results documented in the program’s results framework. As of February 2013, PPDP was working with 10 parties at the national and provincial levels and expected to add two more during Fiscal Year 2013. The program emphasizes including women, youth, and grassroots party members in program activities. The evaluation covered the first 18 months of the project, from July 15, 2011 to February 14, 20135 and took place prior to USAID’s decision on whether to extend the project beyond the initial two-year funding period. It focuses on three main questions:  Evaluation Question 1: How effectively are the partners implementing the planned approach and what changes to the approach might improve the efficiency with which the overall set of activities contribute to achieving intended intermediate results? 1 Agreement number AID-391-A-11-00004. 2 This report generally uses the term “program” instead of “project.” The term “project” is used in the sections on evaluation purpose and evaluation questions, evaluation methods and limitations, and in the evaluation questions in order maintain consistency with the language in the evaluation Statement of Work. 3 USAID RFA 391-11-006: Political Party Development Program, p.18. 4 The term “activity” is used here to refer to “phases” as defined in the PDC and to individual activities outside the PDC. 5 The evaluation reflects program activities and information through March 2013, where available. 2  Evaluation Question 2: How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results?  Evaluation Question 3: How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? Evaluation question 3 asks solely for recommendations, which are based on the findings and conclusions of evaluation questions 1 and 2. The evaluation is based on rigorous qualitative search, including: a desk review of a large volume of PPDP documents; interviews with central and provincial leaders, party researchers and master trainers; and focus group discussions with central, provincial and district-level leaders and workers of ten political parties in all four provinces of Pakistan (133 individuals in all). The evaluation offers eight main conclusions in response to evaluation questions 1 and 2, which inform the recommendations to answer evaluation question 3. The conclusions are: 1. The involvement of party leadership in PPDP is overall positive, while moderate. 2. The IIWs effectively engaged a broad range of grassroots party members and party leaders in the process of identifying issues, including those related to the Millennium Development Goals (MDGs), and the party member surveys served as a party building exercise. Both activities contributed to the aggregation of local concerns, as the project intended. The party member surveys employed some of the techniques used in formal large-sample surveys but the results of the surveys do not have statistical validity since they did not use standard research methods. 3. The use of PWGs for policy drafting reflects an important change in the standard top-down approach to policy development. 4. The composition of the PWGs and Research Units do not match the skills required, which limits the potential effectiveness of policy and research processes, two main elements of the program. 5. The RUs and the RTA curriculum, public opinion polls and the CSO forums have modestly increased the capacity of parties to define and utilize research. The low attendance rates in RTA modules, weaknesses in the composition of Research Units, disconnect between these units and PWGs, and problems with the quality of speakers and facilitation at some CSO forums have inhibited the effectiveness of these activities. 6. The TFP has effectively supported parties to enhance their internal training capacity. The Training Fellows do not yet formally contribute to other PPDP activities. 7. The investment in IPOR has shown results in terms of building IPOR’s technical and organizational capacity and providing technical training for political parties. IPOR appears to be progressing towards eliminating its dependence on IRI. Travel delays for the US-based survey research consultant could inhibit planned technical training for IPOR. 8. The multiparty roundtables resulted in concrete recommendations to enhance the integrity of the 2013 electoral process and appear to have achieved the objective of strengthening democratic political party practices. 3 The evaluation offers the following recommendations for program management based on the conclusions summarized above and elaborated in the main report: 1. In order to enhance the constructive involvement of party leaders in PPDP, NDI could seek written (not just verbal) commitment from party leadership for the second program cycle. This document could outline roles for party leaders and PPDP. 2. Depending on the design of the IIWs for the second program cycle (and particularly if many of the same people participate) NDI could request that parties select PWG members ahead of the IIWs. PWGs could provide an overview of the policy issues in the province, such as MDG themes, during the IIWs. The information could serve as a starting point for discussion and demonstrate to participants the benefits of using secondary sources of information, including national, provincial and district-level data, as a low cost alternative to parties conducting their own research and data collection. It is understood that this addition to the IIWs would need to be informative rather than prescriptive and facilitate grassroots participants to brainstorm and contribute their ideas without influence. 3. PPDP program documents and practice could be modified to clearly articulate the intent for party member surveys to be a party mobilization/outreach exercise, as distinct from a statistically reliable tool to inform policy drafting. Political parties need to understand how to solicit member input that is not meant to provide statistically reliable results and how to use the results of such exercises in tandem with other informal and formal data collection methods. As an alternative means to inform policy drafting, IIW participants could meet with party district committees to seek their input on the priorities emerging from IIWs and share this feedback with PWGs. 4. NDI could develop clear criteria to provide to party leadership for selecting new PWG members. The members could include issue experts from the party who can effectively contribute to policy drafting (and legislative drafting, should the focus change to legislative drafting for the next program cycle). The criteria could also include research background. PPDP could explore the option of integrating Research Unit members into the PWGs to promote more effective coordination between the research and policy development processes. 5. PPDP could develop clear criteria to provide to the party leadership for selecting/replacing researchers to ensure that Research Units have the appropriate composition of talent and committed participants to support the work of the PWGs and the party as a whole. (PPDP has already flagged this as an issue that the program intends to address ahead of the second program cycle.) The criteria might include a stronger emphasis on low and mid-ranking party members with demonstrated aptitude for research, writing and communication. In coordination with party leadership, IRI could screen the nominees and conduct regular assessments of the progress and commitment of individual researchers to determine their particular training needs and whether or not to keep them in the program. Selection could be on the basis of capacity, interest and record of commitment. The training could emphasize the benefits of using secondary research to inform PWG/party activities and focus on participants becoming better consumers – not necessarily producers – of research. 6. To ensure that CSO forums achieve the goal of informing policy development, NDI will need to ensure that the speakers who are selected are knowledgeable about the provinces and well￾briefed on the purpose and policy priorities of PWGs. 4 7. PPDP could explore ways to engage selected Training Fellow graduates in other aspects of the program, such as future communication training for national, provincial and district party officials. This could improve the effectiveness of these activities and demonstrate to the party members the skills of these individuals. 8. IRI could encourage IPOR to explore options for a corporate governance structure in line with Pakistani law and international good practice that can replace IRI oversight when PPDP ends. This could help to promote the integrity of the company, attract business, and provide direction on legal, financial and personnel matters. In addition, in order to allow the IPOR Executive Director to focus on the technical side of the business, it would be very helpful to hire additional senior staff to take on some of the operational responsibilities. Should the U.S.-based survey research consultant face further travel delays, PPDP may need to identify other experts who can provide planned in-country technical training for IPOR. 5 PROGRAM SUMMARY The Political Party Development Program (PPDP) works with the political parties of Pakistan to help them engage their members and leaders in policy development and selected additional aspects of political participation. The program6operates under a cooperative agreement. The continuation of the program for the remaining three years is subject to a successful performance evaluation of the initial program period. The evaluation covers the first 18 months of the program, from July 15, 2011- February 14, 2013. Table 1 summarizes key facts about PPDP. TABLE 1: PROJECT SUMMARY 6 This report generally uses the term “program” instead of “project.” The term “project” is used in the sections on evaluation purpose and evaluation questions, evaluation methods and limitations, and in the evaluation questions in order maintain consistency with the language in the evaluation Statement of Work. Program Name/Title Political Party Development Program Agreement Number AID-391-A-11-00004 Agreement Officer’s Representative (AOR) Humaira Ashraf (Ms.), Program Management Specialist Program Start Date July 15, 2011 Program Completion Date July 14, 2013 Program Location Nationwide USAID Objective Addressed Cross-cutting Objective1: Improved governance in response to citizen concerns. Name of Implementing Organization National Democratic Institute (NDI). NDI has two partners:  International Republican Institute (IRI), which partners with the Institute for Public Opinion Research (IPOR); and  Strengthening Participating Organization (SPO). Budget US$21.5 million 6 PROGRAM BACKGROUND THE PROBLEM STATEMENT The problem which USAID plans to address through PPDP is the lack of open, well-informed and democratic policy making and governance processes by political parties in Pakistan.7 As described in the USAID Request for Application (RFA) 391-11-0068 “political parties in Pakistan have historically been viewed as personality-driven, out of touch with public priorities and lacking in evidence￾based policy development.” The personal interests of the party leader or the vision that leader has for the party and the country tend to determine the policy orientation of the party.9 This top-down approach leads to policies that have little input from rank-and-file party members and ordinary citizens. As a result, citizens have little stake in the success of policies and little confidence in the parties or the political process. Several factors have impeded the development of strong democratic political parties in Pakistan. For more than half of its history as an independent nation, Pakistan struggled to operate under the structural constraints imposed by military rule and executive dominance.10 Parties have lacked internal party democracy, choosing to elect their top leadership unopposed rather than through intraparty elections.11 They have also had highly centralized decision-making that excludes local party branches, party members and citizens from engaging in party politics or informing policy development.12 THE THEORY OF THE INTERVENTION As articulated in NDI’s technical proposal13 and PPDP’s Year 1 Work Plan14, PPDP aims to introduce a paradigm change, referring to the new paradigm as the Policy Development Cycle (PDC). The paradigm shift is described in the Year 1 Work Plan as follows: [The PDC] will incorporate a diverse array of party members from the grassroots activists to the party leadership into the policy development process. Participating in the Policy Development Cycle will provide party members with increased capacity to undertake key party functions including opinion research, skills training for party members and policy development. The Cycle is intended to be a model from which individual political parties would ultimately adopt and modify different elements for their own use. 7 USAID RFA 391-11-006: Political Party Development Program, p.18. 8 USAID RFA 391-11-006: Political Party Development Program, p.19. 9 Democracy within parties by Rasul Bakhsh Rais, http://www.pk.boell.org/web/111-309.html. 10 USAID RFA 391-11-006: Political Party Development Program, p.19. 11 Ibid, p.19. 12 Ibid, p.20. 13 NDI Technical Proposal: Political Party Development Program in Pakistan (USAID-Pakistan RFA 391-11-006), May 2011, p. 5. 14 NDI PPDP, Work Plan Year 1, February 2012, p. 2. 7 The paradigm shift envisaged through the PDC is expected to result in an impact in terms of the public’s trust and confidence in political parties. The logic linking project activities to eventual impact is illustrated in Figure 1. FIGURE 1: POLITICAL PARTY DEVELOPMENT PROGRAM – THEORY OF INTERVENTION THE DESIGN OF THE PROGRAM The overall goal of PPDP is to “enhance the ability of political parties to contribute to democratic policy-making and governance processes at the national and local levels in a more open, research-driven and representative manner and thereby generate greater public trust and confidence in their ability to govern and effect positive policy change.”15 As described above, the framework for the program is the PDC. The PDC includes multiple phases, from the identification of policy issues to the drafting, adoption and communication of new policies. Each phase is designed to build on the previous one. The PDC is intended as a model that individual parties would ultimately adopt and modify for their own use. See Figures 2 and 3 below. 15 USAID RFA 391-11-006: Political Party Development Program, p.18. EFFECT GREATER PUBLIC TRUST AND CONFIDENCE IN PARTIES’ ABILITY TO GOVERN CAUSE POLITICAL PARTIES CONTRIBUTE TO DEMOCRATIC POLICY MAKING AND GOVERNANCE PROCESSES IN A MORE OPEN, EFFECT RESEARCH-DRIVEN AND REPRESENTATIVE MANNER CAUSE DIVERSE ARRAY OF PARTY MEMBERS INVOLVED IN POLICY DEVELOPMENT PARTIES TRAINED TO DEFINE AND UTILIZE RESEARCH 8 FIGURE 2: THE ORIGINAL POLICY DEVELOPMENT CYCLE16 The original PDC was arranged as a series of 11 sequential steps. 16 The content of this PDC is the same as in the NDI Technical Proposal: Political Party Development Program in Pakistan (USAID-Pakistan RFA 391-11-006), May 2011, p. 6. The evaluation team has redesigned the figure to distinguish the various steps and program objectives if printed in black and white. STEP 6 Provincial Policy Review STEP 7 National Policy Convention STEP 8 Integration of Policy Proposals into Party Policies STEP 9 National Level Public Opinion Research and Message Develop STEP 10 Message Desimination at Grassroots Level STEP 11 Individual Cycle Components Formally Adopted by Party STEP 1 Grassroots Issue Identification STEP 2 Policy Working Group and Research Unit Formation STEP 3 CSO, Think Tank and Expert Engagement STEP 4 Policy Drafting STEP 5 Provincial Level Public Opinion Research 1 2 3 4 Objectives 3 1 1 1 1 2 3 1 3 1 1 2 3 3 4 4 4 4 4 9 FIGURE 3: THE REVISED POLICY DEVELOPMENT CYCLE AS PRESENTED IN THE SOW FOR THE PPDP EVALUATION17 The revised PDC is shown as 12 phases with the Training Fellowship Program and the single and multiparty roundtables supplementing the PDC. PPDP is comprised of 15 major activities, including 12 phases contained in the PDC, two activities that supplement the PDC, and one activity outside of the PDC associated specifically with the 2013 elections. PPDP expects to cycle through the components of the PDC twice during the five-year program. The first cycle emphasizes learning-by-doing, in which the program partners facilitate each activity and is scheduled to be completed ahead of the May 2013 elections. The second cycle is designed to focus on developing policy that can be integrated into legislation in the provincial and national assemblies following elections. 17 USAID Political Party Development Program (PPDP) Evaluation Statement of Work, March 19, 2013, p. 10. Phase 1: Parties commit to PPDP Phase 2: Issue Identification Phase 3: PWG Formed Phase 4: RU formed and begin RTA Phase 5: Party member surveys Phase 6: Public Opinion Research Phase 7: Consultation at CSO forums Phase 8: Preparation of policy draft Phase 9: Leadership review of policy draft Phase 10: Policy conference Phase 11: Comm. workshop (Leaders) Phase 12: Comm. workshop (district) Activities designed to supplement the PDC PPDP NDI will also provide electoral assistance in Pakistan in FY13 by training party poll watchers. Single/Multiparty Roundtable Training Fellowship Program Completed On-Going/In Progress Will begin and be completed in FY13 10 PPDP starts the cycle by seeking buy-in from party leaders in one-on-one meetings. The 15 activities of PPDP are described below.18 Phases of PDC Phase 1: Parties Commit to PPDP. NDI invites USAID-approved political parties to participate in PPDP. If the leadership of a political party assigns a PPDP liaison and participates in one or more phases of the PDC, NDI considers that party to be committed to PPDP. Phase 2: Issue Identification. Each party holds an Issue Identification Workshop (IIW), organized and jointly facilitated by NDI and SPO. IIWs include approximately 75 participants (most of them grassroots leaders) identified by the senior (that is, provincial and/or central) leadership and representing a diverse array of party members from within a province with a particular emphasis on representation from the district level or below. At least 35 percent of the IIW members must be women and 35 percent young members (18-35 years old). Democratically and through discussion, the IIW prioritizes three to four issues for policy development.19 Phase 3: Policy Working Groups (PWGs) Activated. The priority issues are taken up by the PWG, which is composed of 10 members (at least 50 percent women), most of whom also participated in the IIW. While IIW participants elected PWG members in the first year of the program, NDI has since modified the PWG selection process and now requests that party leaders select the PWG members based on specific criteria. PWGs are responsible for drafting policies that are research-based and developed in consultation with various strata of the party and external experts. They also conduct party member surveys, which they develop with NDI support. By sanctioning and supporting the creation of a PWG, a political party is informally institutionalizing its commitment to inclusive, responsive, and research-based party policy-making. Phase 4: Research Units (RUs) Form and Start Research Training Academy. RUs are internal party entities that support PPDP- related PWGs and focus on parties’ internal research capacity. A "formed" RU refers to the designation of party members as agreed upon by the party and PPDP implementers. RUs receive training (in six modules) from PPDP’s Research Training Academy (RTA) on public opinion research (using both quantitative and qualitative methods), a process that is led by IRI and may be assisted by IPOR. By forming a RU, a political party is informally institutionalizing its commitment to advancing internal research capacity. Phase 5: Party Member Surveys. NDI trains approximately 75 IIW participants to conduct surveys of 15-20 party members each in their districts to gather feedback on the 3-4 issues prioritized at the IIW. Although not representative,20 the surveys serve as a party-building exercise and survey results provide an opportunity for the PWGs to consider and incorporate grassroots party-member input into their policy drafting. 18 The descriptions are drawn from the PPDP Evaluation Statement of Work, March 9, 2013, pp. 9-13 and have been updated where more recent data was available. Complete data for January-March 2013 was not available, given the PPDP reporting cycle. 19 These issues have included health, education, unemployment, poverty, the economic crisis, law and order, the energy crisis, peace, justice, the rule of law and youth empowerment. 20 During the evaluation, PPDP clarified that the party member surveys are intended to be a party-building and party outreach mechanism and should not be viewed as a scientific tool or survey. The proposal and work plans describe the member surveys as simple surveys designed to inform policy development. PPDP intends to use the term ‘party member input forms’ (‘party forms’) instead of ‘survey’ beginning in April 2013. 11 Phase 6: Public Opinion Research. This refers to scientifically conducted polls and focus group discussions (FGDs). IRI prepares and conducts quarterly public opinion polls. The poll methodology oversamples each of Pakistan’s four provinces once a year to obtain provincial-level results. Polling results are used to verify that issues identified by PWGs reflect citizen concerns. In addition to quantitative research, IRI and IPOR conduct four FGDs each quarter to develop qualitative insights into specific issues identified by the PWGs, IRI, and NDI following analysis of polls. Party leadership and the PWGs are briefed on the poll results and FGDs as appropriate. This is an ongoing activity and extends beyond the PDC. Phase 7: Civil Society Organization (CSO) Forums. CSO forums, organized by NDI with SPO assistance, are designed to be venues for political parties and civil society to identify and discuss mutual areas of concern, in particular community-based problems and possible solution-oriented interventions. PWGs may also choose to continue consulting with CSOs, viewing them as issue-experts, while designing and drafting their party policies. Phase 8: Policy Drafting Based on Research and Party Input. The PWG prepares a policy draft with the technical assistance (TA) of PPDP. While NDI’s PDC and relevant activities are designed to encourage policy drafting based on research and party input, the depth and quality of research and solicitation of party member input depends on the PWG’s and party’s commitment and level of effort. Until the RUs reach the level of maturity required to support their parties, IRI and IPOR will share public opinion research with the PWGs. PPDP supports the PWGs during all phases of the PDC. However, the PDC is designed to be a self-directed and party initiated process. This approach is intended to build-in party ownership and could enhance the likelihood that the PDC, in whole or in￾part, would be incorporated into party practices, as appropriate to each respective party. NDI’s Policy Development Index (PDI) will quantify the qualitative policy development process on an individualized party basis.21 For the first PDC, all of the PWGs created work plans with the goal of completing their policy drafts before the elections, instead of the PPDP work plan target of completing them by September 2013.22 Phase 9: Leadership Review of Policy Draft. The senior leadership of the party reviews and approves the draft policy. Initiation and completion of this phase depends on each party’s leadership. NDI will proactively assist those PWGs that face challenges getting feedback from their leadership. In a manner determined appropriate by the PWGs, NDI will assist the PWGs in bringing the policy drafts to their party leadership’s attention for review and consideration. Among other possible mediums, NDI may facilitate a meeting for PWG representatives to present the draft policies directly and in-person to party leadership.23 Where appropriate, NDI may also discuss with leadership the possible options for incorporating PWG members into existing party structures (e.g., manifesto committee) or creating a permanent policy drafting unit should such a body not exist. Depending on a party’s response to the draft policies, NDI 21 Phases 1-8 of the first round of the PDC have been completed, while the subsequent phases are planned for the early part of 2013. Source: PPDP staff. 22 NDI reported that September 2013 was the original target date for completing the policy drafts. The evaluation team was unable to verify this in the available documents. 23 Ten party units had initiated the leadership review of completed policy drafts as of March 2013. Source: PPDP staff, April 2013. 12 will also inquire whether the party intends to incorporate, wholly or in part, any of the policies into its electoral platform or legislative agenda. Phase 10: Policy Conferences. The policy document is presented at the party’s policy conference. NDI will organize a policy conference with each participating party. The conference will serve as a venue to present the policies produced by that party’s PWGs. Grassroots, middle, and central level leaders will participate in the conference. The actual format will depend on party preferences. However, it is anticipated that PWG members will have the opportunity to present the policies as well as highlight the process undertaken to prepare the policies (i.e., the PDC). The agenda will likely also include an opportunity to debate the presented policies. One policy conference will be organized for each party. Therefore, for larger parties, all of their PWGs will be at the same conference. Following the conference, NDI will provide appropriate ongoing support geared towards facilitating party members’ advocacy efforts to include approved policies into legislative proposals or into their party manifestos.24 Phase 11: Communication Workshops (Leadership). IRI, in coordination with NDI, will conduct communication workshops with the central and provincial leadership of the party focusing on the skills required to disseminate party messages effectively within the party as well as externally. Where parties already have strategic communication plans in place, IRI will provide appropriate technical assistance on a customized basis as requested by the parties.25 Phase 12: Communication Workshops (District Level). Similar workshops are held at the district level. NDI, in coordination with IRI, will hold communication workshops with grassroots members of the party. The workshops will focus on the technical communication skills required to disseminate party messages effectively and will be tailored to the needs of party members.26 Activities to Supplement the PDC Activity 1: Training Fellowship Program (TFP). The purpose of the TFP is to build the parties’ capacity to conduct training for its members, activists, and candidates. Working with party leadership from up to 12 political parties, NDI invites the nomination of at least four party members, 35 years or younger, at least half of whom are women, from each provincial party unit to serve as Training Fellows. NDI has prepared four workshop modules, which are presented in a different province every four to five months. The modules reflect training-of-trainers (ToT) curriculum with an emphasis on adult training techniques, internal and external party communications, effective party organizing, leadership skills, fundraising, membership recruitment, volunteer recruitment and management, strategic planning and implementation, event planning, and election preparation. The TFP training methodology emphasizes group work and learning by doing. All Fellows receive ongoing written and verbal coaching and skills support from NDI to maximize knowledge retention and results. Each module also requires the Fellows to practice their newly learned knowledge and skills by training other party members in their home districts. Training Fellows are provided graduation certificates upon completion of the TFP. By holding the TFP in a multiparty format with substantial 24 One party held a party conference during the period covered by the evaluation. Source: PPDP Activity Tracking Sheet as of February 12, 2013. A second party held a party conference at the end of February 2013. 25 Six parties participated in national level communication training during the period covered by the evaluation. Source: PPDP Activity Tracking Sheet as of February 12, 2013. 26 Three parties participated in district-level communication training during the period covered by the evaluation. Source: PPDP Activity Tracking Sheet as of February 12, 2013. 13 group work, young party members are introduced to the concept of working across party lines—a first for many of them. Activity 2: Multi-Party and Single- Party Roundtables. NDI has been convening multiparty roundtables on elections on a regular basis to allow leaders to discuss issues relevant to political party strengthening and recommend measures to enhance the integrity of the 2013 electoral process.27 PPDP is also programmed to organize single-party roundtables, focusing on internal party democracy. The parties have informed PPDP that it is unrealistic to expect changes in internal operations before the 2013 elections; thus, single-party roundtables are expected to begin after the 2013 elections. Activity for 2013 Elections Activity 3: Party Poll Watchers. Specifically for the 2013 electoral process, PPDP will also train party poll watchers to enable them to participate effectively in the forthcoming elections. This activity has not yet taken place, but it is likely to occur before Election Day.28 Table 2 below summarizes the results framework for PPDP according to the program proposal and Year 1 Work Plan. The approach has evolved over time as noted below. The adjustments to the program approach between Year 1 and 229 include: 1. The think tank component was removed (IR 2.2); 2. Communications training sessions with national, provincial, and district party officials originally planned to follow the policy conferences will be conducted after the 2013 elections. A new set of communications trainings focused on elections were added (IR 3.2); 3. Activities under Objective 4 were delayed until after the 2013 elections at the request of the parties; 4. Objective 5 “Democratic political party practices are strengthened through effective multiparty participation in Pakistan’s electoral process” was added for the 2013 elections at USAID’s request30; and 5. The roundtables (IR 4.1) were removed from the PDC and the roundtables and TFP (IR 1.4) were defined as supplemental to the PDC. 27 All multiparty roundtables focused on electoral issues under IR 5.2. 28 As described above, this activity is beyond the scope of the evaluation. 29 The Year 2 Work Plan was approved in April 2013. 30 Objective 5 includes IR 5.1: “Poll watchers are enabled to participate effectively in the 2013 electoral process” and IR 5.2: ‘Party leaders discuss electoral issues and recommend concrete measures to enhance the integrity of the 2013 electoral process.’ IR 5.1 is beyond the scope of this evaluation. 14 TABLE 2: RESULTS FRAMEWORK FOR PPDP Objectives and IRs Activities as in the Year 1 Work Plan Objective 1: Parties conduct their own research, analysis, and training for the formulation of increasingly responsive and informed platforms and policies; and parties contribute more actively and effectively in the policy￾making of government institutions such as legislatures and commissions. 1.1 Party leadership participates in the policy development cycle 1. Leadership commitment to PDC 2. Leadership review of policy drafts 1.2 Parties increase policy development capacity at the party and legislative levels 1. Formation of PWGs 2. Development of policy drafts (by PWG) 3. Conduct of policy conventions (PWG presents policy drafts to leadership and grassroots) 1.3 Parties have increased capacity to define and utilize research 1. Collaboration with CSOs, thinks tanks and experts (to inform policy drafting) 2. Use of public opinion polls (to inform policy drafting) 3. Use of RUs (to support PWGs and conduct research for the party) 4. RTA (to enhance research capacity of party research units. Training by IRI/IPOR) 1.4 Parties increase internal training capacity 1. TFP (to develop master trainers for the party) 2. Use of Training Fellows (to conduct training for party members at district level) Objective 2: An independent opinion research facility is established that adheres to international research standards. 2.1 IPOR eliminates its dependence on IRI 1. Building IPOR’s technical capacity 2. Building IPOR’s organizational capacity 2.2 IPOR engages political parties and think tanks 1. Technical training for political parties (survey briefings, technical training and consultations with PWG members). Objective 3: Parties communicate more effectively with their members, constituents and the general public at the national and local levels, both in terms of articulating messages and aggregating and responding to concerns, requests and ideas. 3.1 Parties aggregate and respond to local concerns 1. IIWs (to engage grassroots leaders in policy development- identification of policy priorities) 2. Party member surveys (to provide grassroots feedback on policy issues and priorities discussed at IIW- to inform policy development. Conducted in coordination with PWGs.) 3.2 Parties more effectively articulate messages internally and externally 1. Party communications strategy and messaging: a. Briefings for central and provincial leaders on quarterly polls (to improve receptivity to PWG policy drafts) b. Sessions with central and provincial representatives on developing party messages, introducing two-way party communications mechanisms, developing communication plans, effective message delivery. c. Linking party leaders with PWGs. d. Facilitating opportunities for PWGs to report to party leadership. e. Encouraging party leadership to attend program activities and acknowledge efforts of PWG. 15 Objectives and IRs Activities as in the Year 1 Work Plan 2. Party messaging at grassroots – training for district party officials on communication of party policy, messages and general news Objective 4: Parties demonstrate movement toward implementing internationally recognized standards for internal democracy and transparency. 4.1 Party leaders support inclusion of international standards for democratic parties 1. Compliance with Political Parties Order a. Single and multiparty roundtables. 4.2 Party by-laws are reviewed and recommendations made to support elements of the policy development cycle Undefined (planned for succeeding years of the program) PROGRAM IMPLEMENTATION PPDP is led by the NDI in partnership with IRI, IPOR, and SPO. NDI and IRI were established to promote and strengthen democracy throughout the world and have been involved in democracy and governance programming in Pakistan for many years. IRI is a sub-recipient under the program and is responsible for the research and polling components. IRI provides a sub-grant to IPOR, a Pakistani research firm, and works with IPOR to build its capacity to conduct and analyze public opinion research. IPOR assists IRI in conducting polling and research training for the parties. SPO is a Pakistani non￾governmental organization working in the areas of democratic governance, social justice, and peaceful conflict resolution. SPO is a sub-recipient under the program, assisting NDI with grassroots issue identification and supporting PWGs to engage relevant CSOs, think tanks, and experts on policy matters. NDI leads all other parts of the program. The program is implementing a “rolling baseline” approach, where baseline data for each program component is collected as different phases of the program are initiated in each of the provinces. The starting baseline is “0” for most activities. The preliminary baseline collection for the initial participating parties is scheduled to end in October 2013. Target Areas and Groups As of February 2013, PPDP was working with ten parties at the national and provincial levels and expects to add two more during Fiscal Year 2013 (FY13). In instances in which a party is national, PPDP might work with the party in one or more provinces (units). PPDP originally planned to work with 16 different party units. By March 2013, 17 party units from ten parties had participated in at least one routine project-facilitated activity (Table 3), while these ten parties and one more had also participated in election-related multiparty roundtables described below. In the project’s second year (FY13), PPDP aims to increase the number of units among the pool of participating parties and expand the project’s geographic reach. 16 TABLE 3: POLITICAL PARTIES PARTICIPATING IN PPDP31 Party Province 1 Province 2 Province 3 Province 4 Party 3 X X X X Party 7 X X Party 1 X X X Party 4 X Party 10 X Party 9 X Party 5 X Party 8 X Party 2 X Party 6 X X Total 4 4 6 3 Current Status of Activities PPDP has so far engaged most of the political parties in seven phases of the PDC (excluding the first step, which is to get buy in from the party leadership) and two supplementary activities (TFP and multi￾party roundtables). By January 2013, 10 party units of eight parties had been fully engaged in the first seven phases of the PDC, whereas one more party had been partially engaged (Table 4). The TFP had been conducted in two provinces, where it engaged 10 party units from nine parties. 31 This table has been reproduced from the Evaluation Statement of Work, March 9, 2013. Party, province, district and city names have been replaced by numbers in the report in order to maintain confidentiality. 17 TABLE 4: STATUS OF ACTIVITIES BY PARTY32 Party Policy Development Cycle TFP IIW PWG RU Training Member Survey Opinion Research CSO Forum Policy Drafted Province 1 Party 3 X X X X X X X Party 7 X Party 1 X X X X X X X Province 2 Party 3 X X X X X X X X Party 7 X Party 1 X X X X X X X Party 4 X X X X X X X X Party 5 X X X X X X X X Province 4 Party 3 X X X X X X X X Party 7 X X Party 10 X X X X X X X X Party 9 X X X X X X X Party 8 X X X X X X X X Party 6 X X X X X X Province 4 Party 3 X X X X X X X Party 7 X Party 1 X X X X X X X Party 2 X X X X X X Total 14 14 13 13 14 11 10 No. of party members involved 924 140 27 110 110 23 32 Ibid. 18 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The evaluation covers the first 18 months of the project, from July 15, 2011 to February 14, 2013 and is taking place prior to USAID’s decision on whether to extend the project beyond the current two-year funding period. The evaluation results are meant to contribute to this decision and, if USAID decides to continue the project, the results include recommendations about changes that could improve future performance. Finally, the evaluation captures lessons learned that might be relevant to designing and implementing other projects in USAID/Pakistan’s portfolio or the Agency’s portfolio more broadly. The evaluation reflects project activities and information through March 2013, where available. The evaluation objectives are to:  Assess the effectiveness of the overall project approach and of individual activities in contributing to the project’s intermediate results33 and the efficiency with which the partners are implementing the approach and activities; and  Develop recommendations for improving project design and implementation. Audience and Intended Use The primary audience for the evaluation includes: (i) USAID decision-makers and (ii) program leaders and implementers at NDI and its partner organizations. The following indicates USAID/Pakistan’s expectations for other staff participation in the evaluation (in addition to USAID’s Democracy and Governance Team):  USAID staff: Program Office’s Performance Management Unit, especially the Contracting Officer’s Representative (COR) for the MEP contract to monitor evaluation activities and MEP’s role in monitoring the team’s progress, supporting their logistic needs, and implementation of the evaluation.  USAID partners (NDI, IRI, and SPO): The prime implementer NDI and its sub-grantee IRI are key players in providing information about program activities, participants’ details and protocols and sensitivities involved in approaching the political parties. The beneficiaries, i.e. political parties, will also have a key role in providing information for evaluating the program. 33 The focus is on intermediate results rather than objectives because the project’s results framework provides indicators for intermediate results but not for objectives. Most if not all of these indicators are output indicators. 19 EVALUATION QUESTIONS Evaluation Question 1: How effectively are the partners implementing the planned approach and what changes to the approach might improve the efficiency with which the overall set of activities contribute to achieving intended intermediate results? Explanation: This question focuses on the overall project approach and not on individual activities within the approach. The planned approach describes a coordinated set of activities which, if appropriately implemented (e.g., sequence, scale, participants), will contribute to achieving higher level results. This question does not address the relevance of the planned approach per se but only whether the partners are implementing the planned approach effectively. It asks whether the partners are implementing activities in a manner (e.g., sequence, scale) that is consistent with the planned approach and with achieving results in an efficient manner. Efficiency does not mean a rigorous analysis of cost efficiency but, rather, whether implementation is consistent with using resources efficiently to achieve higher level results (i.e., not outputs). For example, the question might address whether the timing and scale of a particular activity is consistent with the timing and scale of subsidiary and dependent activities in the hypothesized development process. The evaluation will require assessing the extent to which specific activities are necessary to achieving intended intermediate results. Evaluation Question 2: How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results? Explanation: This question focuses on how well the partners are implementing individual activities. It does not address the relevance of particular activities to achieving results. Answering the question will involve examining how the partners are implementing each activity, their rationale for the implementation approach (e.g., location/venue, method, choice of participants), and the efficiency of the approach. Efficiency does not mean a rigorous analysis of cost efficiency. Instead, it means whether the partners could implement the activity in a manner that uses resources more efficiently to accomplish the same, or a similar, result. For example, for training activities, the evaluation might examine whether the training engages the appropriate people, uses effective methods, uses an appropriate approach (e.g., direct training or training of trainers), or occurs in the right location. Evaluation Question 3: How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? Explanation: This question draws from the previous questions to craft recommendations for enhancing the project’s overall contribution to achieving intermediate results and the efficiency with which it does so. Recommendations may include modifying (e.g., adding, dropping, scaling, changing targeted participants) individual project activities and aspects of the approach. 20 EVALUATION METHODS AND LIMITATIONS The evaluation questions focused broadly on assessing the effectiveness and general (not financial) efficiency of the overall project approach and activities. Neither of these avenues of inquiry lend themselves well to quantitative analysis nor is much relevant quantitative data available. Therefore, the evaluation relied largely on project records and primary qualitative data collected through focus group discussions (FGDs), group interviews and key informant interviews (KIIs) described below. Baseline data for the project is being collected on a rolling basis as different parties join the project and activities are launched. A formal review of baseline data was beyond the scope of the evaluation. Prior to the evaluation team’s arrival in Islamabad, a team from MEP conducted seven FGDs with party members who had participated in IIWs. MEP selected parties and locations for FGDs on the basis of criteria outlined in the evaluation SOW, including representation from all four provinces, most of the parties and participation in each FGD by men, women and youth and three levels of party leadership (central, provincial and lower). Due to election￾related political activity, however, some FGDs drew less than the 8-12 participants that MEP considered optimal and two of the planned FGDs could not be held. The FGDs were short and focused on the IIW. At the end of each FGD, the MEP team conducted KIIs with two PWG members who attended the FGD and had participated in subsequent phases of the PDC, at least up to the policy drafting stage. During its field work, the evaluation team conducted KIIs with the key staff of PPDP and34: a. Group interviews with two-to-four provincial leaders (from the same party) at a time, focusing on their participation in PPDP activities, utilization of IRI/ IPOR public opinion research and Training Fellows by their party, and the overall PPDP approach; b. Group interviews and KIIs with one or two members each of the RUs of all parties participating in the research component; c. KIIs (one-on-one) with central leaders, focusing on their participation in PPDP activities, the utilization of IRI/ IPOR public opinion research and Training Fellows by their party, and the overall PPDP approach; and d. KIIs with Training Fellows. Most of the field work needed to be completed by March 22, 2013 in view of election￾related activities that began on March 25. This necessitated some reduction in the number of 34 The protocol for inviting party members to FGDs, KIIs and group interviews is laid down in the evaluation SOW. 21 FGDs and interviews planned at the SOW stage of the evaluation. The field activities undertaken are summarized in Table 5. TABLE 5: FIELD ACTIVITIES FOR FOCUS GROUP DISCUSSIONS, GROUP INTERVIEWS AND KEY INFORMANT INTERVIEWS DATA COLLECTION METHOD Total FGDs Group Interviews KIIs Location 1 7 IIW members: Party 1 4 Provincial leaders: Party 1, 2 4 Central leaders: Party 1, 2 27 6 IIW members: Party 2 3 Researchers: Party 1, 2 3 PWG members: Party 1, 2 Location 2 8 IIW members: Party 3 2 Provincial leaders: Party 4 3 Central leaders: Party 3, 4 9 IIW members: Party 4 4 Researchers: Party 3, 4, 5 6 Training Fellows: Party 1, 3, 4, 5 38 6 PWG members: Party 3, 4 Location 3 9 IIW members: Party 3 5 Provincial leaders: Party 1, 3 3 Central leaders: Party 1, 5 33 12 IIW members: Party 1 4 PWG members: Party 1, 3 Location 4 2 Researchers: Party 8, 9 4 Central leaders: Party 1, 6, 7 7 1 Training Fellow: Party 8 Location 5 8 IIW members: Party 8 5 Provincial leaders: Party 8, 9 4 Central leaders: Party 8, 9 1 Researcher: Party 10 28 7 Training Fellows: Party 7, 8, 9, 10 3 PWG members: Party 8 Total Participants: 59 7 FGDs 5 parties 4 locations Total Participants: 26 16 Provincial Leaders (7 parties; 4 locations) 10 Researchers (8 parties; 4 locations) Total Participants: 48 18 Central Leaders (9 parties; 5 locations) 16 PWG Members (5 parties; 4 locations) 14 Training Fellows (8 parties, 3 locations) 133 22 DATA ANALYSIS METHODS The team employed rigorous analytical methods appropriate to the various types of qualitative data collected. For group interviews, KIIs and FGDs, the team identified key themes, coded responses according to these themes, and reported frequencies and qualitative evidence for the responses. METHODOLOGICAL STRENGTHS AND LIMITATIONS The main strength of the methodology was the diversity of data collection sources (by role, party affiliation, and geography) and methods, which enhances the validity of findings. In particular, as illustrated in Table 5: a. Primary data were drawn from five groups of program participants, including three levels of the parties and two groups which were the beneficiaries of training provided by PPDP. b. The sample covered all four provinces and ten political parties. c. Data collection used three different methods of qualitative research. Perhaps the greatest limitation of the evaluation approach was the substantial reliance on data collected from program participants and partners which may lead to a biased view of the project. However, the methodology was specifically designed to mitigate this possibility by capturing a broad range of perspectives. 23 FINDINGS AND CONCLUSIONS FOR EVALUATION QUESTION 2 This chapter reports findings and conclusions for evaluation questions one and two. It presents findings and conclusions for evaluation question two first because the question deals with the individual program activities, which leads naturally into question one, which examines the overall program approach and the extent to which individual activities contribute to achieving the intended intermediate results. Evaluation question three asks solely for recommendations, which are based on the findings and conclusions of evaluation questions one and two. The report thus addresses this question in the “Main Conclusions and Recommendations” chapter. Evaluation Question 2: How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results? The question addresses the extent to which individual project activities are being effectively implemented. Findings and conclusions are organized by activity35 according to the PDC approach as illustrated in Figure 3 (revised PDC). The question addresses all activities including those inside and outside of the PDC. The conclusions are listed first, followed by the detailed findings for each activity. LEADERSHIP COMMITMENT TO THE POLICY DEVELOPMENT CYCLE Activity 1: Leadership commitment to the Policy Development Cycle (PDC) (IR 1.1- leadership participates in the PDC) Conclusion Party leaders have participated (a proxy for commitment) in PPDP at different levels. A majority (64 percent) were aware of their party’s involvement with PPDP. Fewer than half of all respondents (41 percent) said that leaders were directly involved in PPDP activities and 30 percent of all respondents said that leaders were proactively adopting PPDP innovations. Central leaders were less aware than provincial leader groups of the party’s involvement in PPDP. When it occurred, party members found the participation of party leaders in PPDP activities motivating. Findings 1. Sixteen of the 25 (64 percent) party leaders (central leaders and provincial leader groups)36 interviewed in KIIs and group interviews, respectively) were generally aware of the PPDP and at least some of the activities that the party was participating in. The 36 percent who were not familiar with PPDP were not aware of the program purpose, had not attended events, or could not articulate which activities were more or less beneficial to the party. All of the respondents 35 The term “activity” is used here to refer to “phases” as defined in the PDC and individual activities outside of the PDC. 24 who were not familiar with the party’s involvement were central leaders, equivalent to 50 percent of all central leaders interviewed. 2. Twenty-eight of 70 (40 percent) of all respondents reported that party leaders were directly involved in PPDP (e.g. selection of participants, participation in activities.) IIW participants, PWG members and Training Fellows reported that the presence of central leadership in activities was motivating. Respondents had mixed views about the selection of participants for various activities. All seven focus groups for the IIWs reported that the selection process was fair and transparent, while some party leaders, PWG members and researchers, as well as PPDP staff, identified challenges with the selection of researchers and PWGs, in particular. A central leader from Quetta commented that, “Party leadership invites nominations from people of their preference. The selection process[es] should be more transparent and competitive, not nepotistic.” 3. Fourteen of 70 (20 percent) of all respondents including 44 percent of leaders and 29 percent of RUs reported that party leaders regularly informed themselves of PPDP activities. A provincial leader from Quetta commented that the party received DVDs of event proceedings from NDI and observed which party members were actively participating and made a determination about their participation in future events. The leader reported that the party replaced one Training Fellow who was not effective. 4. Twenty of 70 (29 percent) of all respondents reported that leaders were proactively applying innovations from PPDP activities. Ten percent reported that leaders had not yet applied innovations, but intend to. Leaders from five parties in four provinces provided examples of adopting PPDP innovations such as setting up think tanks, creating a permanent training office, use of social media, creating a membership database, use of master trainers to train party members, dissemination of policies to district level, and use of public opinion data to inform policy making. A provincial leader from Peshawar reported that the party had established training units in two districts and plans to strengthen and expand the units to the entire province following the 2013 elections. A provincial leader from Lahore said that PPDP had contributed to an awakening within the party and that the party will continue to invite party workers to brainstorm on policy using indigenous sources and conducting sessions in houses or in the streets. He noted however, that the party would not be able to continue to engage party workers at the same level as PPDP without external financing. A provincial leader from Quetta suggested that NDI could arrange a session for parties to present what they have learned through PPDP. An IIW participant from Karachi reported that the provincial leadership had decided to continue to include grassroots workers in the policy development process and that the party had established a study circle where IIW and PWG members will share their learning. Figure 4 below summarizes participant responses related to leadership participation in PPDP, differentiating between the responses of party leaders and other respondents. 25 FIGURE 4: VIEWS OF PARTY LEADERSHIP INVOLVEMENT IN PPDP ISSUE IDENTIFICATION WORKSHOPS Activity 2: Issue Identification Workshops (IIW) (IR 1.3 Parties aggregate and respond to local concerns) Conclusion The IIWs have been effective forums for engaging a broad range of grassroots party members and party leaders, including women and youth, in the process of identifying issues, including those related to the MDGs. All (100 percent) IIW participants reported that the workshops gave them a voice in policy development and informed the policy-making process, while a minority (20 percent) of party leaders agreed. IIWs were carried out in an inclusive and democratic manner and contributed to the aggregation of local concerns. It could have been useful to review existing research and data on policy matters for the country and provinces to inform the IIW discussions and output. Many grassroots participants may not have been well prepared to engage in the dialogue; participants suggested more advance notice of the IIW agenda and a longer event to fully absorb the content. Findings 1. NDI and SPO engaged grassroots and party leaders in identifying three to five policy priorities in each IIW. Participants were encouraged to develop consensus on a limited number of priorities in order to keep focused.37 37 Conversation with NDI staff member, April 15, 2013. 60% 72% 44% 68% 20% 22% 7% 7% 0% 10% 20% 30% 40% 50% 60% 70% 80% Familiar with party's involvement Directly involved in activities Keeping informed of activities Applying PPDP innovations Party Leaders (n=25) Other Respondents (n=45) 26 2. All (100 percent) IIW focus groups reported that the IIW selection process was clear and fair. All reported that the party included additional selection criteria to ensure broad district-level representation in the event. 3. All (100 percent) IIW focus groups reported that women and youth actively participated in the IIWs. A provincial leader form Peshawar noted that the party joined PPDP late and did not participate in the IIW due to the lack of mahram expenses,38 but that NDI has since addressed the issue. Two provincial leaders from different parties and provinces commented that they would like more women to participate. The leader from Peshawar mentioned the need for communications and polling agent training for women. A female IIW participant from Lahore stated that women used to have a limited and traditional role in the party, but are now actively participating and have learned to engage at all levels. A senior male IIW participant from Quetta noted that women raised most of the issues at IIWs, even though women’s participation was limited due to challenges associated with traveling from rural areas. 4. All (100 percent) IIW focus groups reported that leaders were aware of the purpose of the IIWs before the workshop. Only one in seven focus groups reported that grassroots participants were aware of the purpose of the workshop beforehand. The invitation letter to IIW participants that the evaluation team has seen is in English which many participants do not understand. An IIW participant from Quetta commented that a number of women from rural areas were confused about the purpose of the IIW. 5. All (100 percent) IIW focus groups and five of 25 (20 percent) party leaders interviewed separately reported that the IIW gave grassroots workers a voice in the policy development process. All focus groups of IIW participants reported that the policy priorities they had identified for their parties included subjects (such as education, health, poverty and women’s concerns) that are addressed by the MDGs. An IIW participant from Karachi noted the differences in the issues raised by participants from rural and urban parts of the province, but said they all agreed that education was a priority. A provincial leader from Karachi said the program is building the capacity of individual party workers, not just the party as an institution. One provincial leader from Quetta felt that there were too many grassroots participants and said the party had to send an uneducated participant in order to meet the PPDP participant selection requirements. 6. All (100 percent) IIW focus groups and five of 25 (20 percent) party leaders interviewed separately reported that IIWs informed the policy development process. A male IIW participant in Karachi noted that party workers have never been involved in the policy-making process and said this change is just the beginning of a major shift in increasing the influence of grassroots opinions on policy. A central leader in Karachi noted that diverse participants (urban, rural, women, youth) contributed to the discussion, developed a clear understanding of the issues, and will be able to use the information to mobilize people for the 2013 elections. 7. The IIW documents39 and discussions with NDI40 indicated that IIW mobilized the grassroots to identify key policy priorities through a structured brainstorming and voting exercise. Party 38 Provisions for male relative to accompany female participant. NDI’s internal memo of August 2012 addresses mahram expenses for IIWs and PWGs. 39 NDI IIW activity reports. 27 leaders participated in the IIW; at one workshop, a senior party official informed participants of major issues outlined in the party manifesto to inform the discussion. The IIWs did not involve reviewing existing policy research or data on key national and provincial concerns (e.g. the MDGs) to inform issue identification. 8. PPDP’s assessment of SPO at seven IIWs conducted since March 2012 judged performance as positive in four cases, mixed in two cases and poor in one.41 In the three cases of mixed and poor performance, one involved SPO providing only one facilitator instead of the two planned. In one case, SPO misunderstood the NDI policy for travel reimbursement. In the other example, the SPO facilitators arrived late. The assessment identified no problems with the quality of SPO’s facilitation at the workshop. 9. All IIW focus groups and nine of 16 (56 percent) PWG respondents recommended improvements in the organization and facilitation of IIW. In particular, they suggested: a. All IIW focus groups and three of 16 (19 percent) PWG respondents suggested that IIW be a two-day rather than one-day event to allow participants time to absorb the information. b. Four of seven IIW focus groups and two of seven (29 percent) PWG respondents said the agenda should be shared with participants in advance of the workshop. c. Two of seven IIW focus groups and one (6 percent) PWG respondent suggested that the number of participants in IIW should be increased to more than 75. FORMATION OF POLICY WORKING GROUP Activity 3: Formation of Policy Working Group (PWG) (IR 1.2 Parties increase policy development capacity at the party and legislative levels) Conclusion Utilizing party members to draft policy is an efficient approach in terms of developing in-house capacity. However, the process for selecting PWG members has often resulted in PWGs that do not have the appropriate combination of experience, skills, availability and stature to effectively prepare and present policy options to the party leadership. Not all PWG members knew why they were selected and only a small number (six percent) reported that they were selected on the basis of systematic or relevant criteria. A small minority of party leaders and PWG members (4 percent and 6 percent, respectively) reported regular interaction with each other. This seems to reflect both the entrenched party structures and approaches to policy development, and the limitation of the current PWG composition to facilitate access to central leadership. PPDP was well aware of the challenges with PWG composition and was actively considering changes in the selection criteria, number of PWG members and focus of PWGs, as well as ways to enhance PWG capacity. 40 Meetings with NDI, March- April 2013. 41 According to activity reports NDI introduced written assessments of SPO performance in IIW beginning in March 2012. 28 Findings 1. Ten of 16 (62 percent) PWG members reported that they were selected by party leaders while five of 16 (31 percent) reported that they were selected by IIW participants42. Not all respondents knew the reasons they were selected. They cited their active role in the party, status in the party, education level and communications skills, access to party leaders, experience with policy research and surveys,43 and the need for balance within the PWGs with respect to sex, age, and regional representation. Figure 5 below shows the evaluation findings regarding the selection process for PWG members according to the members. FIGURE 5: METHOD OF PWG SELECTION 2. Two of 16 (13 percent) PWG members reported that they were selected using systematic/relevant criteria. 3. NDI reported that in some cases the person selecting the PWG members may not have been fully cognizant of the purpose of the PWG.44 NDI noted that the program has faced challenges with how parties selected PWG members and is discussing how to improve the selection criteria so that PWG members and party leadership better understand the various components of PPDP.45 NDI also commented that there have been issues with representation at PWG meetings because members have other commitments. 46 NDI is exploring the option of increasing the number of PWG members and possibly having one PWG per party to focus on national-level policies, rather than have a PWG for each party unit.47 NDI is also exploring the option of conducting more than the two planned PWG meetings in order to improve the capacity of PWGs.48 4. One of 25 leaders (4 percent) and one of 16 PWG members (6 percent) reported regular interaction with each other. The two respondents were from the same party and province. The 42 One PWG respondent did not know how the PWG was formed. 43 Three respondents- two selected by party leadership and one selected/ endorsed by IIW participants mentioned these skills. 44 Email from NDI staff member, April 10, 2013. 45 Email from NDI staff member, April 11, 2013. 46 Meeting with NDI staff, April 11, 2013. 47 Ibid. 48 Ibid. 31% 62% 7% By party leader By IIW participant Respondent not sure 29 PWG member felt that the party leadership had fully supported PWG members and that the policy draft will be reflected in the manifesto. NDI staff noted that provincial, divisional, and district-level party members who make up the vast majority of PWGs (76 percent according to the original composition of PWGs49) are very reluctant to contact central leadership to present policy drafts or discuss policy matters.50 To bridge the gap between PWGs and party leadership, PPDP has encouraged PWGs to share policy drafts with the PPDP provincial liaison (typically the provincial leadership), who have more direct access to central leadership, while PPDP directly shares drafts with central leadership.51 PARTY MEMBER SURVEY Activity 4: Party Member Surveys (IR 3.1 Parties aggregate and respond to local concerns) Conclusion The party member surveys served as a party building exercise and contributed somewhat to the aggregation of local concerns. The surveys employed some of the techniques used in formal large-sample surveys but did not use standard research methods, and, therefore the results are not statistically reliable for informing policy development. While the findings suggest that the surveys did inform policy development in some instances, they also reveal weaknesses in the survey approach that severely limit their utility. An adherence to survey design elements (e.g., questionnaire design, sampling, data quality control, and analysis) would have increased the potential for relevant and quality data to inform policy development, and would have been an opportunity to demonstrate the rigors of research-driven policy development to program participants. Findings 1. PWG members from seven party units in four provinces reported that the IIW participants from their parties administered formal questionnaires to 1,000 to 1,500 (in one case, 2,250) respondents using convenience sampling, while ensuring a high degree of representation (30 to 40 percent of the sample) of women party members; each IIW member administered 15-30 questionnaires to party members. The total return rate on party member surveys was 74 percent, with a low of 31 percent and high of 92 percent.52 2. Three out of 16 (18 percent) PWG members reported that member surveys covered most of the geographic reach of the party. PWG members from three parties in two provinces reported that the surveys were conducted province-wide. The evaluation team could not verify this on the basis of available data. A PWG 253 activity report mentions, “The group admitted that the sample of survey was not so representative. For some reason, party liaisons failed to nominate participants from the under-developed areas. As a result survey could not be conducted in 49 These figures are based on the composition of PWGs at the first of two meetings for each PWG. There have been changes to PWG members over the course of the program. 50 Ibid. 51 Ibid. 52 NDI PPDP PowerPoint presentation, March 8, 2013. 53 NDI PWG 2 activity report, October 10-12, 2012. 30 those areas. Next time, the program staff needs to double check whether or not given criteria has been followed while nominating the participants of IIW.” 3. Eleven out of 16 (68 percent) PWG members reported that NDI helped with the design of the questionnaire. The PPDP quarterly reports confirm that NDI assisted the PWGs with the questionnaires. NDI informed the evaluation team that the institute initially prepared issue￾specific surveys based on the results of the IIWs and presented them as a draft for discussion at PWG 1, along with other research (e.g. policy briefs on the relevant issues, government statistics, World Bank statistics, etc.54). For the two most recent IIWs, NDI noted that it has adjusted its approach by first introducing a sample survey at the IIW that PWG members can use to develop the survey questionnaire with NDI assistance.55 4. Five out of 16 (31 percent) PWG members reported that NDI did data analysis. The PPDP quarterly reports and PWG 2 activity reports confirm that NDI assisted the PWGs with data analysis. NDI informed the evaluation team that the RUs were meant to assist with data analysis but were unable to because they were not yet up to speed at the time the surveys were conducted.56 The majority of PWGs (9 of 13, 70 percent) analyzed the party member survey results from July to November 2012, at which stage RTAs had completed two training modules,57 including a tutorial on statistical research and training on FGDs.58 5. One out of 16 (6 percent) reported that the party got help from other experts for data analysis. This example is from a PWG member in Karachi who said that the Statistics Department at the University of Karachi assisted with data analysis. 6. Only three of 16 (18 percent) PWG members, one of 18 central leaders (6 percent) and two of seven (29 percent) provincial leader groups reported that party member surveys effectively captured and aggregated local concerns. Three leaders from different parties and provinces noted that the survey reports were useful for understanding peoples’ views at the grassroots level. A provincial leader from Lahore said the party felt the changes when participating in the IIW and the membership surveys and, “when we get information from our own people, our workers, [we're] getting authentic data.” 7. Seven of 16 (44 percent) PWG members, three of 18 (17 percent) central leaders and two of seven (29 percent) provincial leader groups reported that party member surveys informed policy development. PWG members from four parties in three provinces said that the PWG used the survey data to identify issues for inclusion in the policy drafts. A PWG member in Karachi said that on the basis of the survey results the PWG decided not to abolish the quota system for providing government jobs to the rural population in the province. A provincial leader from Lahore reported that the party “learned the problems of the people then formed policies.” The PPDP proposal and quarterly reports describe the party member surveys as 54 The evaluation team found evidence of these materials being provided at PWG 2, not PWG 1. See NDI PPDP quarterly report October- December 2012 and party-wise activity reports for PWG 2. 55 Meeting with PPDP staff member, March 15, 2013. 56 Ibid. 57 Six RTA modules are planned for the first and second years of the program. See NDI PPDP Year 2 Work Plan, p. 12. 58 See PPDP Activity Tracking Sheet February 12, 2013 and RTA sections of NDI PPDP quarterly reports October-December 2011 and April- June 2012. 31 providing information to the PWGs to be considered in the development of policy proposals.59 Similarly, the PPDP work plans60 describe the surveys as an activity to inform policy development. PPDP staff strongly believed, however, that the parties were using the surveys as a party mobilization/outreach exercise rather than as an instrument to reliably inform policy development.61 The evaluation team reviewed 28 policy drafts62 prepared by ten party units from July 2012- February 2013. Fourteen percent of the drafts refer clearly to results of the party member surveys. 8. IIW participants who conducted the party member surveys were not trained as enumerators and PWG members who conducted data analysis were not trained as data analysts. The IIW agendas show that one to one and a half hours were dedicated to discussing the party member surveys during the workshops.63 The IIW activity reports and exchanges64 with PPDP staff confirmed that the surveys were not ultimately designed for statistical reliability. The language in the program proposal, quarterly reports and work plans creates some confusion on this point since these documents describe the surveys as an activity to inform policy development. Likewise, as described above, some party units used the results of the party member surveys to inform the policy drafts. A PWG activity report65 noted that, “There should have been a pre￾test of the survey. The group indicated that some of the questions were vague. The questions should have been made easier for the respondents. It was also suggested that an orientation session should have been organized for the surveyors, many of whom forget the tips on survey by the time they receive the questionnaire.” A PWG member from Karachi said that more guidance on survey methodology from NDI would be helpful. FORMATION OF RESEARCH UNITS, RTA CURRICULUM Activity 5: Formation of Research Units, RTA Curriculum (IR 1.3 Parties aggregate and respond to local concerns) Conclusion The addition of Researchers has increased the capacity of parties to define and utilize research. The majority of RUs could recall the RTA subject matter and have applied their research skills in the party (75 percent and 63 percent, respectively). A sizable majority of RUs (63 percent) also had a more favorable impression of public opinion research after participating in training and half were interested in receiving additional training on research and non-research-related topics. However, the low attendance rate in RTA modules (43 percent for all modules), identification of a number of RU members who do not meet the criteria set by PPDP, and the disconnect between RUs and PWGs indicate that the research component is not as effective as it could be. 59 NDI PPDP Technical Proposal, May 2011, p. 15; NDI PPDP quarterly reports, October 2011- December 2012. 60 NDI PPDP Year 1 and Year 2 work plans, IR 3.1. 61 Meetings and email correspondence with NDI staff, March-April 2013. 62 NDI shared 28 policy drafts with the evaluation team. The NDI Policy Development Tracking Memo compares 18 policy drafts against language in the 2013 party manifestos. 63 All IIW agendas. 64 Meetings and email correspondence with NDI staff, March-April 2013. 65 NDI PWG 2 activity report, November 8-9, 2012. 32 Findings 1. Based on their recall of the topics covered in the RTA, six of eight (75 percent) RUs representing three provinces demonstrated awareness of the RTA subject matter. 2. Five of eight (63 percent) RUs from five parties in two provinces reported that they have applied research skills in their parties. Examples included: conducting FGDs66 to identify party strengths at the constituency level and identify qualities that people are looking for in their elected leaders; incorporating the results of party surveys67 into the party manifesto; utilizing public opinion research to inform party leadership about policy changes that are needed; reaching out to the public through different forums to gather opinions on a legislative bill;68 suggesting ways that the party could make existing RUs more effective; and conducting informal training sessions with party workers to share research skills. A central leader from Lahore noted that research was the first step towards creating “a buzz” in the party. He said there are a lot of “firsts” because of PPDP and that research [has given] the party a greater understanding of the issues and instilled the PWG with a research sensibility. 3. Three of eight (38 percent) RUs reported that PWGs used data from the RUs. None of the PWGs reported using data from the RUs. 4. Five of eight (63 percent) RUs had a more favorable impression of public opinion research after participating in research training. These respondents represented five parties in three provinces. Several of the groups noted that public opinion research was a new concept for them. A researcher in Quetta said that that initially 70 percent of the participants did not understand the training modules, but now 100 percent do understand. He noted that some of the “non-serious” participants were no longer participating in the RTA. He said, “When we learned the methodology and importance of surveys, we now believe that a sample size of 3,000 is sufficient to gauge public opinion” and “On an individual level, I am connected to modern scientific thinking and [a] scientific way of politics. It is important for the party to have [a] hand on the pulse of the public. If you don’t understand the mood, you can’t develop good policies. These surveys are the best way to do this. If you know these things, you can develop relevant policies. At a technical level, we now understand [the] importance of surveys.” 5. Just two of eight (25 percent) RUs (from different parties and provinces) believed that the party followed a competitive process to select RTA participants. 6. Based on attendance data from three of the four training modules conducted to date, 12 of 28 (43 percent) RU members attended all three RTAs.69 The rest of the RTA members (57 percent) did not attend all modules (Figure 6). One Researcher from Peshawar could not recall attending any of the RTAs. 66 A respondent from Quetta noted that FGDs are less expensive than surveys. 67 A researcher reported that this party is conducting monthly public opinion surveys online. 68 A researcher in Sindh provided an example of a controversial bill that was removed as a result of public opinion and inter￾party efforts against the bill. 69 Data are available for three of the four RTA modules held to date. 33 FIGURE 6: TRAINING MODULES ATTENDED BY RESEARCH UNIT MEMBERS 7. The PPDP has identified a number of RU members who do not meet the criteria laid out by PPDP, such as the interest, commitment and capacity to conduct opinion and issue-based research.70 In addition, the program has recognized financial constraints on the research activity created by the addition of new parties to PPDP, and potential limitations on the effectiveness of the activity due to its current focus on building provincial, rather than central-level, capacity. PPDP has identified options for re-configuring the RUs once the first cycle of the RTA has been completed, including focusing efforts on building research capacity at the central rather than provincial level.71 Ninety-three percent of researchers are mid and high ranking officials (61 percent and 32 percent, respectively).72 It is understood that senior party members will not have the time to dedicate to research73. Likewise, they are unlikely to take on research requests from PWG members, who are largely (76 percent) mid-ranking and lower-level party members74 (39 percent and 37 percent, respectively). Figure 7 below shows the composition of PWGs and RUs according to the rank of members. 70 The October 2012 IRI memo on reconfiguring Research Units refers to the selection criteria set out in the NDI PPDP Technical Proposal of May 2011. These criteria are included in p.8 of the proposal. 71 IRI memo on reconfiguring Research Units, October 23, 2012. 72 Data from RTA participant lists and pre-PWG survey data of February-June 2012. 73 In this finding, senior party members include members of the National Assembly, national-level secretaries and president of the women’s wing, members of the federal cabinet, and members of a party’s central executive committee, 74 Mid-ranking party members are those who hold a provincial-level office in the party or the provincial assemblies, while district- and local-level members are referred to here as lower-level members. 18% 39% 43% 1 module 2 modules 3 modules Percentage of Researchers 34 FIGURE 7: PWG AND RU COMPOSITION BY MEMBER RANK75 8. Four of 8 (50 percent) RUs, representing multiple parties and provinces, were interested in receiving additional training on the topics already covered during the RTA. 9. Four of 8 (50 percent) RUs, representing four parties in three provinces, said they would be interested in receiving training on other topics. Additional topics included media campaigns, current affairs, politics, youth-focused policy development, conflict resolution, membership recruitment strategies, election campaigns, and establishing think tanks research units. 10. Five of eight (63 percent) RUs representing five parties in three provinces offered recommendations for improving the organization and logistics of the RTAs. The recommendations included more regular contact and guidance from PPDP outside of the RTA modules, better screening of researchers, advance notice of trainings, more local trainings, and better training venues. A majority of participants in each group supported these recommendations. PUBLIC OPINION RESEARCH Activity 6: Public Opinion Research (IR 1.3- Parties have increased capacity to define and utilize research) Conclusion While there has been progress, parties lack capacity to understand and effectively utilize public opinion research. A sizable number (40 percent) of party leaders did not have confidence in the IRI/IPOR 75 There have been changes to PWG members over the course of the program. This figure shows the composition of PWGs at the first of two meetings for each PWG. 24% 39% 37% 32% 61% 7% High-ranking Mid-ranking Low-ranking PWG Members (n=16) RUs (n=28) 35 methodology. Just a small number (12 percent) of leaders used IRI/IPOR public opinion research to inform party activities and none of the PWGs, including those who have been briefed by IRI/ IPOR, have used this opinion research to inform policy drafting. Findings 1. Ten of 25 (40 percent) party leaders76 reported that they do not have confidence in the IRI/IPOR polling methodology. This finding reflects responses from six of 18 (33 percent) central leaders from three parties in three provinces, and four of seven (57 percent) provincial leader groups from four parties in three provinces. Only one central leader and one provincial leader group were from the same party unit. A central leader from Peshawar said, “We believe in our performance and votes of people, not opinion polls. They are helpful for providing information, but we don’t strictly believe in them.” A provincial leader from Lahore said that the party doesn’t necessarily disagree with the polls, but that the party should be informed of the methodology. 2. Three of 25 (12 percent) party leaders, two central leaders from the same party unit and one provincial leader from a different party and province, reported that they utilize IRI/IPOR public opinion research to inform party activities. A central leader in Peshawar reported that a party worker successfully targeted undecided voters after learning from the IRI/IPOR polls of the high percentage of undecided voters. A provincial leader in Lahore commented that the party works harder to address the issues if the opinion research shows a decline in party strength. 3. None of the PWGs reported using IRI/ IPOR research in policy drafting. The evaluation met with three of six (50 percent) PWGs who have been briefed on the IRI/IPOR polls. While none of them reported using IRI/IPOR research in policy drafting, the evaluation team identified that a possible explanation for this is that they had completed or nearly completed their drafts before they were briefed on the polls.77 4. Two of 16 (12 percent) PWGs reported using external (non PPDP-sourced) information for policy drafting. A PWG member from Lahore reported that the party contracted IPOR outside of PPDP to conduct a survey and used the results for policy drafting. A PWG member from Karachi said they used survey results from other sources. COLLABORATION WITH CSOS, THINK TANKS AND EXPERTS Activity 7: Collaboration with CSOs, Think Tanks and Experts (IR 1.3 Parties have increased capacity to define and utilize research) Conclusion The success of the CSO forums varied; on the whole CSO forums have made a modest contribution to the parties’ capacity to utilize research. The majority of PWG members (68 percent) reported that party priorities were discussed at CSO forums. Some PWG members (32 percent) believed that the 76 These leaders included those who were briefed directly by PPDP on the opinion polls as well as others who were not briefed directly, but did not include the party heads and key leaders who were directly briefed but were not available for interviews because of their election commitments. 77 The poll briefings were held in November and December 2012 while these party units completed their policy drafts between November 2012 and January 2013. See PPDP Activity Tracking Sheet February 12, 2013 and policy drafts. 36 CSO forum contributed to policy formation; while more (44 percent) reported that it did not. Just a small number of PWG members (18 percent) used information from CSOs, think tanks and other experts for policy drafting. PWG members and party leaders identified problems with the quality of speakers and facilitation for some forums. Findings 1. Three of 16 (18 percent) PWG members from two parties in Karachi reported that they used information from CSOs, think tanks or other experts for policy drafting. The examples of information they used for policy drafting included website references recommended by NDI and discussions with intellectuals, bureaucrats, and professionals working in the relevant policy area. As noted under Activity 4 above, the evaluation team reviewed 28 policy drafts prepared by 10 party units from July 2012-February 2013. Twenty-nine percent of the drafts clearly incorporated secondary data such as MDGs, World Bank statistics and government statistics. 2. Eleven of 16 (68 percent) PWG members, representing six parties in all four provinces, reported that party priorities were discussed at CSO forums. 3. Five of 16 (32 percent) PWG members reported that the CSO forum contributed to policy formation. These respondents represented four parties in four provinces. Participants from Karachi offered many comments about the forum. Two participants from different parties commented that the forum introduced new perspectives and solutions to policy issues. Another participant commented that the forum should have taken place earlier in the PDC because the PWG had nearly completed its policy draft by the time the forum took place. 4. Seven of 16 (44 percent) PWG members reported that the CSO forum did not contribute to policy formation. The seven PWG respondents represented four parties in three provinces. 5. Participants from Peshawar commented that key CSO speakers’ level of understanding of issues was lower than that of participants and the speakers were unable to provide solutions. The CSO activity report reflects the problems with this event and strongly criticizes SPO for failing to properly brief the speakers and understand the purpose of the forum.78 Two participants from Karachi said the forum was of no help and a participant from Lahore noted that the policy drafts did not include any ideas from the forum. Leaders79 also commented on the CSO forums. Two central leaders from different parties in Peshawar and Islamabad reported that the speakers were not qualified. As noted above, the CSO activity report confirms problems with this forum. The leader from Islamabad said the party had its own policy experts and that the and that the forum was not up to their level, such that “some of the participants stood up and said, ‘we are wasting our time.’” One provincial leader from Lahore said SPO was not properly utilized and involved in the workshop; another provincial leader from a different party in Lahore said that NGOs are just interested in getting funding. The CSO activity report for this forum does not capture these critiques.80 78 NDI CSO forum activity report for Khyber Pakhtunkhwa, September 18, 2012. 79 Central and provincial leaders were secondary respondents for this issue. 80 NDI CSO forum activity report for Punjab, September 20, 2012. The Punjab forum included participants from Islamabad and Lahore. 37 POLICY DRAFTING Activity 8: Policy Drafting (IR 1.2 Parties increase policy development capacity at the party and legislative levels) Conclusion PWGs have taken the initiative to draft party policies, signaling the start of a more decentralized approach to policy development. Nearly all PWG members (87 percent) reported that the PWG prepared policy drafts and a few members reported that they consulted party leadership or policy experts ahead of drafting. Half of PWG members (50 percent) reported that NDI provided assistance with typing the drafts on the basis of content provided by the PWGs, as well as supporting discussion of the results of party member surveys and in providing policy drafts and other reference materials for consideration. Findings 1. Fourteen of 16 (87 percent) PWG members from all seven PWGs interviewed in four provinces reported that party policy drafts were prepared. 2. Fourteen of 16 (87 percent) PWG members reported that policy drafting was done by PWG members. Respondents provided few details about their approach. Members from one party in Karachi said they divided the drafting amongst PWG members. A member from a different party in Karachi indicated that one PWG member took the lead in drafting. Two PWG members from one party in different provinces said they consulted with party leadership; two others from another party said they consulted policy experts ahead of drafting. 3. Eight of 16 (50 percent) PWG members reported that policy drafting was done with assistance from NDI. Eight respondents from five parties in four provinces mentioned NDI assistance, including typing the drafts based on the language provided by the PWG members, discussing the results of the party member surveys, and providing sample policy drafts for reference. The program reports indicate that NDI also presented information on standards for policy drafting as well as policy briefs on priorities identified at IIWs.81 NDI made clear to the evaluation team that all of the language in the drafts came from PWGs themselves, not NDI.82 LEADERSHIP REVIEW OF POLICY DRAFTS Activity 9: Leadership Review of Policy Drafts (IR 1.2 Parties increase policy development capacity at the party and legislative levels) Conclusion While difficult to verify at this stage in the policy development process, it appears that party leadership has utilized PWGs for policy drafting and to a limited extent for policy revision. This is a change from 81 See e.g. NDI PPDP quarterly reports July- September and October- December 2012, and PWG 2 activity reports from July 11 and November 8-9, 2012. The policy briefs included data from the Pakistan Social and Living Standards Measurement Survey (PSLM) 2011-12, the Human Development Report (HDR) 2011-12, government statistics and other sources. 82 Debriefing with PPDP staff and USAID, April 24, 2013. A PWG 2 activity report from July 11, 2012 notes that some PWG members were not computer literate. 38 the standard top-down approach to policy development. A majority of PWG members (56 percent) reported that party leadership reviewed the drafts presented to them and a small number (18 percent) said that the leadership requested revisions to the drafts. A sizable number of PWGs (43 percent) believed that party leadership incorporated the drafts into the party manifesto. Findings The evaluation team was not able to verify the following set of findings through other sources. Based on available data, all party units from one party have completed their policy drafts.83 The Policy Development Index (PDI)84 developed by NDI scores PWG progress and the quality of policy drafts after the drafts have been completed and sent to leadership for review. The PDI considers whether the PWG shares the draft with the party leadership, but does not examine the policy review or revision processes in detail or whether the policy drafts are incorporated into the party manifesto. The evaluation team interviewed 16 PWG members and examined their responses regarding policy review, requested revisions and incorporation of policy drafts into the party manifestos by party leaders. The total number of clear responses regarding leadership review of the policy drafts were 12 (out of which seven confirmed that party leadership had reviewed policy drafts), for revisions to the policy drafts it was six (half of them reporting revisions to drafts requested by party leaders), and for the incorporation of the policy drafts into the party manifesto it was eight (including seven observing that party leadership incorporated the drafts in party manifestos). FIGURE 8: REVIEW, REVISION, AND INCORPORATION OF POLICY DRAFTS 83 This data is accurate as of December 31, 2012 as reported in the NDI PPDP quarterly report October- December 2012. 84 The PDI is designed to assess party progress around the PDC, the degree of policy development capacity within the party and in the legislature, and the quality of party effort and achievement. The PDI considers five main criteria to determine overall achievement on the Index: issue identification and prioritization, use of public opinion research, the policy drafting process, internal and external communication of policies, and the composition of the PWGs. See Policy Development Index Guidelines v6, December 21, 2012 and Policy Development Index worksheet v3, January 23, 2013. 43% 18% 56% 6% 18% 18% 0% 10% 20% 30% 40% 50% 60% 70% 80% Party leadership incorporated draft into party manifesto Party leadership requested revisions to policy draft Party leadership reviewed policy draft Percentage of Respondents Yes No 39 POLICY CONFERENCES Activity 10: Policy Conferences (IR 1.2 Parties increase policy development capacity at the party and legislative levels) Conclusion While only one policy conference was completed during the evaluation period, there was solid evidence that the party manifestos reflect the work of PWGs. A solid majority of PWG members (68 percent) believed that the policy draft was or would be reflected in the party manifesto and the evaluation team identified a strong correlation between the language in a selection of policy drafts and the party manifestos in 27 percent of cases. The findings suggest that the PWGs are developing capacity to contribute to policy development. This signals a change to the standard top-down approach to policy development. Findings 1. Three of 16 (18 percent) PWG members from the one party that had held a policy conference by the time of the KII with the evaluation team reported that the draft was approved at the policy conference. The activity report85 indicated that four policy drafts were presented at the conference, that revisions were made on the spot, and that the party leadership planned to incorporate between one and four of the drafts into the party manifesto. Two policy conferences have now been held with PPDP assistance.86 The activity report on the second policy conference87 showed that the three policy drafts presented were approved at the conference with some amendments, and that participants expected up to two of the drafts to be reflected in the party manifesto. 2. Eleven of 16 (68 percent) PWG members from seven party units in four provinces reported that the policy draft is/ would be reflected in the manifesto. 3. Three of 16 (18 percent) PWG members from the same party reported that the policy draft was not reflected in manifesto. As highlighted by NDI, “some PWGs may not have concluded policy drafting or have obtained party leadership buy-in with sufficient time to be considered in manifesto development.”88 4. Based on a memo89 provided by PPDP, the evaluation team compared 18 policy drafts to language in the 2013 party manifestos and identified five examples of a strong correlation90 between the language in the draft and the party manifesto (27 percent), 10 examples of a 85 NDI activity report for policy conference held January 30, 2013. 86 These conferences were held in January and March 2013. 87 NDI activity report for policy conference held March 23, 2013. 88 Email from NDI staff member, April 11, 2013. 89 NDI Policy Development Tracking Memo, April 9, 2013. 90 The evaluation team “scored” the analysis findings as: “strong correlation” = the true spirit and sometimes wording of the policy recommendations are incorporated into the respective manifesto; “moderate correlation” = only partial policy recommendations incorporated into the respective manifesto or the subjects are mentioned in the policy draft and manifesto but content/ recommended solutions are dissimilar; “no correlation” = no apparent correlation between the policies offered by the PWG and the respective party manifesto. 40 moderate correlation (56 percent), and three examples of no discernible correlation between the language in the drafts and the party manifesto (17 percent). For the first policy conference held, two of the policy drafts are strongly reflected in the manifesto; for the second policy conference held two of the policy drafts are moderately reflected in the manifesto. COMMUNICATION TRAINING Activity 11: Communication Training (central and district level) (IR 3.2 Parties more effectively articulate messages internally and externally) Conclusion PPDP has effectively assisted parties to improve internal and external communications ahead of elections through the addition of extra trainings. While reporting data was not yet available for these particular training events, a considerable number of party leaders reported PPDP activities helped the party to articulate messages internally and externally (28 percent and 32 percent, respectively). The planned communications trainings to assist parties with the dissemination of party policies and messaging are scheduled for after the policy conferences, following May 2013 elections and were therefore not part of the evaluation. Findings 1. The communications trainings held to date have been added to support the parties ahead of elections. Program reporting and participant feedback on the trainings was not available at the time of the evaluation. The planned communications trainings to assist parties with the dissemination of party policies and messaging are scheduled for after the policy conferences, which are expected to be held after May 2013 elections. 2. IRI conducted six election-focused communications trainings for central level party members in January/February 2013. The content appears clear and well organized. IRI indicated that the format of the training was interactive.91 The agenda for the February 2-3 training in Islamabad is dense, including: election campaign communications, message developing techniques, core elements of [a] media strategy, internal communications, and structures/mechanisms for effective internal communications. It is not clear if participants were able to absorb this amount of content in a two-day session. Participant feedback forms were not available at the time of the evaluation. 3. NDI conducted a series of communications trainings for district level party members on messaging and public speaking in February/March 2013. No reporting or data was available for review at the time of the evaluation. 4. Seven of 25 (28 percent) party leaders reported that PPDP has helped the party articulate messages internally and eight of 25 (32 percent) said PPDP has helped them articulate messages externally. These findings reflect the general impressions of leaders and are not limited to feedback on formal trainings. 91 IRI staff member, April 24, 2013. 41 TRAINING FELLOWSHIP PROGRAM Activity 12: Training Fellowship Program (TFP) (IR 1.4 Parties increase internal training capacity) Conclusion The Training Fellowship Program has effectively supported parties to enhance their internal training capacity. All Training Fellows demonstrated a strong understanding of concepts learned in TFP and the vast majority (79 percent) indicated that the party is utilizing them for election and non-election related activities. However, there was no indication that Training Fellows formally contribute to other PPDP activities, which could increase the effectiveness of these activities. Participants in the two provinces where TFPs have been conducted exceeded the program targets for improvements in thematic knowledge. A strong majority of Training Fellows (92 percent) and some leaders (16 percent) recommended improvements in the organizations and logistics of the TFPs, mostly related to scheduling. Findings 1. Fourteen of 14 (100 percent) Training Fellows interviewed showed good understanding of the concepts learned in TFP. This finding relates to participant recall of the training topics and elements of the training that they found more or less useful. The most frequently cited “useful” topics or themes included: public speaking and communications skills, volunteer recruitment, leadership, use of social media, election campaigning and the multiparty format of the training. Three participants said they found the second module the least useful; otherwise there was no consistent assessment of the content. Only three fellows cited their previous experience as a trainer; the most frequent explanation given for why they were selected was their active involvement in party activities. 2. Eleven of 14 (79 percent) Training Fellows reported that the party is utilizing them for party activities. These respondents included Training Fellows who had been either specifically tasked by the party or were actively contributing to the party by applying their new skills on their own. The common examples included conducting training or activities related to: voter outreach (including voter registration), volunteer recruitment, and use of social media for disseminating party messages. A Training Fellow in Islamabad said she used to stay away from politics, but now feels differently and said the party can use her skills to maintain contact with party members and recruit new members. Some Training Fellows felt the party has not effectively used them. One respondent in Karachi (where the TFP concluded in October 2012) said the party had not responded to suggestions about how they could utilize the Training Fellows graduates, such as helping the party to prepare for elections. Another graduate from Karachi felt the parties might lack confidence in the Training Fellows or be unaware of the skills they have gained. A graduate from Peshawar (where the TFP concluded in February 2013) said the party had not followed up to ask about the training. Respondents were eager for PPDP to help promote the skills of the fellows to party leadership and said that interaction with the party leader at the Training Fellow graduation event was encouraging. 3. Three of 14 (21 percent) Training Fellows reported that the party has established, or plans to establish, an internal training unit. Two Training Fellows from different parties in Karachi and Peshawar (where TFPs have been completed) reported that the party plans to establish an internal training unit. A third Training Fellow from another party in Peshawar said the party has already established a training unit that is involved in voter outreach, volunteer recruitment, party development and communications. NDI confirmed that some parties formally notified 42 them about the formation of training units and several other parties announced their intentions to form training units.92 4. Fifty-five percent of participants in module 1 and 69 percent of participants in module 2 showed an increase in thematic knowledge as a result of the training against a target value of 50 percent. This finding represents the improvement in Training Fellows’ knowledge, comparing pre- and post-training data for both provinces where the TFPs have been completed.93 5. Twelve of 13 (92 percent) Training Fellows and four of 25 (16 percent) party leaders provided recommendations for improving the organization and logistics of the TFP. Most recommendations from the fellows related to scheduling. They recommended: 15-20 days’ advance notice of events; three to four hour trainings rather than full-day events (in part to help accommodate women traveling from other districts to attend the events); shorter modules because some fellows are not available for multi-day events; and a preference for weekend events so that participants do not have to take leave from their regular work. They also suggested that NDI maintain contact with the Training Fellows between modules and after graduation. The leaders suggested: one-day events instead of multi-day events in order to facilitate the participation of women; more regional/rural events; and lifting the age requirement for female participants so that it is possible to tap a larger pool of qualified (literate) women. MULTIPARTY ROUNDTABLES Activity 13: Multiparty Roundtables (IR 5.2 Party leaders discuss electoral issues and recommend concrete measures to enhance the integrity of the 2013 electoral process) Conclusion The multiparty roundtables resulted in concrete recommendations to enhance the integrity of the 2013 electoral process. A sizable number of party leaders (33 percent) believed that the roundtables fostered interparty dialogue and a number of the recommendations that emerged from the roundtables were included in the report of the senate special committee on elections. While the report recommendations cannot be attributed to PPDP alone and may or may not be adequately addressed by the relevant bodies, the process itself of developing consensus amongst roundtable participants indicates the effectiveness of this activity. Findings 1. Three multiparty roundtables on elections were held94 and six of 18 (33 percent) central leaders reported that the roundtables fostered interparty dialogue. A central leader in Islamabad said, “the mood of the interparty event was ‘patriotic’ and above party politics and about helping the people.” A central leader from Lahore commented that multiparty activities help the parties develop synergies that are important to the survival of the country. 92 Email from ND staff member, April 10, 2013. 93 This finding is based on thematic knowledge portion of the NDI Training Index for first two modules in Sindh and Khyber Pakhtunkhwa as of January 18, 2013. It indicates the average increase in knowledge across the participating parties. Comparable data is not yet available for modules 3 and 4. 94 On July 3, July 24 and November 12, 2012. See PPDP Activity Tracking Sheet February 12, 2013 and NDI election roundtable reports. 43 2. Participants from 12 different parties participated in the roundtables and developed written consensus and recommendations for seven priority issues: the quality of the electoral rolls; the independence and autonomy of the Election Commission of Pakistan (ECP); election security; registration of voters; polling station access for voters; arrangements and access for election observers; and election dispute resolution. 95 In addition, roundtable participants from ten parties submitted consensus comments, recommendations, and concerns to the ECP related to the draft code of conduct for political parties in the form of a written statement.96 The statement addressed provisions in the code that the parties considered vague, unclear or requiring further negotiations with the parties.97 3. The report of the Senate Special Committee on Election Issues98 incorporated several of the recommendations highlighted during the multiparty roundtables. Given the numerous contributors to the report it is not possible to assign attribution to PPDP alone. NDI reported that the relevant committees in the National Assembly and Senate intended to address several of the concerns and recommendations raised under NDI’s multiparty banner through legislative amendments.99 The evaluation team understands that no legislative action has been taken yet despite the motion which obliges the government to act on the report. IPOR CAPACITY DEVELOPMENT Activity 14: IPOR Capacity Development (IR 2.1 IPOR eliminates its dependence on IRI) Given the nature of this activity, the following set of findings draws largely on interviews with IRI and IPOR staff and a review of related program documents, including the two indexes – the Research and Training Skills Index and the Organizational and Management Skills Index 100 developed by IRI to measure IPOR’s progression towards institutional independence and sustainability. Conclusion IPOR has made progress towards eliminating its dependence on IRI. In terms of technical capacity, IRI conducted the majority (83 percent) of planned trainings for IPOR and IPOR achieved high standards in its ability to independently conduct public opinion research and focus group discussions, while relying on IRI for data analysis and report writing. With respect to organizational capacity, IRI conducted the majority (67 percent) of planned trainings for IPOR and IPOR secured numerous research projects from 95 Consensus Electoral Priorities Resulting from Two Multiparty Roundtable Discussions, November 2012. This is included in the Joint Party Comments to the Election Commission of Pakistan, November 15, 2012. 96 Multiparty Roundtable Statement on ECP Draft Code of Conduct for Political Parties. This is included in the Joint Party Comments to the Election Commission of Pakistan, November 15, 2012. 97 Ibid. 98 Report of the Senate Special Committee on Election Issues, adopted March 5, 2013. http://www.senate.gov.pk/uploads/documents/1362572374_677.pdf 99 NDI PPDP quarterly report, October- December 2012, p. 63. 100 The Research and Training Skills index is designed to measure IPOR’s technical progression for conducting surveys, including data collection and analysis, enumerator protocol, data entry, report writing and stakeholder briefings. The Organizational and Management Skills Index is designed to measure IPOR’s organizational progression though IRI assistance and includes a cross-section of business sustainability and growth topics such as organizational design, business unit strategy, program management, human resource management, operations and portfolio management. These indexes reflect the so-called “gap analysis consultation” held between IRI and IPOR in November 2012 to benchmark IPOR’s institutional development. 44 diverse clients and developed a draft business plan that includes expanding its target market. IPOR depends on further support to develop its organizational capacity; IPOR has a system of internal communications, but does not yet have effective management hierarchy or systems or a sustainable organizational structure. Findings Technical capacity 1. IRI conducted five of six (83 percent) planned technical trainings for IPOR.101 There were some delays and adjustments in technical trainings due to travel delays for the U.S.-based survey research consultant.102 In addition to formal technical trainings, IRI provides technical advice and on-going mentoring to IPOR on issues such as questionnaire design and data analysis.103 2. IPOR was approved for membership in the European Society for Opinion and Market Research (ESOMAR), the largest global membership organization for marketing and social research. The evaluation team understands that ESOMAR membership is a recognized mark of excellence in the industry for market, social and opinion research and that ESOMAR reviews the work of members on an annual basis, including their methodology.104 IPOR confirmed that it is interested in expanding its work beyond the political sector to also do consumer research.105 3. The IPOR Research and Training Skills Index, which measures IPOR’s technical progression for conducting surveys, including data collection and analysis, enumerator protocol, data entry, report writing and stakeholder briefings indicates: a. IPOR is capable of independently conducting public opinion research in line with international standards, while not yet capable of independently analyzing the research. b. IPOR independently conducts focus group discussions, but does not yet independently write focus group reports. Organizational capacity 1. IRI conducted four of six (67 percent) planned organizational development trainings for IPOR.106 2. IPOR has secured scores of external survey research projects since the inception of the PPDP program from national and international clients.107 These include constituency, regional and 101 This figure reflects trainings held as of February 2013. The PPDP Year 1 and Year 2 work plans set a performance milestone of one technical training per quarter beginning in Year 1. This is equivalent to six trainings for the first 18 months of the program covered by the evaluation. 102 IPOR Research and Training Skills Index, February 2013. IRI noted, for example, that the February 2013 training was held via web interface rather than in person. Source: Exchange with IRI staff member, April 2013. 103 Meetings with IRI and IPOR, March-April 2013. 104 Meeting with IPOR staff member, March 27, 2013 and ESOMAR website, http://www.esomar.org/membership.php. 105 Ibid. 106 This figure reflects trainings held as of February 2013. The PPDP Year 1 and Year 2 work plans set a performance milestone of one organizational training per quarter beginning in Year 1. This is equivalent to six trainings for the first 18 months of the program covered by the evaluation. 107 IPOR External Project Tracker, February 2013. 45 provincial level polls; an impact evaluation; a focus group series and an in-depth interview series.108 3. IPOR’s business development plan (originally called organizational development plan) is expected to be completed in March 2013. The final business development plan was not available for review during the evaluation period.109 The draft business plan110 indicates that IPOR intends to expand its target market to include telecom and media groups. The plan does not identify any current staff with expertise in marketing, bid proposals, or quality control. 4. The IPOR Organizational and Management Skills Index indicates that IPOR has a system for internal communications. The Index notes recent improvements in internal communications structures and plans for targeted training to address communications deficits.111 5. The IPOR Organizational and Management Skills Index indicates that IPOR does not yet have an effective management hierarchy or system. The Index shows that IPOR lacks standardized procedures and does not yet have a formal organizational chart or a systematic hiring or performance review system.112 6. IPOR’s current structure does not promote organizational sustainability. IPOR’s draft business development plan and discussions with IPOR and IRI staff indicate that IPOR lacks senior staff and that technical, managerial, business development and marketing responsibilities are concentrated in the position of executive director.113 Likewise, the evaluation team did not find evidence of plans to introduce a corporate governance structure following PPDP. The report from the February 2013 business development training provided by an independent organization noted that it was important for IPOR to begin to implement systems and mechanisms that allow for delegation and accountability within the organization. “Putting systems in place that allow for cleaner delegation and building trust with his team is key for IPOR as it continues to grow and prosper.”114 7. IRI and IPOR developed an activity plan for IPOR’s organizational capacity development through July 2013. The activity plan115 outlines a number of targets and steps for IPOR capacity development from October 2012 to July 2013. In addition to steps already undertaken, this plan includes completion of an employee handbook outlining policies and procedures116, training on brand awareness and recognition, training on accounting and financial management, and the development of a three-year strategic plan for IPOR. 108 Ibid. 109 Complete program data is not yet available for January-March 2013. 110 IPOR business plan working draft, March 1, 2013. 111 IPOR Organizational and Management Skills Index, February 2013. 112 Ibid. 113 IPOR business plan working draft of March 1, 2013 and meetings with IPOR and IRI staff March-April 2013. 114 IRI-Pakistan Trainer Report, Business Development Training for IPOR by Invest2Innovate, p.2. 115 Activity Plan for Capacity Development of IPOR, December 2012. 116 The draft was completed in November 2012. The evaluation team is not aware of a more recent version. 46 TECHNICAL TRAINING FOR POLITICAL PARTIES Activity 15: Technical Training for Political Parties (IR 2.2 IPOR engages political parties and think tanks117) Conclusion IPOR has engaged political parties though various technical trainings and discussions, in partnership with IRI. IPOR assists IRI in training political parties through the RTA framework. IPOR has acquired the skills necessary to independently conduct FGDs, while relying on IRI for related planning and reporting expertise. IPOR is still developing its capacity to independently brief stakeholders on the results of IRI/IPOR research. IRI/IPOR briefings with RUs were held on an ad hoc basis and focused on the technical aspects of polls rather than on the poll results as such. Continued training for parties is important for achieving their buy-in and building their knowledge. Similarly, IPOR requires continued training and opportunities to practice its skills, including the presentation of survey findings. Findings 1. The quarterly program reports and IPOR Research and Trainings Skills Index indicate that IPOR assisted IRI with trainings to the RUs. Researchers did not explicitly comment on IPOR’s role in the trainings. 2. The IPOR Research and Training Skills Index shows that IPOR was capable of independently briefing stakeholders on the topline results of IRI/ IPOR opinion research; however, IRI said that IPOR needs further training and practice before it does this.118 Each quarterly training that IRI conducts for IPOR is meant to include a component on the presentation of poll results to stakeholders.119 3. The IPOR Research and Training Skills Index shows that IPOR recently began conducting FGDs independently, while IPOR provides significant guidance with the planning and report drafting. 4. There was limited evidence that IRI/IPOR conducted poll briefings with the RUs. IRI clarified that given the sensitivities around providing poll briefings in a multiparty forum, briefings with RUs are conducted on an ad hoc basis and focus more on technical aspects of the polls than on the results of the polls.120 117 The think tank component of the program has been removed. 118 Debriefing with PPDP and USAID, April 24, 2013. 119 See e.g. NDI PPDP Year 2 Work Plan, p.17. 120 Debriefing with PPDP and USAID, April 24, 2013. 47 FINDINGS AND CONCLUSIONS FOR EVALUATION QUESTION 1 Evaluation Question 1: How effectively are the partners implementing the planned approach and what changes to the approach might improve the efficiency with which the overall set of activities contribute to achieving intended intermediate results? This section of the report discusses findings and conclusions for evaluation question 1. The findings and conclusions are grouped by relevant IRs in order to illustrate linkages between the overall set of activities, as understood by the evaluation team. For stand-alone activities, including those specific to the 2013 elections, the analysis addresses the extent to which the activity is necessary for achieving intended IRs or contributes towards the achievement of the relevant program objective. IR 3.2 is not addressed under question 1 because the activity data is not yet available. IRs 4.1, 4.2 and 5.1 are also not addressed here because no activities have taken place under these IRs. This section draws on the findings under question 2. Therefore, only the relevant findings are repeated here. The conclusions are listed first followed by the findings or a synthesis of the findings where appropriate. The syntheses begin with an overall assessment statement followed by more detailed findings. This is done to establish a link between the detailed findings and the conclusion. IR 1.1-1.3 & 3.1 (Party leadership participates in the policy development cycle- IR 1.1; Parties increase policy development capacity at the party and legislative levels- IR 1.2; Parties have increased capacity to define and utilize research- IR 1.3; Parties aggregate and respond to local concerns IR 3.1) Conclusions 1. The sequence of activities ranging from IIWs, PWG activities, party member surveys, formation of RUs to leadership review of policy drafts promotes inclusive policy development; continued efforts are needed to inform participants of the agenda and purpose of activities well in advance and to select participants with the necessary skill sets. 2. The planned research and policy development processes are not yet coordinated efficiently and this could weaken the effectiveness of policy development. 3. The PWG policy drafts are regularly informing party platforms (manifestos). The CSO forums, party member surveys and public opinion research do not yet efficiently inform policy development. 4. PPDP is progressing towards the achievement of objective 1 and objective 3 in terms of engaging the grassroots and leadership in policy formulation and introducing a more systematic approach to this work. Several challenges remain for building the capacity of parties to conduct research and analysis. 48 Synthesis of Findings 1. Parties have broadly introduced the PDC approach to policy development; party leaders were directly involved in PPDP through the selection of participants and their own participation in some activities, and grassroots party members had opportunities to engage through IIW and party member surveys. Program participants were not always aware of the agenda or purpose of activities in advance and there were challenges with the selection process for Researchers in particular. This assessment reflects: a. The 41 percent of all respondents who reported that party leaders are directly involved in PPDP; b. The 100 percent of IIW participants who reported a good selection process for IIW participants and the contribution of women, youth, and grassroots participants in policy development; c. The 74 percent return rate on party member surveys; the 63 percent of RUs who reported that they applied research skills to assist the party; d. The 87 percent of the PWGs who said they prepared policy drafts; and, e. The 56 percent of PWG members who said party leadership reviewed the policy drafts that PWGs prepared. Along with the positive trends, the finding also reflects the only 14 percent of grassroots IIW participants who said they were aware of the purpose of the workshop in advance and the limited geographic reach of the party member surveys; and challenges with selecting Researchers with the interest, commitment and capacity to conduct research. 2. The profiles of PWG and RU members, entrenched structures, and different timelines for research and policy-related activities impacted the coordination of research and policy development processes. This assessment reflects the 76 percent of PWG members who are low and mid-ranking party members relative to the 93 percent of RU members who are mid and high-ranking party members; the limited 43 percent of RU members who attended the three training modules for which data is available; the mere four percent of party leaders and six percent of PWG members who reported regular interaction with each other; and the 0 percent of PWG members who reported using data from the RUs. It also reflects the considerable role that NDI played in assisting PWGs with analysis of party member surveys and in providing policy briefs and secondary research data to PWGs. 3. The party manifestos regularly reflected the PWG policy drafts while the CSO forums, party members surveys and public opinion research appear to have had limited impact in informing policy development. This assessment reflects the strong correlation between PWG policy drafts and the language in the 2013 party manifestos in 27 percent of cases and moderate correlation in 56 percent of cases. It also reflects the diverging opinions amongst PWG members regarding the utility and quality of the CSO forums and 18 percent of PWG members who reported using information from the CSO forums and other experts to inform policy drafts; as well as the just 12 percent of party leaders who articulated ways in which the party is utilizing the IRI/ IPOR 49 opinion polls and none of the PWG members who reported using the polls to inform policy drafting, including those PWG members who had been briefed on the polls.121 IR 1.4 (Parties increase internal training capacity) Conclusions 1. The TFP has helped the parties to effectively enhance their internal training capacity and appears to be on track for achieving objective 1 in terms of the ability of parties to conduct training. 2. The Training Fellows do not yet formally contribute to other aspects of the PDC or PPDP activities which could increase the effectiveness of these activities. Findings 1. One hundred percent of Training Fellows showed an understanding of the concepts learned in the TFP and 79 percent reported that the party is utilizing them for party activities, particularly ahead of May 2013 elections. The first two TFPs concluded in December 2012 and February 2013. The current program design, according to which the TFP is outside of the PDC, does not suggest that Training Fellows will be integrated into other PPDP activities. IR 2.1- 2.2 (IPOR eliminates its dependence on IRI- IR 2.1; IPOR engages political parties and think tanks- IR 2.2) Conclusions 1. The investment in IPOR has shown results in terms of building IPOR’s technical and organizational capacity and providing technical training for political parties. 2. While continued, targeted support is needed to build IPOR’s organizational capacity, in particular, as well as its capacity to analyze and report on research findings and brief stakeholders, PPDP appears to be on track towards achieving objective 2 in terms of establishing an independent research facility. Findings 1. IPOR is progressing towards institutional independence but requires continued technical support to achieve this goal. This assessment reflects the 83 percent of planned technical trainings conducted for IPOR; IPOR’s recent approval for ESOMAR membership; its ability to independently conduct public opinion research and FGDs while relying on IPOR for assistance with analysis and reporting; its role in delivering technical training to political parties; and its need for further training and practice to brief stakeholders on the findings of opinion research. 121 As reported in more detail under Question 2, a possible explanation for this is the PWGs had completed or nearly completed their policy drafts before they were briefed on the polls. 50 2. IPOR is progressing towards institutional independence but requires continued organizational development support to achieve this goal. This assessment reflects the 67 percent of planned organizational trainings held for IPOR; the numerous research projects for national and international clients that IPOR has secured; its structure for internal communications; its development of a draft business plan; and the existence of a priority activity plan for enhancing IPOR’s organizational capacity. It also reflects IPOR’s lack of standardized procedures, formal organizational chart, and systematic hiring and performance review system; as well as a lack of senior staff to share functional responsibilities with the executive director and the absence of evidence of an alternative corporate governance structure once IRI support ends. IR 5.2 (Party leaders discuss electoral issues and recommend concrete measures to enhance the integrity of the 2013 electoral process) Conclusion 1. The multiparty roundtables resulted in concrete measures to enhance the integrity of the 2013 electoral process. The roundtables appear effectively to have contributed to objective 5 in terms of strengthening democratic political party practices. Findings 1. The multiparty roundtables resulted in concrete outputs related to electoral reform and were an example of democratic political party practices. This assessment reflects the 33 percent of central leaders who reported that the multiparty roundtables fostered interparty dialogue; the written consensus and recommendations for seven priority electoral issues that the roundtable participants from 12 parties achieved; the written consensus comments, recommendations, and concerns related to the draft code of conduct for political parties that participants from ten parties submitted to the ECP; and the inclusion of a number of recommendations from these roundtables (and other sources) in the report of the Senate Special Committee on Elections. 51 MAIN CONCLUSIONS AND RECOMMENDATIONS MAIN CONCLUSIONS This section highlights the main conclusions for evaluation questions one and two, which, in turn, inform the recommendations to answer evaluation question three. 1. The findings related to leadership commitment to the PDC, the selection processes for various activities, communication between leaders and PPDP participants, leaders’ use of public opinion research and their review of policy drafts show that the involvement of party leadership in PPDP is overall positive, while moderate. The appropriate involvement of leaders can contribute to overall program efficiency and effectiveness and increase the likelihood that parties adopt a more inclusive, research-driven approach to policy development over the long term. 2. The findings show that the IIWs effectively engaged a broad range of grassroots party members and party leaders in the process of identifying issues, including those related to the MDGs, and that the party member surveys served as a party building exercise. Both activities contributed to the aggregation of local concerns. The use of issue-specific secondary research data for the IIW discussion could have improved the effectiveness of the activity as far as informing policy development. The party member surveys employed some of the techniques used in formal large￾sample surveys but the results of the surveys do not have statistical validity since they did not use standard research methods. While the findings suggest that the surveys did inform policy development in some instances, they also reveal weaknesses in the survey approach that severely limit their utility. Moreover, primary data collection is less efficient compared to secondary data collected by sector specialists. 3. As evidenced by the findings on policy drafting, leadership review of policy drafts and policy conferences, the use of PWGs for policy drafting reflects a change in the standard top-down approach to policy development. The findings show that nearly all PWGs prepared policy drafts and there is solid evidence that the party manifestos reflected the work of PWGs. 4. As indicated by the findings related to the formation of PWGs and the RUs and the RTA curriculum, the composition of the PWGs and RUs do not match the skills required, which limits the potential effectiveness of policy and research processes, two main elements of the program. In addition to challenges with the selection processes for PWG members and Researchers, entrenched party structures and approaches to policy development restrict coordination between PWGs and RUs. 5. The RUs and the RTA curriculum, public opinion polls and the CSO forums have modestly increased the capacity of parties to define and utilize research. The low attendance rates in RTA modules, weaknesses in the composition of RUs, disconnect between RUs and PWGs and problems with the quality of speakers and facilitation at some CSO forums appears to have inhibited the effectiveness of these activities. 52 6. The TFP has effectively supported parties to enhance their internal training capacity. The Training Fellows do not yet formally contribute to other PPDP activities, which could increase the effectiveness of these activities. 7. The investment in IPOR has shown results in terms of building IPOR’s technical and organizational capacity and providing technical training for political parties. While continued, targeted support is needed to build IPOR’s organizational capacity (structure), in particular, as well as its capacity to analyze and report on research findings and brief stakeholders, IPOR appears to be progressing towards eliminating its dependence on IRI. Travel delays for the U.S.- based survey research consultant could inhibit planned technical trainings for IPOR. 8. The multiparty roundtables resulted in concrete recommendations to enhance the integrity of the 2013 electoral process and were an example of democratic political party practices. RECOMMENDATIONS Evaluation Question 3: How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? This section includes proposed actions to be taken by program management based on the main findings and conclusions above. The recommendations are designed to be forward-looking. Therefore, this section does not include recommendations for activities undertaken specific to the 2013 elections. 1. In order to enhance the constructive involvement of party leaders in PPDP, the project could seek written (not just verbal) commitment from party leadership for the second program cycle. This document could outline roles for party leaders and PPDP. 2. Depending on the design of the IIWs for the second program cycle (and particularly if many of the same people participate) PPDP could request parties to select PWG members ahead of the IIWs. PWGs could provide an overview of the policy issues in the province, such as MDG themes, during the IIWs. Alternatively, party leaders who participate in the IIWs could present such information. The information could serve as a starting point for discussion and demonstrate to participants the benefits of using secondary sources of information, including national, provincial and district-level data as a low cost alternative to parties conducting their own research and data collection. It is understood that adding this presentation element to the IIWs would need to be weighed against the goal of energizing grassroots participants to brainstorm and contribute their ideas without influence. 3. PPDP program documents and practice could be modified to clearly articulate the intent for party member surveys122 to be a party mobilization/outreach exercise, as distinct from a statistically reliable tool to inform policy drafting. Political parties need to understand how to solicit member input that is not meant to provide statistically reliable results and how to use the results of such exercises in tandem with other informal and formal data collection methods. As an alternative means to inform policy drafting, IIW participants could meet with district committees to seek their input on the priorities emerging from IIW and share this feedback with PWGs. 122 PPDP intends to use the term ‘party member input forms’ (‘party forms’) instead of ‘survey’ beginning in April 2013. 53 4. PPDP could develop clear criteria to provide to party leadership for selecting new PWG members. The criteria could include issue experts from the party who can effectively contribute to policy drafting (and legislative drafting, should the focus change to legislative drafting for the next program cycle). The criteria could also include some members with a research background. PPDP could explore the option of integrating RU members into the PWGs to promote more effective coordination between the research and policy development processes. 5. PPDP could develop clear criteria to provide to the party leadership for selecting/replacing Researchers to ensure that RUs have the appropriate composition of talent and committed participants to support the work of the PWGs and the party as a whole. (PPDP has already flagged this as an issue that the program intends to address ahead of the second program cycle.) The criteria might include a stronger emphasis on low and mid-ranking party members with demonstrated aptitude for research, writing and communication. In coordination with party leadership, PPDP could screen the nominees and conduct regular assessments of the progress and commitment of individual Researchers to determine their particular training needs and whether or not to keep them in the program. PPDP also could consider offering parties a menu of options for participating in different levels or modules of research training. Selection could be on the basis of capacity, interest and record of commitment. The training could emphasize the benefits of using secondary research to inform PWG/party activities and focus on participants becoming better consumers - not necessarily producers - of research. 6. To ensure that CSO forums achieve the goal of informing policy development, PPDP will need to ensure that the speakers who are selected are knowledgeable about the provinces and well￾briefed on purpose and policy priorities of PWGs. 7. PPDP could explore ways to engage select Training Fellow graduates in other aspects of the program, such as future communication trainings for national, provincial and district party officials. This could improve the effectiveness of these activities and demonstrate to the party members the skills of these individuals. 8. PPDP could encourage IPOR to explore options for a corporate governance structure in line with Pakistani law and international good practice that can replace IRI oversight when PPDP ends. This could help to promote the integrity of the company, attract business, and provide direction on legal, financial and personnel matters. In addition, in order to allow the IPOR Executive Director to focus on the technical side of the business, it could be very helpful to hire additional senior staff who could take on some of the operational responsibilities. For example, this could be a counterpart to the executive director, who would be responsible for bid proposals, quality control and marketing. 9. Should the U.S.-based survey research consultant face further travel delays, PPDP may need to identify other experts who can provide planned in-country technical trainings for IPOR. 54 ANNEXES Note: Party names and other identifying data have been removed from the annexes in order to maintain confidentiality. Numbers are used instead of party names. The same party numbering is used throughout the report. 55 ANNEX I: EVALUATION STATEMENT OF WORK POLITICAL PARTY DEVELOPMENT PROGRAM (PPDP) EVALUATION STATEMENT OF WORK MARCH 9, 2013 56 POLITICAL PARTY DEVELOPMENT PROGRAM (PPDP) EVALUATION STATEMENT OF WORK 57 I.BACKGROUND INFORMATION A. Identifying Information about the Project This Statement of Work (SOW) outlines the requirements for the Monitoring and Evaluation Program (MEP) to conduct a mid-term performance evaluation of the United States Agency for International Development (USAID)’s Political Party Development Program (PPDP), earlier called the Political Parties Development Program. The PPDP works with the political parties of Pakistan to help them engage their members and leaders in policy development and selected additional aspects of political participation. The project operates under a cooperative agreement the continuation of which for the remaining three years is subject to a successful performance evaluation of the initial project period. The table below summarizes key facts about the PPDP. TABLE 1: PROJECT SUMMARY B. Development Context 1. Problem or Opportunity Addressed Pakistan is a federal state still in the process of establishing a fully functioning democracy. The military has ruled Pakistan for more than half of the 65 years since its creation and controlled and guided the political process for much of the country’s existence. Furthermore, the political culture encourages dynastic politics and patronage, tolerates corruption and often discriminates against women, youth and other groups that lack power and influence. These factors have made it difficult for political parties to emerge as effective democratic institutions. The 2013 national and provincial elections, along with potential local elections in the near future, present an opportunity to introduce change. With mechanisms for rank and file members at the district, provincial and the central levels to participate in shaping the parties’ platforms, the parties could become more representative and responsive to citizen Project Name/ Title Political Party Development Program Agreement Number AID-391-A-11-00004 Agreement Officer’s Representative (AOR) Humaira Ashraf (Ms.), Program Management Specialist Project Start Date July 15, 2011 Project Completion Date July 14, 2013 Project Location Nationwide USAID Objective Addressed Cross-cutting Objective1: Improved governance in response to citizen concerns. Name of Implementing Organization National Democratic Institute (NDI). NDI has two partners:  International Republican Institute (IRI), which partners with the Institute for Public Opinion Research (IPOR); and,  Strengthening Participating Organization (SPO) Budget USD 21.5 Million 58 concerns. Developing the capacities of the parties at all levels to engage in research could also provide an empirical foundation for policy formulation, which can lead to more effective governance. More responsive, open, democratic and effective parties can, in turn, lead to greater public confidence in the parties themselves and in the democratic process. 2. Target Areas and Groups The PPDP is currently working with ten parties at the national and provincial levels and expects to add two more in its second year, Fiscal Year 2013 (FY13). In instances in which a party is national, PPDP might work with the party in two, three or four provinces (units). PPDP originally planned to work with 16 different party units. By March 2013, 17 party units from 10 parties had participated in at least one routine project-facilitated activity (Table 2), while these 10 parties and the Pakhtunkhwa MilliAwami Party had also participated in election-related multiparty roundtables described below. In the project’s second year, PPDP aims to increase the number of units among the pool of participating parties and expand the project’s geographic reach. TABLE 2: POLITICAL PARTIES PARTICIPATING IN PPDP123 Party Province 1 Province 2 Province 3 Province 4 Party 3 X X X X Party 7 X X Party 1 X X X Party 4 X Party 10 X Party 9 X Party 5 X Party 8 X Party 2 X Party 6 X X Total 4 4 6 3 PPDP works with several groups of party members and leaders in each party, including the following, whose involvement in project activities is outlined in Section I.D124: a. 813 grassroots party members who serve as activists or office bearers at the district level and below and participate in the Policy Development Cycle of PPDP, introduced in Section I.D), 47 percent of whom are young men and women (18-35 years old) and 41 percent are women; b. 58 divisional leaders (leaders with responsibility at the administrative unit called the civil division, which includes two or more districts); c. 237 provincial leaders (party office bearers at the provincial level); d. 106 central leaders (party office bearers at the national level); 123 Party, province, district and city names have been replaced by numbers in the SOW in order to maintain confidentiality. 124 The numbers below refer to the period up to March 2013. 59 e. 20 researchers from various political parties who were trained by PPDP to analyze and utilize public opinion research for their parties; and, f. 25 Training Fellows from various parties who were trained as master trainers by PPDP to train grassroots party members in aspects of party operations. Another 28 Training Fellows will graduate from the program in KP in February 2013. C. Intended Results PPDP contributes to USAID/Pakistan’s Mission Strategic Framework (MSF), which has recently been finalized, through Cross-cutting Objective 1: Improved governance in response to citizen concerns. Within the MSF, PPDP contributes directly to the achievement of the Intermediate Result 1.4: Strengthened credible political institutions. PPDP also is a key contributor to the achievement of the sub￾Intermediate Results 1.4.1. Political parties more responsive, 1.4.2 Increased women’s participation in the electoral process, and 1.4.3 Improved integrity of electoral process. The overarching goal of PPDP is to increase the willingness and enhance the ability of political parties to contribute to democratic policy-making and governance processes at the national and local levels in a more open, research-driven and representative manner and thereby generate greater public trust and confidence in their ability to govern and affect positive policy change. To achieve this, the PPDP is organized under five primary objectives (the fifth of which was added in the year two work plan and is not yet approved by USAID) and 10 (12 including the new objective) intermediate results (IRs) which contribute to various steps of the PDC. Table 3 summarizes the project’s results framework and expected impacts. Key deliverables corresponding to the objectives are listed in Annex 1. TABLE 3: PPDP RESULTS FRAMEWORK Objectives Intermediate results Objective 1: Parties conduct their own research, analysis, and training for the formulation of increasingly responsive and informed platforms and policies; and parties contribute more actively and effectively in the policy-making of government institutions such as legislatures and commissions.  Intermediate Result 1.1: Party leadership participates in the policy development cycle.  Intermediate Result 1.2: Parties increase policy development capacity at the party and legislative levels.  Intermediate Result 1.3: Parties have increased capacity to define and utilize research.  Intermediate Result 1.4: Parties increase internal training capacity. Objective 2: An independent opinion research facility is established that adheres to international research standards.  Intermediate Result 2.1: IPOR eliminates its dependence on IRI.  Intermediate Result 2.2: IPOR engages political parties and think tanks on public opinion research. Objective 3: Parties communicate more effectively with their members, constituents and the general public at the national and local levels, both in terms of articulating messages and aggregating and responding to concerns, requests and ideas.  Intermediate Result 3.1: Parties aggregate and respond to local concerns.  Intermediate Result 3.2: Parties more effectively articulate messages internally and externally. Objective 4: Parties demonstrate movement toward implementing internationally recognized standards for internal democracy and transparency.  Intermediate Result 4.1: Party leaders support inclusion of international standards for democratic parties in party by-laws  Intermediate Result 4.2: Party by-laws are reviewed and recommendations are made to support elements of the policy development cycle Objective 5: Democratic political party  Intermediate Result 5.1: Party poll watchers are enabled to participate effectively in the 2013 electoral process. 60 Objectives Intermediate results practices are strengthened through effective multi-party participation in Pakistan’s electoral process (added for FY 13 and subject to USAID approval).  Intermediate Result 5.2: Party leaders discuss electoral issues and recommend concrete measures to enhance the integrity of the 2013 electoral process. D. Approach and Implementation 1. Policy Development Cycle PPDP is programmed to undertake 15 major activities, 12 of which are part of the Policy Development Cycle (PDC), two supplement the PDC and one is associated specifically with the 2013 elections.125 The PDC and other activities are illustrated in Figure 1. The PDC is a bottom-up policy development process which starts with PPDP seeking buy-in from party leaders in one-on-one meetings. The phases of the complete process are described below: Phase 1: Parties Commit to PPDP. NDI invites USAID-approved political parties to participate in PPDP. If the leadership of a political party assigns a PPDP liaison and participates in one or more phases of the PDC, NDI considers that party to be committed to PPDP. Phase 2: Issue Identification. Each party holds an Issue Identification Workshop (IIW), organized by NDI and jointly facilitated by NDI and SPO, in which the participants represent a diverse array of party members from within a province with a particular emphasis on including women and men who represent their party at the district level or below. More precisely, there are approximately 75 participants (most of them grassroots leaders) identified by the senior (that is provincial and/or central) leadership. At least 35 percent of the IIW members must be women and 35 percent young members (18-35 years old). Democratically and through discussion, the IIW prioritizes 3-4 issues for policy development.126 Phase 3: Policy Working Groups (PWGs) Activated. The priority issues are taken up by the PWG which comprises of 10 members (at least 50 percent women), most of whom also participated in the IIW. While some PWGs were elected by IIW participants in Year 1 of the program, NDI has since modified the PWG selection process and now requests that party leaders select the PWG members based on specific criteria. The PWGs are responsible for drafting policies that are research-based and developed in consultation with various strata of the party and experts external to the party. They also conduct party member surveys, which they develop with NDI support. By sanctioning and supporting the creation of a PWG, a political party is informally institutionalizing its commitment to inclusive, responsive, and research-based party policy-making. Phase 4: Research Units (RUs) Form and Start Research Training Academy. Research Units are internal party entities that will support PPDP-supported PWGs and focus on party internal research capacity. A "formed" Unit refers to the designation of party members as agreed upon by the party and PPDP implementers. RUs receive training (in six modules) from the PPDP’s Research Training Academy (RTA) on public opinion research (using both quantitative and qualitative methods), a process that is led by IRI and may be assisted by IPOR. By forming a unit through the designation of party member 125 The PPDP approach is elaborated in its Year 1 and Year 2 work plans. 126 These issues have included health, education, unemployment, poverty, the economic crisis, law and order, the energy crisis, peace, justice, the rule of law and youth empowerment. 61 participants at the RTA, a political party is informally institutionalizing its commitment to advancing internal research capacity. FIGURE 1: PPDP POLICY DEVELOPMENT CYCLE (PDC), FY 12-13 Phase 5: Party Member Surveys. Trained by NDI, approximately 75 IIW participants survey 15-20 party members each in their districts to gather feedback on the three to four issues prioritized at the IIW. Although not scientific, the surveys serve as a party-building exercise and survey results provide an opportunity for the PWGs to consider and incorporate grassroots party-member input into their policy drafting. Phase 6: Public Opinion Research. This refers to scientifically conducted polls and Focus Group Discussions (FGDs). IRI prepares and conduct quarterly public opinion polls. The poll methodology is designed so that each of Pakistan’s four provinces is oversampled once a year to obtain provincial-level results. Polling results are used to verify that issues identified by PWGs are indeed reflective of citizen concerns. In addition to quantitative research, IRI and IPOR conduct four focus groups each quarter to develop qualitative insights into specific issues identified by the PWGs as well as IRI and NDI following analysis of polls. Party leadership and the PWGs are briefed on the poll results and FGDs as appropriate. This is an ongoing activity and extends beyond the PDC. Phase 7: Civil Society Organization (CSO) Forums. CSO forums, organized by NDI with SPO assistance, are designed to be venues for political parties and civil society to identify and discuss mutual Phase 1: Parties commit to PPDP Phase 2: Issue Identificati on Phase 3: PWG Formed Phase 4: RU formed and begin RTA Phase 5: Party member surveys Phase 6: Public Opinion Research Phase 7: Consultati on at CSO forums Phase 8: Preparatio n of policy draft Phase 9: Leadership review of policy draft Phase 10: Policy conference Phase 11: Comm. workshop (Leaders) Phase 12: Comm. workshop (district) Completed On-Going/In Progress Will begin and be completed in FY13 PPDP Single/Multiparty Roundtable Training Fellowship Program Activities designed to supplement the PDC NDI will also provide electoral assistance in Pakistan in FY13 by training party poll watchers. 62 areas of concern, in particular community-based problems and possible solution-oriented interventions. PWGs may also choose to continue consulting with CSOs, viewing them as issue-experts, while designing and drafting their party policies. Phase 8: Policy Drafting Based on Research and Party Input. The PWG prepares a policy draft with the technical assistance (TA) of PPDP. While NDI’s PDC and relevant activities are designed to encourage policy drafting based on research and party input, the depth and quality of research and solicitation of party member input is dependent on the PWG’s and party’s commitment and level of effort. Until the RUs reach the level of maturity required to support their parties, IRI and IPOR will share public opinion research with the PWGs. PPDP supports the PWGs during all phases of the PDC; however, the Cycle itself is designed to be a self-directed and party initiated process. This approach is intended to build-in party ownership and could enhance the likelihood that the PDC, in whole or in-part, would be incorporated into party practices, as appropriate to each respective party. The qualitative policy development process will be measured and captured quantitatively on an individualized party basis by NDI’s Policy Development Index (PDI).127 For the first PDC, all of the PWGs created work plans with the goal of completing their policy drafts before the elections, instead of the work plan target of completing them by September 2013. Phase 9: Leadership Review of Policy Draft. The draft is presented for review and approval by the senior leadership of the party. Initiation and completion of this phase will be dependent on each party’s leadership. NDI will proactively assist those PWGs that may face challenges with getting feedback from their leadership. In a manner determined appropriate by the PWGs, NDI will assist the PWGs in bringing the policy drafts to their party’s leadership attention and for their review and consideration. Among other possible mediums, NDI may facilitate a meeting for PWG representatives to present the draft policies directly and in-person to party leadership.128 Where appropriate, NDI may also discuss with leadership possible options for the incorporation of PWG members into existing party structures (e.g., manifesto committee) or creating a permanent policy drafting unit should such a body not exist. Depending on a party’s response to the draft policies, NDI will also inquire whether the party intends to incorporate wholly or in part any of the policies into its electoral platform or legislative agenda. Phase 10: Policy Conferences. The policy document is presented at the party’s policy conference. NDI will organize a policy conference with each participating party. The conference will serve as a venue to present the policies produced by that party’s PWGs. Grassroots, middle, and central level leaders will participate in the conference. The actual format will depend on party preferences. However, it is anticipated that PWG members will have the opportunity to present the policies as well as highlight the process undertaken to prepare the policies (i.e. the PDC). The agenda will likely also include an opportunity to debate the policies once presented. One policy conference will be organized for each party; therefore, for larger parties, all of their PWGs will be at the same conference. Following the conference, NDI will provide appropriate ongoing support geared towards supporting party members’ advocacy efforts to include approved policies into legislative proposals or into their party manifestos.129 127 Phases 1-8 of the first round of the PDC have been completed, while the subsequent phases are planned for the early part of 2013. 128 Six party units have initiated the leadership review of completed policy drafts. 129 One party has held a party conference so far. 63 Phase 11: Communication Workshops (Leadership). IRI, in coordination with NDI, will conduct communication workshops with the central and provincial leadership of the party focusing on the skills required to disseminate party messages effectively within the party as well as externally. Where parties already have strategic communication plans in place, IRI will provide appropriate technical assistance on a customized basis as requested by the parties.130 Phase 12: Communication Workshops (District Level). Similar workshops are held at the district level. NDI, in coordination with IRI, will hold communication workshops with grassroots members of the party. The workshops will focus on the technical communication skills required to disseminate party messages effectively and will be tailored to the needs of party members. The roles played by PPDP and the political parties in the PDC are summarized in Table 4. 2. Activities to Supplement the PDC Training Fellowship Program (TFP). The purpose of the Training Fellowship is to build the parties’ capacity to conduct training for its members, activists, and candidates. Working with party leadership from up to 12 political parties, NDI invites the nomination of least four party members, 35 years or younger, at least half of whom are women, from each provincial party unit to serve as Training Fellows. NDI has prepared four workshop modules which are held in a different province every four to five months. The modules reflect training-of-trainers (ToT) curriculum with an emphasis on adult training techniques, internal and external party communications, effective party organizing, leadership skills, fundraising, membership recruitment, volunteer recruitment and management, strategic planning and implementation, event planning, and election preparation. The TFP training methodology emphasizes group work and learning by doing. All Fellows receive ongoing written and verbal coaching and skills support from NDI to maximize knowledge retention and results. Each module also requires the Fellows to practice their newly learned knowledge and skills by conducting their own trainings in their home districts for other party members. Training Fellows are provided graduation certificates upon completion of the TFP. By holding the TFP in a multiparty format with substantial group work, young party members are introduced to the concept of working across party lines—a first for many of them. Multi-party Roundtables. NDI has been convening multiparty roundtables on a regular basis to provide senior-level party members with a venue to discuss key issues relevant to political party strengthening and internal party democracy. Party Poll Watchers. Specifically for the 2013 electoral process, PPDP will also train party poll watchers to enable them to participate effectively in the forthcoming elections. This activity has not yet taken place, but it is likely to occur 60 days before election day. Single-party Roundtables. PPDP is also programmed to organize single-party and multi-party roundtables, the former focusing on internal party democracy and the latter on common (particularly electoral) issues. The parties have informed PPDP that it is unrealistic to expect changes in internal operations before the 2013 elections; thus, single-party roundtables are expected to begin after the 2013 elections. 130 One party has held this workshop so far. 64 TABLE 3: PARTY AND PPDP ROLES IN THE POLICY DEVELOPMENT CYCLE Phases of PDC Senior Party Leaders Grassroots Party Members PPDP 1. Parties commit to PPDP Provide commitment. Introduce PPDP (NDI’s responsibility). 2. IIW held Identify 60-75 IIW members. Participate in IIW. Convey profile of IIW members to Senior Leaders. Organize (NDI’s role) and facilitate IIW (NDI and SPO). 3. PWG activated Identify 10 PWG members. Selected members participate in PWG. Convey profile of PWG members to Senior Leaders and organize PWG (NDI). 4. RU formed, training started Identify researchers Organize RTA modules and one-on-one support (IRI). 5. Party member survey of issues conducted PWG members conduct informal survey of grassroots members. Assist with review of survey results, if needed (NDI). 6. Public opinion research discussed Attend survey briefings highlighting public opinion on PWG-identified policy issues PWG members discuss public opinion research. Prepare RU and IPOR to support PWG and discuss public opinion research (IRI). 7. Consultation at CSO forum completed PWG members discuss policy issues in CSO forum. Organize CSO forum (SPO). 8. Policy draft prepared PWG members prepare policy draft. Provide TA (NDI). 9. Leadership review of policy draft completed Review policy draft, provide guidance for finalizing it. PWG members present policy draft. 10. Policy conference held Present policy at conference. Participate in conference. Observe and provide TA, as required by a party, in organizing the event (NDI). 11. Communication workshop (for leadership) completed Attend training on communication. Organize workshops (IRI). 12. Communication workshop (district-level) completed Attend training on communication. Organize workshops (NDI). 65 3. Management Structure NDI is responsible for the overall management of PPDP, led by the Chief of Party (COP) who serves as the senior Point of Contact (POC) for USAID and for senior level party engagement. The team is divided as follows:  The DCOP manages M&E integration, the election related roundtables, and provides overall support to the COP, including serving as POC for IRI and USAID coordination.  One NDI Senior Program Manager leads programming at the grassroots level (i.e. district level communication workshops and IIWs) and manages NDI’s external communication officer.  One NDI Senior Program Manager leads programming around policy process and serves as POC for SPO coordination (i.e., direct support to the PWGs).  IRI leads the components related to public opinion research and IPOR in coordination with NDI. 4. Current Status of Activities As indicated above, PPDP has so far engaged most of the political parties in seven phases of the PDC (excluding the first step, which is to get buy in from the party leadership) and two supplementary activities (TFP and multi-party roundtables). By January 2013, 10 party units of eight parties had been fully engaged in the first seven phases of the PDC, whereas one party had been partially engaged (Table 4). The TFP had been conducted in two provinces, where it engaged 10 party units from nine parties (Table 4). TABLE 4: POLITICAL PARTY UNITS REPRESENTED IN THE POLICY DEVELOPMENT CYCLE AND TRAINING FELLOWSHIP PROGRAM Party Policy Development Cycle TFP IIW PWG RU Training Member Survey Opinion Research CSO Forum Policy Drafted Province 1 Party 3 X X X X X X X Party 7 X Party 1 X X X X X X X Province 2 Party 3 X X X X X X X X Party 7 X Party 1 X X X X X X X Party 4 X X X X X X X X Party 5 X X X X X X X X Province 3 Party 3 X X X X X X X X Party 7 X X Party 10 X X X X X X X X Party 9 X X X X X X X Party 8 X X X X X X X X 66 Party Policy Development Cycle TFP IIW PWG RU Training Member Survey Opinion Research CSO Forum Policy Drafted Party 6 X X X X X X Province 4 Party 3 X X X X X X X Party 7 X Party 1 X X X X X X X Party 2 X X X X X X Total 14 14 13 13 14 11 10 No. of party members involved 924 140 27 110 110 23 Taking into account the 2013 elections and the possibility of polls being held early, the parties determined that the agenda of multi-party roundtables in the lead up to and in the aftermath of the elections will focus on electoral issues and their impact on political party participation in the electoral process. The multi￾party roundtables and a portion of public opinion poll briefings involved both the central and provincial leaders of participating parties. Participation in the multi-party roundtables is reported in Table 5. A total of 65 party members have participated in the roundtables. TABLE 5: MULTI-PARTY ELECTION RELATED ROUNDTABLES Party Round￾table 1 Round￾table 2 Approved Consensus Elections Recommendations Round￾table 3 Approved Consensus Code of Conduct Recommendations Party 3 X X X X X Party 7 X X X X 131 Party 1 X X X X X Party 4 X X X X X Party 10 X X X X X Party 9 X X X X X Party 5 X X X X X Party 11 X X X X X Party 8 X X X X X Party 2 X X X X X 131 Although Party 7 did not formally endorse the multi-party roundtable consensus Code of Conduct recommendations prior to their being submitted to the Election Commission of Pakistan (ECP), the party did endorse many of the multi-party roundtable recommendations as incorporated in the official Code of Conduct comments to the ECP submitted by the Senate Special Committee on Election Issues. 67 Party Round￾table 1 Round￾table 2 Approved Consensus Elections Recommendations Round￾table 3 Approved Consensus Code of Conduct Recommendations Party 6 X X X X X Party 12 X X X X X Total 12 12 12 11 12 II. RATIONALE FOR EVALUATION A. Purpose of Evaluation The evaluation will cover the period from July 2011 to February 2013 and take place prior to USAID’s decision on whether to extend the project beyond the current two-year funding period. The evaluation results will contribute to this decision and, if USAID decides to continue the project, the results will help form recommendations about changes that could improve future performance. Finally the evaluation will capture lessons learned that might be relevant to designing and implementing other projects in USAID/Pakistan’s portfolio or the agency’s portfolio more broadly. The evaluation objectives are to:  assess the effectiveness of the overall project approach and of individual activities in contributing to the project’s intermediate results132 and the efficiency with which the partners are implementing the approach and activities; and,  develop recommendations for improving project design and implementation. B. Audience and Intended Use The primary audience for the evaluation includes: (i) USAID decision-makers; and (ii) project leaders and implementers at NDI and its partner organizations. The following indicates USAID/Pakistan’s expectations for other staff participation in the evaluation (in addition to the Democracy and Governance Team):  USAID staff: Program Office’s Performance Management Unit, especially the COR for the MEP contract to monitor the evaluation activities and MEP’s role in monitoring the team’s progress, supporting their logistic needs and implementation of the evaluation; and,  USAID partners (NDI, IRI and SPO): The prime implementer NDI and its sub-grantee IRI are key players in providing information about project activities, participants’ details and protocols and sensitivities involved in approaching the political parties. The beneficiaries, i.e. political parties, will also have a key role in providing information for evaluating the program. 132 The focus is on intermediate results rather than objectives because the project’s results framework provides indicators for intermediate results but not for objectives. Most if not all of these indicators are output indicators. 68 C. Evaluation Questions The evaluation focuses on how effectively the partners are implementing the project. It examines this broad question from the perspective of implementation of the planned approach and implementation of individual activities. The evaluation does not address the relevance of the planned approach, it addresses only how effectively the partners are implementing the planned approach. The evaluation will interpret findings in the context of the current status of implementation. The first two evaluation questions address the effectiveness of implementation while the third asks for overall recommendations. Because the third question draws on findings and conclusions from the first two, there is not a particularly clear distinction between the first two questions and the third. Data collection and data analysis methods for answering these questions are elaborated below and summarized in the Getting to Answers (G2A) table in Annex 2, which the evaluation team will review and finalize. 1. Evaluation Question 1: How effectively are the partners implementing the planned approach and what changes to the approach might improve the efficiency with which the overall set of activities contribute to achieving intended intermediate results? Explanation: This question focuses on the overall project approach and not on individual activities within the approach. The planned approach describes a coordinated set of activities which, if appropriately implemented (e.g., sequence, scale, participants), will contribute to achieving higher level results. This question does not address the relevance of the planned approach per se but only whether the partners are implementing the planned approach effectively. It asks whether the partners are implementing activities in a manner (e.g., sequence, scale) that is consistent with the planned approach and with achieving results in an efficient manner. Efficiency does not mean a rigorous analysis of cost efficiency but, rather, whether implementation is consistent with using resources efficiently to achieve higher level results (i.e., not outputs). For example, the question might address whether the timing and scale of a particular activity is consistent with the timing and scale of subsidiary and dependent activities in the hypothesized development process. The evaluation will require assessing the extent to which specific activities are necessary to achieving intended intermediate results. 2. Evaluation Question 2: How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results? Explanation: This question focuses on how well the partners are implementing individual activities. It does not address the relevance of particular activities to achieving results. Answering the question will involve examining how the partners are implementing each activity, their rationale for the implementation approach (e.g., location/venue, method, choice of participants), and the efficiency of the approach. Efficiency does not mean a rigorous analysis of cost efficiency. Instead, it means whether the partners could implement the activity in a manner that uses resources more efficiently to accomplish the same, or a similar, result. For example, for training activities, the evaluation might examine whether the training engages the appropriate people, uses effective methods, uses an appropriate approach (e.g., direct training or training of trainers), or occurs in the right location. 3. Evaluation Question 3: How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? Explanation: This question draws from the previous questions to craft recommendations for enhancing the project’s overall contribution to achieving intermediate results and the efficiency with which it does so. Recommendations may include modifying (e.g., adding, dropping, scaling, changing targeted participants) individual project activities and aspects of the approach. 69 III. DESIGN AND METHODOLOGY A. Evaluation Process The evaluation is highly time-sensitive and field work needs to be completed by the third week of March 2013.USAID has committed to conducting an evaluation of this activity prior to making a decision on whether to extend its length of time but with election campaigns set to begin soon, political party members, the key participants in the evaluation, will be less available and participation rates could diminish significantly. Moreover, as the elections approach, the timing of specific meetings could become uncertain and subject to change or cancellation at short notice. The evaluation process reflects the need to complete as much of the field work as possible before these factors result in serious limitations in primary data collection. The first step of the evaluation process (which will take about one week) will entail: a. A review of documents, including those listed below; b. Finalizing a focus group discussion (FGD) guide and key informant interview (KII) instrument which MEP will draft and share with the team; and, c. Preparing a draft of the initial sections of the evaluation report. To ensure that the evaluation is completed appropriately within the time constraints, prior to arriving in Pakistan, team members will interact with each other electronically (including email and web-based conference calls) and contribute to the draft of the initial sections, which will be prepared by the team leader. This exercise is aimed at ensuring that team members: a. Read available information to familiarize themselves with the project, its context and intended results before the team assembles in Islamabad and starts an in-depth review of the SOW; b. Produce an initial draft of the evaluation report section on project context, with credible references; and, c. Relate the above-mentioned information specifically to the SOW and the technical and editorial requirements of report writing. MEP will make the documents available to the evaluation team through a secure cloud-based file storage environment and provide a note that guides team members on the relevance of various documents. Team members will review the documents before assembling as a team in Islamabad and ensure that they are familiar with all aspects of the project that are described in available documents. The draft document will include, at a minimum, the following sections, which are further discussed in the Reporting Guidelines sections of this SOW: Acronyms, Project Summary (which will include the project goal, objectives and Table 1 of this SOW), Evaluation Purpose and Evaluation Questions, Project Background, Evaluation Methods and Limitations, Bibliography, and an annex that reproduces the Evaluation Statement of Work. The initial sections document will be prepared in the style and format of the draft final report. MEP will provide an example of a formatted and branded report to the team. In view of the time constraints, one component of primary data collection will be initiated before the arrival of the lead evaluation team members in Islamabad. For this component, two or three long-term MEP staff led by the Director of Evaluation will conduct FGDs and KIIs with members of various political parties who have participated in the PDC. The scope of this exercise is outlined below. The MEP team will prepare FGD summaries and KII notes as well as provide briefings for the lead evaluation team members to ensure that all members of the team have sufficient understanding of the data from this component of the evaluation and can adequately use the information for analysis. 70 As soon as the evaluation team comes together in Islamabad, MEP will facilitate a Team Planning Meeting. This meeting is a process that includes the following main tasks: a. In-depth review of the evaluation SOW, including the key questions, methodology and G2A; b. Identify any changes that may be required in the SOW with the approval of MEP and USAID; c. Discuss USAID and MSI evaluation standards and requirements, including, in particular, the evaluation logic that links findings, conclusions and recommendations; d. Review the initial sections drafted by the team leader; e. Determine roles and responsibilities within the team and between the team and MEP staff; f. Plan field work with inputs from MEP staff responsible for security and travel; g. Develop instruments for the additional data collection components of the evaluation (additional group interviews and key informant interviews); and, h. Meet with USAID and the implementing partners. Following the team meeting, the team will begin field work for collecting data, with the data collection methods outlined below. Team members will then transition into report writing under the direction of the team leader and with continuing guidance from MEP staff. During the report writing process, the team will discuss initial findings and tentative conclusions with USAID and the implementing partner. Implementers may provide additional information, help correct factual errors in the evaluation findings and provide feedback on conclusions. The team will incorporate feedback from USAID and the implementing partner in the course of preparing a draft report. MEP will carry out an internal review of the draft report, after which the team will prepare a second draft which, after another review within MEP, will go through an editing and branding process. MEP will submit the edited and branded draft final report to USAID for review, which will share it with the implementing partner. The team leader, with inputs from MEP, will then prepare a final report that incorporates comments from USAID and the implementing partner. Full breakdown of the work schedule can be found in Section VI.B B. Existing Data PPDP has a comprehensive database which includes key project documents, baseline and monitoring information and a number of background articles. Documents on the operations of the project include but are not limited to:  Logical Framework Matrix, PMP, and narrative M&E Plan.  Work plans for Year 1 and Year 2 of the project.  Performance Indicator Reference Sheet (PIRS).  Documents on the Policy Development Index.  Documents on the Training Index.  Project Performance Reports: Quarterly, Monthly and Activity Reports.  PPDP Participants List by Activity.  National/Provincial Survey and Focus Group Reports  IPOR Organizational and Technical Development Index  Party-wise Program Material. PPDP collected baseline data through questionnaires administered to party members who participated in the IIW, PWG, CSO forum and TFP. It maintains this information in Excel spreadsheets, each of which has several (15-27) tables and charts reporting the information collected from the participants. The following baseline data and data collection forms are available: 71  IIW Province-wise Pre-Workshop Survey Data.  Political Working Group Meeting: Baseline Survey  Survey Data for PWG Meetings – All parties.  Baseline Data: CSO Forum Survey Data for Punjab, Balochistan and KP.  Training Fellowship Program Data on Sindh.  Civil Society Organization Forum, Survey Form.  Issue Identification Workshop Survey Form.  Training Fellowship Program: Pre-Assessment Form and Post-Assessment Form for KP and Sindh. General background documents include the following:  United States Institute of Peace. Progress in Peace-building: Pakistan. January, 2011.  United States Institute of Peace. Peace Brief: Correcting America’s Image Problem in Pakistan. August 20, 2010.  United States Institute of Peace. Special Report: Who Controls Pakistan’s Security Forces. December, 2011.  National Democratic Institute. A Compilation of Political Party Statutes, 2011 (http://www.ndi.org/node/19347).  Statute and Internal Rules of PPP.  Hassan Abbas. Asia Society Pakistan 2020 Study Group Report: Pakistan 2020. A Vision for Building a Better Future. May 2011.  International Crisis Group. March 30, 2011. Reforming Pakistan’s Electoral System.  Center for Global Development. Beyond Bullets and Bombs: Fixing the U.S. Approach to Development in Pakistan. June, 2011.  International Crisis Group. Islamic Parties in Pakistan. December 12, 2011.  International Republican Institute. Public Opinion Research and Democracy Promotion, 2006.  International Republican Institute. Handbook on Monitoring and Evaluation. 2012-2013. C. Data Collection Methods The evaluation questions focus broadly on assessing the effectiveness and general (not financial) efficiency of the overall project approach and activities. Neither of these avenues of inquiry lend themselves well to quantitative analysis nor is much relevant quantitative data available. Therefore, the evaluation will rely largely on project records (including baseline information) and primary qualitative data collected through FGDs, group interviews and KIIs described below. After the Team Planning Meeting, the evaluation team will conduct KIIs with the key staff of PPDP and133:  Group interviews with 2-4 provincial leaders (from the same party) at a time, focusing on their participation in the PDC and multi-party roundtables, the utilization of IRI-IPOR public opinion research and communication training by their party and the overall PPDP approach.  Group interviews with one or two members each of the RUs of all parties; 133 The protocol for inviting party members to FGDs, KIIs and group interviews is laid down in Annex 4. 72  KIIs (one-on-one) with central leaders, focusing on their participation in the PDC and multi-party roundtables, the utilization of IRI-IPOR public opinion research and communication training by their party and the overall PPDP approach; and,  KIIs with Training Fellows. Prior to the evaluation team’s arrival in Islamabad, a team from MEP will conduct FGDs with party members who have participated in IIWs. The criteria for selection of parties and locations for FGDs are as follows and applied with the help of information summarized in Annex 3:  The number of FGDs in which a party is included should be in approximate proportion to the number of members who have participated in IIWs.  The FGDs as a whole should address the regional and ethno-linguistic diversity reflected in the participants of IIWs.  The FGDs should include participants from the early, middle and later phases of PPDP, starting with November 2011 and going up to December 2012.  The FGD location should be the home district of at least 8-12 IIW participants. Each FGD is expected to involve 8-12 IIW participants selected through stratified random sampling among each party’s IIW participants, including the following:  At least six young participants (age 35 years or less);  At least four women; and,  At least 3-5 PWG members who have participated in the PDC up to the policy drafting stage. The FGDs will be short and focus on the IIW. At the end of each FGD, the MEP will hold KIIs with three of those grassroots members who attend the FGD and have participated in subsequent phases of the PDC, at least up to the policy drafting stage. The overall plan for FGDs, group interviews and KIIs is presented in Table 7. D. Data Analysis Methods The evaluation team will employ rigorous analytical methods appropriate to the various types of qualitative data it collects. For group interviews and semi-structured KIIs with individuals, the team will summarize responses around themes relevant to the evaluation questions – quantitatively when possible or appropriate. For the FGDs, which generate a relatively large quantity of consistent data, the MEP team will employ a more structured approach to analysis – identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses. E. Methodological Strengths and Limitations A quantitative, survey-based approach to data collection was considered but not found feasible for this evaluation because:  Party leaders and members, particularly the grassroots members, who represent the largest number of project beneficiaries, are scattered throughout the country and a survey would have been extremely time-consuming and expensive. 73  There is a need to complete the field work for the evaluation before the parties become involved with their election campaigns, as discussed above. The main strength of the methodology, which allows for strong validation of findings, is the combination of various methods of data collection and the diverse sources of information from all provinces of the country. As illustrated in Table 7:  Primary data are drawn from five groups of project participants, including three levels of the parties and two groups which are the beneficiaries of training provided by PPDP.  The sampling plan covers all four provinces and 10 political parties.  Data collection uses three different methods of qualitative research. Perhaps the greatest limitation of the evaluation approach is the substantial reliance on data collected from project participants and partners which may lead to a biased view of the project. TABLE 7: SAMPLING PLAN FOR FOCUS GROUP DISCUSSIONS, GROUP INTERVIEWS AND KEY INFORMANT INTERVIEWS Method of Collecting Data from Project Participants Total FGDs Group Interviews KIIs Participants IIW participants Include 8-12 in each FGD. Average 10, to be included: 90 PWG participants To be included 27, Include 1 in each KII. Provincial leaders To be included 30, Include 2-4 in each interview. Average 3. Central leaders To be included 13 Include 1 in each KII. Researchers To be included 20, Include 2 in each interview. Training Fellows To be included 9, Include 1 in each KII. Location 1 Party 1 Provincial leaders: Parties 1, 2 Central leaders: Parties 1, 2 38 Party 2 Researchers: Parties 1,2,3 Training Fellows: Party 7 Location 2 Party 3 Provincial leaders: Parties 3,4,5 Central leaders: Parties 3 (2), 4 (2), 5 46 Party 4 Researchers: Parties 3,4,5 Training Fellows: Parties 3,4,5 Location 3 Party 3 Provincial leaders: Parties 1,3 Central leaders: Parties 1,3 35 Party 1 Researchers: Parties 1,3 Training Fellows: Party 7 Location 4 Central leaders: Parties 1,6,10 3 Location 5 Party 10 Provincial leaders: Parties 8,9,10 Central leaders: Parties 8,9,10 45 Party 9 Researchers: Parties 8,9,10 Training Fellows: Parties 6, 8,9,10 Location 6 74 Method of Collecting Data from Project Participants Total FGDs Group Interviews KIIs Party 6 10 Total: 9 FGDs 7 parties 5 locations Total: 10 with provincial leaders (8 parties; 4 locations) 11 with Research Units (9 parties, 4 locations) Total: 27 with PWG members 13 with central leaders (10 parties; 5 locations) 9 with Training Fellows (8 parties, 4 locations) 177 IV. EVALUATION PRODUCTS A. Deliverables The evaluation will produce the following deliverables: 1. A draft Statement of Work (SOW) for review and comment by USAID. The SOW will define the evaluation questions, describe the evaluation approach, specify the evaluation deliverables, and describe the management of the evaluation; 2. A final SOW consistent with USAID standards for SOWs; 3. Proposed instruments for FGDs, group interviews and KIIs; 4. A debriefing discussion with USAID and the implementing partner on initial findings; 5. A draft evaluation report; 6. A presentation to USAID and the implementing partners on the draft report, 7. A final evaluation report consistent with USAID standards for evaluation reports; and, 8. All data collected by the evaluation provided in an electronic file in easily readable format, organized and fully documented for use by those not fully familiar with the project or the evaluation. Field work will not start until USAID has approved the final SOW and commented on the instruments. B. Reporting Guidelines The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.2.8 (Documenting Evaluations), reproduced in Annex 4. The evaluation report will follow the structure given below (the section titles and order are illustrative):  Title page  Table of Contents  List of any acronyms, tables, or charts (if any)  Acknowledgements or Preface (optional)  Executive Summary (ideally not to exceed 3-4 pages)  Evaluation Purpose and Evaluation Questions (2-3 pages) 75  Project Background (1-3 pages). This information provides important context for understanding the evaluation purpose, questions, methods, findings and conclusions and includes: o an outline of the project or program being evaluated; o the original problem or challenge the project is designed to address; o underlying development hypothesis or causal logic; o the results framework, if available; and, o a description of the approach and current status of the project.  Evaluation Methods and Limitations (1-3 pages), describing in detail the evaluation methods and why they were chosen, with additional information provided in the annexes, if so required.  Findings and Conclusions – This section will include findings and conclusions for each evaluation question. If there is a large number of findings, there will be a synthesis of findings for each question that establishes the connection with the conclusions that follow.  Main Conclusions and Recommendations – This section will recapitulate the main conclusions, including those that form the logical basis for recommendations. It will highlight a few key recommendations, or clusters of recommendations, that include actionable statements of what remains to be done, consistent with the evaluation’s purpose, and based on the evaluation’s findings and conclusions. It will provide judgments on what changes need to be made for future USAID financial and cooperative development programming. This section will also recommend ways to improve the performance of future USAID programming and project implementation; ways to solve problems this project has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by management.  Annex o Evaluation Statement of Work o Evaluation Methods and Limitations o Data Collection Instruments o Bibliography of Documents Reviewed o List of individuals and agencies contacted134 o Disclosure of Any Conflicts of Interest o Statement of Differences (only if applicable) o Evaluation Team Bios 134 This information will be removed from the report before USAID places it in the public domain. 76 V. TEAM COMPOSITION A Team Leader/Evaluator (internationally recruited) will lead a three-person team to conduct this mid￾term performance evaluation in accordance with the USAID Evaluation Policy and directives. S/he is expected to possess at least fifteen years of experience in international development, experience of evaluating USAID (preferably governance) projects, short-term consulting experience in Pakistan or neighboring countries, a master’s degree in political science or another relevant field and demonstrated leadership and report writing abilities and communication skills. An internationally recruited Governance Specialist will be the second member of the team. S/he is expected to possess at least fifteen years of relevant experience in implementation, academic, research or consulting positions, in-depth experience of political party and governance issues, short-term consulting experience in Pakistan or neighboring countries, a master’s degree in political science or another relevant field and demonstrated report writing abilities. An Organizational Development (OD) Specialist will be the third team member. S/he is expected to possess at least ten years of relevant experience in implementation, academic or consulting positions, in￾depth experience of (preferably political party) organizational development issues in Pakistan, short-term consulting experience in Pakistan or neighboring countries, a master’s degree in a relevant field and demonstrated facilitation skills and report writing abilities and fluency in English and Urdu. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. VI. EVALUATION MANAGEMENT A. Logistics In terms of logistics, this assignment requires:  travel to the locations identified in Table 7 by the three team members and the MEP staff responsible for the evaluation;  access to project-related information and key informants representing various stakeholders, for which the good offices of USAID and PPDP will be required; and,  organizing and reporting on the FGDs. The evaluation team will work under contract to MEP, which will be responsible for all travel, lodging, and other arrangements related to the team’s work. MEP has responsibility for identifying all key informant and discussion group participants, inviting them to participate, organizing meeting schedules, arranging venues, and handling any necessary transport and lodging arrangements for participants. Prior to MEP issuing invitations, USAID will send a written notice to party leaders and other key participants to inform them about the evaluation and introduce them to MEP. 77 B. Scheduling The evaluation is expected to start in mid-February with FGDs and KIIs covering grassroots party members. The evaluation team is expected to assemble in Islamabad circa March 4. A discussion of initial findings with the evaluation team could be arranged on or around April 15. The draft report is expected to be submitted by May 10, 2013. A presentation will be given by MEP after the submission of the draft report. A tentative schedule for the evaluation is shown in the form of a Gantt chart in Table 8. The schedule can be affected by circumstances beyond the control of USAID and MEP, such as delays in obtaining visas, unforeseen loss of time due to security factors, general elections and public holidays. TABLE 8: EVALUATION SCHEDULE Activity Feb Mar Apr May W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 Initial FGDs and KIIs Pre-arrival team work Team Planning Meeting Field work – Additional KIIs Report writing by team Internal review, revision Branding and editing Submission of draft report X C. Budgeting Table 9 reports estimates of the Level of Effort (LOE) of the team. TABLE 9: LEVEL OF EFFORT OF TEAM MEMBERS Tasks Level of Effort (days) Team Leader/ Evaluator (STTA) Gover￾nance Specialist (STTA) OD Specialist(STTA) Evaluation Manager (LTTA) Evaluation Co￾Manager (LTTA) FGDs 16 16 Pre-arrival: 3 4 2 3 3  Review of documents 2 1 1 3 3  Review of FGD guide 1 1 1  Draft of initial chapters 2 Team Planning Meeting 3 3 3 3 3 Field work 15 15 15 7 7 Draft final report 18 18 18 Internal review and revision 3 3 Final report 2 Presentation to USAID and implementers 2 Travel 4 4 Total 48 44 38 34 29 78 SOW Annex 1: PPDP Deliverables Objective 1: Political parties conduct their own research, analysis and training for the formulation of increasingly responsive and informed platforms and policies; and parties contribute more actively and effectively in the policy-making of government. Key Deliverables – Expected Performance Milestone for First Cycle:  NDI will conduct up to 12 single-party meetings with up to 12 political parties to discuss the PPDP; all parties commit to participation in the Policy Development Cycle.  IRI completes eight opinion poll reports; IRI and IPOR present polling results to leadership from 12 political parties NDI has a representative at each poll briefing.  NDI assists parties in the formation of up to 27 PWGs (including 15 groups based on the additions proposed in year two).  NDI conducts a first consultative meeting with up to 27 PWGs to provide training on use of research in inclusive policy design and drafting. (This includes the 15 groups based on additions proposed in year two). NDI conducts a second consultative meeting with up to 27 PWGs to analyze research, draft policy proposals and design polling questions.  NDI will assist 10 parties to conduct policy conventions with representation from PWG members.  SPO will organize and facilitate up to four CSO forums. The 27 PWGs each have consultations with issue experts.  A national-level public opinion poll will be conducted each quarter. Briefing meetings for each poll will be conducted with up to 12 political parties, USAID, research units and PWGs.  A provincial-level public opinion poll will be conducted each quarter. Briefing meetings for each poll will be conducted with up to 12 political parties, USAID, research units and PWGs.  IRI conducts meetings with up to 12 political parties to discuss research units and the Research Training Academy (RTA). Parties establish up to16 research units.  IRI will conduct briefing meetings on eight polls with the 12 parties in PPDP.  IRI and IPOR develop six training modules and conduct six RTA sessions.  NDI develops four training modules for Training Fellowship curriculum. NDI invites participants and conducts four modules each four-month period (one week per month). A total of approximately 30 individuals will participate in each fellowship.  NDI obtains commitment from up to 12 political parties to participate in the Training Fellowship program and to use Training Fellows for party member training following the four-month training period. NDI will launch the Training Fellowship program in year one. Objective 2: Establishment of an independent opinion research facility that adheres to international research standards. Key Deliverables – Expected Performance Milestone for First Cycle: 79  IRI conducts one, multi-day technical research training for IPOR staff every quarter. IPOR applies certain technical skills with guidance from IRI during the conduct of each quarterly public opinion poll.  IRI conducts 16 focus group discussions per year.  IRI and IPOR develop a business development plan that is reviewed and readjusted by IRI on a quarterly basis. IRI conducts one organizational development training for IPOR staff every quarter.  Each quarterly research training provided by IRI to IPOR includes a component on delivery and dissemination of polling results to PWGs and research units. Objective 3: Political parties communicate more effectively with their members, constituents and the general public at the national and local levels, both in terms of articulating messages and aggregating and responding to concerns, requests and ideas. Key Deliverables – Expected Performance Milestone for First Cycle:  NDI organizes and conducts up to 27 IIWs.  Up to 900 Grassroots Leaders gather feedback from rank-and-file party members regarding issues and policies discussed at the IIWs and submit feedback to PWGs via electronic or paper submission.  IRI will conduct strategic communications training sessions with senior officials of each party at the central and provincial level. Up to 12 parties will receive this training.  NDI will conduct communications training sessions for the grassroots party members of up to 12 parties. Objective 4: Parties demonstrate movement towards implementing internationally recognized standards for internal democracy and transparency. Key Deliverables:  Party by-laws and practices are beginning to reflect international standards for democratic parties.135  Party by-laws and practices comply with Pakistan’s Political Parties Order of 2002, with an emphasis on candidate selection procedures, participatory decision-making, democratic internal structures and greater inclusiveness of women and youth.  Expected Performance Milestone: NDI will conduct one multi-party roundtable and up to 12 single-party consultations136 with party leadership every quarter. NDI will assist interested parties in revising their by-laws and devising action plans for translating such changes into party practice. 135 The parties have informed PPDP that it is unrealistic to expect changes in by laws and party constitutions before the 2013 elections. 136 Single-party roundtables are expected to begin after the 2013 elections. 80 SOW Annex 2: Initial Getting to Answers (G2A) Matrix Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/ Selection Data Analysis Methods 1. How effectively are the partners implementing the planned approach and what changes to the approach might improve the efficiency with which the overall set of activities contribute to achieving intended intermediate results? Descriptive  Document review  FGDs  Group Interviews  KIIs  Review project documents and activity records to document relative progress on activities.  FGDs and KIIs with party members.   Group interviews with Provincial Leaders and Researchers from Research Units   KIIs with Central Leaders and Training Fellows.  KIIs with relevant staff of NDI, USAID, IRI, SPO, and IPOR.  Purposive/ Convenience sampling for FGDs  Purposive sampling for Group Interviews and KIIs Document the planned approach from document review and KIIs with project staff. Conduct content analysis of FGD transcripts. Summarize responses of KIIs and Group Interviews around themes to learn details of actual implementation and opinions about the effectiveness and efficiency of the approach. 81 Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/ Selection Data Analysis Methods 2. How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results? Descriptive  Document review  FGDs  Group Interviews  KIIs  Review project documents and activity records to document relative progress on activities   FGDs and KIIs with party members.   Group interviews with Provincial Leaders and Researchers from Research Units   KIIs with Central Leaders and Training Fellows.  KIIs with relevant staff of NDI, USAID, IRI, SPO, and IPOR  Purposive/ Convenience sampling for FGDs  Purposive sampling for Group Interviews and KIIs Document implementation of activities from document review and KIIs with project staff and participants. Conduct content analysis of FGD transcripts. Summarize responses of KIIs and Group Interviews around themes to learn details of actual implementation and opinions about the effectiveness and efficiency of the individual activities. 82 Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/ Selection Data Analysis Methods 3. How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? Descriptive  Document review  FGDs  Group Interviews  KIIs  Review project documents and activity records to document relative progress on activities   FGDs and KIIs with party members.   Group interviews with Provincial Leaders and Researchers from Research Units   KIIs with Training Fellows and Central Leaders.  KIIs with relevant staff of NDI, USAID, IRI, SPO, and IPOR  Purposive/ Convenience sampling for FGDs  Purposive sampling for Group Interviews and KIIs Synthesize results from evaluation questions 1 and 2 to craft overall recommendations. 83 SOW Annex 3: Participants of Issue Identification Workshops ANNEX TABLE 1: NUMBER OF PARTICIPANTS IN ISSUES IDENTIFICATION WORKSHOPS, BY PARTY, PROVINCE AND LEVEL IN PARTY Party Province/Region Number 2 Number 1 Number 3 Number 4 Number 5 Total Percent Note Party 3 61 59 61 75 4 260 28% A Grassroots 41 48 39 57 1 186 72% Divisional 15 0 8 4 1 28 11% Provincial 1 10 14 14 2 41 16% Central 4 1 0 0 0 5 2% Party 1 64 69 0 68 2 203 22% A Grassroots 43 29 36 1 109 54% Divisional 8 14 0 0 22 11% Provincial 13 23 30 0 66 33% Central 0 3 2 1 6 3% Party 4 77 1 1 0 1 80 9% C Grassroots 39 1 0 1 41 51% Divisional 12 0 1 0 13 16% Provincial 6 0 0 0 6 8% Central 20 0 0 0 20 25% Party 10 0 0 62 0 4 66 7% C Grassroots 44 2 46 70% Divisional 1 0 1 2% Provincial 15 0 15 23% Central 2 2 4 6% Party 9 0 1 62 0 4 67 7% C Grassroots 1 43 3 47 70% Divisional 2 0 2 3% Provincial 15 0 15 22% Central 2 1 3 4% Party 5 66 0 0 0 0 66 7% C Grassroots 38 38 58% Divisional 1 1 2% Provincial 20 20 30% Central 7 7 11% Party 8 0 0 61 0 7 68 7% C Grassroots 46 6 52 76% Divisional 0 0 0 0% Provincial 13 13 19% Central 2 1 3 4% 84 Party Province/Region Number 2 Number 1 Number 3 Number 4 Number 5 Total Percent Note Party 2 0 0 0 65 0 65 7% C Grassroots 23 23 35% Divisional 0 0 0% Provincial 12 12 18% Central 30 30 46% Party 6 0 0 43 0 6 49 5% C Grassroots 34 4 38 78% Divisional 7 1 8 16% Provincial 2 0 2 4% Central 0 1 1 2% Total 268 130 290 208 28 924 100% Percent 29% 14% 31% 23% 23% 100% Notes: A. Party share in the total is more than 20 percent. B. Party share in the total is 11 to 20 percent. C. Party share in the total is 10 percent or less. ANNEX TABLE 2: SUMMARY BY PROVINCE AND LEVEL IN PARTY Level of Leadership Province/Region Number Total 2 Number 1 Number 3 Number 4 Number 5 Number of Participants Grassroots 161 79 206 116 18 580 Divisional 36 14 19 4 2 75 Provincial 40 33 59 56 2 190 Central 31 4 6 32 6 79 Percent of Total Grassroots 60% 61% 71% 56% 64% 63% Divisional 13% 11% 7% 2% 7% 8% Provincial 15% 25% 20% 27% 7% 21% Central 12% 3% 2% 15% 21% 9% 85 ANNEX TABLE 3: NUMBER OF GRASSROOTS MEMBERS (DISTRICT AND LOWER LEVEL) WHO PARTICIPATION IN ISSUE IDENTIFICATION WORKSHOPS, BY PARTY AND PROVINCE Party Province/Region Number Total Percent Note 2 Number 1 Number 3 Number 4 Number 5 Party 3 41 46 39 57 1 184 32% 1 Date 10-Apr-12 29-Mar-12 27-Mar-12 25-Apr-12 A Party 1 43 29 0 36 1 109 19% 2 Date 11-Feb-12 29-Nov-11 24-Apr-12 B Party 4 39 1 0 0 1 41 7% 3 Date 14-Dec-11 C Party 10 0 0 44 0 2 46 8% 4 Date 10-Jan-12 C Party 9 0 1 43 0 3 47 8% 5 Date 6-Sep-12 C Party 5 38 0 0 0 0 38 7% 6 Date 13-Dec-11 C Party 8 0 0 46 0 6 52 9% 7 Date 18-Dec-11 C Party 2 0 0 0 23 0 23 4% 8 Date 22-Dec-11 C Party 6 0 0 34 0 4 38 7% 9 Date 29-Dec-12 C Total 161 77 206 116 18 578 100% 10 Percent 28% 13% 36% 20% 3% 100% Notes: A: Party share in the total is more than 20 percent. B: Party share in the total is 11 to 20 percent. C: Party share in the total is 10 percent or less. 86 ANNEX TABLE 4: POSSIBLE AND PROPOSED LOCATIONS FOR SINGLE-PARTY FOCUS GROUP DISCUSSIONS Province District Parties With 9 or More Participants in District Proposed FGDs Coverage Province 4 District 1 Party 3 Party 1 Party 2 Party 1 Party 2 Provinces: Province 2 District 2 Party 3 Party 1 Party 4 Party 3 Party 4 4 Province 2 District 7 Party 3 Party 5 Parties: Province 2 District 8 Party 5 9 Province 1 District 3 Party 3 Party 1 Party 3 Party 1 FGDs: Province 3 District 5 Party 10 Party 8 Party 9 Party 6 Party 10 Party 9 9 Province 3 District 6 Party 6 Party 6 Note: Some IIW participants gave their party's provincial headquarters as their address. 87 SOW Annex 4: Ground Rules for Inviting Political Party Representatives to Focus Groups and Interviews 1. MEP will select the participants of FGDs, KIIs and group interviews as described in this SOW and summarized in Table 7. 2. MEP will attach the list of participants to a draft letter which the Democracy and Governance Team (DG) of USAID will send out to the party focal persons with copies to PPDP. The letter will: a. Introduce MEP and the evaluation and request that selected party members meet the evaluation team on the dates and in locations proposed in the letter. b. Clarify that MEP will pay the travel and other expenses of participants whose presence is required for a day or longer according to specified criteria which are consistent with the standard practices of PPDP. c. Ask whether the focal person will personally coordinate the participation of party members with MEP or assign the task to other focal persons located in the cities where the evaluation team plans to visit. 3. To follow up and clarify relevant messages, DG will arrange meetings between the focal persons and a team representing DG, MEP and PPDP. 4. MEP will follow up with designated focal persons in order to ensure the participation of invited party members. 5. In case the designated focal person advises that the required number of invitees is unable to confirm participation, MEP will provide the focal person with additional names from the same category of party members (such as grassroots members, leaders, researchers and training fellows). 6. In case any problems arise in this process, MEP will inform DG accordingly and DG will propose appropriate steps. 88 SOW Annex 5: Reporting Guidelines 1. The evaluation report must represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action. Note: These guidelines are taken from ADS 203.3.2.8 (Documenting Evaluations) - http://transition.usaid.gov/policy/ads/200/203.pdf - which is based on Appendix 1 of USAID Evaluation Policy: Criteria to Ensure the Quality of the Evaluation Report. 89 ANNEX II: DATA COLLECTION INSTRUMENTS FGD GUIDE FOR PARTY MEMBERS WHO PARTICIPATED IN ISSUE IDENTIFICATION WORKSHOPS (IIWS) Questions Prompts General Questions/Ice Breakers 1. Can you tell us how you balance your political work with your occupational activities and social life? 2. In what ways is your party working with PPDP?  How do the two partners decide what to work on?  What is the value of PPDP to your party? Specific Questions 3. How were participants selected for the workshop?  What were the selection criteria?  Who did the selection?  What are your views about the selection process?  What are your views about the inclusion of women and youth in the workshop? 4. What can you tell us about the purpose of the Issue Identification Workshop?  How did PPDP explain the purpose?  How did your party explain it?  How do you understand the purpose after attending the workshop? 5. What was covered in the workshop?  What was your expectation about what the workshop will cover?  Did the workshop meet your expectations? 6. How did participants identify policy issues for discussion and what were the issues?  In what ways were you involved in the discussion?  To what extent did women and youth participate in the discussion?  What kind of issues came up for discussion?  To what extent were issues raised by women and youth? 7. How did participants establish priorities for policy development from the issues identified?  How did you decide which issues were more important than others?  Who decided the priorities?  What were the priorities you established as a group?  How can these priorities help your party prepare for the forthcoming election? 8. What role did PPDP play in the workshop?  What kind of help did PPDP provide in getting you all together?  What kind of advice and guidance did PPDP provide during the workshop? 9. What role did your party play in the workshop?  How did your party facilitate your participation in the workshop?  What kind of advice and guidance did your party give you for the workshop? 90 Questions Prompts 10. What kind of results have you seen or might see arising from the workshop?  For your party?  For yourself?  For people in the district or city where you live? 11. How useful did you find the workshop?  What did you like the most about it?  What did you like the least about it? 12. How could the workshop be improved?  What would you keep the same?  What would you change?  Can you decide as a group on the most important improvements?  What benefits might come up as a result of these improvements? GUIDE FOR KIIS OF MEMBERS OF POLICY WORKING GROUPS (PWGS) 1. What stages of the Policy Development Cycle have you participated in? 2. In what ways did you participate in the following steps of the Policy Development Cycle, and what was your experience? (a) the formation of the PWG (b) conducting the party member survey (c) using public opinion research (d) participation in Civil Society Organization (CSO) forum (e) policy drafting (f) getting the party leadership’s approval of the draft policy (g) participating in the party conference 3. Do you think that your work will be reflected in your party’s policy discussions? Do you think it will be reflected in the party manifesto? 4. Would you suggest any improvements in the way that the PWGs function? 5. What would strengthen your commitment for participation in the PWG? 91 GUIDE FOR GROUP INTERVIEWS OF RESEARCHERS 1. Why do you think your party selected you as a researcher to receive training from PPDP? 2. What kind of qualitative and quantitative research training have you received from PPDP and from whom specifically? 3. What did you find most useful and least useful about the training? 4. In what ways have you applied the research skills acquired through PPDP? 5. How do you think your party can benefit from the research skills you have learned through PPDP? 6. In which topics do you feel the need for further training to enhance your research skills? 7. Do you have any comments or suggestions for improving PPDP training for researchers? 8. Is there anything which is preventing you or your party from more fully participating in research training? GUIDE FOR KIIS OF TRAINING FELLOWS 1. What was your experience as trainer before PPDP? 2. Why do you think your party selected you as a Training Fellow for PPDP? 3. What kind of training have you received from PPDP? 4. What was most useful and least useful about the training? 5. In what ways have you applied the master trainer skills acquired through PPDP? 6. On average, how many hours do you spend per week on training, outside the NDI training program? 7. How do you think your party can benefit from the training skills you have learned through PPDP? 8. In which topics do you feel the need for further training to enhance your master trainer skills? 9. Do you have any other comments or suggestions for improving training fellowship program? 10. Is there anything which is preventing you or your party from more fully participating in TFP? 92 GUIDE FOR KIIS OF CENTRAL LEADERS AND GROUP INTERVIEWS OF PROVINCIAL LEADERS 1. In what ways is your party working with PPDP? 2. In which activities of the PPDP have you personally participated? (For example IIW, PWG, TFP, RUs, multi-party Round Tables, communication workshops … Note: Evaluation team will prompt participants) 3. Which activities have been most useful and least useful for your party? 4. How does your party develop its policies? 5. In what ways has your party changed the way it develops policies after participating in PPDP (e.g. opinions of grassroots party workers, attention to the priorities identified by women and youth, support among members, use of data for policy making)? 6. How has your party used the public opinion research brought to your attention by IRI, IPOR?; 7. How has your party used or plans to use the Training Fellows trained by PPDP? 8. Is there anything which is preventing you or your party from more fully participating in PPDP? 9. How could PPDP better support your party in policy development in the future? 93 ANNEX III: SOURCES OF INFORMATION LIST OF PERSONS INTERVIEWED In the interest of confidentiality, this list will be provided to USAID separate from this report. BIBLIOGRAPHY OF DOCUMENTS REVIEWED The evaluation team reviewed hundreds of documents made available by PPDP via Dropbox and in hard copy, as well as external reports and information. The key documents are listed below. NDI PPDP, “Technical Proposal,” May 2011 NDI PPDP, “Work Plan Year 1,” February 2012 NDI PPDP, “Work Plan Year 2,” April 2013 NDI PPDP, “Monitoring & Evaluation Plan,” August 2011 SPO, reports October 2011- January 2012 NDI PPDP, quarterly narrative reports, October 2011-December 2012 NDI PPDP, “Log frame,” November 2012 NDI PPDP, “Performance Monitoring and Evaluation Plan, target v. actual,” January 2013 IRI, Monthly Reports, October & December 2012 PPDP, “Program activity materials & reports (IIW, PWG, CSO forums, RTAs, poll briefings, TFP, multiparty roundtables, communication trainings, policy conferences)” NDI PPDP, “Key Data thru January 2013” NDI PPDP,” Activity Tracking Sheet,” February 12, 2031 NDI PPDP, PWG Pre-workshop surveys, February-June 2012 NDI PPDP, “Policy Tracker,” April 9, 2013 PPDP, Political party policy drafts, July 2012-February 2013 (28) NDI PPDP, “Policy Development Index” NDI PPDP; “Training Index” IRI/IPOR, “Activity Plan for Capacity Development of IPOR,” December 2012 94 IRI, “Reference Sheet: PPDP Baseline Data,” January 2013 IR/IPOR, “Trainer Report IPOR Business Development Training,” February 2013 IRI/IPOR, “Organizational Chart,” February 2013 IRI, “Research and Training Skills Index for IPOR,” February 2013 IRI, “Organizational and Management Skills Index for IPOR,” February 2013 IRI/IPOR, “External Tracker,” March 2013 IRI/IPOR, “Business Development Plan, working draft,” March 2013 NDI PPDP, “Report of the Senate Special Committee on Election Issues,” March 2013 http://www.senate.gov.pk/uploads/documents/1362572374_677.pdf 95 ANNEX IV: NDI STATEMENT OF DIFFERENCES ON THE EVALUATION OF USAID/PAKISTAN’S POLITICAL PARTY DEVELOPMENT PROJECT 96 • Evaluation Question 2: How effectively is the project implementing activities and what changes to the approach to implementing individual activities might enhance their contribution to efficiently achieving intended intermediate results? • EvaJuation Question 3: How, if at all, could the implementation approach be modified to enhance the contribution of project activities to achieving intended intermediate results and the efficiency with which it produces results? NDI's approach to responding to the evaluation report is two-fold: 1) to provide any corrections of factual errors the Institute has found in the draft report; and 2) to provide broad responses and clarifications to the main conclusions and recommendations. The following statements of difference reflect these objectives: Evaluation Finding or Recommendation 1. The findings related to leadership commitment to the P DC, the selection processes for various activities, communication between leaders and PPDP participants, leaders' use of public opinion research and their review of policy drafts show that the invol'vement of party leadership in PP DP is overall positive, while moderate. The appropriate involvement of leaders can contribute to overall program efficiency and effectiveness and increase the likelihood that parties adopt a more inclusive, research-driven approach to policy development ov1ir the long term. Statement of difference NDI agrees that the appropriate involvement of leaders can contribute to overall program efficiency and effectiveness and increase the likelihood tlhat parties adopt a more inclusive, research-driven approach to policy development over the long term. However, it is important to understand that the parties themselves take the lead in this aspect. Some party leaders are more engaged in the process than others. This leaves the responsibility of the day to day, detailed program engagement with the chosen liaisons. Additionally, as noted in the quarterly covering January to March 2013, PPDP liaisons - chosen by party leadership - are the main point of contact for program activities, particularly for any large-scale training, meeting or Policy Working Group (PWG) event. NDI meets with the liaisons lo describe the criteria and purpose of various activities, and then the liaisons are responsible for proposing responsive participant lists to NDI. NDT, with regular liaison input, makes the necessary logistical and other arrangements to ensure the participants are present at the activities. The liaisons or their equivalent senior-level designee regularly attend the various events. This has the dual benefit of the participants seeing their leadership's interest in their work under PPDP, but also ensures that participants actually attend and complete the entire activity. Additionally, NDI sends written reports to the party leadership and liaisons after each activity and periodic updates on party progress in the process overall. Evaluation Finding or Recommendation 2. The findings show that the JJWs effectively engaged a broad range of grassroots party members and party leaders in the process of identifying issues, and that the party member surveys served as a party building exercise. Both activities contributed to the aggregation of local concerns. The use of issue-specific secondary research data for the JJW discussion could have improved the effectiveness of the activity as far as informing policy development. The results of the party 2 97 98 99 100 101 ANNEX V: MEP RESPONSE TO NDI STATEMENT OF DIFFERENCES ON THE EVALUATION OF USAID/PAKISTAN’S POLITICAL PARTY DEVELOPMENT PROJECT 102 103 104 105 106 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523