CITIZENS’ VOICE PROJECT (CVP) FINAL PERFORMANCE EVALUATION JUNE 27, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International, a Tetra Tech Company. CITIZENS’ VOICE PROJECT (CVP) FINAL PERFORMANCE EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Citizens’ Voice Project Final Performance Evaluation iii CONTENTS Acronyms...................................................................................................................................... vi Project Summary .......................................................................................................................... 1 Executive Summary...................................................................................................................... 3 Findings and Conclusions........................................................................................................................................3 Key Recommendations............................................................................................................................................4 Lessons Learned........................................................................................................................................................4 Evaluation Purpose and Questions.............................................................................................. 5 Project Background ...................................................................................................................... 6 Project Context........................................................................................................................................................6 CVP Theory of Change ...........................................................................................................................................7 Intended Results........................................................................................................................................................7 Evaluation Methods and Limitations........................................................................................... 8 Evaluation Limitations............................................................................................................................................11 Interpretation of Findings.....................................................................................................................................12 Findings ........................................................................................................................................ 12 Findings for Evaluation Question 1: Relevance................................................................................................12 Findings for Evaluation Question 1: Effectiveness...........................................................................................16 Findings for Evaluation Question 2: Sustainability...........................................................................................26 Findings for Evaluation Question 3: Grants Under Contract ......................................................................30 Conclusions .................................................................................................................................. 35 Recommendations ...................................................................................................................... 37 Lessons Learned.......................................................................................................................... 40 Annexes........................................................................................................................................ 41 Annex 1: Evaluation Statement of Work..........................................................................................................41 Annex 2: Assignment Work Plan........................................................................................................................48 Annex 3: Data Collection Instruments..............................................................................................................65 Annex 4: Data Collection Team and Fieldwork Coordination ...................................................................85 Annex 5: List of Documents Reviewed.............................................................................................................86 Citizens’ Voice Project Final Performance Evaluation iv Annex 6: Conflict of Interest Statements.........................................................................................................87 Annex 7: Grants with Grant Value and Duration by Province and District ............................................88 Annex 8: Grants and Themes by Province.......................................................................................................94 Annex 9: Grants That Reported Communication Problems with CVP ....................................................96 Annex 10: Grants That Reported Payment Problems with CVP................................................................97 Citizens’ Voice Project Final Performance Evaluation v LIST OF TABLES AND FIGURES Table 1: Project Summary................................................................................................................................................1 Table 2: Grant Selection by CVP Thematic Area ......................................................................................................8 Table 3: Distribution of Interviews and Respondents by Data Source ..............................................................12 Table 4: Themes with Insufficient Grant Duration and Amount .........................................................................15 Table 5: Perceptions of the GUC Grant Mechanism..............................................................................................31 Table 6: CVP Payment Problems and Recommendations......................................................................................38 Figure 1: CVP Evaluation Respondents.........................................................................................................................2 Figure 2: CVP Results Framework.................................................................................................................................7 Figure 3: Number of Interviews (Individuals and Groups) by Respondent Type...............................................9 Figure 4: Number of Interviews (Individuals and Groups) by Respondent Type and Theme (n=197).......10 Figure 5: Distribution of Respondents by Type and Sex (n=622)........................................................................10 Figure 6: Methods of Selecting Grant Activities.......................................................................................................13 Figure 7: Sufficienty of Grant Amount and Duration (Percent of Grantees)....................................................14 Figure 8: Most Effective Awareness Activities By Respondent Sex.....................................................................16 Figure 9: Perceived Level of Citizen Participation by Respondent Type............................................................17 Figure 10: Participation in Grant Activities by Type of Participant (N=298)....................................................18 Figure 11: Citizen Participation by Respondent Type.............................................................................................19 Figure 12: Citizen Participation By Sex of Respondent..........................................................................................19 Figure 13: Most Effective Linkages (Grantees and Beneficiaries) (N=154)........................................................20 Figure 14: Most Effective LInkages by Theme...........................................................................................................21 Figure 15: Perceptions of the Level of Government Engagement by Respondent Type................................22 Figure 16: Perceived Level of Government Engagement by Theme....................................................................23 Figure 17: Change in Government’s Response to Citizens’ Needs.....................................................................25 Figure 18: Most Useful Trainings (Percent of Respondents).................................................................................26 Figure 19: Reported Sustainability of Grant Results................................................................................................28 Figure 20: What Respondents Would Do Differently............................................................................................30 Figure 21: Grantees’ Experience with the Grant-Making Process.......................................................................32 Figure 22: Main GUC Improvement Areas................................................................................................................33 Citizens’ Voice Project Final Performance Evaluation vi ACRONYMS AASR-GT Anjum Asim Shahid Rahman – Grant Thornton ABKT Association for Behavior and Knowledge Transformation AJK Azad Jammu and Kashmir AWP Assignment Work Plan CARD Coastal Association Research and Development COR Contracting Officer’s Representative CPDI Center for Peace and Development Initiatives CSO Civil Society Organization CVP Citizens’ Voice Project DELTA Development, Environmental, Legal Aid, Technical Support and Advocacy Associates DRS Development Resource Solutions DPG District Pressure Group DVC District Vigilance Committee FAA Fixed Amount Award FATA Federally Administered Tribal Areas FCR Frontier Crimes Regulation FGD Focus Group Discussion GEESE Global Educational, Economic, and Social Empowerment (Grant) GINI Governance Institutes Network International GMIS Grant Management Information System GUC Grants Under Contract GUC-FAA Grant Under Contract - Fixed Amount Award HRDN Human Resource Development Network HDO Hamdam Development Organization IEC Information, Education, and Communication KII Key Informant Interview KP Khyber Pakhtunkhwa M&E Monitoring and Evaluation MIS Management Information Systems MNA Member of the National Assembly MOU Memorandum of Understanding MPA Member of the Provincial Assembly MSI Management Systems International OSG Office of Stabilization and Governance OTI Office of Transition Initiatives, USAID PERFORM Performance Management Support Contract PMP Performance Management Plan PRDP Pak Rural Development Program RFA Request for Applications RTI Right to Information SHER Society for Human Empowerment and Rural Development SOW Statement of Work SPO Strengthening Participatory Organization TAF The Asia Foundation TDEA Trust for Democratic Education and Accountability TMA Tehsil Municipal Administration TPW Team Planning Workshop Citizens’ Voice Project Final Performance Evaluation vii UNDP United Nations Development Program USAID United States Agency for International Development WESS Water, Environment and Sanitation Society YMSESDO Yar Muhammad Samejo Educational Society Citizens’ Voice Project Final Performance Evaluation 1 PROJECT SUMMARY Table 1 summarizes basic information about the Citizens’ Voice Project (CVP). TABLE 1: PROJECT SUMMARY Title / Field Project information Contract number AID-391-C-11-00001 Contracting Officer’s Representative (COR) Start date May 27, 2011 Completion date May 2016 (with possible two-year extension) Location Pakistan Implementing partner(s) Trust for Democratic Education and Accountability (TDEA), prime contractor The Asia Foundation (TAF), subcontractor through December 2015 Anjum Asim Shahid Rahman – Grant Thornton (AASR-GT), subcontractor through April 2014 USAID/Pakistan Mission Strategic Framework objectives addressed Strengthened voice (policy advocacy and government oversight) Improved accountability (public-private connections) Improved capacity (organizational development and targeted trainings) Budget $45,000,000 Citizens’ Voice Project Final Performance Evaluation 2 The CVP evaluation team interviewed 622 individual respondents from 32 districts in 7 provinces/territories (Figure 1). FIGURE 1: CVP EVALUATION RESPONDENTS Citizens’ Voice Project Final Performance Evaluation 3 EXECUTIVE SUMMARY Management Systems International (MSI), through its Performance Management Support Contract (PERFORM), conducted the final performance evaluation of the $45 million USAID-funded Citizens’ Voice Project (CVP). The evaluation developed findings, conclusions, and recommendations from a desk review of 40 purposively selected grants and other grantee documents and group and individual interviews with 622 CVP beneficiaries, grantees, government officials, and civil society and media stakeholders. The evaluation focused on 1) the relevance of grant-funded activities to stakeholders and the target population, 2) the effectiveness of grant-funded activities in achieving project objectives, 3) the extent to which grant results are sustainable, and 4) the utility of the Grants Under Contract (GUC) model. Findings and Conclusions CVP is an effective program across most dimensions measured in this evaluation. 1. Relevance: Grant activities were highly relevant to stakeholder and beneficiary interests, mostly due to grantees’ consulting with local communities and relying on their local knowledge when selecting grant activities. However, 60 percent of respondents would have preferred grants with a longer duration to improve prospects for sustainable results. Activities were designed to be highly participatory, targeting disenfranchised groups and empowering citizens through awareness activities such as information campaigns and community-level meetings. 2. Effectiveness: Grants effectively facilitated engagement between citizens and the government. Ninety-four percent of respondents reported medium or high levels of citizen engagement in grant-funded activities, and 86 percent reported medium or high levels of government engagement, especially in information campaigns and community-level meetings. The government engaged across the spectrum of grant themes but was most engaged in three areas, local governance systems, implementation of the Frontier Crimes Regulation (FCR), and legislative governance. Four key USAID priority areas—education reforms, improving water rights, tax collection, and energy sector reforms—showed low to medium levels of government engagement. Grantees and beneficiaries valued linkages with the government above all other relationships, and for good reason: grantees with policy objectives frequently engaged with the government in order to achieve policy change. Seventy-four percent of respondents across respondent types believed grant-funded activities had improved governments’ openness to citizens, including improving governments’ responsiveness to citizens’ needs at the district level (67 percent of respondents). Seventeen grantees with policy change objectives contributed to policy discussions through various initiatives, and four achieved specific policy changes. 3. Sustainability: CVP encouraged sustainability from the start of the grant process by requiring grantees to submit sustainability plans. Capacity-building activities improved internal grantee operations and helped some grantees win follow-on funding from other organizations. Ninety￾two percent of respondents believed that grant results were sustainable due to grantee efforts to maintain linkages and awareness created because of grant activities. 4. Grants Under Contract: The two delivery mechanisms that supported CVP’s successful programming were 1) the GUC contractual grant framework between USAID and implementing partner Trust for Democratic Education and Accountability (TDEA) and 2) the Fixed Amount Award (FAA) grant funds disbursal system to grantees. GUC-FAA was an efficient contract mechanism for CVP and a model for similar future programming. The grant-making process was Citizens’ Voice Project Final Performance Evaluation 4 transparent and positive for most grantees; however, poor communication and an unreliable payment process frustrated some. Major communication problems led to several payment and approval delays and caused significant negative ripple effects across the grant lifecycle. Key Recommendations Based on the findings and conclusions, the evaluation team developed the following recommendations for future projects similar to CVP: 1. The implementing partner should initiate formal linkages with the government at the beginning of the implementation process. 2. The implementing partner of future similar projects should emphasize including high-impact activities, such as information campaigns and community-level meetings, in the request for applications (RFA) process and throughout the activity planning stage. 3. The implementing partner should increase the number of grantee capacity-building trainings relative to CVP and conduct the trainings early in the grant startup phase to build the capabilities of grantee organizations before grants begin. 4. The implementing partner should create an effective payment system to improve the grantee experience relative to that under CVP by increasing payments for initial milestone deliverables, reducing the total number of milestone deliverables, and creating an escrow account so the project can access additional program funds when necessary to continue project activities. 5. The implementing partner should develop a communications manual alongside a Grant Management Information System (GMIS) training presentation to improve communication between the project and grantees and deploy both during an off-site retreat with grantees. The evaluation team’s recommendations for USAID/Pakistan include: 1. If it chooses to extend CVP, USAID/Pakistan should continue to actively manage TDEA but should reduce the Mission’s role in approving grant awards. 2. If it chooses to extend CVP, USAID/Pakistan should both use the GUC mechanism with TDEA and specify that TDEA use the FAA process to award grants under the GUC mechanism for the two-year CVP extension. There are no practical alternatives. 3. In future similar projects, USAID/Pakistan could speed up the grantee review and approval process by developing a “core COR” team within USAID/Pakistan regional offices to work in tandem with designated implementing partner staff. Lessons Learned The evaluation team drew the following lessons from the evaluation exercise: 1. CVP laid the foundation for positive beneficiary and grantee experiences by selecting efficient grantees and engaging the local community early in the grant activity design phase. 2. Citizens who understand their rights, are informed about issues, and know how to petition the government to redress grievances are sustainable change agents in their communities. Citizens’ Voice Project Final Performance Evaluation 5 3. The government and the local community are the life support system of grant effectiveness and sustainability. Creating and reinforcing these linkages will improve prospects for grant success and future sustainability. EVALUATION PURPOSE AND QUESTIONS The final evaluation of CVP determines the extent to which grantees achieved the project’s goal of improving engagement between citizens and the government on priority program-supported initiatives designed to advance good governance objectives. Objectives included 1) strengthening citizen engagement in policy advocacy and state oversight; 2) enhancing grantee organizational capacity, transparency, and accountability; and 3) creating productive linkages between state and non-state actors for increased accountability. The evaluation assesses whether activities implemented by CVP were relevant, effective, and sustainable. It provides recommendations and lessons learned to help USAID/Pakistan design a future project with a similar objective to enhance civil society capacity to hold the Government of Pakistan accountable at the local, provincial, and national levels. The Evaluation Statement of Work (SOW) (Annex 1) specifies the following evaluation questions: 1. To what extent have the outputs and outcomes of the grants’ interventions contributed to improved engagement between citizens and government on priority program-supported initiatives to advance good governance objectives? Explanation: This question examines whether the grant activities supported under CVP were relevant to CVP’s objectives and took government priorities into consideration. It also examines the extent to which the activities were effective at improving engagement between the citizens and the government. It links with Outcomes 1 and 3: strengthening citizen engagement in policy advocacy and state oversight and creating productive linkages between state and non-state actors for increased accountability. 2. To what extent are the outputs and outcomes of the grants’ interventions able to be maintained (both financial and human capacity) beyond the life of the grant? Explanation: This question examines whether grants improved the organizational capacity and systems of the grantees to be more transparent and accountable during project management and the extent to which grant results were sustainable after grant closure. This question links with project Outcome 2: enhancing grantee organizational capacity, transparency, and accountability. 3. In what ways (both positive and negative) has the Grants Under Contract (GUC) model affected the implementation of interventions and their alignment with the stated objectives of the project? Explanation: This question examines the extent to which the GUC model was an effective mechanism for implementing CVP. It assesses whether the model is the best mechanism for a similar future project, and if yes, how it can be strengthened. The answer focuses largely on the Citizens’ Voice Project Final Performance Evaluation 6 GUC model and not specifically on the mechanisms the implementing partner, TDEA, used to award grants under the GUC model.1 PROJECT BACKGROUND The USAID/Pakistan Citizens’ Voice Project (CVP) started in May 2011 and is implemented by TDEA. The three-year contract was extended for an additional two years and is scheduled to end in May 2016. At the time of the evaluation, USAID was considering extending the project for an additional two years. CVP’s main objective is “to improve engagement between citizens and government on priority program￾supported initiatives in order to advance good governance objectives.”2 Grants support innovative ideas and high-quality, competitive applications from eligible organizations. As of October 31, 2015, 222 grants were completed and closed (activities completed and final milestone payment disbursed), and 48 remained open. Project Context ThCVP responds to Pakistan’s historical and current need for improved accountability across all levels of government. While an impoverished civil society was the result of early political stagnation and the failure of the government to deliver adequate public services, the passage of the 18th Amendment and the first democratic transition in the nation’s history have reinvigorated public awareness and desire for government accountability. Historical Political Stagnation and Poor Public Service Delivery The current political stagnation in Pakistan has its roots in pre-partition British rule. Under the British Raj, civil society was not sufficiently empowered to demand government accountability. After independence, the antiparliamentarian “viceregal” system replaced the colonial system of governance but did little to encourage additional civil society development. A succession of democracies between 1947 and 2013 attempted to introduce measures to improve accountability and strengthen civil society, but these attempts were stifled by three coups d’état and long periods of military dictatorship. As a result, Pakistan continued to suffer under corrupt governments and sub-standard delivery of public services across many sectors, including education, energy, tax collection, agriculture, and legislative reform. The 18th Amendment and Peaceful Democratic Transition Two watershed events drove Pakistan to address its longstanding governance problems. First, in April 2010, the National Assembly passed the 18th Amendment to Pakistan’s constitution. It curtailed the president’s authority to dissolve Parliament or to declare a state of emergency, thereby curbing executive abuses of power. The 18th Amendment also devolved 17 federal ministries to the provinces with corresponding financial, administrative, and political authority. Second, Pakistan achieved its first peaceful democratic transition in 2013 with the election of Prime Minister Nawaz Sharif, raising hopes for a sustainable parliamentary democracy. 1 The final sentence does not appear in the SOW or Assignment Work Plan (AWP) (Annex 2). The authors added it to reflect a discussion with USAID to clarify the question while writing the report. 2 CVP Performance Management Plan, p. 8. Citizens’ Voice Project Final Performance Evaluation 7 USAID/Pakistan’s Support for Civil Society USAID awarded CVP in order to improve the government’s responsiveness to citizens’ concerns. To encourage accountability through increased civil society participation, CVP initially worked with civil society to improve water rights and access to education. Over the life of the project, USAID/Pakistan and TDEA identified additional thematic areas, expanding the project’s focus into areas such as increasing citizens’ awareness of their rights and their participation in democratic processes (e.g., encouraging voter participation in the 2013 elections), promoting energy reforms, and supporting the evolution of civil society organizations (CSOs). CVP Theory of Change CVP documents did not articulate a clear theory of change. The project is based on the premises that 1) the links between the government and the governed in Pakistan are weak; and 2) the absence of public debate and discussion about key social, economic, and political issues often exacerbates the lack of public trust and confidence in government institutions.3 Intended Results Figure 2 illustrates the CVP Results Framework. FIGURE 2: CVP RESULTS FRAMEWORK Project Goal: Improved engagement between citizens and government on priority program-supported initiatives to advance good governance objectives Result 1: Strengthened citizen engagement in policy advocacy and state oversight Result 2: Enhanced organizational capacity, transparency, and accountability Result 3: Productive linkages between state and non-state actors for increased accountability Intermediate Results: 1.1. Increased number and diversity of citizen engagement in policy advocacy on priority themes 1.2. Increased number and diversity of citizen participation in policy processes and oversight of state institutions Intermediate Results: 2.1. Improved grantee systems for transparent and accountable project management 2.2. Improved grantee CSOs’ organizational capacity to implement activities related to citizens’ voice Intermediate Results: 3.1. Increased number and diversity of constructive public-private (state￾CSO) linkages 3.2. Increased responsiveness and accountability of state institutions to citizens 3 The evaluation team used the CVP Results Framework to develop the following theory of change: If engagement between citizens and the government is increased and the capacity of civil society organizations (CSOs) is strengthened, then good governance objectives will be advanced. Citizens’ Voice Project Final Performance Evaluation 8 EVALUATION METHODS AND LIMITATIONS The evaluation relies entirely on qualitative evidence. The evaluation team examined 40 closed CVP grants whose activities spanned 32 districts in 7 provinces/territories.4 The team randomly selected 2 grants from each of the 16 thematic areas and selected additional grants from USAID’s 5 priority thematic areas. When the randomized process selected grants to grantees located in inaccessible or highly insecure areas, the evaluation team replaced them with randomly selected grants awarded in safer, more accessible locations. The evaluation team also reviewed CVP’s contract, grant files, annual reports, performance management plan (PMP), and other pertinent documents.5 After collecting all respondent data and reviewing grantee documents, the team used MAXQDA qualitative analysis software to code themes in interview responses and identify patterns in the coded data. Table 2 shows the number of grants examined and interviews conducted for each of 16 thematic areas with the 5 priority themes USAID identified for the evaluation in bold. TABLE 2: GRANT SELECTION BY CVP THEMATIC AREA Grant thematic areas (priority themes in bold) Number of grants selected/number of grants in thematic area Number of interviews Advocacy for effective implementation of the Frontier Crimes Regulation (FCR) 2 / 3 10 Citizens’ voice for effective legislative governance (priority theme) 4 / 7 18 Citizens’ advocacy for implementation of 18th Amendment 2 / 7 12 Citizens’ awareness for higher female voter turnout 2 / 25 12 Citizens’ engagement and accountability for an effective local government system 2 / 6 11 Citizens’ oversight of municipal services 2 / 7 11 Citizens’ voice and accountability for youth development 2 / 8 9 Citizens’ voice and public accountability in tax collection sector (priority theme) 2 / 2 7 Citizens’ voice for effective grievance redress through the offices of ombudsmen 2 / 5 3 Citizens’ voice for free, fair, and peaceful elections 2 / 42 10 Citizens’ voice for independent, free, and responsible media 2 / 5 9 Citizens’ voice for strengthening transparency and accountability mechanisms 3 / 7 15 4 Gilgit-Baltistan was the only territory the evaluation team did not visit. Closed grants are grants for which grantees have completed all grant activities and TDEA has disbursed final milestone payments. As of October 31, 2015, 222 CVP grants were closed. The evaluation team’s selection of 40 grants represents 18 percent of all closed grants. 5 For a complete list of the documents the evaluation team reviewed, see Annex 5: List of Documents Reviewed. Citizens’ Voice Project Final Performance Evaluation 9 Grant thematic areas (priority themes in bold) Number of grants selected/number of grants in thematic area Number of interviews Education sector reforms (priority theme) 3 / 10 17 Energy sector reforms (priority theme) 5 / 12 21 Importance of local government systems 2 / 72 7 Improving water rights (priority theme) 3 / 4 18 Total 40 / 222 190 To evaluate CVP’s performance, the evaluation team conducted 42 focus group discussions (FGDs) with beneficiaries of grant-funded activities and 155 key informant interviews (KIIs) with grantees, government officials, civil society and media, USAID, TDEA, and AASR-GT.6 Figure 3 and 4 illustrate the distribution of interviews by respondent type and theme. FIGURE 3: NUMBER OF INTERVIEWS (INDIVIDUALS AND GROUPS) BY RESPONDENT TYPE 6 Annex 3 contains the interview instruments. Additional information on the field team responsible for gathering interviewee data and the fieldwork schedule can be found in Annex 4: Data Collection Team and Fieldwork Coordination. Annex 5 lists the documents the evaluation team reviewed for the evaluation. Citizens’ Voice Project Final Performance Evaluation 10 FIGURE 4: NUMBER OF INTERVIEWS (INDIVIDUALS AND GROUPS) BY RESPONDENT TYPE AND THEME (N=197) The evaluation team interviewed 622 individual respondents: 417 beneficiaries of grant-funded activities and 205 key informants. Key informants included 60 government stakeholders; 57 members of grantee organizations; 80 grant stakeholders from civil society and media; and 8 representatives from USAID and the sub-contractors, TDEA and Grant Thornton (Figure 5). Of the beneficiaries interviewed, 30 percent were women (124) or transgender (2). CVP encouraged the participation of disadvantaged groups from the beginning of Cycle. All successful applications included a plan for addressing the needs of underprivileged groups. FIGURE 5: DISTRIBUTION OF RESPONDENTS BY TYPE AND SEX (N=622) Citizens’ Voice Project Final Performance Evaluation 11 Evaluation Limitations Several limitations associated with the evaluation methods had the potential to introduce bias in the results. The limitations and the actions the evaluation team took to mitigate the limitations and reduce the potential for bias include: 1. Because of the relatively small number of grants selected overall and from each theme, the evaluation team cannot confidently generalize findings to all 222 closed grants or to all of the individual thematic areas. Results at the thematic area level are generalizable only in cases where the sample contained all or most grants in a given theme. These themes include tax collection, legislative governance, Frontier Crimes Regulation, and water rights. The team discussed this limitation with USAID at the outset and determined that the design was adequate to address the evaluation questions. 2. During the analysis, the evaluation team disaggregated data from the selected grants by theme, grant size, and duration. The report presents findings across these dimensions only if the analysis identified discernable differences for each dimension. If any section of the report does not present findings by theme, it means that there were no identifiable differences in findings across the themes. In general, time and resource constraints prevented the team from disaggregating all results by grant cycle, although it did so for selected findings when the differences seemed most relevant. As with results by theme, grant size, and duration, the report presents results by cycle only when the differences were meaningful. 3. The instrument questions did not focus clearly on the historical timeline of the challenges faced by the respondents, so it was not possible to explore details such as the stage of grant activity at which problem(s) occurred and when they were resolved (if at all). 4. Based on the evaluation team’s understanding of the evaluation questions, clarified by USAID, the evaluation focused on the suitability of the GUC model to achieving CVP objectives, which necessarily included examining TDEA’s ability to manage the GUC approach and the FAA mechanism through which TDEA awarded grants. 5. Several selected grants closed years prior to the evaluation. The evaluation team worked closely with TDEA and grantees, but it was not always possible to locate documents and beneficiaries. As a result, the team was able to conduct 42 of a planned 66 FGDs with grant beneficiaries. 6. While the study randomized initial grant selection, purposively selected grants sometimes replaced randomly selected grants to achieve broader geographic coverage or to accommodate evaluator accessibility and safety. Results therefore overly represent grant activities and interviewees located near more accessible and safer urban areas. The evaluation team does not believe this limitation significantly biased results. 7. Grantee organizations usually selected beneficiary interviewees, so the evaluation team could not ensure random selection. Whenever possible, however, the team randomly selected participants from lists of beneficiaries supplied by grantee organizations. 8. Female and transgender interviewees made up 30 percent of the respondent sample. There were only marginal differences between male and female responses across all questions disaggregated by sex. 9. Group dynamics, including “alter leaders” who influence what individuals say (or do not say) and extroverts who overpower introverts, may influence responses during FGDs. To mitigate this effect, moderators did their best to engage all participants in the discussions. Citizens’ Voice Project Final Performance Evaluation 12 Interpretation of Findings The evaluation team conducted both individual interviews and FGDs and coded FGD data in a manner that identifies individual respondents. The analysis is thus able to determine both the number of individual respondents who expressed a particular opinion and the number of interviews (group or individual) in which at least one respondent articulated a particular opinion—although interview-level analyses can be problematic. Analysis based on individual responses may be more appropriate to understanding opinions specific to an individual (e.g., beneficiaries’ opinions of the relevance of a grant activity), while an interview-level analysis may be more appropriate to understanding experiences specific to an organization (e.g., grantees’ experiences with the grant process). The report presents interview- and response-level analyses as appropriate. Table 3 summarizes the distribution of interviews and respondents by data source. TABLE 3: DISTRIBUTION OF INTERVIEWS AND RESPONDENTS BY DATA SOURCE Data source Number of organizations Number of interviews (interview level) Number of respondents (response level) Grantees 40 40 57 Government N/A 55 60 Civil society/media N/A 53 80 Beneficiaries 40 42 417 USAID/TDEA/AASR-GT 3 7 8 The evaluation team corroborated evaluation findings with evidence from interviews and project documents. The report disaggregates analyses of interview data by respondent type, which may include grantees, civil society/media, beneficiaries, government, and USAID/TDEA/AASR-GT. For findings that describe binary responses (e.g., yes/no or sufficient/not sufficient), the analyses represent individual respondents. For findings that describe open-ended responses (e.g., what are the most effective awareness activities?), the analyses allowed for multiple responses per respondent. FINDINGS Findings for Evaluation Question 1: Relevance Evaluation Question #1: To what extent have the outputs and outcomes of the grants’ interventions contributed to improved engagement between citizens and government on priority program-supported initiatives to advance good governance objectives? Relevance to the Needs of the Target Population Ninety-seven percent of the 245 respondents representing grantees, civil society/media, and beneficiaries who answered the question agreed that grant-funded activities responded to the needs of the target population. Beneficiaries stated that grant activities improved their understanding of their rights as citizens and addressed issues relevant to their daily lives. For example, an FGD participant who was a beneficiary of the Strengthening Participatory Organization (SPO) energy reform grant said that Citizens’ Voice Project Final Performance Evaluation 13 the grantee and beneficiaries worked together to identify the community’s shared energy problems and jointly advocated for improved government regulation of electricity consumption through better metering. In response to a multiple response question about how they selected grant activities, representatives from 75 percent of the 40 grantees interviewed reported consulting with local communities and/or using in-house expertise to select grant activities (Figure 6). 7 Forty-five percent said they consulted with the local population to select relevant activities for the grants; 30 percent said they relied mainly on their own knowledge; and 28 percent said they followed USAID guidelines. In contrast, only 10 percent of grantees reported relying on government and CSOs, and 3 percent sought TDEA guidance to identify priority grant activities. FIGURE 6: METHODS OF SELECTING GRANT ACTIVITIES Fifty-eight percent of the 12 grantee respondents who said that they used their own knowledge of the sector to design grant activities reported that they incorporated lessons learned from previous projects. For example, an energy sector reform grantee shared that prior experience working with USAID taught him the importance of educating citizens on the complexities of the electricity market so they understand how to reduce electricity consumption and petition the government to address load￾shedding and metering problems. As a result of these experiences, the grantee incorporated these lessons into grant activities.8 Relevance to Government Priorities Eighty-nine percent of the 134 combined (i.e., grantee, government, and civil society/media) respondents who answered the question believed that grant objectives aligned with government priorities at the time of implementation. A number of grantees (the evaluation team was not able to quantify the precise 7 Local consultations refer to informal meetings/group discussions held with beneficiaries on the nature of the grant and the types of challenges faced by the community. 8 Grantee: Semiotics Consultants. Theme: Energy sector reforms. Citizens’ Voice Project Final Performance Evaluation 14 number) said that grant activities stemmed directly from new government priorities after the successful 2013 democratic transition and that grant themes also reflected these new priorities. 9 For example, one grantee explained that the recently created Punjab government Commission for Compulsory and Free Education complemented CVP grant activities in Lahore, which included improving beneficiary, CSO, and government knowledge of the decentralized education system and how it benefits Punjabi students.10 Two grantee respondents who thought grant activities were not relevant to government priorities believed that the government was not yet ready to be fully accountable to citizens and remained relatively opaque. Perceived Adequacy of Grant Amount and Duration As Figure 7 illustrates, 78 percent of grantees believed that the grant amount was sufficient to implement grant activities. However, only 40 percent thought that the duration of the grant was sufficient. Of the 23 respondents who thought the grant duration was insufficient, 9 (39 percent) cited the need for more time to accomplish their grant’s governance, policy reform, and advocacy objectives. By their very nature, grant activities such as advocacy and relationship building with stakeholders— especially the government—require considerable time to successfully influence policy. A quote from the grantee Human Resource Development Network (HRDN) illustrates this point: “The time was also insufficient as there were limitations of working with the government. An ideal time for such a grant is at least two years.”11 Despite their dissatisfaction with the grant duration, the 23 grantees who thought the grant duration was insufficient expressed no consensus about what constituted a suitable timeframe. FIGURE 7: SUFFICIENTY OF GRANT AMOUNT AND DURATION (PERCENT OF GRANTEES) Table 4 summarizes data on grantee dissatisfaction with grant size and duration disaggregated by theme. Because the themes in the first three rows have little in common, there does not appear to be any discernable pattern in these data. 9 Grant themes such as free and fair elections, implementation of the 18th Amendment, and the importance of local government echo various laws passed around the time of the 2013 democratic transition. 10 Grantee: LEAD. Theme: Education sector reforms. 11 Grantee: HRDN. Theme: Effective grievance redress through offices of ombudsmen. Citizens’ Voice Project Final Performance Evaluation 15 TABLE 4: THEMES WITH INSUFFICIENT GRANT DURATION AND AMOUNT Grant theme Grant duration insufficient (>50% grantee responses) Grant amount insufficient (≥50% grantee responses)12 Citizens’ oversight of municipal services X X Citizens’ advocacy for implementation of 18th Amendment X X Citizens’ voice and accountability for youth development X X Citizens’ voice for free, fair and peaceful elections X - Education sector reforms X - Energy sector reforms X - Importance of local government system X - Advocacy for effective implementation of the Frontier Crimes Regulation - X Citizens’ voice for effective grievance redress through offices of ombudsmen - X Citizens’ voice for effective legislative governance - X Citizens’ awareness for higher female voter turnout - X Citizens’ engagement and accountability for an effective local governance system - X 12 No themes had a majority of grantees (more than 50 percent) who said that the grant amount was insufficient. Rather, one of two grants in a theme, for example, reported that the grant amount was insufficient. Citizens’ Voice Project Final Performance Evaluation 16 Findings for Evaluation Question 1: Effectiveness Effectiveness of Awareness Activities Interviews with representatives of government, beneficiaries, and civil society/media asked respondents to identify the most effective awareness activities. The greatest percentage of responses13 (59 percent of those from men and 57 percent from women) mentioned information campaigns, especially door-to￾door advocacy and distribution of information, education, and communication (IEC) materials. The second largest percentage of responses (27 percent of those from men and women) mentioned community-level meetings, which include small group consultations and informal committee discussions. Figure 8 illustrates how responses ranked awareness activities by effectiveness. There is no meaningful difference between the responses from men and women. FIGURE 8: MOST EFFECTIVE AWARENESS ACTIVITIES BY RESPONDENT SEX Respondents explained that information campaigns and community-level meetings were particularly effective awareness-raising activities because they informed and mobilized the public to advocate for a common cause. For example, one respondent said that the Global Educational, Economic, and Social Empowerment (GEESE) grant accomplished its goal of improving the electability of female candidates through door-to-door campaigning and distributing pamphlets and banners for upcoming elections. 14 Notably, however, 5 percent of responses (17 responses) ranked information campaigns as the least effective awareness activity. This is because some information campaign activities were poorly executed. For example, Governance Institutes Network International (GINI) used a local television channel to spread information on improving water rights for disadvantaged farmers. However, most of the farmers lived in rural areas and had no access to cable television. In another case, Coastal Association Research and Development (CARD) organizers in Balochistan printed IEC pamphlets to distribute among fisherman, even though the fishermen were illiterate. In a third case, a respondent from grantee Center for Peace and Development Initiatives (CPDI) in Abbottabad said that informative materials were initially 13 The data represent responses and not respondents. Because some of the data come from FGDs and some from individual interviews, it was not feasible for the evaluation team to report on the basis of respondents. 14 Grantee: GEESE. Theme: Higher female voter turnout. Citizens’ Voice Project Final Performance Evaluation 17 printed in English, which was difficult for some beneficiaries to understand. The grantee later printed the materials in literary Urdu, which was also very difficult for beneficiaries to understand. Citizen Participation in Grant Activities The data collection instruments asked for respondents’ perceptions of how involved citizens had been in grant-funded activities. Responses from the 222 (of 622) respondents who answered the question suggest that citizens were very involved in all types of grant activities, with the strongest involvement in public meetings with government departments, seminars, and public dialogues with civil society and media. Of 222 respondents, 68 percent described the level of citizen participation in grant activities as “high,” while only 6 percent described participation as “low” (Figure 9). 15 FIGURE 9: PERCEIVED LEVEL OF CITIZEN PARTICIPATION BY RESPONDENT TYPE Government and civil society/media respondents reported proportionally more “medium” and “low” citizen participation levels (39 percent and 50 percent, respectively) than beneficiaries or grantees (29 percent and 15 percent, respectively).16 Respondents reported that citizens learned about their own rights and the role of the government as a result of participating in these activities. One beneficiary at a polling station in Baffa Khurd, Khyber Pakhtunkhwa (KP), emphasized his enthusiasm for participating in a female voter turnout activity in the following words: “No woman ever cast her vote in my family, but this time I will personally take my daughter and wife to the polling station to cast their votes.”17 15 The levels of citizen participation reflect the evaluation team’s subjective interpretation of answers to the question about level of participation. The team defined the categories as follows: - High: All or most targeted citizens attend and actively participate in activities. - Medium: Some targeted citizens attend and actively participate in activities. - Low: Few targeted citizens attend and actively participate in activities. 16 While the reasons for these proportionally high “medium” and “low” response rates are unclear, government and civil society/media respondents are not as actively engaged in the grant implementation process as beneficiaries and grantees. On the other hand, government and civil society/media may be more objective observers of citizen participation because of their indirect participation. 17 Grantee: GEESE. Theme: Higher female voter turnout. Citizens’ Voice Project Final Performance Evaluation 18 The data collection instruments for all types of respondents asked which types of groups participated in grant-funded activities. The data depicted in Figure 10 reflect the percentage of the 298 responses that mentioned each type of participant. Responses highlighted several disadvantaged groups among the most common types of participants in grant activities, including women (20 percent of responses), minorities (14 percent), and youth (13 percent). FIGURE 10: PARTICIPATION IN GRANT ACTIVITIES BY TYPE OF PARTICIPANT (N=298) Note: The figure represents the percentage of responses, not the percentage of respondents, since the underlying question allowed multiple responses, and some of the data came from FGDs. While a greater percentage of grantees than other respondent types engaged citizens through community-level meetings, the difference is not statistically significant. Similarly, differences in participation between men and women are not statistically significant. While a greater percentage of grantees than other respondent types engaged citizens through community-level meetings, the difference is not statistically significant. Similarly, differences in participation between men and women are not statistically significant. Figure 11 and 12 illustrate citizen participation in various types of grant-funded activities. The non￾beneficiary instruments asked respondents for the types of activities in which citizens participated. Seventy-three percent of the 141respondents who answered the question reported that information campaigns and community-level meetings were the most effective activities for engaging citizens. These results corroborate earlier findings that information campaigns and community meetings were the most effective activities for raising awareness, demonstrating a strong overall preference (regardless of respondent sex) for activities that both inform and mobilize the citizenry. While a greater percentage of Citizens’ Voice Project Final Performance Evaluation 19 grantees than other respondent types engaged citizens through community-level meetings, the difference is not statistically significant. Similarly, differences in participation between men and women are not statistically significant. FIGURE 11: CITIZEN PARTICIPATION BY RESPONDENT TYPE FIGURE 12: CITIZEN PARTICIPATION BY SEX OF RESPONDENT Linkages to Other Stakeholders The grantee and beneficiary instruments asked respondents which linkages to other stakeholders they found most effective in contributing to achieving grant objectives. The highest proportion (55 percent) of the 154 responses cited linkages formed with the government as the most effective, followed by Citizens’ Voice Project Final Performance Evaluation 20 linkages with local communities (13 percent) and linkages with media organizations (10 percent) (Figure 13). FIGURE 13: MOST EFFECTIVE LINKAGES (GRANTEES AND BENEFICIARIES) (N=154) Note: The figure represents the percentage of responses, not the percentage of respondents, since the underlying question allowed multiple responses, and some of the data came from FGDs. Disaggregated by respondent type, both beneficiaries and grantees identified linkages with the government as the single most effective linkage. Sixty-five percent of the 101 responses from beneficiaries and 36 percent of the 53 responses from grantees18 cited government linkages as the most effective. While grantees and beneficiaries agreed on the primacy of government linkages, grantees placed significantly more emphasis than beneficiaries on the role of the media (21 percent of grantee responses relative to 4 percent of beneficiary responses).19 For example, the grantee Water, Environment, and Sanitation Society (WESS) appreciated media linkages because the press “promoted our activities and provided media coverage of our activities and articles on legislative governance in the newspapers,” which helped WESS to improve local government transparency and accountability in KP. 20 It is difficult to overstate the positive contribution of government linkages to accomplishing grant objectives. For example, an Aghaz grantee respondent said that because the government provided a list of villages surrounding election polling stations, the organization was able to analyze the reasons for low female voter turnout in Tando Muhammad Khan district in Sindh province.21 18 Results reflect multiple response data, and a larger percentage of grantee respondents (36 percent) identified linkages with government as more effective than any other single linkage. 19 The difference is statistically significant at the 0.01 level. 20 Grantee: WESS. Theme: Effective legislation and accountability. 21 Grantee: Aghaz. Theme: Higher female voter turnout. Citizens’ Voice Project Final Performance Evaluation 21 The perceived effectiveness of government, local community, media, and other linkages varied across grant themes. Beneficiaries, grantees, and civil society/media respondents associated with grants under the redress through offices of ombudsmen, accountability for effective local government, energy sector reforms, and education sector reforms themes found linkages with government more effective than respondents from other themes. Although the number of observations is small, linkages with local communities featured prominently for the Frontier Crimes Regulation; free and responsible media; and free, fair, and peaceful elections themes. Figure 14 illustrates the percentage of responses associated with the grants under each theme that identified the various types of linkages as being most effective.22 FIGURE 14: MOST EFFECTIVE LINKAGES BY THEME Note: The figure represents the percentage of responses, not the percentage of respondents, since the underlying question allowed multiple responses, and some of the data came from FGDs. Government Engagement in Grant-Funded Activities Ninety-two percent of the 125 grantee, civil society/media, and beneficiary respondents23 who answered the question reported some level of government engagement in the grant activities with which they were associated. According to respondents, government engaged with citizens in many activities, including public forums with citizen activists and election awareness campaigns. When asked about the level of engagement, however, only 55 percent of respondents reported that the government had a “high” level of engagement, 31 percent reported a “medium” level of engagement, and 13 percent 22 The number of observations on each theme were too small to identify statistically significant differences across themes. 23 Beneficiary data reflects responses of individual beneficiaries. Data from grantees and civil society/media may reflect the response of an individual or of a small group. “Respondents” in the context of this analysis may therefore mean individuals or groups. Citizens’ Voice Project Final Performance Evaluation 22 reported “low” engagement. Civil society and media respondents in particular perceived lower levels of government engagement than beneficiaries and grantees (Figure 15).24 FIGURE 15: PERCEPTIONS OF THE LEVEL OF GOVERNMENT ENGAGEMENT BY RESPONDENT TYPE While the government nearly always supported grant activities to some degree, it did not engage evenly across thematic areas (Figure 16). The government engaged most extensively with the following six themes, only one of which (i.e., effective legislative governance) is a USAID priority theme:  Effective local governance systems,  Effective implementation of FCR,  Effective legislative governance,  Transparency and accountability, and  Youth development. 24 The evaluation team subjectively assessed levels of government engagement as follows: - High: Government mostly responds to grantee requests, is accessible to citizens, and furthers grantee objectives. - Medium: Government listens to grantee concerns and sometimes furthers grantee objectives. - Low: Government sometimes listens to grantee concerns but rarely furthers grantee objectives. Citizens’ Voice Project Final Performance Evaluation 23 FIGURE 16: PERCEIVED LEVEL OF GOVERNMENT ENGAGEMENT BY THEME A representative from Strengthening Participatory Organization (SPO), a grantee organization in Azad Jammu and Kashmir (AJK), noted positive engagement from the highest levels of the provincial government: “We have been successful in getting our voice heard by the Prime Minister of AJK. Our concerns and issues are being heard.”25 Despite strong government engagement in many thematic areas, respondents perceived low to medium government engagement in the other four USAID priority themes, improving water rights, tax collection, energy sector reforms, and education sector reforms. Grant Contributions to Policy Change Most grants (30 of 40) did not have policy change among their objectives. Of the 10 grants that specified a policy change or development objective, 4 (40 percent) achieved their objectives, according to grantee 25 Grantee: SPO. Theme: Energy reforms. Citizens’ Voice Project Final Performance Evaluation 24 respondents. The policy-focused grants describe a broad range of policy objectives, and while some grant activities achieved specific reforms, others only moved the needle on policy discussions. Policy changes were generally limited in scope, with most contributions taking the form of drafting policies and recommendations, charters of demand, or action plans delivered to government officials. For example, a Right to Information (RTI) commission representative with the Hamdam Development Organization (HDO) brought attention to local and national governance problems in KP: “[A] public hearing with [Members of the Provincial Assembly] MPA and [Members of the National Assembly] MNA led to a Charter of Demand which documented issues and demands of citizens and a copy was handed over to MPA….The minorities’ issues were raised by their Minister in the Assembly and they have gone to court.”26 Strong advocacy helped other grantees achieve more concrete policy objectives. For example, CARD’s amendment was proposed in the Balochistan 2012 Industrial Relations Ordinance to define fishermen as workmen.27 In another case, GINI grantees said they successfully lobbied to change the tax law (Section 40-B of the Sales Tax Act of 1990) to reduce government monitoring of businesses suspected of reducing production to avoid tax payments.28 Beneficiary Level Results Two-thirds of beneficiaries (67 percent) reported that government had improved its responsiveness to citizens’ needs (Figure17). Responses ranged from increased citizen awareness to changes in how the government operates. For example, an Aghaz grantee beneficiary and member of the District Vigilance Committee (DVC) in Tando Muhammad Khan district noted an “increase in female voter turnout in 46 polling stations in the election of 2013 as compared to the election of 2008 and an increase of 30 to 40 percent in female voter turnout in targeted polling stations.”29 26 Grantee: HDO. Theme: Effective legislative governance. 27 The ordinance was passed on February 22, 2011. Fishermen likely fit the following definition of workmen: a “person not falling within the definition of employer who is employed (including employment as a supervisor or as an apprentice) in an establishment or industry for hire or reward either directly or through a contractor whether the terms of employment express or implied, and, for the purpose of any proceedings under this Act in relation to an industrial dispute includes a person who has been dismissed, discharged, retrenched, laid-off or otherwise removed from employment in connection with or as a consequence of that dispute or whose dismissal, discharge, retrenchment, lay-off, or removal has led to that dispute but does not include any person who is employed mainly in managerial or administrative capacity.” Balochistan Industrial Relations Act, 2010, p. 6. 28 The evaluation team cannot confirm that the GINI advocacy effort directly influenced reforms to the Sales Tax Act of 1990. 29 Grantee: Aghaz. Theme: Higher female voter turnout. Citizens’ Voice Project Final Performance Evaluation 25 FIGURE 17: CHANGE IN GOVERNMENT’S RESPONSE TO CITIZENS’ NEEDS In addition, 74 percent of respondents believed the government had improved its openness to citizens as a result of grant activities, including an improvement in government responsiveness to citizen needs at the district level. Government seemed most responsive when grantees advocated for specific reforms, such as the 18th Amendment, Article 25, and the Citizens’ Charter of Demands. In some instances, citizens’ knowledge of their own rights and the role of the government contributed to improved government responsiveness. For example, one female beneficiary with grantee Association for Behavior and Knowledge Transformation (ABKT) in Peshawar (KP) said that her organization formed committees to improve government services: “[W]e have experienced changes in their [the Tehsil Municipal Administration’s (TMA)] performance to some extent…they were not regular in collecting waste from the bazaars before the project. But after formation of our committees, they perform their duties well and collect garbage on regular basis.”30 30 Grantee: ABKT. Theme: Oversight of municipal services. Citizens’ Voice Project Final Performance Evaluation 26 Findings for Evaluation Question 2: Sustainability31 Evaluation Question #2: To what extent will the outputs and outcomes of the grants’ interventions sustain (both financially and in terms of human capacity) beyond the life of the grant? Grantee Capacity Building Eighty-seven percent of the 40 grantees interviewed felt that project-supported capacity-building trainings were relevant to their organization’s needs and had improved their organization’s capacity and strengthened their administrative and functional systems. Specifically, grantees said CVP built their capacity in the following ways: 1. Improved understanding of grant activity proposal writing, documentation, and reporting; 2. Strengthened financial, human resources, procurement, and general administrative skills; and 3. Improved program management, communications, monitoring and evaluation (M&E), and management information systems (MIS) skills. Respondents found training in the areas of finance, program management, and M&E particularly beneficial (Figure 18). Thirty-eight percent of respondents said that program management training was the most useful, 37 percent identified financial training, and 13 percent M&E.32 FIGURE 18: MOST USEFUL TRAININGS (PERCENT OF RESPONDENTS) The following quote from a representative of grantee Development, Environmental, Legal Aid, Technical Support and Advocacy Associates (DELTA) demonstrates that CVP’s technical and management 31 USAID defines sustainability as “the capacity of a host country entity to achieve long-term success and stability and to serve its clients and consumers without interruption and without reducing the quality of services after external assistance ends” (USAID Website February 2016). In the context of this report, the evaluation team uses the same definition but replaces “host country entity” with “grantee organization.” In addition, the evaluation team directly answers the second evaluation question, which ties grant sustainability to future outputs and outcomes (results). 32 Grantee responses mostly referred to the mandatory operational and technical trainings, with some mention of other unspecified trainings. Therefore, the findings are limited to the mandatory trainings and cannot be generalized to include the needs-based trainings unless specifically stated. Citizens’ Voice Project Final Performance Evaluation 27 capacity-building activities helped some grantees win other grant funds and apply acquired skills to new grant activities. “As an organization, DELTA was greatly helped by the CVP grant and it helped us win another UNDP grant as well as one from the Australian Commission. We are now a registered taxpayer. We learned from CVP on documentation requirements. There were many things we were weak at, such as proper procurement systems, and the CVP team taught us about a lot of these things.”33 Anecdotal evidence also suggests that the timing of training and poor communication between TDEA staff and grantees caused some dissatisfaction. Respondents from one grantee, AWAZ, shared that they received PMP training only a month before the project was closing, and they were required to complete the project PMP during the training. Grantees such as WESS, Yar Muhammad Samejo Educational Society (YMSESDO), and Change Thru Empowerment noted that trainings should have been provided at the start of the project, but did not mention which trainings. Another grantee, DELTA, complained that TDEA had not shared the training schedule, and the sudden onset of training increased the management burden for the grantee, as it had to send nominations very quickly. Six of 40 grantees (15 percent) recommended that TDEA could improve trainings in the future by increasing the frequency of the trainings, with some grantees specifying that financial management, human resources, procurement, information technology, MIS, and M&E trainings, in particular, should be given more frequently. Grant Contribution to Grantee Mission Fifteen of the 40 grantee organizations interviewed (37 percent) said that the CVP grant helped them achieve their organizational mission. For example, grantee Society for Human Empowerment and Rural Development (SHER) achieved its mission to “improve the quality of life and governance” in the education sector by improving the quality of education in its target schools in Khushab district.34 Similarly, grantee Pak Women achieved its mission to improve “good governance and human rights” through the grant’s focus on increasing the political participation of marginalized women and by “providing access to information, which improved their engagement with the MPAs [Members of the Provincial Assembly], strengthening their political development and rights.”35 Sustainability of Grant Results TDEA required each grantee to write a sustainability plan to submit with its grant application. Examples of grantees’ strategies to improve the chances of sustainability include building strong linkages with grant stakeholders, conducting awareness campaigns, and nurturing supportive community networks. Only 3 of the 40 grantees the team interviewed said that they never intended the results of their grant activities to be sustainable. For example, representatives of the grantee GEESE said that grant activities had exclusively focused on an upcoming election and results were not expected to sustain beyond that date.36 The 37 grantees that expected their grant activities to be sustainable cited active community networks, an aware and empowered citizenry, continued engagement with grant stakeholders (such as media, civil 33 Grantee: DELTA. Theme: Transparency and accountability. 34 Grantee: SHER. Theme: Education sector reforms. 35 Grantee: Pak Women. Theme: Effective legislative governance. 36 Grantee: GEESE. Theme: Higher female voter turnout. Citizens’ Voice Project Final Performance Evaluation 28 society, or government agencies), and individual beneficiary efforts as reasons for sustainability after grant closure. Of 282 responses from representatives of government, beneficiaries, civil society/media, and grantees, 217 (77 percent) reported that results from grant activities continued even after the grant ended (Figure 19). FIGURE 19: REPORTED SUSTAINABILITY OF GRANT RESULTS Respondents provided a number of examples of why results are sustainable. For example, one believed that awareness of citizen rights would help to sustain grant results because knowledge can empower individuals to be agents of change in their communities. A beneficiary of grantee HDO said that because the community understands its rights, people now request birth certificates and petition the government to address agricultural issues.37 The grantee Network for Consumer Protection sustained its activities with additional funding from Oxfam International: “As a result of their citizen level approach to taxes, Oxfam took over the activities once the grant ended. They also got one year of additional funding from Oxfam to continue working the taxpayer/citizen level. Oxfam built upon their work through the linkages with the people trained in the Network and the knowledge of the Network.”38 Strong linkages to stakeholders also contributed to sustainability. For example, after the grant to Pak Women ended, the Khyber Pakhtunkhwa Finance Department Civil Secretariat conducted “pre-budget consultations with its own resources” and found ways “to reduce the blocked funds in the KP development budget.”39 The 17 percent of responses suggesting that results were unsustainable included reasons such as lack of funding to continue activities, weak informal community networks, frequent government transfers that disrupted interpersonal relationships, and a lack of government investment in project success. For 37 Grantee: HDO. Theme: Effective legislative governance. 38 Grantee: Network for Consumer Protection. Theme: Tax collection. 39 Government interview: Finance Department, KP. Citizens’ Voice Project Final Performance Evaluation 29 example, a beneficiary of Development Resource Solutions (DRS) grant activities in Balochistan Province said that the lack of funds and a weak community network made results unsustainable: “Networks formed during the project duration were not functional anymore. Networks were independent and formed by volunteers. We never had funds so this has big constraints…it’s [the network] not as active as it should be.”40 A majority of civil society and media organizations (64 percent) reported continuing their engagement with grantees through other projects that continued CVP’s work of building linkages between communities and the government. Civil society and media organizations that discontinued their engagement with grantees cited as a rationale the fact that individual grant activities had ended. Enhancing Results As depicted in Figure 20, grantees, beneficiaries, and civil society/media respondents believed that awareness activities and government and civil society engagement were the most important considerations for improving the performance of future grant activities. Suggested awareness activity improvements included refresher trainings, expanding media coverage of awareness issues, and the use of innovative communication methods such as theatrical productions. To increase government engagement in the future, respondents wanted to see improvements in government communication, including improved district office responsiveness, more frequent community visits, and increased access to higher levels of government. Suggestions for improving civil society participation included holding more regular community meetings, inviting district pressure groups (DPGs) to observe parliamentary proceedings, and involving village counselors in awareness campaigns. 40 Grantee: DRS. Theme: Transparency and accountability. Citizens’ Voice Project Final Performance Evaluation 30 FIGURE 20: WHAT RESPONDENTS WOULD DO DIFFERENTLY Findings for Evaluation Question 3: Grants Under Contract Evaluation Question #3: In what ways (both positive and negative) has the Grants Under Contract (GUC) model affected the implementation of interventions and their alignment with the stated objectives of the project? Implementation of CVP relied on two grant mechanisms: the GUC contractual grant framework between USAID and TDEA and the Fixed Amount Award (FAA) grant funds disbursal system between TDEA and grantees. The answer to this question addresses and examines the relationship between USAID and TDEA and also TDEA’s management of the grant process for grantees. The GUC-FAA Mechanism TDEA’s contract with USAID authorized it to award GUCs to local organizations, and it used the FAA mechanism to make these awards. USAID’s “Fixed Amount Awards to Non-Governmental Organizations: An Additional Help Document for ADS Chapter 303” outlines why FAAs are a useful grant mechanism for inexperienced organizations (such as TDEA at the outset of CVP): “[W]hen awarding to a non-U.S. NGO with little or no experience in receiving and implementing [U.S. Government] USG grants, a fixed amount award can provide a manageable vehicle for both USAID and the recipient and assist in building institutional capacity as the recipient completes the grant activities. During the period of a fixed amount award, these NGOs, including local and community organizations, have the opportunity to strengthen and improve their internal procedures, systems, and policies. By doing so, they can then Citizens’ Voice Project Final Performance Evaluation 31 improve their ability to access and manage resources from USAID or other donors.”41 The GUC-FAA mechanism is generally a better implementation choice than in-kind grants (i.e., goods and services procured directly by the grantor rather than the grantee) for grantors with limited administrative capacity. “In-kind grants involve a much larger administrative burden [for TDEA],” according to MSI’s Contracts Manager, Pushmeet Bhatia.42 With in-kind grant models, grantors must have large procurement, oversight, and logistics teams to ensure that supplies are competitively purchased and efficiently distributed. TDEA did not have the budget or the manpower to successfully execute an in-kind grant model. By tying payments to completion of predetermined milestones, the GUC-FAA model also gave TDEA the ability to “push the implementing partners” to achieve objectives.43 Without the burden of procuring supplies and managing a large number of administrative staff, TDEA could instead focus on technical grant management, capacity building, and potentially improving the quality of milestone deliverables. The overall positive programmatic data gathered over the course of this evaluation suggests that the GUC￾FAA model was effective in the context of achieving program objectives but does not support conclusions about effectiveness relative to alternative models. The GUC-FAA mechanisms were beneficial to USAID because they provided the flexibility to support diverse grant awards to small organizations while also giving USAID strong approval and oversight powers44 to manage a “high-risk” organization such as TDEA.45 Under this arrangement, the Mission worked alongside TDEA to jointly develop thematic areas, participate in the grant approval process, and build the grant management capacity of the TDEA team. USAID and TDEA Perceptions of GUC-FAA TDEA and USAID had quite different perceptions of the advantages and disadvantages of the GUC model. In general, TDEA managers believed the model allowed USAID an excessive oversight role that sometimes caused delays in approvals with subsequent delays in project activities. USAID staff, on the other hand, appreciated the flexibility of the GUC model and the hands-on management role it gave the Agency. Table 5 summarizes the opinions of TDEA and USAID staff on the grant mechanism. TABLE 5: PERCEPTIONS OF THE GUC GRANT MECHANISM46 Perception USAID TDEA Slow grant approval processes Yes Yes Funding delays (to TDEA and to grants) Yes (outside their control) Yes TDEA deserves greater decision-making authority No Yes Too much control from USAID No Yes 41 USAID. (December 24, 2014). Fixed Amount Awards to Non-Governmental Organizations: An Additional Help Document for ADS Chapter 303, pp. 2- 3. 42 P. Bhatia, MSI Contracts Manager, personal communication, April 27, 2016. 43 TDEA staff member, personal communication. 44 Interviews with USAID/Pakistan personnel. 45 At the project’s inception, TDEA had numerous “moderate to high risk” categories of operation as determined by USAID/Pakistan’s Office of Financial Management. 46 The table summarizes data from interviews with TDEA and USAID staff. Citizens’ Voice Project Final Performance Evaluation 32 USAID and TDEA personnel the evaluation team interviewed both recognized that funding and grant approval delays have a negative ripple effect throughout the project lifecycle—the longer and more frequent the delays, the more likely that grantee deliverables and milestones are delayed as well. It is clear, however, that TDEA prefers less USAID oversight (especially relating to approvals) to reduce these delays, while USAID believes that more hands-on, full-time management can help address the delays. Grant-Making Process When asked about their experience with the grant process, more grantees mentioned positive than negative experiences, but the evidence is quite mixed. Sixty percent of the 40 grantees interviewed said the grant evaluation process was transparent, 53 percent said the grant making process was satisfactory, and 50 percent appreciated the specificity of grant criteria and limitations (Figure 21). Sizable minorities, however, cited problems with various aspects of the grant-making process. Sixty-eight percent appreciated that they could pursue grant activities without spending their own money. FIGURE 21: GRANTEES’ EXPERIENCE WITH THE GRANT-MAKING PROCESS When asked specifically about the areas of the grant process that could be improved, respondents mainly expressed concerns about the payment/milestone process and poor communication with TDEA. Figure 22 illustrates the most common complaints and recommendations respondents mentioned regarding GUC-FAA processes. Citizens’ Voice Project Final Performance Evaluation 33 FIGURE 22: MAIN GUC IMPROVEMENT AREAS Thirty-seven percent (21of 56) of the grantee respondents expressed frustrations over TDEA’s lack of communication and seemingly arbitrary acceptance or rejection of grant applications. Communication failures were systemic throughout the grant process, from the application’s inception to the final milestone payment. Limited internal TDEA staff capabilities contributed directly to delays in reviews and approvals. Grantees working in 10 of the 16 thematic areas reported communication problems, but no single theme emerged as particularly problematic. Half of the 10 grantees that experienced communication issues were in the Khyber Pakhtunkhwa (KP) province, with the rest divided across Punjab, Balochistan, and the Federally Administered Tribal Areas (FATA). A statistical test of differences between proportions found that grantees in KP were significantly more likely to report communication problems than grantees in the other districts combined.47 There was no strong link between CVP communication problems and the grant cycle. Cycle 3 stood out with the largest number of problems: 4 of the 10 grantees reporting problems were from Cycle 3, while 2 belonged to Cycle 4, and 4 were divided among Cycles 1, 2, 5, and 6. The differences among cycles were not statistically significant. Refer to Annex 9 for more detail. Examples of communication problems grantees mentioned included little timely feedback on submitted reports and the long time it took TDEA to respond to email queries. Other examples included a lack of 47 The Z score for the difference between proportions was 2.16, which indicates a significant result at the 0.05 confidence level. Citizens’ Voice Project Final Performance Evaluation 34 clarity on non-allowable expenses outside the project scopes of work and how to handle situations where actual participants in the activities were fewer than planned. 48 Thirty-five percent of grantees interviewed complained about a problematic payment process, including a lack of timely milestone payments.49 While CVP used the Grant Management Information System (GMIS) to review activity documentation, track milestone completion, and approve milestone payments, a lack of GMIS capacity among grantees contributed to payment delays. For example, one grantee from KP expressed frustration about milestone payment delays: “We had to do extensions at their [CVP] request. We didn’t want extensions and they forced us to take several extensions even though we were done and not yet paid. We had to apply for the extension after the fact, at the request of CVP, just to get paid for work that was already over. Our experience was so cumbersome that we would not apply for another grant.” 50 The most common reasons grantees provided for milestone payment delays included:  Lack of grantee capacity to use GMIS,  Miscommunication between CVP and the grantee,  Grantee performance delays,  Grantee staff changes,  Inadequate documentation of completed activities,  Excessive number of milestones requiring onerous documentation, and  Payments incorrectly tied to completion of activities other than specified milestones. Twenty of the 40 grantees cited payment problems as a main area for GUC improvement. Grantees cited payment problems in 13 of the 16 thematic areas, with the most cases occurring in the effective legislative governance theme (4 of the 20 grants). These 20 grants were implemented in KP, Punjab, Sindh, Balochistan, and AJK provinces, with the highest number of grants falling in KP (6 of 20). Most of the 20 grants were disbursed in the earlier grant cycles (1- 4), with the largest number belonging to Cycle 2 (6 of 20), followed by Cycle 3 (5 of 20). One grant was disbursed in Cycle 5, and one in Cycle 6. There were no statistically significant differences across themes, provinces, or grant cycles. Refer to Annex 10 for more detail. 48 These two examples were shared by grantees DELTA and NEEDS. 49 A milestone-based payment system relies on completion of activity milestones (such as activity completed or documentation submitted). If milestones are not completed, grantees cannot receive payment. This payment mechanism can function smoothly when grantee milestone information submission, review, and approval are efficient and timely. Unfortunately, reviews and approvals were frequently delayed, resulting in delayed payments for milestone completion. 50 Grantee: Individual Land Trust. Theme: Independent, free, and responsible media. Citizens’ Voice Project Final Performance Evaluation 35 CONCLUSIONS CVP’s activities were highly relevant for all stakeholders, including the government. Male and female beneficiaries had similarly positive views on every aspect of the program. The government is clearly the most useful linkage to achieve project objectives and especially important for those grants seeking policy change. CVP effectively supported grant activities targeting disadvantaged groups and conducted trainings that improved grantees’ chances of sustaining results after activities ceased. CVP communication lapses undergird nearly all of the problems with grant selection delays, complaints concerning arbitrary deadlines in the grantee selection, and milestone payment processes delays. CVP programming was highly relevant to grantees, beneficiaries, and the government. Grantees enhanced relevance by combining their own local knowledge with input from the local population when selecting activities. Most grants considered both CVP objectives and government priorities in the activity design phase. The government is a useful partner for achieving grantee objectives. Grantees leveraged positive relationships with government to reduce red tape, enforce existing regulations, or even introduce new reforms into the legislative body.51 While informal linkages were useful, formalized (with MOUs or other written agreements) connections were the most effective for accomplishing grant objectives. In addition, beneficiaries believed that the government responded positively to their needs as a result of grant activities. Government involvement in grant activities was high, even though it could have been higher in some USAID priority areas (e.g., improving water rights and tax collection). While the reasons for government’s perceived lack of involvement in these areas are unclear, beneficiaries profited when the government participated in grant activities. CVP was prudent to select grantees that focused on bringing together disadvantaged populations through information campaigns, community-level meetings, and other awareness activities. CVP awarded many grants to organizations that included provisions for addressing the problems of disadvantaged groups such as women, youth, and minorities. CVP encouraged the participation of disadvantaged groups from the beginning of Cycle 1 in the grant application process. All successful applications included a plan for addressing the needs of underprivileged groups. In addition, these grantees used information campaigns and community-level meetings to effectively advocate for various causes. CVP commendably emphasized these causes throughout the grant application process and empowered grantees through awareness activities. CVP encouraged sustainability throughout the grant process, from grant selection through closedown. CVP required grantees to design a plan for project sustainability as a part of the RFA submission. Selected grantees received capacity-building trainings that improved internal operations and project management skills, although some trainings could have taken place even earlier in the grant startup phase. By encouraging sustainability from the grant’s inception, CVP taught grantees to create durable linkages and look for additional grant opportunities after CVP ended. The GUC-FAA model is a useful mechanism for implementing CVP grant activities despite operational challenges. While other grant mechanisms (such as in-kind grants) are available to USAID, the GUC-FAA model is the most practical for a small-grants program such as CVP. Given TDEA’s 51 Each grantee addresses 1 of 16 thematic areas, all of which entail advocating for change or reforming some aspect of the established laws. By their very nature, the established laws are enforced (or not, in some cases) by the government. Improved linkages with the government, therefore, help grantees build the political capital necessary to change established laws. Citizens’ Voice Project Final Performance Evaluation 36 relatively small size, the organization does not have the staff or in-house expertise necessary to implement activities across the country, nor does it have the administrative capacity to provide in-kind grants to awardees. GUC-FAA is also a strong mechanism for supporting “high-risk” organizations such as TDEA. FAAs are practical for CVP because they provide financial resources to organizations best suited to accomplish CVP objectives with minimal cost and administrative burden. The implementing partner and USAID can address emerging operational challenges with minimal disruption to the grant process. USAID grant approvals require a substantial amount of time, resulting in delays to grant awards. However, USAID can both actively manage TDEA and reduce approval delays. USAID grant management and TDEA grant award delivery are not zero-sum in the sense that additional USAID management and oversight must lead to further delays in grantee awards. Both USAID’s need for additional management and TDEA’s desire to reduce delays can be accommodated by allowing USAID to manage aspects of the TDEA grant selection process while reducing and possibly eliminating USAID approvals for grant awards. The CVP grantee payments system is not working for grantees and reduces grantee trust in CVP staff and processes. The grantee payments system was problematic because 1) TDEA lacked the liquidity necessary to select grantee awards in a timely manner, 2) TDEA could not process milestone payments quickly enough, and 3) grantees lacked experience with GMIS processes. These significant challenges reduce the effectiveness of grantee organizations and discourage successful grantee organizations from applying for additional CVP funding. Communication problems cause significant negative ripple effects across the grant lifecycle. Communication problems are a result of a lack of GMIS training and a lack of internal TDEA staff capacity to monitor payments and approvals. When TDEA fails to communicate with grantees in a timely fashion on a host of issues—including acceptance or rejection of grant applications, reduction in grant amount and duration, and late milestone payments—grantees may be unprepared or unmotivated to conduct grant activities. Better communication between TDEA and grantees could improve the timing of grant activities, reduce confusion about grant milestone deadlines, and enhance grantees’ confidence in TDEA to provide on-time payments. Citizens’ Voice Project Final Performance Evaluation 37 RECOMMENDATIONS 1. Grantees in future similar projects should follow the same inclusive process to select grant activities that many CVP grantees used, with small tweaks. Grantees should continue to involve local communities and other stakeholders in developing activities, and continue to follow CVP guidelines to design grants that are relevant to USAID and government objectives. However, future similar projects should consider grantee concerns about grant duration on a case-by-case basis, especially for policy-oriented grants that may require more time to effect positive change. 2. Initiate formal linkages with the government early in the implementation process. Where appropriate, future similar projects should facilitate formalized linkages between grantees and relevant government bodies; for example, the project could provide template MOUs to grantee organizations and plan an initial organizational meet-and-greet between government officials and local grantee organizers in the early stages of activity planning and implementation. The implementing partner should request that grantee planners nominate a government liaison both to inform the government of grantee activities and to encourage additional government participation. 3. Emphasize information campaigns and community-level meetings in the RFA process and throughout the activity planning stages. Grantees of future similar projects should emphasize information campaigns and community-level meetings. While no organization can fully mitigate against activity failure, the implementing partner can limit the number and extent of failed awareness activities (such as the distribution of IEC pamphlets to illiterate farmers) by reviewing the scope of specific grant activities and questioning activities that seem unlikely to succeed. 4. Increase the number of capacity-building trainings and conduct trainings early in the grant startup phase. While grantees were generally happy with the quality of capacity-building trainings, a future implementing partner should increase the frequency of these trainings and ensure that the trainings are customized to grantee requirements. Initial trainings should take place as soon as possible after the grant awards so grantees can use their new skills over the lifetime of grant implementation. The implementing partner should also announce training dates well in advance of the event so grantees can ensure that the right people participate. After trainings, the implementer should solicit constructive feedback and prepare for additional trainings if grantees request follow￾on assistance. 5. Fixed Amount Awards under the GUC mechanism are the best implementation option for USAID to use for future similar projects. There are no practical alternatives. CVP demonstrated that the mechanism is effective, and USAID should specify it for future programs of similar design, size, and scope. Other types of awards, such as in-kind grants, are not practical for small organizations because of high administrative costs. In addition, both USAID and TDEA support the grant model and could not come up with better alternatives. 6. In future similar projects, USAID should reduce its role in approving grant awards. USAID prefers additional management and oversight, and implementing partners (TDEA, at least) prefer less USAID control. Both of these can be accommodated. USAID can actively manage the implementing partner without requiring approvals that the partner considers to be onerous and time-consuming. If USAID extends the project, it should phase out its approvals for individual grant awards over time as a trial during the first period of the project extension, and reduce them further pending positive performance from TDEA over the intervening months. See Recommendation 9 to further reduce USAID approval delays. Citizens’ Voice Project Final Performance Evaluation 38 7. Future similar projects should implement an improved payment system to enhance the grantee experience. Table 6 outlines the payment problems CVP grantees encountered and offers recommendations for improving the process. TABLE 6: CVP PAYMENT PROBLEMS AND RECOMMENDATIONS Payment problem Recommendation TDEA lacked liquidity, which delayed grantee award selection. USAID/Pakistan should make payments to the contractor within 30 days of receiving an invoice. USAID/Pakistan should make 10 percent of the monthly pending invoice amount available to the contractor to draw on as needed, as a form of escrow account, for up to 30 days until the next invoice is received. Milestone payments were not delivered on time and delayed activity implementation. Initial deliverable payments should be sizeable enough to cover both completed activities and all planned activities until the next milestone payment. The Contracting Officer should allow up to a 10 percent “mobilization” payment for nascent grantees with low cash reserves. The grantee continues to shoulder the risk of overspending. USAID/Pakistan should negotiate a reduction in the number of milestone payments to a level that both grantees and CVP consider reasonable but still provides necessary USAID control. Milestones should represent significant events within the grant’s work plan. Grantees and staff did not understand GMIS well, causing milestone payment approval delays. The implementing partner should organize an internal “train the trainers” GMIS session for all applicable staff. These new trainers would then be responsible for teaching GMIS to all grantees and for resolving technical problems. Grantees would be responsible for all GMIS inputs, while the implementer would review documentation and promptly approve milestone payments.52 8. Develop a communications manual alongside the GMIS training presentation to improve communication between the implementing partner and grantees. Deploy these tools at an off-site retreat with grantees. At the beginning of each grant cycle, the partner should gather all newly selected grantees at one or several off-site retreats for a two- to three-day training seminar. The seminar should cover implementing partner-grantee communication and GMIS. The communications manual should 1) identify staff roles and responsibilities, 2) list alternate points of contact, 3) describe communications procedures from the RFA stage through to final payment, 4) clearly outline capacity-building training schedules, and 5) describe measures to hold responsible parties accountable for poor communication. The GMIS course should outline the technical knowledge necessary to update milestone deliverables, request payment, and send messages to implementing partner staff. At the end of the seminar, grantees and the implementing partner should understand how to submit documentation, process milestone payments, and communicate efficiently about technical issues. 9. Improve the grantee review and approval process by developing a “core COR” team within USAID/Pakistan to work in tandem with designated implementing partner staff. The workload 52 USAID’s Office of Transition Initiatives (OTI) uses a similar grant management system, in which the contractor enters information into the database and OTI administrators are responsible for reviewing submissions and approving payments. Citizens’ Voice Project Final Performance Evaluation 39 associated with reviews and approvals of such a broad scope of grants is more than a single COR can reasonably handle. The evaluation team recommends a “core COR” team approach in the future, in which implementing partner staff work with a dedicated team of CORs within USAID/Pakistan from each of the regional offices—Islamabad, Karachi, Lahore, and Peshawar. The team should jointly decide which grants to award and submit grant award information as a complete package to the implementing partner. Applying additional USAID personnel and pooling resources from different USAID hubs can mitigate review and approval delays. In addition, if grantee applications are so numerous that grant approvals are unreasonably delayed, USAID/Pakistan should consider restricting grantee eligibility for select RFAs.53 Allowing restricted eligibility to proven and experienced implementers may be prudent if USAID decides to extend CVP. 10. If USAID/Pakistan extends CVP, it should pay critical attention to internal TDEA staff hiring and training, especially for dedicated communications, administrative, and finance specialists. Maintaining internal TDEA capabilities is critical now that TDEA subcontractors TAF and AASR-GT are no longer working with the project. TDEA should dedicate full-time resources to financial, communication, and administrative oversight to ensure timely milestone payments and professional and responsive communication. USAID/Pakistan and TDEA should review core skill competencies and plan for critical hires going forward to fill identified gaps. To the extent possible, USAID/Pakistan should provide TDEA the flexibility to make staffing decisions, and TDEA should quickly fill openings according to USAID requirements. 11. If USAID/Pakistan extends CVP, future grant cycles should connect and build upon themes and results obtained in preceding cycles. Grant activities developed in the extension period should link to previous grant and grantee successes. For example, future grant activities could emphasize activities already proven effective, such as citizen awareness and community networking activities. In addition, CVP should consider expanding the scope or geographic area of activities that were successful in previous grant cycles. 53 Authority for restricting eligibility can be found in ADS 303.3.6.5. Restricted eligibility is justified in a number of cases, including follow-on awards and extensions, local or regional entities, and exclusive or predominant capability, among others. Citizens’ Voice Project Final Performance Evaluation 40 LESSONS LEARNED CVP laid the foundation for positive beneficiary and grantee experiences by selecting efficient grantees and engaging the local community in the grant activity design phase. While the grantee selection process faced delays, CVP eventually chose highly qualified grantee organizations with proven experience implementing activities. With strong grantees and through consultations with local communities, CVP developed activities that were relevant and responded to citizen needs. These early good decisions paid dividends throughout the grant cycle. Citizens who understand their rights, are informed about issues, and know how to petition the government to redress grievances are powerful and sustainable change agents in their communities. Grant results can become sustainable when citizens are aware of their rights and when they are able to think broadly about the ways their rights are being violated. CVP showed that some beneficiaries continued to work with the government after grants had closed to continue to solve problems. Sustainability becomes possible when grantees and citizens institutionalize habits formed during grant activities and when citizen action requires no outside assistance. The government and the local community are the life support system of grant effectiveness and sustainability. Connection to political power structures and to target communities is a prerequisite to both grant effectiveness and sustainability. Grants cannot be implemented in a vacuum or from a distance. CVP demonstrated that an understanding of the target population through frequent consultations and building relationships with powerful individuals and political organizations enhances effectiveness. Because the government proved to be so effective at powering and sustaining grant activities, local communities, grantees, and the government should meet as early and as often as possible in the grant process to create goodwill and buy-in. Citizens’ Voice Project Final Performance Evaluation 41 ANNEXES Annex 1: Evaluation Statement of Work PERFORM Contract Assignment #XX: Evaluation DO3: Stabilization, Citizen’s Voice Project (CVP) Scope of Work Period of Performance: Late October 2015 – Early February 2016 Relevant/Target Decision Timelines: 8/2015 – SOW finalized 8/2015 – Initial AWP finalized 11/2015 – Data Collection begins 1/2016—Presentation of preliminary findings, conclusions and recommendations 1/2016 – Draft evaluation report submitted 2/2016 – Final report submitted Purpose The purpose of this assignment is to inform the design of a future project to enhance the capacity of civil society to advocate for a more democratic, transparent and accountable GOP at all levels of government--national, provincial and local. Relevant/Target Decision Timelines: This will be the first evaluation of the CVP. The evaluation report will be submitted in January 2016 and exclusively focus on a sample of the 144 closed grants. The evaluation will be used to help inform the design of the successor project. Background Activity/Project description: USAID’s Citizens’ Voice Project seeks to support the evolution of democratic governance by strengthening citizens’ voice and public accountability in government planning and decision-making. Under the project, approximately 500 grants are to be awarded to Pakistani entities across the country. On a quarterly basis, applications for grants will be solicited through print and other media for selected priority themes. Grants will support innovative ideas and high-quality competitive applications from eligible organizations. The second objective of the Project is to build capacity of private and civil society sector organizations in program implementation and organizational development. The capacity building responsibilities are performed with a focus on grantees, though the project has also been proposing to expand the scope of trainings to include potential grantees as well. Citizens’ Voice Project Final Performance Evaluation 42 As of August 2015, there are 100 active grants and 144 grants have since closed. The final tranche of approximately 50 grant awards is expected to be made in October 2015. The contract for the Strengthening Citizens Voice and Public Accountability Project (hereinafter “Citizens’ Voice Project”) was signed between the United States Agency for International Development (USAID) and the Trust for Democratic Education and Accountability (TDEA) on May 27, 2011. The Asia Foundation (TAF) and Anjum Asim Shahid Rahman-Grant Thornton (AASR-GT) are sub-contractors involved in project implementation. Initially awarded for three years, the Project received an extension for two more years in March 2013 and will now be completed in May 2016. Development Hypothesis and Theory of Change: The project is based on the understanding that the links between the government and the governed in Pakistan are weak; and that the absence of substantive public debate and discussion about key social, economic and political issues often exacerbates the lack of public trust and confidence in government institutions. The goal of the project is to “improve engagement between citizens and government on priority program-supported initiatives to advance good governance objectives”. The Results Framework of the Citizens’ Voice Project is shown in Figure 1. The Citizens’ Voice Project will pursue its overall goal by focusing on three results sub-divided into intermediate results. Figure 1: Results Framework of the Citizens’ Voice Project Citizens’ Voice Project Final Performance Evaluation 43 Anticipated Results and Associated Performance indicators: Please refer to attached project M&E plan for the full performance indicator matrix. List of Existing Project Documents: The below project documents are available on PakInfo and further documents can be provided to the assignment team per request.  Citizens Voice Project Success Story  Annual Report 2011-12  Full Annual Report 2012-2013  Full Annual Report 2013-2014  Quarterly Progress Report (May 27-June 30, 2011)  Quarterly Progress Report (July-Sept 2011)  Quarterly Progress Report (October 1 - December 31, 2011)  FINAL 2014 12 CVP Full Quarterly Report  Quarterly report Jan-Mar 2012  Quarterly Report July-Sept 2012  Quarterly Report Oct-Dec 2012  Quarterly Report Jan-Mar 2013  Quarterly Report July-Sep 2013  Quarterly Report Oct-Dec 2013  Quarterly Report January -March 2014  2014 CVP Jul - Sept Quarterly Report  Citizens' Voice Project Activity M&E Plan  M&E Manual  Citizens' Voice Project Annual Work Plan 2012-2013  Citizens' Voice Project Annual Work Plan 2013-2014  Grants Manual Citizens’ Voice Project Final Performance Evaluation 44 Purpose, Audience and Learning Objective The final evaluation of the CVP project will help determine the degree to which the project’s grantees achieved the results of the project, as stated in the above results framework. This information will be used to inform the design of the follow-on project. Assignment Purpose Intended Audience Learning Objective Information Sources Timeline To inform the design of future projects that seek to improve CSO capacity to advocate for a more transparent and accountable government. USAID/Pakistan Analyze the extent to which the grants under the contract mechanism enabled civil society to increase their ability to effectively advocate and engage in policy making. USAID/OSG, Trust for Democratic Education and Accountability, Grantees representing the 144 closed grants, their beneficiaries, and other stakeholders including representatives of GOP at local, provincial and national levels. Non-specific. While there is not a current plan for designing a new project in this area, it is likely to happen following approval of the DO3 PAD. List of Sites and Frequency for Monitoring (required for monitoring and if applicable, GIS tasks): A random sample of the 144 closed grants will be evaluated. The evaluation team will determine the size of the sample during the TPW. PERFORM will ensure that the sample will represent the priority thematic areas of the grants as determined by USAID. The data collection plan and interview protocols will be developed by the evaluation team during the Team Planning Workshop, and incorporated into a revised AWP. Key Evaluation Questions: Data collected from both current and closed grantees will be used to answer the following key analysis questions: 1) To what extent have the outputs and outcomes of the grants’ interventions aligned with and contributed to improved engagement between citizens and government on priority program￾supported initiatives to advance good governance objectives? 2) To what extent are the outputs and outcomes of the grants’ interventions able to be maintained (both financial and human capacity) beyond the life of the grant? 3) In what ways (both positive and negative) have the grants under this contract model affected the implementation of interventions and their alignment to the stated objectives of the project? Methodology: Below is an illustrative methodology for data collection and analysis. The methodology will be further refined during the Team Planning Workshop and incorporated into a revised Assignment Work Plan (AWP). Citizens’ Voice Project Final Performance Evaluation 45 Data Collection methods a) Review project performance monitoring data b) Structured observation—using site visit report form to be developed by PERFORM as part of the AWP. c) Unstructured observations—assignment team should include general notes and observations from their visits beyond the specifically requested questions/data points. d) Key Informant interviews—with relevant staff of grantees, TDEA, USAID/OSG, local, provincial and national governments, community leaders, beneficiaries and other stakeholders. Data Analysis methods a) Descriptive statistics: in each quarterly report as well as in the final synthesis report i. Frequency ii. Percentages iii. Ratios b) Content or pattern analysis to describe patterns in qualitative responses for the evaluation report. Team Composition:  Team Leader, Expat or local STTA Evaluation expert  4 local STTA evaluation specialists  Evaluation Specialist, PERFORM LTTA  Data Analyst, PERFORM LTTA Deliverables: 1. Assignment Work Plan (AWP): Proposed approach, methodology, timeline, staff composition, and estimated budget for completion of the work requested in the SOW. The AWP draft will be reviewed and approved by USAID before any work begins on the assignment. The AWP draft will be revised and finalized during the team planning workshop (TPW) once an assignment team is in country. The finalized AWP can be modified with PERFORM COR approval throughout implementation of the assignment if conditions or needs change. The finalized AWP with attached COR-approved amendments will be used as the basis for assessing completion and quality of the assignment. 2. Data Rehearsal: A presentation on the proposed methodology, data limitations, and potential challenges of data analysis will be provided to staff of the PMU and OSG near the conclusion of the TPW. 3. Preliminary Presentation of findings and conclusion: A presentation will be provided to relevant USAID/Pakistan staff at the conclusion of the data collection and analysis to solicit feedback and assist in the development of recommendations. 4. Draft Evaluation Report: To be submitted to the PMU following the preliminary presentation of evaluation findings and conclusions. 5. Final Evaluation Report: To be submitted following review of the draft by USAID/Pakistan. 6. Submission of data to USAID: Per ADS 579 - USAID Development Data –all primary data (both quantitative and qualitative) collected for this assignment will be submitted to USAID in electronic format within 30 days of completion of the evaluation. Citizens’ Voice Project Final Performance Evaluation 46 Estimated Level of Effort and Timeline Below is the estimated level of effort for all deliverables required under this assignment. The estimated LOE and Timeline may be adjusted during the TPW and will be reflected in the final AWP. Frequency TASK Team member(s) responsible LOE (days) of Team members Total LOE per team member One time AWP draft Performance Monitoring Advisor 5 5 One Time Finalize methodology, data collection instruments, and data collection and analysis plans Evaluation Team 42 6 One Time over 3 weeks Data collection Evaluation Team 126 18 One time Data Analysis and drafting of initial findings and conclusions Evaluation Team 112 16 One time Submit Draft Report Evaluation Team Lead 6 6 One time Revise draft and submit Final Report 6 6 One time Formatting and submission of raw data files Data analyst 3 3 Citizens’ Voice Project Final Performance Evaluation 47 Estimated Timeline 2015 2016 Task/Deliverable Aug Sep Oct Oct Nov Dec Jan Feb Draft SOW Approved AWP approved Team Recruitment Document Review Team Planning Workshop Revision to AWP Field Work Data Analysis and presentation of initial findings and conclusions Draft Report Final Report Citizens’ Voice Project Final Performance Evaluation 48 Annex 2: Assignment Work Plan Citizens’ Voice Project Final Performance Evaluation 49 Citizens’ Voice Project Final Performance Evaluation 50 Citizens’ Voice Project Final Performance Evaluation 51 Citizens’ Voice Project Final Performance Evaluation 52 Citizens’ Voice Project Final Performance Evaluation 53 Citizens’ Voice Project Final Performance Evaluation 54 Citizens’ Voice Project Final Performance Evaluation 55 Citizens’ Voice Project Final Performance Evaluation 56 Citizens’ Voice Project Final Performance Evaluation 57 Citizens’ Voice Project Final Performance Evaluation 58 Citizens’ Voice Project Final Performance Evaluation 59 Citizens’ Voice Project Final Performance Evaluation 60 Citizens’ Voice Project Final Performance Evaluation 61 Citizens’ Voice Project Final Performance Evaluation 62 Citizens’ Voice Project Final Performance Evaluation 63 Citizens’ Voice Project Final Performance Evaluation 64 Citizens’ Voice Project Final Performance Evaluation 65 Annex 3: Data Collection Instruments Pakistan Citizens’ Voice Project (CVP) Evaluation Instrument for Government Interviews Interviewee Name: Designation: Grantee Name: Grant Name: Department Name: Gender (Please check) Male Female Date : Time Start: Time End: Venue : City: Interviewer Name: Introduction Thank you very much for meeting us today. My name is __________. We are conducting a series of interviews with organizations that have been associated with the Citizens’ Voice Project (CVP) to better understand how to further encourage citizens’ participation and government accountability. We thank you for your time and recognize that your prior involvement is an important contribution to helping understand and shape future civic engagement activities that promote a stronger citizens’ voice and public sector accountability. The collected information will remain strictly confidential. As you will hear from our questions, the focus of our conversation will be on your knowledge of the prior CVP grant, grant-funded activities, and future directions. Do we have your permission to audio record the proceedings? Y/N If you are ready may we start? RELEVANCE: 1. How were the issues being addressed under the grant identified? PROMPTS: a. What were the issues being addressed under the grant? Citizens’ Voice Project Final Performance Evaluation 66 b. Were you involved in identifying the issues? R: 2. How were the grant activities relevant to key government priorities at the time? R: 3. What was the level of government engagement in grant activities? PROMPTS: a. Which departments were involved during the engagement? b. What was the duration of the engagement? c. Was the duration enough to achieve the grant objectives? d. Were any changes made in the partnership during the duration of the engagement? R: EFFECTIVENESS: 4. Did the grant activities include citizens' awareness about the rights of citizens and their understanding of the role of government? If yes, what was the nature of the awareness activities and their results? If no, please proceed to Q.5 PROMPTS: a. Which awareness activities did you find most and least effective, and why? b. What are some of the success stories that can be shared on awareness activities? c. If yes, did awareness lead to policy formulation and legislation? If yes, please give examples (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.)? R: 5. What was the nature and level of citizen participation during the engagement? PROMPTS: a. Did participation lead to policy formulation and legislation? If yes, give examples (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.) b. Which activity did you find most effective in increasing citizen participation? Why? R: 6. What types of civic-public linkages were developed during the engagement and with whom? PROMPTS: Citizens’ Voice Project Final Performance Evaluation 67 a. Public Private Partnership (PPP), media, corporate sector, business associations, bars, other CBOs etc.? b. Has the engagement continued? R: 7. What were the results of engaging with the grantee? PROMPTS: a. Was there for example, an increase in the number and diversity of citizen’s engagement, increased policy advocacy, network established etc. b. If there were no significant results, what were the reasons? c. Were there any lessons learnt as a result of this engagement? R: 8. Did the grant engagement lead to any policy change during or after the grant? If yes, what kind? If no, why do you think it didn’t? PROMPTS: a. What initiatives were undertaken to influence policy change? b. Please state examples of policy change (e.g. draft law, Charter of Demands, public petition). c. Could the grant engagement have been more effective at policy change? If yes, how? R: 9. Did the grant engagement lead to any change in government practices during or after the grant? If yes, what kind? If no, why do you think it didn’t? PROMPTS: a. If government practices changed, please give examples (government accessibility, improved service delivery, functioning complaint systems, etc.)? b. To what extent did the linkages influence the government to be more responsive? c. Could the grant engagement have been more effective at changing government practices? If yes, how? R: SUSTAINABILITY: 10. Has your engagement continued with the grantee or any other non-state actor that was involved during the implementation of the grant, after the grant closed? Citizens’ Voice Project Final Performance Evaluation 68 R: 11. Which of the results continued beyond completion of the grant? PROMPTS: a. To what extent have targeted government institutions sustained results of grant interventions? b. What were the key factors that contributed to sustainable results? (Please specify in terms of outputs and outcomes) R: 12. Did you receive any assistance under the project? If yes, what was the nature of the assistance? If no [end of the interview]. PROMPTS: a. Did the assistance involve trainings, orientations, study visits etc.? R: 13. Did you see any results from the assistance given under the project? If yes, what kind? If no, why do you think that was so? PROMPTS: a. Did you make any assistance requests that were not fulfilled by the project? b. Do you have any specific suggestions for such assistance component for the future? R: Citizens’ Voice Project Final Performance Evaluation 69 Pakistan Citizens’ Voice Project (CVP) Evaluation Instrument for Non-State Actors Interviews Interviewee Name : Designation: Grantee Name : Grant Name: Other Stake holder Name: Gender (Please check) Male Female Date : Time Start: Time End: Venue : City: Interviewer Name: Introduction Thank you very much for meeting us today. My name is __________. We are conducting a series of interviews with organizations that have been associated with the Citizens’ Voice Project (CVP) to better understand how to further encourage citizens’ participation and government accountability. We thank you for your time and recognize that your prior involvement is an important contribution to helping understand and shape future civic engagement activities that promote a stronger citizen voice and public sector accountability. The collected information will remain strictly confidential. As you will hear from our questions, the focus of our conversation will be on your knowledge of the prior CVP grant, grant-funded activities, and future directions. Do we have your permission to audio record the proceedings? Y/N If you are ready, may we start? RELEVANCE: 1. How did you get engaged with the CVP Project? PROMPTS: a. What was the level of the engagement? b. Did this engagement change over time? R: 2. What were the activities undertaken during the engagement? Citizens’ Voice Project Final Performance Evaluation 70 PROMPTS: a. Were the activities relevant to the local needs? b. Did the engagement facilitate in project implementation? If yes how? If no, why not? R: 3. What was your role in carrying out the activities? PROMPT: a. Were there any other partners involved in carrying out the activities? If yes, who all? R: 4. Was the government involved during the engagement activities? If yes, what was the level of government engagement? If no, please go to Q.5 PROMPT: a. How were the grant activities relevant to key government priorities at the time? b. Which departments were involved during the engagement? c. What was the duration of the engagement? d. Was the duration enough to achieve the grant objectives? e. Were any changes made in the partnership during the duration of the engagement? R: EFFECTIVENESS: 5. Did grant activities include citizens' awareness about the rights of citizens and their understanding of the role of government? If yes, which awareness activities did you find most and least effective? If no, please proceed to Q.6 PROMPTS: a. What are some of the success stories that can be shared on awareness activities? b. If yes, give examples of how awareness led to policy formulation and legislation (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.)? R: 6. What was the nature and level of citizen participation during the engagement? PROMPTS: Citizens’ Voice Project Final Performance Evaluation 71 a. Did participation lead to policy formulation and legislation? If yes give examples (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.) b. Which activity did you find most effective in increasing citizen participation? Why? R: 7. What types of civic-public linkages were developed during the engagement and with whom? PROMPTS: a. Public Private Partnership (PPP), media, corporate sector, business associations, bars, other CBOs etc.? b. Has the engagement continued? R: 8. What were the results of engaging with the grantee? PROMPTS: a. Was there, for example an increase in number and diversity of citizen’s engagement, increased policy advocacy, network established IEC materials etc. b. If there were no significant results, what were the reasons? c. Were there any lessons learnt as a result of this engagement? R: 9. Did the grant engagement lead to any policy change during or after the grant? If yes, what kind? If no, why do you think it didn’t? PROMPTS: a. What initiatives were undertaken to influence policy change? Please state examples of policy change (e.g. draft law, Charter of Demands, public petition) R: 10. Did the grant engagement lead to any change in government practices during or after the grant? If yes, what kind? If no, why do you think it didn’t? PROMPTS: a. If government practices changed, please give examples (government accessibility, improved service delivery, functioning complaint systems, etc.)? b. To what extent did the linkages influence the government to be more responsive? Citizens’ Voice Project Final Performance Evaluation 72 R: SUSTAINABILITY: 11. Has your engagement with the grantee or any other non-state actor that was involved during the implementation of the grant, continued after grant closure? R: 12. Which of the results continued beyond completion of the grant? PROMPTS: a. To what extent have targeted government institutions sustained results of grant interventions? b. What were the key factors that contributed to sustainable results? (Please specify in terms of outputs and outcomes) R: 13. Did you receive any type of assistance under the project? If yes, what was the nature of the assistance? If no, please proceed to Q.15 PROMPTS: a. Did the assistance involve trainings, orientations, study visits etc.? b. How many of your colleagues benefitted from the assistance? c. Did you make any assistance requests that were not fulfilled by the project? d. Do you have any specific suggestions for such assistance component for the future? R: 14. Did you see any results from the assistance given under the project? If yes, what kind? If no, why do you think that was so? PROMPTS: a. Did you make any assistance requests that were not fulfilled by the project? b. Do you have any specific suggestions for such assistance component for the future? R: 15. If you were involved in a similar project again, what would you do differently? PROMPT: a. Based on your experiences, what activities would you replicate, add or drop in the similar project and why? Citizens’ Voice Project Final Performance Evaluation 73 R: 16. What are the lessons learned from the project? If there are any success stories, please share. R: Citizens’ Voice Project Final Performance Evaluation 74 Pakistan Citizens’ Voice Project (CVP) Evaluation Instrument for Grantee Interviews Interviewee Name : Designation: Grantee Name: Grant Name: Gender (Please check) Male Female Date : Time Start: Time End: Venue : City: Interviewer Name: Introduction Thank you very much for meeting us today. My name is _________. We are conducting a series of interviews with organizations that have been associated with the Citizens’ Voice Project (CVP) to better understand how to further encourage citizens’ participation and government accountability. We thank you for your time and recognize that your prior involvement is an important contribution to helping understand and shape future civic engagement activities that promote stronger citizens’ voice and public sector accountability. The collected information will remain confidential. As you will hear from our questions, the focus of our conversation will be on your knowledge of the prior CVP grant, grant-funded activities, and future directions. This is a semi-structured interview and will be the same for all other Grantees. Do we have your permission to audio record the proceedings? Y/N If you are ready, may we start? RELEVANCE: 1. How were the grant activities selected? Were they relevant to the needs of the target population? If yes, how do you think they were relevant? If no, why do you think they weren’t? PROMPTS: Citizens’ Voice Project Final Performance Evaluation 75 a. Were the activities based on the civic engagement needs of community, beneficiaries, or specialized expertise of the grantee? b. Was there any change from the original set of activities implemented? If yes, why? c. What changes are needed to improve the selection and implementation of activities relevant to the actual needs of the community? R: 2. What is your feedback on the amount and duration of the grant? PROMPTS: a. Did you have enough time to conduct the activities as planned? If not, why? b. What in your opinion is the ideal time frame for such grants? c. If the funds were not sufficient, how did you manage the planned activities? R: 3. Were the grant objectives aligned with the objectives of CVP? [Note: Give them examples of CVP objectives to stimulate the conversation]. If yes, how were they aligned? If no, why do you think that is so? PROMPT: a. How does the goal of the grant relate to CVP objectives? R: 4. Were grant activities relevant to the requirements of the government? If yes, how do you think they were? If no, why do you think they were not? PROMPTS: a. What were the issues being addressed under the grant? b. How were the issues identified? c. How were the grant activities relevant to key government priorities at the time? d. Were you involved in identifying the issues? R: EFFECTIVENESS: 5. Did grant activities include citizens' awareness about the rights of citizens and their understanding of the role of government? Citizens’ Voice Project Final Performance Evaluation 76 If yes, what was the nature of the awareness activities? If no, can you explain why they weren’t included? [Go to Question 7]. PROMPTS: a. Which awareness activities were undertaken in the project? R: 6. Which awareness activities did you find most and least effective? PROMPTS: a. What are some of the success stories that can be shared on awareness activities? b. What were some of the most successful citizens' awareness activities: IEC materials, media outreach, better turnout of elections, organization of citizen’s groups, advocacy etc. c. What difference have the awareness activities made in terms of reaching the grant objective? d. Did awareness activities lead to policy formulation and legislation? If yes, give examples (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.)? If no, why not? R: 7. What was the nature and level of citizen participation in the project? How effective was the citizen participation? Which activities were the most successful in increasing citizen participation? PROMPTS: a. Did participation involve disadvantaged groups (women, minorities, disabled, other socially excluded groups, etc.) b. Did participation lead to policy formulation and legislation? If yes, give examples (legislation, change in laws, rules of business, oversight policies, membership of commissions etc.). If no, why not? c. Which activity did you find most effective in increasing citizen participation? Why? R: 8. What types of linkages were developed under the project and with whom? PROMPT: a. Public Private Partnership (PPP), media, corporate sector, business associations, bars, other CBOs etc.? R: 9. Which linkage(s) proved most effective in terms of project objectives? R: 10. Was the government involved during the engagement activities of the grant? Citizens’ Voice Project Final Performance Evaluation 77 If yes, what was the level of government engagement and the results? If no, please go to Q.11 PROMPTS: a. What was the result of engaging with the government? (E.g. increase in number and diversity of citizen’s engagement, increased policy advocacy, network established etc.) b. What were the reasons for the ineffectiveness, if any? c. Were there any lessons learnt for the government as a result of this engagement? R: 11. Did the grant lead to any policy change during or after the grant? If yes, what kind? If no, why do you think it didn’t? PROMPTS: a. What initiatives were undertaken to influence policy change? b. Was any policy change initiated due to grant interventions? If yes, please state examples (e.g. draft law, Charter of Demands) c. Were any initiatives converted into law or policy change? If yes, please give examples. If not, please state reasons. R: 12. Did government practices change during or after the grant duration? If yes, how? If no, why do you think they didn’t? PROMPTS: a. If government practices changed, please give examples (government accessibility, improved service delivery, functioning complaint systems, etc.)? b. To what extent did the linkages influence the government to be more responsive? R: SUSTAINABILITY: 13. How did you intend on sustaining the grant’s results? PROMPTS: a. How was the grant intended as a sustainable operation? (e.g. advocacy, awareness, trainings, partnerships, networks, collaborations)? R: Citizens’ Voice Project Final Performance Evaluation 78 14. What were the results that continued beyond completion? PROMPTS: a. To what extent have targeted government institutions sustained results of grant interventions? b. What were the key factors that contributed to sustainable results? Prompt: please specify in terms of outputs and outcomes. R: 15. Were the capacity building/institutional trainings relevant to the organization needs? If yes, how? If no, why not? R: 16. How many participants were trained in the trainings? PROMPT: a. Did the grant facilitate grantee staff participation in different training events for enhancing their skills in communication, leadership, team building, media relations, oversight of public bodies, project themes, proposal writing, IEC, etc.? R: 17. Which training and capacity building activity was most beneficial for grantee civil society officials? PROMPT: a. What specific suggestions can you provide for improving the capacity building component for the future? R: 18. How effective has CVP been in improving CSOs’ ability in strengthening their organizational, administrative and functional capacities? R: 19. To what extent has the CVP program increased the capability of the grantee to achieve its organizational mission? R: 20. Describe your experience with the grant making process. PROMPTS: a. How did you find the grant process? (question for both contractor/IP and grantee) b. What was the grant evaluation process? Was it a transparent process? (question for both contractor/IP and grantee) Citizens’ Voice Project Final Performance Evaluation 79 c. Did the grant process require spending your own money and time in preparing an application? (grantee only) d. Did you require outside resources (consultants, for example) to prepare the proposal? (grantee only) e. Were grant criteria and limitations specified? (both) f. What is your feedback on grant size and duration? g. How was the size of the grant determined? In your opinion, should there be a minimum size and duration of the grant? Why and what size? h. Was the duration of the grant adequate to accomplish the proposed activities? R: 21. What were the most effective and least effective aspects of the grant management model? PROMPT: a. Discuss respect to implementation such as timeliness of decision making, disbursements of funds, reporting (#s of reports and complexity), target beneficiaries, etc. R: 22. What improvements would you like to make to the grant management model? PROMPT: a. Discuss respect to with respect to implementation such as timeliness of decision making, disbursements of funds, reporting (#s of reports and complexity), target beneficiaries, etc. (question for both contractor/IP and grantee) R: 23. If you apply for a grant again, what would you do differently? PROMPTS: Broad Areas a. Are there better ways of focusing grant funds for improving citizen engagement with government, and improving advocacy and citizen participation? b. Should grants be government-centered or people-centered? Why? Narrow Areas c. Having discussed the grant making process what are the two key/critical areas of concern to you in doing things differently in the future? Why? d. How would you change these two or three areas to do them better in the future? e. Are there areas of doing grants and grant processes that we have not discussed that you see as important? R: Citizens’ Voice Project Final Performance Evaluation 80 CLOSING QUESTIONS: 24. If you were involved in a similar project again, what would you do differently? PROMPT: a. Based on your experiences, what activities would you replicate, add or drop in the similar project and why? R: 25. What are the success stories or lessons learned from the project? R: Citizens’ Voice Project Final Performance Evaluation 81 Pakistan Citizens’ Voice Project (CVP) Evaluation Instrument for Focus Groups and Beneficiaries Introduction Thank you very much for meeting us today. Our names are __________ and we are working with a team of evaluators searching for ways in which to enhance civic participation in public decision making. We are conducting a series of interviews with those who have been associated with the Citizens’ Voice Project (CVP) to better understand how to further encourage citizens’ participation and government accountability. We thank you for your time and recognize that your prior involvement is an important contribution to helping understand and shape future civic participation activities that promote stronger citizens’ voice and public sector accountability. The collected information will remain confidential. It is for the purpose of this focus group discussion only. As you will see from our questions, the focus of our conversation will be on your knowledge of the prior CVP grant, grant-funded activities and results, and future directions. Do I have permission to record? Y/N ______ INTRODUCTORY QUESTIONS: These will be broad contextual questions relating to the theme of each grant. For instance, if it is a water project, the question(s) could be water related issues. 1. What are the major ____ related issues in your area? 2. Who are the key stakeholders in this regard? FOCUS GROUP DISCUSSION: Section 1: RELEVANCE 3. What was this project about? 4. What were the problems/issues that this project addressed? 5. Were the project activities relevant to your needs? If yes, please explain how? If not, why do you think they weren’t? Section 2: EFFECTIVENESS 6. Did you see any results from the activities you were involved in? If yes, what were the results? If there were no results, why do you think that was so? Citizens’ Voice Project Final Performance Evaluation 82 PROMPTS: a. How did you participate in this project - as an individual or a group? b. Please provide examples of the most successful activities to accomplish the results/benefits? c. Have you experienced any changes as a result of your participation? d. Which awareness related activities were undertaken in the project? e. Which awareness activities did you find most and least effective? f. Did citizen awareness change as a result of this project? If Yes/not, how? g. What was the nature and level of citizen participation with government in the project? h. Did participatory activities involve disadvantaged groups (women, minorities, disabled, other socially excluded groups, etc.) If yes, how? If no, why do you think the project did not involve disadvantaged groups? i. What types of linkages were developed under the project and with whom? j. Which linkage(s) proved most effective in terms of project objectives? 7. Did you and/or your organization receive any training under the grant? If yes, how effective were these trainings? If no please proceed to Q.8 PROMPTS: a. Which were the most beneficial trainings and why? b. How were these trainings relevant to the grant objectives? c. Did you make any training requests that were not fulfilled by the project? d. How effective was the training in improving beneficiaries’ capacity? (organizational, administrative, financial management etc.) 8. Did the grant include engagement with the government and citizens (local, provincial and federal)? If yes, what was the nature and level of engagement? What was your role in these activities? If no, please proceed to Q.9 PROMPTS: a. How did your engagement with the government change after being involved in this project? b. Did the engagement facilitate project implementation? If yes, how? If no, why not? c. Have any participatory mechanisms been developed? If yes, how? If no, why not? d. Has provision of services improved as a result of this project? If yes, how? If no, why not? Citizens’ Voice Project Final Performance Evaluation 83 9. Has the government changed the way it responds to your needs since your involvement in the project? If yes, how? If no, why do you think that is? PROMPTS: a. How do you think the government could show improved responsiveness to the citizens? Please explain your answer by giving examples. b. What changes have you experienced with the government in terms of transparency and accountability? c. What changes in government practices have been experienced? Section 3: SUSTAINABILITY 10. If the activity you were involved in generated any results/benefits, did they continue after the grant activity ended? If yes, please explain which of these /benefits have continued and how? If no, why not? PROMPTS: a. Which factors contributed to the continuity/discontinuity of the /benefits? How will the discontinuity affect the grant benefits? 11. Have the targeted local level institutions sustained results/benefits of grant activities? If yes, how do you think they have? If no, why do you think they haven’t? PROMPTS: a. Have both government and non-government institutions sustained results/benefits? b. How have the results/benefits of the grant been sustained by state/non-state institutions? c. Were the linkages developed sustainable? If yes, are these continuing? If not, what are the reasons for discontinuity? 12. Were you able to expand upon the project results/benefits through your own efforts? PROMPTS: a. Did you attempt to mobilize additional resources (financial, human, networks, etc.)? If yes, what kind? b. If you couldn’t, why not? 13. If you were involved in a similar project again, what would you do differently? Citizens’ Voice Project Final Performance Evaluation 84 14. Based on your experiences, what activities would you replicate, add or drop in the similar project and why? 15. What are the success stories or lessons learned from the project? MODERATOR OBSERVATIONS: Citizens’ Voice Project Final Performance Evaluation 85 Annex 4: Data Collection Team and Fieldwork Coordination A team of five sector specialists covered five regions/provinces: Islamabad, Khyber Pakhtunkhwa (KP), Balochistan, Sindh, and Punjab. Each specialist team included two moderators responsible for conducting focus group discussions (FGDs). TDEA arranged the evaluation team’s visits with the selected grantee organizations. The grantees identified the relevant government and civil society/media stakeholders engaged with their grants and organized the team’s interviews with the stakeholders. The evaluation team gave grantees general criteria for selecting beneficiaries, which included:  Equal representation of men and women, with special attention to minorities and other marginalized groups;  Beneficiaries should have participated in grant activities; and  Beneficiaries should be accessible and willing to participate in the FGDs. The team capped participation in the FGD at 12 individuals. In some cases, grantees selected the 12 people themselves according to the criteria. In other cases, grantees provided a list of 20 or more potential FGD participants who met the criteria, and the evaluation team selected 12 participants randomly from the list. Citizens’ Voice Project Final Performance Evaluation 86 Annex 5: List of Documents Reviewed During the course of the evaluation, the team reviewed the following documents. 1. 40 CVP grant proposals and project completion reports 2. CVP project documents: a. Annual Report 2011-12 b. Full Annual Report 2012-2013 c. Full Annual Report 2013-2014 d. Quarterly Progress Report (May 27-June 30, 2011) e. Quarterly Progress Report (July-Sept 2011) f. Quarterly Progress Report (October 1 - December 31, 2011) g. FINAL 2014 12 CVP Full Quarterly Report h. Quarterly Report Oct-Dec 2012 i. Quarterly Report Oct-Dec 2013 j. Quarterly Report January -March 2014 k. 2014 CVP Jul - Sept Quarterly Report l. Citizens' Voice Project Performance Management Plan m. M&E Manual n. Citizens' Voice Project Annual Work Plan 2012-2013 o. Citizens' Voice Project Annual Work Plan 2013-2014 p. CVP Grants Manual Citizens’ Voice Project Final Performance Evaluation 87 Annex 6: Conflict of Interest Statements Material intentionally removed to protect the confidentiality of team members. Citizens’ Voice Project Final Performance Evaluation 88 Annex 7: Grants with Grant Value and Duration by Province and District Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) FATA Islamabad Nowshera Advocacy for effective implementation of the Frontier Crimes Regulation (FCR) Center for Research and Security Studies (CRSS) Moong Qabail (We the Tribals) 82,501 13 FATA Mardan Khyber Agency Advocacy for effective implementation of the Frontier Crimes Regulation (FCR) Step Towards Empowerment of Pupil Nawae Sahar (New Dawn) 144,452 15 AJK Muzaffarabad Muzaffarabad Energy sector reforms Global Prosperity Network (GPN) Citizens’ Voice on Energy Crises 64,165 12 AJK Islamabad Hattian Energy sector reforms Strengthening Participatory Organization (SPO) Enhancing Access to and Control Over Energy Resources 76,184 18 Punjab Islamabad Rahim Yar Khan Education sector reforms Human Development Foundation (HDF) Advocacy Campaign for Education Sector Reforms in Pakistan 82,173 12 Punjab Islamabad Lahore Education sector reforms Leadership for Environment and Development– Pakistan Strengthening Civil Society Oversight of Education Decentralization in Punjab 82,444 12 Punjab Multan Multan Improving water rights AWAZ Foundation Pakistan: Centre for Development Services Improving Water Rights of Rural Communities in District Multan and Bahawalpur 97,115 12 Punjab Lodhran Lodhran Improving water rights Development Through Awareness and Motivation (DAMAN) Transforming the Thinking-A Project for Materializing the Water Rights and Entitlements 98,076 12 Citizens’ Voice Project Final Performance Evaluation 89 Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) Punjab Islamabad Bahawalpur Citizens’ voice for effective grievance redress through the offices of ombudsmen Development Through Awareness and Motivation (DAMAN) Empowering Citizens to Make Public Service Department Accountable Through Office of Ombudspersons in Multan Region 68,460 13 Punjab Islamabad Sargodha Citizens’ voice and public accountability in tax collection sector Governance Institutes Network International Citizens’ Voice and Public Accountability in Tax Collection Sector 107,607 12 Punjab Islamabad Rawalpindi Citizens’ voice and public accountability in tax collection sector The Network for Consumer Protection in Pakistan Pay Income Tax 119,736 12 Punjab Islamabad Faisalabad Improving water rights Governance Institutes Network International Improving Water Rights in Central Punjab 0 12 Punjab Lahore Lahore Energy sector reforms Research Institute of Natural Resources of Pakistan Creating Social Impacts of Electricity Reforms in Pakistan 66,232 12 Punjab Khushab Khushab Education sector reforms Society for Human Empowerment and Rural Development (SHER) Making Schools Accountable to Performance 70,261 12 Punjab Islamabad N/A Importance of local government systems Inventure Private Limited Awareness and Education of Communities on Local Government System 14,416 4 Punjab Islamabad Lahore Energy sector reforms Semiotics Consultants (Private) Limited Energy Sector Reforms 75,087 12 Sindh Larkana Larkana Citizens’ voice for free, fair, and peaceful elections Community Development Network Forum Towards Ensuring Free, Fair and Peaceful General Elections 2013 17,456 4 Citizens’ Voice Project Final Performance Evaluation 90 Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) Sindh Tando Muhammad Khan Tando Muhammad Khan Citizens’ awareness for higher female voter turnout Aghaz Social Welfare Association Increased Female Turnout in Election 2013 25,551 4 Sindh Hyderabad Tando Allahyar Citizens’ voice and accountability for youth development Civil Society Support Program (CSSP) Empowering the Voice of a New Generation 62,688 13 Sindh Sukkur Jacobabad Citizens’ oversight of municipal services Goth Seengar Foundation Work Together for Improvement of Solid Waste Management System 71,417 15 Sindh Karachi Karachi Citizens’ voice for independent, free, and responsible media Institute of Business Management Media Responsibility and Independence Index (MRII) Survey 64,709 13 Balochistan Quetta Ziarat Citizens’ voice for effective legislative governance Water, Environment and Sanitation Society Citizens’ Voice for Enhanced and Effective Legislation and Accountability 134,135 12 Balochistan Quetta Killa Abdullah Citizens’ voice for free, fair, and peaceful elections Youth in Action Balochistan Improving Female Turnout Rate in General Election 2013 16,448 5 Balochistan Quetta Quetta Citizens’ voice for strengthening transparency and accountability mechanisms Development Resource Solutions (DERS) Giving Voice to the People Against Corruption in Balochistan 64,926 12 Balochistan Quetta Killa Abdullah Citizens’ engagement and accountability for an effective local government system Awareness on Human Rights Social Development and Action Society (AHSAS) Promoting Local Government Systems Through Citizens’ Engagement 130,087 12 Citizens’ Voice Project Final Performance Evaluation 91 Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) Balochistan Quetta Pishin Citizens’ voice and accountability for youth development Change Thru Empowerment Initiative for Mainstreaming Youth in Development and Accountability (IMYDA) Balochistan 67,134 13 Balochistan Quetta Quetta Citizens’ advocacy for implementation of 18th Amendment Yar Muhammad Samejo Educational Society (YMSESDO) Improving Social Service Delivery in Education Sector in Relation to 18th Constitutional Amendment 69,053 14 Balochistan Lasbela Lasbela Citizens’ advocacy for implementation of 18th Amendment Coastal Association Research and Development (CARD) Citizens’ Advocacy for Implementation of 18th Amendment 124,026 14 KP Charsadda Charsadda Importance of local government systems Network of Education and Economic Development Services (NEEDS) Importance of Local Government Systems 16,938 4 KP Mansehra Mansehra Citizens’ awareness for higher female voter turnout Global Educational, Economic and Social Empowerment Citizens’ Awareness for Higher Female Voter Turnout 21,291 4 KP Islamabad Dir-Lower Citizens’ oversight of municipal services Association for Behavior and Knowledge Transformation (ABKT) Improvement and Strengthening of TMA’s Municipal Service through Advocacy, Citizens’ Involvement and Oversight 68,596 9 KP Islamabad Nowshera Citizens’ voice for strengthening transparency and accountability mechanisms DELTA Association Strengthening Citizens’ Voice through Advocacy 77,966 12 Citizens’ Voice Project Final Performance Evaluation 92 Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) KP Islamabad Abbottabad Citizens’ voice for strengthening transparency and accountability mechanisms Centre for Peace and Development Initiatives (CPDI) Strengthening Transparency and Accountability Mechanisms in KP 127,952 12 KP Peshawar Peshawar Citizens’ voice for effective legislative governance Institute of Management Sciences/Center for Public Policy Research Strengthening Legislative Governance in Khyber Pakhtunkhwa 67,829 13 KP Islamabad Nowshera Citizens’ voice for independent, free, and responsible media Individualland Trust Engagement of Media on Responsibility towards Citizens in Khyber Pakhtunkhwa 74,963 13 KP Mansehra Manshera Citizens’ engagement and accountability for an effective local government system Pak Rural Development Program (PRDP) Active Citizen Engagement for Effective and Transparent Local Government System 53,530 14 KP Dera Ismail Khan Dera Ismail Khan Citizens’ voice for effective legislative governance Hamdam Development Organization (HDO) Awareness and Advocacy Campaign Aimed at Helping Society Better Interact with the Legislature at Members of Parliament of Provincial (KP) and National Assembly 134,320 18 KP Mardan Mardan Citizens’ voice for effective legislative governance Pak Women Citizens’ Voice for Effective Legislative Governance Project KP 152,394 18 Islamabad Capital Territory Islamabad N/A Energy sector reforms Sustainable Solutions Private Limited Raising Peoples’ Voice for Power Sector Reforms 77,340 12 Citizens’ Voice Project Final Performance Evaluation 93 Province Interview location for grantees Interview location for beneficiaries Theme Grantee organization Grant title Grant value (USD) Grant duration (months) Islamabad Capital Territory Islamabad Islamabad Citizens’ voice for effective grievance redress through the offices of ombudsmen Human Resource Development Network Mera Haqq (My Right) 74,569 13 Citizens’ Voice Project Final Performance Evaluation 94 Annex 8: Grants and Themes by Province Province Districts covered Themes covered No. of grants FATA  Khyber Agency  Advocacy for effective implementation of the Frontier Crimes Regulation (FCR) 2 AJK  Muzaffarabad  Hattian  Energy sector reforms 2 Punjab  Rahim Yar Khan  Lahore  Multan  Lodhran  Bahawalpur  Sargodha  Rawalpindi  Faisalabad  Khushab  Education sector reforms  Improving water rights  Citizens’ voice for effective grievance redress through the offices of ombudsmen  Citizens’ voice and public accountability in tax collection sector 12 Sindh  Larkana  Tando Muhammad Khan  Hyderabad  Tando Allahyar  Sukkur  Jacobabad  Karachi  Citizens’ voice for free, fair, and peaceful elections  Citizens’ awareness for higher female voter turnout  Citizens’ voice and accountability for youth development  Citizens’ oversight of municipal services  Citizens’ voice for independent, free, and responsible media 5 Citizens’ Voice Project Final Performance Evaluation 95 Province Districts covered Themes covered No. of grants Balochistan  Quetta  Killa Abdullah  Pishin  Lasbela  Ziarat  Citizens’ voice for effective legislative governance  Citizens’ voice for free, fair, and peaceful elections  Citizens’ voice for strengthening transparency and accountability mechanisms  Citizens’ voice and accountability for youth development  Citizens’ advocacy for implementation of 18th Amendment 7 Khyber Pakhtunkhwa  Charsadda  Mansehra  Lower Dir  Nowshera  Abbotabad  Peshawar  Dera Ismail Khan  Mardan  Importance of local government systems  Citizens’ awareness for higher female voter turnout  Citizens’ oversight of municipal services  Citizens’ voice for strengthening transparency and accountability mechanisms  Citizens’ voice for effective legislative governance  Citizens’ voice for independent, free, and responsible media  Citizens’ engagement and accountability for an effective local government system 10 Islamabad Capital Territory  Islamabad  Energy sector reforms  Citizens’ voice for effective grievance redress through the offices of ombudsmen  Importance of local government system 2 Citizens’ Voice Project Final Performance Evaluation 96 Annex 9: Grants That Reported Communication Problems with CVP Grantee organization Theme Region Cycle number Grant title Network of Education and Economic Development Services (NEEDS) Importance of local government system KP 6 Importance of Local Government System Global Educational, Economic and Social Empowerment Citizens’ awareness for higher female voter turnout KP 4 Citizens’ Awareness for Higher Female Voter Turnout Youth in Action Balochistan Citizens’ voice for free, fair, and peaceful elections Balochistan 5 Improving Female Turnout Rate in General Election 2013 Research Institute of Natural Resources of Pakistan Energy sector reforms Punjab 3 Creating Social Impacts of Electricity Reforms in Pakistan DELTA Association Citizens’ voice for strengthening transparency and accountability mechanisms KP 3 Strengthening Citizens’ Voice through Advocacy Leadership for Environment and Development–Pakistan Education sector reforms Punjab 2 Strengthening Civil Society Oversight of Education Decentralization in Punjab Development Through Awareness and Motivation (DAMAN) Improving water rights Punjab 1 Transforming Thinking - A Project for Materializing Water Rights and Entitlements Individualland Trust Citizens’ voice for independent, free, and responsible media KP 4 Engagement of Media on Responsibility Towards Citizens in Khyber Pakhtunkhwa Center for Research and Security Studies (CRSS) Advocacy for effective implementation of the Frontier Crimes Regulation (FCR) FATA 3 Moong Qabail (We the Tribals) Pak Rural Development Program (PRDP) Citizens’ engagement and accountability for an effective local government system KP 3 Active Citizen Engagement for Effective and Transparent Local Government System Citizens’ Voice Project Final Performance Evaluation 97 Annex 10: Grants That Reported Payment Problems with CVP Grantee organization Theme Region Cycle number Grant title Aghaz Social Welfare Association Citizens’ awareness for higher female voter turnout Sindh 4 Increased Female Turnout in Election 2013 Civil Society Support Program (CSSP) Citizens’ voice and accountability for youth development Sindh 4 Empowering the Voice of a New Generation Coastal Association Research and Development (CARD) Citizens’ advocacy for implementation of 18th Amendment Balochistan 3 Citizens’ Advocacy for Implementation of 18th Amendment Development Through Awareness and Motivation (DAMAN) Improving water rights Punjab 1 Transforming Thinking - A Project for Materializing Water Rights and Entitlements Global Prosperity Network (GPN) Energy sector reforms AJK 3 Citizens’ Voice on Energy Crises Goth Seengar Foundation Citizens’ oversight of municipal services Sindh 1 Work Together for Improvement of Solid Waste Management System Governance Institutes Network International Improving water rights Punjab 1 Improving Water Rights in Central Punjab Governance Institutes Network International Citizens’ voice and public accountability in tax collection sector Punjab 2 Citizens’ Voice and Public Accountability in Tax Collection Sector Hamdam Development Organization Citizens’ voice for effective legislative governance KP 2 Awareness and Advocacy Campaign Aimed at Helping Society Better Interact with the Legislature and Members of Provincial (KP) and National Assemblies Individualland Trust Citizens’ voice for independent, free, and responsible media KP 4 Engagement of Media on Responsibility Towards Citizens in Khyber Pakhtunkhwa Institute of Business Management Citizens’ voice for independent, free, and responsible media Sindh 4 Media Responsibility and Independence Index (MRII) Survey Institute of Management Sciences/Center for Public Policy Research Citizens’ voice for effective legislative governance KP 2 Strengthening Legislative Governance in Khyber Pakhtunkhwa Network of Education and Economic Development Services (NEEDS) Importance of local government system KP 6 Importance of Local Government System Citizens’ Voice Project Final Performance Evaluation 98 Grantee organization Theme Region Cycle number Grant title Pak Rural Development Program (PRDP) Citizens’ engagement and accountability for an effective local government system KP 3 Active Citizen Engagement for Effective and Transparent Local Government System Pak Women Citizens’ voice for effective legislative governance KP 2 Citizens’ Voice for Effective Legislative Governance Project KP Research Institute of Natural Resources of Pakistan Energy sector reforms Punjab 3 Creating Social Impacts of Electricity Reforms in Pakistan Society for Human Empowerment and Rural Development (SHER) Education sector reforms Punjab 2 Making Schools Accountable to Performance Water, Environment and Sanitation Society Citizens’ voice for effective legislative governance Balochistan 2 Citizens’ Voice for Enhanced and Effective Legislation and Accountability Yar Muhammad Samejo Educational Society (YMSESDO) Citizens’ advocacy for implementation of 18th Amendment Balochistan 3 Improving Social Service Delivery in Education Sector in Relation 18th Constitutional Amendment Youth in Action Balochistan Citizens’ voice for free, fair, and peaceful elections Balochistan 5 Improving Female Turnout Rate in General Election 2013 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523