October 10, 2014 This publication was produced for review by the United States Agency for International Development by Marc Shiman, Janet Robb, Ahmad Jameel, Vardah Malik, and Haider Ali Daud Khan. It was prepared by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP). Assessment and Strengthening Program (ASP): Interim Evaluation Report ACKNOWLEDGMENTS The authors would like to express their thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:  Officials of Assessment and Strengthening Program (ASP) implementing partners, i.e., Rural Support Programmes Network (RSPN), Associates in Development (AiD), and Lahore University of Management Sciences (LUMS), who shared their valuable time and insights with the evaluation team, provided a wealth of information through discussion and relevant documents, and facilitated successful field visits;  Officials of ASP beneficiary organizations, including the Small and Medium Enterprise Development Authority (SMEDA), the Provincial Reconstruction Rehabilitation and Settlement Authority (PaRRSA), the Finance Department of the Government of Khyber Pakhtunkhwa (GOKP), the Municipal Services Program of Khyber Pakhtunkhwa (KP), the Municipal Services Delivery Program of Sindh, Jamshoro Power Company Limited, and the Federal Public Procurement Regulatory Authority (FPPRA), who participated in meetings and discussions with the evaluation team, and provided valuable information;  Officials of ASP-sponsored civil society organizations, including Faces Pakistan, STEP, Abaseen, Peace and Development Organization (PADO), The Layton Rahmatullah Benevolent Trust (LRBT), Integrated Rural Awareness and Development Organization (IRADO), Saranga Literary and Cultural Society, Friends Development Organization (FDO), Rahma Islamic Relief, Foundation for Integrated Development Action (FIDA) and Hashoo Foundation, who participated in group interviews and provided valuable information relevant to the evaluation;  USAID officials, who shared their insights and information through interviews; and,  Mr. Imran Baloch, who facilitated the team’s work. ASSESSMENT AND STRENGTHENING PROGRAM (ASP) INTERIM EVALUATION REPORT Contracted under: Order No. AID-391-C-13-00005 Monitoring and Evaluation Program (MEP) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Project Summary ..................................................................................................................................................................................... ii Executive Summary................................................................................................................................................................................. 1 Program Background .............................................................................................................................................................................. 5 Evaluation Purpose and Evaluation Questions ...............................................................................................................................12 Evaluation Methods and Limitations .................................................................................................................................................13 Findings and Conclusions ....................................................................................................................................................................17 Overall Conclusions and Recommendations ................................................................................................................................. 39 Bibliography ............................................................................................................................................................................................41 Annexes ...................................................................................................................................................................................................43 Annex 1: Evaluation Statement of Work ..............................................................................................................................43 Annex II: Bios of Team Members ...........................................................................................................................................86 Annex III: Data Collection Instruments ................................................................................................................................87 Annex IV: Manual Review Checklists ...................................................................................................................................116 Annex V: Selected Websites on Updated Government of Pakistan Policies and Procedures ..............................129 Annex VI: Office Memorandum for Revised Accounting Procedure for Revolving Fund Accounts ...................130 Annex VII: Disclosure of Any Conflicts of Interest ..........................................................................................................132 Annex VIII: Case Studies .........................................................................................................................................................137 Tables and Figures Table 1: Project Summary ..................................................................................................................................................................... ii Table 2: LUMS Training Activities by Tier Level and Field of Study ........................................................................................10 Table 3: RSPN and AiD Interventions (Ongoing and Completed) ...........................................................................................11 Table 4: Summary of ASP’s Capacity Building Interventions .....................................................................................................24 Table 5: Participants’ Perception of LUMS Courses Results.....................................................................................................31 Table 6: Percentage Particpation in LUMS Courses by Gender ...............................................................................................37 Table 7: Percentage Female Participation in LUMS Courses by Job Rank .............................................................................38 Figure 1: ASP Project Locations by Implementing Partner.......................................................................................................... iii Figure 2: ASP Theory of Intervention ................................................................................................................................................ 6 Figure 3: The Design of ASP and its Implementing Partners ....................................................................................................... 9 Figure 4: Training Participants by Gender ......................................................................................................................................36 ii PROJECT SUMMARY1 The United States Agency for International Development (USAID)’s Assessment and Strengthening Program (ASP) works with Pakistan’s government agencies and non-governmental organizations (including non-profit civil society organizations and for-profit private sector organizations) to provide institutional development support so that their management capacity meets standards required for transparent, accountable, and effective utilization of USAID resources. The program operates under three cooperative agreements. Table 1 summarizes key facts about ASP. TABLE 1: PROJECT SUMMARY 1 The information in this section is based on ASP project documents, including the cooperative agreements and amendments thereof. 2 One each, respectively, for the three implementing partners listed later in the table. Title / Field Program Information Agreement Numbers2 391-A-00-11-01201 391-A-00-11-01202 391-A-00-11-01203 Agreement Officer’s Representative (AOR) Feroz Shah (Mr.), Lead Financial Analyst, Office of Financial Management (OFM) Start Date October 12, 2010 Completion Date October 11, 2015 Location Nationwide Name of Implementing Partners (IPs)  Rural Support Programmes Network (RSPN)  Lahore University of Management Sciences (LUMS)  Associates in Development (AiD) USAID Objective Addressed Cross-cutting Objective 3: Pakistan’s Development Capacity Improved Budget  RSPN: US$20 million  LUMS: US$15.5 million  AiD: US$8.9 million Total US$44.4 million iii FIGURE 1: ASP PROJECT LOCATIONS BY IMPLEMENTING PARTNER iv ACRONYMS AiD Associates in Development ASP Assessment and Strengthening Program ASP-AiD Assessment and Strengthening Program/Associates in Development ASP-LUMS Assessment and Strengthening Program/Lahore University of Management Sciences ASP-RSPN Assessment and Strengthening Program/Rural Support Programmes Network CSO Civil Society Organization ERP Enterprise Resource Planning FPPRA Federal Public Procurement Regulatory Authority G2G Government-to-Government GENCO Power Generation Company GOKP Government of Khyber Pakhtunkhwa GOP Government of Pakistan GOS Government of Sindh HRM Human Resources Management IPs Implementing Partners IR Intermediate Result IT Information Technology KLB Kerry-Lugar-Berman Bill KP Khyber Pakhtunkhwa LUMS Lahore University of Management Sciences M&E Monitoring and Evaluation MEP Monitoring and Evaluation Program MSI Management Systems International NRSP National Rural Support Programme NGO Non-governmental Organization OFM Office of Financial Management (USAID) PC-1 Planning Commission Form Number 1 PC-V Planning Commission Form Number V PFM Public Financial Management PMU Program Management Unit PaRRSA Provincial Reconstruction Rehabilitation and Settlement Authority QPR Quarterly Progress Report RSPN Rural Support Programmes Network SGAFP Small Grant Ambassador Fund Program SMEDA Small & Medium Enterprises Development Authority SOW Statement of Work UNDP United Nations Development Program USAID United States Agency for International Development UPU Urban Policy Unit 1 EXECUTIVE SUMMARY Following the Paris Declaration3 and the Accra Agenda,4 the United States Agency for International Development (USAID) made it a priority to work with host country organizations. Under the Kerry-Lugar-Berman (KLB) Bill,5 the USAID/Pakistan Mission is required to consider using host country governmental organizations and non￾governmental organizations (NGOs) as implementing partners (IPs). Subsequently, the new Pakistan Assistance Strategy Report (2010-2014)6 stated, “U.S. assistance will be directed increasingly through a broad range of Government of Pakistan (GOP) institutions, as well as local non-governmental organizations (including the private for-profit sector), with the capacity to implement programs effectively and accountably.”7 The strategy report also noted “Pakistan’s limited capacity to absorb and effectively use external resources and public sector corruption” as “challenges to the effective implementation of U.S. and other donor assistance programs.”8 Given this context, the USAID/Pakistan Mission needed a way to systematically assess its potential partners’ capacity, to identify areas of risk, and to provide capacity development assistance. USAID/Pakistan, therefore, launched ASP with the objective of making “Pakistani organizations more transparent, accountable, and effective.”9 The Assessment and Strengthening Program (ASP) is designed to assist governmental organizations and NGOs (including civil society organizations [CSOs] and private sector organizations) selected to implement USAID￾financed projects in all sectors and throughout Pakistan. ASP is implemented through three implementing partners (IPs): Rural Support Programmes Network (RSPN), Associates in Development (AiD), and Lahore University of Management Sciences (LUMS). As of April 30, 2014, ASP assistance included support for 43 public sector, 99 non-profit, and nine for-profit organizations. The non-profit organizations include 78 CSOs, which are small, community-based organizations. Forty-two of the beneficiary organizations are located in Khyber Pakhtunkhwa (KP), 24 in Sindh, 28 in Islamabad, 24 in Punjab, 23 in Balochistan, three in Azad Jammu and Kashmir, and seven in Gilgit-Baltistan, and two have nationwide coverage. The evaluation assessed activities that had been completed and almost completed by March 2014, the evaluation cut-off date, including three of the five ASP components, as outlined below:10  Pre-award Assessment: assessment to determine whether a proposed recipient’s internal controls and administrative and financial management systems are adequate for effective utilization of USAID funds;  Capacity Building: interventions to address the institutional shortcomings identified in the ASP beneficiaries’ pre-award assessments; and 3 In the Paris Declaration (2005), donors committed “to use country systems and procedures to the maximum extent possible.” Available on page 5 at URL: http://www.oecd.org/development/effectiveness/34428351.pdf. 4 In the Accra Agreement for Action (2008), donors agreed “to use country systems as the first option for aid programmes in support of activities managed by the public sector.” Available on page 3 at URL: http://siteresources.worldbank.org/ACCRAEXT/Resources/4700790- 1217425866038/AAA-4-SEPTEMBER-FINAL-16h00.pdf. 5 The Enhanced Partnership with Pakistan Act of 2009 (popularly known as the Kerry-Lugar-Berman Act) states that “…as appropriate, to utilize Pakistani firms and community and local nongovernmental organizations in Pakistan, including through host country contracts…” Available on page 8 at URL: http://www.gpo.gov/fdsys/pkg/BILLS-111s1707enr/pdf/BILLS-111s1707enr.pdf. 6 According to section 302 (a) of the Enhanced Partnership with Pakistan Act of 2009, “Not later than 45 days after the date of enactment of this Act, the Secretary of State shall submit to the appropriate congressional committees a report describing United States policy and strategy with respect to assistance to Pakistan under this Act.” 7 Pakistan Assistance Strategy Paper (2009). Available on page 14 at URL: http://www.state.gov/documents/organization/134114.pdf. 8 Ibid, page 10. 9 The ASP Performance Management Plan updated in January 2013, page 8. 10 Program descriptions of cooperative agreements for ASP-AiD, ASP-RSPN, and ASP-LUMS. 2  Research: Studies on institutional aspects of host country organizations and development of best practices in the CSO sector. The evaluation fieldwork focused on ASP IPs and beneficiary organizations in Islamabad, Peshawar, Lahore, Karachi, and Jamshoro. The evaluation methodology included individual interviews with USAID and IPs, eight case studies of intensive capacity building project beneficiaries, individual interviews with Lahore University of Management Sciences (LUMS) personnel involved in the ASP training, and an online survey of 1,846 alumni of the ASP-LUMS courses. In total, the team conducted 67 individual interviews with USAID/Pakistan, ASP beneficiaries, and IPs, and four group interviews with CSOs. In addition to the interview and survey data, evaluation findings and conclusions are based on an extensive review of documents, including those of USAID/Pakistan, federal and provincial levels of the GOP, and beneficiary organizations. The evaluation’s main strength is the multiple layers of triangulation11 across data sources, information that was comparatively analysed with respect to each project component and then synthesized to assess the overall project effectiveness and institutionalization. The main limitation is the case study approach used, because some of the organizations included were not comparable to others. USAID/Pakistan identified three main evaluation questions, as follows: 1) To what extent have the program’s overall approach and individual components contributed to achieving the results set out in the ASP results framework? 2) To what extent are ASP results and the services provided by ASP IPs likely to be sustainable? 3) To what extent, if any, has ASP included gender as a consideration in its interventions? The evaluation addressed each of these questions, and many related sub-questions, and derived 10 main conclusions, on the basis of which three sets of recommendations are proposed. The overall conclusion is that ASP has been effective in improving compliance of beneficiary organizations through improved documentation and beneficiary employees’ awareness. ASP beneficiaries now have operational manuals in place and/or upgraded in the areas of finance, human resources, procurement, and/or monitoring and evaluation (M&E), thus meeting Intermediate Result (IR) 3: Improved Public Financial Management (PFM) Policies and Procedures. While enhanced compliance documentation alone may or may not make a significant contribution to improved organizational effectiveness, addressing IR 2: Human Resources Enabled for Enhanced Performance, improvements in beneficiaries’ employees’ skills through LUMS training will eventually contribute to improved organizational effectiveness. ASP has contributed to CSOs institutional capacity building, by providing them with the skills and knowledge to establish their own systems, thus achieving IR 1: Improved Management Systems Implemented. Conversely, IR 1 has not been fully achieved for public sector beneficiaries because the relevant governing bodies in some of the beneficiary organizations have not approved improved policies and procedures. Over the four years that ASP has been operating, it has transformed into a program of multiple projects with very different intended outcomes and methodologies. CSOs, in particular small grant awardees, had a different experience from public sector beneficiaries. With CSOs, ASP took a more demand-driven approach by providing beneficiaries with access to training classes and allowing them to plan and to lead their own institutional capacity building processes. Many of the CSOs that participated in ASP took part in their institutional assessment and planning for their capacity building through ASP. CSOs were assisted in a limited way in planning for their capacity building through two one-day consultative workshops on planning of capacity building and its implementation. CSOs were given training first, and then received generic manuals that they were to customize themselves only after they received skills training. CSOs were given a calendar of ASP-LUMS training events, and it was up to them to nominate relevant personnel for trainings. There was limited follow-up assistance in the form of combined sessions/trainings/workshops after organizations customized the ASP-provided manuals to ensure that there were no obvious gaps or to help with the development of forms and other tools (in particular, M&E tools). As a result, respondents feel that they fully 11 “Triangulation is a powerful technique that facilitates validation of data through cross verification” (Bogdan, R.C. and Biklen, S.K. (2006). Denzin (2006) identified four basic types of triangulation: 1) data triangulation – different sources, locations and time periods; 2) investigator `triangulation – multiple researchers independently examining the same data; 3) theory triangulation – use of multiple theoretical schemes; and 4) methodological triangulation – using more than one method to gather data. The more types of triangulation that are used the greater the reliability and validity of research results. 3 understand the contents of their manuals (which is not surprising given that they developed them) and that they are implementing them completely. They feel confident in their ability to update them as needed. Conducting the training in the local languages would have benefited smaller NGOs working in remote areas. In contrast, ASP took a relatively supply-side approach with public sector organizations, as the IPs, apart from providing some technical assistance in developing strategic plans, typically planned and conducted much of the capacity building process on behalf of most of the beneficiaries included in the evaluation. Public organizations received customized manuals followed by training on their contents. While document review indicates acceptance of the ASP-developed products by public sector organizations, ASP’s approach to CSOs likely resulted in a higher degree of ownership of the final product and changed some of the ways the CSOs went about doing business.12 ASP-LUMS training provided skills to many employees of the ASP beneficiaries. However, there was no formal training needs analysis. Instead ASP-LUMS conducted a two-day opening workshop in December 2011 to identify the topics that should be included in each course and obtain feedback on the design of the courses suggested by ASP-LUMS faculty for government and CSO beneficiaries. The topics of the courses were also based upon the recommendations of around 150 pre-award assessments conducted for the potential beneficiary organizations. While there was no evidence of a gender analysis in the original three ASP cooperative agreements, gender equality was given consideration, to varying degrees, in all three partners’ agreements or modifications. Each of the ASP partners strives for gender equality within their respective organizations. They do so in varying degrees through HR manuals and policies, employment terms, and creating a supportive workplace environment. The attention to gender concerns, especially gender equality, has been handled differently in the ASP-developed manuals for CSOs and those for public sector organizations. For public sector organizations the attention is far less significant and is limited to the need for gender-disaggregated data in carrying out the M&E function and to statements of equal employment opportunities when preparing job notifications. For CSOs, there is significant attention given to gender concerns through policies and procedures detailed in the HR manual. As all ASP-LUMS courses are gender-neutral, they developed a special course on “Leadership: Gender Dimension” to address the importance of gender and gender-based barriers to career mobility. However, the efforts of ASP￾LUMS to encourage females to attend training programs have had meager, though positive, effects in increasing the enrollment of females in its training programs. Nonetheless, ASP-LUMS female alumni are represented at a significantly lower percentage than their male counterparts, particularly as the job scale/rank grows higher. However, without comparing these statistics with industry-wide statistics, no conclusions can be made about the effect (if any) of the LUMS training on women's positions in the workplace. RECOMMENDATIONS For improved accountability and transparency, ASP should greatly increase its efforts and investment to establish internal audit departments in provincial governments in a similar vein to its efforts in KP. This will strengthen the crucial oversight/enforcement piece necessary to ensure that improved management systems and PFM policies and procedures are implemented sustainably as documented. ASP should continue and accelerate efforts to work in what ASP-RSPN refers to as “strategic institutions”, particularly those that have an oversight role such as regulatory authorities at the provincial and federal levels. ASP should also identify other opportunities to work with other regulatory oversight bodies (i.e., Ministry of Finance, Establishment Division, the Planning Commission, and the Auditor General of Pakistan) to enhance beneficiaries’ compliance with documentation. ASP took different strategies to working with CSOs and public sector organizations, and both had their shares of success and significant gaps in their service delivery. Lessons should be applied to each to make the other more robust. 12 Although, documents provided (in Dec 2014) by ASP-RSPN indicate that some technical assistance was provided to some public sector beneficiaries, the findings from case studies indicate that much of the capacity development process was done by the IPs on behalf of the beneficiaries. 4 The following are the specific recommendations resulting from this evaluation:  ASP should provide CSOs with more assistance in developing their own capacity building plans.  ASP should increase CSOs’ access to experts in each field to review the manuals they have created and to get questions answered.  ASP should prepare CSO training in local languages, not just English. Though ASP-LUMS courses are gender-neutral, LUMS should review its courses, wherever applicable, to identify opportunities to include issues of gender equality, including the understanding of gender gaps within a particular technical area, a profile of the status of men and women in terms of their leadership roles, the gender norms and strategies to mitigate the gaps and discrepancies. Strategies to address gender equality should extend beyond participation and inclusivity to include laws, policies, regulations, and institutional practices. 5 PROGRAM BACKGROUND The United States Agency for International Development (USAID) Assessment and Strengthening Program (ASP) is designed to assist governmental organizations and non-governmental organizations (NGOs) (including civil society organizations [CSOs] and private sector organizations) selected to implement USAID-financed projects in all sectors and throughout Pakistan. THE PROBLEM STATEMENT Following the Paris Declaration13 and the Accra Agenda,14 USAID made it a priority to work with host country organizations. Under the Kerry-Lugar-Berman (KLB) Bill,15 the USAID/Pakistan Mission is required to consider using host country governmental organizations and NGOs as implementing partners (IPs). Subsequently, the new Pakistan Assistance Strategy Report (2010-2014) stated, “U.S. assistance will be directed increasingly through a broad range of Government of Pakistan institutions, as well as local non-governmental organizations (including the private for-profit sector), with the capacity to implement programs effectively and accountably.”16 The strategy report also noted “Pakistan’s limited capacity to absorb and effectively use external resources and public sector corruption” as “challenges to the effective implementation of U.S. and other donor assistance programs.”17 THE THEORY OF THE INTERVENTION To comply with the Pakistan Assistance Strategy Report (2010-2014), USAID/Pakistan designed ASP with the objective of making “Pakistani organizations more transparent, accountable, and effective.” In addition, the report states, “This approach will help increase host country ownership and U.S. commitment to building a long-term partnership with the Pakistani government and people.”18 ASP’s results framework contains three intermediate results (IRs): IR 1: Improved Management Systems Implemented Sub-IR 1.1: Institutional capacity gaps understood by stakeholders Sub-IR 1.2: Management systems adopted IR 2: Human Resources Enabled for Enhanced Performance Sub-IR 2.1: Knowledge and skills of human resources enhanced Sub-IR 2.2: Application of knowledge and skills supported Sub-IR 2.3: Increased opportunities for young professionals in development sector IR 3: Improved Public Financial Management (PFM) Policies and Procedures Sub-IR 3.1: Policy recommendations advocated 13 In the Paris Declaration (2005), donors committed “to use country systems and procedures to the maximum extent possible.” Available on page 5 at URL: http://www.oecd.org/development/effectiveness/34428351.pdf. 14 In the Accra Agreement for Action (2008), donors agreed “to use country systems as the first option for aid programmes in support of activities managed by the public sector.” Available on page 3 at URL: http://siteresources.worldbank.org/ACCRAEXT/Resources/4700790- 1217425866038/AAA-4-SEPTEMBER-FINAL-16h00.pdf. 15 The Enhanced Partnership with Pakistan Act of 2009 (popularly known as the Kerry-Lugar-Berman Act) states that “as appropriate, to utilize Pakistani firms and community and local nongovernmental organizations in Pakistan, including through host country contracts.” Available on page 20 at URL: http://www.gpo.gov/fdsys/pkg/BILLS-111s1707enr/pdf/BILLS-111s1707enr.pdf. 16 Pakistan Assistance Strategy Paper (2009). Available on page 14 at URL: http://www.state.gov/documents/organization/134114.pdf. 17 Ibid, page 10. 18 Pakistan Assistance Strategy Paper (2009). Available on page 14 at URL: http://www.state.gov/documents/organization/134114.pdf. 6 Improved effectiveness and sustainability of U.S. assistance Increased host country ownership U.S. assistance increasingly directed through local organizations Improved transparency and accountability of selected Pakistani institutions Improved capacity of Pakistani institutions ASP also contributes to USAID/Pakistan’s Mission Strategic Framework cross-cutting Objective 4: “Pakistan’s Development Capacity Improved.” Figure 2 illustrates the ASP theory of intervention, depicting how project interventions are expected to achieve the goals set out in the Pakistan Assistance Strategy Report. FIGURE 2: ASP THEORY OF INTERVENTION APPROACH AND IMPLEMENTATION ASP is implemented through three IPs:  Rural Support Programmes Network (RSPN) generally focuses on, but is not limited to, providing support to USAID’s host government partners;  Associates in Development (AiD) focuses primarily on CSOs and for-profit organizations, but also assists government partners in Khyber Pakhtunkhwa (KP); and  Lahore University of Management Sciences (LUMS) specializes in providing on-campus training to participants from USAID partners, including ASP beneficiary organizations. 1. Pre-award/Other Assessments The purpose of pre-award/other assessments is to determine whether a proposed recipient’s internal controls and administrative and financial management systems are adequate for effective utilization of USAID funds. The pre￾award assessments are carried out by one of the chartered accounting firms that work with RSPN and AiD. The relevant USAID/Pakistan office selects potential IPs, and the Office of Financial Management (OFM) commissions either RSPN or AiD to conduct the pre-award assessment. Before ASP, USAID/Pakistan conducted pre-award assessments of host country organizations through Blanket Purchase Agreements, which USAID/Pakistan had in place with several Regional Inspector General-approved chartered accounting firms. The chartered accounting firms conduct the pre-award assessments in accordance with the assessment framework19 and the corresponding scopes of work20 developed by ASP. The pre-award assessment report provides an assessment of an organization’s financial management, procurement, administration and human 19 The framework assesses beneficiary organizations’ control environment, financial management, and accounting systems, procurement management systems, personnel policies and payroll systems, and M&E. 20 Assessment framework forms the basis for pre-award assessment scopes of work. 7 resources management (HRM) systems and controls, and absorptive capacity, concluding with a risk assessment for each system (i.e., low, medium, and high). This report is provided to OFM and discussed with the relevant USAID/Pakistan office. It provides the basis for a capacity development plan, as outlined below. In addition to pre-award assessments, ASP-RSPN also conducts “other assessments,” which include assessing, evaluating, reviewing, and validating other areas related to financial management, institutional governance, procurement, and management. 2. Capacity Building The key objective of this component is to address the capacity building issues of host country organizations identified in the pre-award assessments. These organizations can be classified both by type and by relationship with USAID. The three types are (1) Pakistani governmental or quasi-governmental organizations, (2) for-profit organizations, and (3) non-profit (often community-based) organizations. All three types of organizations are further classified as direct or indirect beneficiaries. RSPN classifies direct beneficiaries as partners that receive direct funding from USAID, while indirect beneficiaries do not receive direct USAID funding, but are strategically important for the performance of direct partners. ASP provides capacity building support in four key areas: 1) financial management, 2) procurement, 3) administration and HRM, and 4) Monitoring and Evaluation (M&E). RSPN provides capacity building support to USAID’s direct and indirect partners. AiD provides capacity building support to direct USAID partners, including Small Grant Ambassador Fund Program (SGAFP) awardees. LUMS provides classroom training to participants from all ASP beneficiary organizations and other USAID partners. RSPN and AiD’s support to partners is based on the risks identified in the pre-award assessments.21 ASP uses the pre-award assessment to develop an organization-specific capacity building plan.22 Capacity building support to USAID partners includes development of manuals in the key areas mentioned above; on-the-job staff training in these areas; and help in preparing Planning Commission Form Number 1 (PC-1s), which is the standard project planning document for public sector organizations. Once the concerned organization accepts the capacity building plan, it is sent to USAID/Pakistan for formal approval. After its approval, RSPN or AiD implements the plan through a chartered accounting firm, individual consultant, or members of its own staff. To avoid conflict of interest, the same ASP partner or chartered accounting firm does not carry out pre-award assessments and capacity building interventions. CSOs capacity building is based on either the pre-award assessments or institutional assessments23 conducted by the U.S. Embassy. AiD has prepared two capacity building plans, one each for the two groups (Batch1 and Batch 2) of CSOs. Capacity building support to these CSOs is provided by developing and sharing manuals AiD has developed in the fields of internal audit, financial management, HRM, information technology (IT), and procurement. Following this, AiD conducts hands-on training for CSO staff on how to use these manuals. LUMS provides capacity building support by conducting standard classroom (on-campus) training in the areas of financial management, procurement, administration and HRM, M&E, and related areas. In addition, LUMS provides training from time to time on areas such as communication strategy and management, fiscal decentralization, leadership, and cultural change. LUMS training programs target three tiers of USAID partner organization employees:  Tier 1: senior managers,  Tier 2: middle managers, and  Tier 3: operational executives. 21 The assessments used for developing Risk Mitigation Frameworks and capacity building plans to date are pre-ASP pre-award assessments conducted by various chartered accounting firms through the Blanket Purchase Agreement. 22 The capacity building plans are customized for each organization and for each institution. 23 These assessments are carried out by SGAFP using a standardized checklist. 8 In Years 2 and 3, ASP-LUMS offered three tiers of training. Based on the experience of those two years, Tiers 1 and 2 were merged into one single tier through Modification #6.24 From Year 4 onwards, Tier 1 represents a mix of senior and middle managers and Tier 2 represents operational executives. The courses are organized throughout the year, and each course is 3-5 days long. 3. Validation According to ASP cooperative agreements (Aid and RSPN), this component aims at conducting validation of beneficiary organizations that have received capacity building assistance to assess whether the organizations are in compliance with the standards and procedures developed under the capacity building plans. The annual validation component focuses on three specific objectives: 1) to ensure compliance with the standards and procedures identified by USAID and ASP; 2) to refer concerned organizations for further capacity building when gaps are identified; and 3) to develop benchmarks for improvement. So far, ASP has completed five validations. 4. Research This component focuses on conducting research on institutional aspects of host country organizations and development of best practices in the CSO sector in the following fields: 1) financial management, 2) procurement management, 3) administrative management, and 4) M&E. Through this component, ASP assesses the government’s existing policies and procedures in these fields and advises the government on changes in these policies and procedures with the aim of improving project planning and implementation. 5. Awardee Capacity Building The purpose of this component is to enhance the capacity of ASP IPs and their staff. To attain this, IPs prepare a detailed needs assessment of their respective organizations and identify areas where capacity building is required. The interventions under this component include ASP staff training and upgrading the IPs’ accounting systems, operational manuals, and IT infrastructure. ASP staff training fields include governance, financial management, procurement, HRM, management information systems, and USAID rules and regulations. Figure 3 below depicts the design of ASP and the primary activities of each of its IPs. 24 ASP-LUMS Cooperative Agreement, Modification #6 (January 2014). 9 FIGURE 3: THE DESIGN OF ASP AND ITS IMPLEMENTING PARTNERS COMPONENT MAIN ACTIVITIES/PURPOSE Pre-Award Assessment Validation Research on Policies and Procedures RSPN and AiD  Preparation of manuals on financial management, HRM, procurement and M&E  On-the-job training and mentoring exercise (for CSOs) in the four key fields  Help in PC-1 preparation  Establishing institutional infrastructure LUMS  Classroom training for beneficiary organizations in the four key fields Institutional Capacity Building Awardee Capacity Building RSPN and AiD  Assess organizations’ financial, procurement, and administration/HRM systems and controls. RSPN, AiD, and LUMS  Development of best practices in the fields of financial management, procurement and administrative management RSPN, AiD, and LUMS  Training of ASP IPs’ staff  Institutional capacity building of IPs RSPN and AiD  Ensuring compliance with pre-award assessment recommendations and standards and procedures developed under the institutional capacity building component. 10 CURRENT STATUS OF ACTIVITIES As of April 30, 2014, ASP assistance included support for 43 public sector, 99 non-profit, and nine for-profit organizations. The non-profit organizations include 78 CSOs, which are small, community-based organizations. Forty-two of the beneficiary organizations are located in KP, 24 in Sindh, 28 in Islamabad, 24 in Punjab, 23 in Balochistan, three in Azad Jammu and Kashmir, and seven in Gilgit-Baltistan, and two have nationwide coverage. The status of ASP activities as of April 2014 is as follows:  Pre-award assessments of five public sector, one non-profit, and nine for-profit private sector organizations have been completed.  Capacity building interventions in 80 non-profit and 25 public sector organizations have been completed.  Capacity building interventions in 11 public sector organizations either have been partially completed or are ongoing.  Validations of four public sector and one for-profit organization have been completed. Validations of 36 non-profit organizations and one public sector organization are ongoing.  Four research activities have been completed.  A total of 2,392 participants have received LUMS training, including 1,069 in Tier 1, 859 in Tier 2, and 464 in Tier 3. Table 2 below summarizes LUMS trainings, and Table 3 summarizes RSPN and AiD ASP interventions. TABLE 2: LUMS TRAINING ACTIVITIES BY TIER LEVEL AND FIELD OF STUDY Training Tier 1 (Senior Managers) Tier 2 (Middle Managers) Tier 3 (Operational Executives) Grand Total Financial Management 194 223 81 498 Procurement Management 139 255 105 499 HRM & Administration 148 203 133 484 M&E 88 145 145 378 Project Management 163 0 0 163 Leadership: Gender Dimension 111 0 0 111 Forensic Audit 31 0 0 31 Fiscal Decentralization 54 0 0 54 Procurement Audit 38 0 0 38 Training of Trainers 54 0 0 54 Communication Strategy 49 33 0 82 TOTAL 1,069 859 464 2,392 11 TABLE 3: RSPN AND AID INTERVENTIONS (ONGOING AND COMPLETED) Component Type of Partner Location Azad Jammu and Kashmir Balochistan Countrywide Gilgit￾Baltistan Islamabad KP Punjab Sindh Grand Total Pre-Award Assessments For-profit 3 4 2 9 G2G* 1 1 2 1 5 Non-profit 1 1 Total 1 4 6 4 15 Capacity Building: Direct Partners For-profit G2G 5 10 5 5 25 Non-profit 2 13 6 8 22 13 16 80 Total 2 13 6 13 32 18 21 105 Capacity Building: Indirect Partners G2G 1 4 5 1 11 Non-profit Total 1 4 5 1 11 Validation For-profit 1 1 G2G 2 3 5 Non-profit 1 9 1 1 1 5 18 Total 1 9 1 1 3 5 1 3 24 Research G2G 4 4 Total 4 4 Grand Total 3 23 2 7 28 42 24 26 155 *G2G: Government-to-Government 12 EVALUATION PURPOSE AND EVALUATION QUESTIONS The overall goals of this interim evaluation, according to the evaluation Statement of Work (SOW; see Annex I of the report), are to assess progress toward achievement of the program’s results, to determine successes and shortcomings and their underlying causes, and to provide recommendations for improving the program and implementation strategy. USAID/Pakistan and the ASP IPs (RSPN, AiD, and LUMS) are particularly focused on the effectiveness of the program approach and components, with the aim of identifying opportunities for improvement. USAID/Pakistan also is concerned with the project’s sustainability, which depends on developing the capacities of the selected beneficiary organizations and also the extent to which these organizations are institutionalizing improvements introduced with ASP assistance. Ways in which ASP and beneficiary organizations are addressing issues of gender equality are also an important area of inquiry for this evaluation. To address the purposes described above, the evaluation is expected to answer the following three key questions: 1. To what extent have the program’s overall approach and individual components contributed to achieving the results set out in the ASP results framework? What factors have influenced the program’s achievement of results? 2. To what extent are ASP results and the services provided by ASP implementing partners likely to be sustainable? 3. To what extent, if any, has ASP included gender as a consideration in its interventions? 13 EVALUATION METHODS AND LIMITATIONS DATA COLLECTION AND ANALYSIS METHODS The evaluation applies a mixed-method approach, using both quantitative and qualitative techniques to collect data from multiple sources in a variety of locations to ensure multiple levels of triangulation.25 The two broad areas addressed by the evaluation include the effectiveness and institutionalization of research and capacity building. The institutionalization of research and capacity building can be further subdivided into intensive work with direct and indirect beneficiaries, limited work with the CSOs, and the LUMS training program. Each of these components was examined through a unique set of data collection and analysis methodologies. The overall effectiveness was then analyzed by synthesizing the findings for each individual component. Intensive Capacity Building: A Case Study Approach The effectiveness and institutionalization of capacity building interventions of AiD and RSPN direct and indirect beneficiaries (except the CSOs) was assessed through case studies. This entailed drawing on information from pertinent documents and individual interviews with relevant USAID technical teams and key personnel within selected beneficiary organizations. A case study, according to the USAID Glossary of Evaluation Terms, is a systematic description and analysis of a single project, program, or activity. Accordingly, a case study is a qualitative method of inquiry, which is used for a comprehensive examination of a single observation or phenomenon.26 A case study uses context-specific, non￾survey-based evidence such as process-tracing, historical, documentary, and participant observation. It also uses multiple sources for triangulation to investigate the properties of phenomena.27 A case study usually focuses on questions of how and why.28 USAID selected the case studies based on the following criteria:  Beneficiaries who have completed (or nearly completed) enough interventions that they will be responsive to evaluation questions;  Both direct and indirect program beneficiaries; and  Both governmental and autonomous organizations under the Federal Government. The case studies included the following eight public sector organizations: Direct Beneficiaries 1. Provincial Reconstruction, Rehabilitation, and Settlement Authority (PaRRSA) 2. Small and Medium Enterprise Development Authority, Lahore (SMEDA) 3. Municipal Services Delivery Program, Sindh 25 Triangulation is the use of multiple methods to examine one variable/study question. It is the process of assessing a study question from several different perspectives (Schutt, Russel. 2006) 26 John Gerring, John. Case Study Research: Principles and Practices (Cambridge: Cambridge University Press, 2007). Case Study Research: Principles and Practices. Cambridge: Cambridge University Press. Page 17 27 Ibid. 28 Yin, Robert K. Yin, Case Study Research: Design and Methods (California: SAGE Publications, 2003). Case Study Research: Design and Methods. California: SAGE Publications. Page 5 14 4. Municipal Services Delivery Program, KP 5. Urban Policy Unit (UPU) 6. Jamshoro Power Company Limited Indirect Beneficiaries (Strategic Partners) 7. Federal Public Procurement Regulatory Authority, Islamabad (FPPRA) 8. Department of Finance–Internal Audit Function, KP For each case, the team used the exact same instruments to examine the same set of issues pertaining to the effectiveness and institutionalization of capacity building. Using a checklist based on the evaluation questions, the team conducted a thorough desk review of all available manuals for financial management, procurement, and HRM, as well as M&E. In addition, other relevant documents were examined. The team also conducted individual interviews with key personnel from each beneficiary organization, including the chief executive or equivalent, and managers in charge of and staff working in each of the four ASP intervention areas. In total, 67 individuals were interviewed. Finally, they conducted document spot checks at beneficiary organization offices to assess the extent to which procedures established through the ASP interventions are actually being implemented. The team wrote notes summarizing each individual and group interview and conducted an in-depth content analysis of relevant documents. Based on these, they wrote a case study report assessing the effectiveness and institutionalization of ASP’s capacity building interventions for each of the eight case study beneficiaries separately (see Annex IX). They then used content analysis to conduct cross-case comparison by coding the case studies according to themes relevant to the evaluation questions. They then prepared tally sheets identifying the themes that emerged in the case studies to facilitate systematic and rigorous data analysis aimed at identifying key evaluation findings with respect to the effectiveness and institutionalization of this project component overall. LUMS Training Assessment To examine the effectiveness and institutionalization of the training provided by LUMS, the evaluation combined both quantitative and qualitative approaches. This included document review of all training materials and curriculum, interviews with LUMS staff, and an online survey of LUMS training beneficiaries. Individual interviews included 11 ASP-LUMS staff members and faculty, including the ASP-LUMS Chief of Party, the Program Director, and full- and part-time ASP-LUMS faculty. In-depth content analysis was used to analyze both the documents and interview data (summary notes) to assess both the content of the training and perceptions of LUMS staff and faculty regarding its effectiveness and institutionalization. The evaluation also included a short online survey targeting all (1,846)29 ASP-LUMS training participants. Of these, 1,023 responded, resulting in a 55 percent response rate. These data were then analyzed using descriptive statistics (frequencies and crosstabs) to assess the relevance and effectiveness of the ASP-LUMS training. All knowledge gains shared by survey respondents are self￾reported, as ASP-LUMS did not conduct any pre- or post-test. Results from each of the three methodologies were then compared to assess the overall effectiveness and institutionalization of this component of ASP. Limited Capacity Building: CSO Training Assessment The effectiveness and institutionalization of the limited capacity building provided to the CSOs was assessed through document review, particularly of the manuals provided to the awardees, and group interviews. Five to eight representatives of each CSO were invited to participate in group interviews conducted at each location the evaluation team visited.30 Based on their availability, a total of 16 people representing 11 organizations participated in the four group interviews held in AiD and RSPN offices in Lahore, Karachi, Peshawar, and Islamabad. These discussions included representatives from the following organizations: Formation, Awareness and Community Empowerment Society, Lahore; Friends Development Organization, Gujranwala; The Layton Rahmatullah 29 The total number of ASP-LUMS trainees is 1,846, of which 546 received more than one training, taking the total number of trainings to 2,392. 30 Of the five to eight representatives invited per awardee, only a few showed up for these group interviews. In some cases a single representative for an awardee organization participated in a group interview. 15 Benevolent Trust, Karachi; Integrated Rural Awareness and Development Organization, Hyderabad; Saranga Literary and Cultural Society, Hyderabad; Special Talent Exchange Program, Mardan; Abaseen, Peshawar; Peace and Development Organization, Peshawar; Hashoo Foundation, Islamabad; Foundation for Integrated Development Action Foundation, Islamabad; and Rahma Islamic Relief, Islamabad. Content analysis of the group interview data was conducted by first identifying themes relevant to the evaluation questions that emerged in the interviews. The data were then coded in a tally sheet according to these themes to identify key findings regarding the project component’s effectiveness and institutionalization. Research Through the research component, ASP assesses the government’s existing policies and procedures in the fields of 1) financial management, 2) procurement management, 3) administrative management, and 4) M&E. Subsequently, ASP advises the government on changes to these policies and procedures with the aim of improving project planning and implementation. ASP research projects were assessed through a comparative analysis of the final report against the original concept note for the project. In addition, the utility of the research projects was discussed in interviews with relevant RSPN and USAID staff. If the research achieved its intended results as identified in the concept note and was being used by the intended parties, it was deemed a success. Overall Assessment In addition to examining each of these project components, the team interviewed key personnel from both USAID/Pakistan, particularly OFM, involved in the project and ASP IPs. These individual interviews provided both project background and examined the projects’ overall effectiveness from the perspective of these stakeholders. These interview data also were examined through content analysis and were compared with those of other project stakeholders. In addition, the team examined the following documents:  Assessment framework and corresponding scopes of work for assessing effectiveness of pre-award assessments;  Pre-award assessment, risk mitigation frameworks, capacity building plans, and other system documents (i.e., manuals and PC-1) for assessing effectiveness of capacity building support provided to direct beneficiary partners;  Relevant assessments, capacity building plans, and other system documents (i.e., manuals) for assessing effectiveness of capacity building support provided to indirect beneficiary partners;  Institutional assessments and manuals to assess capacity building support provided to CSOs; and  Completed research documents for assessing the research component. METHODOLOGICAL STRENGTHS AND LIMITATIONS The great strength of this evaluation methodology is the multiple layers of triangulation across information sources and data collection and analysis methods, which ensures the validity and reliability of findings and conclusions. Specifically, the data were gathered from multiple information sources with varying perspectives, including USAID, the IPs and key personnel within each of the eight case study organizations and representative CSO recipients. These data were comparatively analyzed. In addition, these data were triangulated against the document reviews, survey results, and spot checks to assess each project component. These steps are critical to mitigating any potential selection bias inherent in the purposive sampling approach associated with the qualitative data collection methods used, especially the interviews, which were not necessarily representative of the entire population of interest. Selection bias was also a risk with the online survey, because some respondents chose to participate while others did not. However, with a response rate as high as 55 percent of all LUMS training participants, the survey was, in 16 fact, representative of the overall population. Typically a response rate of 30 percent to an online survey is considered average, and 40 percent is considered good.31 The evaluation’s greatest limitation results from the case study approach. As it only made sense to include organizations that had fully or largely completed ASP interventions, these were not necessarily comparable to each other or to other organizations benefiting from the project. Specifically, the study included a federal regulatory and monitoring organization, two provincial-level organizations for municipal services delivery (primarily water and sanitation), an urban policy and planning unit, an organization for disaster management, a power generation company, a provincial-level oversight and auditing structure, and an enterprise development authority. Given the very different nature of the organizations, it was difficult to comparatively assess the effectiveness and institutionalization of the interventions. In most instances, there was only one of a particular type of organization. Like the case studies, the group discussions held with CSOs32 also included a diverse group of organizations that were not necessarily comparable to each other or to other organizations benefiting from the project. Represented in these discussions were organizations that were within easy travel distance to the discussion site and were representative of large and small organizations, newly formed and well-established organizations, as well as those with a variety of different missions and goals. In addition, there was a limitation in evaluating the pre-award assessment component as carried out by the ASP IPs in relation to their capacity building efforts for the selected case study organizations, since those pre-award assessments were conducted by other USAID-funded organizations prior to the start of ASP. 31 “Instructional Assessment Resources,” University of Texas, accessed October 7, 2014, http://www.utexas.edu/academic/ctl/assessment/iar/teaching/gather/method/survey-Response.php. 32 Each of the CSOs interviewed were recipients of the SGAFP of the U.S. Embassy. 17 FINDINGS AND CONCLUSIONS The findings, conclusions, and recommendations presented in this section are organized by evaluation question. For question number 1, the findings, conclusions, and recommendations are organized by ASP component. In addition, conclusions for each sub-question (where they exist) follow findings from that particular sub-question. Conclusions on each primary question appear at the end of the section dedicated to the primary question. QUESTION 1: EFFECTIVENESS To what extent have the program’s overall approach and individual components contributed to achieving the results set out in the ASP results framework? What factors have influenced the program’s achievement of results? The evaluation SOW looks to assess effectiveness from two perspectives—by individual component and overall for the project. For the first perspective, this question addresses the effectiveness of the pre-award assessment, capacity building, and research components in contributing to ASP’s IRs and Sub-IRs (i.e., to what extent are the individual components contributing to achieving overall results). From the second perspective, this assessment identifies the specific factors both contributing to and detracting from the project’s overall effectiveness as well as that of its individual components. Component 1: Pre-Award Assessments As outlined in the ASP Interim Evaluation SOW, the evaluation assesses the effectiveness and use of pre-award assessments in relation to the respective USAID/Pakistan assessment frameworks and scopes of work. As stated in the limitations, pre-award assessments for the case study organizations were not carried out by the ASP IPs, but were conducted by other USAID-funded organizations prior to ASP. Therefore, the evaluation of this component of ASP was handled in two distinct ways: 1) as a process that informed the capacity building needs for the case study organizations, and 2) as a process carried out to provide assurances of an organization’s ability to manage USAID resources. Findings Pre-award assessments were introduced by USAID/Pakistan as a means to provide reasonable assurances regarding the host country institutions’ sufficiency of financial management, accountability, procurement systems, and management capacity to manage USAID funds in accordance with USAID requirements, thereby ensuring compliance with applicable laws and regulations. The assessment is based on a USAID framework that addresses the financial control environment and risk assessments, financial management and accounting, procurement, personnel and payroll, M&E, and financial absorptive capacity. The assessment framework provides guidelines for pre-award assessments that address the requirements of the KLB Bill. The evaluation found that the pre￾assessment award scopes captured what was intended from the framework. Through a variety of chartered accounting firms, ASP conducted 14 pre-award assessments of government institutions, private companies, CSOs, and universities.33 The evaluation reviewed pre-award assessments relative to their SOWs. For review, the evaluation selected one organization from each category of beneficiary assisted by ASP. Except for a minor observation in the pre-award assessment of the Marie Stopes Society (as mentioned below), each of the pre-award assessments was conducted in accordance with its respective SOW. Findings on each organization for which a pre-award assessment was conducted are presented below. 33 ASP-RSPN Quarterly Progress Report (QPR), March 31, 2014. 18 National Engineering Services Pakistan Established in 1973, National Engineering Services Pakistan is wholly owned by the Government of Pakistan (GOP). The agency offers a broad spectrum of consultancy services ranging from conception to completion and operation of development projects. The scope of these services covers pre-feasibility and feasibility studies, surveying and mapping, investigations, design, tender and contract documentation, construction/installation supervision, contract technology, and outsourcing functions. KPMG conducted the pre-award assessment in October 2011. The assessment recommended improvements in the agency’s governance structure (internal audit manual, Board of Directors' meetings), capacity building of the Finance Department (including updating accounting and finance manuals and accounting software), strengthening of the Human Resource Department (policies, job descriptions), and strengthening of the procurement function (M&E functions and risk assessment of prospective clients). Abacus Consulting Technologies Abacus Consulting Technologies (Private) Limited was established in 1987 and is engaged in management consulting. KPMG conducted the pre-award assessment in August of 2011. Key recommendations for improvement comprised strengthening of governance structure (internal audit function), capacity building of the Finance Department (strengthening human resources, segregation of duties, compliance with policies), strengthening of the procurement processes (policies, segregation of duties), capacity building of the Human Resource Department (strengthening of policies), and setting up an independent M&E mechanism. Marie Stopes Society Marie Stopes Society is a subsidiary of Marie Stopes International, a UK-based organization with partners in 43 countries around the globe. Registered in 1990, the Marie Stopes Society is a social enterprise that provides reproductive health and family planning services to the people of Pakistan. Rafaqat Mansha Mohsin Dossani Masoom & Co Chartered Accountants conducted the pre-award assessment in January 2013. Recommendations included strengthening the governance structure (general body strengthening, approval of manuals on finance, supply chain, human resources, and strengthening of internal audit function), financial management and accounting, procurement, personnel, and compliance with applicable laws and regulations. The evaluation observed that M&E was not covered in the assessment despite the requirements of the SOW: “The assessment should take into account the capacity of the organization to manage and organize community involvement; capacity for subcontracting and sub-grants including assessing … what monitoring and oversight mechanisms are anticipated to ensure compliance and impact of sub-grants. …”34 University of Agriculture Faisalabad The University of Agriculture Faisalabad was established in 1961 by upgrading the Punjab Agricultural College and Research Institute, Lyallpur, which was founded in 1906. The university comprises six faculties, nine institutes, and three centers. Its mission is to provide instruction in Agricultural, Social, and Natural Sciences, and Humanities, and to make provisions for research and for the advancement and dissemination of knowledge. KPMG conducted the pre-award assessment in February 2014. Recommendations included strengthening of governance and legal structure (independent M&E, independent internal audit function), strengthening of the project performance function (policies and procedures), capacity building of the Treasury (finance) Department, and strengthening procurement and human resource functions. Conclusion There was a clear line of logic linking the KLB Bill, the Pakistan Assistance Strategy Report, the assessment framework, and the pre-award assessment SOWs. With one exception, the pre-award assessments comprehensively addressed the SOWs. In this respect, the first component, pre-award assessments were effective. 34 Paragraph 12.4, Alternative Recommendations section. 19 Capacity Building As outlined in the SOW, the evaluation assesses: The effectiveness of direct and indirect partners’ capacity building (through AiD and RSPN) in comparison with the requirement for risk mitigation identified through pre-award assessments35 and corresponding risk mitigation frameworks. This will include assessing the usefulness of manuals, on-the-job-training, and PC-1s in promoting compliance with USAID and government regulations. For the small grants partners, the evaluation will assess effectiveness in terms of the suitability and usefulness of manuals and handholding assistance.36 According to interviews with OFM, “suitability” means “in compliance with Pakistani laws.” “Usefulness” is determined by the beneficiaries of the manuals and mentoring. Public Sector Organizations Brief descriptions of the eight case studies follow, and the full case studies, resulting from document reviews and interview responses, are included in Annex IX. In addition to a brief introduction to each featured organization and background on its involvement with USAID and ASP, each case study covers findings for the Capacity Building Component of Evaluation Question 1. Following the case studies are common findings from across the cases. FINDINGS FROM THE EIGHT CASE STUDIES Small and Medium Enterprise Development Authority, Lahore (SMEDA) SMEDA was established in October 1998, under the Ministry of Industries and Production, and was given the status of an autonomous body in 2002. Its mission is to promote and facilitate the small and medium enterprise sector in Pakistan by creating a conducive environment, as well as providing and facilitating service delivery to small and medium enterprises for enhancing their capacities and competitiveness. USAID initiated a pre-award assessment of SMEDA in 2010 (prior to ASP). Although the USAID project for which the pre-award assessment was conducted never materialized, ASP-RSPN approached SMEDA in 2011 to participate in the ASP process. Documents reviewed and interviews with seven SMEDA representatives reveal that ASP and SMEDA collaborated to develop a two-phase capacity building plan in September 2012. Several aspects of the plan were later cancelled (including the development of Enterprise Resource Planning [ERP] and on-the-job training), leaving the emphasis to be on the development of policies and procedures manuals. The human resources manual incorporated new forms and processes as the consultants worked closely with SMEDA. SMEDA formed M&E and procurement departments as part of the ASP project and received manuals for each department. A senior SMEDA representative described the finance manual as a representation of procedures already in place. It appears from the feedback from SMEDA that the ASP intervention provided value-added to the organization in terms of creating ready access to rules and regulations in three of the four areas in which manuals were written. However, because the manuals have yet to be approved by the SMEDA Board, those areas including organizational design and hiring in procurement have yet to be implemented. Board approval is not a guarantee, so it will be difficult to assess whether ASP’s activities ultimately will be effective. The manuals were followed by a total of three days of training that were attended by the majority of the professionals in SMEDA. Over 30 SMEDA employees have attended LUMS training, and management reports that the training has been very useful in terms of knowledge gain. Federal Public Procurement Regulatory Authority (FPPRA) The FPPRA, established by the GOP, is an autonomous body under the Public Procurement Regulatory Authority Ordinance of 2002. FPPRA’s mandate is that of a regulatory and monitoring body with the responsibility of prescribing regulations and procedures for public procurements by federal government-owned public sector organizations. In 2010, ASP-RSPN began working with four public procurement regulatory authorities including 35 This will include pre-award assessments conducted before ASP, which were the basis for developing some of the capacity building plans implemented through ASP. 36 USAID/Pakistan ASP Evaluation Scope of Work, pages 14-15 20 FPPRA. A pre-award assessment of FPPRA was not carried out; however, reports from the ASP-RSPN workshop (2012) became guiding documents for the dialogue between FPPRA and ASP-RSPN, and a capacity building plan was developed in November 2012. The specific objectives of the capacity building plan for FPPRA included preparation of a national procurement strategy, revision of standard bidding documents for works, goods, and services, development of templates for procurement planning, a procurement code of ethics, system development for FPPRA for ISO 900-200837 certification, establishment of an M&E system, a grievance redress mechanism, revamping of the FPPRA website, an ERP system with a human resources module, a Learning Management System, and building FPPRA’s in-house capacity to organize trainings for procuring agencies. Document review and interviews with four FPPRA officials indicate that the development of the ERP system was viewed as the major intervention under the ASP project and a critical step towards enhancing public procurement monitoring by FPPRA as a regulatory body. However, the actual implementation of ASP at FPPRA digressed from the original capacity building plan, as the ERP system was an all-encompassing system covering four main modules on human resources, accounting and financial, learning management system, and M&E. All modules are currently at various stages of development. As per the capacity building plan, the development of the National Procurement Strategy provided a common vision and outlook on making the public procurement system efficient and transparent in the country. The development of a human resources module on government rules and regulations and development of the accounting and financial management modules on the basis of FPPRA accounting procedures indicate that the development of the ERP system has been so far effective in achieving its objectives of consolidating FPPRA operations under one decision-support system. What was not delivered to the satisfaction of FPPRA under the ASP project was the set of standard bidding documents for procurement of civil works, services, and goods. The drafts of standard bidding documents prepared by ASP-RSPN consultants were not used by FPPRA, which is now working with Asian Development Bank consultants on the development of standard bidding documents. To date, the ASP-RSPN trainings mainly focused on the human resources module, and most staff members have received training on the module. Two officers had attended training on procurement at LUMS. A FPPRA official suggested training of trainers as a way to overcome the lack of training provided so far to FPPRA staff and other public procurement officials. This was also part of the capacity building plan and stills needs to be delivered. Municipal Services Delivery Program, Sindh The Sindh Municipal Services Delivery Program is a joint project of the Government of Sindh (GOS) and the U.S. Government. This program is a Program Management Unit (PMU). USAID began its intervention with the GOS by conducting a pre-award assessment of the Planning and Development Division, which suggested that there were many areas that represented high risk to USAID financing. The first intervention for ASP-RSPN was to assist the Planning and Development Division (representing the Sindh Municipal Services Delivery Program) to complete the PC-1 for the program, required by Pakistani law for the creation of a PMU and establishing rules for the operations of a donor-funded project. A capacity building plan was completed for the joint project entailing supply of technical assistance for an undetermined time in management information system, M&E, human resources, compliance, internal audit, and procurement. The second part of the plan called for technical assistance in developing “frameworks” through a process of accommodating GOP, GOS, and U.S. Government rules and regulations, and incorporating focus groups. Based on extensive document reviews and interviews with four Municipal Services Delivery Program representatives, the evaluation team discovered that except for providing technical assistance for the development of PC-1, ASP-RSPN did not mobilize the technical assistance from the outset as envisaged by the plan. ASP-RSPN did embark on developing manuals for human resources, finance, procurement, and M&E for the Municipal Services Delivery Program. However, interviews with program officials indicate that the Municipal Services Delivery Program leadership felt that the process would be duplicative of the existing manuals. Manuals were developed 37 ISO 9000-2008 is the International Standards Organization’s protocol for quality management. It is a set of “good management practices” that an organization adopts in order to ensure consistency in its procedures and outputs. Organizations compliant with ISO 9000 can seek certification from the International Standards Organization. 21 that comply with the rules and regulations of the GOP and the GOS on all four areas of intervention. While the Municipal Services Delivery Program team initially was resistant to the notion of these manuals, they report that they now appreciate the condensed version of the rules into single manuals making access to reference documents easy, although they do not represent anything new. No training took place on the manuals, as the Municipal Services Delivery Program still has not hired its workforce. Only one staff member other than the leadership has taken a LUMS course. The Municipal Services Delivery Program officials reported that the development of M&E manuals was an important value-add from the ASP intervention, as they had never been exposed to the underpinning of an effective M&E system. They had never been exposed to the notion of outcomes, having always counted outputs for Government. Another aspect of the ASP intervention cited by Municipal Services Delivery Program leadership that might eventually pay dividends is that the existence of the manuals will make on-boarding non-civil servants considerably easier, particularly those unfamiliar with current government rules and regulations. Jamshoro Power Company Limited/GENCO-I Jamshoro Power Company Limited was incorporated in 1998 as a public company limited. It was established as part of the reform and restructuring of the Pakistan Power Sector, when the power generation companies (GENCOs) and distribution companies were converted into autonomous and commercially viable enterprises by the GOP. In 2010, a pre-award assessment of Jamshoro Power Company Limited was commissioned by USAID. The overall assessment was ranked as moderate risk, with the procurement system and some parts of accounting and financial management rated as high risk areas. The capacity building initiative under the ASP project developed in 2012 was to respond to the gaps and issues identified in the pre-award assessment. ASP-AiD was assigned to support Jamshoro Power Company Limited/GENCO-I, and it was hoped that since the company was one of 14 corporate entities created to reform Pakistan’s energy sector, the tools developed and practices established would have the potential of being adopted by other similar corporate entities. The capacity building plan had the governance structure of Jamshoro Power Company Limited, as well as HRM, financial management, internal audit, procurement, and IT manuals as key action points. While governance was later dropped from the plan, a fully operational ERP system was included under the IT component. Implementation of ASP was largely the development of the ERP system, manuals, and provision of trainings to the selected staff members at Jamshoro Power Company Limited. Findings from the document review and seven interviews with Jamshoro Power Company Limited officials show that a Budget and Accounting Manual was developed in late 2012 for all three GENCOs supported under ASP. The manual is considered to be standard and not specific to the functions required by this GENCO. For instance, the manual talked about a public works accounting system, which was not required by Jamshoro Power Company Limited and other GENCOs, since they were all following international financial report standards post incorporation as companies. It was also observed that although there is an internal audit manual in place, internal audit staff are not aware of it. Jamshoro Power Company Limited followed Public Procurement Regulatory Authority rules and still use them. The new procurement manual covers all the relevant Public Procurement Regulatory Authority rules. Regarding the effectiveness of ASP intervention, data entry into the ERP system started a year ago, but has not been completed as yet so the system has not been able to generate any reports, thus undermining the use of ERP as a complete decision-making support system. With the contractor’s engagement ended, frustration was noticed during the interviews with relevant staff at Jamshoro Power Company Limited. The board has not yet approved any of the manuals. Expected changes have not materialized due to pending implementation of the ERP system that integrates all the four areas of intervention. Jamshoro Power Company Limited appreciated the Budget and Accounting Manual. Findings from the study suggest that a little effort was made by Jamshoro Power Company Limited to institutionalize changes brought about through ASP interventions. Around 15-20 employees in financial management were trained on the manual; five administrative staff, two accounts officers, and two people from the field were trained on the human resources module under the ERP system; and 35 people have been trained on the ERP system (mostly users of the system). The CEO and three assistants in the department attended the LUMS training in 2013. 22 Internal Audit, Government of KP (GOKP) Pursuant to the pre-award assessment of the GOKP by Deloitte in March 2010, in which internal control was identified as an area of high risk and low confidence level, GOKP took the decision to establish an Internal Audit Function across all Provincial Government departments and offices. The Internal Audit Charter was approved by the Provincial Cabinet on June 16, 2012, with a view to determine whether governance processes, internal controls, and risk management systems, as designed and represented by the management, were adequate and functioning to ensure application of risk management policies, conformance with all applicable policies, procedures, and manuals and that the overall control environment is up to the mark. ASP-RSPN carried out its technical assessment and governance structure reports in March 2013, which laid out the foundations for the establishment of Internal Audit Function across the identified departments/organizations within GOKP. For the case study, the evaluation relied on document review and two interviews with officials of the GOKP Finance Department. The Finance Department was the main counterpart for ASP-RSPN interventions. Together they drew up a three-phase capacity building plan that comprised 1) ASP-RSPN providing technical assistance to conduct internal audit as per the existing charter and implementation plan, 2) improving/developing policies and procedures to conduct internal audit, and 3) replicating the audit function throughout the province. It was envisioned that that Internal Audit Function would be developed for and introduced into four departments: Education, Health, Communication and Works, and Police (USAID does not work with Departments of Police). ASP-RSPN support resulted in getting 18 permanent sanctioned posts created for the GOKP Internal Audit Function. However, none of these sanctioned posts have been filled. In addition, ASP-RSPN developed audit tool kits for the Education Department, and subsequently the UK Department for International Development (DFID) has taken over the responsibility for the development support of the department, thus leaving in question the implementation/use of the audit tool kits. At this point in time, training is envisioned, along with mentoring for the 18 sanctioned positions, but until the positions are filled, no capacity building efforts can proceed. Municipal Service Program, KP The Municipal Services Program is an initiative of the GOKP aimed at improving municipal services delivery in Peshawar, Dera Ismail Khan, and Malakand Divisions. While the bulk of the program funding, (i.e., US$84.5 million) is to be provided by USAID, the GOKP has agreed to allocate around US$12.7 million as counterpart funding. The objectives of the program are closely aligned with the Provincial Comprehensive Development Strategy as well as other national and provincial policies with respect to drinking water and sanitation. The implementing agency for the program is the Local Government and Rural Development Department of the GOKP. The Municipal Services Program as a PMU ensures adherence to rules, regulations, and policies of the GOP and USAID. A dedicated pre￾award assessment was not conducted for the program; however, gaps in organizational capacity were identified through the Risk Mitigation Framework in December 2011 and the pre-award assessment of Local Government, KP in 2010. These gaps comprised financial management/internal controls, procurement, administration and HRM, and M&E. For the case study, the evaluation relied on document review and five interviews with officials of the Municipal Services Program in KP. ASP-AiD took up the task of capacity building for the program in 2011 and came up with a capacity building plan. However, owing to significant delays in resource development at the PMU, there were a number of objectives of the initial plan that were not met, so a revised capacity building plan was devised in 2013. The Municipal Services Program Director and technical area heads commented that they were not involved in the development of the plan and were not aware that a capacity building plan existed that detailed who within the organization should be trained, on what, and when. Manuals were developed by ASP-AiD, which complied with Government requirements and USAID rules and regulations. However, they offered nothing new or innovative to the technical teams. Technical support on M&E from ASP-AiD was not provided. Training was provided to the PMU through both in-house training conducted by ASP-AiD technical experts and at LUMS. Some of the key challenges to the program were lack of adequate positions filled within the program prior to capacity building, lack of conditions placed on the money received by USAID, and therefore, lack of incentives to embrace capacity building efforts, and excessive delays in obtaining approvals for the newly developed manuals by the GOKP. 23 PaRRSA PaRRSA, which was established in 2009 under the Provincial Disaster Management Authority by the GOKP, was created to look after the reconstruction, rehabilitation, and settlement of those affected by crisis. PaRRSA is to provide the requisite speed, ease, facilitation, coordination, supervision, and linkages to all parties involved and to help the provincial government in its endeavor to rehabilitate affected areas. The main contributors to the resource needs for PaRRSA activities have been the World Bank, the Asian Development Bank, the United Nations Development Program (UNDP), and USAID. A Strategic Oversight Council responsible for strategic and policy-level actions administers PaRRSA’s oversight and operations. In December 2009 Deloitte Touche was contracted by USAID to conduct the Host Country Contracting & Fixed Reimbursement Arrangement Capabilities Assessment of PaRRSA, which served as the pre-award assessment for PaRRSA. Responsibility for capacity building was transferred from the Khyber Pakhtunkhwa Reconstruction Program to ASP-RSPN on January 31, 2012. For the case study, the evaluation relied on document review and two interviews with PaRRSA officials. ASP-RSPN developed a capacity building framework in 2012 that contained five objectives. These revolved around financial management, procurement, human resources and administration, M&E, operations, and governing policies. Reasonable progress was made against these objectives as manuals were made, LUMS trainings were given, and job descriptions were drawn up. Capacity Building activities worth US$250,000 were conducted by June 30, 2014, across all the identified areas. However, according to PaRRSA officials the selection process for training participants requires further improvement to ensure better utilization of funds. It was observed that staff were sent for training outside their area of technical expertise and many had not yet received the initial training in their technical area. Manuals were largely a compilation of relevant/applicable government rules and regulations. However, their regular use was not observed, and no version control or updating provisions exist, revealing a major deficiency. UPU/KP UPU was established through a PC-1 for the PMU of the KP Municipal Services Program, with a view to working in an integrated manner toward improving urban governance in the province. Its focus is on achieving integration of urban planning and target capacity development of line departments. Being a new set-up within the GOKP Planning & Development Department, there was no pre-award assessment. For the case study, the evaluation relied on document review and five interviews with seven officials of the UPU. The role of ASP-AiD was understood to be helping UPU get started, providing some training if needed, and assisting with the establishment of policy and procedure manuals. ASP-AiD hired UPU/Punjab to write the PC-1 for UPU/KP and to ensure that its policies and procedures were in line with UPU/Punjab. ASP-AiD initially devised a capacity building plan for UPU/KP in June 2011; however, due to considerable delays in the approval of PC-1 and then recruitment of the required staff, ASP-AiD submitted a revised capacity building plan by late 2012. Key officials at UPU seemed unaware of these, which reduced the effectiveness of the plans. ASP-AiD provided initial positions before UPU recruitment took place. Trainings took place in the form of in￾house consultants, mentoring through emails, phone calls, person-to-person, and LUMS courses. Manuals were prepared, and these contributed to enhanced compliance with government and USAID rules and regulations. Some of the key challenges faced by the program were delayed approvals of manuals/positions and lack of proper support for M&E. Common Findings The pre-award assessment largely served as the basis for capacity building plans of the eight public sector case study beneficiaries. As seen in the table below, of the eight case study beneficiaries, two pre-award assessments specifically identified the needs of their respective beneficiary organizations: SMEDA and Jamshoro Power Company Limited. Four of the case study organizations were PMUs, and one was a department within GOKP. Therefore, the pre-award assessments upon which needs were assessed were conducted for parent government departments. There was no pre-award assessment done for the FPPRA; however, the need for intervention at 24 FPPRA was identified in other beneficiaries’ assessments. The specific intervention design in FPPRA came about as the result of meetings between USAID, ASP-RSPN, and FPPRA. TABLE 4: SUMMARY OF ASP’S CAPACITY BUILDING INTERVENTIONS Case Study Basis for Capacity Building Plan Manuals Other Capacity Building FPPRA “Stakeholders’ Consultation Workshop on Public Procurement Regulatory Regime in Pakistan” organized by ASP-RSPN in April 2012  Manuals not yet developed  Assistance in developing the National Procurement Strategy  ERP system SMEDA Pre-award assessment conducted  Employees Service Regulations (Human Resources)  Financial Regulations and Procedures Manual  M&E Manual  Procurement Manual Not conducted Municipal Services Delivery Program, Sindh Pre-Award Assessment of Planning and Development Division, Sindh  Financial Management Framework  Human Resource Manual  M&E Manual  Procurement Manual  Assistance in developing the PC-1 for the Municipal Services Delivery Program. Jamshoro Power Company Limited Pre-award assessment conducted  Budget and Accounting Manual  Internal Audit Manual  Procurement Manual  ERP system PaRRSA Report on host country contracting and fixed￾amount reimbursement arrangement capabilities assessment  Finance and Accounting Manual  Human Resource and Administration Manual  M&E Manual  Procurement Manual  Internal Audit Manual  Training Needs Assessment  IT Needs Assessment Municipal Services Program, KP  Financial Management Manual  Human Resource Manual  M&E Manual  Procurement Manual UPU/KP  Financial Management Manual  Human Resource Manual  M&E Manual  Procurement Manual  Assistance in developing PC-1 for the project 25 Case Study Basis for Capacity Building Plan Manuals Other Capacity Building Internal Audit, KP Technical Assessment Report for Establishment of Internal Audit Function  Internal Audit Framework  Job Descriptions for Internal Audit Cell Staff  Internal Audit Governance Structure for Internal Audit Function for GOKP  Internal Audit Guidelines, Communication and Works Department  Internal Audit Report, Board of Intermediate and Secondary Education, Peshawar  Internal Audit Guidelines, Elementary and Secondary School Department  Internal Audit Report, Lady Reading Hospital, Peshawar ASP-RSPN and ASP-AiD prepared capacity building plans for all eight case study beneficiaries and shared the plans with the beneficiaries upon completion. For the seven beneficiaries with pre-award assessments, or whose parent organization received a pre-award assessment, a review of the capacity building plans indicated that they focused on gaps identified in the pre-award assessments in finance, governance, HRM, M&E, and/or procurement. The IP submitted the capacity building plans to USAID for approval. Upon receiving USAID’s approval of the capacity building plan, ASP began developing manuals in the focal areas. In seven of the eight case studies, IPs provided beneficiaries with manuals as part of the capacity building program (the FPPRA capacity building plan did not call for the development of manuals). The process was that ASP-RSPN and ASP-AiD identified focal points in each technical discipline, interviewed the focal points, and collected forms, procedures, and other relevant documents. Beneficiaries reported that ASP consultants would then compile the manuals offsite and present a draft to the technical heads of the department for concurrence. Depending upon how technical heads viewed the accuracy of the manual, they would either work with ASP to finalize it or would accept it with comments and suggestions. The evaluation's review of the manuals, using a checklist the team developed for this purpose, found that they are comprehensive and compliant with GOP regulations and policies. Five of the seven beneficiary organizations reported that the manuals provide easy access and reference to the relevant policies and regulations. Many of the manuals provide new templates under financial management and procurement and help organizations streamline their capture of financial and technical information. Human resource manuals introduced recruitment, retention, compensation, and severance practices that are useful, particularly for those beneficiaries for which most employees are recruited from the private sector. Employees recruited from the private sector are not typically as aware of government rules and regulations as are civil servants, and the manuals serve to onboard them more quickly. Nonetheless, common knowledge of effective compliance tells us that for manuals to be effective tools in promoting compliance there must be effective oversight and enforcement, typically done by a combination of internal and external audit. The problems in Pakistan’s oversight environment for USAID partners are characterized by ASP’s efforts in KP to improve the Internal Audit Function within the GOKP as a strategic initiative. Staff of four of the seven beneficiary organizations noted that the manuals ASP developed are a snapshot of existing government policies and procedures at the time they were written, and staff of five beneficiaries said they prefer to refer to rules, regulations, and circulars that come from the GOP, which they consider to be more appropriate 26 and up-to-date than printed manuals. The evaluation team’s review of the relevant Pakistani regulatory bodies’ websites finds that the laws, regulations, and circulars are searchable and up-to-date on the Internet.38 Each of the seven case study beneficiaries that received manuals also received training39 from ASP on the contents of the manuals. Beneficiary organizations’ personnel were notified well in advance of training opportunities, which ensured participation by the relevant staff. Staff was cross-trained to familiarize them with manuals outside of their own technical function. Although no training needs analyses40 were carried out by the IPs in support of the trainings, all public sector case study organizations thought that the training was relevant, and six of the eight deemed the training practical and appropriate. Staff from all public sector case study organizations felt that the ASP trainers were well qualified in their respective technical areas. However, some beneficiaries found deficiencies in the ASP-RSPN and ASP-AiD training. Beneficiaries criticized the ASP training on manuals as limited to “manual orientation” (two beneficiaries) and inadequate (five beneficiaries). Each of the beneficiaries interviewed stated that they required more comprehensive training. All beneficiaries also noted that they continue to have training needs for staff that did not participate in the training or recently joined the organization. Beneficiaries said they do not have the human resources available to deliver their own training to new recruits. The evaluation's research found conflicting data on the subject of “on-the-job training.” IPs stated that during the process of developing the manuals, they provided technical assistance to their counterparts as part of on-the-job training. Four of the capacity building plans call for ASP to provide personnel to PMUs that have been unable to hire their own staff, and interviews with IPs and the beneficiaries slated to have assigned personnel suggest that there seems to be disagreement on ASP-deployed human resources’ role to provide continued on-the-job assistance. Review of documents provided by ASP IPs indicates that six of the eight case studies beneficiaries received some mentoring after the manuals were delivered. However, all beneficiaries agreed that none of the case study beneficiaries received technical assistance after the manuals were delivered and the trainings were completed. One said that while it offered additional assistance, and included it in the capacity building plan, the beneficiary organization did not feel that they could spare their employees for as long as was prescribed in the capacity building plan. The PC-1s played an important role in the process, as well. The evaluation’s review of the five PMUs in the sample that were established as a result of PC-1s found that the PMUs were established in accordance with the instructions and guidelines issued by the Pakistan Planning Commission. Three of the PC-1s contained clauses that enhanced beneficiaries’ compliance: two of the PC-1s contained recommendations for the formation of internal audit functions that were followed up by the respective PMUs; another specified that an M&E section was to be set up to meet USAID and GOS monitoring and reporting requirements. ASP conducted validation reports41 for two of the eight case study beneficiaries. These validation reports updated the risk profiles of organizational components, including governance and internal control structure, internal audit, financial management, budgeting and accounting system, procurement, preparation of PC-1s, human resources, and M&E. The evaluation reviewed the two validation reports conducted for the case study beneficiaries and found inconsistencies in the Jamshoro Power Company Limited Validation Report. The assessors provided a high score based on the presence of an internal audit manual. Although there is an internal audit manual in place, the Jamshoro Power Company Limited’s Internal Audit Department was unaware of it. Jamshoro Power Company Limited’s procurement staff said they were not trained on the procurement manual, in contradiction to the validation report’s finding and findings from the AiD document review that indicated training was conducted. The Municipal Services Program-KP Validation Report did not contain inconsistencies. 38 The team arrived at this determination through a review of the websites for FPPRA, the Planning Commission, the Ministry of Finance, the Establishment Division (responsible for government human resources), and the Auditor General of Pakistan. 39 We observed three different types of trainings in ASP: “Skills Development Training” refers to LUMS training and the ASP-AiD training provided to CSOs. “Manual Awareness Training” refers to the training offered by ASP-AiD and ASP-RSPN to public sector beneficiary organizations upon delivery of the manuals. “On-the-job-training” refers to one-on-one training, coaching, and mentoring post-manual delivery provided by the IPs in select cases to the beneficiaries. 40 According to USAID, such analysis was part of the Risk Mitigation Framework. 41 Municipal Services Program-KP Validation Report, March 2014; and Jamshoro Power Company Limited Validation Report, February 2014. 27 Conclusions The process of identifying needs through the pre-award assessment and developing a capacity building plan on the basis of its findings was an effective approach to building a set of activities that addressed compliance with USAID and GOP regulations. Capacity building plans were effective in addressing the requirements of risk mitigation identified through pre-award assessments. Therefore, the pre-award assessments were useful in developing the capacity building plans, thereby contributing to ASP sub-IR 1.2: Management systems adopted. The overall usefulness of the manuals and PC-1s in promoting compliance is also positive. Manuals are primarily useful in increasing beneficiaries’ compliance with regulations, and simplify introducing new employees to the organization, particularly those projects that are drawing their employees from outside the civil service (employees unfamiliar with government regulations). However, they are only useful if they are kept up-to-date. As new regulations and government circulars are released, they may not be as useful as the comprehensive electronic versions hosted on government websites, unless the manuals are continually updated to keep pace with regulatory changes. PC-1s proved useful in promoting compliance with USAID and GOP regulations, and in some cases were also useful as a means of introducing important new compliance dimensions to the projects. The implementation of capacity building plans and manuals, in particular, contribute to ASP’s Sub-IR 1.2: Management systems adopted. Manuals’ awareness trainings have been generally useful to public sector beneficiaries. To the extent they increased knowledge, they contributed to ASP results IR 2.1: Knowledge and skills of human resources enhanced and IR 2: Human resources enabled for enhanced performance. Some on-the-job-trainings and mentoring after the trainings was provided to six of the eight case study beneficiaries. Because two case study beneficiaries (SMEDA that did not opt for on-job assistance and Internal Audit KP that did not have staff)42 did not receive this support, the usefulness of on-the-job trainings cannot be fully assessed (Sub-IR 2.2: Application of knowledge and skills supported). The effectiveness of the manuals, trainings, and PC-1s in promoting compliance will be largely tied to the oversight environment in which they operate. In a strong oversight environment, manuals combined with employees’ awareness can be a very effective tool and are a critical component of internal and external audit. In Pakistan, while there is a strong network of international audit firms capable of conducting high-quality external audits, there are problems in the internal audit environment as characterized by the struggles of the Internal Audit Department (e.g., hiring staff for the internal audit cells)in KP. As long as the internal audit function is weak in KP, the effectiveness of the manuals in promoting compliance will be limited. Therefore, the full achievement of IR 1: Improved Management Systems Implemented is subject to strong internal audit function. Civil Society Organizations Findings All 11 of the CSOs reported that the ASP process began with an assessment, typically conducted outside of ASP. The Institutional Assessment Questionnaire served as most CSOs’ initial assessments, while a few others were outsourced to a CPA firm, which used their own instrument. The two instruments mainly differ in the manner in which they are administered: the Institutional Assessment Questionnaire is a checklist largely completed by CSOs with support from National Rural Support Programme (NRSP), while an external audit firm conducted the Assessment before the award. The process for ASP support was uniform for all 11 interviewed CSOs. Each received annual schedules of trainings to be held at LUMS and at select locations throughout Pakistan by ASP-AiD. Each organization made its own selection of who should attend the trainings and which trainings they should attend. None of the trainings were mandatory for any of the CSOs. Capacity building plans were not designed and developed specifically for ASP interventions by/for any of the CSOs interviewed. Only three of the 11 CSOs had developed their own action plans and timeframes to rectify the risks identified in their pre-award assessments, and one received assistance from ASP-AiD in completing its plan. 42 Document review (Jan 2015) indicates that ASP plans to provide on-the-job training to the Internal Audit department in KP once staff for the internal audit cells have been hired. On the request of SEMEDA, ASP provided a two-day orientation training (instead of on-the-job training) to relevant SMEDA employees. 28 ASP-AiD CSO training took place at select centralized locations on the topics of workplace ethics and code of conduct; procurement management; HRM; monitoring, evaluation, and reporting; CSO compliance; and IT skills and computer use. With the exception of the IT and M&E training, all CSOs reported that the ASP-AiD trainings were beneficial. One of the CSOs is a training organization itself, and they reported that the ASP-AiD training was some of the best they have come across, so far, due to AiD’s expertise in training and engagement level with the participants. Three of the 11 CSOs specifically noted that the IT training was not long enough, not at the appropriate level for the participants, and that the software introduced had flaws in implementation and application. At least half of the CSOs reported that the IT training was not appropriate for their needs, as the software introduced was not applicable to their needs and the instructor was not able to adapt to the various levels of the participants. One CSO reported that the M&E training was not comprehensive enough and was not offered enough times in the schedule of courses. Others, in the group interviews, expressed that the M&E training was not as beneficial as other trainings they received. LUMS course offerings were in financial management, procurement management, M&E, HRM/ administration, and leadership and cultural change. There was a consensus among CSOs that LUMS training was more specific to government rules and best practices and not specific for CSOs, while ASP-AiD training was more geared toward CSOs. However, it was also a consensus that, as stated by one CSO representative, “The professionalism of both ASP-AiD and LUMS trainers was greatly appreciated.” Two major challenges to how training was delivered surfaced during the CSO group interviews (concurred by all CSO representatives): 1) there was no sequential order to the training; and 2) small and large CSOs were both represented in the same trainings.43 The lack of sequence left participants feeling as though there was no opportunity to build on what they had learned before, and they struggled to find logic in how the material was presented. When both large (more established) and small (newly formed) CSOs were grouped together for training, it was difficult to meet the training needs of both. Either the larger CSOs were held back because of the lack of experience or knowledge of the smaller CSO, or the smaller CSO was “lost” and unable to keep up with the training. It was also noted in three of the four group discussions that training materials should have been prepared in multiple languages—not just English. CSOs felt that it was difficult for those who did not have a good command of English to follow the training, even when conducted in their local language, because they were unable to follow along in the manuals. This became evident in the interviews, as two of the four groups required facilitation in Urdu as the heads of the organizations were not sufficiently acquainted with English to understand the questions. In two of the four groups, CSO representatives mentioned they would have preferred more frequent and geographically dispersed trainings at less costly sites. At the group discussions held in Karachi and Lahore, CSO representatives expressed concern that ASP-AiD training was not conducted at sites readily accessible to CSOs located in remote areas. They felt that training sites that were a bit more far-flung would be more appropriate and more easily accessible. ASP-AiD developed five manuals for use in the capacity building efforts for CSOs: Internal Audit Manual, Financial Management Manual, Human Resources Manual, IT Policy Manual, and Procurement Manual. The intent of the manuals was to form a comprehensive set of guidelines and procedures for each of the technical areas and to make these available to the CSOs with whom they were directed to work. The manuals were then to be adapted, adopted, or otherwise modified to fit the particular needs of each organization. All CSOs interviewed received manuals from ASP-AiD, and all CSOs understood the intent of the manuals and the need for them to be appropriately adapted for their use. During two group discussions, concern was raised that no support was provided by ASP-AiD in the modification/adaption of the manuals. Of particular concern was the “overly comprehensive” nature of the Financial Management and IT Policy Manual and that “there is a lot of stuff copied from the corporate sector” that is not applicable for CSOs. It is important to note here that even though 43 ASP-LUMS does not nominate participants for training. The process of nominating participants rests with AiD and RSPN. Final screening and approval of nominations is done by USAID. 29 the CSOs expected mentoring in the modification of the manuals, the CSO capacity development plans did not include such assistance.44 Ultimately, CSOs in all group interviews were highly appreciative of ASP and felt that the training had enabled them either to introduce new manuals or to upgrade their existing operating manuals in a manner beneficial to the organization, thereby achieving Sub-IR 1.2: Management systems adopted. They felt that the manuals provided by ASP-AiD proved useful as a strong reference point for streamlining management systems, and CSO representatives in all four group interviews said that organizational change had taken place in their respective CSOs and that management systems had been improved. CSO representatives in each of the four group interviews expressed that they are better able to apply for resources from other donors and USAID, because they feel they have the systems in place to prove their accountability and transparency. Specific changes were noted, and concurred by those participating in the group discussions. The following are the most notable changes:  Modified policy and procedure manuals were approved by the CSO’s governing body and have become instrumental in carrying out their day-to-day duties.  More transparent operations and clearer guidelines for procurement have been institutionalized.  Timesheets are newly required in CSOs where they were not previously.  Gender policies are now in existence with support by policies on workplace harassment and a code of workplace ethics. Conclusions ASP’s intervention with CSOs was effective in establishing documentation and building awareness to move the beneficiaries toward a higher degree of compliance. Because the CSOs were so closely involved in the process and, therefore, exhibited a high degree of ownership, the manuals should become an integral part of CSO operations and will result in improvements to the effectiveness of their operations. The improved management systems and organizational change that CSOs identified are evidence that ASP has likely achieved IR 1: Improved Management Systems Implemented. CSOs were given training first, and then received generic manuals that they were to customize themselves only after they received skills training. As a result, respondents feel that they fully understand the contents of their manuals (which is not surprising given that they developed them) and that they are implementing them completely. They feel confident in their ability to update them as needed. Public organizations, in contrast, received customized manuals followed by training on their contents. While document review indicates acceptance of the ASP developed products by public sector organizations, the differing approaches likely resulted in a higher degree of ownership of the final product and changed some of the ways the CSOs went about doing business.45 CSOs received limited assistance in planning for their capacity building mainly through two one-day consultative workshops on planning of capacity building and its implementation. CSOs were given a calendar of ASP-LUMS training events, and it was up to them to nominate relevant personnel for trainings. ASP-LUMS conducted a two￾day opening workshop in December 2011 to identify the topics for each course and obtain feedback on the design of the courses from government and CSO beneficiaries. The topics of the courses were also based upon the recommendations of around 150 pre-award assessments conducted for the potential beneficiary organizations. However, there was no formal training needs analysis. There was limited follow-up assistance for CSOs in the form of combined sessions/trainings/workshops after organizations customized the ASP-provided manuals to ensure that there were no obvious gaps or to help with the development of forms and other tools (in particular, M&E tools). Conducting the training in the local languages would have benefited smaller NGOs working in remote areas. 44 Contrary to what CSO representatives reported during the evaluation interviews, AiD correspondence (shared in Dec 2014) indicates that some mentoring did take place supporting adaption of new standards/policies. However the extent to which such support was provided cannot be ascertained. 45 Although documents provided by ASP-RSPN indicate that some technical assistance was provided to some public sector beneficiaries, the findings from case studies indicate that much of the capacity development process was done by the IPs on behalf of the beneficiaries. 30 LUMS Training The evaluation assesses the: suitability and usefulness of the LUMS’s training program in improving USAID/Pakistan partners’ human resources and management systems. The evaluation also examines whether there are synergies between the training provided by LUMS and other capacity building efforts (i.e., does exposure to both types of capacity building enhance results?). Findings The LUMS training seeks to provide trainees with the tools and techniques that will make Pakistani organizations transparent and accountable. While the beneficiaries and participants were largely uniform in their praise of the program, the LUMS staff and faculty gave the program mixed reviews. Six out of the eight case study beneficiaries and participants in all four CSO group discussions said the training offered by LUMS was relevant. CSOs in three group interviews were pleased that their technical staff were able to take courses outside of their technical area, taking into account the interrelated nature of courses, such as finance and procurement. The online and telephone survey results show that LUMS trainees considered their skills inadequate prior to attending a course. The highest percentage of trainees reporting inadequacy of skills in the four core course areas were in M&E (36.7 percent), followed by Procurement Management (34.5 percent), Financial Management (22.5 percent) and HRM (17.3 percent). The survey results indicate that trainees believe their skills have improved considerably as a result of attending the LUMS training. The highest percentage of trainees that reported improvement in their skills were from the Financial Management course (97.5 percent), followed by M&E (96.9 percent), Procurement Management (96.5 percent), and HRM (96.1 percent). These results indicate that ASP achieved Sub-IR 2.1: Knowledge and skills of human resources enhanced and IR 2: Human resources enabled for enhanced performance. The LUMS faculty and staff had mixed reviews of the training. Four out of 12 considered the LUMS training generic, theoretical, and/or academic and conceptual, while six of the 12 thought that in addition to providing a theoretical perspective, the LUMS training courses are also helpful in upgrading job-related skills. Five of the 12 LUMS faculty and staff interviewed said the courses on procurement, forensic audit, and procurement audit focus primarily on transparency and accountability in procurement and financial processes. Four of the 12 stated that LUMS did not conduct a formal training needs assessment of potential trainees prior to designing each course. According to six ASP-LUMS faculty, the courses were designed as a result of consultations with the ASP IPs and USAID. Further, feedback on course content from the other ASP IPs is received through meetings, from IPs' staff attending ASP-LUMS training, and from the stakeholders’ validation conferences, and is incorporated into course curricula to ensure that the courses remain relevant to the needs of trainees (according to six LUMS interviewees). A review of LUMS training materials revealed that a training folder on HRM had 22 documents, out of which 15 were from international sources, including one UN document and one USAID/Pakistan document. An M&E course folder included 9 USAID/Pakistan TIPS on topics related to M&E and a USAID/Pakistan paper titled Conducting Mini Survey. In contrast, one Procurement Management course folder contained only locally developed reading materials.46 A high proportion of ASP-LUMS alumni reported in the survey that they were able to make changes to their work as a result of attending a course. The highest percentage of participants who reported positively on being able to make a change were those who attended the Human Resource Management and Administration Course (94.9 percent), followed by Procurement Management (94.2 percent), Monitoring and Evaluation (92.2 percent) and Financial Management (89.7 percent). 46 Sources: on-site review of ASP-LUMS training materials and one of the four USAID respondents. 31 TABLE 5: PARTICIPANTS’ PERCEPTION OF LUMS COURSES RESULTS Participants Felt Skills were Improved by the Course Felt Skills Improved Able to Make Changes Monitoring & Evaluation 97% 92% Financial Management 98% 90% Procurement Management 97% 94% Human Resources Management & Administration 95% 94% Project Management 99% 96% Leadership: Gender Dimension 95% 94% Forensic Audit 100% 90% Fiscal Decentralization 93% 83% Procurement Audit 100% 92% Despite the ability of the participants to make changes to their work, two of the three IPs stated that training participants did not employ their newly acquired knowledge in the ASP capacity building process. As noted earlier, beneficiaries reported that ASP consultants would compile manuals offsite, leaving little opportunity for those trained at LUMS to contribute. In addition, trainees were not necessarily trained in a timely manner to support ASP capacity building. In a review of the schedules for training and capacity building, there did not appear to be any intended alignment between when training classes took place at LUMS and the schedule for other IPs' capacity building activities with beneficiaries. Conclusions The ASP-LUMS training programs are suitable and useful in improving USAID/Pakistan partners’ human resources, thereby achieving ASP IR 2.1: Knowledge and skills of human resources enhanced. Improvements in beneficiaries’ employees’ skills through LUMS training will eventually contribute to improved organizational effectiveness, addressing IR 2: Human Resources Enabled for Enhanced Performance. The synergies between the LUMS training and other capacity building efforts were present, but limited by partial alignment between LUMS training and IPs’ capacity building schedules. Only those CSO employees that participated in the trainings before they implemented new manuals were able use their newly acquired knowledge to implement changes. It is entirely possible that changes implemented by training participants improved the management systems, but there is no clear linkage that they improved the management systems selected for other ASP interventions. Research Components The evaluation assesses the effectiveness of completed research in relation to its objective and concept note. Findings The research component’s intended role is to identify and address systemic issues that affect the performance of GOP projects by calling on best practices to provide recommendations to the GOP to streamline processes. ASP-RSPN conducted four research activities on: 47 1) Revolving Fund Account/Assignment Account, 2) PMU Approach, 47 All four research projects, including the three covered under this evaluation, were undertaken by ASP-RSPN. 32 3) The GOP Project Cycle Planning Architecture, and 4) PFM Assessment Framework (Institutional Performance Assessment Framework). Each of the research projects were G2G-level interventions spanning over a year, with the exception of research on the Revolving Fund Account, which was completed in 4-5 months. The topics for the research component— identified by USAID/Pakistan during the development of ASP’s results framework—largely related to the difficulties and issues faced in the implementation of USAID/Pakistan-funded development projects. The GOP was not involved in the identification process; however, close interaction and consultations with the government took place throughout the research process and at its completion to obtain the government’s feedback and buy-in. The evaluation SOW identified three of the research projects to be sampled and examined for effectiveness relating to Sub-IR 1.1 (Institutional capacity gaps understood by stakeholders) and Sub-IR 3.1 (Policy recommendations advocated). Presented below is the analysis of findings sourced from the review of concept notes, research reports, relevant correspondence, and interviews with USAID/Pakistan and ASP-RSPN staff familiar with each research project. Revolving Fund Account/Assignment Account: This research provides a step-by-step guide for the establishment and operationalization of revolving fund accounts. ASP-RSPN and USAID/Pakistan indicated that the research has resulted in significantly reduced time to open a revolving fund account—from an average of 9 months to less than 10 days in some cases. ASP-RSPN closely worked with the office of Controller General of Accounts Pakistan to seek feedback from all the relevant stakeholders in the GOP. The Ministry of Finance released an official notification on the revised accounting procedure for Revolving Fund Accounts in August 2013, which helped streamline opening and operating the accounts. The research also resulted in the development of a training toolkit addressing a long-outstanding demand from key stakeholders, including ministries, provincial departments, offices of the Accountant and Auditor Generals, development partners etc., resulting in over 120 GOP projects’ staff being trained. Research on PMU Approach: Prior to this research, GOP did not have PMU-specific establishment or operational guidelines. The objective of this research was to make specific recommendations for streamlining procedures and instructions governing PMU establishment and operations; making improvements to conform to international project management best practices; and proposing policies to ensure the sustainability of the proposed reforms. An advocacy workshop with Planning Division and Provincial Planning and Development Departments was also held by ASP-RSPN. Research on GOP Project Cycle Planning Architecture: The research provided the Planning Commission with a diagnostic study of the procedures involved in planning development projects, including developing PC-1s through Planning Commission Form Number V (PC-V). The report identifies issues and makes key recommendations on the development project planning system in Pakistan. It also recommends a restructuring and empowering of the Pakistan Planning and Management Institute to develop and implement needs-based training programs to implement donor-funded development projects in the country. Following the completion of this research, the Institute and ASP-RSPN developed and redesigned the courses. ASP shared each of the research reports with the relevant GOP counterparts and other donors, according to ASP￾RSPN and USAID interviewees. ASP shared the revolving fund accounts research report with the donor coordination group. Since the research on GOP project cycle and PMU approach reports were considered complementary to one other, both were shared with the GOP Planning Commission, but ASP has not received formal feedback so far, according to ASP-RSPN interviewees. Conclusion The research component has met USAID's objective of advising the GOP on improving project planning and implementation procedures and therefore has contributed successfully to IR 1.1 (Institutional capacity gaps understood by stakeholders). Close interaction and consultations with the government took place throughout the research process and at its completion to obtain the government’s feedback and buy-in and, therefore, the research component contributed to IR 3.1 (Policy recommendations advocated). 33 QUESTION 2: SUSTAINABILITY To what extent are ASP results and the services provided by ASP implementing partners likely to be sustainable? The evaluation seeks to answer this question in two parts. QUESTION 2.1 The first part of the sustainability question focuses on the sustainability of results and the extent to which beneficiary partners have institutionalized capacity building measures introduced by ASP in the areas of financial, procurement, administration and human resource management and monitoring and evaluation. Institutionalization includes a beneficiary organization’s recognition of the usefulness of an ASP intervention, a decision to implement it, its reflection in system documents, allocation of resources required for implementation, and actual organizational practice. Findings Sustainability of any capacity building program is partially tied to an organization’s sense of ownership of the project outcomes. The measure of an organization’s ownership is typically tied to the degree of participation of the individuals or organizations in charting their own path for change and their involvement in the change process itself. In ASP, public sector and CSO beneficiaries had very different experiences in this regard. In the case of ASP public sector beneficiaries, targets for pre-award assessments and capacity building were selected by USAID; pre￾award assessments were conducted by external organizations and did not have self-assessment components; capacity building plans were designed externally by the IPs; and consultants wrote manuals externally. Conversely, CSOs in all four group interviews said they had to conduct a self-assessment through the institutional assessment questionnaire; they were free to choose their own level of participation in ASP; and they customized their own manuals utilizing their senior management’s time. Another indicator of sustainability are the internal and external forces that compel the organization to either continue or abandon reforms. Public sector ASP beneficiaries and CSOs demonstrated that they have both internal48 and external49 incentives to maintain their manuals. ASP beneficiaries have the ongoing external incentive of USAID funding and funding from other donors to remain in compliance with GOP laws and USAID policies. In all four of the group discussions, CSOs expressed confidence in knowing that they now have the documentation to respond favorably to pre-award assessments in ways that will demonstrate their ability to transparently and responsibly manage external funds. For internal incentives, manuals developed by ASP were integrated into the systems of most of the beneficiary organizations (six of eight public sector beneficiaries and participants in all four CSO group interviews), and interviewees cited these manuals as ingredients of their future success. For example, SMEDA indicated that they would use the manual to establish their procurement department, the department’s roles, and its responsibilities (upon board approval). All public sector beneficiaries and CSOs interviewed also see the manuals as important to the effectiveness of their operations. For example, one participant heading a small CSO in Karachi observed that they had rudimentary systems, policies, and procedures, but participation in ASP provided them with proper manuals in each of the four areas that are now driving the operations of their organization. Keeping the manuals up-to-date and ensuring that employees are aware of changes require particular organizational capabilities. In reviewing the manuals and speaking to employees of the seven public sector beneficiaries interviewed whose assistance included manuals, beneficiaries do not demonstrate a system of version control for manuals, and technical personnel feel they will need further assistance to update the manuals in the future and to reproduce training on the manuals (per five public sector beneficiaries). Conversely, the CSOs customized their own manuals without mentoring as a result of ASP’s training, and all of them reported having the capacity to update them in the future. 48 Internal incentives are typically defined as improvements in effectiveness that would be rolled back in the absence of the manuals. 49 External incentives are defined as opportunities to receive benefits from outside organizations as long as the improvements remain in place. 34 Manuals were comb-bound (or, in one case, permanent hard-cover bound), which will require the beneficiaries to print entirely new manuals when changes are made. Best practice suggests that manuals are printed and placed in binders in which pages can easily be replaced with records of version control. Also, one beneficiary stated that when additional copies of manuals are needed, they simply ask the ASP IP to do it for them. The sustainability of the capacity building programs will be dependent not only on the employees that have already been trained, but future employees’ access to the same training as they join the organization. Each of the public sector beneficiaries said that the consultants left no training materials behind or trained trainers to facilitate the replication of the orientation sessions for organizational newcomers. The presence of oversight will play an important role in ensuring the ongoing sustainability of the ASP reforms. ASP is addressing oversight through its efforts with what USAID/Pakistan refers to as “strategic partners” (in this case, the Internal Audit Department of KP and the FPPRA). ASP initiated the process of implementing reforms and initiatives in the Public Procurement Regulatory Authorities50 and Internal Audit in Peshawar, oversight bodies that will be critical to ensure manuals are being followed. The organizational structure developed by ASP for the Internal Audit Department with 18 permanent positions is approved by GOKP. ASP delivered an internal audit manual to the Higher and Secondary Education Department and rooted internal audit functions in PMUs through their respective PC-1s (UPU/KP, Municipal Services Program/KP). However, as described under the case study for the KP Department of Finance/Internal Audit Function, the evaluation’s spot checks and interviews suggest there still exists a lack of clarity and consensus as to the positioning of the overall Internal Audit Department for the province, its powers, and reporting lines, and the department has yet to be institutionalized in the GOKP. Conclusions “Sustainability” is a measure of a future state and something that cannot be proved, but only predicted. In evaluating sustainability, we are left to measure those factors that will determine the likelihood that an intervention will be sustained beyond the end of the assistance. Completed manuals and the associated awareness training have been institutionalized in the different types of organizations (public and CSO sectors), inasmuch as they are summaries of the relevant laws and regulations that govern the body. Beneficiaries certainly recognized the usefulness of ASP's interventions, as mentioned throughout this report, and most also reported implementing the manuals and training they received as evidenced in the systems documents, though to varying degrees. It is difficult to assess the degree to which organizations will commit resources to maintaining and updating the reforms; however, the opportunities for external funding provide strong incentives to continue to implement ASP’s reforms, despite a lack of ownership on the part of the public sector beneficiaries. However, as laws and regulations are modified/revised from time to time, and there is a regular flow of circulars and directives from government regulators, these manuals will have a short shelf life before they become obsolete. It is likely that both public sector organizations and CSOs will update the manuals as laws change and knowledge increases, as they have the internal and external incentives to keep the manuals current. However, some public sector organizations will need to develop the skills and systems to maintain a regular schedule of manual updates, and this might compromise the quality of future updates. QUESTION 2.2 Explanation: The second part of the question will focus on the assessment of any plans prepared by the implementing partners for continuing to provide ASP-type services after the end of ASP. Findings ASP-RSPN felt that without any external source of funding, it was unlikely that RSPN would continue to provide services to increase the compliance and/or effectiveness of public sector institutions. ASP-RSPN opined that government departments would not pay for the types of services that have been provided to ASP beneficiaries and, therefore, would need an external funding source. 50 ASP has worked with the Public Procurement Regulatory Authority in each of the provinces and the FPPRA. This evaluation studied only the case of capacity building within the FPPRA. 35 ASP-AiD is preparing a business plan to continue to provide capacity building services along the lines of what they offered under ASP. In the plan, the NGO that they will form will need donor support to supplement the anticipated fee-for-service, particularly in the beginning. ASP-AiD’s capacity building with CSOs was viewed very positively by all 13 CSOs interviewed. As ASP-LUMS has enjoyed a degree of success in their training programs. The Chief of Party of ASP-LUMS reported that LUMS has established the Center for Governance and Public Management to sustain the ASP-LUMS program after ASP ends. LUMS is seeking to establish international accreditation for the Center in the next two to three years and has established a foreign advisory committee consisting of professors to guide the process. They will also likely seek donor funding initially to support the Center. Conclusion ASP-AiD and ASP-LUMS are actively taking measures to establish sustainable post-ASP programs that are based on the ASP design. RSPN is unlikely to continue with the services they are offering under ASP. QUESTION 3: GENDER To what extent, if any, has ASP included gender as a consideration in its interventions? The evaluation SOW looks to address the question of gender by answering the following sub-questions:  Do the Cooperative Agreements reflect the findings of a gender analysis and/or considerations for gender equality?  To what extent and by what means (HRM policies, terms of employment, supportive workplace, etc.) do the ASP IPs strive for gender equality?  How are gender concerns, particularly gender equality, incorporated in the PC-1s and the HRM and M&E manuals prepared by ASP for beneficiary organizations?  To what extent are concerns with gender equality reflected in the curricula of LUMS courses on communication strategy and management, leadership and cultural exchange, M&E, project management and training of trainers?  What steps has ASP taken to encourage women’s participation in LUMS courses, and to what effect in terms of enrollment (in various courses and for all tiers)?  How do the current positions and roles of LUMS program alumni compare across male and female alumni? Findings While it is stated in the ASP Activity Approval Document that “each new activity started by USAID must outline the significant gender issues that need to be considered during activity implementation,”51 the evaluation did not find in any of the original Cooperative Agreements52 for the three ASP IPs such an outline, and none contain a reference to, nor reflect, any gender analysis conducted prior to the award. Of the three original cooperative agreements, ASP-RSPN’s agreement had specific references to how and where gender should be addressed in ASP interventions (gender mainstreaming, anti-sexual harassment policy, and code of conduct for the workplace); ASP￾AiD’s was silent on the issue; and the LUMS chartered accountant indicated that LUMS should include gender as part of “other trainings offered on a need basis.”53 51 USAID Automated Directives System (ADS) 201.3.12.6. 52ASP-RSPN Original Cooperative Agreement (A-11-01201); ASP-AiD Cooperative Agreement Original (A-11-01203); ASP-LUMS Original Cooperative Agreement (A-11-01202). 53 Ibid. 36 Modification #454 to ASP-AiD’s cooperative agreement included how attention to gender issues should be addressed in its interventions with beneficiary organizations, through such topics as gender mainstreaming, anti￾sexual harassment policies, and workplace codes of conduct. Each of the three ASP partners addresses gender in their internal operations manuals. RSPN’s commitment to gender equality is reflected in its Human Resources Manual.55 RSPN has a position in its organizational structure56 of Manager of Gender and Development and has gender sensitivity as one of its stated core values. RSPN has a comprehensive Gender Policy that is referenced in the Harassment Policy and throughout the operations and procedures of its HRM. RSPN scores all employee applicants/candidates on “gender knowledge and sensitization.” To support its ASP implementation efforts, RSPN has developed two manuals to guide their work: Gender Mainstreaming in ASP-RSPN Program57 and Gender Analysis.58 To support its own need for strengthened capacity in the area of gender awareness and sensitivity, RSPN arranged for all staff within their Central Program Office to receive general training on workplace-related gender issues and specialized, advanced training on gender.59 FIGURE 4: TRAINING PARTICIPANTS BY GENDER AiD’s commitment to gender equality is reflected in its Policy Document,60 which includes a policy on gender equity that provides the principles on which the policy is based. These principles provide operational guidelines for how to achieve gender equity in the workplace. AiD’s Annual Work Plan (2013-2014)61 calls for gender training as part of its ASP Awardee Capacity Building component. ASP-LUMS has given due attention to gender concerns in its human resources manual. ASP-LUMS has an equal opportunity policy and is committed to providing equal opportunities regardless of gender, gender reassignment, and other considerations. There is a detailed section on sexual harassment in the human resources manual.62 The ASP-LUMS Chief of Party and Director of Programs, in interviews, mentioned that they strive to create gender￾balanced teams in all their activities. 54 ASP-AiD Cooperative Agreement, Modification #4 (March 2013). 55 Human Resources Manual, NRSP. Reviewed and upgraded by Anjum Asim Shahid Rahman Chartered Accountants. 56 Ibid. 2. 57 ASP-RSPN, Gender Mainstreaming in ASP-RSPN Program: An In-Depth Analysis (June 18, 2014). 58 ASP-RSPN, Gender Analysis of Assessment and Strengthening Program (May 11, 2012). 59 ASP-RSPN Summary of Activities as of April 30, 2014, items 56.1 & 57.1. 60AiD (Pvt) Ltd. Policy Document (June 1, 2007). 61 ASP-AiD Annual Work Plan 2013-14 (September 19, 2013), 4. 62 ASP-LUMS HR Manual (July 2014; revised). 87% 13% Male Female 37 The PC-1s reviewed by the evaluation team include strategies for inclusivity, gender targets for participation, and gender-specific communication and outreach activities.63 In addition, they identify the need to identify a gender specialist64 and to improve the Gender Parity Index, to reduce gender gaps, and to increase gender-sensitive indicators.65 The document Human Resource Policies & Procedures: Manual for Civil Society Organizations66 is a comprehensive ASP￾developed human resources manual provided to every beneficiary CSO for their reference, adoption, and/or adaptation. The manual includes policies and procedures to address gender equality and gender sensitivity through recruitment, staff training, and appraisal; workplace safety; codes of ethics; and management. The M&E and HRM manuals developed by ASP for public sector organizations give limited attention to gender equality, with the only mention being to disaggregate data by gender (M&E) and to include statements of equal opportunity for both genders in job notifications (HRM). Gender equality concerns were not reflected in the curricula of LUMS courses on communication strategy and management, leadership and cultural exchange, M&E, project management and training of trainers.67 Interviews with ASP-LUMS personnel indicate that gender issues were rarely, if at all, incorporated in ASP courses. The following statement was, in general, repeated by most lecturers regardless of topic area: “The participation of females in training or any specific concerns relating to gender have not been highlighted as a major issue in procurement training. Had it been mentioned as an issue, something would have been done to address these concerns. Procurement is a gender-neutral topic, and the application of procedures is done irrespective of gender.” 68 As all ASP-LUMS courses are gender-neutral, they developed a special course on“Leadership: Gender Dimension” to address the importance of gender and gender-based barriers to career mobility. In its email to RSPN and AiD inviting nominations for its courses, ASP-LUMS specifically includes the suggestion to nominate female candidates for trainings. The ratio of female trainees in various courses increased modestly from 12 percent of all trainees in the first year to 13 percent in the second year and 17 percent in the third year. TABLE 6: PERCENTAGE PARTICPATION IN LUMS COURSES BY GENDER Current Professional Field69 Numbers % Female Male Female Finance/Financial Management 24 314 7.1 Procurement Management 14 170 7.6 Human Resources Management & Administration 37 262 12.4 M&E 33 185 15.1 Academics 4 22 15.4 Project Management 8 44 15.4 Other 19 115 14.2 63 RSPN, PC-1 Need Based Merit Scholarships for Pakistani University Students with USAID’s Grant Assistance (Higher Education Commission, Pakistan: December 2012). 64 RSPN, PC-1 DGAP’s Capacity, Accountability and Verification Improvement Initiative (January 2013). 65 RSPN, Government of Sindh Education and Literacy Department. PC-1 Sindh Basic Education Program Sindh Province (October 2011). 66 AiD, Human Resource Management-Policies & Procedures: Manual for Civil Society Organizations. 67 On-site document review of course curriculum. 68 Notes on Interview with LUMS Faculty by Ahmad Jameel (August 15, 2014). 69ASP Interim Evaluation, Survey of LUMS Training Participants (August 2014). Survey respondents were allowed to indicate more than one current position, if applicable, so the total number of females and males are slightly higher than the total number of participants. 38 On average, 12.5 percent of the current professional fields of LUMS program alumni are occupied by women, with Academics and Project Management having the highest percentages of females at 15.4 percent and Finance/Financial Management with the lowest percentage at 7.1 percent. TABLE 7: PERCENTAGE FEMALE PARTICIPATION IN LUMS COURSES BY JOB RANK The percentage of female alumni who participated in the ASP-LUMS program decreases as the job scale/rank in their current position increases, with a greater percentage (17.5 percent) of female participation at the entry level and a lesser percentage (10.5 percent) of female participation at the senior level. Conclusions While there was no evidence of a gender analysis in the original three ASP cooperative agreements, gender equality was given consideration, to varying degrees, in all three partners’ agreements or agreement modifications. Each of the ASP partners strives for gender equality within their respective organizations. They do so in varying degrees through human resources manuals and policies, employment terms, and creating a supportive workplace environment. The attention to gender concerns, especially gender equality, has been handled differently in the ASP-developed manuals for CSOs and public sector organizations. For public sector organizations, the attention is far less significant and is limited to the need for gender-disaggregated data in carrying out the M&E function and to statements of equal employment opportunities when preparing job notifications. Gender concerns are incorporated into PC-1s through suggested targets and strategies to promote participation of females and to reduce gender gaps. For CSOs there is significant attention given to gender concerns through policies and procedures detailed in the human resources manual. Gender equality concerns are not reflected in the content of the ASP-LUMS course offerings. However, the efforts of ASP-LUMS to encourage females to attend training programs seem to have had a positive effect in slightly increasing the enrollment of females in its training programs. Nonetheless, ASP-LUMS female alumni are represented at a significantly lower percentage than their male counterparts, particularly as the job scale/rank grows higher. However, without comparisons with industry statistics as a whole, it is impossible to conclude to what extent, if any, LUMS alumni gender balance statistics are any different from non-beneficiary statistics. The low representation of females in LUMS trainings is due to the fact that the beneficiary organizations, not LUMS, nominated trainees for ASP LUMS trainings. 70 Ibid. Job Scale/Rank70 Numbers % Female Male Female Entry Level 7 33 17.5 Mid-Level 79 523 13.1 Senior Level 40 341 10.5 39 OVERALL CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS ON ASP'S OVERALL EFFECTIVENESS The overall conclusion is that ASP has been effective in improving compliance of beneficiary organizations through improved documentation and beneficiary employees’ awareness. ASP beneficiaries now have operational manuals in place and/or upgraded in the areas of finance, human resources, procurement, and/or monitoring and evaluation (M&E), thus meeting Intermediate Result (IR) 3: Improved Public Financial Management (PFM) Policies and Procedures. Enhanced compliance documentation may or may not make a significant contribution to improved organizational effectiveness addressing IR 2: Human Resources Enabled for Enhanced Performance. However, improvements in beneficiaries’ employees’ skills through LUMS training will eventually contribute to improved organizational effectiveness. ASP has contributed to CSOs institutional capacity building, by providing them with the skills and knowledge to establish their own systems thus achieving IR 1: Improved Management Systems Implemented. Conversely, IR 1 has not been fully achieved for public sector beneficiaries because the relevant governing bodies in some of the beneficiary organizations have not approved improved policies and procedures. CSOs, in particular small grant awardees, had a different experience from public sector beneficiaries. Many of the CSOs that participated in ASP took part in their institutional assessment and planning for their capacity building through ASP. While the public organizations received customized manuals followed by training on their contents, CSOs were given training first, and then received generic manuals that they were to customize themselves only after they received skills training. While document review indicates acceptance of the ASP-developed products by public sector organizations, the differing approaches likely resulted in a higher degree of ownership of the final product and changed some of the ways the CSOs went about doing business.71 Respondents feel that they fully understand the contents of their manuals and that they are implementing them completely. They feel confident in their ability to update them as needed. However, there were a series of missed opportunities to provide further assistance to the CSOs. CSOs were assisted in a limited way in planning for their capacity building through two one-day consultative workshops on planning of capacity building and its implementation. CSOs were given a calendar of ASP-LUMS training events, and it was up to them to nominate relevant personnel for trainings. While ASP-LUMS conducted a two-day opening workshop in December 2011 to identify the topics for each course and obtain feedback on the design of the courses, there was no formal training needs analysis. There was limited follow-up assistance for CSOs in the form of combined sessions/trainings/workshops after organizations customized the ASP-provided manuals to ensure that there were no obvious gaps or to help with the development of forms and other tools (in particular, M&E tools). Conducting the training in the local languages would have benefited smaller NGOs working in remote areas. Over the four years that ASP has been operating, it has transformed into a program of multiple projects with very different intended outcomes and methodologies. ASP took a relatively supply-side approach with public sector organizations, as IPs, apart from providing some technical assistance in developing strategic plans, typically planned and conducted much of the capacity building process on behalf of most of the beneficiaries included in the evaluation. With CSOs, ASP took a more demand-driven approach by providing beneficiaries with access to 71 Although documents provided by ASP-RSPN indicate that some technical assistance was provided to some public sector beneficiaries, the findings from case studies indicate that much of the capacity development process was done by the IPs on behalf of the beneficiaries. 40 training classes and allowing them to plan and to lead their own institutional capacity building processes. Overall, ASP-LUMS training provided skills to many employees of the ASP beneficiaries. While there was no evidence of a gender analysis in the original three ASP cooperative agreements, gender equality was given consideration, to varying degrees, in all three partners’ agreements or modifications. Each of the ASP partners strives for gender equality within their respective organizations. They do so in varying degrees through HR manuals and policies, employment terms, and creating a supportive workplace environment. The attention to gender concerns, especially gender equality, has been handled differently in the ASP-developed manuals for CSOs and those for public sector organizations. For public sector organizations the attention is far less significant and is limited to the need for gender-disaggregated data in carrying out the M&E function and to statements of equal employment opportunities when preparing job notifications. For CSOs, there is significant attention given to gender concerns through policies and procedures detailed in the HR manual. As all ASP-LUMS courses are gender-neutral, they developed a special course on“Leadership: Gender Dimension” to address the importance of gender and gender-based barriers to career mobility. However, the efforts of ASP￾LUMS to encourage females to attend training programs have had meager, though positive, effects in increasing the enrolment of females in its training programs. Nonetheless, ASP-LUMS female alumni are represented at a significantly lower percentage than their male counterparts, particularly as the job scale/rank grows higher. However, without comparing these statistics with industry-wide statistics, no conclusions can be made about the effect (if any) of the LUMS training on women's positions in the workplace. RECOMMENDATIONS For improved accountability and transparency, ASP should greatly increase its efforts and investment to establish internal audit departments in provincial governments in a similar vein to its efforts in KP. This will strengthen the crucial oversight/enforcement piece necessary to ensure that improved management systems and PFM policies and procedures are implemented sustainably as documented. ASP should continue and accelerate efforts to work in what ASP-RSPN refers to as “strategic institutions”, particularly those that have an oversight role such as regulatory authorities at the provincial and federal levels. ASP should also identify other opportunities to work with other regulatory oversight bodies (i.e., Ministry of Finance, Establishment Division, the Planning Commission, and the Auditor General of Pakistan) to enhance beneficiaries’ compliance with documentation. ASP took different strategies to working with CSOs and public sector organizations, and both had their shares of success and significant gaps in their service delivery. Lessons should be applied to each to make the other more robust. For example, CSOs should receive some support for organization-specific manuals, particularly M&E. The following are the specific recommendations resulting from this evaluation:  ASP should provide CSOs with more assistance in developing their own capacity building plans.  ASP should increase CSOs’ access to experts in each field to review the manuals they have created and to get questions answered.  ASP should prepare CSO training in local languages, not just English. Though ASP-LUMS courses are gender-neutral, LUMS should review its courses, wherever applicable, to identify opportunities to include issues of gender equality, including the understanding of gender gaps within a particular technical area, a profile of the status of men and women in terms of their leadership roles, the gender norms and strategies to mitigate the gaps and discrepancies. Strategies to address gender equality should extend beyond participation and inclusivity to include laws, policies, regulations, and institutional practices. 41 BIBLIOGRAPHY AiD, Human Resource Management-Policies & Procedures: Manual for Civil Society Organizations. AiD (Pvt) Ltd. Policy Document (June 1, 2007). ASP-AiD, Activity Tracker, April 30, 2014. ASP-AiD, Annual Progress Report, October 2012–September 2013, Assessment and Strengthening Program. ASP-AiD Annual Work Plan 2013-14 (September 19, 2013), ASP-AiD Cooperative Agreement Original (A-11-01203). ASP-AiD, Quarterly Progress Reports, 2013–2014. ASP-AiD, Revised Performance Management Plan, 2010–2015, January 2013. ASP-LUMS Original Cooperative Agreement (A-11-01202). ASP-RSPN, Activity Tracker, April 30, 2014. ASP-RSPN, Annual Program Performance Report 2012–13, Assessment and Strengthening Program (ASP) for Civil Society Organizations (CSOs) and Government of Pakistan (GOP). ASP-RSPN, Cooperative Agreement and its amendments. ASP-RSPN, Gender Analysis of Assessment and Strengthening Program (May 11, 2012). ASP-RSPN, Gender Analysis - Assessment & Strengthening Program (ASP), November 2012. ASP-RSPN, Gender Mainstreaming in ASP-RSPN Program: An In-Depth Analysis (June 18, 2014). ASP-RSPN, Gender Mainstreaming in ASP-RSPN Program, June 2014. ASP-RSPN Original Cooperative Agreement (A-11-01201); ASP-RSPN, Performance Management Plan For the period October 2012 to September 2015, January 2013. ASP-RSPN, Quarterly Progress Reports, 2013–2014. ASP-LUMS HR Manual (July 2014; revised). Bogdon, R.C. and Biklen, S.K. Qualitative research in education: An introduction to theory and methods. Allyn and Bacon 2006. Denzin, N. Sociological Methods: A Sourcebook. Aldine Transaction (5th Edition), 2006. Human Resources Manual, NRSP. Reviewed and upgraded by Anjum Asim Shahid Rahman Chartered Accountants. “Instructional Assessment Resources,” University of Texas, accessed October 7, 2014, http://www.utexas.edu/academic/ctl/assessment/iar/teaching/gather/method/survey-Response.php. Gerring, John. Case Study Research: Principles and Practices (Cambridge: Cambridge University Press, 2007). Case Study Research: Principles and Practices. Cambridge: Cambridge University Press. Municipal Services Program-KP Validation Report, March 2014; and Jamshoro Power Company Limited Validation Report, February 2014. Office of Inspector General, Audit of USAID/Pakistan’s Assessment and Strengthening Program, Audit Report No. G-391- 12-009-p, Islamabad, September 2012. Public Procurement Regulatory Authority, National Procurement Strategy 2013-2016, Government of Pakistan. Available at http://www.ppra.org.pk/doc/nps.pdf 42 RSPN, PC-1 Need Based Merit Scholarships for Pakistani University Students with USAID’s Grant Assistance (Higher Education Commission, Pakistan: December 2012). RSPN, PC-1 DGAP’s Capacity, Accountability and Verification Improvement Initiative (January 2013). RSPN, Government of Sindh Education and Literacy Department. PC-1 Sindh Basic Education Program Sindh Province (October 2011). The Enhanced Partnership with Pakistan Act of 2009, (popularly known as the Kerry-Lugar-Berman Act). Available at http://www.gpo.gov/fdsys/pkg/BILLS-111s1707enr/pdf/BILLS-111s1707enr.pdf. The Paris Declaration (2005), at URL: http://www.oecd.org/development/effectiveness/34428351.pdf. The Accra Agreement for Action (2008), Available at URL: http://siteresources.worldbank.org/ACCRAEXT/Resources/4700790-1217425866038/AAA-4-SEPTEMBER-FINAL￾16h00.pdf. The Pakistan Assistance Strategy Paper (2009). Available at URL: http://www.state.gov/documents/organization/134114.pdf. USAID Automated Directives System (ADS) 201.3.12.6. USAID, ADS Chapter 220 - Use and Strengthening of Reliable Partner Government, Systems for Implementation of Direct Assistance, July 2014. USAID Pakistan, Mission Strategic Framework, Fiscal Year 2013–2017, Performance Management Plan, January 2014. Yin, Robert K. Yin, Case Study Research: Design and Methods (California: SAGE Publications, 2003). Case Study Research: Design and Methods. California: SAGE Publications. 43 ANNEXES ANNEX 1: EVALUATION STATEMENT OF WORK Political Parties Development Program (ASP) – Evaluation Statement of Work 44 ASSESSMENT AND STRENGTHENING PROGRAM (ASP) INTERIM EVALUATION STATEMENT OF WORK MAY 27, 2014 45 ASSESSMENT AND STRENGTHENING PROGRAM (ASP) INTERIM EVALUATION STATEMENT OF WORK 46 I. BACKGROUND INFORMATION A. IDENTIFYING INFORMATION ABOUT THE PROGRAM This Statement of Work (SOW) outlines the requirements for a consulting firm (the consultant) to conduct an interim performance evaluation of the United States Agency for International Development (USAID)’s Assessment and Strengthening Program (ASP). ASP works with Pakistan’s government agencies and non-governmental organizations (including non-profit Civil Society Organizations (CSOs) and for-profit private sector organizations) to provide institutional development support so that their management capacity meets standards required for transparent, accountable and effective utilization of USAID resources. The program operates under three cooperative agreements. Table 1 summarizes key facts about ASP. TABLE 1: PROJECT SUMMARY B. DEVELOPMENT CONTEXT 1. Problem or Opportunity Addressed Following the Paris Declaration73 and the Accra Agenda,74 USAID made it a priority to work with host country organizations. Under the Kerry-Lugar-Berman75 (KLB) Bill, the USAID/Pakistan Mission is 72 One each, respectively, for the three implementing partners listed later in the table. 73 In the Paris Declaration (2005) donors committed “to use country systems and procedures to the maximum extent possible”. Available on page 5 at URL: http://www.oecd.org/development/effectiveness/34428351.pdf. 74 In the Accra Agreement for Action (2008), donors agreed to “to use country systems as the first option for aid programmes in support of activities managed by the public sector”. Available on page 3 at URL: http://siteresources.worldbank.org/ACCRAEXT/Resources/4700790- 1217425866038/AAA-4-SEPTEMBER-FINAL-16h00.pdf. Title / Field Program Information Agreement Numbers72 391-A-00-11-01201 391-A-00-11-01202 391-A-00-11-01203 Agreement Officer’s Representative (AOR) Feroz Shah (Mr.), Lead Financial Analyst, Office of Financial Management Start Date October 12, 2010 Completion Date October 11, 2015 Location Nationwide Name of Implementing Partners (IPs)  Rural Support Programmes Network (RSPN)  Lahore University of Management Sciences (LUMS)  Associates in Development (AiD) USAID Objective Addressed Cross-cutting Objective3: Pakistan’s Development Capacity Improved Budget  RSPN: US$20 million  LUMS: US$15.5 million  AiD: US$8.9 million Total US$45 million 47 required to consider using host country governmental and non-governmental organizations as implementing partners. Subsequently, the new Pakistan Assistance Strategy Report76 (2010-2014) stated that “U.S. assistance will be directed increasingly through a broad range of Government of Pakistan institutions, as well as local non-governmental organizations (including the private for-profit sector) with the capacity to implement programs effectively and accountably. This approach will help increase host country ownership, and U.S. commitment to building a long- term partnership with the Pakistani government and people”.77 The strategy report also noted “Pakistan’s limited capacity to absorb and effectively use external resources and public sector corruption” as “challenges to the effective implementation of U.S. and other donor assistance programs”.78 ASP implementing partner Rural Support Programmes Network (RSPN)’s Cooperative Agreement identified several additional challenges to carrying out development projects that arise from government procedures. These include the following:79  Absence of separation of administrative and financial powers reportedly undermines accountability;  Lack of clear stakeholder roles and responsibilities in the planning and implementation of development projects results in lack of effective and transparent decision making; and  Lack of knowledge and skills among staff regarding rules and regulations, which is reflected in lack of effective planning and implementation. The RSPN Cooperative Agreement also identifies the following problems associated with non￾governmental organizations (including CSOs and for-profit organizations):  Absorptive capacity, as most CSOs deal with small amounts of development funds and have limited geographical and sectoral outreach;  Absence of robust financial management and accountability mechanisms, and lack of segregation of important functions;  Absence of standardized and relevant system documents in the areas of governance, internal control and financial management; and  Lack of familiarity with USAID regulations and systems for accountability, reporting and transparency. 2. Target Areas and Groups ASP is designed to assist governmental and non-governmental organizations (including CSOs and private sector organizations) selected to implement USAID-financed projects in all sectors and throughout Pakistan. ASP assistance to date has included support for 48 public sector, 49 non-profit and 11 for￾profit organizations. The non-profit organizations include 43 CSOs, which are small community based 75 The Enhanced Partnership with Pakistan Act of 2009, (popularly known as the Kerry-Lugar-Berman Act) states that “as appropriate, to utilize Pakistani firms and community and local nongovernmental organizations in Pakistan, including through host country contracts”. Available on page 20 at URL:http://www.gpo.gov/fdsys/pkg/BILLS-111s1707enr/pdf/BILLS-111s1707enr.pdf. 76 According to section 302 (a) of the Enhanced Partnership with Pakistan Act of (2009), “Not later than 45 days after the date of enactment of this Act, the Secretary of State shall submit to the appropriate congressional committees a report describing United States policy and strategy with respect to assistance to Pakistan under this Act.” 77 Pakistan Assistance Strategy Paper (2009). Available on page 14 at URL:http://www.state.gov/documents/organization/134114.pdf. 78 Ibid, page 10. 79Attachment B (Program Description) in the ASP-RSPN Cooperative Agreement No. 391-A-00-11-01201, Modification No.2. Challenges facing government and non-governmental organizations are discussed on pages 5-7 of this document. 48 organizations. Twenty-seven of the beneficiary organizations are located in Khyber Pakhtunkhwa (KP), 23 each in Sindh and Islamabad, 16 in Punjab, 14 in Balochistan, two in Azad Jammu and Kashmir (AJK), one in Gilgit-Baltistan (GB) and two have nation-wide coverage. C. INTENDED RESULTS In view of the problems and challenges outlined above, the USAID/Pakistan mission felt the need to systematically assess its potential partners’ capacity, identify areas of risk and provide capacity development assistance. USAID/Pakistan subsequently launched ASP with the objective of making “Pakistani organizations more transparent, accountable and effective”80. ASP’s results framework contains three Intermediate Results (IRs):  IR 1: Improved Management Systems Implemented  IR 2: Human Resources Enabled for Enhanced Performance  IR 3: Improved Public Financial Management (PFM) Policies and Procedures. ASP also contributes to USAID/Pakistan’s Mission Strategic Framework (MSF) cross-cutting Objective 3: “Pakistan’s Development Capacity Improved”. A summary of the ASP results framework is presented in Table 2.81 TABLE 2: ASP RESULTS FRAMEWORK Objective Indicators Pakistani organizations more transparent, accountable and effective 1. Percentage change in risk level of ASP partner organizations as result of validation exercise 2. Percentage change in satisfaction level of key organizational constituencies 3. Number of ASP introduced reforms and initiatives contributing towards transparent, accountable & effective Pakistani organizations Intermediate Results Indicators IR 1:Improved Management Systems Implemented 1. Percentage organizations using improved management systems 2. Percentage organizations which continue to meet predefined management standards Sub-IR 1.1: Institutional capacity gaps understood by stakeholders 1. Number of organizational assessments and researches conducted 2. Number of Pakistani organizations whose capacity needs have been established Sub-IR 1.2: Management systems adopted 1. Number of capacity building plans developed 2. Number of policies and procedures developed 3. Number of IT based management systems developed 4. Number of new tools and techniques introduced and adopted Number of project design activities supported IR 2: Human Resources Enabled for Enhanced Performance 1. Percentage trainees with improved knowledge and skills 2. Percentage change in relevant competencies of trainees 80 The ASP Performance Management Plan was updated in January 2013. 81 All the indicators of the results framework (Table 2) are numerical indicators on which the three implementing partners are expected to report. However, reporting on these indicators has not yet commenced. 49 Sub-IR 2.1: Knowledge and skills of human resources enhanced 1. Number of training programs developed 2. Number of individuals who benefitted from training 3. Number of trainings conducted 4. Number of programs developed through international collaboration Sub-IR 2.2: Application of knowledge and skills supported 1. Number of individuals who receive on-job assistance, refreshers and coaching 2. Number of consultants provided for short-term technical assistance Sub-IR 2.3: Increased opportunities for young professionals in development sector 1. Number of students who join government and CSOs as interns 2. Number of student internship fairs conducted IR 3: Improved PFM Policies and Procedures 1. Number of policy reforms adopted 2. Number of rules and regulations improved Sub-IR 3.1: Policy recommendations advocated 1. Number of research studies conducted 2. Number of reforms/recommendations developed 3. Number of policy advocacy activities carried out 4. Number of policy level institutions receiving policy reform recommendations or attending advocacy events D. APPROACH AND IMPLEMENTATION ASP is implemented through three IPs:  Rural Support Programmes Network (RSPN) generally focuses on, but is not limited to, providing support to USAID’s host government partners;  Associates in Development (AiD) focuses primarily on CSOs and for-profit organizations, but also assists government partners in KP; and  Lahore University of Management Sciences (LUMS) specializes in providing on-campus training to participants from USAID partners including ASP beneficiary organizations. ASP is comprised of five components as outlined and elaborated below:82  Pre-award Assessment: The purpose of pre-award/other assessments is to determine whether a proposed recipient’s internal controls and administrative and financial management systems are adequate for effective utilization of USAID funds;  Capacity Building: The key objective of this component is to address the capacity building issues of host country organizations identified in the pre-award assessments;  Validation: This component aims at conducting validation of beneficiary organizations that have received capacity building assistance to assess whether the organizations are in compliance with the standards and procedures developed under the capacity building plans;  Research: This component focuses on conducting research on institutional aspects of host country organizations and development of best practices in the CSO sector; and  Awardee capacity building: The purpose of this component is to enhance the capacity of ASP implementing partners and their staff. 82 Program descriptions of cooperative agreements for ASP-AiD, ASP-RSPN and ASP-LUMS. 50 1. Pre-award/Other Assessments The purpose of pre-award/other assessments is to determine whether a proposed recipient’s internal controls and administrative and financial management systems are adequate for effective utilization of USAID funds. The pre-award assessments are carried out by one of the Chartered Accountant (CA) firms that work with RSPN and AiD. The relevant USAID/Pakistan office selects potential implementing partners and the Office of Financial Management (OFM) commissions either RSPN or AiD to conduct the pre-award assessment. Before ASP, USAID/Pakistan conducted pre-award assessments of host country organizations through Blanket Purchase Agreements (BPA), which USAID/Pakistan had in place with several Regional Inspector General (RIG) approved CA firms. The CA firms conduct the pre-award assessments in accordance with the assessment framework83 and the corresponding scopes of work84 developed by ASP. The pre-award assessment report provides an assessment of an organization’s financial management, procurement, administration and human resource management systems and controls, and absorptive capacity, concluding with a risk assessment for each system (i.e., low, medium and high). This report is provided to OFM and discussed with the relevant USAID/Pakistan office. It provides the basis for a capacity development plan, as outlined below. In addition to pre-award assessments, ASP-RSPN also conducts “other assessments,” which includes assessing, evaluating, reviewing and validating other areas related to financial management, institutional governance, procurement and management. 2. Capacity Building The key objective of this component is to address the capacity building issues of host country organizations identified in the pre-award assessments. ASP provides capacity building support in four key areas: 1) financial management; 2) procurement; 3) administration and human resource management; and 4) monitoring and evaluation (M&E). RSPN provides capacity building support to USAID’s direct85 and indirect partners. AiD provides capacity building support to direct USAID partners including Small Grant Ambassador Fund Program (SGAFP) awardees. LUMS provides classroom training to participants from all ASP beneficiary organizations and other USAID partners. RSPN and AiD’s support to direct partners is based on the risks identified in the pre-award assessments86/other assessments. USAID/Pakistan develops a Risk Mitigation Framework (RMF), wherever applicable, for the concerned beneficiary and shares it with AiD and RSPN, which, in consultation with the concerned beneficiary organization, uses the RMF to develop an organization￾specific87 capacity building plan. Capacity building support to direct and indirect USAID partners includes development of manuals in the key areas mentioned above; on-the-job staff training in these areas; and help in preparing PC-1s (Planning Commission Pro-forma 1, which is the standard project planning document for public sector organizations). Once the concerned organization accepts the capacity building plan, it is sent to USAID/Pakistan for formal approval. After its approval, RSPN or AiD 83 The framework assesses beneficiary organizations’ control environment, financial management and accounting systems, procurement management systems, personnel policies and payroll systems, and monitoring and evaluation. 84 Assessment framework forms the basis for pre-award assessment scopes of work. 85 RSPN classifies beneficiary organizations as direct and indirect USAID partners. Direct partners are those that receive direct funding from USAID and indirect partners are those that do not receive direct USAID funding, but are strategically important for the performance of direct partners. 86 The assessments used for developing Risk Mitigation Frameworks (RMF) and capacity building plans to date are pre-ASP pre-award assessments conducted by various Chartered Accountant firms through the Blanket Purchase Agreement. 87 The capacity building plans are customized for each organization and for each institution. 51 implements the plan through a CA firm, individual consultant or members of its own staff. To avoid conflict of interest, the same ASP partner or CA firm does not carry out pre-award assessments and capacity building interventions. Capacity building of SGAFP awardees (all of them CSOs) is based on the capacity88 and institutional assessments89 conducted by SGAFP. AiD has prepared two capacity building plans, one each for the two groups (Batch 1 and Batch 2) of SGAFP awardees. Capacity building support to these CSOs is provided by developing and sharing manuals, which AiD has developed in the fields of internal audit, financial management, human resource, information technology and procurement. Following this, AiD conducts hands-on training for CSO staff on how to use these manuals. RSPN capacity building support to indirect USAID partners to-date is based on needs or other relevant assessments carried out by CA firms before ASP was established. Based on these assessments, ASP develops capacity building plans. Capacity building support to these indirect USAID partners includes development of manuals in the above-mentioned key areas, on-the-job training to staff in these key areas, and establishing institutional infrastructure. LUMS provides capacity building support by conducting standard classroom (on-campus) training in the areas of financial management, procurement, administration and human resource management, monitoring and evaluation and related areas. In addition, LUMS provides training from time to time on areas such as communication strategy and management, fiscal decentralization, leadership and cultural change. LUMS training programs target three tiers of USAID partner organization employees: tier 1- senior managers, tier 2-middle managers and tier 3-operational executives. In Year 2 and 3 ASP-LUMS offered three Tiers of Training. Based on the experience of the first two years of training, Tier 1 and 2 are merged into one single Tier through Modification number 6. From Year 4 onwards Tier 1 represents a mix of Senior and Middle Managers and Tier 2 represents Operational Executives. The courses are organized throughout the year and each course is three-to-five days long. 3. Validation According to ASP cooperative agreements (Aid and RSPN) this component aims at conducting validation of beneficiary organizations that have received capacity building assistance to assess whether the organizations are in compliance with the standards and procedures developed under the capacity building plans. The annual validation component focuses on three specific objectives: 1) ensures compliance with standards and procedures identified by USAID and ASP; 2) refers concerned organizations for further capacity building when gaps are identified; and 3) develops benchmarks for improvement. So far ASP has completed five validations. 4. Research This component focuses on conducting research on institutional aspects of host country organizations and development of best practices in the CSO sector in the fields of: 1) financial management; 2) procurement management; 3) administrative management; and 4) monitoring and evaluation. ASP through this component assesses the government’s existing policies and procedures in these fields and advises the government on changes in these policies and procedures with the aim of improving project planning and implementation. 88 CA firms commissioned by SGAFP carry out these assessments. 89 These assessments are carried out by SGAFP using a standardized checklist. 52 5. Awardee Capacity Building The purpose of this component is to enhance the capacity of ASP implementing partners and their staff. To attain this, ASP implementing partners prepare a detailed needs assessment of their respective organizations and identify areas where capacity building is required. The interventions under this component include training of ASP staff and upgrading the implementing partners’ accounting systems, operational manuals and IT infrastructure. ASP staff training fields include governance, financial management, procurement, human resource management, management information systems, and USAID rules and regulations. Activities undertaken by the three implementing partners under the five components are summarized in Figure 1. 53 FIGURE 1: ASP COMPONENTS AND MAIN ACTIVITIES Pre-Award Assessment Validation Research on Policies and Procedures RSPN and AiD  Preparation of manuals on financial management, human resources, procurement and monitoring and evaluation  On-the-job training in and mentoring exercise (for CSOs) in the four key fields  Help in PC-1preparation  Establishing institutional infrastructure LUMS Capacity Building Awardees Capacity Building Component RSPN and AiD  Assess organizations’ financial, procurement, administration and human resource management, and systems and controls. RSPN, AiD and LUMS  Development of best practices in the fields of financial management, procurement and administrative management RSPN, AiD and LUMS  Training of ASP Implementing Partners staff  Institutional capacity building of ASP Implementing Partners RSPN and AiD  Ensuring compliance with pre-award assessment recommendations, and standards and procedures developed under the institutional capacity building component. Main Activities/Purpose 54 E. CURRENT STATUS OF ACTIVITIES The status of ASP activities as of June 2013 is as follows:  Pre-award assessments of three public sector, two not-for-profit and nine for-profit private sector organizations have been completed.  Capacity building interventions in one for-profit and eight public sector organizations have been completed.  Capacity building interventions in 32 public sector and 49 not-for-profit are either partially completed or ongoing.  Validations of four public sector and one for-profit organization have been completed.  One research activity is completed and three are in the draft stage.  1,284 participants have received LUMS training including 330 in tier 1, 552 in tier 2 and 355 in tier 3. Table 3 below summarizes LUMS training activities and Table 4 summarizes RSPN and AiD ASP interventions. TABLE 3: LUMS TRAINING BY TIER LEVEL, YEAR AND FIELD OF STUDY Course Name Year 2 Year 3 Year 4 TOTAL GRAND Tier TOTAL 1 Tier 2 Tier 3 Tier 1 Tier 2 Tier 3 Tier 1 Tier 2 Tier 3 Tier 1 Tier 2 Tier 3 Financial Management 41 43 42 40 130 39 113 50 0 194 223 81 498 Procurement Management 24 148 49 40 48 56 75 59 0 139 255 105 499 HR Management & Administration 22 53 70 60 64 63 66 86 0 148 203 133 484 Monitoring & Evaluation 22 49 79 41 66 66 25 30 0 88 145 145 378 Project Management 0 0 0 89 0 0 74 0 0 163 0 0 163 Leadership: Gender Dimension 18 0 0 55 0 0 38 0 0 111 0 0 111 Forensic Audit 0 0 0 0 0 0 31 0 0 31 0 0 31 Fiscal Decentralization 0 0 0 27 0 0 27 0 0 54 0 0 54 Procurement Audit 0 0 0 38 0 0 0 0 0 38 0 0 38 Training of Trainers 29 0 0 25 0 0 0 0 0 54 0 0 54 Communication Strategy 0 0 0 15 33 0 34 0 0 49 33 0 82 TOTAL 156 293 240 430 341 224 483 225 0 1,069 859 464 2,392 689 995 708 2,392 55 TABLE 4: AS OF APRIL 2014, PROJECT INTERVENTIONS BY COMPONENT AND LOCATION Component Location Type of Partner AJK Balochistan Countrywide Gilgit￾Baltistan Islamabad KP Punjab Sindh Grand Total Pre Award Assessments For Profit - - - - 3 - 4 2 9 G2G 10 14 - 9 1 24 13 14 85 Not for profit - - - - - - - 1 1 Pre Award Assessments Total 10 14 - 9 4 24 17 17 95 Capacity Building - Direct Partners For Profit - - 2 - - - - - 2 G2G - - - - 2 7 3 5 17 Not for profit - - 1 - 2 - - - 3 Capacity Building - Direct Partners Total - - 3 - 4 7 3 5 22 Capacity Building - Indirect Partners For Profit - - - - - - - - - G2G - 1 - - 4 3 - 1 9 Not for profit 8 66 - 13 61 121 117 160 546 Capacity Building - Indirect Partners Total 8 67 - 13 65 124 117 161 555 Validation For Profit - - - - - - 1 - 1 56 Component Location Type of Partner AJK Balochistan Countrywide Gilgit￾Baltistan Islamabad KP Punjab Sindh Grand Total G2G - - - - 1 1 1 3 6 Not for profit 1 9 - 1 2 9 9 6 37 Validation Total 1 9 - 1 3 10 11 9 44 Research G2G - - 4 - - - - - 4 Research Total - - 4 - - - - - 4 Grand Total 19 90 7 23 76 165 148 192 720 57 II. RATIONALE FOR EVALUATION A. PURPOSE OF EVALUATION ASP has completed three years of its five-year implementation period. Ample time remains to adapt the program approach to enhance performance and the prospects for producing sustainable results. USAID/Pakistan and the ASP implementing partners (RSPN, AiD and LUMS) are particularly interested in learning about the effectiveness of the program approach and components with the aim of identifying opportunities for improvement. USAID/Pakistan is also particularly interested in sustainability. Sustainable results will depend not only on developing the capacities of the target organizations, but also the extent to which these organizations are institutionalizing improvements introduced with ASP assistance. Ways in which ASP and beneficiary organizations are addressing issues of gender equality are also an important area of inquiry for this evaluation. Looking toward the future, USAID/Pakistan is also interested in potential demand among donors and donor-assisted implementing institutions for the kind of services that ASP has been providing. It is interested in such demand among donor and government organizations as well as non-government organizations, both for-profit and non-profit. This kind of assessment, however, goes beyond an evaluation that is based on project-related evidence and will be reported separately from the evaluation. B. AUDIENCE AND INTENDED USE The primary audience for the evaluation includes: (i) USAID program managers and relevant decision￾makers; (ii) ASP implementers; and (iii) beneficiary organizations. The evaluation is intended to provide feedback for improvement of ASP. C. EVALUATION QUESTIONS USAID is interested in addressing four main questions. Three of the questions can be answered with evidence from the project and its implementing partners. The fourth one is a forward-looking question that is not strictly an evaluation question and entails research beyond the institutions that are engaged with the project. The fourth question will be undertaken separately from the evaluation itself. The four questions and corresponding explanations are described below. Question No. 1: To what extent have the program’s overall approach and individual components contributed to achieving the results set out in the ASP results framework? What factors have influenced the program’s achievement of results? Explanation: This question will address effectiveness from two perspectives –by individual component and overall for the project. With the exception of validation, the question will address, the effectiveness of individual components (i.e., to what extent are the individual components contributing to achieving overall results) 58 and the effectiveness of the overall approach (i.e., is the program achieving the results expected)90 at this stage of implementation. For the first perspective, the evaluation will assess the effectiveness of the pre-award assessment, capacity building and research components in contributing to ASP’s Intermediate Results (IRs) and Sub￾IRs. It will, in particular, examine: (a) The effectiveness and use of pre-award assessments in relation to the respective USAID/ Pakistan assessment frameworks and scopes of work; (b) The effectiveness of direct and indirect partners’ capacity building (through AiD and RSPN) in comparison with the requirement for risk mitigation identified through pre-award assessments91 and corresponding risk mitigation frameworks. This will include assessing the usefulness of manuals, on-the-job-training and PC-1s in promoting compliance with USAID and government regulations. For the small grants partners, the evaluation will assess effectiveness in terms of the suitability92 and usefulness93of manuals and mentoring; (c) The suitability and usefulness of the LUMS’s training program in improving USAID/Pakistan partners’ human resources and management systems. The evaluation will also examine whether there are synergies between the training provided by LUMS and other capacity building efforts (i.e., does exposure to both types of capacity building enhance results?); and (d) The effectiveness of completed research in relation to its objective and scope of work. This assessment will identify the specific factors both contributing to and detracting from the project’s overall effectiveness as well as that of its individual components. Along with a number of other sources, validation reports prepared through ASP will be used as data contributing to this evaluation, including this evaluation question. Question No. 2: To what extent are ASP results and the services provided by ASP implementing partners likely to be sustainable? Explanation: The first part of this question, focusing on the sustainability of results, will assess the extent to which beneficiary partners have institutionalized capacity building measures introduced by ASP in the areas of financial, procurement, administration and human resource management and monitoring and evaluation. Institutionalization includes a beneficiary organization’s recognition of the usefulness of an ASP intervention, a decision to implement it, its reflection in system documents, allocation of resources required for implementation, and actual organizational practice. The second part of the question will focus on the assessment of any plans prepared by the implementing partners for continuing to provide ASP-type services after the end of ASP. Question No. 3: To what extent, if any, has ASP included gender as a consideration in its interventions? 90 All the indicators of the results framework (Table 2) are numbers intended for reporting by the IPs. The evaluation will assess progress in achieving results in a qualitative manner, based on a sample of the beneficiary organizations. 91 This will include pre-award assessments conducted before ASP, which were the basis for developing some of the capacity building plans implemented through ASP. 92 Suitability will be assessed with reference to USAID/Pakistan requirements. 93 Usefulness from the point of view of beneficiary partners. 59 Explanation: The evaluation will address this question by answering the following sub- questions:  Does the Cooperative Agreements reflect the findings of a gender analysis and/or considerations for gender equality?  To what extent and by what means (human resources management (HRM) policies, terms of employment, supportive workplace, etc.) do the ASP IPs strive for gender equality?  How are gender concerns, particularly gender equality, incorporated in the PC-1s and the HRM and M&E manuals prepared by ASP for beneficiary organizations?94  To what extent are concerns with gender equality reflected in the curricula of LUMS courses on communication strategy and management, leadership and cultural exchange, monitoring and evaluation, project management and training of trainers?  What steps has ASP taken to encourage women’s participation in LUMS courses and to what effect in terms of enrollment (in various courses and for all tiers)?  How do the current positions and roles of LUMS program alumni compare across male and female alumni? III. DESIGN AND METHODOLOGY A. DATA COLLECTION METHODS The evaluation team will apply a mixed-methods approach, using both quantitative and qualitative techniques to collect data from multiple sources in a variety of locations to ensure multiple levels of triangulation. The two broad areas addressed by the evaluation include the effectiveness and institutionalization of research and capacity building. The institutionalization of research and capacity building can be further sub-divided into intensive work with direct and indirect beneficiaries, limited work with the CSOs that comprise the SGAFP awardees, and the LUMS training program. Each of these will be examined through a unique set of methodologies. The overall effectiveness can then be examined by synthesizing the findings for each component. Intensive Capacity Building: A Case Study Approach The effectiveness and institutionalization of capacity building interventions of AiD and RSPN direct and indirect beneficiaries (except the SGAFP awardees) will be assessed through case studies. This will entail drawing on information from relevant documents and key informant interviews from selected beneficiary organizations and relevant USAID technical teams. A case study, according to the USAID Glossary of Evaluation Terms is “a systematic description and analysis of a single project, program, or activity.” Accordingly, a case study is a qualitative method of inquiry, which is used for a comprehensive examination of a single observation or phenomenon.95A case study uses context-specific, non-survey based evidence such as process-tracing, historical, documentary, and participant observation. It also uses multiple sources for triangulation to investigate properties of phenomena.96A case study usually focuses on questions of how and why.97 94 This will only include direct beneficiary organizations and will not include the indirect beneficiary organizations. 95 Gerring, John. (2007). Case Study Research: Principles and Practices. Cambridge: Cambridge University Press. 96 ibid. 97 Yin, Robert K. (2003). Case Study Research: Design and Methods. California: SAGE Publications. 60 The case studies will include 6 public sector organizations. The selection criteria for these organizations include:  Beneficiaries who have completed (or nearly completed) enough interventions that they will be responsive to evaluations questions;  Both direct and indirect program beneficiaries;  Both governmental and non-governmental organizations;  At least one beneficiary organization for each USAID/Pakistan participating office; For each case included the team will use the exact same instruments to examine the same set of issues pertaining to the effectiveness and institutionalization of capacity building. Each case study will include three stages: first, using a checklist based on the evaluation questions, the team will conduct a thorough desk review of documents mentioned in the section on available documents below. Next, using a similar, but not exactly the same instrument, the team will conduct key informant interviews with relevant staff from USAID technical offices and five key officials from each beneficiary organization – the chief executive or equivalent, and the managers in-charge of financial, human resource, and procurement management, and monitoring and evaluation. Finally, they will do spot checks at beneficiary organization offices to assess compliance with the procedures they should have adopted as a result of ASP interventions. Each case will be written up separately in order to facilitate both within case and cross￾case comparison. LUMS Training Assessment To evaluate the effectiveness and institutionalization of the training provided by LUMS, the evaluation will draw upon documents, including training materials, interviews with LUMS staff and quantitative data from an online survey of LUMS training beneficiaries. Key informant interviews at LUMS will include up to 10 ASP-LUMS staff members and faculty who will be chosen by the evaluation team at the time of the fieldwork. The quantitative portion of the evaluation will rely on a short online survey targeted towards all (1,284) the participants of trainings conducted by LUMS. The survey will be outsourced. The evaluation team will design the survey questionnaire, which the subcontractor will implement by first pre-testing the questionnaire and then sending emails to all participants for whom email contacts are available, with a URL for the questionnaire. Table 3 above provides a summary of LUMS training participants by training area and management tier. Limited Capacity Building: SGAFP Awardee Training Assessment The effectiveness and institutionalization of the limited capacity building provided to the SGAFP awardees will be assessed through document review, particularly of the manuals provided the awardees, and group interviews. Based on their availability, five to ten SGAFP awardees will be invited to participate in two to five group interviews, of which one will held in each location the evaluation team visits. Overall Assessment In addition to examining each of these project components, the team will need to speak with key personnel from both USAID/Pakistan, particularly OFM, involved in the project and ASP Implementing Partners. These key informant interviews will provide both project background and facilitate the assessment of overall project effectiveness. 61 Table 5 below details the complete proposed key informant interview plan. As noted above document review is a key component of the evaluation methodology. The required documents include, but are not limited to the following:  Assessment framework and corresponding scopes of work for assessing effectiveness of pre￾award assessments;  Pre-award assessment, risk mitigation frameworks, capacity building plans and other system documents (i.e. manuals and PC-1) for assessing effectiveness of capacity building support provided to direct beneficiary partners;  Relevant assessments, capacity building plans and other system documents (i.e. manuals) for assessing effectiveness of capacity building support provided to indirect beneficiary partners;  Institutional assessments and manuals to assess capacity building support provided to SGAFP supported CSOs; and  Completed research documents for assessing the research component. USAID/Pakistan and ASP will provide all the required secondary data in advance of the evaluation team’s arrival in Islamabad, except for the training manuals and other training materials developed by LUMS, which the evaluation team will review when it visits the university. 62 TABLE 4: PROPOSED KEY INFORMANT INTERVIEW PLAN Organization USAID/ Pakistan Office IP Type of Beneficiary Budget (US$) City/ Province Total Interviews USAID/Pakistan (OFM only), AiD and RSPN OFM Islamabad 9 For Case Studies on Capacity Building by AiD and RSPN USAID/Pakistan KP/FATA, Sindh and Technical Teams KP/FATA Sindh Technical Islamabad Karachi Islamabad 3 3 4 1. Provincial Reconstruction, Rehabilitation and Settlement Authority (PaRRSA), KP RSPN Direct 238,084 KP 5 2. Small and Medium Enterprise Development Authority, Lahore (SMEDA) EGA RSPN Direct 92,238 Punjab 5 3. Municipal Services Delivery Program, Sindh (MSP-Sindh) Sindh RSPN Direct 118,040 Sindh 5 4. Municipal Services Delivery Program, KP (MSP-KP) KP/FATA AiD Direct 55,565 KP 5 5. Urban Policy Unit (UPU) KP/FATA AiD Direct 20,000 KP 5 6. Northern Power Generation Company Limited Jamshoro Energy AiD Direct 50,000 Province 5 7. Public Procurement Regulatory Authority, Islamabad (PPRA) OFM RSPN Indirect 179,556 Islamabad 5 8. Department of Finance –Internal Audit Function in KP OFM RSPN Indirect 344,271 KP 5 For Capacity Building by LUMS LUMS staff members and faculty OFM Lahore 10 For the Research Component Research on Revolving Fund Account (RFA)/Assignment Account OFM RSPN - 13,647 Islamabad 2 Research on Project Management Units Approach OFM RSPN - 14,900 Islamabad 2 Research on GOP Project Cycle OFM RSPN - 22,340 Islamabad 2 Total 85 63 B. DATA ANALYSIS METHODS For analysis of qualitative data, the evaluation team will:  Prepare summary reports of the document desk review.  Summarize key informant interview notes and code them according to themes relevant to the evaluation questions.  Prepare tally sheets identifying the themes that emerge in the document review and key informant interviews to facilitate systematic and rigorous data analysis aimed at identifying key evaluation findings.  Prepare individual reports for each case study summarizing key findings.  Prepare cross-case analysis report summarizing findings across the case studies. Prepare summaries of roundtable discussions, listing all relevant issues and summarizing the main findings and recommendations. The latter will be based on the findings and the expert views of the consultant’s staff. The team will analyze the quantitative data by preparing cross-tabs and frequency distributions from the online survey. Finally, the team will prepare a detailed outline summarizing key findings, based on all the data analysis, and conclusions for each evaluation question and overall recommendations. C. METHODOLOGICAL STRENGTHS AND LIMITATIONS The evaluation methodology relies on triangulation of sources and methods to ensure the validity and reliability of results. The scope of this evaluation is limited to answering the evaluation questions to the extent that they can be operationally defined and data are available. The methods proposed for collecting and analyzing data are potentially subject to selection bias, which occurs when the subjects of surveys or interviews are not representative of the population of interest. In this case, selection bias is most likely to result from the online survey when some respondents choose to participate while others do not. The evaluation relies on the timely response of training participants to the online survey. Low response rates due to their professional and personal time commitments will be beyond the consultant’s control. This limitation can affect the representativeness of the sample and reliability of results. It may also occur through the purposive sampling used for the qualitative data collection methods. However, in both cases, they will be offset by rigorous triangulation of methods and participants as well as variation in the geographic location of participants. Moreover, any selection bias in the survey results can be addressed through the data analysis. D. EXISTING DATA A summary of the documents available to the evaluation team is listed below. A complete list of documents that will be reviewed by the evaluation team is attached in Annex 3. A complete list of documents available for direct and indirect partners is attached in Annex 4. 64 Program Documents  USAID’s latest Mission Strategic Framework (MSF)  ASP cooperative agreements and Program Description amendments for RSPN, AiD and LUMS  PMPs for RSPN, AiD and LUMS  Annual and quarterly reports RSPN, AiD and LUMS  Project’s current status in form of Activity Trackers for RSPN and AiD  Annual Work-plans RSPN, AiD and LUMS  List of training course applicants, consultants and faculty (LUMS)  List of cases studies used in the training (LUMS)  LUMS training-Mid-term Impact Assessment Report, 2012 Documents on Components Pre-Award Assessments  Before ASP o RSPN- assessment reports (nine) o AiD- assessments reports (five) and institutional assessments of CSOs (24) of Small Grants and Ambassador Fund Program (SGAFP)  RSPN o Pre-award assessment frameworks, scopes of work and pre-award assessment reports (12) o Third part validation reports (two) Capacity Building  RSPN o Various documents including, PC-Is, need assessment reports and capacity building plans for public sector organizations (13)  AiD o Manuals in the fields of internal audit, financial management, procurement, human resources and IT for CSOs (five) o Various manuals in the fields of budget and accounting, internal audit, procurement, financial management, human resource, IT, and monitoring and evaluation for public sector organizations (seven) Awardee Capacity Building  RSPN o Training reports (seven) o Capacity Building Plan for Institutional Development Awardees o Manuals in the fields of internal audit, procurement, financial management, human resource, and IT (nine) Validation  RSPN o Scopes of work and validation reports (four) Research  RSPN o Research reports (three completed) 65 E. EVALUATION PROCESS The evaluation process consists of five main stages: Stage 1: Preparation and planning by the consultant’s staff. Stage 2: Initial review of priority documents by the evaluation team. Working from their home stations, the evaluation team members will review all project background documents. Stage 3: Teamwork in Islamabad. Upon deployment in Islamabad, the evaluation team will participate in a team-planning meeting facilitated by the consultant and undertake specific preparatory tasks as a team. The team will adopt a case-study approach for one part of this evaluation. The consultant and evaluation team will finalize the SOW in collaboration with USAID during the team-planning meeting. Following the team-planning meeting and before the start of fieldwork, the team will conduct a thorough desk review of the documents related to various components of ASP. Stage 4: Fieldwork. The evaluation team will begin its fieldwork after it completes its teamwork in Islamabad. Team members will conduct key informant interviews and on the spot document review for case studies where the beneficiary organizations are located. At least one team member will be based exclusively in Lahore to carry out document review98 and interviews at LUMS. In addition, the consulting firm will administer an online survey of ASP-LUMS training participants, so that data can be available for analysis in time for the evaluation team’s report writing activity. It will also organize key informant interviews and roundtable discussions for answering the forward-looking questions. Stage 5: Data analysis and report writing. IV. EVALUATION PRODUCTS A. DELIVERABLES The evaluation will produce the following deliverables: 1. Finalized draft SOW in collaboration with USAID/Pakistan and USAID/Pakistan Program Office. The SOW will meet USAID expectation for quality evaluation Statement of Work; 2. Survey questionnaire, key informant interview instruments, and a list of issues for discussion at roundtables; 3. A debriefing discussion with USAID and the implementing partner on initial findings; 4. A draft evaluation report; 5. A final evaluation report consistent with USAID standards for evaluation reports (outlined below); 6. A presentation to USAID and the implementing partners on the final report. Fieldwork will not start until USAID has approved the final SOW and commented on the instruments. 98 LUMS will make its training material and faculty available to the evaluation team at its campus in Lahore. 66 B. REPORTING GUIDELINES The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced in Annex 2. The evaluation report will follow the structure given below (the section titles and order are illustrative):  Title page  Table of contents and table of tables and charts  List of acronyms  Acknowledgements  Program summary  Map showing the location of program activities  Executive summary (ideally not to exceed five pages)  Evaluation purpose and evaluation questions  Program background. This information provides important context for understanding the evaluation purpose, questions, methods, findings and conclusions and includes:  the problem statement;  the theory of intervention;  the design of the program;  the program’s results framework; and,  program implementation, including the current status of the project.  Evaluation methods and limitations, describing in detail the evaluation design and methods with explanation as to why they were chosen, with additional information provided in the annexes, if required.  Summary of data analysis, including methods and other relevant observations.  Findings and conclusions. This section (or sections) will include findings and conclusions for each evaluation question. If there are a large number of findings, there will be a synthesis or summary of findings for each question that establishes the connection with the conclusions that follow.  Main conclusions and recommendations. This section will recapitulate the main conclusions, including those that form the logical basis for recommendations. It will highlight a few key recommendations, or clusters of recommendations, that include actionable statements of what remains to be done, consistent with the evaluation’s purpose, and based on the evaluation’s findings and conclusions. It will provide judgments on what changes need to be made for future USAID financial and cooperative development programming. This section will also recommend ways to improve the performance of future USAID programming and project implementation; ways to solve problems this program has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by management.  Annex  Evaluation Statement of Work  Evaluation Methods and Limitations 67  Data Collection Instruments  Data Analysis Plan  Bibliography of Documents Reviewed  List of individuals and agencies contacted  Disclosure of Any Conflicts of Interest  Statement of Differences (only if applicable)  Evaluation Team Bios V. TEAM COMPOSITION A Team Leader/Evaluator will lead a four-person evaluation team to conduct this interim performance evaluation in accordance with the USAID Evaluation Policy and directives. S/he is expected to possess at least fifteen years of experience in international development, experience of evaluating USAID projects, short-term consulting experience in Pakistan or neighboring countries, a master’s degree in business/economics or another relevant field and demonstrated leadership and report writing abilities and communication skills. An Institutional/G2G Specialist will be the second team member. S/he is expected to possess at least fifteen year of relevant experience in implementation of USAID funded projects in Pakistan, particularly those focusing on a G2G modality. The consultant is expected to have a good knowledge and understanding of USAID regulations and contracts. A Chartered Accountant will be the third member of the team. S/he is expected to possess at least fifteen years of relevant experience in the areas of financial management, human resource management, procurement and corporate governance in Pakistan. An Education Specialist will be the fourth team member, focusing mainly or exclusively on LUMS. S/he is expected to possess at least ten years of relevant experience in implementation, academic or consulting positions, in-depth experience of evaluating higher education/ training programs in Pakistan, short-term consulting experience in Pakistan, a master’s degree in a relevant field and demonstrated report writing abilities. In addition to this evaluation team, the consultant will provide expertise for the online survey and the forward-looking question. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. 68 VI. EVALUATION MANAGEMENT A. LOGISTICS In terms of logistics, this assignment requires inputs and actions from the evaluation partners as summarized below: USAID/Pakistan  The Program Office’s Performance Management Unit (PMU) will facilitate the preparation of the evaluation SOW in accordance with USAID standards and good practices, review the instruments and the draft report, and provide technical inputs on the contractual matters. ASP IPs  The three IPs –RSPN, AiD and LUMS –will provide all relevant information required for the evaluation and facilitate meetings and interviews which the evaluation teams may require with their staff and beneficiary organizations. Beneficiary Organizations  The staff from selected beneficiary organizations is expected to cooperate with the evaluation team by giving time for meetings and interviews and providing relevant documents. Those who will have participated in the LUMS training will be included in an online survey. The Consulting Firm  The consulting firm will provide support for travel, lodging and other arrangements related to evaluation team’s work. 69 B. SCHEDULING Activity W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 W13 W14 W15 W16 W17 W18 Pre-arrival team work Team Planning Meeting Desk Review of documents Field work – KIIs and roundtables Field work – Online Survey Data Processing (quantitative data) Data Analysis (quantitative and qualitative) and FCR Workshop Initial findings debriefing and report writing Internal review & revision Branding and editing Submission of draft report Comments from USAID Final Report submission to USAID Presentation to USAID 70 C. BUDGETING Level of Effort (days) (The consultant may propose additional staff for the online survey and the forward-looking question) Tasks Team Leader (STTA) Chartered Accountant (STTA) Institution Specialist (STTA) Education Specialist (STTA) Pre-arrival  Review of documents 2 2 2 2 Review of instruments 3 2 2 2  Drafting initial chapters 3 Team Planning Meeting 7 7 7 7 Desk Review/Analysis of Docs 10 10 10 10 Field work – KIIs 10 10 10 10 Field work – Online Survey Quantitative/ Qualitative Data Analysis and Internal FCR workshop 15 15 15 15 Report Writing and initial findings debriefing 6 3 6 3 Internal review/ revisions 3 Final revisions 3 Presentation to USAID and IPs Travel 4 4 Total 66 49 56 49 71 SOW Annex 1: Getting To Answers (G2A) Matrix Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/Selection Data Analysis Methods Question No. 1: To what extent have the program’s overall approach and individual components contributed to achieving the results set out in the ASP results framework? Descriptive  Description of what each case with respect to evaluation the question Comparative  Comparative analysis across cases  Documents review  Key informant interviews  Online survey  Group Interview Review of Documents:  Project records  Pre- ASP assessment reports  Assessment frameworks and scopes of work  Assessment Reports prepared by ASP  Capacity building plans developed by ASP for partner organizations  Various manuals developed by ASP for partner organizations  Training materials prepared by LUMS for tier 1, 2 and 3 participants in various fields Key Informant Interview:  Relevant ASP and USAID officials  Relevant partner organizations officials (direct and indirect public sector, CSOs and private sector organizations)  Participants in trainings conducted by ASP Survey:  Online survey of participants in trainings conducted by ASP  Purposive sampling for selection of partner organizations  Purposive/ Convenience sampling for selection of key informants and Group Interview participants  Universe of training participants for online survey (subject to availability of email addresses)  Frequency tables and cross-tabs of survey data  Case study development for each organization and comparative analysis across case studies and group interviews  Checklist and content analysis of capacity building materials prepared by ASP 72 Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/Selection Data Analysis Methods Question No. 2: To what extent are ASP results and the services provided by ASP implementing partners likely to be sustainable? Descriptive  Description of what each case with respect to evaluation the question  Key informant interviews  Group Interview Key informant interview:  Relevant ASP and USAID officials  Relevant partner organizations officials (direct and indirect public sector, CSOs and private sector organizations)  Participants in trainings conducted by ASP  Purposive sampling for selection of partner organizations  Purposive/ Convenience sampling for selection of key informants and Group Interviews with participants  Case study development for each organization and comparative analysis across case studies and group interviews Question No. 3: To what extent, if any, has ASP incorporated gender equality in its interventions? Descriptive  Description of what each case with respect to evaluation the question  Document review  Key informant interviews  Group Interview Review of Documents:  Project records  Cooperative agreements  Various manuals developed by ASP for partner organizations  Training materials prepared by LUMS for tier 1, 2 and 3 participants in various fields Key Informant Interview:  Relevant ASP and USAID officials  Relevant partner organizations officials (direct and indirect public sector, CSOs and private sector organizations)  Participants in trainings conducted by ASP Survey:  Online survey of participants in trainings conducted by ASP  Purposive sampling for selection of partner organizations  Purposive/ Convenience sampling for selection of key informants  Universe of training participants for online survey (subject to availability of email addresses)  Frequency tables and cross-tabs of survey data  Case study development for each organization and comparative analysis across case studies and group interviews  Checklist and content analysis of building materials prepared by ASP LUMS 73 Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/Selection Data Analysis Methods Forward-looking question: To what extent are donors and the present and potential implementing partners of donor-assisted projects interested in using and willing to pay for the services that ASP is providing?  Descriptive  Key informant interviews  Roundtable discussions Representatives of USAID, other donors, government organizations and non￾government for-profit and non￾profit organizations  Purposive/ Convenience sampling  Summarize interviews and roundtable discussions and conduct content analysis 74 SOW Annex 2: Reporting Guidelines 1. The evaluation report must represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action. Note: These guidelines are taken from ADS 203.3.2.8 (Documenting Evaluations) - http://transition.usaid.gov/policy/ads/200/203.pdf - which is based on Appendix 1 of USAID Evaluation Policy: Criteria to Ensure the Quality of the Evaluation Report 75 SOW Annex 3: List of Available Documents Following is the list of documents currently available for the evaluation team: ASP RSPN Agreement and Performance Management Plan  ASP-RSPN: Modification No 02 Cooperative Agreement No. 391-A-00-11-01201  ASP-RSPN: Presentation for MSI, May 8, 2013  Performance Management Plan of RSPN, January 2013. Awardee Capacity Building Training  Advanced Specialized Gender Training, November 24-25, 2011  Disaster Management Training, October 26, 2011  Workshop on Effective Report Writing, June 7-9, 2012  Gender Sensitization Training, November 11, 2011  Government Rules & Regulations Training, September 26-28, 2011  Training Workshop on Effective Monitoring and Evaluation Skills, February 23-25, 2012  USAD Rules and Regulations Training, December 13-15, 2011 Plan  Capacity Building Plan for Institutional Development Awardees, September 23, 2011 Manuals  Financial Management Manual (Draft), RSPN, 2012  Human Resource Policies and Procedures Manual (Draft), RSPN 2012  Human Resources Manual, RSPN, 2012  Information Technology Manual (Draft), RSPN, 2012  Information Technology Manual, RSPN  Internal Audit Manual (Draft), RSPN, 2012  Internal Audit Manual, RSPN  Procurement Manual (Draft), RSPN, 2012  Procurement Manual, RSPN 76 Pre-Award Assessment Pre-award assessment frames, scopes of work and pre-award assessment reports for:  AA Associate  Abacus Consulting  EA Consulting  ECO West Consulting  Halcrow Consulting  Izhar Construction  Marie Stopes Society  Matracon Pakistan  National Development Consultants  NESPAK,  National Highway Authority (NHA)  Qavi Engineering Validation  Pakistan’s Citizens’ Damage Compensation Program, Third Party Validation Revised Inception Report, June 01, 2011  Citizens’ Damage Compensation Program, Third Party Validation Final Report, 2012 Research Assignment Account (folder name)  Chief of Party (COP), RSPN letter to Controller General of Accounts (CGA) regarding ASP’s study on Revolving Account Fund, December 10, 2012  Controller General of Accounts acknowledgement for meeting with COP RSPN, December 20, 2012  Controller General of Accounts’ notification regarding Revolving Account Fund study  Controller General of Accounts and ASP team meeting minutes, December 28, 2012  Reply to Controller General of Accounts from COP, January 10, 2012  Working Paper on changes in the procedure for Revolving Fund Account (RFA) (Assignment Account) for Foreign Funded Projects (Notification dated January 17, 2012)  CGA comments on ASP study, February 13, 2012  Comparative Table on proposed amendments by ASP and Accountant Generals of provinces, February 26, 2012  Notification of CGA on ASP-NRSP study on Revolving Fund, March 12, 2012  Concept note on Assignment Account study 77  Revolving Fund Accounts for Development Projects in Pakistan: Implementation Options and Way Forward, Draft Reports of Research Study, October 2011 Research Planning Architecture  Concept Note: Research Topic 02: Evaluate, streamline & improve the GOPs Project Design, Approval & Implementation Process  Planning and Development Architecture in Pakistan: A Strategic Review, October 2012 Project Management Unit  Concept Note: Research Topic 02: Evaluate, streamline and improve the GOPs Project Design, Approval & Implementation Process  Final Report: Evaluate, Streamline and Improve the Government of Pakistan’s Project Management Units Approach: An Insight into the Issues and Implementation of Project Management through Project Management Units, October 2012 Public Financial Management  Agenda and attendance sheet of participants of meeting ‘Advisory Group On Public Financial Management Assessment Framework, November 5, 2012.  Members list of Advisory Group on Public Financial Management Assessment Framework’, August 6, 2012  Institutional Performance Assessment Framework (IPAF): End of Project Report, January 22, 2103.  Inception Report: Study on Public Financial Management  Institutional Performance Assessment Framework Module 1, final draft, February 8, 2013. Validation Abacus Consulting  Scope of Work, Program Description For Validation Of Abacus Consulting Technologies (Pvt) Ltd  Validation Repot Of Abacus Consulting Technologies (Private) Limited, Rafaqat Mansha Mohsin Dossani Masoom & Co. December 04, 2012 Trust for Democratic Education and Accountability  Scope of Work, Program Description For Validation of Trust for Democratic Education and Accountability (TDEA)  Validation / Follow Up Review Of Trust For Democratic Education And Accountability, ASP-RSPN, June 27, 2012 ASP- AiD  Revised Performance Management Plan 2010–2015, January, 2013  Revised Program Description for Modification No 04. ASP-AiD, Assessment and Strengthening Program (ASP) for Civil Society Organizations (CSO) and Government of Pakistan (GOP) 78 CSOs in Batches 1 and 2  Risk status sheet called rating chart for 2nd batch  Agenda: Institutional Capacity Building for Civil Society Organizations, Human Resource Management Training Workshop, 13th – 16th October 2012  Agenda: Institutional Capacity Building for Civil Society Organizations, Walk through IT Manual, 21st October 2012, Pearl Continental Hotel, Karachi  Agenda: One Day Training on Workplace Ethics & Code of Conduct, July 6th, 2012, PC Hotel Karachi  Agenda: Institutional Capacity Building for Civil Society Organizations, USAID-SGAFP Compliance Training Workshop, 17th-18th, 2012  Agenda: Institutional Capacity Building for SGAFP Grantee Organizations, Financial Polices & Procedure Training Workshop, dated: 29-31 January 2013Venue: Marriot Karachi  Participant Manual on procurement  Contact list of partner organizations – two documents (partial)  Final Training Needs Assessment (TNA) analysis  Capacity Building Plan USAID Small Grants Program’s Awardees, January 24, 2012.  Capacity Building Plan for Small Grants and Ambassador Fund yearly plan May 2012-June 2103 Manuals  Internal Audit, Financial Management, Procurement, Human Resources, IT Policies and Procedures manuals AiD Procurement Documents  Around 26 procurement related forms are provided which include Supplies Request, Request for Quotation Forms, Purchase Order, Tender Notice, Bank Guarantee etc. Pre-ASP Pre-award assessments and institutional assessments Pre-award assessment reports  Balochistan Rural Development & Research Society (BRDRS)  RAHMA Islamic Relief  PADO  MRDS  FIDA Institutional Assessment of Small Grants and Ambassador Fund Program (SGAFP)  Hashoo Foundation  Research and Development Solutions (RADS)  FACES Pakistan 79  SAFE  TWO  The Citizen Foundation  AWS  Balochistan Boys Scout Association  Relief Pakistan  Friends Development Organization  LBRT  Tareen Education Foundation  Bam Development Organization  MTM  Falah Foundation  SMJH  Sahara Welfare Development Organization  SDO  NVCDDF  Himalayan Development Foundation  Maalar Kolwa Local Support Organization  SAMI Foundation  Sargana Literacy and Cultural Society  NRDF Public Sector Organizations  Budget and Accounting Manual, Capacity Building Plan, Internal Audit Manual, Procurement Manual for CGPL Guddu, JPCL Jamshoro, NGPL Muzaffargarh.  Financial Management Plan, HR Manual, IT Policies and Procedures, M&E Manual, Capacity Building Plan, Procurement Manual for FATA Secretariat (FS)– PMU  Financial Management Plan, HR Manual, M&E Manual, Capacity Building Plan, Procurement Manual for Municipal Services Program (MSP) KP – PMU, Urban Policy Unit (UPU) KP  Capacity Building Plan For Local Government and Rural Development Department LGRDD ASP LUMS  Modification #3 in Cooperative Agreement ASP-LUMS  Performance Management Plan, October 2011  List of certified applicants 80  ASP- LUMS Training Calendar  ASP – LUMS presentation to MSI, May 13, 2013  List of Consultants for MSI  List of Faculty  List of case developed  Mid-term Impact Assessment Report, 2012  PMU Employee List Annual and Quarterly Progress Reports  Year 2 Annual Progress Report, October 2011 – September 2012  Quarterly Progress Reports:  October – December 2010  January – March 2011  April – June 2011  July – September 2011  October – December 2011  January – March 2012  April – June 2012  July – September 2012  October – December 2012  January – March 2013 81 SOW Annex 4: Available Documents for Direct and Indirect Partners TABLE 5: LIST OF AVAILABLE DOCUMENTS FOR DIRECT PARTNERS 99 Institutional assessments for 24 CSOs have been conducted by SGAFP and five pre-award assessments conducted by CPA firms are available. Capacity Building- Direct Partners Pre-award Assessment/ Other Assessment Risk Mitigation Framework Capacity Building Plan Validation Manuals PC-1 RSPN Auditor General of Pakistan (AGP) × - × - × Health Services Academy (HAS) - - - - - - PaRRSA × - × - × Municipal Services Program, Sindh (MSP-Sindh) × - × - × × Sindh Basic Education Program (SBEP) × - × - × × Small and Medium Enterprise Development Authority (SMEDA) - - × - × - Higher Education Commission (HEC) - - × - - × AiD FATA Secretariat-Project Management Unit - - × - × - Municipal Services Program-KP (MSP-KP) - - × - × - Urban policy Unit-KP - - × - × - Local Government Rural Development - - × - - - Department-KP - - × - × - GENCO I (Jamshoro) - - × - × - GENCO II (GUDDU) - - × - × - GENCO III (Muzaffargarh) - - × - × - Rural Support Programs - - - - - - SGAFP99 (Batch one and two) × - × - × - 82 TABLE 6: LIST OF AVAILABLE DOCUMENTS FOR DIRECT PARTNERS Beneficiary Partners Capacity Building- Indirect Partners (RSPN) Pre-award Assessment/ Other Assessment Risk Mitigation Framework Capacity Building Plan Validation Manuals/Other Public Procurement Regulatory Authority (PPRA) - - × - - PPRA, KP × - × - × PPRA, Sindh × - × - PPRA, Balochistan × - × - Internal Audit, KP × - × - × Strategic Planning, KP × - × - × STI - - × - - 83 SOW Annex 5: Current Status of Activities TABLE 7: STATUS OF CURRENT PROJECT INTERVENTIONS BY LOCATION Support/Activity Location USAID Office Grand Total Total (After accounting for Sindh DG DOCS duplication) Education EG Energy FATA/KP Health OFM OIE Program Office Pre-Award Assessments Countrywide - - - - - - - - - 1 1 13 Islamabad - - - - - - 1 - - 4 - 5 Punjab - - - - - - 1 - - 4 - 5 Sindh 1 - - - - - - 1 - 2 - 4 Pre Award Assessments Total 1 - - - - - 2 1 - 10 1 15 Capacity Building - AJK - 1 - - - - - - - - 1 2 64 Direct Partners Balochistan - 11 - - - - - - - - 1 12 Countrywide - 1 - - - - - - - - - 1 Gilgit Baltistan - 1 - - - - - - - - - 1 Islamabad - 6 - 1 1 1 - - - 9 KP - 14 - - 7 - - - - 21 Punjab - 7 - - 1 1 - - - - - 9 Sindh 3 7 1 1 2 - 1 2 - - 17 Capacity Building - Direct Partners Total 3 48 1 2 1 4 7 2 2 - 2 72 Capacity Building - AJK - - - - - - - - - - - - 10 Indirect Partners Balochistan - - - - - - - - 1 - - 1 84 Support/Activity Location USAID Office Grand Total Total (After accounting for Sindh DG DOCS duplication) Education EG Energy FATA/KP Health OFM OIE Program Office Islamabad - - - - - 1 - - 4 - - 5 KP - - - - - - - - 3 - - 6 Punjab - - - - 1 - - - - - 1 Sindh 1 - - - - - - 1 - - 2 Capacity Building - Indirect Partners Total 1 1 1 3 - 9 - - 15 Validation Islamabad - 1 - - - - - - - - - 1 Punjab 2 - - - - - - 1 - - - - 1 Validation Total - 1 - - - - 1 - - - - 2 Research Islamabad - - - - - - - - 4 - - 4 4 Research Total - - - - - - - - 4 - - 4 Grand Total 5 49 1 2 2 5 13 3 15 10 3 108 93 85 TABLE 8: CURRENT STATUS OF PROJECT INTERVENTIONS BY TYPE OF ORGANIZATION Support Type Sindh DG DOCS Education EG Energy FATA/KP Health OFM OIE Program Office Grand Total Grand Total (After accounting for duplication) Pre-Award Assessments For Profit - - - - - - 1 - - 8 9 13 G2G - - - - - - 1 - - 2 1 4 Not for profit 1 - - - - - - 1 - - - 2 Total 1 - - - - - 2 1 - 10 1 15 Capacity Building - Direct Partners For Profit - - 1 - - - - - - - - 1 64 G2G 3 5 - 2 1 4 7 2 - - - 24 Not for profit - 43 - - - - - - 2 - 2 47 Total 3 48 1 2 1 4 7 2 - - - 72 Capacity Building - Indirect Partners G2G 1 - - - 1 1 3 9 - - 15 Not for profit - - - - - - - - - - - - 10 Total 1 1 1 3 10 - 2 15 Validation For Profit - - - - - - 1 - - - - 1 G2G - 1 - - - - - - - - - 1 2 Total - 1 - - - - 1 - - - - 2 Research G2G - - - - - - - - 4 - - 4 4 Total - - - - - - - - 4 - - 4 Grand Total 5 49 1 2 2 5 13 3 15 10 3 108 93 86 ANNEX II: BIOS OF TEAM MEMBERS Team Leader/Institutional Specialist/Government-to-Government (G2G) Specialist – Marc Shiman Marc Shiman is a project leader of institutional development projects for government and non￾governmental organizations worldwide and is an expert in capacity building. He has over 15 years of experience managing projects in the areas of public administration, economic growth, and democracy and governance. Mr. Shiman has demonstrated evidence of leadership of cross-cultural teams in 35 countries over 4 continents. Evaluation Specialist – Dr. Janet Robb Dr. Janet Robb is a highly talented, executive-level international development expert with visionary and critical thinking skills who has grown and managed significant business centers for both for-profit and not-for-profit professional service organizations. She exhibits advanced skills in senior-level development and strategic partnership building with organizational teams, government officials, host country implementers, and beneficiaries. One of her many areas of expertise includes program evaluation. Executive Education/Training Specialist – Mr. Ahmad Jameel Mr. Ahmad Jameel will primarily focus on the Lahore University of Management Sciences (LUMS) capacity building component. He has more than 40 years of experience, starting with public sector training institutions in Pakistan and the Ministry of Education. He has also undertaken consulting assignments with UNODC, UNICEF, and the German Aid Program (GIZ), involving capacity building and training. He also has almost 25 years of evaluation experience, including work for UNFPA and the World Bank/ILO. He has also worked as a consultant on four MEP evaluations. Mr. Jameel has a Master’s Degree in Administrative Science from the University of the Punjab and has studied public administration at the American University of Beirut, Lebanon. Management System Specialist – Vardah Malik Vardah Malik has over 17 years of experience, starting with the banking sector in Pakistan and later in the financial sector. She has worked as a financial management consultant for over 4 years with the World Bank and later joined the International Finance Corporation (investment banking arm of the World Bank Group), where she worked as an investment officer for over 6 years in the manufacturing, agribusiness, and services sectors. More recently, she has undertaken research and technical advisory assignments with the World Bank and DFID in the education sector in Pakistan. She has also worked as an economist for the design of the Punjab Skills Development Program funded by DFID. Institutional Development/Change Management Specialist/PFM Specialist – Haider Ali Daud Khan Haider Ali Daud brings 17 years of experience in financial management, institutional development, change management, capacity building, and reforms for both the public and private sectors. A chartered accountant by degree and a management consultant by experience, Haider has led an international development organization, besides advising on projects by all major international donors like USAID, UKAiD/DFID, World Bank. Asian Development Bank, European Commission, and JICA. He specializes in organizational change through organizational assessments/diagnostics, change management, and capacity building that leads up to sustainable solutions. 87 ANNEX III: DATA COLLECTION INSTRUMENTS Assessment and Strengthening Program (ASP) Interim Evaluation Qualitative Interview Protocol Implementing Partners Introduction: We work for Management Systems International (MSI), a USAID contractor that monitors and evaluates the work of various development projects in Pakistan. This interview is being conducted to gain further knowledge and insight into the strategies and interventions of the USAID-sponsored Assessment and Strengthening program (ASP) and the role you have played as an Implementing Partner. In particular, we are interested in learning about the effectiveness of the program approach with the aim of identifying opportunities for improvement. Of the five program components – 1) Pre-award Assessment, 2) Capacity Building, 3) Validation, 4) Research, and 5) Awardee Capacity Building – this evaluation will focus on components 1, 2, and 4. Interviews will be conducted in confidence and no individual response will be attributed to any one person being interviewed. As a knowledgeable person within your organization we value your input to our evaluation. It is important to note that this is not an evaluation of your organization, but an evaluation of the overall Assessment and Strengthening Program. Before the interview begins, I would like to ask your permission to take notes to capture your responses and to tape record the interview so that I can further enhance my notes once I am writing my report. Interview Questions: # Questions Prompts Overall Understanding 1. a. In your opinion, were the objectives of the ASP project clear and attainable?  USAID-funded project  Works with Pakistan’s government agencies and non-governmental organizations  Provides institutional development support  In the areas of: Finance, Procurement, HR, and M&E  To help meet standards of effective use of USAID resources If some were not, explain and give examples of how your organization responded. 88 b. What unique role did your organization have in carrying out the objectives? With regard to the three IPs operating under ASP. c. Please explain the specific strengths of your organization to perform your defined role in ASP. Please provide specific examples. Pre-Award Assessment 2. a. Please explain the process of accessing and utilizing pre-assessment award information for the development of Capacity Building Plans. Prior to ASP, pre-award assessments were contracted directly by USAID through a BPA with various accounting firms. b. In your opinion, please discuss the accuracy of the findings and recommendations in the pre-award assessments. Were they aligned with the assumptions stated in your Cooperative Agreement? Could you give me some examples? Did any focus on gender? c. How were these findings and recommendations communicated to the beneficiary organizations? d. Please provide some examples of how organizations received the information from the pre-award assessments? Did they agree with the findings? Or disagree? Provide examples. e. In your opinion, what factors contribute to a thorough and accurate pre-award assessment? f. Please discuss the advantages and disadvantages of having the pre-award assessment conducted by an outside auditing firm or by the ASP IPs. These might include: Willingness of the organization? Transparency in process? Inclusion of all levels of employee and both genders? Validation of findings through follow-on dialogue and on-site observations. Capacity Building Plans 3. a. Describe the approach your organization used in the development of Capacity Building Plans. b. To what extent did your approach to capacity building reflect the recommendations from the pre￾award assessment? We are interested in: • Did you play a facilitator or a leadership role? • What organization leadership and people were involved • Was involvement inclusive of men and women? • Where did final approval lie? • How long did the process take? 89 c. Please explain how the Capacity Building Plans were communicated within the organizations. d. What challenges did you face in the development and communication of the Capacity Building Plans? What strategies were used to gain acceptance and motivation for the implementation of the Capacity Building Plans? How were the specific tasks of the Plan communicated throughout the organization? Were special efforts made to ensure that female employees were also made aware of the Plan? Training 4. a. Describe the training your organization provided in support of the Capacity Building Plan. b. Describe the coordination between your organization and LUMS with regard to the training provided to the beneficiary organizations. Who were the trainers? How frequently did they meet with the participants? How was knowledge transfer facilitated throughout the organization to non￾participants? How did you ensure a good gender balance among participants Was any other training involved? Small groups? Mentorship? What steps were taken to ensure participation of women? c. In what ways did the LUMS experience complement the training provided by your organization? LUMS courses: Financial Management Procurement Management M&E HRM & Administration Leadership (including gender) Project Management d. How was a participant’s acquisition of knowledge assessed? Formal exams? On-the-job observation? Dialog with supervisors? Did participants receive remedial training if necessary? Were participants observed in practice? 90 e. Could you describe the way in which these trainings—both in-house and LUMS—contributed to the overall change efforts in the beneficiary organization? f. Can you suggest recommendations to improve the in￾house or LUMS trainings? Probe for specific examples. Manuals 5. a. Please describe the process of manual development. Where they developed by technical experts? Was a consultative team used in the development process? From where were the policies and procedures derived? USAID? Pakistan? What was the piloting and revision process? What was the official approval process? b. How were new manuals – policies and procedures – rolled-out through the organization? How were you able to ensure: Relevance? Adequacy? Practicality? Gender balance? c. How do you confirm that the new manuals were adopted and used within the organization? Progress reports, interviews with managers, etc. Changes 6. a. Please describe changes you observed in operations within the beneficiary organizations after the implementation of their Capacity Building Plan. Did accountability improve? Did procedures become more transparent? Was gender equity and sensitivity more apparent? Are changes now fully inculcated into the culture of your organization? Has there been resistance to the changes? b. What mechanism did you use to verify changes within the operation of the beneficiary organizations? Observation tool? Internal Change Agents? Interviews? c. What other factors beyond the ASP Capacity Building Plan may have contributed to these changes? Probe for specific examples 91 Sustainability 7. a. How did you build sustainability into the Capacity Building efforts? b. Without the further assistance of the ASP Program, what will likely happen going forward with the new systems, procedures, and manuals within the beneficiary organizations? What procedures introduced by ASP do you feel will be sustainable within the beneficiary organization? Do you see vast discrepancies among the beneficiary organizations? Differences between government and CSOs? What type? Which do you feel will not be sustainable? Sustainable means that policies and procedures will continue after the life of ASP. Also Inquire about the sustainability of conducting further assessments and developing ongoing capacity building plans. Challenges to Capacity Building 8. a. Identify the challenges you faced in the capacity building efforts. b. What further steps are required to ensure that beneficiary organizations meet the requirements to receive or continue to receive project funds from USAID and/or other outside donors? This could be those introduced anywhere throughout the entire process. Research 9. a. Please describe the intent and focus of the research component of ASP. This component should focus on best practices in the CSO sector and on changes in the government policies and procedures. b. Within your organization, what role has research played in assessing existing policies and procedures and in advising government and CSOs on best practices? In terms of efficiency, transparency, relevance, etc. c. With the research you have conducted, how did you inform government and CSOs of the research findings? How were the findings received? 92 d. In your opinion, what are some of the key areas for concern in Pakistan’s development capacity that could benefit from further research and understanding? e. What challenges does your organization face in implementing a research agenda? Areas of policies and procedures? Gender inclusion? Recommendations 10. a. If there were a “next phase” of ASP, what aspects of the current ASP would you keep and which would you change, and why? Probe for examples dealing with efficiency, transparency, accountability, motivation, and gender sensitivity. Seek recommendations that are specific and well defined on the design, the model, implementation, etc. “Thank you for your time and efforts to provide us with the information we need for our evaluation of the Assessment and Strengthening Program.” 93 Assessment and Strengthening Program (ASP) Interim Evaluation Qualitative Interview Protocol Heads of Organizations Introduction: We work for Management Systems International (MSI), a USAID contractor that monitors and evaluates the work of various development projects in Pakistan. This interview is being conducted to gain further knowledge and insight into the strategies and interventions of the USAID-sponsored Assessment and Strengthening program (ASP) as they relate to your organization. In particular, it is anticipated that your responses will help us to further understand the extent to which ASP services for capacity building – in the areas of financial management, procurement, administration and human resources management, monitoring and evaluation, and gender – were useful and are being implemented within your organization. With your permission, we will be interviewing approximately five individuals within your organization – yourself, Director/Manager of HR, Director/Manager Finance, Director/Manager Procurement, and Director/Manager M&E. Interviews will be conducted in confidence and no individual response will be attributed to any one person being interviewed. As a knowledgeable person within your organization we value your input to our evaluation. It is important to note that this is not an evaluation of your organization, but an evaluation of the overall Assessment and Strengthening Program. Before the interview begins, I would like to ask your permission to take notes to capture your responses and to tape record the interview so that I can further enhance my notes once I am writing my report. Interview Questions: # Questions Prompts 1. When did you begin working in this position? Know the date that ASP began working with the organization. Identify in your notes whether this was before or after the ASP activities were underway. Overall Understanding 2. a. What is your understanding of the objectives of the ASP project? b. What did you or do you expect the ASP project to do for your organization?  USAID-funded project  Works with Pakistan’s government agencies and non-governmental organizations  Provides institutional development support  In the areas of: Finance, Procurement, HR, and M&E  To help meet standards of effective use of USAID resources 94 Pre-Award Assessment 3.* a. To your knowledge, what were the findings of the pre-award assessment of your organization’s capacity? Prompts should be specific to the assessment findings and recommendations for the organization. Could you give me some examples? Did any focus on gender? b. In your opinion, please discuss the accuracy of the findings in the pre-award assessment. * Where no pre-award assessments were conducted, skip to question 4. If the response was “inaccurate” or “not useful,” determine whether they have another means of identifying gaps and probe further as to what those are. Did you find the findings and recommendations useful? Did they help define changes that needed to be made within your organization? Capacity Building Plan 4. a. Describe the process of development and implementing the Capacity Building Plan. We are interested in: • The leadership and people involved • The role of the IP • The need for approvals • Resources dedicated to the plan • How long it took For both development and implementation b. In what ways does the Capacity Building Plan address the needs of your organization? c. In your opinion, what needs do you feel should have been addressed but were not? Probe for examples in terms of policies and procedures and in terms of communicating those new policies and procedures. Was gender addressed in the plan? d. Describe some interventions that took place as part of the Capacity Building Plan and which ones you felt were the most important. Examples: Process mapping Job descriptions Forms redesigned IT 95 5. a. What were the priorities given inside your organization to implementing the Capacity Building Plan? In terms of time, resources, leadership, etc? Probe as to how high these priorities were relative to other activities. b. Describe the demands placed on your organization to implement the Capacity Building Plan? Employee’s time, leadership’s time, other resources expended Did you have sufficient absorptive capacity? Training 6. a. Describe the way employees were selected for training related to the Capacity Building Plan. By position? Availability? They were next in line for training? Other? Were considerations given to gender balance? What other factors may have been considered? b. Describe the type of trainings received by the employees in support of implementing the Capacity Building Plan. Was it both in-house and at LUMS? Was other training involved? c. What were the criteria for nomination of participants to LUMS training? By position? Availability? They were next in line for training? Other? Were considerations given to gender balance? What steps were taken to ensure participation of women? d. In what ways did the LUMS experience complement their in-house training? e. Could you describe the way in which these trainings—both in-house and LUMS—contributed to the overall change efforts in your organization? LUMS courses: Financial Management Procurement Management M&E HRM & Administration Leadership (including gender) Project Management f. Can you suggest recommendations to improve the in￾house or LUMS trainings? Probe for specific examples. 96 Manuals 7. a. What do you think of the manuals developed under ASP? Relevance? Adequacy? Practical? Gender balanced? b. How are the manuals used within your organization? Can you provide me with examples that support your impressions? c. Does the use of the manuals extend beyond the actual participants in the training? If yes, who else uses them? As a daily “how to” guide? As occasional reference? As a guide to train others? Everyone within a technical area? As reference for everyone in the organization? Only senior managers? Changes 8. a. Please describe changes in operations within your organization after the implementation of the Capacity Building Plan. Did accountability improve? Did procedures become more transparent? Was gender equity and sensitivity more apparent? b. What other factors beyond the ASP Capacity Building Plan may have contributed to these changes? Change-over in employees Individual employee? Motivation to see change happen? Changes within overall government policies? c. Please describe the extent to which your organization has adopted the new procedures and manuals. Are they now fully inculcated into the culture of your organization? Has there been resistance to the changes? Sustainability 9. a. Without the further assistance of the ASP Project, what will likely happen going forward with the new systems, procedures, and manuals? What procedures introduced by ASP do you feel will be sustainable within your organization? Do you think your organization will close 97 after current funding? Will it be absorbed within existing structures? b. What procedures introduced by ASP do you feel will be sustainable within your organization? Which do you feel will not be sustainable? Sustainable means that policies and procedures will continue after the life of ASP. c. What steps have you taken or will you take to ensure that the changes introduced are sustained? Also Inquire about the sustainability of conducting further assessments and developing ongoing capacity building plans. Challenges 10. a. Identify the challenges you saw in implementing the new policies and procedures introduced. b. What further steps are required to ensure that your organization meets the requirements to receive or continue to receive project/program funds from USAID and/or other outside donors? This could be either those introduced through training or through the manuals. Recommendations 11. a. In your own words, please describe ways in which ASP interventions have affected your organization. Probe for examples dealing with efficiency, transparency, accountability, motivation, and gender sensitivity. b. What changes would you recommend be made to the way ASP is implemented? Seek recommendations that are specific and well defined. “Thank you for your time and efforts to provide us with the information we need for our evaluation of the Assessment and Strengthening Program.” 98 Assessment and Strengthening Program (ASP) Interim Evaluation Qualitative Interview Protocol Technical Managers Introduction: We work for Management Systems International (MSI), a USAID contractor that monitors and evaluates the work of various development projects in Pakistan. This interview is being conducted to gain further knowledge and insight into the strategies and interventions of the USAID-sponsored Assessment and Strengthening program (ASP) as they relate to your organization. In particular, it is anticipated that your responses will help us to further understand the extent to which ASP services for capacity building, especially in your area of technical focus, were useful and are being implemented within your organization. We will be interviewing approximately five individuals within your organization. Interviews will be conducted in confidence and no individual response will be attributed to any one person being interviewed. As a knowledgeable person within your organization we value your input to our evaluation. It is important to note that this is not an evaluation of your organization or your performance, but an evaluation of the overall Assessment and Strengthening Program. Before the interview begins, I would like to ask your permission to take notes to capture your responses and to tape record the interview so that I can further enhance my notes once I am writing my report. Interview Questions: # Questions Prompts Overall Understanding 1. a. What is your understanding of the objectives of the ASP project? b. What did you or do you expect the ASP project to do for your organization and, in particular, [technical area]?  USAID-funded project  Works with Pakistan’s government agencies and non-governmental organizations  Provides institutional development support  In the areas of: Finance, Procurement, HR, and M&E  To help meet standards of effective use of USAID resources Finance Human Resources Mgmt Procurement M&E 99 Capacity Building Plan 2. a. Describe your understanding of how your organization’s Capacity Building Plan was developed. We are interested in:  Were there findings and recommendations from a pre-award assessment?  Was the process inclusive?  Was input sought from you or someone in your technical area? b. In your opinion, in what ways does the Capacity Building Plan address the needs of your organization and, in particular, [technical area]? In terms of policies and procedures and in terms of communicating those new policies and procedures. Also in terms of gender sensitivity and equity. c. In your opinion, what needs do you feel should have been addressed but were not? Probe for examples such as: job descriptions for levels below manager, a harassment policy, etc. 3. a. What were the priorities given inside [technical area] to implementing the Capacity Building Plan? In terms of time, resources, leadership, gender, etc. Probe as to how high these priorities were relative to other activities. b. Describe the demands placed on you and your team to implement the Capacity Building Plan? Employee’s time, leadership’s time, other resources expended Training 4. a. Describe the type of trainings provided in support of implementing the Capacity Building Plan. In-house? LUMS? Workshop? Mentoring? b. What was the participant selection process? By position? Availability? The next in line for training? Other? Were considerations given to gender balance? 100 c. Describe those positions that were selected to receive training through ASP in [technical area]. Senior managers? Program officers? Clerks? What trainings did each of these receive? d. In what ways did the LUMS experience complement their in-house training? e. Could you describe the way in which these trainings—both in-house and LUMS—contributed to the overall change efforts in your organization? LUMS courses: Financial Management Procurement Management M&E HRM & Administration Leadership (including gender) Project Management f. In what ways have the trainings contributed to how your technical team functions as a whole? g. Can you suggest recommendations to improve the in￾house or LUMS trainings? Probe for specific examples. Senior Management 5. a. From the activities in which you were personally involved, what were the key takeaways for you during the ASP process? Examples might be:  Additional steps in the procurement process  Greater gender sensitivity b. Explain which areas of your job could be enhanced by further learning. Examples might be:  Better communication and more transparency  Greater acceptance of women in the workplace Manuals 6. a. Please describe your impressions of the manuals developed under ASP for [technical area]. Relevance? Adequacy? Practical? Gender balanced? Useful? Easy to understand and use? Provide examples b. How are the manuals used within your department? As a daily “how to” guide? As occasional reference? 101 c. Does the use of the manuals extend beyond those who received direct training? If yes, who else uses them? If no, can you explain why? Everyone within a technical area? As reference for everyone in the organization? Only senior managers? Change 7. a. Describe the changes that have taken place after the ASP project in [technical area]. Provide specific examples b. Describe how the changes were communicated to others within [technical area] and throughout the organization. Through circulars, group meetings, casually by word of mouth, through the actual implementation, etc. c. Describe ways in which you do your job differently after the ASP process. Provide examples in terms of knowledge, efficiency, workload, etc. 8. a. In what ways has your job description evolved or changed during these capacity building efforts? b. Explain how you have been able to prepare for or otherwise accommodate the changes in your job description. c. Describe the type of career progression you see as an outcome of the ASP project. If possible, review job descriptions prior to the interviews so that follow￾on questions can be relevant to the individual. 9. a. Since the ASP capacity building efforts within your organization have taken place, in what ways do you feel you are able to demonstrate accountability and transparency in your own work? Accountability is the obligation of an individual or organization to account for its activities and accept responsibility for them. b. In what ways is your group’s importance within the organization felt among members of the group and felt by others? Transparency is operating in such a way that it is easy for others to see what actions are performed. 102 Sustainability 10. a. Can you describe how your colleagues have reacted to the new procedures and manuals? Have they been receptive? Eager to understand and implement? Angry that changes are being introduced? b. Without the further assistance of the ASP project, what will likely happen going forward with the new systems, procedures, and manuals? c. What procedures introduced by ASP do you feel will be sustainable within your organization? Which do you feel will not be sustainable? Sustainable means that policies and procedures will continue after the life of ASP. d. What has your organization done in terms of institutionalizing the ASP interventions? Human Resources and staff beyond the project Challenges 11. a. What challenges did you face in implementing the new policies and procedures? b. What steps could be taken to mitigate those challenges? This could be either those policies and/or procedures introduced through training or through the manuals. c. In your opinion, what overlaps might have emerged from the ASP project in policies between your department and others within the organization. Please elaborate on how these might be streamlined. Recommendations 12. a. In your own words, please describe ways in which ASP interventions have affected your technical area and your organization, as a whole. Probe for examples dealing with efficiency, transparency, accountability, motivation, and gender sensitivity. b. What changes would you recommend be made to the way ASP is implemented? Seek recommendations that are specific and well defined. “Thank you for your time and efforts to provide us with the information we need for our evaluation of the Assessment and Strengthening Program.” 103 Assessment and Strengthening Program (ASP) Interim Evaluation Qualitative Interview Protocol SGAFP Awardee Group Discussions Introduction: We work for Management Systems International (MSI), a USAID contractor that monitors and evaluates the work of various development projects in Pakistan. This discussion is being conducted to gain further knowledge and insight into the strategies and interventions of the USAID-sponsored Assessment and Strengthening program (ASP) as they relate to your organizations. In particular, it is anticipated that your responses will help us to further understand the extent to which ASP services for capacity building – in the areas of financial management, procurement, administration and human resources management, monitoring and evaluation, and gender – were useful and are being implemented within your organization. We will be interviewing a sampling of SGAFP Awardees in the geographic areas of Islamabad, Peshawar, Karachi, and Lahore. No individual response will be attributed to any one person being interviewed or to any one organization represented. As a knowledgeable person within your organization we value your input to our evaluation. It is important to note that this is not an evaluation of your organization, but an evaluation of the overall Assessment and Strengthening Program. Before the discussion begins, I would like to ask your permission to take notes to capture your responses and to tape record the discussion so that I can further enhance our notes once I am writing my report. Interview Questions: # Questions Prompts Overall Understanding 1. a. What is your understanding of the objectives of the ASP project? b. What did you or do you expect the ASP project to do for your organization?  USAID-funded project  Works with Pakistan’s government agencies and non-governmental organizations  Provides institutional development support  In the areas of: Finance, Procurement, HR, and M&E  To help meet standards of effective use of USAID resources 104 Pre-Award Assessment 2.* a. To your knowledge, what were the findings of the pre-award assessment of your organization’s capacity? Prompts should be specific to the assessment findings and recommendations for the organization. Could you give me some examples? Did any focus on gender? b. In your opinion, please discuss the accuracy of the findings in the pre-award assessment. * Where no pre-award assessments were conducted, skip to question 3. If the response was “inaccurate” or “not useful,” determine whether they have another means of identifying gaps and probe further as to what those are. Did you find the findings and recommendations useful? Did they help define changes that needed to be made within your organization? Capacity Building Plan 3. a. Describe your understanding of the process of implementing the Capacity Building Plan. We are interested in: • The leadership and people involved • The role of the IP • The need for approvals • Resources dedicated to the plan • How long it took b. In what ways does the Capacity Building Plan address the needs of your organization? c. In your opinion, what needs do you feel should have been addressed but were not? Probe for examples in terms of policies and procedures and in terms of communicating those new policies and procedures. Was gender addressed in the plan? 4. a. What were the priorities given inside your organization to implementing the Capacity Building Plan in terms of time, resources, leadership, etc.? Probe as to how high these priorities were relative to other activities. 105 b. Describe the demands placed on your organization to implement the Capacity Building Plan? Employee’s time, leadership’s time, other resources expended Training 5. a. Describe the way employees were selected for capacity building training. By position? Availability? They were next in line for training? Other? Were considerations given to gender balance? What other factors may have been considered? b. Describe the type of trainings received by the employees in support of implementing the Capacity Building Plan. Was it both in-house and at LUMS? Was other training involved? c. What were the criteria for nomination of participants to LUMS training? By position? Availability? They were next in line for training? Other? Were considerations given to gender balance? What steps were taken to ensure participation of women? d. In what ways did the LUMS experience complement their ASP training? e. Could you describe the way in which these trainings—both in-house and LUMS—contributed to the overall change efforts in your organization? f. Can you suggest recommendations to improve the in￾house or LUMS trainings? LUMS courses: Financial Management Procurement Management M&E HRM & Administration Leadership (including gender) Project Management Probe for specific examples. Manuals 6. a. Please describe your impressions of the manuals developed under ASP. Relevance? Adequacy? Practical? Gender balanced? Can you provide me with examples that support your impressions? 106 b. How are the manuals used within your organization? As a daily “how to” guide? As occasional reference? As a guide to train others? c. Does the use of the manuals extend beyond the actual participants in the training? If yes, who else uses them? Everyone within a technical area? As reference for everyone in the organization? Only senior managers? Changes 7. a. Please describe changes in operations within your organization after the implementation of the Capacity Building Plan. Did accountability improve? Did procedures become more transparent? Was gender equity and sensitivity more apparent? b. What other factors beyond the ASP Capacity Building Plan may have contributed to these changes? Efforts by other donors? Hiring of new staff? c. Please describe the extent to which your organization has adopted the new procedures and manuals. Are they now fully inculcated into the culture of your organization? Has there been resistance to the changes? Sustainability 8. a. Without the further assistance of the ASP Program, what will likely happen going forward with the new systems, procedures, and manuals? What procedures introduced by ASP do you feel will be sustainable within your organization? Do you think your organization will close after current funding? Will it be absorbed within existing structures? Sustainable means that policies and procedures will continue after the life of ASP. 107 b. What procedures introduced by ASP do you feel will be sustainable within your organization? Also Inquire about the sustainability of conducting further assessments and developing ongoing capacity building plans. Challenges 9. a. Identify the challenges you saw/see in implementing the new policies and procedures introduced. b. What further steps are required to ensure that your organization meets the requirements to receive or continue to receive project/program funds from USAID and/or other outside donors? This could be either those introduced through training or through the manuals. Recommendations 10. a. In your own words, please describe ways in which ASP interventions have affected your organization. Probe for examples dealing with efficiency, transparency, accountability, motivation, and gender sensitivity. b. What changes would you recommend be made to the way ASP is implemented? Seek recommendations that are specific and well defined. “Thank you for your time and efforts to provide us with the information we need for our evaluation of the Assessment and Strengthening Program.” 108 LUMS Training Survey ASP-LUMS Training Survey Guidelines for surveying the trainees Before the survey communicate the following: “We work for Management Systems International (MSI), a USAID contractor that monitors and evaluates the work of various development projects in Pakistan. Currently MSI is assessing the USAID’s Assessment and Strengthening Program (ASP). Under ASP, Lahore University of Management Sciences (LUMS) offered training for professionals from public, private and non-profit sectors. In order to help them better understand the effectiveness of the ASP-LUMS trainings, we are surveying all the trainees who attended ASP-LUMS trainings. You have been chosen to participate in this survey because as a participant and beneficiary of the training program we value your input to our assessment. The survey will take less than 15 minutes to complete. Your answers will be strictly confidential and kept by MSI for data analysis purposes only. No respondents will be identified by name in the report; therefore your input will also be strictly anonymous. Thank you in advance for your assistance with this important survey.” _______________________ 1. Respondent’s gender a.  Female b.  Male 2. Age ____________________ (write age in years) 3. Job scale/rank (mark only one) c. Entry level d. Mid-level e. Senior level 4. Sector a. Government b. Private c. Non-profit/NGO d. AiD/RSPN 5. Current professional field (mark all that apply) a. Finance/Financial Management b. Procurement Management c. HR Management & Administration d. Monitoring and Evaluation e.  Other (please specify) _____________________ 109 6. Which capacity/professional development training courses did you attend that were offered by LUMS under the USAID’s Assessment Strengthening Program (ASP)? (Mark all that apply) Year a.  Monitoring & Evaluation b.  Financial Management c.  Procurement Management d.  HR Management & Administration e.  Project Management f.  Leadership: Gender Dimension g.  Forensic Audit h.  Fiscal Decentralization i.  Procurement Audit j.  Training of Trainer (Questions 7-9 have sample learning objectives from Monitoring and Evaluation training.) The following three questions focus on the core learning objectives of the course you took at LUMS. Please circle the number that best describes your impression. 7. Learning Objective 1: Developing Performance Indicators i. To what extent is this topic relevant to your work? Not Relevant Somewhat Relevant Relevant Very Relevant 0 1 2 3 ii. How familiar were you with this topic before you took this course? No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 iii. How familiar were you with this topic following this course? No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 110 8. Learning Objective 2: Results Framework i. To what extent is this topic relevant to your work? Not Relevant Somewhat Relevant Relevant Very Relevant 0 1 2 3 ii. How familiar were you with this topic before you took this course? No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 iii. How familiar were you with this topic following this course? No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 9. Learning Objective 3:Evaluating Development Programs i. To what extent is this topic relevant to your work? Not Relevant Somewhat Relevant Relevant Very Relevant 0 1 2 3 ii. How familiar were you with this topic before you took this course? (Circle the number the best describes your impression) No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 iii. How familiar were you with this topic following this course? No knowledge Not enough to do my job Enough knowledge to do my job More than enough to do my job 0 1 2 3 111 10. How useful has the (insert manual name i.e., course title) manual been to your work at the office? a. Not useful at all b. Not useful c.  Not sure d. Useful e. Very useful f. I didn’t receive a manual 11. How often do you consult the manual? a. Never b. Rarely c.  Sometimes d. Frequently e. Almost daily Please look at the following statements and rate them by circling the appropriate number that best describes your impression. As a result of this training… Strongly Disagree Disagree Neither agree or disagree Agree Strongly Agree 12. I changed the way that I did my work 1 2 3 4 5 13. This course was instrumental in making changes to my work 1 2 3 4 5 14. This course was instrumental in making changes to my organization’s work 1 2 3 4 5 15. This course was instrumental in making changes to my organization’s work 1 2 3 4 5 (Note: For each course selected in question 6 the participant will answer the set of questions 7-17). 112 USAID Assessment and Strengthening Program (ASP) Interim Evaluation Protocol for Interview with COP/ASP, LUMS Introduction: I am a consultant working for the Management Systems International (MSI), an organization that has been contracted to undertake the monitoring and evaluation of various USAID funded development projects in Pakistan. MSI is conducting an interim evaluation of the USAID’s Assessment and Strengthening Program (ASP). This interview is being conducted to gain further knowledge and insight into the various capacity building initiatives implemented by LUMS under ASP. As the leader of LUMS Team, your input to our assessment would be very valuable. Your answers to a series of questions will enhance our understanding and analysis. Could you allow recording our conversation to support our notes? Your answers will be strictly confidential and used by MSI for data analysis purposes only. No informants will be identified by name in the report therefore your input will be kept strictly anonymous. Overall perspective Questions Prompts 1. What is your understanding of the overall objective for Assessment and Strengthening Program (ASP) and how LUMS is contributing to achieving this objective?  To what extent and in what ways, LUMS is contributing to making the Pakistani organizations transparent, accountable and effective? (ASP Objective - Page 7 SOW) Training Design & Feedback 2. In what ways have other ASP implementing partners, RSPN and AiD contributed to the process of designing the training courses?  Was a formal needs assessment conducted before designing training courses?  Did IPs formally participate in training needs assessment and how? 3. How do you incorporate feedback from other IPs in the training courses?  How frequent is the feedback from IPs?  What is the process of receiving feedback and incorporating that in the training courses? 4. To what extent have the training courses been modified in the light of feedback from IPs?  How and to what extent there is flexibility to modify the training course contents by the instructor should a majority of the participants find the training courses not very relevant? 5. How do you measure the changes in knowledge and skills that come from participation in the training courses?  Is there a pre and post-test? If not why not?  Any other method used? 6. How and to what extent do you think LUMS training courses are helpful in upgrading job￾related skill sets?  Were they more academic in nature  What is the share of theory and practice in the training courses?  How can the training course be made more relevant to the job requirements?  What proportion of training materials (manuals/case studies/exercises/handouts) is based on local experience?  The importance of meeting the standards/requirements of USAID/donor assistance 113 7. Since IPs have their own training programs, to what extent do the LUMS training complement other ASP training programs?  What is the value-added by LUMS training courses?  How would you argue that the LUMS training course is absolutely essential for the participants to prepare them for efficiently performing their duties? Participant Selection Process 8. What are your views about the participant nomination/selection process?  Effectiveness  Should LUMS have a say in accepting the participants based on an agreed criteria. 9. Can you describe the differences in attitudes and receptivity between the participants in Tiers 1 and 2?  Participants active, participating, retaining knowledge 10. What differences have you observed in the interest taken in the training courses by the participants from the public sector and the CSOs?  In which areas the interests of the public sector and CSOs overlap/differ? Training Follow up 11. How does LUMS follow up on participants and to what extent is the current system of follow up contributing to their continued professional growth?  How feedback from participant annual conference is used in revising the training course contents? Sustainability 12. Please discuss how do you anticipate ensuring the future of ASP funded LUMS training beyond USAID funding?  What’s the sustainability plan of LUMS training initiatives?  Role of setting up of the Centre for Governance & Public Management (CGPM).  Challenges in ensuring sustainability? Gender 13. Please describe how attention to gender aspects is incorporated in LUMS training courses?  What has LUMS done or can do to: a. Increase the number of females in its training courses b. Incorporate appreciation of gender aspects or concerns in its training courses. Recommendations 14. What changes would you recommend be made to the way LUMS is contributing to the achievement of the objective of ASP?  How can you enhance the value added by LUMS training courses in achieving the ASP objective of making Pakistani organizations more transparent, accountable and effective? Thank you for your time. 114 USAID Assessment and Strengthening Program (ASP) Interim Evaluation Protocol for Interview with LUMS FACULTY Introduction: I am a consultant working for the Management Systems International (MSI), an organization that has been contracted to undertake the monitoring and evaluation of various USAID funded development projects in Pakistan. MSI is conducting an interim evaluation of the USAID’s Assessment and Strengthening Program (ASP). This interview is being conducted to gain further knowledge and insight into the various capacity building initiatives implemented by LUMS under ASP. As a key member of the LUMS Team, your input to our assessment will be very valuable. Your answers to a series of questions will enhance our understanding and analysis. Could you allow recording our conversation to support our notes? Your answers will be strictly confidential and used by MSI for data analysis purposes only. No informants will be identified by name in the report therefore your input will be kept strictly anonymous. Overall perspective Questions Prompts 15. What is your understanding of the overall objective for Assessment and Strengthening Program (ASP) and how the training course you conduct is contributing to achieving this objective?  To what extent and in what ways, the training course you conduct is contributing to making the Pakistani organizations transparent, accountable and effective? (ASP Objective - Page 7 SOW) Training Design & Feedback 16. In what ways have other ASP implementing partners, RSPN and AiD contributed to the process of designing the training courses?  Was a formal needs assessment conducted before designing training courses?  Did IPs formally participate in training needs assessment and how? 17. How do you incorporate feedback from other IPs in the training courses?  How frequent is the feedback from IPs?  What is the process of receiving feedback and incorporating that in the training courses? 18. To what extent have the training courses been modified in the light of feedback from IPs?  How and to what extent there is flexibility to modify the training course contents by the instructor should a majority of the participants find the training courses not very relevant? 19. How and to what extent do you think LUMS training courses are helpful in upgrading the job related skill sets?  Were they more academic in nature  What is the share of theory and practice in the training courses?  What proportion of training materials (manuals/case studies/exercises/handouts) is based on local experience?  The importance of meeting the standards/requirements of USAID/donor assistance 20. Since IPs have their own training programs, to what extent the LUMS training courses complement other ASP training programs?  What is the value-added by the training course you conduct?  How would you argue that this training course is absolutely essential for the participants to prepare them for efficiently performing their duties? 115 Participant Selection Process 21. What are your views about the participant nomination/selection process?  Effectiveness  Should LUMS have a say in accepting the participants based on an agreed criteria. 22. How do you measure changes in knowledge and skills that come from participation in the training course?  Is there a pre and post-test? If not why not?  Any other method used? 23. How and to what extent do you think LUMS training courses are helpful in upgrading job related skill sets?  Were they more academic in nature  What is the share of theory and practice in the training courses?  How can the training course be made more relevant to the job requirements?  What proportion of training materials (manuals/case studies/exercises/handouts) is based on local experience?  The importance of meeting the standards/requirements of USAID/donor assistance 24. Can you describe the differences in attitudes and receptivity between the participants in Tiers 1 and 2?  Participants active, participating, retaining knowledge  Are participants in each tier heterogeneous with similar knowledge and skill levels? 25. What differences have you observed in the interest taken in the training courses by the participants from the public sector and the CSOs?  In which areas the interests of the public sector and CSOs overlap/differ? Training Follow up 26. How does LUMS follow up on participants and to what extent is the current system of follow up contributing to their continued professional growth?  How feedback from participant annual conference is used in revising the training course contents? Gender 27. Please describe how attention to gender aspects is incorporated in LUMS training courses?  What has LUMS done or can do to incorporate appreciation of gender aspects or concerns in its training courses. Recommendations 28. What changes would you recommend be made to the training course you conduct? Thank you for your time. 116 ANNEX IV: MANUAL REVIEW CHECKLISTS Assessment and Strengthening Program (ASP) Interim Evaluation Manual Checklist Financial and Accounting Systems Manual Introduction This manual puts in place the key accounting, finance and governance policies and procedures designed to protect and safeguard the organization as well as its assets and to ensure the maintenance of accurate records of financial/business activities in order to comply with all necessary statutory requirements. In the table below, in the left-hand column check all that apply and use the right-hand column for notes. 1. General Requirements The policies and procedures developed for organizations take account of the size, scale and type of organization. In the event that the organization is bound to follow other guidelines (government regulations, public procurement) these are incorporated /referred to in the policies and procedures. The procedures take account of the size of the enterprise and pay due regard to the nature of its operations. The overall objective of policies and procedures is to safeguard the assets of the organization and manage risk. The organization is expected to operate in a transparent manner and follow good governance models. 1A General Review Do the policy and procedure manuals (manuals) have a version control? Have the policies and procedures been approved by the Governing body? Do the Manuals contain Delegation of Authority? Do the Manuals contain an Authority Matrix specifying, authority limits for each level? Are transactions requiring approval of the Governing Body clearly shown? Do the manuals refer to applicable laws/regulations that apply to the entity? 117 1B. Responsibilities – Does the manual outline responsibilities of the Board, CEO and other Senior Members? Examples: The Board of Directors is responsible for the overall stewardship of the Organization’s finances. The Board delegates the day-to-day financial management to the Senior Management but continues to review the operations and activities of the organization on a regular basis. The Audit Committee appointed by the Board plays an active role in Audit and other matters related to Internal Control. The CEO has the overall responsibility for all operations and activities, including financial management. The CFO is responsible to the CEO for all financial matters. The Department Heads are responsible for all aspects of their departments including the management of their allocated budgets. 2. General Policies 2A. Overall Policies of the Organization Including Adequate segregation of duties The work of the Accounting Team including the bank reconciliations and the preparation of the monthly accounts to be reviewed by the CFO on a regular basis. Independent Reviews that are carried out by the CFO, CEO and the Department Heads on all payments, prior to authorization. The chart of accounts and its structure to maintain separate accounts for the Departments on a departmental and project basis. The approval process for changes to the chart of accounts Passwords protocols for the accounting team. 2B. Compliance with External Policies - Consistency with: Generally Accepted Accounting Principles (GAAP). International Accounting Standards and International Financial Reporting Standards. The relevant laws and regulations governing the entity. 3. Checklist 3A. Overview of the Accounting System Includes: Data Backup Funds Received Funds Disbursed Payroll System Investment of Funds 118 Business Planning and Budget Process 3B. Detailed Accounting Procedures Include: Funds Received: Receipts Petty Cash Fund Disbursements: Approvals and Authorization Process Cheque Writing Procedures Cheque Approval Process Internet Banking and Transfers Withholding Taxes Tax Reporting Payroll Preparation and Payment Procurements Bank Accounts Accounts Receivable Departmental Allocation of Expenses: Cost Centers Fixed Assets Depreciation Audits: Financial Audit Process Internal Audit Process Budgets: Planning & Budgeting Policy End of Month Accounting Procedures End of Year Accounting Procedures Record Retention Investment Policies and procedures Matrix of Authority Cash and Liquidity Management Bank Account 3C Exhibits Detailed process maps Standard forms Precedent documents 119 Assessment and Strengthening Program (ASP) Interim Evaluation Manual Checklist Procurement Systems Manual Introduction Procurement is a complex as well as a high-risk area for most organizations. It is imperative that the policies are well thought out and take account of the nature of the operations of the organization. Most organizations tend to treat procurement as “chore”, however a well-managed and proactive procurement function can add significant value through cost effective procurement and more efficient operations. Procurement function needs to develop an annual procurement plan for all known projects, thus minimizing last minute pressure and allowing them to better plan their workload and ensure that no delays occur in the operations. The procurement methods (direct purchase, tenders, single source) along with authority limits should be clearly spelt out including the thresholds where the approval of the Board is required. In the table below, in the left-hand column check all that apply and use the right-hand column for notes. Gender is a factor that should be considered throughout the procurement process. [Gender] has been placed throughout this checklist to remind the reviewer that special attention should be given to ensure gender issues have been addressed, at a minimum, in these key locations. 1. General Requirements 1A. Manual Overview The organizational structure of the procurement department [Gender] The policies and procedures developed for the procurement department take account of the size, scale and type of organization.[Gender] External rules governing the procurement process are referenced The procedures pay due regard to the nature of its operations. The overall objective of policies and procedures is to safeguard the assets of the organization and manage risk. The organization is expected to operate in a transparent manner and follow good governance models. 1B General Review Do the policy and procedure manuals (manuals) have a version control? 120 Have the policies and procedures been approved by the Governing body? Do the manuals refer to applicable laws/regulations that apply to the entity? 2. Procurement Specific 2A. Procurement Policies and Processes: Procurement Process Model/Options: Direct Sourcing Thresholds and requirements Single Source Procurement thresholds and justification Procurement General Policies and guidelines: [Gender] Conflicts of Interest Ethical Behavior Delegation of Authority Authority Approval Process and Thresholds: Board Approval Thresholds CEO Thresholds Department Head Thresholds Procurement Planning Procurement Sourcing Plan Policies Process Flow / Processes 2B. Vendor Management Nomination and Selection of Vendors – Policies, Procedures and Processes [Gender] Vendor Administration and Maintenance - Policies, Procedures and Processes Vendor Performance Evaluation– Policies, Procedures and Processes [Gender] Selection of preferred vendors/approved vendors [Gender] 2C. Procurement Operations Manage – Purchase Requisition (PR) - Policies, Procedures and Processes Approve PR and Check Budget Authority Limits/Thresholds 2D. Non-Tender Procurement Manage Purchase Order (PO) - Policies, Procedures and Processes Manage PO (Urgent Purchases / Single Sole Source Procurement) Manage Variation Orders in PO Receive Materials and Services Manage Reverse Logistics Including Product Returns and Exchanges 2E. Product and Services Tendering Establishing a Tender Committee [Gender] Terms of Reference for Tender Committee [Gender] 121 Perform Product and Services Tendering￾Policies, Procedures and Processes [Gender] Log Bids Receipt and Perform Bids Tech. Evaluation – Policies, Procedures and Processes [Gender] Perform Bids Commercial Evaluation – Policies, Procedures and Processes [Gender] Selection of preferred bidder and notification process [Gender] Manage and Resolve Bid Disputes [Gender] 2F. Other Manage Variation Orders in Contracts 3. Standard Forms Annual Procurement Plan Supplier Evaluation Form Purchase Requisition PR RFQ (Request for Quotation) RFQ Bid Responses Evaluation Document Purchase Order (PO) Single Source / Urgent Requisition Justification and Approval Form Materials Inspection Report/ Materials Return Voucher Bids Technical Evaluation Form/ Bids Commercial Evaluation Form Tender Evaluation Summary Form 122 Assessment and Strengthening Program (ASP) Interim Evaluation Manual Checklist Human Resources Manual Introduction The HR manual is a critical document for most organizations as it covers a wide range of areas relating to employees and how they are expected to operate within the workplace. HR Policies are expected to cover everything from recruitment, training, remuneration, leave and termination. In the table below, in the left-hand column check all that apply and use the right-hand column for notes. Gender is a factor that should be considered throughout the HR manual. [Gender] has been placed throughout this checklist to remind the reviewer that special attention should be given to ensure gender issues have been addressed, at a minimum, in these key locations. 1. General Requirements: The policies and procedures developed for organizations take account of the size, scale and type of organization. [Gender] In the event that the organization is bound to follow other guidelines (government regulations, public procurement) these are incorporated /referred to in the policies and procedures. The procedures take account of the size of the enterprise and pay due regard to the nature of its operations. The overall objective of policies and procedures is to safeguard the assets of the organization and manage risk. The organization is expected to operate in a transparent manner and follow good governance models. [Gender] 1A General Review Do the policy and procedure manuals (manuals) have a version control? Have the policies and procedures been approved by the Governing body? Do the Manuals contain Delegation of Authority? Do the Manuals contain an Authority Matrix specifying, authority limits for each level? Are transactions requiring approval of the Governing Body clearly shown? Do the manuals refer to applicable laws/regulations that apply to the entity? [Gender] 123 2. Specific HR Checklist 2A. Background The Operating Structure of Human Resources Department [Gender] Access to the Manual [Gender] Amendments to the Manual 2B. HR Policies General Working Conditions [Gender] Attendance Policy – Working Hours Manpower Planning/Recruitment of additional staff [Gender] Organizational Development Policies Recruitment And Selection Policies [Gender] Compensation and Benefits: [Gender] Grading structure Compensation & Benefits Business Travel Benefits Policy Bonuses Leave Policy: [Gender] Public Holiday Policy Unauthorized Absence Policy Training Policy [Gender] Discipline and Termination of Employment [Gender] Performance Appraisal Policy [Gender] Confidentiality and non-compete Governing Body’s involvement in senior recruitment and termination [Gender] Policies on Gender [Gender] Hiring Special needs (maternity, etc) Protections 2C. Processes and Procedures for Key Areas [Gender] Disciplinary Process Performance Appraisal Employee Grievance Recruitment Process Probation Period Leave Application Termination of Employment Resignation 124 Assessment and Strengthening Program (ASP) Interim Evaluation Manual Checklist Monitoring & Evaluation Manual Introduction A well-functioning M&E system is a critical part of good project/program management and accountability. A robust M&E system should be designed in such a way that resources are dedicated towards collection of raw data, which can be processed into information and evidence that guides policy action, which in turn generates an impact. Timely and reliable M&E provides information to support project/program implementation with accurate, evidence based reporting that informs management and decision-making to guide and improve project/program performance. The M&E manual should provide concise guidance for project planning, implementation, and evaluation including key components of an M&E system that trace a logical train of thought from hypotheses on how the project will bring about change in a specific sector, to the specific objectives needed for these changes, methods for measuring the project’s achievement of its stated objectives, and protocols for collecting and analyzing data and information used in the measurement. An effective M&E system should meet internal and external reporting requirements, and inform future programming. In the table below, in the left-hand column check all that apply and use the right-hand column for notes. Gender is a factor that should be considered throughout the M&E manual. [Gender] has been placed throughout this checklist to remind the reviewer that special attention should be given to ensure gender issues have been addressed, at a minimum, in these key locations. 1. General Requirements: The policies and procedures developed for the organization take account of the size, scale and type of organization. [Gender] The overall objective of the Manual is stated clearly and the context in which the project/program is being implemented. 1A General Review Do the policy and procedure manuals (manuals) have a version control? Have the policies and procedures been approved by the Governing body? Do the Manuals contain Delegation of Authority? Do the manuals refer to applicable laws/regulations that apply to the entity? [Gender] 125 How effectively does the M&E system address the results chain of the project? [Gender] Does the manual identify roles and responsibilities for data collection and analysis, collection of baseline data, establishment of targets and mechanisms for revising/refining proposed indicators once implementation begins? [Gender] Does the manual recognize that baseline or other relevant data may not be available and clear targets may not be established from the outset and how are these issues addressed in the manual? [Gender] How does the manual link to other key management documents? [Gender] Is the M&E system flexible to incorporate i) changes in project design and priorities of key stakeholders; ii) lessons learned through implementation; and, iii) changes in the environment in which the project is being implemented. [Gender] Identifies key stakeholder informational needs and expectations. [Gender] 2. Specific M&E Checklist 2A. Background Purpose and scope of the M&E system Resources for adequate and regular monitoring. [Gender] Access to the Manual [Gender] Amendments to the Manual 2B. M&E Policies Data collection, information and frequency. Are the following defined:  Results  Outcomes  Outputs  Inputs M&E standards and ethics [Gender] Attention to gender and vulnerable groups Results Framework 2C. M&E Tools Data collection Analytical frameworks used to convert the data into evidence and knowledge Performance indicators 2D. Data Collection Methods Baseline Regular Data Collection Secondary sources Plan for data collection and management Quantitative and qualitative data Triangulates data collection sources and 126 methods Sampling requirements Data Quality Standards 2E. Data Analysis Purpose Frequency Responsibility Process 2F. Evaluation Impact Effectiveness Efficiency Sustainability Relevance Types of Evaluation undertaken 2G. Resources M&E teams in the field (at the sub-national level) [Gender] M&E focal points (central location) [Gender] Itemized M&E budget 2H. Templates Needs/audience [Gender] Frequency Formats People responsible [Gender] 127 USAID Assessment and Strengthening Program (ASP) Interim Evaluation LUMS TRAINING MATERIALS QUALITY CHECKLIST Date reviewed: Title of the Training Material: ______________________________________________________________________________ _____________________________________________________________________ No. Indicator Observations 1. Developed on the basis of a needs assessment? 2. Reviewed for appropriateness of technical contents by _____ 3. Separate trainer manual and trainee manual or a combined manual 4. Pretest – Posttest (or other knowledge/skills assessment tool included) 5. Share of theoretical contents 6. Share of materials focusing on practice (exercises, case studies, role play, others) 7. Number of locally developed exercises 8. Number of exercises from foreign sources 9. Number of locally developed case studies 10. Number of case studies from foreign sources 11. Coverage of subject matter100 12. Can the participants use the training material/manual for reviewing the subject after training? 13. Was the training material/manual provided in the beginning of the course or at the end? 14. Can/does it serve as a reference document in the workplace? 100 There will be a separate sheet to review the contents based on the checklists to review the procurement, HR, financial management and M&E Manuals 128 15. To what extent the concerns with gender equality reflected in the curricula of the training course and the manual? (Check for all manuals, but specifically of training courses on communication strategy and management, leadership and cultural exchange, monitoring and evaluation, project management and training of trainers. 16. Is the manual easy to handle and durable 129 ANNEX V: SELECTED WEBSITES ON UPDATED GOVERNMENT OF PAKISTAN POLICIES AND PROCEDURES 1. Ministry of Planning, Development & Reforms (Planning Commission)  Rules of Business and PC forms http://www.pc.gov.pk/?page_id=163 2. (Federal) Public Procurement Regulatory Authority (FPPRA)  Standard Bidding Documents, Forms, PP Rules and Regulations, Policy Guidelines http://www.ppra.org.pk/ 3. Cabinet Secretariat, Establishment Division  Esta Code http://www.establishment.gov.pk/gop/index.php?q=aHR0cDovLzE5Mi4xNjguNzAuMTM2L2VzdGFiL3B 1YkRldGFpbHMuYXNweA%3D%3D  Policies and Service Rules http://www.establishment.gov.pk/gop/index.php?q=aHR0cDovLzE5Mi4xNjguNzAuMTM2L2VzdGFiL3B vbGljaWVzRGV0YWlscy5hc3B4 4. Ministry of Finance  Regulations http://www.finance.gov.pk/circulars.html 5. Accountant General Pakistan Revenues (AGPR)  Accounting Policies and Procedures http://www.agpr.gov.pk/APPM.pdf  Chart of Accounts http://www.agpr.gov.pk/download.html 130 ANNEX VI: OFFICE MEMORANDUM FOR REVISED ACCOUNTING PROCEDURE FOR REVOLVING FUND ACCOUNTS 131 132 ANNEX VII: DISCLOSURE OF ANY CONFLICTS OF INTEREST 133 Name Janet Robb Title Evaluation Specialist Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Assessment and Strengthening Program (ASP) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date September 11, 2014  x      X X 134 135 136 137 ANNEX VIII: CASE STUDIES Case Study on Capacity Building of Internal Audit Function, Government of Khyber Pakhtunkhwa (GOKP) Evaluation Team: Dr. Janet Robb Mr. Haider Ali Daud Khan Interviewees: Mr. Sahibzada Sahib, Executive Secretary Finance Mr. Naib Khan, Director Local Audit Fund Javed Iqbal, Deputy Chief of Party, ASP-RSPN Nazar Rathore, Diretor Capacity Building, ASP-RSPN Farhan Ghafoor, Advisor International Development, ASP-RSPN Waqar Ali Shah, Manager Internal Audit, ASP-RSPN 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and human resources management (HRM), monitoring and evaluation (M&E), and gender useful and implemented (effective and institutionalized)? 2. Context The Government of Khyber Pakhtunkhwa (GOKP) took the initiative and the Provincial Cabinet decided to establish an Internal Audit Function across the entire Provincial Government departments and offices. The Provincial Cabinet took the following steps:  Approved the establishment of an Internal Audit Function;  Approved an Internal Audit Charter;  Prioritized some provincial departments for the establishment of an Internal Audit Function on a pilot basis; and  Availed technical assistance from ASP-RSPN to propose a governance structure for internal audit. The Internal Audit Charter approved by the Provincial Cabinet on June 16, 2012, defines the purpose, responsibility, and authority of the Internal Audit Function. The charter establishes the position of the Internal Audit Function, authorizes access to relevant engagement performances, defines the role/scope of activities, and provides the basis for its evaluation. Placement of Internal Audit within Organization The Chief Internal Auditor shall be functionally and administratively responsible/report to the Administrative Secretary or to the Head of the entity concerned. The Internal Audit Function shall perform its duties independently and objectively, and there shall be no interference with the mandate or performance of audit duties or other assignments. The number of staff members shall range between two to five depending upon the size of the entity and audit requirements. The Chief Internal Auditor shall hold a BPS-18101 position in big departments and BPS-17 or equivalent in comparatively smaller departments, organizations, autonomous bodies, corporations, institutions, etc. 101 BPS, or Basic Pay Scale, refers to the civil servant salary structure of the Government of Pakistan 138 Objectives The Internal Audit Function shall determine whether the respective department’s governance processes, internal controls, and risk management systems, as designed and represented by the management, are adequate and functioning in a manner that ensures the following:  The effective application of risk management procedures and methodologies (including risk identification and management) and the functioning of the internal control system.  Significant financial, managerial, and operating information is accurate, reliable, and timely.  Employees’ actions are in compliance with the internal control policies, standards, and procedures, and applicable laws and regulations.  The department’s resources and assets are economically acquired, efficiently used, properly accounted for, and adequately safeguarded.  Programs, plans, and objectives are achieved.  Quality and continuous improvements are fostered in the department’s control processes.  Significant legislative or regulatory issues impacting the department are recognized and appropriately addressed.  The department’s policies, procedures, and manuals in relation to the above are adequate and complied with. 3. Initial Assessment Deloitte carried out a pre-award assessment of the GOKP in March 2010 in which internal control was identified as an area with high risk and low confidence levels. The report emphasized the need for effective internal audit and reporting lines. As an Internal Audit was a new function within the GOKP, ASP-RSPN was asked to carry out technical assessment and governance structure reports. These were delivered in March 2013 and laid out the foundations to establish an Internal Audit Function across the identified departments/organizations within the GOKP. 4. ASP Strategies and Interventions Capacity Building Plan The activities and results of ASP-RSPN interventions consisted of the following three phases: Phase I: To provide technical assistance via human resources to the Finance Department to conduct an internal audit as per the existing charter, and in accordance with the implementation plan for salaries and pensions for the Communication and Works, Planning and Development, Education, and Health Departments. Phase II: To improve/develop policies and procedures to conduct internal audits. Phase III: To replicate the establishment of the Internal Audit Function horizontally and vertically (under the provincial departments as well as district governments) across the province. Given the limitation in scope of the ASP Interim Evaluation (specifically, no site visits or meetings with current government officials), we cannot conclusively comment upon the progress made or otherwise toward the achievement of the three phases. However, based upon our discussions with the Implementing Partner (IP) and former government focal person(s), we believe that low to moderate progress was achieved on Phase I and Phase II, whereas low progress was made on Phase III. 139 Manuals Introduced/Developed An Internal Audit Toolkit was developed and approved in June 2013 by the Finance Department for Elementary and Secondary Education Department, GOKP. The toolkit is the Finance Department’s recommended approach to carrying out internal audit activity at the departmental level. Expected Goals/Outcomes Sanctioned posts were created, and the Internal Audit Toolkit was developed and approved for the Elementary and Secondary Education Department. Performance Measurement Criteria There are no performance measurement criteria for this initiative. The MEP team would need to rely on the progress of the envisaged outputs that initially included four line departments. Progress against the outputs was not achieved fully, owing to lack of buy-in and broad-based consensus on the purpose and rationale of the initiative. 5. Operational Changes (from IP interviews and review of documents they provided in support) ASP support for Internal Audit in Khyber Pakhtunkhwa (KP) has been different than the support for the other public sector organizations in that Internal Audit is a cross cutting intervention that is implemented through a number of departments within the Province. Both GOKP and USAID identified the Internal Audit Function as a necessary entity to be created. GOKP requested that USAID’s ASP facilitate the establishment of Internal Audit, with a focus on four departments: Education, Health, Police, and Communication and Works. USAID opted not to work with the Police Department (additional details are below). ASP-RSPN conducted a detailed technical assessment and then developed the methodology, governance structure, job descriptions, and reporting requirements for Internal Audit. These were shared with the government through a series of workshops and discussions. These were then forwarded to the GOKP for approval. The approval resulted in the creation of 18 positions for Internal Audit (2013-2014). Each position now has to have the rules of engagement (standard operating procedures) established. All of the positions will be paid by the GOKP. The GOKP had a clear idea of what they wanted from ASP-RSPN. This included identification and documentation of every process—HR, Finance, M&E, Procurement—and each step within each process. This resulted in the development of a robust set of Internal Audit Toolkits. Each of the three identified departments (Education, Health, and Communication and Works) was divided into sub-entities and the Finance function within each became the functional area responsible for Internal Audit. Previously there was no high-level forum where issues of coordination among various GOKP departments could be discussed; as of January 2014 GOKP has created a Steering Committee for ASP-RSPN interventions in support of the Internal Audit Function. Training will begin when the 18 positions are filled. First there will be classroom training followed by mentoring support by government officials or ASP advisors. 6. Challenges (interviews on site) Although 18 posts for GOKP's Internal Audit Function have been sanctioned, at the time of this evaluation, none have been filled. Therefore, for the purposes of this Interim Evaluation, discussions were only conducted with the IP and former government officials involved in this process. 140 7. Key Observations Given the scope limitations for this case study assignment (i.e., no meetings with current government stakeholders), it was observed that the initiative achieved only limited success, and overall performance was below satisfactory. The evaluation team sensed (but could not substantiate) that ASP-RSPN was not able to generate momentum to achieve the desired objectives/phases. As a result, there is not much to discuss about the initiative beyond the creation of the 18 positions and the development of the Internal Audit Toolkits. Per discussions with former GOKP focal persons for internal Audit, the Finance Department was selected to pilot the initiative, which was financed through government resources. The four departments selected to introduce Internal Audit were Education, Health, Communication and Works, and Police. However, USAID opted not to work with the Department of Police, and the UK Department for International Development (DFID/UK Aid) took up the support of the Education Department. This left the ASP-RSPN initiative to cover Health and Communication and Works. The initiative was not able to make any visible or meaningful progress. While meetings and workshops were held, there were few tangible results that emerged. The capacity of the staff involved from ASP-RSPN was not adequate for the task, and contained mainly former government employees and current civil servants who were on leave. The composition of the team from ASP-RSPN should have contained a balanced mix of people from both the private and public sectors, with the former in the majority. However, the ASP-RSPN team was exactly the opposite. This hampered the team’s ability to take a fresh approach to the tasks at hand and to attain meaningful progress. The GOKP Finance Department did not feel the need for the Internal Audit Toolkits in their current form. They had reservations about the usefulness and purpose of the toolkits. The initiative focused more on outputs than outcomes, and that led to its lack of effectiveness. In parallel, the GOKP did not have a unified stance on who owned the initiative. The Finance Department had wanted to lead the initiative. The departments selected for the intervention agreed that the Finance Department should lead, but wanted their own Principal Accounting Office to have greater input and authority in the development and review of the reports the Internal Auditor produced on their department. Lack of ownership and clarity within the GOKP also contributed to the lack of effectiveness of the initiative. ASP￾RSPN did not play a significant role in addressing these issues. 141 Case Study on Capacity Building of the Small and Medium Enterprise Development Authority (SMEDA) Evaluation Team: Mr. Marc Shiman, Technical Director Ms. Vardah Malik, Evaluation Consultant Interviewees: Dr. Naeem Rauf, General Manager, Central Support Waheed Iqbal Kayani, Finance Manager Shaheen Tahir, Manager, Monitoring & Evaluation (M&E) Mohammad Asif, Monitoring & Evaluation Officer Tania Buttar, Manager-Incharge, Human Resources (HR) Liaqat Ali Gohar, Deputy General Manager, Administration & PR (also oversees Procurement) Arif Chaudhry, Manager, Legal Services 1. Purpose The purpose of this case study is to determine the extent to which ASP capacity building services in financial management, procurement, human resources management (HRM), and M&E were useful and implemented (effective and institutionalized). 2. Background SMEDA was initially established by the order of the Prime Minister in October 1998, under the Ministry of Industries and Production, to take on the challenge of developing small and medium enterprises in Pakistan. Under the SMEDA Ordinance 2002, SMEDA was given the status of an autonomous body attached to the Ministry of Industries and Production. SMEDA’s mission is to function as the promoter and facilitator of the small and medium enterprise sector in Pakistan. SMEDA aims to create an environment conducive to SME development, provide business development, corporate, and legal advisory services, and facilitate service delivery. Apart from its role as the small and medium enterprise policy-advisory body for the Government of Pakistan (GOP), it also facilitates other stakeholders in addressing their small and medium enterprise development agendas. In 2010, SMEDA and USAID were in discussions to conduct a joint project. As a result, USAID initiated a pre￾award assessment (prior to the launching of ASP). Despite the fact that the project never materialized, ASP-RSPN approached SMEDA in 2011 to become a recipient of ASP services. 3. Pre-Award Assessment Deloitte carried out a pre-award assessment for SMEDA in January 2010. The objectives of the pre-award assessment were to assess: SMEDA’s management structure; accounting and financial management systems and other systems of internal controls; quality assurance capabilities; and whether policies, procedures and practices were in line with USAID/Pakistan’s established criteria. As a whole, they found that SMEDA was being run professionally, and was effectively developing the small and medium enterprise sector. However, it was also observed that some instances of non-compliance with the SMEDA Ordinance were not being addressed. Specifically, these included issues in the areas of M&E, finance, procurement, and HRM. 4. Planning Following multiple meetings, in September of 2012 SMEDA and ASP-RSPN agreed on a plan to build the capacity of SMEDA in two phases. It appears the plan was developed in close cooperation between ASP-RSPN and the beneficiary. This plan included the following phases: Phase I A) Staffing Requirements Assistance: Because SMEDA had filled only approximately two-thirds of its allotted staffing slots; ASP-RSPN was to assist SMEDA with drafting new job descriptions and organizational charts. 142 B) IT Software Development: SMEDA requested the development of an Enterprise Resource Planning (ERP) system to replace its outdated financial software. C) Development of Policies and Procedures Manuals: SMEDA requested assistance developing policies and procedures manuals in the four areas of HRM, Finance, Procurement, and M&E. Also, ASP-RSPN was supposed to help with other areas of risk identified in the Pre-Award Assessment. Phase II The second phase of capacity building was meant to include training on the policies and procedures manuals and hands-on assistance to SMEDA. Specifically, the capacity building plan stated: On-the-job technical assistance and training will be introduced, after the above mentioned regulatory manuals have been developed. ASP-RSPN will deploy a consulting firm with expertise in the fields of Financial Management, HR/Admin, Procurement and Monitoring & Evaluation (M&E) for a period of five (5) months… There was no reference in the capacity building plan to LUMS trainings or any other type of training to be provided by ASP-RSPN. 5. Implementation of Capacity Building Ultimately, only the manuals were developed, followed by 1-day “orientations” (the term given by SMEDA participants) for the manuals. ASP-RSPN never assisted with job descriptions or an ERP system, and did not assist with other findings in the pre-award assessment. There was no follow-up implementation assistance, although the HR Manager reported that the consulting firm has been generous with its time, ensuring that any problems with interpreting the manual are resolved. ASP-RSPN engaged four different consultants and consulting firms to assist SMEDA in developing its manuals. The finance manual was developed after research by the consultant on the procedures currently in place. The M&E manual was developed for SMEDA by a consultant following regular consultations with all of the department heads. The M&E Department within SMEDA was newly established prior to the development of the manual. The beneficiaries found the first draft of the manual to be generic and lacking specific content relevant to SMEDA’s objectives. While the second draft was better tailored to SMEDA, the M&E Department has said that the consultant was unable to respond to requests to develop organization-specific M&E forms. The M&E Department is currently tailoring the manual to meet their needs. ASP-RSPN’s contractor worked very closely with the personnel in the HR Department to finalize the HR manual and introduced many new (and needed) forms and policies. In absence of those policies, SMEDA reverted to Pakistani law, but the existence of a SMEDA-specific manual is thought to be of enormous assistance to SMEDA. The Procurement Manual was a particularly valuable document to SMEDA because there was no Procurement Department (it was part of Public Relations and Administration), no procurement personnel, and no rules of procurement (although the PPRA rules apply). The manual calls for the formation of a department, employment of procurement specialists, and the application of forms and procedures. SMEDA does not do much large-scale procurement except to meet the needs of its offices. Their programs are of a technical assistance nature. The Board of SMEDA has not yet approved the manuals, as the Chairman position has been vacant for some time. As such, the procurement department has not been created, and the relevant employees have not been hired. Nonetheless, most of the departments are referring to and applying the manuals, particularly those components that are summaries of existing rules and regulations and new forms that do not require Board approval. Over the course of three days, the consultants hired by ASP-RSPN conducted what the organization has termed “orientation sessions” on all four manuals. Department heads and other officers were invited to participate in all three days of orientation (a total of 50-60 participants). Each session ranged from a half-day to a full day. These sessions helped participants gain awareness about the contents of the manuals; however, the department heads said it added little value to their knowledge of their respective manuals. SMEDA has sent over 30 participants to LUMS training during the life of ASP. It appears that only the department heads’ participation in the LUMS training has played a role in influencing the impact of the rest of the ASP activities. It is not clear that their participants undertook any activities as a direct result of LUMS trainings. 143 6. Challenges The quality of the consultants and their topical expertise varied by subject. Of the four, the consultant most criticized was the M&E consultant, who did not appear to have a strong grasp of the subject matter to the point that he was unable to customize solutions specifically to SMEDA’s context. The other consultants were thought to be knowledgeable, and the HR consultants were thought to have added significant value. In April of this year, SMEDA’s building was destroyed in a fire, and SMEDA was operating out of tents until its recent move into its new premises. Recovering from the fire has become a higher priority to SMEDA than the implementation of their manuals. In addition, SMEDA’s role in implementing the Prime Minister’s project on youth employment has commanded senior management’s attention from fully implementing the ASP Manuals. 7. Effectiveness It appears from SMEDA’s feedback that the ASP intervention provided value-added to the organization by creating ready access to rules and regulations in three of the four departments in which manuals were written. However, because the manuals have yet to be Board approved, those areas including the creation and staff of the Procurement Department have yet to be implemented. Board approval is not a guarantee, so it is difficult to assess whether ASP’s activities in that area will ultimately be effective. 8. Sustainability SMEDA has demonstrated that it has not only employed the manuals, but that it has done so with enthusiasm. It is likely that SMEDA will continue to implement the manuals to the extent possible absent Board approval. In addition, SMEDA has also shown the internal capacity to further develop the manuals as needs arise. 9. Gender The ASP-RSPN HR consultants regularly referred to ensuring gender equality within their organization; however, SMEDA felt that in this regard, they are already one of the leaders in progressive policies toward gender among public sector organizations. 10. Conclusions While SMEDA was disappointed that ASP-RSPN did not implement the whole of the capacity building plan, they felt that the assistance provided by ASP-RSPN has had a material impact on the way they operate. They also praised the LUMS training, although they cannot point to any specific changes in the organization as a result of the training. Nonetheless, SMEDA feels that significantly more capacity building, particularly in terms of IT automation, is needed to modernize and streamline the operations of the organization. 144 Case Study on Capacity Building of the Federal Public Procurement Regulatory Authority (FPPRA) Evaluation Team: Mr. Marc Shiman, Technical Director Ms. Vardah Malik, Evaluation Consultant Interviewees: Nazrat Bashir, Managing Director Alam Zeb Khan, Director General Rizwan Mehmood, Assistant Director & Incharge IT and Monitoring (Technical Head) Yasir Shamim, Head of Finance Department (Technical Head) 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Background The Federal Public Procurement Regulatory Authority (FPPRA), established by the GOP, is an autonomous body under the Public Procurement Regulatory Authority Ordinance of 2002. FPPRA’s mandate is that of a regulatory and monitoring body with the responsibility of prescribing regulations and procedures for public procurements by federal government-owned public sector organizations to improve governance, management, transparency, accountability, and quality of public procurement of goods, works, and services. FPPRA regulates the public sector organizations through exercising the “Public Procurement Rules 2004,” which states the following: Procuring agencies of the Federal Government, whether within or outside Pakistan, while engaging in procurements, shall ensure that the procurements are conducted in a fair and transparent manner, the object of procurement brings value for money to the agency and the procurement process is efficient and economical.102 FPPRA is a result of the World Bank’s Country Procurement Assessment Report of 2000, which resulted in the World Bank’s assistance to FPPRA in the form of a three-year capacity building program and the establishment of the National Institute of Procurement. In addition, the Asian Development Bank is supporting the strengthening of the public procurement system in Pakistan through a major initiative entitled “Use of Procurement Agents in Public Procurements.” In 2010 USAID became involved through ASP. As an ASP IP, RSPN began working with four public procurement regulatory authorities in Pakistan, with the main focus on institutional capacity building, policy advocacy, and development of management systems for effective and transparent procurement processes in the country, particularly at the federal level. The formation of an Advisory Group on Public Procurement and its inaugural meeting in April 2013, under USAID’s ASP project, was an important milestone in the harmonization of different procurement regimes in the country, whereby a National Procurement Strategy was initiated and recently finalized.103 The constitution of the Advisory Group is the outcome of the Stakeholders’ Consultation Workshop on Public Procurement Regulatory Regime in Pakistan, organized by ASP-RSPN in April 2012, to bring all the four public procurement regulatory authorities and FPPRA onto a single platform to create partnerships so as to facilitate harmonization of procurement laws, procedures, and practices, and to deliberate on options for strengthening procurement systems in the country. 3. Initial Assessment/Analysis/Identification of Issues A pre-award assessment of FPPRA was not carried out to provide a baseline for the ASP interventions. However, the report from the April 2012 consultation workshop by all stakeholders on countrywide procurement practices 102 ASP-RSPN, Capacity Building Plan for Public Procurement Regulatory Authority (PPRA) (November 7, 2012). 103 Inaugural Meeting of Advisory Group on Public Procurement (April 8, 2013). 145 became a guiding document for the dialogue between FPPRA and ASP-RSPN. This document acted as the basis for the FPPRA ASP Capacity Building Plan. 4. ASP Strategies and Interventions Under the USAID-funded ASP project, its IP RSPN is helping the FPPRA and four provincial public procurement regulatory authorities to improve and develop policies, procedures, and management structures, as well as to establish a voluntary Advisory Group on Public Procurement. Since public procurement is a provincial subject in Pakistan, the federal and provincial public procurement regulatory authorities operate independent of each other. As such, prior to ASP, there was no consultative forum among the five regulatory organizations to facilitate harmonization of a regulatory framework. The ASP project provided this forum to share knowledge and experiences among all the public procurement regulatory authorities; to create linkages at the national level in public procurement; to identify required changes in the regulatory framework for making the public procurement effective, efficient, and transparent; to harmonize various rules and regulations at the federal and provincial levels according to international best practices; and to contribute to the development and implementation of the National Procurement Strategy. To implement ASP’s envisaged interventions, a capacity building plan was developed in November 2012 by ASP￾RSPN. It states the following: This capacity building plan shall impact Sub Intermediate Result 1.1 of the performance management plan (PMP) through the provision of trainings and technical assistance to PPRA. Specifically it shall address output indicator 1.1.1 by executing this capacity building plan, 1.1.2 by executing capacity building initiatives, 1.1.4 through the development of a capacity building plan to strengthen the institutional capacity of PPRA, 1.1.7 through policy and advocacy workshops, 1.1.8 by developing training programs for effective utilization of USG resources, 1.1.9 by conducting training programs and 1.1.10 by imparting trainings to PPRA staff.104 The capacity building plan was developed when FPPRA management approached ASP-RSPN for assistance, as FPPRA did not have the budget to undertake major capacity building activities. The specific objectives of the capacity building plan for FPPRA included the preparation of the National Procurement Strategy; revision of Standard Bidding Documents for works, goods, and services; development of templates for procurement planning; a procurement code of ethics; system development for FPPRA for ISO 900-2008 Certification; establishment of an M&E system; a Grievance Redressal Mechanism; revamping of the FPPRA website; an ERP system with an HR module; a Learning Management System; and building FPPRA’s in-house capacity to organize trainings for procuring agencies. The capacity building plan was to be implemented in two phases, with Phase I focused on ASP-RSPN providing technical assistance in all of the above areas and Phase II primarily involving implementation of manuals, rules and regulations, and trainings. Specifically, ASP-RSPN was to assist in organizing and training a cadre of trainers. The capacity building plan specifically stated that gender integration within ASP-RSPN’s activities and outputs would continue to be an important theme during all phases of planning and implementation, resource use, management, and associated decision-making. Actual Implementation under ASP: It was found that the development of the ERP system for FPPRA was viewed as the highest priority intervention under the ASP project and a critical step toward enhancing public procurement monitoring by FPPRA as a regulatory body. Development of the ERP system by ASP-RSPN experts involved contributions by FPPRA staff to ensure that the system was customized and implementable within FPPRA and that it addressed FPPRA’s needs. FPPRA management was satisfied with the development of the ERP system and its modules, minus some delays experienced in its implementation. 104 ASP-RSPN, Capacity Building Plan for Public Procurement Regulatory Authority (PPRA) (November 7, 2012). 146 Manuals: The four main modules within the ERP system were as follows: (i) HR Module: fully operational now and being used by all FPPRA employees; (ii) Accounting and Financial Module: interlinked with the sub-module on tender management, partially implemented as of yet; (iii) Learning Management System: an e-learning system currently under development; and (iv) M&E Module: largely under development, but is the core module related to FPPRA operations. There is also a Grievance Redressal System being developed at FPPRA under the ASP project by ASP-RSPN experts for complaints management that will be part of the M&E module. Actual implementation of ASP at FPPRA digressed from the original capacity building plan. The ERP system was an all-encompassing system covering four main modules. As for manuals, each module will have a user manual that will be treated as the manual for the four areas of interventions, namely, HRM, financial management, M&E, and learning management. The ERP system has been designed to track procurement from start to end. Its main purpose is to enable FPPRA to play its public procurement regulatory role. One observation was that the HR module developed by ASP-RSPN is based on government rules and regulations that govern HR-related matters pertaining to all government employees. There are no approved service rules that govern the HR activities of FPPRA, being an autonomous government authority. The Esta Code (HR policies and procedures developed by the Establishment Division that governs all government employees) does not apply to FPPRA, and to date, FPPRA cannot hire employees on permanent contracts, and hence it is difficult to attract good professionals to work for FPPRA. Trainings Conducted: To date, there have been ASP-RSPN trainings mainly on the HR module, and most staff have been trained on the module. Other trainings have not taken place as yet on ERP, as it is still in the implementation phase. Two officers had attended training on procurement at LUMS, while the head of the IT & Monitoring Department is scheduled to attend LUMS training. The Finance head did not have any knowledge of the LUMS training schedule. It was felt that key staff involved in the implementation of the capacity building plan, including the finance head, had not had the opportunity or access to LUMS training as yet. The management recognized that the capacity building plan requires LUMS training on specific modules, but this has been delayed. The management explained that FPPRA is short-staffed, with only eight officers out of a total staff of 48. It is difficult with such a small number of operational staff to be able to send people to training with no substitutes available in their absence. ERP is expected to be fully executed and implemented by the end of this year, and that is when most of the trainings are expected to take place at LUMS. In general, the officers found LUMS training useful. Part of FPPRA’s mandate is also to train public procurement officials in various government departments, so there is a need for a Training of Trainers, which was one of the actions under the capacity building plan. The scope and details of such trainings have not been discussed with ASP-RSPN as yet. There is a National Institute of Procurement within FPPRA that provides 2-day trainings to procurement staff in the government agencies. FPPRA plans to link programs under the institute to LUMS training by offering similar courses online, helping them reach a larger number of government employees. With this in mind, the Training of Trainers should have been an important part of ASP implementation. Other Capacity Building Interventions: As per the capacity building plan, other capacity building interventions by ASP-RSPN included development of the National Procurement Strategy, which provided a common vision to make the public procurement system efficient and transparent in the country. The Strategy endeavors to lay down a roadmap for federal and provincial regulators as well as the procuring agencies that can help them realize the objectives and principles of procurement. Expected Goals/Outcomes: The ERP system will deliver promised changes, as it includes the entire procurement process starting from planning of procurement and ending with payment to the contractor. It will collate information on the tenders and payments against them and also will provide information on the suppliers that is missing in the existing system at FPPRA. The ERP system will link the accounting and financial management function with tenders management, thus it is expected that the procurement function will be better managed at FPPRA. The system was scheduled to go online and to be fully implemented by December 2013 (within six months of hiring the consultant). Since it is still in progress, any changes that may emerge as a result are yet to be seen. 147 Gender considerations: The senior management of FPPRA does not consider gender relevant in terms of the application of procurement laws. We were also informed that gender was not included in the proposed service rules. Performance Measures: No performance measures were established to assess new knowledge gained or change in performance. 4. Operational Changes/Results As recently as three to four months ago, HR functions at FPPRA were entirely manual. The implementation of the ASP-RSPN HR module, which is now fully functional, has provided access to the web-based HR portal for all FPPRA employees, leading to paperless HR management, and this was well appreciated by the staff. FPPRA clerical staff’s capacity to use the HR system was a challenge, as the system requires basic knowledge of the English language. FPPRA accounting procedures were approved in July 2014, and were used by ASP-RSPN consultants to develop the accounting module. This module is partially implemented, as parts are linked to a tender module that is still under development. The accounting module includes preparation of five key financial statements for FPPRA (ledgers, cash flow statement, balance sheet, etc.). The finance head was key in identifying FPPRA’s need for an accounting module and was particularly happy that an accounts receivable system (linked with the tender management, as procuring agencies pay a fee of PKR 1,500 for uploading tenders on the FPPRA website) was part of the accounting module, since the old accounting software (Quickbooks) lacked this provision and did not fully capture the PKR 90 million in revenue generated through tender fees by the FPPRA. The Managing Director was personally involved and interacted closely with USAID and ASP-RSPN and had positive feedback on what had changed since the intervention rolled out. ASP-RSPN intervened and assisted FPPRA with the HR module, implementation of the ERP system, and development of the National Procurement Strategy. Hiring two consultants and supporting FPPRA in consultations with the four independent provincial public procurement regulatory authorities achieved the latter. The National Procurement Strategy recently has been approved. ASP￾RSPN helped FPPRA set up the Advisory Group on Public Procurement, composed of representatives from donors, four provincial public procurement regulatory authorities, a contractors’ association, and other stakeholders. 5. Challenges One part of the capacity building plan under the ASP project for FPPRA was to develop standard bidding documents for procurement of civil works, services, and goods. Departments such as the Pakistan Engineering Council are still using the existing standard bidding document for civil works already in place at FPPRA. Different formats are currently being used by various procuring agencies and need to be standardized. The new drafts of standard bidding documents prepared by ASP-RSPN consultants were scrapped by FPPRA. ASP-RSPN hired a consultant for only 7-10 days to develop the standard bidding documents (which clearly required a higher level of effort by an expert). The quality of the ASP-RSPN consultant’s deliverables for this component was not satisfactory. Standard bidding documents, which constituted a major part of the capacity building plan, still have not been delivered. FPPRA is now working with consultants from the Asian Development Bank for 10 months to develop standard bidding documents. It would have been better if FPPRA had been involved in the selection of experts with a background in public procurement. FPPRA could have helped ASP-RSPN identify suitable candidates, or ASP-RSPN could have advertised the position to attract better consultants. FPPRA did not think highly of the consultants hired under the ASP project, and there was a perception that the ASP-RSPN team did not contribute anything other than hire ERP consults, a process that saw significant delays. FPPRA leadership recommended that consultants with the required background in procurement should have been hired to work with FPPRA, which would have made the whole process easier for the organization. FPPRA staff were not involved in identifying the consultants under the ASP project. Furthermore, ERP consultants lacked expertise in the relevant fields, such as financial management for those consultants charged with developing the 148 accounting module. Finally, ASP-RSPN’s practice of hiring only chartered accountant firms is of concern to FPPRA, as chartered accountant firms may or may not have public procurement expertise. ASP is a short-term project, and the interventions required long-term support. The short-term nature of the project seemed to drive the selection of consultants, whose level of effort was not adequate for the requirements of FPPRA. 6. Key Observations/Sustainability/Effectiveness Sustainability: FPPRA is an organization that takes ownership of the development and implementation of its own capacity building under the ASP project. Being a small yet effective organization with the support of top management, significant changes are expected as a result of the ASP project. For the ERP system to be as functional as envisioned, technical support is needed by FPPRA for another 2-3 years. With the Head of IT and Monitoring actively involved in debugging the ERP system with the consultants, and the Head of Finance involved in the accounting module rollout, it was felt that the consultants should be retained for some time. This will help ensure successful implementation of the ERP system and allow for any future maintenance/updates of the modules. FPPRA does not have the in-house IT capacity to ensure maintenance of the ERP system once fully developed. FPPRA top management has been very supportive of the ASP project; however, any change in the top management is likely to put the sustainability of the capacity built under the project at risk. FPPRA has not been able to train key staff on the system and many new recruits/hires are expected in the near future (positions for four director generals and many directors have been sanctioned recently). Effectiveness: FPPRA is the beneficiary of various interventions by three donors, including USAID, the World Bank, and the Asian Development Bank. FPPRA also has its own ongoing procurement reform agenda. The evaluation team was informed that the World Bank has helped FPPRA develop the world’s first one-stop portal dedicated to training, knowledge-sharing, and networking in the public procurement field, as well as their e-procurement strategy and the National Procurement Strategy. ASP has helped FPPRA develop the Learning Management System and e-learning module, which will be fully implemented once the standard bidding documents are developed by the Asian Development Bank consultant. FPPRA felt that donor coordination for such interventions should be strengthened to avoid duplication, and to ensure the effectiveness of interventions, given that many activities are interlinked and being supported by various donors. Training of Trainers was suggested as a way to overcome the lack of training provided so far to FPPRA staff and other public procurement officials. FPPRA would like the ASP-RSPN team to provide this training. This was also part of the capacity building plan and still needs to be delivered. FPPRA, under the National Institute of Procurement, has the capacity to train up to 480 people a year on public sector procurement and could take capacity building to scale if provided the opportunity for Training of Trainers. 149 Case Study on Capacity Building of the Provincial Relief, Rehabilitation and Settlement Authority (PaRRSA) Evaluation Team: Dr. Janet Robb Mr. Haider Ali Daud Khan Interviewees: Kamran Khan, HR and Procurement Johar Ali, Finance Javed Iqbal, Deputy Chief of Party, ASP-RSPN Nazar Rathore, Director Capacity Building, ASP-RSPN Farhan Ghafoor, Advisor International Development, ASP-RSPN Iqtikhar Amad Rao, Advisor International Development, ASP-RSPN 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Context The Provincial Relief, Rehabilitation and Settlement Authority (PaRRSA) was established in 2009 under the Provincial Disaster Management Authority by the Government of Northwest Frontier Province, now GOKP. PaRRSA was created to look after the reconstruction, rehabilitation, and settlement of those affected by crisis. PaRRSA is to provide the requisite speed, ease, facilitation, coordination, supervision, and linkages to all parties involved and to help the provincial government in its endeavor to rehabilitate the affected areas. The initial focus of PaRRSA has been the rebuilding and rehabilitation of the public infrastructure destroyed during the conflict in 2007–2009 and the floods in 2010 in the Malakand Division of KP Province. This was an estimated need for US$1.09 billion to rectify the damage (as identified in the Damage and Needs Assessment of 2009). The main contributors to the resource needs for PaRRSA activities have been the World Bank, the Asian Development Bank, the United Nations Development Program (UNDP), and USAID. PaRRSA’s Annual Work Plan is a compilation of District Work Plans, and a PC-1 is prepared for each activity in the Annual Work Plan. Through the PC-1s, administrative approval for the activity/project is sought and, once approved, funds are released. For those non-developed departments seeking funds, PaRRSA is responsible for the procurement of machinery, equipment, and furniture. PaRRSA’s oversight and operations are administered by a Strategic Oversight Council responsible for strategic and policy level actions. Three committees provide the functional support: A Provincial Steering Committee serves as a decision-making forum; the Division Security and Coordination Committee provides leadership and supervision; and an Advisory Committee identifies gaps, suggests improvements, and provides linkages between the Government and communities. Special Support Groups are used to create synergy and support for various activities and projects. 3. Initial Assessment The below concerns/issues/gaps in the capacity of PaRRSA were derived from the Host Country Contracting & Fixed Reimbursement Arrangement Capabilities Assessment of PaRRSA, December 2009, conducted by Deloitte Touche Tohmatsu (contracted by USAID), which essentially served the purpose of a pre-award assessment for ASP. These are summarized under seven thematic clusters: financial management/internal controls; procurement; administration and HRM; M&E; grant handling and reporting; and gender. The Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation Draft Report (July 9, 2014) augmented original source information. The responsibility for capacity building within PaRRSA was transferred from KPRP to RSPN under ASP, on January 31, 2012. 150 The concerns/issues/gaps are as follows: Financial Management/Internal Controls  Manual bookkeeping process coupled with lack of appropriate and sufficient human resources.  Lack of capacity of the office of KP’s Director General of Audit to conduct audits that respond to the requirements of myriad donors, in particular to the satisfaction of USAID.  Operations Manual in draft form, awaiting approval by the Provincial Government. Procurement  The North West Frontier Province Procurement of Goods, Works and Services Rules, 2003 are less stringent than USAID standard procurement procedures.  Lack of requisite procurement resources.  Burden of procurement of machinery, equipment, and furniture for non-developed departments. Administration and HRM  Inadequate and limited absorptive capacity to handle the magnitude of project funds coming to them.  Ratio of contractual staff to civil servants is extraordinarily high, and there are no plans in place to reduce the number of contractual employees.  A point of contact (focal person) is required for each functional area, but none exist.  Ill-defined job descriptions and segregation of duties. Training  There are unfulfilled training needs for PaRRSA officials on different subjects, in particular in budgeting and accounting, procurement, HRM and administration, M&E, and operations and governing policies. M&E  Need for independent oversight to avoid any corruption and manage public perception of PaRRSA.  Poor quality of reports, poor management of process, and limited or no link to field-based reports. Grant handling and reporting  Powers of waiver from tendering requirements given to the Director General of PaRRSA under certain circumstances; ineffective for USAID funding utilization. Gender  Attempts at gender balance within PaRRSA have not achieved their goal.  There have been 3 women out of 106 contracted hires, after public notices stating that qualified women would be given priority in hiring. 151 4. ASP Strategies and Interventions Capacity Building Plan After its commencement, ASP-RSPN developed a capacity building plan in 2012. The objectives of the plan were as follows. Objective 1: To evolve a robust financial management system with a focus to strengthen budgeting and accounting processes. The following is the progress made to date:  A Finance Manual was developed, and almost 80 percent of it is implemented.  An internal audit team, with qualified personnel, is operational and has developed numerous valuable reports based on their internal audit. An internal audit manual is also developed and delivered to PaRRSA and is in review stage.  Training on the Finance Manual was arranged in January 2012 and attended by all the staff.  Trainings included one-on-one training of PaRRSA staff by ASP-RSPN, and an ASP-LUMS training attended by various finance staff.  A detailed budget process as mentioned in the Finance Manual has been developed.  In October 2012 ASP finalized the procurement process for the outsourcing of IT activities; however, in April 2013 the Director General of PaRRSA and USAID decided that PaRRSA will pay the cost of IT and ASP will provide technical assistance. The activity is still in process. Objective 2: To develop and implement a transparent and efficient procurement policy for emergency and rehabilitation/reconstruction phases.  Overall, progress is encouraging as the manual, strategy, and training modules have been developed. The only exception is the automation of the procurement process, for which PaRRSA shall provide the hardware cost, whereas ASP-RSPN shall bear the technical assistance costs. Objective 3: To strengthen the HRM and administration function to provide uninterrupted quality support to program activities.  Some progress was made in these areas through the development of job descriptions, manual preparation and approval, training modules, etc. However, training needs assessment, training selection for more inclusiveness, and obtaining approval of a revised organogram are areas that require further progress. Objective 4: To establish a central M&E system for evidence-based strategy formulation, management, and accountability. Objective 5: To improve Provincial Disaster Management Authority/PaRRSA operations and governing policies for achievement of desired outcomes. Trainings Conducted Trainings worth US$250,000 were conducted by June 30, 2014, for the functional areas listed below:105  Financial management  Internal Audit  IT  Procurement 105 ASP-RSPN, Activity Tracker (April 30, 2014). 152  Administration and HRM  M&E Selection for trainings requires further improvement to ensure better utilization of funds. We observed that there were staff that were yet to be sent on training in their core areas, but had been sent on training in areas outside of their technical expertise. No dedicated trainings on gender were noted. Manuals Introduced/Developed  Financial management  Procurement  Administration and HRM  M&E All the manuals are largely a compilation of relevant/applicable government rules and regulations. They are useful in this respect; however, their regular use was not observed during the field visit and discussions. The manuals do not have version control or provisions for updates, and this is a major deficiency. Gender PaRRSA is striving to hire more women and to make progress in achieving a better gender balance among staff; however, so far this has not been met, owing to lack of applications from women even when statements such as “Female candidates with equal qualifications will be given preference” were made.106 Positions Trained The Activity Tracker did not mention the positions of the persons trained thus far. Expected Goals/Outcomes These were articulated in the capacity building plan established in 2012. Performance Measurement Criteria There is a lack of specific performance measurement criteria for PaRRSA engagement. The evaluation team did not observe any institutionalized mechanism in place to measure performance and progress, even against the capacity building plan. 5. Operational Changes (from IP interviews and review of documents they provided in support) Based upon the IP interviews, it was understood that significant progress has been made in trainings and manual development. Document reviews confirm development and approval of the various manuals. 106 ASP-RSPN, Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation Draft Report (July 9, 2014). 153 6. Challenges (interviews on site) Interventions/changes PaRRSA is not a permanent government structure yet has to follow government rules and regulations in its day-to￾day operations. Unlike a typical fast-track operation through a Program Management Unit (PMU) type of structure (which PaRRSA actually is), built-in bureaucracy and the pace of government hinder the effectiveness of PaRRSA’s changes. In many ways PaRRSA suffers from the very challenges that it is designed to overcome for public sector￾led initiatives receiving USAID funding. Manual development and approval is claimed as one of the key successes of ASP-RSPN, yet the usefulness of manuals in day-to-day operations remains uncertain given that these are compilations of pre-existing government rules and regulations. The newly developed manuals are devoid of proper mechanisms to update them and maintain version control. These are more systemic risks for the entire ASP-RSPN project as a whole rather than being PaRRSA-specific. Sustainability Regarding the sustainability of interventions by ASP at PaRRSA, it consistently emerged that the Provincial Disaster Management Authority is the ultimate beneficiary. It was noted that PaRRSA is a “special purpose vehicle” and not intended to be permanent. It was developed to respond to the special circumstances of the time: the 2005 earthquake, the 2007 floods, and civil unrest. PaRRSA is a “spin-off” from the Provincial Disaster Management Authority, whose legal framework was initially established by GTZ. If funding for PaRRSA were discontinued, it is widely understood by those working for PaRRSA that the Provincial Disaster Management Authority would absorb their roles and responsibilities. 7. Key Observations ASP-RSPN achieved development and approval of various operational manuals and ensured select trainings for staff. For the purposes of this case study, the interim evaluation was limited by the inability to schedule meetings with the PaRRSA M&E point person and the Director General of PaRRSA despite numerous attempts by the evaluation team. However, interviews with the IP and other PaRRSA technical managers, a site observation, and manual review were carried out. Particularly noteworthy, it was noted that if the ASP-RSPN team is truly to bring about change within GOKP, former and serving GOKP officials might too heavily dominate it. 154 Case Study on Capacity Building of the Project Management Unit–Municipal Services Program (PMU-MSP), Khyber Pakhtunkhwa Evaluation Team: Dr. Janet Robb Mr. Haider Ali Daud Khan Interviewees: Hassan Tariq, Finance Shafiq Alam, HR & Institutional Development Specialist Amjad Hussain, M&E Specialist Wajid Khan, Internal Auditor Muhammad Ilyas, Procurement Specialist Mr. Riaz Khan, Project Director M. Mohsin Khan, ASP-AiD Mudassar Salim, ASP-AiD 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Context The Municipal Services Program (MSP) is an initiative of the GOKP aimed at improving municipal services delivery in Peshawar, Dera Ismail Khan, and Malakand Divisions. The bulk of the program funding, i.e., US$84.5 million, is to be provided by USAID, whereas GOKP has agreed to allocate around US$12.7 million as counterpart funding. The objectives of MSP are closely aligned with the Provincial Comprehensive Development Strategy as well as other national and provincial policies with respect to drinking water and sanitation. The implementing agency for MSP is the Local Government and Rural Development Department of the GOKP. Since this department lacks the necessary wherewithal to effectively implement a large-scale complex program like MSP, which requires strict adherence to the rules, regulations, and policies of the Government and USAID, a dedicated Project Management Unit (MSP-PMU) has been established for its implementation. According to the Implementation Strategy as envisaged in the Program PC-1, the MSP-PMU is responsible for the overall implementation of MSP in KP, including the following: 1. Plan, design, implement, and monitor various interventions under MSP, e.g., improvements in urban drinking water supply, sewerage, drainage, and solid waste management, in small and medium towns and cities of Peshawar, Malakand, and Dera Ismail Khan Divisions. 2. Build the capacities of city/town municipal administration for effective urban municipal services planning and through local citizens’ participation in the planning and execution of various activities. 3. The MSP-PMU is to act as secretariat to the Program Steering Committee, which makes all key decisions and accords approval of sub-projects under MSP. 3. Initial Assessment The following concerns/issues/gaps in capacity of organization were identified: a. Financial Management/Internal Controls107  Limitation of budgeting techniques and lack of coordination among departments 107 Risk Mitigation Framework, KP (December 2011). 155  Non-existence of internal audit function  Non-existence of automated financial information system b. Procurement108  Preparation of PC-1  Inadequate procurement rules  Non-compliance of prevailing procurement rules c. Administration and HRM109  Vacancy of senior staff positions  High employee turnover  Inadequate qualifications and experience of staff in HR section d. Training and Capacity Building110  Need to conduct Training Needs Assessment to prioritize training areas e. M&E111  Non-compliance with procedures of Planning Commission  Lack of M&E staff and technological support f. Gender  Gender was not specifically addressed 4. ASP Strategies and Interventions Capacity Building Plan ASP-AiD was assigned the task of capacity building for PMU-MSP in 2011, and it came up with a capacity building plan. However, owing to significant delays in resource development at the PMU-MSP, there were a number of objectives of the initial plan that were not met, and so a revised capacity building plan was devised in 2013. Capacity building was to be provided to MSP in the form of policy and guideline development (manuals) and training related to the content of the manuals and how to use the manuals. No formal capacity building plan was shared with MSP that mapped out what/who would be trained, nor where/how. There was no involvement by MSP in the details of the capacity building plan or the revised capacity building plan. 108 Ibid. 109 Ibid. 110 Capacity Building Plan (2012). 111 Risk Mitigation Framework, KP (December 2011). 156 Trainings Conducted Trainings were conducted by ASP-AiD for PMU-MSP for all the categories below:112  Financial Management  Procurement Management  HRM  M&E There were no dedicated trainings on Gender. Training was provided to PMU-MSP through both in-house training conducted by ASP-AiD technical experts and at LUMS. Responses from half of the MSP technical staff on the training received by ASP-AiD were negative. The M&E Specialist noted, “ASP-AiD totally failed with the M&E Department.” There was no proper training given to M&E. When assistance was offered, it was felt that the ASP-AiD team was not very helpful and that they did not know what they were talking about. Any training received was very basic and was not up to the standard of quality training. It was the USAID M&E team that was the most helpful to MSP. They were very responsive to any request from MSP for assistance. The Finance Manager reported that no training was designed specifically for MSP. MSP was invited to take part in other trainings that were being conducted. The HR Specialist reported this as well. The HR Specialist did take courses at LUMS. KPMG was hired by ASP-AiD to do the training for Internal Audit. KPMG started with an inception report, working together with the Internal Auditor. A manual was prepared by KPMG for Internal Audit. They did step-by￾step training. They were open and responsive to the needs of MSP’s Internal Auditor. Training took place in MSP offices and was given by a two-person KPMG team. The Procurement Specialist attended training at LUMS in both Procurement and Financial Management. ASP-AiD provided technical assistance to the procurement team and responded to requests for assistance when the team contacted them. For the Procurement Specialist, the LUMS training was not “new,” but there were some practical assignments that were helpful. He felt that the case study approach was extremely beneficial. The Procurement Specialist felt that the ASP-AiD technical consultants and LUMS instructors were highly knowledgeable. Manuals Introduced/Developed Manuals were prepared by ASP-AiD in the following areas and then were handed over to MSP for their use:  Financial management  Procurement 112 ASP-AiD, Report on Validation of PMU-MSP, Khyber Pakhtunkhwa (March 2014). 157  Administration and HRM  M&E  Internal audit There was no input by MSP in the preparation of the manuals. It was strongly communicated that the manuals should have been a collaborative process involving MSP. The manuals were merely a compilation of GOKP rules and regulations. There was nothing new included in the manuals. As one technical manager said, “ASP put the manual together, and it is good to have, but if it did not exist, we would have no problem moving forward.” Positions Trained Staff that received training spanned from entry-level positions such as Accounts Assistant to senior-level positions such as Project Director.113 Performance Measurement Criteria When ASP-AiD was asked it was confirmed that the new manuals were actually used throughout the organization, the response of one ASP-AID interviewee was, “We keep an eye, but whether or not they are actually using the manuals, we have no process for follow-up.” No performance measures were put in place to assess knowledge gained or change in behavior. 5. Operational Changes Without any performance measures and since MSP was an entirely new entity, identifying operational changes was not possible. ASP-AiD expressed that, as an IP, they are not responsible for tracking whether the manuals or training are being utilized within the beneficiary organization. It was felt that responsibility lies with the Project Managers within the organization. The Procurement Specialist noted that ASP-AiD support through technical assistance did not give them anything new. He stated, “It is the good people in the organization who know how to do their jobs.” The HR & Institutional Development Specialist stated that MSP has made progress against the deficiencies noted in the original risk management report. The risks are now being mitigated. It is the Internal Auditor who, with the relevant department, documents progress against tasks/challenges to be tackled. 6. Challenges Challenges identified by those interviewed include:  Positions within the organizations were vacant when the capacity building was to begin.  There were no conditionalities placed on the money the organizations received from the U.S. Government. As a result, the incentives were not there to make changes or accelerate reform.  The working culture of the GOKP and the PMU was, at times, problematic. 113 ASP-AiD, Report on Validation of PMU-MSP, Khyber Pakhtunkhwa (March 2014). 158  ASP-AiD technical support was not competent in the area of M&E. MSP had to rely on the help of USAID’s M&E team to fill the gap left by ASP-AiD.  ASP-AiD did not have the ability or knowledge to move things forward for approval. There were significant delays in hiring MSP staff, and ASP-AiD could not move that forward. 7. Key Observations PMU-MSP appears to be in a position to continue their work. They have key personnel in place, and they believe they will be able to move MSP efforts into more districts beyond the original three. The GOKP funds the administrative costs of PMU-MSP; all other resources are paid for by USAID funds. Everyone interviewed within the organization felt that if ASP-AiD support was no longer available to them, there would be no negative effect. 159 Case Study on Capacity Building of Municipal Services Delivery Program–Sindh (MSDP-Sindh) Evaluation Team: Mr. Marc Shiman, Technical Director Ms. Vardah Malik, Evaluation Consultant Interviewees: Khalid Hyder Shah, Program Director Muhammad Naeem Waheed, Procurement Specialist Mohammad Arab Shaikh, Director General (Works) Asad Zamin, Director Finance 1. Purpose The purpose of this case study is to determine the extent to which ASP capacity building services in financial management, procurement, HRM, and M&E, are/were useful and implemented (effective and institutionalized). 2. Background The Municipal Services Delivery Program (MSDP) in Sindh is a joint project of the Government of Sindh (GOS), represented by the Planning and Development Department, and the U.S. Government, as represented by USAID. Its purpose is twofold:  To improve basic service delivery and to provide services in a more transparent and accountable manner, and  To upgrade provincial information systems for planning, operations, and M&E. The first intervention for ASP-RSPN was to assist the PND to complete the PC-1, which is required by Pakistani law. ASP-RSPN provided a specialist who helped the GOS to complete and submit the PC-1 in July of 2011. The review by the GOP took over a year before it was approved in September of 2012. Upon its approval, the Government formed a PMU called MSDP. Prior to its formal approval, the Government appointed four civil servants to lead the development of the project in July 2012. ASP began its capacity building efforts at the start of the project. 3. Pre-Award Assessment USAID began its intervention with the GOS by conducting a pre-award assessment. The results of the assessment suggested that there were many areas of the PND that represented high risk to USAID. As a result, USAID engaged the ASP and selected RSPN as the IP. Because MSDP was a new organization, there was never an assessment of the direct beneficiary of the capacity building program. 4. Planning The Risk Mitigation Framework that resulted from the pre-award assessment identified the weaknesses of the PND. Because MSDP was to be a new office, USAID engaged RSPN to build the systems within the organization, particularly those lacking in the PND. ASP-RSPN completed a capacity building plan for MSDP-Sindh. The plans called for the supply of technical assistance for an undetermined time in MIS, M&E, HR, compliance, Internal Audit, and Procurement. The second part of the plan called for technical assistance in developing “frameworks” through a process of accommodating GOP, GOS, and U.S. Government rules and regulations, and incorporating focus groups. 160 5. Implementation of Capacity Building Except for providing technical assistance for the development of PC-1, ASP-RSPN did not mobilize the technical assistance from the outset as envisaged by the plan. ASP-RSPN did embark on developing manuals for MSDP. Initially, the MSDP leadership was resistant to the notion of manuals, as the Government already documented all of the rules. MSDP leadership felt that the process would be duplicative of the existing manuals. Nonetheless, the leadership acquiesced to the intervention as part of the conditions of USAID’s participation. ASP-RSPN engaged consultants in each of the areas of HR, Finance, Procurement, and M&E to pull together manuals. The consultants worked closely with their counterparts in MSDP (in HR, Finance, and Procurement; MSDP did not, and still does not have an M&E specialist on staff) to create manuals that comply with the rules and regulations of the GOP and the GOS. While the team initially was resistant to the notion of these manuals, they report that they now appreciate the condensed version of the rules into single manuals, although they feel that these manuals do not represent anything new. No training took place on the manuals, as MSDP still has not hired its workforce. The MSDP leadership felt that it was unnecessary to have training on the manuals, as they participated in their design. While the manuals were being created, the leadership of MSDP participated in a few courses sponsored by ASP￾RSPN and several implemented by ASP-LUMS. They reported significant learning in those courses. Only one staff member other than the leadership has taken a LUMS course, as reported by the MSDP team. The MSDP leadership said the most valuable intervention was in the area of M&E. The long-term civil servants had always collected output data in the past as part of their efforts in government, but had never learned about the components of what makes an effective M&E system. They had never been exposed to the notion of outcomes, having always counted outputs for Government. It is hard to identify whether the combination of the training and manuals have resulted in changes, as the organization did not exist when the intervention began. The MSDP leadership said that it follows the manuals to the letter (something the evaluation team could not verify), but that because what was in the manuals represented the letter of the law, they would have followed them anyway. The M&E manuals, which the leadership of MSDP identified as being the most valuable, appear not to have made any impact on their procedures, as they continue to measure outputs only. No validation has taken place since the intervention by ASP-RSPN. Because of MSDP’s difficulty in hiring staff, they have stated that they intend to take up contract employees from the ASP-RSPN project for a period of 12-18 months. 6. Challenges MSDP is short-staffed with a current workforce of nine officers and 11 subordinates. The sanction of a requested 24 officers and 44 subordinates has been granted, and they need to hire all of them before the end of the MSDP, in the next one and a half years. Not only must they hire staff, but train them and familiarize them with the manuals. MSDP seems to need ASP engagement more now than before. 7. Effectiveness It is difficult to gauge the effectiveness of the efforts, as this is a new organization and, therefore, there are no baselines against which the evaluation team can compare their current program. The leadership of MSDP have praised the manuals and the training, but have admitted that they would not be doing things differently if they did not have the ASP-RSPN intervention. They have mentioned that the trainings at LUMS and those sponsored by ASP-RSPN were very valuable, but they did not cite any changes they made as a result of their learning. The one aspect of the ASP intervention cited by MSDP leadership that might eventually pay dividends is that the existence of the manuals will make onboarding non-civil servants considerably easier, particularly those unfamiliar with current government rules and regulations. However, since the majority of the staff has not yet come on board, this is yet to be seen. 161 8. Sustainability The leadership of MSDP feels confident in their ability to update the manuals as needed, including when GOS rules and regulations are updated. The team is quite experienced, and because they closely participated in the design of the manuals, they feel familiar enough with them that making changes should come about easily. The same characteristic of the process, the leadership’s participation in the manuals’ design, should also have a positive impact on the likelihood that they will be implemented. However, that is no guarantee that they will be implemented; if the leadership of the team chooses to sidestep the rules and regulations of the GOS, then the presence of the manuals, regardless of leadership’s buy-in, will not control behaviors. MSDP leadership has stated that it does not intend to train its staff on topics that were delivered by ASP-RSPN or LUMS, but will send staff to future trainings. This means that the sustainability of the training will be largely limited to those few individuals that participated and remain in government employ beyond the life of the project. 9. Gender The MSDP leadership felt that there was very little mention of gender in the trainings or in the manuals and that their own policies (those of GOS) were probably more stringent than what was recommended in the manuals. 10. Conclusions USAID’s role in setting up the MSDP may ultimately be critical to the success of the project, but at this point it is difficult to see where the ASP project has added value through its capacity building. The leadership of MSDP spoke highly of USAID and the project, but had difficulty citing specific examples of where the project has had an identifiable impact on the organization. 162 Case Study on Capacity Building of Jamshoro Power Company Limited (JPCL)/GENCO I Evaluation Team: Vardah Malik, Evaluation Consultant Haider Ali Daud, Evaluation Consultant Interviewees: Iftikhar Aziz, CEO (on phone) Shams Ul Arfin, Finance Director and Company Secretary Abdul Satter Mullah, Deputy Manager, HR (on phone) Engr. Rustam Ali Ghouri, Additional Manager, ICB/International Procurement Syed Muneer Hussain, Deputy Director, MIS Muhammad Hussain Memon, Internal Audit Consultant/Head of the Department Mujahid Hussain Channa, Assistant Audit Officer 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Background As part of the reform and restructuring of the Pakistan Power Sector, the GOP entrusted Pakistan Electric Power Company (Private) Limited (PEPCO) with the task of managing the transition of the Water and Power Development Authority (WAPDA) from a public sector entity to a corporate and a commercially viable entity. The power generation companies (GENCOs) and distribution companies (DISCOs) were converted into autonomous and commercially viable enterprises through the induction of effective corporate management, best business and utility practices, and well-engineered systems to meet customers’ electric energy requirements on a sustainable basis. For its part, the U.S. Government announced a Signature Energy Program to increase electricity output and to conserve wasted energy, which involved the rehabilitation, refurbishment, and upgrading of units 1-4 at the Thermal Power Station in Jamshoro managed by Jamshoro Power Company Limited (JPCL), incorporated in 1998 as a public company limited. In 2010, a pre-award assessment of JPCL was commissioned by USAID to assess the financial management, procurement, and accountability environment at the company to determine the effectiveness of financial management capacity, internal control structure, procurement capacity, host country contracting, and fixed￾amount reimbursement arrangement capabilities for managing USAID funds in accordance with U.S. Government and USAID standards. The capacity building initiative under the ASP project was to respond to the gaps and issues identified in the pre-award assessment. 3. Initial Assessment/Analysis/Identification of Issues The pre-award assessment included a review of Governance and Management, Finance and Accounts (including budgeting and internal audit), Procurement and HRM. The overall assessment was ranked as moderate risk. Individual functions were rated as below:  Governance: Moderate risk  Accounting and Financial Management: Moderate Risk (accounting and financial reporting and internal audit within finance were ranked high risk)  Procurement System: High risk  HRM: Moderate Risk. It was noted that being a corporate entity, JPCL is under the direct control of PEPCO, which is an inherent limitation. JPCL is a fully owned company of the GOP with the direct involvement of PEPCO, which necessitates 163 that any systems and procedures adopted be in line with those at WAPDA/PEPCO, including the financial powers. While the organization was ranked moderate to high risk on the above functions, it was largely due to its limitations as a public company to adopt international best practices and code of corporate governance. 4. ASP Strategies and Interventions ASP-AiD was assigned to support JPCL/GENCO-I in line with the strategic objective of the ASP project to increase its management capacity. It was hoped that since JPCL was one of 14 corporate entities created to reform Pakistan’s energy sector, the tools developed and practices established were to have the potential to be adopted by other, similar corporate entities. In response to the risk ratings assigned in the pre-award assessment on the four core areas of governance, financial management, procurement, and HRM, a capacity building plan was developed in 2012. The execution of the plan entailed different capacity building initiatives planned to contribute to policy/procedure manuals’ improvement, training programs, and support by providing temporary experts and professionals to JPCL. The capacity building plan had the following key action points:  Review of Pre-Award Assessment Report and implementation arrangements.  Governance and Management: Since the Board of Directors at JPCL could not perform its mandated function being a public sector company, ASP-AiD will facilitate review of the existing governance structure and policies to determine the gaps and make recommendations. PEPCO was abolished last year and a GENCO Holding Company was created in its stead. ASP-AiD was also to facilitate implementing an improved governance structure and policies.  Accounting and Financial Management System: With the overall risk rating at moderate, some functions were considered to be high risk, such as accounting and financial reporting system and policies. Manuals and trainings were to be provided along with temporary staff support through short-term consultancies by ASP-AiD.  Internal Audit: Review and update of existing internal audit procedures will be facilitated by ASP-AiD in view of the high-risk rating assigned in the pre-award assessment. An internal audit manual, followed by trainings, were to be provided by ASP-AiD.  Procurement Management: Due to the absence of a specific procurement manual and non-compliance with FPPRA rules in some cases, procurement management at JPCL was considered high-risk. ASP-AiD was to facilitate preparation of a separate Procurement Policies and Procedures Manual and to suggest improvements with a special focus on addressing the shortcomings identified in the pre-award assessment. This would be followed by preparation of a training program and training of the procurement staff.  HRM: ASP-AiD will review and improve existing HR policies and practices. Subsequently the staff will be provided with training in improved policies and practices. In critical areas of capacity gaps, ASP-AiD will provide services of temporary staff if needed.  IT: GENCO I has manual systems for financial management, procurement, M&E, and HRM. ASP-AiD will assess the IT needs; design MIS, integrate HR, Financial Management, Procurement, and Assets Management; and provide IT training for relevant staff. Actual Implementation under ASP: While governance was later dropped from the capacity building plan, a fully operational ERP system was included under the IT component. Very few staff members were aware of the pre-award assessment and capacity building plan or had actually looked at it. Most of those interviewed were not aware of the risk ratings and the factors contributing to the ratings. Implementation of ASP was largely the development of the ERP system, manuals, and provision of trainings to the selected staff members at JPCL. 164 Manuals: Financial management: A Budget and Accounting Manual was developed in late 2012 for all three GENCOs supported under ASP. A consultant hired by ASP-AiD developed the manual independently. The draft was shared with the Chief Financial Officer, and his comments were incorporated into the manual by ASP-AiD. However, the manual is considered to be generic and not specific to the functions required by a GENCO. For instance, the manual talked about a public works accounting system, which was not required by JPCL and other GENCOs since they were all following international financial reporting standards post-incorporation as companies. Related to financial management, the ERP includes modules on accounts receivable, accounts payable, general ledger, inventory management, payroll, and plan maintenance. Data entry into the ERP system started a year ago, but the system has not been able to generate any reports, thus undermining the use of ERP as a complete decision-making support system. ASP-AiD was contacted to re-engage the consultants, but they could not help, as the consultants’ contracts had expired. JPCL also made a formal request to USAID seeking help on this. Internal Audit: JPCL’s scorecard against the action plan for Internal Audit gives quite a different picture from what we observed on the ground. As per the Action Plan, two out of three actions seem to have been achieved: assessment of the current mechanism of internal audit functions and development of an internal audit manual. It was observed that although there is an internal audit manual in place, internal audit staff interviewed reported that they are not aware of it. Procurement: JPCL followed FPPRA rules and still do. In addition to FPPRA rules, JPCL follows WAPDA purchase procedures for their procurements. After the development of the new procurement manual, feedback was obtained from the relevant staff members. The new manual covers all the relevant FPPRA rules. Once approved, the new procurement manual will, in principle, replace the WAPDA purchase procedures. HRM: The ERP system has an HR module. The content of the module was considered adequate. It does provide a sound decision-making support system. All GENCOs have reviewed and provided comments on the HR module that were incorporated. After it was developed in 2012, data entry continued for over one year, however, the software was never brought online. The company wrote to the Director of the USAID Energy Policy Project stating great concern that the payroll and HR modules were still not functional despite repeated requests. Furthermore, backup software for the system was not provided for reinstallation in case there was an error. The contractors had asked for a new contract to provide this support. IT: For the implementation of the ERP system (automation of HR, finance, procurement, assets management, inventory, and computerized maintenance management systems), a third party consultant was hired, who began implementing an ERP system called SAGE in early 2013. However, the process did not move beyond data entry after almost 15 months. Turnover of the CHHC team coupled with lack of collaboration by ASP-AiD not only caused a considerable waste of time and resources but has essentially put JPCL in a deadlock wherein they cannot proceed further, as the contract of the consultant firm ended around that time. A board for the GENCO holding company, under which all three GENCOs operate, has recently been formulated, and it must approve the manuals. The Board has not approved any of the manuals yet. Trainings Conducted: Financial Management: Around 15-20 employees were trained on the manual during a one-day training session. There are 50 people in the Finance Department. The training was actually an orientation to the manual and not training on the accounting function. The CEO and three assistants in the department attended the LUMS training in 2013. The feedback on LUMS training was positive, as it gave them an overview of the best practices in finance across the private corporate sector; however, there has been no change in the way they perform their duties, and no significant changes were witnessed. This is partly because of the pending ERP system implementation. It is expected that once ERP is operational, much of what has been learned through the training could be used. Thirty-five people have been trained on the ERP system; these are mostly the users of the system. ASP-AiD consultants provided the training. The training was good and included all relevant modules. 165 IT: The limited training conducted by CHHC for JPCL staff on the SAGE system was more of an orientation than a proper training. Even that has lost its value after the considerable delay due to multiple factors. CHHC did hand over some specialized manuals on SAGE; however, since the training was nominal only, they are hardly of any value. Internal Audit: The head of the Internal Audit Department is a short-term consultant himself, hired three months back with only three more months to go. As a newly recruited consultant, he has had no training. There is only one other staff member in the department who had received a session on internal audit and accounting by the ASP-AiD consultants. He thought the trainings should have been customized. None of them had received LUMS training. It was apparent that Internal Audit is not receiving due attention within the company. HRM: Five administrative staff, two accounts officers, and two people from Kotri power station went to Lahore for training by the consultants on the HR module under the ERP system. None of the HR staff has had any LUMS training. Procurement: Trainings were not received by any of the staff members in the department. Gender: Gender was not a priority at JPCL. Being a power company, the role of women in its core operations was limited, given that local women in Sindh do not obtain the skills required generally. Performance Measures: No performance measures were established to determine if new knowledge was gained or practiced by those using new manuals or those who had received training. 4. Operational Changes/Results Expected changes have not materialized due to the pending implementation of the ERP system that integrates all the four areas of intervention. JPCL appreciated the budget and account manual. The Finance Department and others were excited at how the ASP project will bring changes to the company, but the last year has been frustrating for them. Validation: None of the staff interviewed were aware of the validation, except for the CEO, who had not seen it yet. 5. Challenges Consultants visiting JPCL for ERP system installation changed frequently. Each time, they requested similar information that had already been provided to the previous consultant representatives, which frustrated JPCL staff. It is strongly felt that ASP-AID did not play its due role in resolving the situation and bringing about agreed-upon changes in the IT domain of JPCL, which, if properly implemented, could have changed the way day-to-day operations are conducted to everyone’s advantage. 6. Key Observations/Sustainability/Effectiveness Given the energy crisis in Pakistan, JPCL’s primary concern is to improve the power generation capacity of its two plants in Jamshoro and Kotri and also to meet targets under the USAID energy project. Nevertheless, JPCL is keen to make changes, to implement manuals, to get more training, and, finally and most importantly, to have the ERP system operational in the next six months. Their hands are tied, however, as they have no access to the consultants who developed the ERP system. All the heads interviewed had involved their relevant staff in data entry for over 15 months, but no reports have been generated so far to support decision-making for which the ERP system was developed. They also are unable to implement the manuals without approval, and turnover of staff in key positions has undone the efforts made so far under ASP. It seems partly the responsibility of JPCL management and to a great extent the inability of ASP-AiD to resolve issues. Willingness to implement the capacity building plan is there, but the capacity of JPCL is weak to do it on its own. 166 One interesting observation was that most of the staff interviewed considered JPCL to be a corporation. They wanted to move away from WAPDA financial, procurement, and internal audit-related rules to establish that they were very much an independent company that adhered to private corporate sector best practices. Sustainability: Implementation of the key deliverable, i.e., ERP, has not happened. Even after full implementation, trainings of more than 1,700 staff will be required at both power stations (Jamshoro and Kotri). It was requested that future trainings should be on site and on-the-job; otherwise, sustainability of this intervention would be in question. Specialists will be required to update/edit the manuals, and significant mentoring is required on ERP system. It seemed that little effort was made by JPCL to institutionalize changes brought about through ASP interventions. It was also obvious that ASP-AiD engagement was not very inclusive and persistent. This is required for a technically driven organization like JPCL, which places importance on its core activity of power generation, and its key people need to be drawn into institutional reforms through continuous engagement. Recommendations:  The budget and accounting manual should be approved either by the board of the GENCO holding company or by all three GENCO boards, as it is fairly standard for all three GENCOs. It was requested that ASP-AiD should assist in getting these approved.  The ERP system is expensive, and considerable time and effort has been made by JPCL to enter data over the last year. It would be a waste of time and money if there were no follow-up to ensure it is operational.  Better consultants could have been hired. Consultants working on the budgeting and accounting manual did not have expertise in power projects. The level of reporting by a power company needs output-based costing; the consultants produced simple Excel sheets to report on those. This had already been done by the Finance Department.  JPCL needs an expert on the HR module for 6 months in order to run the module and use it effectively.  Since JPCL is a power company, fire safety and health training should be part of the capacity building. 167 Case Study on Capacity Building of Urban Policy Unit (UPU), Khyber Pakhtunkhwa Evaluation Team: Dr. Janet Robb Mr. Haider Ali Daud Khan Interviewees: Muhammad Zubair Asghar Qureshi, Executive Director Shandana Azizullah, Internal Audit Officer Faiz Muhammad, Past HR Officer Humayun Hilal, Communication Specialist/HR Officer Hina Gul Wazid, M&E Officer Tariq Mehmood, Procurement Officer Sami Ullah Khattak, Senior Finance Officer M. Mohsin Khan, ASP-AiD Mudassar Salim, ASP-AiD 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Context The major objective for the establishment of the Urban Policy Unit (UPU) is to work in an integrated manner towards improving urban governance in the province of KP. Its focus is on achieving integration at urban planning and targeting capacity development of line departments. The intent of the UPU is to work as a demand responsive unit, firewalled from implementation, which will be resourced with sector experts capacitated to respond to the needs of the provincial departments and entities. UPU is to formulate economic regeneration and growth strategies for urban clusters and urban regions, maintain urban data sets and GIS, and generate research and disseminate innovations in urban sectors.114 The mandate of UPU is as follows:  The broad purpose and mandate of the UPU is to work on the formulation and improvement of integrated and strategic Urban Policy, Urban Planning, and Urban Management (institutional), including an urban regulatory (legal) framework.  UPU will ensure and lead integrated interdepartmental, inter-provincial and inter-city coordination for synergies and integration. This also includes coordination with donors on urban matters.  UPU will ensure that the line departments do not lack capacity to ensure implementation of solutions proposed, and it will provide capacity building and technical support to line departments in managing urban affairs and institutional reforms.  UPU will formulate economic regeneration and growth strategies for urban clusters and urban regions.115 3. Initial Assessment Being a new set-up within the Planning & Development Department of GOKP, there was no pre-award assessment of the UPU. According to ASP-AiD, “There were no employees or projects in place on which to do a pre-award assessment.” However, a UPU was called for in the PC-1 for the PMU-MSP in KP. 114 UPU official website. 115 Ibid. 168 The role of ASP-AiD was understood to be helping UPU get started, providing some training if needed, and assisting with the development of policy and procedure manuals. UPU/KP was modeled after UPU/Punjab. ASP-AiD hired UPU/Punjab to write the PC-1 for UPU/KP. Therefore, it was clearly understood what procedures needed to be put in place: those based on the UPU/Punjab model. 4. ASP Strategies and Interventions Capacity Building Plans ASP-AiD initially devised a capacity building plan for UPU in June 2011. The plan was to be implemented by the end of 2012; however, due to considerable delays in the approval of the PC-1 and then recruitment of required staff, ASP-AiD submitted a revised capacity building plan to have the interventions extended by six months, i.e., until the end of June 2013. However, when asked, no one in the organization had ever seen a capacity building plan—initial or revised. Technical managers reported that it was “everyone’s understanding that capacity building was to take place through on-the-job training, mentoring, and courses at LUMS.” Capacity building was delivered in the form of policy and guideline development (manuals) and training related to the content of the manuals and how to use the manuals. Until positions within UPU could be filled through a recruitment process, ASP-AiD supplied the personnel for the positions. Trainings Conducted Training by ASP-AiD was carried out in one of three ways:  In-house consultants;  Mentoring through emails, phone calls, person-to-person; or  LUMS courses. The schedule of courses offered by LUMS was made available to UPU, and the UPU Executive Director made nominations to LUMS of staff to attend various courses. However, it was reported by the Executive Director that LUMS rejected candidates if they were not from a particular technical area; in other words, the Executive Director felt that LUMS did not allow for cross-training of staff. As articulated by one interviewee, in general, ASP-AiD felt that LUMS training was “too academic, theoretical, and boring to most participants.” They felt that it needed to be of a more practical nature to truly support the ASP goals, and they advocated for more “mentoring” as opposed to classroom training. Prior to UPU having a Procurement Office, ASP-AiD helped others on the UPU team to step in and understand the processes needed for procurement. UPU technical managers stated, “ASP-AiD was there to help whenever we needed them.” When UPU needed to hire consultants, ASP-AiD would come to help write job descriptions, to advertise for positions, to interview consultants, and to vet the applicants. For the Internal Audit function, ASP-AiD hired KPMG to provide on-the-job training to the UPU Internal Auditor. KPMG had someone on site for more than two months. ASP-AiD felt that more training on the “governance side” should be done with UPU staff. However, this was thought to be outside of their mandate. They also felt that training of government officials should be done to support the functions of UPU when it transitions from a PMU. Again, they felt this was outside of their mandate. 169 Those UPU officials interviewed felt fairly strongly that “even if training wasn’t there by ASP-AiD or LUMS, staff would have found ways to learn what they needed.” There were no dedicated trainings on Gender. Manuals Introduced/Developed ASP-AiD reported that, headed by subject specialists at AiD, the best practices in HR, Finance, Procurement, and M&E were indigenized to UPU. Manuals were prepared by ASP-AiD and handed over to UPU. The UPU Executive Director reported that the manuals prepared by ASP-AiD were not extremely helpful. The technical area managers concurred, stating that “sitting inside the government and using the government funds, we must follow the government rules. Therefore, the originally drafted manuals, which contained some innovation and new thinking, were not approved. They were too idealistic.” UPU officials moved forward with their jobs without manuals. They followed government rules and regulations but did not need them written down in a manual. The manuals were tied up for a year and a half in the approval process. This delay in manual approval seems to have been a result of ASP-AiD hoping to infuse some procedures that were in addition to those of the GOKP. The ASP-AiD HR manual proved to be particularly problematic. A consultant for ASP-AiD wrote the manual without consulting UPU. It was delivered to UPU and sent forward for vetting and approval. Too many changes were required, and it was sent back to UPU, which sent it back to ASP-AiD, which sent it back to the consultant, and this continued many times over. The UPU HR Officer was never allowed to speak with the ASP-AiD consultant directly. This was extremely frustrating and seemed to UPU to be unnecessary. “Frustration was all around about these manuals.” Although the manuals are completed, they are nothing new— they only restate existing GOKP policy and practice. Only the M&E manual is not required to follow government regulations, since there is no precedent that exists within the GOKP. However, UPU personnel found the ASP-AiD-provided M&E manual to be too basic. UPU also felt that ASP-AiD has not been very helpful in helping them fully understand the USAID-required M&E process. Positions Trained Training in one form or another was provided to senior management in the areas of HRM, Finance, Procurement, M&E, and Internal Audit. Performance Measurement Criteria There were no performance measurement criteria established for the capacity building needs of UPU. 5. Operational Changes UPU/KP is an entirely new entity; therefore showing changes in operations is not relevant. However, UPU does appear to be a motivated group of professionals who believe in the value of their organization. An enthusiastic Executive Director leads them, and it appears that this keeps them working as a coherent team. There was a stated willingness to support one another and to strive to make UPU a lasting entity. 6. Challenges  The manual development and approval process was a huge challenge. Because ASP-AiD wanted to develop the manuals, and they included more than GOKP rules and regulations, the approval process was long and drawn out. 170  A challenge of UPU, in general, was winning over the support of other GOKP departments/agencies, which felt that UPU might be taking over their mandates.  Beyond organizational management, the UPU staff stated that they felt UPU is too “technical” for ASP capacity building assistance. In other words, their capacity building needs in the areas encompassed in urban policy and planning are outside of ASP’s mandate. This is felt to be true even in the area of M&E.  Areas that still need support:  M&E. Training for all staff is required since the technical teams need to write their own log frames.  Record Keeping. UPU is finding it difficult to keep track of their myriad documents when some are electronic; some are in paper form, etc.  Communications/public relations. UPU has the need to communicate effectively with the public regarding its various activities to gain buy-in, support, and understanding.  Internal Audit still needs guidance and training.  UPU is not aware of the capacity building plan developed by ASP-AID. This seems to reflect lack of coordination/communication on one end and absence of measurability of progress on capacity building on the other end.  ASP-AID consultants were not felt to be conversant with the requirements/needs of UPU, and their recommended manual versions had to be considerably revised. 7. Key Observations Probably the most significant statement of sustainability is documented in the PC-1 for UPU: During the first five years, it shall be run as a Project Management Unit, to be financed from the Development Budget. However, subsequently, the Unit shall be converted into a permanent entity whether as a separate department/authority or within the P&D Department. The exact nature and structure of that would depend upon requirements at that time. The logic of running it in the project mode for five years is to allow flexibility for innovations and research, which is usually difficult in a standardized department mode.116 In addition, the following points support the notion that UPU will be a sustainable entity:  Government pays all staff.  UPU started with equal funding from USAID and GOKP; USAID funding is now less than 10 percent of the total.  UPU drafted a Physical Planning Law for KP, and UPU is identified as the policy and regulatory unit. Passage of this law will secure UPU as a permanent entity.  UPU is a strategic agency for urban planning with no restrictions placed on it by USAID with regard to the sectors in which UPU works.  USAID initially wanted UPU to be restricted to working in three districts. UPU refused and wanted the Unit to work in all districts and wanted no restriction of technical areas. GIS remains the only sticking point.  The process followed for the establishment of UPU speaks to strategies to ensure sustainability: 116 ADP. No 810 PC-1 171 1. UPU was identified in MSP’s PC-1. 2. Potential senior leadership was consulted in the very early stages of design, in collaboration with ASP-AiD and USAID. 3. The proposed Executive Director was part of most, if not all, decisions. 4. The proposed Executive Director took the PC-1 forward for approval. 5. The proposed Executive Director led a donors’ consultative dialogue. 6. The proposed Executive Director held consultative meetings with all GOKP line departments.  Plans are in the works for UPU to be a regular department within the GOKP, responsible for writing laws, establishing institutions, doing feasibility studies, developing master planning strategies for cities, etc.  UPU has multiple donors providing funds. UPU staff noted that apart from selective infrastructure gaps filled by ASP-AiD, they do not see much sustainability in the ASP interventions. The almost complete absence of a capacity building plan at UPU is an alarming indicator and does not bode well for sustainability. Finance focal persons said that as per the PC-1, USAID funding constituted around 50 to 64 percent of the total budget outlay of UPU. This largely covers consultancies and infrastructure-related expenses. All the recurring expenses (especially salaries, etc.) are borne by the government itself. Sustainability of UPU as a department seems promising considering its embedded position within the Planning & Development Department. However, the sustainability of ASP-AiD interventions remains a question mark as they have so far fallen short of making a mark, and moreover there has not been institutionalization of the changes made as a result of the program. 172 Case Study on Capacity Building of Small Grants and Ambassador’s Fund Program (SGAFP) Recipients Evaluation Team: Mr. Marc Shiman Dr. Janet Robb Ms. Vardah Malik Mr. Haider Ali Daud Khan Interviewees: ASP-AiD Nighat Kamdar, Manager Capacity Building Sana Hussain Faces Pakistan, Lahore Maria Jabeen, Deputy General Manager Ayub Ejaz, Manager HR Friends Development Organization, Gujranwala Abdul Khaliq, President Rahat Austin, Project Officer and Lawyer The Layton Rahmatullah Benevolent Trust, Karachi Abdul Shakoor, G.M. Administration IRADO, Hyderabad Imran Bhand, Head Monitoring & Evaluation Saranga Literary and Cultural Society, Hyderabad Ishaq Samejo, Chairman STEP, Mardan Ali Mohammed, HR Manager Fakhre Alam, Finance Manager Abaseen Peshawar Shahid Khan, Program Manager Asif Jan, Finance Manager Peace and Development Organization, Peshawar Furqan Ullah, Program Manager Hashoo Foundation, Islamabad Shahzad Ali Mirza, Manager Strategy & Proposal Development FIDA Foundation, Islamabad Kashif Khatak, Manager Operations Mohammad Wasif, Manager Rahma Islamic Relief, Islamabad Zulgarnain Ansari, Senior Programme Manager 173 1. Purpose To what extent were/are ASP services for capacity building in financial management, procurement, administration and HRM, M&E, and gender useful and implemented (effective and institutionalized)? 2. Context USAID/Pakistan’s Small Grants and Ambassador’s Fund Program (SGAFP) funds time-limited development projects under US$150,000 throughout Pakistan. Worthy proposals, selected from Pakistani NGOs, civil society organizations (CSOs), and communities, receive grants to support one- to three-year projects. A competitive proposal must meet the following criteria:  Be innovative and unique;  Be independently developed by the applicant;  Be consistent with and supportive of USAID/Pakistan’s strategy and objectives;  Be able to meet a specific programmatic need; and  Not be an advance proposal for a forthcoming USAID solicitation that will be competitively awarded. ASP-AiD is actively engaged in the capacity building of a large number of CSOs across Pakistan, with an intentional focus on SGAFP recipients. ASP-AiD’s key interventions include organizational assessments to identify gaps and assistance to fill those gaps through training, coaching, and the provision of policies and procedures manuals. To assess the effectiveness and institutionalization of the limited capacity building provided to the SGAFP awardees, the ASP Interim Evaluation team conducted interviews in Peshawar with representatives from three SGAFP awardees in KP, and conducted group interviews of three awardees each in Karachi and Islamabad, and two awardees in Lahore, for a total of 11 SGAFP awardees. As the ASP IP responsible for CSO capacity building, two representatives from AiD were also interviewed. 3. Initial Assessment USAID was to have all SGAFP recipients undergo a financial review by SGAFP prior to the award. For some, this was carried out as a pre-award assessment conducted by a local chartered accounting firm contracted by USAID/Pakistan. In some cases, however, it was the completion of a basic Institutional Assessment Questionnaire. In all cases, it was to provide each of the SGAFP awardees with some sense of where they had capacity gaps in light of the requirements for managing their USAID award. Of the 11 SGAFP awardees interviewed, three reported that they had no pre-award assessment done for them, five had full pre-award assessments conducted by KPMG, and three only completed the Institutional Assessment Questionnaire provided by SGAFP. As one interviewee noted, ASP-AiD conducted a baseline survey from a sample of 500 CSOs throughout Pakistan “in order to identify institutional strengths and weaknesses, and to gain a better understanding of organizational management issues in the CSO sector in Pakistan.” The findings were used to help identify training and capacity building needs for the CSO sector. In particular, two key findings set the stage for the interventions used to build the capacity of SGAFP recipients:  CSOs generally do not have documented policies, procedures, and systems for key organizational management functions such as financial, procurement, HRM, M&E, and IT. In addition, only a minority of board and staff members of CSOs have reportedly received training in these areas. 174  Professional accountants reportedly audit the majority of the CSOs annually. Automation of various organizational management functions is still limited to very few large CSOs. ASP-AiD reportedly received the pre-award assessments that were conducted on SGAFP recipients. They followed by conducting post-award assessments for each organization. These post-award assessments, along with their baseline survey information, formed “the basis from which all other activities were to flow.”117 ASP-AiD conducted joint meetings for all of the selected SGAFP recipients in three cohorts. Findings from both the baseline surveys and the post-award assessments were shared with the group, and plans for further strategies and interventions were presented. 4. ASP Strategies and Interventions Capacity Building Plans Capacity building plans, as a result of the pre- and post-award assessments, were not developed for any of the SGAFPs interviewed. Three organizations, however, had developed their own action plan and timeframe to rectify the risks identified in their pre-award assessments. In lieu of formal capacity building plans developed for ASP, organizations were presented with a schedule of ASP￾AiD workshops and LUMS courses. Organizations selected the appropriate workshops and courses and identified the participants. One organizational representative stated, “As a small organization, the core team just runs from one thing to another. It is impossible to really detail a capacity building plan.”118 Organizations commented that ASP-AiD was extremely diligent in following up with emails and phone calls to be certain that they selected people for the workshops and trainings being offered. They stressed that it was ASP-AiD who was really proactive, not them. With the exception of two SGAFP recipients, the majority of the organizations felt that they did not have the resources to continue capacity building of their staff in any formal way without relying on donor funds. As one interviewee explained, “If it was not fully paid, our organization would never have been able to afford such training.” One organization had developed an endowment fund that paid for additional training and capacity building needs of their core staff. The other also pays for training out of their own funds and has sent staff to Sustainable Development Policy Institute and to some international training. After training and as a result of their own internal assessment, one organization developed their own capacity building plan and submitted it to ASP-AiD for review and comment. However, at the time of the interview, they still had not received any feedback. Trainings Conducted Trainings were conducted by both ASP-AiD and LUMS to support the capacity building efforts of the SGAFP awardees. ASP-AiD held workshops at centralized locations on the topics of workplace ethics and code of conduct; procurement management; HRM; monitoring, evaluation and reporting; SGAFP compliance; and IT skills and computer use. LUMS course offerings were in financial management; procurement management; M&E; HRM and administration; and leadership and cultural change. 117 N. Kamdar, ASP-AiD. 118 S. Khan, Abaseen. 175 SGAFP awardees were provided with a schedule of ASP-AiD offerings as well as LUMS offerings. Each organization made their own selection of who should attend the trainings and which trainings they should attend. None of the trainings were mandatory for any of the SGAFP awardees. With the exception of the IT and M&E training, in general, it was reported that the ASP-AiD trainings were, overall, beneficial. One of the organizations is a training organization itself, and they thought that ASP-AiD training was some of the best they have come across, so far, due to its expertise in training and engagement level with the participants. Another had received training in similar areas of capacity building through another USAID-funded project, yet they felt that ASP-AiD training was much better, yet complementary. One interviewee stated, “The professionalism of both ASP-AiD and LUMS trainers was greatly appreciated.” Some reported that the IT training was not long enough and that the software that was introduced had some demonstrated flaws in implementation and application. The M&E training was viewed by at least one organization as not being comprehensive enough or not offered enough times in the schedule. They sought M&E training elsewhere. There was a consensus on the following points:  LUMS training was more specific to government rules and best practices.119  ASP-AiD training was more geared toward CSOs. Manuals Introduced/Developed ASP-AiD developed five manuals for use in the capacity building efforts for SGAFP awardees:  Internal Audit Manual for CSOs  Financial Management Manuals for CSOs  HR Manual for CSOs  IT Policy Manual for CSOs  Procurement Manual for CSOs These manuals were designed specifically for use by CSOs; they are not the same manuals that would be used by Government, and they were not intended to be. ASP-AiD and hired technical consultants developed the manuals. The intent of the manuals was to form a comprehensive set of guidelines and procedures for each of the technical areas and to make these available to the SGAFP awardees with whom they were directed to work. The manuals were then to be adapted, adopted, or otherwise modified to fit the particular needs of an organization. All SGAFP awardees reported that they received the set of manuals from ASP-AiD, some via email before the trainings and some immediately following the trainings. In all cases, the manuals needed to be modified to be appropriate and relevant for their individual organizations. The manuals are comprehensive, although at places there is a lot of content copied from the corporate sector that may not be directly applicable for the CSOs. Other than that, the manuals seem to be acceptable. Some observations/suggestions for improvement follow. 119 While the findings from the SGAFP case study suggest that ASP-LUMs training were relatively more specific to government rules and best practices, ASP-LUMS maintains that their courses were specific to both government and CSOs. 176 HR Manual  The HR Manual covers specific job descriptions of key officials of CSOs. Among these, a CEO is mentioned, although this is not an appropriate designation in CSOs, where the head of the organization is termed an Executive Director.  Organizational Structure is intentionally left blank for CSOs to fill in. It is suggested that the Manual should have contained an example to assist CSOs in developing/designing their structure.  CSOs can take different paths to registration as an official organization within Pakistan. The HR Manual should provide a description of the different options and provide guidance for each. Finance Manual The manual is overly comprehensive. Its applicability/relevance vis-à-vis the limited resources generally available within CSOs need to be considered so that they are not overly burdensome, yet still achieve the purposes of the manual. IT Manual Just like the Finance Manual, the IT investments expected to be made from CSOs prescribed in the manual need to be rationalized and made possible through discussion with stakeholders on the ground. Other Capacity Building Interventions ASP-AiD also made some site visits, but not to all SGAFP awardees. Due to the large number of SGAFP awardees, ASP-AiD capacity building interventions were mostly confined to the manual development and group trainings at centralized locations. However, as ASP-AiD has developed, they are now offering a set of online webinars and other online tools to be used in capacity building efforts available to all CSOs in Pakistan. Positions Trained In small CSOs, it is often the case that one official “wears more than one hat,” meaning that they are responsible for more than one functional area within the organization. In other CSOs, they have grown to such a size that it takes more than one person to handle all of the demands of a single functional area. While not always specified by position or title, ASP-AiD provided training to managers in the areas of HR, Procurement, Finance, M&E, and IT. LUMS did not provide training in IT, but they included Leadership and Cultural Change to their list of offerings for CSOs. Performance Measures There were no performance measures used to assess whether the trainings resulted in increased knowledge or not. Likewise, there were no measures to determine whether or not the appropriate people within a CSO received training, if it was compatible with their area of responsibility, or if any cross-training took place within the organization (e.g., procurement officials being trained in finance or HR). 5. Operational Changes In all cases, SGAFP awardees reported modifying their procedures and policy manuals/documents to be more comprehensive and appropriate for their organization. Modified policy and procedure manuals were approved by their respective governing bodies and have become instrumental in carrying out their day-to-day duties. Procurement seemed to be the area that saw the most significant operational changes. CSOs are now exhibiting more transparent operations and clearer guidelines. Most have introduced “slabs” or procurement thresholds for the first time. The process is competitive, with clear criteria and transparent selection results. 177 HR Management also reported significant changes. For example, timesheets are now required in most of the organizations where they were not previously; employee performance appraisal systems are being developed and used; employee files contain complete sets of documents from job announcements through termination; and gender policies are in existence and supported by policies on workplace harassment and a code of ethics. In Finance, changes were identified in instituting the use of a Master Payroll and financial software. Financial software has facilitated organizations ability to move away from simple Excel format reporting. However, the software selected for use varies, and the software introduced by ASP-AiD could not be made operational. There seemed to be fewer references to M&E policy and operational support made by the SGAFP awardees. Where comments were made, they were mixed. Some felt that the M&E support was useful, while others pointed to frustration. They felt that the support in this area was not detailed enough and did not help them to fully understand the process. All interviewed SGAFP awardees were questioned as to their approach to gender equity and sensitivity within their organization. Most reported that prior to a review of their policies and procedures, they were silent on the matter. Now they have statements referring to equal opportunities. In some cases, they have adopted a policy of stating in all of their job advertisements that “priority will be given to qualified females who apply for this position.” As a result of gender awareness through the ASP-AiD HR manual and trainings, organizations have also begun to adopt policies on sexual harassment and a respectful workplace. IT also proved to be another problematic area for change as a result of ASP-AiD interventions. While it was reported by two organizations that the IT support for new financial software was valuable, others reported that the ERP software introduced by ASP-AiD was not operational and that it has now been taken off-line by ASP-AiD. Some felt that the training in this area was simply too short and contained too little useful information. A positive by-product of ASP-AiD and LUMS trainings was the opportunity for participants to meet people with similar interests and jobs from organizations located throughout the country. This allowed participants the chance to establish informal networks where they can share information, seek guidance, and provide input. 6. Challenges There were two major challenges to how interventions were delivered: 1) there was no sequential order to the training, and 2) small and large CSOs were both represented in the same trainings.120 The lack of sequence left participants feeling as though there was no opportunity to build on what had been learned before, and they struggled with finding any logic in how the material was presented. When both large (more established) and small (newly formed) CSOs were grouped together for training, it was difficult to meet the training needs of both— either the larger CSOs were held back because of the lack of experience and knowledge of the smaller CSO, or the smaller CSO was “lost” and unable to keep up with the training. Another challenge to the capacity building interventions being delivered was the limited size of the core teams within organizations and the time to fit in training with all of their other responsibilities. It was also noted that training materials should have been prepared in multiple languages—not just English. It was difficult for those who did not have a good command of English to follow the training even when conducted in their local language, because they were unable to follow along in the manuals. A few awardees felt that the ASP-AiD training was not conducted at sites readily accessible to CSOs and their staff located in remote areas when training was held in the larger cities. They felt that training sites that were a bit more far-flung would be more appropriate. 120 As per the feedback received from ASP-LUMS, ASP-LUMS does not nominate participants for training. The process of nominating participants rests with AiD and RSPN. Final screening and approval of nominations is done by USAID. 178 There seemed to be few challenges to sustainability of changes within the SGAFP awardees. They saw the ASP interventions as appropriate and helpful to moving their organizations forward in their operational growth. A number of CSOs stated that they now have greater confidence in applying for resources from other donors and again from USAID because they feel they have the systems in place to prove their accountability and transparency. 7. Key Observations While appreciation for training opportunities was expressed by all CSOs, the underlying message was that training provided by ASP-AiD should have been more customized to individual CSOs or common groups of CSOs. The training should progress as the maturity of the CSO progresses—it should not be a “one size fits all” solution. Providing CSOs with comprehensive manuals in each of the technical areas gave each organization the ability to “pick and choose” what was most appropriate for their organization, after first reviewing all of the possibilities. The combination of ASP-AiD and LUMS training was complementary, with the former being more practical and the latter more theoretical. Without a project such as ASP, many CSOs would not have the resources to fund their own capacity building. ASP-AiD is perhaps an ideal conduit for establishing communities of practice for each of the technical areas. The concept would be to provide a network link for professional counterparts in the various CSOs to share knowledge, lessons learned, and best practices, and to ask questions of one another when they are faced with new challenges or new procedures. 179 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523