May 15, 2014 This publication was produced for review by the United States Agency for International Development by Ahmad Jameel, Zameer Haider, and Tariq Husain. It was prepared by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP). FATA Institutional Strengthening Project: Mid-term Performance Evaluation ACKNOWLEDGMENTS The authors would like to express their thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:  Officials of the Federally Administered Tribal Areas (FATA) Secretariat, the FATA Development Authority and the Khyber Pakhtunkhwa Governor’s Secretariat, who provided a wealth of information through interviews, focus group discussions and relevant documents;  The staff of Abacus Consulting, the FISP implementing partner, for demonstrating the automated systems they have developed and promptly providing documents and other information at all stages of the evaluation process; and,  USAID officials responsible for FISP, who facilitated the team’s work and shared their insights and information through interviews. FATA INSTITUTIONAL STRENGTHENING PROJECT MID-TERM PERFORMANCE EVALUATION May 15, 2014 Contracted under Order No. AID-391-C-13-00005 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Executive Summary................................................................................................................................................................................. 1 Project Summary ..................................................................................................................................................................................... 5 Program Background .............................................................................................................................................................................. 7 Evaluation Purpose and Evaluation Questions ...............................................................................................................................13 Evaluation Methods and Limitations .................................................................................................................................................14 Findings and Conclusions ....................................................................................................................................................................18 Recommendations ................................................................................................................................................................................35 Bibliography ............................................................................................................................................................................................37 Annexes ...................................................................................................................................................................................................38 Annex 1: Current Status of FISP Activities ..........................................................................................................................38 Annex 2: FISP Evaluation Statement of Work .....................................................................................................................40 Annex 3: Owners, Users, Data Providers and Current Status of Systems Developed by FISP .............................67 Annex 4: Data Collection Instruments ..................................................................................................................................70 Tables and Figures Table 1: Project Summary ........................................................................................................................................................... 5 Table 2: Individual Interviews with USAID/Pakistan Program Managers and Abacus Consulting ..........................15 Table 3: Individual Interviews with Owner-Cum-Users of FISP Systems .....................................................................15 Table 4: Individual Interviews with Other Users and Future Users of Systems .........................................................16 Table 5: Individual Interviews with Commissioners of Studies/Reports .......................................................................16 Table 6: Organizational Needs Expressed By Owner-cum-Users of the Systems .....................................................18 Table 7: Factors Facilitating Systems’ Progress in Meeting Organizational Needs ....................................................22 Table 8: Factors Inhibiting Systems’ Progress in Meeting Organizational Needs.......................................................24 Table 9: Steps Taken to Institutionalize Systems ................................................................................................................28 Table 10: Assessment of Seven FISP Policy-related Reports ...........................................................................................32 Table 11: Gender Balance in FISP Training ...........................................................................................................................33 Figure 1: Map of FISP Project Area........................................................................................................................................... 6 Figure 2: FISP Theory of Intervention ...................................................................................................................................... 9 Figure 3: Organizational Needs and Their Operational Meanings As Reported by System Owner-cum-Users20 Figure 4: Factors Facilitating Progress in Meeting Organizational Needs .....................................................................23 Figure 5: Factors Inhibiting Progress in Meeting Organizational Needs .......................................................................25 ii ACRONYMS ACS Additional Chief Secretary AI&C Administration, Infrastructure and Coordination COR Contracting Officer’s Representative DRAS Digital Resource Access System EAS Employee Attendance System FAMS FATA Assets Management System FATA Federally Administered Tribal Areas FDA FATA Development Authority FDMA FATA Disaster Management Authority FISP FATA Institutional Strengthening Project FMS Financial Management System FS FATA Secretariat FTS File Tracking System HRMIS Human Resource Management Information System IR Intermediate Result IT Information Technology KP Khyber Pakhtunkhwa LCMS Litigation Cases Management System M&E Monitoring and Evaluation MEP Monitoring and Evaluation Program MSI Management Systems International NOC No Objection Certificate NOCMS No Objection Certificate (NOC) Management System OPFS Online Public Feedback System P&D Planning and Development PC-FMS Planning Commission Forms Management System PMIAM Public and Media – Intercommunication and Awareness Mechanism PMP Performance Management Plan RFP Request for Proposal SOW Statement of Work SRS System Requirement Specification USAID United States Agency for International Development ZMIS Zakat Management Information System 1 EXECUTIVE SUMMARY The Federally Administered Tribal Areas (FATA) Institutional Strengthening Project (FISP) is a US$17.96 million USAID-funded project to support three FATA institutions responsible for governance and development, namely, the FATA Secretariat (FS), the FATA Development Authority (FDA) and the Khyber Pakhtunkhwa Governor’s Secretariat. The project started in August 2011 and will end in September 2016. The FS represents the core of the FATA administration, which extends to seven tribal agencies and six frontier regions. The FS is a relatively new entity consisting of six departments, each of which includes several subordinate units, most of which are called Directorates. The FS is responsible for public services in FATA, including but not limited to security, basic utilities, and infrastructure. The FDA executes specific development projects assigned by the government. The Governor of Khyber Pakhtunkhwa (KP) Province administers FATA as a representative of the President of Pakistan, who is the chief executive of FATA. The KP Governor’s Secretariat assists the KP Governor in coordinating the government machinery in FATA and KP. EVALUATION PURPOSE AND QUESTIONS FISP is a demand-driven project; its activities are initiated at the request of the above-mentioned FATA institutions and, in the case of some studies it has undertaken, also from USAID/Pakistan. Its support to FATA institutions includes the automation of office processes and improvement of related functions, strengthening information technology (IT) infrastructure, staff training, and policy-related studies. Over time, these interventions are expected to help FATA institutions become viable service-oriented institutions, managing their duties in a transparent and well-planned manner. This, in turn, is expected to eventually lead to improved service delivery in FATA. The purpose of this evaluation was to assess the project’s performance, determine the extent to which the project is proceeding as anticipated, and identify course-corrections for the remainder of the project in order to improve project implementation. The evaluation focused on five questions that can be summarized as follows: (1) to what extent are the systems developed and deployed by FISP addressing the organizational needs of the departments using the systems; (2) to what extent have FATA institutions institutionalized these systems; (3) to what extent are the policy-related reports prepared with FISP assistance useful; (4) to what extent are gender issues and concerns addressed by FISP interventions; and (5) recommendations for course-corrections for the remainder of the project. FINDINGS AND CONCLUSIONS The evaluation’s findings and conclusions are based on extensive review of qualitative data drawn from project documents, interviews, and focus group discussions. The evaluation team conducted 27 interviews with the owner￾cum-users, other users, and future users of the systems deployed with FISP assistance1 , and eight interviews with officials who had commissioned the policy-related studies. Three focus group discussions were conducted, two with a total of 28 male participants, and one with six female participants employed by FATA institutions who received training through FISP. Abacus Consulting Technology, Limited, hereinafter referred to as Abacus, the FISP implementing partner, demonstrated automated systems to the evaluation team before the start of data collection. The evaluation methodology helped capture the diversity of perspectives across multiple groups of stakeholders, including project managers, system developers, implementers, and current and future system users, and allowed 1 “Owner-cum-user” refers to a department or subordinate unit that manages a system that is used by the owner and, possibly, a number of other entities. “Other user” refers to an organization that is not an owner but uses a system within the limits of access provided by the owner. “Future user” refers to an organization that was not using a system at the time of the evaluation because the system had not yet been introduced in the department, data entry was incomplete, or the organization lacked the skills, resources, or willingness to use the system. 2 triangulation across stakeholder groups and secondary sources of information. Triangulation across different sources of information helped mitigate bias in respondent selection. By the time the evaluation was conducted, FISP had deployed 11 automated systems2 in FATA institutions. Respondents interviewed by the evaluation team identified efficiency, accountability, and transparency as the most common organizational needs to be addressed by the systems deployed by FISP. They defined efficiency, accountability and transparency in terms of the operational needs of the civil service and in the case of one system, accountability to the public for development projects. The evaluation report provides specific examples of government departments’ understanding of organizational needs and the utility of systems in addressing them in terms of day-to-day operations. The evaluation found that six of the seven systems in use were being used to address some or all of the organizational needs expressed by the owner-cum-user departments and other users. They included the File Tracking System, Employee Attendance System, No Objection Certificate Management System, Online Public Feedback System, Public and Media Inter-communication and Awareness Mechanism, and the Zakat Management Information System.3 A seventh, the Digital Resource Access System, was also working as intended, although it is meant for individuals and does not serve any particular organizational need. Four systems, the Planning Commission Forms Management System, Human Resource Management Information System, FATA Assets Management System and Litigation Cases Management System, were not in use at the time of the evaluation and are at various stages of implementation. The evaluation found that top-level support for the systems deployed so far is the most important factor facilitating implementation of the systems and progress toward meeting organizational needs. The training of staff and the provision of IT equipment and infrastructure have also contributed to progress. However, several factors have slowed implementation including the perception among owner-cum-user departments and subordinate units that five of the systems lack usefulness due to design-related issues. Other impediments are institutional issues, including lack of communication, motivation and staff capacity, high staff turnover, unclear roles and responsibilities, lack of IT infrastructure, and data entry and cleaning problems. The criteria used to assess the degree of institutionalization of the systems deployed included senior management support for the systems, availability of trained staff and resources, use of manuals, and the actual use of a system in the work of the department or organization. Based on these criteria, four out of the seven systems in use were determined to be institutionalized. The KP Governor’s Secretariat has fully institutionalized the File Tracking System and the FS has institutionalized it to a considerable extent. The FS has fully institutionalized the Employee Attendance System; the FATA Disaster Management Authority has institutionalized the No Objection Certificate Management System; and the FDA has institutionalized the Online Public Feedback System. A fifth, the Digital Resource Access System, was determined to not be relevant for institutionalization. The remaining six systems (the Public and Media Inter-communication and Awareness Mechanism, the Planning Commission Forms Management System, the Zakat Management Information System, the Human Resources Management Information System, and the FATA Asset Management System) are expected to be institutionalized based on the plans currently being pursued by the owner-cum-user departments. The recommendations of six out of the seven policy-related studies conducted with FISP assistance have been useful for their commissioners (USAID/Pakistan and/or the FATA Secretariat); only one fell short of expectations. All but one of the studies’ commissioners indicated their general satisfaction with the methodology and outcomes of the studies. None required any amendments or modifications to the study designs and methodologies. 2 These systems and the purposes for which they are intended are described in the main report. 3 Zakat is 2.5 percent of the accumulated wealth and property that an adult Muslim who is considered to be eligible to pay zakat according to specified criteria must pay annually to help the poor of their community. In Pakistan, in addition to zakat that is paid by individuals to recipients of their choice, since 1980 zakat is also collected by the government from savings bank accounts (having a balance above a threshold limit that is announced every year) and distributed to the needy through official channels, including provincial and district administrators. 3 The extent to which FISP interventions have addressed gender issues or concerns has been limited. Positive steps include progress toward partial implementation of the recommendations of the Women Empowerment Wing report, participation of FATA female staff in FISP training programs (women represent 3 percent of FATA staff and 4 percent of the training participants), and three automated systems that can produce sex-disaggregated data. However, FISP gender initiatives have been constrained by the demand driven orientation of the project and the fact that FATA institutions and USAID/Pakistan counterparts have not prioritized gender initiatives among their requests for FISP assistance. There is persuasive evidence in this evaluation that FISP’s demand-driven orientation has thus far turned out to be a double-edged sword. On the one hand, FISP’s responsiveness to demands from key stakeholders has elicited the highest level of government ownership and decisive action for introducing new systems and commissioning reports aimed at changing how government works in FATA. On the other hand, relying on its demand-driven orientation has kept FISP at a distance from the operating environment that will ultimately determine the success or failure of systems introduced by FISP, as well as from engaging proactively to support women’s participation and gender initiatives in FATA institutions. RECOMMENDATIONS In terms of course corrections, USAID/Pakistan should take the necessary steps (including consultations with FATA institutions) to enable FISP to adopt a more proactive posture for the remainder of the program and consider implementing a change management initiative to deal with the broader operating environment that affects systems implementation and utilization and gender issues. Such an initiative would enable FISP to better address the perceptions, expectations and recommendations of system users, as well as the role and participation of women in FISP interventions and in FATA institutions. Accordingly, USAID/Pakistan should also consider providing FISP with greater flexibility in planning, staffing, and budgeting in order to undertake course corrections and facilitate the formation of a committee including USAID/Pakistan, Abacus and government representatives to promote and monitor FISP progress. More specifically, USAID/Pakistan and FISP should take a more systematic, longer-term view of the implementation process. This entails assessing the lack of IT infrastructure in relevant departments and developing an action plan to overcome it with government and FISP resources for the remainder of the project, and then monitoring progress and flagging issues that require the attention of USAID/Pakistan and the government. Moreover, before introducing a new system, FISP should specify arrangements for data entry or data transfer and ensure that government counterparts are well-equipped to handle these responsibilities. In terms of training, FISP should promote merit-based nomination of trainees and arrange convenient training hours to address the mobility and safety considerations of women participants. As an enabler of systems and system users, FISP needs to address new requirements identified by system users and investigate the veracity of user perceptions about specific system￾related issues. In addition, attention should be given to managing the expectations associated with concepts such as transparency, accountability, and quality by focusing on clear operational definitions of these terms. FISP also should focus more systematically on institutionalizing and sustaining the systems deployed by the project. In this connection, FISP needs to undertake an across-the-board assessment and analysis of the resources needed and available to sustain the systems introduced. This analysis should be accompanied by cost estimates and an action plan to ensure that any resource gaps are covered. Finally, as indicated above, USAID/Pakistan and FISP should adopt a more proactive posture to address gender concerns and women’s participation in FISP interventions and FATA institutions. In this connection, it would be useful to discuss with government counterparts the provision of sex-disaggregated data for at least two more systems, the Online Public Feedback System and the Human Resources Management Information System and modify the systems accordingly. In addition, FISP needs to sensitize staff to the fact that training activities need to take into account mobility and safety considerations that particularly affect women. This may require shorter training hours during the day. FISP has an opportunity, based on the report it funded for the Women Empowerment Wing of the FATA Secretariat, to discuss the capacity building needs of the Women Empowerment Wing with the government, to provide appropriate support for capacity building, and to take forward the report’s recommendation to establish a FATA Women Empowerment Council. FISP also needs to make greater effort to 4 sensitize staff through training and situation analysis to support the role and opportunities for women in FATA institutions. 5 PROJECT SUMMARY The Federally Administered Tribal Areas (FATA) Institutional Strengthening Project (FISP) is a five-year project funded by USAID/Pakistan. FISP is assisting the FATA Secretariat (FS), the FATA Development Authority (FDA) and the Governor’s Secretariat in their efforts to develop their capacity through systems development,4 improvement of related functions, and the strengthening of the information technology (IT) infrastructure.5 According to the FISP Monitoring and Evaluation (M&E) Plan, the strengthened capacity of the FS and FDA will help achieve the strategic objective of ensuring their institutional viability as service-oriented government bodies capable of managing their duties in a transparent and well-planned manner. TABLE 1: PROJECT SUMMARY 4 This entails the automation of office processes, including the government’s planning process. 5 FISP M&E Plan, originally called the Performance Monitoring Plan 2011-2016, Draft 2, November 4, 2011. Assistance to the Governor’s Secretariat is mentioned in FISP progress reports but not in the M&E Plan. Project Name/Title FATA Institutional Strengthening Project (FISP) Agreement Number AID-391-C-11-00003 USAID Contracting Officer’s Representative Muhammad Anwar (Mr.) Project Start Date August 24, 2011 Project Completion Date September 6, 2016 Project Location Peshawar in the Khyber Pakhtunkhwa Province and the Federally Administered Tribal Areas USAID Objective Addressed USAID/Pakistan Mission Strategic Framework (approved July 30, 2013) Development Objective 3 “Increased stability in target areas,” primarily through contributing to the achievement of Intermediate Result 3.2 “Governance Improved.” Name of the Implementing Partner Abacus Consulting Technology (Private) Limited Budget US$17.96 Million 6 FIGURE 1: MAP OF FISP PROJECT AREA 7 PROGRAM BACKGROUND FATA is in the northwest frontier region of Pakistan, which also includes the Khyber Pakhtunkhwa (KP) Province. With a geographical area of 27,220 square kilometers, FATA has a population of approximately 3.18 million, according to the 1998 census and an annual growth rate of 2.19 percent.6 FATA consists of seven tribal agencies and six frontier regions, subdivided into 43 administrative units (called tehsils) and approximately 3,000 villages. The FATA region is represented by elected representatives in the National Assembly and the Senate of Pakistan, but does not have its own legislature, cabinet, or chief minister7 and comes under the direct executive authority of the President of Pakistan. It is administered by the Governor of KP Province in his capacity as an agent of the President of Pakistan, under the overall supervision of the Ministry of States and Frontier Regions in Islamabad.8 Pakistan’s justice system does not extend to FATA; civil and criminal cases are decided under the Frontier Crimes Regulations of 1901 by a jirga or council of elders.9 Until 2002, service delivery in FATA was managed through departments of the Government of KP. In 2002, a small FATA Secretariat headed by a Secretary for FATA was established in Peshawar, which is located in KP Province. In 2006, the Secretariat was upgraded to a full-fledged multi-departmental secretariat, headed by an Additional Chief Secretary reporting to the Chief Secretary of KP, who reports to the Governor of KP on FATA-related matters. The FATA Secretariat (FS) consists of six departments; each headed by a Secretary, and includes several subordinate units, called directorates, in most cases. It is responsible for the delivery of public services in FATA, including security (law and order), basic utilities, and infrastructure. The FATA Development Authority (FDA) is a smaller organization focusing on specific development projects assigned by the government, and is organized by departments. The Governor’s Secretariat assists the Governor of KP in coordinating the government machinery in development and political matters. PROBLEM OR OPPORTUNITY ADDRESSED The performance of the FS has a direct bearing on the state of affairs in FATA, with consequences for government legitimacy there. As a nascent entity, the FS requires support for institutional development. Realizing the importance of capacity building for FATA institutions, particularly the FS, USAID/Pakistan funded the FATA Capacity-Building Project that was completed in 2010. USAID/Pakistan continued its capacity-building support through FISP, which started in August 2011. The project aims to strengthen the ability of the FS and FDA to deliver essential government services in FATA by improving the human and technical capacity of FATA institutions, along with improving accountability, monitoring, state responsiveness, and effective communication with citizens. TARGET AREAS AND GROUPS The target groups of FISP include staff working in the FS, FDA, and the Governor’s Secretariat in Peshawar, as well as government officials based in the tribal agencies and frontier regions of FATA. FISP also provides IT-related services to the FATA administration based in the tribal agencies and frontier regions. 6 Information reported in this paragraph is taken from the official FATA website, www.fata.gov.pk. A population census has not been conducted in Pakistan since 1998. 7 Unless indicated otherwise, information in this and the next two paragraphs is taken from USAID’s Statement of Work in the Request for Proposals for the FATA Capacity-Building Program Phase II, RFP No. 391-11-000001, dated 2011. 8 Official FATA website, www.fata.gov.pk. 9 Official FATA website, www.fata.gov.pk. 8 INTENDED RESULTS AND THE THEORY OF INTERVENTION FISP supports USAID/Pakistan’s Development Objective 3, Increased stability in focus areas. According to the FISP Results Framework10, the strategic objective of the project is to ensure the institutional viability of the FS and FDA as service-oriented government bodies capable of managing their duties in a transparent, well-planned manner. The project is pursuing the following intermediate results (IRs):  IR 1: Improved capacity of the FS to govern through systems development, training, and strengthening of the IT infrastructure.  Sub-IR-1.1: Strengthened planning, monitoring and evaluation, budgeting, finance, human resources management, office management, and communication functions of the FS and FDA.  Sub-IR-1.2: Strengthened IT infrastructure and IT capacity of the FS and FDA. By achieving the above-mentioned intermediate results, the project expects11 to improve efficiency,12 transparency,13 accountability,14 and the quality of project plans, proposals, monitoring, and reporting in government departments. 10 This is part of the FISP M&E Plan, originally called the Performance Monitoring Plan 2011-2016, Draft 2, November 4, 2011. 11 The terms “efficiency,” “transparency” and “accountability” are mentioned in different contexts in USAID’s Statement of Work, in the Request for Proposals for the FATA Capacity-Building Program Phase II, RFP No. 391-11-000001, dated 2011, the FISP M&E Plan, and the minutes of the meeting between USAID and MEP on November 27, 2013 for planning the evaluation. The terms are not defined in the project documents. 12 USAID’s Glossary of Evaluation Terms prepared by Planning and Performance Management Unit, Office of the Director of U.S. Foreign Assistance, dated March 25, 2009, defines “efficiency” as “a measure of how economically resources/inputs (funds, expertise, time etc.) are used to achieve results.” In the context of this evaluation, however, efficiency has been assessed in terms of time saved, absenteeism controlled and other organizational needs of government departments. 13 Transparency International defines “transparency” as a characteristic of governments that are open in the clear public disclosure of information, rules, plans, processes and actions. Source: Mapping Transparency, Accountability and Integrity in Primary Education in South Africa published by Transparency International, dated 2011. In FISP, systems development is aimed mainly at making a department’s actions transparent within that department and to other relevant departments. 14 USAID’s Glossary of Evaluation Terms defines “accountability” as “an obligation to demonstrate what has been achieved in compliance with agreed rules and standards. This may require a careful, legally defensible, demonstration that work is consistent with the contract.” In FISP, the new systems are aimed at enhancing accountability of government employees and departments to government officials. 9 FIGURE 2: FISP THEORY OF INTERVENTION APPROACH AND IMPLEMENTATION FISP is a demand-driven project for which requests for assistance are received from the government and, in the case of some studies, from USAID/Pakistan. FISP undertakes its activities within the scope of the project in response to these requests. To achieve its results, FISP develops systems and manuals, conducts training, provides IT support, and undertakes studies. FISP has six components, the details of which are provided below. 1. IT Support: FISP is providing IT maintenance and support services to the FS and FDA. The key tasks in IT support include setting up a help desk at the FS and FDA, maintenance, repair and connectivity of the Local Area Network, and provision of IT hardware and IT training. It also includes support for strengthening the FS Data Center and FS Conference Room. FISP has also been responsible for preparing the FATA IT Policy and the corresponding IT Action Plan. 2. Form Automation: FISP is providing support for automation of the planning process, which is based on standard forms and guidelines of the Planning Commission of Pakistan. The automated system is called the Planning Commission Forms Management System (PC-FMS). The key tasks include development, testing, and deployment of and operational assistance for PC-FMS and the training of staff on the systems. Under this component, the project has also developed the FATA Web Portal, the Litigation Cases Management System for the FS, and a No Objection Certificate (NOC) Management System for the Directorate of Projects in the Planning and Development (P&D) Department and for the FATA Disaster Management Authority (FDMA). 3. Human Resources (HR) Support: FISP is providing support to strengthen human resource management at the FS, Governor’s Secretariat, and FDA. The key tasks include: development and deployment of the File Tracking System (FTS); Human Resource Management Information System (HRMIS); capacity building of FS and FDA officials; preparation of job descriptions; installation and expansion of an Employee Attendance System at the FS and FDA; digitization of FS files and archives; and the provision of a Digital Resource Access Facility. Improved service delivery for the people of FATA Greater efficiency, transparency and accountability in FATA institutions, better quality of plans, project proposals, monitoring and reporting, and more effective communication between FATA institutions and citizens Technical assistance to develop and implement systems and IT infrastructure for FATA institutions and capacity building of staff 10 4. Communication: FISP is providing support to build the capacity of the FATA Information Directorate and the Governor’s Secretariat to communicate effectively with FATA communities and local and foreign media. The key tasks include strengthening the communication function and public information/awareness and image-building programs for FATA communities. 5. Systems Development and Training: FISP is providing support for development and strengthening of the business processes of the FS and FDA. The key tasks under this component include: development and deployment of the Zakat Management Information System15, FATA Assets Management System, Financial Management System, and the Procurement System, as well as the training of FS staff on these systems. Under this component, FISP has also undertaken various studies/reports, including those on the strengthening of the Women Empowerment Wing of the FS, the management set-up of the Social Welfare Directorate, and a handbook on simplified zakat procedures. 6. Monitoring and Evaluation: FISP is providing support to develop a monitoring and reporting mechanism for the FS and FDA. The key tasks include the development of a monitoring and evaluation framework, software for performance data entry and reporting on service delivery, development of performance indicators and reporting forms, training of FS and field staff on newly-developed/improved monitoring and evaluation (M&E) mechanisms and processes16, and development of the Project Evaluation and Verification Module. AN OVERVIEW OF THE SYSTEMS INTRODUCED BY FISP17 File Tracking System (FTS): Formal communication in government offices in Pakistan is through files that contain all relevant documents on a particular matter, which are sent from one office to another, or within different sections of the same office. Under the current manual system, how a file is handled and how long it stays with a particular officer is not monitored so it is difficult to determine a file’s location or ensure timely attention. There are reports of government employees delaying the movement of files for personal reasons. The FTS is intended to electronically track the delivery, receipt, and location of files. The FTS also alerts the department head when an officer does not clear a file within the time specified. Employee Attendance System (EAS): Late arrivals, early departures, and unauthorized absences from the office are major personnel issues that affect efficiency of government offices. Attendance registers are maintained in offices, but do not inform senior management about staff attendance. These registers are also subject to tampering and proxy marking. The EAS requires employees to mark attendance on biometric devices that record their arrival and departure times. Reports are generated for senior officials responsible for monitoring staff attendance. 15 Zakat is 2.5 percent of the accumulated wealth and property that an adult Muslim who is considered to be eligible to pay zakat according to specified criteria must pay annually to help the poor of their community. This practice is one of the five pillars (basic acts) of Islamic belief. In Pakistan, in addition to zakat that is paid by individuals to recipients of their choice, since 1980 zakat is also collected by the government from savings bank accounts (having a balance above a threshold limit that is announced every year) and distributed to the needy through official channels, including provincial and district administrators. 16 FISP is developing a monitoring and evaluation manual that will describe data collection and reporting processes and mechanisms. 17 This section is intended to introduce various systems to readers who are not familiar with the way government works in Pakistan and how these systems are intended to improve the business of government. This is an impressionistic contribution from the report’s authors that may not necessarily reflect the understanding of FISP or the government. 11 No Objection Certificate Management System (NOCMS): There are several national and international organizations working or planning to work on development projects in FATA. Due to security concerns, and the need to track these initiatives, the government has mandated that organizations and their employees working in FATA obtain No Objection Certificates (NOCs) from the FATA Disaster Management Authority (FDMA). Due to the volume of applications, manually processing and tracking applications is time-consuming and inefficient. The NOCMS offers automated submission and processing of NOC applications, and enables applicants to follow their application status online. Online Public Feedback System (OPFS): There are few organized channels for public participation in development activities or the expression of views about development projects initiated by the government’s FDA. The OPFS offers citizens an online platform to provide feedback on development projects and offer suggestions for improving project implementation. Public and Media – Intercommunication and Awareness Mechanism (PMIAM): The Governor’s Secretariat is responsible for promoting the Governor’s activities in the local press and electronic media. During the Governor’s visits to FATA, the remoteness and inaccessibility of many tribal areas caused delays in communicating these activities to the Secretariat in Peshawar and then to the media. The PMIAM has provided the Secretariat with equipment and connectivity to perform its duties efficiently. The system also allows the Governor’s Secretariat to monitor FATA-related news coverage provided by television and print media. Digital Resource Access System (DRAS): There are not enough libraries in FATA for government employees and staff from teaching institutions to access the latest knowledge and information sources. The DRAS connects employees of the FS and staff from the Cadet College in Razmak (located in FATA), with the latest international knowledge and information sources through a web link managed by the Higher Education Commission of Pakistan in Islamabad. It is intended for individual use only and is not related to any particular department’s organizational needs. Planning Commission Forms Management System (PC-FMS): Four forms (or templates) introduced by the Planning Commission of Pakistan in the 1960s are used by government departments to propose, monitor, and evaluate public sector projects. The Planning Commission Form-I (PC-1) is the prescribed template in which government departments prepare project proposals for approval by higher government authorities. The Planning Commission Form-III (PC-III) has two separate templates, namely, PC-III (A) and PC-III (B). The PC-III (A) is used for reporting on physical targets based on Public Sector Development Program allocations. The PC-III (B) is used for monthly progress reporting on approved projects. The Planning Commission Form-IV (PC-IV) is the project completion report template, while the Planning Commission Form-V (PC-V) is used to prepare annual performance reports after the project completion report is submitted. In addition to these four project-related forms, the Planning Commission Form-II (PC-II) is used to propose and approve surveys and feasibility studies. The PC-FMS aims to automate the preparation and processing of these forms, focusing initially on the PC-I. The PC-I form, once completed, has to pass through various approvals by different government authorities within and outside the FS; hard copies require written sign off/approval. The preparation of the PC-I is a time-consuming process and requires relevant expertise. Officers responsible for approval often complain about the lack of quality of PC-Is. Approval delays are often due to failure to meet PC-1 requirements. There is also anecdotal evidence of tampering with approved PC-Is with mala fide intent. The PC-MFS offers the template online and provides examples from properly complete PC-Is, as well as explanations to facilitate completing the documentation. The PC-I approval process will also be channeled through the system, which will allow for faster approvals, as well as locking the document electronically once it is finally approved, thereby preventing access to or tampering with the document. 12 Zakat Management Information System (ZMIS): The Social Sector Department of the FS is charged with overseeing and monitoring the distribution of zakat to poor and deserving people in FATA. Reportedly, there were issues in recordkeeping and retrieval of information relating to zakat beneficiaries, as well as some manipulation of the service delivery process. The ZMIS has been designed to improve service delivery and increase efficiency and transparency in the management and distribution of zakat in FATA. Human Resource Management Information System (HRMIS): The majority of employees in the FS are on deputation (temporary transfer) from the KP Secretariat and the Federal Government. These employees are posted to the FS for a period of three years, after which they are repatriated (sent back) and replacements are sought. This practice requires managing the appointments, postings, transfers, and repatriation of almost 800 employees. The HRMIS will keep detailed records and generate reports relating to personnel management in the FS. FATA Assets Management System (FAMS): Inventories of assets in storage and assets issued to various offices and individuals in the FS are maintained in hard copy in multiple files, making it extremely difficult to access and track information. Officials sometimes procure items that are already in stock because inventory information is not readily available. It is also difficult to track the repair and maintenance status of assets through the manual system. The FAMS provides automated recordkeeping, making it possible to manage stock, allocate assets and plan procurement, repair, and maintenance of assets more efficiently. Litigation Cases Management System (LCMS): There are currently more than 300 active cases filed against the FS by employees or other parties in various courts. The Litigation Section of the FS, which has only three staff, represents the FATA government in court. Given the workload and the absence of a mechanism to track court hearing dates, staff occasionally fails to appear in court on the scheduled dates. This adversely affects the efficiency of the department and prompts negative remarks and findings by the courts. The LCMS is intended to provide an automated case management system that generates alerts on upcoming court hearings. 13 EVALUATION PURPOSE AND EVALUATION QUESTIONS PURPOSE OF EVALUATION FISP reached the halfway mark of its planned duration in February 2014. USAID/Pakistan sought an evaluation to assess the project’s performance, document achievements and propose course corrections for the remainder of the project. The evaluation aims to assess progress toward institutional strengthening in relation to the organizational needs of the three key FATA institutions, the FA, FDA and the Governor’s Secretariat. The broad organizational needs, which FISP identified through consultation with relevant institutions, are improving efficiency, transparency and accountability, along with improving the quality of project plans, proposals, monitoring, and reporting in government departments. The evaluation is also expected to document the government’s steps to institutionalize the systems introduced by FISP within its various departments. Institutionalization of systems is defined as an individual governmental department’s capacity to use a system, as well as the sustainability of the new system. In addition to assessing the systems introduced by FISP, the evaluation is expected to assess the seven policy￾related reports prepared with FISP assistance at the request of USAID/Pakistan and the FS. The evaluation Statement of Work (SOW) is included in this report as Annex 2. EVALUATION QUESTIONS The evaluation focuses on five main questions:  Question 1: How and to what extent are the systems developed by FISP and currently in use addressing the organizational needs of the departments that own and use the systems, and what factors explain progress or lack thereof in this regard?  Question 2: How and to what extent have the FS, the FDA, and the Governor’s Secretariat institutionalized the systems introduced by FISP?  Question 3: What has been the usefulness of the policy-related reports prepared with FISP assistance?  Question 4: What evidence is there of women’s participation and a concern for gender issues in FISP interventions?  Question 5: What changes could better serve the project objectives and the priority organizational needs of the FS, FDA, and the Governor’s Secretariat? The evaluation SOW included in Annex 2 includes explanations for each question, which are summarized below in the findings and conclusions for each question. 14 EVALUATION METHODS AND LIMITATIONS A three-person team, consisting of MEP evaluation staff and a consultant, conducted this evaluation. The evaluation team consisted of:  Ahmad Jameel, Evaluation Team Leader;  Zameer Haider, MEP staff and Evaluation Team Member; and  Tariq Husain, MEP Director of Evaluation and Team Member. Brief biographies of the team members are provided in Annex 5. In addition to its own data collection and analysis, the team received inputs from the MEP IT unit. DATA COLLECTION METHODS The information used in this evaluation is drawn from FISP documents and individual interviews, group interviews, and focus group discussions conducted by the team. The documents include project work plans and reports, the System Requirement Specification (SRS) documents for the systems developed by FISP, and seven studies completed with FISP assistance. The bibliography lists the documents reviewed by the evaluation team. Information provided by FISP through electronic messages is cited in relevant parts of the report and not included in the bibliography. The overall approach to sources of data and data collection is outlined in the getting-to-answers table (Appendix 1) of the evaluation SOW. FISP’s implementing partner, Abacus, demonstrated the systems it has developed to the evaluation team, which interviewed Abacus system developers and relevant managers with the help of a semi-structured instrument to better understand the objectives and uses of the systems. The evaluation team conducted 11 group interviews with Abacus. The interview instrument is included in Annex 4. The evaluation team also used a semi-structured instrument to interview USAID/Pakistan staff and key representatives of government departments that are either using or expected to use the systems introduced by FISP. The team conducted 35 individual interviews with USAID/Pakistan managers, FISP program managers, and government officials working with the FS, FDA and the Governor’s Secretariat. The instrument used in these interviews is included in Annex 4. To assess the seven policy-related studies, the evaluation team reviewed the studies using a specific review questionnaire and interviewed the commissioners of the study (and their successors, in some cases) with the help of a semi-structured instrument. Annex 4 includes both the review questionnaire and the interview instrument. The team also conducted three focus group discussions, two with men and one with women working in FATA institutions who have received training through FISP. Twenty-eight men and six women participated in these discussions. The focus group discussion guide is included in Annex 4. The sample for individual interviews is summarized in Tables 2-5 below. 15 TABLE 2: INDIVIDUAL INTERVIEWS WITH USAID/PAKISTAN PROGRAM MANAGERS AND ABACUS CONSULTING Organization Interviewee Number of Individual Interviews USAID/Pakistan Relevant Managers 2 Abacus Consulting Chief of Party, FISP 1 TABLE 3: INDIVIDUAL INTERVIEWS WITH OWNER-CUM-USERS OF FISP SYSTEMS18 System Owner-cum-User Departments/Subordinate Units Number of Individual Interviews 1. Zakat Management Information System (ZMIS) Social Sector Department, FATA Secretariat 1 2. No Objection Certificate (NOC) Management System FATA Disaster Management Authority (FDMA) 1 3. Litigation Cases Management System (LCMS) Litigation Cell, Administration, Infrastructure, and Coordination (AI&C) Department, FATA Secretariat 2 4. FATA Assets Management System (FAMS) 5. Digital Resource Access System Administration Wing, AI&C Department of FATA Secretariat 2 6. Human Resource Management Information System (HRMIS) 7. File Tracking System (FTS) 8. Employee Attendance System (EAS)  Services Wing, AI&C Department, FATA Secretariat  Governor’s Secretariat  FATA Development Authority 5 9. Public and Media Inter- communication and Awareness Mechanism (PMIAM) Governor’s Secretariat, Governor House 1 10. Planning Commission (PC) Forms Management System (PC-FMS) - Module 1: PC-I Planning and Development (P&D) Department 2 11. Online Public Feedback System (OPFS) FATA Development Authority 1 Total 8 15 18 “Owner-cum-user” refers to an organization that manages a system that is used by the owner and, possibly, a number of other organizations. For example, the Administration, Infrastructure and Coordination (AI&C) Department is the owner of six systems and part￾owner of a seventh. The Planning and Development Department is the owner of two major systems. “Other user” refers to an organization that is not an owner but uses a system within the limits of the access provided by the owner. For example, all departments are users of the FATA Web Portal, which is owned/managed by AI&C, and also have access for changing their respective web pages in the portal. . “Future user” refers to an organization that was not using a system at the time of evaluation because the system had not yet been introduced in the department or data entry was incomplete or the organization lacked skills, resources or willingness to use the system. The term “future users” in the evaluation report replaces “non-users” in the evaluation SOW with no change in meaning. 16 TABLE 4: INDIVIDUAL INTERVIEWS WITH OTHER USERS AND FUTURE USERS OF SYSTEMS System Other User Department Future User Department Number of Individual Interviews File Tracking System (FTS) 1. Directorate of Local Government and Rural Development 1. Directorate of Fisheries 2. Directorate of Forests 3 Planning Commission (PC) Forms Management System (PC-FMS) - Module 1: PC-I 1. Directorate of Health 2. Directorate of Education 3. Communication and Works Department 4. Directorate of Agriculture Extension 5. Directorate of Sports, Culture and Youth Affairs 6. Directorate of Livestock and Dairy Development 6 Total 1 8 9 TABLE 5: INDIVIDUAL INTERVIEWS WITH COMMISSIONERS OF STUDIES/REPORTS Title of Study Commissioner of Study/Successor Number of Individual Interviews 1. Study on Strengthening of Women Empowerment Wing in FATA Secretariat Social Sector Department 1 2. Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat P&D Department 1 3. Draft Local Government Regulations for the FATA Local Government Local Government and Rural Development Department 1 4. FATA Information Technology Policy 2012 AI&C Department 2 5. South Waziristan Agency Development Plan USAID/Pakistan 1 6. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA USAID/Pakistan 1 7. FATA Investment Plan Study USAID/Pakistan 1 Total 8 17 DATA ANALYSIS METHODS The evaluation team employed rigorous analytical methods appropriate to the individual and group interviews and focus group discussions. The team used a structured approach by deconstructing each of the first four evaluation questions into sub-questions and organizing data on all the assessed systems and studies according to these sub￾questions. This allowed for analyzing similarities and differences across systems and studies and aggregating the findings at a higher level to support the conclusions. The analysis of the qualitative data yielded an in-depth understanding of the level of improvements in terms of achieving efficiency, transparency and accountability, along with improving the quality of project plans, proposals, monitoring, and reporting in the FS, FDA, and Governor’s Secretariat. The team also compared the findings from the interviews with data from interviews with current and future users of the selected systems. The comparison provided further insight into the level of improvements, factors facilitating utilization of the systems, and factors hindering use of the systems. METHODOLOGICAL STRENGTHS AND LIMITATIONS The evaluation employed multiple levels of triangulation in terms of information sources and methods to ensure validity and reliability of findings. The individual and group interviews and focus group discussions helped the evaluation team gain valuable insight into project performance and to capture the diversity of how the owners-cum-users, other users, and future users viewed the FISP components. The purposive sampling used had the possible shortcoming of selection bias. Since the results of this evaluation are not intended to be generalizable, however, this does not compromise the findings in any way, particularly given that the method used captures the diversity of perspectives relevant to addressing the evaluation questions. To assess the automated systems, each department or subordinate unit (except two) was represented by only one individual, the person designated to manage the system and/or ensure compliance with it, which limited the range of diversity and validation. This was mitigated, however, through the diversity of perspectives across multiple other relevant stakeholder groups, including project managers, system developers, implementers and current and future system users, which allowed triangulation across stakeholder groups and secondary sources of information. Another challenge the team faced was protecting the anonymity of some participants. Since the implementing partner works closely with these individuals, the views expressed and reflected in the report could be ascribed to a particular person, given the limited number of people involved in each of the various systems. This lack of confidentiality pertains only to USAID/Pakistan and the implementing partner, however, and every effort was made to protect their identities. 18 FINDINGS AND CONCLUSIONS QUESTION 1 How and to what extent are the systems developed by FISP and currently in use addressing the organizational needs of the departments that own and use the systems, and what factors explain progress or lack thereof in this regard? The explanation provided for this question in the evaluation SOW requires attention to four issues: organizational needs as articulated by the owner-cum-users, needs expressed by other key users, the extent to which the systems are addressing these needs, and factors explaining progress or lack thereof in meeting organizational needs. According to the SOW, FISP had developed and deployed 11 systems (listed in Table 6) for which data entry had been completed or initiated by the time the SOW was prepared, one system in which it was expected to start, and one (a web portal) for which content development was underway. FISP had also started developing two systems after completing the “as is” analysis for these systems. 19 The evaluation focuses on the 11 systems listed in Table 6. TABLE 6: ORGANIZATIONAL NEEDS EXPRESSED BY OWNER-CUM-USERS OF THE SYSTEMS System Number of Departments/Subordinate Units Expressing Need For: Efficiency Transparency Accountability Quality of Plans and Projects (Shading indicates that the system has started meeting organizational needs) 1. File Tracking System (FTS) 3 2 2. Employee Attendance System (EAS) 1 1 3. No Objection Certificate Management System (NOCMS) 1 4. Online Public Feedback System (OPFS)1 1 1 5. Public and Media – Inter communication and Awareness Mechanism (PMIAM) 1 6. Digital Resource Access System (DRAS)2 7. Zakat Management Information System (ZMIS) 1 1 1 8. Planning Commission Forms Management System (PC-FMS) Module-1 (the PC-I Form) 4 3 2 19 “As-is” analysis is the analysis of an existing system and its problems on the basis of which a new system is designed. 19 System Number of Departments/Subordinate Units Expressing Need For: Efficiency Transparency Accountability Quality of Plans and Projects (Shading indicates that the system has started meeting organizational needs) 9. Human Resource Management Information System (HRMIS) 2 10. FATA Assets Management System (FAMS) 1 1 11. Litigation Cases Management System (LCMS) 1 1 Total Number of Reporting Departments and Subordinate Units 153 2 103 2 Notes: 1. The OPFS is the only system for which accountability and transparency mean accountability to and transparency for the public. In all other cases, the expressed need is for accountability of government employees and transparency within the government system. 2. The DRAS was not introduced as a result of any expressed work related need, and hence has not been evaluated for meeting any organizational need. Moreover, it is not an office automation system but a means for connecting individuals to the electronic library of the Higher Education Commission. 3. There is double-counting in the totals because three units of the Administration, Infrastructure and Coordination (AI&C) Department of the FATA Secretariat are the owner-cum-users of six systems (FTS, EAS, DRAS, HRMIS, FAMS, and LCMS) on which they provided responses, and the FATA Development Authority (for HRMIS and OPFS) and the Governor’s Secretariat (for FTS and PMIAM) are the owner-cum-users of two systems each. FINDINGS In participants’ answers corresponding to the first issue raised in question 1, i.e., the identification of organizational needs, improved efficiency featured 15 times among the needs identified by eight owner-cum-users20 and six other users and future user departments and subordinate units21 (Table 6). The need for improved accountability was mentioned 10 times as an expressed organizational need, by seven owner-cum-users22 and two other/future users.23 Two departments24 identified transparency, and two others25 mentioned the improvement of plans and project proposals as important organizational needs. 20 The eight owner-cum-user departments and subordinate units are the Services Wing of the AI&C Department, the Governor’s Secretariat, P&D Department, FDMA, the Zakat and Ushr Section of Social Sector Department, FDA, the Administration Wing of AI&C Department and the Litigation Section of AI&C Department. 21 Directorate of Local Government and Rural Development, Directorate of Health, Communication and Works Department, Directorate of Sports, Culture and Youth Affairs, Directorate of Livestock and Dairy Development and Directorate of Agriculture Extension, 22 The Services Wing of AI&C Department, the Governor’s Secretariat, FDA, P&D Department, the Zakat and Ushr Section of Social Sector Department, and the Administration Wing and Litigation Section of AI&C Department. 23 Communication and Works Department and Directorate of Livestock and Dairy Development. 24 FDA and the Zakat and Ushr Section of Social Sector Department. 25 P&D Department and the Directorate of Agriculture Extension. 20 Interviewees for eight systems26 described the operational meaning of efficiency in terms of reducing the time required for completing a work-related process. For the EAS, the need for efficiency was mentioned in terms of reducing staff absenteeism and ensuring punctuality. For six systems,27 interviewees articulated accountability as holding employees accountable for handling documents and records. For the EAS, however, accountability was expressed as holding staff accountable for attendance and punctuality. The operational meaning of accountability for the OPFS was described as making project implementation accountable to the public. The need for transparency was expressed as making development activities transparent to the public through the OPFS and bringing transparency in service delivery through the ZMIS. These findings are reflected in Figure 3. The evaluation found no evidence, however, of how respondents related the PC-FMS to improvements in the quality of any substantive aspect of planning and PC-I preparation. FIGURE 3: ORGANIZATIONAL NEEDS AND THEIR OPERATIONAL MEANINGS AS REPORTED BY SYSTEM OWNER-CUM-USERS Organizational Need Operational Meaning According to Interviewees Number of Departments/ Subordinate Units Efficiency 8 Reducing the time required for completing a work-related process 1 Reducing staff absenteeism and ensuring punctuality Accountability 6 Holding staff accountable for handling documents and records 1 Holding staff accountable for attendance and punctuality Making project implementation accountable to public 1 Transparency 1 Making development activities transparent to the public 1 Bringing transparency in service delivery Note: The total number of reporting departments/subordinate units is 19, including double counting, as some departments/subordinate units articulated more than one organizational need. 26 FTS, NOCMS, PMIAM, PC-FMS, ZMIS, HRMIS, FAMS, and LCMS. 27 FTS, EAS, NOCMS, PC-FMS, ZMIS and FAMS. 21 PC-FMS 113 PC-Is prepared by 17 departments/ subordinate units using the system. The submission, approval and locking of the approved document through the system that would address the organizational needs of efficiency and accountability is not yet operational. The system is likely to be fully operational from July, 2014. HRMIS Data on 473/843 employees entered. The data of remaining 370 employees are being entered as received from different department/subordinate units. System is likely to become functional in six months. The second issue in question 1 was about the sharing of needs between the owner-cum-user and other users and is assessed by comparing the needs expressed by the owner-cum-users with those expressed by the other users/future users. The evaluation found that this comparison was possible for only two systems, FTS and PC-FMS, which are being used by both owner-cum-users and other users/future users. However, no comparison of expressed needs was made for the other nine systems28 that do not have any other user departments/future users except owner-cum-user departments. The needs of improving efficiency and accountability identified by the owner-cum-users of FTS and PC-FMS were shared by five other users/future users.29 The third issue pertained to the extent to which the systems are addressing the organizational needs. The evaluation found that five systems30 were able to meet the organizational needs expressed by the owner-cum-user departments/other users, while a sixth (DRAS) was also working as intended, although it was not designed to serve any particular organizational need. The extent to which the six systems in use are meeting organizational needs is as follows:  The FTS is fully meeting the articulated need of efficiency in the Governor’s Secretariat and to a considerable extent, in the FS as well. In the FS, some departments are not using the FTS.  The EAS is reducing staff absenteeism and ensuring punctuality in the FS to a considerable extent, but has not fully covered the FS as data on some employees has not yet been entered and one biometric attendance machine is yet to be installed.  The NOCMS is fully meeting the need for efficiency in the FDMA, as all NOCs are now processed using the system.  The OPFS has started receiving feedback on projects from the public, although it is not always relevant (only five or six relevant messages out of 120 received in a year).  The PMIAM is also reported to be functioning, but only partially.  The ZMIS has been used for the allocation of zakat funds from the FS to the Agency Zakat Offices, and for data entry of zakat beneficiaries for 2011-12 and 2012- 13. It is meeting the organizational needs of efficiency and transparency, although not entirely in the manner articulated in the SRS for this system. Of the four systems that are not yet in use (PC-FMS, HRMIS, FAMS, and LCMS), the progress and likelihood of becoming fully operational are detailed below.  The PC-FMS has been used to prepare 113 PC-Is by 17 departments/subordinate units. However, the features of the system that enable online submission, approval, and locking the approved document are not yet operational. The 113 PC-Is were submitted manually for approval without using the system. The Additional Chief Secretary (ACS) and the P&D Department have advised all departments that from the next financial year (beginning in July 2014), any PC-I not prepared on PC-FMS will not be entertained. These instructions reflect the likelihood of the system becoming fully operational by that date. 28 EAS, NOCMS, OPFS, PMIAM, ZMIS, DRAS, HRMIS, FAMS, and LCMS. 29 Directorate of Local Government and Rural Development, Directorate of Health, Communication and Works Department, Directorate of Livestock and Dairy Development and Directorate of Agriculture Extension. 30 FTS, EAS, NOCMS, OPFS, and PMIAM. 22  HRMIS. The data of 473 FS employees out of 843 have been entered. According to the owner-cum-user, different departments/subordinates of FS need to provide data for the remaining employees. Data entry for these employees could take almost six months to complete, after which the system will become fully operational and useful.  FAMS. The data of all assets in FS have been entered and data cleaning is in progress. According to the owner-cum-user, the system is likely to become functional once the data cleaning has been completed and a formal order from the ACS has been issued.  LCMS. The data entry of approximately 200 out of 300 active court cases has been completed. FISP is assisting with the data entry process and no timeline could be determined for the system to become fully operational. The fourth issue in question 1 relates to factors facilitating or inhibiting progress. Interviewees highlighted the following factors facilitating the progress of systems in meeting organizational needs:  Top-level support (from the Secretary or department head, the ACS of FATA or a higher level);  Availability of trained staff; and  Availability of adequate resources. Participants affirmed that all 11 systems have benefitted from top-level support, eight from the availability of trained staff, and five from the availability of resources. Table 7 summarizes factors facilitating progress in meeting the organizational needs. TABLE 7: FACTORS FACILITATING SYSTEMS’ PROGRESS IN MEETING ORGANIZATIONAL NEEDS System Facilitating Factors Top-level Support Trained Staff Resources Available 1. File Tracking System (FTS)  In the FATA Secretariat    In the Governor’s Secretariat   2. Employee Attendance System (EAS)   3. No Objection Certificate Management System (NOCMS)    4. Online Public Feedback System (OPFS)  5. Public and Media – Inter communication and Awareness Mechanism (PMIAM)  6. Digital Resource Access System (DRAS)    7. Zakat Management Information System (ZMIS)   8. Planning Commission Forms Management System (PC-FMS) Module-1 (the PC-I Form)    9. Human Resource Management Information System (HRMIS)  In the FATA Secretariat    In the FATA Development Authority   10. FATA Assets Management System (FAMS)   11. Litigation Cases Management System (LCMS)  Reporting Number of Owner-cum-User Departments/Subordinate Units 13 8 5 23 The findings summarized in Table 7 are illustrated in Figure 4. FIGURE 4: FACTORS FACILITATING PROGRESS IN MEETING ORGANIZATIONAL NEEDS Participants cited the following factors inhibiting progress in meeting organizational needs:  Lack of IT infrastructure;  Institutional issues in the operating environment;  Perception that a system is deficient in some way (including design) or needs improvement; and  The data entry that is required to make a system useful has not been completed. Table 8 and Figure 5 summarize the factors inhibiting progress in meeting the organizational needs. 12 7 7 Top-level Support Trained Staff Resources Available Factors Facilitating Systems’ Progress in Meeting Organizational Needs – Reporting Number of Owner-cum-User Departments/Subordinate Units 24 TABLE 8: FACTORS INHIBITING SYSTEMS’ PROGRESS IN MEETING ORGANIZATIONAL NEEDS System Inhibiting Factors Lack of IT Infrastructure Institutional Issues Technical Issues or Need for System Improvement Data Entry Not Completed 1. File Tracking System (FTS)  In the FATA Secretariat    In the Governor’s Secretariat 2. Employee Attendance System (EAS)  3. No Objection Certificate Management System (NOCMS)  4. Online Public Feedback System (OPFS) 5. Public and Media – Inter communication and Awareness Mechanism (PMIAM)    6. Digital Resource Access System (DRAS) 7. Zakat Management Information System (ZMIS)    8. Planning Commission Forms Management System (PC-FMS) Module-1 (the PC-I Form)   9. Human Resource Management Information System (HRMIS)  In the FATA Secretariat    In the FATA Development Authority   10. FATA Assets Management System (FAMS)  31 11. Litigation Cases Management System (LCMS)   Reporting Number of Owner-cum-User Departments/Subordinate Units 5 4 6 4 31 Data quality and cleaning issues reported. 25 EAS The biometric device does not work during load-shedding hours when electricity is not available, so employees cannot use the device during that period. Five or six employees have experienced difficulty in marking attendance through the biometric device as it does not register their hands. FISP is trying to solve this problem by installing a thumb-reading device. ZMIS The system owner-cum-user requested higher authorities for staff to be assigned to run the system. Government staff was not available, so the authorities requested the help of FISP, and FISP provided a person to run the system until August 2014. FIGURE 5: FACTORS INHIBITING PROGRESS IN MEETING ORGANIZATIONAL NEEDS Interviewees cited the lack of IT infrastructure and other equipment (see box on this page) as adversely affecting five systems (FTS and EAS in FS, PMIAM, ZMIS, and PC-FMS) in four owner-cum-user departments and subordinate units. Interviewees specifically mentioned the lack of computers, biometric devices, uninterruptible power supply and problems with internet connectivity. Internet connectivity issues and prolonged periods of power outages are reported to be common in the tribal areas, and create a particularly adverse operating environment for the PC-FMS and ZMIS, both of which depend on extensive outreach in these locations. The evaluation did not have the means to assess the extent of these problems or how adversely they affect the systems deployed. Participants from four owner-cum-user departments and subordinate units responsible for four systems (PMIAM, ZMIS, PC-FMS, and LCMS) reported that institutional issues such as lack of communication, motivation, and staff capacity, as well as high staff turnover and unclear roles and responsibilities have affected progress in using the systems to address organizational needs. These issues are reported to particularly affect the progress of the PC-FMS and ZMIS (see boxes), and to a lesser extent the PMIAM and LCMS. According to the interviewees, six systems (FTS in FS, NOCMS, PMIAM, ZMIS, HRMIS in FS and FDA) in five owner-cum-user departments and subordinate units suffer from system-related issues or need improvement to become more useful. The NOCMS is reported to need improvement as and when the owner-cum-users identify the need for additional requirements, particularly for reporting. While the FTS is working as designed, the system allows only the sender of a file to 5 4 6 4 Lack of IT Infrastructure Institutional Issues Technical Issues or Need for System Improvement Data Entry Not Completed Factors Inhibiting Systems’ Progress in Meeting Organizational Needs – Reporting Number of Owner￾cum-User Departments/Subordinate Units 26 record its dispatch to the recipient, which has created problems in tracking, and does not allow file tracking in instances in which a senior officer asks a subordinate to leave a file with the officer for further consideration.32 ZMIS reportedly has some design-related issues that have arisen because the system does not adequately reflect the actual process followed by those responsible for zakat distribution. There is a mismatch between the expectations of the owner-cum-user and what the system is designed to do according to the implementing partner. The PMIAM is said to require modifications to make it as effective as expected by the owner-cum￾user and these have reportedly been communicated to FISP. Interviewees in FDA reported that they have communicated to FISP that the HRMIS system does not keep track of leave records. The interviewee in the FS said that the need for additional features has already been communicated to FISP. Three systems (HRMIS, FAMS, and LCMS) are not in use because of data-related issues. Data entry has not been completed in HRMIS and LCMS, and interviewees reported data quality and data cleaning issues with FAMS. ZMIS and PMIAM are perceived by their owner-cum-users to be facing three sets of inhibiting factors, lack of IT infrastructure, institutional issues, and system-related issues. The lack of IT infrastructure and institutional issues are reportedly affecting the PC-FMS, and the HRMIS is facing system-related and data-related issues. CONCLUSIONS Efficiency and accountability, defined in terms of the operational needs of the civil service, including, in the case of one system, accountability to the public for development projects, are reported to be key organizational needs for the systems deployed by FISP. FISP has obtained support from the highest level of departmental management for all the systems it has deployed so far. The support is the single most important factor contributing to progress toward meeting organizational needs through these systems. FISP contributions through staff training and resources have also facilitated progress. These factors, however, have not ensured uniform progress across all systems and departments. Part of the problem is the perception among owner-cum-user departments and subordinate units that five of the systems lack usefulness because of some system-related issues. This perception may or may not turn out to be correct upon further examination, but it is an articulated (though not uniformly well-articulated) perception among FISP counterparts and, as such, a matter of concern for FISP. Three sets of issues reported to be inhibiting progress (institutional issues, lack of IT infrastructure, and data￾related problems) fall within the domain of government decision-making and operations, which is beyond the direct influence of FISP and underscores the importance of the operating environment in whether the systems deployed by FISP will be able to meet organizational needs. QUESTION 2 How and to what extent has the FS, the FDA, and the KP Governor’s Secretariat institutionalized the systems introduced by FISP? 32 It is standard practice that an officer who needs to discuss an issue with his/her senior takes the relevant file to the senior’s office, who sometimes asks for the file to be left with him/her for further consideration and/or decision. The system does not allow file tracking in such instances. PC-FMS Some departments/ directorates have not communicated the instructions of the P&D Department for preparing PC￾Is on the PC-FMS to their staff. Internet issues and the lack of uninterruptible power supply in the tribal areas are inhibiting the use of the system. 27 Institutionalization  FTS fully institutionalized in Governor’s Secretariat and institutionalized to a considerable extent in FS  EAS fully institutionalized in FS  NOCMS fully institutionalized in FDMA  OLPFS fully institutionalized in FDA The explanation provided in the SOW for question 2 anticipates that institutionalization will be assessed on the basis of recognition of the new system in the departments’ operating polices, the official decision to implement the system, training of staff on the system, adoption of relevant manuals, actual use and integration of the system in the work of the department and allocation of resources for implementation, maintenance, upgrades and redundancy. Participants indicated that directives from the department head or meeting minutes are considered operational policies, and in some cases verbal instructions suffice in lieu of written directives. Moreover, the use of manuals is not enforced through high-level (written or unwritten) instructions for adopting the manual. FINDINGS All interviewees reported that the decision to implement a system was reflected in the directives or verbal instructions provided by the department head and in the case of the EAS, PC-FMS, and HRMIS, by the Additional Chief Secretary. The evaluation team conducted three focus group discussions (two for men and one for women) that reported that the training of staff was a key step to institutionalizing nine systems that required training to use them. Officials who received training provided positive feedback on the training content and the trainers who conducted the training. Focus group participants felt, however, that the departments’ nomination and selection process of training participants was ad hoc, suggested favoritism, and did not take into account merit or provide clear nomination/selection criteria. The evaluation team did not assess the nomination process and is unable to provide additional evidence about this issue. Focus group participants also felt that the training hours (from 9:00 a.m. to 5:00 p.m.) were too long and took up too much of the day. Respondents reported that the availability and use of training and user manuals have also contributed to the institutionalization of eight systems. Respondents for two systems, PC-FMS and LCMS, were not knowledgeable about the availability and use of the manuals. Participants reported that the required resources, specifically computers, are in place, further facilitating the institutionalization of seven systems. Some of these resources have been provided by FISP. There is anecdotal information about resource gaps inhibiting implementation, such as lack of resources for systems maintenance, and upgrades and how system owners and users will fill these gaps. Five systems (FTS in the Governor’s Secretariat, EAS and ZMIS in FS, NOCMS in FDMA, and OPFS in FDA) have already been incorporated into work flows and organizational practice, contributing to their institutionalization. An assessment of the five indicators of institutionalization (i.e. the decision to implement the system, training of staff, use of manuals, allocation of resources, and making the system part of organizational practice) suggests adequate or considerable progress toward institutionalization for the four systems in the five organizations/departments mentioned below:  The Governor’s Secretariat has fully institutionalized the FTS. The FS has institutionalized FTS to a considerable extent.  The FATA Secretariat has fully institutionalized EAS.  The FATA Disaster Management Authority has fully institutionalized the NOCMS and all applications for NOCs are now received and processed using the system.  The FATA Development Authority has fully institutionalized the OPFS. The detailed assessment of five indicators of institutionalization for this question is summarized in Table 9. 28 TABLE 9: STEPS TAKEN TO INSTITUTIONALIZE SYSTEMS Name of the System/ Organization Decision to Implement System Training of Staff Use of Manual Allocation of Resources System Made Part of Organizational Practice Conclusion 1. File Tracking System (FTS)  In the FATA Secretariat     - Institutionalized  In the Governor’s Secretariat      Fully institutionalized 2. Employee Attendance System (EAS)      Fully institutionalized 3. No Objection Certificate Management System (NOCMS)      Fully institutionalized 4. Online Public Feedback System (OPFS)      Fully institutionalized 5. Public and Media – Inter￾communication and Awareness Mechanism (PMIAM)     - Likely to be institutionalized 6. Digital Resource Access System (DRAS)   Not evaluated for institutionalization33 7. Zakat Management Information System (ZMIS)    -  Likely to be institutionalized 8. Planning Commission Forms Management System (PC-FMS) Module-1 (the PC-I Form)   -  - Likely to be institutionalized 9. Human Resource Management Information System (HRMIS) 33 The DRAS is not an office automation system but a means for connecting individuals to the electronic information/knowledge resources. It does not require any steps on part of the owner-cum￾user to institutionalize it. 29 Name of the System/ Organization Decision to Implement System Training of Staff Use of Manual Allocation of Resources System Made Part of Organizational Practice Conclusion  In the FATA Secretariat     - Likely to be institutionalized  In the FATA Development Authority     - Likely to be institutionalized 10. FATA Assets Management System (FAMS)    - - Likely to be institutionalized 11. Litigation Cases Management System (LCMS)  - - - - Not likely to be institutionalized Total 12 11 8 9 4 30 CONCLUSIONS Judged in terms of the criteria for assessing institutionalization used in the evaluation, the Governor’s Secretariat has fully institutionalized the FTS, and the FS has to a considerable extent institutionalized the FTS. The FS has fully institutionalized the EAS; the FDMA has institutionalized the NOCMS; and the FDA has institutionalized the OPFS. A process and plans are in place to institutionalize the PMIAM, PC-FMS, ZMIS, and FAMS. However, information about the resource gaps related to maintenance, redundancy, and upgrades, and how system owners and users are planning to fill these gaps, is not systematic or reliable, creating uncertainty about the future of these systems. QUESTION 3 What has been the usefulness of the policy-related reports prepared with FISP assistance? The evaluation SOW expects that the answer to this question will include an assessment of the reports in relation to their purpose (as evidenced by their terms of reference and the perceptions of the institutions that requested them). It also anticipates findings and conclusions on the continuing relevance of the reports and plans to implement the report recommendations. The evaluation team assessed the policy-related reports listed below. 1. Study of Functions, Structure, and Planning Processes of the Planning and Development Department, FATA Secretariat 2. Draft Local Government Regulations for the FATA Local Government 3. FATA Information Technology Policy 2012 4. South Waziristan Agency Development Plan 5. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA. 6. FATA Investment Plan Study 7. Study on Strengthening of Women Empowerment Wing in FATA Secretariat34 FINDINGS Study of Functions, Structure, and Planning Processes of the Planning and Development Department The interviewee in the FS found the recommendations to be “very relevant and useful” for proceeding with restructuring the P&D Department.35 The interviewee reported that some important recommendations had already been implemented informally. Other recommendations involved amendments to constitutional provisions and related regulations, which entails a lengthy process of discussion and approval from various federal ministries and entities. The FS is actively pursuing this course of action, but the final outcome is not currently clear. 34 In accordance with the evaluation SOW, findings and conclusions relating to this report are placed under question 4, but the report is included in Table 10 here to provide a complete overview of the usefulness and status of recommendations of policy-related studies. 35 There are no terms of reference for this report. 31 Draft Local Government Regulations for the FATA Local Government The evaluation team’s desk review confirmed that the report addressed its main purpose by providing a legal framework for municipal bodies in FATA based on a consultative and participatory process. Two participants consider it to be perhaps the best FISP product because of its potential for bringing about change in FATA. According to another interviewee, the draft regulation proposed by the study was cleared by the Federal Law Division and submitted to the President of Pakistan for his signature in August 2013. The President’s approval is pending. FATA Information Technology Policy 2012 The commissioner of the report found its recommendations very “useful and actionable”. According to two other participants, the Governor of KP approved and notified this policy, but nothing has happened since. The terms of reference for the study also required it to provide a technical approach and standards for e-government systems, which were not included in the report. An interviewee who is now overseeing the implementation of the report’s recommendations said that they are now being pursued vigorously. The respondent believed that all the recommendations included in the report are important and relevant and are likely to be implemented, but a timeframe for implementation has not been finalized. South Waziristan Agency Development Plan A participant associated with the study indicated that it was completed through a comprehensive consultation with stakeholders, including government officials and the private sector. The interviewee added that the report was presented to the FS to serve as a roadmap for development of South Waziristan Agency, although the interviewee conceded implementation of the recommendations will be challenging. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA According to the interviewee, the objective of the study was to gain in-depth knowledge of the capacity of training institutions, as well as to assess market needs and potential, and supply-and-demand issues of the workforce in KP and FATA. According to the interviewee, most of the recommendations (approximately 70-80 percent) were relevant, useful, and actionable. He added that the implementation depends on available resources. FATA Investment Plan Study According to the interviewee, USAID/Pakistan requested this study to feed into the upcoming USAID/Pakistan Economic Stabilization Project in FATA, which seeks to increase private sector participation in the economy of FATA. Based on stakeholder consultations involving the government and the private sector, the report identified five priority sectors for investment (agribusiness, minerals, trade, education, and health). Not all of the recommendations in the report were considered useful or actionable, as some lacked specificity from the user’s point of view. Key findings about the usefulness of the seven reports and the relevance of their recommendations are summarized in Table 10. 32 TABLE 10: ASSESSMENT OF SEVEN FISP POLICY-RELATED REPORTS Report Addressed the Terms of Reference of the Report Provided Useful and Actionable Recommendations Report Commissioner or Successor Followed up on Recommendations Opened up Potential for Significant Change 1. Study of Functions, Structure, and Planning Processes of the Planning and Development Department, FATA Secretariat   2. Draft Local Government Regulations for the FATA Local Government     3. FATA Information Technology Policy 2012    4. South Waziristan Agency Development Plan    5. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA    6. FATA Investment Plan Study  7. Study on Strengthening of Women Empowerment Wing in FATA Secretariat    Total 5 6 6 2 Participants interviewed for this question were satisfied with the process for commissioning and implementing the reports/studies, except for the FATA Investment Plan Study, for which the respondent suggested that FISP should strengthen its quality assurance process to ensure that a report/study meets the requirements for which it is commissioned and provides specific and actionable recommendations. CONCLUSIONS The policy-related studies have been useful (with one exception) for USAID/Pakistan or the FS. Specifically, two of the studies have the potential to create significant change in FATA, and two have generated recommendations that are relevant but challenging to implement. QUESTION 4 What evidence is there of women’s participation and a concern for gender issues in FISP interventions? The evaluation SOW anticipates this question’s answer on the basis of sex-disaggregated data on the training provided by FISP, as well as findings and conclusions on the report prepared with FISP support on the Women’s Empowerment Wing in the FATA Secretariat. 33 FINDINGS Women trained by FISP represent 4 percent of all employees of FATA institutions trained by FISP (Table 11). This is approximately the same as the percentage (3 percent) of women among all those employed in these institutions. TABLE 11: GENDER BALANCE IN FISP TRAINING Name of Organization Number of Employees Number of Employees Trained by FISP Total Men Percentage of Men in Total Women Percentage of Women in Total Total Men Percentage of Men Among Trained Women Percentage of Women Among Trained FATA Secretariat 883 850 96 33 4 795 767 96 28 4 FATA Development Authority 212 208 98 4 2 62 59 95 3 5 Governor’s Secretariat 89 87 98 2 2 7 7 100 0 0 Total 1,184 1,145 97 39 3 867 833 96 31 4 Source: Information provided by FISP in email messages dated April 14 and 25, 2014. 34 A focus group discussion with female government staff trained by FISP indicated that the hours during which the training is held do not suit them. The training extends into late afternoon or early evening, and women reported that they prefer to return to their homes earlier due to the difficulty of finding available transportation and concerns about their safety traveling in the evening. A review of the “Study on Strengthening of Women Empowerment Wing in FATA Secretariat” shows that the report recommended two main options: upgrading the existing Women Empowerment and Child Protection Cell to a Women Empowerment Wing, and building its capacity in order to be effective; and establishing an independent Women Empowerment Directorate that reports directly to the Secretary of the Social Sectors Department. The study also reiterated the recommendation included in the “Social Welfare and Women Empowerment Strategy - 2010” calling for the formation of a FATA Women Empowerment Council. The objective of this council is to provide stewardship and strategic focus for the improvement of women-related conditions in its respective domain. According to a government interviewee, the FATA Secretariat has partially implemented the first option, and has prepared a PC-I to continue supporting the already existing Women Empowerment and Child Protection Cell, arguing that it will be difficult to manage an independent directorate. The evaluation team asked all individual and group interviewees and focus group participants if and how the systems introduced by FISP are addressing gender issues. Three individual and two group interviewees said that three systems (NOCMS, HRMIS, and ZMIS) have provisions for disaggregating data by gender. All others said the remaining eight systems are gender-neutral and do not need to address any specific gender concerns. CONCLUSIONS FISP assisted the FS with the satisfactory completion of the report on the Women Empowerment Wing and includes women of FATA institutions in its training programs. The only other evidence of FISP addressing gender issues is the implementation of three systems that provide sex-disaggregated data. The remaining systems are reported by users to be gender-neutral and do not need to address specific gender concerns. FISP interventions that promote women’s participation and address gender concerns have been few in number and reflect a passive role vis-à-vis gender initiatives and women’s participation in project implementation reflective of the demand-driven orientation of the project. 35 RECOMMENDATIONS QUESTION 5 What changes could better serve the project objectives and the priority organizational needs of the FS, the FDA, and the Governor’s Secretariat? The evaluation SOW expects the answer to this question to consist of recommendations based on the answers to the preceding questions. An attempt has been made to suggest course-corrections for the remainder of the project. There is persuasive evidence in this evaluation that FISP’s demand-driven orientation has thus far turned out to be a double-edged sword. On the one hand, FISP’s responsiveness to demands from key stakeholders has elicited the highest level of government ownership and decisive action for introducing new systems and commissioning reports aimed at changing how government works in FATA. On the other hand, relying on its demand-driven orientation has kept FISP at a distance from the operating environment that will, in the final analysis, determine the success or failure of systems introduced by FISP, and from engaging proactively to support women’s participation in development and governance. As a matter of course correction, FISP, with the help of USAID/Pakistan and the government, needs to conceive a change management initiative for: (a) dealing with the broad operating environment that affects the use of systems and users’ perceptions in FATA institutions; and (b) better addressing the role of women in FATA and its governance institutions. This is the overall course of action recommended by the evaluation for the remainder of the project’s duration. The specific recommendations offered below are aimed at both USAID/Pakistan and Abacus for addressing the limitations in development effectiveness that has resulted from following a strictly demand-driven approach, while eschewing a proactive role for the project. For USAID/Pakistan, the main recommendation is to facilitate FISP and Abacus to adopt a more proactive role as catalyst and enabler, as described below, by: (a) providing the flexibility that will be required in work planning, staffing, and budgeting for undertaking specific course corrections; and (b) facilitating the formation of a committee consisting of USAID/Pakistan and government representatives to facilitate and monitor progress. The first main recommendation is for Abacus to take a more systematic, longer-term view of the implementation process. Two broad directions for change are recommended, one focusing on the resources and approaches needed for improving implementation, and the other on addressing user perceptions of system-related constraints and benefits during implementation. In consultation with USAID/Pakistan and the government, and as a matter of priority, Abacus should conduct a needs assessment of the IT infrastructure required to support the new systems and improvements and develop an action plan, including cost estimates, for addressing any resource gaps for the duration of the project and share the plan with USAID/Pakistan and the government. Assuming the plan is approved, Abacus should closely monitor progress and flag any lack of action for the attention of USAID/Pakistan and the government. Priority in the allocation of resources should be given to the needs of the PC-FMS and ZMIS, both of which have more direct implications for service delivery than the other systems and depend on extensive outreach in the tribal areas, where IT infrastructure problems have created a particularly adverse operating environment for these systems. The PMIAM should also be considered a priority because of its importance to governance. Approaches to data entry and data migration are other aspects of the operating environment that need attention to improve implementation. Before introducing a new system, FISP should specify arrangements for data entry or data transfer and ensure that government counterparts are well-equipped for these purposes. In addition, with the help of government counterparts, FISP needs to promote a merit-based nomination process for trainees and arrange more convenient training hours, particularly for women. In line with its proposed role as an enabler of systems and system users, Abacus also needs to address new requirements expressed by system users and investigate the veracity of specific user perceptions about system-related issues and the root causes of actual technical problems, if any, and their consequences for the design and use of the systems. Taking steps to address technical problems and showing users how to better 36 use the systems that do not have technical problems would be part of this exercise. In addition, attention should be given to managing the expectations associated with concepts such as transparency, accountability, and quality. As illustrated by responses to evaluation question 1, operational definitions can be found to clarify and address these concepts in practical terms that are consistent with the requirements of an enlightened civil service. Discourse and action that focuses on clear operational definitions is one way of managing expectations. At the same time, there is a need to revisit the expectation that the PC-FMS is a tool for improving the quality of planning and PC-I preparation. Recognition needs to be developed among the system owner and users that these quality issues cannot be addressed by simply automating PC forms. The second main recommendation is to focus more systematically on institutionalizing and sustaining the systems that FISP is introducing in order to enhance the likelihood of the sustainability of these systems. In this connection, FISP needs to undertake an across-the-board in-depth analysis of the resources needed and available to sustain the systems. This analysis should be accompanied by an action plan for the government and USAID/Pakistan and FISP to fill the resource gaps to enhance sustainability. FISP should subsequently provide periodic reports to USAID/Pakistan and the government on progress made in filling the identified resource gaps. These efforts are needed to increase the payoff of USAID/Pakistan investments in terms of benefits and costs, as well as improvements in FATA institutions and the governance of FATA. The third main recommendation is for FISP to take on a more proactive role in addressing gender concerns and women’s participation in the development and governance of FATA. In this connection, it would be useful to discuss the provision of sex-disaggregated data for at least two systems (OPFS and HRMIS) with government counterparts and ensure that the systems are modified and enabled for this purpose. In addition, FISP needs to sensitize its staff to the fact that women training participants have different requirements than their male counterparts related to mobility and safety considerations. This may require reducing the hours of training or otherwise scheduling training sessions so that women are able to travel to and from the training without safety concerns. Lastly, FISP has an opportunity, based on the report it funded for the Women Empowerment Wing of the FATA Secretariat, to discuss the capacity building needs of the Women Empowerment Wing with the government, provide appropriate support for capacity building, and take forward the recommendation of its report to establish a FATA Women Empowerment Council to “provide stewardship and strategic focus for the improvement of women-related conditions in its respective domain” (page 37 of the report). This body should include high-level government officials and representatives of women’s rights organizations working in FATA and KP. More broadly, FISP needs to sensitize its staff, through training, situation analysis, day-to-day interaction with relevant women’s rights organizations, or other means, to appreciate the role of women in the development and governance of FATA and include measured and constructive initiatives in its work plan whenever possible. 37 BIBLIOGRAPHY  Statement of Work of Request for Proposal of FATA Capacity Building Program – Phase 2.  FISP Midterm Review and Planning Workshop Report dated 26 August, 2013, Islamabad.  FISP Performance Monitoring Plan, 2011-2016, Draft 2, November 04, 2011.  FISP Year 2 Work Plan, September 2012- August 2013.  FISP 5th PMP Quarterly Report, September – December 2012.  FISP 6th PMP Quarterly Report, January – March 2013.  FISP Annual Report, September 2012 – August 2013.  System Requirements Specification (SRS) for Human Resource Management Information System  (HRMIS) - July 2013  System Requirement Specification (SRS) for Zakat Management Information System (ZMIS) - July 2012  System Requirements Specification (SRS) for Assets Management System (AMS) - April 2013  System Requirement Specification (SRS) for PC Automation Software (Activity-2) - March 2012  Software Requirements Specification of NOC Automation of Directorate of Projects and Fata Disaster Management Authority, FATA - 20th June 2013  Study on Strengthening of Women Empowerment Wing in FATA  Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat  Draft Local Government Regulations for the FATA Local Government  Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA  FATA Information Technology Policy 2012  South Waziristan Agency Development Plan  FATA Investment Plan Study 38 ANNEXES ANNEX 1: CURRENT STATUS OF FISP ACTIVITIES The current status of project activities is as follows: 1. The information technology (IT) component of FISP has provided maintenance, repair and connectivity services to the FATA Secretariat (FS) and FATA Development Authority (FDA). An IT Help Desk has been setup for the FS and FDA. Local Area Networks have been installed at the FS and the new office building of FDA. FISP has supported the strengthening of the FS Data Centre and the FS Conference Room. Two hundred and thirty three personnel of the FS, the Governor’s Secretariat and FDA have been trained in IT skills. FISP has also developed a detailed IT Action Plan for the implementation of the FATA IT Policy. 2. Under the PC Forms36 Automation component, the project has developed and deployed Module 1 of Planning Commission Forms Management System (PC-FMS). Module 2 (for the PC-III (A) and PC-III (B) forms) of the PC-FMS has also been developed. Operational assistance and training is being provided for Module 1. Advanced features of PC-FMS Module 1 have also been developed. The FATA web portal has been designed and developed. The Administration, Infrastructure and Coordination (AI&C) Department of the FS is collecting data for the web portal. The Litigation Case Management System for the FS and the No Objection Certificate (NOC) Management System for the FATA Disaster Management Authority have also been designed, developed and deployed. 3. Under the HR component, FISP is providing support to the FS and FDA for strengthening of institutions for using the systems developed and deployed during project Year 1. This includes operational assistance, software de-bugging and system improvement support. Ongoing support is also being provided for the Human Resource Management Information System (HRMIS), the Employee Attendance System and the Public Feedback System for FDA. The File Tracking System (FTS) has been installed in the Governor’s Secretariat and the FS. FISP has also trained staff in public sector project management, effective communication and presentation skills, and code of conduct and ethical standards in public sector. FISP has also assisted in developing job descriptions for the staff of the Directorate of Social Welfare and the Social Sector Department. Access to online digital resources at the Higher Education Commission of Pakistan, located in Islamabad, has been provided to the staff of the FS and users at the Cadet College, Razmak, located in FATA, and staff has been trained in using this resource. In addition, FISP is managing the day-to-day issues of USAID/Pakistan￾paid support staff working in FS and FDA through the HR component. 4. Under the communication component, FISP has developed and deployed the Public and Media Intercommunication and Awareness Mechanism at the Governor’s Secretariat, which will be extended to the FS when the FATA Information Directorate is fully functional. In addition to this, FISP has provided assistance to FS in developing a fifteen-minute daily awareness program in Pashto, called Da Khushalay Safar, which is broadcast by Radio Pakistan for the FATA public and is about the development initiatives being undertaken by the Government of Pakistan and supporting development agencies, especially USAID/Pakistan. FISP has also produced a video documentary on the development initiatives in Bajaur Agency in FATA. Assistance was also provided to FATA Information Directorate and other FS employees to enhance their internal and external communication skills. 36 The Planning Commission Form PC-1 is the prescribed template in which government departments prepare project proposals for approval by higher government authorities. The PC-II is used for proposing and approving surveys and feasibility studies. The PC-III has two separate templates, namely, PC-III (A) and PC-III (B). The PC-III (A) is used for reporting on physical targets based on Public Sector Development Program allocations. The PC-III (B) is used for monthly progress reporting on approved projects. The PC-IV is the project completion report template, while the PC-V is used to prepare annual performance reports after the project completion report is submitted. 39 Under the Training and Systems Development component, FISP has assisted the FS in developing the handbook Simplified Zakat Procedures and its Urdu version and guidelines for the Chairmen of Local Zakat Committees in FATA. The Zakat Management Information System (ZMIS), which was developed with FISP assistance, has been deployed in the FS. FISP has provided extensive training on ZMIS and the simplified procedures of zakat management. The project has conducted seven policy-related studies and reports at the request of USAID/Pakistan and FATA authorities (details given in the table below). Moreover, the FATA Assets Management System (FAMS) has been developed and deployed in the FS. STUDIES FUNDED BY FISP No. Name of the Study Requested by Completion Date 1. Study on Strengthening of Women Empowerment Wing in FATA Deputy Secretary, Social Sector Department 2012 2. Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat37 Secretary, Planning and Development Department 2012 3. Draft Local Government Regulations for the FATA Local Government Local Government and Rural Development Directorate 2012 4. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA USAID/Pakistan 2013 5. FATA Information Technology Policy 2012 Principal Secretary to the Governor; approved by the Additional Chief Secretary, FATA 2012 6. South Waziristan Agency Development Plan USAID/Pakistan 2012 7. FATA Investment Plan Study USAID/Pakistan Jan. 2014 5. Under the Monitoring and Evaluation component FISP has developed the Monitoring and Evaluation framework for FS. It has also developed the Performance Monitoring and Reporting System (PMRS) that comprises goals, outcomes, outputs and respective performance indicators for all FS directorates. Performance indicators for the directorates of Education, Health, and Agriculture have been developed. The PMRS software has also been developed for data entry, analysis and reporting of performance data from the field. 37 FISP conducted this study with its in-house staff and there are no terms of reference for it. 40 ANNEX 2: FISP EVALUATION STATEMENT OF WORK JANUARY 28, 2014 FATA INSTITUTIONAL STRENGTHENING PROJECT (FISP) MID-TERM PERFORMANCE EVALUATION EVALUATION STATEMENT OF WORK 41 FATA INSTITUTIONAL STRENGTHENING PROJECT (FISP) MID-TERM PERFORMANCE EVALUATION EVALUATION STATEMENT OF WORK 42 I. BACKGROUND INFORMATION A. Identifying Information about the Project The Federally Administered Tribal Areas (FATA) Institutional Strengthening Project (FISP) is a five-year project funded by USAID and implemented by Abacus Consulting Technology (Private) Limited, a Pakistani consulting firm. FISP is assisting the FATA Secretariat (FS), the FATA Development Authority (FDA) and the Governor’s Secretariat in their efforts to develop their capacity through systems development38, improvement of related functions and strengthening of the information technology (IT) infrastructure39. According to the FISP Monitoring and Evaluation (M&E) Plan, the strengthened capacity of the FS and FDA will help achieve the strategic objective of ensuring their institutional viability as service-oriented government bodies capable of managing their duties in a transparent and well-planned manner. FISP supports the USAID/Pakistan Mission Strategic Framework through Development Objective 3 “Increased stability in focus areas” primarily through contributing to the achievement of Intermediate Result 3.2 “Governance Improved.” TABLE 1: PROJECT SUMMARY B. Development Context 1. Problem or Opportunity Addressed FATA is located in the northwest frontier region of Pakistan, which also includes the Khyber Pakhtunkhwa (KP) Province. FATA has a geographical area of 27,220 square kilometers and the 1998 38 This entails the automation of office processes, including the planning process. 39 FISP Performance Monitoring Plan 2011-2016, Draft 2, November 4, 2011. Assistance to the Governor’s Secretariat is mentioned in FISP progress reports, but not in the PMP. Project Name/ Title Fata Institutional Strengthening Project (FISP) Agreement Number AID-391-C-11-00003 USAIDUSAID/PAKISTAN Contracting Officer’s Representative (COR) Muhammad Anwar (Mr.) Project Start Date August 24, 2011 Project Completion Date September 6, 2016 Project Location Peshawar and Federally Administered Tribal Areas (FATA) USAID Objective Addressed  Intermediate Result (IR) 1: Improved capacity of FATA Secretariat (FS) to govern through systems development, training and strengthening of information technology (IT) infrastructure  Sub-IR 1.1: Strengthened planning, monitoring and evaluation, budgeting, financing, human resources (HR), office management and communication functions of FS and FATA Development Authority (FDA)  Sub-IR 1.2: Strengthened IT infrastructure and IT capacity of FS and FDA Name of the Implementing Partner (IP) Abacus Consulting Technology (Private) Limited 43 Census reported its population to be 3.18 million, with an annual growth rate of 2.19 percent.40 It consists of seven tribal agencies and six frontier regions (FRs), sub-divided into 43 tehsils and approximately 3,000 villages. The FATA region has representation in the National Assembly and the Senate of Pakistan, but does not have its own legislature, cabinet or chief minister.41 Until 2002, FATA was under the direct control of the Governor of Khyber Pakhtunkhwa, who is appointed by and represents the President of Pakistan according to the country’s constitution. Service delivery in FATA was managed through line departments of the Government of Khyber Pakhtunkhwa (then called the North-West Frontier Province). In 2002, the FATA Secretariat was established under a Secretary for FATA. In 2006, the Secretariat was upgraded to a Civil Secretariat, headed by an Additional Chief Secretary reporting to the Chief Secretary of KP, who reports to the Governor of KP in matters relating to FATA. The FATA Secretariat consists of six departments, each of which is headed by a secretary and includes several functional entities, which are called directorates in most cases. It is responsible for the delivery of public services in FATA, including provision of security (public law and order), as well as basic utilities and infrastructure. The FATA Development Authority is a smaller organization focusing on specific development projects assigned by the government and is organized in departments. The Governor’s Secretariat assists the Governor of KP in coordinating the government machinery in development and political matters. The performance of the FS has a direct bearing on the state of affairs in FATA. This, in turn, has consequences for government legitimacy in FATA. As the FS is a nascent entity, it requires support for institutional development. Realizing the importance of capacity building for FATA institutions, particularly the FS, USAID funded the FATA Capacity Building Project that was completed in 2010. USAID continued its capacity building support through FISP, which started in August 2011. The project is aimed at strengthening the ability of FS and FDA to deliver essential government services in FATA. This will be achieved by improving the human and technical capacity of FATA institutions, improving accountability and monitoring mechanisms, improving state responsiveness and supporting FATA institutions in communicating effectively with the citizens that it serves and beyond. II. Target Areas and Groups The target groups of FISP include staff working in the FS, FDA and the Governor’s Secretariat. FISP also provides IT-related services to the FATA Administration based in the tribal agencies and frontier regions. C. Intended Results and the Theory of Intervention According to the FISP results framework, the strategic objective of the project is to ensure the institutional viability of the FS and FDA as service-oriented government bodies capable of managing their duties in a transparent, well-planned manner. Related to this, the project is pursuing the following intermediate results (IRs):  IR 1: Improved capacity of the FATA Secretariat (FS) to govern through systems development, training and strengthening of IT infrastructure.  Sub-IR-1.1: Strengthened planning, monitoring and evaluation, budgeting, finance, HR, office management and communication functions of the FS and FDA.  Sub-IR-1.2: Strengthened IT Infrastructure and IT Capacity of the FS and FDA. 40 Information reported in this paragraph is taken from the official FATA website www.fata.gov.pk. 41 Most of the information in this and the following paragraph is taken from USAID’s Statement of Work in the Request for Proposals for the FATA Capacity Building Program Phase II, RFP No. 391-11-000001, dated 2011. 44 By achieving the above mentioned intermediate results the project aims42 at improving efficiency43 , transparency44, accountability45 and the quality of project plans, proposals, monitoring and reporting in government departments. FIGURE 1: FISP THEORY OF INTERVENTION D. Approach and Implementation FISP is a demand-driven project for which requests for assistance are received from the government. FISP undertakes its activities within the scope of the project in response to these requests. To achieve its results, FISP develops systems and manuals, conducts training and provides IT support. FISP has six components, the details of which are given below: 1. IT Support: FISP is providing information technology (IT) maintenance and support services to the FS and FDA. The key tasks in IT support include setting up a help desk at the FS and FDA, maintenance, repair and connectivity of the Local Area Network (LAN) and IT hardware and IT training. It also includes support for strengthening the FS Data Center and FS Conference 42 The terms efficiency, transparency and accountability are not defined in the project documents. 43 The USAID Glossary of Evaluation Terms prepared by Planning and Performance Management Unit, Office of the Director of U.S. Foreign Assistance dated March 25, 2009 defines efficiency as “a measure of how economically resources/inputs (funds, expertise, time etc.) are used to achieve results”. In the context of this evaluation, however, efficiency will be assessed in terms of time saved, absenteeism controlled and other institutional needs. 44 Transparency International defines transparency as a characteristic of governments that are open in the clear public disclosure of information, rules, plans, processes and actions. Source: Mapping Transparency, Accountability and Integrity in Primary Education in South Africa published by Transparency International, dated 2011. In FISP, systems development is aimed mainly at making a department’s actions transparent within that department and to other relevant departments. Transparency to the public is apparently the aim of only one system, namely, the FATA Web Portal. 45 The USAID Glossary of Evaluation Terms defines accountability as “an obligation to demonstrate what has been achieved in compliance with agreed rules and standards. This may require a careful, legally defensible, demonstration that work is consistent with the contract”. In FISP, the new systems are aimed at enhancing accountability of government employees and departments to officials within the government. Improved service delivery for the people of FATA Greater efficiency, transparency and accountability in FATA institutions and better quality of plans, project proposals, monitoring and reporting Technical assistance to develop and implement systems and IT infrastructure for FATA institutions 45 Room. FISP has also been responsible for preparing the FATA IT Policy and the corresponding IT Action Plan. 2. Form Automation: FISP is providing support for automation of the planning process, which is based on standard forms and guidelines of the Planning Commission of Pakistan. The automated system is called the Planning Commission Forms Management System (PC-FMS). The key tasks include development, testing, deployment of and operational assistance for PC-FMS and the training of staff on the system. Under this component, the project has also developed the FATA Web Portal, the Litigation System for the FS and a No Objection Certificate (NOC) management system for the Directorate of Projects in the P&D Department and for the FATA Disaster Management Authority (FDMA). 3. HR Support: FISP is providing support to strengthen human resource management at the FS and FDA. The key tasks include: development and deployment of the File Tracking System, Human Resource Management Information System (HRMIS), capacity building of FS and FDA officials, preparation of job descriptions, installation and expansion of an Employee Attendance System at the FS, digitization of FS files and archives and the provision of digital resource access facility. 4. Communication: FISP is providing support to build the capacity of the FATA Information Directorate and the Governor’s Secretariat to communicate effectively with FATA communities and local and foreign media. The key tasks include strengthening the communication function, strengthening public information/awareness and image-building programs for FATA communities. 5. Systems Development and Trainings: FISP is providing support for development and strengthening of the business processes of FS and FDA. The key tasks under this component include; development and deployment of the Zakat Management Information System,46 FATA Assets Management System, Financial Management System and the Procurement System and training of FS staff on these systems. Under this component, FISP has also commissioned various reports, including those on the strengthening of the Women Empowerment Wing of the FS, management setup of the Social Welfare Directorate, and a handbook on simplified zakat procedures. 6. Monitoring and Evaluation (M&E): FISP is providing support to develop a monitoring and reporting mechanism for the FS and FDA. The key tasks include the development of a monitoring and evaluation framework, software for performance data entry and reporting on service delivery, development of performance indicators and reporting forms, training of FS and field staff on newly developed/improved M&E mechanisms and processes,47 and development of the Project Evaluation and Verification Module (PEVM). 46 Zakat is 2.5 percent of the accumulated wealth and property that an adult Muslim who is considered to be “well-to-do” according to specified criteria must pay annually to help the poor of their community. This practice is one of the five pillars of Islam. In Pakistan, in addition to zakat that is paid by individuals to recipients of their choice, since 1980 zakat is collected by the government from savings bank accounts and distributed to the needy through provincial and district administrators. 47 FISP is developing a monitoring and evaluation manual that will describe data collection and reporting processes and mechanisms. 46 E. Current Status of Activities The status of the project’s activities at the end of November 2013 is as follows: 1. The IT component of FISP has provided maintenance, repair and connectivity services to FS and FDA. An IT Help Desk has been setup at FS and FDA. Local Area Networks (LAN) have been installed at FS and FDA’s new buildings. Support for strengthening has been provided to the FS Data Centre and the FS Conference Room. Two hundred and fifty-three personnel of FS and FDA have been trained in IT skills. FISP has also developed a detailed IT Action Plan for the implementation of the FATA IT Policy. 2. Under the PC Forms48 Automation component, the project has developed and deployed Module 1 of Planning Commission Forms Management System (PC-FMS). Module 2 (for PC-III (A) and PC-III (B)) of the PC-FMS has also been developed. Operational assistance and training is being provided under Module-1. Advance features of PC-FMS Module 1 have also been developed. The FATA web portal has been designed and developed. The AI& C Department of the FS is collecting data for the web portal. The Litigation System for the FS and the NOC Management System for the FS and FDMA have also been designed, developed and deployed. 3. Under the HR component, FISP is managing day-to-day issues of USAID-paid support staff working in FS and FDA. Support for strengthening institutions is being provided to FS and FDA for the installed systems developed during Year 1. The support includes operational assistance, software de￾bugging and system improvement support. Ongoing support is also being provided to the Human Resource Management Information System (HRMIS), the Employee Attendance System and the Public Feedback System for FDA. The File Tracking System (FTS) has been installed in FDA and FS. FISP has also trained staff in public sector project management, effective communication and presentation skills, and code of conduct and ethical standards in public sector. FISP has also assisted in developing Job Descriptions for the Directorate of Social Welfare and Social Sector Department. Access to online digital resources the Higher Education Commission (HEC) has been provided to FS and 44 FS employees have been trained in using this resource. 4. Under the communication component, FISP has developed and deployed the Public and Media Intercommunication and Awareness Mechanism at the Governor’s Secretariat, which will be extended to the FS when the FATA Information Directorate (FID) is fully functional. In addition to this, FISP has provided assistance to FS in developing a fifteen-minute daily awareness program in Pashto, called Da Khushalay Safar, which is broadcast by Radio Pakistan for the FATA public and is about the development initiatives being undertaken by the Government of Pakistan and supporting development agencies, especially USAID. FISP has also produced a video documentary on the development initiatives in Bajaur Agency in FATA. Assistance was also provided to FATA Information Directorate and other FS employees to enhance their internal and external communication skills. 5. Under the Training and Systems Development component, FISP has assisted the FS in developing the handbook Simplified Zakat Procedures and its Urdu version and guidelines for the Chairmen of Local Zakat Committees in FATA. The Zakat Management Information System (ZMIS), which was developed with FISP assistance, has been deployed in the FS and is now fully operational. FISP has provided extensive training on ZMIS and the simplified procedures of zakat management. The project has conducted seven policy-related studies at the request of USAID/Pakistan and FATA 48 The Planning Commission Form PC-1 is the basic template in which government departments prepare project proposals for approval by higher government authorities. The PC-II is used for proposing and approving surveys and feasibility studies. The PC-III has two separate templates, namely, PC-III (A) and PC-III (B). The PC-III (A) is used for reporting on physical targets based on Public Sector Development Program allocations. The PC-III (B) is used for monthly progress reporting on approved projects. The PC-IV is the project completion report template, while the PC-V is used to prepare annual performance reports after the project completion report is submitted. 47 authorities (detail given in Table 2). Moreover, the FATA Assets Management System (FAMS) has been developed and implemented in FS. TABLE 2: STUDIES FUNDED BY FISP No. Name of the Study Requested by Completion Date 1. Study on Strengthening of Women Empowerment Wing in FATA Deputy Secretary, Social Sector Department 2012 2. Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat49 Secretary, Planning & Development Department 2012 3. Systems Development for Directorate of Local Government & Rural Development Local Government & Rural Development Directorate 2012 4. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA USAID/Pakistan 2013 5. Development of IT Policy and E-Government Roadmap for FATA Requested by Principal Secretary to the Governor; approved by Additional Chief Secretary FATA 2012 6. South Waziristan Agency Development Plan USAID/Pakistan 2012 7. FATA Investment Plan Study USAID/Pakistan Jan. 2014 6. Under the Monitoring and Evaluation component FISP has developed the Monitoring & Evaluation framework for FS. It has also developed the Performance Monitoring and Reporting System (PMRS) that comprises goals, outcomes, outputs and respective performance indicators (PIs) for all FS directorates. PIs for the directorates of Education, Health and Agriculture have been developed. The PMRS software has also been developed for data entry, analysis and reporting of performance data from the field. The detail of the owners, users, data providers and current status of the systems developed by FISP is given in Table 350. FISP has so far developed 13 systems while system development is underway for two more. At this time, 11 of the 13 developed systems are being used for the purpose for which they were developed. There are some systems in which data entry has started, but not completed, so that the users have not yet started using the system for the purpose for which it was intended. These systems are not considered to be “systems currently in use” for the purposes of this evaluation. 49 FISP conducted this study with its in-house staff and there are no terms of reference for it. 50 “Owner-cum-user” refers to an organization that manages a system that is used by the owner and, possibly, a number of other organizations. For example, the Administration, Infrastructure and Coordination (AI&C) Department is the owner of six of the systems in use and part-owner of a seventh. The Planning and Development Department is the owner of two major systems. “User” refers to an organization that uses a system within the limits of the access provided by the owner. For example, all departments are users of the FATA Web Portal, which is owned/managed by AI&C, and also have access for changing their respective web pages in the portal. These two categories of organizations are direct beneficiaries of the systems in use. A third category consists of data providers whose role is mainly to provide data for data entry by an owner. For example, all departments provide data in hard copy for data entry by AI&C, which is the owner of the FATA Asset Management System and the Employee Attendance System. 48 TABLE 3: OWNERS, USERS, DATA PROVIDERS AND CURRENT STATUS OF SYSTEMS DEVELOPED BY FISP (Systems currently in use are identified by *) No. System Name Organizational Needs Addressed by System (as Reported by Implementing Partner) Owner-cum-User Other Users Main Role in Providing Data Current Status 1.* Zakat Management Information System (ZMIS) - Contains details of zakat beneficiaries and the amount paid to them - Brings transparency, efficiency and coherence in zakat management Zakat and Ushr Directorate of the Social Sectors Department Agency Zakat and Ushr officials Agency Zakat and Ushr officials System is currently in use 2. FATA Web Portal Will act as an information hub for the general public AI&C Department Viewing and editing access to all departments and directorates for their respective web-page All departments of FATA Secretariat and FDA Web Portal has been developed and content development is underway 3.* No Objection Certificate (NOC) Management System - Makes NOC management more efficient - Facilitates NOC applicants in receiving status updates Directorate of Projects (DoP) and FATA Disaster Management Authority (FDMA) All donor and civil society projects in FATA Donor and civil society projects in FATA System is currently in use 4.* Litigation Cases Management System Facilitates management of legal cases Litigation Cell of AI&C Department Litigation Cell of AI&C Department System is currently in use 5.* Human Resource Management Information System (HRMIS) Facilitates human resources (HR) management and makes it more efficient - FATA Secretariat: Data being entered by AI&C - FDA: Data being entered by Manager IT All departments of FATA Secretariat and FDA - All the FS directorates and departments - All FDA departments System is currently in use 49 No. System Name Organizational Needs Addressed by System (as Reported by Implementing Partner) Owner-cum-User Other Users Main Role in Providing Data Current Status 6. Performance Management and Reporting System (PMRS) Makes performance monitoring of service delivery in FATA more structured and efficient M&E Directorate of the P&D Department The following will start data entry in January Social Sectors Department: Health and Education Directorates; Production and Livelihood Development Department: Agriculture Directorate. The remaining directorates will start later on. (Please see the attached organogram for the names of remaining directorates) - All the FS directorates and departments Data entry is expected to start in January 2014 7.* File Tracking System (FTS) Automates and facilitates file tracking in FATA Secretariat. Also installed at the Governor’s office AI&C Department P&D Department: Directorate of Projects and Monitoring and Evaluation Directorate; AI&C Department: Local Government, Irrigation and FATA Information Directorates; Law and Order Department. All Users System is currently in use 8.* FATA Assets Management System (FAMS) Improves record keeping, tracking and stock taking of all assets of FATA Secretariat AI&C Department Viewing rights to all the FS directorates and departments All departments of the FATA Secretariat System is currently in use 9.* Public and Media – Inter communication and Awareness Mechanism (PMIAM) - Facilitates swift communication between Governor’s Secretariat and the media - Provides information to the public about Governor’s activities Governor’s Secretariat The FATA Information Directorate of AI&C is a potential user - Governor’s Secretariat - FID of AI&C (if it becomes user) System is currently in use 50 No. System Name Organizational Needs Addressed by System (as Reported by Implementing Partner) Owner-cum-User Other Users Main Role in Providing Data Current Status 10.* Planning Commission (PC) Forms Management System (PC-FMS) - *Module 1: PC-I - Module 2: PC-III (A&B) - Module 3: PC-II, IV & V - Automates Planning Commission Forms (PC-I to PC￾V) - Improves quality and efficiency of project proposals, plans, monitoring and reporting P&D Department Module 1 is being used by: AI&C Department: Works and Services, Irrigation, and Local Government Directorates; Social Sectors Department: Education, Health and Social Welfare Directorates; Production and Livelihood Development Department: Agriculture (Extension), Forests, Fisheries, and Livestock & Dairy Development Directorates. All users Module I only is currently in use 11.* Employee Attendance System (EAS) - Automates and tracks employee attendance - Helps improve attendance and timely arrival of employees AI&C Department All departments of the FATA Secretariat and FDA All users System is currently in use 12. Public Procurement System (PPS) - Automates procurement process - Ensures adherence to procurement rules AI&C Department All departments of FATA Secretariat and FDA All users “As-Is” 51 analysis has been carried out and system development is underway 13.* Digital Resource Access System Provides access to the e-library of the Higher Education Commission AI&C (the system is installed in a library) All officers of FATA Secretariat have access Access to Higher Education Commission E￾library System is currently in use 14.* Online Public Feedback System Allows public to give online opinion and feedback about development projects being carried out in FATA FDA The public has access Public System is currently in use 51 “As-Is” analysis is the analysis of an existing system and its problems on the basis of which a new system is designed. The new design is called the “To-Be” proposal. 51 No. System Name Organizational Needs Addressed by System (as Reported by Implementing Partner) Owner-cum-User Other Users Main Role in Providing Data Current Status 15. Financial Management System - Automates financial management (budgeting and expenditure management) at the FATA Secretariat - Improves transparency and efficiency in financial management Finance Department All departments and directorates All users “As-Is” analysis has been carried out and system development is underway 52 II. RATIONALE FOR EVALUATION Purpose of Evaluation FISP will be halfway completed by February 2014. USAID/Pakistan wants to assess the project’s performance and determine the extent to which the project is proceeding as anticipated. This evaluation is expected to identify course-corrections for the remainder of the project in order to improve project implementation. The evaluation will assess progress toward institution strengthening in relation to the organizational needs of the focused FATA institutions (FATA Secretariat, FATA Development Authority and Governor Secretariat). Organizational needs have been identified through consultation with the owners and users institutions with emphasis on improving efficiency, transparency, accountability and the quality of project plans, proposals, monitoring and reporting in government departments. The evaluation is also expected to document the steps taken by the government to institutionalize the systems introduced by FISP within its various departments. Institutionalization of government systems is defined as both an increased capacity within an individual governmental department as well as sustainability of the new system. In addition to assessing the systems introduced by FISP, the evaluation will also assess the seven policy￾related reports prepared with FISP assistance at the request of USAID/Pakistan and the FATA Secretariat. USAID/Pakistan is interested in learning about the actions resulting from these reports and whether they are still relevant to the institutions that requested them. The findings of this mid-term evaluation will be used during the remainder of the implementation of the program. These findings will lead to refinements in the activity design and Monitoring and Evaluation Plan to ensure more effective implementation and improved results. A. Evaluation Questions The evaluation purpose and questions have been agreed to and finalized after a thorough discussion among all stakeholders (FATA Secretariat, FATA Development Authority, Abacus consulting and USAID) in a half day workshop which took place in the very outset of SOW development process. The evaluation questions have been arranged in priority order: Question 1: How and to what extent are the systems developed by FISP and currently in use addressing the organizational needs of the departments that own and use the systems, and what factors explain progress or lack thereof in this regard? Explanation: The organizational needs to which the new systems respond are not documented and the evaluation will first identify these needs (such as improving efficiency, transparency, accountability and the quality of project plans, proposals, monitoring and reporting in government departments), as expressed by the owners of these systems. The evaluation will also investigate whether these needs are shared by the key users of the systems. The answer to the question will include findings and conclusions about the perceptions of owners and users about the extent to which the systems currently in use are meeting organizational needs. Question 2: How and to what extent have the FATA Secretariat, the FDA and the Governor’s Secretariat institutionalized the systems introduced by FISP? This question will assess the extent to which beneficiary partners have institutionalized the new systems introduced by FISP. Institutionalization includes recognition of the new system in the operating policies 53 of a department, an official decision to implement the system, training of staff in the new system, adoption of relevant manuals, allocation of resources required for implementation, maintenance, redundancy and upgrades, and the actual organizational practice. If all the concrete steps required for institutionalization have been taken, or have been planned for a specific future date, the systems are likely to be sustainable once FISP comes to an end. Question 3: What has been the usefulness of the policy-related reports prepared with FISP assistance? Explanation: The answer to this question will include an assessment of the reports in relation to their purpose (as evidenced by their terms of reference and the perceptions of the institutions that requested these reports). It will also include findings and conclusions on the continuing relevance of the reports and plans to implement the recommendations presented in the reports. Findings and conclusions relating to the report on the Women Empowerment Wing are placed under Question 4. Question 4: What evidence is there of women’s participation and a concern for gender issues in FISP interventions? Explanation: FISP has included women in its training activities. The answer to this question will examine sex-disaggregated data on the training provided by FISP. As mentioned under question 3, the answer to question 4 will also include findings and conclusions on the report prepared with FISP support on the Women’s Empowerment Wing in the FATA Secretariat. Question 5: What changes could better serve the project objectives and the priority organizational needs of the FATA Secretariat, FDA and the Governor’s Secretariat? Explanation: The answer to this question will consist of recommendations based on the answers to the preceding questions. The evaluation will recommend course-corrections within the design outlined in the FISP RFP and the duration and resources of the project. While USAID/Pakistan is not interested in changing the design of the information technology systems introduced by FISP, changes in the use of these systems to better suit the organizational needs for which they were designed will be considered. The methods by which data will be collected and analyzed for each question are summarized in Appendix 1 (the draft getting-to-answers table) and described below in greater detail. III. DESIGN AND METHODOLOGY A. Evaluation Process Evaluation activities will include a team planning meeting, desk review of policy-related studies, demonstration by the Implementing Partner (Abacus Consulting) of the systems currently in use, group interviews with Abacus Consulting on these systems, key informant interviews with USAID/Pakistan and government officials on the systems currently in use and the policy-related studies, focus group discussions with government officials trained by FISP, and data analysis and report writing. These activities are described below in greater detail. Stage 1: Preparation and planning by MEP staff. Before the evaluation team assembles in Islamabad, MEP’s full-time staff, led by the Evaluation Manager, will:  Draft a data analysis plan and instruments for group and key informant interviews and focus group discussions.  Prepare a detailed plan for the evaluation team’s training in qualitative data collection, recording and analysis. 54  Draft the initial sections and supporting annexes of the evaluation report based on available information. Stage 2: Initial review of priority documents by evaluation team. Working from their home stations, evaluation team members will review documents provided by MEP, including the drafts prepared by MEP staff for the evaluation team in Stage 1, and conduct a desk review of the seven policy-related studies prepared by FISP. Stage 3: Team work in Islamabad. Upon deployment in Islamabad, the evaluation team will participate in a team planning meeting facilitated by the MEP Evaluation Manager and undertake specific preparatory tasks as indicated below.  Discuss in detail the evaluation SOW and identify changes (if any) that need to be discussed with and approved by MEP and USAID/Pakistan.  Review available information to identify gaps and finalize data collection plans.  Refine data collection instruments for focus group discussion and key informant interviews, and finalize them after pre-testing and review by USAID/Pakistan.  Finalize the field work plan with inputs from MEP staff responsible for evaluation, security and travel.  Discuss evaluation standards and requirements, including, in particular, the evaluation logic that links findings, conclusions and recommendations.  Determine roles and responsibilities within the team and between the team and MEP staff, including for data collection, data analysis and report writing.  Participate in training on qualitative data collection, documentation and analysis, which will be facilitated by MEP.  Identify preliminary themes for analysis of qualitative data and develop a detailed analysis plan (including table formats) for analyzing and reporting survey data.  Meet USAID/Pakistan to discuss the SOW, data analysis plan and data collection tools in detail. Stage 4: Fieldwork. MEP will share the fieldwork schedule with relevant USAID/Pakistan and government officials in advance of the fieldwork. As soon as possible during the evaluation team’s Islamabad-based activities, the evaluation team will begin its fieldwork by interviewing USAID/Pakistan program managers (previous and present) responsible for FISP to obtain answers to evaluation questions in general, as well as their assessment of the seven studies conducted by FISP, in particular. The team will then go to Peshawar for data collection at FISP and government organizations. At the start of its field work in Peshawar, the team will attend demonstrations by FISP of the systems developed under the project and currently in use to understand the systems and their functions. Each demonstration will be followed by a group interview with the Abacus Consulting system developer and the manager responsible for system implementation. There will be 13 group interviews, one for each of the systems currently in use listed in Table 3. The team will then conduct 32 key informant interviews in Peshawar with the owner-cum-users, other users, and future users of systems currently in use and the commissioners of the studies conducted by FISP. The team will also conduct three focus group discussions with staff trained by the project, two with men, perhaps covering different kinds of training, and one with women staff members of the FS who have been trained by the project. 55 During Stage 4, the team will also provide MEP with initial findings and conclusions and possibly recommendations on a weekly basis. The following table summarizes the data collection methods and locations during the fieldwork: TABLE 4: DATA COLLECTION METHODS AND LOCATIONS Data Collection Method Location Initial review of key documents Consultant’s home station Key informant interview with USAID/Pakistan officials in Islamabad Islamabad Key informant interviews with owners and users, and adopters and non-adopters of IT systems FATA Secretariat, FATA Development Authority and Governor’s House at Peshawar Focus group discussions with male and female staff trained by the project FATA Secretariat, Peshawar Collect and take photographs showing systems developed by the project Peshawar Stage 5: Data analysis and report writing. The evaluation team will analyze data and write its report in Islamabad. It will work closely with MEP and interact with USAID/Pakistan as outlined below:  The evaluation team will initiate analysis of qualitative data during its field work and continue the work in Islamabad after all data collection is completed. The MEP Evaluation Unit will facilitate the team in this process.  Based on the analysis of the data, the team will prepare a summary of findings, preliminary conclusions and possible recommendations for discussion with MEP, USAID/Pakistan and the implementing partner, Abacus Consulting. MEP will facilitate a debriefing with USAID/Pakistan and Abacus Consulting. The discussion may lead to additional information, correction of factual errors in the evaluation team’s findings and feedback and perspectives on the team’s conclusions.  The team will utilize feedback from USAID/Pakistan and Abacus Consulting in preparing its draft report. USAID/Pakistan may share the report with Abacus Consulting for additional feedback.  The team, with inputs from MEP, will then prepare a final report that addresses comments from USAID/Pakistan and Abacus Consulting. B. Existing Data Documents available about the operation of the project include, but are not limited to the documents listed below. 1. The available baseline data - for most indicators, the baseline data is zero, since the systems being developed by FISP are new 2. FISP M&E Plan 2011-2016, Draft 2, November 04, 2011 3. FATA Capacity Building Program II, Year 1 Work Plan, October 2011 – September 2012, Draft 1. 4. FISP, Year 2 Work Plan, September 2012- August 2013. 5. FISP 5th M&E Plan Quarterly Report, September – December 2012. 6. FISP 6th M&E Plan Quarterly Report, January – March 2013. 7. FISP Annual Report, September 2012 – August 2013. 8. System Requirements Specification (SRS) for all the systems developed by FISP. 56 9. FISP Standard Operating Procedures (SOPs), IT Office Help Desk. 10. Study on Strengthening Women Empowerment Wing in FATA 2013. 11. List of Trainees: Information Gathering, Sharing, and Media Speaking Skills; HR Support; IT Training; PC Form Automation; Training on Zakat Management and others on office management. 12. Monitoring/Verification of FISP, Task Order Plan #100, MEP. 13. FISP Monthly Monitoring Report July 2013, Draft, dated August 16, 2013, MEP. USAID/Pakistan will provide MEP the FISP Technical Proposal of Abacus Consulting and help MEP to obtain any additional information that may be required during the evaluation. C. Data Collection Methods The evaluation team will apply qualitative techniques to collect data from multiple sources to ensure multiple levels of triangulation. The evaluation team will collect qualitative data through approximately 13 group interviews and a total of 34 key informant interviews with USAID/Pakistan, FISP program managers, and government officials working with the FATA Secretariat, FATA Development Authority and Governor’s Secretariat. It will also conduct three focus group discussions among male and female government staff who have received training through the project. The sampling frame for interviews is summarized in Tables 5-7 below. TABLE 5: KEY INFORMANT INTERVIEWS SAMPLING FRAME FOR USAID/PAKISTAN, ABACUS CONSULTING AND OWNER-CUM-USERS OF FISP SYSTEMS CURRENTLY IN USE Project Components/ Organization Key Informants Number of Key Informant Interviews Total Islamabad Peshawar USAID/Pakistan Current and Previous Program Managers 2 - 2 Abacus Consulting Chief of Party, FISP - 1 1 Zakat Management Information System (ZMIS) 1. Section Officer, Social Sector Department 2. Data Entry Officer - 2 2 No Objection Certificate (NOC) Management System 1. Director Projects in Planning &Development Department 2. Official from FATA Disaster Management Authority (FDMA) - 2 2 Litigation Cases Management System In-charge, Litigation Cell of AI&C Department 1 1  Human Resource Management Information System (HRMIS)  File Tracking System (FTS)  FATA Assets Management System (FAMS)  Employee Attendance System (EAS)  Digital Resource Access System 1. Deputy Secretary, AI&C Department or another representative – interviews on five systems 2. Representative of the FDA – interview on the HRMIS 3. Secretary to the Governor, Governor’s Secretariat – interview on the FTS 7 7 Public and Media Inter￾communication and Awareness Secretary to the Governor, Governor’s Secretariat 1 1 57 Project Components/ Organization Key Informants Number of Key Informant Interviews Total Islamabad Peshawar Mechanism (PMIAM) Planning Commission (PC) Forms Management System (PC-FMS) - Module 1: PC-I 1. Additional Secretary, Planning &Development Department 2. Chief Economist, Planning & Development Department 2 2 Online Public Feedback System Manager IT, FDA 1 1 Total 19 TABLE 6: KEY INFORMANT INTERVIEWS SAMPLING FRAME FOR USERS AND NON-USERS (FUTURE USERS) OF FISP SYSTEMS Project Components Key Informant from Total Users Non-Users File Tracking System (FTS) 2. Director Administration in Governor’s Secretariat 3. Director Health, Health Directorate 3. Director Forests, Forest Department 4. Director Fisheries, Fisheries Department 4 Planning Commission (PC) Forms Management System (PC-FMS) - Module 1: PC-I 7. Deputy Secretary, Social Welfare Directorate 8. [to be determined] 1. Director Minerals, Industries and Technical Education 2. Director Agriculture 4 Total 8 TABLE 7: KEY INFORMANT INTERVIEWS SAMPLING FRAME FOR COMMISSIONERS OF FISP POLICY-RELATED STUDIES Title of Study Commissioner of Study Number of Key Informant Interviews Study on Strengthening of Women Empowerment Wing in FATA Secretariat Deputy Secretary, Social Sector Department 1 Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat Former Secretary,52 Planning & Development Department 1 Draft Local Government Regulations for the FATA Local Government Director, Local Government & Rural Development 1 Development of IT Policy and E-Government Roadmap for FATA Director IT in AI&C Department 1 South Waziristan Agency Development Plan USAID/Pakistan 1 Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA USAID/Pakistan 1 FATA Investment Plan Study USAID/Pakistan 1 Total 7 52 This officer is currently posted in Balochistan and will be interviewed by electronic means. 58 MEP will conduct three focus group discussions with male and female staff members that received training from the project. Two focus group discussions will be conducted with male and one with female participants of these training. Each focus group discussion will have 10-12 participants. MEP will convey participants’ requirements to FISP for onward communication to the departments concerned in advance of the field work, so that participants can be mobilized in time for the evaluation team’s field work. D. Data Analysis Methods The evaluation team will employ rigorous analytical methods appropriate to the focus group discussion and key informant interviews. The team will use a structured approach based on identifying key themes, coding the responses and reporting the frequencies of responses by theme. The analysis of the qualitative data will yield an in-depth understanding of the level of improvements in terms of achieving efficiency, transparency, accountability and the quality of project plans, proposals, monitoring and reporting in FATA Secretariat, FATA Development Authority and Governor’s Secretariat. The team will also compare the findings from the interviews with users and non-users of the selected systems. The comparison between the users and non-users of these systems will provide further insights into the level of improvements in a comparative manner. E. Methodological Strengths and Limitations The scope of this evaluation is limited to the evaluation questions and does not extend to other aspects of the project. The evaluation will employ multiple levels of triangulation in terms of information sources and methods to ensure validity of findings. The key informant interviews and focus group discussions will help gain valuable insight into the project performance and help capture diversity in the views of the owners-cum-users, users and non-users about the FISP components. However, purposive sampling used in qualitative methods has the shortcoming of selection bias. To address this, the team will make every effort to ensure that diverse views are captured through the data collection. IV. EVALUATION PRODUCTS A. Deliverables The evaluation will produce the deliverables listed below. 1. Finalized draft SOW in collaboration with USAID/Pakistan FATA/KP Office and USAID/Pakistan Program Office. The SOW will meet USAID expectations for quality evaluation Statements of Work. 2. Instruments for document review, focus group discussions, group interviews and key informant interviews. 3. All data and records (qualitative and quantitative) from the evaluation in electronic and hard copies. 4. A debriefing discussion with USAID/Pakistan and Abacus Consulting on initial findings; 5. A draft evaluation report. 6. A final evaluation report consistent with USAID standards for evaluation reports (outlined below). 59 7. A presentation to USAID/Pakistan and Abacus Consulting on the final report. Field work will not start until USAID/Pakistan has approved the final SOW and commented on the instruments. B. Reporting Guidelines The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced in Appendix 2 of the SOW. The evaluation report will follow the structure given below (the section titles and order are illustrative).  Title page  Table of contents and table of tables and charts  List of acronyms  Acknowledgements  Project summary  Map showing the location of project activities  Executive summary (ideally not to exceed five pages)  Evaluation purpose and evaluation questions (2-3 pages)  Project background. This information provides important context for understanding the evaluation purpose, questions, methods, findings and conclusions and includes:  the problem statement;  the theory of intervention;  the design of the project;  the project’s results framework; and,  project implementation, including the current status of the project.  Evaluation methods and limitations (1-3 pages), describing in detail the evaluation design and methods with explanation as to why they were chosen, with additional information provided in the annexes, if so required.  The evaluation report should include a summary of data analysis and any relevant issues.  Findings and conclusions for each evaluation question. If there are a large number of findings, there will be a synthesis or summary of findings for each question that establishes the connection with the conclusions that follow.  Main conclusions and recommendations including those that form the logical basis for recommendations. It will highlight a few key recommendations, or clusters of recommendations, that include actionable statements of what remains to be done, consistent with the evaluation’s purpose, and based on the evaluation’s findings and conclusions. It will provide judgments on what changes need to be made for future USAID/Pakistan financial and cooperative development programming. This section will also recommend ways to improve the performance of future USAID/Pakistan programming and project implementation; ways to solve problems this project has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by management. 60  Annexes  Evaluation Statement of Work  Evaluation Methods and Limitations  Data Analysis Plan, including Data Collection Instruments  Bibliography of Documents Reviewed  List of individuals and agencies contacted  Disclosure of Any Conflicts of Interest  Statement of Differences (only if applicable)  Evaluation Team Bios V. TEAM COMPOSITION The evaluation team will be led by a consultant (Short-term Technical Assistance (STTA)) who should be an experienced evaluator and preferably an institutional development specialist with KP experience. The team will also include three MEP full-time staff (Long-term Technical Assistance (LTTA)). The team’s experience will include familiarity with GOKP and donor assisted projects, demonstrated experience in institutional development experience developing and implementing IT systems as well as expertise in qualitative research and evaluation. The requirements of individual team members are given below in detail: A Team Leader/Evaluator (STTA) will lead the team throughout the evaluation in accordance with the USAID Evaluation Policy and directives. S/he is expected to possess prior experience evaluating USAID projects preferably in KP/FATA. S/he should have a master’s degree in social sciences or another relevant field, demonstrated team leadership, report writing abilities, communication skills, and fluency in English and Urdu. The team leader/evaluator should also have demonstrated experience in institutional/organizational development. A Senior Evaluator/Capacity Development Specialist (LTTA) is expected to possess eight to twelve years of relevant experience in research and evaluation. He should have a master’s degree in social sciences and demonstrated experience in qualitative research methods and analysis, familiarity with gender issues, and must have fluency in English and Urdu. An IT Systems Specialist (LTTA) will assist the team by providing inputs on the designs of IT systems, data collection instruments and analysis of data. He is expected to have diverse experience in designing, developing and deploying IT systems and knowledge of FATA/KP institutions and procedures. The Director of Evaluation/Moderator (LTTA) will provide guidance on the design of the evaluation and provide overall supervision of the evaluation. He will also moderate focus group discussions. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. 61 VI. EVALUATION MANAGEMENT A. Logistics In terms of logistics, this assignment requires:  Travel to the project locations by the team leader/evaluator and the MEP staff responsible for the evaluation;  Travel cost reimbursement to the participants of focus group discussions (and key informant interviews, if the evaluation team cannot access certain locations in view of travel restrictions); and,  Organizing the key informant interviews and focus group discussions. The evaluation team will work under contract to MEP, which will be responsible for all travel, lodging, and other arrangements related to the team’s work. Prior to MEP issuing invitations, USAID/Pakistan will send a written notice to relevant government officials to inform them about the evaluation and introduce them to MEP. MEP has responsibility for identifying all key informants and focus group discussion participants, inviting them to participate, organizing meeting schedules, arranging venues, and handling any necessary transport and lodging arrangements for participants. 62 B. Scheduling TABLE 8: EVALUATION SCHEDULE Activity Jan. 2014 Feb. 2014 Mar. 2014 Apr. 2014 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Drafting of instruments Review of documents by consultant Drafting of initial chapters Team Planning Meeting (TPM) Field work – Focus group discussions and key informant interviews Data analysis Findings Conclusions and Recommendations (FCR) workshop with USAID/Pakistan and implementing partner X Report writing Internal review and revision Branding and editing Submission of draft final report X USAID/Pakistan comments, revision and submission of final report May 15- 25 63 C. Budgeting TABLE 9: LEVEL OF EFFORT OF TEAM MEMBERS Tasks Level of Effort (days) Team Leader/ Evaluator (STTA) Director Evaluation/ Moderator (LTTA) Senior Evaluator/Capacity Development Specialist (LTTA) IT Systems Specialist (LTTA) Evaluation Co￾Manager (LTTA) Pre-arrival  Review of documents 1 1 3 1 2  Drafting of instruments 1 1 2 1  Drafting of initial chapters 1 Team Planning Meeting 5 5 5 1 5 Field work – Focus group discussions, key informant and group interviews 15 4 15 3 3 Data Analysis 11 4 11 1 5 FCR workshop 1 1 1 1 1 Report writing – complete draft report 9 3 5 Internal review and revision 1 1 1 1 Final report 3 3 3 Total 48 23 46 9 16 64 SOW Appendix 1: DRAFT Getting To Answers (G2A) Matrix Evaluation Question Type of Answer/ Evidence Data Collection Data Analysis Methods Method Sources Sampling 1. In what ways and to what extent are the systems developed by FISP and currently in use addressing the organizational needs of the departments that own and use the systems, and what factors explain progress or lack thereof in this regard? Descriptive and Analytical  Approximately 13 group interviews and 27 Key informant interviews  Three focus group discussions with government officials trained by FISP  Review of relevant project records  Key informant interviews with FISP, USAID, officials from FS, FDA and Governor’s Secretariat  Focus group discussions with male and female training participants from FS, FDA and Governor’s Secretariat  Review of relevant project records such as reports and System Requirements Specifications (SRSs)  Purposive sampling for key informant interviews and focus group discussions Content analysis of key informant interviews and focus group discussions transcripts and secondary data 2. In what ways and to what extent have the FATA Secretariat, the FDA and the Governor’s Secretariat institutionalized the systems introduced by FISP? Descriptive and Analytical  Approximately 13 group interviews and 25 key informant interviews  Review of relevant project records and secondary data  Key informant interviews with FISP, USAID, officials from FS, FDA and Governor’s Secretariat  Review of relevant project records such as reports and memos and government documents such as notifications for institutionalizing the systems introduced by the project  Purposive sampling for key informant interviews. Content analysis of key informant interviews transcripts and secondary data 3. What has been the usefulness of the policy-related studies prepared with FISP assistance? Descriptive and Analytical  Approximately 7 key informant interviews  Review of relevant project records  Key informant interviews with FISP and USAID staff, and officials from FS, FDA and Governor’s Secretariat  Review of relevant project records such as studies and their SOWs.  Purposive sampling for key informant interviews. Content analysis of key informant interviews transcripts and secondary data such as studies and their SOWs 65 Evaluation Question Type of Answer/ Evidence Data Collection Data Analysis Methods Method Sources Sampling 4. What evidence is there of women’s participation and a concern for gender issues in FISP interventions? Descriptive and Analytical  Document review of study on Women Empowerment Wing  Three key informant interviews  One focus group discussion for female government officials trained by FISP  Review of FISP training records and FS employee numbers  Key informant interviews with FISP, USAID, one senior level official each from FS, FDA and Governor’s Secretariat  Review of relevant project records such as study on strengthening of Women Empowerment Wing and its SOWs.  Purposive sampling for key informant interviews and focus group discussion. Content analysis of key informant interviews transcripts and secondary data 5. What changes could better serve the project objectives and the priority organizational needs of the FATA Secretariat, FDA and the Governor’s Secretariat? Descriptive and Analytical  All methods mentioned above  All sources mentioned above  Methods mentioned above  Methods mentioned above 66 SOW Appendix 2: Reporting Guidelines 1. The evaluation report must represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action. Note: These guidelines are taken from ADS 203.3.2.8 (Documenting Evaluations) - http://transition.usaid.gov/policy/ads/200/203.pdf - which is based on Appendix 1 of USAID Evaluation Policy: Criteria to Ensure the Quality of the Evaluation Report. 67 ANNEX 3: OWNERS, USERS, DATA PROVIDERS AND CURRENT STATUS OF SYSTEMS DEVELOPED BY FISP No. System Name Owner-cum-User Other Users Main Role in Providing Data Current Status 1. File Tracking System (FTS) Services Wing of AI&C Department, FATA Secretariat AI&C Department; P&D Department; Law and Order Department. Directorate of Projects; Directorate of Monitoring and Evaluation; Directorate of Local Government and Rural Development; Directorate of Irrigation; Directorate of Information FATA; Owner-cum-user and all other users System is in use for the purposes for which it was intended 2. Employee Attendance System (EAS)  Administration Wing of AI&C Department, FATA Secretariat  FATA Development Authority All departments of the FATA Secretariat and FDA Owner-cum-user and all other users System is in use for the purposes for which it was intended 3. No Objection Certificate (NOC) Management System FATA Disaster Management Authority (FDMA) Donor and civil society projects in FATA System is in use for the purposes for which it was intended 4. Online Public Feedback System (OPFS) FATA Development Authority General public System is in use for the purposes for which it was intended 5. Public and Media – Inter communication and Awareness Mechanism (PMIAM) Governor’s Secretariat The FATA Information Directorate of AI&C is a potential user Governor’s Secretariat System is in use for the purposes for which it was intended 6. Digital Resource Access System (DRAS) Administration Wing of AI&C Department, FATA Secretariat All officers of FATA Secretariat have access Access to Higher Education Commission E￾library System is reportedly in use 7. Zakat Management Information System (ZMIS) Zakat and Ushr Section of the Social Sectors Department, FATA Secretariat Agency Zakat and Ushr officials Agency-level Zakat and Ushr officials System is in use for some of the purposes for which it was intended 68 No. System Name Owner-cum-User Other Users Main Role in Providing Data Current Status 8. Planning Commission (PC) Forms Management System (PC-FMS) - *Module 1: PC-I - Module 2: PC-III (A&B) - Module 3: PC-II, IV & V P&D Department, FATA Secretariat Other users/futures users of are: AI&C Department; Communication & Works Department; Social Sectors Department; Production and Livelihood Development Department; Directorates of Irrigation; Directorate of Local Government & Rural Development; Directorate of Education; Directorate of Health; Directorate of Social Welfare;: Directorate of Agriculture Extension; Directorate of Forests; Directorate of Fisheries; Directorate of Livestock & Dairy Development. All users System has been used for preparing PC-Is but is not yet functioning as intended. The Additional Chief Secretary of the FATA Secretariat has issued directives for all departments to use the system for submission of PC-Is from the financial year 2014-15. 9. Human Resource Management Information System (HRMIS)  Services Wing of AI&C Department, FATA Secretariat  FATA Development Authority All departments of FATA Secretariat and FDA - All the FS directorates and departments - All FDA departments Data entry ongoing 10. FATA Assets Management System (FAMS) Administration Wing of AI&C Department, FATA Secretariat Viewing rights to all the FS directorates and departments All departments of the FATA Secretariat Data cleaning not yet completed 11. Litigation Cases Management System (LCMS) Litigation Section of AI&C Department, FATA Secretariat Litigation Cell of AI&C Department Data entry ongoing 12. FATA Web Portal AI&C Department, FATA Secretariat Viewing and editing access to all departments and directorates for their respective web page All departments of FATA Secretariat and FDA Web Portal has been developed and content uploading is underway 13. Performance Management and Reporting System M&E Directorate of the P&D Department, FATA Secretariat The following will start data entry in January Social Sectors Department: Health and Education - All the FS directorates and departments Data entry ongoing 69 No. System Name Owner-cum-User Other Users Main Role in Providing Data Current Status (PMRS) Directorates; Production and Livelihood Development Department: Agriculture Directorate. The remaining directorates will start later on. (Please see the attached organogram for the names of remaining directorates) 14. Public Procurement System (PPS) AI&C Department, FATA Secretariat All departments of FATA Secretariat and FDA All users “As-Is” 53 analysis has been carried out and system development is underway 15. Financial Management System Finance Department, FATA Secretariat All departments and directorates All users “As-Is” analysis has been carried out and system development is underway 53 “As-Is” analysis is the analysis of an existing system and its problems on the basis of which a new system is designed. The new design is called the “To-Be” proposal. 70 ANNEX 4: DATA COLLECTION INSTRUMENTS FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Desk Review of Policy-Related Studies Desk Review Note Number: XXXX Title of Study Official and Organization Requesting the Study Name of Official Position Organization Name and Position of Reviewer Zubair Faisal Abbasi, Senior Evaluation Specialist, MEP Question 1 What is the purpose (management utility) of the study, as stated in its Terms of Reference (TORs) or equivalent document and in the study itself? Answer  Question 2 What is the study’s target audience, as stated in its TORs and in the study itself? Answer  Question 3 What is the methodology of the study (sources of information, data collection and methods, stakeholders identified and consultation required) as stated in its TORs? Answer  Question 4 What is the methodology followed by the study and how does it compare with the methodology described in the TORs? Answer  Question 5 What is the nature (for example, strategic or operational, brief or detailed, long-term or short￾term) of the recommendations? Answer  Question 6 To what extent are the recommendations actionable (specific, feasible and time-bound with responsibility specified) in relation to the purpose of the study? Answer  Question 7 When and in what ways were the stakeholders engaged in consultations during the study? [Prompts:(a) at the time of finalizing TORs; (b) during the time the study was being conducted; (c) during review of the draft report; (d) during post-study discussion and implementation of the study] Answer  Question 8 To what extent has the study addressed the purpose and management utility as stated in its TORs? Answer  Question 9 In what ways, if any, does this study discuss and address gender needs and issues? Answer  71 FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Individual Interviews on Policy Studies Introduction Thank you very much for joining us today for this session. My name is Ahmad Jameel and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. I would like to ask you a few questions about [name of study] and Mr. Zameer Haider will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as a government officer. Individual Interview Code: II nn Title of Study Date of Interview City Name and Position of Interviewer Name and Position of Note-taker Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Zameer Haider, Evaluation Officer, MEP Question 1 To what extent and how were you personally involved in the study? [Prompts:(a) in developing TORs; (b) during the time the study was being conducted; (c) review of draft report; (d) in post-study discussion and implementation of recommendations] Answer  Question 2 To what extent did you find the methodology adequate for addressing the purpose of the study? Answer  Question 3 In what ways were the stakeholders consulted and to what extent do you consider the consultations adequate? [Prompts:(a) in developing TORs; (b) during the time the study was being conducted; (c) review of draft report; (d) in post-study discussion and implementation of recommendations] Answer  Question 4 How useful (including actionable) are the recommendations of the study? [Prompts: any ideas to make it useful/actionable] Answer  Question 5 To what extent have the study recommendations been implemented or are likely to be implemented? When? Answer  72 Question 6 If recommendations have not been implemented, what are the reasons for the non￾implementation of recommendations? [Prompts: financial constraints, not in sync with the organizational culture, transfer of requesting official, lack of ownership by the management and changes in priorities] Answer  Question 7 What changes are required in the process of commissioning similar studies? Answer  Question 8 What changes are required in the process of conducting similar studies? Answer  Question 9 In what ways does this study discuss and address gender needs and issues? Answer  73 FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Individual Interviews with USAID Program Managers (Current and Previous), Chief of Party (FISP), Additional Chief Secretary of FATA and Director of Projects, Planning and Development Department, FATA Secretariat Introduction Thank you very much for meeting us today. My name is __________________ and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. I would like to ask you a few questions about FISP and Mr. _______________ will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as a government officer. Individual Interview Code: II nn System Discussed Date of Interview City Name and Position of Interviewer Name and Position of Note-taker Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Zameer Haider, Evaluation Officer Question 1 How would you describe the organizational needs of FS that are addressed by the systems developed by FISP? [Prompt: cover all systems, one by one.] 1. Planning Commission Forms Management System (PC-FMS) Module-1 2. Litigation Cases Management System 3. No Objection Certificate (NOC) Management System 4. Zakat Management Information System (ZMIS) 5. FATA Assets Management System (FAMS) 6. HR Management Information System (HRMIS) 7. Employee Attendance System (EAS) 8. File Tracking System (FTS) 9. Digital Resource Access System (Digital Online Library) 10. Online Public Feedback System 11. Public and Media – Inter communication and Awareness Mechanism (PMIAM) Answer  Question 2 In what ways, if at all, have the systems developed by FISP influenced efficiency? [Prompts: time saved, punctuality improved, absenteeism controlled]. 74 Answer  Question 3 In what ways, if at all, have the systems developed by FISP influenced accountability? [Prompt: adherence to rules and regulations] Answer  Question 4 In what ways, if at all, have the systems developed by FISP influenced transparency? [Prompts: public disclosure of information, rules, plans, processes and actions] Answer  Question 5 In what ways, if at all, have the systems developed by FISP influenced the quality of the system owner’s and user’s products? [Prompts: plans, PC-Is, monitoring reports] Answer  Question 6 In what ways, if at all, have the systems developed by FISP influenced service delivery by the FS, FDA, and Governor’s Secretariat? Answer  Question 7 How does USAID/the government/FISP [refer to one, depending on the interviewee] assure itself that: (a) systems; (b) policy studies; and (c) training developed by FISP are aimed at organizational needs articulated by the government or USAID? [Prompts: consultations, written requests from government, verbal requests, organizational analysis by FISP.] Answer  Question 8 Which systems have started meeting organizational needs as anticipated and how? Answer  Question 9 Which systems are not yet meeting the anticipated organizational needs and what factors explain lack of progress? Answer  Question 10 What successes have you observed in the institutionalization/ sustainability of the systems by the government and to what do you attribute these successes? [Prompts: Resources the users deployed – or planned to deploy – for: (a) maintenance of system; (b) redundancy; and (c) upgrades.] Answer  Question 11 What obstacles have you observed in the institutionalization/ sustainability by the government of the systems introduced by FISP and to what do you attribute these obstacles? [Prompts: Resources the users deployed – or planned to deploy – for: (a) maintenance of system; (b) redundancy; and (c) upgrades.] Answer  Question 12 Please explain whether and how the policy related studies prepared with FISP assistance have been useful. [Prompt: Discuss all studies one by one.] 1. Study on Strengthening of Women Empowerment Wing in FATA 75 2. Study of Functions, Structure and Planning Processes of the Planning & Development Department, FATA Secretariat54 3. Systems Development for Directorate of Local Government & Rural Development 4. Workforce/Job Market Assessment for Workforce Development Program in KP Province and FATA 5. Development of IT Policy and E-Government Roadmap for FATA 6. South Waziristan Agency Development Plan 7. FATA Investment Plan Study Answer  Question 13 What explains the lack of progress in relation to public-private partnerships and social audit/report cards included in the RFP under Component 5? What more can FISP do to facilitate progress? Answer  Question 14 What changes in project design and implementation could make it more effective in helping FS, FDA and the Governor’s Secretariat become service-oriented entities working in a transparent and well-planned manner55? Answer  Question 15 How useful (including actionable) are the recommendations of the study? [Prompts: Useful, actionable, any ideas to make it useful/actionable] Answer  Question 16 In what ways do the systems introduced and studies undertaken by FISP raise and address gender needs and issues? Answer  54 FISP conducted this study with its in-house staff and there are no terms of reference for it. 55 Respondents should feel free to interpret the terms “service oriented,” “transparent” and “well-planned” in their own ways. 76 FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Group Interviews on System Presentations by the Implementing Partner (Abacus Consulting) [Each Interview will include the Systems Developer and the Manager Responsible for the System] Introduction Thank you very much for joining us today for the presentation and interview. My name is __________________ and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. I would like to ask you a few questions about the system you have just presented to us and Mr. _______________ will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as a government officer. Group Interview Code: GI nn System Discussed Date of Interview Venue Name and Position of Interviewer Name and Position of Note-taker Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Zameer Haider, Evaluation Officer, MEP Question 1 Who is the: (a) owner-cum-user; and (b) other users of the system? Answer  Question 2 What is the problem or need in the department(s) concerned for which this system has been designed? Which official or institution identified the problem/need (USAID, FS, FDA, Governor’s Secretariat or Abacus? [Prompts: (a) efficiency: time saving, punctuality, control of absenteeism; (b) accountability: adherence to rules and regulations; (c) transparency: public disclosure of information, rules, plans, processes and actions; (d) quality of the system owner’s and user’s products: plans, PC-Is, monitoring reports; (e) service delivery by the FS, FDA, and Governor’s Secretariat; and others?] Answer  Question 3 In what ways, if at all, the systems developed by FISP have influenced efficiency? [Prompts: time saved, punctuality improved, absenteeism controlled]. Answer  Question 4 In what ways, if at all, have the systems developed by FISP influenced accountability? [Prompt: adherence to rules and regulations] Answer  77 Question 5 In what ways, if at all, have the systems developed by FISP influenced transparency? [Prompts: public disclosure of information, rules, plans, processes and actions] Answer  Question 6 In what ways, if at all, the systems developed by FISP have influenced the quality of the system owner’s and user’s products? [Prompts: plans, PC-Is, monitoring reports] Answer  Question 7 In what ways, if at all, have the systems developed by FISP influenced service delivery by the FS, FDA, and Governor’s Secretariat? Answer  Question 8 To what extent and in what ways has the system been incorporated into the government’s operational policies? How has this been done? [Prompts: in writing, verbal orders] Answer  Question 9 At what level of government has an official decision been made to implement the system among the users? What is the documentary evidence of this decision? [Prompts: (a) office order; (b) minutes of meeting; (c)noting on file; (d) verbal order] Answer  Question 10 What kind of training has been given to the system owners and users? [Prompts: (a) focus of training; (b) duration; (c) content; (d) participating organizations] Answer  Question 11 Which of the owners and users have: (a) received; and (b) started using the relevant manuals for implementing the system? Please provide copies of manuals. [Prompts: discuss and obtain evidence of use of manuals] Answer  Question 12 What resources have the owners and users: (a) deployed or (b) planned to deploy for implementation? What is the evidence for the answer? Answer  Question 13 To what extent and in what ways are the owners and users using the system? Answer  Question 14 What resources have the users deployed – or planned to deploy – for: (a) maintenance of system; (b) redundancy; and (c) upgrades? What is the evidence for the answers? Answer  Question 15 What other evidence is there that the systems introduced will continue to be maintained and used after the FISP project is over? Answer  Question 16 To what extent are the results of the designed system different from those expected initially? Answer  78 Question 17 What changes in the implementation and design of this system will better suit the needs of FS? Answer  Question 18 In what ways, if any, does the system address gender needs and issues? Answer  79 FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Individual Interviews with System Owners and Users Introduction Thank you very much for meeting us today. My name is __________________ and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. I would like to ask you a few questions about the [name of system] that was developed by FISP. My colleague Mr. _______________ will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as a government officer. Individual Interview Code: II nn System Discussed Individual Interview Represents Owner-cum-User/Other User Date of Interview City Name and Position of Interviewer Name and Position of Note-taker Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Zameer Haider, Evaluation Officer, MEP Question 1 What is the problem or need in your department for which this system has been designed? Which official or institution identified the problem/need (USAID, FS, FDA, Governor’s Secretariat?) [Prompts: (a) efficiency: time saving, punctuality, control of absenteeism; (b) accountability: adherence to rules and regulations; (c) transparency: public disclosure of information, rules, plans, processes and actions; (d) quality of the system owner’s and user’s products: plans, PC-Is, monitoring reports; (e) service delivery by the FS, FDA, and Governor’s Secretariat; and others?] Answer  Question 2 When the system was designed, in what ways was it expected to improve: (a) efficiency; (b) accountability; (c) transparency; (d) quality of your department’s products; and (e) service delivery in your department? Please give examples of the improvements you expected in specific processes and products. [Prompts: (a) efficiency: time saving, punctuality, control of absenteeism; (b) accountability: adherence to rules and regulations; (c) transparency: public disclosure of information, rules, plans, processes and actions; (d) quality of the system owner’s and user’s products: plans, PC-Is, monitoring reports; (e) service delivery by the FS, FDA, and Governor’s Secretariat; and others?] 80 Answer  Question 3 In what ways, if any, is the system actually improving: (a) efficiency; (b) accountability; (c) transparency; (d) quality of your department’s products; and (e) service delivery at this time in your organization? Please give before-and-after comparisons. [Prompts: (a) efficiency: time saving, punctuality, control of absenteeism; (b) accountability: adherence to rules and regulations; (c) transparency: public disclosure of information, rules, plans, processes and actions; (d) quality of the system owner’s and user’s products: plans, PC-Is, monitoring reports; (e) service delivery by the FS, FDA, and Governor’s Secretariat; and others?] Answer  Question 4 To what extent and in what ways has the system been incorporated into your department’s operational policies? How has this been done? [Prompts: in writing, verbal orders] Answer  Question 5 At what level has an official decision been made to implement the system in your organization? What is the documentary evidence of this decision? [Prompts: (a) office order; (b) minutes of meeting; (c) noting on file; (d) verbal order] Answer  Question 6 What kind of training has been given to the staff of your organization in the use of the system? [Prompts: (a) focus of training; (b) duration; (c) content; (d) participating units] Answer  Question 7 Has the staff in your organization: (a) received; and (b) started using the relevant manuals for implementing the system? What is the evidence for the answer? [Prompt: physical verification of presence of manual. Please discuss and obtain evidence of use] Answer  Question 8 What kind of resources has your organization: (a) deployed or (b) planned to deploy for implementation? If the answer is positive, what is the evidence for it? Answer  Question 9 To what extent is your organization using the system? Answer  Question 10 What kind of resources has your organization deployed – or planned to deploy – for: (a) system maintenance; (b) redundancy; and (c) upgrades? If the answer is positive, what is the evidence for it? Answer  81 Question 11 What other evidence is there that the system introduced will continue to be maintained and used after the FISP project is over? [Prompts: (a) funds allocated/budgeted, permanent staff assigned, staff trained, maintenance contract/responsibility formalized] Answer  Question 12 What changes in the implementation and design of this system will better suit the organizational needs? Answer  Question 13 In what ways the systems introduced by FISP discuss and address gender needs and issues? Answer  82 FATA Institutional Strengthening Project (FISP) Mid-term Evaluation Instrument for Individual Interviews with Non-Users (= Future Users) Introduction Thank you very much for meeting us today. My name is __________________ and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. I would like to ask you a few questions about the [name of system] that has been developed by FISP, which we understand is not yet fully functional in your department. My colleague Mr. _______________ will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as a government officer. Individual Interview Code: II nn System Discussed Interviewee Represents Future owner-cum-User/Future other User Date of Interview City Name and Position of Interviewer Name and Position of Note-taker Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Zameer Haider, Evaluation Officer, MEP Question 1 What are the needs of the department for which FISP has designed this system? Answer  Question 2 In what ways, if at all, do you expect the system to influence efficiency? [Prompts: time saved, punctuality improved, absenteeism controlled]. Answer  Question 3 In what ways, if at all, do you expect the system to influence efficiency? [Prompts: adherence to rules and regulations] Answer  Question 4 In what ways, if at all, do you expect the system to influence transparency? [Prompts: public disclosure of information, rules, plans, processes and actions] Answer  83 Question 5 In what ways, if at all, do you expect the system to influence the quality of the outputs of system owners and users? [Prompts: plans, PC-Is, monitoring reports] Answer  Question 6 Without this system being fully functional, what did does your organization lack in terms of: (a) efficiency; (b) accountability; (c) transparency; (d) quality of your department’s products; and (e) service delivery in your department? Please give examples with reference to specific processes and outputs. Answer  Question 7 At what level of government has an official decision been made to implement the system in your organization? What is the documentary evidence of this decision? [Prompts: (a) office order; (b) minutes of meeting; (c)noting on file; (d) verbal order] Answer  Question 8 What kind of resources has your organization deployed – or planned to deploy – for: (a) system maintenance; (b) redundancy; and (c) upgrades? If the answer is positive, what is the evidence? [Prompts: (a) funds allocated/budgeted, permanent staff assigned, staff trained, maintenance contract/responsibility formalized] Answer  Question 9 What changes in the implementation and design of this system will better suit the organizational needs? Answer  Question 10 In what ways does the system address gender needs and issues? Answer  84 FATA Institutional Strengthening Project (FISP) Mid-term Performance Evaluation Focus Group Discussion (FGD) Guide For Officials who Participated in FISP Training Programs FGD Code: FGD No. 1 Main Topic of Discussion Date of FGD Venue City Group Male 1/Male 2/Female Name and Position of Moderator Name and Position of Note-taker Tariq Husain, Director, Evaluation Unit, MEP Ahmad Jameel, STTA/Team Leader FISP Evaluation, MEP Introduction Thank you very much for joining us today for this focus group discussion. My name is _____________ and I represent the USAID Monitoring and Evaluation Program (MEP). This program helps USAID by independently monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MEP to conduct a mid-term evaluation of FISP and identify possible improvements for the remainder of the project duration. USAID is keen that this should be a learning exercise for USAID as well as Abacus (the Implementing Partner) and the government departments with which FISP is working. The evaluation report will be disclosed to the public when it has been finalized. The main subject of this discussion is the training you have received through FISP. We are also interested in knowing about your experiences with the new systems introduced through FISP. We would like all of you to discuss these subjects frankly as a group and in so doing share your thoughts with us. I will facilitate discussion by asking a few questions, one at a time, and try to ensure that everyone feels free to participate in the discussion. Please allow everyone to express their views and focus on the questions. My colleague _________________ will take notes. Confidentiality With your permission, we would like to record this discussion so that we accurately capture what you say. Please be assured that your responses will be kept strictly confidential. MEP will not share the data with anyone and will not identify individual responses in the evaluation report by name. We can now start with by introducing everyone. Please mention your name, position and department. Thank you for your participation in this discussion. Questions Prompts General Question/ Ice Breaker 1. Most or perhaps all of you are working with computers in government offices. In what ways are computers changing the work  Efficiency  Accountability  Quality of work/products 85 Questions Prompts environment in government organizations?  Service delivery  Cost of operations  Employment  Flexibility Specific Questions 2. On what basis were you selected for the training?  By virtue of participant’s job position  Because of participant’s own interest  Because the participant is a user of the system 3. What arrangements were made to facilitate your participation in the training?  Transportation  Accommodation  Convenient venue and timing  Special arrangements 4. What did the training program include?  Practical work  Lectures and question-answer sessions  Discussion and group work  Demonstration of new system  Other features of training 5. What did you learn from the training?  To use the a new system introduced by FISP  Communicate better with the media  Improve plans and project proposals  Improve monitoring and evaluation  Improve management 6. How do you feel about various aspects of the training?  Contents  Trainer  Duration  Group work  Practical training  Venue  Ask what they liked the most and the least 7. In what ways are you applying what you learned during the training?  For data entry  For generating reports using the new system  For making better products, plans, projects  General management 8. What factors influence the extent to which you are using the skills you learned through training?  Opportunity  Office environment  Equipment and other resources (specify)  Cultural barriers  Required data not yet entered 9. In what ways, if any, has the training affected your status in the government?  Increased work load  Earned recognition as an expert  Helped with a promotion  Helped move to a more desirable organization 10. How does your organization deal with the transfer of trained staff to other departments?  Arrange training for incoming staff prior to inducting them on the job 86 Questions Prompts  Ask trained staff to train incoming staff on the job  Does nothing about it 11. What improvements should FISP consider in organizing future training activities?  Duration  Content  Trainers and interaction with them  Logistics and timings  Venue (particularly outside of the office) 12. What kinds of changes have taken place in your organization as a result of the new systems introduced with the help of FISP?  Affected efficiency (how?)  Affected transparency (how?)  Affected accountability (how and to whom?  Affected quality of products (which ones?)  Affected service delivery 13. To what extent are the new systems accepted by your organization?  Accepted by all or almost all  Accepted by specific groups of employees  Generally ignored  Why?/Why not? 14. What factors have facilitated the implementation of the new systems in your organization?  Commitment of top officials  Interest of system users  Cooperation of staff  Features of the system  Training  Allocation of necessary resources 15. What are the hurdles in implementing new systems in your organization?  Financial constraints  Logistical constraints (e.g., internet & electricity breakdown, difficulties in maintenance)  Non-availability of equipment such as computers  New system/recommendation not in sync with the rest of the system  New system not user friendly  Mindset and resistance to change  Transfer of trained staff 16. What needs to be done to ensure that the new systems continue to work efficiently in future?  Ensure its full implementation  Arrange more training and refresher training  Ensure adequate funding  Make it more user-friendly (how?)  Overcome resistance to change (how?)  Repair and maintenance of system 87 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523