ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT ASSESSING THE THEORY OF CHANGE AND IMPACT OF USAID/PAKISTAN’S ECONOMIC GROWTH AND AGRICULTURE PROGRAM 16 MARCH 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP). ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT ASSESSING THE THEORY OF CHANGE AND IMPACT OF USAID/PAKISTAN’S ECONOMIC GROWTH AND AGRICULTURE PROGRAM Contracted under Order No. AID-391-C-13-00005 Monitoring and Evaluation Program (MEP) DISCLAIMER This study/report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 i ACKNOWLEDGMENTS Responsibility for the development and content of this impact assessment lies with the assessment team, but it would not have been possible without input from a host of stakeholders and supporters. Primary among these, we acknowledge Randy Chester and Nazim Ali of USAID/Pakistan for their invaluable support and technical guidance on this assessment. We would also like to thank Matthew Hermerding, Kaleem Ullah Khan and Sherry Ward for promoting M&E at USAID/Pakistan. This assessment would not have been possible without the participation and support of hundreds of Pakistani men and women who volunteered their time in responding to the survey and providing feedback on our research questions in individual interviews and focus groups. We are grateful for their confidence. Authors: John A. Berry Holly Hapke Hussain Samad Eshya Mujahid Katharine Wheatley Hoffman Fatima Abbas Jennifer L. Mandel ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 ii CONTENTS ASSESSMENT PURPOSE ................................................................................................................... V PROGRAM BACKGROUND .............................................................................................................. V EXECUTIVE SUMMARY .....................................................................................................................1 PROGRAM BACKGROUND ............................................................................................................... 2 ASSESSMENT PURPOSE AND RESEARCH QUESTIONS ............................................................ 6 THEORETICAL FRAMEWORK ........................................................................................................ 7 ASSESSMENT DESIGN AND METHODOLOGY ...........................................................................14 Methodological Limitations.................................................................................................................................... 25 FINDINGS AND CONCLUSIONS................................................................................................... 27 Question 1 ................................................................................................................................................................. 27 Question 2 ................................................................................................................................................................. 32 Question 3 ................................................................................................................................................................. 40 RECOMMENDATIONS .................................................................................................................... 54 ANNEXES ........................................................................................................................................... 56 Annex I: Map of USAID/EGA Intervention Zones ........................................................................................ 56 Annex II: EGA Results Framework ..................................................................................................................... 57 Annex III: Descriptions of USAID EGA Methodologies ................................................................................ 58 Annex IV: Impact Assessment Indicators ........................................................................................................... 60 Annex V: Impact Assessment Survey Questions ............................................................................................... 63 Annex VI: Sources of Information ....................................................................................................................... 68 Tables and Figures Table 1: Summary of USAID/Pakistan Economic Growth Projects ........................................................................ 5 Table 2: Initial Household Survey Sample Size and Composition ............................................................................17 Table 3: Actual Survey Sample Size and Composition ...............................................................................................21 Table 4: Actual FGD Sample Size..................................................................................................................................22 Table 5: Opportunity and Empowerment Indices ......................................................................................................24 Table 6: Control Variables Used in Regression ...........................................................................................................25 Table 7: Household Access to Markets and Financial Services (Percent of Respondents with Access)..........................................................................................................................................................33 Table 8: Perceptions of Community Values .................................................................................................................39 Table 9: Outcome Indices by Beneficiary Status .........................................................................................................43 Table 10: Difference in Outcome Indices Between Beneficiary Types by Province .............................................44 Table 11: Difference in Outcome Indices Between Beneficiary Types by Gender ...............................................45 Table 12: Regression Estimates of EGA Program Effects on Outcome Indices ..................................................46 ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 iii Table 13: Regression Estimates of EGA Program Effects on Outcome Indices by Province ............................48 Table 14: Regression Estimates of EGA Program Effects on Outcome Indices by Gender ..............................49 Figure 1: Beneficiaries Surveyed by EGA Impact Assessment ................................................................................... 6 Figure 2: Theory of Change ............................................................................................................................................10 Figure 3: Impact Assessment Design Components ....................................................................................................15 Figure 4: Selection of Non-program Villages ...............................................................................................................20 Figure 5: Ownership of Productive Assets by Beneficiary Types (Percent)............................................................28 Figure 6: Household Dwellings and Sanitation by Beneficiary Types (Percent) ....................................................29 Figure 7: Sources of Income by Beneficiary Types .....................................................................................................30 Figure 8: Respondents' Mobility and Freedom by Beneficiary Types (Percent) .....................................................35 Figure 9: Individual Decision-making Ability on Household Spending (Percent) .................................................36 Figure 10: Decision-making Ability on Children’s Issues (Percent).........................................................................37 Figure 11: Respondents' Freedom on Various Issues by Beneficiary Type (Percent) ...........................................38 Figure 12: Revised Theory of Change ...........................................................................................................................53 ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT iv ACRONYMS ASF ANOVA Agribusiness Support Fund Analysis of Variance DRDF DO DB Dairy and Rural Development Foundation Development Objective Direct Beneficiary EGA Economic Growth and Agriculture FATA Federally Administered Tribal Areas FEG Farmer Enterprise Groups FY FGD GB Fiscal Year Focus Group Discussion Gilgit-Baltistan G2G Government-to-Government GDP Gross Domestic Product GOP IB Government of Pakistan Indirect Beneficiary IR Intermediate Result KP MEDA Khyber Pakhtunkhwa Mennonite Economic Development Associates M&E Monitoring and Evaluation MEP Monitoring and Evaluation Program MSI NB Management Systems International Non Beneficiary PPL PIA Policy, Planning and Learning Portfolio Impact Assessment PSU Primary Sampling Unit SME Small and Medium Enterprises SOW Statement of Work USAID U.S. Agency for International Development ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 v ASSESSMENT PURPOSE To expand the depth and breadth of the United States Agency for International Development (USAID)/Pakistan’s learning agenda, 1 the Economic Growth and Agriculture (EGA) Office has commissioned portfolio-wide research on the cumulative impact of its programming at the individual, household, and community levels. Management Systems International’s (MSI’s) Monitoring and Evaluation Program (MEP) designed and implemented the first phase of an impact assessment to explore how EGA activities have collectively contributed to the impact of USAID programming in Pakistan as part of its ongoing support for EGA’s learning agenda activities. The underlying purpose of this assessment is three￾fold: to test the validity of EGA’s theory of change, to measure the impact of EGA programming on the economic and social empowerment of program beneficiaries, and to offer insights for improving the planning and implementation of EGA activities. PROGRAM BACKGROUND The EGA program is a key component of USAID/Pakistan’s Results Framework Development Objective-2 (DO-2)2 : Improved Economic Status of Focus Populations. EGA’s program strategy focuses on increasing income￾generating and employment opportunities, especially in the agricultural sector. The program is grounded in the belief that the modernization of Pakistan’s agriculture sector is the most direct pathway to generating broad income growth, increasing food security and fostering the rural population’s stake in a stable government.3 The EGA program works with local and U.S.-based partners to strengthen the competitiveness of Pakistan’s agricultural value chains, increase technological innovation, improve management practices and promote water-use efficiency.4 Other USAID/EGA programs also seek to improve the legal and regulatory environment for businesses, with a focus on policy issues such as trade with neighboring countries.5 The intended beneficiaries of EGA activities include small farm holders and entrepreneurs in agricultural and non-agricultural value chains. Additionally, EGA is providing support to small and medium enterprises (SMEs), business management organizations, and farmer/community organizations. Although EGA works nationally to promote income and job growth, several districts are the subjects of direct interventions. The provinces selected for this assessment are areas in which EGA is currently active and where it anticipates concentrating future activities. 1 According to USAID’s Learning Lab, a learning agenda is defined as a set of questions related to an organization’s work that, when answered, will help the organization work more effectively. In the development context, learning agendas are often used to prove or disprove untested assumptions in development hypotheses. Learning agendas help shape research and evaluation plans. 2 EGA’s Development Objective and Intermediate Results are summarized graphically in Annex II. 3 USAID, “Economic Growth Strategy” for Pakistan, Draft Working Paper, February 19, 2011. 4 Ibid. 5 Ibid. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT 1 EXECUTIVE SUMMARY This assessment addresses a key EGA program learning agenda question: “How effective have USAID/EGA interventions been at empowering beneficiaries and promoting social and economic change?” To answer this question, the framework concentrates on three main lines of inquiry:  What is the current economic status of EGA project beneficiaries?  What is the current level of economic and social empowerment of EGA beneficiaries?  Have USAID-supported activities influenced EGA program beneficiaries’ individual and household economic opportunities, economic and social empowerment and communal social change? Implementation of this research took place in two stages: an individual-level survey comprised of 5,419 respondents and focus group discussions (FGDs) with 240 participants in three provinces.6 Survey data were compiled and analyzed using a range of statistical techniques including descriptive analysis, factor analysis, and regression analysis. FGD notes were analyzed using comparative content pattern analysis, in which the frequency of themes was scored to assess patterns and trends across and within participant types. Research findings then were further analyzed to identify data that were statistically significant, socio-economically significant, and programmatically significant for USAID. The results of this analysis were mapped to the research questions to assess the influence of EGA activities on individual and household economic opportunities, economic and social empowerment, and communal social change. Based on a portfolio-wide analysis of direct, indirect, and non-beneficiaries, the assessment found that EGA has had deep direct impact and broad indirect impact at the individual, household and community levels. Further, EGA has increased beneficiaries’ economic and social empowerment and well￾being and improved their ability to make and act on decisions, control resources and advance economically and socially. Positive change in a range of inter-connected economic and social empowerment indicators appears linked to access to education, markets and services and, most importantly, to USAID beneficiary status. In addition, the influence of women’s education and participation in the workforce extends across a spectrum of development outcomes. While EGA does not directly intervene in some of these domains, the influence of beneficiary status complements and supports other socioeconomic changes. For example, EGA does not support access to primary education, but EGA beneficiaries have a greater tendency to ensure education for their girls than do non-beneficiaries. Complex circles of diverse indicators influence a range of development outcomes. Consider a virtuous circle where girls’ education leads to improved skills among women, which increases their economic opportunities, thereby offering them greater voice in household decisions regarding educating their children and a stronger position to promote girl’s access to education. Both survey and focus group data demonstrate strong positive links between beneficiary status and economic opportunity, asset ownership, housing and sanitation. Beneficiaries also reported better access to markets and ability to do business than non-beneficiaries. In addition, EGA project beneficiaries had greater access to both productive and non-productive assets. They were also more likely to be self-employed (as opposed to wage laborers) and to engage in non-agricultural occupations. Female beneficiaries, in particular, reported more and better employment opportunities than female non-beneficiaries. 6 Provinces covered include Punjab, Sindh and Khyber Pakhtunkhwa (KP). ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 2 The assessment found that EGA has had positive direct and indirect influence on the complex interrelations between opportunity, empowerment and social change.7 After controlling for confounding influences and biases, survey data demonstrated statistically that EGA program participation has its greatest direct influence on economic empowerment and on individual perceptions of social change. In addition, economic empowerment has an important influence on social empowerment and social change. The highest-level development outcome, social change, is influenced by economic opportunity, social empowerment and to a lesser degree economic empowerment. Thus the influence of economic opportunities created by EGA interventions can be traced to both economic empowerment and social change. Additionally, although EGA participant status does not have a statistically significant impact on social empowerment, it does have an indirect influence through economic empowerment. Statistical analysis demonstrates that while EGA beneficiaries have a greater probability of economic opportunity, economic empowerment and positive social change in general, the nature of this impact varies by gender. For example, male beneficiaries are more likely to have economic opportunities, but female beneficiaries are more likely to experience economic empowerment and social change. EGA program influence also varies by province. For example, beneficiary status has the greatest influence on development outcomes in Punjab, followed by Sindh and the least impact in Khyber Pakhtunkhwa (KP). At the same time, indirect beneficiaries in Punjab report significantly higher probability of reaping benefit, particularly in terms of social empowerment and social change, than those in the other two provinces. In documenting the complex and interrelated impact of EGA programs on different social groups and regions, this study provides USAID with information and perspective to improve future planning and direct future research on program impact. Additionally, the second and third rounds of this study will enable researchers to accurately measure the impact of EGA interventions over time and to demonstrate the continuum of influences of EGA programming. PROGRAM BACKGROUND Pakistan Context Political instability, a deteriorating macroeconomic environment and low levels of investment have slowed Pakistan’s economic development, leading to chronically low incomes and growth rates. In addition, Pakistan has experienced average inflation of 10.2 percent against average annual growth in gross domestic product (GDP) of 3.6 percent over the past five years (2010–2014). 8 As a result, purchasing power has declined across the country as inflation has continued to drive up the cost of living, leaving poor families with less money to buy increasingly expensive goods. Low growth rates, low productivity, declining terms of trade and high inflation have contributed to widespread poverty rates. High poverty rates have also disproportionately affected rural areas; while 63 percent of the Pakistani population lives in rural areas, more than 90 percent of Pakistan’s poor are rural. Food insecurity typically affects close to half the population, a situation that was worsened for many 7 For the purposes of this study social empowerment is defined as an individual’s or group’s capacity to make purposive independent choices regarding to key aspects of their lives (e.g., health care, education and marriage). Economic empowerment is considered as possessing the ability to advance economically and to make and act on economic decisions. Social change is considered as the positive transformation of norms and institutions within the household and at the community level that support the power and agency of less-empowered groups of individuals. 8 Pakistan Bureau of Statistics, Trading Economics http://www.tradingeconomics.com/pakistan/inflation-cpi ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 3 households by the 2010 floods. In addition, the areas of Pakistan with the highest level of food insecurity are both the hardest to access and most susceptible to extremist influence.9 While the official unemployment rate for 2013 was 6.6 percent, much of the economy is informal (particularly in rural areas) and underemployment remains a persistent problem. In addition, more than half of Pakistan’s population is under 22 years of age, which creates both a present and future challenge to promote new employment opportunities for the 2 million young people who enter the labor market every year.10 Although small and medium enterprises could help to provide new employment, they often lack access to finance and non-financial support, substantially reducing their growth potential.11 Agriculture, which represents the second-largest economic sector in the country, is one of the largest employers in Pakistan. However, inefficient farming and irrigation practices and declining agricultural R&D have limited agricultural productivity. Agriculture accounts for more than 45 percent of employment and it contributes 25 percent to Pakistan’s GDP.12 Weak agricultural production/supply chains from producer to consumer and deficiencies in rural support service providers have contributed to Pakistan’s inability to achieve its full potential in both competitiveness and growth rates.13 Program History The United States Agency for International Development’s (USAID’s) Economic Growth and Agriculture (EGA) Office aims to facilitate Pakistan’s economic development by improving enterprise productivity (especially in agriculture), enhancing trade and promoting an enabling environment that supports market-led economic growth. The EGA portfolio is based on the theory of change that increased economic opportunity and job creation will improve the economic status of ordinary Pakistanis. The EGA results framework (see Annex II) articulates the development strategy in terms of one Development Objective (DO), two Intermediate Results (IRs), and five sub-IRs. Economic Growth and Agriculture Results Framework DO 2: Improved Economic Status of Focus Populations IR 2.1: Improved Economic Performance of Selected Enterprises IR 2.1.1: Increased Access to Finance IR 2.1.2: Improved Skill Development and Job Placement IR 2.1.3: Increased Use of Modern Technology and Management Practices IR 2.2: Improved Business Enabling Environment IR 2.2.1: Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations IR 2.2.2: Strengthened Private Sector and Civil Society Engagement in Policymaking 9 Sustainable Development Policy Institute, 2009. 10 Rana, 2012. http://tribune.com.pk/story/370522/short-of-expectations-unemployment-surges-as-growth-falls-short-of-target/. 11 SME Development in Pakistan: Issues and Remedies; SME Policy of Pakistan (2005) http://www.gcu.edu.pk/publications/vc￾sme.pdf 12 Sarwar, 2014. Problems of Agriculture in Pakistan http://www.bookhut.net/problems-of-agriculture-in-pakistan/ 13 Ibid. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 4 Linking this results chain with larger development objectives, the EGA program's theory of change states that improved economic status (increased in employment, income and expenditure) leads to increased economic empowerment (greater decision making power related to economic matters at household level) and social empowerment (greater decision making power related to social matters at household level) as well as social change (transformation of norms and institutions supporting less empowered individuals and groups). Indicators from the EGA program’s Results Framework highlight the importance of agricultural value chains and policy reform as well as capacity building (e.g. access to finance, skill development and job placement, use of modern technology and management practices). The EGA program also seeks to improve the business enabling environment (e.g. new/revised policies, increased political participation in policy making), and overall household economic status (e.g. household income/expenditure and employment). Although the specific approaches used by EGA’s value chain development projects vary, they are all defined by where in the value chains they intervene, the kinds of markets they target, whether they are oriented primarily toward production or marketing, the size of enterprises targeted, the type of assistance or training provided, and the extent to which they emphasize women’s participation. EGA Program Interventions EGA currently manages a portfolio of eleven projects worth a total value of $407.7 million. Almost half (5 of 11) of the projects in the EGA portfolio employ value chain development approaches. While many of these focus on agricultural products, a few also emphasize non-agricultural sectors (e.g., marble and hand￾embellished fabrics). In addition, two projects focus on policy-level issues in the business-enabling environment for agricultural value chain development. Specific EGA interventions are aimed at improved access to markets, workforce development, introduction of new technology and best practices, increased agricultural productivity and improved water management (see Annex III for a complete list of EGA Methodologies).14 The projects chosen for this assessment were selected by the EGA program as offering the best perspective on the individual- and household-level impact of USAID activities.15 Projects selected include the Agribusiness Project, the Dairy Project, the Entrepreneurs Project and the Firms Project. A brief description of each project follows in Table 1 below. 14 USAID, “Economic Growth Strategy” for Pakistan, Draft Working Paper, February 19, 2011 15 Policy reform projects were excluded from the sample as they do not have any direct beneficiaries and it was therefore impossible to measure their beneficiary-level impact. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 5 TABLE 1: SUMMARY OF USAID/PAKISTAN ECONOMIC GROWTH PROJECTS Project Name Implementing Partner Value Provincial Presence Dates Agribusiness Project Agribusiness Support Fund (ASF) $39.9 million16 KP Punjab Sindh GB Nov 2011 – Nov 2015 Creates employment opportunities and reduces poverty through increasing the competitiveness of horticulture and livestock value chains. Dairy Project Dairy and Rural Development Foundation (DRDF) $14.0 million KP Punjab Sindh Jul 2011 – Jul 2014 Increases incomes of smallholder milk producers by improving the quantity and quality of milk production. Entrepreneurs Project Mennonite Economic Development Associates (MEDA) $30.0 million KP Punjab Sindh June 2009 - June 2014 Designed to build capacity to scale up the operations of micro- and small enterprises, particularly those led by women, and provide business support services as well as emergency grants. Firms Project Chemonics International $92.3 million KP Punjab Sindh May 2009 – Dec 2014 Encourages increased exports, expanded employment, higher-quality products and services, and a supportive policy environment. Total $176.2 million Source: USAID, http://www.usaid.gov/pakistan In designing the survey and focus group discussions (FGDs), no intentional distinction was made among beneficiaries from these four different EGA projects or among the three different provinces where they are implemented. However, in implementing this research, significant differences emerged between the numbers and types of project beneficiaries in each of the three regions. As a result, the project with the greatest number of beneficiaries overall did not have the greatest number of FGD participants. For example, although the Dairy Project has the smallest budget of all four projects, it had the largest overall number of beneficiaries in the survey sample. The distribution of beneficiaries sampled also varied by province. On a provincial basis, Punjab had the largest overall number of beneficiaries, followed by Sindh and KP. Additionally, the Dairy Project had the most beneficiaries in Punjab, while the Agribusiness Project had the largest number of beneficiaries in KP. The majority of the Firms project beneficiaries are in KP (see Figure 1). 16 This figure represents a reduction in budget to $39.95 million from the original total value of the Agribusiness project, which was $89.4 million. This was the result of a reduction in the project’s scope that took place in mid-2013. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 6 FIGURE 1: BENEFICIARIES SURVEYED BY EGA IMPACT ASSESSMENT Source: Impact Assessment Survey Note: Number of participants in Agribusiness, Dairy, Firms and Entrepreneurs are 201, 335, 1 and 25 respectively in Punjab; 185, 180, 5 and 105, respectively in Sindh; and 265, 143, 143, and 78, respectively in KP. ASSESSMENT PURPOSE AND RESEARCH QUESTIONS The overall objective of the assessment is to measure whether and how USAID-funded activities have changed beneficiaries’ lives. Specifically, the assessment aims to test the EGA program’s theory of change, assess the impact of its interventions on project beneficiaries and offer insights for improving the planning and implementation of future USAID/Pakistan programs. The assessment does not seek to measure the progress of individual projects toward EGA’s intermediate results, but rather to take a broader view of the EGA program’s overall impact on social and economic empowerment and social change. The main audience for this impact assessment is USAID/Pakistan’s EGA program. The findings and conclusions regarding the impact of EGA’s portfolio are intended to help improve future USAID/Pakistan and EGA programming. This assessment also should encourage the development of similar learning agendas in other USAID offices and missions. Implementation of the assessment was structured to include a sample large enough to allow for a mixture of analytical techniques and to yield as accurate and comprehensive a picture of EGA program impact as possible. The assessment was designed to address the EGA program’s overarching learning agenda question, ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 7 “How effective were USAID/EGA economic growth and agriculture interventions at empowering beneficiaries economically and socially, and promoting social change?” To do so, the research framework follows three main lines of inquiry:  What is the current status of EGA project beneficiaries in terms of economic opportunity, income and expenditures?  What is the current level of economic and social empowerment (e.g. control over livelihoods, participation in decision-making at the household and community level, etc.) of the beneficiaries of EGA projects?  To what extent, and in what ways, have USAID-supported activities influenced EGA program beneficiaries’ employment, income, expenditure, economic empowerment and social empowerment? These questions can only be addressed with a clear understanding of the pathways through which interventions lead to longer-term program goals and objectives. In order to create a conceptual foundation for this research framework and questions, the assessment team worked with the EGA program to fully flesh out a theory of change that hypothesizes how these pathways of cause and effect impact USAID’s development objectives. The framework that follows operationalizes this theory of change, using academic literature to define and describe the relations between the key concepts underlying the research questions: opportunity, empowerment and social change. The results chain captured by the EGA theory of change frames the relationship between EGA program intentions, interventions and impact. The framework provides definitions of terms embedded within the theory of change and informs the research by explaining, theoretically, how the theory of change is supposed to operate. THEORETICAL FRAMEWORK Theory of Change The fundamental hypothesis of this assessment follows the EGA development results, which are to stimulate economic opportunity, creating new jobs and increasing beneficiary incomes. Following this causal chain, increases in economic opportunity are expected to lead to economic and social empowerment of beneficiaries and their households, and to positive social change at the community level17 (see theory of change in Figure 2). Embedded in this theory of change is a set of assumptions about the relationships among development, empowerment and social change. These assumptions are:  Enhanced economic opportunity leads to economic and social empowerment for both individuals and households;  Enhanced economic and social empowerment at the individual and household levels lead to social change at the community level;  Increased employment and income-earning opportunities for women lead to their greater economic and social empowerment within the household; and  Increased empowerment of women within households will result in positive social change in gender norms at the community level. The overarching goal of this assessment is to test the extent to which this theory of change is valid using qualitative and quantitative methods, allowing us to understand the dynamic relationships among 17 USAID Pakistan 2013. Mission Strategic Framework – Pakistan, Fiscal Year 2013-2017, Background and Narrative. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 8 socioeconomic opportunity, empowerment and social change. To evaluate these relationships, the assessment posed several key questions.  Does enhanced economic opportunity for individuals and households lead to greater economic and social empowerment?  If so, what kinds of social transformations take place when individuals and households have increased economic opportunity?  Do increased income-earning opportunities for women lead to greater economic and social empowerment within the household?  What factors (both endogenous and exogenous to EGA interventions) are most important in the process of empowerment?  Does women’s empowerment at the household level lead to social change and improved social status for women at the community level? What Is Empowerment? The theoretical framework adopted for this assessment conceptualizes empowerment as “an individual’s or group’s capacity to make purposive choices and to transform those choices into desired actions and outcomes.”18 In a socioeconomic context, individuals are empowered when they possess “the ability to succeed and advance economically and to make and act on economic decisions”19 and are able to make decisions independently related to other key aspects of their lives (e.g., health care, education and marriage). The three interrelated and inseparable elements of empowerment are resources (preconditions), agency (autonomy to act) and achievements (outcomes).20 (These elements track closely with the key research questions related to economic opportunity, socioeconomic empowerment and social change.) From a socioeconomic development perspective, empowerment begins with individuals and their access to resources, capacity for agency and ability to achieve. Ideally, individual empowerment aggregates across households over time to result in a positive transformation in values, norms and ideologies at the community level, which, in turn, supports the agency of previously disempowered individuals. Empowerment may, therefore, be understood as both a process of change and an outcome. Empowerment changes people’s ability to make strategic life choices by gaining greater control over both the material circumstances (physical, human, intellectual, financial resources) and ideological underpinnings (beliefs, values, attitudes) of their lives.21 When empowerment is achieved for individuals or groups within a society, the outcome is social change. However, because empowerment is fluid and dynamic, it may either advance or reverse, depending on factors and events within particular local contexts. In this framework, “opportunity structure” refers to the institutional context22 in which an individual lives. It encompasses economic factors and institutions (local economy-market-resource base) as well as the rules, norms, and ideologies (social, political and cultural institutions) that govern access to economic resources and institutions. “Agency” refers to the ability to make and act on decisions and control resources and profits that 18 Alsop, Bertelsen and Holland, 2006: 1. 19 Golla et al., 2011. 20 Kabeer, 1999. 21 Kabeer, 1999; Sen and Batliwala, 2000. 22 “Institutions” in this sense refer to any structure or mechanism of social order governing the behavior of individuals within a given community, including organizations such as the market or financial institutions as well as laws, customs and practices. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 9 allow one to advance economically. Our definition of empowerment is conceptualized not simply in the conventional sense as “the ability to exert power over institutions, resources and people” 23 but as:  Power to: have decision-making authority, make choices, solve problems and be creative and enabling;  Power with: organizing with a common purpose or understanding to achieve collective goals; and  Power within: self-confidence, self-awareness and assertiveness; ability to recognize how power operates in one’s life and gain confidence to act to influence and change this.24 The ability to advance economically requires skills, resources and access to economic and social institutions.25 Critical social and material resources include knowledge and information, education, credit, food, health care, employment and ownership of assets such as land, tools, etc.26 Psychological resources include self￾awareness, self-confidence and self-efficacy. If popular empowerment is to be achieved social, cultural and political institutions need to support people’s capacity to envisage options, and their agency to act on them. Based on the above theoretical framework, Figure 2 graphically depicts the assessment team’s assumed relationships among economic opportunity, economic and social empowerment and social change. It is from this theory of change that the assessment framework and the questions in the survey and FGDs are derived. 23 Parpart, Rai and Staudt 2004, p. 5. 24 Green 2008; Parpart, Rai and Staudt 2004; Kabeer, 1999. 25 Golla et. al., 2011. 26 Saigol, 2011; Kabeer, 2005. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT 10 FIGURE 2: THEORY OF CHANGE EMPOWERMENT: (Power and agency; “effective choice”, i.e., the ability to envisage and enact choice) SOCIAL EMPOWERMENT Scales: Household & Individual Expansion in ability to define, make and act on choices; to engage in collective action to assert interests or challenge social hierarchies; and to influence governing policies.  Individual agency within array of institutions  Enhanced power and agency in non￾economic household decision-making such as health care, education, life partner, etc.  Freedom from gender- and child￾based violence ECONOMIC OPPORTUNITY Scales: Community & Household Within local and regional economy, refers to availability of economic resources supporting household livelihoods: employment and income￾generating opportunities; market development; availability and use of modern technologies; availability of environmental resources (water/energy). As a result of USAID-funded projects, these resources expand, thereby increasing employment, income and expenditures. ECONOMIC EMPOWERMENT Scales: Household & Individual  Ability to access economic opportunities  Ability to advance and succeed economically  Economic Security – sustained income earning potential; food security  Greater ability to access, control and share in resource use and economic decision￾making SOCIAL CHANGE Scale: Community Positive transformation in norms, ideologies and institutions that support the power and agency of individuals and households in economic and social matters.  Freedom from social violence  Increased mobility  Enhanced rights of access to resources  Democratic and inclusive institutions  Change in community attitudes about gender to support women’s participation in economy and community decision-making ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 11 Assessment Framework To test EGA’s theory of change, the concept of empowerment was operationalized using a model representing its four dimensions:  Economic opportunity;  Economic empowerment;  Social empowerment; and  Social change. Within each of these four dimensions the assessment team identified components and indicators at the micro (individual/household) and mezzo (community) levels.27 A brief description of each dimension follows. (A fuller definition of the indicators used for each dimension is included in Annex IV). 28 Economic opportunity is a structural dimension that captures the availability of resources, information, infrastructure, employment opportunities, etc. It is defined by the local economy and its resource base, employment structure and opportunities for generating incomes, market development and availability of key environmental and physical resources such as roads, electricity, telecommunications, water and energy. The existence of development projects and other interventions to support livelihoods also factors into economic opportunity. Toward this end, support services tailored to the needs of women (i.e. extension services focused on gender-specific tasks such as seed preparation) are critical to support women’s empowerment.29 In this framework, expanded economic opportunity is indicated by increases in:  Employment (jobs);  Incomes and assets;  Expenditures:  Skill development;  Use of modern technology and management practices;  Improved physical infrastructure. Economic empowerment derives from enhanced access to economic opportunity and refers to improved economic status and the ability to advance and succeed economically. It entails a positive transformation in which households and individuals enjoy enhanced economic security (sustained income-earning and asset￾accumulation potential) and greater ability to access, control and share resource and economic decision￾making. Components of economic empowerment include:  Increased access to markets and financial services (e.g., credit, savings);  Increased access to information about economic matters (e.g., markets, incentive programs);  Increased ability to accumulate and control assets (productive resources such as land, tools, savings, water, skills and education); 27 There is a third level — the macro level, or society at large, which is not incorporated in this framework since social change at the national scale is too far removed from project interventions to be meaningfully linked. 28 See Rozan, 2010. 29 Saigol, 2011. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 12  Ability to manage debt;  Sense of economic security; and  Ability to participate in economic decision-making. For women, economic opportunity and economic empowerment are influenced by the extent to which gender norms and ideologies within the family and community support their right to obtain education and skills training, work outside the home, control their income and participate in household economic decisions. The effect of gender norms and ideologies is captured in the third and fourth dimensions of social empowerment and social change. For children and youth, improved household economic status should ideally result in a reduction in child labor and increased school attendance, particularly for girls. Economic empowerment would mean that children and youth (young adults) enjoy the ability to access education and skills training, freedom of choice over employment and control over their own incomes. Social empowerment entails a positive transformation of social hierarchies in which previously less￾empowered individuals experience an expansion in their ability to define, make and act on choices, participate in household and community decision-making, occupy and move freely through public space, engage in collective action and influence governing policies. In this assessment framework, social empowerment is broadly conceptualized to encompass social, political and cultural factors, such as norms/values/ideologies/practices, power structures and support networks. It includes the following components:  Self-confidence/self-efficacy;  Autonomy and mobility;  Gender norms and ideologies that support women’s access to resources, autonomy and mobility;  Freedom from social violence (gender-, child-based, minorities, socially exclude groups);  Access to health care and autonomy in health care decisions;  Access to information and media;  Ability to make decisions regarding children’s health care, education and marriage;  Ability to vote independently; and  Ability to participate in community affairs. For women, social empowerment also implies that they have control over their own bodies and they can make decisions about their own health care and reproductive activity. Social change refers to a positive transformation in values, norms and ideologies both within the household and at the community level that support the on-going power and agency of less-empowered groups of individuals. The household in this sense is both an agent within a community and an opportunity structure within which individuals are situated. Within the household, positive social change entails a change in family ideologies and practices that previously constrained individual family members, such as women and children. An example is families becoming more willing to allow women to seek employment, control their own incomes and participate in household decision-making. At the community level, social change means greater opportunity, autonomy and agency for poor households and particular groups of previously disempowered individuals, such as women and youth. Social and political space becomes more open to the participation of ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 13 poor households in civil society and community affairs, and tolerance for religious and sectarian differences and for differences in political opinions expands. With respect to women, this implies a change in gender norms toward increased acceptance of women’s participation in social and economic activities, including:  Freedom from domestic violence;  Recognition of inheritance rights;  Involvement in community leadership;  Community support for integration of minorities into community action; and  Community support for women’s autonomy and participation in decision-making. For minorities and other socially excluded groups this includes:  Freedom from social violence;  Involvement in community leadership;  Community support for integration of minorities into community action. Positive social change for children and youth would result in a decrease in child labor, increased school attendance, freedom from violence, increased opportunities for youth and a greater role in key decisions affecting their lives, such as marriage. For men and boys, positive social change would entail change in the way masculinity and masculine identity are socially constructed, disassociating it from asserting power and control over members of the family and society and thereby reducing domestic violence, child sexual abuse, pressure to defend family “honor”, and exposure to and engagement with violence on the streets. Empowerment in the Pakistani Context The dynamics of empowerment in Pakistan are driven by complex and interrelated factors, such as income opportunities, human and material resources and physical security. Conversely, it is the lack of these opportunities, assets and security that drives disempowerment. The lack of income opportunities results from unemployment, underemployment, low wages, preponderance of informal sector employment, poor market development, low agricultural productivity and poor infrastructure.30 Human resources remain underdeveloped because of lack of access to educational institutions, low school enrollment and attendance (especially for girls) and poor access to information, education and skills training. Lack of access to credit and other business support services, on top of a low asset base, inhibits the ability of small businesses to grow and contribute to poverty alleviation.31 In addition, the poor security situation, especially in rural areas, limits people’s ability to take advantage of economic opportunities. For impoverished households to overcome these obstacles and gain greater control over their material conditions, they must strengthen their economic security. Ideally, as households improve their economic status, their level of economic empowerment also improves. They are then better positioned to increase their social empowerment, lobby for stronger social and political rights and find a greater place in civil society. 30 USAID, “Economic Growth Strategy” for Pakistan, Draft Working Paper, Feb. 19, 2011. 31 SME Development in Pakistan: Issues and Remedies; SME Policy of Pakistan (2005) http://www.gcu.edu.pk/publications/vc￾sme.pdf. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 14 In addition to constraints to empowerment for the poor in general, inherent inequalities in autonomy and power exist between men and women. Pakistani culture is strongly patriarchal, with women often confined to inferior roles. Observance of purdah32 can exert a strong check on women’s mobility, access to employment and even access to friends and family. In addition, domestic violence also impacts a significant proportion of the female population in Pakistan.33 Even when such women’s rights as property ownership, inheritance and participation in civil society are encoded in law, household and community-level gender norms often negate women’s ability to assert their rights.34 Other social factors, such as child marriage, family composition (e.g., the presence of in-laws versus a nuclear family unit) and age, also influence the extent to which women enjoy autonomy and participate in household decision-making. The end result of these and other constraints on women is that Pakistan has one of the lowest levels of women’s participation in the formal labor force in the world.35 Gender systems at all levels of society impact women’s economic and social empowerment. However, because empowerment is a multifaceted phenomenon, and because Pakistan has a diverse society, the particular ways women are empowered or disempowered varies in different economic, social and cultural contexts. Furthermore, because empowerment is a dynamic process embedded within complex social, cultural, political and economic frameworks, it can reverse as well as advance, sometimes moving in different directions simultaneously. For example, a growing body of literature indicates that economic development, particularly among lower-middle-income households, has a potentially negative impact on women’s autonomy, mobility and social status.36 According to these studies, when lower-income households achieve middle-class status, they are able to withdraw women from remunerative work. They often do so because the households are considered to have greater social status when women are at home than when they work in low-status jobs. The implications of such developments for women’s empowerment have yet to be systematically investigated, so it is imperative to remain cognizant of possible reversals of women’s empowerment as household incomes increase. ASSESSMENT DESIGN AND METHODOLOGY This assessment was designed to analyze the effect of USAID/EGA program interventions on beneficiaries' economic opportunities (i.e., assets, income and expenditures) and the influence changes in these opportunities have on beneficiaries' economic and social empowerment and community social change. The research design and methods were intended to be sufficiently rigorous to allow a broad and in-depth analysis of the impact of EGA’s programs at the individual and household levels through a mixed methods approach, that is, one using both qualitative and quantitative methods. These methods were theoretically grounded in current development literature, methodologically grounded in best practices and empirically grounded in the Pakistani context. They were selected to complement and build upon each other and designed to be replicable. The impact assessment methodology was designed with two integrated components: a survey and focus group discussions. The purpose of the survey is to assess change in economic opportunities, economic and social empowerment at the household and individual levels, and social change at the community level as perceived by the respondents. (See Annex V for a list of indicators representing each of these dimensions as 32 “Purdah” refers to the seclusion of women to prevent men from seeing them, which is practiced by some Muslim and Hindu communities in Afghanistan, Pakistan, India and Bangladesh. Purdah takes a number of forms, of which the two most common are: physical segregation of the sexes and the requirement that women cover their bodies to conceal their skin and form. 33 Saigol, 2011; Sathar and Kazi, 2000. 34 Saigol, 2011; Sathar and Kazi, 2000. 35 Saigol 2011, p. 16. 36 See Derne, 2008; Hapke, 2006, 2012; Mencher 1988; see also Sathar and Kazi, 2000. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 15 reflected in the survey.) The findings from the survey are subjected to rigorous quantitative analysis using a mix of statistical techniques. The purpose of the Focus group discussions is to provide contextual enrichment to the survey results, corroborate the survey findings, and offer perspective on how and why the trends identified by the survey data are occurring. The findings from the focus groups are analyzed for their qualitative content. The results of these two analytical processes are then triangulated to offer a more holistic perspective on the relationships and dynamics of the findings. These processes are shown in Figure 3. FIGURE 3: IMPACT ASSESSMENT DESIGN COMPONENTS A pseudo-panel design was selected for the survey to ensure the comparability of data and consistency in analyses during subsequent rounds of the assessment. A pseudo-panel design uses the same survey locations (i.e., villages), but not necessarily the same respondents for each round of the data collection. The major strength of this approach is that analysis does not suffer from sample attrition, which is a major issue in panel design.37 By using the same locations, the pseudo-panel design also tends to result in lower sampling errors than a repeated cross-section design using different locations. The assessment team designed this research framework and survey with a clear understanding of the challenges of measuring and attributing impact. The first set of challenges stems from the non-randomized nature of this study. More specifically, program areas and participants were not selected randomly and there is no pre-intervention baseline survey. Thus, the task of identifying a credible counterfactual is not straightforward, unlike in a randomized intervention. To tackle this issue, sample selection included rigorous procedures to ensure that non-program areas and non-beneficiaries are comparable to program areas and direct beneficiaries, respectively. The second set of challenges arises from the heterogeneous nature of the projects within the EGA program. There are four different projects with hundreds of activities, some of which have already ended and some that have yet to begin implementation, intervening in different regions and sectors, using a variety of approaches and focused on unstable beneficiary populations. The assessment is not intended to test the impact of individual projects under EGA’s program, but rather to examine the average effect of the four projects studied. For the sake of simplicity, manageability, and comparability across projects, beneficiaries and locations, a number of analytical conventions were adopted. First, no distinctions were made among the four projects selected for the first round of the study (Agribusiness, Dairy, Firms, and Entrepreneurs). Second, training participation was used as a proxy measure for participation in EGA interventions, even though EGA projects offer an array of different forms of programmatic support. Training was chosen because it was common to all beneficiaries across all regions, regardless of other types of other support they received. Finally, a distinction was made between the non-participants from the villages where programs were present (indirect beneficiaries) 37 Since the same households or respondents are not necessarily interviewed over successive rounds, there will be no impact on future surveys if respondents move out of the area/enumeration unit between survey rounds. 1A STEP 1 1B 2A 2B STEP 2 STEP 3 Individual Survey Quantitative Analysis Triangulation Focus Group Discussions Qualitative Analysis ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 16 and those from villages where program were not present (non-beneficiaries). This was done to capture spillover effects of the EGA program benefits among non-participants in program villages, if any, in addition to the direct program benefits received by training participants.38 Hence, the respondents constitute three groups based on program participation status: direct beneficiaries, indirect beneficiaries, and non￾beneficiaries.39 To develop a data set that is rich enough to fully capture EGA program effects, the survey should be implemented three times over five years on the following schedule:40  Round 1, baseline: survey conducted in late 2014, 41  Round 2, midterm: survey to be implemented at the end of 2016 or early 2017, and  Round 3, endline: survey to be carried out in mid-2019. With the implementation of Rounds 2 and 3 of the assessment, the richness of the data will increase, allowing for more robust analysis. By replicating this research methodology in two additional rounds of data collection,42 the EGA program will have new and comparable data sets over three time periods, allowing them to much more accurately assess trends in the performance of its portfolio. By the end of the third round of surveys, the Mission will have sufficient data to compare the impact of beneficiary status across groups over time, assessing and comparing sustainability, progress over time and the influence of exogenous variables (e.g. disasters, conflict, etc.) on program outcomes. This analysis will allow EGA to draw portfolio￾wide lessons learned, as well as offering an important contribution to the USAID/Pakistan Mission's learning agenda. Sample Size Determination In order to address the EGA program’s overarching learning agenda questions, the sampling strategy for this assessment was designed to capture a representative data set capable of measuring economic opportunities, and economic and social empowerment at the individual and household levels and respondents’ perception of social change at the community level. An important goal in choosing a sampling strategy was to identify approaches that would produce data of sufficient quantity and quality to allow for a rigorous statistical analysis of the linkages among development interventions, outcomes and social change. To determine the optimum sample size for the survey, a reasonable trade-off was required between a technically desirable level of precision and accuracy and a logistically and financially practical number of enumeration units (villages) and respondents. A sample size of 600 per respondent type (i.e., direct beneficiary [DB], indirect beneficiary [IB], non-beneficiary [NB]) per province was chosen as a cost-effective, 38 Spillover effects from EGA training are certainly possible. For example, non-participants in program areas can benefit when their friends participate in training on improved farming techniques and share what they have learned. 39 Since participation in EGA program activities is not random, there may be differences (observed and unobserved) between direct and indirect beneficiaries, which are likely to impart selection bias in the estimated program impacts. While we did not control for selection bias using techniques such as instrumental variable regressions or regression discontinuity design (which are difficult to implement in the current context), we controlled for observable differences between direct and indirect beneficiaries, assuming that they determine self-selection into EGA program activities. Also, since we did not compare direct and indirect beneficiaries directly, but compare these two groups with the non-beneficiaries, selection bias may not be a serious issue. 40 Findings based on one-time survey (cross-sectional survey) cannot capture the dynamics of program effects, which can only be observed through repeated surveys spanned over a longer period. Moreover, without repeated surveys it is difficult to say whether the impacts estimated during the cross-sectional survey are just short-term or sustainable over time. 41 Since the EGA interventions have already started, this is not a true baseline, but can be considered as a proxy baseline. 42 This assessment is intended to be implemented three times: Round 1was conducted in mid-2014, Round 2 is to be implemented at the end of 2016 or early 2017, and Round 3 is to be carried out in mid-2019. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 17 yet statistically valid sample size for the survey. Since the actual share of direct beneficiaries in the total population is very low (about two-tenths of one percent of the total population in the three provinces), a randomly selected proportional sample of the entire population would have yielded a beneficiary size too small for rigorous, in-depth investigation of beneficiaries and EGA program effects. Therefore, a stratified random sample was used with disproportionate sampling (over-sampling) of beneficiaries. Using this technique, beneficiaries were systematically selected in larger numbers than their proportion in the general population to ensure a sufficiently large sample size for statistical analysis. More specifically, beneficiaries comprised one-third of the whole sample. For analytical purposes, any bias created by disproportionate sampling was corrected using population weight to make the findings representative of the underlying province population. Specific districts, villages and respondents were systematically sampled to ensure that each had an equal probability of being selected within a specific category, thereby ensuring randomness in sample selection. Large enumeration units (with more than 150 households) were segmented into different areas based on mohallas (neighborhoods), with one mohalla chosen at random from a complete list of all neighborhoods in the enumeration unit. Based on the above sampling strategy, 200 enumeration units (villages) were selected in each province (100 from project areas and 100 from non-project areas) for the survey. In each project area three men and three women were selected for each of the direct and indirect beneficiary respondent categories. Likewise, in each non-project area three men and three women were selected for the non-beneficiary category. This amounted to 600 interviews per category per province or 600 direct beneficiaries, 600 indirect beneficiaries, and 600 non-beneficiaries from Punjab, Sindh and KP. With 1,800 respondents per province, the total sample size was therefore 5,400 individuals43 (see Table 2 below). This represents just over 10 percent of the 53,000 beneficiaries in the three provinces. TABLE 2: INITIAL HOUSEHOLD SURVEY SAMPLE SIZE AND COMPOSITION Punjab Sindh KP Total Total EGA Beneficiaries44 17,516 9,593 25,867 52,976 Direct beneficiaries interviewed (Project areas) 600 600 600 1,800 Indirect beneficiaries interviewed (Project areas) 600 600 600 1,800 Non-beneficiaries interviewed (Non-project areas) 600 600 600 1,800 Total Interviews 1,800 1,800 1,800 5,400 Enumeration units (Project areas) 100 100 100 300 Enumeration units (Non-project areas) 100 100 100 300 Total Enumeration Units 200 200 200 600 43 However, in reaction to the on-the-ground realities of survey implementation, both the sample size and household distribution of the data collected varied slightly from the original design. These data are presented in Table 3. 44 Figure based on beneficiary data available from implementing partners. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 18 Survey Implementation Design The same survey questionnaire was administered through face-to-face interviews to all selected direct, indirect and non-beneficiary respondents. The survey, comprised of 89 questions, collected data on a range of indicators covering economic opportunity, economic and social empowerment and social change. The main themes are described in the table below (see Annex V for a list of the survey questions and the EGA Impact Assessment Design Document for the complete survey questionnaire). Survey Research Themes Economic Opportunity Economic Empowerment Social Empowerment  Income, Assets, and Access to Services  Food Consumption  Project Participation  Household Access to Economic Resources  Debt  Long Term Economic Security  Individual Access to/Control over Resources  Self-Confidence/Self-Efficacy  Autonomy and Mobility  Opportunity Structures/ Institutional Access  Health Care In addition to the individual survey, a community questionnaire was administered to identify factors at the enumeration unit level that could influence survey outcomes. (See the EGA Impact Assessment Design Document for the Community Survey.) Community characteristics that could influence the outcomes were considered to include physical infrastructure (i.e., roads, markets, and electricity), proximity to markets, urban or district centers, major livelihood activities, external shocks (such as natural disasters) and non-EGA development activities implemented by non-governmental organizations (NGOs) or other donors. Survey supervisors administered this questionnaire to village leaders or tehsil administration officers. Project Area Sampling In project areas, the survey focused on both direct beneficiaries (people who are participating or have recently participated in USAID-funded projects) and indirect beneficiaries (people who live in the same locality or enumeration unit, but have not participated in USAID-funded projects since 2010). The total number of direct beneficiaries in an enumeration unit was determined from beneficiary lists obtained from USAID/EGA local implementing partners. From this list, 6 beneficiaries (3 males and 3 females) were selected randomly.45 To identify the six indirect beneficiaries to be interviewed, a systematic sampling approach was used based on the total number of indirect beneficiaries in a village and using a random sampling interval (see the detailed methodology in the Sampling Strategy Implementation section of EGA Impact Assessment Design). Non-project Area Sampling If individuals participated in EGA programs only from villages where the program operates (project areas) then identifying non-program villages would have been fairly straightforward – all villages other than project areas would have been potential candidates for non-program villages. However, in reality, individuals from neighboring villages also participate in the program, making all such villages the program villages. We can call this whole area of program villages ‘the zone of influence’. Clearly, any non-program villages must be outside the zone of influence. 45 This can be done using any statistical package, such as STATA or SPSS. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 19 Ideally, to find out the non-program villages, a census type survey would be conducted in all villages, which unambiguously determines which village have program participants and which do not. Then non-program villages, which are comparable to program villages, would be selected using techniques such as propensity￾score matching. However, such a survey is both time-consuming and resource-intensive. An alternate way, which was used for this assessment, is to make a reasonable assumption about the expanse of the zone of influence and select villages from outside that area. The advantages of this approach are that it is both fast and much less costly. Moreover, the PakInfo GIS database, maintained by USAID, allowed us to locate program and non-program villages without the need to physically visit them. We selected non-project villages from areas located about 7.5 km away from the project sites within the same district. This is based on the assumption that individuals from villages beyond 4-5 km of the project sites are not likely to participate. This is fairly common in rural areas of the developing countries where roads and transport are not well-developed and people are unlikely to commute more than 5 km on a daily basis to attend trainings. By selecting non-project villages from even farther (that is, 7.5 km), we are reasonably sure that no one from those villages has participated in EGA interventions. At the same time, we did not want to go farther than 7.5 km or outside the districts where project villages are located to ensure that, while these villages are far enough away to avoid spillover effects from EGA program interventions but close enough to be comparable to project villages.46 This process is illustrated in Figure 4. Non-project villages were selected in this way specifically so that they would reasonably approximate the observable socio-economic and demographic characteristics of project villages (again, see the detailed methodology in the Sampling Strategy Implementation section of EGA Impact Assessment Design). Within each non-program village thus selected, individual respondents are identified for interviews through a random procedure similar to the one used to select indirect beneficiaries. As mentioned before, the community questionnaire was fielded in all villages, including the non-program ones, to collect information that would facilitate controlling for observable differences between program and non-program villages in the data analysis. 46 To check the comparability of the selected non-program villages with program villages, we compared a wide range of community characteristics between the two types of villages (for example, village size, population density, proximity to urban center, village infrastructures such availability of electricity, paved roads, banks, markets, schools, health care facilities, microcredit and other development organizations, and various exogenous shocks experienced by villagers during the last 12 months such as floods, droughts, earthquake, crop or livestock diseases, etc.). We found that for a good majority of these variables (about 75 percent), program and non-program villages are similar or comparable (based on t-statistics of the differences in those characteristics). ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 20 FIGURE 4: SELECTION OF NON-PROGRAM VILLAGES As implemented, the sample composition varied from the original design (Table 2) because of changes made in response to conditions on the ground that could not have been anticipated in the design phase. Also, in order to compensate for observation losses due to possible non-response and non-cooperation of the respondents, enumeration errors, and data entry and data cleaning errors, a 5-10 percent oversampling was implemented during data collection. As a result, after data cleaning and processing the total survey sample included 5,419 interviews instead of the planned 5,400 (see Table 3 for actual sample composition).47 Differences were also found in the number of interviews per region, project and beneficiary type. For example, KP represented slightly more than a third (35.5 percent) of the total interviews, while Punjab and Sindh each comprised slightly less than a third. While the survey intended to maintain equity between the number of male and female respondents, differences in the gender balance between EGA projects, the realities of survey implementation in the field, as well as oversampling and the discarding of unusable observations during data cleaning, led to slightly different results. Male respondents (2,786) slightly outnumber females (2,633), resulting in a 6 percent overrepresentation of males. Because of the large size of the sample, these minor differences did not compromise data analysis. In addition, population weighting was used to ensure that findings were representative of the underlying population. 47 Even when required to deviate from the original survey design, the assessment team tried as best as possible to meet the original design criteria (e.g. geographically representative, equal numbers of males and females, etc.). This is discussed in detail in the Methodological Limitations section. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 21 TABLE 3: ACTUAL SURVEY SAMPLE SIZE AND COMPOSITION Punjab Sindh KP Total Male Total Female Total Direct beneficiaries (project areas) 588 547 665 943 857 1,800 Indirect beneficiaries (project areas) 576 594 655 922 903 1,825 Non-beneficiaries (non-project areas) 601 587 606 921 873 1,794 Total Interviews 1,765 1,728 1,926 2,786 2,633 5,419 Source: EGA-PIA first round survey, 2014. Focus Group Discussions and Community Surveys FGDs were implemented in all three provinces in order to provide qualitative data on economic and social empowerment at the individual and household levels and to triangulate and enrich the survey results. The FGDs were further intended to validate the survey results and provide explanations for survey trends. Focus groups also offer a valuable opportunity to gain a deeper understanding of Pakistani perspectives and interpretations of economic and social change at the household and community levels. Finally, FGDs helped researchers to better understand the opportunity structures that underlie Pakistan’s socioeconomic fabric, especially opportunities created by USAID-funded projects. Each focus group discussed 18 questions covering a wide range of topics related to economic opportunity, economic and social empowerment and community change. While these themes were not replicated exactly from the survey, they were comparable. In addition, initial survey analysis was used to revise the FGD guide to reflect issues and questions that emerged from the survey data. The FGD main themes are detailed in the table below (The Impact Assessment Design for EGA guide contains the full FGD questionnaire). Focus Group Discussion Themes FGD participants were selected from a list of survey respondents who provided phone numbers along with demographic information. Potential FGD participants were organized into a list according to the sampling frame (by region, sex and beneficiary status). Enumerators then, starting from the top of the list, called to ask Economic Opportunity Economic Empowerment Social Empowerment  Occupations/activities  Changes in economic activities  Changes in employment for women  Changes in employment for youth  Overall community prosperity  Educational and skills training facilities and opportunities  Household access to markets  Change in ability to do business  Change in ability to cope with shocks  Change in access to financial services  Women’s mobility  Household division of labor  Participation in household decision-making  Voting in elections  Violence against women  Violence against children  Types of community-based organizations  Participation in community￾based organizations ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 22 survey respondents if they would be willing to participate in a focus group. Twelve participants were selected for each focus group, with the understanding that some would drop out, and the goal of having at least eight participants in each focus group. Due to factors beyond the research team’s control, minor adjustments were required to the proposed sample size and composition while implementing the FGDs. While these changes did not impact the total number of focus groups, which remained at 24, they did influence total number of participants (191). The quality of the data from these discussions was not affected by these changes (see table below). TABLE 4: ACTUAL FGD SAMPLE SIZE Data Analysis Analytical Approaches Each of the assessment’s three research questions was addressed using a mix of analytical methods and tools to describe the data and establish patterns and relationships. This section describes how these different tools were used to capture findings and draw conclusions from both survey and FGD data. Question 1: What is the current status of EGA project (actual and potential) beneficiaries in terms of employment, income and expenditures? While some welfare indicators, such as employment status, can be determined through straightforward questions in survey interviews, income and expenditure cannot because of their sensitive nature. Given the difficulties in collecting accurate data on household income and expenditure, they were captured by proxy variables such as household’s productive assets, housing type, access to finance and market, consumption frequency of food items such as meat, fruits and vegetables, etc. To estimate financial well-being, descriptive statistics for these proxy variables were analyzed for each of the three beneficiary types and the statistical significance of their differences assessed. In addition to survey data, FGD data provided insight to respondents’ perceptions of their current economic status and the various factors that influence it. FGD results were also scored and comparatively analyzed to assess differences between beneficiaries and non￾beneficiaries. Question 2: What is the current level of economic and social empowerment of the beneficiaries of EGA projects (i.e. control over their livelihoods and participation in decision-making at the household and community level)? Location Number of Beneficiary FGDs Number of Non￾Beneficiary FGDs Total Number of FGDs Participants Total Participants Beneficiaries Non￾Beneficiaries Men Women Men Women Men Women Men Women KP 2 2 2 2 8 18 18 17 16 69 Punjab 2 2 2 2 8 13 14 27 7 61 Sindh 2 2 2 2 8 16 14 18 13 61 Totals 6 6 6 6 24 47 46 62 36 191 ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 23 Because empowerment is an abstract concept, we cannot observe or measure it directly – we only perceive it through a range of indicators, which are included in the survey instrument (see Annex IV). Indicators such as economic security, control of resources, self-confidence, and spatial mobility were selected to represent economic and social empowerment. Factor analysis was then used to construct indices from these indicators for the various dimensions of the EGA theory of change. Once these indices were constructed, both descriptive and regression analyses were carried out to assess and compare variations in the levels of economic and social empowerment indices among the three respondent groups. Again, FGD data provided insight into people’s perceptions of social and economic empowerment. Question 3: To what extent and in what ways have USAID-supported activities changed/influenced EGA program beneficiaries’ lives in terms of employment, income, expenditure, economic empowerment and social empowerment? This question addresses this assessment’s fundamental issue. To examine it, both descriptive and regression analyses were used. Descriptive analysis established the correlation among respondents’ beneficiary status, outcome indicators (collected through the survey) and indices (developed through factor analysis). Regression analysis then used these indices to measure the contribution of EGA projects to higher-level outcomes (e.g. economic opportunity, social and economic empowerment and social change), after controlling for all exogenous factors (at individual-, household- and community-level). Statistical Analysis Techniques A variety of analytical techniques were used to interpret the survey data. The most basic form of data analysis used was descriptive statistics of individual and household indicators. Besides the observed indicators, indices capturing the four dimensions of the theory of change were also analyzed using descriptive statistics. Descriptive analysis compared mean values of the outcomes (both indicators and indices) by beneficiary types. Such comparisons indicated whether the differences in the outcomes across three types of respondents (direct beneficiaries vs. non-beneficiaries and indirect beneficiaries vs. non-beneficiaries) were statistically significant.48 The statistical technique most commonly used to establish relationships between dimensions of change is factor analysis. Using factor analysis allowed us to create indices of change from observable indicators that were otherwise difficult-to-quantify. Four indices were created representing the four dimensions of the EGA theory of change: economic opportunity, economic empowerment, social empowerment and social change.49 Table 5 shows the four indices and corresponding indicator variables. 48 If they are not statistically significant, any differences in outcomes between direct beneficiaries and non-beneficiaries, or indirect beneficiaries and non-beneficiaries can be viewed as random, and no correlation between the beneficiary type and the outcomes can be implied. 49 A brief description of factor analysis and how it is used to construct the indices for four dimensions is given later in the Findings section). ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 24 TABLE 5: OPPORTUNITY AND EMPOWERMENT INDICES INDICES Economic Opportunity Economic Empowerment Social Empowerment Social Change Indicators Source(s) of income Project participation Self-confidence/self￾efficacy of respondent Women’s status and rights in society Productive assets Household access to resources/markets Vision of a better future Violence against children Non-productive assets Sources of finance Respondent’s mobility Community and shared goals Land owned Safety nets Respondent’s freedom to move freely Status of marginalized groups House features/facilities Household debt status Respondent’s use of media and phone Safety and security challenges to community Food consumption patterns Economic security Opportunity structures/institutional access Individual access to markets Individual access to/control over resources Health care As descriptive analysis can only establish correlations, more sophisticated analyses, such as regression analysis, were conducted to go beyond correlation. For example, while descriptive analysis can measure differences in outcomes between direct beneficiaries and non-beneficiaries, it cannot say what other factors, besides EGA interventions, might have contributed to such differences. Regression analysis, on the other hand, takes into account a wide range of observable differences between respondent types, not just their EGA participation status, in assessing differences in outcomes. Regression analysis was, therefore, then performed to determine the effects of participation in EGA programs on these indices and the inter-relationships among them. As mentioned before, such differences can come from individual-, household-, and community-level factors, information on which was collected through the individual and community surveys. We have controlled for a wide range of these characteristics in the regression. The list of control variables used in the regression analysis is shown in the Table 6. In addition to these variables, dummy variables for provinces were also used in the regressions to control for unobserved province-level effects. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 25 TABLE 6: CONTROL VARIABLES USED IN REGRESSION Methodological Limitations One limitation of this assessment is that the survey is not a true baseline for the EGA program because discrete projects/activities within the program have different starting dates and many had already started befor the implementation of the survey. If baseline data had been collected before project interventions began, it would have offered better control for confounding factors that might affect program outcomes. In the absence of baseline data, we used the variables listed in Table 6 as control variables. 50 Since we assume these variables were not affected by program interventions or participant status, their effects on the outcomes are the same as they would have been if they were collected in a true baseline survey. While some of the methodological limitations of this research were inherent to the nature of the study itself, some were due to data limitations and some associated with challenges in survey implementation. On several occasions during the course of survey implementation, the assessment team was confronted with issues that forced it to alter the survey sample. Such issues included security, logistics and efficiency. For example, some districts identified during the design phase were simply too risky for the enumerators to visit for safety’s sake, 50 For example, we did not use income- or employment-related variables as controls because these variables were most likely affected by program participation. At Individual Level At Household Level At Village Level Sex Age Education Sex of head Age of head Education of head Occupation of head Number of adult males in household Number of adult females in household Whether rural or urban area Area of the village Population density in the village Land price in the village Distance of village from district center Whether village has paved roads Whether village has electricity Whether village has good mobile phone signal Whether village has primary schools Whether village has markets Whether village has banks Whether village has a healthcare facility Whether village has a bus stop Number of micro-credit organizations operating in the village Number of safety net programs operating in the village Number of NGOs operating in the village Number of non-USAID donors operating in the village If the village had droughts during last 12 months If the village had floods during last 12 months If the village had earthquakes during last 12 months If the village had crop diseases during last 12 months If the village had livestock diseases during last 12 months If the village had sudden increase in food prices during last 12 months ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 26 some were too remote to make them logistically feasible, and some had too few beneficiaries to produce statistically viable data. However, the number of these cases was too few to compromise the randomness of the overall sample. The analysis in this study may also have suffered from biases that are typically difficult to control for in cross￾sectional studies, such as program placement bias and self-selection bias.51 To counteract any such biases, as noted above, the analysis controlled for a wide range of individual, household and community characteristics, assuming they are determinants of program placement, self-selection or other unobserved biases that may affect EGA program benefits. For example, characteristics such as respondents’ age and education levels can influence them to participate in the EGA program activities, and thus can indeed be good predictors of self￾selection into the program. Similarly, community characteristics such as infrastructure development, incidence of natural disasters, proximity to urban centers, etc. can have a strong influence on program placement. Since non-program villages were not selected by matching a process with program villages using techniques such as propensity-score matching, their comparability may be questioned. However, as noted earlier in this section, by selecting non-program villages from within the same districts as program villages, but from locations outside program’s zone of influence (7.5 km away from project sites), we made every endeavor to ensure that they remain comparable with the program villages. Moreover, comparison of a host of community characteristics between program and non-program villages made sure that the two types of villages were indeed comparable for all practical purposes. Another limitation that the survey teams had to overcome was the lack of comprehensive beneficiary data for all EGA projects, including beneficiary contact information. Because this information was not available in time, the beneficiary sample frame could not identified in advance. Consequently, the survey teams were often required to adapt data collection upon arrival at a project village. For example, the team sometimes arrived in a village to find only male beneficiaries. Instead of bypassing that village, the team interviewed six (male) beneficiaries to complete the direct beneficiary quota for the village, hoping that they would find later a female-only (or at least female-dominated) village from which to make up for the gender gap. In the end, such villages (female-only or female-dominated) were not always found, resulting in male-female disparity in the overall sample. This gender disparity also reflected the fact that EGA project interventions themselves were not gender-balanced (some projects serving significantly more women and others serving more men), and that gender distribution varied by province (some provinces simply had more male beneficiaries than women). Gender discrepancy was still quite low compared to the sample size, and in addition, analysis was weighted to adjust for gender variations. Overall, such differences did not affect analysis. An additional limitation to the methodology was that actual income and expenditure variables were not captured, rather proxy variables were used instead. (These included house construction, access to water, productive assets, food consumption patterns, etc.) Such income proxies are often used in field surveys because respondents are typically sensitive about revealing their income (and income sources) to strangers. (Collecting information on expenditures can also be challenging, although not necessarily as sensitive as income information.) The proxies used offered a reasonable approximation of economic status and were demonstrated to be significant through statistical analysis. Finally, because of the small sample size for urban populations (less than 300 observations or 5 percent of the sample), it was not possible to perform separate analyses of rural-urban differences. During the design phase, the assessment team assumed that urban communities would represent a sufficiently large share of observations (at least 15-20 percent) to allow for separate analysis. Since about 35 percent of Pakistan’s 51 Program placement effects are effects of the communities where programs operate. Since residents from a developed area may have improved outcomes to begin with, the influence of program placement needs to be controlled for in order to isolate the effects of EGA interventions. In addition, people who are relatively well-off and/or intelligent may tend to participate (self￾select) more in program activities than those who are not so motivated. Intelligent people are likely to do well even without program participation. So, it is important to disentangle the effects of intelligence or motivation (which are often unobserved) from that of EGA program participation. These issues are usually handled better with panel data than with cross-sectional data. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 27 population is urban, this assumption was not unrealistic. Moreover, EGA intervention areas were found to be overwhelmingly rural, and consequently survey teams found very few urban respondents. Similarly, it was difficult to find urban beneficiaries for focus group discussions. (FGD enumerators were able to interview two urban focus groups (16 percent of the sample), both from Punjab and both non-beneficiaries.) Even though the number of urban respondents included in the study is relatively low, this trend reflects the inherent reality of where EGA programs operate and does not impact the significance of the findings. However, in order to compare and analyze urban-rural populations in the future, urban areas could be oversampled in second and third survey rounds. FINDINGS AND CONCLUSIONS The survey and FGD data offered powerful insights into the nature and interrelationships of the socioeconomic status and levels of empowerment of the studied populations. Empowerment is an inherently complex, multifaceted phenomenon that often moves in multiple directions simultaneously. It is therefore necessary to analyze the social, cultural and economic variables that influence empowerment (and the effectiveness of EGA interventions at promoting empowerment) using a mixture of methodologies. Using a variety of analytical techniques across data sets also helps to triangulate findings offer a broader picture of trends in economic and social empowerment. Because of the different nature of the data collected by the survey and focus groups, findings from each were not always mutually reinforcing. However, where FGD and survey data findings were comparable, they were consistent. Although the survey data captured did provide statistically significant measurement of the current situation for different respondent types in EGA program and non-program areas, they are, in the end only a snapshot of cross-sectional data at one point in time. The explanatory power of these data is expected to increase significantly with Rounds 2 and 3 of the study when comparisons and analysis of change over time are possible. Question 1 What is the current economic status of EGA project (actual and potential) beneficiaries in terms of economic opportunity, income and expenditures?52 Key Findings Overall, EGA project beneficiaries have better access to both productive and non-productive assets, and are more likely to be self-employed (as opposed to wage laborers) and engaged in non-agricultural occupations than non-beneficiaries. Interestingly, despite the fact that more beneficiary women report being employed than non-beneficiaries, the overwhelming majority of FGD respondents, both beneficiary and non￾beneficiary, feel their economic well-being has deteriorated over the last five years (primarily due to inflation and natural disasters). Thus, while FGD participants frequently described the overall economic situation as one of increasing opportunities and incomes, they also complained that rising prices negated any gains in their standard of living. 52 Note: This question does not focus on attribution. The data collected describes the current status of beneficiary and non￾beneficiary populations, without attempting to make any inferences about linkages between their status and EGA programming. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 28 Key Findings on Question 1  A complex situation exists today where economic opportunities are increasing while economic well￾being is declining.  EGA program beneficiaries tend to be better off in terms of assets, housing and self-employment.  Beneficiary women report more and better job opportunities than non-beneficiary women.  Beneficiary FGD participants describe an increase in women doing income-earning work (8 out of 12) while only 3 of 12 non-beneficiary focus groups felt the same. According to the survey, 41 percent (354 of 857) of female direct beneficiaries reported they were employed.  Ability to do business increased (frequently due to improved infrastructure and access to information).  Women have an easier time doing business now than five years ago.  Of the three provinces studied, FGD participants in Punjab cited relatively greater improvement in overall prosperity, women’s employment and ability to do business, and youth employment. Availability of Assets and Resources EGA project beneficiaries consistently reported greater ownership of various productive assets ranging from cash to tractors. Indirect beneficiaries report the second highest rate of asset ownership among the three groups, except for livestock, where non-beneficiaries reported a higher ownership rate than indirect beneficiaries. Livestock is the most widely owned asset, followed by land or other real estate assets (see Figure 5). FIGURE 5: OWNERSHIP OF PRODUCTIVE ASSETS BY BENEFICIARY TYPES (PERCENT) Source: EGA-PIA first round survey, 2014. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 29 Ownership of pucca houses (with walls made of concrete, bricks or stones, and foundations of concrete), as well as improved roofing (concrete, bricks, stones, tiles, slate, metal or asbestos sheets) and access to sanitary latrines are all highest among direct beneficiaries. Differences between beneficiaries and non-beneficiaries are statistically significant for these housing quality indicators. Indirect beneficiaries report the highest level of access to in-house tap water, followed by direct beneficiaries and non-beneficiaries, although this difference is not statistically significant (see Figure 6). FIGURE 6: HOUSEHOLD DWELLINGS AND SANITATION BY BENEFICIARY TYPES (PERCENT) Source: EGA-PIA first round survey, 2014. Major Occupations/Activities All focus groups report that agriculture represents the most important economic activity in their community (either in self-employment or wage employment). This is not surprising since 22 out of 24 focus groups were held in rural areas. Survey findings confirmed that agriculture is the major source of employment for 55 percent of direct beneficiaries, 54 percent of indirect beneficiaries, and 52 percent of non-beneficiaries. Other economic activities reported include (in order of significance) daily wage labor, business/trade/shop keeping, services (e.g., tailor, barber, etc.) and government work. There was no significant variation in these responses across provinces (see Figure 7). Survey data revealed differences between respondents in terms of major economic activities, with significantly more direct beneficiaries reporting being self-employed. Specifically, self-employment was reported as a source of income by 66 percent of direct beneficiaries, 59 percent of indirect beneficiaries, and 51 percent of non-beneficiaries. Another reflection of this trend is the fact that 32 percent of non-beneficiaries reported income from agricultural daily wage labor, while only 22 percent of direct beneficiaries reported the same. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 30 FIGURE 7: SOURCES OF INCOME BY BENEFICIARY TYPES Source: EGA-PIA first round survey, 2014. Changes in Economic Activities The majority of FGD participants, men and women, across all three provinces, reported positive change in economic activities and income earning opportunities (17 of 24). The most frequent reasons cited for this improvement were more non-agricultural employment opportunities (15 of 24) and shift to higher paying jobs (14 of 24). Beneficiary women were more likely to report improvement in non-agriculture opportunities (5 of 12) and more likely to report shifting from low-paying to better-paying jobs (5 of 12) than female non￾beneficiaries. Although this was not a specific question in the FGDs, many women described engaging in needlework or embroidery. In addition, FGD participants cited improved access to markets through better roads (14 of 24) and better transportation (16 of 24) and improved means of communication (13 of 24) as contributing to an improved ability to do business. Economic Changes in the Past Five Years Despite optimism about improved opportunities, an overwhelming majority of all focus group participants noted deterioration in their economic well-being over the past five years (20 of 24). The most frequently cited factors contributing to this decline were high inflation (19 of 24) and lower incomes (15 of 24). Focus group participants explained that inflation had destroyed any benefit from additional earnings. In addition, natural disasters, particularly floods, were cited as a reason for economic decline (10 of 24). Natural disasters were reported more often in Punjab and Sindh than in KP. All female non-beneficiary focus groups (6 out of 6) reported that their economic well-being had deteriorated compared to only half of female beneficiary FGDs (3 out of 6). More men and more non-beneficiaries reported deterioration in well-being than women and beneficiaries. Regionally, lower income was described as a factor responsible for the deterioration in economic well-being more often in Punjab and KP than in Sindh. Changes in Women’s Employment While the majority of focus groups said that more women are now employed, differences between beneficiaries and non-beneficiaries regarding women’s employment are significant. For example, three quarters of beneficiary focus groups (8 out of 12) said that more women are engaged in paid work and that women and girls have better skills; only one quarter of non-beneficiary focus groups (4 out of 12) said the same. Similarly, a quarter of non-beneficiary focus groups (3 out of 12) saw an increase in women working ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 31 outside the home, while twice as many beneficiaries (6 of 12) noted the same trend. However, only two female non-beneficiary focus groups said women have more employment opportunities and only one said that women and girls have better skill levels. Interestingly, there is no significant difference in the responses of beneficiary and non-beneficiary males to the same questions. The majority of all focus groups cited the following factors as enabling women to work: women’s education/skills/awareness (16 of 24) and poverty/need for additional income (12 of 24). Among male focus groups, more non-beneficiaries cited poverty/need for additional income as a factor enabling women to work. All female beneficiary focus groups cited education as a factor enabling women to work. Interestingly, the only male focus groups to state that women want to work were non-beneficiaries. In terms of regional differences, more focus groups in Punjab (7) reported that more women are now employed than in KP (5) and Sindh (3). Punjabi focus groups also reported more employment opportunities for women than other provinces. Focus groups in Sindh were the least likely to agree that women generally want to work. More focus groups in KP (7) cited education levels/awareness as a reason for increased women’s employment, while a significantly higher number of respondents in Punjab (6) cited poverty than in Sindh (2). Changes in Youth Employment Just under half of all focus groups reported improvement in youth employment (11 of 24). Most of these said that youth are mainly employed in daily labor (10 of 24), while slightly fewer stated that youth were shifting to higher-paying jobs in the non-farm sector (9 of 24). It is notable, however, that almost half of the groups reporting improved youth employment were located in Punjab (6 of 11) and more than half describing worsened youth employment were in KP (4 of 7). Conclusions Across a broad range of indicators, EGA program beneficiaries are better off economically than non￾beneficiaries. Female beneficiaries, in particular, have more employment opportunities than female non￾beneficiaries. In addition, many female beneficiaries are getting out of agriculture and into higher-paying occupations, whether as owners of their own small businesses or as employees. The fact that these trends are taking place in the context of predominantly agricultural communities that have seen inflation outstrip increases in wages, makes the advancement of women beneficiaries all the more impressive. A positive change that EGA activities appear to have influenced is a tendency to move towards self￾employment, which often implies independence and is positively linked with economic welfare. For example, when a rural economy is subject to a systemic shock (such as a flood), self-employed households are less likely to suffer, as they are more likely to have savings and other assets to live on. Survey and focus group data indicate that participants’ economic status is influenced by a complex and varied range of factors, some of which are not economic nor tied specifically to EGA activities. Many of the variables influencing economic status are, predictably, tied to economic activities (i.e., the predominance of agriculture activities and day labor among respondents). However, economic status often varies by region and sometimes manifests complex/conflicting tendencies (such as simultaneous improvements in economic activities and women’s employment with deterioration in overall economic well-being). Data also reveal that education and skills are prime influences in increasing the ability to obtain employment, especially among women and youth. However, while women’s employment and employment opportunities have generally increased over the last five years, stable employment opportunities for youth are limited, especially in KP and Sindh. Since agriculture was the major economic activity in EGA project communities, interventions related to agriculture have the potential for widespread impact on economic opportunity. Despite the importance of ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 32 agriculture, it is important to keep in mind that non-agriculture activities also offer important opportunities for women’s and youth employment and for higher incomes. Question 2 What is the current level of economic and social empowerment (e.g., control over their livelihoods, participation in decision-making at the household and community level, etc.) of the beneficiaries of EGA projects? Key Findings Findings regarding economic and social empowerment can be grouped into three major categories: access, participation and decision-making. Not surprisingly, relationships among different empowerment indicators are strong and, as with Question 1, frequently tied to variables outside the strict domains of economic growth and agriculture. Again, increased access to education (particularly for women and girls) and better access to markets due to improved infrastructure emerged prominently in focus groups as exogenous variables influencing empowerment outcomes. Interestingly, FGD respondents cite improved education and awareness as major reasons for the decline in domestic violence against both women and children. Regional differences also influence empowerment outcomes, with Punjab again ahead of the other two provinces. Key Findings on Question 2  Beneficiaries have seen more advancement in women’s social empowerment than non-beneficiaries.  Higher skill levels for women and girls have created more employment opportunities.  Better education and skills, as well as need, have enabled women to work.  Women’s improved economic status has driven advancement in social empowerment, at least indirectly.  Punjab is generally ahead of KP, and both are far in advance of Sindh, in economic and social empowerment.  Violence against women has become less acceptable among all groups in all provinces, but predominantly among beneficiaries. Violence against women was less acceptable among focus group beneficiaries in Punjab and Sindh than in KP.  Male FGD participants are more likely to cite increased awareness and better education as reasons for reduced acceptability of violence against women.  Women’s FGDs are almost equally divided between increased awareness and better education, less tolerance by victims, and better income levels of women as reasons for decreased violence against women.  Focus group respondents cite infrastructure (e.g., roads, transportation and electricity) as well as education and skills training and access to credit as key determinants of enhanced economic empowerment. Better roads and means of transportation/communication have improved access to markets. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 33 Access to Markets The majority of all focus groups indicated that market access had improved over the last five years. Contributing factors cited are better transportation and roads, increased knowledge and awareness and better means of communication. This trend is confirmed by survey data (Table 7), which indicate that beneficiaries are significantly more likely to have access to markets for sale of products (32 percent), followed by indirect beneficiaries (21 percent) and non-beneficiaries (19 percent). Similarly, beneficiaries are more likely to have access to markets for purchase of inputs (30 percent) than indirect beneficiaries (22 percent) or non￾beneficiaries (18 percent). The reasons cited for improved market access vary both by beneficiary status and by region. For example, more beneficiaries than non-beneficiaries (both men and women) state better transportation and means of communication as important factors for improved access to markets. Interestingly, more KP respondents list better roads and transportation as important factors for improved market access, while more respondents in Punjab cite better knowledge/awareness/means of communication. In general, FGD respondents in Punjab and KP say they have better market access more often than those in Sindh. Access to Financial Services All FGD respondents describe a lack of access to formal sources of credit in their communities. Lack of access to financial services represents a significant potential constraint to economic opportunities, especially on the ability to do business. Among the types of financial services cited as available (to varying degrees) were banks, informal rotating credit groups, family, friends, and neighbors. Survey data indicate that beneficiaries have the best access to financial services, followed by indirect beneficiaries and non-beneficiaries. For example, 17.7 percent of direct beneficiaries surveyed describe having access to commercial banks, while only 13.3 percent of indirect beneficiaries and 9.8 percent of non￾beneficiaries have the same access (see Table 7). More focus groups in Punjab (7) and KP (7) say they have access to banks than their counterparts in Sindh (4). Focus groups in KP report access to rotating credit groups most often (6) and in Sindh, FGD respondents were more frequent in relying on credit from relatives, friends and neighbors (6). A significantly higher number of beneficiaries (17) state that the purpose for taking loans is business or emergency (as opposed to consumption) than non-beneficiaries (9). Further, focus groups in Punjab and KP take loans for business purposes more often than those in Sindh. Respondents in KP and Sindh take loans for emergencies more often than those in Punjab. TABLE 7: HOUSEHOLD ACCESS TO MARKETS AND FINANCIAL SERVICES (PERCENT OF RESPONDENTS WITH ACCESS) Variables Direct Beneficiaries Indirect Beneficiaries Non-Beneficiaries Markets for selling products 31.8 21.0 19.2 Markets for purchasing inputs 29.8 21.6 18.2 Commercial banks 17.7 13.3 9.8 Microfinance Institutions 13.6 9.9 4.8 Informal savings and lending groups 9.8 5.9 7.4 Source: EGA-PIA first round survey, 2014. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 34 Ability to do Business Although half of all focus groups state that people’s ability to do business has increased over the past five years, there are significant differences in the responses of beneficiaries and non-beneficiaries. Significantly more beneficiaries stated that their ability to do business has increased (10 of 12) than non-beneficiaries (2 of 12). A third of all FGD participants cite better awareness and education as the driving factors behind the improved ability to do business, but again, seven times more beneficiaries feel this way than non-beneficiaries. Close to half of all focus groups cite social norms and domestic responsibilities as the most significant challenges to women’s ability to do business. Significantly, non-beneficiary focus groups were 4.5 times more likely to describe these constraints than beneficiary groups. There was no significant difference in responses to this question across sex or province. An interesting corollary to this trend is that more than twice as many beneficiary focus groups state that it has become easier for women to do business as non-beneficiaries; the only groups stating that there has been no change in women’s ability to do business were non-beneficiaries. From a regional perspective, the largest numbers of respondents who feel that their ability to do business, in general, and women’s ability to do business, in particular, has improved are from Punjab and the smallest from KP. Access to Education Unsurprisingly, the most widely accessible educational institutions are public schools, with 22 out of 24 focus groups saying that they have a public school in their community. Close to half of all focus group participants stated that they send their children to public schools. Private schools are also prevalent, with 17 of 24 of all focus groups stating that they also have a private school in their communities (the majority of the private schools are in Punjab and KP). There were no non-beneficiaries that said they send their girls to private school, whereas a quarter of beneficiaries (3 of 12) said they sent girls to private schools (all in Punjab and KP). Affordability is also mentioned as a significant factor affecting school choice, contributing to greater attendance in public schools. Close to two thirds of all focus groups (15 of 24) say that boys and girls have equal access to education, but non-beneficiaries were only half as likely to respond positively to this question as beneficiaries. Furthermore, the only respondents who stated that boys and girls do not have equal access were non-beneficiaries. Additionally, while greater than half of all respondents (13 of 24) stated that more girls currently go to school than in the past, beneficiaries were three and a half times more likely to agree than non-beneficiaries (10 of 24). 53 When asked about the factors influencing the decision to send their children to school, close to two thirds of focus groups mentioned future returns on education (15 of 24). However, future returns were cited twice as often as the motivation for boys to attend school as for girls (16 vs. 8). Notably, those respondents who mentioned future returns on girls’ education were equally divided by gender (4 men, 4 women), but the majority were non-beneficiaries. One focus group stated they viewed education for girls is not only an investment providing future financial returns, but also as the only way to find a husband for their girls. Finally, cultural barriers and family opposition were mentioned by 5 of 24 focus groups as barriers to sending girls to school; all of these focus groups were in KP. 53 A question that neither the survey nor focus group discussions addressed directly was the difference in access to education between primary versus secondary school (particularly for girls). While a majority of focus groups described access to education as equal across genders, this equality applied primarily to the community-level primary school. Groups that mentioned the topic indicated that if attending secondary school required traveling outside the community, girls’ access to education declined dramatically. This important distinction requires further exploration in future surveys. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 35 Mobility and Ability to Move Freely Both survey and focus group data indicate that freedom of mobility and freedom to socialize and communicate are predominantly male prerogatives. Survey data indicate that on a range of empowerment indicators such as ability to work outside the home, to visit friends alone, to travel in public alone and to use mobile phones, typically 80 percent of males enjoy these freedoms while only 20 percent of females enjoy the same freedoms (see Figure 8). Focus group discussions offer interesting insights into how beneficiary status influences these basic freedoms. In their mobility and freedom to use mobile phones, direct beneficiaries, both male and female, described higher levels of independence than non-beneficiaries. Further, while two-thirds of focus groups said that women’s mobility has improved over the last five years, only a third of non-beneficiary groups agree. In addition, while 7 out of 12 beneficiary focus groups said women can move with no restriction, no non￾beneficiary focus group respondents said the same. In fact, all non-beneficiaries said that women must travel with a male escort (12 of 12). FIGURE 8: RESPONDENTS' MOBILITY AND FREEDOM BY BENEFICIARY TYPES (PERCENT) Source: EGA-PIA first round survey, 2014. Participation in Household-level Decision-making Survey data show that direct beneficiaries, both males and females, have higher levels of decision-making power over spending on food, clothing, health and education than non-beneficiaries. It is particularly interesting that beneficiary women indicate more decision-making power on household issues than non￾beneficiary women across all types of decisions. While male beneficiaries demonstrate the most decision￾making authority, indirect beneficiaries are generally worse off in this regard than non-beneficiaries (see Figure 9). It is also worth mentioning that household decision-making shows significantly greater gender equity than freedom of mobility. These findings were confirmed in focus group discussions where beneficiary women’s groups cited greater decision-making power in general, and particularly concerning health, education and their children’s marriage ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 36 than non-beneficiaries. For example, beneficiaries were three times more likely than non-beneficiaries to say that all decisions are made jointly, that a woman’s income earning ability gives her more of a role in household decision-making, and that women’s participation in decision-making has increased significantly over the past five years. FIGURE 9: INDIVIDUAL DECISION-MAKING ABILITY ON HOUSEHOLD SPENDING (PERCENT) Source: EGA-PIA first round survey, 2014. Participating in Decision-making Regarding Children Direct beneficiaries demonstrated more decision-making ability than non-beneficiaries on a range of key issues regarding child rearing (see Figure 10). Across all survey respondents, women described having more decision-making ability than men on all child-related issues, particularly health care and education. The sole exception to this trend was the issue of marriage, where non-beneficiary men have higher decision-making power than non-beneficiary women. On the specific issue of marriage, it is worth mentioning that male indirect beneficiaries reported the same level of decision-making ability as direct beneficiaries and that women indirect beneficiaries reported the highest level of influence of all groups. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 37 FIGURE 10: DECISION-MAKING ABILITY ON CHILDREN’S ISSUES (PERCENT) Source: EGA-PIA first round survey, 2014. Household Division of Labor Responses among all focus group participants regarding division of household labor are predictably skewed towards women doing household chores (20 of 23) and men working outside the house (23 of 24). Only a quarter of all groups say that men participate in household chores; perhaps tellingly those saying so were five times more likely to be men than women. Despite a strong gender imbalance in the division of household labor, it is encouraging to note that half of all focus groups say women do not just do household chores, but also engage in income-earning work. Importantly, beneficiaries were twice as likely (8 of 12) as non-beneficiaries (4 of 12) to report women engaging in income-earning work. This is particularly encouraging considering the influence that a woman’s ability to earn income has on empowerment indicators such as participation in household-level decision making. It is also positive to note that greater than 15 of 24 FGD respondents agreed that more women are employed now and that there are more employment opportunities for women (14 of 24). Focus groups indicated that beneficiary status has a significant influence on employment, with beneficiaries twice as likely to say that women pursue income-earning work as non-beneficiaries (8 to 4). It is notable that 10 out of 12 of the focus groups that said this are from KP and Sindh. Voting in Elections Focus group discussions indicate that, in general, direct beneficiaries vote more and choose candidates more independently than non-beneficiaries. Beneficiary focus groups are twice as likely to state that women make totally independent voting decisions and a third less likely to state they are told how to vote by their husbands. Nonetheless, the opinions of family and community elders or leaders have a strong influence on both men and women’s voting choices across all beneficiary strata. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 38 Participation in Community-based Organizations While many focus group participants indicated that there are no community-based organizations in their village, this apparently did not include jirgas54 and other traditional organizations. This trend was not, however, universal, as non-beneficiaries were three times as likely to say there are no organizations in their communities. Although it is not surprising that there are more community organizations in beneficiary villages than non-beneficiary ones, the magnitude of the difference in participation is notable. While survey data indicate that 67 percent of female beneficiaries reported participating on community organizations, only 13 percent of non-beneficiary women participate. This gap is also reflected in male beneficiaries whose participation rate in community organizations is 71 percent compared to non-beneficiary males whose participation rate is 28 percent (see Figure 11). Regionally, women generally have more freedom and independence in KP and Punjab than in Sindh. FIGURE 11: RESPONDENTS' FREEDOM ON VARIOUS ISSUES BY BENEFICIARY TYPE (PERCENT) Source: EGA-PIA first round survey, 2014. Perceptions of Social Change In a final analysis of survey data on empowerment, a representative set of indicators is grouped to assess perceptions of social values at the community level. These include indicators both of freedom to (e.g. to work) and freedom from (e.g. from violence). Based on these indicators, it is interesting to note that although direct beneficiaries do not always have a more optimistic outlook than their counterparts in terms of their current economic status, they are universally more positive in terms of how the situation in their communities has improved over last five years (see Table 8).  Fewer beneficiaries said that women are free to work outside home, but a higher percentage said that women’s freedom to work has increased. 54 A jirga is a tribal assembly of elders that makes decisions by consensus. There are several types of jirga, some of which are government initiated, and others that are initiated at the tribal level. The sarkari jirga is a government jirga, while the shaksi and Olasi jirgas are traditional tribal jirgas charged with various responsibilities. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 39  Slightly fewer beneficiary women said that they were free to choose a spouse, but significantly more said that women’s freedom to choose spouse has increased ( > 8 percent). This also implies freedom to choose a spouse for their children.  A majority of respondents said that violence against women is not acceptable, but significantly more beneficiaries said violence against women has decreased over the past five years (almost 10 percentage points more than non-beneficiaries).  Significantly more beneficiaries said that violence against children is not acceptable than non￾beneficiaries and that violence against children decreased (by almost 10 percentage points).  Beneficiaries were almost seven percent more likely to say that marginal groups are included in the community than non-beneficiaries. TABLE 8: PERCEPTIONS OF COMMUNITY VALUES Issues DB (%) IB (%) NB (%) Women are free to work outside home 11.7 13.6 16.1 Women’s freedom to work outside home increased during last 5 years 39.5 37.9 29.4 Women are free to choose spouse 6.3 7.1 7.5 Women’s freedom to choose spouse increased during last 5 years 30.4 30.2 22.0 Violence against women is not acceptable 50.1 48.1 50.6 Violence women has decreased during last 5 years 46.6 44.2 37.0 Violence against children is not acceptable 46.9 43.8 40.6 Violence against children has decreased during last 5 years 46.4 43.2 37.7 Conclusions Pakistan is experiencing a period of rapid change in economic and social empowerment, in large part due to education and women’s participation in the workforce. Current trends indicate that women, men and their children (particularly beneficiaries) are enjoying greater social freedom, sowing the seeds for a more equal society in the future. While inflation and natural disasters will continue to negatively affect rural communities, the rising participation of women in income-generating activities and the tendency to move towards higher paying work will help to mitigate and, hopefully, to reduce the negative impact of these externalities over the long term. Positive changes in a range of interwoven economic and social empowerment indicators appears linked to access to education, markets and services and to USAID beneficiary status. While some of these indicators are linked to EGA-supported activities, others are exogenous to USAID interventions. For example, the majority of respondents (particularly beneficiaries) stated that their access to markets has improved and cited factors such as improved roads, transportation and communication (which are not part of EGA’s portfolio). Although both beneficiaries and non-beneficiaries enjoy the benefits of activities that are not directly funded by EGA (such as primary education), there is a notable synergy with beneficiaries taking greater advantage of non-EGA funded activities than non-beneficiaries. For example, although EGA does not fund access to primary education, EGA beneficiaries have a greater tendency to ensure access education for their girls than non-beneficiaries. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 40 Both survey and focus group data demonstrate that social empowerment in the form of freedom for positive action and freedom from negative actions is linked to economic empowerment and to EGA project participation. As an example, focus group discussions clearly indicate that beneficiary women are able to exercise greater freedom in choosing to work outside the home than non-beneficiary women. Female beneficiaries also tend to have greater freedom of mobility than their non-beneficiary counterparts, providing them with greater potential for social interaction and for acquiring resources to improve their employment opportunities and business prospects. Along this same line, although the acceptability of domestic violence has decreased in general, beneficiaries are significantly less likely to accept violence against children than non￾beneficiaries. The interrelated nature of economic and social empowerment (and the implicit obligation to view empowerment from a holistic perspective) is demonstrated by the influence that higher incomes among women and greater awareness among men have on increased decision-making ability for women. As education and skills training improve women’s ability to find work, increased income positively influences women’s decision-making power in the household. In addition, higher incomes among both men and women appear to increase girls’ access to education, thereby improving their ability to obtain employment and become decision-makers in the future. Question 3 To what extent and in what ways have USAID-supported activities influenced EGA program beneficiaries’ employment, income, expenditure, economic empowerment and social empowerment? Key Findings Overview This question directly addresses the fundamental challenge and primary focus of this research - what has USAID’s role been in fomenting change in economic opportunity, economic and social empowerment and community change outcomes? The response to this question encompasses a broad range of interrelationships and influences among social and economic variables at the individual, household and community levels, drawing them together analytically to measure USAID’s influence on development outcomes. Completing the circle, the answer to this question feeds back to confirm or revise the theory of change that underlies and frames this research. Descriptive analysis of the survey data and contextual analysis of the focus group data were sufficient to effectively address the first two questions of this assessment on economic opportunity and economic and social empowerment. However, to address the key final question of USAID’s influence on social outcomes, more sophisticated regression analysis was required. Because these findings are based on complicated statistical analysis, a brief review of the methodologies used to produce them is presented first to aid in their interpretation and to demonstrate their statistical validity. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 41 Key Survey Findings on Question 3  Participation in EGA program activities positively influences economic opportunities, economic empowerment and perceptions of social change.  Like direct beneficiaries, indirect beneficiaries fare better on a range of opportunity and empowerment outcomes than non-beneficiaries, although not as much.  Individuals with economic opportunity are more likely to be economically empowered.  Individuals who are economically empowered are more likely to be socially empowered.  Individuals who are economically and socially empowered are more likely to perceive positive social change in their communities.  EGA program activities had greater influence on development outcomes in Punjab and Sindh and less in KP.  Male beneficiaries have greater economic opportunity, while female beneficiaries score best on economic empowerment and social change. Factor Analysis To effectively address Question 3, factor analysis was used to construct four indices to score respondents on the four dimensions of the theory of change (i.e. economic opportunity, economic empowerment, social empowerment, and community change) and to identify these otherwise difficult-to-quantify concepts. These indices were created by selecting and grouping coherent subsets of the 50 + indicators captured in the survey (e.g. asset ownership, access to markets, household decision-making, domestic violence, etc.) and assigning them to each of the four theory of change components (see Table 5 above). Each of these indicators was grouped with a germane index, for example, it was assumed that variables such as sources of income, productive assets and monthly expenditures on particular food items would be among the indicators comprising the economic opportunity index. Factor Analysis Factor analysis uses correlations among measurable (observed) indicators to infer a latent, usually un-measurable, behavior or attribute. By creating a single score to represent many indicators, factor analysis also offers insights into underlying trend that would otherwise be difficult, if not impossible, to measure. The score is a weighted combination of the observed variables (indicators that are measured through the survey), where weights are correlation coefficients (also called factor loadings). For example, the score for economic opportunity can be expressed as, scoreeo = a1x1 + a2x2 + a3x3 + . . . + anxn where, x1, x2, x3, . . . xn are indicator variables such as household ownership of productive assets, land ownership, housing facilities, and so on, and a1, a2, a3, . . . an are correlation coefficients between the score and each of the x’s. Each of the correlation coefficients takes a value from 0 to 1, and shows to what extent the change in one indicator is associated with change in the overall score. For example, a correlation factor (value of a’s) of 0.70 implies that the score will increase by 0.70 percent with a one percent increase in the value of the indicator. Since correlation does not imply causality, we can also say that the indicator will increase by 0.70 percent with a one percent increase in the score. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 42 Indices were created from the indicators in two steps. First, factor analysis was used to create a score for each of the four dimensions. These scores were then used to rank each survey respondent, with the scores reflecting the respondents’ rank for each dimension. One difficulty with these scores is that, while they are useful in comparing one respondent relative to another, they cannot determine if a respondent has reached a meaningful status with respect to the dimension it represents. For example, a respondent with a score of 0.55 for social empowerment is evidently better off than one with a score of 0.44, but these scores offer no indication whether either has (or both have) a high enough score to be called socially empowered. To resolve this ambiguity, a 0/1 index was created for each score. A value of 0 or 1 was then assigned to each survey respondent based on his or her score, using a cut-off point at the median overall score.55 For example, if the median value for social empowerment score is 0.75 then all respondents with a score above 0.75 are given a value of 1 for the social empowerment index, indicating that they are considered socially empowered. In contrast, respondents with a score below 0.75 will get 0 for their social empowerment index, implying a lack of social empowerment. Using this technique, indices were created for all four dimensions of opportunity and empowerment.56 One of the most important relationships that this assessment sought to measure was that of USAID beneficiary status relative to the development outcomes comprising the theory of change. Table 9 shows how a respondent’s welfare on the four indices representing the components of the EGA theory of change varies by his or her status as a direct beneficiary (DB), indirect beneficiary (IB) or non-beneficiary (NB). More specifically, the table shows the share (percentage) of each beneficiary type that has economic opportunity, is economically and socially empowered and has positive perception about social change. This table clearly indicates that direct beneficiaries are better off (that is, have a higher share of respondents with economic opportunity, empowerment, and so on) than non-beneficiaries in terms of all development outcomes, except for social empowerment (where t-statistics of the difference are not significant). The most notable difference is in the perception of social change, where direct beneficiaries were almost 16 percentage points more likely to see positive social change than non-beneficiaries. The smallest difference is observed for economic opportunity, which direct beneficiaries were 9.4 percentage points more likely to have than non-beneficiaries. It is interesting to note that even though the relationship between direct beneficiaries and non-beneficiaries in terms of social empowerment is not statistically significant, it is significant between indirect beneficiaries and non-beneficiaries, with the former group being almost 4 percentage points more likely to be socially empowered than non-beneficiaries. 55 The use of indices (or indexes) is fairly common in development communities. For example, the World Bank’s Doing Business index and the State Department’s Trafficking in Persons index are two well-known indices compiling multiple variables into scored rankings. 56 Converting a continuous-value score to a dummy variable (0/1) index is a common practice in the social sciences. For example, in a study to measure vulnerability, Khandker, Khalily and Samad (2012) created an index by setting a cut-off point at 0.50 of a continuous variable (ranging 0 to 1). A household was considered vulnerable if it had a value 0.5 or above for the continuous variable, and non-vulnerable otherwise. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 43 TABLE 9: OUTCOME INDICES BY BENEFICIARY STATUS (Share of respondents attaining the four dimensions of Theory of Change) Outcome Indices DB IB NB tDB_NB tIB_NB Percentage of respondents with economic opportunity 54.9 49.5 45.5 5.63** 2.41** Percentage of respondents that are economically empowered 68.1 60.9 53.3 9.13** 4.62** Percentage of respondents that are socially empowered 60.1 64.3 60.4 -0.22 2.39** Percentage of respondents with positive perception of social change 56.1 52.9 40.4 9.57** 7.61** Source: EGA-PIA first round survey, 2014. Note: DB, IB and NB refer to direct beneficiaries, indirect beneficiaries and non-beneficiaries, respectively. The last two columns refer to t-statistics of the difference in indices between two types of respondents as denoted by the subscripts. **Implies that the difference is statistically significant at a level of 5 percent or less on a two-tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Table 10 examines the influence of province on development outcomes for beneficiaries and non￾beneficiaries. .As the table shows, the difference in outcome indices between direct beneficiaries and non￾beneficiaries is highest in Sindh and lowest in KP. For example, differences in the social change index is 20.5 percentage points in Sindh, followed by 14.4 percentage points in Punjab, and 10.6 percentage points in KP. At the same time, the difference in outcome indices between indirect beneficiaries and non-beneficiaries in Punjab is higher than that in the other two provinces. In both comparisons, KP has come out last. There are few cases where non-beneficiaries have higher index values than beneficiaries (indicated by negative values), however they do not count and can be considered zero for all practical purposes since they are not statistically significant. Finally, it is important to remember that these statistics do not necessarily indicate that one province is better or worse off than another, but rather possibly how direct and indirect beneficiaries fair compared to non-beneficiaries in those provinces. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 44 TABLE 10: DIFFERENCE IN OUTCOME INDICES BETWEEN BENEFICIARY TYPES BY PROVINCE Outcome Indices Punjab Sindh KP Difference in Indices Between Direct Beneficiaries and Non-Beneficiaries Difference in economic opportunity index (percentage points) 11.3** 15.5** 1.8 Difference in economic empowerment index (percentage points) 16.7** 16.6** 11.5** Difference in social empowerment index (percentage points) 0.4 -2.0 0.4 Difference in index for positive perception of social change (percentage points) 14.4** 20.5** 10.6** Difference in Indices Between Indirect Beneficiaries and Non-Beneficiaries Difference in economic opportunity index (percentage points) 9.6** 7.7** -3.8 Difference in economic empowerment index (percentage points) 10.6** 10.0** 3.0 Difference in social empowerment index (percentage points) 7.2** -2.3 1.9 Difference in index for positive perception of social change (percentage points) 16.2** 11.9** 8.9** Source: EGA-PIA first round survey, 2014. Note: **Implies that the difference is statistically significant at a level of 5 percent or less on a two-tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Also, the figures in this table are differences in indices, not the actual values of the indices Table 11 breaks down the influence of gender on development outcomes for beneficiaries and non￾beneficiaries. Again, the tale shows differences in index values, rather than the indices themselves. Findings show that for the economic opportunity index, the difference between male direct beneficiaries and non￾beneficiaries (23.5) is much higher than that for females (-6.1 and statistically insignificant). For the other three indices, however, such differences are higher for females than for males. In the comparison between indirect beneficiaries and non-beneficiaries, differences in economic opportunity are again higher for males, while the difference in economic empowerment is same for males and females. Overall (considering all indices and both comparisons), the index-gap for female beneficiaries is higher than that for male beneficiaries in the majority of cases. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 45 TABLE 11: DIFFERENCE IN OUTCOME INDICES BETWEEN BENEFICIARY TYPES BY GENDER Outcome Indices Males Females Difference in Indices Between Direct Beneficiaries and Non-Beneficiaries Difference in economic opportunity index (percentage points) 23.5** -6.1 Difference in economic empowerment index (percentage points) 12.4** 16.6** Difference in social empowerment index (percentage points) 6.2** 8.3** Difference in index for positive perception of social change (percentage points) 11.0** 20.5** Difference in Indices Between Indirect Beneficiaries and Non-Beneficiaries Difference in economic opportunity index (percentage points) 10.4** -2.6 Difference in economic empowerment index (percentage points) 7.8** 7.8** Difference in social empowerment index (percentage points) 4.8** 3.4 Difference in index for positive perception of social change (percentage points) 9.8** 15.6** Source: EGA-PIA first round survey, 2014. Note: **implies that the difference is statistically significant at a level of 5 percent or less on a two-tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Regression Analysis Descriptive statistics of the four dimensions of the theory of change (economic opportunity, economic empowerment, social empowerment and social change) (reported in Tables 9-11) unambiguously demonstrate that direct beneficiaries are overall better off than non-beneficiaries. Similarly, indirect beneficiaries are also better off than non-beneficiaries, but to a lesser degree. However, these descriptive statistics cannot be interpreted as anything more than simple correlation. Although survey data indicate that beneficiaries have improved outcomes, without regression analysis we cannot say that such improvement is due to program participation. A host of factors, besides EGA program interventions, could potentially have affected these outcomes. Without further analysis we cannot be sure which factors contributed to these differences and to what extent. Potential confounding exogenous factors include respondents’ individual characteristics (i.e. age, education, and sex) and household and community characteristics and development activities not directly related to EGA program (e.g. household head’s age, and/or gender, village population density, infrastructure development, proximity to urban centers, presence of various NGO and donor-funded development programs and aggregate shocks or natural disasters such as floods, earthquakes in recent time, and so on). Please see Table 6 for a list of such factors. To measure the true effects of EGA program interventions, these potentially confounding factors were controlled for using regression analysis, thereby allowing for a more accurate measurement of the effects of EGA interventions on economic opportunity, economic and social empowerment, and perception of social change. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 46 Table 12 shows the influence of program participation (direct and indirect) on development outcomes.57 Besides reporting the effects of program participation, this table also shows the impacts of lower level outcomes on higher level outcomes – such as effects of economic opportunity on economic empowerment, social empowerment, effect of social empowerment on social change, and so on. Data indicate that both direct and indirect participation increase the probability of having economic opportunity, being economically empowered, and perceiving positive social change. For example, being a direct beneficiary increases the probability of having economic opportunity by 8.8 percentage points and of being economically empowered by 12.7 percentage points. Similarly, being an indirect beneficiary increases the probability of attaining those outcomes by 6.2 percentage points and 7.1 percentage points, respectively. Economic opportunity, in turn, increases the probability of being economically empowered by 8 percentage points and of being socially empowered by 4.8 percentage points. Social empowerment is the only outcome that is not directly affected by program participation (although it is directly affected by economic empowerment, which is affected by program participation). While these changes may be small in absolute terms, they are substantial in terms of percentage change. For example, the 11.6 percentage point difference in perception of social change for direct beneficiaries implies a change of more than 26 percent over their pre-participation status. TABLE 12: REGRESSION ESTIMATES OF EGA PROGRAM EFFECTS ON OUTCOME INDICES Independent Variables Change in Outcome Indices Economic Opportunities Economic Empowerment Social Empowerment Social Change Change Percent Change Change Percent Change Change Percent Change Change Percent Change Direct EGA beneficiary status 8.8** 19.1 12.7** 22.9 -3.0 - 11.6** 26.1 Indirect EGA beneficiary status 6.2** 14.3 7.1** 13.2 2.5 - 9.5** 21.9 Economic opportunity - 8.0** 13.5 2.0 - 6.5** 8.8 Economic empowerment - - 4.8** 7.6 4.4** 9.0 Social empowerment - - - 5.6* 11.5 Source: EGA-PIA first round survey, 2014. Note: Change is expressed in percentage points. * and ** imply that the difference is statistically significant at a level of 10 percent, and 5 percent or less, respectively, on a two-tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Regression estimates control for various individual, household and community characteristics as reported in Tables 6. 57 For this table we used a seemingly unrelated regression (SUR) statistical model to run four equations simultaneously, each having its own dependent variable and set of exogenous explanatory variables, and each equation being a valid linear regression on its own. This is technically a better option than running four Ordinary Least Square (OLS) regressions separately because OLS cannot (and SUR can) control for correlations among error terms of the equations, which can potentially bias the estimates. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 47 Like Table 12, Table 13 reports findings on program effectiveness, but with data disaggregated by province. According to these data, program effectiveness varies distinctively by province. While being a direct or indirect beneficiary of the EGA program increases economic opportunity in Punjab and Sindh, it does not do so in KP. Similarly, having economic opportunity or being economically empowered improves social empowerment in both Punjab and Sindh, but not in KP. In fact, EGA program interventions do not appear to be effective at enhancing social empowerment in KP. Interestingly, in Punjab, social change is significantly influenced by program participation, economic opportunity and social empowerment, but less so by economic empowerment. Overall, EGA program interventions have improved development outcomes the most in Punjab, and the least in KP. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 48 TABLE 13: REGRESSION ESTIMATES OF EGA PROGRAM EFFECTS ON OUTCOME INDICES BY PROVINCE Explanatory Variables Economic Opportunity Economic Empowerment Social Empowerment Social Change Punjab Sindh KP Punjab Sindh KP Punjab Sindh KP Punjab Sindh KP Respondent is a direct beneficiary 11.9** 10.3** 5.5 13.0** 13.8** 12.6** -5.4 -1.6 -4.0 18.2** 13.2** 4.5 Respondent is an indirect beneficiary 11.3** 6.0* 2.0 7.1* 7.8** 9.6* 1.1 2.4 1.0 19.4** 5.7* 5.7 Respondent has economic opportunity - - - 9.1 6.2** 9.4** 6.7** 6.2** -2.2 12.5** -1.8 8.9** Respondent is economically empowered - - - - - - 6.2** 11.2** -1.4 4.4* 3.2 2.1 Respondent is socially empowered - - - - - - - - - 10.5** 4.9** 3.7 Source: EGA-PIA first round survey, 2014. Note: Change is expressed in percentage points. * and **imply that the difference is statistically significant at a level of 10 percent, and 5 percent or less, respectively, on a two￾tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Regression estimates control for various individual, household and community characteristics as reported in Tables 6. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 49 The influence of gender on outcome indicators is detailed in Table 14. Among males, direct and indirect beneficiaries have the greatest probability of having economic opportunity (23 percentage points and 13 percentage points, respectively). Although the relationship between beneficiary status and economic opportunity for women is not statistically significant, program participation had a greater influence on economic empowerment and social change for women than for men (women beneficiaries had about a 20 percentage point higher probability of attaining economic empowerment and 18 percentage point higher probability of perceiving positive social change). These findings mean that while male beneficiaries enjoy more economic opportunities, female beneficiaries do better in economic empowerment and perception of social change. TABLE 14: REGRESSION ESTIMATES OF EGA PROGRAM EFFECTS ON OUTCOME INDICES BY GENDER Explanatory Variables Economic Opportunity Economic Empowerment Social Empowerment Social Change Men Women Men Women Men Women Men Women Respondent is a direct beneficiary 22.5** -3.7 6.0** 19.8** 2.7 -3.2 5.8* 17.7** Respondent is an indirect beneficiary 13.4** -0.7 3.5 11.9** 1.8 3.3 8.4** 10.9** Respondent has economic opportunity - - 9.5** 7.8** 2.9* -0.5 3.3 11.1** Respondent is economically empowered - - - - 3.3* 6.7** -1.0 7.7** Respondent is socially empowered - - - - - 1.0 9.8** Source: EGA-PIA first round survey, 2014. Note: Change is expressed in percentage points. * and **imply that the difference is statistically significant at a level of 10 percent, and 5 percent or less, respectively, on a two-tailed test. Please note that the critical values of t-statistics to be considered statistically significant are: 1.63 or higher for a significance level of 10 percent, 1.99 or higher for a significance level of 5 percent or less. Regression estimates control for various individual, household and community characteristics as reported in Tables 6. Findings from both descriptive and regression analyses suggest that direct beneficiaries are better off than non-beneficiaries in terms of economic opportunity, economic empowerment and perception of social change. Indirect beneficiaries are also better off, although not by as much as direct beneficiaries. Interestingly, while social empowerment is directly improved by economic empowerment, it is only indirectly influenced by program participation. However, perception of social change, the highest-level outcome, is influenced directly by EGA program interventions, and by other outcomes such as economic opportunity and economic empowerment that are directly influenced by project participation. At the same time, the influence of EGA program participation on development outcomes is not consistent across provinces. EGA interventions appear to have the greatest influence on development outcomes in Punjab and the least influence in KP. Finally, women seem to benefit more than men from EGA program participation, particularly in terms of economic empowerment and social change. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 50 Conclusions The interrelated nature of economic opportunity, empowerment and social change is demonstrated in the complex patterns of influence that each has on the other and that EGA programming has on all three. From these findings we can concluded that:  EGA has had both deep direct impact and broad indirect impact at the individual, household and community levels.  EGA has increased beneficiaries’ economic and social empowerment and well-being and improved their ability to make and act on decisions, control resources and advance economically and socially.  Positive change in a range of inter-connected economic and social empowerment indicators is linked to access to education, markets and services and, most importantly, to USAID beneficiary status.  USAID programs have had greater impact on economic empowerment of women than on economic opportunity for women.  For women, social empowerment is most strongly linked to economic empowerment but is not significantly linked to economic opportunity.  Program participation has its greatest impact on economic opportunity for men, but significantly less impact on economic empowerment for men.  Social change is much more strongly influenced by program participation, economic opportunity and economic and social empowerment for women than for men. From these data it is reasonable to assume that USAID program participation has a greater overall impact on women than on men, particularly on economic empowerment and social change.  The impact of USAID programs extends beyond direct program beneficiaries.  As three of the four USAID programs surveyed for this study are implementing activities in more than one region, it is likely that regional-level differences in program impact are not driven by differences in program technical approach or management.  The lack of impact of USAID program participation on economic opportunity, social empowerment and social change in KP presents a programmatic challenge for the Mission. The influence of EGA program participation on development outcomes is broad and strong, but it is not homogenous. EGA appears to be particularly effective at promoting economic opportunity for men and economic empowerment for women. However, it is important to keep in mind where EGA programming has had the least impact, on social empowerment, and to consider how other initiatives (such as girls education) could offer synergistic complements to EGA activities. Better understanding of the differential impact of EGA programs on women and men will be an important goal for rounds 2 and 3 of survey research. It is also important to consider the dynamics behind regional-level differences in program impact. Assuming that it is not differences in USAID programs themselves that cause these differences, how can EGA adjust is program design and implementation to take into account the impact of regional location on development outcomes. How, for example, can USAID capitalize on the relative success of its programs in Punjab and improve the impact of its programs in KP. Again, further survey rounds should shed additional light on these trends. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 51 Reviewing the Assessment’s Theory of Change One of the fundament objectives this assessment was to test the validity of EGA's theory of change. The basic hypothesis of this theory of change is that EGA project interventions supporting policy reform, access to markets, workforce development, introduction of new technology and agricultural best practices will increase agricultural productivity improve resource management, stimulate economic growth and job creation and increase incomes. In turn, increased income and job creation (economic opportunity) is expected to lead to economic and social empowerment of program beneficiaries at the individual and household level and social change at the community level (see Figure 2 for EGA’s hypothesized theory of change). Embedded within this theory of change was an initial set of assumptions about the relationships among development, empowerment and social change. These were:  Enhanced economic opportunity leads to economic and social empowerment for both individuals and households;  Enhanced economic and social empowerment at the individual and household levels lead to social change at the community level;  Increased employment and income-earning opportunities for women lead to their greater economic and social empowerment within the household; and  Increased empowerment of women within households will result in positive social change in gender norms at the community level. Based on an extensive literature review on empowerment and the relationships among economic opportunity, economic and social empowerment and social change, the assessment team added the following corollary assumptions:  Economic and social empowerment at the individual and household levels is mutually reinforcing. That is, as the ability to advance economically increases, so does the autonomy to make and act on choices. At the same time, increased social autonomy allows one a greater ability to take advantage of economic opportunities and become economically empowered.  Similarly, positive social change at the community level reinforces the economic and social empowerment of individuals and households, as previously marginalized groups (e.g., women, the poor) advance economically and influence the social constructs of both household and community life.  Positive social change at the community level reinforces economic opportunity, as formerly marginalized groups increase their participation in the local economy and expand the community’s potential to increase employment opportunities and expand markets. Based on the quantitative analysis of survey data and qualitative analysis of focus group data presented above, the theory of change was revised. While many of the initial assumptions about the relationships among opportunity, empowerment and social change were proven valid, others required revision and some were proven unfounded by the data. The revised theory of change is described as follows and depicted graphically in Figure 12:  Enhanced economic opportunity leads to economic empowerment for both individuals and households, as well as to social change at the community level.  Enhanced economic and social empowerment at the individual and household levels lead to social change at the community level; ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 52  For women, increased economic empowerment leads to their greater social empowerment within the household.  As groups of women become empowered within households, positive social change in gender norms and ideologies at the community level ensue.  Economic empowerment at the individual and household levels leads to social empowerment. As the ability to advance economically increases, so does one's autonomy to make and act on choices. While the findings of the survey and focus groups offer an in depth perspective on how development outcomes influence each other (and the role of EGA programs in promoting opportunity, empowerment and social change), there are insufficient data at this point to explain why the EGA theory of change was not fully borne out by these findings. It is possible that social empowerment is simply more complex than originally expected. In addition, unsurprisingly, data indicate that other (non-EGA) interventions such as health and education have a strong influence on overall development outcomes. Because this assessment is a baseline for future study rounds, it is not yet possible to compare change over time, which would certainly shed light on the dynamics at play in empowerment and social change. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 53 FIGURE 12: REVISED THEORY OF CHANGE ECONOMIC OPPORTUNITY Scale: Household Within local and regional economy, refers to availability of economic resources supporting household livelihoods: employment and income￾generating opportunities; market development; availability and use of modern technologies; availability of environmental resources (water/energy). As a result of USAID-funded projects, these resources expand, thereby improving Household Economic Status. ECONOMIC EMPOWERMENT Scales: Household & Individual  Ability to access economic opportunities  Ability to advance and succeed economically  Economic Security – sustained income earning potential; food security  Greater ability to access, control and share in resource use and economic decision￾making SOCIAL EMPOWERMENT Scales: Household & Individual Expansion in ability to define, make and act on choices; to engage in collective action to assert interests or challenge social hierarchies; and to influence governing policies.  Individual agency within array of institutions  Enhanced power and agency in non￾economic household decision-making such as health care, education, life partner, etc.  Freedom from gender- and child-based violence SOCIAL CHANGE Scale: Community Positive transformation in norms, ideologies and institutions that support the power and agency of individuals and households in economic and social matters.  Freedom from social violence  Increased mobility  Enhanced rights of access to resources  Democratic and inclusive institutions  Change in community attitudes about gender to support women’s participation in economy and community decision-making ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 54 RECOMMENDATIONS This assessment offers the following recommendation to EGA: 1. Fund and implement rounds 2 and 3 in 2017 and 2019. Although the findings from Round 1 are interesting and valuable in and of themselves, after two more rounds of data collection and analysis USAID/Pakistan will be able to much better understand and explain changes in beneficiary and non￾beneficiary populations due to program interventions. Round 1 demonstrated what is happening, but rounds 2 and 3 will provide greater insight as to how and why changes are occurring (and into USAID’s influence in promoting them). Also, while round 1 findings provide a short-term measure of EGA program benefits, findings of rounds 2 and 3 will capture long-term measures of such benefits and possibly sustainable ones. 2. Leverage the current impact assessment (and future assessments) to develop additional knowledge products (such as briefing notes and beneficiary impact case studies) for different USAID internal and external audiences. 3. Expand training programs on income-generating and other activities for women and girls. Such interventions have proven to be effective in increasing both the current and future empowerment of women in Pakistan. 4. Develop training programs on income-generating activities for youth. The strong influence of poverty (need) on increasing women’s employment implies a growing need for additional income at the household level. Increased youth employment can improve current household income levels, and also promote future socioeconomic stability. 5. Increase programmatic focus on lesser-developed provinces, in particular, KP, which lags behind the other provinces studied on almost every outcome indicator, both in the survey and in the focus group findings. At the same time, USAID should investigate why its program activities appear to have less of an influence on development outcomes in KP. 6. Support increased access to financial services, particularly for women. The lack of access to formal financial services was a hindrance to economic growth for the studied populations, both beneficiary and non-beneficiary. 7. Coordinate with other non-agricultural projects to increase the potential for synergistic impact. For example, roads are vital for easing access to markets where beneficiaries can buy inputs and sell their goods. Market proximity is especially important for women, since most are not allowed to travel long distances on their own. Also, access to education, particularly for women and girls, has a broad and diverse impact on both individual and household development outcomes. 8. Ensure that project managers measure common indicators consistently across all EGA projects. Lack of reliable data on beneficiaries, such as expenditures, farmers' gross income, value of sales and gender perspectives58 made the study more difficult than it could have been, especially when creating valid comparison groups. Training and regular (at least annual) data quality assessments would help tremendously. 9. Verify that project implementers maintain and report accurate records on all project beneficiaries. In some cases, lists of project beneficiaries were incomplete; in others, they were unavailable; and in others, the implementers were reluctant to release data to the assessment team. All of these hurdles 58 These indicators are all Standard Foreign Assistance ("F") indicators. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 55 delayed data collection and compressed the resources available for developing a robust beneficiary sample, data collection, analysis and report writing. The following general recommendations should be considered in USAID’s funding of further impact assessment research: 1. Explore the reasons behind the lack of influence of EGA project participation on social empowerment. Beneficiary status has a significant influence on almost every other outcome indicator, why is there no statistical linkage between project participation and social empowerment? 2. Compare individual and household economic status and empowerment, especially differences between women and men. Key questions include: What happens to women’s status and individual empowerment when household economic status improves? When access to education for women increases? Why? In what specific ways are women empowered or disempowered in this process? 3. Compare outcomes for households and individuals in rural and urban settings: Are there key differences in which certain indicators factor strongly between rural and urban households? Given the limited number of EGA beneficiaries in urban locations, additional effort will be required to ensure sufficient data from urban beneficiaries. 4. Investigate the dynamics of economic and social empowerment (especially for women and children) in nuclear family households versus joint family households. 5. Research differences between women’s employment and economic and social empowerment in tenant farmer and landless households. 6. Explore the impact of different types of employment (formal-informal, skilled-unskilled, paid-unpaid, low wage-high wage) on economic and social empowerment and social change. 7. Investigate the influence of non-EGA activities (in particular access to education for women and girls) on EGA beneficiary outcomes. 8. Assess the compare the impact of different projects over time to assess the relative effectiveness of different intervention strategies. ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 56 ANNEXES Annex I: Map of USAID/EGA Intervention Zones ECONOMIC GROWTH AND AGRICULTURE PORTFOLIO IMPACT ASSESSMENT: ROUND 1 57 Annex II: EGA Results Framework IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 58 Annex III: Descriptions of USAID EGA Methodologies EGA activities are segmented into the following four broad categories:59 One: Technical assistance to strengthen business policy and environment through:  Improved financial stability and fiscal discipline, liberalized trade policy  Improved business regulatory environment, particularly in agriculture. Two: Agricultural assistance to strengthen markets and institutions in the agricultural sector through:  Increased value chain productivity in processing, marketing, and product diversification, particularly in regions susceptible to extremist influence60  Strengthened farmer organizations and business linkages and introduction of improved production and processing technology and practices  Improved quality standards, sanitary and phytosanitary systems61  Strengthened Pakistani research and policy institutions and agricultural and market information systems. Three: Private Sector Assistance to small and medium enterprises through:  Increased access to bank finance and improved infrastructure and finance systems  Improved management practices and linkages to global marketing and distribution channels  Improved access to markets, information, technology, commodities and equipment  Increased women’s economic participation in microenterprises and agricultural production  Improved workforce development systems to identify skill gaps and train workers  Support for reform and increased accountability through advocacy by civil society institutions. 59 USAID Pakistan, “Mission Strategic Framework – Pakistan, Fiscal Year 2013-2017”, Background and Narrative. (2013). 60 The Development Objective-3 (DO-3) of the Mission Results Framework (Increased stability in target areas) has four Intermediate Results (IR3.1: Governance improved, IR3.2: Essential services delivered, IR3.3: Economic opportunities expanded, and IR3.4: Socially constructive values and beliefs embraced). 61 Sanitary and phytosanitary measures are defined as any measures applied: to protect human or animal life from risks arising from additives, contaminants, toxins or disease-causing organisms in their food; to protect human life from plant- or animal￾carried diseases; to protect animal or plant life from pests, diseases, or disease-causing organisms; and to prevent or limit other damage to a country from the entry, establishment or spread of pests. IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 59 Four: Water Programs to increase the availability for water, particularly in agricultural zones through:  Increased water retention and irrigation  Increased efficiency of water use for livestock and horticultural production  Strengthened academic and research institutions and policy making  Improved national water resources information and management systems  Strengthened local systems and institutions to manage water resources. IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 60 Annex IV: Impact Assessment Indicators Question 1 What is the current status of EGA project (actual and potential) beneficiaries in terms of economic status (employment, income and expenditures, assets)? 1. Employment status  Number of employed/economically active persons, and  Type of employment (e.g. employer, own account worker, contributing family worker, employee). 2. Income  Household income sources (e.g. wages and salaries, crop, livestock, non-agricultural activities, property, social insurance benefits/pension, gifts and assistance, foreign and domestic remittances, other income),  Types of economic activities,  Contributions of different activities to household income. 3. Expenditure  Average monthly and annual distribution of expenditure on housing, food, clothing, transport and communication, recreation and entertainment, education, housing, fuel and lighting, health, economic activity inputs and purchase of assets. 4. Assets  Cash, savings, and precious metals,  Livestock,  Land/real estate,  Tractor, farm equipment, and tools,  TV, Radio, Phone/cell phone,  Fan, Refrigerator, Washing machine,  Bicycle, Motorcycle, Car. Question 2 What is the current level of economic and social empowerment (e.g. measure of control over their livelihoods and participation in decision making at the household and community level) of the beneficiaries of EGA projects? 1. Economic empowerment  Control over assets (ownership of land, housing, livestock; own source of income; share of household income provided; control over how to spend cash/savings, share in), IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 61  Capabilities (skills, education),  Economic condition (employment, income, expenditure) of direct vs. indirect male/female beneficiaries or the demonstration/ spillover effect of USG assisted activities),  Control over resource allocation (e.g. for basic household needs such as among food, health, education, housing, assets and social status), and  Participation in economic decision-making (e.g. women’s influence within the household and marketplace on economic decisions such as household expenditures). 2. Social Empowerment  Individual capabilities (social belonging, sense of identity, autonomy and self-confidence to act),  Quality of life in terms of, buying power and the ability to imagine and aspire to a better future,  Collective assets and capabilities (such as voice, organization and representation),  Access to basic services/facilities (health, education, utilities, microfinance, skills development, housing and decent work),  Gender norms (ability to negotiate sexual and reproductive decisions and selection of spouse), and  Participation (individual participation in household decision making on issues other than economic ones and individual/household participation in community based organizations, producer groups and marketing/business associations), disaggregated by sex. Question 3 To what extent and in what ways have USAID-supported activities influenced EGA program beneficiaries’ employment, income, expenditure, economic empowerment and social empowerment? 1. Status of USAID program direct and indirect beneficiaries  Economic status (employment, income and expenditures, assets)  Economic and social empowerment (e.g. measure of control over their livelihoods and participation in decision making at the household and community level) of the beneficiaries of EGA projects 2. Status of USAID non-beneficiaries  Economic status (employment, income and expenditures, assets)  Economic and social empowerment (e.g. measure of control over their livelihoods and participation in decision making at the household and community level) of the beneficiaries of EGA projects IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 62 3. Statistical analysis of relationships between the component elements of the EGA program’s theory of change  Economic opportunity,  Economic empowerment,  Social empowerment, and  Social change. IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 63 Annex V: Impact Assessment Survey Questions Q# Survey Question ECONOMIC OPPORTUNITY Opportunity Structure Q8 Write the name of all household members above the age of 16 years Q9 Age of household members? Q10 Is household member male or female? Q11 Write Employment Status of HH members? Q12 Write Employment Sector of employed HH members Q13 If agriculture, what type? Q14 If agriculture, what is the tenure status? Q15 If non agriculture, what type? Employment Q17 Information about all members of your household? Income and Assets Q18 & Q19 From what sources does your household get income? What is the HH’s dominant source of income? Q20 How has your household income changed in the last 5 years? Q21 Which of these productive assets does your household own? Q22 Which of these other assets does your household own? Q23 How much land does your household own? Q24 & Q25 Do you own the house you live in? If no, do you pay rent? Q26. What type of house do you live in? Q27 What kind of roof your house has? Q28 What is the total number of enclosed rooms in your house excluding washrooms and kitchens? Q29 & Q30 Do you have a toilet in your compound? What type of toilet is used by the household? IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 64 Q# Survey Question Q31 What is the main source of drinking water? Q32 How much time does it take for you/household member to fetch drinking water? Q33 Does your house have an electricity connection? Consumption Pattern/Expenditures Q34 In a typical month, how often does your household consume the following food items? Project Participation Q35 Since 2010, have you participated in any work-related training activities? Q36 If yes, please identify each training activity that you have participated in. Q37 Did you complete the training? Q38 If no (to Q35), what is the primary reason preventing you from participating? ECONOMIC EMPOWERMENT Household Access to Economic Resources Q39 & Q40 To what extent, if at all, does your household have access to markets for sale of your products? To what extent, if at all, does your household have access to markets to purchase product inputs/raw materials? Q41 To what extent does your household have access to the following sources of finance? Q42 Which, if any, of the following social safety nets help your household? Debt Q43 Q45 Does your household owe any debt? Does anyone owe your household a debt? Q44 Q46 Is your household able to make regular loan repayments? Do they (your debtors) make regular repayments? Long Term Economic Security Q47 How economically secure do you think your household is? Q48 Do you think your household has the ability to improve its economic status in the next few years? Individual Access to/Control Over Resources IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 65 Q# Survey Question Q49 Are you currently engaged in any income-generating activity? Q50 If yes, which income-generating activity are you engaged in? Q51 Why are you engaged in this particular activity? Q52 Which of these reasons is the most important? Q53 Who decides how to spend the income you earn? Q54 To what degree are you free to use any of your household’s productive assets to earn an income? Q55 If not engaged in any income-generating activity, why not? Q56 To what degree are you free to make decisions about how any of your household’s productive assets will be used? Q57 Do you have access to other household assets? Q58 & Q59 Who decides who should have access to household financial assets)? Who makes the final decision on how these assets are used? Q60 Who makes the final decision about spending on food; clothing; health; education; and marriage? SOCIAL EMPOWERMENT Self Confidence/Self-Efficacy Q61, Q62 & Q63 To what extent do you feel respected in your family? To what extent do you feel comfortable expressing disagreement with your spouse? To what extent do you feel comfortable expressing disagreement with your family elders? Q64 & Q65 Can you imagine a better future for yourself? Can you imagine a better future for your children? Q66 & Q67 To what extent are you confident in your ability to improve your future? To what extent are you confident in your ability to improve the future of your children? Autonomy and Mobility Q68 To what extent are you free to work outside your home? Q69 & Q70 To what extent are you able to visit friends, family and neighbors on your own without anyone else’s company? To what extent are you able to associate with your business contacts on your own IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 66 Q# Survey Question without anyone else’s company? Q71 & Q72 To what extent are you able to freely use public transport? To what extent are you able to travel freely in public places? Q73 To what extent are you able to freely use media (TV, Radio, Internet)? Q74 To what extent are you able to freely use a mobile phone? Opportunity Structures/Institutional Access Q75 To what extent do you agree or disagree with the following statements? [The response options include “strongly agree”; “agree”; “disagree”; “strongly disagree”]. A. I participate in decision-making about children’s healthcare; education; and marriage B. I participate in decision-making about children’s education C. I participate in decision-making about children’s marriage D. I am comfortable approaching informal authority figures (e.g. village elders, jirgas and maulvis, etc.) with problems? E. I am comfortable approaching formal authority figures (e.g. police, court, councilor, government services, etc.) with problems? Q76 Did you vote in the last general election? Q77 If yes, were you able to independently choose who you voted for? Q78 If you did not vote, why not? Q79 Do you participate in any community based organization (NGOs; support groups; savings groups; social welfare; religious; sports; etc.)? Q80 If not, why not? HEALTH CARE Q81 To what extent are you able to access healthcare facilities for your personal health needs? Q82 To what extent are you able to make personal healthcare decisions independently? Social Change Q83 To what extent are women in your community allowed to work outside their homes? Q84 To what extent has women’s ability to work outside their homes changed in the past five years? IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 67 Q# Survey Question Q85 To what extent are women’s rights to inheritance accepted in your community? Q86 To what extent are women’s rights to choose their spouse accepted in your community? Q87 To what extent has acceptance of women’s rights to choose their spouse changed over the past 5 years? Q88 To what extent are women able to participate in and take leadership roles in community organizations (e.g. NGOs)? Q89 To what extent has women’s ability to participate in and take leadership roles in community organizations changed over the past 5 years? Q90 To what extent is it acceptable in your community to beat children? Q91 To what extent has violence against children changed in the past 5 years? Q92 To what extent is it acceptable in your community to beat women? Q93 To what extent has violence against women changed in the past 5 years? Q94 To what extent does your community come together to work towards shared goals? Q95 To what extent are women included in this community action? Q96 To what extent are marginalized groups included in this community action? Q97 & Q98 What are the major challenges to community security? Which one is the most important challenge? General Definitions: 1. The "reference period" is the month preceding the date of the interview. Age brackets are as follows: (1) under 7 years old; (2) 7 to 15 years old; (3) 16 to 29; (4) 30 to 59; (5) 60 and up 2. “Cluster” refers to the sub-set of sampled households by type of “beneficiaries” in an area e.g. sampled “Direct Beneficiaries” in Punjab; or “Indirect Beneficiaries” in Sindh; or “Non-beneficiaries” in KP. 3. The respondent is the male or female respondent of the household survey. IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 68 Annex VI: Sources of Information A. Secondary Data Sources Government of Pakistan, “Year Book 2011-2012,” Federal Board of Revenue, (2012). Government of Pakistan, “Pakistan Agricultural Census 2010,” Islamabad, (2012). Government of Pakistan, “Pakistan Social and Living Standards Measurement Survey 2011-2012,” Pakistan Bureau of Statistics, Islamabad, (2012). Government of Pakistan, “Labour Force Survey Annual Report 2012-2013," Islamabad, (2013). Government of Pakistan, “Economic Census of Pakistan,” Federal Bureau of Statistics, Islamabad, (2005). Government of Pakistan, “Pakistan Economic Survey: 2011-2012,” Ministry of Economic Affairs, Islamabad, (2012). Government of Pakistan, “SME Development in Pakistan: Issues and Remedies,” SME Policy of Pakistan (2005). Government of Pakistan, “Small and Microenterprise (SME) Policy,“ Small and Medium Enterprise Development Authority, Islamabad (2007). Government of Pakistan, “SME Development Report (2010 – 11),“ Small and Medium Enterprise Development Authority, Islamabad. (2012). Government of Pakistan, “SME Sector Genesis, Challenges and Prospects,“ Small and Medium Enterprise Development Authority, Islamabad. (2012). OECD, “Criteria for Evaluating Development Assistance,” Development Assistance Committee, (2005). USAID, “Pakistan Agriculture Sector Strategy, (2011-2015)” USAID, “Pakistan Food and Agriculture Project Report to USAID/Pakistan,” Weidemann Associates, Inc., (2009). USAID Pakistan, “Mission Strategic Framework – Pakistan, Fiscal Year 2013-2017”, Background and Narrative. (2013). USAID, “US Government Economic Growth Strategy for Pakistan”, (February 2011). USAID, “EGA Results Framework” USAID, “Feed the Future (FTF) Learning Agenda” USAID, Annual and Quarterly Reports from the following projects: Agribusiness Project, Balochistan Agriculture Project, Satpara Development Project, Entrepreneurs Project, Dairy Project, Firms Project) USAID, “Economic Growth Strategy” for Pakistan, Draft Working Paper, (2011). World Bank, Online demographic data. IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 69 B. Bibliography Alsop, R. Bertelsen, M.F. and Holland, J. Empowerment In Practice: From Analysis to Implementation. The World Bank, (2006). Alsop, R. Heinsohn, N. “Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators”, World Bank Policy Research Working Paper 3510, (February 2005). Amin, R. Becker, S. and Bayes, A. “NGO-Promoted Microcredit Programs and Women's Empowerment in Rural Bangladesh: Quantitative and Qualitative Evidence”, Journal of Developing Areas, 221-236, (1998). Asian Development Bank, “Impact Evaluation: Methodological and Operational Issues,” Economic Analysis and Operations Support Division, (2006). Derne, S. “Globalizing gender culture: Transnational cultural flows and the intensification of male dominance in India”. In K.E. Ferguson & M. Mironesco (Eds.), Gender and globalization in Asia and the Pacific. (pp. 121– 137). Honolulu: University of Hawaii Press, Honolulu. (2008). Golla, A.M., Malhotra, A., and Nanda, P. and Mehra, R. “Understanding and Measuring Women’s Economic Empowerment: Definition, Framework and Indicators,” International Center for Research on Women, (2011). Green, J.H. “Measuring women’s empowerment: development of a model”. International Journal of Media and Cultural Politics. Vol. 4, No. 3: 369-389, (2008). Hapke, H.M. “Theorizing Patriarchy: Development Paradoxes and the Status of Women in India.” Gender, Technology and Development. Vol. 17, No. 1: 1-29. (2013). Hapke, H.M. “Women’s Labor And Contribution To Household Income In The Informal Economy Of South India.” Paper presented at International Association for Feminist Economics conference, Sydney. (2006). Kabeer, N. “Resources, Agency, Achievements: Reflections on the Measurement of Women's Empowerment,” Development and Change Vol. 30 (1999), 434-464. Kabeer, N. “Gender equality and women’s empowerment: a critical analysis of the third Millennium Development Goals”. Gender and Development, Vol. 13, No. 1:13-24, (2005). Khandker, S., M. A. Khalily, and H. Samad. “Seasonal Hunger and Its Mitigation in North-West Bangladesh,” Journal of Development Studies, Vol. 48, No. 12:1750-1764, (2012). Mencher, J. “Women’s work & poverty: Women’s contribution to household maintenance in South India”. In Daisy Dwyer & Judith Bruce (Eds.), “A Home Divided: Women and Income in the Third World” (pp. 99–119). Palo Alto, CA: Stanford University Press. (1988). Pakistan Bureau of Statistics, Trading Economics. Accessed at: http://www.tradingeconomics.com/pakistan/inflation-cpi Parpart, J.L., Rai, S.M., and Staudt, K. “Rethinking empowerment, gender and development: An Introduction”. In Parpart, J.L., Rai, S.M., and Staudt, K. (eds.). Rethinking Empowerment: Gender and Development in a Global/Local World. Routledge. London and New York, (2004). IMPACT ASSESSMENT DESIGN FOR ECONOMIC GROWTH AND AGRICULTURE 70 Rana, Shahbaz. Short of expectations: Unemployment surges as growth falls short of Expections. Tribune (April 27, 2012). Accessed at: http://tribune.com.pk/story/370522/short-of-expectations-unemployment￾surges-as-growth-falls-short-of-target/. Saigol, R. “Women’s Empowerment in Pakistan: A Scoping Study.” USAID, (2011). Sathar, Z.A. and Kazi S. “Women’s Autonomy in the Context of Rural Pakistan”, Pakistan Development Review, 39:2, (2000). Sen, G. and Batliwala, S. “Empowering women for reproductive rights” in Pressure, H.B. and Sen, G. (eds.). Women’s Empowerment and Demographic Processes. Oxford University Press, New York, (2000). SME Development in Pakistan: Issues and Remedies; SME Policy of Pakistan (2005) http://www.gcu.edu.pk/publications/vc-sme.pdf Sraboni, E., Quisumbing, A. and Ahmed, A., “The Women’s Empowerment in Agriculture Index: Results from the 2011-2012 Bangladesh Integrated Household Survey”, International Food Policy Research Institute, (2003). Sustainable Development Policy Institute, “Food Insecurity in Pakistan”, (2009). Accessed at: http://vam.wfp.org.pk/Publication/food%20insecurity%20in%20Pakistan%202009%20- %202%20Aug%202010.pdf USAID Pakistan. Mission Strategic Framework – Pakistan, Fiscal Year 2013-2017, Background and Narrative (2013). USAID, “Economic Growth Strategy” for Pakistan, Draft Working Paper, February 19, 2011