REPORT ON EVALUATION OF THE ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International. APRIL 2008 REPORT ON EVALUATION OF THE ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under Cooperative Agreement 391-A-00-03-01011-00 REPORT ON EVALUATION OF THE ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The evaluation team is grateful for the support provided b y USAID/Pak istan in con ducting this evaluation. The perceptive comments and in put provided by Amy Meyer, USAID/Pakistan’s Director for the Economic Growth Office, were invaluable during the course of our work. Also, the team appreciates the informati on which MEDA and E CDI shar ed with us and the ti me they devoted to assi sting our evaluation. The ECDI Director and staff took tim e to accompany the evaluation tea m to the field for an informative day trip to visit so me of the project beneficiaries and see first hand what the pro ject is doing for them. The PEGED staff is also grateful to Shunila Da vid and Suhail Sattar of MSI/Pakistan for their good￾natured logistical support. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT i CONTENTS ACKNOWLEDGEMENTS ...........................................................................................................I PROJECT SUMMARY...............................................................................................................IV ACRONYMS................................................................................................................................. V EXECUTIVE SUMMARY .........................................................................................................VI SUMMARY EVALUATION REPORT..................................................................................VIII ANNEX 1: FULL DRAFT EVALUATION REPORT ON THE ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT ............................................................................... 1 I. INTRODUCTION ....................................................................................... 2 Background................................................................................................. 2 II. THE DEVELOPMENT PROBLEM ......................................................... 3 A. USAID’s Intervention in Response..................................................... 4 B. Purpose of the Evaluation ................................................................... 4 III. EVALUATION METHODOLOGY .......................................................... 7 IV. FINDINGS AND CONCLUSIONS ON THE OVERARCHING QUESTIONS................................................................ 9 A. Relevance ............................................................................................ 9 B. Effectiveness..................................................................................... 10 C. Impact ............................................................................................... 15 D. Efficiency.......................................................................................... 19 E. Sustainability..................................................................................... 22 F. Replication ........................................................................................ 24 G. Gender............................................................................................... 25 H. Reporting........................................................................................... 26 I. Outreach and Communications......................................................... 26 J. Coordination...................................................................................... 27 V. RECOMMENDATIONS .......................................................................... 27 VI. LESSONS LEARNED............................................................................... 29 VII. NEXT STEPS............................................................................................. 30 ANNEX 2: GETTING TO ANSWERS MATRIX .................................................................... 31 ANNEX 3: LIST OF REFERENCES......................................................................................... 41 ANNEX 4: CONTACTS.............................................................................................................. 43 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT ii ANNEX 5: MEDA/ECDI PROJECT STRUCTURE ............................................................... 45 ANNEX 6: KEY STAKEHOLDERS ......................................................................................... 46 ANNEX 7: PRELIMINARY MAP OF THE EMBROIDERED GARMENT SUB￾SECTOR ................................................................................................................. 47 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT iii PROJECT SUMMARY Project Name From Behind the Veil: Access to Con temporary Markets for Homebound Women Embroiderers in Pakistan Life of the Project (LOP) November 2004 – November 2007 Project Location Quetta, Balochistan; Thatta, Hy derabad & Karachi, Sindh; and Multan, Punjab Implementing Partners Recipient: Mennonite Economic Development Association (MEDA) Sub-Awardee: Enterprise and Career and Development Institute (ECDI) Activity Number REE-A-00-04-00053-00 Type of contract: Cooperative Agreement Budget USAID Share: US$ 506,362 MEDA Share: US$ 127,033 Total: US$ 633,395 FIGURE 1: MAP OF PAKISTAN SHOWING PROJECT AREAS (*) USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT iv ACRONYMS ANE Bureau Asia Near-East Bureau BDS Business Development Services BH Buy ing House CIDA Canadian International Development Agency CSA Co mmunity Sales Agent DE Designers ECDI Enterprise and Career Development Institute EGAT/PR/MD USAID Bureau Economic Growth, Agriculture, and Trade Office of Poverty Reduction/Micro-Enterprise Development Team FS Financial Services GEMINI Growth and Equity through Microenterprise Investments HIV Hu man Immunodeficiency Virus IDPs Internally Displaced Person IGP Implementation Grant Program IS Input Suppliers JOBS Job Opportunities and Business Support LSA Local Sales Agent M&E Monitoring & Evaluation Officer MBP Micro-enterprise Best Practices MC Marketing Coordinator MD Microenterprise Development Team ME Microenterprise MEDA Mennonite Economic Development Association MIP Microenterprise Innovation Project MSE Micro and Small Enterprises MSME Micro, Small and Medium Enterprises NAPT North American Project Team NGOs Non-Governmental Organizations PC Project Coordinator PPM Pakistan Project Manager PVOs Private Voluntary Organizations RE Rural Embroiderer RF Rural Facilitator SA Sales Agent SEEP Small Enterprise Education Promotion SMEDA Small and Medium Enterprise Development Agency STC Short-term Consultants TD Tracer Designer UAI Usage, Attitude, Image market assessment tool UF Urban Facilitator UGM Urban Garment Maker USAID United States Agency for International Development USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT v EXECUTIVE SUMMARY The CIA World Factbook reports that 24% of Pakistan’s population lived below the country’s established poverty line in fiscal year 2005/2006.1 The World Bank adds that 35.9% of rural Pakistanis belong to that portion of th e populatio n under the p overty line. 2 Women, in particular, make up a large share of Pakistan’s ru ral poor. T he UN reported that the adult economic activity rates for men and women in Pakistan in 2006 were 83 and 33, respectively, and that the percentage of women in Pakistan in the adult labor force was 27%. 3 A USAID-fu nded m arket stud y i n Pa kistan determined that while there are high value markets for e mbroidery which poor, homebound, rural women were capable of creating, these wo men were not rea ching the markets. Monopolistic bu ying cha nnels, lack of inform ation on market demand for patterns, male control over purchasing inputs, few female sales agents, and lack of garment makers’ access to lucrativ e customer bases curtailed these talented women’s salaries. In 2004, MEDA and ECDI stepped in to fix these probl ems through their “Behind the Veil” project that trained women sales agents to provide the market linkages the embroiderers needed to raise their incomes and their social status. From 2004 to 2007, Behind the Veil worked in Multan in Punjab Pro vince; Quetta in Balochistan Province; and Karachi, Hyderabad, and Thatta in Sindh Pro vince to help disadvantaged women engage in productive economic and social activity. What did the project accomplish? As su mmarized in the table, Behind the Veil raised the incomes of more than 9,700 women rural em broiderers and sales agents. The proj ect brought significant social changes to beneficiaries’ lives. Wom en are m ore mobile, they are better respected a t home, and they are improving the well being of thei r fam ilies th rough decisi on m aking, purchasing household pro ducts, and sending their children to school. Behind the Veil created a self-sustaining s ystem of m arket linkages thr ough directly inv olving beneficiaries as agents of change, planni ng for sustainability before the project e ven began, and adapting to environmental challenges and opportunities. The key conclusions and recommendations resulting from this evaluation are as follows: 1 CIA World Factbook 2008 2 As reported by the Rural Poverty Portal http://www.ruralpovertyportal.org/english/regions/asia/pak/index.htm 3 UN Statistics Division http://unstats.un.org/unsd/demographic/products/indwm/tab5a.htm Summary Measures at Project Completion Measure Value Benefit cost ratio 4.8 Number of direct beneficiaries 9,783 Average annual benefit/beneficiary $81 Em broiderers $2,745 Sales agents Benefit as % of average annual income 5% Em broiderers 174% Sales agents USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT vi CONCLUSIONS Relevance: The project was highly focused on the needs of its beneficiaries and took them into account at every step. Effectiveness: Behind the Veil accomplished its objectives and goal, and exceeded most of its targets. Impact: The project has produced quantifiable monetary benefits for 9,800 rural Pakistani women. Based upon available data and assumptions about the level of pre-project income, we conclude that MEDA has contributed to increasing the income of all of its targeted beneficiaries, especially the women sales agents. Efficiency: The project generated benefits in excess of costs (4.8 times costs). Sustainability: As long as demand for the rural embroiderers’ (REs) work is su stained (no guarantees in the free market) and there are no unforeseen supply li ne disruptions, benef iciaries should continue to enjoy what the project has established for them and modify it to suit their own ambitions and capacities. Replication: The Behind the Veil model is replicable and could be replicated both in Pakistan and in other countries, within and without the em broidery sector. It is already being replicated with fundi ng from the Canadian International Development Agency (CIDA). Gender: Wo men were the targeted b eneficiaries of Be hind the Veil and they have realized significant monetary and social bene fits from their participation in the model (see Effectiveness and I mpact). The project has primarily benefited women by increasing their incomes, mobility, skills, self-confidence, and family respect. Reporting: Reporting has been timely, useful, and informative. Outreach and Communications: The project’s co mmunications have been adequate and pro bably appropriate, considering the beneficiaries’ situations. Media interactions were minimal in order to protect the privacy of beneficiaries, but the project exhibite d its pro ducts both nationally and internationally, a form of publicity. Coordination: The project coordinated effectively internally, with USAID, and with the Government of Pakistan. It did not interact significantly with other projects or NGOs. RECOMMENDATIONS ™ The value chain approach, especially in addressing the needs of p oor women, has great potential to em power women. Futu re projects should follow the Behind the Veil model of inclu ding beneficiaries in project design and im plementation, and conducti ng thor ough market research prior to project design. Doing so increases the rele vance of a project to the needs of its inte nded beneficiaries. ™ Following Behind t he Veil’s successful exam ple, future projects should create a sustainability plan at the very beginning and foll ow it closely throug hout the life of the project, making modifications based upon the findings of project monitoring. ™ Gender-focused projects should develop a design th at works directly for women and shows a solid understanding of th eir needs. USAID should consider using dem onstrated knowledg e of women’s needs amongst the target population as a key criterion in proposal evaluation. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT vii SUMMARY EVALUATION REPORT4 INTRODUCTION Studies show that Pakistani women are economically and socially disadvantaged compared to Pakistani men, thereby depressing their ability to pursue inco me-generating activities. The CIA World Factboo k reports that 24% of Pakistan’s population lived below the country’s established poverty line in fiscal year 2005/2006.5 The World Bank adds that 35.9% of rural Pakistanis belong to that portion of the population under the poverty line.6 Women, in particular, make up a large share of the poor in Pakistan. Though it is difficult to measure exactly what percentage of Pakistan’s poor are women, several indicators suggest that poor female to male pri mary and secondar y enrollmen t rates, literacy rates, and em ployment rat es contribute to the gender inequality of the dist ribution of wealth. The World Bank’s Millennium Development Goals Portal indicates th at in 2005, the ratio of g irls to boy s i n primary and secondary education was 75.4, the ra tio of young literate females to m ales (ages 15-24) was 69.4 in 2005, and t he share of women e mployed in the non-agricultural sector was 8.6%. 7 Finall y, t he UN reported that the adult economic activity rates for men and women in Pakistan in 2006 were 83 and 33, respectively , and that the percentage of women in Pakistan in the adult labor force was 27%. 8 Against this backdrop, i n Jul y 20 03, USAID i ssued a Request for Applications (RFA) for the Microenterprise I mplementation Grant Program—Business Deve lopment Services (BDS), focusing on BDS market development, defined as, “the creation of well-functioning BDS markets as the best means of providing micro-, small and mediu m enterprises (M SMEs) with a wide array of useful, aff ordable and high-quality services with which to gr ow or stab ilize their businesses. ” In response, the Mennonite Economic Development Association (MEDA) and th e Enterpri se and Career Develop ment Institute (ECDI) submitted a competitive proposal to provide market access to homebound women embroiderers in Pakistan.9 Development Problem MEDA and ECDI had conducted a USAID-funded market assessment in Pakistan, which found a demand for high-end embroidered garments both in Pakistan and in neighboring cou ntries, such as Dubai. At the same time, rural women were creating quality embroidery that could be sold at premium prices, but were receiving little income from their work for the following reasons:10 a) Monopolistic buying channels: the bu yers set the price at whatever they wished, and som etimes did not pay the women for months. b) Male fam ily mem bers conduct m ost transactions, which m eant that i nformation on market demand for particular styles, patterns, and high quality inputs never reached t he e mbroiderers, thereby cutting them out of more lucrative markets. 4 The full evaluation report is included as Annex 1. 5 CIA World Factbook 2008 6 As reported by the Rural Poverty Portal http://www.ruralpovertyportal.org/english/regions/asia/pak/index.htm 7 http://devdata.worldbank.org/idg/idgprofile.asp?ccode=pak&cname=pakistan&selectedcountry=pak 8 UN Statistics Division http://unstats.un.org/unsd/demographic/products/indwm/tab5a.htm 9 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File. 10 Ibid. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT viii c) There were few female sales agents dealing directly with women embroiderers, and women who would potentially be inter ested in play ing this role did not have the contacts n eeded to esta blish commercial relationships. d) Garment makers were not reaching retailers, exporters, and exhibitions. USAID’s Intervention in Response MEDA/ECDI developed a program to respond to the market limitations driving down the incomes of the rural women em broiderers, with the purpose of r educing “the povert y of poor homebound wom en embroiderers in three conservative areas of rural Pakistan by helping them access higher value markets in the embroidered garment subsector.” To do so, MEDA/ECDI proposed establishing three BDS markets:11 1. Embedded s ervice pack age to hom ebound rural em broiderers consisting of market access, product development, and input supply; 2. Market acc ess s ervices for urban garment makers to hi gh val ue retailers, exhibitions, and exporters; 3. Product development and design services to mobile women sales agents, garment makers, retailers, and exporters. MEDA/ECDI’s model involved training women sales agents to provide the em bedded service package to the rural e mbroiderers and the market access services to urban m icro garment makers, making them the agents of change.12 The program, which was popularl y called “Behind th e Veil,” proposed work in five areas and three provinces: Multan in Punjab Province; Quetta in Ba lochistan Province; and Karachi, Hy derabad, and Thatta in Sindh Province. USAID funded the ME DA/ECDI program , formally called Access to Contem porary Markets for Homebound Women Embroiderers, in 2004. Purpose of the Evaluation In 2007, USAID/Pakistan asked the Pakistan Economic Growth Evaluation and Design (PEGED) Project to evaluate all eight of the projects that comprised the Economic Growth (EG) portfolio. Because projects were at differ ent stages, some evaluations were final and so me were mid-term. The evaluati on exercise had several purposes including: • Identifying opportunities for improving performance of ongoing projects, • Extracting lessons learned that can help USAID and the contractor improve performance of future interventions, and • Providing input to the design of the new EG portfolio. The Behind the Veil project closed in 2007, and PE GED therefore conducted a final evaluation focused on lessons learned for future projects. USAID asked that the evaluation address 14 specif ic questions, each with a number of su b-questions. These include six questio ns on overarching issues, f our on cross-cutting issues, and four project specifi c questions, which are included in Annex 2. 11 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File, pp. 1-3. 12 Ibid. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT ix Evaluation Methodology The PEGED evaluation took place from S eptember 25, 2007 through Febr uary 2008, with fieldwork conducted fr om Septem ber throug h November. The evaluation team used a com prehensive, m ixed￾methods evaluation design that was consistent (t hough not identical) across all eight EG e valuations.13 The design started with a Getting to Answers ( GTA) matrix, a planning to ol that helps one to determ ine for each evaluation question the data needed to answer it, possib le sources of the data, method for data collection, an d anal ysis techniques. Th e Behind the Veil GTA is included in Annex 2. Th e evaluation relied on document review; interviews with project staff, beneficiaries, and USAID; field visits to Karachi and Thatta; and compilation and analysis of project-provided quantitative data for our evaluation. The evaluati on tea m was asked not t o travel to most project sites (Thatt a and K arachi being the exceptions), which meant that we had to rely heavily on project reports for our data. Limitations therefore include: (a) not all of the information we would have liked was available; and (b) we did not have control over the collection methods or quality of the data contained in those reports. In addition, the evaluation team could not obtain complete financial statements for the project from either the im plementer or USAID 14, and while project reporting im proved during t he project, it was not of consistent quality . The evaluation the refore em ployed a conservative approach in deali ng with the benefits data. For the ten months for which we did not have data, we considered benefits to be zero rather than interpolating data to fi ll the gaps. T he greatest deficiency in the benefit calculation was the absenc e of baseline data on incomes, and we therefore relied on anecdotal information reported by the project. FINDINGS AND CONCLUSIONS Relevance How well was the project focused on the needs of the beneficiaries? The project f ocused intently on the needs of its be neficiaries and took them i nto account at every step. The project design was based upon rur al women embroiderers’ needs to access high qualit y markets for their products and receive a reasonable price, which MEDA/ECDI identified through a market assessment conducted prior to writing a response to the RFA that initiated funding for this project. The project design addressed those needs through the following: Beneficiaries’ Needs Project Design Access to market information Train w omen sales a gents (SAs) to prov ide em bedded services package to embroiderers Reduction of monopolistic buying practices Train SAs to provide m arket a ccess ser vices to urban garment makers Access to high-end and export markets Provide information on design services to SAs, garment makers, retailers, and exporters The project also adjusted its model to address the re alities of the environments in which the wo men live. For exam ple, it found t hat SAs could not reach or co mmunicate directly wit h hom ebound women in certain communities. To address this is sue, Behind th e Veil created two tiers of sales agents: local S As 13 See MSI’s Evaluation Methodology Report, September 2007. 14 We requested financial statements from MEDA and received two. We also repeatedly requested the records from USAID Washington and have not received a response. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT x with home boutiques based in urban areas and co mmunity SAs based in rural areas and w orking directly with the embroiderers.15 Effectiveness Has the project accomplished its objectives? Behind the Veil accomplished its objectives and goal, and exceeded most of its targets, with the exception of increased revenues for gar ment makers and desig ners. It increased the dollar value of SAs’ annual sales, linked 9,295 rural embroiderers (REs) with SAs, and increased the annual revenues of its REs. The project has also incr eased the quality of em broidery produced by homebound rural women and empowered t hem to become real businesswo men who design based on m arket demand. It has enabled some of these women to become sales agents and trained many more women sales agents to serve as the link between hom ebound em broiderers and high value markets. It has increased the income and confidence of these sales agents and given women a higher stat us within their households and in the market pla ce. The project has given wo men multiple outlets for selling their war es and a collectiv e bargaining vehicle through “joint ventures” to combat monopolistic buying practices. The project was not so successful with the micro garment makers and designers it targeted, indicating the model was not right for t hem. MEDA/ECDI explain in their last Sem i-Annual Report that they designed the program based upon the assumption that the garment makers produced at a similar level of quality and skill as the REs and therefore, MEDA/ECDI expected products fro m both groups to sell in the same markets. However, they discovered that the gar ment makers di d not have the skill level assu med and therefore they were confined to lower value markets.16 MEDA/ECDI had intended to work with formal designers, who generally have higher incomes, but found they were not a good fit for the SAs. Therefore, Behind the Veil turned to tracer designers, who command a lower price. The project also reported that some of the SAs began developing their own designs. Marketing Strategies Behind the Veil used exhibitions, bot h overseas and in Pakistan, to m arket the work of its REs. The quality of the work is a marketing tool in and of itself, and both wholesalers/retailers and SAs agreed that the quality of the embroidered products improved during the project. A further indicator of the project’s marketing success is the fact that a hom e-based boutique owner in Karachi reduces the price of MEDA-supported prod ucts sold overseas b ecause she more than compensates for it through the volume of products she sells.17 Impact The project has produced quantifiable m onetary bene fits for 9,800 rural Pakistani wo men. Based upo n available data and assumptions about the level of pre-project income, we conclude that MEDA has contributed to increasing the income of all of its targeted beneficiaries, especially the women sales agents. Table 3 summariz es curre nt project be nefits where benefits are the difference between pre-project and post-project income. The benefit meas ures assume that rural embroiderers and sales agents would hav e earned Rs. 360 per month without the project. Consistent with the information presented in Effectiveness, 15 Interviews with ECDI staff, ECDI presentation on the project, project documents 16 MEDA Semi-Annual Report (January 1 to June 30, 2007), p. 6. 17 Interview with home-based boutique owner in Karachi. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xi the sales agents have realized the most growth in their monthly incomes, whereas urban garment makers and designers have benefited the least in monetary terms. TABLE 1: CURRENT PROJECT BENEFITS Benefits Beneficiary Number Total nominal benefits ($ thousands) Average annual per beneficiary ($)a As percent of average annual household incomee Rural embroiderers 9,330 $1,930 $81b 5.2% Rural embroiderers 6,150 $1,930 $167c 10.5% Sales agents 174 $969 $2,745c 174.0% Urban garment makers 200 $60 n.a.d -- Designers 40 $47 n.a. d -- Totals 9,783 $3,006 a. Calculated by dividing benefits by participant year, not the number of current beneficiaries. b. Reflects average income over all embroiderers. c. Reflects average income in a month that an embroiderer or sales agent is active. d. Data are not adequate to calculate average annual benefits. e. Based o n $ 1,584 a verage an nual h ousehold i ncome i n rural areas from 200 5-05 H ousehold I ntegrated Economic Survey (HIES). Note: The $229 implied average monthly income for sales agents is somewhat misleading because some very high sales ag ent profits (the m aximum is $ 3,047 per m onth) skew th e d istribution. Th e median of $178 per month better represents benefits for a typical sales agent. The project ended in Decem ber, 2007. But sales agents and rural em broiderers are not likely to sto p making and marketing em broidery without t he project . To acco unt m ore fully f or project benefits the evaluation also considered the benefits likely to accr ue in the next five y ears – through 2012. Assuming an annual growth rate of 10%, Figure 7 illustrates the calculation of potential fut ure benefits for SAs and REs. The present value of the area between the two lines represents the benefit of the project in terms of increased beneficiary incomes. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xii FIGURE 2: TOTAL PROJECT BENEFITS (NOMINAL $, 10% ANNUAL GROWTH) $0 $100,000 $200,000 $300,000 $400,000 Jan-05 Jan-06 Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Aggregate incomes ($ Aggregate incomes with project Aggregate incomes without project PV of benefits = $13.2 million The project has also brought significant social chang es to beneficiaries’ lives. Women are more mobile, they are better respected at home, and they are improving the well being of their families through decision making, purchasing household products and sending their children to school. Efficiency The project has been effi cient based u pon available da ta. It gen erated benefi ts in excess of costs (4.8 times costs). MEDA operated on a sm all budget using predominantly local labor, thereby minimizing its costs, while it maximized benefits through building local capacity and skills. Sustainability MEDA/ECDI created a sel f-sustaining BDS market with a little facilitation and capacity building. They designed an exit strategy from the beginning and followed it succ essfully. Beneficiaries have embraced the model and are discovering the i mportance of their roles and mobility within it. By training SAs to provide the market linkages needed to bring dem and information to REs an d REs’ em broidery to th e market, MEDA/ECDI can step back and let their BDS markets run. As long as dem and for the REs’ work is sustained (no guarantees in the free market) and there are no unforeseen supply line disruptions, beneficiaries should continue to enjoy what the project has established for them and m odify it to suit their own ambitions and capacities. MEDA/E CDI even transferred their own facilitation role to six Buying Houses (BHs), which provide quality control, marketing, and generally link large numbers of REs with customers. Entreprene urs’ investments established the BHs, and som e of the project-supported SAs have established their own BHs. It is still an open question of whether REs’ highe r inco mes will remain protected fro m monopolistic practices, but given that they have m ultiple buyers, some collective bargaining power through form ing joint ventures, and ongoing monitoring from ECDI, chances are high that they will be protected. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xiii Replication The Behind the Veil model could be replicated both in Pakistan and in other countries, within and without the e mbroidery sector. MEDA/ECDI are alre ady rep licating the model in other sectors in the four provinces of Pakistan with fundi ng from the Cana dian International Development Agency, and MEDA had conducted a similar project in Afghanistan prior to Behind the Veil. The essential ingredients needed for replicating the Behind the Veil model include:18 ¾ Market research-based program design; ¾ Commitment to reaching marginalized populations through sustainable market systems; ¾ Skills to conduct training and capacity building for pro-poor value chai n development and to manage such a complex program. ¾ Funding that matches the scale of the market. ¾ Market economy with sufficient supply and demand signals and demand for a product or service that the targeted beneficiaries can make and deliver. Gender Women were the targeted beneficiaries of Behind the Veil and, as discussed in effectiveness and impact, they have re alized significant monetary and social benefits from their partici pation i n the m odel. The project has primarily benefited women by increasing their incomes, mobility, skills, self-confidence, and family respect. Their incomes grew an d they discovered capabilities and skills the y did not know they had. The proj ect’s extensive preparations in the design phase and m onitoring throughout ensured that it appropriately targeted and benefited women. 19 Former e mbroiderers beca me sal es age nts and travel ed, some for the first time, outside of their own villages to places like Karachi. Some women opened boutiques, shops, and buy ing houses. They learned to work with other business people an d with each other. MEDA/ECDI obse rved that CSAs and LSA s became “women’s advocates, community mobilizers, and social entrepreneurs.”20 Through training sessions, work groups, participati on in exhibitions, joint ve ntures, and AWESOME, women have found more freedom to meet in groups. Other Issues Project Reporting Reporting ha s been tim ely, useful, an d inform ative. MEDA/ECDI produced a series of sem i-annual progress reports that wer e submitted in a ti mely manner, alway s 6 weeks after the close of a 6- month period. Quarterly reports were also submitted, and USAID/Pakistan was pleased with the reporting. Not all of the quantitative data was colle cted at the outset, however. Income data collected at th e beginning was limited, but expanded over time. Communications/Media Coverage The project’s co mmunications have been adeq uate and probabl y appr opriate, considering its beneficiaries’ situations. Media interactions were minimal in order to protect the privacy of beneficiaries, but the proj ect exhibited its products both nation ally and inte rnationally, a form of publicit y. The 18 SEEP Case Study #6, pp. 54-55. 19 See Relevance. 20 MEDA Impact Analysis, p. 5. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xiv implementers also participated in networks such as the SEEP Network and published articles about their work. Coordination The project coordinated ef fectively internally, with USAID, and with the Gov ernment of Pakistan. It did not interact significantly with other projects. RECOMMENDATIONS The following reco mmendations identify Behind the Veil’s success factors and suggest their application to future projects. Relevance The value chain approach, especially in addressing the needs of poor women, has great potential and can and should be replicated, by USAID/Pakistan and its partners. However, expe rience shows that not all value chain projects create a stable market that survives the project’s end. As the Behind th e Veil project shows, projects can be made most relevant to ben eficiaries by including the m in project design and implementation, and by conducting thorough market research prior to project design. Future projects that focus on individual (rather than systemic) benefits should adopt this model. Effectiveness Projects designed b y US AID/Pakistan and its partners can at the design stage identify possible weaknesses or issues that can arise in the future, and ensure their m itigation. In sim ilar value chain models for poor wom en, steps should b e taken, such as the establish ment of producers’ coo peratives, so that the possibility of monopolistic pricing by buyers is diminished. Impact At the design stage, USAID/Pakistan and its partners should ensure that the value chain process for th e particular product being considered is workable, tried and tested, so that project impact is maximized. The preparation and dissemination of case studies through the life of the project would be effective in showing impact. Efficiency Future projects of a si milar nature should maximize cost benefit ratios b y using local labor and building local capa city to conduct the project’ s activities. T he indicators for ensuring that cost b enefit analy sis takes place prior to and during the project’s implementation should be identified in the project design, and their collection and analysis should be a part of the contractual agreement with the implementers. Sustainability Future projects should create a sustainabilit y plan at the very beginning and follow it closely throughout the life of the project, making modifications base d upon ex perience. The project design team must consider issues of eventual sustainability of the project, and make the needed adjustments/interventions a part of the indicators to be monitored through the life of project. Project designers and im plementers should consider providing microfinance facilities to assi st women at the sales agent and buying house level to respond t o more orders, participate in more exhibitions, and increase sales by accessing loans for equipment, input supplies, and travel, thus enhancing sustainability. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xv Gender Gender-focused projects should develop a design that works directly for women and sh ows a solid understanding of their needs. USAID should co nsider using demonstrated kno wledge of women’s needs amongst the target population as a key criterion in proposal evaluation. Outreach and Communications In projects where there is sensitivity about photographs and other media coverage, the project implementers can use other forums, such as the SEE P Network, to publicize results. Efforts should be made, by USAID and th e im plementers, to present pr oject results at different forums, and generate discussion and interest in developm ent groups. In future, similar projects, funding should be adequate to ensure that the basic requirements are met. Coordination The project im plementers, using USAID/Pakistan as sistance when needed, should coordinate fully with civil society, donors and the government, and keep them informed through meetings and newsletters, thus enhancing a common understanding and ownership of best practices. LESSONS LEARNED ƒ Projects can benefit by precisely defining what economic and social empowerment of women entails, and developing indicators to track progress. ƒ Research and market assessments help to design value chain development. ƒ Beneficiary involvement is critical for successful pr oject design and implementation, particularly for micro-level projects. ƒ Development of wo men’s as sociations in th e val ue chain process can p rotect wo men from monopolistic buying practices. ƒ Using predominantly loc al labor reduces project costs and maximizes bene fits by building local capacity. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xvi NEXT STEPS This evaluation pro duced a num ber of specific re commendations for enhancing future p rojects. The evaluation will be useful onl y if USAID and its project partners learn fro m t he reco mmendations and implement th em. This cha pter sets fort h procedur es for evaluating the reco mmendations and deciding how to use them to best advantage. A possible course of action for methodically processing the evaluation results is: Activity Timeframe Responsible person/office Assign a person to review the reco mmendations of all eight evaluations and separate the reco mmendations into: (1) those that need to be handled internally within USAID, (2 ) those that need to be handled internally within E G, and (3) those that are project specific. Immediately USAID EG Recommendations internal to USAID Convene a meeting within USAID to review the recommendations that need to be ha ndled i nternally within USAID. Use the meeting to: • Decide which recommendations to address and which to ignore. • Discuss how to address the reco mmendations dee med important. • Identify an individual or office responsible for implementing each recommendation. • Establish a timeframe for implementation. • Determine a proces s for tracking progress on implementation of each recommendation. Third priority after initial meeting USAID Reconvene every six m onths (in whatever groups are appropriate) to review progress on implementation. Six month intervals USAID Recommendations specific to EG Convene a meeting within EG to review the reco mmendations that need to be handled with in EG. Follow the procedures outlines above. Second priority after initial meeting EG Reconvene every six m onths (in whatever groups are appropriate) to review progress on implementation. Six month intervals EG USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT xvii ANNEX 1: FULL DRAFT EVALUATION REPORT ON THE ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 1 I. INTRODUCTION Background The CIA World Factbook reports that 24% of Pakistan’s population lived below the country’s established poverty line in fiscal y ear 2005/2006.21 The World B ank adds that 35.9% of r ural Pakistanis belong t o that portion of the po pulation under the poverty line.22 Other measures of pove rty in Pakista n over th e past decades indicate that the percent of the Pa kistani population living under US$2 a day was 65.6 million, with 13.4 million of those living under US $1 a day.23 Women, in particular, make up a large share of th e poor in Paki stan. Though it is difficult to measure exactly what percentage of Pakistan’s poor are wome n, several indicators suggest that poor fem ale to male primary and secondary enr ollment rates, literacy rates, and em ployment rates contribute to the gender inequality of the distribution of wealth. The World Bank’s Millennium Development Goals Portal indicates that in 2005, the ratio of girls to boys in primary and secondary education was 75.4, the ratio of young literate females to males (ages 15-24) was 69.4 in 2005, and the sh are of women employed in the non-agricultural sector was 8.6% .24 Finally, the UN reported that the adult econom ic activity rates fo r men and wo men in Pakistan in 2006 were 83 and 33, r espectively, and that the percentage of women in Pakistan in the adult labor force was 27%. 25 Founded i n 1994, USAID’ s Microenterprise Devel opment (M D) office focu ses on providing incom e￾generation opportunities to the poor worldwide.26 Its interventions include business development services (BDS), financial service s (m icrofinance), enterpris e developm ent services, and enabling environm ent resources.27 In July 2003, USAID issu ed a Re quest for Applications (RFA) for the Microenterprise I mplementation Grant Program—Business Development Services, focusing on BDS market development, defined as, “the creation of well-functioning BDS markets as the best means o f providi ng micro-, small and medium enterprises (MSMEs) with a wide array of useful, affordable and high-quality services with which to grow or stabilize their businesses.” Unlike direct service provision, the impact of which may fade after funding ends, BDS market development strives to create sustainable impact by creating private sector markets for the products MSMEs produce.28 The goal of the IGP was, “to test and docum ent new and innovati ve approaches to BDS market development that can achieve sustainable results.” 29 In response, the Mennonite E conomic Development Association (MEDA) and the Enterprise and Ca reer Develo pment Institute (ECDI) submitted a competitive proposal to provide market access to homebound women embroiderers in Pakistan.30 21 CIA World Factbook 2008 22 As reported by the Rural Poverty Portal http://www.ruralpovertyportal.org/english/regions/asia/pak/index.htm 23 Ibid. 24 http://devdata.worldbank.org/idg/idgprofile.asp?ccode=pak&cname=pakistan&selectedcountry=pak 25 UN Statistics Division http://unstats.un.org/unsd/demographic/products/indwm/tab5a.htm 26 USA ID R equest for Applications (R FA) Num ber M /OP/EGAT/EMD03-1610, M icroenterprise Im plementation Grant Program-Business Development Service, released July 3, 2003. 27 www.microlinks.org 28 USAID RFA 29 Ibid 30 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 2 II. THE DEVELOPMENT PROBLEM MEDA and ECDI had co nducted a USAID/MD-fu nded ma rket assessment i n Pakistan, which found a demand for high-end embroidered garments both in Pakistan and in neighboring countries, such as Dubai. At the same time, rural women were creating quality embroidery that could be sold at premium prices, but were receiving little income from their work because:31 a) Monopolistic buying channels: the bu yers set the price at whatever they wished, and som etimes did not pay the women for months. b) Male fam ily mem bers conduct m ost transactions, which m eant that i nformation on market demand for particular styles, patterns, and high quality inputs never reached t he e mbroiderers, thereby cutting them out of more lucrative markets. c) There were few female sales agents dealing directly with women embroiderers, and women who would potentially be inter ested in play ing this role did not have the contacts n eeded to esta blish commercial relationships. d) Garment makers were not reaching retailers, exporters, and exhibitions. These conditions provi ded an opport unity for a creative model to link homebound r ural wo men embroiderers with high end domestic and export markets. FIGURE 3: TYPICAL PRICES IN THE HOME EMBROIDERY VALUE CHAIN Source: MEDA/ECDI Pakistan Presentation, October 14, 2006, based upon data collected during their market assessment on which their proposal for this project was based. 31 Ibid. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 3 A. USAID’s Intervention in Response MEDA/ECDI developed a program to respond to the market limitations driving down the incomes of the rural women em broiderers, with the purpose of r educing “the povert y of poor homebound wom en embroiderers in three conservative areas of rural Pakistan by helping them access higher value markets in the embroidered garment subsector.” To do so, MEDA/ECDI proposed establishing three BDS markets:32 4. Embedded s ervice pack age to hom ebound rural em broiderers consisting of market access, product development, and input supply; 5. Market acc ess s ervices for urban garment makers to hi gh val ue retailers, exhibitions, and exporters; 6. Product dev elopment an d design services to m obile women sales agents, garm ent mak ers, retailers, and exporters. MEDA/ECDI’s model involved training women sales agents to provide the em bedded service package to the rural e mbroiderers and the market access services to urban m icro garment makers, making them the agents of change.33 The program, which was popularly called “From Behind the Veil,” proposed work in five areas and three provinces: Multan in Punjab Province; Quetta in Ba lochistan Province; and Karachi, Hy derabad, and Thatta in Sindh Province. USAID funded the MEDA program , formally called Access to Contem porary Markets for Homebound Women Embroiderers, in 2004. USAID provided j ust over $500,000, while MEDA/ECDI kicked in an additional $127,000 for three years. B. Purpose of the Evaluation In 2007, USAID/Pakistan asked the Pakistan Economic Growth Evaluation and Design (PEGED) Project to evaluate all eight of the projects that comprised the Economic Growth (EG) portfolio. Because projects were at differ ent stages, some evaluations were final and so me were mid-term. The evaluati on exercise had several purposes including: • Identifying opportunities for improving performance of ongoing projects, • Extracting lessons learned that can help USAID and the contractor improve performance of future interventions, and • Providing input to the design of the new EG portfolio. The Behind the Veil project closed 200 7, and PEGED therefore conducted a fi nal evaluation focused on lessons learned for future projects. USAID asked that the evaluations address ten spe cific questions, each with a number of su b-questions. These include six questio ns on overarching issues, f our on cross-cutting issues, and four project specifi c questions as described below. 32 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File, pp. 1-3. 33 Ibid. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 4 Overarching Issues 1. Relevance: How well was the project focused on the needs of the beneficiaries? a. Was the project well designed to address the needs of the beneficiaries? b. How well was the project adjusted to address the needs of the beneficiaries? c. To what extent did the design of the activity utilize participatory techniques? d. Was the activity designed to meet a felt need of a specific community, target audience, or influential stakeholder? e. Were stakeholders involved in a substantive way throughout the project life cycle? f. Was the targeting appropriate in hindsight? 2. Effectiveness: Has the project accomplished its objectives? a. How were the initial targets established for each activity? b. Were the targets realistic and appropriate? c. To what extent were the targets achieved? d. What are the lessons learned for setting targets in future activities in accordance with the requirements of USAID’s Performance Monitoring Plan (PMP)? 3. Impact: To what extent has the project benefited the people of Pakistan? a. How has the program benefited the intended beneficiaries? b. What were the primary and secondary positive and negative impacts of the projects? c. How large have the impacts been or are likely to be? d. To what extent can the impacts be attributable to the project? e. How were the impacts distributed by region, sector and gender of the beneficiaries? f. Were any of these benefits or losses unexpected? 4. Efficiency: How efficient has the project been in utilizing its resources to achieve results? a. To the extent possible, what is the internal rate of return for this project, as calculated in a cost benefit analysis? b. How cost-effective has the project been? c. How do overhead and administrative costs for this activity com pare to oth ers across differing t ypes of implementation mechanisms (e.g. Contract, Lim ited Scope Grant Agreement, Grant, Cooperative Agreement) a nd for the differen t types of implem enting entities (e.g. local vs. international firms, non-profits vs. for-profits, etc)? 5. Sustainability: Are the activities and results likely to be sustained after the project is completed? a. Were the act ivities designed in a manner which fo cuses on their sustainabil ity after project completion? b. Were the a ctivities implemented in a manner which focuses on their sustainabi lity after project completion? c. Was the initial timeframe for the activity realistic to achieve sustainable results? d. Were any of the activitie s fundam entally designed and im plemented in a way which creates donor dependence? e. Is it reasonable to expect the project to ach ieve sustainability i n the project life given internal and external factors? 6. Replication: To what extent can the activities and results of the project be replicated? a. Were the activities designed in a manner which focuses on their replication? b. Were the activities implemented in a manner which focuses on their replication? c. Can the activities be replicated in other areas with similar socio-economic features? d. Can the activities be replicated in dissimilar areas? e. To what quantified extent can the project be replicated? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 5 Cross-Cutting Issues 7. Gender: To what extent has the project benefited women? a. To what extent has the project included women in its staff, partners, agents, etc.? b. To what extent has the project systemically targeted women in its activities? c. To what extent have project resources been used to benefit women? d. How effective has the project been in reaching women? e. What are the direct quantified benefits of the project for women? 8. Reporting: Have the prime contractors and grantees reported on time and in a useful manner? a. Have the partners fulfilled all of their reporting requirements? b. Were the reports useful to USAID staff? c. Were all branding guidelines followed? d. Were the reported results accurate and verifiable? e. How can the reporting requirements and formats be improved? 9. Public Relations/Media Coverage: How effective has the project been in getting its story out? a. Have the project’s work plans contained public relations and media activities? b. Was the branding strategy clear? c. Has the project highlighted success stories? d. How active h as the project been in public rela tions efforts in terms of events/activities – frequency, nature, profile, content and design, branding and participation? e. To what extent have they raised awareness of the activity among intended beneficiaries? f. To what extent has the project followed branding guidance? g. How can the im pact of the public relations com ponent of future progra mming b e improved? 10. Coordination: How effectively has the project coordinated with other parties? a. How effectively has the project coordinated with the Government of Pakistan? b. How effectively has the project coordinated with other USG projects? c. How effectively has the project coordinated with other donors? d. How effectively has the project coordinated with other stakeholders? e. To what extent were sy nergies developed between the project and other individ ual USAID EG activities, other donor programs, and/or GOP initiatives? f. What concrete steps should be taken to improve coordination and maximize synergies in future activities? Behind the Veil Specific Questions • What has been the quantitative impact on the beneficiaries’ incomes? • Are the project results sustainable without additional support? • Can this activity for embroiderers be replicated on a larger scale across Pakistan? • Can this activity be replicated in other areas beyond embroidery? • What successful strategies have been adopted for marketing these women’s products? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 6 III. EVALUATION METHODOLOGY The PEGED evaluation took place from S eptember 25, 2007 through Febr uary 2008, with fieldwork conducted from September through November. The following individuals formed the evaluation team: • Team Leader, Sarah Tirmazi: Created evaluation design, arrange d and condu cted interviews in Islamabad, Karachi, and Thatta, drafted evaluation report. • Quantitative Analysis Expert, Doug Krieger: Collected and analyzed available quantitative impact data; computed benefits, costs, and net present values. • Competitiveness Expert, Michael Dan: Participated in interviews in Islamabad and Karachi • Report Writer, Katharine Wheatley: Wrote the final evaluation report The evaluation team used a comprehensive, mixed-methods evaluation design that was consistent (though not identical) across all ei ght EG evaluations. 34 The design started with a Getting to Answers (GTA) matrix, a planning tool that helps one to determine for each evaluation question the data needed to answer it, possible sources of the data, method for data collec tion, and analysis techniques. The Beh ind the Veil GTA is included in Annex 2. The evaluation re lied on bot h qualitative and quantitative methods as follows: • Document re view35: The evaluation team reviewed an array of documents on the project, including the original RFA, MEDA/E CDI’s respon se, i mplementation plans, quarterly reports, market assessm ent, and i nternal im pact assessmen t, am ong others. We also consulted re ports produced by outside sources, such a s r elevant reports fro m the S mall Enterpri se Education and Promotion (SEEP) Network. • Interviews 36: The team conducted se mi-structured, pr edominantly group i nterviews with 11 MEDA st aff, 13 Sales Agents, 1 home-based b outique owner, and more than 30 rural embroiderers during Sept ember and October 2007. We used the GTA as a refer ence f or the interviews. We also met with USAI D/Pakistan st aff, including the Direct or of the Office of Economic Growth and our Cognizant Technical Officer. • Field Visits to Karachi an d Thatta: The team interviewed ECDI project staff and sales age nts in Karachi; sales agents, rural em broiderers, and facilitators in Thatta. We also visited a hom e-based boutique and two stores selling Behind the Veil products in Karachi, as well as an output shop in downtown Thatta. We attended an exhibition of Behind the Veil -supported products in Karachi, during which we spoke with sales agents and inspected the quality of the products displayed. • Compilation and Analysis of Qu antitative Data: MEDA/ECDI collected information on their project’s outputs and outcomes, which they provided to the evaluators. They include: ¾ Numbers of active women sales agents; ¾ Number of r ural e mbroiderers represented by each sale s agent on a month by m onth basis; ¾ Volume of embroidered units acquired from rural embroiderers per month; ¾ Amount paid to embroiderers for each transaction; ¾ Number of urban garment makers represented or sub-contracted by each sales a gent on a month-by-month basis; ¾ Volume of embroidered units stitched into outfits by garment makers per month; ¾ Amount paid to garment makers for each transaction; ¾ Amount received by sales agents fro m retail ers, exporters and exhibits, f or each transaction; 34 See MSI’s Evaluation Methodology Report, September 2007. 35 A list of references consulted is in Annex 2. 36 A list of contacts is in Annex 3. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 7 ¾ Costs of materials on a monthly basis; ¾ Number of sales agents acquiring design services on a fee-for-ser vice or embedded basis per month; ¾ Amount paid for fee-for-service design work on a month-by-month basis; and ¾ Other expenditures of sales agents on a monthly basis. Data Limitations The team had planned to c onduct short surveys of project beneficiaries in Multan and Quetta, but project staff requested that we not do so. The y stated that, given (a) the culturally difficult situation of poor rural women; (b) too m any teams/individuals (including from USAID) had already visited; and (c) the project was coming to an end and the staff was busy with close-out, they could not support our site visits. We did have permission to visit Karachi (where the project office was) and Thatta, which is close to Kara chi, but we cannot say that the situations we observed there are representative of those in the other project areas. Given the resulting lack of primary data, the team had to rely heavily on project reports, which means that (a) not all of the information we would have liked was available; and (b) we did not have control over the collection methods or quality of data contained in those reports. Nonetheless, we believe from our inspection of MEDA’s reports and interviews with staff that the data in the reports were conducted methodically and with an acceptable level of quality control The evaluati on tea m cou ld not obtain co mplete fi nancial stat ements for th e project fro m either the implementer or from USAID. The onl y consistent sour ce of data on project costs obtained by the team was the budget realignment associated with the no cost extension. That budget revisi on reported cumulative p roject expenditures as of March 31, 2007 and forecasted expenditures fro m April 1, 2007 through Decem ber 31, 2007. Costs data thus cons ists of two data point s: March 31, 200 7 and December 31, 2007. Sales agents reported m onthly activity to project administrators. These re ports consisted of data on the number of rural embroiderers (participating and active in that month), payments to rural embroiderers, the number of urban garm ent makers engaged, payments to urban garm ent makers, the number of designers engaged, payments to desi gners, the num ber of pi eces ordered an d delivered, other expenses, and total sales. The se data provided the information nece ssary to estimate inco mes for sale s agents, rural embroiderers, urban garment makers, and designers. The sophistication of rep orting im proved as the pr oject matured. For example, early re ports did not contain information on expenses incurred by sales ag ents and thus may have overstated sales agent incomes.37 Also, the evaluation team was not able to obta in sales agents’ reports for 10 of the 29 months that project was active. E stimates of project benefits are thus based on spotty and inconsistent data. The evaluation em ployed a c onservative a pproach in dea ling with the benefit data. For the ten months for which data did not exist, it considered benefits to be zero rather than interpolating data to fill the gaps. The greatest deficiency in the benefit calculation w as the absence of baseline da ta on incomes. The sales agents’ repor ts provide reasonable data on i ncomes w ith the project but there is no firm esti mate of incomes prior to the project fro m whi ch to estimat e the i mpact of the project on incom es. Anecdotal information reported b y the project sug gests that women earned an average of Rs. 360 per month from embroidery before the project began. In the absenc e of a better estimate, impact estimates are based on this figure. 37 The reports did note sales agent profit which may have ac counted for expenses even though expenses were not explicitly reported. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 8 IV. FINDINGS AND CONCLUSIONS ON THE OVERARCHING QUESTIONS A. Relevance How well was the project focused on the needs of the beneficiaries? Conclusions: The project was highl y focused on the needs of its beneficiaries and took them into account at every step. The project design was based on rural women embroiderers’ needs to access hig h quality markets for their products and receive a reasonable price, which MEDA/ECDI identified through a market assessment prior to writing a r esponse to the RFA that initiated fundi ng for this pr oject. The project also adjusted its model to address the realities of the environments in which the women live. Beneficiaries and their Needs As identified in MEDA/ECDI’s response to USAID’s RFA, the intended beneficiaries of this project were “poor, homebound women embroiderers in three conservative areas of rural Pakistan.” MEDA/ECDI’s RFA response implicitly identified these women’s needs as: 1. Access to market information on the designs, patterns, and inputs for which premiums are paid; 2. Reduction of monopolistic buying practices that depress their incomes; and 3. Access to high-end and export markets. Project Design MEDA/ECDI designed the Behind the Veil project based on a market a ssessment of the needs of its targeted beneficiaries. To address that, it proposed a value chain approach to de velop three BDS markets with both demand- and supply-side interventions:38 1. Embedded p ackage to r ural women embroiderers: MEDA planned t o introd uce rural embroiderers to women sales agents and their services (demand); and recruit and train women t o serve as sales agents and provide the embedded services package (supply). 2. Market access services fo r urban garment makers to high-value urban markets: MEDA planned to introduce urban garment makers to sales agents and the m arket access services they provide (demand); identify and train mobile women to serve as sales agents between the garment makers, retailers, exporter, and exhibitio n organizers; and facilitating in formation sharing am ong market players (supply). 3. Product development and design services to mo bile women sales agents, garment makers, retailers, and exporters : MEDA planned to provide information to sales agen ts, small garmen t makers, and exporters on co mmercial design services and develop service delivery and payment models (demand); and su pport the flow of product design information through designers to sales agents, or thr ough garment makers, buyers, and i nput suppliers to sales agents; develop to ols to help sales agents purchase design services; and he lp designers de velop affordable service s f or microenterpreneurs (supply). Figure 4 summarizes MEDA’s design. 38 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File, pp. 5-7. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 9 FIGURE 4: THE MEDA VALUE CHAIN APPROACH Enabling Environment • Business environment • Socio￾economic context Producers Input Suppliers Wholesalers Retailers Support Products / Service Markets Source: M ary McVay, SEEP Net work’s Poverty Out reach Working Group, C ase St udy #6, “B ehind t he Vei l,” September 2007, p. 14. Adjusting to Beneficiaries’ Needs To solicit be neficiary input, the pr oject tasked its Sales Agen ts (SAs) wit h collecting quantitative, income-related data from rural embroiderers (REs) with whom they work on a monthly basis. In addition, the project monitorin g and evaluation ( M&E) officer conducted bi-monthly interviews with 120 REs t o measure changes in their quality of life since the project began working with them. The M&E officer also interviews a random sample of 60 sales agents (SAs).39 During the course of the project, ECDI found that t he SAs could not reach or communicate directly with homebound women in certain communities. To address this issue, From Behind the Veil created two tiers of sales agents: local SAs with hom e boutiques based in urban areas and co mmunity SAs based in rural areas and working directly with the embroiderers.40 B. Effectiveness Has the project accomplished its objectives? 39 Interviews with ECDI staff, Behind the Veil project documents 40 Interviews with ECDI staff, ECDI presentation on the project, project documents USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 10 Conclusions: Behind the Veil accomplished its objectives and goal, and exceeded most of its targets. It has increased the m arket value and quality of embroidery pr oduced by hom ebound rural women and empowered them to become re al businesswomen who design ba sed on marke t demand. It has enable d some of these women to become sales agents and trained many more women sales agents to serve as the link between homebound em broiderers and high value markets. It has increased the i ncome and confidence of these sal es agents and gi ven wo men a higher status within their households and in th e market place. The project has given women multiple outlets for selling their w ares and given them a collective bargaining vehicle. The project was not so s uccessful with the m icro garment makers and designers it targeted, indicating the model was not right for them. Objectives The project defined its goal as, “to economically empower poor homebound women embroiderers in three conservative areas of rural Pakistan, and urban mi cro garment makers, by h elping them access higher value markets in the embroidered garment subsector .” Although the project di d not specifically define intermediate results, its proposal for funding stated that it intended to achieve the following:41 1. Create co mpetitive, sustainable BDS markets th at support the developm ent of a high value productive chain based on the outputs of poor rural women. 2. Homebound women will become economically empowered. 3. “Monopolistic buying patters will be replaced with fair competition.” 4. Gender relations will be adjusted. 5. “All participants in the value chain will becom e more aware of the fi nal goods and se rvices markets and the demands of affluent consumers, realizing greater success and job satisfaction.” MEDA/ECDI’s proposal for funding included performance indicators and targets to measure its progress, and Table 1 below presents information on the quantitative targets met by June 2007. 41 MEDA: “FROM BEHIND THE VEIL: Access to Contemporary Markets for Homebound Women Embroiderers in Pakistan,” IGP Narrative File, pp. 1 and 9. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 11 TABLE 1: PERFORMANCE INDICATORS, TARGETS, AND RESULTS Performance Indicators Targeted Year 3 (June 30, 2007) Project-to-date Percent over/under Target Number of Sales Agents (SAs) participating in program 60 213 255% Number of Sales Agents actively receiving and selling orders N/A 174 … Number of Sales Agents purchasing design services 60 73 22% Annual sales of Sales Agents $450,000 $1,973,704 339% Profitability Ratio for Sales Agents 15.4% 22.8% Number of Rural Embroiders linked with Sales Agents 6,000 9,295 55% Number of embroiders receiving embedded services from sales agents (monthly average) N/A 6,544 … Annual revenues of embroiders from program SAs $450,000 $1,289,407 187% Number of Urban Garment Makers linked with Sales Agents 180 200 11% Number of garment makers receiving embedded services from SAs (monthly average) N/A 200 … Annual revenues of garment makers from program SAs $337,500 $30,874 (91%) Number of designers involved in program42 6 40 567% Annual revenue of designers from program SAs $19,200 $17,600 (8%) Source: MEDA Semi-Annual Report (January 1 to June 30, 2007), dated August 15, 2007 The targeted total income in Year 3 for REs and SAs was the same, but SA income increased 152% more than that for REs. The revenue of urban garment makers is 9% of t hat targeted for Year 3, and the actua l revenue of designers in Year 3 is slightly less, at 92% of revenue targeted. MEDA/ECDI explain in t heir last Semi-Annual Repor t that the y designed the program based upo n the assumption t hat the gar ment makers p roduced at a si milar level of quality and skill as the REs and therefore, MEDA/ECDI expected products from both groups to s ell in the same markets. However, they discovered that the garment makers did not have the skill level assumed and therefore they were confined to lower value markets.43 MEDA/ECDI had intended to work with formal designers, who generally have higher incomes, but found they were not a good fit for the SAs. Therefore, Behind the Veil turned to tracer designers, who command a lower price. The project also reported that some of the SAs began developing their own designs. MEDA/ECDI targeted 6,000 rural em broiderers, and by June 2007, 9,295 were already in the program, a 55% increase over the target. On average, their in comes were targeted to increase to $450, 000, b ut actually increased to $1,289,407 by June 30, a 186% increase. Figure 5 shows the change in the per item 42 Note that these figures represent sales and numbers of tracer designers and formal designers. 43 MEDA Semi-Annual Report (January 1 to June 30, 2007), p. 6. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 12 price paid to embroiderers for each of the three project districts (excluding Karachi, where activity did not include embroiderers) and the average change over all districts.44 FIGURE 5: GROWTH IN EARNINGS PER PIECE FOR RES 200 300 400 500 600 700 800 900 1000 Mar-06 May-06 Jul-06 Sep-06 Nov-06 Jan-07 Mar-07 May-07 Average earnings per piece (Rs. Quetta Multan Thatta All locations 54% growth 34% growth 45% growth 38% growth The price paid to em broiderers increased substantially in each of the districts with Quetta experiencing the largest growth. A dow nturn i n Mul tan in the fi nal m onth affected the average over all locations. Growth in the price paid to em broiderers appears t o have leveled off. The increas e in prices is likely attributable to the project but, since data are not avai lable for a control gro up, it is not possible to say for sure. MEDA/ECDI also tracked more qualitative, outcome indicators, though these never had targets. By June 30, MEDA/ECDI reported that they had accomplished the following: 44 Calculated from sales agents’ reports by dividing amount paid to embroiderers by the number of pieces sold. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 13 TABLE 2: OUTCOME INDICATORS AND RESULTS Indicator Percent Percent of wholesalers / retailers int erviewed who say q uality and de sign of products have improved (Total 18 interviewed) 81% Percent of sales age nts who say quality and de sign of produ cts have imp roved (Total 174 interviewed) 100% Percent of sales agents who are satisfied with design services of Tracer/Designer (Total 72 interviewed) 100% Percent of embroiderers/garment makers who report higher monthly income as a result of program participation (Total of 117 interviewed) 29% Percent of embroiderers/garment makers who say their quality of life has improved as a result of increased revenues related to the prog ram (e.g., not working longer hours for more income, better food) (Total of 117 interviewed) 78% Percent of embroi derers/ gar ment make rs who report t hat their statu s in the household h as risen a s a re sult of g reater e conomic contribution (T otal of 117 interviewed) 65% Percent of embroi derers/garment makers who report number of child ren going to school increased after project (Total of 117 interviewed) 26% The indicators related to product quality have gone up for almost all o f the interviewees, and wholesalers/retailers agree to a large extent (though not completely) with t he sales agents’ estimation of product quality. While a low percentag e of em broiderers and garment makers reported monthly income increases, most reported quality -of-life improvements, the quantitative value of which is difficult to measure. It should be remem bered that according to the project ’s figures shown in Table 1 , while the project exceeded its target for increase d revenue for e mbroiderers, it fell short of its target for garm ent makers, which could explain the low percentage of embroiderers/garment makers who repo rted monthly income increases. To address the monopolistic buying practices issue, the project linked embroiderers directly with women sales agents, and instructed the SAs to provide fair prices. The project also established joint venture (JV) groups of RE s with the theor y that the J Vs would have some collective bargaining power. According t o project staff, ECDI follows up on proj ects for three years after c lose-out, during which they reinforce principles like fair price setting. MEDA’s internal impact assessment of Behind the Veil found that average incomes for embroiderers and SAs increased by approximately the same percentag e (50%), indicating that monopolistic practices were averted.45 The evaluators spoke with an SA and an RE in Karachi who said t hat previously, there were occasions when they were not paid for their work. Now they consistently are, and at better rates. Regarding gender roles, th e project’s i mpact assessment found that women became more mobile, met in groups for the first ti me, and engage with men in business. S ome of the REs became SAs, ther eby increasing their mobility by default. 46 The evaluators spoke with some of these converts in Karachi and Thatta. One SA proudl y said that she did not think she could beco me a lead er, and now she is selling embroidery at exhibitions. Marketing Strategies Behind the Veil used exhibitions, bot h overseas and in Pakistan, to m arket the work of its REs. The quality of the work is a marketing tool in and of itself, and as the indicators above illustrate, both 45 Jones, Linda. MEDA—Internal Report From Behind the Veil: Impact Analysis May 2007, July 20, 2007. 46 Ibid. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 14 wholesalers/retailers and SAs agree that the quality of the embroidered products has improved during the project. A further indicator of the project’s marketing success is the fact that a hom e-based boutique owner in Karachi reduces the price of MEDA-supported prod ucts sold overseas b ecause she more than compensates for it through the volume of products she sells.47 C. Impact To what extent has the project benefited the people of Pakistan? What has been the quantitative impact on the beneficiaries’ incomes? Conclusions: The project has produced quantifiab le monetary benefits for 9,800 rural Pakistani women. Based upon available data and assumptions about the level of pre-project income, we conclude that MEDA h as contributed to increasin g the inco me of all of its t argeted beneficiaries, especially the women sales agents. The project h as also brought significant social changes to their beneficiaries’ lives. Wo men are more mobile, they are better respected at home, and they are improving the well being of their families. Current Monetary Impacts Monthly fi nancial reports compiled by the sales ag ents docum ent 174 active sales agents, 9,295 rural embroiderers, 200 ur ban garment makers, and 36 de signers engaged in the pr oject. The project’s 200 7 impact analysis reports 213 sales agents, a discrepancy caused by the fact that not all sales agents report in each month. Sales agents’ reports provided the data for estimating monetary impacts. The reports document total sales; payments to rural em broiderers, urban garm ent makers, and designers ; and materi als and other costs. Because the sophistication of the reporting evolved over time, the data series is not particularly consistent prior to April, 2006. The data series do seem consistent from that date forward. Payments to beneficiaries for embroidery products or services are not project benefits. The benefit of the project is the difference bet ween the income wo men earn from participating in t he project and what the y would have e arned had th ey not participated. The project’s 2003 market assessment reported that rural embroiderers typically make about six pieces per month and sell them for about Rs. 60 each for gross monthly earnings of Rs. 3 60.48 If this is correct on average, then embroidery revenue without the project would have been Rs. 360 per month.49 The project’ s impact analysis report st ates that most sales agents were embroiderers or urban garm ent makers who were more mobile and took on the role of sales agents. The impact analysis thus assumes that sales agents’ income without the project would also have been Rs. 360 per month. These measures of pr e-project income are no t particularly reliable but are the be st available in the absence of a good baseline. 47 Interview with home-based boutique owner in Karachi. 48 The pre-project incomes presented in the market assessment report appear to be largely illu strative but probably came from the project’s initial market analysis. Estimates are not based on a rigorous survey of participants. 49 This figure has a number of deficiencies as a “ without project” measure of in come. First, it is no t clear th at it is based o n a re presentative sa mple of wom en. Sec ond, i t appea rs t o be a gr oss value rat her t han a val ue net of expenses for materials. Third, it is a m easure of only one possible component of income. Although women in the project areas have few income earning opportunities, a better measure would assess chan ges in women’s’ income from all sources to account for possible substitution between income earning activities. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 15 Table 3 summariz es curre nt project be nefits where benefits are the difference between pre-project and post-project income. The benefit meas ures assume that rural embroiderers and sales agents would hav e earned Rs. 360 per month without the project. Consistent with the information presented in Effectiveness, the sales agents have realized the most growth in their monthly incomes, whereas urban garment makers and designers have benefited the least in monetary terms. TABLE 3: CURRENT PROJECT BENEFITS Benefits Beneficiary Number Total nominal benefits ($ thousands) Average annual per beneficiary ($)a As percent of average annual household incomee Rural embroiderers 9,330 $1,930 $81b 5.2% Active rural embroiderers 6,150 $1,930 $167c 10.5% Sales agents 174 $969 $2,745c 174.0% Urban garment makers 200 $60 n.a.d -- Designers 40 $47 n.a. d -- Totals 9,783 $3,006 a. Calculated by dividing benefits by participant year, not the number of current beneficiaries. b. Reflects average income over all embroiderers. c. Reflects average income in a month that an embroiderer or sales agent is active. d. Data are not adequate to calculate average annual benefits. e. Based on $1 ,584 ave rage ann ual household i ncome in rural areas from 20 05-05 Household Integrated Economic Survey (HIES). Note: The $2 29 im plied averag e monthly in come fo r sal es ag ents i s some what mislea ding b ecause some very high sales agent profits (the maximum is $3,047 per month) skew the distribution. The median of $178 per month better represents benefits for a typical sales agent. Project benefits are not distributed evenly across project area s. Figure 6 illustrates the distribution of benefits and beneficiaries across project regions. Multan conta ins half of t he project’s benefits and beneficiaries. The 26% o f beneficiarie s in Thatta capture more than their share of benefi ts at 32%. Beneficiaries in Quetta get less than their share of benefits. 50 Beneficiaries in Karachi are largely garment makers rather than embroiderers. 50 This may seem counterintuitive as price paid to embroiderers are highest in Quetta. However, the proportion of the sale price retained b y em broiderers in Qu etta (56 %) is sm aller th an Quetta th an in Mu ltan (65%) or Thatta (86%). The difference probably reflects the distance of Quetta from markets in Karachi. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 16 FIGURE 6: BENEFITS BY REGION 50% 26% 21% 2% 50% 32% 15% 3% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Multan Thatta Quetta Karachi Percent of beneficiaries Percent of benefits Future Monetary Impacts The project ended in Decem ber, 2007. But sales agents and rural em broiderers are not likely to sto p making and marketing em broidery without the proj ect (see the sustainabilit y chapter for findings on the prospect for continuation of market activity). To acc ount more fully for proje ct benefits th e evaluation also considered the benefits likely to accrue in the next five years – through 2012. Projecting future activity requires judgments and assumptions about the sustainability of project activities and likely growth or decay rates over tim e. The primary determinants of growth are the num ber of sales agents and em broiderers. The question is whether sal es agents will continue t o enter the market sy stem without the training and support provided by the project. The project’s im pact analysis suggests that this market entry is happening now as embr oiderers and others affiliated with the p roject move to the sales agent level. I ncreasing pri ces for e mbroidery can al so contribute to growth in sale s but pri ces seem to have plateaued (see Effectiveness chapter). To account for uncertainty about future activity, the analysis considers three possible scenarios for annual growth rates of sales agents and embroiderers: no future growth, 10% growth, and 20% growth. The “no growth” scenario represen ts a lower bound on grow th as it assu mes that no additional sales agents or embroiderers enter the market and current participants continue their activities. The 10% growth scenario is what the project’ s im pact analy sis considered a conservative esti mate. The 20% growth scenario i s intended as an upper bound esti mate. Figure 7 illustrates the cal culation of benefits for the 10% growth rate sc enario. The vertical axis repres ents aggregat e income to all beneficiaries in a given month. The present value of the ar ea between the two lines represents the benefit of the project in ter ms of increased beneficiary incomes. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 17 FIGURE 7: TOTAL PROJECT BENEFITS (NOMINAL $, 10% ANNUAL GROWTH) $0 $100,000 $200,000 $300,000 $400,000 Jan-05 Jan-06 Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Aggregate incomes ($ Aggregate incomes with project Aggregate incomes without project PV of benefits = $13.2 million Table 4 reports total benefits and the distribution of the estimated number of b eneficiaries for the three growth scenarios. All monetary values are expressed as present values as of 2007. TABLE 4: TOTAL BENEFITS FOR ALTERNATIVE GROWTH SCENARIOS Beneficiaries Growth scenario Present value of benefits ($ million) Embroiderers Sales agents Current benefits $3.3 9,295 174 Growth scenarios No growth $10.7 9,295 174 10% annual growth $13.2 16,074 301 20% annual growth $16.9 27,672 518 Non-Monetary Benefits The project also reported non-monetary benefits for its beneficiaries. According to MEDA/EC DI’s internal i mpact a ssessment, al most all of the wo men bene ficiaries they interviewed said that they contributed more to household decision-making and that their husbands and in￾laws respected them more. They said that they now have greater control over their own incomes, are more self confident, and envision a brighter future for themselves.51 As captured in the quantitative indicators in Effectiveness, increased women’s incomes contribute to more children attending school, better family nutrition and co mfort. The evalua tors’ visit to Thatta found that women used their extra inco mes to buy basic househol d items such as fans, em broidery input supplies, goats, and sometimes jewelry . Ho wever, other cond itions had no t improved, as the water quality was dubious and some children were responsible for lugging firewood. We were told that whereas the m en in the villages used to be suspicious of the project sta ff visiting their female family members, they now stay out of the way because they realize that their wives and sisters are benefiting from the assist ance, which also benefits the household. 51 MEDA/ECDI Internal Impact Assessment, pp. 5-6. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 18 D. Efficiency How efficient has the project been in utilizing its resources to achieve results? Conclusions: The project has been efficient based upon available data. It generated benefits in excess of costs (4.8 times costs). However, this measure does not capture any potential negative im pacts of th e project, such as opportunit y costs, which would depr ess the benef it measures. It also does n ot include other benefits, such as improved health, better food, etc. There are a num ber of possible measures of project e fficiency. They have in c ommon a com parison of costs to so me measure of output. The econom ic measures presented in Table 5 all depend on estim ated monetary benefits. Because not all pr oject benefits can be expressed in monetary terms and there are no data available on any negative impacts of the project, the economic measures of efficiency are only partial measures and not full internal rates of return (IRR). Table 5 documents project costs and benefits and presents t hree co mmon measures of econom ic efficiency: net present value (NPV), the benefit cost ratio, and the internal rate of return (IRR). USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 19 TABLE 5: PROJECT COSTS AND BENEFITS BY YEAR Costs ($) Benefits ($) Year Actual Discounted Actual Discounted 2004 $0 $0 $,000 $,000 2005 $0 $0 $308,000 $373,000 2006 $0 $0 $895,000 $985,000 2007 $658,095 $658,095 $1,803,000 $1,803,000 2008 $0 $0 $2,091,000 $1,901,000 2009 $0 $0 $2,310,000 $1,909,000 2010 $0 $0 $2,552,000 $1,917,000 2011 $0 $0 $2,819,000 $1,925,000 2012 $0 $0 $3,114,000 $1,934,000 Totals to date $658,095 $3,160,275 Totals to 2012 $658,095 $12,745,969 Current (2004-2007) Net present value $2,502,000 Benefit cost ratio 4.8 Internal rate of return n.a. Projected (2004-2012) Net present value $12,087,874 Benefit cost ratio 19.4 Internal rate of return n.a. Note: The evaluation team has not yet been able to obtain annual cost data for the project. Without these data it is not possible to calculate an accurate IRR. Note: The project budget includes $501,540 in USAID funds, $115,603 in contributions from MEDA and ECDI, and $40,952 in third-party contributions. Net Present Value (NPV) – The project’s net present value is the difference between the present value of benefits and the present value of costs. A positive valu e indicates that a project has returned benefits in excess of costs. Since the mea sure is in dependent of the size of a project’s budget, however, it provides no evidence of the return on the project investment. For example, the returns implied by a NPV of $1 million are very different for a $5 million project than for a $20 million project. Benefit Cost Ratio – The benefit cost ratio is the present value of benefits divided by the present value of costs. A bene fit cost ratio of one im plies that a project returned one dollar for each dollar expended. A benefit cost ratio provi des a rough m easure of eco nomic r eturns because it represents th e proportional return on inv estment. For example, a benefit cost ratio of 1.5 implies that project benefits are 150% of costs. It is no t, however, a rigoro us measure of re turns as it does not account f or the timing of costs and benefits. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 20 Internal Rat e of Retu rn (IRR) – The IRR is the r ate of return on a strea m of costs that equates th e present value of costs wit h the present value of benef its. It thus represents t he financial r eturn on the investment. The IRR, however, provides no information on the distribution of benefits. Based upon these measures, the project generated benefits in excess of costs (4.8 times costs). Another way to assess efficiency is to exa mine how costs are distributed across different cost categories. Figure 8 summarizes project expenditures by category. FIGURE 8: DISPOSITION OF EXPENDITURES $92 $173 $81 $111 $45 $502 $29 $0 $41 $87 $0 $157 $121 $173 $122 $198 $45 $658 $0 $100 $200 $300 $400 $500 $600 $700 Ex-pat labor Local labor Outputs ODCs in Pakistan ODCs outside Pakistan Total Expenditures ($ thousands) USAID Other All sourrces The categories in which co sts are reported facilitate calculation of several expenditure ratios documented in Figure 9 . The ratios provide an in dication of ho w the project spent its m oney. It spent 37% of the budget on administration and 45% on la bor. Of the la bor expenses, 59% was for local labor and 41% for ex-pat labor. It was not possible fro m the inform ation provided to separate expenditure within Pakistan from that outside Pakistan. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 21 FIGURE 9: EXPENDITURE RATIOS 37% 45% 59% 19% 0% 10% 20% 30% 40% 50% 60% 70% Administrative costs as % of total Labor costs as % of total Local labor as % of all labor Outputs as % of total Percent of expenditure in Pakistan E. Sustainability Are the activities and results likely to be sustained after the project is completed? Are the project results sustainable without additional support? Conclusions: MEDA/ECDI created a self-sustaining BDS market with a little facilitation and capacity building. Beneficiaries have embraced the model and are dis covering the im portance of their roles and mobility within it. As long as demand for the REs’ work is sustained (no guarantees in the free market) and there ar e no unforeseen suppl y line disruptions, be neficiaries should conti nue to enjoy what t he project has established for them and modify it to suit their own ambitions and capacities. It is still an open question of whether REs’ highe r inco mes will remain protected fro m monopolistic practices, but given that they have multiple buyers, some collective bargaining power through the JVs, and ongoing monitoring from ECDI, chances are high that they will be protected. Behind the Veil created a sustainability plan in its or iginal design that centered on the r ole of the SAs as follows:52 ¾ SAs provide em bedded services to rural em broiderers and receive a fee through the markup on embroidered garments. ¾ SAs link micro garment makers to retailers, exporters, and exhibition planners. ¾ Commercial designers provide services to micro garment makers, retailers, exporters, and SAs for a fee. “Once these groups have confidence in the market, they will be willing to invest.” MEDA/ECDI provided ca pacity development and f acilitation of inform ation sharing and relationship building, but the SAs performed the actual work of establishing the value chain and once the sy stem was in place, MEDA/ECDI expected to withdraw. The income i ncreases the project beneficiarie s ha ve realized are based upon t he project model of devel opment of a value chain that responds to market demand and supply. 52 MEDA IGP Narrative File, p. 7. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 22 In addition, MEDA/ECDI transferred the facilitation role they had been playing to Buying Houses (BHs). Entrepreneurs’ investments established the BHs, and some of the project-supported SAs have established their own BHs. The six Buying Houses (BHs) consolidate the work of groups of REs by working with the CSAs. They provide quality control, marketing, and generally link large numbers of REs with custom ers. BHs provide merchandise samples to a range of buyers and distribute large orders through the value chain system. The BHs charge a reasonable fee for their services and the project’s impact analysis contends that Behind the Veil incre ased BHs’ capa cities in planni ng, financi al management, staff management, and merchandise displays.53 Therefore, theoretically, sustainability of the buying houses means sustainability of the value chain. ME DA/ECDI had intended fro m the beg inning t o use BHs as a sustainability mechanism.54 Figure 10 depicts the market linkages MEDA established. REs either sell to SAs or form “joint ventures” to sell as groups (see discussion in Effectiveness). FIGURE 10: MAP OF THE EMBROIDERED GARMENT SECTOR Source: MEDA Semi-Annual Report (July 1 – December 31, 2006), February 15, 2007. With multiple channels and links, Behind the Veil created a web of market access, supply, and demand, providing many options for the products that REs produce. 53 MEDA/ECDI Impact Analysis, p. 3. 54 Discussion with ECDI project manager and staff, October 1, 2007 in Karachi. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 23 As described under effectiveness, many REs have become SAs as they have dis covered an interest in and capacity to be businesswomen. Some of the SAs have opened input supply shops, boutiques, retail outlets, and buying houses55, further strengthening the model Behind the Veil built. The project also established the Association of Wo men Entrepreneurs in Sm all and Micro-enterprise s (AWESOME) in H yderabad, Quetta, Karachi, and Mu ltan as part of its exit strategy . Me mbers meet regularly to plan events , share infor mation, co llaborate on orders, and plan expansi on of their businesses.56 We assu me t hat the wo men will continue to meet as long as they find the association beneficial. The evaluators found that AWESOME is tied to Ch ambers of Commerce, wh ich give the m recognition as business people. ECDI did acknowledge to the evaluators that prices of embroidery are lower in some markets, although the sa me car e may not have been take n to ensure that designs match high v alue market demand. In a market system it would be acc eptable for prices to be driven down. However, this project is geared toward economic empowerment of poor women, while at the same time driven by market demand. Some of the REs have beco me empowered enough to start to establish themselves as SAs; howev er this is not true for the majority, given field realities. ECDI told the evaluators that they had begun to take the model further by linking beneficiaries with the National Rural Support Program (NRSP) to obtain micro finance for the sales agents. F. Replication To what extent can the activities and results of the project be replicated? Can this activity for embroiderers be replicated on a larger scale across Pakistan? Can this activity be replicated in other areas beyond embroidery? Conclusions: The Behind the Veil model is replicable and could be replicated both in Pakistan and in other countries, within and without the embroidery sector. The SEEP Network found that the project’ s replic ability was en visioned from the beginning because MEDA/ECDI’s market assessment informed their des ign.57 Therefore, assu ming market conditions have not changed, MEDA/ECDI’s model as modified through experience would be replicable in other parts of Pakistan, as well. In fact, the model is already being replicated. ME DA and ECDI recently received a large grant from the Canadian International Development Agency (CI DA) to develop additi onal value chains in all fou r provinces of Pakistan, with the goal of reducing poverty for marginalized women. A large com ponent of the project will be value chain training by ECDI for other groups. 58 When the evaluation team last spoke with Helen Loftin of MEDA in Novem ber 2007, she said that they had not y et settled on the specific sectors for value chain work. MEDA has d eveloped a toolkit to support replication of their model by other institutions in other places and markets. The Pakistan work is already a replication, since MEDA conducted a similar project in Afghanistan. 55 MEDA internal Impact Assessment, p. 6. 56 MEDA/ECDI reporting 57 SEEP Case Study #6. 58 Interview with Perveen A, Shaikh (ECDI), and Helen Loftin (MEDA) on September 14, 2007. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 24 The essential ingredients needed for replicating Behind the Veil include:59 ¾ MEDA/ECDI’s market research and program design process; ¾ Commitment to reaching marginalize d populations through sustainable market sy stems and conviction that markets can be influenced to benefit the poor; ¾ Sufficient skills to conduct training and capacity building for pro- poor value chain developmen t and manage such a complex program. ¾ Funding that matches the scale of the market. ¾ Understanding that value chain develo pment is not a social enterprise, and donors sh ould not expect cost recovery. ¾ Market economy with sufficient suppl y and demand signals and d emand for a product or service that the targeted beneficiaries can make and deliver. ¾ “Visionary and entrepreneurial leadership.” It should be remembered that Behind the Veil was able to generate significant quantitative and qualitative benefits for its beneficiaries with few resources ($633, 000 for three years). To do so again with a similar funding level, the im plementing organi zation woul d need talent, co mmitment, and a slim operating budget. Like MEDA/ECDI, it would also need to rely on predominantly local labor that receives payment through the services provided, not through the project (i.e., SAs). G. Gender To what extent has the project benefited women? Conclusions: The project has primarily benefited women by increasing their incomes, mobility, skills, self-confidence, and family respect. Women were the targeted beneficiaries of Behind the Veil and, as discussed in effectiveness and impact, they have realized significant m onetary and social be nefits from their participation in the model. Their incomes gre w, as did their status in their families, they discovered capabilities and skills they did not know they had. Former e mbroiderers beca me s ales ag ents and traveled, some for the first time, outside of their own villages to places l ike Karachi. The 14 mobile SAs i n the Thatta region, the evaluators were told, are all former REs. Some wo men opened bout iques, shops, and buy ing houses. They learned to work with other business people and with each other. MEDA/ECDI observed that CSAs and LSAs beca me “wo men’s a dvocates, community mobilizers, and social entrepreneurs.”60 Through tr aining sessions, work groups, participation in exhibiti ons, joint vent ures, and AWESOME, women have found more freedom to meet in groups. Photo 1: Input supply sh op in Rahib Abro village and the girls (taken by evaluator) 59 SEEP Case Study #6, pp. 54-55. 60 MEDA Impact Analysis, p. 5. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 25 H. Reporting Have the prime contractors and grantees reported on time and in a useful manner? Conclusions: Reporting has been timely, useful, and informative. MEDA/ECDI produced a series of sem i-annual progress reports that were sub mitted in a tim ely manner, always 6 weeks after the close of a 6-month period. Quarterly reports were also submitted. The project’s Cognizant Technical Officer (CTO) at USAID/Pakistan was pleased with the reporting. The reports include progress against quantifiable indica tors that w ere developed at the outset. The project developed an extensive questionnaire to capture qualitative changes, which it administered twice a year to record changes in the beneficiaries’ incomes; sense of em powerment; decision making authority in the household; and effects on savings, investments, and children’s education. Not all of the quantitative data was collected at the outset, however. Income data collected at th e beginning was limited, but expanded over time. The project pre-dates the branding guidelines for cooperative agreements, according to the CTO. Even if it had not, th e CTO said he probabl y would have requ ested a w aiver due to the context in which the project was operating. I. Outreach and Communications How effective has the project been in getting its story out? Conclusions: The project’s co mmunications have be en adequate and probabl y appropriate, considering its beneficiaries’ situations. Media interactions were minimal in order to protect the privacy of beneficiaries, but t he pro ject exhibited its pro ducts both nationally and i nternationally, a form o f publicity. The im plementers also participated in networks such as the SEEP Network and published articles about their work. The project staff has been effective, but low ke y in getting the project’s story out. According to the CTO, the project has intentionally kept a low profile inside Pakistan in order to develop and maintain stron g relationships and trust with co mmunities and the private sector. By contrast, outside Pakistan, the project has broadly promoted its approach, successes, challenges, and lessons learned to donors and practitioners. The project has published a series of articles on the SEEP Network. Both through the evaluation tea m’s visit to the fie ld and the CTO’ s visit, the i mpression is that the beneficiaries were well aware of the project’s objectives, role, and activities. MEDA/ECDI used the following activities to publicize the project:61 • Exhibitions. Project beneficiaries h ave partic ipated in govern ment-subsidized national and international trade shows i n Lahore, Karachi, a nd Islam abad. The evaluators attended one s uch exhibition in Karachi and were i mpressed with the qualit y of goods produced by pr oject beneficiaries. • SMEDA. On e of the Behind the Veil’s groups in Ba lochistan was selected to participate in the “One village-one product” program of the Sm all and Medium Enterprise Development Authority (SMEDA). Through the program , p articipants receive market support an d opp ortunities to participate in exhibitions. 61 MEDA Impact Assessment, p. 5. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 26 • Trade fairs outside Pakistan. From Behind the Veil offered program participants the opportunity to attend subsidized trade fairs in India. One Karachi SA participated in one such fair. According to the CTO, the project had a strong sense of the message and media to communicate with its stakeholders to maximize project impact among sensitive communities. However, more could have been done to comm unicate the project’s activities and resu lts more b roadly within Pakistan fro m the U.S. Government perspective, which was not a priority from the project’s perspective. J. Coordination How effectively has the project coordinated with other parties? Conclusions: The project coordinated effectively internally, with USAID, and with the Government of Pakistan. It did not interact much with other projects or NGOs. MEDA and ECDI had worked together prior to this project and told the evaluators that their coordination and relationship is based on trust. 62 The two organizations had worked toget her in the pa st in Pakista n and are continuing to work together on a CIDA-funded project. According to the project CTO, the project coordinate d well with USAID/Pakistan. Since sit e visits w ere not possible, the CTO he ld quarterly conference c alls with both HQ and field staff to discuss st atus, issues, and plans. The CTO told the evaluators, “the project was extremely proactive in raising questions and issues.” 63 On the other hand, he commented g enerally that better coordination woul d be usefu l between USAID/Washington or US AID/Washington-funded projects (like Behind the Veil) and t he USAID/Pakistan Mission. The project has attempted to coordinate with other USG projects when required, but it has n ot developed synergies with other econo mic growth projects. The project staff s aid that they attempted synergies with the Government of Pakistan’s (GOP) Ahan project, but nothing materialized. Behind the Veil did wor k with SMEDA as discussed above, and received subsidies fr om the Export Promotion Bureau for exhibitions outside of Pakistan.64 V. RECOMMENDATIONS The following reco mmendations identify Behind the Veil’s success factors and suggest their application to future projects. Relevance The value chain approach, especially in addressing the needs of poor women, has great potential and can and should be replicated, by USAID/Pakistan and its partners. However, expe rience shows that not all value chain projects create a stable market that survives the project’s end. As the Behind th e Veil project shows, projects can be made most relevant to ben eficiaries by including the m in project design and implementation, and by conducting thorough market research prior to project design. Future projects that focus on individual (rather than systemic) benefits should adopt this model. 62 Interview with MEDA and ECDI representatives, September 15, Islamabad. 63 Communication with Jason Wolfe, MEDA CTO, USAID/Pakistan 64 MEDA Impact Assessment, p. 5. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 27 Effectiveness Projects designed b y US AID/Pakistan and its partners can at the design stage identify possible weaknesses or issues that can arise in the future, and ensure their m itigation. In sim ilar value chain models for poor wom en, steps should b e taken, such as the establish ment of producers’ coo peratives, so that the possibility of monopolistic pricing by buyers is diminished. Impact At the design stage, USAID/Pakistan and its partners should ensure that the value chain process for th e particular product being considered is workable, tried and tested, so that project impact is maximized. The preparation and dissemination of case studies through the life of the project would be effective in showing impact. Efficiency Future projects of a si milar nature should maximize cost benefit ratios b y using local labor and building local capa city to conduct the project’ s activities. T he indicators for ensuring that cost b enefit analy sis takes place prior to and during the project’s implementation should be identified in the project design, and their collection and analysis should be a part of the contractual agreement with the implementers. Sustainability Future projects should create a sustainabilit y plan at the very beginning and follow it closely throughout the life of the project, making modifications base d upon ex perience. The project design team must consider issues of eventual sustainability of the project, and make the needed adjustments/interventions a part of the indicators to be monitored through the life of project. Project designers and im plementers should consider providing microfinance facilities to assi st women at the sales agent and buying house level to respond t o more orders, participate in more exhibitions, and increase sales by accessing loans for equipment, input supplies, and travel, thus enhancing sustainability. Gender Gender-focused projects should develop a design that works directly for women and sh ows a solid understanding of their needs. USAID should co nsider using demonstrated kno wledge of women’s needs amongst the target population as a key criterion in proposal evaluation. Outreach and Communications In projects where there is sensitivity about photographs and other media coverage, the project implementers can use other forums, such as the SEE P Network, to publicize results. Efforts should be made, by USAID and th e im plementers, to present pr oject results at different forums, and generate discussion and interest in developm ent groups. In future, similar projects, funding should be adequate to ensure that the basic requirements are met. Coordination The project im plementers, using USAID/Pakistan as sistance when needed, should coordinate fully with civil society, donors and the government, and keep them informed through meetings and newsletters, thus enhancing a common understanding and ownership of best practices. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 28 VI. LESSONS LEARNED ƒ Projects can benefit by precisely defining what economic and social empowerment of women entails, and developing indicators to track progress. ƒ Research and market assessments help to design value chain development. ƒ Beneficiary involvement is critical for successful pr oject design and implementation, particularly for micro-level projects. ƒ Development of wo men’s as sociations in th e val ue chain process can p rotect wo men from monopolistic buying practices. ƒ Using predominantly loc al labor reduces project costs and maximizes bene fits by building local capacity. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 29 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 30 VII. NEXT STEPS This evaluation pro duced a num ber of specific re commendations for enhancing future p rojects. The evaluation will be useful onl y if USAID and its project partners learn fro m t he reco mmendations and implement th em. This cha pter sets fort h procedur es for evaluating the reco mmendations and deciding how to use them to best advantage. A possible course of action for methodically processing the evaluation results is: Activity Timeframe Responsible person/office Assign a person to review the recommendations of all eight evaluations and separate t he reco mmendations into: (1) those that need to be handled in ternally within USAID, (2) those that need to be handled internally within EG, and (3) those that are project specific. Immediately USAID EG Recommendations internal to USAID Convene a meeting within USAID to review the recommendations that need to be handled in ternally wit hin USAID. Use the meeting to: • Decide which recommendations to address and which to ignore. • Discuss how to address the reco mmendations deemed important. • Identify an indiv idual or office responsible for implementing each recommendation. • Establish a timeframe for implementation. • Determine a process for tracking progress on implementation of each recommendation. Third priority after initial meeting USAID Reconvene every six m onths (in whatever groups are appropriate) to review progress on implementation. Six month intervals USAID Recommendations specific to EG Convene a meeting within EG to review the reco mmendations that need to be handled with in EG. Follow the procedures outlines above. Second priority after initial meeting EG Reconvene every six m onths (in whatever groups are appropriate) to review progress on implementation. Six month intervals EG ANNEX 2: GETTING TO ANSWERS MATRIX Evaluation Questions Type of Answer or Evidence Needed Method of Data Collection Data Source Selection Criteria Data Analysis Methods Relevance: How well was the project focused on the needs of the beneficiaries? Describe the motivation for initiating the project. (who, why, where, for/to whom) Descriptive Interviews Interviews with USAID Interviews with stakeholders Interviews with key informants Content analysis – look for examples of identifying a need, stakeholder input into design, participatory approach How well did the project design incorporate the views of all relevant stakeholders? Describe the logic and process behind the joint USAID/MEDA/ECDI venture. Was the activity designed to meet a felt need of a specific community, target audience, or influential stakeholder? Was the project well designed to address the needs of the beneficiaries? To what extent did the design of the activity utilize participatory techniques? How well was the project adjusted to address the needs of the beneficiaries? Was the targeting appropriate in hindsight? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 31 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods In what ways has the project helped or hindered the benefits for women? How well did the project design incorporate the views of all relevant stakeholders? Scale (5 point) Interviews Interviews with USAID Interviews with stakeholders Interviews with key informants Aggregate scale Were stakeholders involved in a substantive way throughout the project life cycle? Scale (5 point) Interviews Interviews with USAID Interviews with stakeholders Interviews with key informants Aggregate scale Has stakeholder input led to changes in implementation? Provide examples. Descriptive Interviews Interviews with USAID Interviews with stakeholders Interviews with key informants Content analysis – document concrete examples of instances where stakeholder input guided implementation What successful strategies have been adopted for marketing these women’s products? Effectiveness: Has the project accomplished its objectives? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 32 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods What were the stated objectives and targets? Descriptive Document review Project planning documents Tabulate objectives and targets How were the initial targets established for each activity? Were the targets realistic? To what extent were the targets achieved? Were the targets appropriate? What are the lessons learned for setting targets in future activities in accordance with the requirements of USAID’s Performance Monitoring Plan (PMP)? What was the impact on the stakeholder of training materials or programs? (by region and gender) Comparison of before and after the training for those trained (skills, job responsibilities, etc.), their organizations and operations (changes and likely effects). Survey structure interviews Those trained and their bosses Energy experts Random sample of those trained and their bosses, stratified by type of training and organization Expertise of the knowledgeable persons Simple tabulation and estimation of the range of potential economic benefits How many community organizations have been mobilized? Comparison Interviews Number of community organizations mobilized How has the product done more than create the linkages and the middlemen? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 33 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods Impact: To what extent has the project benefited the people of Pakistan? What are annual budgets for the project, and what does USAID/Pakistan provide? Cost breakdown for each activity How has the program benefited the intended beneficiaries? What were the primary and secondary positive and negative impacts of the projects? How large has the impacts been or are likely to be? To what extent can the impacts be attributable to the project? How were the impacts distributed by region/province/area? Disaggregated data Were any of these benefits or losses unexpected? What have been the impact (positive or negative) on women due to the project’s implementation? Interviews Efficiency: How efficient has the project been in utilizing its resources to achieve results? What are the costs associated with the project Total costs associated with MEDA activities Document review or request from MEDA Interview with MEDA or request to MEDA staff Input to cost benefit analysis What was USAIDs contribution to project costs (by quarter and region) Total USAID financial support by quarter and region Document review or MEDA/USAID records Review of project financial documents Input to cost benefit analysis Break down by (1) ex-pat labor based in US, (2) ODCs (if any) USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 34 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods spent in US, (3) ex-pat labor in Pakistan, (4) local labor, (5) outputs and activities, (6) ODCs What are the monetary benefits of the project? (by quarter, region, and gender) Impact analysis – see questions to assess impacts in impacts section Input to cost benefit analysis To the extent possible, what is the internal rate of return for this project, as calculated in a cost benefit analysis? How cost-effective has the project been? How do overhead and administrative costs for this activity compare to others across differing types of implementation mechanisms (e.g. Contract, Limited Scope Grant Agreement, Grant, Cooperative Agreement) and for the different types of implementing entities (e.g. local vs. international firms, non￾profits vs. for-profits, etc)? What are the estimates of benefits from bringing individuals into the value chain Importance, relevance and impact of the value chain inclusion Structured interviews Those included Sustainability: Are the activities and results likely to be sustained after the project is completed? Were the activities designed in a manner which focuses on their sustainability after project USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 35 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods completion? Was the initial timeframe for the activity realistic to achieve sustainable results? Were any of the activities fundamentally designed and implemented in a way which creates donor dependence? Is it reasonable to expect the project to achieve sustainability in the project life given internal and external factors? Is project financially sustainable? Why or why not? Descriptive Interviews Key informant interviews Narrative with content analysis to identify common reasons What are the key elements that will sustain this project after MEDA and/or USAID withdraw? Replication: To what extent can the activities and results of the project be replicated? Were the activities designed in a manner which focuses on their replication? Descriptive Interviews Interviews with stakeholders Narrative with content analysis to identify common challenges Were the activities implemented in a manner which focuses on their replication? Can the activities be replicated in other areas with similar socio￾economic features? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 36 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods Can the activities be replicated in dissimilar areas? To what quantified extent can the project be replicated? Descriptive Interviews Interviews with stakeholders Narrative with content analysis to identify common constraints Can the activity for embroiders be replicated on a larger scale across Pakistan? Which parts of the project best lend themselves to scaling up? Can this activity be replicated in other areas besides embroidery? Have other partners/groups been trained in value chain methodology? What are the prospects of such training and how would it be achieved? Gender: To what extent has the project benefited women? What efforts have been made to include men in this process of women’s economic empowerment, and what have been the successes and challenges? To what extent has the project included women in its staff, partners, agents, etc.? How effective has the project been in reaching women? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 37 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods What are the direct quantified benefits that the project has had on women? What is the direct benefit that each program has had on women based on quantitative analysis of available data? With and without comparison Collected in impacts section Described in impacts section Reporting: Have the prime contractors and grantees fulfilled all of their reporting requirements in an accurate, comprehensive and timely manner? Have the partners fulfilled all of their reporting requirements? Comparison Document review Interview Project reports USAID Percent of reporting requirements fulfilled How useful is the format of reports been to you (USAID staff)? Scale Interview USAID How can the reporting format be improved? Descriptive Interviews Interviews with project staff Interviews with USAID Narrative with content analysis to identify common concerns How can the reporting requirement and formats be improved? Descriptive Interviews Interviews with project staff Interviews with USAID Narrative with content analysis to identify common concerns Have all branding guidelines been followed? Yes/no Document review Interview Project reports USAID Yes/no with descriptive detail Are the reported results accurate and verifiable? Yes/no Document review Interview Project reports USAID Yes/no with descriptive detail Public Relations/Media Coverage: How effective has the project been in getting its story out? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 38 Type of Answer or Evidence Method of Data Selection Evaluation Questions Needed Collection Data Source Criteria Data Analysis Methods Describe the project’s public relations and media coverage outputs. (e.g., number of news stories published, number of press events held, etc.) Numeric Document review Interview Project documents Project staff Frequencies of different types of public relations and media activities To what extent have they raised awareness of the MEDA/ECDI project? Comparison Document review Public awareness survey Percent increase in awareness of project How effectively has the project highlighted success stories? How effective were the project’s public relations events/activities in terms of frequency, nature, profile, content and design, branding and participation? How can the impact of the public relations component of future programming be improved? What would make future public relations activities more effective? Descriptive Interview Project staff Implementing partners USAID media personnel Coordination: How effectively has the project coordinated with other parties? How effectively has the project coordinated with the Government of Pakistan? How effectively has the project coordinated with other USG projects? USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 39 EMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 40 Evaluation Questions Type of Answer or Evidence Needed Method of Data Collection Data Source Selection Criteria Data Analysis Methods How effectively has the project coordinated with other donors? How effectively has the project coordinated with other stakeholders? To what extent were synergies developed between the project and other individual USAID EG activities, other donor programs and/or GOP initiatives? What concrete steps should be taken to improve coordination and maximize synergies in future activities? What evaluations have already been done for this project? What is the strength of relationship between MEDA and ECDI? What is strength of relationships between partners through the value chain? What are the key strengths of MEDA and ECDI’s management style? USAID/ACCESS TO CONT ANNEX 3: LIST OF REFERENCES 1. Blumberg, Rae Lesser. August 2 005. Women’s Ec onomic Em powerment as the Magic Potion of Development? Paper presented at the annual meeting of the American Sociological Association. 2. Eagly, Alice H. and Carli, Linda L. September 2007. Leadership. Harvard Business Review. 3. ECDI. 2005. Quarterly Progress Report (January – March 2005). 4. ECDI. 2007. Quarterly Progress Report (January – March 2007). 5. Economist.com. Finance and Economics. April 2006. “A Guide to Womenomics.” 6. Gloekler, Aalya and Janet Seeley. 2006. Gender a nd AKRSP – Mainstreamed or Sidelined? in Geof Wood, Sum aira Sagheer and Abdul Malik (ed.) Lessons Lear ned in Devel opment in Pakistan: Twenty years of the Aga Khan Rural Support Programme Oxford University Press. 7. Jones, Linda, and Snel grove, Alexandra. June 2006. From Behind the Veil: Industry -Level Methodologies and the Im plications for Disadva ntaged Commu nities – the Case of Seq uestered Women in Pakistan. Small Enterprise Development Journal, Vol. 17, No. 2. 8. Jones, Linda, et al. 2006. The Double-X Factor : Harnessing Female Hu man Capital for Econom ic Growth. International Journal of Emerging Markets. Vol. 1, Issue 4. 9. Management Sy stems International. October 2007. Report on Design Methodolo gy for Pakistan Economic Growth Evaluation and Design (PEGED). Washington, DC. 10. Management S ystems International. September 2007. Repor t on E valuation Metho dology f or Pakistan Economic Growth Evaluation and Design (PEGED). Washington, DC. 11. McVay, Mary, et al. Septem ber 2007. “Behind th e Veil: Case Study # 6”, SEE P Network’s Poverty Outreach Working Group. 12. MEDA. August 15, 2005. Semi-Annual Report (January 1 – June 30, 2005). 13. MEDA. August 15, 2006. Semi-Annual Report (January 1 – June 30, 2006). 14. MEDA. August 15, 2007. Semi-Annual Report (January 1 – June 30, 2007). 15. MEDA. August 27, 2003. From Behind the Veil: Access to Contemporary Markets for Homebound Women Embroiderers in Pakist an. Implementation Grant Program (IGP) Narrative File, Executive Summary. 16. MEDA. February, 2007. Semi-Annual Report (July 1 – December 31, 2006). 17. MEDA. February 15, 2006. Semi-Annual Report (July 1 – December 31, 2005). 18. MEDA. May 2007. Internal Report - From Behind the Veil: Impact Analysis. 19. MEDA. July 13, 2007. Report to USAID for Data Quality Assessment (DQA) Purposes. 20. MEDA/ECDI. 2004 . Access to Contem porary Mark ets for Homebound W omen Em broiderers in Pakistan. Presentation. USAID Proposal. 21. MEDA/ECDI. Access to Contem porary Markets for Homebound Women Embroiderers in Pakistan. Presentation. 22. MEDA/ECDI. July 2005. Embroidered Garment Value Chain. Presentation. 23. MEDA/ECDI. February 2004. Follow-up Report. Revised. 24. MEDA/ECDI. January 14, 2004. Implementation Grant Program (IGP) Implementation Plan. 25. MEDA/ECDI. June 13, 2003. MEDA ECDI Market Assessment Report. SEEP PLP. 26. The SEEP Network. June 2005. The Role of the Facilitator: Taking a Systems Perspective. Technical Note # 3. 27. USAID. 2006. Briefing Paper: Value Chain Approach to Economic Growth with Poverty Reduction. USAID website. 28. USAID. July 3, 20 03. R equest for Applications ( RFA) Nu mber M/OP/EGAT/EMD03-1610 – Microenterprise Implementation Grant Program: Business Development Services. Washington, DC. 29. Wolfe, Jason. October 18, 2007. Gender Sensitive Value Chain Develop ment: Learning fr om the Embroidered Garment Value Chain in Pakistan. USAID, Washington. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 41 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 42 30. World Bank. Operations Evaluation Department. Summer 2002. The Next Ascent: An Evaluation of the Aga Khan Rural Support Program. Washington, DC. 31. World Bank. Pakistan Co untry Gender Assessme nt 2005. Bridging the Gender Gap: Opportunities and Challenges. Islamabad. ANNEX 4: CONTACTS No Last Name First Name Title Organization Address Cell Telephone Email/WWW Date Met 1 Loftin Helen Senior Consultant/ Manager Production and Marketing Linkages Mennonite Economic Development Associates (MEDA) 155 Frobisher Dr, STE 1- 106, Waterloo, Ontario N2V 2E1 Canada g 9/14/2007 2 Asif Shazia Marketing Coordinator Enterprise and Career Development Associates (ECDI) Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 3 Asreen Rozina Urban Facilitator ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 4 Dewan Rubina Project Coordinator ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 5 Khan Nida Zaman Data and Training Coordinator ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 6 Khatoon Momina M&E Officer ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 7 Latif Kashif Finance Officer ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 8 Nisa Zaib-un Rural Facilitator ECDI Thatta, Sindh 10/1/2007 9 Shaikh Amneh M&E Manager ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 10 Shaikh Perveen A. Chief Executive Officer ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/14/2007 11 Waqar Saima Merchandizer /Trainer ECDI Awami Markaz, Shara-e￾Quaid-e-Azam, Karachi 9/30/2007 12 Khan Tanvir A. Mrs Owner Nines Boutique Punjab. 1158-C Mumtazabad, Multan 9/29/2007 13 Khan Yasmeen Worker Nines Boutique Punjab. 1158-C Mumtazabad, Multan 9/29/2007 14 Akhtar Mrs. Worker Nines Boutique Punjab. 1158-C Mumtazabad, Multan 9/29/2007 15 Choudhry Zahida Director Designing She's Boutique Punjab. 2142 Erum Colony, Vehari Road, PC 60500, Multan 9/29/2007 16 Naqvi Hina Owner Rida's Fashion Boutique Punjab. 8/9 Ali Murtaza Street, #2 Haiderai Road, Gulgasht Colony, Multan 9/29/2007 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 43 EMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 44 No Last Name First Name Title Organization Address Cell Telephone Email/WWW Date Met 17 Tahira, Mrs. Owner Tahira Boutique Punjab. Double Phatak, Purana Shaja Abadar Road, Multan & 0 6625 9/29/2007 18 Baji Farzana Abdullah Boutique Centre Punjab. House# 30, Street # 14, AlFarooq Grammar School Road, Muslim Town, Bahawalpur 9/29/2007 19 Ms. Sabiha Owner Sabihaz Punjab. Multan 9/29/2007 20 Kiani Asia Owner Kashmala's Collection Punjab. Band Gali, Mahmood Abad Colony, Near Bismillah Chowk, Khanewal Road, Multan 9/29/2007 21 Shaheen, Mrs. Owner Shaheen Botieuque Sindh. House 336, Unit # 10, Block # D, Latifabad, Hyderabad 9/29/2007 22 Worker Mehran Rang Boutique Sindh. Shop No 6, Main Road, Qasimabad, Hyderabad 9/29/2007 23 Sand Sheheryar Owner Marvee Boutique Sindh. Shop No. 57, 1st Floor, Fawad Shopping Center, North Nazimabad Block G, Haidery, Karachi 9/29/2007 24 Worker Hubba Collection Sindh. House # G/11, Al Waheed Colony, Hali Road, Hyderabad 9/29/2007 25 Owner Anjum's Designer Collection Sindh. Suite A-271, Block￾L, North Nazimabad, Karachi 9/29/2007 26 REs of Village Jar, Thatta, Sindh 27 REs of Village Rahib Abro, Thatta, Sindh USAID/ACCESS TO CONT ANNEX 5: MEDA/ECDI PROJECT STRUCTURE Head Office (Karachi) Balochistan Facilitating Office Punjab Facilitating Office Sindh Facilitating Office Quetta Multan Thatta Hyderabad Karachi Source: ECDI, Access to Contemporary Market for Homebound Women Embroiderers in Pakistan, Project Brief/Presentation, March 2006. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 45 ANNEX 6: KEY STAKEHOLDERS ƒ Buying house owners ƒ Credit marketing institutions ƒ Embroiderers ƒ Event organizers ƒ Exporters ƒ Fashion designer institutions ƒ Fashion designers ƒ Foreign buyers ƒ Home based boutiques ƒ Immobile sales agents ƒ Infrastructure (venues for exhibitions) ƒ Input suppliers ƒ Market based boutiques ƒ Merchandisers ƒ Mobile sales agents ƒ Retailers / wholesalers ƒ Sign board makers ƒ Tracer designers ƒ Trainers (marketing, finance, design) ƒ Web site developers ƒ Community sales agents Source: ECDI, Access to Contemporary Market for Homebound Women Embroiderers in Pakistan, Project Brief/Presentation, March 2006. USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 46 ANNEX 7: PRELIMINARY MAP OF THE EMBROIDERED GARMENT SUB-SECTOR Source: MEDA ECDI, Market Assessment Report, SEEP, Practitioner Learning Program, December 2003 USAID/ACCESS TO CONTEMPORARY MARKETS FOR HOMEBOUND WOMEN EMBROIDERERS PROJECT 47