BANGLADESH PROGRAM EVALUATION: LEADERS OF INFLUENCE PROGRAM April 6, 2011 This publication was produced for review by the United States Agency for International Development Mission in Dhaka, Bangladesh. It was prepared by Millennium Partners under its Building Recovery and Reform Through Democratic Governance Contract DFD-I-00-06-0028-00 Task Order AID-388-TO-11-00006 LEADERS OF INFLUENCE EVALUATION REPORT Submitted by: Millennium Partners Two Boar’s Head Place Suite 130 Charlottesville, Virginia 22903 Submitted to: USAID/Bangladesh April 6, 2011 Task Order No. AID-388-TO-11-00006 Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page ii TABLE OF CONTENTS ACRONYMS ............................................................................................................................................... iii EXECUTIVE SUMMARY ......................................................................................................................... iv I. INTRODUCTION ............................................................................................................................... 1 A. The Country and Its People ..................................................................................................... 1 B. Improving Governance ............................................................................................................ 2 II. EVALUATION PURPOSE ................................................................................................................. 4 A. Leaders of Influence Overview ............................................................................................... 4 B. Evaluation Results ................................................................................................................... 4 III. THE DEVELOPMENT CHALLENGE & USAID’S RESPONSE .................................................... 5 A. Development Challenge Addressed by the LOI Program ....................................................... 5 B. The Leaders of Influence Program .......................................................................................... 6 C. Program Assumptions and Components ................................................................................. 7 D. Democratic Governance and Institution Building ................................................................... 9 IV. EVALUATION METHODOLOGY ................................................................................................. 10 A. Evaluation Preparatory Phase ................................................................................................ 10 B. Bangladesh Fieldwork ........................................................................................................... 10 C. Methodological Constraints .................................................................................................. 11 V. EVALUATION FINDINGS .............................................................................................................. 12 A. Relevance .............................................................................................................................. 12 B. Effectiveness ......................................................................................................................... 14 C. Relevance and Effectiveness ................................................................................................. 18 D. Efficiency .............................................................................................................................. 21 E. Impact .................................................................................................................................... 23 F. Sustainability ......................................................................................................................... 25 VI. CONCLUSIONS AND RECOMMENDATIONS ............................................................................ 28 A. Conclusions ........................................................................................................................... 28 B. Recommendations ................................................................................................................. 29 Tables Table 1 LOI Partners by Type ................................................................................................... 7 Table 2 Religious Groups by Projected Outputs ....................................................................... 8 Table 3 Secular Groups by Projected Outputs .......................................................................... 8 Table 4 Religious Groups Projected Outputs by Results Achieved ........................................ 15 Table 5 Secular Groups Projected Outputs by Results Achieved ........................................... 15 Figures Figure 1 LOI Results Framework ............................................................................................... 9 Annexes Annex 1 Scope of Work Annex 2 LOI Team Member Biographical Summaries Annex 3 Literature Cited Annex 4 Persons Interview Annex 5 LOI Trainee Recruitment by Group and Year Annex 6 Persons Surveyed Responding to Questions on Democratic Values Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page iii ACRONYMS BCCP Bangladesh Center for Communication Programs BCDJC Bangladesh Center for Development Journalism and Communication BRWT Buddhist Religious Welfare Trust CCDB Christian Commission for Development in Bangladesh DESH Democracy Enhanced, Sustained and Harnessed (USAID) DG democracy and governance DI Data International DW Democracy Watch EOP survey End-of-Project survey GOB Government of Bangladesh HRWT Hindu Religious Welfare Trust IFB Islamic Foundation Bangladesh ITA Imam Training Academy LEBs Local Elected Officials or Bodies LOI Leaders of Influence Program LTQ Long-term Questionnaire MDG Millennium Development Goals M&E Monitoring and Evaluation MACCA Masjid Council for Community Advancement MACH Management of Aquatic Ecosystems through Community Husbandry MORA Ministry of Religious Affairs SOW Statement of Work SPSS Statistical Package for the Social Sciences SSFP Smiling Sun Franchise Program STQ Short-term Questionnaire SUCCEED Early Learning for School Success Program TAF The Asia Foundation UNFPA United Nations Population Fund USAID U.S. Agency for International Development USG United States Government YWCA Young Women’s Christian Association Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page iv EXECUTIVE SUMMARY The Leaders of Influence (LOI) is a four-year program designed to enhance the capacity of religious and secular leaders to contribute to: 1) national development, and 2) democratic reform efforts. Under this program, up to 20,000 leaders including more than 10,000 Imams (Muslim religious leaders) have received training and hands-on orientation to U.S. Agency for International Development (USAID) partner programs, including democracy and governance, gender equality, health, nutrition, family planning, HIV/AIDS, employment generation, and disaster management. LOI was designed to preserve and promote the values of democracy, tolerance, diversity, social harmony, and understanding in Bangladeshi society. The basic assumption underpinning the LOI program was that the knowledge of different aspects of development, gained through training and exposure in regional settings will help these leaders of influence to increase tolerance, dispel erroneous myths about the United States, and decrease the attractiveness of extremist appeals. Additionally, it was assumed that participating leaders, both religious and secular, are well placed in society to contribute to development efforts and to advance democratic values within their communities. Evaluation Purpose and Objective The evaluation purpose is to identify strengths and weaknesses in the LOI program. The objective is to assess key issues of relevance, effectiveness, management efficiency, impact, and sustainability of the program as it is due to be completed in April 2011. The evaluation findings will inform decisions to be made by USAID/Bangladesh about any similar follow-on program and the nature of its design. In keeping with USAID’s renewed focus on program evaluation, the evaluation seeks to provide “systematic, meaningful feedback about the successes and shortcoming” of LOI. Evaluation Methods The LOI Evaluation Team (comprised of two expatriate evaluation specialists and one Bangladeshi evaluation specialist) conducted the evaluation in January–March 2011, including three weeks in Bangladesh during February. Three methods were used to assess the LOI program. First, the Team conducted a thorough and on-going literature review of all relevant documents and reports made available to the Team to understand the operations of the LOI program. Secondly, The Team developed a semi￾structured interview guide for interviewing core implementing partners, facilitating partners, and individuals from participating USAID partner programs. Using the guide, the Team conducted a broad range of interviews, collecting data in a systematic manner for subsequent analysis. Third, the Team undertook field research at program implementation sites in order to observe its operation and develop a deeper understanding of key evaluation issues. These site visits provided additional opportunities for the Team to conduct key informant interviews and focus group meetings. Key Findings Relevance — When the LOI program was launched, it was assumed that Imams (and other religious and secular leaders) could play a major role as stabilizing figures in their communities — even serving as change agents to introduce new ideas and technologies. This assumption has proven to be correct, based on survey data as well as anecdotal information provided by implementing partners. Given the continuing leadership role of local Imams in communities throughout Bangladesh, there is reason to believe this intervention will continue to be useful. Moreover, involving local officials along with other secular leaders has provided a synergistic effect to the community revitalization process — all while improving the local social conditions, promoting democratic ideas, and reducing the risk that alternative, extremist views might gain popularity. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page v Effectiveness — The LOI program was successful in meeting its goal of training up to 20,000 religious and secular leaders and was able to adhere closely to the numbers estimated for each leader category as agreed to with USAID. In fact, it exceeded its targets by training a total of 20,743 participants, conducting 96 regional exchanges, and organizing 48 community outreach events. The LOI program did a good job of measuring numeric data on training, regional exchanges, and outreach events—referred to as “measures of effort.” Attempts to measure the effects of the aforementioned interventions were less successful. Perception surveys captured the level of attitudinal change that took place over the life of the project, but the team had to rely on anecdotal data to gain an understanding of how LOI interventions affected behavioral change within communities. Efficiency — The LOI management structure in the implementing organization, The Asia Foundation (TAF) was appropriate for effective relationships with the partner organizations that have been vital to the success of the program. However, the staffing level funded by this grant (only 3.5 persons in TAF to manage and oversee the entire program) placed a strain on staff and limited their availability to foster effective teamwork with some of the partners — especially those not based in Dhaka. Despite staffing constraints, a substantial multiplier effect was achieved based on the number of persons reached in communities after LOI orientation training, resulting in a very positive return on investment. Impact — LOI training, outreach, and exchange events reached a wide breadth of leadership types and a broad geographical distribution of participants and partners. Participants were exposed to a range of priority development concepts emphasized in the development strategy of the Government of Bangladesh (GOB) and in United States Government (USG) programs of development cooperation. They were able to visit a number of USAID-funded projects in the vicinity of their communities that were applying and advocating shared development concepts and demonstrating effective implementation. Participants and partners who were interviewed agreed that these visits provided valuable experiences. Sustainability — Interviews with the core, facilitating, and participating partners revealed a universal belief that continued assistance would be required over the next three to four years to ensure LOI impacts will be sustained beyond the USAID-funded period, which ends in April 2011. These partners also felt that LOI could become more self-sustaining in some form as its results became better known in Bangladesh and in the international community. It is doubtful that this type of program could become completely self-sustaining from a financial standpoint. Nevertheless, its multiplier effect, low operating costs and contributions by many partners have shown it to be a positive investment. Conclusions The LOI program has been increasing knowledge and building capacity at the local level over the past four years, leading to a civil society increasingly prepared to embrace national development goals and democratic values. In spite of a few discordant comments, almost all of the partners interviewed expressed positive views about the value of the program. They were unanimous in their desire that this type of program be continued in some fashion. Overall, the performance of LOI was strong in achieving its targeted outputs although the systematic measurement of the impact of the program was not fully achieved. • All LOI activities have been consistent with and contributed to the USAID democracy and governance (DG) country strategy’s Intermediate Result 1.4 to build a “vibrant civil society promoting democratic development.” • LOI has exceeded its target of training 20,000 religious and secular leaders — a total of 20,743 participants have been trained (10,787 Imams, 821 mosque committee members, 2,014 other religious leaders, and 7,121 secular leaders). • Given the relatively small amount of funding ($US 3.6 million), LOI’s return on investment was very positive. At a minimum, for example, each Imam serves as a disseminator to five mosque Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page vi committee members and quite likely many more persons in the community. The “multiplier” effect was thus at least five times and probably higher. • LOI had widespread support for its activities among partners, especially with Imams. All of the partners interviewed were positive in their praise for LOI goals and activities. None of the partners interviewed felt the program was sustainable at this time or in the foreseeable future. Several speculated that it would take at least three to four years for enough support to be generated to sustain this type of program. • Considerable thought and effort went into conducting a baseline survey and an end-of-project (EOP) survey. The findings clearly validated that LOI participants changed their perceptions in their appreciation of tolerance, diversity, and the role of religious leaders in community affairs. • The monitoring and evaluation (M&E) procedures for documenting LOI performance were stronger on paper than in their actual implementation. Documentation of the views of LOI participants on their perceptions of the value of the training they received was only partially completed. As a result, TAF was unable to provide documented analysis of the short-term questionnaires (STQs) and the response rate for returning the long-term questionnaires (LTQs) was so low that virtually no data results were evident. • TAF staff were overextended in coordinating, overseeing and appraising all activities. This appears to have limited opportunities for visiting, supporting, and providing and receiving feedback, apart from those events that TAF conducted jointly with partners. Recommendations 1. USAID should consider continuing the activities conducted under LOI with a continued emphasis on training participants in areas consistent with the GOB national development strategy and with the priorities of USAID’s country development cooperation strategy. 2. A future LOI program should build on the experience of LOI to date. In addition to new participant leaders and communities, a future program should consider enhancing the experience and multiplier effect of past activities. For example, intergroup workshops might be extended for an additional day or follow-up workshops could be organized to sustain communication and reinforce understanding among participants. 3. The value of site visits to projects could be enhanced further by including the GOB development activities and relevant activities supported by other donors in the local areas. 4. Future efforts like the LOI program should be based on rigorous contractual requirements that more closely link tasks, outcomes, and results to projected costs, based on thoroughly documented budget procedures. 5. Improved procedures should be developed to document what activities are undertaken by LOI participants after their orientation training once they return to their communities. Far greater attention is needed to document behavioral changes of participants along with capturing changes in perception. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 1 LEADERS OF INFLUENCE EVALUATION I. INTRODUCTION This report provides an evaluation of the USAID program in Bangladesh for Leaders of Influence (LOI), a $US 4 million, four-year effort to promote values of democracy, tolerance, diversity, social harmony, and understanding in Bangladeshi society. USAID Bangladesh requested the evaluation to assess the impact and lessons learned from LOI for consideration in the design of a follow-on program, if appropriate. The report describes the background context, the design and implementation of the program, and the purpose and methodology of the evaluation. It then sets out the evaluation team’s findings and conclusions, identifies lessons learned, and sets out recommendations for consideration by USAID. The evaluation took place in January–March 2011. The Statement of Work (SOW) is included in Annex 1. The evaluation team consisted of Dr. William Millsap (team leader), Stephen Strobach, and Samier Mansur; biographical summaries of the team members are provided in Annex 2. A bibliography of principal sources used in the evaluation team’s research is in Annex 3 and a list of persons interviewed in the course of the evaluation is in Annex 4. A. The Country and Its People Bangladesh is located at the northeast of the Indian subcontinent, at the northern edge of the Bay of Bengal. Its land boundaries are with India to the west, north and east, and with Burma to the southeast. The national territory of about 144,000 square kilometers (about the size of the state of Iowa or of England and Wales) is mostly a flat, alluvial plain. The land is traversed by the confluence of the Ganges, Brahmaputra, and Meghna rivers and is subject to extensive flooding during annual monsoons. Bangladesh is the world’s most densely populated country (leaving aside the urban enclaves of Malta, Monaco, and Singapore), with a population exceeding 160 million.1 It has the seventh largest population of all countries, and the fourth largest Muslim population (behind Indonesia, Pakistan, and India). While Bangladesh is experiencing urban migration, about 70 percent of the people still live in rural areas. The population is consists predominantly Bengali ethnicity; almost 90 percent of this population state they are Muslims, while an estimated nine percent declare themselves Hindus and the remaining one percent Buddhist, Christian or members of other religions. Life expectancy in Bangladesh is about 68 years, with approximately 80 percent of the nation’s population under the age of 40. Current per capita income is about $US 750. Life for the average Bangladeshi is a continuous struggle to overcome economic hardship, with about 40 percent of the population living in poverty. Adult literacy is estimated to be approximately 60 percent. Although male literacy is slightly higher, Bangladesh has now achieved gender parity in school enrollment, which is diminishing the gender-based discrepancy in literacy.2 1 With a population of 160 million, Bangladesh has a population density of 2878 persons per square mile (i.e., 144,000 sq. kilometers. = 55,598 sq. miles). 2 There is some variance among national and international sources of economic and social data about Bangladesh. See the website of the Bangladesh Bureau of Statistics, http://www.bbs.gov.bd; World Bank Country Data for Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 2 Bangladesh’s path to sovereignty was marked by political struggle and periods of intense violence – especially at the time of partition and independence from British rule in 1947 and again in the 1971 conflict for independence from Pakistani sovereignty. The new nation of Bangladesh emerged in December 1971 with many challenges and few resources. Its initial 20 years were characterized by recurrent political upheavals and instability. However, especially over the past two decades, the country has experienced impressive development progress. GDP has tripled, with a doubling of per capita income and dramatic gains in life expectancy, poverty reduction, literacy, and child survival. Bangladesh is among the top performers in human development and is on track to meet several of the Millennium Development Goals.3 The economy continues to grow at a rate of close to six percent and is rapidly evolving as the rate of growth in the industrial sector (primarily in apparel assembly) is more than twice that of agriculture. B. Improving Governance Advances in economic and social conditions have taken place despite persistent low rankings on indicators of good governance. For example, the World Bank Governance Index places Bangladesh in the bottom 25 percent of countries for political stability, government effectiveness, regulatory quality, and control of corruption, with deterioration in all these categories from its rankings in 1998.4 This divergence between economic and social progress, on the one hand, and continued political polarization and weak governance, on the other, has been described as Bangladesh’s development paradox.5 There is broad agreement that Bangladesh needs to continue, and even accelerate, its economic and social progress over the next decade if it is to realize the aspiration of becoming a well governed, democratic, middle-income country of opportunity by 2021, the 50th anniversary of its independence. There is also broad agreement that sustained economic and social progress will require that Bangladesh respond effectively to the challenge of improving the quality of governance. Highly credible elections in December 2008 have provided a broad base of support for an ongoing transition to renewed democratic values and, with it, an opportunity for Bangladesh to reform political practices and institutions of governance. Promoting good governance features prominently in the country’s development planning. 6 The international community strongly supports this objective. In Bangladesh, http://data.worldbank.org/country/bangladesh; World Fact Book, 2011, https://www.cia.gov/library/publications/the-world-factbook/geos/bg.html. 3 See “MDGs and Bangladesh,” UNDP, http://www.undp.org.bd/mdgs.php. A recent study estimates that more than 40 million people in Bangladesh will escape from extreme poverty in the period 2005–2015. Chandry, Laurence, and Geoffrey Gertz, “Poverty in Numbers: The Changing State of Global Poverty from 2005 to 2015,” Brookings Institution, January 2011, http://www.brookings.edu/papers/2011/01_global_poverty_chandy.aspx. 4 See “Country Data Report for Bangladesh: 1996–2009,” in Worldwide Governance Indicators, 2010, World Bank, http://info.worldbank.org/governance/wgi/pdf/c20.pdf. See also the Millennium Challenge Corporation’s Bangladesh Scorecard for 2011, which compares Bangladesh with other low-income countries, http://www.mcc.gov/documents/scorecards/score-fy11-bangladesh.pdf. 5 See, e.g., Mahmud, Wahiduddin, Sadiq Ahmed, Sandeep Mahain, “Economic Growth and Governance: The Political Economy Aspects of Bangladesh’s Development Surprise,” Commission on Economic Growth and Governance Working Paper 22, 2008, http://www.growthcommission.org/storage/cgdev/documents/gcwp022web.pdf; World Bank, “Governance and Growth: The Bangladesh Conundrum,” in Bangladesh: Strategy for Sustained Growth, July 2007, pages 125–136, http://siteresources.worldbank.org/SOUTHASIAEXT/Resources/Publications/44813-1185396961095/4030558- 1185396985915/fullreport.pdf. 6 See “Strategy II: Promoting Good Governance,” in Steps Towards Change: National Strategy for Accelerated Poverty Reduction II (Revised) FY 2009-2011, pages 71–77, Government of the People’s Republic of Bangladesh, December 2009; “Economic Growth and Corruption Free Good Governance,” in Outline Perspective Plan of Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 3 particular, support for pluralistic and responsive governance is a top priority for the United States in its relations with Bangladesh. These priorities are reflected in USAID’s country development cooperation strategy and exemplified in the LOI program. Bangladesh 2010–2021: Making Vision 2021 a Reality, pages 12–16, Government of the People’s Republic of Bangladesh, June 2010 (DRAFT). Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 4 II. EVALUATION PURPOSE USAID Bangladesh requested this evaluation in order to assess the impact and lessons learned from the LOI program. Accordingly, evaluation findings will inform decisions to be made by USAID/Bangladesh about any follow-on program and the nature of the design. In keeping with USAID’s renewed focus on program evaluation, the evaluation seeks to provide “systematic, meaningful feedback about the successes and shortcomings” of LOI.” As delineated in the SOW, the evaluation objectives are to:7 • Test the technical and programmatic validity of the hypothesis on which LOI was designed; • Assess the relevance, effectiveness, management efficiency and cost effectiveness, impact, and the sustainability and client satisfaction of LOI implementation in meeting program objectives; • Examine the manner and extent to which LOI has addressed cross-cutting issues of gender, inclusion, and transparent and accountable governance; • Evaluate the performance of USAID and its implementing partners to date, comparing results achieved with performance goals and indicators, and identify lessons learned; and • Make recommendations to USAID Bangladesh. A. Leaders of Influence Overview LOI is a four-year program designed to enhance the capacity of religious and secular leaders to contribute to national development and democratic reform efforts. In building this capacity, LOI seeks to preserve and promote values of democracy, (i.e., tolerance, diversity, and social harmony) and increase understanding and appreciation of these values in Bangladeshi society. This program provided training and exposure to USAID partners’ programs including democracy and governance, gender equality, health, nutrition, family planning, HIV/AIDS, employment generation and disaster management for up to 20,000 leaders. The basic assumption underpinning the LOI program is that the knowledge of different aspects of development, gained through training and exposure in regional settings, will help local leaders of influence to increase tolerance, dispel erroneous myths about the United States, and decrease the attractiveness of extremist appeals. USAID also assumed that these leaders of influence, both religious and secular, were well placed in society to contribute to the development efforts and advance democratic values within their communities throughout Bangladesh. B. Evaluation Results The evaluation reviews, analyzes, and evaluates the LOI program. It is guided by assessment criteria cited in the SOW 8 and, where applicable, identifies opportunities and makes recommendations for improvement. In answering the evaluation questions, the Team assessed both the performance of USAID and that of the implementing partner(s). 7 See the SOW in Annex 1. 8 Ibid, pages: 3–4. These criteria include; relevance, management and administration, cost effectiveness, impact, sustainability, cross-cutting issues, synergy with other USAID and donor funded programs, and performance measurement systems. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 5 III. THE DEVELOPMENT CHALLENGE & USAID’S RESPONSE The last decade has proven to be a formidable challenge for the GOB. The economy has grown at a consistent pace and many have benefited from this growth. Yet, considerable economic inequities are in evidence and numerous observers cite weak governance as the greatest obstacle to broad-based economic and social development due to partisan political tensions, widespread corruption, public administrative malaise, inefficient regulatory mechanisms, public security concerns, continued centralized administration of local governance, and a lack of public participation in decision-making.9 A. Development Challenge Addressed by the LOI Program After the terrorist attacks on the United States on September 11, 2001, and the subsequent military responses in Afghanistan and Iraq, there was concern that these events would be exploited by extremist elements within Bangladeshi society. It was in this context that the United States Embassy in Dhaka began to increase its efforts to work informally with the Islamic Foundation on shared interests. Building on these initial efforts, USAID supported an incipient effort (Sept. 2004) by The Asia Foundation (TAF) to introduce Imams at the Islamic Training Academy to a broad range of perspectives regarding development throughout South Asia. Called the “Leaders Outreach Initiative,” this program exposed Imams to principles of development, democratic governance and citizen participation through an orientation program and exchange visits to South-Asia countries.10 In 2005, Bangladesh experienced a series of bomb attacks by a domestic organization (the Jama’atul Mujahideen).11 These acts forced the GOB to acknowledge that extremist elements were at work and needed to be addressed. One of tasks carried out under the Leaders Outreach Initiative was a national Islam study seeking to examine the perceived relationship between religious/moral values and political integrity. The findings revealed that: A greater than expected support for the prospect of Bangladesh becoming an Islamic State . . . While the overall findings affirmed the conventional view that Bangladesh remains more secular, pluralistic, and tolerant than most countries with large Muslim populations, the risk remains that growing public frustration with partisan political tensions, weak governance, and economic burdens could lead disillusioned young people to seek alternatives that—in espousing violence, resisting the empowerment of women, or opposing democratic institutions—could undermine values of democracy, religious tolerance, pluralism, and social harmony.12 The possibility that extremism was on the rise in Bangladesh was reflected in the USAID Country Strategy for 2005–2010. One of the key Strategic Objectives (SO11) identified was to promote “more effective democratic institutions and practices.”13 Directly confronting the issue of denying space for potential terrorism, USAID approved modest funding of $US 3.6 million for a Leaders of Influence (LOI) 9 See “Country Data Report for Bangladesh: 1996-2009,” and the Millennium Challenge Corporation’s Bangladesh Scorecard for 2011, note 5, supra. See also Blair, Harry, “Party overinstitutionalization, contestation, and democratic degradation in Bangladesh,” in Brass, Paul R., editor, Handbook of South Asian Politics, Routledge, 2010, pages 98–117. 10 This effort operated from Sept. 2004 through Sept. 2006. 11 LOI Agreement, The Asia Foundation, April 15, 2007 Final Document. 12 See Leaders Outreach Initiative, USAID Cooperative Agreement #388-A-00-04-00113-00, National Perception Study on Islam in Bangladesh. 13 See USAID Strategy 2005–10, USAID/Bangladesh, Dhaka. September 2005; the other strategic objectives were SO12 Expanded Economic Growth, SO13 Better Educated, Healthier Population, and SO14 Improve Food Security and Disaster Mitigation. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 6 program for the period of April 16, 2007 through April 15, 2011.14 This program was designed to address the development challenge of confronting and mitigating the rise of extremism in Bangladesh. B. The Leaders of Influence Program LOI was implemented from April 2007 through April 2011 under a Cooperative Agreement with TAF. It was designed to enhance the capacity of religious and secular leaders to contribute to: 1) national development, and 2) democratic reform efforts. LOI promotes values of tolerance, diversity, social harmony by training religious and secular leaders on various aspects of national development and providing opportunities for them to visit various participating USAID partner programs. The operating premise is that an expanded base of informed citizens will make people aware of the diversity that exists in Bangladeshi society, leading to increased tolerance and social harmony and thereby diminishing the attractiveness of extremist appeals. Furthermore, USAID assumed that local leaders of influence, both religious and secular, were well placed in society to contribute to the development efforts and advance democratic values within their communities throughout Bangladesh. Under this program, up to 20,000 leaders (half religious leaders and half secular leaders) were to receive training and exposure to USAID partners’ programs including democracy and governance, gender equality, health, nutrition, family planning, HIV/AIDS, employment generation and disaster management. 1. LOI Antecedents As noted above, the Leaders Outreach Initiative (2004–2006) was funded by USAID as a grant for a total ceiling amount of $US 656,022.15 TAF had submitted an unsolicited proposal with three major activities taking into consideration growing concerns by the GOB and international donors regarding extremist elements emerging in Bangladeshi society. These activities included; 1) developing an Imam Activity Program, 2) conducting an Islam Study, and 3) carrying-out a South-Asia exchange program. TAF developed orientation materials for Imams that focused on raising awareness of the principles of development and democratic governance at the grassroots level, and in promoting greater citizen participation in public decision making processes. 16 During this period, orientation sessions were conducted with 5,000 Imams at the Islamic Training Academy, a National Perception Study on Islam in Bangladesh was completed (April 2006), and a limited number of regional exchanges took place. Based on this initial outreach effort, an Activity Approval Amendment recommended continuing the program — subsequently renamed the Leaders of Influence program. 2. LOI Activities Refocused The Activity Amendment identified three areas of adjustment; 1) focusing more on the development challenges facing Bangladesh, 2) expanding the activities to include not only Muslim leaders but also leaders of other religious groups as well as community and professional leaders of influence, 3) incorporating the successful experience (e.g., anti-trafficking, anticorruption, gender equality, youth empowerment) of others working with religious and community leaders. Most importantly, the amendment called for extending LOI for four additional years (2007–2011) to run concurrently with USAID’s Country Strategy 2005–2010 in support of SO11 promoting “more effective and responsive democratic institutions and practices.” TAF, which had administered the Leaders Outreach Initiative, was awarded a four-year Cooperative Agreement to implement LOI for the period starting on April 16, 2007 14 Implementation was made through an award to The Asia Foundation in April 2007 of $US 3,598,582, to be complemented by a TAF cost sharing commitment of $US 392,160 (total funding of $US 3,990,742). Refer to USAID/Bangladesh Cooperative Agreement #388-A-00-07-00029-00. 15 The grant sources were $US 457,000 in regional ESF funds and $US 200,000 from DA funds to be spent over a two-year period (Sept. 2004-Sept. 2006). See Leaders of Influence, Activity Approval Amendment, April 30, 2006. 16 Ibid. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 7 and continuing through April 15, 2011. Over this period, TAF projected the training of 20,000 religious and secular leaders of influence from all parts of Bangladesh who would be exposed to USAID program activities along with site visits to USAID partner organizations. The refocused and expanded LOI program goal was to mitigate any latent potential for extremism and promote national development goals — including democratic reform, health, food security, and energy security. C. Program Assumptions and Components The underlying logic of LOI is that building knowledge, through training and exposure to development activities, will lead to increased tolerance by participants. This premise is augmented by three sub￾premises. First, LOI participants are well-placed to contribute to national development efforts. Secondly, these leaders are positioned to offer a better understanding of the needs and interests of local communities. Thirdly, these leaders come from local community settings are also well-positioned to communicate with other members of their communities on national development goals and democratic values. Operationally, the LOI program consists of: • Conducting a three-day orientation training program for religious leader and secular leaders; • Promoting outreach activities with representatives of the print and broadcast media for producing articles or media segments on local development efforts for dissemination to a wider national audience; • Supporting international exchange programs; and • Implementing robust monitoring and evaluation procedures. TAF as the LOI program implementer worked with four core implementing partners to provide various types of services provided to the program (survey design and implementation, media contacts, training module development, and setting-up training venues). TAF also worked with over a dozen facilitating partners to carry-out LOI activities along with other participating USAID partner programs that were frequently visited by LOI participants to view USAID-supported activities. The names of these organizations are provided in Table 1. TABLE 1: LOI Partners by Type CORE IMPLEMENTING PARTNERS Imam Training Academy (Ministry of Religious Affairs [MORA]) Rupantar (Local elected officials) Democracy Watch (Democratic local governance) BCCP (Bangladesh Center for Communication Programs; materials and outreach events) Data International (Baseline and End of Project Surveys) FACILITATING PARTNERS Hindu Religious Welfare Trust; Buddhist Religious Welfare Trust; Christian Diaspora (CCDB and Caritas); Teachers, YWCA, Rotary & Lions Clubs, BCDJC (Bangladesh Center for Development Journalism & Communications), Chambers of Commerce, Women, Youth and Professional Associations SELECTED PARTICIPATING USAID PARTNER PROGRAMS Smiling Sun Franchise Program (SSFP), Family Health International, Save the Children USA, CARE Bangladesh, National Democratic Institute (NDI), International Republican Institute (IRI), Grameen Shakti, Rights Jessore, Winrock-MACH, and other partners as available Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 8 The Cooperative Agreement with TAF listed four program objectives to be carried-out over the four-year life of project.17 These program objectives are cited below along with the expected outputs that TAF expected to achieve. Objective 1: To engage leaders of influence in dialogue on and practical exposure to modern development practices and democratic values, including those to which USAID contributes through its support for democracy and human rights, economic prosperity, and investment in human capital. The target groups and the projected outputs are provided in Tables 2 and 3. Table 2 lists the respective religious groups to be engaged, and Table 3 displays the types of secular groups to be provided with exposure to democratic values. TABLE 2: Religious Groups by Projected Outputs Religious Leaders: Target Groups Projected Orientation Outputs Imams (Imam Training Academy) 8,000 (2,000 per year) Mosque Committee Members 800 (200 per year) Hindu Religious Leaders 2,600 (600-700 per year) Buddhist Religious Leaders 600 (100-200 per year) Christian Religious Leaders 200 (50 per year) Sub-Total 12,200 TABLE 3: Secular Groups by Projected Outputs Secular Leaders: Target Groups Projected Orientation Outputs Local Elected Officials 1,800 (450 per year) 50% women Community Service Clubs 400 (100 per year) 50% women Professional Leaders 400 (100 per year) 50% women Media Leaders 1,600 (400 per year) 50% women Youth Leaders 1,200 (300 per year) 50% women Women Leaders 400 (100 per year) Intergroup/Interfaith Leaders 2,000 (500 per year) Sub-Total 7,800 Objective 2: To facilitate dialogue and exchange of experience between Bangladeshi leaders of influence and their counterparts in other South and Southeast Asian countries. TAF projected that 48 leaders of influence would complete regional study programs and share their experience through outreach initiatives on return to Bangladesh with an equal number of leaders from other South and Southeast Asian countries visiting Bangladesh to interact with Bangladeshi counterparts. Objective 3: To promote broader public understanding of and dialogue on modern development practices, democratic values, and the respective contributions of leaders of influence and USAID program activities in advancing them. TAF expected that the projected 20,000 LOI graduates would serve as agents of change by communicating their experiences with others in their respective communities. Part of their orientation experience was to include practical guidance on how to effectively share and disseminate their knowledge and experience. TAF, in seeking a broader understanding by the general public, estimated that 17 USAID/Bangladesh Cooperative Agreement No. 388-A-00-07-00029-00, Leaders of Influence Program with The Asia Foundation signed April 16, 2007. See pages 22-24. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 9 36 outreach print and media campaigns would be launched to cover USAID programs along with 48 media outreach events convened to profile the LOI program in the promotion of development and democracy. Several community outreach campaigns were planned to raise popular awareness and drive media coverage. Objective 4: To monitor, analyze, and better understand the perceptions of leaders of influence, the communities they serve, and the broader population regarding national development issues and the societal values that underline traditions of democracy, tolerance, diversity, and social harmony. TAF also undertook to design and use rigorous monitoring and evaluation methodologies to measure process and outcomes in order to refine the LOI program implementation, and to conduct pre and post￾perception surveys to measure project impact. D. Democratic Governance and Institution Building In support of the Mission’s SO11, the office of Democratic Governance revised its Strategic Plan in December 2009 giving even greater emphasis to the development of democratic institutions.18 The new plan stresses the need for institutional development. Intermediate Result 1.4 calls for creating a “vibrant independent civil society promoting democratic values.” In recognition of this goal, the need to involve local leaders is expressly acknowledged. Sub IR 1.4.3 speaks to “enhanced engagement of leaders of influence in promoting values of democratic pluralism, social harmony and tolerance.”19 Based on the TAF Cooperative Agreement, the LOI program is graphically displayed below in Figure 1. The balance of this evaluation assesses how well the LOI program is working vis-à-vis Sub IR 1.4.3. Attention will be directed at the relationships between the design, intervention activities, outputs achieved, and outcomes in terms of performance and impact. Figure 1: LOI Results Framework Baseline Survey End of Project Survey 3 year+ Interval Effect on Initial Problem/Issue Problem Need PROJECT DESIGN Interventions M&E Plan Budget INTERVENTIONS Training Activities Outreach Activities Regional Exchanges Outputs • Training #s • Groups #s • Gender #s • M&E-STQs Outcomes • M&E-LTQs • Partner Rpts IMPACT 18 SO11 is slightly modified to “Effective and viable institutions of democratic governance” in lieu of “more effective democratic institutions and practices.” USAID/Bangladesh Activity Approval Document 2009–2014: Democracy Enhanced, Sustained, and Harnessed (DESH) Dec. 2009. 19 Id at page 13. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 10 IV. EVALUATION METHODOLOGY The LOI evaluation was conducted by two Expatriate Evaluation Specialists and one local Bangladeshi Evaluation Specialist. The evaluation strategy primarily relied on qualitative data collection approaches (i.e., review of relevant documents, interviews with the primary implementer staff [TAF] and other core partner personnel, and visits to field sites by the Evaluation Team). While the team did not have the time to collect qualitative data in the field, TAF did provide the team Statistical Package for the Social Sciences (SPSS) data files to verify the data provided in the Annual Reports on the baseline perception survey as well as the EOP survey findings. The Evaluation Team conducted its work in Bangladesh from February 3rd through February 21st . A. Evaluation Preparatory Phase The Evaluation Team began reviewing basic documents on January 26. Some of these initial documents included the USAID Country Strategy for 2005–2010, the 2009 Democracy Enhanced, Sustained and Harnessed (DESH) Strategy paper,20 and materials on recent USAID evaluation policy and guidelines. The team mobilized in Washington, D.C. on January 30 and departed for Dhaka on the January 31. B. Bangladesh Fieldwork The LOI Evaluation Team conducted initial meetings with the USAID Contracting Officer Technical Representative (COTR) overseeing the LOI program along with staff in the Democracy and Governance Office and corresponding USAID monitoring and evaluation (M&E) staff overseeing LOI implementation activities. After meeting with USAID, the Team developed a list of key persons to be interviewed in organizations that worked with the LOI program: 1) core implementing partners, 2) facilitating partners, and 3) participating USAID partner programs. 1. Literature Review The LOI Evaluation Team reviewed existing USAID/Bangladesh data on the LOI program and associated programs as appropriate for understanding cross-cutting issues. After initial briefings with USAID, the Team began a thorough review of voluminous key project documents (i.e., LOI annual work plans, M&E plans, quarterly and annual reports, and other relevant documents. TAF and the LOI partners provided additional documents covering their LOI activities over the last four years. 2. Field Research and Observation After developing a program of interviews, meetings, and site visits in Dhaka, two members of the team conducted a site visit to Khulna in southern Bangladesh. The Team met with representatives of Rupantar and a focus group of elected officials. Rupantar, a core participating LOI implementing partner, works with community groups and local government officials to improve their capacity in the area of governance. It has provided LOI training to 1,815 officials in three districts. In the field, the Team also met with Mosque Committee members who had been through the LOI orientation as well as two local Imams that had received training. In addition, the Team was able to observe some of the community projects promoted by these Imams undertaken by some of the local communities (e.g., fish ponds, poultry and dairy products). Thus, the field site visit enabled the Team to conduct interviews, hold group meetings, and directly observe some of the resulting activities underway at three different sites where local officials, Imams, and Mosque Committee member had been involved with the LOI training. The team made additional site visits to the Imam Training Academy (ITA) in Dhaka, an interfaith meeting of religious and secular leaders in Rajandrapur, and a USAID-supported health clinic being visited by LOI participants. 20 Ibid. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 11 3. Key Informant Interviews An interview schedule was developed concurrently with the document review with the aim of conducting interviews with key informants most knowledgeable with LOI. These interviews focused on addressing the evaluation questions set out in the SOW, with an emphasis on LOI issues of relevance, effectiveness, efficiency, impact and sustainability.21 Cross-cutting factors of gender, coordination with other programs, and transparency and accountability were also addressed during the interview sessions. The Team also carried out interviews with other individuals in organizations that had worked with the LOI program. To insure a degree of standardization in the interview process, the Team developed guides for interviewing core implementing partners, facilitating partners, and individuals from participating USAID partner programs. These guides focused on answering the following key questions consistent with the issues deemed important to investigate as set out in the SOW. Relevance—Were the technical and programmatic hypotheses of the program valid at the time of initiation and to what extent are they valid now? What is current relevance of the program to original and modified hypotheses? Effectiveness—How has program implementation by USAID and implementing partners met or failed to meet program objectives, performance goals, and indicators? Efficiency—Has the program been managed in an efficient and cost-effective manner? Coordination—How has program implementation addressed cross cutting issues (gender, inclusion of disadvantaged, transparent and accountable governance) and achieved synergies with other programs and development objectives? Impact—What have participants said about the benefits they received by participating in the LOI program and to what extent has this participation changed their involvement in their organizations. Further inquiry will be made into how LOI participating partner organizations are incorporating democratic values into their development activities. Sustainability—Are the direct beneficiaries satisfied with the program activities and has sufficient progress been achieved towards LOI program goals to justify continued client support? How sustainable is the program’s impact? C. Methodological Constraints In accordance with the SOW (page 5), the Evaluation Team has sought to identify the impacts of LOI to the extent feasible. However, LOI was neither designed nor implemented so as to meet the rigorous standards of the recently approved USAID Evaluation Policy. As that policy notes, “impact evaluations are based on models of cause and effect” and in essence can control for extraneous variables that might account for change other than the actual program intervention.22 Within the context of USAID policy, the present evaluation is in the nature of a performance evaluation—descriptive in nature and focused on how the program was implemented and how it was valued by the intended beneficiaries. It examines program design, management, and what results are observable. In short, a performance evaluation, while assessing impact as best it can, necessarily lacks the experimental rigor of an impact evaluation. The following section presents the Team’s findings in accordance with the key issues and evaluation criteria cited above. 21 See Annex 1. 22 USAID Evaluation Policy, Bureau for Policy Planning and Learning, January 2011, p. 4, http://www.usaid.gov/evaluation/USAID_EVALUATION_POLICY.pdf?020911. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 12 V. EVALUATION FINDINGS A. Relevance Were the technical and programmatic hypotheses of the program valid at the time of initiation and to what extent are they valid now? The decision to fund the LOI program appears to have been a very relevant response to the sporadic terrorist events of 2005. Indeed, the efforts to sensitize not only Imams but also other religious leaders and secular leaders to development issues and democratic values was quite appropriate given the concerns by the government that extremism was on the rise. Moreover, TAF’s “National Perception Study on Islam in Bangladesh” conducted as part of the Leaders Outreach Initiative documented a growing concern about the level of perceived corruption in the nation as well as a limited satisfaction with most public institutions. Interestingly, the study also found “an overwhelmingly positive view of religious leaders, particularly Imams at the local level.”23 While bombing and other terrorist activities have subsided, a potential for an increase in extremism remains. There continue to be public concerns about corruption and the effectiveness of government institutions. When the LOI program was launched, it was assumed that certainly Imams (and other religious and secular leaders) in their local villages could play a major role as stabilizing figures in their communities—even serving as change agents for introducing new ideas and technologies. The validity of this assumption has been confirmed by available data. Given the continuing leadership role of Imams and other religious leaders, there is reason to assume this intervention would continue to be useful. Moreover, involving local officials and other secular leaders has provided a synergistic effect to the community revitalization process—all while improving the local social conditions and thereby reducing the appeal of extremist alternatives. 1. What development challenge did the program respond to and how? The LOI program was designed mitigate social conditions conducive to promoting extremist activities and to broaden the base of support for development and democratic values. Intervention efforts to mitigate extremism included providing orientation training on national development issues and democratic values to religious and secular leaders along with conducting complementary community outreach events and regional exchange programs. 2. Has the Mission’s LOI approach achieved desired results? The LOI program’s overall goal was to provide new knowledge and skills related to Bangladesh’s development challenge to a target population of 20,000 religious and secular leaders from diverse communities throughout Bangladesh. LOI participants would then return to their respective communities to influence a wider population. Though never labeled as a “trainer of trainers” approach, the LOI program operated on the idea that a single trainee who was a community leader would be a multiplier of ideas—reaching a much wider audience with the knowledge learned. In this regards, LOI reached its goal of training 20,000 individuals—actually 20,743, 10,787 of whom were Imams.24 TAF was able to achieve extensive media coverage documenting LOI activities—indeed, the Bangladesh Center for Development Journalism & Communication (BCDJC) developed materials on LOI activities and conducted 20 orientations with 1,004 journalists.25 With respect to being relevant to USAID’s Country Strategy, the 23 Leaders Outreach Initiative, USAID Cooperative Agreement #388-A-00-04-00113-00, National Perception Study on Islam in Bangladesh, refer to Chapter 4, Public Institutions and Religious Leaders. 24 Leaders of Influence: Key Output Results, The Asia Foundation, “Summary of Results,” Feb. 13, 2011. 25 Interview with BCDJC personnel on February 8, 2011. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 13 LOI program was carrying out activities consistent with the Democracy and Governance IR 1.4, “to build a vibrant civil society promoting democratic development.”26 3. How has the environment (political, economic, and social) evolved and how has the program responded? It is difficult to draw conclusions on changes in the overall sociopolitical and economic environment since the advent of the LOI program. This period has seen a change from caretaker to elected government and the economy has adjusted to a global financial crisis. However, the Team visited with the ITA in Khulna and learned about some of the following changes since some of their Imams had completed their LOI orientation:27 • Imams expressed appreciation for the opportunity to learn development concepts, receive practical information about projects, and have the opportunity to make contact visits to some of these ongoing activities. Concepts of gender, trafficking, transparency, and anti-corruption were specifically highlighted as valuable by Imams. • Imams are now able to talk openly about important issues like HIV awareness, gender rights and health, and trafficking since they have access to the information and resources to guide them. • LOI training led to the formation of anti-drug, anti-trafficking community policing. In speaking with secular leaders at Rupantar (an organization that works with local officials) one of the summary comments was: At beginning of this project, I was confused somewhat about the program’s assumptions. Following 3-4 meetings with TAF, I realized the program was well-conceived. Before participating in LOI, Local Elected Officials or Bodies (LEBs) lacked information and had negative ideas about the USA. But given information and opportunities to visit projects, they now see the value of these USAID programs and ideas. LEBs are opinion leaders and it’s valuable to include them in the LOI program.”28 The above comments are typical of the positive responses the team heard from virtually all LOI participants and partners in training sessions and during the field visits. The LOI program has emboldened local Imams, local officials, and other religious and secular leaders to embrace new ideas not only in the context of technical change but also behavioral change—greater participation of women in community life and more interaction between religious leaders and local officials. The net effect has been a greater openness to diversity and tolerance in support of social harmony. 4. How appropriate was the selection of program components and participants? The selection of LOI program participants was consistent with the projected outputs by group type (refer to Tables 2 and 3, above). The actual selection of participants was left to the facilitating partners working with TAF, with TAF normally playing only a minor role. The only point of discord heard by the Team was from the Ministry of Religious Affairs (MORA). The Ministry felt it should have been more involved in the overall LOI program operations with TAF. Program components in terms of training materials were decided in consultation between the Bangladesh Center for Communication Programs (BCCP) and TAF. Currently there are nine training modules that were develop for the Islamic Training Academy. These include segments on health, education, economic activity, alternative energy, youth empowerment, gender equality, human rights, national development goals, and environment and climate change. The ITA has expressed interest in more modules in areas like anti-corruption and trafficking. 26 See USAID/Bangladesh Activity Approval Document 2009–2014: Democracy Enhanced, Sustained, and Harnessed (DESH) Dec. 2009, p. 13. 27 Personal Interview with representatives at the Imam Training Academy, Khulna February 9, 2011. 28 Personal Interview conducted with Rupantar representatives in Khulna, February 8, 2011. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 14 5. What is the current relevance of the program to the original and modified hypothesis? As noted in the introduction to this Findings section, it was assumed that Imams and other religious leaders could play major roles as stabilizing figures in their communities—even serving as change agents for new ideas and technologies. This assumption is still valid—the overwhelming majority of organizations interviewed were in agreement with this proposition. Over the course of the LOI program, this basic premise underlying the program’s operations has not changed. B. Effectiveness How has the program implementation by USAID and implementing partners met or failed to meet program objectives, performance goals, and indicators? 1. What objectives, performance goals, and indicators were established at the outset? How have they changed? As illustrated in Figure 1 above, there were three interventions to be carried out by the LOI program; 1) training activities, 2) outreach activities, and 3) regional exchanges. TAF further elaborated on its anticipated intervention efforts in its Baseline Report for 2008.29 • To train 20,000 leaders from diverse communities, demonstrating new knowledge and skills related to the development challenges and the opportunities provided through USG assistance in their respective communities and beyond. • To conduct outreach activities to the wider populace of Bangladesh on the values of tolerance, diversity, social harmony and understanding through the influence that leaders oriented under the program have on them, and through a robust media and public events campaigns highlighting opportunities provided through USG assistance in development and democracy. • To enable Leaders of Influence across South and Southeast Asia have met on a regular basis and share ideas on USG programs to promote development and democracy in their respective countries. The following outputs were achieved based on the latest data provided by TAF as of February 23, 2011. Final training outputs are displayed in Tables 4 and 5. In both tables, note that column one represents the original training projections in the TAF Cooperative Agreement with USAID.30 Column three of both tables shows revisions based on later agreements with USAID. 31 TAF also established a goal of conducting 48 LOI Regional Exchanges—including 48 inbound as well as 48 outbound participants. This target was achieved, both for inbound and outbound participants, for a total of 96 participants. No specific target was set for community outreach events. As of this writing, 48 community outreach events were completed. These targets were not revised over the life of the project. TAF’s efforts to recruit and train participant in development issues and democratic concepts were fairly consistent over the life of this four￾year project. This consistency is demonstrated in Annex 5 that displays LOI trainee recruiting numbers by group over four years. 29 LOI Program: USAID Cooperative Agreement No. 388-A-00-07-002900. LOI BASELINE REPORT, 2008:4. 30 Ibid 31 The projected number of religious groups to be trained was adjusted in a Memorandum of Understanding between the MORA and USAID/Bangladesh dated March 9, 2008. Adjustments to the number of secular leaders to be training were revised in accordance with TAF’s Performance Management Plan (actually their M&E Plan) submitted to USAID/Bangladesh on May 6, 2008. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 15 TABLE 4: Religious Groups Projected Outputs by Results Achieved Religious Leaders Original Projected Outputs Revised Projected Outputs Results Achieved* Female Participants (%) Imams-Islamic Train Academy 8,000 10,000 10,787 -- Mosque Committee Members 800 800 821 -- Hindu Religious Leaders 2,600 1,500 1,537 47 Buddhist Religious Leaders 600 300 300 30 Christian Religious Leaders 200 200 173 43 Sub-Totals 12,200 12,800 13,622 *TAF exceeded its projected target by 822 religious leaders, an increase of 6.4 percent. TABLE 5: Secular Groups Projected Outputs by Results Achieved Secular Leaders Projected Outputs Revised Projected Outputs Results Achieved* Female Participants (%) Local Elected Officials 1,800 (Same) 1,815 49 Community Service Clubs 400 200 184 44 Professional Leaders 400 900 806 28 Media Leaders 1,600 1,000 1,000 7 Youth Leaders 1,200 1,000 998 45 Women Leaders 400 500 520 94** Intergroup/Interfaith Leaders 2,000 1,800 1,798 23 Sub-Totals 7,800 7,200 7,121 *TAF missed its target of 7,200 secular leaders by 79, a shortfall of 1.1 percent. **Note: Curiously, 31 of the “women leaders” group were males 2. Performance Shortcomings While the aforementioned performance goals were achieved, TAF had developed STQs to evaluate the views of participants on LOI orientation training (see Annex 5). TAF informed the Team that 5,000 STQs had been completed and entered into a database (i.e., Excel files by training group) for analysis. However, the Team found only a few references summarizing some of the findings of LOI orientation training based on the STQs. The Team’s principal source of data, therefore, was comments made during interviews with numerous organizations.32 Equally disappointing was the absence of documentation to gauge the impacts on participants after they returned to their communities. We were advised by TAF that a useful Long Term Questionnaire (LTQ) had been developed and distributed but that the response rate was less than one percent. Of particular importance, there were questions that asked what participants actually did in their communities as a direct result of their LOI training. TAF had earlier called attention to the STQs and LTQs as important procedures for insuring the quality of orientation training. It stated in its first annual report (for 2007): To ensure the quality of orientations, the Foundation team closely monitored all sessions through the participation of team members in the orientation sessions, including the project site visits. The team also continuously analyzed and reviewed feedback from the 32 The schedule and names of the organizations are provided in Annex 4. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 16 questionnaires collected from participants to identify improvements for future orientation programs (see TAF LOI First Annual Report 4-07 to 3-08). In its Second Annual Report TAF stated that information from the STQs and LTQs was being entered in a data file, stating that a “detailed analysis of the STQs from a sample pool of 5,000 oriented LOIs will be conducted in Year Three.”33 However, as noted above, very little evidence of analysis of these data was provided in response to the Evaluation Team’s request. TAF reported to USAID that problems had been encountered. For example: There has been poor response from the LOIs oriented under the program with respect to the LTQs. This has necessitated some re-thinking on the usefulness of the LTQs, as statistically valid conclusions cannot be drawn from the small sample so far available. Discussion with USAID will be undertaken to devise suitable alternative monitoring tools for some of the outcome indicators.34 The last Semi-Annual Report, covering April through September 2010, acknowledged the same problem. Responses from participating leaders of influence on the long-term questionnaires (LTQs) have been poor. Essentially only the more pro-active trainees have responded in a timely fashion. To overcome these challenges, the LOI Team plans to build a more elaborative system with creating opportunities and awarding for the best performing LOIs in the next iteration of the LOI program. This will provide stronger incentives for the LOIs to report more frequently on their ongoing activities drawing from their LOI experiences. 35 When the Team inquired about this issue, TAF responded that the low response rate for the LTQ was probably attributable, as least in part, to the absence of sufficient program resources to provide participants with postage for returning the questionnaires. This appears to be a missed opportunity by TAF and USAID to collect systematic data about the LOI program’s impact on participants. 3. What baseline data were established with respect to those objectives, performance goals, and indicators? The M&E Plan, approved by USAID in May 2008, discussed measures of process or “measures of effort” that captured numeric data on training, outreach events, and regional exchanges. The baseline for these process data was set at zero with incremental targets predicated on annual targets reported semi-annually (refer to Tables 3 and 4, above). Again, as noted earlier, in the absence of STQ data for comparison with LTQ data, no baseline could be established with respect to training quality and/or effect. As shown in Figure 1, TAF proposed to assess the overall impact of the LOI program by conducting pre￾project and post-project perception surveys of leader of influence groups and their communities—referred to as “measures of effect.”36 TAF sub-contracted this effort to Data International (DI) which conducted a highly reliable baseline survey during the first year of the LOI program and an EOP survey in year four of the program. The survey was designed to assess the perceptions of those who had participated in the LOI program versus a sample of persons who had not been exposed to the LOI program. Essentially, the design focused on documenting attitudinal changes of those “with” the intervention (i.e., LOI orientation training) in contract to those “without” that intervention. Key areas of inquiry focused on probing participants’ attitudes on tolerance, diversity, social harmony, and the role of religious leaders. 33 TAF, LOI Second Annual Report 4/08-3/09:2. 34 Ibid. 35 TAF Semi-Annual Rpt. 4/2010-9-2010:39 36 The Asia Foundation: Leaders of Influence: Monitoring & Evaluation Plan, May 6, 2008. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 17 4. How adequate are the internal systems and indicators for tracking, monitoring, and reporting? TAF did a very credible job of capturing accurate information on “measures of effort,” that is, process data on training numbers, outreach events, and regional exchanges. The output data (i.e., measures of effort) were carefully tracked and monitored as reflected in TAF’s semi-annual and annual reports. As previously noted, apart from the baseline and EOP surveys, data apparently was not collected and analyzed that would permit meaningful tracking, monitoring, and reporting on “measures of effect.” 5. Has performance been measured to compare results with baseline data? Were the program objectives achieved? TAF’s M&E plan was to measure the overall impact of the LOI program via the implementation of perception surveys conducted during Years One and Four of the project. Two key questions were investigated in the survey. First, has the LOI Program changed the orientation and perception of leaders of influence who are exposed to USAID development programs, as compared to the orientation and perception that they would have held if they had not been exposed to LOI activities? Secondly, has the LOI program changed the orientation and perception of members of the general public who are influenced by leaders of influence trained under the program?37 Overall, the EOP report documented, on average, a positive shift on such issues as human rights, tolerance, and a positive view of Imams as changes agents of 14.2 percent. The shift in perception for ordinary citizens was, on average, 11.8 percent. Most impressively, the attitudes of Imams surveyed shifted 22.9 percent.38 The EOP survey demonstrates that the LOI program did change perceptions. However, since there was a three year interval between the initial baseline survey and the EOP survey, there were undoubtedly other extraneous factors, not addressed in the EOP report, that affected the reported changes in perception. At the same time, the survey results are fully consistent with the anecdotal evidence the Team received in its extensive interviews and focus group meetings. It is fair to conclude that the LOI program did play an important role in changing the perception of LOI program participants. 6. How have program components, individually and in relation to each other, contributed to results? The program components—ranging from the LOI interventions to measuring outputs (e.g., training, group types, gender, etc.) —worked effectively. The baseline and EOP surveys were innovative efforts to document the results of the LOI program in terms of changes in perception. Regretfully, the Team only found anecdotal data on behavioral change attributable to those changed perceptions. 7. Were there other results—intended or not, positive or negative? Based on the documents reviewed and interviews conducted with core and facilitating partners, the Evaluation Team found no negative unintended consequences. However, a positive unintended consequence reported by CARE and Smiling Sun Franchise Program (SSFP) was that, “as a result of the positive reports and referrals by LOI religious leaders (particularly Imams), they have decided to invite LOI participants to visit their respective programs throughout Bangladesh.” 8. What major factors influenced achievement of program objectives? Two major factors stand out. First, the orientation training was well-designed and well-received by religious and secular leaders. The inclusion of site visits to various USAID supported projects provided especially valuable information, based on the comments of various participants cited in TAF semi-annual 37 LOI End of Project Evaluation Report, January 2011: 8. 38 See Annex 5 for a summary of “Persons Surveyed Responding Positively to Democratic Values of Tolerance, Diversity, and the Role of Religious Leaders. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 18 and annual reports and confirmed in our interviews with both partners and participants. Second, in achieving its objectives—especially in terms of process outputs—much of the credit has to be extended to the core and facilitating partners that TAF worked with on implementing the LOI program and carrying out its many activities. Given the limited number of TAF staff working on the LOI program, much of the burden of implementation and attendant costs was shared by these partners. C. Relevance and Effectiveness How has program implementation addressed cross-cutting issues (gender, inclusion of disadvantaged, transparent and accountable governance) and achieved synergies with other programs and development objectives? Gender empowerment, along with the inclusion disadvantaged persons, and transparent and accountable governance were overarching themes that the LOI program sought to address. Particular issues included: human rights; good governance and anticorruption; health and family planning; education; gender equality and women’s empowerment; tolerance and a terror-free environment; renewable energy; youth development; HIV/AIDS awareness; anti-drug and anti-trafficking; and disaster management. Orientation workshops provided training on these topics to 13 categories of leaders: Muslim, Hindu, Buddhist, and Christian religious leaders; mosque committee members; local elected officials; print and broadcast journalists; youth, women, and professional leaders; community service club members; and intergroup and interfaith leaders. 1. How does the program relate to development priorities identified by the Government of Bangladesh, USG, other informed analysis and perspectives? The majority of the themes addressed by the LOI program track closely with the National Development Strategy of Bangladesh, the core of which is attainment of the Millennium Development Goals (MDGs).39 Also, LOI was designed to substantially address USAID’s Strategic Objectives for 2005-2010 and it has remained current, as well, for DESH, USAID’s democracy and governance plan for 2009–14, and the emerging new country development cooperation strategy. Participants in thirteen follow-up intergroup workshops run by Democracy Watch (DW)—which reinforce LOI orientations—cite becoming involved in a range of endeavors supporting GOB and USAID development priorities following their orientation workshops. Some examples: pre-primary education awareness; establishment of libraries for the youth; formation of committees to encourage youth employment; greater involvement of females in society; advocacy against child-marriage and the dowry system; platforms to discuss interfaith relations; and efforts to educate others on reproductive health and safe-motherhood by guiding them to local USAID partner clinics. Visits to and talks given by USAID-funded projects formed an integral part of the LOI training. The various groups interviewed, including the partner programs visited by trainees, cited the importance of this LOI component. By complementing the in-class training with site visits to health clinics (SSFP, Family Health International), pre-primary education institutions (Save the Children’s SUCCEED program), renewable energy service providers (Grameen Shakti), or anti-trafficking groups (Rights Jessore), to name a few, trainees were exposed to a variety of service providers that: 1) corresponded well with the training modules, and 2) increased exposure of the programs to local communities, thereby increasing their utilization and contribution to local development. 39 See Steps Toward Change: National Strategy for Accelerated Poverty Reduction II (Revised) FY 2009-2011, Government of the People’s Republic of Bangladesh, December 2009. The completion and publication of a new national development strategy is reportedly imminent. See “Bangladesh country paper on progress made in developing and implementing MDG-based national development strategies,” United Nations Economic Commission for Asia and the Pacific, December 2010, http://www.unescap.org/pdd/calendar/CSN-Cambodia￾Dec10/Papers-Presentations/Bangladesh.pdf. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 19 This synergy further increased the visibility throughout Bangladesh of USAID projects. For example, according to the numbers provided by one project holder—SSFP—the estimated number of yearly customers referred by LOI participants was 27,300, 40 and the overwhelming majority of these new customers were women. (SSFP’s 27 clinics treated over 22 million patients nationally in 2009 and 21 million in 2008 of which 90 percent were women. 41) The leadership of the Islamic Foundation, ITA, and Masjid Council for Community Advancement all related information that Imams were so impressed by the quality of SSFP’s services, and by the transparency and affordability of treatment, that they advertised the presence of local branches to their communities. The Hindu Religious Welfare Trust (HRWT) and the Buddhist Religious Welfare Trust (BRWT) provided similar anecdotes, and the Young Women’s Christian Association (YWCA) forged their own organizational relationships with SSFP after LOI training, working out pricing discount schemes for particular treatments. Rupantar, for their part, had initially assumed that local elected officials would be knowledgeable about USAID projects in their local areas and their services. However, experience quickly demonstrated that they did not possess this knowledge. After training, officials relayed the information they had obtained through open forums and community gatherings as a part of their public service duties.42 When asked why trainees received and relayed the information so readily, both the religious and non-religious participants answered that the training themes were both “practical” and “relevant” in terms of addressing Bangladesh’s most pressing development challenges. Another significant reported impact of LOI was the integration of religious leaders as agents of development. According to the Masjid Council for Community Advancement (MACCA), prior to LOI there was a deep divide between the development community (NGOs and international donor and developmental agencies) and the religious institutions of Bangladesh. The development community viewed the religious community as resistant to progress, being unwilling to address fundamental development issues. The religious bodies, in turn, expressed suspicion regarding the intentions of the development community, which they viewed as consisting of agents of Western influence. According to CARE, another partner agency involved with LOI, by engaging the religious leaders of the various faith groups, the program helped to bridge the divide. Religious leaders became more sympathetic and “friendly” towards the development sector, while the development organizations became more sensitive to the concerns of the religious community. The enthusiasm with which both religious and secular implementing partners continue to endorse the program is indicative of the relevance and perceived value of LOI in Bangladeshi society. 2. How has program fostered gender equality? To advance gender equality, LOI facilitating partners responsible for sourcing trainees attempted to achieve 50 percent female participation in all secular categories43 with the exception of the Imams and the mosque committee members, whose demographics are 100 percent male. Though extensive efforts were made to reach this target, female participation in the program (apart from the Imams and mosque committee members), stood at 38 percent. Despite the final female figures being lower than anticipated, it was significant in the context of Bangladesh. The Evaluation Team believes that the LOI program contributed to the empowerment and enhanced participation of women in society by 1) training community leaders on the increased social returns from the education of girls and women and the increased participation of women in society; 2) providing the necessary tools and encouragement for 40 From document entitled: Report on Customers Referred by LOI to Surjer Hashi [Smiling Sun] Clinic, January￾December 2010, presented to LOI Evaluation Team. 41 Smiling Sun Franchise Program Health Bulletin 2010 and 2009. 42 From a focus group discussion with six local elected officials from Khulna, who undertook LOI training organized through Rupantar. 43 TAF’s target, per the Cooperative Agreement 2007. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 20 women leaders to feel empowered enough to speak up in public forums and contribute actively in society; and 3) providing a platform of engagement during intergroup/interfaith exchanges where women leaders worked side by side with other secular and religious leaders to brainstorm effective strategies to tackle local development challenges. The YWCA commented that the LOI training sessions encouraged women to speak in front of the workshops, leading to an increase in self confidence, and a “feeling of empowerment to speak up on issues of social justice, health, sex, drugs, environment, and trafficking.” Also, workshops brought together women leaders from various locations and enabled them to form networks to address their concerns to enhance their work. 3. How has program contributed to broader participation by disadvantaged members of society? Creating social tolerance and harmony regarding the participation of disadvantaged members of society was another key goal of the LOI program. The HRWT and YWCA addressed issues of minor religious discrimination in Bangladesh. The HRWT ranked interreligious tensions between their community and the larger Muslim community at a 4 (based on a ten point scale; 1=lowest, 10=highest). The YWCA expressed concern about hidden tensions that could result in open hostilities, or violence and the BWRT expressed the least concern—citing minor issues, but overall ranking the relationship at 8.5 out of 10. All groups expressed the belief that there is relative social harmony between the religious groups, but more could be done. As such, they all referred to LOI’s intergroup/interfaith workshops as an effective strategy toward enhancing religious tolerance and pluralism in society. The interaction by the different minorities of Bangladesh with the majority Muslim leaders facilitated the breakdown of barriers and the acknowledgement that the various faith communities face similar development challenges that are best met when working together. The MORA expressed enthusiasm for this particular aspect of LOI, hoping that it would continue and expand in scope in the future. 4. How has the program integrated values of transparency and accountability? Trainees participating in intergroup workshops also provided examples of their post-LOI involvements that specifically address themes of transparency and accountability. The cited examples included promoting open-forums on increasing fiscal transparency by local elected officials; and creating public information boards pertaining to good governance and open budgets. In addition, the Evaluation Team received reports of LOI participants who urged the collaboration of community leaders and elected officials to build basic infrastructure such as roads and sanitation systems and to enlist local law enforcement authorities to combat drug and human trafficking. 5. How has program advanced aid effectiveness principles (local ownership, alignment, harmonization, managing for results, mutual accountability)? Numerous LOI partners confirmed to the Team during interviews that trainees absorbed the information well and that many later reported disseminating LOI themes through their various public platforms afterwards. Religious leaders addressed these themes through sermons during services, religious festivals, and community gatherings at mosques, temples, monasteries and churches. According to the Islamic Foundation Bangladesh (IFB), which oversees the training of Imams through seven ITA centers throughout the country, Imams felt the information was practical in terms of addressing local developmental challenges, and felt it to be their moral responsibility to spread the information amongst their congregations. Whereas in the past Imams would restrict the role of women to the household, following LOI many report to partners that they are encouraging women to take employment opportunities while advocating responsible family planning, disseminating information on where to go in order to receive quality health treatment, speaking on HIV/AIDS awareness and prevention, and addressing the need to educate children Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 21 in pre-primary education programs. In one particular example, which is indicative of the multiplier effect of the LOI program, an Imam from Khulna was inspired by LOI to learn about women’s rights, anti￾trafficking, anti-terrorism, and promoting interfaith tolerance. Through his research, he developed scripts on these topics, and delivers them through a weekly radio show on the Radio Bangladesh channel, which he told the Team reaches 5–6 million people. Among minority religious communities of Bangladesh, similar outcomes are reported by leaders, according to the HWRT, the BWRT, and the YWCA, as well non-religious affiliated groups such as Rupantar. In each instance, the Team was told about increased awareness among the religious and secular leaders, which in turn translated into practical implementation of LOI themes at the local level. D. Efficiency Has the program been managed in an efficient and cost-effective manner? Efficiency and cost-effectiveness of the LOI Program can be measured in several ways. Before LOI, TAF had established collaborative relations with national and regional organizations working throughout Bangladesh and already engaged with the leader categories targeted for this program. This enabled inexpensive and efficient sourcing, recruitment and participation of these leaders. 44 As previously cited, TAF and its Partners were able to exceed the targeted total number of leaders trained in orientations, exceeding projections for some categories—notably Imams and Professional Leaders, but falling short in other categories, notably Buddhists, media and community service leaders, as provided above in Tables 4 and 5. 1. Is the management structure conducive to efficient and timely direction of the program? The management structure was appropriate for effective relationships with the partner organizations that have been vital to the success of the program. However, the staffing level funded by this grant—only 3.5 persons in TAF to manage and oversee the entire program—placed a strain on staff and limited their availability to foster effective teamwork with some of the partners—especially those not based in Dhaka. TAF was unable to monitor some LOI activities or give feedback to its partners. Inadequate staffing was probably also a factor in the missed opportunity to assess the impact of training through the use of LTQs for participants. Beginning in the Second Annual Report in early 2009, TAF reported problems with the LTQs: Overall the response from participating imams and other LOIs on the LTQs have been poor…. To overcome these challenges, the LOI Team will, after consulting with USAID counterparts, undertake small sample surveys as well as spot visits to a pool of oriented Leaders.45 The value of this instrument and the difficulties in obtaining responses was repeated with identical words in subsequent semi-annual and annual reports. 46 The Year 4 Work Plan suggested that additional budget for LOI to provide trainees postage stamps and mailing envelopes, while the Year 4 Semi-Annual Report suggested an incentive system to encourage LOIs to report activities after orientations—“in the next 44 Refer to Table 1 for a list of partners that TAF worked with on the LOI program. 45 Year 2 Annual Report to USAID, Page 42. 46 Year 3 Semi-Annual Reports and Annual Report, pages 41 and 42, respectively. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 22 iteration of the project.”47 Nonetheless, effectively no corrective action was taken, according to TAF’s reports, and interviews with TAF officials.48 2. Does the management structure facilitate coordination with related activities, opportunities for leveraging and cost sharing, and achievement of synergies? The management structure and the extensive involvement of partners was very appropriate and set the basis for efficient coordination, leveraging, cost sharing and achieving synergies, by utilizing each partner’s particular strengths and audiences and complementing these with attributes other partners possessed. Visits to participating USAID funded projects also were very positive: trainees commented in STQs these exposures as a very valuable part of their experience, while in meetings with project officials the evaluation team was told of increased interest and support from religious and elected leaders, and new clients referred by participants. In contrast, LOI had very modest funding, compared to the quantity of outputs projected to achieve its stated objectives nationwide. As a result, TAF and its core and facilitating partners were underfunded and overextended. TAF cited its small staff dedicated to LOI and how this challenged their ability to lead and manage all aspects of the program. All TAF’s partners expressed their conviction that the LOI program is important for Bangladesh and valuable for both the trainees and their organizations. But most also commented that the funding provided proved inadequate to the responsibilities they undertook. This required them to make substantial in-kind contributions in terms of staff time and administrative expenses. While they like working with TAF, several felt they weren’t given enough information or feedback about overall progress of the program. And some facilitating partners expressed frustration at not having the opportunity to participate more in planning, implementation and program review. Also, had they received preparatory training, they could have a more active role as trainers in the orientation workshops. 3. Are the available implementation instruments adequate in achieving optimum results? The mechanisms of sub-grants and subcontracts were vital to the success of LOI: the partners had an established presence countrywide, long experience in working with the targeted leadership groups, and sometimes provided expertise beyond TAF’s capabilities; e.g., the surveys conducted by DI. Most had pre-existing collaborative relationships with TAF. It was efficient that TAF staff shared facilitation responsibilities in training orientations and inter-group meetings with several of their partners. Later on, some partners assumed principal responsibility for implementing some training activities. The biggest challenges to achieving optimum results were not at all implementation instruments, but rather limited staffing and funding, as described throughout this section of the report. 4. Were USAID requirements consistent with efficient and timely management and did USAID and implementing partners comply with those requirements? TAF diligently prepared and submitted detailed Program Plans, Semi-Annual and Annual Reports on time. Although much of the content was repeated verbatim in succeeding reports, these provided a good summary of the program to date. They informed on activities undertaken, outputs generated, coordination efforts and problem areas with partners. Likewise, a sampling of reports obtained from implementing partners were informative, provided output data, and had useful evaluative content and suggestions concerning the implementation. In each of TAF’s progress reports, the several components of LOI’s M&E plan were carefully presented, described in the same way as previously. However, beginning with the Year 2 Annual Report, poor return of the important LTQs was repeatedly reported in later pages, under “Monitoring and Evaluation 47 Id. At pages 8 and 39, respectively. 48 Meetings at TAF office, February 6 and 13, 2011. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 23 Challenges.” Corrective measures to be taken were described each time, but in effect nothing was subsequently carried out to date, other than to duly note this challenge in each subsequent report. 5. Has the implementation approach been cost-effective in achieving timely results? The implementation approach has been very cost-efficient in achieving all the outputs stated in the Cooperative Agreement, even surpassing the total projected number of LOI participants. Despite the very modest amount of $US 3.6 million donor funding and staffing level, as of March 7, 2011 the program has trained 20,743 participants in 294 orientation trainings throughout the country. Most of these trainees also visited USAID partner projects. The program was also cost efficient for USAID because it depended so much on the goodwill of TAF and its partners in achieving these ambitious numbers. In our meetings with the core and facilitating partners, most of them reported to the evaluators that the funding received did not cover their real costs; still, they contributed ad honorem to the achievement of objectives because they believe in the importance of the program to the country. Furthermore, a substantial multiplier effect was achieved in the LOI program resulting in a very positive return on investment given the low costs of reaching much larger populations. As leaders, participants were in a position to disseminate LOI ideas and information about USAID partner projects in the communities they serve and among other leaders. Each Imam, for example, works closely with an average of five Mosque Committee Members. Also, many Imams and other religious leaders informally tell LOI partners they include LOI social messages to attendants at religious services. Anecdotal information the Evaluation Team was able to gather (although a very small sample of the 10,787 participating religious leaders) indicated this is happening frequently. Similarly, as called for, 48 outreach events were conducted and 96 persons participated in international exchange programs, 48 traveling to other countries and 48 coming to Bangladesh from South and East Asian countries. 6. Might changes in management approach have realized more efficient, timely performance and/or cost savings? As discussed above, cost efficiency appears to have been maximized in this program, and no unnecessary costs are noted. However, as also observed earlier in this section, this economy in costs seems to have impeded adequate attention to aspects of the program relating to quality of interaction with some partners, and especially, attention to measuring impact. This will be treated in the following section of this report on conclusions and lessons learned. E. Impact What has been the impact of the program on the lives of direct beneficiaries and participants as well as on the achievement of broader policy objectives? It is early to measure differences in the lives of beneficiaries and the absence of hard data precludes measuring impact at this point. However, reports by TAF and its partners and anecdotal information cited in them, along with comments received during the Team’s numerous interviews and site visits provide significant data on outputs and evidence of other results. These outputs and results will be the basis of the following findings relating to impact. 1. What real impacts have been achieved in terms of differences in the lives of beneficiaries and the performance of institutions? Project reports provide important anecdotal evidence of changes in attitude and actions undertaken as a result of LOI orientations by a number of the leaders. The positive thrust of these anecdotes was consistently echoed in the Team’s meetings with the core partners, almost all the facilitating partners, and Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 24 a number of the participating USAID partners. 49 Participating leaders who spoke to the Team in the field, in focus groups, and at meetings uniformly attested to the positive influence on their subsequent attitudes and actions. Several organizations also commented to the Team how their experience with LOI aided their programs and influenced their approaches and key persons to engage. As previously cited, CARE found religious leaders now support their programs and SSFP found LOI participants to be important sources of referrals of modest income female clients they especially seek. 2. What has been the breadth of impact in the population and what are the prospects for scaling up to broaden the impact? LOI managed to include a wide breadth of leadership types and broad geographical distribution of participants and partners, despite the centralization characteristic of public administration in Bangladesh. This is an important achievement. Trainees were exposed to a range of priority development concepts given prominence in the development strategies of the GOB and the USG. They also visited and had talks with personnel at a number of USAID-funded projects applying and advocating community development concept. This type of project could certainly be expanded into a larger more extensive program, possibly by follow-up activities with the population already reached or, alternatively, by expansion to include additional participants. Bangladesh’s environment and traditional openness to diverse influences seem quite favorable to this kind of undertaking. Both governmental and civil society institutions that have been involved speak of LOI’s value and its importance for the country’s development. Almost all express a strong desire to continue participating and some would welcome an expanded role. The potential participant population of leaders is substantial, especially Muslim religious leaders and local government officials. Moreover, there are many potentially apt project activities to visit, including those supported by other donors. Recruiting female leaders for LOI will be a continuing challenge. Partners responsible for recruiting told of to the difficulties in reaching a 50 percent female participation rate despite their best efforts. Any scaling up of an LOI-type project will need to take this challenge into account and include measures to address it. 3. What principal factors have contributed to the impact achieved? A number of factors contributed to achieving the results and value of participating in LOI, as perceived by leaders. Among them: • The overall environment in Bangladesh, which appears to be very receptive to this type of intervention, combined with the great need perceived by many for the country to integrate democratic values and practices more fully in the development agenda. • Selection of strong partners for this kind of undertaking, with established, funded programs, audiences and presence throughout Bangladesh. • Good leadership and coordination by TAF, despite some shortcomings in follow-up communications and feedback, probably because staff is overworked. • Diplomatic, patient handling of sensitive relationships, especially by TAF. • Belief in the program and a spirit of cooperation among partners, even to the point of making personnel available pro-bono and providing unplanned organizational in-kind contributions to ensure success. • Receptive participants, many hungry for the opportunity to learn, who are eager to visit projects, work with colleagues, and apply what they learn. 49 See Table 1, page 9 for a list of these partners, and Annex 4 for the listing of interviews held. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 25 • Access to projects funded by USAID throughout the country, permitting trainees to see activities and engage with implementing partners in their areas. • Participants were able, following training, to become engaged with the projects they had visited through LOI, which has been valuable to the project implementers as well. 4. Might a different allocation of program activities or other alternative approach in design or implementation have achieved a better impact? Program design and startup as well as the implementation of LOI activities generally were carried out efficiently without any real problems being cited by core or facilitating partners. However, implementation of the M&E plan was deficient, and the long time interval between the baseline and subsequent EOP survey made it likely that other factors affecting attitudinal changes among the individuals surveyed. Early attention might have remedied this weakness in time, especially the poor returns of questionnaires, which TAF repeatedly pointed out in their reports. In retrospect, it is possible that additional participating partners could have played more active roles in workshops if TAF had provided additional training (based on some interview comments). However, this is at best an untested speculation. In point of fact, TAF did coach some partners to become adept at training in LOI themes; perhaps, these partners were the ones viewed by TAF with the most potential for having an expanded role. It also appears that TAF staff was overextended and additional training activities would have required additional resources. F. Sustainability Are clients satisfied with the program and how sustainable is the program’s impact? Clients of the LOI program have expressed appreciation for the program, which they view as an innovative strategy to address local and national development needs. Assessing the impact of the LOI program and its probable sustainability is difficult given that: 1) some M&E components of the program did not produce the desired data; and 2) attitudinal changes and social transformation are difficult to measure so soon after program completion. However, based on site visits made by the Team and anecdotes from beneficiaries, trainees, and program partners suggest that LOI themes have been implemented on the local level, thereby raising awareness and increasing knowledge about developmental issues. One core and facilitating partner, Rupantar, informed the Team that verbal feedback from trainees was always positive but suggested that the LOI program is at a relatively early stage of its development and still has only a limited reach. Rupantar argued that the sustainability of the program depends on: 1) the continuation of the orientation training for newly elected local government officials after the May 2011 elections; 2) reinforcement of initial training through follow-up sessions; 3) and expansion of LOI eligibility criteria to include persons at the sub-district level who hold considerable influence over their communities, such as teachers, doctors, business leaders, NGOs and sports club leaders (Rupantar believes this could double or triple the number of participants in small rural areas). An additional three years of LOI support by an external donor would establish a larger and broad base of local leaders capable of becoming self-sustaining. The ITA, which coordinated the training of Imams (over 50% of participants) felt sustaining the program would be difficult for them without the continued assistance of LOI implementer staff and resources as it does not have the capability or the financial resources to train Imams on its own. 1. Have the Government of Bangladesh and other local stakeholders demonstrated ownership of program activities? While the IFB, HRWT, and BRWT were actively involved in recruiting participants and conducting workshops, the MORA, which oversees the religious trusts, complained they were not adequately brought Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 26 into the LOI program by TAF. MORA said their role must be larger in the oversight and implementation of LOI if the program is continued. They know the government’s development strategy and want to be the main implementing counterpart for the religious training component of the program,50 with full oversight over the training activities and financial transactions carried out between the main implementing partner and the religious trusts. This role, the Ministry contends, would enable MORA to better address the development strategy of the GOB as well as providing an improved M&E process. Despite meetings and briefings with TAF, attendance at some LOI events, and one regional trip, MORA stated “there was a major information gap” between its offices and USAID regarding TAF’s LOI activities. Local stakeholders such as the local elected officials, religious leaders, women’s and youth leaders demonstrated ownership of program by implementing the themes of LOI through various practical methods that involved raising communal awareness and level of activity to address the key development themes. These were well documented by DW during the Intergroup/Interfaith workshops they organized, in which returning trainees listed, according to the themes of LOI, the various ways in which they applied LOI training in their communities since their first training session. CARE reported engaging religious leaders after observing the influence and impact of engaging religious leaders in the development process; and the YWCA related having forged independent working relationships with USAID partner programs such as SSFP recruited women leaders from other development organizations across Bangladesh. 2. Is there support within Bangladesh for continuation of similar activities for their own value or as a complement to programs of others? There was wide support and expressed need for the continuation of LOI programming. All the core implementing, and facilitating partners, and the MORA recognize the value of the program and its contribution to national development. When asked, IFB and ITA indicated no concern that USG funding for LOI training might compromised the integrity of the training. They thought the Imams supported LOI because it resonated with their reading of the Quran and Hadith, especially in advocacy for justice and humanitarian solidarity. Other known programs that engage LOI in Bangladesh include CARE’s SHUHARDO II program outreach to religious and community leaders (as mentioned above) as well as the United Nations Population Fund’s (UNFPA) Human Resources Development project which provides a four (4) day training to Imams through the ITA’s 45-day Imam training course (sometimes in a sequence with LOI’s three-day Imam training program). UNFPA training covers topics such as gender equality; drug prevention; early marriage prevention; youth and development; reproductive health; and HIV/AIDS. The BRWT also mentioned the UNFPA program’s work with the Buddhist communities, as well as Save the Children pre-primary education. 3. How have direct beneficiaries responded to their experience with the program? Partners told the Team how local elected officials, religious leaders, women and youth leaders demonstrated ownership of the program by implementing the themes of LOI through raising community awareness and activities to address key development themes. These were reported by returning trainees on flipcharts during follow up intergroup workshops. 4. How have media representatives, civil society, elected officials, local and regional government officials, and other stakeholders responded? The IFB/ITA, HRWT, BRWT all confirmed that religious leaders take the opportunity to relate LOI themes during prayer gatherings and religious festivals; Rupantar and a focus group of LEBs trained by Rupantar told us that they and colleague LEBs advocate themes at public gatherings, official events, and 50 During the course of the program TAF dealt directly with the IFB, HRWT, and BWRT in the coordination of LOI training, disbursing funds to them directly, bypassing MORA. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 27 cultural events through open forum discussions and entertainment programs with embedded LOI themes; BCCP has provided information that there have been over 139 newspapers and electronic articles written about the LOI program; YWCA initiated a policy that members who underwent LOI training hold at least one classroom session with the youth, during which they go over LOI themes; as part of LOI, as of February 26, 2011, LOI/DW had brought together 877 religious and secular leaders in follow up intergroup workshops to collaborate on development strategies in resonance with LOI themes; and all groups related that members of their respective communities availed the services of USAID partner programs. In this way the LOI program achieved exposure and sustainability, at least in the short-term. 5. Are processes, systems, and programs now in place to support the sustainability of program impact? As mentioned above, Rupantar does not believe enough members of society have been trained and suggests directing LOI investment towards the sub-district levels and targeting “opinion leaders” alongside the local elected officials, plus expanded training as a way to make LOI impact sustainable. Again, the ITA and HRWT expressed lack of resources to continue LOI programming without external funding. In this regard, they suggested that members of ITA and the Hindu religious leadership be taught to train others. A key process recommended by the ITA, MAACA, HRWT, BRWT, YWCA, Rupantar, and DW was an effective follow-up mechanism through which trainees can be: 1) monitored in terms of what they are doing to implement training; and 2) recruited for a second follow-on training (anywhere from three months to one year after the original training) to create an on-going dialogue with trainees and help motivate them to contribute towards LOI themed development goals. 6. What major factors influenced the achievement of program sustainability? Enthusiasm for program goals by LOI partners and participants and the enthusiasm by trainees seemed to be the most significant drivers of program sustainability. This enthusiasm was derived in part from the themes of LOI, which were deemed “practical” and “relevant” towards addressing pressing development challenges. Additionally, follow-on training for 877 religious and secular leaders proved to be a significant aspect of LOI as it allowed returning trainees to share their experiences of LOI implementation, and strategize practical ways to further apply LOI themes in their societies. This process contributed to program sustainability in the following ways: 1) it provided a platform for networks and relationships to develop between members of different faiths and social backgrounds in their work towards shared development goals; 2) it created an ongoing dialogue between the facilitators of LOI and the trainees, who felt responsibility to show something for the LOI training they had received; and 3) it asked of the trainees that they list challenges they encountered and to help each other think about how to overcome them through specific actions and within a stated time-period. Through these steps, the intergroup workshops allowed for a sort of M&E process among the participants, empowering and challenging their social development efforts. It was, perhaps, for these reasons that the core and implementing partners of LOI cited this activity as one of the most successful components of LOI. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 28 VI. CONCLUSIONS AND RECOMMENDATIONS A. Conclusions LOI has been building capacity at the local level for four years. During that time, civil society has been embracing national development priorities and democratic values the program addresses in locations where it has substantial operations. Almost all the partners interviewed expressed positive views on the value of LOI. They were unanimous in their desire that this type of program be continued and many advocated is expansion in some fashion. Overall, the performance of the LOI program was strong in achieving its targeted outputs. Outcomes attributable to the program could only be estimated in general terms, based on limited evidence in the absence of systematic data on follow-on activities and impacts. 1. Relevance and Effectiveness of the LOI Program The goal of LOI was to mitigate conditions that might be conducive to extremism. The operating premise was that providing knowledge to religious and secular leaders on national development activities and visits to projects funded by USAID would lead to increased tolerance and understanding—thereby reducing extremist risks, broadening support for development and democratic values, and dispelling misconceptions about the U.S. All LOI activities have been consistent with the USAID’s strategic objective to help build a “vibrant civil society promoting democratic development.” The LOI program has exceeded its target of training 20,000 religious and secular leaders—a total of 20,743 participants have been trained (10,787 Imams, 2,658 other religious leaders, and 7,298 secular leaders). LOI orientation training has been instrumental in assisting participants to embrace the notion that they can become agents of change in their communities. As proposed, TAF completed 96 regional exchanges (48 inbound, 48 outbound) to nations in South Asia. In promoting the LOI program, 48 community outreach events were conducted over the life of the program. 2. Efficiency and Impact of the LOI Program Given the relative small amount of funding ($US 3.6 million) for the LOI program, the return on investment was very positive, in substantial part because the participants were local leaders who had the opportunity to share knowledge and influence the opinions of others in their communities. For example, at a minimum, each Imam serves as a disseminator to five mosque committee members and speaks with authority to a larger group of followers. The “multiplier” effect is significant. 3. Sustainability Issues LOI had widespread support for its activities among partners and participants. All the partners interviewed were positive in their praise for the goals and activities of the program. None of the partners, however, felt the program was sustainable at this time or that it would be so in the foreseeable future. They believed that continued international funding would be needed for orienting additional leaders and reinforcing those already trained. A few partners speculated that it would take at least three to four years for enough support to be generated to sustain this type of program. It is doubtful that this type of program could become completely self-sustaining from a financial standpoint. Nevertheless, its multiplier effect, low operating costs and contributions by many partners have shown it to be a positive investment 4. Monitoring and Evaluation Systems Considerable thought and effort went into conducting a baseline survey and an EOP survey. The findings clearly validated that LOI participants positively changed their perceptions in the areas of tolerance, diversity, and the role of religious leaders in community affairs. Unfortunately, the time between the baseline and EOP surveys was three years and extraneous factors in all likelihood contributed to some extent to the changes in perception. Notwithstanding costs constraints, it would have been preferable that Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 29 smaller surveys have been conducted annually to more accurately gauge changes in perception among LOI participants. The M&E procedures for documenting LOI impact were stronger on paper than in their actual implementation. Documentation of participant orientations was limited. There was little to no documented analysis of the STQs and the poor response rate for returning the LTQs was so low that virtually no data results were evident. The Team found no evidence either that TAF took remedial steps on its own or that USAID took action on TAF’s reports of these problems. 5. Management Issues TAF LOI staff was overextended in coordinating, overseeing, and appraising all activities. This appears to have limited visiting, supporting and providing feedback among partners apart from the conduct of orientation trainings. While overall results of the project were in compliance with its stated targets, these results could have been even better with more frequent and quality interaction with all of the partners. Indeed, some partners were critical about the limited interaction they had with TAF over the life of the project. The program was designed to conduct three-day orientation training programs for religious leaders of all faiths and also for secular leaders. Yet, in many cases orientation was for only one or two days. TAF told us they found certain groups of leaders unable or unwilling to dedicate three full days to the training. Again, the Team saw no reporting of this or of any written acknowledgement by USAID of this change from the design. 6. Lessons Learned Future M&E efforts seeking to assess the impact of any LOI program should concentrate on validating behavioral changes at the community level—using ethnographic data collection and/or case study procedures—in addition to tracking attitudinal change among program participants. For the amount of activities undertaken, it is clear in retrospect that the LOI program was underfunded— the Evaluation Team was not able to ascertain how TAF arrived at the budget level set in the LOI Activity Approval Document (April 30, 2006). B. Recommendations • USAID should strongly consider a continuation the LOI program with continued emphasis on training participants in areas consistent with the GOB national development strategy and consistent with USAID’s new country development cooperation strategy. • The selection of USAID training themes and USAID implementing partner participation might be more strategic in the setting of training priorities for LOI program content in order to reflect evolving priorities of GOB and USAID strategies. For example, if food security or access to justice is to be a greater priority in the future, that priority might be given consideration in making judgments about the content of future LOI program activities. • In addition to new participant leaders and communities, a future program should consider enhancing the experience and multiplier effect of past leader participants; e.g., more one day intergroup workshops or follow up workshops to reinforce understanding, and other mechanisms to keep in touch and document interactions with and among participants. • The value of visits to projects could be enhanced further by including the GOB and other donor funded activities in the local areas. • Future efforts like the LOI program should be based on rigorous contractual requirements that more closely link tasks, outcomes, and results to projected costs based on thoroughly documented budget development procedures. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 30 • Improved procedures should be developed to monitor the impact of the program on the behavior of participants, including by documenting activities they undertake after they return to their communities. Far greater attention in M&E efforts must document behavioral change along with capturing changes in perception. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 31 ANNEX 1. STATEMENT OF WORK SECTION C –STATEMENT OFWORK C. 1 BACKGROUND As the seventh most populous country in the world and the fourth largest Muslim-majority country, Bangladesh is of significant strategic interest to the United States. Bangladesh is poor, vulnerable to natural disasters, and susceptible to social upheaval and political conflict. Since independence, it has held democratic elections sporadically, including three successive peaceful transfers of power from 1991 to 2006. Yet, Bangladesh’s development as a democracy has been interlaced with military rule, debilitating political polarization, ineffective institutions of governance, and endemic corruption. The triumphant free and fair parliamentary elections organized by the military-backed caretaker government on December 29, 2008 led to Bangladesh’s successful transition to democratic rule and opened a new chapter in the country’s history. This transition comes in the wake of a history of political acrimony, Parliament boycotts, bitter recriminations over electoral manipulations and a tradition of street violence. The difficult but successful transition back to elected government has brought about new opportunities for democratic development and a new sense of urgency exists among Bangladeshi stakeholders to reform political practices and institutions of governance. This new chapter also presents the United States with an historic opportunity to help Bangladesh improve the country’s governance including the rule of law. C.2 OBJECTIVES USAID/Bangladesh seeks to purchase two project evaluations and a rule of law assessment for Democracy and Governance programs. The project evaluations will gauge the impact and lessons learned from two programs that will end in 2011and, if appropriate, provide USAID/Bangladesh with recommendations for follow-on programming. The rule of law assessment will provide a targeted analysis of the status of rule of law development in Bangladesh, and an assessment of the primary opportunities and constraints to the development of the rule of law in Bangladesh. The assessment will lead directly into a strategy and potential activity design for rule of law assistance in Bangladesh to include the identification of priority areas that could benefit from USAID interventions and prioritized recommendations for future programming. C.3 SCOPE OF WORK Task One Evaluation of Leaders of Influence (LOI) USAID Bangladesh’s Leaders of Influence (LOI) is a four-year program designed to enhance the capacity of religious and secular leaders to contribute to national development and democratic reform efforts. In so doing, LOI sets out to preserve and promote values of democracy, tolerance, diversity, social harmony, and understanding in Bangladeshi society. Under this program, at least 20,000 leaders will receive training and hands-on orientation to Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 32 USAID partners’ programs including democracy and governance, gender equality, health, nutrition, family planning, HIV/AIDS, employment generation and disaster management. The basic assumption underpinning the LOI program is that the knowledge of different development aspects, gained through training and exposure, will help these leaders of influence to increase tolerance, dispel erroneous myths about the U.S. and decrease the opportunities for extremism and potential terrorism. USAID also assumes these leaders of influence, both religious and secular, are well placed in society to contribute to the development efforts and advance in democratic values within their communities and throughout Bangladesh. In April 2007, USAID awarded a $US 3,598,582 cooperative agreement to The Asia Foundation to implement the planned four-year LOI program. TAF will provide an additional $US 392,160 under a cost-share arrangement. The program is scheduled to end in April 2011. The objective of this external evaluation is to assess the impact and lessons learned from LOI for consideration in the design of a follow-on program, if appropriate. The evaluation will: 󲐀 Test the technical and programmatic validity of the hypotheses on which the LOI program was designed; 󲐀 Assess the efficacy, cost-effectiveness, and impact of the LOI implementation tools and management structure in meeting the objectives; 󲐀 Evaluate LOI implementers’ performance to date and assess results vs. goals and indicators; and 󲐀 Make necessary recommendations to USAID Bangladesh. The audience for this evaluation is USAID Bangladesh. The evaluation should review, analyze, and evaluate the LOI program along the following illustrative criteria, and, where applicable, identify opportunities and make recommendations for improvement. In answering these questions, the Evaluation Team should assess both the performance of USAID and that of the implementing partner(s). Relevance. The Evaluation Team should assess the relevance of the LOI approach. 󲐀 Has the Mission’s LOI approach achieved desired results? 󲐀 Would a different approach achieve the same or enhanced results? Describe. 󲐀 Is the original hypothesis on which the LOI approach was designed still valid? 󲐀 Is the LOI program approach on track to respond to the needs of program beneficiaries and to achieve their stated objectives? 󲐀 Were the LOI program assumptions accurate? Was capacity enhanced among participants? Were the desired democratic values preserved and promoted? 󲐀 Did the selected components (orientation, regional exchange programs, outreach, and monitoring and evaluation) under LOI prove amenable to program goals? 󲐀 Were the target groups (religious leaders, local elected officials, community service clubs, professionals, media, youth, women, and intergroup/interfaith) correctly selected? 󲐀 Are the program and its various components and activities relevant from the perspective of the democracy and governance context? 󲐀 How did political events in Bangladesh during the implementation period affect the success/failure of LOI interventions? Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 33 󲐀 Are the processes and procedures currently in place appropriate to guarantee LOI implementation is efficient, timely, and in accordance with USAID regulations? Management and Administration. The Evaluation Team should assess management and administration of the implementing partner(s), as well as that of USAID. 󲐀 To date, how effective has the LOI management structure including TAF and their implementing partners (i.e., Imam Training Academy, Rupantar, Bangladesh Center for Communications Programs, Data International, etc.) been in obtaining program results? 󲐀 Has the TAF management structure helped to ensure optimal coordination and decision making to achieve stated objectives and avoid duplication of effort with other USAID or other donor programs? 󲐀 What alternative management structure and tools would/could increase effectiveness? 󲐀 What impact does the existing management/administrative structures have on the flexibility of the LOI program? 󲐀 Did start-up activities (e.g., building rental, staffing, salary approval, agreements with progressive layers of implementers, and other administrative and financial actions) occur at an appropriate pace? Cost Effectiveness. The Evaluation Team will assess cost-effectiveness and efficiency against program impact, both in terms of time and money. 󲐀 Are results achieved under LOI being produced at an acceptable cost compared to alternative approaches accomplishing the same objectives? 󲐀 To what extent is LOI appropriately taking advantage of cost share and/or leverage opportunities? 󲐀 What, if any, alternative approaches exist which could achieve results at greater efficiency and what mechanisms can be recommended for implementing the alternative approaches? Impact. The Team should conduct an impact evaluation. 󲐀 What has been the impact of activities implemented under LOI? 󲐀 Are implementing partners planning their individual activities with the broader LOI objectives and sub-objectives in mind? 󲐀 How well have implementing partners worked as a team to coordinate work plans and activities towards achieving the overall objectives of the LOI program? 󲐀 How are communications and outreach components contributing to program impact? 󲐀 Are there any externalities or unintended consequences related to implementation of LOI that USAID should consider? Sustainability. 󲐀 Are the processes, systems, and programs in place to ensure that the results and impact of LOI activities will be sustainable? 󲐀 What evidence has there been of the Government of Bangladesh and other Bangladeshi partners taking ownership of LOI activities? 󲐀 What obstacles exist for achieving sustainability? 󲐀 What measures should be taken to increase sustainability? Cross Cutting Issues. 󲐀 How well has LOI integrated support to issues of gender, the disabled, and minorities? Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 34 󲐀 What improvements have been made to increase program effectiveness in integrating these issues? Synergy with other USAID and Donor Funded Programs. 󲐀 How effective is coordination within the various USAID Bangladesh teams (i.e., Economic Growth, Population Health Nutrition and Education, Food Disaster and Humanitarian Assistance) -- as well as within other DG activities -- in achieving LOI objectives that cut across all USAID Bangladesh Strategic Objectives? 󲐀 How effective is LOI coordination with other donor activities? 󲐀 To what extent have other donor activities been successfully integrated into LOI? 󲐀 How have LOI interventions supported/complemented other donor efforts? Client Satisfaction (GOB, beneficiaries, other stakeholders). 󲐀 How have direct beneficiaries responded to the program? 󲐀 What do media representatives, civil society and citizen organizations think about LOI? 󲐀 What do elected officials, local and regional governments officials, and other stakeholders think about LOI? Performance Measurement Systems. Measuring program impact requires the existence of sound performance monitoring systems at the level of individual partners as well as at the level of program management. The Evaluation Team should investigate whether systems have been established internally for tracking, monitoring, and reporting results attributable to LOI activities, and whether these systems are effective and utilize independently verifiable information. 󲐀 Do performance monitoring systems at all levels effectively measure program impact? 󲐀 Are the indicators being used by USAID and the implementing partners meaningful? Why? 󲐀 Do indicators create positive/negative incentives for implementing partners? How? Other. Are there other concerns by the program stakeholders (GOB, local governments, other beneficiaries), not mentioned above, that USAID Bangladesh should consider? Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 35 ANNEX 2. LOI TEAM MEMBER BIOGRAPHICAL SUMMARIES Dr. William Millsap has over twenty-five years of experience in the development, management, and evaluation of international development programs and community development programs in the United States. His expertise includes the application of quantitative and qualitative research approaches for conducting socioeconomic impact assessments, performance appraisals, case studies and cross-sector studies of socioeconomic transition. These experiences have been carried out for federal and state agencies, private consulting firms, and academic institutions. Dr. Millsap is also skilled in the application of management information systems to support and facilitate management and research endeavors, including statistical packages. Highlights of his work experience include conducting evaluation and performance measurement studies for USAID, designing and carrying out policy reviews, program/project assessments for various government agencies (e.g., ACTION, Peace Corps, DHHS/OCS, CDC, USDA/FNS), International PVOs, and the World Bank providing technical assistance on a wide range of research/evaluation studies. Stephen Strobach has over 30 years of experience working with civil society and other organizations in developing countries. He is an experienced advisor and evaluator of development assistance efforts and of multicultural teams. Mr. Strobach has twenty years overseas management experience and nine years consulting experience, designing and evaluating programs supporting efforts of CSOs (civil society organizations), host country government entities and local NGOs. He also has extensive rural development experience, including over five years in rural areas of Nepal, Bangladesh and Pakistan. He is a skilled developer of community and youth participation and gender inclusiveness. Additionally, he has provided guidance to coaching of partners and technical staff in designing strategies, projects and proposals; also management, oversight and evaluation of the entire project cycle. Mr. Strobach has led and managed programs in Nepal, Colombia, Bolivia, Guatemala, Honduras and Dominican Republic; consultancies in these countries and Bangladesh--where he worked with Muslim community leaders as a civil society expert. Samier Mansur is an honors graduate of James Madison University where he Studied International Relations, Economics, and Religion. He has applied his academic background and intellectual drive in various outfits which have included: working alongside former U.S. Ambassador William Milam in the publication of a book on the political and economic evolution of Pakistan and Bangladesh at the Woodrow Wilson International Center for Scholars; serving as an analyst for a strategic communications firm working on conflict mitigation projects in Iraq and Afghanistan; consultation and analysis for the U.S. Trade and Development Agency to promote development aims and business interests in developing and middle income nations of South and Southeast Asia; appointed as a Research Scholar for the Mahatma Gandhi Center for Global Nonviolence at James Madison University where he taught a course on Political Islam, and conducted research and writing on Islam and contemporary issues of global peace and justice. In addition to serving as a private consultant on the issue of engagement with the “Muslim World,” Mr. Samier currently serves as an Economic Analyst for the Policy Research Institute of Bangladesh, and writes a Religion and Society column for the Independent Magazine. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 36 ANNEX 3. LITERATURE CITED The Asia Foundation, Leaders of Influence Annual and Semi-Annual Reports from Sept 2008 thru Sept 2010. • Monitoring & Evaluation Plan, May 6, 2008. • Program Overview, March 2010, http://www.leadersofinfluence￾asia.org/uploads/517f8d28b3d287953937f5703b4b45f88b2aa3a9.pdf. • “Midterm Review of Bangladesh Country Partnership Strategy,” August 2009, http://www.adb.org/Documents/CPSs/BAN/2006-2010/CPS-BAN-2006-2010.pdf. • Key Output Results, “Summary of Results,” 2011. • End of Project Evaluation Report, January 2011. Centre for Policy Dialogue, “Bangladesh Vision 2021,” 2007, http://www.cpd.org.bd/html/Policy%20Brief/sub%20folders/downloads/Vision_2021_English.pdf. Chandry, Laurence, and Geoffrey Gertz, “Poverty in Numbers: The Changing State of Global Poverty from 2005 to 2015,” Brookings Institution, January 2011, http://www.brookings.edu/papers/2011/01_global_poverty_chandy.aspx Constitution of the People’s Republic of Bangladesh, as amended. European Community, “Bangladesh Country Strategy Paper for the Period 2007-2013,” http://www.eeas.europa.eu/bangladesh/csp/csp_07_13_en.pdf. Freedom House, “Bangladesh Country Report,” in Freedom in the World, 2010, http://www.freedomhouse.org/template.cfm?page=22&year=2010&country=7778. Government of the People’s Republic of Bangladesh, “Framework of National Integrity Strategy: An Inclusive Approach to Fight Corruption,” October 2008, http://www.cabinet.gov.bd/file_upload/CD_POLICY_bn_1_113.doc. Planning Commission, Steps Towards Change: National Strategy for Accelerated Poverty Reduction II (Revised) FY 2009-2011, December 2009, http://www.lcgbangladesh.org/prsp/docs/PRS%20Bangladesh%202010%20final.pdf. Ministry of Finance, “Bangladesh Joint Cooperation Strategy, 2010-2015,” June 2010, http://www.erd.gov.bd/JCS/JCS_Signed_document.pdf. “Outline Perspective Plan of Bangladesh 2010-2021: Making Vision 2021 a Reality,” DRAFT, June 2010. Institute of Governance Studies, State of Governance in Bangladesh: Confrontation, Competition, Accountability, BRAC University, August 2009, http://dspace.bracu.ac.bd/bitstream/10361/582/1/SOG_2008.pdf. Mahmud, Wahiduddin, Sadiq Ahmed, Sandeep Mahain, “Economic Growth and Governance: The Political Economy Aspects of Bangladesh’s Development Surprise,” Commission on Economic Growth and Governance Working Paper 22, 2008, http://www.growthcommission.org/storage/cgdev/documents/gcwp022web.pdf. Millennium Challenge Corporation, “Bangladesh Scorecard for 2011”, http://www.mcc.gov/documents/scorecards/score-fy11-bangladesh.pdf. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 37 Mintoo, Abdul Awal, Bangladesh: Anatomy of Change, Athena Press, 2006. Netherlands, “Bangladesh Multi-Annual Strategic Plan, 2010-2011,” http://www.netherlandsembassydhaka.org/newpdf/MASP%202010-11.pdf. Odhikar, “Human Rights Report 2010,” January 2011, http://www.odhikar.org/documents/2010/English_Reports/Annual_Human_Rights_Report_2010_Od hikar.pdf. Quibria, M.G., and Ahmad Shafi, “Aid Effectiveness in Bangladesh,” Conference Paper, June 2008, http://mpra.ub.uni-muenchen.de/10299/1/MPRA_paper_10299.pdf. • Smiling Sun Franchise Program, Estimated Report on Customers Referred by LEADERS OF INFLUENCE, January-December 2010. • Health Bulletin 2010 and 2009. Swiss Cooperation , “Bangladesh Country Case Study” in Joint Evaluation of Citizens’ Voice and Accountability, Overseas Development Institute, 2008, http://www.norad.no/en/Tools+and+publications/Publications/Publication+Page?key=124740. USAID/Bangladesh Strategy 2005-10, September 2005. • Cooperative Agreement #388-A-00-04-00113-00, Leaders Outreach Initiative, “National Perception Study on Islam in Bangladesh” • “Strategic Statement, FY 2006-2010,” September 2005, http://www.usaid.gov/bd/files/strategic_statement.pdf. • Cooperative Agreement No. 388-A-00-07-00029-00, Leaders of Influence Program with The Asia Foundation, April 16, 2007. • “Leaders Of Influence Baseline Report,” 2008 • Democracy and Governance Assessment, August 2009. • Activity Approval Document 2009-2014: “Democracy Enhanced, Sustained, and Harnessed (DESH),” Dec. 2009. • Bangladesh Gender Assessment, April 2010. • Bangladesh Economic Growth Assessment, June 2010. USAID Evaluation Policy, Bureau for Policy Planning and Learning, Washington, D.C., January 2011, http://www.usaid.gov/evaluation/USAID_EVALUATION_POLICY.pdf?020911. • “Issue Brief: The Role of Religious Leaders and Communities in Development Efforts in Asia and the Middle East,” November 2009, http://www.usaid.gov/locations/asia/documents/Religion_AME_brief.pdf. USG Department of State Reports on Human Rights, Money Laundering, Narcotics Control, Religious Freedom, Terrorism, and Trafficking in Persons, http://www.state.gov/r/pa/ei/rls/dos/221.htm. Van Schendel, Willem, A History of Bangladesh, Cambridge University Press, 2009. World Bank, “Governance and Growth: The Bangladesh Conundrum,” in Bangladesh: Strategy for Sustained Growth, July 2007, pages 125-136, http://siteresources.worldbank.org/SOUTHASIAEXT/Resources/Publications/44813- 1185396961095/4030558-1185396985915/fullreport.pdf. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 38 • Country Data for Bangladesh, Population and Economic Indicators 2005-2009, http://data.worldbank.org/country/bangladesh. • “Country Data Report for Bangladesh: 1996-2009,” in Worldwide Governance Indicators, 2010, http://info.worldbank.org/governance/wgi/pdf/c20.pdf. • “Bangladesh Country Assistance Strategy, 2011-2014,” http://siteresources.worldbank.org/BANGLADESHEXTN/Resources/295759- 1271081222839/6958908-1284576442742/BDCASFinal.pdf. Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 39 ANNEX 4. PERSONS INTERVIEWED United States Government Habiba Akter, Human Rights and Rule of Law Advisor, USAID Bangladesh Dianne Cullinane, Democracy Officer, USAID Bangladesh Nicholas J. Dean, Deputy Chief of Mission, United States Embassy Dhaka Lily P. Kak, Senior Health Advisor, Bureau of Global Health, USAID Shumana Masud, Governance Advisor, USAID Bangladesh James Moriarty, United States Ambassador to Bangladesh Denise Rollins, Mission Director, USAID Bangladesh Dennis Sharma, Deputy Mission Director, USAID Bangladesh Sherina Tabassum, Governance Advisor, Office of DG, USAID/Bangladesh Implementing Organizations Minhaj Alam, Director, Human Resources Program, Democracy Watch Shahidul Alam, Assistant Director of Training, Bangladesh Center for Communication Programs Farouk A. Chowdhury, Senior Economist, LOI Database Manager, The Asia Foundation Muhammad Aminul Ehsan, Team Leader, Rupantar, Khulna Khan Muhammad Farouk, Training Coordinator, Imam Training Academy, Khulna Mohammed Taher Hussain, Director of the Islamic Training Academy Nazrul Islam, Chief of Party, Leaders of Influence Program, The Asia Foundation Mir Junayed Jamal, Program Officer, LOI Program, The Asia Foundation John Karr, Director, Digital Media, The Asia Foundation, San Francisco Muhammad Shahabuddin Khan, Director of Planning Department, Islamic Foundation Rafiqul Islam Khokan, Executive Director, Rupantar, Khulna Hasan Mazumdar, Country Representative, The Asia Foundation Matthew Pendergast, Producer, Digital Media, The Asia Foundation, San Francisco Russell Pepe, Deputy Chief of Party, Promoting Democratic Institutions & Practices, State University of New York at Albany, former LOI Chief of Party A.F.M. Azizur Rahman, Director, Data International A.K. Shafiqur Rahman, Sr. Deputy Director, Bangladesh Center for Communication Programs Mohammad Shahjahan, Director and CEO, Bangladesh Center for Communication Programs K.M. Shamsuzzaman, Manager, Data International Nancy Yuan, Vice President, The Asia Foundation Facilitating Partners Proshonto Kumar Biswas, Field Officer, Hindu Religious Welfare Trust Muhammad Nurul Hassan, Joint Director, Bangladesh Centre for Development Journalism and Communication Nayeemul Islam Khan, President, Bangladesh Centre for Development Journalism and Communication Khaled Muhuddin, Director, Bangladesh Centre for Development Journalism and Communication Bipul Chandra Roy, Secretary, Hindu Religious Welfare Trust Participating USAID Partner Programs Mr. Binoy, Rights Jessore Anti-Trafficking Group M A Shahid Khan, Managing Director, Smiling Sun Franchise Program Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 40 Palash Mondal, Technical Manager - Humanitarian Assistance, CARE Bangladesh Abdul Motin, Franchise Operations Officer, Smiling Sun Franchise Program Juan Carlos Negrette, Chief of Party, Smiling Sun Franchise Program Rehan Uddin Ahmed Raju, Smiling Sun Franchise Program Civil Society and Subject Matter Experts Dr Azra Abidi-Assistant Professor of Sociology, Jamia Millia Islamia, New Delhi – LOI International Exchange Program visitor from India Mr. Abul Kalam Azad, Moulana, Chairman, Masjid Council for Community Advancement Shah Abdul Baten, Director (Research, Advocacy and Campaign), Masjid Council for Community Advancement Sorowar Hossain Chowdhury, Technical Advisor, Masjid Council for Community Advancement Zillur Khan, Senior Fellow, Policy Research Institute of Bangladesh, Professor Emeritus, University of Wisconsin Focus Groups and Other Meetings Focus Group with LOI Local Elected Officials from 4 UPs in Khulna and 2 in Bagerhat Districts, at Rupantar Office, Khulna: Sanjit K Mondal-Jalma Chair, and Members Md. S Sheik-Baliadanga, Md. Salim Hossain-Ganarampur, Kalpona Sarkar-Batiaghata, Moriam Begum-Jatrapur and Debala Sarkar￾Chandpai Observe Inter Faith & Inter Group follow-up training for LOI graduates in Rajendrapur Conversations during breaks with: Mahfuz Mondal-Bogra Correspondent, New Nation Daily Shyamal Sarkar-Asst Professor of English at Jhitka Khaja Rahmat Collete Manikganj Dr Sunil C Mistry-Assistant Professor of Sanskrit, National University at Gazipur Visit with participants during their training at Imam Training Academy, Agargaon, Dhaka Visit with participants during their training at Imam Training Academy, Khulna Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 41 ANNEX 5. LOI TRAINEE RECRUITMENT BY GROUP AND YEAR LOI Group Type Year-wise Participant Numbers Year 1 Year 2 Year 3 Year 4 Total All Years Muslim Religious Leaders (Imams) 1,998 2,697 2,495 3,597 10,787 Mosque Committee Members 207 202 201 211 821 Buddhist Religious Leaders 50 100 50 100 300 Hindu Religious Leaders 100 500 578 359 1,537 Local Elected Officials 397 443 504 471 1,815 Print and Broadcast Journalists 186 306 323 185 1,000 Christian Religious Leaders 50 48 0 79 177 Youth Leaders 250 297 254 197 998 Women Leaders 108 203 53 156 520 Professional Leaders 0 350 145 311 806 Community Service Club Members 0 87 50 47 184 Inter-faith/Inter-group Leaders 308 205 666 619 1,798 Total All Groups 3,654 5,438 5,319 6,332 20,743 Leaders of Influence Evaluation Report Task Order No. AID-388-TO-11-00006 April 6, 2011 Millennium Partners Page 42 ANNEX 6. PERSONS SURVEYED RESPONDING TO QUESTIONS ON DEMOCRATIC VALUES Key Survey Questions on Democratic Values Baseline Survey Values (%) End of Project Survey (%) % of Change from Baseline to EOP Values Imams All LOIs Ordinary Citizens Imams All LOIs Ordinary Citizens Imams All LOIs Ordinary Citizens T-3 Strong belief in equal employment for people of all religions 55.0 64.0 56.3 85.7 80.6 69.7 30.7 16.0 13.4 T-5 Strong belief in equal employment opportunities for women 29.5 56.9 49.8 57.5 62.4 66.5 28.0 5.5 16.7 T-7 Strong belief in protection of minority rights by government 37.5 43.4 41.4 49.5 55.9 44.2 12.0 12.5 2.8 T-9 Strong belief in the notion that religious & cultural minority groups should be allowed to hold their religious services & festivals peacefully and not be disturbed 60.5 70.2 61.6 79.5 80.5 70.2 19.0 10.3 8.6 T-18 People’s belief in the notion that a religious leader should not limit himself or herself to religious matters in giving advice to people 31.5 24.3 16.8 60.0 46.2 35.2 28.5 21.9 18.4 T-20 Consulting religious leaders on problems facing their communities 28.5 12.7 13.4 47.6 31.9 24.5 19.1 19.2 11.1 Average levels of change over six value dimensions: 22.9% 14.2% 11.8%