Evaluation of the Burma Outreach and Distance Education Project: A Foundation for the Future December 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). EVALUATION OF THE BURMA OUTREACH AND DISTANCE EDUCATION PROJECT A FOUNDATION FOR THE FUTURE December 2012 Cooperative Agreement 486-A-00-07-0007-00, September 21, 2007 – November 30, 2012 Prepared by: Simon Richards DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acronyms ................................................................................................................................................ a Executive Summary.................................................................................................................................. i 1. Evaluation Purpose & Evaluation Questions..................................................................................... 1 1.1 Evaluation Purpose........................................................................................................................ 1 1.2 Evaluation Questions ..................................................................................................................... 1 2. Project Background.......................................................................................................................... 2 2.1 Identifying Project Information........................................................................................................ 2 2.2 Problem or Opportunity Addressed................................................................................................. 2 2.3 Target Program Areas and Groups................................................................................................. 2 2.4 Burma Outreach and Distance Education Project Goal................................................................... 2 2.5 The Burma Outreach and Distance Education Project Intended Results ......................................... 2 3. Evaluation methods & limitations...................................................................................................... 3 3.1 Methods......................................................................................................................................... 3 3.2 Limitations ..................................................................................................................................... 4 3.2.1 Limitations concerning sampling and choice of locations visited............................................... 4 3.2.2 Limitations concerning access to stakeholders ........................................................................ 4 3.2.3 Limitations concerning data quality.......................................................................................... 5 3.2.4 Limitations with the Perception Survey .................................................................................... 5 3.2.5 Limitations of the overall scope and design of the evaluation................................................... 6 4. Findings, Conclusions & Recommendations..................................................................................... 6 4.1 Findings......................................................................................................................................... 6 4.1.1 The Context – External and Internal ........................................................................................ 6 4.1.3 Efficiency of the Project.........................................................................................................10 4.1.4 Effectiveness of the project....................................................................................................13 4.1.5 Impact and the Evaluation Questions ....................................................................................15 4.2 Conclusions ................................................................................................................................. 21 4.3 Recommendations.......................................................................................................................22 4.3.1 Recommendations Part 1: Operating in Oppressive Environments ........................................22 4.3.2 Recommendations Part 2: Emerging from the Project Experience .........................................22 4.3.3 Recommendations Part 3: Strengthening Civil Society - Strategic Choices for the Future ......23 Annex I: Evaluation Statement of Work ..........................................................................................27 Annex II: Evaluation Design Strategy .............................................................................................40 Annex III: Data Collection Instruments............................................................................................49 Annex IV: Sources of Information...................................................................................................58 Annex V: Project Performance Tables FY 2009-2011.....................................................................65 Annex VI: Disclosure of any Conflicts of Interest ............................................................................67 a ACRONYMS AC American Center BC British Council BP Best Practice BODEP Burma Outreach and Distance Education Project CBI Capacity Building Initiative CBO Community Based Organization CNF Chin National Front CSO Civil Society Organization CSU Colorado State University DE Distance Education FCToT Foundation Cycle Training of Trainers INGO International Non-Government Organization IU Indiana University KNU Karen National Union OCA Organization Capacity Assessment OTI Office of Transition Initiatives MDRI Myanmar Development Resource Institute MSR Myanmar Survey Research NLD National League for Democracy PMP Performance Management Plan QuiLT Qualification in Language Training TDP Teacher Development Program TOT Training of Trainers TP Teaching Practice TNI Transnational Institute RDMA Regional Development Mission for Asia SCS Strengthening Civil Society SIT School for International Training SSAS Shan State Army South SLORC State Law and Order Restoration Council SOW Statement of Work USAID U.S. Agency for International Development USG U.S. Government WL World Learning i EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This performance evaluation of the Burma Outreach and Distance Education Project (BODEP) implemented by World Learning (WL) took place in October to November 2012 just prior to its November 30, 2012 completion. The evaluation was intended primarily to learn to what extent the project’s goals and objectives have been achieved. It builds upon previous monitoring data already captured by Performance Management Plans (PMPs), previous internal evaluations and provides a “bigger-picture” insight on the validity of the project’s development hypothesis and results framework. Because the project’s transformation occurred during a rapidly changing environment in Burma, the evaluation also provides some insight into the contextual changes that happened simultaneously, and how they may have affected programming. On a higher level, USAID requested this evaluation to understand the efficacy of WL’s approaches, insights into best practices and barriers to civil society development to add to the growing body of knowledge informing strategic future civil society programming in Burma. As a secondary purpose, the evaluation explores possibilities for working with and building the capacity of local Burmese evaluators for future evaluations. To accomplish this aspect of the evaluation, Myanmar Survey Research was brought on to collect evaluation data and receive training from the evaluation team. The evaluation questions informing the shape of the evaluation are as follows: 1. Is the training of community leaders actually contributing to strengthened organizational capacity1 in the organizations where project alumni are working? 2. What are the key barriers to community leaders using acquired learning and networking competencies in their work? 3. In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Are there signs that the project alumni will continue to apply knowledge and skills attained after the project’s end? 4. During the different stages of the project, and in light of the rapidly changing contexts within Burma, what approaches have been the most effective and efficient in achieving the project’s goals? Which approaches have been the least effective and efficient? 5. What were the unintended consequences of the project, if any? PROJECT BACKGROUND BODEP aimed to address the increasing need for civil society leaders who can manage change, create networks, mobilize communities and build coalitions in Burma. This program fit within the space that was opening for civil society activities in Burma particularly following the natural disaster of cyclone Nargis in 2007 and has taken advantage of the more recent positive political changes moving away from a military form of government. The goal of the project, as articulated in the program description in 2011, is to develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society. This has been addressed through five major components, including: 1) English Teacher Training, 2) Training of Trainers, 3) Distance Education, 4) Leadership, and 5) Networking. Project activities are nearly wholly capacity building inputs to a range of individual Burmese community leaders, with some support to Burmese civil society organizations. The project evolved as the program description was refined. An initial emphasis on supporting the education needs of political dissidents and their families and professionalizing the English Language Teaching sector changed to utilizing an education and training platform as the entry point to discussing and addressing issues affecting civil society. The project developed an expanded range of courses tailored to suit different needs of civil society. Project management also evolved, developing increasing rigor in targeting, recruiting and monitoring project participants. 1 Strengthened organizational capacity will be used as a proxy for strengthened civil society, the ultimate goal of the project; Evaluators may choose to explore other relevant proxies for strengthened civil society. ii EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation employed a mixed methods approach designed to collect data to inform a rich response to each of the evaluation questions. Three main methods were used: • A review of project literature and academic literature in the public domain provided background analysis on the state and evolution of civil society in Burma over the past 10 years or so. • Quantitative data from project monitoring and evaluation data, where available, was analyzed along with a quantitative collation and analysis of qualitative data that was obtained through interviews and focus group discussions. • Qualitative data obtained through a broad perception survey conducted by local partner Myanmar Survey Research (MSR), focus group discussions and key informant interviews was analyzed. The survey questionnaire and interview guide for alumni were informed by Kirkpatrick’s model of assessing training levels among participants. A number of limitations affected the evaluation design, the collection of data and the quality of data obtained, potentially impacting on the findings. Specifically, the evaluation design and management was constrained by the sensitivity of the project given that it did not operate formally or publicly, which meant the expatriate evaluation team was not able to travel outside of Rangoon so as to avoid drawing unwanted attention to the process. MSR staff also had to make choices concerning sites visited informed by a combination of insecurity concerns and trade-offs between time, costs and the numbers of participants that could be reached. Data collection proved challenging with communication limitations and difficulties accessing project participants. The quality of the data itself was influenced by a number of factors that potentially present them in a more positive light than perhaps is the reality. These include the limited opportunity for triangulation of self-reported change, the Hawthorn effect where participants attempt to please the interviewer, as well as other potential cultural biases. The data has also necessarily passed through several layers of translation and processing. FINDINGS AND CONCLUSIONS A combination of external and internal contextual features hampered the project in achieving its goals effectively and efficiently. These have also constrained the project from making more strategic choices in project design to address broader needs of civil society within a rapidly changing environment. The project was late in developing its final, more focused form resulting in missed opportunities to increase its impact. It was also not able to create synergies with other civil society strengthening initiatives and could have taken a more innovative approach in the delivery of the training programs. The data analyzed from the field survey is consistent with previous WL internal findings and suggests that there has been significant positive change at the individual level, with a large number (75%) of participants reporting personal transformations in their behaviors and illustrating these changes in applied learning from the courses that they attended through strong examples. The changes in behavior include the application of technical skills in the workplace, sharing knowledge and skills with colleagues both in the workplace, as well as more broadly in the community and with other organizations formal and informal. At the organization level of impact, the picture is a more mixed with 25% of the participants reporting positive change and citing strong examples from their experience. The level, depth and longevity of these changes are difficult to assess given some of the complexities associated with attribution and a lack of project definition in what sort of institutional change was expected. Barriers to change encountered by participants included inadequate targeting of the right participants, cultural resistance and inertia due to rigid hierarchies and course participants often being junior with little opportunity to be able to implement change more systemically. Later in the project, WL paid more attention to an organizational approach to leadership change. iii At the community level, the data reveals that 80%2 of respondents consider they have had no impact at the community level. However, a number of activities inspired by the project took place in the community more broadly. While these were sometimes limited in scope to the sphere of influence of the individual alumni, there were others that demonstrate effective collaboration of several alumni working together for change. The project was relevant to the needs of civil society in the country context but suffered in its management as a result of varied agendas, understandings of the project and its goal, and communications between and from the various institutional stakeholders involved. Lack of consensus in whether WL was implementing an education program under the guise of a civil society program or a civil society program under the guise of an education program caused it to struggle to achieve its goal in the early years. The project was most effective in the last 18 months, with dramatic improvement in documentation, management and program focus. During this time the project undertook a number of initiatives that showed potential in addressing some of the needs of civil society. It provided new courses for different types of civil society leadership, and improved targeting, advertising and recruiting course participants. Other limited but innovative activities such as the annual alumni conferences and the civil society round tables also took place during this phase. These were forums where trust and relationships between different sectors of civil society were developed and they provided a much needed opportunity for leaders to discuss and debate pertinent issues affecting society during transition. While it was not possible to quantify the impact of these events, it was clear that they were servicing a need. In conclusion, BODEP contributed positively to civil society strengthening in Burma. It had a strong impact at the individual level, though with diminishing levels of evidence for impact at the organization and community levels. It did however, miss opportunities for increasing its levels of effectiveness and impact because of a lack of strategic leadership. If the improvements that were made later in the project were commenced sooner, other initiatives could have been developed to better achieve the project goal. SELECTED RECOMMENDATIONS Operating in Oppressive Environments 1. Deliberate opportunities for strategic reflection should be integrated into projects. It is easy to absorb and unintentionally reflect a narrow view of what is happening or what might be possible in oppressive environments. It is therefore imperative to triangulate information regularly with a range of stakeholders to ensure contextual analysis informing project decisions is firmly rooted in a consensual reality. This may allow more informed risk taking to enable the exploitation of strategic opportunities. 2. Project goals, intentions, and expectations should be clearly documented. Ensure that particular attention is made to articulating clear project goals, intentions and expectations in project documentation even if they reflect a diverse set of objectives that are not cohesive. Strong project documentation reduces the risk of misunderstandings and shifts the emphasis onto the safe management of information rather than avoidance of documenting project issues and learning. 3. Build flexible mechanisms into project that continually assess project relevance and adapt to context. Flexible mechanisms need to be built in to programs to take advantage of opportunities and opening space. The project did certainly adapt as it evolved; however, it is also important to reflect the actual circumstances and potential impacts more appropriately in the evaluation model. Emerging from the Project Experience 1. Ensure foundational civil society mappings and assessments are undertaken to inform programs. The project did not have a strong idea of the range of different organizations and the state of civil society across Burma to inform its recruitment strategy or positioning. A mapping exercise, undertaken at either the sector level or State by State, would provide the foundation for a more informed program design for any future civil society strengthening program. 2 This figure combines those who definitely say that they have not had any impact at the community level (47%), with the figures for those who say that they have had impact (33%) but the examples that are given are not significant (e.g. at the most extreme end – change in one student is cited as community ‘change’) or are examples of organisational change rather than community change, or changes that cannot be attributed, even by a stretch of the imagination, to the courses. iv 2. Concentrate program efforts outside of Rangoon at the State level. The project provided a degree of training for those outside of Rangoon; but in the future this needs to be increased in the form of affirmative action for those who have not been able to benefit from these opportunities. 3. Increase the opportunities for people to come together across divides. The project started networking initiatives around the fundamentals of a strong society - tolerance, diversity and building trust between ethnicities and religions. These initiatives should be developed significantly but with the intention of addressing and discussing them jointly. A series of forums should be developed at Division, State and National levels that deliberately bring together leaders from different sectors, government, civil society, religious and business leaders in a safe environment. 4. Target organizations rather than individuals for capacity building. It is more efficient and effective to target organizations rather than individuals to build capacity and leadership because there is a formal structure within which participants can practice their skills and with more chance of developing impacts at higher levels. 5. Invest in Developing a Cadre of Top Class Local Trainers. Whatever form of civil society strengthening program is undertaken, there will be a need for capacity building skills in the sector. It would be sensible for USAID to invest in capacity building in whatever types of projects it supports. Strengthening Civil Society - Strategic Choices for the future 1. Ensure that ‘Do No Harm’ and Conflict Sensitive programming principles are applied at all levels of USAID programming. There are no shortage of opportunities for USAID to make a positive contribution to the development of civil society but almost universal feedback from key informants was the anxiety that USAID may potentially do damage unintentionally in its programs. The fear expressed was that this will occur either through an overestimation of the absorptive capacity of the sector or through the potential creation of distortions in the environment. Therefore, ensure that whatever elements are incorporated into the USAID portfolio, a combination of ‘Do no Harm’ principles coupled with conflict sensitivity are integrated into all aspects of USAID programming. 2. Foundations for a balanced society. All members of Burmese society have been damaged by their experience of oppression and the military regime. When considering the cumulative effect of the military government’s oppression of the people one can express two broad approaches that are desperately needed across the country to counter the legacies and long term enculturation of distrust, and fear found in the current relationships and interactions between people: (a) the ‘normalization’ of discussion, or the ability of people to be able to discuss, debate, talk and express opinions about any topic without fear of reprisals or negative consequences; and (b) the ‘normalization’ of organization and gathering together when people can gather, travel, and meet others without suspicion and fear. These two elements are fundamental to democratic and healthy governance environments. 3. Recommendations for the Longer Term. To address the legacies of oppression and the development of a viable long term governance model, it is recommended that USAID invest heavily in two streams of programming: Invest in Peace-building and Stabilization programming and Invest in the Provision of Information and Development of local media. 4. Recommendations for the Medium Term. There are three critical issues looming on the horizon that can become the triggers for rapid violence if the potential sting is not taken from them starting now. This is (a) the ‘pilot’ census in 2013 (b) the full census in 2014 (c) the elections in 2015. Even if violence is avoided at each stage, poor processes and issues can become latent proximate causes for conflict that await a later trigger. Given the lack of information about these processes in the past and the poor communication network in Burma, it would be prudent to start addressing this issue now before time slips away. This would allow a far more deeply informed process to take place and a more informed process is likely to be one less characterized by rumor or manipulation. 5. Recommendations for the Immediate Term. Given the rapid investment of foreign money and the relative richness of Burma, resources are a potential touchstone for conflict. This means that natural resource management, the environment and land reform are critical areas of governance. These sorts of issues (perhaps lower profile ones) offer opportunities now to learn and practice positive engagement processes between government, business and civil society sectors. In other words, an issue-based development approach at the State level would bring stakeholders together over local issues that would enable learning and ‘managed’ communication to take place around a process that is aimed at finding solutions, through negotiation and mediation while continuing the process of breaking down mistrust and fear and enhancing collaboration. 1 1. EVALUATION PURPOSE & EVALUATION QUESTIONS International Business and Technical Consultants, Inc. (IBTCI) submits this performance evaluation report to USAID/RDMA as a deliverable under AID-486-TO-12-00003 following the Final Performance Evaluation (Burma Outreach and Distance Education Project) Scope of Work. 3 The Burma Outreach and Distance Education Project (BODEP) has been implemented by World Learning (WL) since 2007 in a rapidly changing context, and has aimed to address the increasing need for civil society leaders who can manage change, create networks, mobilize communities and build coalitions. The drivers of these context changes have been a combination of factors including: opportunities arising from the opening up of the operating space in the broader context, utilization of learning gleaned from project monitoring and evaluations fed back into adaptive management loops, as well as considerations from strategic reflections and planning sessions. In this regard the project has sensibly adopted an evolutionary, iterative process and project progression to respond to the sensitive and challenging environment of Burma over the last five years. As a result, this final evaluation, which reflects the environment and complexity of the cultural and political context in which the project is situated, employs a variety of methodologies to assess the success and impact of the project at higher levels, while simultaneously being mindful of the differing ecosystems within the broader civil society landscape in which the participants and expected beneficiaries are operating. 1.1 EVALUATION PURPOSE This performance evaluation, which took place in October to November 2012 prior to the November 30, 2012 completion of BODEP in Burma, is intended primarily to learn to what extent the project’s goals and objectives have been achieved. The evaluation builds upon previous monitoring data already captured by Performance Management Plans (PMP), previous internal evaluations conducted by WL, and it provides insight on the validity of the project’s development hypothesis and results framework. The project evolution reflects a desire to move impact of the project’s activities from an individual level to organizational change to wider community change. The evaluation captures how this impact may have been made. Because the project’s transformation occurred during a rapidly changing environment in Burma, this evaluation also provides some insight into the contextual changes that have happened simultaneously, and how these may have affected programming. On a higher level, USAID requested this evaluation to understand the efficacy of WL’s approaches in Burma to add to the growing body of knowledge informing strategic future civil society programming in Burma. While the cooperative agreement with WL ended officially on November 30, 2012, this evaluation provides insights into best practices and barriers to civil society programming in Burma. As a secondary higher level purpose, the evaluation is also a learning opportunity to explore possibilities for working with and building the capacity of local Burmese evaluators for future evaluations. To accomplish this aspect of the evaluation, Myanmar Survey Research (MSR) was brought on was brought on to collect evaluation data and receive training from the evaluation team. 1.2 EVALUATION QUESTIONS 1. Is the training of community leaders actually contributing to strengthened organizational capacity4 in the organizations where project alumni are working? 2. What are the key barriers to community leaders using acquired learning and networking competencies in their work? 3. In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Are there signs that the project alumni will continue to apply knowledge and skills attained after the project’s end? 4. During the different stages of the project, and in light of the rapidly changing contexts within Burma, 3 Evaluation Scope of Work (see Annex I.) 4 Strengthened organizational capacity will be used as a proxy for strengthened civil society, the ultimate goal of the project; Evaluators may choose to explore other relevant proxies for strengthened civil society. 2 what approaches have been the most effective and efficient in achieving the project’s goals? Which approaches have been the least effective and efficient? 5. What were the unintended consequences of the project, if any? 2. PROJECT BACKGROUND 2.1 Identifying Project Information 1. Program: Strengthening Civil Society 2. Project Title: Burma Outreach and Distance Education Project: Capacity Building for Change 3. Award Number: Cooperative Agreement 486-A-00-07-0007-00 4. Award Dates: September 21, 2007 – November 30, 2012 5. Funding: $2,454,632 6. Implementing Organization: World Learning 2.2 Problem or Opportunity Addressed There exists an increasing need and demand for trained Burmese nationals to be development and community leaders in the rapidly changing environment of Burma. The influx of foreign aid into Burma after the 2007 Cyclone Nargis was a key in-road for initial programming in Burma, and also a key trigger for this demand for trained Burmese nationals who are able to gain access to areas otherwise inaccessible to foreign workers. With the recent developments in Burma, this need to go beyond relief efforts to sustainable community development and ultimately with leadership from Burmese civil society, is more essential than ever. Burma’s civil society, however, is still in its beginning stages of redevelopment and currently lacks the leadership necessary to effectively move towards an advocacy-centered civil society. BODEP was initiated to address the need for civil society leaders who understand and practice modern theories and practices of change management, networking and coalition building, mobilization and leadership. 2.3 Target Program Areas and Groups The target population was modified through various iterations of the project in response to myriad contextual changes in Burma, as well as staffing and programmatic adjustments. The project was originally intended to provide opportunities for influential Burmese citizens with restricted access to higher education opportunities due to the political environment. Original recruitment efforts cast a broad net for individuals working in related professional fields (namely English language teaching) and demonstrating specific technical skills. In some cases, particularly during the project’s nascent stages, there were pre￾determined populations from which to recruit participants without a formalized recruitment plan. This later changed to include participants from a much broader field of civil society and underlying criteria to target participants in each of the project components requiring all successful applicants to be connected in some manner with a wider network of individuals and/or organizations. 2.4 Burma Outreach and Distance Education Project Goal The goal of the project, as articulated in the program description in 2011, to develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society, was addressed through five major components, including: 1) English Teacher Training, 2) Training of Trainers, 3) Distance Education, 4) Leadership, and 5) Networking. The project activities were nearly wholly capacity building inputs to a range of individual Burmese community leaders, with some support to Burmese civil society organizations. 2.5 The Burma Outreach and Distance Education Project Intended Results The 2011 project goal was based on the following development hypothesis: If a diverse cadre of competent community leaders is developed through participatory training and networking, then civil society will be strengthened in Burma. While this hypothesis gives an overall picture of the project’s goals, it is difficult – if not impossible – to prove causality and effectively measure strengthened civil society. A more testable hypothesis is the following, which can be considered as a proxy development hypothesis of the aforementioned: If learning and networking opportunities are created for a diverse cadre of competent community leaders, then community leaders will use leadership competencies and networks in their work. This evaluation took it a step further to test the hypothesis that trainees will actually contribute to 3 increased organizational capacity where they work, used as a proxy for strengthened civil society. 3. EVALUATION METHODS & LIMITATIONS 3.1 Methods The evaluation employed a mixed methods approach designed to collect data to inform a rich response to each of the evaluation questions5 . Three main methods were used: • A review of project literature and academic literature in the public domain provided background analysis on the state and evolution of civil society in Burma over the past 10 years or so. • Quantitative data from project monitoring and evaluation data, where available, was analyzed along with a quantitative collation and analysis of qualitative data that was obtained through interviews and focus group discussions. • Qualitative data obtained through a broad perception survey conducted by local partner MSR, focus group discussions and key informant interviews was analyzed. The survey questionnaire and interview guide for alumni were informed by Kirkpatrick’s model of assessing training levels among participants as described in the following text box. 6 The data collected using the above methods consisted of 13 focus group discussions, 202 face-to-face alumni interviews, as well as 35 key informant interviews. 7 Table 1 below illustrates which data collection methods and tools were used and the input into the evaluation by the various evaluation stakeholders. Table 1: Stakeholders and their role in the provision of data Stakeholder Data Collection Methodology and Tool Input into the Evaluation WL Project Staff Methodology: Individual Interviews Tool: Interview Guide 1 (in￾depth semi- structured) WL staff provided: • insights into areas of learning from the project programmatic experience • reflections on challenges and successes in project logistical implementation • insights into the potential future directions that civil society strengthening projects might take to build on this project • guidance on project timelines and history Alumni from the different training programs: • Leaders/founders of Civil Society organizations • CSO staff • Community leaders • Teachers and others Methodology: Individual Interviews Tool: Interview Guide 2 (in￾depth semi- structured) Alumni provided: • Insights into how the project has been useful in their work and lives • Insights into how they have applied course content • Their perceptions on the extent of impact they feel has been achieved • Analysis of data from the perception surveys was also intended to understand variations in findings dependent on variables such as geography, gender, ethnicity, and course. 5 See Annex II for the Evaluation Design Strategy. 6 See Annex III for Data Collection Instruments. 7 See Annex IV for Sources of Information. Kirkpatrick’s Four Levels of Change Reaction: What were your initial impressions towards the training? Learning: What additional knowledge have you gained with the training? Behavior Change: How has the training affected the local community? Results: How is the training knowledge being passed on to those who have not experienced it? 4 Key external Informants: • USAID technical staff • INGOS who may also be partners of the community groups • External Trainers of other similar programs • Civil society leaders Methodology: Individual Interviews Tool: Interview Guide 3 This set of stakeholders were able to provide: • perspectives on technical content of the trainings and methodologies, • Triangulation on program efficacy as well as strengths and gaps in courses. • Perspectives on future directions for civil society strengthening programming. • Partner organizations were also able to provide perspectives on levels of attribution to change, where other complementary inputs or supports may have been provided and importantly observations from potential changed behaviors in the field from the trainings. Staff and leaders from the organizations that benefited from the project Methodology: Individual interviews Unfortunately it proved difficult accessing this group but a few leaders of organizations were interviewed. They were able to provide perspectives on course participants’ behavior change and the extent to which they have implemented change in the organization. Communities that have benefited from services or programs undertaken by the organizations Methodology: Focus Group Discussions Possible Tools: Participatory tool 3. Unfortunately it was not possible to access groups of beneficiaries, partly due to sensitivities regarding the informal nature of the project and partly due to logistical issues. 3.2 Limitations 3.2.1 Limitations concerning sampling and choice of locations visited There are a number of limitations to the data, trade-offs and constraints affecting data collection. The initial intended sampling approach was to obtain a random sample of alumni to interview, while simultaneously trying to obtain a balance between a sample of participants who participated in both short and long courses, a range of ethnicities and a reflection of the geographic scope of the project. In practice this was not easy to achieve. Firstly geographic sampling and locations visited were shaped by trade-offs around budget issues, time constraints, and security issues. 8 This meant that locations visited by the local partner MSR to conduct face to face interviews and focus groups only included Rangoon, Lashio and Mandalay; although some participants from Sagaing and areas further afield were able to travel to these locations to be interviewed. The external evaluation team members, unfortunately, were not able to travel outside of Rangoon for the purposes of this evaluation as they were not in a position to request formal travel permission. 9 Permission, strictly speaking, is also required for local organizations such as MSR, but there is considerably more leeway provided to nationals in this regard. Secondly, it proved exceedingly difficult to obtain a sound array of alumni from each of the courses due to the difficulties of contacting and following up with them. On the positive side, this resulted in a relative randomization by default with so many people being contacted to reach the agreed sample size. It also meant, however, that the number of respondents sampled in Rangoon was higher than planned, resulting in diminished representation of other regions. 3.2.2 Limitations concerning access to stakeholders External key informants were chosen for their deep knowledge of the state of civil society in Burma and strategic overview of the political situation. Their names emerged as a result of informed suggestions from a range of project staff, USAID staff as well as referrals from Burmese civil society leaders. The main difficulty was accessing them due to the extraordinarily busy nature of these key informants, the rush of 8 For instance it was inadvisable to travel to Rakhine State and Kachine State due to the ongoing conflicts in those areas and associated difficulties. 9 As the WL project had no formal MOU with a counterpart ministry, which meant that permission to travel to these locations to conduct interviews or focus groups was not possible. 5 external interest and the recent number of visitors in Burma trying to position for business or provision of aid. Organizations visited to triangulate alumni reporting were initially chosen for purposive reasons, but again due to difficulties in communication and availability of key persons meant that at the practical level, the lead evaluator often had to make do with those who were able to make the time. This was not necessarily a problem, but it meant that a different set of organizations were visited than intended. Access to communities and peer staff from the organizations that work with participants was not possible for a variety of reasons, including the informal nature of the project, the fact that many of the participants did not necessarily belong to organizations, or the organizations were difficult to access. 3.2.3 Limitations concerning data quality There were also a number of factors that potentially shaped the type of data obtained through the sampling methodologies: (a) Self-Reporting: Most of the data is based on self-reporting. Given the context, it was extremely difficult to triangulate effects with any certainty. This was partly because of the diffuse nature of the project and how individual participants were selected (particularly in the earlier part of the project). It was also partly due to the difficulties associated with not being able to visit communities and many of the organizations10 to discuss their experiences. Even with hindsight it is difficult to see how this may have been mitigated effectively. (b) ‘Hawthorn Effect:’ WL acknowledged in a number of their reports the difficulties associated with the ‘Hawthorn Effect’. This is where respondents’ behavior and responses during the course of the exercise are as a result of the social situation and context itself. In this case it takes the cultural form of ‘trying to please’ the interviewer. Thus a person being interviewed is more likely to say something positive about the project that they think the interviewer wants to hear. It is difficult to mitigate this bias, but an attempt was made to shape questions in the perception survey to confirm responses and draw out contradictions. On the ground it proved even harder than anticipated to address these issues as triangulation methodologies were difficult to undertake. (c) Contextual Biases: Some informants considered that interviews and focus groups undertaken by Burmese might not reveal rich information compared to those undertaken by externals. This was considered to be due the lack of experience and trust between and among nationals compared to the possible openness in expressing oneself to a foreigner. Foreigners may also be able to ‘dig’ for information more forcefully being less bound by cultural sensitivities. Foreign interviewers however, may lack the ability to ‘read’ or interpret responses and signals correctly and there are inherent communication difficulties in interviewing in a second language. (d) Cultural Biases: Similarly there are potential cultural biases and issues faced by national interviewers. For instance it is very difficult for a young interviewer to question deeply and insistently an older respondent as this may be considered rude. National interviewers undertaking interviews with those from an ethnic minority may also be affected by prejudices and the interview may be biased from both sides in how they react to each other. 3.2.4 Limitations with the Perception Survey The data collected through the perception survey passed through a number of filters before being analyzed by the evaluation lead and this likely affected its quality. The data capturing process for the perception survey was as follows: An interviewer noted down a respondent’s answer, most likely in Burmese (which may also be the second language for some of the respondents). These notes were then translated from Burmese into English by a translator. The translation was then reviewed and ‘cleaned’ by MSR senior staff and finally sent to the lead evaluator for collation and analysis. He, then, applied a ‘cleaning’ course of action by filtering out what was relevant data and transforming it into succinct 10 For instance because the project was not official, it was not possible for the lead evaluator to openly talk with communities as without formal government permission this would raise issues. Some organizations, especially in the early days, were also unaware that their staff had even received training. 6 information for reporting purposes. This process was inescapably reductionist in nature and most likely meant that some of the nuanced aspects of the experiences being reported may have been lost. It is also important to note that the monitoring and evaluation data from WL was not as comprehensive as anticipated. This limited the ability of the evaluation team to use it for the purposes originally described in the evaluation design strategy through a time panel data analysis. Discussions with WL revealed that the reason for the lack of data is partly due to the same challenges encountered by MSR including difficulties in contacting and communicating with alumni (e.g. changed email addresses and phone numbers, lack of access to internet facilities for alumni in rural areas, poor physical communications infrastructure, etc.) and the lack of responses from respondents. This meant that WL could not collect as much data from post-course surveys as originally intended. 3.2.5 Limitations of the overall scope and design of the evaluation As noted by some key informants and the evaluation team leader the scope of the design and methodology of the evaluation may not have been the most cost effective and appropriate way to learn lessons from this project for a number of critical reasons including the following: • The continual evolution of the project approach, content of the materials, intent and design of the project to achieve the objectives, meant that there were no constant benchmarks against which to measure impact over the course of the project. The evolution and iterative approach of the project was not a weakness in itself – in fact it was a strength – it was more a reflection of the context, perspectives of the key stakeholders and personnel at that time, as well as the legal and contractual background informing (and sometimes constraining) what was possible. • Importantly, the evaluation was considering impact at higher levels than the individual to gain insight into the extent to which the project contributed to civil society strengthening more broadly in Burma. The project was shaped to achieve this objective most directly in the last year and a half, with objectives prior to that also being informed by additional unwritten objectives (though important in their own right) and approaches. Nevertheless it is fair to say that organizational change through an approach such as this takes a relatively long time to achieve, let alone broader societal change that can take many years, being multi-causal and involving multiple factors. It is therefore optimistic to expect to see societal change in a period of just over a year for such a project. This meant that the investment in the methodology to capture this change was over ambitious. It is ironic that adopting a very extensive and rigorous evaluation approach mirrors some of the tensions within the project approach itself. BODEP was ostensibly a civil society strengthening project with a goal to develop society, however the project design and self-imposed parameters within which it was set, made it very difficult to achieve the possibility of large scale impact on civil society until the latter part of the project. As a result, the evaluation methodology was more purposefully designed to emphasize a qualitative approach on the ground. This resulted in the information from key informants playing a more significant role in the data analysis than was originally anticipated. The qualitative approach generally ensured that the evaluation attempted to capture the nuance of change articulated by participants and observers as much as possible. This previously risked being lost in the search for higher impact given the short time frame of the project in its current form. 4. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 Findings 4.1.1 The Context – External and Internal There were major contextual features, both positive and constraining, that shaped the evolution and implementation of the project. This can be understood both in the context of this evaluation and in light of USAID’s intent to learn lessons for operating in such difficult environments. The environment inevitably affects the thinking, perceptions and behaviors of all actors that are working within it. This project is no exception. The relationship between a project and the environment, at its best, is a two-way street with programs both responding to the environment and where possible, deliberately attempting to affect the environment positively. For the WL project there were a number of ways that mutual influence led to consequences for the impacts, results and the relative import of those results. It is critical to understand these elements in considering the best way to design future civil society strengthening projects. 7 The US Legal Context The ability of BODEP to achieve impact in terms of strengthening civil society was influenced and constrained by the specific US legal framework with respect to Burma. The Omnibus Appropriations Act 1997, re-codified annually, constrained the types of participants involved in the project. For instance, BODEP was unable to work with teachers or principals from government schools. This is not the place to debate the rights and wrongs of sanctions and such mechanisms in terms of their efficacy in achieving their intended ends; however, a greater degree of analysis invested in the approach on how best to strengthen civil society and prepare and support potential long term change in Burma, rather than a punitive approach, may have resulted in a different set of legal statutes, or possibly, exemptions that would have enabled a different set of actors to be involved in this type of project.11 The Context of Oppression The effects of long-term political oppression are expressed differently by people in different cultures around the world. Responses by those suffering are also different, being shaped by culture and history. Nevertheless there are many similarities and commonalities across countries that have experienced political oppression which deeply affect the social and governance environment. Some of the features that have been prevalent over the course of the project (and still are operating) in Burma include: Trust: There is a deep lack of trust between people outside of immediate family and friends resulting from the uncertainty of never being sure to whom one is talking, who may be listening, and whether you may be reported to authorities. The lack of open communication and trust creates dysfunctions and divisions deep within society that then affect relationships between ethnicities, religions and people at all levels. Fear: There is a fear of taking any initiative or decision within your job or leadership in other domains. The source of this fear is intimidation and the potential negative consequences of making a mistake or misreading a senior’s intentions or desires. If outside government, then one fears upsetting someone in government or drawing unwanted attention from authorities. The consequence is institutional inertia and centralized decision-making at the most senior levels, even for the most trivial of decisions. Self-censorship: An atmosphere of self-censorship has been and continues to be present in Burma resulting from (i) a real fear of punishment from the government (encouraged by occasional high profile, disproportionately punitive examples) as well as (ii) the government policy of imposing fiercely restrictive legislation that is effectively impossible to follow. This means that everybody is already ‘guilty’ of not abiding by the law, which provides opportunities for government security forces to arrest people on an ostensible pretext (if they feel they need one). 12 Risk-taking: More broadly these features create an atmosphere where ‘normal’ risk-taking in any field is constrained. It reduces the ‘fields’ of change visible in an environment to the micro-level where enormous energy and time are expended by all actors on small and often apparently inconsequential aspects that become more significant and symbolic. This has been deliberately shaped by government. For example, having over-burdensome bureaucratic fulfillments for simple things (e.g. travel of project staff, or registration of a CBO) so that extensive time and energy are allocated to these tasks that could be better spent on other activities. It also means that the import of small events or processes, superficial and apparently insignificant, is enhanced, so the subtleties and interpretation of what is happening in the broader environment and what it means, becomes more important. The political topography is not uniform 11 For example if one is intentionally trying to change the dynamics between government and civil society, then a program primarily working with freelance teachers and those operating out of private schools may not be the most practical a first choice of target as evidenced by the way that later project iterations shifted dramatically to target a wider range of participants. Here, a clearer development hypothesis would have clarified this. 12 An example can be seen in the old colonial law that requires anyone to register with the local police if staying in a locality that is not where one is officially registered. This means that if you want to stay overnight with a friend somewhere, you basically need to go to the police. In practice no one does this as it is burdensome and bureaucratic but the law is still in place and recent attempts to lift or change it have been resisted. Thus anyone travelling can be apprehended if the authorities wish to hassle them or their friends or exercise petty power. The result is control of movement and people’s ability to organize. Key informants noted that there has been a recent effort to repeal this law, but the effort failed being outvoted in parliament. 8 across the country and this increases the subtleties of interpretation on the meaning of events. Thus from an external perspective, the skill lies in getting the balance right between levels of interpretation, and between specificity and generalization of interpretation from the micro to the macro level. 4.1.2 Relevance of the World Learning Project to Civil Society Needs in Burma Given the rapid changes in the Burma context and the continual evolving nature of the project over its life, it is important to assess the relevance of the intervention as a civil society strengthening program during its different phases. Is it an education or civil society project? This is where the project struggled to overcome its inception and early stages. According to key informants from WL and USAID the project originally emerged as a mechanism to support political dissidents who were imprisoned and whose education was curtailed. It was not a development project. Support was therefore provided to these participants through the distance education component with a further component more broadly being the provision of Qualification in Language Training (QuiLT) through a School for International Training (SIT) set of modules to English Language teachers. This latter component was considered a ‘safe conduit’ within the oppressive environment to build potential leaders of civil society. Two key USAID informants noted that the project suffered from a repeated effort to ‘fit’ the project retroactively into a civil society strengthening project shape rather than an education project. This view is supported by an analysis of the early project documentation that revealed the following: • No formal development hypothesis • Little apparent analysis of or reference to the needs and state of civil society in Burma • Quarterly reports having a focus on the educational progress of participants • The content of the material in the Associate Arts course for Indiana University. This is of a generalist arts nature as opposed to being tailored to the needs of civil society leaders • Little attention to targeting and recruitment of course participants with respect to civil society leadership This is in stark contrast to the strong, positive, project documentation in the latter half of the project where the efforts of WL and USAID from late 2010 were far more rigorous in attempting to consider the needs of civil society and the types of participant, materials and courses that might be useful in the development of potential or existing leaders. The improvements include the following: • The articulation of a different program description • The introduction of a rigorous project monitoring framework including pre and post course surveys • The introduction of a recruitment and targeting strategy to guide who is best able to benefit • The introduction of new or adapted courses, materials and initiatives • The introduction of specific course metrics on skills Reflecting Changes in the Context at the macro-level: Table 2 shows that key project events and contextual events do not reveal any substantial and obvious correlation with the evolution of the courses. This is perhaps not surprising, as one would expect changes in the civil society ‘space,’ either expansions or reductions, to be lagging behind critical political events. It is the case that the project did ‘open up’ in terms of courses, personnel and relevance to civil society more broadly as time went on, and this is indeed a reflection of changes in the context but the question is the timing and timeliness of these changes. It appears that factors such as changes in personnel and project management (including USAID) appear to have been far more important than external factors in determining the relevance, success and different approaches encapsulated in the project. This interpretation is also further borne out by a consideration of when other civil society strengthening projects adopting a different approach were initiated. One would like to be able to say that the BODEP design was informed as a complement to these other approaches, but there is more evidence to suggest that there was little consideration of, adaptation to, or synergies to other initiatives being implemented at the time. In fact a key criticism of the project is that there was little in-depth contextual analysis and needs assessment to inform design and approach. Reflecting Changing Needs at the Micro-level: Given the lack of opportunities in the environment generally, it is true to say that the project was filling a large need even if there was a failure of specificity. Nevertheless confirmation of this is reflected in the feedback obtained from focus groups and the 9 interviews where general satisfaction with the course content was expressed (with some caveats). The most important indicator that the project was important and relevant is the fact that 75% of the sample of participants reported they are using some course content in their work and lives. This is a significant very positive and important finding. There was also an attempt to shape course material to meet the needs of the Burma context (as reported by WL and ex-WL staff13). For example even in the early stages of the project the ‘generic’ QuiLT course was deliberately adapted to overcome the known shortcomings of a very traditional education system in Burma that relies heavily on teacher-centered pedagogies. This resulted in an emphasis on experiential learning and reflection/praxis approaches. Later changes to courses were more mindful of the specific needs of broader civil society as evidenced by the introduction of the Community Development course. Table 2: Mirroring Key Project events and broader contextual events affecting Civil Society Date Key Project Events Key Civil Society / Political Events 1988 • 1988 uprisings pro-democracy 1989 • State Law and Order Restoration Council (SLORC) declares martial law 1990 • National League for Democracy (NLD) wins landslide Victory in elections but it is ignored by the government • Aung San Suu Kyi party wins 392 out of 492 seats 1991 • Aung San Suu Kyi awarded Nobel Peace Prize 1998 • 300 NLD political prisoners released 2000 • Capacity Building Initiative formally begins projects (prior to that it was Myanmar Development Research Institute, or MDRI, started in 1996) 2002 • Aung San Suu Kyi released 2005-6 • Pyoe Pin Civil Society Strengthening Project begins 2007 • Project begins • Paung Ku Civil Society Strengthening Project begins • Monks protest across country – anti-government and over fuel prices (Saffron Revolution) 2008 • Training of Trainers (TOT) Project developed and started • QuiLT extended for civic ed. Modules • Cyclone Nargis occurs • Referendum on new constitution takes place 2009 • Project Description revised (TOT + Distance Education or DE) • 270 activists sentenced to long jail terms for their roles in 2007 protests and helping victims of cyclone Nargis • Aung San Suu Kyi house arrest extended for 18 months • Aung San Suu Kyi begins talks with the military Government • Oxfam Civil Society Development and partnership project begins 2010 • Foundation Cycle Training of Trainers (FCToT) developed • Election occurs, main military backed Union Solidarity and Development Party USDP wins and opposition says election was a sham • Aung San Suu Kyi released 1st qtr 2011 • First Annual Project Conference • First Impact Evaluation • Targeting and recruitment strategy • Recruitment Applications revised • TOT core principles course implemented (intensive) • Tin Oo (NLD Vice chair) freed • Thein Sin sworn in as President and leader of the new civil government 2nd qtr 2011 • QuiLT off-site • Two new modules on civic education and “returning to your 13 Although it is not entirely clear on what basis these choices were made. 10 community” offered • New evaluation criteria designed and applied for FCToT participants regarding course competency • Introduction of Monitoring and Evaluation pre/post course surveys 3rd qtr 2011 • Five-day civil society workshop based on the Community Development course • Use of Burmese in FCToT • First delivery of 10-week 2 days per week FCToT • President Thein Sein meets Aung San Suu Kyi in Nay Pyi Taw 4th qtr 2011 • Civic education module not included in QuiLT for security reasons • First local TOT trainer certified • Myitsone dam suspended • Some political prisoners freed • New labor laws passed allowing unions • Hilary Clinton visits Burma • Law signed to allow peaceful demonstrations 1st qtr 2012 • Three Burmese achieve WL/SIT certification • Second Annual Project Conference • Second WL impact evaluation • First CD course • First local trainers conduct QuiLT • First QuiLT alumni conference • First civil society roundtable • Karen National Union (KNU) sign ceasefire with the government • Chin state government signs peace treaty with Chin National Front (CNF) • Shan State Army South (SSAS) signs 11 point peace agreement 2nd qtr 2012 • First FCToT run by Burmese • By-elections held - NLD sweeps them 3rd qtr 2012 • First QuiLT course run outside Rangoon 4th qtr 2012 • Second Burmese TOT trainer • First TOT Alumni conference held • QuiLT Alumni meet to establish a National QuiLT Association • Violence in Rakhine – over 90 killed 4.1.3 Efficiency of the Project Given the political environment and social context, was this the optimum way to implement and manage the project to achieve the goal? Were the management structures and systems appropriate? Were the project activities cost efficient and was this the most efficient way to contribute to civil society strengthening in Burma during this period? In this regard, like so many of the different aspects of the project given the way that it unfolded, it is very difficult to come to a balanced conclusion, nevertheless one can make a number of comments that provide some limited illumination on the subject. Strategic Direction and Responsiveness of the Project. The choice of education as the platform for a project strengthening civil society at the earlier stages of the project was very sensible, given the governance space available and the perceived ‘neutrality’ of the sector. There was also general consensus among project staff and USAID staff interviewed that this was indeed the most appropriate entry point. Nevertheless it is also reasonable to pose the question as to whether later adaptations could have been implemented sooner, or whether there were opportunities that could have been taken to strengthen civil society in other ways. In this regard, projects in oppressive governance environments need to find ways constantly to triangulate their understanding of the environment to avoid interpreting the environment too subjectively from one perspective. This requires actively seeking input from others, as well as creating deliberate openings for reflection on what are appropriate levels of risk or ways to ‘safely’ pilot or test initiatives. It may well be that the situation is not uniform and that there are other opportunities to take further in some places than others. In this respect the American Center (AC) played a pivotal role in how the project analyzed the context and possible project approaches in the political space available. The AC is an extension of the U.S. Embassy and is guided by their security constraints and interpretations (understandably conservative in its views due to its diplomatic mandate). The major sources of information came from project participants in daily 11 contact with project staff (most significantly the political activists on the DE courses). It is notable that an analysis of the quarterly reports, which listed all the meetings attended by project staff each quarter, reveals a lack of external input from and interaction with other agencies. A third significant factor shaping how WL considered the environment was their lack of formal registration as an NGO, coupled with a simultaneous lack of full support from being part the U.S. Embassy. 14 The combination of these factors created a level of vulnerability for project staff which shaped activities and perceptions of the possible.15 All categories of key informants (project staff, ex project staff, USAID staff, National civil society leaders, and expatriate ‘experts’) were unanimous in their opinion that while the space for civil society strengthening was severely constrained it was certainly possible to implement a range of different training and organizational development activities even as early as 1996 but more realistically following the event of Cyclone Nargis in 2008. This would suggest that the project could have implemented many of its more successful elements far earlier in the project than within its last two years. Important examples of such initiatives include, the Myanmar Development Resource Program (1996) which then became the Capacity Building Initiative (1998), The Pyoe Pin Civil Society Strengthening Project (2005-2006), Paung Ku Civil Society strengthening project (2007), and the Oxfam civil Society Strengthening Project (2009) as well as other umbrella partnership programs supported by organizations such as Swissaid and Trocaire. In other words there were plenty of good examples to link up with, learn from and to build on well before the last phase of BODEP which turned the corner in late 2010. This represents a number of missed opportunities to shape the program in the light of these ongoing civil society strengthening projects. Strategic Leadership. The above situation points to a lack of strategic leadership within the project stakeholders as well. In this case with the number and arrangement of institutions involved in the management of a project (e.g. State Department on the ground, initial distance management by USAID from Bangkok, and the WL program being managed out of their education unit in the US rather than its civil society and governance unit, a clearer approach to strategic leadership could have been structured more formally. It appears as though stakeholders were unable to step outside their institutional perspectives and sit down to agree on a coherent approach to meets their needs. It is clear though that USAID and some WL staff tried hard to guide stakeholders to create a more cohesive programmatic position and approach but this appears to have taken the shape of trying to ‘fix’ the project rather than challenging the fundamental approach. As one WL staff member said, “we are doing now what we should have been doing all along.” In light of this experience and if a similar type of project (in an oppressive environment) is undertaken in future, it would be important to recognize and mandate such a role formally, whether it lies with the Ambassador, an advisory or management committee. Clear responsibility and accountability is required to make the most of the resources and the programming opportunity. At its best interpretation, the BODEP experience can be considered a strong positive example of cross￾fertilization between the stakeholders, as learning was indeed accomplished and the project advanced in quality significantly. Staffing Structure. The staffing structure was not as efficient as it might have been given the fact that all WL staff acknowledged that they were ‘too busy’ on administrative minutiae to invest in strategic training of local staff and the development of new materials which might have alleviated their work loads. Delays in recruitment and a lack of administrative support were key elements in this regard. Until 2010, the WL staff deployed on the ground were not trained in either development studies more broadly or the civil society sector. This bias towards the education sector was further deepened through headquarters oversight from the education unit rather than the civil society and governance unit with the result that the project found it difficult to find the best way to support civil society strengthening through education in the 14 The Embassy, for instance, did not provide visas for WL staff or banking facilities for the project. This meant that staff were reliant on local NGOs sponsoring them and could never be certain that they would be able to return on each of their numerous visa runs. These runs were also combined with having to bring large amounts of money into the country every trip. 15 An example of how this operates: WL staff was strongly (and wisely) requested by State Department not to attend the INGO leaders’ forums that occurred regularly. The basis for this suggestion was that (i) it would draw unwanted attention to WL given their lack of formal registration and (ii) that it would therefore be assumed that they were representing the AC – an extension of the Embassy – and this would be inappropriate for essentially formal government diplomatic staff to be at such a forum. 12 first few years. A further shortcoming was the lack of investment in the development of local trainers and national staff much earlier in the project life. This was one of the key approaches that could have combined a level of sustainability with flexibility on the ground. It was not until expatriate staffs were unable to travel outside of Rangoon, that the emphasis was shifted to development of national training staff (as acknowledged and reported by WL staff). This tardiness was exacerbated by the fact that expatriate staff reported feeling overwhelmed with keeping up with project administration and did not feel they had time or space to concentrate efforts in this critical area. A recurring theme within the perception survey has been the recommendation for local trainers who understand the context, speak Burmese and can travel to rural areas (these issues were raised by 24 respondents). The Role of the American Center in the project. The project has been run under the auspices of the AC in Rangoon. This has been fundamental to the project and has had both a positive and a constraining influence on it. On one hand, it provided a certain level of ‘cover’ and support for all participants; and on the other, it arguably influenced what was perceived to be possible. The most positive element mentioned by key informants and alumni was the creation of a ‘safe space’ (most importantly the political activists on the distance education courses), where the AC was one of the few locations where participants were able to express themselves freely if they choose to. Secondly, the AC offered education and personal development opportunities through this project and its other activities within an environment where such opportunities to expand the mind, congregate, and engage in normal intellectual activities were severely restricted. In the earlier stages of the project these two fundamental aspects were of enormous positive psychological value to a populace ‘under siege’. It was also well known by all, that the AC was under constant scrutiny from security forces and those who entered it were noted down and sometimes even hassled or detained on their leaving the building. Conversely, according to key informants, these factors may have influenced the type of applicant participating in AC programs, with a predominance coming from a certain demographic that can best be described as young, with time to be able to devote to studies, and willing to take the risks from being identified. This demographic was described and noted by several alumni and externals interviewed (six specifically). More recently the project was aware of this issue and actually interviewed applicants regarding their feelings on whether the venue was a significant factor restricting their participation, and on that occasion it was not. Nevertheless others felt that there were community leaders who did not apply for programs, not wanting to be ‘labeled’ by their association with the AC. Those of this opinion noted that if you are an older person attending AC programs, there was a risk of implications for family and friends as well as the leader’s standing in the community. The Project Approach. While the project was definitely filling a need, there were major questions surrounding the choices of approach that were chosen to achieve the project goal as well as the decision￾making processes for selecting participants. BODEP, largely a training and education project, was delivered by expatriates out of the AC. This biased the participant selection towards those from Rangoon in the early stages of the project (perhaps inevitably given the context at the time) as well as a very limited set of potential civil society leaders. Rather than going through each and every facet of the project, to illustrate the point and how restricting the project approach was, one could imagine a very different scenario to achieve the same goal in a difficult context. Imagine the role of expatriates primarily training local trainers and developing new course material. Twenty high quality national trainers (from different ethnicities) who are then are based in pairs in States where they conduct courses – even the same suite of courses. A further person is based in the State with the team to concentrate on developing networks, supporting alumni, mentoring, and assisting with secondary trainings delivered by alumni’. This sort of approach would have made it easier to target appropriate leaders at the State level and enabled development of stronger local networks. The center could then have concentrated on organizing exchanges, national events like annual conferences, civil society roundtables and other such events that could be delivered at the State level. Documentation, Clarity of Intent and Communications. The goal of a project, the way that it is to be implemented, its objectives and the expected results all underpinned by a development hypothesis is the traditional and formal way that stakeholders achieve a common understanding. The foundational documents that express these ideas and intentions become the fundamental reference points, are incorporated in the contract documents, and guide what activities are chosen and how they are undertaken. In the case of this project the early documents are vague, limited and ambiguous in this 13 regard. While this may have been deliberate due to the sensitivities of undertaking a civil society strengthening program in Burma, nevertheless until 2010 the standard of documentation was extremely poor when judged by the norms of development programming. This meant that the ambiguity expressed in different interpretations, intentions and understanding of all stakeholders were potentially distorted when seen through different institutional lenses, changes and turnover of staff, insufficient depth of handover, and distance management (e.g. involvement of USAID/RDMA in Bangkok). Communications between all stakeholders State Department, USAID and WL suffered as a result and contributed to the difficulties. This poor understanding and communication was seen in a number of relationships. For instance there was a communication and understanding gap between WL project staff at field and headquarter levels as to whether this was an education program under the guise of a civil society program or a civil society program in the guise of an education program. Project staff from the ground reported that they had been briefed by their own organization that it was the former rather than the latter and it was not until they were working with USAID staff in Rangoon that they started to understand the intention of the program and how it was funded. Cost Efficiency. The fees for a participant on the distance education IU component cost approximately $11,073 per person while the later DE – CSU course fees were considerably better value at $2552 (i.e. not including administration or staffing of the project). The cost of a QuiLT participant has been estimated (by WL) as being ca $2000 and short courses varied considerably. At the most basic level the average project cost per participant was US$162216. This does not take into account the cost of networking events which are included in the calculation and would reduce the figure per head, but equally costs were reduced through the in-kind contribution of overhead costs associated being located in the AC. A very broad comparison (but not necessarily that significant as it does not take into account other aspects such as quality of the course, quality of the training provision, etc.) can be found by comparing this cost to that of the Capacity Building Initiative (CBI)17 that provided training (and Organizational Capacity Assessments) for 449 participants from 60 different civil society organizations at an estimated $312 per person (in 2011) including all the overheads of the organization and including a holistic organizational capacity assessment approach to the capacity building of the organization. A comparison between the two basic figures, no matter how much it is comparing apples and oranges, does suggest that one has to ask the question of whether value for money was obtained from a cost per participant which was five times as much. 4.1.4 Effectiveness of the project As can be seen from Table 3 and Table 4 below, (as well as tables in Annex V for FY 2009-2011) there is quality data available from WL for the last three years, when sound monitoring practices were in place. It is difficult to get a clear picture of what was going on prior to that in terms of achievements, which has not been included or considered since the data from 2008 appears to be inaccurate. Nevertheless in the last three years the project has basically achieved all the expected project targets. In fact it has exceeded the majority of them. Where the project has underachieved, the numbers are not particularly significant, such as with the opportunities for internships where there is a slight underachievement. Table 3: A Summary of the Project Participants (Compiled from WL participant database) Year 2012 2011 2010 2009 Totals& Averages Total Participants 412 574 240 287 1513 Number of Participants from outside Rangoon 177 420 132 256 985 % of Participants from outside Rangoon 43% 73% 55% 89% 65% Number of Participants by ethnic minority 208 272 111 43 634 16 This is a simple calculation obtained from the total budget divided by the number of participants in the project period. 17 CBI is a local organisation initiated by a group of INGOs inn 1996 (then called Myanmar Development Resource Program) to provide training and organisational development services to their agencies. It is also interesting to note that both projects have trained staff from 27 of the same organizations between the two projects - the CBI total is from one particular year and the BODEP project overlap was from the whole database over the course of several years. Nevertheless this also supports the comment from a CSO leader that it is difficult to attribute any organizational change, if there is any, to AC alone 14 % of Participants by ethnic minority 50% 47% 46% 15% 42% Number of Participants by religious minority 132 157 49 13 351 % of Participants by religious minority 32% 27% 20% 5% 23% Number of Participants by women 294 377 103 94 668 % of Participants by women 71% 66% 43% 33% 44% Table 3 shows an increasing level of performance in quantitative terms. More importantly the percentages of women participants grew annually as did the percentages by religious and by ethnic minority, although there has been a slight drop from 2011 in the number of participants from outside of Rangoon. In many ways the most significant figures are those associated with the networking events. Putting aside what these events consist of and how modest some of them might be, they do nevertheless illuminate and provide insight into a number of important positive facets of the project by proxy. For instance one can consider that these reflect the following: • The extent that some course information is passed on to others who were not included in the original training. • The events serve as a proxy for the level of motivation and ‘leadership’ being displayed by course participants. This latter aspect is also borne out by the large number of events that are held without support from the project, despite the project having some funds and materials available on request. 18 • A degree of opening in society and increasing relative tolerance from the regime to civil society. • An indication of the thirst for knowledge and demand from people. As can be seen from the text box “Examples of Networking Events” (taken from the myriad reported in WL quarterly reports), there is great diversity in the type, size and scope of the networking events. The networking component also addressed some of the underlying issues and legacies of oppression by building trust between people, and opening the space for issues to be discussed in the public domain. Of particular interest were the annual conferences and civil society roundtables (one of three conducted is noted in the text box above). The annual conferences provided an opportunity to strengthen or consolidate the relationships developed in the courses between participants as well as extend them to a broader circle of people. In one of the two annual conferences held, the alumni invited ‘extra’ participants 18 This is interesting as resources were available from the project, but were not accessed as much as might have been expected by Alumni. One interpretation from WL is that this is a reflection of the Buddhist teaching where ‘good’ undertaken in this life will be reflected in the fortune of your next life. A more prosaic interpretation is that despite being informed of the availability of resources at the end of the course, distance or logistics made it difficult or prohibitive to access. Unfortunately the evaluation team did not have time to explore this dynamic more fully. Examples of Networking Events (compiled from WL project quarterly reports) • In 2010, 6 alumni received support to implement a World AIDS Day event in Rangoon for over 200 people. • A Teacher Training event took place in Myikyina in 2011 for a total of 56 participants. • Compassionate Hands, a local NGO founded by a program alumnus, conducted 3 activities in 2011 (a) provision of free medical treatment via a mobile clinic to 200 children in Rakhine State. (b) a further 160 patients in Irrawady Division. (c) Provision of an artisan well in Irrawady Division, serving an estimated 500 villagers. • AC Teacher Development Group held 12 workshops over this 3rd quarter (2011) for a total of 22 participants. • A Community Mobilization networking activity in Myaebon Township, Rakhine State was held 2011 for 40 participants. • A training “Building young leaders in rural communities” was conducted in Pyin Oo Lwin, Mandalay Division in 2011 for 25 participants. • A Civil society roundtable on Sept 2012 entitled “Press Freedom and Journalists’ Responsibility,” examined the critical role of media in an emerging democracy. • Mustard Seed (local NGO from Alumni) conducted a 5 day community based development knowledge and skills to 32 local community leaders and citizens in Chin State. A series of extra trainings covering organizational development, leadership training, community development, environmental awareness and human trafficking was given to more than 2,500 students, community members and local leaders. 15 outside of the alumni, which is a good innovation. It is difficult if not impossible to assess the direct ‘impact’ from such events, but in the opinion of the evaluation Team Leader (and drawn from considerable experience in a variety of transition contexts) these types of activities are critical in developing the foundations for a stable, functioning civil society. These activities served the following functions: • They ‘normalized’ gatherings of people together in large groups for to discuss issues and ideas. • They provided an opportunity for people to understand ‘new’ issues given the paucity of information and access to information in the environment • They provided an ‘injection’ of new content and material and • They provided opportunities for people to start to explore the possibilities for collaboration together. Table 4: Project Performance in Fiscal Year 2012 (Provided by WL Head Office, compiled from project M+E data throughout the project -Performance in the other fiscal years can be found in Annex V) Performance Indicator No. Performance Indicator Narrative FY 12 Targets FY 12 Actual Goal To develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society #1 % of course graduates who practice leadership competencies in their work 75% 98% #2 # of post-course training initiatives completed by course graduates 40 338 IR 1 Potential and current community leaders develop skills and knowledge competencies #3 % of course participants who achieve a rating of 3 or higher on course competency assessments 85% 96% #4 # participants who achieve a rating of 3 or higher on leadership component of course assessment rubric 40 56 IR 1.1 Potential and current community leaders trained on leadership and networking #5 # Course participants - Women 110 319 - Ethnic Minority 66 245 - # by city/town of residence 66 192 - Religious Minority 66 151 - Aggregate 220 456 IR 2 Potential and current community leaders trained on leadership and networking #6 # of participants who complete mentoring/internship Women 3 3 Ethnic Minority 3 4 Religious Minority 3 2 Aggregate 6 4 #7 # networking events held without support from the project 60 153 IR 2.1 Community leaders receive technical and material support for organization of networking events #8 # of networking events supported 20 47 IR 2.2 Networking events implemented directly by the project #9 # of networking events implemented 15 9 Color Code: 1. Achieved Project Targets 2. Overachieved Project Targets 3. Underachieved Project Targets 4.1.5 Impact and the Evaluation Questions Question 1: Is the training of community leaders actually contributing to strengthened organizational capacity in the organizations where project alumni are working? The project description and development hypothesis concerning strengthened organizational capacity is only specifically mentioned from August 2011, which does not provide much time for the project to have 16 altered its design to address this element rather than the broader goal of creating a diverse set of civil society leaders. This meant that the project had only started specifically targeting organizations with a formal recruitment goal very recently in late 2011 for example with the SCS course (this course was implemented from Feb-July 2012). For other courses, WL reported that started looking at the possibility in an “informal way” perhaps in mid-2011. In practice the targeting strategy meant trying to recruit more than one participant from an organization so they can provide each other support after training. As with other aspects of the project, in the absence of clarity on what is meant by organizational capacity, 19 consideration of impact or even influence depends to a certain extent on definitions, expectations and associated perceptions. For the purposes of this evaluation, the evaluation team took a very loose and inclusive definition of organization and organizational capacity. 'Organization’ was any form of structure no matter how loose, or group of people, working together consistently with a similar intent, aim or goal, and that identify themselves in this way as a group. In looking at organizational capacity the approach was to allow people to discuss and explain the work that participants were engaged in beyond individual enrichment and who has assisted others within the 'sector' or more broadly in society. Organizational capacity could take the form of some or any of the following (along a spectrum): a) Formal systems of a structured organization - e.g. finance, administrative systems, policy, HR, etc b) Application of Technical skills improving the quality of work of the organization c) Application of other skills that enable staff to 'grow', be empowered, take an initiative, that they would otherwise not have done e.g. leadership skills, facilitation skills, different ways of learning, shifting of responsibility d) Application of Learning or knowledge that deepens group understanding of social processes, rights, governance, roles and responsibilities of people on both sides of the 'governance equation' e) Attitudinal changes - e.g. questioning of 'authority' (not necessarily government. This could be traditional accepted power relations e.g. with leaders, 'seniors', peers, or gender dynamics), critical thinking, acceptance of and deference to information that has been provided to more active engagement. If one takes this approach, then there is evidence from the data that organizational capacity has been strengthened through the training of community leaders with some additional benefits accruing in the organization either through sharing of skills or transfer of other behaviors such as more participatory decision-making. The perception survey incorporating 202 face-to-face interviews suggests that approximately 25% of those alumni interviewed consider that they have had an impact on the organization they associate themselves with and were able to provide clear examples of this change. 45% of those interviewed definitely did not think that they had had an impact. Some of the reasons given include that they were too junior or leaders were not open or that simply they were not in a relevant job or it was too soon to say. (There were no gender considerations evident at this stage here.) There were a further 29% of the sample who also said that they had had an impact at the organizational level, but the evidence given was not convincing. Interpretation of these figures is subjective, given the difficulties of triangulation and the acknowledged Hawthorn Effect. At the most optimistic level one could consider that the project in fact, had a positive impact at the organizational level with 54% of the respondents. The few organizations that the lead evaluator was able to visit and try and obtain a perspective from the organization’s director concerning the perceived impact from the AC trainings noted the following caveats regarding whether change could be attributed to AC training. Firstly six (of the seven) organizations visited noted that the organization and the individuals, and their staff more broadly, received trainings from a variety of sources including the British Council, the Capacity Building Initiative (CBI), Paung Ku, and internal trainings as well. Four of them also noted that they have had high turnover of staff, including 19 For instance at an extreme it could be argued that any organization that has a staff member who has participated in additional training has had their capacity built in which case one could say that 248 formal and informal organizations have had their capacity built. 17 those who attended the AC training. These pragmatic points suggest that the depth of the long term impact may be less significant than the numbers would indicate. It is also difficult to say anything about the extent of the change and its depth and sustainability from the data. As one would expect there was also considerable blurring of where participants consider that they see some change between the individual, organizational and even community levels. Although there is a noticeable ‘tailing off’ of impact as one goes to the higher levels. Some examples of how participants consider that their organizations have changed are noted in the below text box. There are other considerations that should be kept in mind but are more speculative in nature. The extent of the impact on organizations may be related to some of the following variables (among many possibilities): (a) size of organization; (b) the type of colleague that the alumni is working with and their level of motivation; (c) the type of organization (e.g. school or educational institution versus community development organization); and (d) type of course attended (and by implication length of course). These elements are not immediately clear from the data collected, as variables (a) and (b) were not collected in the survey. The data was also analyzed comparing the different courses and relative impact at the different levels as well as perceived degree of personal change in the participant resulting from attendance on the course. However, no discernible meaningful patterns with clear conclusions were apparent from these exercises. Gender Analysis A basic gender analysis was also conducted on the data collected from the 202 face to face interviews. The Table 5 below summarizes some of the figures. Table 5: Disaggregation by gender of reporting perceived change at the individual, organizational and community levels Gender Total Participants in the sample Reporting positive Change at the individual level – applying learning Overall Percentage reporting change at the individual level Reporting positive Change at the organizational level Overall Percentage reporting change at the organizational level Reporting Change at the Community level Overall Percentage reporting change at the community level Female 141 84% 75% 23% 25% 17% 19% Male 61 71% 31% 25% Unfortunately little significance can be gleaned from the figures. It appears that the number of males who reported positive change and projected impact at the various levels was higher than for females. It is difficult to make any conclusive judgments about this apparent difference. On one hand it may be that for various reasons males were indeed able to instigate or effect more change at the organizational and community levels in this society where gender roles are currently modeled along traditional lines. On the Examples of Perceived Organizational Change (extracted from alumni face to face interview transcripts) • Improved Organizational Communication with staff being able to share opinions openly • Decision-making is more participatory • Motivated staff • Improved rational thinking • Improved relations with the community • Improved strategic planning • Staff now take responsibility for educating pupils and are not just there for their salaries • Staff have improved their teaching • Improved training models are used now by the organization • There is more collaboration and mutual support among staff • The organization is more confident dealing with the government 18 other, it may mean that males were more inclined to exaggerate their impact and ability to apply their knowledge. Unfortunately there is insufficient information to analyze the data in a more meaningful way. Question 2: What are the key barriers to community leaders using acquired learning and networking competencies in their work? Information from the perception survey and the focus group discussions regarding key barriers is remarkably sparse. Very few comments were recorded in the interviews and most of the information had to be drawn from responses to other questions. This may not be so clear partly due to the Hawthorn effect noted above. This would partly explain why there are many answers which do not really reflect true positive impact but which effectively suggest that there have been no barriers to change. It may also be a difficult question to answer culturally as it could be perceived as only identifying negative or critical aspects of their workplace, themselves, their colleagues or their seniors. The question may also be considered as an implicit criticism and recognition of their own failure. Alternatively it may also be a reflection of the difficulty for national interviewers to press or dig more deeply in interviews. The most useful information came from key informant interviews that suggest there are a range of barriers mentioned including the following: 1. Cultural barriers: such as gender, hierarchy, ethnicity and differences in age. 2. Relevance of the learning to everyday life: Content of courses, and targeting of participants 3. Weaknesses in Project Design: Shortness of some courses, lack of support, mentoring, orientation of the organization 4. Physical barriers: such as time, distance, communications, and cost (this last category is relevant only in the case of networking. Where information can be gleaned from the perception survey suggest that the main reason given by those who cannot apply their learning, and hence can’t influence the organization, is due to a mismatch with their work, or being unemployed (26 responses). A few respondents note that being a junior member of staff is a significant barrier and this is illustrated with comments such as ‘I can’t change the leader’s perception yet’ or ‘I need more experience first as there are more senior people…’, or ‘I have suggested this to the principal but she has not applied it yet as I am too junior’. In the light of the barriers noted above that were perceived to have played a role in constraining impact it is worth having a look at how WL has tackled some of the typical elements that are known to influence effectiveness in leadership and/or training programs (adjusted from Leskiew & Singh 2007 and others). Table 6: Influencing Effective Leadership Elements of a successful training or leadership program World Learning Project Other Possible approaches that could have been considered Relevant Course Content or material Ongoing improvements and adjustments to (a) program material developed over the course of the project but particularly post 2011 (b) the modification of the structure of the course (i.e. 2 days per week) to accommodate working leaders who can’t spare the time). • Formal Needs assessment undertaken • Tailoring of course material to the needs of some large organizations • Introduction of the range of courses earlier in the project (given that others were doing this already) Sound targeting of the right participants Comprehensive targeting strategy only developed in late 2011 • Approach existing leaders of CSOs to better understand their needs – e.g. possible leadership forums • Partnering organizations who are working with other types of leaders at the community level (e.g. Village Leaders, youth leaders, etc.) • Collaborating with other service providers e.g. CBI, Puang Ku to create synergistic and complementary inputs • Surveys of different States or Divisions to encompass cadres who can work together • Adopting a Sector approach e.g. leadership 19 Question 3: In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Respondents’ reflections on how they have been applying their learning can be categorized into the following types of activities: • Sharing knowledge and skills with colleagues and peers in the formal workplace • ‘External’ trainings with or for other organizations as resource people or volunteers in their own or other organizations • Some alumni have set up organizations or small groups of volunteers who do voluntary work • Application more broadly in their personal lives In addition to technical skills that were applied, there have also been changes in participants’ self confidence that enabled them to take on more of these roles and activities. While this was only expressed overtly a few times, nevertheless in the opinion of the evaluator it was an underlying element implicit in the examples given by alumni and was also a successful and direct consequence of the experiential reflection process that was taught to participants. Some examples culled from survey forms are in Table 7 below to provide a flavor of the above categories of change. Table 7: Examples of Change at the Individual and Community attributed to the project (extracted from alumni face to face interview transcripts) Change At the Individual level Change at the Community level - Children are more active in my class now - My classes are more participatory - I now think more rationally and apply this to everything - I train other staff on program design - I have shared critical thinking with my subordinates and volunteers - I have conducted TOTs for CBOs and youth impacting on ca. 80- 100 people - Through the trainings we now do on WASH we have trained ca. 170 school volunteers and 160 hygiene promoters - Now there is no more confrontation between ca. 100 CBO members in 4 townships with the local 20 It is acknowledged that this may not have been possible along the way – perhaps only in the latter stages of the project due to the sensitivity of some of the topics. in the health sector, etc. Post-Training Support to participants Some support later provided by: • resources if alumni are giving training • some ad hoc mentoring (except in the case of the QuiLT Alumni initiated ‘Teacher Development Program’ or TDP) • Setting up specific Mentoring programs, these could have been peer to peer or structured differently from AC (see efficiency section) • Provision of ‘kits’ to alumni to set up leadership forums in geographic areas • Start-up kits to undertake extra trainings provided immediately on graduation • Structured local ‘Area meetings’ for alumni on a regular basis • Newsletter or provision of additional material/topics for consideration to teachers. This could have included things like ‘tips’ to overcome specific barriers, gleaned from feedback from alumni. Addressing the enabling environment Not really tackled • Where possible20an orientation project for organizations or leaders of organizations that are having someone trained at the AC on a long course. • This is where training alongside an organization that is undergoing an OCA could have yielded stronger results. 20 - I only use English in my classes now - Tolerance - I teach the other teachers on techniques for skills and grammar - Classes are more practical - I founded an organisation. - I have helped teachers at an orphans school and 250 other people In workshops and trainings authorities because we have altered our approach - 18/25 students have changed in this private school and no longer quarrel with each other according to parents’ feedback. There used to be a mindset of bullying in the students and this has changed to one of tolerance. - I have trained 300 students and 30 members of my church group in some skills - 120 people in political parties have been given the TOT - I have given Leadership training to communities on power and advocacy. - We solved some problems between factory workers and the management by involving the township administration, the ministry of education and some parliamentarians. After this the parliamentarians said to come to them if there were any more issues. This would never have happened before. Question 4: What approaches have been most effective and efficient to achieve goals in the light of rapidly changing context? It was difficult to assess this question given the way it is phrased. The rapidly changing context, as noted elsewhere in this report, in many ways was not the most important constraint to achieving goals in the latter years of the project. In the former years there were significant contextual constraints to achieving goals and the project was slow to adapt, in the opinion of the evaluation team leader. Nevertheless, the most significant elements of the project that enabled it to achieve some success were the following: 1. The addition of the TOT course. This signified an important project shift from the more formal education perspective to that of broader civil society strengthening and through the non-formal education sector. This allowed the project to provide relevant skills to a much greater range of civil society leaders outside of the limited spheres of English language teaching and the small number of political dissidents undertaking distance education courses. It also supported the opening of space resulting from public participation and response to Cyclone Nargis. 2. Changing the curriculum and scope of the Distance Education component. Changing the service provider from IU to CSU with a different curriculum and length of course improved its relevance, usefulness and was far more appropriate to the needs of the participants and the Burmese context. 3. Provision of Courses outside of Rangoon. The needs of civil society were even more significant outside of Rangoon than inside. There was a dearth of opportunities to learn and develop and BODEP rightly started to see how best to address this need. Further, the provision of courses outside of Rangoon was a sensible approach to address the goal of creating a diverse set of civil society leaders. Populations of ethnic minorities are concentrated outside of the Burmese heartland and few can to travel to Rangoon to access opportunities because of cost, time and sometimes security. 4. Training of Local Trainers. Given the difficulties of travel for expatriates outside of Rangoon the project trained local trainers to conduct the courses on offer. This is probably the most noteworthy step that the project took, both in terms of trying to create some degree of sustainability but also as a step to overcome the contextual constraints. It also represents one of the major lost opportunities for the project which could have shifted another gear if this step had been taken earlier and with more intent. 5. Annual Conferences and Civil Society Roundtables. These two initiatives under the networking component were important in shifting the project into the public domain by providing opportunities for broader discussions and perspectives to be aired. Again this reflected both the stronger and weaker aspects of the project. On one hand these were innovative initiatives and on the other they could have been developed had there been stronger strategic leadership displayed earlier in the project. 6. Project Professionalization from 2011. As noted previously, tightening management with respect to targeting and recruitment of participants, adaptive learning instruments such as the Performance Management Plan (PMP), monitoring, Annual Evaluations, and more thoughtful consideration of relevant course material to meet civil society needs all contributed to improving this project. 21 Question 5: Were there any unintended Consequences? There is little evidence of any unintended consequences from the project uncovered through key informant interviews, focus group discussions or the face-to-face interviews. Positive potential outcomes that might fall into this category include the development of new organizations by alumni as well as materials replicated and used. But both of these could have been considered implicit expectations in the latter stages of the project. The only interesting aspect that could arguably fall into this category and that was mentioned by two key informants was the phenomenon of ‘AC/BC Shock.’ This was described as being when participants who took courses run by the British Council or the AC reportedly became distanced from their original constituencies and ‘worse’ leaders. When explored, this apparently means that they lose touch with the grassroots and become more self-centered. 21 In some ways this perception is also not unexpected as someone who undergoes a transformative experience and then behaves in ways that challenge cultural norms (such as questioning an elder or challenging an opinion) is likely to be perceived as having lost touch with the community. 4.2 Conclusions In conclusion BODEP made a contribution to civil society strengthening in Burma. It has had good impact at the individual level with understandably diminishing levels of evidence for impact at the higher organization and community levels. It has however, missed many opportunities for increasing the levels of effectiveness and impact along the way due to lack of strategic leadership. Many of the improvements that were made later in the project could have been commenced sooner and been combined with complementary efforts, as evidenced by other civil society programs already taking place. BODEP has contributed to building a diverse cadre of civil society leaders in Burma during this rapidly evolving context. The project evolved over a period of several years but really started to demonstrate improved management and effectiveness starting in 2011 as a civil society strengthening project, with the change in program description and personnel. Previously dominated by a more formal education perspective it broadened its scope in 2009 with the addition of a training-of-trainers course aimed at addressing broader civil society needs as well as supporting groups responding to cyclone Nargis. Further significant changes, particularly in terms of ‘professionalizing’ project management, assisted in creating a strong project platform from 2011. This included the introduction of a rigorous PMP and systems to collect data, a targeted recruitment strategy and performance rubrics to assess quality and standards of participant achievement. The project also introduced new activities, courses and approaches to strengthening civil society over the last two years that have showed much potential. In terms of impact, it is clear that there has been strong positive change at the individual level for course participants. An impressive 75% of participants surveyed reported they have used their skills following course completion. The most significant reasons cited by those who have not used them included the following: they are not working, or the course was not relevant to the work that they have. A proxy for the level of leadership being evinced by participants is through the number of networking events, which primarily shows that alumni are actively using these skills to pass on information and skills to others outside of their own organizations. It is harder and possibly slightly unfair to try to assess the impact of the project at the organizational and higher levels due to the short time frame that the project attempted to influence this level rather than the broader goal of a diverse set of civil society leaders. The evidence is also somewhat mixed and not so convincing. On one hand, self-reporting suggests that many participants considered they had an impact and changed organizations and indeed communities to a certain extent. On the other, the evidence that they cited is not strongly supportive of these statements. The few participating organizations consulted also suggested that impact at the organizational level in their agencies was minimal in the context of other ongoing development and training. 21 One person in particular was cited as an example of this phenomenon and the Team Leader had interviewed this person as well. While purely a personal observation, it appeared that this person rather than failing at the community level had instead altered their focus of leadership to a higher level of contribution at the national policy level rather than the very grassroots community level. 22 At the community level, there have been examples of impact provided by participants through their own efforts. This was at a modest scale but is an indication of how motivated many of participants were in their own communities. There is little evidence of large scale change at this level, which is expected given the time frame and the way that the project has been shaped; but modest changes indicate potential. While the last two years of the project provided the most significant contributions, throughout its life it addressed some of the underlying societal issues affecting Burma. In conjunction with the practical training in technical skills and approaches, it also started to tackle long term development issues and legacies of oppression. This was through its approaches to leadership in its course material (critical thinking, inclusion, and participation), networking events and the attempted creation of cohorts of alumni. Introducing annual conferences and civil society roundtables and these were important initial efforts to build trust (and breakdown distrust), normalize gatherings and discussion of issues in the public domain. It did, however, also miss significant opportunities to increase its effectiveness along the way. 4.3 Recommendations 4.3.1 Recommendations Part 1: Operating in Oppressive Environments As a result of the contextual analysis the following recommendations can be made to USAID concerning fundamentals on implementing programs in oppressive environments more broadly. It goes without saying that many of these recommendations are basic and are lessons learned from other environments. 1. Deliberate opportunities for strategic reflection should be integrated into projects. It is easy to absorb and unintentionally reflect a narrow view of what is happening or what might be possible in oppressive environments. External stakeholders consider this to be the case with BODEP being based at the AC and influenced by perceptions from stakeholders close to the project. It is therefore imperative to triangulate information regularly with a range of stakeholders to ensure that contextual analysis informing project decisions is more firmly rooted in a consensual reality. This may allow more informed risk taking to enable more opportunities to be exploited. Similarly, deliberate opportunities need to be factored in for strategic reflection so that lessons learned from the project, and adaptive management in the light of contextual changes can better be considered and implemented. 2. Project goals, intentions, and expectations should be clearly documented. Ensure that particular attention is made to articulating clear project goals, intentions and expectations in project documentation even if they reflect a diverse set of disparate objectives that are not cohesive. Sound reference documents are even more important in oppressive or constrained environments even if they are perceived as being sensitive or high risk to stakeholders. Strong documentation reduces the risk of misunderstandings and shifts the emphasis onto the safe management of this information rather than the avoidance of documenting project issues and lessons along the way. 3. Build flexible mechanisms into projects, continually assessing project relevance and adapting to context. Flexible mechanisms need to be integrated into programs, not only to take advantage of opportunities and opening space if it is there, but also to adapt as time goes on. The project did certainly adapt as it evolved; however, it is also important to reflect the actual circumstances and potential impacts more appropriately in the evaluation model. 4.3.2 Recommendations Part 2: Emerging from the Project Experience USAID will not be providing continued funding for this particular project, as the contract ended on November 30, 2012. However, there are some recommendations emerging from the project experience, lessons and mistakes. 1. Ensure foundational civil society mappings and assessments are undertaken to inform programs. The project did not have a strong idea of the range of different organizations and the state of civil society across Burma to inform its recruitment strategy or positioning. There have been a number of misconceptions regarding the absence of civil society in Burma that were later seen to be mistaken. A mapping exercise, undertaken at either the sector level or State by State, would provide the foundation for a more informed program design for any future civil society strengthening program. Certainly there will be a proliferation of new organizations in various forms both formal and informal that will probably mushroom in the next couple of years, but it would be helpful to have the start of a database for more intentional programming. 2. Concentrate program efforts outside of Rangoon at the State level. The project provided a 23 degree of training for those outside of Rangoon, but in the future this needs to be increased in a form of affirmative action for those who were not able to benefit from these opportunities so far. This is important because in the interests of stability the different ethnicities and States need to be developed quickly following neglect during the military regimes’ years in power. Many of the most flammable development issues are likely to occur in the more peripheral States. These are areas where governance issues are sensitive and close to the surface of people’s consciousness. They have already resulted in significant violence and conflict in these areas and this is where the triggers for instability will be realized. This may take the form of natural resources extraction, land grabbing, investment, and failures to consider local perspectives or ensure that local people obtain a fair share of the benefits of national development as well as be able to participate in their own governance. 3. Increase opportunities for people to come together across divides. The project started some good networking initiatives working on the fundamentals of a strong society - tolerance, diversity and building trust between ethnicities and religions. These initiatives should be picked up and developed significantly but with the intention of addressing and discussing issues jointly and developing collaborative leadership. A series of different forums need to be developed at Division, State and National levels that deliberately bring together leaders from different sectors, government, civil society, religious and business leaders in a safe environment. For instance an example might be a facilitated forum specifically for senior state level administrators to look for common solutions without fear of being judged from above or from civil society. This could then be developed by having additional opportunities for ‘cross-sector’ seminars between civil society and business leaders to learn together e.g. about EITI or best practice social development for extractives. This needs to happen across all aspects of Burma’s society. 4. Target organizations rather than individuals for capacity building. It is more efficient and effective to target organizations rather than individuals to build capacity and leadership. This is because there is a resulting formal structure within which the participants can practice their skills and have more chance of developing impacts at higher levels. It also provides opportunities to formalize structures that can provide ongoing support, whether technical or emotional/moral in nature, after the end of any training courses. Mentoring should be a prominent feature of any future civil society strengthening program, to break down the legacies of oppression and create a sense of common purpose in working for the future collaborating together. 5. Invest in developing a cadre of top class local trainers. Whatever form of civil society strengthening program is undertaken, there will be an inevitable need for capacity building and skills in the CS sector. It would be sensible for USAID to invest in capacity building in whatever types of projects it supports. From the start of the project it should be mandated that a cadre of strong local trainers with sufficient technical skills is developed. While this is likely to take a few years realistically, it should be a priority for the long term development of Burma. 4.3.3 Recommendations Part 3: Strengthening Civil Society - Strategic Choices for the Future 4.3.3.1 Foundational Programming Perspectives: As a result of in-depth discussions with key informants from Burmese civil society, donors, practitioners and program managers of other civil society strengthening programs, this section outlines a number of ways forward for USAID to consider in its future programming. The perspectives are necessarily higher level considerations and based on a holistic approach to the development of society rather than only ‘civil society’. This first section is foundational in the sense that these recommendations underpin all of the subsequent recommendations whether they are long, medium or short term approaches. 1. Ensure that ‘Do No Harm’ and Conflict Sensitive programming principles are applied at all levels of USAID programming. There are no shortage of opportunities for USAID to make a positive contribution to the development of civil society but almost universal feedback from key informants was the anxiety that USAID may potentially do damage unintentionally in its programs. The fear expressed was that this will occur either through an overestimation of the absorptive capacity of the sector or through the potential creation of distortions in the environment. This can easily result from overinvestment in some areas coupled with an institutional need to see rapid progress and impact. Hence, whatever elements are incorporated into the USAID portfolio, ensure that ‘Do no Harm’ principles coupled with conflict sensitivity are incorporated into all aspects of the USAID programming. An additional aspect is to consider these issues at the macro levels of national 24 development rather than simply at the programmatic or sector levels where these principles are normally applied. 2. Key features of Burma and considerations to shape a strategic approach. In summary, and given the plethora of needs for support in the development of every facet of the governance landscape in Burma, the USAID strategy for strengthening civil society needs to bear in mind the following, and perhaps obvious facts, that Burma: • Has been isolated from the external world for over 20 years; • Is relatively rich in natural resources; • Is located in the middle of a geographic region characterized by rapid economic growth; and • Has been deeply shaped by the ongoing legacies of oppression. The consequence of these facts is that Burma is not only vulnerable to predatory economic behaviors from its neighbors but also simultaneously vulnerable to the rapidly rising internal expectations from the populace for democracy and change. This latter issue risks creating deep internal tensions that could easily result in further expressions of conflict and instability. This is a particular concern given the history of ongoing conflict in the marginalized peripheries of Burma where ethnic minorities have been struggling for their identity and rights. In light of these elements, it makes sense for USAID to consider a multi-streamed, strategic approach aimed at a facilitating a smooth transition for Burma society simultaneously within a number of timeframes. 3. Foundations for a sustainable balanced society. All members of Burmese society have been damaged by their experience of oppression and the military regime. Those in government at the State and local levels have also been affected and need to be part of the solution going forward. This means that in looking at civil society one should also be mindful of how to balance the optimum elements with government to support healthy societal change. When considering the cumulative effect of the military government’s oppression of the people one can express two broad medicinal approaches that are desperately needed across the country to counter the legacies and long term enculturation of distrust, and fear found in the current relationships and interactions between people: a) The ‘normalization’ of discussion. In other words, the ability of people to be able to discuss, debate, talk and express opinions about any topic without fear of reprisals or negative consequences. b) The ‘normalization’ of organization and gathering together. Similarly people need to feel able to gather together, travel, and meet others without suspicion and fear. These two elements are fundamental to democratic and healthy governance environments. At present, society in Burma is more broadly characterized by division, lack of trust, and lack of understanding across religious and ethnic divides. This was the initial foundation of the WL project and the positive elements within the project now need to be developed and expanded further to utilize the new space opening up. 4.3.3.2 Recommendations for the longer term: From the perspective of stability and conflict management, Burma will eventually need to amend its constitution to reflect the needs of the whole nation. A quick glance across the country reveals active and violent conflicts underway in most of the States around the periphery. Some of these conflicts have been underway for decades. At a simplistic level this reflects the historical social, political and economic marginalization of the ethnic minorities in the nation who have not yet enjoyed the ability to participate fairly in all aspects of the development of the country. If this fundamental set of issues is not addressed with appropriately paced progress made by the center and all stakeholders, then the future will be very difficult with the potential for significantly increased violence and instability spreading. As noted, the most sensible way to enshrine these rights is within a new constitution and probably some form of ethnic federalism. If not then at least a very decentralized form of government that empowers States and local authorities to administer budgets in line with local needs determined by the populace. For this process to have a more favorable chance of success, the seeds for that success need to be sown now with the provision of neutral information around possible governance models that allows all stakeholders to become accustomed to these ideas and to understand all the ramifications. Managing the speed of transition is the difficult aspect. 25 It is in the light of the above two perspectives – addressing the legacies of oppression and the development of a viable long term governance model that it is recommended that USAID invest heavily in two streams of programming: 1. Invest in Peace-building and Stabilization programming: There are a number of long-term political and non-government actors involved in peace-building in Burma, and this is always an exceedingly sensitive and nuanced field to be working in. Failure to appreciate subtleties in positions taken by stakeholders, including spoilers, can easily do harm. Equally it is healthy to delicately try and open some of this space to greater public scrutiny and challenge. Often entrenched political movements on both sides of the trenches think that they are representing the ‘people’ but are also guilty of abusing them and, in fact, often tend to represent themselves and their interests rather than that of their constituencies. Building deeper and more common understandings of positions and situations across the different States is in the best interests of all, so that those in Rakhine can appreciate and compare their situation with that of the Karen or the issues in Kachin State. This prevents rumor and breaks down prejudice. 2. Invest in the provision of information and development of local media: An investment in the development of neutral local media will provide a sound foundation for the development of a strong society and a strong civil society. The ideal would be an initial investment in FM radio stations in every State22 coupled with some sort of ‘newspaper’ or news sheet. If handled correctly both of these approaches allow for all stakeholders (i.e. government and civil society simultaneously) to develop at the same pace without being disadvantaged or developing imbalances in power which might cause a reaction. For instance a radio can start off by having mainly entertainment and music with occasional opportunities for stakeholders to broadcast what they are doing (e.g. have health authorities and NGOs in that sector both explain their programs). Once people become confident that these are not oppositional platforms for ‘government bashing’ then one can move to the next level of having increasingly meaningful, facilitated discussions on how to tackle health issues and the respective roles of stakeholders, etc. as well as broadcasting health education messages. Successful methodologies include the use of dramas and soap operas to ‘depersonalize’ issues into fictitious characters. Similarly a simple news sheet could be distributed just with information on positive steps the government has taken so that all can be literally ‘on the same page’.23 Simple information will then be read and commented on without the need for commentary or critique at this stage. Information is the major need at present. 4.3.3.3 Recommendations for the medium term: There are three critical issues looming on the horizon that can become the triggers for rapid violence if nothing is done now to avert this. These are (a) the ‘pilot’ census in 2013 (b) the full census in 2014 and (c) the elections in 2015. Even if violence is avoided at each stage, poor processes and issues can become latent proximate causes for conflict that await a later trigger. The reason that these processes are important to tackle now through the delivery of balanced information and civic education is because they will inevitably be politicized. Issues surrounding the elections are inevitable but poor processes can result in terrible violence even in apparently stable environments24. Given the lack of information about these processes in the past and the poor communication network in Burma, it would be prudent to start addressing these issues now before time slips away. This would allow a far more deeply informed process to take place and a more informed process is likely to be one less characterized by rumor or manipulation. 4.3.3.4 Recommendations for immediate issues: There is no shortage of issues to address and support. However given the rapid investment of foreign money and the relative richness of Burma, 22 The evaluator knows of one NGO that has already had discussions with State authorities for such a project and has reached the point where they have agreement and just need the funding. This would be an ideal pilot to test the bureaucratic tolerance for this type of project. 23 For instance, the evaluator was told that there have been changes to 30 laws recently and even well-informed civil society leaders did not know what all these changes were and cannot keep up. Imagine what a local authority feels like in an upcountry Township? Not knowing what to do and how to act and especially not knowing what is going on in government even though he is part of it. 24 For instance look at the last elections held in Kenya 26 resources are also a potential touchstone for conflict. This means that natural resource management, the environment and land reform are critical areas of governance. There have been some major civil society successes that have been well publicized in this area. 25 These are heartening but also risk being the ‘lightning rods’ and conduits for the expression of resentments accrued over the many years of oppression. These sorts of issues (perhaps lower profile ones) also offer opportunities now to learn and practice positive engagement processes between government, business and civil society sectors. Once more considering this sector through a stability and conflict lens means that it is important to address the sector in a balanced way that does not empower one side of the governance coin to the perceived disadvantage of the other. To this end it would be interesting and wise to take an issue based development approach at the State level that would bring stakeholders together over local issues that would enable learning and ‘managed’ communication to take place around a process that is aimed at finding solutions, through negotiation and mediation while continuing the process of breaking down mistrust and fear and enhancing collaboration. 1. Invest in multi-stakeholder ethical Natural Resource Management: (a) Invest in an issue based program that brings multiple stakeholders around a common issue at the State level. (b) Support education and policy reform in the natural resource management sector including supporting public￾private partnerships around social development. (c) Promote corporate social responsibility through tools such as the global compact, Equator principles and others. (d) Research and document land issues to inform the development of a land reform process. 2. Continue to invest in Leadership: The WL project rightly identified leadership as a key issue to be addressed in a society where initiative and leadership have been stifled for years. It is recommended therefore that a range of special leadership forums be developed that are appropriate for the Burma context. Such an approach has been developed in the Democratic Republic of Congo by the Wilson Center and this would bear looking at and adjusting for Burma. Opportunities that enable leaders to learn, grow, and work together on common solutions has the added advantage of continuing to address the legacies of oppression, i.e. normalizing discussion, normalizing coming together to tackle common problems. 3. Invest in Organizational Development: There are two options with respect to supporting the inevitable and mostly positive mushrooming of civil society organizations that are being established as people start to take initiatives to address their own community development issues. Option one that civil society leaders recommend is to support and allow these organizations to develop organically at the State level, where they may end up failing or fading away, which is part of the natural process. This option might take the form of a small grants program at the State level, or supporting the expansion of one of the existing mechanisms that is already operating. An alternative option is to recognize the enormous gap in human resource capacity that there will be in the near future. The formal education system while fundamental, takes too long to develop people for this period. To address this, one could support or initiate the service industry for civil society by providing trainings and trainers, translators, information publishing and printing in the many local languages, organizational development, and other support services. Again this approach demands that these services are located out of Rangoon in the more remote States. 25 E.g. the suspension of the Chinese funded investment in the Myitsone Dam 27 ANNEX I: EVALUATION STATEMENT OF WORK BURMA OUTREACH AND DISTANCE EDUCATION PROJECT: CAPACITY BUILDING FOR CHANGE I. BACKGROUND INFORMATION A) Identifying Information 1. Program: Strengthening Civil Society 2. Project Title: Burma Outreach and Distance Education Project: Capacity Building for Change 3. Award Number: Cooperative Agreement 486-A-00-07-0007-00 4. Award Dates: September 21, 2007 – November 30, 2012 5. Funding: $2,454,632 6. Implementing Organization: World Learning 7. Agreement Officer’s Representative (AOR): Thomas White B) World Learning Background World Learning (WL) is a non-profit organization with operations in 77 countries, providing support in education, student exchange, and development programs. World Learning has worked in Burma since 2003 to build civil society capacity through distance learning, training of trainers and teacher training. The cooperative agreement with the United States Agency for International Development (USAID) and World Learning began in 2007, and is scheduled to end on November 30, 2012. The Burma Outreach and Distance Education Project has been managed through USAID’s Regional Development Mission for Asia (RDMA) based in Bangkok, Thailand. C) Development Context 1. Problem or Opportunity Addressed There exists an increasing need and demand for trained Burmese nationals to be development and community leaders in the rapidly changing environment of Burma. The influx of foreign aid into Burma after the 2007 Cyclone Nargis was a key in-road for initial programming in Burma, and also a key trigger for this demand for trained Burmese nationals who are able to gain access to areas otherwise inaccessible to foreign workers. With the recent developments in Burma, this need to go beyond relief efforts to sustainable community development, ultimately with leadership from Burmese civil society, is more essential than ever. Burma’s civil society, however, is still in its beginning stages of redevelopment and currently lacks the leadership necessary to effectively move towards an advocacy-centered civil society. The Burma Outreach and Distance Education Project was initiated to address the need for civil society leaders who understand and practice modern theories and practices of change management, networking and coalition building, mobilization and leadership. 2. Target Areas and Groups The target population has been modified through various iterations of the project, in response to a myriad of contextual changes in Burma, as well as staffing and programmatic adjustments. The project was originally intended to provide opportunities for influential Burmese citizens with restricted access to higher education opportunities due to the political environment. Original recruitment efforts cast a broad net for individuals working in related professional fields (namely, English language teaching) and demonstrating specific technical skills. In some cases, particularly during the program’s nascent stages, there were pre￾determined populations from which to recruit participants without a formalized recruitment plan. Since the project’s inception, World Learning has refined the definition of their target group in order to focus their training efforts more efficiently. From World Learning’s 2012 Performance Management Plan 28 (PMP), “The WL program seeks individuals who are intrinsically motivated to produce positive change in their communities, open to new paradigms of thought regarding the process of change and are able to influence their peers. Participants should also be actively engaged in community development or civic affairs. In short, WL is looking for people who are active, motivated, open and influential and have a platform from which to advocate for change. In addition to the main leadership qualities targeted participants need to possess and demonstrate, participants will be recruited from specific professional profiles. Individuals with these professional profiles are in ideal positions to most benefit from the trainings and then apply new skills and knowledge in their communities or professional organizations upon completion of the trainings.” Professional profiles are specified below for each of the four program components: English Teacher Training Component: • Community education activists: teachers and trainers of English/ teacher training from private, non-formal and/or religious based schools and community centers. These individuals are current leaders in a classroom, institutional entity and/or community. • Experienced teachers who also serve as community leaders (for example, as organizers of community libraries). These individuals may serve as local community leaders in positions to mobilize others for information sharing and collective action. Training of Trainers (TOT) Component: • Mid-level community development officers active in CBOs/NGOs (local CBOs/NGOs are prioritized and may not necessarily be formally incorporated as an NGO, though members of INGOs may also be included); individuals may be paid employees or active volunteers; individuals will be conducting trainings on leadership, capacity building, or technical content. These individuals are currently in positions of change as leaders within their communities and within smaller organizations. • Mid to higher level trainers at CBOs/NGOs: they have the potential to be future civil society leaders; they are currently in or rising into positions of change as organizational managers. • Founder/ managing director of CBOs/NGOs: they are in current civil society leadership positions and have a vision for organizational or social change and development. Distance Education (DE) Component: • Founders, managing directors and/or mid-level active members and volunteers from CBOs and NGOs (local NGOs are prioritized and may not necessarily be formally incorporated as an NGO, though members of INGOs may also be included). These individuals are in current positions of civil society leadership and have visions for organizational or social change and development. • Community activists: leaders of informal CBOs active in community development initiatives. They may also have the flexibility and scope to apply newly acquired tools and skills within smaller CBOs/ NGOs and community initiatives. Leadership Component: • Founders, managing directors and/or mid-level managers of local NGOs/CBOs. These individuals are in current positions of civil society leadership and have visions for organizational or social change and development. Underlying criteria for targeted participants in each of the program components necessitates that all successful applicants are connected in some manner with a wider network of individuals and/or organizations. That is, a successful applicant should be formally affiliated with a particular CBO/NGO(s) or institution(s) or informally part of a broader and looser network. World Learning aims to reach participants already establishing themselves as mid-level civil society leaders who are working in coordination with others for greater impact in a given technical field or geographical area. World Learning’s recruitment also aims to form course cohorts who are from a wide array of backgrounds, in order to create a cadre of community leaders composed of women, minority ethnic and religious groups 29 and geographic diversity. The Performance Management Plan (PMP) 2011 targets indicate targets of 50 percent women as course participants, and approximately 25 percent of all participants would be members of an ethnic, religious, and/or geographic minority.26 (Please see Attachment “Participant Geographic origins” for number of origin of course alumni from 2007-2012.) It is important that this evaluation capture the diversity of the program participants and not simply focus on the urban centers. The following 2012 map shows how the targeted participants have been able to disseminate their learnings through networking events held throughout the country without USG assistance. D) Approach and Implementation The goal of the program - to develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society - has been addressed through five major components, including 1) English Teacher Training, 2) Training of Trainers, 3) Distance Education, 4) Leadership, and 5) Networking. The program activities are nearly wholly capacity building inputs to a range of individual Burmese community leaders, with some support to Burmese civil society organizations. This program has undergone numerous changes since its inception in 2007. These changes to programmatic activities, methodologies and strategies were driven by a number of factors, including: openings within the operating environment; collection and analysis of program monitoring and evaluation data for lessons learned; and strategic reflection and planning sessions. The following are illustrative examples of how the development hypothesis, goals, and activities have been modified throughout the years: 26 An ethnic minority is defined as a person who does not belong to the ethnic majority, Burman; By city/ town of residence is defined as a person located outside of Rangoon at the time of participation in a course and/or who travels to Rangoon for the express purpose of attending a program course. A religious minority is defined as a person who does not belong to the national predominate religion, Buddhism. 2007 The cooperative agreement, enabling World Learning to expand activities related to an earlier agreement, adding a Distance Education component run in association with Indiana University (IU) and extending previous English Teacher Training activities to include the WL/SIT Qualification in Language Teaching (QuiLT) certificate program. Teacher trainings were designed and delivered to model participatory classroom interaction, to introduce teachers to a range of international teaching materials representing a broad spectrum of views on both educational and wider social issues, and to develop participants’ skills in reflective practice and critical thinking about their teaching and its influence on their students’ learning. WL later designed and implemented a series of Training of Trainers (TOT) courses to provide a broader audience of community leaders and activists with the principles of training design and planning, inclusion, participatory methodologies, collaborative practices, and experiential learning. 30 E) Intended Results The project goal (recently updated in 2011) is based on the following development hypothesis: If a diverse cadre of competent community leaders is developed through participatory training and networking, then civil society will be strengthened in Burma. (See graphical representation of results framework, Appendix II.) While this hypothesis gives a broad overall picture of the project’s goals, it is extremely difficult—if not impossible—to prove causality and effectively measure strengthened civil society. A more testable hypothesis is the following, which can be considered as a proxy development hypothesis of the aforementioned. If learning and networking opportunities are created for a diverse cadre of competent community leaders, then community leaders will use leadership competencies and networks in their work. This evaluation takes it a step further to test the hypothesis that trainees will actually contribute to increased organizational capacity where they work, used as a proxy for strengthened civil society. The development hypothesis should be tested to the extent possible in this evaluation. F) Existing Data (Key Documents to Consult) Attached with SOW: • Updated project description (August 2011) • Results Framework • Participant Geographic Origins Spreadsheet 2008 The scope of work focused primarily on providing outreach through language teacher training and providing distance education to give students who for ethnic, religious, and political reasons have been denied access to higher education. 2009 The goal of the project was to strengthen civil society by promoting the principle of inclusive participation and respectful collaboration among diverse citizens in support of community development through the American Center in Rangoon. The program was modified from previous iterations to expand or reshape each of the technical components: English Teacher Training; Training of Trainers; and Distance Education. 2010 The development hypothesis was strengthened to create tighter links between learning/networking opportunities and the use of leadership competencies and networks in community leaders’ work. 2011 A number of further programmatic changes were made based on a series of strategic reflection and planning sessions, monitoring and evaluation data collected and analyzed as part of a 2011 outcome evaluation, and discussions with key stakeholders. See pp. 2-5 of Revised Program Description (August 2011) for details. 2012 An internal World Learning Impact Evaluation Report finding suggests future program changes, including: • Emphasize advocacy skills in all courses. Ensure that all long course trainings continue to include and strengthen “returning to your community” modules to provide tools to influence others and facilitate opportunities for skills application and change. • Enable CSOs to improve communication and information sharing channels between each other, crossing technical and geographical lines (and ethnic or religious lines, if an organization is oriented towards a particular ethnic group or is faith-based). • Expand courses (both short and long courses) outside of Rangoon as much as possible to reach those organizations or community leaders who are unable to access WL programs in Rangoon. • More directly target organizations conducting civil society strengthening programs and members of advocacy groups and/or those working in issues that have seen viable operational space and achievements e.g. the environment, labor rights, women’s issues and HIV/AIDS work. Conduct organization-specific courses to maximize organizational change potential. 31 To be provided within two (2) business days of award: • Performance Management Plans (PMP), including Indicator Tables (2010-2012) • World Learning Outcome Evaluations (2010, 2011) II. EVALUATION RATIONALE A) Evaluation Purpose This performance evaluation, timed before the completion of World Learning’s project in Burma, is firstly intended to learn to what extent the project’s goals and objectives have been achieved. The evaluation should build upon previous monitoring data already captured by PMPs, previous internal evaluations conducted by World Learning, and provide “bigger-picture” insight on the validity of the program’s development hypothesis and results framework. The project modifications aforementioned reflect a desire to move impact of the project’s activities from an individual level to organizational change to wider community change. The evaluation should capture—to the extent possible— how this higher level impact may have been made. It is important to note that the project’s transformation has occurred during a rapidly changing environment in Burma. This evaluation should also attempt to provide some insight into the contextual changes which have happened simultaneously, and how these may have affected programming. On a higher level, USAID also seeks to understand the efficacy of World Learning’s approaches in Burma, in order to add to the growing body of knowledge informing strategic future civil society programming in Burma. While the cooperative agreement with World Learning will officially be ending in November 2012, this evaluation is intended to provide key insights into best practices and barriers to civil society programming in Burma. As a secondary purpose, the evaluation is also a learning opportunity to explore possibilities for working with local Burmese evaluators, and build capacity for future evaluations. B) Audience and Intended Uses Stakeholder Intended Use USG staff, especially USAID governance and program/project development officers Make programmatic and funding decisions which maximize civil society outputs; Improve accountability to program goals and objectives by exploring improved monitoring and evaluation methods through local World Learning project staff Make programmatic adjustments as needed and understand efficacy of approaches for possible future O i ther implementing partners Learn from World Learning civil society programming to focus their own civil society programming on approaches that are effective and efficient C) Evaluation Questions 1. Is the training of community leaders actually contributing to strengthened organizational capacity27 in the organizations where project alumni are working? 27 Strengthened organizational capacity will be used as a proxy for strengthened civil society, the ultimate goal of the program; Evaluators may choose to explore other relevant proxies for strengthened civil society. 32 2. What are the key barriers to community leaders using acquired learning and networking competencies in their work? 3. In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Are there signs that the project alumni will continue to apply knowledge and skills attained after the project’s end? 4. During the different stages of the project, and in light of the rapidly changing contexts within Burma, what approaches have been the most effective and efficient in achieving the project’s goals? Which approaches have been the least effective and efficient? 5. What were the unintended consequences of the project, if any? III. EVALUATION DESIGN AND METHODOLOGY A) Evaluation Design The main evaluation design is most feasibly a snapshot design, as the evaluation will primarily examine the group receiving the intervention (the community leaders/trainees) at one point in time (near the end of the project). However, as the evaluation will be examining and analyzing previous monitoring and evaluation documents, as well as asking certain questions of informants retrospectively to better understand the impacts of the project over time, the evaluation will incorporate aspects of a time series approach. Rapid appraisal methods to understand barriers to specific behaviors, such as Doer/Non￾Doer Analysis or Barrier Analysis,28 are also recommended, particularly to address Evaluation Question #2. The consultant may suggest alternative designs which effectively and rigorously answer the evaluation questions, given the budget and time constraints set forth in this Statement of Work. B) Data Collection and Analysis Methods The following chart gives suggestions for data collection and analysis methods. The data collection and analysis will be primarily supported by a subcontract to an established local Burmese organization with experience in qualitative research methods and professional interpretation and translation. (See Section V for further details on the Evaluation team composition.) The consultant may suggest alternative methods which effectively and rigorously answer the evaluation questions, given the budget and time constraints set forth. Evaluation Questions Type of Answer Needed Suggested Data Collection Method(s) Suggested Sampling or Selection Criteria Data Analysis Method(s) 1. Is the training of community leaders actually contributing to strengthened organizational capacity in the organizations where project alumni are working? Cause-and￾Effect Surveys Focus Group Discussions Randomized sample of project alumni and persons in the organizations where they are working TBD by external consultant Disaggregation by gender and minority groups 28 Academy for Educational Development (Doer/Non-Doer Analysis) and Food for the Hungry International (Barrier Analysis) 33 2. What are the key barriers to community leaders using learning and networking competencies in their work? Descriptive Surveys In-depth interviews Randomized sample of “Doers” (community leaders using competencies in their work); “Non-Doers” (community leaders not using competencies in their work; Randomized sample of persons in the organization where j t l i TBD by external consultant Disaggregation by gender and minority groups 3. In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Descriptive Surveys In-Depth interviews Direct observation Randomized sample of project alumni and community members TBD by external consultant Disaggregation by gender, ethnic group, religion 4. During the different stages of the project, and in light of the rapidly changing contexts within Burma, what approaches have been the most effective and efficient in achieving the project’s goals? Descriptive Key-informant interviews Key program staff TBD by external consultant Disaggregation by gender and minority groups 5. Were there any unintended consequences of the Descriptive Normative In-Depth Interviews Key-informant interviews Selected program staff, project alumni, community members TBD by external consultant Additionally, the Evaluation Team will investigate the validity of the some of the findings of World Learning’s internal 2011 and 2012 evaluation in order to provide input into the aforementioned evaluation questions. A) METHODOLOGICAL LIMITATIONS There were unfortunately no full evaluations of the project before World Learning’s internally-led evaluation in 2011, and monitoring data is often inconsistent due to the various iterations of the project and the highly sensitive operating environment in Burma. Information gathered from this evaluation that asks retrospective questions will be subject to significant recall bias. Without consistent indicators collected over time (including baselines), it is very difficult—if not impossible—to truly prove causality in this final performance evaluation. World Learning staff who completed the 2011 evaluation also expressed concerns of the “Hawthorn effect,” i.e. participants may be responding a certain way (usually positively) simply because they know they are being tested. In examining the outcomes of the World Learning 2011 internal evaluation, most of the findings are overwhelmingly positive. While this may give merit to World Learning’s intervention, Burmese culture which rarely shows dissent to authority may have also led to respondents refraining from expressing any negative outcomes of the project. The 2012 World Learning internal Evaluation also shows high potential for bias due to the fact that it was internally conducted, i.e. project staff were facilitating data collection for their own project. There is also significant selection bias given the focus on Rangoon-based participants. In order to minimize the aforementioned biases in the proposed evaluation, there is a need for culturally￾sensitive in- depth interviews and focus groups, preferably led by local Burmese with no particular relationship to the US government or World Learning. In order to further minimize bias, the evaluation team will be led by an external consultant, and USAID staff comprising the rest of the team will have had 34 no previous involvement with the project. At the same time, it is also important to understand that local capacity for rigorous, objective qualitative data collection may currently be questionable due to lack of familiarity with qualitative evaluation methodology and the terminology used with civil society programming. The Evaluation Team lead would need to design a data collection plan which uses triangulation in order to test the validity of the findings (for example, local Burmese data collectors conducting in-depth interviews and the Evaluation Team lead and/or USAID staff conducting focus groups on similar topics). In order to have a richer understanding of the impact of the program, it will also be very helpful to expand the evaluation’s target population to include community members, co-workers, and other persons in the networks of project alumni. This may necessitate travel to other provinces and the Evaluation Team would need to exercise sensitivity and caution in their approach. The rapidly changing political environment in Burma also may mean restricted travel to certain areas, and notifications for these travel restrictions may be granted with little advanced noticed. The Evaluation Team Lead should aim to produce an evaluation with maximum evaluation rigor, but at the same time understand the flexibility required in the rapidly changing environment of Burma. Questions on feasibility may be directed to USAID/RDMA. Should there be any methodological changes to the Evaluation Strategy or Work Plan, the Evaluation Team Lead must propose these changes in writing to the USAID/RDMA Evaluation Lead and will be subject to further approval. IV. EVALUATION PRODUCTS A) Deliverables Deliverable Description Approximate Due Date Evaluation Strategy (updated from originally submitted proposal) Updated Evaluation Strategy (3-7 pages) submitted electronically to the USAID/RDMA Evaluation Lead and COR, including but not limited to: intended evaluation design, approach, methods, evaluation schedule, sampling plans, data analysis plans. Must provide brief justifications of choices in approach, emphasizing the maximum evaluation rigor possible given the limitations of a complex programming and evaluating environment. Must also include any updated details of subcontract with local Burmese organization for data collection. 5 business days after award Field work plans and draft data collection instruments Detailed overview (3-7 pages, not including data collection instruments) of field work plans, submitted electronically to the USAID/RDMA Evaluation Lead and COR, including but not limited to intended site visits, anticipated contacts, list of key informants, scheduling, etc. Must include draft data collection instruments, such as surveys, focus group discussion questions, etc. as attachments. Must also include draft 8 business days after award 35 Field Work Update Brief update on field work (2-5 pages) submitted electronically to the USAID/RDMA Evaluation Lead and COR, including but not limited to field work completed, preliminary findings, and any changes to field work approach or plans 6 business days after start of field work Draft report and presentation of preliminary findings at the RDMA Draft report of evaluation, findings, and recommendations, submitted electronically to the COR and USAID/RDMA Evaluation Lead. Must be written in English and must not exceed 30 pages in length (excluding appendices). Must also include a draft slide deck (not to exceed 20 slides), for a presentation of preliminary findings, which will be approximately 1.5 hours in length. 5 business days after completion of fieldwork (draft report & slide deck); Actual presentation will be scheduled based on availability of evaluation and USAID staff, but not exceeding the length of the external evaluator’s contract Evaluation final report + Virtual out- briefing with USAID/RDMA Final report (not to exceed 30 pages, excluding appendices), submitted in English electronically and in hard copy (2 copies) to the COR and USAID/RDMA Evaluation Lead. At minimum must include the following contents: a) Executive Summary; b) Methodology; c) Important Findings (empirical facts collected by evaluators); d) Conclusions (evaluators’ interpretations and judgments based on the findings); e) Lessons Learned and Recommendations (proposed actions based on the conclusions, including some emphasis on future designs and strategy). The Final Report must meet the Report Guidelines from USAID’s Evaluation Policy (below) for additional guidance. 14 business days after completion of fieldwork Note: USAID/RDMA Evaluation team staff will be responsible for providing the Evaluation Team lead with comments and suggestions to all the aforementioned deliverables within 3 business days of the deliverable’s submission. B) Reporting Guidelines 36 After the submission of the Final Evaluation Report, USAID/RDMA’s Program Development Office (PDO) will be responsible for submitting the document in full to the USAID Development Experience Clearing House for full and active public disclosure within 3 months of being finalized. Should there be concerns regarding the sensitivity of any of the evaluation findings, USAID’s Office of Learning, Evaluation and Research in the Bureau of Policy, Planning, and Learning will make the determination if an exception to full public disclosure should be made. This must not be the decision of the external consultant or of the RDMA. V. TEAM COMPOSITION Evaluation Team Members will be as follows: Evaluation Team Member Roles/Responsibilities Qualifications External Evaluation Consultant, hereafter “Evaluation Team Lead” Draft evaluation framework, methods and approach; design data collection instruments; lead analysis of data and draft evaluation report; locate and manage subcontract to local Burmese organization; Organize calendar/travel/meetings Level I Evaluation Specialist; graduate level degree in evaluation￾related course; 7+ years experience designing, managing, and conducting evaluations, with emphasis on fieldwork data collection in sensitive programming environments; 5+ years experience in evaluating or managing civil society programs; Strong ability to train others in evaluation data collection; Proficiency in Burmese oral and written skills a plus. USAID program officer, hereafter “USAID Evaluation lead” USAID Evaluation lead Provide guidance in evaluation design and methodology; provide leadership, along with the Evaluation Team lead, in training local data collectors on evaluation methodology and data collection; provide input into drafting of Evaluation Report and presentation; serve as the liaison between the Evaluation Team Lead, Program/Project development officer; USAID evaluation specialist; graduate degree in program design, monitoring, and evaluation; no direct involvement with the implementation of the WL project 37 World Learning, and RDMA, along with the USAID/RDMA AOR. USAID staff (2) Alternate to USAID Evaluation lead; Review and provide input on drafts of evaluation deliverables; participate in evaluation field work; provide input into analysis of evaluation data USAID governance and contracting officers with no direct involvement with the implementation of the WL project Data collectors [local Burmese staff] (3) Carry out in-depth interviews and focus groups; Participate in trainings on evaluation methods and data collection; translate data collection instruments; transcribe and translate evaluation qualitative data; serve as interpreters and cultural mentors, as needed for expat staff Completion of university-level degree required; No direct involvement with the implementation of the WL project; Must be fluent in English and Burmese; Proficiency in a language of one of Burma’s ethnic minorities to be surveyed in this evaluation helpful; Experience in data collection and qualitative evaluation methods preferred; Experience in professional translation and interpretation helpful; Note: Efforts will be made to have an equal balance of genders and representation from not only the ethnic/religious majority. VI. EVALUATION MANAGEMENT A) Logistics All work will be conducted in close coordination with World Learning program’s Agreement Officer’s Representative (AOR) and the Task Order Contracting Officer’s Representative (COR) who will provide guidance to the work of the Team Lead specifically hired for this evaluation. The Program Officer and AOR will provide strategic direction throughout the evaluation process by providing comments and input into the development of the evaluation framework and approach, evaluation report, and presentation to the RDMA. Evaluation Team Lead will keep USAID staff abreast of the evaluation’s proceedings through the deliverables outlined above and must be available to respond to regular communication and feedback. The Evaluation Team lead will work through the AOR and/or Program Officer to coordinate and hold consultations with World Learning staff and beneficiaries. USAID staff included in the Evaluation Team will be an integral part of providing input into the deliverables above and supporting the implementation of the fieldwork in Burma. The Evaluation Team Lead will be responsible for organizing scheduling and other logistics of the evaluation, although USAID administrative staff will be responsible USAID staffs’ own travel and lodging arrangements, etc. The Evaluation Team Lead will not be responsible for supervision of the USAID staffs’ day-to-day responsibilities, but will rather provide leadership to the overall evaluation, in which USAID will participate. The Contractor will be responsible for the selection and management of the services of a sub-contract to a local Burmese organization that will be the main staff responsible for the evaluation data collection. At least one field day will be reserved for a training for the local Burmese data collectors, led jointly by the Team Lead and the USAID Evaluation Lead, on evaluation data collection. This may also include testing of data collection instruments for linguistic and cultural accuracy. The data collectors may provide some insight into the report findings, but will have the primary role of providing objective (to the extent possible), linguistically and culturally appropriate data collection. B) Estimated Scheduling and Level of Effort 38 This evaluation will be performed approximately from September 2012 to November 30, 2012. The field work must be completed no later than November 30, 2012. The Contractor shall provide a tentative schedule of activities such as the one below, based on the approximate deliverable due dates found in Section C, IV (Evaluation products), A (Deliverables). While the Contractor must propose an Evaluation Strategy and corresponding work plan with the highest evaluation rigor possible, the Contractor must also consider the limitations of the complex operating environment in Burma. The Contractor is advised to consult the Illustrative Level of Effort (LOE) chart below to guide the proposed activity dates. Activity/ Deliverable Estimated Date(s) - TBD by the Contractor, in coordination with USAID/RDMA Virtual kick-off meeting between USAID/RDMA Evaluation Team members and Evaluation Team Lead Desk review, development of evaluation strategy, drafting of field work 29 Deliverable: Revised evaluation strategy Deliverable: Field work plans and data collection instruments Fieldwork30 (approximately 6 days Rangoon-based, 5 days Mandalay￾based, and 5 days travel to other provinces), includes approximately 1 day of training of local data collectors in Rangoon and 1 day at the end of fieldwork time for team de-briefing and summary of initial findings Deliverable: Field work update Preparation for presentation of initial findings at RDMA and Draft report 31 Deliverable: Draft final report and presentation Preparation of final report Virtual out-briefing Deliverable: Final Report The estimated LOE for the External Consultant is 45 days, and for 3 local data collectors is 60 person days. The following chart is for illustrative purposes only. The Evaluation will also be supported by 3 USAID staff, for a total LOE of approximately 34 person days, of which approximately 22 days will be field work. Please see Section C, V (Team Composition) for further details on the USAID staffs’ proposed involvement. The Consultant must propose an LOE chart which sufficiently addresses the Statement of Work with the most rigorous evaluation methods and sample sizes possible, but considering the logistical limitations of the operating environment in Burma. USAID/RDMA recognizes, for example, that doing a truly randomized sample of an adequate sample size may not be possible given constraints of in-country travel and sometimes lack of contact means of project participants. The Consultant must consider the cost￾benefit of additional LOE days to the strength of the evaluation. For example, if only a very small number of project participants are from a province to which travel would be costly and time- consuming, the 29 During this period, the Evaluation Team Lead is not expected to be at USAID/RDMA, but is expected to be available for consultations, conference calls and otherwise available by electronic means to facilitate preparation for the evaluation. The Team Leader will also be expected to begin coordination with the field over this period 30 The Evaluation Team is authorized a 6-day work week during the field work period. 31 This preparation period may be at the USAID/RDMA Office in Bangkok for the Evaluation Team lead to continue close communications with USAID Evaluation Team members and for the Evaluation Team Lead to be physically present for the Preliminary Findings Presentation. 39 Consultant must consider if the costs of including these participants in the evaluation outweigh the benefits to the evaluation, or vice versa. Task Evaluation Team Lead (external consultant) Local Data collectors (3) Total # person days Preparation, Review of Documents 2 2 Preparation/update of approach, methods, fieldwork logistics, development and pre-field testing and translation of data collection instruments; including logistics of subcontract to local Burmese data collectors 7 3 x 1 = 3 14 Fieldwork (Rangoon-based)32: Training for local data collectors; survey instrument testing; data collection, including some simultaneous analysis 6 6 x 3 = 18 36 Fieldwork (Mandalay￾based)33: Data collection, including some simultaneous 6 5 x 3 = 15 31 Fieldwork (other provinces): Data collection, including some simultaneous analysis 5 5 x 3 = 15 20 Field de-briefing and summary of initial findings 1 1 x 3 = 3 4 Analysis and draft report & presentation preparation 8 2 x 3 = 6 18 Final report preparation 10 14 Total Person Days 45 60 105 person days 32 Evaluation respondents may travel in from provinces surrounding Rangoon. 33 Evaluation respondents may travel in from provinces surrounding Mandalay. 40 ANNEX II: EVALUATION DESIGN STRATEGY I. Introduction International Business and Technical Consultants, Inc. (IBTCI) submits this evaluation strategy to USAID/RDMA as a deliverable under AID-486-TO-12-00003 following the Final Performance Evaluation (Burma Outreach and Distance Education Project) Scope of Work. The ‘Burma Outreach and Distance Education Project’ has been implemented by World Learning (WL) since 2007. However, since its inception, it has evolved from focusing on relief efforts following the 2007 Cyclone Nargis to its present form, which endeavors to support sustainable community development through leadership from Burmese civil society. The drivers of these changes have been a combination of factors including: opportunities arising from the opening up of the operating space in the broader context, utilization of learnings gleaned from project monitoring and evaluations fed back into adaptive management loops, as well as considerations from strategic reflections and planning sessions. In this regard the project has sensibly adopted an evolutionary, iterative process and program progression to respond to the sensitive and challenging environment of Burma over the last five years. As a result, this final evaluation, which will reflect the environment and complexity of the cultural and political context in which the project is situated, will employ a variety of methodologies to assess the success and impact of the project at higher levels while simultaneously being mindful of the differing ecosystems within the broader ‘civil society landscape’ in which the participants and expected beneficiaries are operating. II. Burma Outreach and Distance Education Project Background A. Burma Outreach and Distance Education Project Goal The goal of the program - to develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society - has been addressed through five major components, including: 1) English Teacher Training, 2) Training of Trainers, 3) Distance Education, 4) Leadership, and 5) Networking. The program activities are nearly wholly capacity building inputs to a range of individual Burmese community leaders, with some support to Burmese civil society organizations. B. Burma Outreach and Distance Education Project Intended Results The project goal (recently updated in 2011) is based on the following development hypothesis: If a diverse cadre of competent community leaders is developed through participatory training and networking, then civil society will be strengthened in Burma. While this hypothesis gives a broad overall picture of the project’s goals, it is extremely difficult – if not impossible – to prove causality and effectively measure strengthened civil society. A more testable hypothesis is the following, which can be considered as a proxy development hypothesis of the aforementioned. If learning and networking opportunities are created for a diverse cadre of competent community leaders, then community leaders will use leadership competencies and networks in their work. This evaluation takes it a step further to test the hypothesis that trainees will actually contribute to increased organizational capacity where they work, used as a proxy for strengthened civil society. C. Burma Outreach and Distance Education Project Evaluation Purpose The primary purposes of this evaluation are the following: Firstly, it is intended to learn to what extent the projects’ goals and objectives have been achieved building upon monitoring data already captured by PMPs, previous internal evaluations conducted by WL, and to provide “bigger-picture” insight on the validity of the program’s development hypothesis and results framework. 41 Secondly the evaluation should capture – as much as possible - how the project has been able to move impact of its activities from an individual level to higher organizational change and on to wider community change. Thirdly the project’s transformation has occurred during a rapidly changing environment in Burma and this evaluation should also attempt to provide some insight into the contextual changes which have happened simultaneously, and how these may have affected programming. Fourthly, at a higher level, USAID also seeks to understand the efficacy WL’s approaches in Burma, to add to the growing body of knowledge informing strategic future civil society programming in Burma. The evaluation will seek to provide insights into best practices and barriers that can inform future civil society programming in Burma. As a secondary purpose, the evaluation is also a learning opportunity to explore possibilities for working with local Burmese evaluators, and build capacity for future evaluations. The Project results and outcomes are being evaluated based on the following questions. Evaluation Questions: 1. Is the training of community leaders actually contributing to strengthened organizational capacity in the organizations where project alumni are working? 2. What are the key barriers to community leaders using acquired learning and networking competencies in their work? 3. In what ways have participants been applying the knowledge/skills gained from the trainings in their local communities and networks? Are there signs that the project alumni will continue to apply knowledge and skills attained after the project’s end? 4. During the different stages of the project, and in light of the rapidly changing contexts within Burma, what approaches have been the most effective and efficient in achieving the project’s goals? Which approaches have been the least effective and efficient? 5. What were the unintended consequences of the project, if any? D. Overall Design Strategy The Burma Outreach and Distance Education Project evaluation is fundamentally a consideration of the success and impact of training programs that are aimed at developing a diverse cadre of competent community leaders. As a result some form of time series analysis is required, and so IBTCI will evaluate all programs under the Kirkpatrick Model of Training Evaluation. This model evaluates the effectiveness of training programs on the following four levels to have concrete evidence of outcomes: Reaction or what was felt during training; Learning or knowledge gained from training; Behavior change in the workplace or community after training; and Results or application of training in the workplace or in non￾implementation areas. Using a combination of quantitative and qualitative data collection and analysis techniques, the evaluation team will analyze patterns of progression through the Kirkpatrick Model of Training levels among participants to produce evidence-based findings on strengthened organizational capacity, application of knowledge, and effective approaches to achieve program objectives. The approach will also provide insights on barriers to success and unintended consequences, if any. D.1. Quantitative Evaluation Methods The evaluation team will leverage the substantial amount of M&E data collected by WL and will build on these resources in further developing the evaluation design. WL has been conducting on-going monitoring at the institutional, context, technical, and programmatic levels using a 2011 baseline, pre- and post-course surveys, competency assessment rubrics, evaluation surveys, annual observations, focus group discussions, and key informant interviews. The evaluation team will design an approach that 42 conducts follow-on surveys on past and present training participants using content from instruments listed above to obtain another round of survey data for this evaluation. The data will then be merged with previous data to have an expanded set with more information across groups (i.e. cross-section) and in more than one period (i.e. time series). D.2. Data Analysis Surveying the same participants will allow the evaluation team to conduct quantitative evaluation methods that are more rigorous than other approaches. Generally, the team can conduct a range of multivariate analysis, which is an examination of several variables at the same time. In this case, data is available on perceptions, proficiency, and other outcome variables. One major advantage of multivariate analysis is that it makes it possible to look at the outcome effects of one variable, such as competency, while holding all other variables constant. The evaluation team’s analysis can also be done over time because data for more than one period are available. An illustrative example of multivariate analysis is panel data analysis, which evaluates outcomes in spatial and temporal dimensions. The spatial dimension refers to a set of cross-sectional units, such as the participants of the five training programs. The temporal dimension refers to the observations done in three periods, i.e. during the baseline, the post-course surveys, and in this evaluation or follow-up. The combination of cross-section and time series data results in a panel data set. A major advantage of panel data analysis is it accounts for effects, sometimes unobservable, that might result in biased measures. In a single period analysis, for example, effects within a group and/or within that period might bias measurements. A relevant example would be a participant’s illness during Distance Education training that led to a low score in a competency assessment. Panel data analysis can control for these possibilities by incorporating techniques to control for those random or non-random influences on outcome variables. Thus, when stating findings among participants, such as changes in competency scores, supporting data are controlled for potential biases that might over- or underreport outcomes. Additional information on panel data analysis is provided in Annex A. The evaluation team will also conduct sub-group analysis to disaggregate findings by gender, following USAID evaluation guidelines. Analyses according to minority groups, ethnic background, and other relevant dimensions could also be done to identify significant findings at the sub-group level. Annex A provides an illustrative approach to sub-group analysis using analysis of variance. While the evaluation team can apply effective approaches in analyzing the program, there are some methodological considerations in the design. Apart from the limitations cited in Section A, Page 11, of the RFTOP, one consideration is the possibility of attrition among respondents, which is a standard concern in follow-up surveys. Respondents may no longer be accessible for various reasons such as transfers or migration, resulting in lower sample sizes in the follow-up round. However, given that there were 1,804 respondents, we are confident that the evaluation will still obtain a sample size that will be sufficient for analysis even with some level of attrition. Related to the possibility of attrition and sampling, it is important to note that the survey does not intend to contact all 1,804 respondents for a follow up. For evaluation purposes, the team will develop the sampling plan to obtain a sample size sufficient for the quantitative analysis. One approach is to have the minimum number needed for inference testing or to have the minimum needed based on acceptable sampling parameters. D.3. Qualitative Evaluation Methods The team will also incorporate a module to obtain qualitative information on the Kirkpatrick levels. Illustrative questions for each level include: • Reaction: What were your initial impressions towards (training)? • Learning: What additional knowledge in community development have you gained with ___ (training)? • Behavior Change: How has (training) affected the local community? • Results: How is (training) knowledge being passed on to those who have not experienced it? 43 Focus group discussions will be conducted with a sample of project alumni and a separate one with persons in the organizations where they are working to obtain this more experiential information so that findings will have more depth. The discussions will provide confirmatory evidence from findings from surveys and interviews and are expected to last two hours. Each group will include homogeneity among participants to ensure more openness and willingness to provide information. Key informant interviews will be conducted with program staff to obtain insights on program design, management, and implementation. Interview agendas will explore issues of effectiveness, or how activity outputs achieved program objectives, and efficiency, or how activity outputs were achieved with the least inputs. Each interview is expected to take approximately one hour. Another set of informant interviews will be conducted with project alumni and community members to have a first-hand perspective on program outcomes. Separate interviews will be conducted with a random selection of “doers,” or community leaders using their competencies at work, and “non-doers,” or community leaders not using their competencies at work. Their community-level insights will provide information on applying their knowledge to the workplace and community as well as local barriers faced and the unintended consequences of the program, if any. Interviews with some alumni and community members will also address efficiency considerations in logistics, such as in areas where surveys cannot be done due to travel and/or logistical constraints in reaching every respondent. Qualitative data will be collected and coded to classify information and identify response trends and patterns. Data will also be disaggregated by gender and minority groups to examine response trends and patterns at the sub-group level. III. Methodology and Data Collection Plan This final evaluation will be conducted in accordance with USAID Evaluation Policy so that reliable data will be produced to support evidence based findings, be a sound basis for analysis that will lead to conclusions and recommendations that are useful for assessing the progress to date and also provide support for recommendations. In line with the policy and the methodologies noted above, the evaluation combines qualitative and quantitative methods and instruments, including key informant interviews, surveys, focus group discussions, as well as a document review, and briefing meetings with Project staff. This approach allows for the verification of the findings through triangulation. The Evaluation Team understands triangulation as a method used during the evaluation process to check and establish validity of conclusions by analyzing the responses to evaluation questions from multiple perspectives, such as desk review, interviews, surveys, etc. Triangulation of data enables evaluators to ensure validity of conclusions, i.e. that the findings of assessment are true (accurately reflecting the situation) and certain (supported by the evidence). It is important that due to the operating context this evaluation will necessarily be flexible and adaptive in nature, evolving as opportunities arise and managing risks and constraints in a pragmatic and sensible manner. The consequence of this situation is that the evaluation team (including USAID members) is fully committed to working together to maximize the collection of data and understanding of the project and its impacts through a collaborative and synergistic approach that may entail ‘tag-teaming’, taking on and sharing multiple roles throughout the assignment, specifically with regards to data collection where the roles and responsibilities of the IBTCI external team lead, the USAID evaluation lead and the USAID evaluation team staff overlap as per the Scope of Work. Tag-teaming, which may take on the form of splitting up the team to conduct several interviews and focus groups simultaneously, will expand the breadth and depth of information collected. Some examples of the issues, risks (and their management) are noted below as illustrative of IBTCIs’ and the evaluation teams’ commitment to deliver a quality product under potentially difficult circumstances: A full and deep understanding of the variability of the context across Burma and the constraints and circumstances in which civil society is operating in the different provinces, is the ideal in this important 44 evaluation. Pragmatic time considerations, however, as well as difficulties in political sensitivities and access to some areas, the apparent ‘migration’ of project alumni to larger urban centers inevitably means that a balance will need to be struck in deciding where it is most appropriate to sample CSOs, their activities and personnel who have participated in the program. WL is not a registered organization in Burma and some of the CSOs operate in very sensitive areas of the country where there are insurgent activities against the government. This means that open discussions naming them or talking about their work indiscreetly may jeopardize their work and at its most serious may place staff in difficult situations. Managing such risks requires ongoing vigilance and sensible caution, taking the advice of those on the ground, triangulating information with other trusted informants before taking action, ensuring that ‘back-up’ plans are in place – for instance using the local sub￾contractor in particular circumstances, or undertaking the interviews or meetings in alternative venues or locations, where visibility is reduced and anonymity increased. Knowledgeable informants within Burma note that the letters of authorization from the relevant counterpart Ministry are required to undertake the evaluation and can take from two weeks to several months to obtain. Without this letter of authorization it may not be possible for the expatriate members of the ET to conduct focus group discussions with CSO organization staff or have meetings directly with the communities. Again management of these risks may involve rearranging the fieldwork to accommodate these conditions, which could entail requesting the subcontractor to lead some focus groups without the having an expatriate ET member present. The evaluation will be carried out in four phases which will overlap and in some cases be running simultaneously: • Inception phase and the document review: these will enable the continued honing and development of the methodology and the evaluation instruments. • Fieldwork phase for the ET: comprises the travel to Burma and within Burma undertaken by the ET over a period of at least 18 working days. The ET will conduct interviews, focus groups with a range of stakeholders as well as engage stakeholders in participatory exercises. • Ongoing Research comprising of face-to face interviews and surveys to develop robust quantitative data, as well as possible focus group discussion, and logistical support to be undertaken by a local research firm, MSR. • Analysis and report writing phase will include two main deliverables: the fieldwork de-brief meeting with USAID; and the evaluation report that is based on the data gathered in the field. The ET will finalize the report upon receipt of comments and inputs from USAID. Inception Phase The inception phase focuses on detailed desk research and familiarization with the project concept, vision and results thus far. This phase includes the following steps: Initial discussions about expectations and current context with USAID, WL, MSR. Preparation of a detailed work plan, timing and scope of the field mission, institutions likely to be met, survey data collection approach, sample and target regions; as well as research instruments; Review of USAID comments and update of the work plan based on these comments. Desk Review Document review includes: Project documents, work plans, PMPs, previous evaluations, annual reports; Studies, reports and other relevant documentation prepared and commissioned by the project; and Relevant policy publications and studies from donors and other institutions. The document review and the analysis of the project interventions are used for the further ongoing development and honing of the evaluation matrix and design of the research methods to be applied in the main assessment stage as they are available. 45 Data Collection Phase The purpose of the data collection phase for the evaluation is to systematically collect the information required to identify findings through answers to the evaluation questions. This data and the information from the previous phases will be used for conclusions and recommendations. The main assessment phase will include coordination with the WL team, contacts with stakeholders and counterparts, and the application of data collection methods. The main data collection methods to be applied, within this phase, are the interviews with stakeholders, focus groups, surveys, and the follow up interviews with the WL project team. For the data collection phase, IBTCI will be working closely with our subcontractor, MSR, to refine the data collection tools so that they are culturally and linguistically appropriate in both English and Burmese. Additionally, MSR will be the primary interface with interlocutors and respondents in localities where outsiders may be viewed with skepticism and suspicion. In this phase, the triangulation strategy based on the interviews will assist in serving to question and/or verify the preliminary findings from the desk phase. In the interviews with the different stakeholders, their assessment of the project and its impacts will be sought. While the individual assessments may be biased in various ways, systematic and consistent questioning will reveal patterns of interest to the overall assessment of the evaluators. Key Informant Interviews: Formal, in-depth, albeit semi-structured interviews based on a pre-decided set of guiding questions will be carried out with the following respondents where possible (as well as others that are identified during the field work phase): WL staff and technical advisors Alumni of the program and CSO leaders; CSO leaders from peer organizations that may not have participated; These interviews will be specialized; however the questions will be submitted to USAID in a following deliverable to ensure parity and equity in the evaluation. Whenever possible, two members of the Evaluation Team will be present at interviews for the purpose of both note taking and also to compare impressions/conclusions after the interview. Follow-up Surveys: As noted above, surveys based on existing models already utilized by the project will be an important component and tool to build on the existing monitoring data as well as complement the direct narrative data collected during the desk review, field visits and the interviews. It will allow respondents to provide information, share views and opinions while remaining anonymous. The purpose for this evaluation is to reach out particularly to those alumni, partners and beneficiaries of the project, with whom the ET will not meet personally during the limited field phase. The survey respondents will be contacted either by phone or face-to-face. IBTCI estimates the sample size to be approximately 200 respondents. This number will be confirmed in consultation with WL and MSR. Focus Group discussions: Wherever possible focus group discussions will be organized to allow multiple participants to share opinions and ideas on the project. The focus groups will primarily use a set of consistent participatory methodologies used to gather information that will place the project within the specific political context at both the national and local levels as well as explore the issues around constraints to greater civil society activity and visioning exercises for the future. Examples of the types of participatory methodologies that will be developed include: • Timeline exercises comparing significant political events at the national and local levels that have had an impact on program activities (both for the CSOs and for the broader program) and how this has affected programs, altered them, or driven adaptations in practice. • Visioning exercises whereby CSO staff and alumni consider how and what their activities might look like in the future as well as considering their longer term needs for support. • ‘Additionality’ exercises which focus on identifying what might have happened or would not have happened if the program support had not been provided by USAID. 46 • A score card approach considering perceptions of communities and / or CSO staff / training program alumni on the services, support, technical expertise, etc. that were provided by WL and the program. Throughout the main assessment stage, data collection methodologies will be adjusted and modified in response to the constraints and opportunities encountered, in order to maintain and improve the quality of the evaluation. Analysis and Reporting The analysis of received information, data and perceptions of the project will be an ongoing activity of the ET. The team members will discuss and compare notes at regular intervals during the data gathering and document review stages. The evaluation questions as stated in the Scope of Work will provide for a fixed reference and guide for the analysis, conclusions and recommendations. The findings at each stage of the in-country visit, will be noted and, as analysis progresses, these notes will be used to triangulate evidence for conclusions. Formulation of conclusions and recommendations will be based on the triangulation of collected documentation and its review; responses to the survey, focus group analysis, discussions and interviews with a broad range of stakeholders. The consultants’ experience in other projects or evaluations within this sphere will also be relevant. The notes on gender implications, and also differences in responses by gender, will be cross tabulated and reported. Sampling Through face-to-face or telephone surveys, the ET will reach out to as many as 200 Alumni, as well as communities, benefitting CSO staff and key informants. It should be noted that compared to the number of individuals who have participated and benefited from project interventions during the two-year period, the evaluation sample size is not large enough to statistically generalize the findings to the rest of the beneficiary population. The sample does, however, offer a general pattern of opinions and perspectives that, when triangulated, offer evaluative “impressions.” Individuals and organizations to be included in the sample will be limited by circumstances on the ground. In order to ensure that the survey findings are based on inputs from as broad a range of stakeholder as possible, we will aim at ensuring that the sample is drawn from a full list of representatives participating in each component of the project (trainings/courses). The sample will be generated from a using a purposive approach rather than a randomized approach as many of the interlocutors may be selected out of convenience of already being in a location where the evaluation team is or because they are in a village that is easily accessible by the evaluation team. For those interlocutors who are in localities where expatriate members of the evaluation team cannot travel, the subcontract will conduct those interventions. The sample may also be purposively chosen to ensure that each course conducted by WL is represented. Survey data analysis will include both an analysis of the entire sample, but also cross tabulations in order to explore views of different types of grantees, trainees and media outlets, men vs. women, etc. Final survey reports will also include the cross-tabulated data, and be presented as an annex to the main evaluation report. Geographic coverage is relevant to this evaluation but it may not be possible to visit as many geographic areas as will reflect the full program for a variety of reasons. Nevertheless where it is not possible to visit the ET will do their utmost to either bring respondents to their location or utilize the local survey partner to visit where access is precluded to international staff. While this may not deliver a 100% perspective across the country, it should enable the ET to glean a sufficient understanding, when combined with data collected from surveys, polls and questionnaires implemented in areas not physically visited, to present valid conclusions and relevant recommendations. Weaknesses and Limitations of the Evaluation Design and How They Will be Managed There are two overall weaknesses and limitations for this evaluation. The following list provides additional details as to how these factors could impact the evaluation. 47 The WL project finishes on the November 30 and project staff may not be easily available for consultation after this time. This is why the contract sensibly determines that field work should be completed by this date, however, the bureaucratic and idiosyncratic reality of the variable length of time for the visa process and the possible need to obtain a letter of authorization from the relevant ministry, means that time is at a premium and may be constrained and outside of the control of the ET. The length of the fieldwork as prescribed in the Scope of Work is limited and combined with the above constraint is insufficient to be able to cover the breadth of the country and the optimum range of project participants. Burma Outreach and Distance Education is a large project with numerous areas of intervention and wide range of partners and more time would be beneficial to ensure more complete coverage. To mitigate this and to complement the meetings and interviews conducted by the ET, IBTCI has partnered with the local firm, MSR, to conduct a range of interviews and surveys that the evaluation team would not be able to do on its own (within the constraints of the budget limitations). Following are limitations and weaknesses of the evaluation methodologies and how IBTCI will manage them. Document Reviews: Limitation/Weakness: Information may be inapplicable, disorganized, unavailable, or out of date; biased because of selective survival of information; incomplete or inaccurate; time consuming to collect, review, and analyze many documents. How it will be managed: IBTCI has provided provide USAID and WL with a supplemental list of the relevant types of documents required to conduct the desk review. When the material is received, IBTCI will cull through the documentation to ensure that it is organized, current, and complete. If any gaps are found, IBTCI will communicate this with to USAID to determine how to fill the gaps and/or alternatives to collecting the necessary information. Besides, the ET will collect and analyze all reports and data that will emerge as relevant once in the country. To ensure that all files are managed and available at all times to the team, a Drop Box file has been created and will be accessible to the team. Key Informant Interviews: Limitation/Weakness: Findings can be biased if the informants are not carefully, or purposively, selected. When only a few people are interviewed, it may be difficult to demonstrate any general, let alone statistically significant, validity of the findings. It is difficult to prove that the interviewees are, in fact, knowledgeable and informed and that they are representative of their peers in their information and recommendations. Findings could be susceptible to interviewer bias. How it will be managed: IBTCI will manage the potential limitations and weaknesses of the key informant interviews by working with USAID and implementing partners to confirm that all key informants selected are relevant to the projects. IBTCI will develop a purposive sample, i.e., a qualitative sample based on those respondents likely to respond to questions and therefore to contribute to findings. IBTCI will interview as many relevant key informants as necessary within the allotted evaluation period to increase confidence in the validity of the evaluation findings. Furthermore, all data received will be triangulated through ongoing desk review and situational analysis, validation of findings through interviews, focus groups and survey as well as reflection and analysis of project reports and meetings with the team. The Evaluation Team Leader will provide an objective, scientifically rigorous set of interview guides and findings to prevent interviewer bias. Focus Groups: Limitation/Weakness: The moderator has less control over the data produced than in quantitative methods. The nature of the participatory methodology means that the moderator has to allow participants to talk to each other, ask questions and express doubts and opinions, while having very little control over the interaction other than generally keeping participants focused on the topic. By its nature focus group research is open ended and cannot be entirely predetermined. How it will be managed: The focus group moderator will come from MSR - the professional survey firm that has been subcontracted by IBTCI and will be guided by the team leader. This will increase the level of confidence in ensuring that the focus groups stay on topic without dictating the direction of the conversation so not to jeopardize the organize nature of a focus group. 48 IV. Subcontractor’s Draft Scope of Work IBTCI has submitted a request for consent to subcontract to the Contracting Officer with a draft contract, budget and scope of work. The scope of work is based on the following list of tasks: Providing travel and logistics inputs for the efficient execution of the survey work plan; Pre-testing and translating qualitative survey modules and, if needed, updating existing WL surveys; Administering all surveys to a sample taken from 1,804 participants; Daily cleaning of survey results and submission of raw and clean data sets to the evaluation teams; Providing calculations of descriptive statistics and other analysis, as needed; Providing brief summaries of data analysis outcomes; Serving as support facilitators, interpreters, and note takers during the key informant interviews and focus group sessions; and providing technical input to the Team Leader in the development of the Draft and Final Evaluation Reports, as needed. 49 ANNEX III: DATA COLLECTION INSTRUMENTS Survey / Interview Guide: Alumni – Program Participants Introduction/Informed Consent: Thank you for taking time to talk with us. My name is ………………. (Interviewer’s name), and I am from Myanmar Survey Research an independent company undertaking market research and social surveys. We are not from the government, nor any INGOs. I would like to talk with you to better understand what you have learned from the American Center Training Programs (Quilt, DE, CD, TOT) and how you are using what you have learned, and any challenges to using the things you have learned. This interview group will be about 1 hour. Everything we talk about today will be confidential. Your name will not be recorded for public disclosure and anything that we talk about in the next 1 hour stays in this room. If there is anything I ask which makes you feel uncomfortable, you do not have to answer me. Your participation and this survey with your suggestions and comments will be very helpful in future program design. General questions 1. Tell me briefly about your organisation, its size, activities and when it started, and is the main target group or beneficiaries (Organisation here means any form of structure or group of people, no matter how loose, that are working towards a common goal or purpose and that identify themselves as a group) 2. If you are an individual then please tell me about your activities and any groups you work with formally or informally? Organizational Behavior Change 1. Have you had an opportunity to apply your learning within your organisation or activities? Yes / No 2. If No – why do you think you have not been able to apply it? 3. If Yes – in what way? E.g. a. Application of technical skills improving the quality of work of the organisation b. Application of empowerment skills e.g. improved leadership, facilitation, inclusion, participation and responsibility 4. Do you think that the organisation has changed as a result of applying what you have learnt? Yes/ No 5. If Yes - In what way has it changed? 6. If NO- What have been the main barriers to organisational change? Community Impact Date today: Name of Course Taken: Year Course Completed: Gender (M/F): Age: What province are you originally from? What city/town do you currently live in? If you have moved from your home province to your current location – when did you move? If any, what is the name(s) of the organization(s) you are working in (paid or volunteer) and your main responsibilities? (If the organization is not well-known, please provide a brief description of the organization.) 50 1. Do you think there has been any impact on the broader community from your organisational / individual change? Yes / No (interviewer to understand whether or not this is (a) organisational (b) individual (c) both 2. If Yes (for each category) – what impacts do you think there have been at the community level? 3. If No – Why do you think you or your organisation has not been able to translate this into community impact? Impact on Government 1. Assuming that your organization has changed and has community impact does this have any influence on the government? Yes / No 2. If Yes - in what way do you think it has influenced the government and at what level? (e.g. improved or worsened relationships, increased space to operate, increased or decreased tolerance from the government to your activities, changed attitudes) 3. If No – why not? 4. To what extent do you think that you and your colleagues are able to influence government or the space that is available to you and for your work? Or is it that you have to respond to opportunities and take advantage of them rather than create them as space opens up? Networking 1. Did you learn about networking in the ACTP? Yes / No 2. If Yes – have you applied these networking skills? Yes / No 3. Has this networking been within your ACTP alumni peers or outside this group or both? 4. Describe your activities in the different networks you are involved in (alumni, non-alumni networks)? E.g. (a) technical skill development (b) mutual support (c) building alliances for other social objectives? (check after pilot) 5. If No - What is preventing you from networking (and working with others on civil society issues)? Future Recommendations 1. What do you think could be done to better build a cadre of skilled civil society leaders in this country apart from improved training programs? 2. Do you have any further suggestions, comments or ideas? Individual Learning 1. What was the most important thing that you learned through your course? And why? 2. Do you think that the course could be improved? If so how? Individual Behaviour Change 1. To what extent have you changed the way that you work after returning from your ACPT course?(0-5 scale 2. In what ways have you changed – give examples 3. How consistent is your behaviour change? All the time, some of the time, occasionally, never? 4. If you have not been able to apply any changes in your work or life why do you think this is so? Note to the Interviewer: 51 • Case Study: If there are any interesting story both positive (success, outstanding impact etc.), or negative (important challenge or where things went wrong) at the organizational, community or government level, please obtain greater detail and information of that particular case. If the interviewee is only able to provide some of the overall picture from this interesting story, then please identify additional informants who can add perspective and detail to the case study e.g. another work colleague, community leader or member or beneficiary. • Quotations: Can either come from the case study or from an ordinary interview. Also if the interviewee says something interesting that is worth quoting directly as it illustrates a point (positive or negative) very well, please not it down exactly. 52 Focus Group Discussion Guide – Alumni (Long-courses) Brief anonymous survey to be collected from FGD participants: Introduction/Informed Consent: Thank you for taking time to talk with us. Our names are Josie and Sam, and we are from the Bangkok office of USAID. We would like to talk with you to better understand what you have learned from the American Center Training Programs (Quilt, DE, CD, TOT) and how you are using what you have learned, and if there are any challenges to using the things you have learned. This focus group will be about 1.5 hours. Everything we talk about today will be confidential. Your names will not be recorded for public disclosure and we ask that anything that we talk about in the next 1.5 hours stay in this room. If there is anything I ask which makes you feel uncomfortable, you do not have to answer me. If you don’t understand something, please ask me to repeat or slow down. We want to be able to accurately capture your perspectives. What are you more comfortable with: us taking notes or recording the conversation? Once we synthesize our findings, the recording and/or transcripts will be fully deleted or destroyed. (Decide as a group if we will record or take notes.) Unfortunately we cannot promise you anything from this discussion, such as payment or follow-on programs. However, we do hope that we can get some ideas from you about what is important to better to strengthen CS in this country, and are committed to conveying recommendations from this evaluation to hopefully guide future project designs. Discussion Questions: 1. Course Relevance: When you took the [name the course], what were some of your main learning needs? (what did you want to get from the course?) a. What are some of the most important topics you remember from the course you took? b. Were the topics you learned things you needed at the time you took the course? 2. Applying Learning: Have you been applying the learning in your work or communities? a. If so, what are some specific examples of how you have been applying your learning? b. If you haven’t been able to apply your learning in your work or communities, what are some of the biggest reasons why you have not been able to? (e.g.: personal reasons, external context, organizational barriers, etc.) 3. Sharing Learning: Have you been able to pass the learning on to others such as co-workers, friends, neighbors, or other acquaintances? a. If so, what are some specific examples? b. If not, what are some of the reasons which keep you from sharing your learning with others? c. Have you personally seen any of your contacts with whom you have shared your learning applying it themselves? What are some examples? 4. Building Organizational Capacity: Let’s talk about the organizations you are working in (these could be places you are formally employed or informal organizations you volunteer with). Do any of you feel like your organization is now stronger because of the things you have learned and are able to apply? (For example, if your organization developed new systems, has had attitude changes, technical competence has improved, etc.) Date today: Name of Course Taken: Year Course Completed: Gender (M/F): Age: What province are you originally from? What city/town do you currently live in? If any, what is the name(s) of the organization(s) you are working in (paid or volunteer)? 53 a. What are some examples? b. How much influence do you feel your organization has on the wider community in which it works? 5. Community Level Impact: What changes, if any, do you think have been felt at the community level as a result of your learning and the learning of other alumni? Please give some specific examples. 6. Networking: Do you know others working in the same subject area as you? a. Do you network with one another (or other course alumni), formally or informally? Do you gather to talk about issues, share problems, and support each other? If so…. b. Is this locally or in other areas? c. How do you communicate as a group and how has this changed over time? d. How regularly do you stay in contact? e. What do you discuss when you meet or talk? f. To what extent do you feel meeting with others in the same course helps you to apply your learning? 7. Project Design: Do you have any suggestions for how the ACTP could have been improved to better meet your needs? [clarify that this could mean non-education programs as well.] a. How could the ACTP be improved to meet the current needs of this country, especially the need to have strong CS leaders? 8. Government: Do you feel that there is an increase in the freedom to carry out trainings/workshops/CD? a. How has this change affected how you work? How has it affected what activities you do? b. Do you think that you and your colleagues have been able to influence government entities? How about the freedom to do your work? c. Or do you respond to and use opportunities as space opens up, rather than create them? d. Depending on the time and place… i. What are your perceptions of the government’s attitude to your work? ii. To what extent can you be open at the local levels? iii. What are your experiences of government at the local level? iv. How do you manage these relationships? 54 Interview Guide 2: World Learning Local Staff Name of organisation: Role of respondent in their institution and main responsibilities: Date: Location: Kirkpatrick’s Model for Evaluating Training Programs: Covering 1. Learning: What additional knowledge has WL learned and beneficiaries? 2. Behaviour Change: How has the program affected the beneficiaries? 3. Results: How is (training) knowledge being passed on to those who have not experienced it? Introduction: Explain who you are and what the purpose of the interview is. Emphasise the confidential nature of the interview and that their anonymity will be respected and nor will any quotes if used be attributed to them personally. Request permission to cite their name at the end of the final report if they are comfortable, in a general list that will not be published Arrange for soft drinks / tea etc General questions 1. Tell us briefly about your role in the organisation and in the training program 2. Please describe the training program on which you work, particularly its evolution, to the extent that you know. 3. How long have you been involved on the ACTP? Learning 1. Please describe your understanding of the goals of the [ToT, DE, CD] project. 2. What are the most important lessons that the project has learned during the course of its life, wrt: a. Project design – if you have been involved in this. b. Working in this political environment c. Content of the program 3. Given your experience, what advice would you give USAID/USG about how to support similar initiatives in the future? 4. How do you think emerging leaders in this country can be better supported to make their organizations better able to carry out their missions? a. Please talk about the selection of participants for the program you managed. Did the IT recruit and select the right people? b. What elements need to be supported to build on the work that the ACTP has done? c. Can you identify existing capacity gaps where civil society would benefit from support? d. Are education-focused projects a useful way to develop civil society leaders? Why or why not? 5. Are there any emerging opportunities in civil society development that you can see that donors and other stakeholders should know about? Results 1. What are you most proud of in terms of your input into the project? 2. What are the achievements of the [ToT, DE, CD] program that you are most proud of? Why is this so significant? 3. In what ways do you think that the program(s) of the ACTP have helped improve the situation in Myanmar at the following levels: a. individual b. organisation c. community and civil society sector levels i.e. networks or associations 55 4. What are the most notable differences in work of the alumni that you can attribute directly to your program? Please give practical examples that illustrate the differences? 5. Have alumni used their experience and contacts in the program to develop lasting networks of alumni? If so, how? If not, why not? 6. Have you seen any unintended consequences, positive or negative, as a result of WL/ACTP? Other questions 1. Do you have any further suggestions, comments or ideas? 56 Interview guide 3: External Key Informants (USAID, INGOs, Context ‘Experts’, etc) Name of organisation: Respondent: Role of respondent in their institution and main responsibilities: Date: Location: A. General questions 1. Please tell us briefly about yourself 2. Please describe your relationship with the BODEP B. Relevance 1. How do you view the developments of civil society in Burma? Present and future role? 2. What are the most important enabling and impeding factors for a vibrant civil society in the region 3. To your knowledge, how does the program contribute to this possibility? 4. To your knowledge how does it differ from other donors/actors support to civil society reforms? 5. What do you consider to be the most important task or role of civil society at present? 6. What is your impression of the major donors for civil society reforms and development? How do they differ? 7. Have you any impression of the specific contributions made by the BODEP program (or the organisations and individuals that it has supported)? Examples? 8. How relevant is the work supported by BODEP (or the organisations and individuals that it has supported) in relation to F. The situation in Myanmar? G. Your organisation’s vision of what is needed here? H. Expanding the space for civil society? C. Effectiveness 1. In the current context what do you think is the best way to achieve the overall goals of a strong civil society in the country? 2. What types of agencies/organisations are able to make a difference? What kind of methods work best? 3. In general, do donors / agencies coordinate among themselves when supporting the civil society sector? How do they do this if at all? D. Efficiency 1. What other ways of channelling the support for a program like this, do you think that there are given the current circumstances in Burma? E. Outcomes (impact) 1. What changes if any have you yourself observed in any of the organisations or individuals that BODEP has supported? 2. Which types of organisations do you think have benefitted most from the program and why? 3. What changes have you seen in civil society and to what extent do you think that BODEP (or the organisations and individuals that it has supported) has / have contributed in terms of their support and actions? Please give examples if you can? 4. What role has BODEP (or the organisations and individuals that it has supported) played in this change processes (facilitation, contribution etc.)? 57 5. To what extent do you think that this program has been able (a) build capacity of civil society (b) build broader networks, relationships and alliances for change? Why do you think that? 6. Do you know of any similar programs or organisations working within the area of civil society strengthening? How do you relate to them? F. Sustainability 1. From your perspective how has the BODEP (or the organisations and individuals that it has supported) developed during your experience of it? 2. In what way do you consider the achievements made by (or the organisations and individuals that it has supported) sustainable, what are the main risks if donor would stop its support? 58 ANNEX IV: SOURCES OF INFORMATION List of BODEP Key Informant Interviews conducted Stakeholder Contact Title Organization City 1 USAID Sandee Pyne - Burma Program Manager, Former AOR USAID ASIA Bangkok 2 USAID Ravi Srisartsanarat – Former AOR USAID ASIA Bangkok 3 USAID Ms. Jessica Davey- Former WL Program Advisor USAID/OTI Rangoon 4 US State Department Adrienne Nutzman - PAO State Department Rangoon 5 World Learning Mr. Andrew Dick- Senior Program Officer, World Learning Washingto n DC 6 World Learning Ms. Meredith McCormac - Deputy Director of Education World Learning Washingto n DC 7 World Learning Ms. Bernice Clark- Teacher Training Program Coordinator World Learning Rangoon 8 World Learning Ms. Lindsey Stein- WL Burma Program Coordinator World Learning Rangoon 9 World Learning Ms. Dim Sian Nem- TOT Trainer and TOT Program Administration World Learning Rangoon 10 World Learning Mr. Han Soe Tun- Program Associate managing DE Program Administration World Learning Rangoon 11 World Learning Mr. Thet Naing Tun- Program Associate and Administrative Assistant World Learning Rangoon 12 World Learning David Root – Ex Distance Education Coordinator Ex - World Learning Sweden Skype 13 Alumni Aung Kyaw Phyo - Previous and current political activist Independent Rangoon 14 Alumni Aung Nay Mong - Previous and current political activist Independent Rangoon 15 Alumni Myo Zaw Aung - Director and Previous and current political activist The Innovative – Community Research and Education Center Rangoon 16 Alumni Kyi Pyar Chit Saw – Research Associate MDRI Rangoon 17 Alumni Thi La Min- Alumni and Previous political activist as well as Artistic Director Thukhuma Khayeethe Theater (Arts Travel Theatre) Rangoon 18 Alumni Jimmy Razar Boi Myanmar Youth Network Rangoon 19 External Person Nay Zin Latt – Governance Program Manager Oxfam Rangoon 20 External Person Dr. Kyaw Thu - Director Paung Ku Rangoon 21 External Person Matt Schissler – Paung Ku advisor Paung Ku Rangoon 22 External Person Mr. Zaw Oo – Director, Loka Ahlin LNGO Loka Ahlin Rangoon 23 External Person Claire Light Swiss Aid Rangoon 24 External Person Nilar Myaing – Director Local Resource Center Rangoon 25 External Person Gerry Fox – Program Director Pyoe Pin - The British Council Rangoon 26 External Person Ngwe Thein - Director Capacity Building Initiative Rangoon 27 External Person Myo Win – Executive Director Smile Education and Development Foundation Rangoon 28 External Person Dr Naing – Director Dear Myanmar Rangoon 29 External Person Sandar Myo – Senior Consultant Empower Rangoon 30 External Person May Htun Aung – Program Coordinator Spectrum Rangoon 31 External Person Chan Nyein Aung - CEO Charity Oriented Myanmar Rangoon 32 External Person Kyaw Hlaing – Director Myanmar Think Tank Rangoon 33 External Person David Tegenfeldt – Senior Advisor Hope International Development Agency Rangoon 59 34 External Person Bobby Mg - CEO Network Activities Group Rangoon 35 External Person Tom Kramer TNI Rangoon 36 External Person Matthew Mc Dermott – Governance Advisor Dfid Rangoon 37 External Person Matt Desmond – Independent Consultant on Civil Society Independent Consultant Rangoon 38 External Person Erica Tubbs – Deputy Chief of Party Pact Rangoon 39 External Person Jan Schollaert – Country Director Pact Rangoon 40 External Person Soe Htun ’88 Generation Rangoon 60 List of Documents Consulted 1. Burma Outreach and Distance Education Project – Module 3 Post Report - June 15, 2012 2. Burma Outreach and Distance Education Project – Module 2 Post Report - April 27, 2012 3. A New World is Possible – Welcome to Week 5 4. Comparative Consumer Prices (Host vs. USA) January 2011 5. Description for: Week 1: Introduction to Community Mobilization 6. Disaster Resistant Cities (Welcome to Week 1) 7. Working with Marginalized Communities (Welcome to Week 3) 8. Destructuring and Restructuring Development (Welcome to Week 4) 9. Support Services (Welcome to Week 4) 10. Community Risk Assessment and Safe Habitation (Welcome to Week 4) 11. Global Case Studies (Welcome to Week 4) 12. Designing a Grassroots Participatory Disaster Management Plan (Welcome to Week 5) 13. Description for Week 5: Synthesis 14. Description of Week 2 15. The History of Aid as an Ongoing Legacy (Welcome to Week 2) 16. Power and Empowerment (Welcome to Week 3) 17. Tactics and Action (Welcome to Week 4) 18. Tools and Resources II 19. Case Study: Disaster Mitigation in the U.S. (Welcome to Week 2) 20. Challenges in Developing Countries (Welcome to Week 5) 21. Supporting Community Involvement (Welcome to Week 2) 22. Approaches to Organizing (Welcome to Week 2) 23. Development as Contestable (Welcome to Week 1) 24. Case Studies in Community-based Mapping (Welcome to Week 1) 25. Welcome to Week 3 – Learning From Below 26. Reading 27. ACTP’s Training of Trainers: Foundational Cycle Trainer’s Handbook 28. BC=Field Observation Form 29. Biak Hlei Sung – The Sept 26 Observation 30. FY11 Analysis 31. Field Observation Form Sept 12 2011 32. Kyaw Win Tun Sept 12 2011 33. Maypale thwe Sept 13 2011 34. Naing Khey Sett Sept 13 2011 35. Naw Htoi Sept 2 Observation 36. Thin Zar Khin Myo Win Sept 28 Observation 61 37. American Center Training Programs: Community Development Program 38. American Center Training Programs: Distance Education Program 39. World Learning Burma: Civil Society Briefing Notes – March 1, 2012 40. American Center Training Programs: Training of Trainers Program TOT 41. American Center Training Programs: English Teacher Training Program 42. American Center Training Programs 43. USAID/Asia: Burma Outreach and Distance Education Project: Capacity Building for Change 44. WL Amb Mitchell Briefing 45. World Learning: Burma Outreach and Distance Education – Program Summary 46. World Learning – Strengthening Civil Society in Burma 47. Gender Equity in Development: Course Description 48. Community-Based Mapping - Welcome: Getting Started 49. Development and the Politics of Empowerment - Welcome: Getting Started 50. Getting Started/Help with RamCT 51. Community Health: Summer 2011 52. Introduction to 1520 53. DevCCommunity-Based Mapping: Welcome 54. Welcome to Development and the Politics of Empowerment – Introduction Letter 55. Training Practice 14, Lashio, Northern Shan State - September 19-20, Report 56. Activity Completion Form – Hopone – October 17, 2010 57. Option 2: Instructions: Displaying GIS data using Google Earth/Google Maps. 58. QuiLT Course May – June 2011: Civil Society and Community Development Pilot Module Report 59. Development as Contestable: Welcome to Week 1 – Readings 60. Myikyina and Lashio Outreach Training Report April 3 – April 10 2011 61. QuiLT Course May – June 2011: Civil Society and Community Development Pilot Module Report 62. Qualification in Language Teaching (QuiLT) Participant Record Booklet 63. Burma Outreach and Distance Education Project: Capacity Building for Change - Module 4 and Overall Course Report, August 17, 2012 64. Burma Outreach and Distance Education Project: Capacity Building for Change – Module 2 Post Report, April 27, 2012 65. Burma Outreach and Distance Education Project: Capacity Building for Change - Module 3 Post Report, June 15, 2012 66. World Learning Burma: Rationale and Strategy for Participant Targeting and Recruitment for American Center Training Courses – February 2011 67. Burma Outreach and Distance Education Project: Capacity Building for Change –Program Description, August 2011 68. World Learning Burma Outreach and Distance Education Project: Capacity Building for Change – Strengthening Civil Society Module 1 Post Report, March 9, 2012 69. GSLL 1508 – Fall 2009, Challenges to Smallholder Agriculture: Weekly Schedule 62 70. Syllabus: Community-Based Organizing GSLL 1517 71. Community-Based Mapping DCE-0713, International Institute for Sustainable Development 72. Syllabus: Development and Politics of Empowerment – GSLL 1509 73. Community Based Disaster Mitigation – GSLL 1513 74. Syllabus – Grassroots Participatory Disaster Management – GSLL 1511 75. Syllabus: Section Information GSLL 1501 201 - Approaches to Community Development (Fall 2010) 76. Synthesis (Welcome to Week 5) 77. World Learning Burma Rationale and Strategy for Participant Targeting and Recruitment for American Center Training Courses February 2011 78. Training Practice 12, March 19-20 Report 79. The Experiential Learning Cycle: A Model 80. The Importance of Perception 81. Communities, Rights, Justice, and Conservation (Welcome to Week 1) 82. Week 1: Introductions, Background, Exceptions, and Case Studies 83. Introduction to Community-Based Organizing (Welcome to Week 1) 84. Community-based Disaster Management (Week 1) 85. Overall Social-Economic Environment (Welcome to Week 1) 86. Smallholder Producers and Technology Transfer (Welcome to Week 2) 87. Case study (Week 2) 88. Case study (Week 2) SAME AS ABOVE!! 89. A Brief History of Development Practice (Welcome to Week 2) 90. Community Health Assessment (Welcome to Week 2) 91. Actions and Tactics for Practice (Welcome to Week 3) 92. Community Disaster Mitigation Programs (Welcome to Week 3) 93. Governance of the World’s Forests (Welcome to Week 3) 94. Sustainability and Support Services (Welcome to Week 3) 95. Protecting Those Most at Risk (Welcome to Week 4) 96. Critical Assessment and Case Studies (Welcome to Week 4) 97. Tactics and Action (Welcome to Week 4) 98. Destructuring and Restructuring Development (Welcome to Week 4) 99. The Elephant in the Room: Gender and Development (Welcome to Week 5) 100. Future Implementation (Welcome to Week 5) 101. Welcome to Community-Based Health - Welcome Letter 102. Welcome Letter – Raul Paz Pastrana 103. Welcome to Approaches to Community Development – Welcome Letter 104. Welcome letter 3 (EXTRA LETTER, NOT THERE!!) 105. Dr. Farmer’s Remedy for World Health 63 106. Burma Outreach and Distance Education Project – Capacity Building for Change: Strengthening Civil Society – Module 1 Post Report, March 9, 2010 107. Beneficiary Data Priority Alumni Identified Revised. 108. Training of Trainers’ Alumni Conference, October 28, 2012 109. Key Informants (Political & Civil Society) Revised 110. Key Contacts for USAID Evaluation 111. TOT Conference Attendees 112. Burma Outreach and Distance Education Project: Capacity Building for Change - Strengthening Civil Society Burma Final Report, Module 4 and Overall Course Report, August 17, 2012 113. Burma Outreach and Distance Education Project: Capacity Building for Change - Strengthening Civil Society Burma Final Report, Module 2 Post Report, April 27, 2012 114. Burma Outreach and Distance Education Project: Capacity Building for Change - Strengthening Civil Society Burma Final Report, Module 3 Post Report, June 15, 2012 115. World Learning Burma: Rationale and Strategy for Participant Targeting and Recruitment for American Center Training Courses – February 2011 116. Burma Outreach and Distance Education Project: Capacity Building for Change – Program Description, August 2011 117. Burma Outreach and Distance Education Project: Capacity Building for Change – Strengthening Civil Society – Module 1 Post Report, March 9, 2012 QUARTERLY REPORTS 118. Quarterly Narrative Report #1 – Strengthening Outreach and Distance Education Opportunities at the American Center, Rangoon, Burma – September 2007 – January 2008 119. IU Distance Education Facilitator Quarterly Report – January – March 2008 120. Burma Outreach and Distance Education Project Quarterly Report - July – September 2008 121. FY 2008 Q4 122. Burma Distance Education and Outreach Project – Quarterly Report – October – December 2008 123. Burma Distance Education and Outreach Project 486-!-00-07-0007-00-Quarterly Report – January – March 2009 124. Burma Distance Education and Outreach Project – Quarterly Report – October – December 2008 (RE: Quarterly Report April – December 2009) 125. Burma Distance Education and Outreach Project – Quarterly Report – July – September 2009 126. Burma Outreach and Distance Education Project – Quarterly Report for October – December 2009 127. Burma Outreach and Distance Education Project – Quarterly Report for January – March 2010 128. Burma Outreach and Distance Education Project – Quarterly Report for April – June 2010 129. Burma Outreach and Distance Education Project – Quarterly Report for July – September 2010 130. Burma Outreach and Distance Education Project – Quarterly Report for October – December 2010 131. Burma Outreach and Distance Education Project – Quarterly Report for January – March 2011 132. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – April – June 2011 64 133. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – July – September 2011 134. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – October – December 2011 135. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – January – March 2012 136. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – April – June 2012 137. Burma Outreach and Distance Education Project – Capacity Building for Change – Quarterly Report – July – September 2012 WL EVALUATIONS 138. Burma Outreach and Distance Education Project: Capacity Building For Change -Outcome Evaluation Report – March 2011 139. World Learning Evaluation 2012 project description(1) 140. Civil Society Strengthening Program Burma Final Report WL SURVEYS 141. WL Local Staff 3 - 10/25/12 142. WL Local Staff Interview 2 -10/25/12 143. WL Local Staff 1 Interview – 10/25/12 144. ANNEX 3: Working List of BODEP Key Informant Interviews to be Conducted 145. ToT (Yangon) FGD 10/27/2012 146. FGD – TOT (outside of Yangon) – 10/27/12 147. QuiLT FGD 2012_10_26.CODED 148. Josie notes 149. Interview Guide 2: World Learning Local Staff 150. Group Discussion Guide: Alumni Program Participants Version 3 151. Focus Group Discussion Guide – Alumni (Long-courses) 152. Distance Education Focus Group Discussion 153. Data collection Respondent roster 154. CD FGD 10/27 155. Best Practice Pre-Course Survey 156. Pre-Course Survey – Distance Education 157. Pre-Course Survey – FCTOT 158. Pre-Course Survey – Community Development 159. Pre-Course Survey QuiLT – July 18 – September 24, 2011 160. World Learning Burma Rationale and Strategy for Participant Targeting and Recruitment for American Center Training Centers – February 2011 161. Training Practice Pre-Course Survey 65 Annex V: PROJECT PERFORMANCE TABLES FY 2009-2011 Performance in Fiscal Year 2011 Performance Indicator No. Performance Indicator Narrative FY 11 Targets FY 11 Actual Goal To develop a diverse cadre of competent community leaders through participatory training and networking to foster the growth of civil society #1 % of course graduates who practice leadership competencies in their work 75% 100% #2 # of post-course training initiatives completed by course graduates 40 67 IR 1 Potential and current community leaders develop skills and knowledge competencies #3 % of course participants who achieve a rating of 3 or higher on course competency assessments 85% 93% #4 # participants who achieve a rating of 3 or higher on leadership component of course assessment rubric 40 62 IR 1.1 Potential and current community leaders trained on leadership and networking #5 # Course participants Women 128 460 Ethnic Minority 76 355 # by city/town of residence34 76 420 Religious Minority 76 184 Aggregate 256 635 IR 1.2 Potential and current community leaders complete mentoring/internship opportunities #6 # of participants who complete mentoring/internship Women 3 1 Ethnic Minority 2 0 Religious Minority 2 0 Aggregate 5 2 IR 2 Potential and current community leaders trained on leadership and networking #7 # networking events held without support from the project 10 67 IR 2.1 Community leaders receive technical and material support for organization of networking events #8 # of networking events supported 40 29 IR 2.2 Networking events implemented directly by the project #9 # of networking events implemented 15 18 Color Code: 1. Achieved Project Targets 2. Overachieved Project Targets 3. Underachieved Project Targets Performance in Fiscal Year 2010 Result Indicators Target Output FY10** Total Output FY10 IR 1: To equip change agents with practical training skills and content knowledge relevant to civil society. 1. # of training courses and networking events completed 2. # of participants in training courses and networking events 18 283 51 117535 34 Inclusive of: 1) course participants permanently based outside of Rangoon; and 2) course participants traveling to Rangoon for the explicit purpose of attendance who received program-funded student stipends. 35 This number is high because it is made up of 458 participants in the trainings and 717 participants in the networking events. 66 IR 1.1 Teacher Training, Distance Education, and Training-of-Trainers Training delivered. 1. # of teaching training course participants 2. # of distance education courses completed 3. # of training of trainers course participants 169 30 84 381 32 69 IR 1.2 Content trainings delivered by course participants through practice training and internships 1. # of practice training hours delivered by TOT and TT course participants 2. # participants in practice training sessions 3. # of project assisted training internships 95 200 5 115 216 1 IR 1.3 Distance Education Project designed and administered according to core SIT principles. 1. # of DE project graduates 8 8 IR 2 To create diverse networks of teachers, trainers, civil society leaders and supporters. 1. # of networking opportunities (not incl. training) provided 2. # of CSOs represented at networking opportunities n/a n/a 11 18 IR 2.1 A diverse mix of participants for all projects recruited. 1. % of ethnic minorities and members of disadvantaged groups enrolled in courses and attending networking events 2. % of women enrolled in courses and attending networking events n/a n/a n/a n/a IR 2.1a Training experiences through field work offered by project participants. 1. # of trainings delivered by TOT and TT course participants 2. # of CSOs receiving training through TOT field work 3. # of CSOs using USG assistance to improve internal organizational capacity 20 n/a n/a 50 n/a n/a *Data collection instruments for IR 2.1a 2 and 3 were under development and not in use yet. **Target Output for FY10 derived from work plan Color Code: 1. Achieved Project Targets 2. Overachieved Project Targets 3. Underachieved Project Targets 4. Not Applicable Performance in Fiscal Year 2009: Note that no targets were set for Fiscal Year 2009 Output Indicators FY 2009 Results Number of students receiving USG-funded scholarships to attend higher education institutions (standard indicator) 13 (2 women) Number of courses completed 18 courses Students performance per course GPA 3.50 – 4.00: 2 persons (0 women) GPA 3.00 – 3.49: 12 persons (2 woman) GPA 2.50 – 2.99: 9 persons (4 women) GPA 2.00 – 2.49: xx persons (x women) Number of teachers enrolled in QuiLT program 0 Number of participants completing QuiLT successfully 0 Number of teachers participating in other teacher training activities 312 Number of participants enrolled in training of trainers course 14 Number of participants completing training of trainers course 11 Number of civil society organizations affected 30 Number of teachers from Burmese ethnic minority groups 32 Number of geographical regions impacted by the trainings 8 67 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Simon Richards Title Team Leader Organization Consultant contracted to IBTCI Evaluation Position? √ Team Leader Evaluation Award Number (contract or other instrument) AID-RAN-I-00-09-00016; Order No. AID-486-TO-12-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Burma Outreach and Distance Education Project Implemented by World Learning Award Number: Cooperative Agreement 486-A-00-07-00007-00 I have real or potential conflicts of interest to disclose. √ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 13 December 2012 USAID/Asia Regional Development Mission Asia Athenee Tower, 25th Floor 63 Wireless Road, Lumpini, Patumwan Bangkok 10330 Thailand U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523