OTI/ANGOLA Support to Angola’s Democratic Transition (SADT) Program Final Evaluation Report Office of Transition Initiatives (OTI) Bureau for Democracy, Conflict and Humanitarian Assistance U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT Submitted by: Tristi Nichols, PhD. SOCIAL IMPACT, INC Contract No. HDA-I-13-03-00124-00 Social Impact, Inc. OTI/Angola Evaluation Report Page i of 76 TABLE OF CONTENTS Acronyms and Glossary of Terms in Angolan Portuguese............................................................................ii List of Tables and Figures ..................................................................................................................................iii I. EXECUTIVE SUMMARY ...............................................................................................................................1 II. INTRODUCTION...............................................................................................................................................6 OFFICE OF TRANSITION INITIATIVES........................................................................................................................ 6 COUNTRY CONTEXT ................................................................................................................................................... 6 INTERVENTIONS IN ANGOLA...................................................................................................................................... 8 III. EVALUATION OBJECTIVES AND METHODOLOGY.......................................................................9 PURPOSE AND OBJECTIVES........................................................................................................................................ 9 METHODOLOGY......................................................................................................................................................... 10 LIMITATIONS.............................................................................................................................................................. 11 IV. PROGRAM CONTEXT AND IMPLEMENTATION............................................................................12 PROGRAM GOALS, OBJECTIVES, AND STRATEGY ................................................................................................ 12 Strengthening Media........................................................................................................................................... 13 Civil Society Advocacy...................................................................................................................................... 13 Community Engagement and Problem Solving............................................................................................. 14 PROGRAM IMPLEMENTATION .................................................................................................................................. 14 Program Structure ............................................................................................................................................... 14 Grant Making....................................................................................................................................................... 14 Merging of OTI and DG Management Roles................................................................................................. 16 Staff Development .............................................................................................................................................. 17 Other Factors........................................................................................................................................................ 17 V. PROGRAM OUTCOMES ..............................................................................................................................18 INCREASING CAPACITY TO ADVOCATE ON KEY ISSUES...................................................................................... 18 INCREASING ENGAGEMENT TO ADDRESS COMMUNITY PROBLEMS.................................................................. 21 RAISING AWARENESS AND CHANGING ATTITUDES............................................................................................. 23 NEW ACTIONS............................................................................................................................................................ 24 VII. LESSONS LEARNED AND RECOMMENDATIONS ..........................................................................25 INCREASING CSO/NGO CAPACITY FOR ADVOCACY AND ENGAGEMENT ....................................................... 26 PROGRAM STRATEGY AND IMPLEMENTATION...................................................................................................... 27 MERGING OF OTI AND DG MANAGEMENT ROLES.............................................................................................. 29 APPENDICES ..............................................................................................................................................................30 APPENDIX 1 – SCOPE OF WORK.............................................................................................................................. 30 APPENDIX 2 – EVALUATION FRAMEWORK........................................................................................................... 35 APPENDIX 3 – LIST OF INTERVIEWEES................................................................................................................... 42 APPENDIX 4 – LIST OF GRANTEES AND PROJECTS SAMPLED............................................................................. 43 APPENDIX 5 – REFERENCES AND WORKS CITED................................................................................................. 46 APPENDIX 6 – SURVEY DEMOGRAPHICS & RESULTS FROM ATTITUDINAL STATEMENTS............................. 48 APPENDIX 7 – QUESTION GUIDE 1: CAII/OTI STAKEHOLDERS BASED INWASHINGTON, DC.................... 52 APPENDIX 8 – QUESTION GUIDE 2: CREA/OTI STAKEHOLDERS BASED IN ANGOLA................................... 53 APPENDIX 9 – QUESTION GUIDE 3: INTERVIEW GUIDES FOR GRANTEES IN ANGOLA................................... 56 APPENDIX 10 – CAPACITY-BUILDING SURVEY.................................................................................................... 58 APPENDIX 11 – SAMPLING....................................................................................................................................... 66 APPENDIX 12 – EVALUATION AND CORRESPONDING DATA SOURCES/METHODOLOGIES............................ 67 APPENDIX 13 – INSTRUMENTS AND INTERVIEWING PROTOCOL........................................................................ 69 APPENDIX 14 – EVALUATION LIMITATIONS......................................................................................................... 71 Social Impact, Inc. OTI/Angola Evaluation Report Page ii of 76 Acronyms and Glossary of Terms in Angolan Portuguese Acronyms CAII Creative Associates International, Inc. CREA Creative Associates International, Inc. (name in Angola) CSO Civil Society Organization CTO Cognizant Technical Officer DCHA Democracy, Conflict, and Humanitarian Assistance DG Democracy and Governance FSN Foreign Service National FY Fiscal Year IDP Internally Displaced Person JEA Joventude Ecológica de Angola MOU Memorandum of Understanding MPLA Movement for the Liberation of the People of Angola NGO Non-Governmental Organization OTI Office of Transition Initiatives PDO Program Development Officer PTG Political Transition Grant SADT Support to Angola’s Democratic Transition SWIFT Support which Implements Fast Transition Initiatives USAID US Agency for International Development USG United States Government Terms in Angolan Portuguese Cobrança Bribe Payment in Angolan Portuguese Fazenda Large plantation Kwanza Angolan currency (as of mid-September 2004, 86 kwanza=US$1.00) Soba Traditional leader Social Impact, Inc. OTI/Angola Evaluation Report Page iii of 76 List of Tables and Figures Page 1. Map of Angola 7 2. Demographic Information on Angola 7 3. Number of Grants and Funds Obligated, By Program Objective 16 4. Thematic Areas in which NGOs Advocated for Change 19 5. Thematic Areas in which NGOs Plan to Advocate for Change 19 6. Advocacy Methods, in Rank Order 19 7. Target Audiences, in Rank Order 19 Social Impact, Inc. OTI/Angola Evaluation Report Page 1 of 76 I. EXECUTIVE SUMMARY Background The Office of Transition Initiatives (OTI) has operated two major programs – or interventions – in Angola to help the country make a successful transition to a peaceful and democratic society. OTI was created within the US Agency for International Development (USAID) Bureau for Humanitarian Response to respond to situations where rapidly deployed aid can make a critical difference to a country’s transition from crisis to recovery and stability. Angola is one of the poorest countries in the world. Nearly four decades of war, conflict and economic mismanagement have exacted a heavy toll on the country, ravaging political and social institutions, and internally displacing over four million people. In response to the opening brought about by the unexpected peace following Jonas Savimbi’s death, USAID undertook a review of its Angola program in 2002. Though the window of opportunity for an OTI intervention was narrow, it was clear that there was a compelling need to support Angola during the transition from war to peace. The Support to Angola’s Democratic Transition (SADT) Program was designed to support a wide range of activities to strengthen participatory democratic practices, through a small grants mechanism. This program was administered by OTI with Creative Associates International, Inc. (CAII – known as CREA in Angola) as the implementing partner. The OTI/Angola program’s stated goal was to help create a foundation for the transition to an open and participatory democracy. This goal was the measure for evaluating the program’s impact in Angola. Evaluation Objectives and Methodology Two specialists (an American and an Angolan woman) designed a mixed-method approach to evaluate specific aspects of the program. Three principal questions guided this evaluation: 1. To what extent did OTI/Angola’s program meet its stated goal and objectives? 2. How did the management and operation of the program contribute to or detract from achievement of the program goal and objectives? 3. Based on the evaluation findings, what are the lessons learned and ways OTI can improve its programs? The focus of the first question was on two of the OTI/Angola Strategic Plan objectives: · strengthening the capacity of Civil Society Organizations (CSOs) / Non-Governmental Organizations (NGOs) to advocate on key issues; and · increasing engagement between citizens and local authorities to address community problems. Social Impact, Inc. OTI/Angola Evaluation Report Page 2 of 76 Because management issues were examined in a separate external review, the second question for this evaluation focused on one particular aspect of program management: the Country Representative’s dual role as the OTI Representative and DG Team Leader. Program Context and Implementation The SADT program was administered in Angola by the OTI Country Representative, who in a newly merged management approach was also the DG Team Leader. The grant making process was implemented by the CREA Program Development Officers (PDOs), under the leadership of the Country Representative (although they reported to the CREA Chief of Party for administrative purposes). OTI recruited a broad spectrum of potential grantees, and the PDOs worked closely to assist grantees throughout the life of the grant. An “in-kind” small grants procurement system (in which grantees received necessary inputs supplied directly through CREA staff, rather than cash) was designed to enable even groups with limited administrative capacity to receive support through OTI to conduct their work. Fifty-two grants were made, for a total of $ 1.9 million, with approximately 60 percent going to projects under the Advocacy objective, and 40 percent toward the Engagement objective. Evaluators found that the merged management role for the OTI Country Representative did seem to facilitate better coordination between DG and OTI. However, this was offset by the administrative demands of the DG Team Leader role, and by insufficient information-sharing of OTI strategy and activities with the DG staff. Constraints related to the program’s implementation included: the low skill level of the PDOs, high operating costs, and communication difficulties which at times hampered the in-kind procurement mechanism (logistical constraints related to procurement were eased in one area by the contractor’s working with a local NGO to facilitate procurement for other sub-grantees). Program Outcomes Evaluators found that OTI/Angola’s goal of helping build a foundation for democratic transition was achieved to a large extent. The data clearly indicates that the program opened or initiated participatory and democratic processes where it was active. The evaluation data, both quantitative and qualitative, suggest that the program objectives of strengthening CSO/NGO capacity to advocate on key issues, and to increase citizens’ and local authorities’ engagement to address community problems, were met to a large extent. There were many specific examples of positive changes in awareness, attitudes, and capacity for advocacy and democratic action by grantees and beneficiaries. These results were all the more impressive for having been achieved in an operating context fraught with many challenges. Lessons Learned and Recommendations This summary of lessons learned from the OTI/Angola experience and related recommendations is organized under three categories: Social Impact, Inc. OTI/Angola Evaluation Report Page 3 of 76 · Increasing CSO/NGO Capacity for Advocacy and Engagement · Program Strategy and Implementation · Merging of OTI and DG Management Roles. The evaluators hope that these lessons and recommendations will prove useful to other USAID/OTI programs with similar goals and challenges. Increasing CSO/NGO Capacity for Advocacy and Engagement Lesson · Success and positive reinforcement strengthened grantees’ confidence and credibility, helping them attract and inspire more participation. Recommendations · OTI should build in more systematic opportunities and modes for providing positive reinforcement to grantees for their efforts at every stage. · OTI should do more to help its grantees gain wider visibility and recognition for successes. Lessons · Efforts to increase engagement in community problem-solving met with varying degrees of local government responsiveness – and CSOs/NGOs can be proactive about improving their engagement strategies. · Responsiveness to engagement was higher in areas more affected by war. Recommendations · OTI should include assessment of local government responsiveness in the grant making process, to capitalize on opportunities for engagement and apply funds where they will have the greatest impact. · USAID/OTI should gather best practices, establish guidelines, and develop training on successful strategies for engaging local government (along with citizens) in addressing community problems. · Opportunities should be created for sharing of successful engagement strategies among CSOs/NGOs, and of successful engagement experiences among local authorities, across regions – in other OTI programs worldwide. Lesson · Raising awareness, changing attitudes, encouraging advocacy, and new forms of action are overlapping and mutually reinforcing activities. Social Impact, Inc. OTI/Angola Evaluation Report Page 4 of 76 Recommendation · OTI should develop criteria in order to better identify and support CSOs/NGOs that are working in ways that take advantage of synergies between awareness, attitudes, advocacy, and action. · Such synergies should also be encouraged by OTI through grant making that supports CSOs/NGOs to align their work on separate but interrelated elements of democracy building. Lesson · Grantees’ capacity for advocacy was most strengthened by program participation when they had either limited or extensive advocacy experience in the past. There seemed to be two groups of activists motivated to gain capacity with support, one of committed advocacy veterans and another of passionate newcomers. Recommendations · OTI should figure out what it takes to more consistently recruit and select grantees from these two high-impact groups. · At the same time, OTI needs to understand better how to motivate and develop the capacity of those grantees in the mid-range of experience. Program Strategy and Implementation Lessons · The speed and flexibility of OTI’s small-grants program approach helped OTI respond rapidly to emerging opportunities in Angola. · An inclusive, flexible grant making approach allowed for adapting program activities across a wide range of conditions and cultures. · OTI’s participatory, grassroots strategy resulted in initiatives that were grounded in grantees’ knowledge of – and accountability to – their communities. · Difficulties with fulfilling USG procurement process standards were eased by partnering with an NGO familiar with those procedures. · The in-kind distribution system was perceived differently by grantees with differing degrees of financial/administrative capacity –– and those perceptions affected their program outcomes. Recommendations · OTI should apply this flexible small-grants model in countries with similar characteristics, but should mitigate its significant risks by: - providing continuity of personnel throughout the life of the program in designated countries. - developing clearer standard operating procedures for the Implementing Partner, and ensuring that these operating procedures are understood by all. Social Impact, Inc. OTI/Angola Evaluation Report Page 5 of 76 - requiring informal cost analysis as part of the program assessment process. · The Implementing Partner/Contractor should proactively seek out innovative approaches to meeting procurement challenges. This could include seeking out potential partners (NGOs) with a field presence to assist in the delivery of inputs where procurement along USG standards presents unreasonable challenges. · Rather than implementing a uniform, in-kind distribution system, OTI should conduct a rapid administrative and financial assessment of grantees. If a uniform in-kind distribution system must be utilized, the Implementing Partner should take the necessary time to explain the rationale to CSO/NGOs with strong financial and administrative capacity. Merging of OTI and DG Management Roles Lesson · Merging dual management responsibilities into one role is a workable approach to aligning OTI and DG programs, if constraints are addressed. Recommendations · Streamline the administrative load on the DG Program at the field level. · Expand the scope of permissible responsibilities for competent FSNs. · Conduct regular (monthly or more often) information meetings to share vital program information. · To streamline effectively, USAID offices need to look at field level as well as Headquarters in order to ensure lateral communication and coordination between DG and OTI management, and to reduce duplication of effort in program administration. Social Impact, Inc. OTI/Angola Evaluation Report Page 6 of 76 II. INTRODUCTION Office of Transition Initiatives The Office of Transition Initiatives (OTI) has operated two programs – or interventions – in Angola to help the country make a successful transition to a peaceful and democratic society. The US Agency for International Development (USAID) Administrator created OTI in the Bureau for Humanitarian Response (now the Bureau for Democracy, Conflict and Humanitarian Assistance) to assist priority countries to make successful transitions from crisis to recovery and stability. The volatile political and economic nature of transitioning countries requires fast, emergency-type political responses that show immediate, visible and positive effect. Countries experiencing complex crises resulting from internal conflict and civil war have special needs that are often not addressed by traditional emergency assistance programs. OTI enables USAID to capitalize on ‘windows of opportunity’ where quickly deployed aid can make a critical difference to a country’s transition to peaceful, democratic government. Interventions are tied to pivotal events, such as cease-fires, peace accords, or the advent of progressive leadership, often through key elections. OTI responds swiftly to these events with near-term, high-impact actions that support a country’s transitional needs. While operating in a country, OTI works to bring new groups into the transition process, tests new activities for advancing democratic governance, and provides fast and flexible support for immediate transition needs. OTI’s program options for transition responses include: 1) expanding democratic political processes; 2) building citizen security; 3) promoting reconciliation; 4) support peace negotiations; and 5) cross-cutting themes, including community-based approaches and media activities. As appropriate and necessary, relationships and practices that prove productive may be handed off to the USAID mission or other donors for further development when OTI phases out its assistance. Country Context The Republic of Angola is located on the South Atlantic Coast between Namibia and the Republic of the Congo, bordered by the Democratic Republic of Congo and Zambia to the east. Angola’s population of roughly ten million1 occupies an area slightly less than twice the size of the state of Texas (see inset). The capital, Luanda, is situated on the north coast. 1 Angola’s population is estimated to range between ten and eleven million though such estimates are based on scantily collected birth and death rate data. Social Impact, Inc. OTI/Angola Evaluation Report Page 7 of 76 The country has a variety of climates, including an arid coastal strip stretching from Namibia to Luanda; a dry savanna in the interior south and southeast; a wet, interior highland; and rain forest in the north. The northern part of Angola experiences a rainy season from September to April; the southern part from November to about February; and typically, it is warm and wet in the Cabinda Province. The Zambezi River and several tributaries of the Congo River have their sources in Angola. Ethnically, the Republic comprises three main groups, each speaking a Bantu language: Ovimbundu (37%), Kimbundu (25%), and Bakongo (13%). Other groups include Chokwe, Lunda, Ganguela, Nhaneca-Humbe, Ambo, Herero, Xindunga, mixed racial (European and African), and a small population of whites – primarily ethnically Portuguese. Portuguese is both the official and predominant language, but vernaculars are spoken throughout the provinces. Principal religions include: Roman Catholic (68%); Protestants sects (20%); and indigenous beliefs (12%). Angola is one of the poorest countries in the world. Nearly four decades of war, conflict and economic mismanagement have exacted a heavy toll on the country. Despite abundant natural resources (particularly oil and diamonds) and considerable potential for growth, output per capita remains among the world's lowest. The majority of Angola’s population lives in poverty: 68 percent of urban dwellers live below the poverty line2 and the rural economy produces at subsistence level. 3 As Angola struggled for independence from Portugal – 2 Human Development Report 3 http://www.unicef.org/about/execboard/files/04-PL1_Angola.pdf Republic of Angola Demographic Information Size: 1,246,700 sq km or slightly less than twice the size of the state of Texas Population*: 10,978,552 (July 2004 estimate) Age Structure: 0-14 years: 43.5% 15-64 years: 53.7% 65 years and over: 2.8% (2004 estimates) Median Age: 18.1 years Annual Growt h Rate:1.93% Life Expectancy: 36.79 to 40 years Total Fertility Rate: 6.33 children born/woman (2004 estimate) Literacy Rates: Defined as age 15 and over can read and write Total population: 42% Male: 56% Female: 28% (1998 estimate) Source: www.cia.gov and UNICEF * Not a reliable figure Social Impact, Inc. OTI/Angola Evaluation Report Page 8 of 76 achieved in 1975 – it found itself immersed in a violent 27-year civil war fought between two opposing groups: the Movement for the Liberation of the People of Angola (MPLA) and the National Union for the Total Independence of Angola (UNITA). On two occasions during the civil war, negotiations between MPLA and UNITA resulted in peace agreements, first in 1991 (the Bicesse Accord) and then again in 1994 (the Lusaka Protocol). Neither agreement ushered in a lasting peace. The war finally ended in 2002 shortly after the death of Jonas Savimbi, which provided the impetus for the negotiation of a third agreement (the Luena Accord). The long civil war ravaged the country’s political and social institutions, internally displacing over four million people. Approximately, three-fourths of the internally displaced persons (IDPs) have now returned to their areas of origin. Interventions in Angola First Intervention: 1994 - 1998 From 1994-1998, OTI managed a $28 million program in Angola to support the Lusaka peace process. OTI’s interventions targeted communities hardest hit by the war and included funding for independent media, mine awareness campaigns, and community￾based reintegration programs, principally focused on demobilized ex-combatants. Creative Associates International, Inc. (CAII, known as CREA in Angola) was OTI’s implementing partner during this first intervention. OTI Assessment and the Second Intervention: February 2003 - September 2004 In response to the opening brought about by the peace following Jonas Savimbi’s death, USAID undertook a review of its Angola program in 2002. The situation in Angola passed the OTI Criteria for Engagement – a process guiding OTI’s decision to intervene. 4 Though the window of opportunity for an OTI intervention was narrow, it was felt that there was a compelling need to support Angola during the transition from war to peace. A multi-sector USAID team assessed specific options for assisting Angola in the post￾conflict period. Taking into consideration that some needs were already being addressed through humanitarian programs (e.g., reintegration of ex-combatants and assistance to IDPs), the team concluded that OTI’s program would be more effectively targeted to promoting civil society and democratic processes, key areas to strengthen in anticipation of the potential challenges and changes in Angola. The Support to Angola’s Democratic Transition (SADT) Program was conceived as a collaborative partnership between OTI and Democracy and Governance (DG) to support a wide range of transition activities, employing a small grants mechanism. 5 4 Affirmative answers to four questions help ensure that a transition is indeed in progress and that the necessary conditions are in place to give an OTI country program a good chance to succeed. 5 USAID, OTI- Transition and Development Assessment, Washington DC October, 2002, pg. 28. Social Impact, Inc. OTI/Angola Evaluation Report Page 9 of 76 III. EVALUATION OBJECTIVES AND METHODOLOGY Purpose and Objectives The purpose of this evaluation was to provide USAID/OTI with an assessment of the impact of its SADT program and to capture lessons learned from the program. The Evaluation Scope of Work (see Appendix 1) identified three fundamental questions to guide the direction of the evaluation: 1. To what extent did SADT meet its stated goal and objectives? 2. How did the management and operation of the program contribute to or detract from achievement of the program goal and objectives? 3. Based on the evaluation findings, what were the key lessons learned and ways OTI can improve its programs? Under the first question, the most important objective of this evaluation was to review the impact of specific grant activities, focusing on two of the OTI/Angola Strategic Plan objectives: · strengthening the capacity of Civil Society Organizations (CSOs) / Non-Governmental Organizations (NGOs) to advocate on key issues; and · increasing engagement between citizens and local authorities to address community problems. Because management issues were examined in a separate external review, the second question for this evaluation focused on one particular aspect of program management: the Country Representative’s dual role as the OTI Representative and DG Team Leader. Many USAID/Washington stakeholders were specifically interested in how this experiment in integrating OTI’s and DG’s program to achieve “complementarity” influenced OTI’s program strategy, and whether the dual management role could be handled effectively by one person. Meeting the objectives of this evaluation called for gathering information from a number of sources, employing multiple review methods, to understand: · the purpose and implementation of each program component, · specific outcomes achieved, and · obstacles or constraints to meeting program goals. Based on this research, the lessons learned and resulting recommendations may prove useful to other OTI programs elsewhere. Social Impact, Inc. OTI/Angola Evaluation Report Page 10 of 76 Methodology The evaluation team (both women) included an expatriate evaluation specialist with USAID evaluation experience and an Angolan evaluator familiar with social science methods and sensitive to research modalities in Angola. The scope of the evaluation work called for a mixed-method approach combining qualitative and quantitative research. Qualitative data drew from organizational documents, the database, and interviews/focus groups with OTI and CAII (in Angola and Washington, DC) and program participants/grantees. Quantitative methods were also employed to understand how and what types of inputs and resources offered through the program influenced grantees’ capacity to advocate. The Evaluation Framework consisted of seven tasks (see Appendix 2): 1. Thorough review of OTI documents related to the OTI/Angola program. 2. Sampling frame of all stakeholders in Washington, DC and in the field. 3. Question guides and surveys for each stakeholder group consistent with the three broad evaluation questions. 4. Site visits in Angola where OTI-supported activities were being undertaken. 5. Processing and analysis of data collected on SADT program operations. 6. Final draft evaluation to the Evaluation Cognizant Technical Officer (CTO). 7. Final report for USAID/OTI responding to review comments provided by the OTI CTO. Task 1: Document Review OTI provided all documents pertaining to the OTI/Angola program. Prior to departing for Angola, the evaluation team reviewed these documents and conducted interviews with people who were directly involved in establishing the OTI/Angola SADT program. They also consulted the OTI database and documents provided by the field office in Angola. Task 2: Sampling Frame The qualitative sample included (1) OTI/Washington, USAID/Washington staff (CTOs, Africa Bureau and those involved in the OTI assessment); (2) Creative Associates International Inc. (CAII) in Washington (program assistance and support, first Chief of Party, and Vice-President of Program); (3) OTI/Angola staff (Country Representative/DG Team Leader and others involved in program implementation); (4) CAII staff in Angola (Logistics Officer, Program Development Officer, and second Chief of Party); and OTI grantees. (See Appendices 3 and 4 for more details.) Task 3: Question Guides and Surveys Stakeholders in Washington, DC were asked about their roles and activities in the OTI program, and their thoughts about difficulties, influencing factors, program impact, and type of evaluation information that would be most informative. In Angola, however, respondents were asked more detailed questions about the overall program approach and Social Impact, Inc. OTI/Angola Evaluation Report Page 11 of 76 implementation process. Grantees were asked detailed questions about the grant application, implementation, and service delivery. (See Appendices 7, 8, 9, and 10 for more details.) Task 4: Site Visits Over three weeks, evaluation team members carried out focus groups, document reviews, a survey and interviews with OTI and CAII field office personnel and OTI grantees involved in community development and democracy building programs. During the site visits, arrangements were made for carrying out a field survey of SADT participants. Task 5: Processing and Analysis of Data Both qualitative and quantitative methods were used in the evaluation, however, the impact analysis is primarily qualitative. OTI/Washington stakeholders specifically requested a review of case examples, which were used to illustrate grantees’ impact on democracy building. In addition, text analysis was conducted for all qualitative data collected, and “working hypotheses” were drawn from recurrent themes. The evaluation team ran descriptive statistics on all data. Where possible, triangulation of data sources and analysis was used to increase the validity of data interpretation. Task 6-7: Reporting Preparation of draft final report and final report to OTI/Washington. Data Sampling On the qualitative side, of 50 persons sampled, the majority (64 percent) were grantees. Approximately 16 percent of the stakeholders came from Washington DC (OTI and CAII), while 10 percent were based in Angola (the CREA staff combined and OTI Country Representative). Ten percent, categorized under ‘other’, represented individuals from the Africa Bureau or DG in Angola. (See Appendices 3, 4, and 11 for more detail.) The quantitative sample was smaller, with a sample size of only 35 people, comprised of grantees representing all program regions. The sample could be characterized as relatively young concerned citizens, with some level of formal education. The majority (about 63 percent) of individuals surveyed came from NGOs. (See Appendices 4, 6, and 11 for more detail.) Schedule The activities of this evaluation were carried out over a three-month period from September through December 2004, with three weeks spent in the field. Limitations Several key limitations may restrict the degree to which inference, causation, and correlations can be drawn between inputs and outcomes in the Final Evaluation Report of OTI/Angola’s SADT program. First, CREA staff was not available for in-depth interviews, limiting the evaluation team’s ability to fully understand the program implementation process. Second, survey instruments were not sufficiently pre-tested Social Impact, Inc. OTI/Angola Evaluation Report Page 12 of 76 despite the evaluation team’s observations as to which questions presented difficulties for respondents. Third, measurement of CSO/NGO “capacity to advocate” is an experimental approach. Fourth, the margin of error for attitudinal questions was high. Fifth, the database used to record and document the amounts issued to grantees may not be considered an actual reflection of grants disbursed. Examination of management issues in this evaluation was limited to the Country Representative’s dual role in managing OTI and DG, because a separate external review of program management was conducted. (See Appendix 14 for further details on evaluation limitations.) IV. PROGRAM CONTEXT AND IMPLEMENTATION Program Goals, Objectives, and Strategy The USAID/OTI program in Angola sought to build on the increased stability brought by the transition to peace. Recognizing that “achieving a fully democratic society in Angola [would] be a slow process taking place over many years,” OTI’s Country Strategy was directed at “helping to build a foundation to begin a real transition to an open and participatory democracy.” 6 This short-term, realistic goal was the measure for analyzing the program’s impact in Angola. To meet it, OTI employed a flexible, small-grants program, supporting activities that spread and strengthened participatory democratic practices and promoted greater political competition, accountability, and transparency. Field observations indicate that OTI cast a wide net when soliciting proposals from CSOs/NGOs and was receptive to feedback from those working on USAID’s DG programs. Based on the reviewed projects, OTI’s strategic approach is best characterized as participatory – responsive to local needs, and oriented toward capacity building. CSOs/NGOs were encouraged to set the issue agenda for advocacy efforts. This flexible and inclusive strategy was particularly appropriate in Angola, where there are important cultural variations from province to province. USAID/OTI’s initial objectives were to: 1. Strengthen media and access to information; 2. Increase/improve CSOs capacity to advocate on key issues; and 3. Increase citizens’ and local authorities’ engagement in addressing community problems. Each objective is discussed further below. 6 OTI, Angola Program Strategy, August 2003, pp. 8 (pg. 25) Social Impact, Inc. OTI/Angola Evaluation Report Page 13 of 76 Strengthening Media It was anticipated that media in Angola would be strengthened by supporting efforts toward improved media quality, professionalism, a stable and favorable legal environment for journalists, and increasing the number of media outlets. OTI detailed each component 7 and listed activities that it was likely to support (i.e., training; community debates; distribution of newspapers or cassettes; and support to journalists for legal fees, training, and travel). Following a midterm program review in February 2004, however, there was a strategic determination regarding the media strengthening objective. Given limited opportunities and resources available through OTI for effective media grant making, a decision was made to consolidate the program’s objectives and concentrate funds for greater impact. Several factors contributed to this decision: 1) a number of media activities were adopted under the program’s advocacy and CSO/NGO cooperation components; 2) only a handful of independent media outlets had been established in Angola (OTI worked with most of them in some form); and 3) government control over the media was a major obstacle to creating new outlets. Given these factors, discussion of program implementation and impact in this report focuses on the second and third objectives: “Civil Society Advocacy” and “Community Engagement and Problem Solving.” Civil Society Advocacy Advocacy – defined as “fight[ing] for or stand[ing] for a belief or perspective” - is considered a key component of democracy. OTI’s program aimed to strengthen CSO/NGO capacity to advocate, 8 defined as: 1. Articulating and demanding change; 2. Organizing and mobilizing citizens in action that brings greater attention to key issues by those who have the power to institute change (i.e., policy makers); and 3. Educating citizens and elected officials on key issues, while applying effective, transparent and nonviolent pressure to generate the political will needed to enact desired reforms. OTI used a participatory approach in its advocacy programming: with the local CSOs/NGOs determining specific issues to advocate. Tables 4 and 6 in the Program Outcomes section of this report provide a list of the issues and advocacy methods observed by the evaluation team. OTI grants financed activities that helped CSOs/NGOs gain and learn from experience, network among organizations, conduct information campaigns on key issues, and other activities encouraging broader citizen participation. 7 OTI, Grant Worthiness Guide, May 2003. 8 OTI/Angola Program Strategy, August 2003. Social Impact, Inc. OTI/Angola Evaluation Report Page 14 of 76 Community Engagement and Problem Solving OTI’s program theory emphasizes the importance of “engagement” characterized as a “process that includes identifying community needs in a participatory and inclusive fashion, negotiating and compromising to determine priorities, collaborating to acquire financial, material and human resources, and implement[ing] projects in a transparent and participatory fashion.”9 To satisfy this objective, OTI sought to support activities that included small community infrastructure and active involvement of local authorities. With limited resources, OTI’s strategy was to demonstrate how Angolans could start constructively engaging their government to rebuild and plan, rather than trying to pursue major reconstruction. Only 12 percent (2 out of 17) of the projects reviewed in the evaluation related directly to infrastructure. Most activities observed within this objective focused on giving CSOs/NGOs assistance to promote community participation and engage local authorities (at communal and municipal levels) around other topics. Program Implementation This section of the report describes how OTI was organized to meet its objectives: the program structure and grant making process, including the “in-kind” grants procurement mechanism. The merging of OTI and DG management roles is also discussed, and staff development efforts are described, along with other factors in the operating environment influencing OTI’s ability to fulfill program objectives. Program Structure OTI/Angola staff consisted solely of the Country Representative. DG was staffed by two Foreign Service Nationals (FSNs), who were overseen by the OTI Country Representative in the dual role of DG Team Leader (see “Merging of OTI and DG Management Roles” below). The implementing partner for OTI/Angola was Creative Associates International, Inc., which in Angola was called CREA (the name is an invented word, not an acronym). CREA was led by a Chief of Party (COP), a role held by four different people during the program period. Three CREA Program Development Officers (PDOs) (initially two, then one was added) reported to the COP for administrative purposes, but to the OTI/Angola Country Representative on programmatic responsibilities. CREA staff also included two Logistics Officers (initially one, then one was added), who reported solely to the COP. Grant Making OTI advertised program opportunities directly (through flyers or meetings) and indirectly (through referrals from DG/FSNs or word of mouth). The implementing partner received resulting proposals and worked with grantees to nurture their idea(s). Interviews indicate that this process took from three weeks to five months. In two cases, the grantee 9 Ibid., pg. 11. Social Impact, Inc. OTI/Angola Evaluation Report Page 15 of 76 mentioned that a proposal was sent back two times for revisions and further clarifications. 10 From this point, the OTI country representative and the CREA Program Development Officers (PDOs) jointly decided if the project was a ‘go’ through various stages of the grant making process. OTI’s grant selection process weighed a number of factors in addition to considering the relevancy of proposed activities. Potential grantees had to meet the following conditions: 4. Relevant organizational experience; 5. Ability to hire appropriate and experienced personnel who met proposed project requirements; 6. Demonstrate understanding ofthe responsibilities and regulations associated with managing the grant; and 7. Present a sound and realistic budget and timeframe. Questions designed to elicit such information from potential grantees were supplied to PDOs. Once a proposal received a ‘go,’ the PDO developed and prepared the Political Transition Grant (PTG) with the grantee, including the project description, goals and objectives, evaluation methods, and the grantee’s and CREA’s responsibilities. The Memorandum of Understanding (MOU) – a contract between CREA and the grantee – included a detailed budget that had been reviewed and revised by the Procurement Officer (PO). On final approval, CREA provided either in-kind service delivery or first installment of a cash advance (depending upon the nature of the agreement). The in-kind distribution system was designed to assist grantees with limited administrative capacity to focus on programming their work, rather than having to deal with the administration and financial aspects of executing a project. The PDOs negotiated what kind of inputs a grantee required to complete his/her project. This list of items and/or services was then given to the Logistics Officer, who was in turn responsible for purchasing the actual items for the grantee and delivering them to the grantee on site and on time. For example, for a workshop project, the grantee would need to pay for venue rental, lunch for participants, etc. In this case, CREA's Logistics Officer would come to the site before the workshop, pay for everything, then leave. The PDO would attend the workshop to see to it that the programmatic objectives of the event were met. 10 Note: reported with caveat that many grantees had notable difficulty recalling the time of proposal submission through proposal acceptance. Social Impact, Inc. OTI/Angola Evaluation Report Page 16 of 76 CREA had the option to issue cash advances to grantees rather than using the in-kind system, and in the second year of program implementation, CREA increasingly used the cash advance system. However, most support was through the in-kind system. Of the grantees interviewed, 90 percent received in-kind service delivery. PDOs were responsible for program-related assistance: approving materials used for advocacy purposes; providing technical assistance; establishing links with local government administrators; attending meetings and major activities outlined in the proposal; coordinating procurement activities; and conducting monthly site visits. In Angola, each PDO was responsible for two or three regions and a portfolio of approximately 10-15 grantees. 11 At the close of each grant, PDOs conducted an evaluation, reviewed the final report with the grantee, and assessed any other expected deliverables. In addition, OTI’s Country Representative made a monthly field trip to verify program activities and meet with other potential grantees who might be eligible for program support. Table 3 below presents the number of grants and corresponding funds obligated according to program objective. The proportion of grants and obligated funds was approximately 60/40 for the Advocacy and Engagement objectives respectively. Table 3: Number of Grants and Funds Obligated, By Program Objective Program Objective Grants Approved (#) Funds Obligated as of August 2004 Civil Society Advocacy 32 $1,138,461 Engagement in Problem Solving 20 $784,788 Total: 52 $1,923,249 Source: Figures from the program database. Merging of OTI and DG Management Roles Qualitative data from the field indicates many areas of overlap between DG and OTI programming. Synchronization of these interventions was the primary rationale for assigning the dual role of both OTI Representative and DG Team Leader to one person, the Country Representative. The two program staffs (FSNs and PDOs) shared the following activities: 1. Joint field visits; 2. Strategic planning sessions; and 3. Providing contact information about national CSO/NGOs. 11 These figures are estimates, as staff interviews were limited and subsequent data quality may have been compromised. Social Impact, Inc. OTI/Angola Evaluation Report Page 17 of 76 Having both staffs report to a single manager on these programmatic responsibilities did seem to facilitate coordination and alignment between DG and OTI, and therefore to support achievement of program objectives. Evaluators found two key constraints to achieving a high degree of synchronization with this new approach. First, the DG Team Leader’s administrative tasks and other work requirements were quite demanding. This limited the time available for the programmatic aspects of the dual role. Second, although there was joint work between FSNs and PDOs, qualitative data indicates that there was insufficient information-sharing about OTI’s program strategy and activities to ensure consistent integration of program duties. As a result, picking up where OTI has left and building on top of that may be a challenge for DG staff. Further, while the FSNs working on the DG program were skilled, experienced, and clearly understood their role(s), it appeared that their contributions were limited as some of the more supervisory tasks were designated exclusively for expatriates. Staff Development Program staff received orientation about OTI and on-the-job training, including: One-on-one meetings Individualized training with the Country Representative and other experienced OTI staff who intermittently substituted for the Country Representative. Joint field visits The PDO and Country Representative (and sometimes a DG officer) traveled together to identify potential grantees and monitor ongoing activities. Conducted almost monthly. Workshops (in English and/or Portuguese, with an external facilitator) April 2003 Topics: the grant-making system, planning activities, roles and responsibilities, program strategy (for all stakeholders). September 2003 Topics: the grant cycle, roles and responsibilities. April 2004 Topics: evaluation, developing data collection tools. Despite multiple opportunities for program orientation, training, and individualized guidance, PDOs encountered difficulties fully meeting their oversight responsibilities. Other Factors Logistical constraints, shortage of highly-skilled workers, high operating costs, and communication difficulties contributed to delays in grant making and administration. However, while all grantees (with two exceptions), reported delays in receiving inputs and supplies from CREA, all but two from the entire sample were able to satisfy agreed contractual terms. Though dealing with the procurement processes required by CREA Social Impact, Inc. OTI/Angola Evaluation Report Page 18 of 76 was difficult for some CSOs/NGOs and led to delays, this difficulty was mitigated in Benguela by a larger and more experienced NGO that was able to act as a conduit to facilitate procurement for some grantees. V. PROGRAM OUTCOMES The OTI/Angola program’s stated goal was to help create a foundation for the transition to an open and participatory democracy. Evaluators found that this goal was achieved to a large extent. The data clearly indicates that the program opened or initiated participatory and democratic processes in all regions where it operated (except Luanda, where results were inconclusive due to small sample size).12 The program’s stated objectives (consolidated into two by discontinuing grants for media strengthening) were to strengthen CSO/NGO capacity to advocate on key issues, and to increase citizens’ and local authorities’ engagement to address community problems. The evaluation data, both quantitative and qualitative, suggest that these objectives were substantially met. Given the challenging operating context for OTI’s work in Angola, this achievement was notable. The following sections discuss the program outcomes under each objective. Particular attention is given to the impact of OTI-supported activities on grantees’ and beneficiaries’ awareness, attitudes, and actions. Increasing Capacity to Advocate on Key Issues The program aimed to enable CSOs/NGOs to advocate on issues they themselves identified as priorities. OTI’s flexible and participatory programming approach received high marks for accommodating different interests, contexts, and approaches. Issues of interest varied widely in the five regions surveyed. Overall, the most popular topic mentioned was land rights (23 percent), followed by women’s rights in the markets and elections (19.6 percent), and free (of cost) education and human rights (12 percent). Table 4 shows that issues related to land rights are comparably more prominent in Huila and Huambo. Women’s rights are very important to those in Benguela and Luanda, as is the right to a free (primary) education in Luanda. Similarly, the areas in which CSOs/NGOs planned to advocate in the future vary. Notably, of those surveyed, elections ranked highest as illustrated in Table 5 below. 12 Note that Luanda is excluded from this analysis because the sample of projects was too small to be able to state with confidence that participatory and democratic processes initiated can be attributed directly to OTI-supported projects. Social Impact, Inc. OTI/Angola Evaluation Report Page 19 of 76 Table 4: Thematic Areas in which CSOs/NGOs Advocated for Change Region Land Rights Free Education Women’s Rights Workers Rights Human Rights Elections Huila 23% 3% 3% 11% 14% 20% Benguela 9% 6% 26% 6% 11% 9% Huambo 14% 3% 3% 9% 3% 6% Luanda 11% 20% 20% 11% 0% 11% Benguela/Huambo 6% 0% 0% 0% 3% 6% Table 5: Thematic Areas in which NGOs Plan to Advocate for Change Thematic Areas Percent Elections 29.5% Women's rights in market place 18.9% Land rights 18.9% Free education 16.8% Worker's rights 11.6% Human rights 2.1% Civil education 2.1% To raise awareness about issues and educate citizens about their rights, CSOs/NGOs used advocacy methods ranging from workshops to lectures, radio and newspaper reports, forums, theater, and one rally. Tables 6 and 7 show the variety of advocacy methods used and audiences targeted. Women were the audience most targeted, at 29 percent. Tables 6 and 7: Advocacy Methods and Target Audiences, in Rank Order Advocacy Method Percent Formal Lectures 28% Workshop 21% Radio 17.1% Informal meetings 15% Posters/pamphlets 12.4% Forum 8.6% Target Audience Percent Women 29.3% Farmers 22.8% Rural youth 14% Urban youth 17% NGOs 5.4% Parents 3.3% Government workers 3% Journalists 2.2% Direct persuasion of public officials was another method of advocacy used to good effect by grantees, as in the example of the following reported conversation between a grantee and a communal administrator about the IDP’s need for land: Social Impact, Inc. OTI/Angola Evaluation Report Page 20 of 76 Activist: “If there is land that no one is tilling, why not give it to the IDPs?” Administrator: “This land does belong to someone, and they have already claimed it…” Activist: “If it is claimed, fine, but we still need to help these people (IDPs). They really need help, and this land is the answer …” Other grantees working on land rights issues also reported speaking out with increasing directness about the connection between land and poverty, a subject that used to be ‘taboo.’ Grantee reports also indicate that target audience(s) were not aware of their legal rights (e.g., to free primary education, to own land, to work in the market free of harassment), suggesting that the selected projects were both warranted and suitable to the program’s objectives. One clear example of positive impact was that some SADT grantees gained visibility and were recognized for their advocacy. For instance, the staff from Okitiuka – The Coalition had the opportunity to meet with the Vice-Minister of Education to voice concerns about the universal rights of children in Angola to a free primary education. This encounter prompted an official letter sent to all schools in Lobito and Benguela informing school administrators that the Coalition should be regarded as “partners in increasing the quality of education for all children…” This breakthrough inspired the grantee to continue in advocacy work. There are other similar examples, which suggests that there is a “virtuous circle” effect – grantees that received positive reinforcement for their advocacy efforts seem to have increased their capacity more than those that did not. This can lead to further recognition, and greater confidence and credibility. The visibility of such successes can inspire other CSOs/NGOs to more vigorous advocacy, as well. Thus, OTI-supported successes may have produced a multiplier effect that serves the overall program goal of building a foundation for a democratic transition. In another interesting finding, based on in-depth interview data, it appeared that CSO/NGO capacity was most strengthened when the grantee had either limited or extensive advocacy experience in the past. Those in the middle range, i.e., not new to advocacy work but also not highly knowledgeable, seemed to benefit less from SADT support in terms of increased capacity for advocacy. For instance, those involved in advocacy two years or less prior to funding found their OTI-supported projects highly rewarding, in spite of challenges encountered. And when asked about future plans, they reported submitting new project proposals to other specific funding sources (e.g., the European Union and United States Embassy). Citizens with more advocacy experience dating back to 1999 or earlier also articulated positive experiences managing their project(s). Social Impact, Inc. OTI/Angola Evaluation Report Page 21 of 76 When middle-range grantees were asked about future plans, they had either 1) “no plans” or 2) limited knowledge of available funding sources for follow-on activities. Thus, gaining experiential knowledge from the OTI-supported project did not always lead to increased capacity to advocate. The evaluators believe that grantees’ internal commitment to advocacy is likely an important factor, as well (e.g., the more experienced citizens were highly committed to advocacy work as evidenced by their persistence in it over time). A positive impact of the SADT program may then be that it helped provide an important bridge between two highly motivated cohorts of committed activists, one of committed advocacy veterans and another of passionate newcomers. The question then may be what OTI can do to recruit and select that type of grantee, while better motivating and developing those in the middle, lower-performing group. Another factor which closely correlated with the grantee’s learning experience was his/her perceptions of the in-kind distribution system. The number of grantees who either valued or complained about the system was roughly split. This viewpoint respectively reduced or enriched the grantee’s skills development and success at advocacy. For example, for those who appreciated the distribution system, data associated with the learning experience (i.e., successes in advocating for a change) was positive; the grantee appeared fulfilled. Conversely, for those who felt as though their financial and administrative capacity was undermined because a) ‘[OTI/CREA] did not trust us…’ or b) they were deprived ‘of the opportunity to gain experience or training in financial management’, the data suggests that the grantee felt disempowered to successfully advocate for change. Grantees’ process of gaining capacity to advocate seems to have been either helped or hindered, depending on their feelings about the in-kind system. Overall, OTI/Angola’s program was successful in increasing CSO/NGO capacity to advocate on key issues through supported projects. Much of this success can be traced to its strategy of flexible, participatory grant making and sensitivity to local contexts and regions. Several other external and internal factors contributed to individual success along with a grantee’s commitment to change and advocacy playing a strong role in successful capacity building. The quantitative data – although mixed – supports this conclusion. Finally, grantee perceptions about the in-kind system and experience of positive reinforcement also shaped their capacity building experience. Increasing Engagement to Address Community Problems Evaluators found that CSOs/NGOs supported by USAID/OTI’s SADT program under this objective played a prominent role in improving engagement between citizens and local authorities to solve community problems. Support was effective in opening or initiating participatory and democratic processes in all regions, although comparably less in Huila. This observed variation has been attributed to contextual differences among local authorities. Social Impact, Inc. OTI/Angola Evaluation Report Page 22 of 76 Data from in-depth interviews indicate that grantees appeared to have a solid understanding of problems/issues, how to address them, and how to increase participation from the community. All grantees interviewed reported community participation as ‘good’, ‘high’, and ‘participatory.’ Communication difficulties including lack of writing skills were reported to have constrained some community engagement efforts. The quantitative data on community engagement grantees comprises the same thematic areas, methods, and audience(s) discussed above under Advocacy, for the most part. Grantees’ intentions were clear, with 97 percent of those surveyed agreeing (67 percent strongly) that “Networking with local authorities is very important to achieving advocacy goals.” OTI supported engagement activities that included: · Seminars and workshops with beneficiaries in Huila; · Creating councils where none existed before in Huila and Benguela; · Hosting a rally with parents in Benguela; · Negotiating land rights with administrators; and · Distributing cassettes to villagers about land rights. Qualitative data indicates that grantees working in Benguela and Huambo achieved multiple levels of cooperation, open dialogue, and willingness on the part of local officials to work with the community in the problem-solving process. This positive engagement extended to support from the police, city administrators, and even the governor. The following are good examples of cooperation and responsiveness: “…The municipality gave us access to a tractor (and fuel) to take construction materials back and forth, and they even gave fuel for our motorbikes when needed.” Another grantee reported: “…approaching the Police Chief was not a problem….We explained our project mandate as well as the law which entitles the woman to work without harassment in the markets, and these people [authorities] executed this instruction or directive down the ranks all the way to the police in the markets…” Those working in Huila Province, however, experienced consistent and repeated setbacks when eliciting participation from city administrators, provincial representatives, and others invited to comment on land rights issues. Variation in government participation rates across different regions could be attributed to the war. In areas where heavy fighting existed (Huambo) and population movements occurred (Benguela), it is likely that the local authorities were willing to take risks to meet community’s expressed needs as an initial step to securing peace and demonstrating commitment. In the case of Huambo, there is a rich donor community, which may have been providing training to local Social Impact, Inc. OTI/Angola Evaluation Report Page 23 of 76 authorities. Conversely, in Huila, a grantee described the local authorities as “the same old guys and they are tired.” Raising Awareness and Changing Attitudes Effective advocacy and community engagement both require raising awareness and changing attitudes and beliefs. For a war-torn country such as Angola, transition to democracy can be seen as requiring people at many levels (beneficiaries, CSOs/NGOs, local authorities, community leaders) to journey from apathy to awareness, from resignation to empowerment, and from fear to commitment. OTI’s program clearly helped move some people along that journey, and into new forms of action. Grantees report that before participation in OTI-supported activities, there were many beneficiaries who lacked awareness of their basic rights, and whose attitudes were not conducive to advocating change or to engaging with others to solve community problems. One grantee described this state of mind as ‘very calm’ [muito calmo]. Yet discouragement and fear are clearly also factors inhibiting people’s openness to new ideas and new actions. The following interview quotes illustrate this outlook: · [Before the grantee’s advocacy efforts], “Parents asked ‘how much is the ‘cobrança’13 this year?” · “When a woman saw a crime in the market, she would look the other way.” · “Women were scared of the police.” · “The laws don’t get distributed to the provinces, this is Angola.” · “The leader in the village is the LEADER.” · “People think that ‘to vote means war.’” Grantees reported working directly with people to raise awareness and change attitudes. As an example, an activist told a parent “You have the right as a parent to send your child to school without paying…” According to this grantee, parents did not know that paying a cobrança was not mandatory and were not aware that a law even existed which protected their children. Another grantee reported that women did not realize that they were entitled to work in the market place free of police harassment. The grantee reached audiences of women through formal lectures and political theater to educate them about their rights. The grantee reported that the women now openly discuss their rights with others in the markets, saying “the police have no right to harass us…” Further examples of shifts in awareness and attitudes resulting from program participation can be seen in the following interview quotes: 13 ‘Cobrança’ means ‘bribe’ or payment in Angolan Portuguese Social Impact, Inc. OTI/Angola Evaluation Report Page 24 of 76 · “The women would not hesitate to report a crime when observed which is different from the past. In the past, they would look the other way and going to the police was not even part of their thinking.” · “Now, a parent will …say you did not cover this or that topic well (criticizing) or ‘I thought that one area was too complicated for this class’. Before, holding teachers accountable for their instruction was not even considered.” · “The schools [administrators] used to pull a number out of the sky – about what to charge – and now this number is discussed amongst school administrators, parents, and at the parent assembly. The process is much more transparent.” · [After the community meetings] “…the beneficiaries were more informed and showed that they had learned something – new sense of consciousness, saying let’s continue this.” Fear remains a factor limiting people’s willingness to pursue advocacy or open civic engagement, especially in times of instability and violence, and until civic society can be strengthened further. One grantee described how “their (women in the markets) heads have changed, but there is still the need to eat…they do not have the money to take the action.” Another grantee recounted that some parents spoke out against the grantee’s advocacy work, stating that they were against corruption, but they wanted their children to receive instruction. New Actions Even given the challenging program context and alongside beneficiaries’ justifiable fears, the evaluation team uncovered a number of examples of positive change, which occurred as a direct result of OTI-supported initiatives. These changes took place at the school, community, and government levels. Three examples illustrate grantee actions. By distributing a bi-monthly bulletin to schools, grantees encouraged teachers to introduce and discuss key topics such as democracy, domestic violence, and elections, which had never taken place before. And students, in turn, could engage their parents in discussion. To get the project off the ground, the grantee described that it was first necessary to educate local authorities as they were not aware of citizen rights within the law. The second example represents new actions at the community level. One grantee described how lectures were offered twice a month to communities with high IDP populations (on land rights, peace and conflict resolution, constitutional law, and human rights). These lectures were geared to changing discriminatory attitudes toward IDPs. The grantee described how the teachers, the ‘soba’, police, administrators, and beneficiaries attended lectures. Grantees noted that the most vulnerable members of the community, who wanted a change, and who needed the help, consistently participated in grantee activities. Social Impact, Inc. OTI/Angola Evaluation Report Page 25 of 76 As a result of these efforts, 600 IDP families were able to negotiate with the administration for ten hectares of land. Moreover, two different farmer associations were created, which have assisted 350 other families. A community seed bank was also established, and people have increased access to seeds, thereby reducing food shortages. The third action took place within the city administration in Huila Province. A professor complained on the radio about a trash dumpster located near an elementary school, pointing out the inefficiency of the Department of Public Works in the city of Lubango. A number of journalists followed the case, visiting the Public Service office, where the authorities responded by stating that the trash wasn’t that close to the school. The journalists then visited the school and noted that trash blocked the school driveway. They also conducted interviews with the parents and the students to collect their opinions on the issue. Shortly after the broadcast, another grantee hosted a separate discussion on health care issues. Participants asked consistently: “Where is all the garbage going to and, how can the garbage from a market be in such close proximity to a [school and] hospital?” While people knew that there were sites where garbage is dumped, it was not clear where those sites were located. Grantee reports indicate that an association, Joventude Ecológica de Angola (JEA), followed up with this issue by continuing to pressure the city’s health department. The government imported trucks and new dumpsters in response to the crisis, and the next week, there were more garbage trucks emptying dumpsters and hauling away trash. In this case, concerned citizens held the city government responsible for public services. These examples show how OTI-supported projects had a decidedly positive impact through both advocacy and community engagement that catalyzed new actions. VII. LESSONS LEARNED AND RECOMMENDATIONS The OTI/Angola program aimed to help build a foundation for the transition to an open and participatory democracy. The Support to Angola’s Democratic Transition (SADT) Program was designed to support a wide range of transition activities, through a small grants mechanism administered by CAII/CREA as OTI’s implementing partner. Evaluators found that the program objectives, to strengthen CSO/NGO capacity to advocate on key issues, and to increase citizens’ and local authorities’ engagement to address community problems, were met to a large extent. These results were all the more impressive for having been achieved in an operating context fraught with many challenges. The following are lessons learned from the OTI/Angola experience, along with recommendations that the evaluators hope will prove useful to other USAID/OTI programs: Social Impact, Inc. OTI/Angola Evaluation Report Page 26 of 76 Increasing CSO/NGO Capacity for Advocacy and Engagement Lesson Success and positive reinforcement strengthened grantees’ confidence and credibility, making it easier for them to attract further participation. Those grantees who receive public recognition may also serve as visible models of successful advocacy, resulting in a multiplier effect. Recommendations · OTI should build in more systematic opportunities and modes for providing positive reinforcement to grantees for their efforts at every stage.. The SADT program may have provided an important bridge between two highly motivated cohorts of committed activists, one of committed advocacy veterans and another of passionate newcomers. OTI should more systematically try to recruit and select that type of grantee, while better motivating and developing those in the middle, lower-performing group. · OTI should do more to help its grantees gain wider visibility and recognition for successes. For example, program staff could publicize good news, advise and train grantees in media relations, encourage authorities to publicly recognize grantee efforts, etc. Lessons Efforts to increase engagement in community problem-solving met with varying degrees of local government responsiveness – and CSOs/NGOs can be proactive about improving their engagement strategies. Even where local officials are unresponsive, methods for community engagement can be tested and refined, in anticipation of opportunities for further engagement. Responsiveness to engagement was higher in areas more affected by war. This may have been due both to greater incentives for officials to reduce conflict in those areas, and to a more substantial presence of international aid organizations that provide training to local officials. Recommendations · OTI should include assessment of local government responsiveness in the grant making process, to capitalize on opportunities for engagement and apply funds where they will have the greatest impact. · USAID/OTI should gather best practices, establish guidelines, and develop training on successful strategies for engaging local government (along with citizens) in addressing community problems. Guidelines should be designed to help CSOs/NGOs decide which methods to pursue given specific local conditions; training should be based on best practices and lead to practical application of specific engagement methods. · Opportunities should be created for sharing of successful engagement strategies among CSOs/NGOs. Social Impact, Inc. OTI/Angola Evaluation Report Page 27 of 76 · Opportunities should be created for sharing of successful engagement experiences among local authorities and across regions. Lesson Raising awareness, changing attitudes, encouraging advocacy, and creating new forms of action are overlapping and mutually reinforcing activities. Many grantees reported substantial progress in raising beneficiaries’ awareness of their rights, particularly in relation to land rights and women’s rights. Awareness of one’s rights and a sense of empowerment to assert them go along with more favorable attitudes toward advocacy, which counteract fears that inhibit action. These synergy effects are a cornerstone of democracy building. Recommendation · OTI should develop criteria for helping to identify and support CSOs/NGOs that are working in ways that take advantage of synergies among awareness, attitudes, advocacy, and action. · Such synergies should also be encouraged by OTI through grant making that supports CSOs/NGOs to align their work on separate but interrelated elements of democracy building. Lesson Grantees’ capacity for advocacy was most strengthened by program participation when they had either limited or extensive advocacy experience in the past. Those in the middle, neither new to advocacy work nor highly knowledgeable, seemed to benefit less from OTI support in terms of increased capacity. Thus, there seem to be two cohorts of activists highly motivated to learn and to gain capacity with support: one of committed advocacy veterans and another of passionate newcomers. Recommendations · OTI should figure out what it takes to more consistently recruit and select grantees from these two high-impact cohorts. The benefits of helping to connect these two cohorts should also be explored, as it seems this might have favorable implications both for the mentoring of those newer to advocacy and the reinvigoration of veteran activists. · At the same time, OTI needs to understand better how to motivate and develop the capacity of those grantees in the mid-range of experience. Program Strategy and Implementation Lessons The speed and flexibility of OTI’s small-grants program approach helpedOTI respond rapidly to emerging opportunities in Angola. This strategy also made it possible to shift the focus of grant making toward areas of greater effectiveness as Social Impact, Inc. OTI/Angola Evaluation Report Page 28 of 76 those became clear (for example, away from media strengthening and toward advocacy and engagement). An inclusive, flexible grant making approach allowed for adapting program activities across a wide range of conditions and cultures. Observations from the field indicate that the OTI Country Representative ‘cast a wide net’ when financing various NGOs/CSOs. This strategy was appropriate and effective in responding to the high degree of cultural variation in Angola. OTI’s participatory, grassroots strategy resulted in initiatives that were grounded in grantees’ knowledge of – and accountability to – their communities. Encouraging CSOs/NGOs to set the agenda for their advocacy efforts reinforces other aspects of capacity building, and makes it more probable that grantees will achieve self-sufficiency. In Angola, it also led to agendas that were arguably better adapted to their respective local contexts than OTI-defined agendas might have been. Difficulties with fulfilling USG procurement process standards were eased in one province by partnering with an NGO familiar with those procedures. This was a notable success since many grantees in other areas did not have the benefit of this assistance, and experienced significant problems and delays. The in-kind distribution system was perceived differently by grantees with differing degrees of financial/administrative capacity –– and those perceptions affected their program outcomes. Some grantees – those with a more advanced financial capacity - saw the in-kind system as disempowering and as depriving them of a learning opportunity. Those grantees had a less positive experience and outco Recommendations · OTI should continue to apply this flexible small-grants model in countries with similar characteristics, but should mitigate its significant risks by: ? Providing continuity of personnel throughout the life of the program in designated countries. High staff turnover from every level of this program contributes to the inconsistencies observed within the administrative system and programming. ? Developing clearer standard operating procedures for the Implementing Partner. OTI should provide better documentation, in addition to the contract, describing field office management, lines of authority between OTI and implementing partner(s)/contractor, and specific details about what level of performance is considered acceptable/unacceptable. The ‘four corners’ should meet together prior to initiating the contract in country, and should clarify roles, responsibilities, and the steps each party would take to manage conflict. ? Requiring informal cost analysis as part of the program assessment process. During the assessment period, the Contractor/Implementing Partner should conduct an informal consumer price survey to substantiate the costs and quality for goods/consumables and services related to operations. The process should be quick and simple, perhaps captured by a structured email to the USAID/OTI office. Social Impact, Inc. OTI/Angola Evaluation Report Page 29 of 76 · The Implementing Partner/Contractor should seek out potential partners (NGOs) with a field presence to assist in the delivery of inputs where procurement along USG standards presents unreasonable challenges. Such NGOs can serve as communication and information conduits on behalf of the implementing partner, greatly facilitating input delivery. · Rather than implementing a uniform in-kind distribution system, OTI’s implementing parter should conduct a rapid administrative and financial assessment of grantees. Field observations revealed that some CSO/NGOs had the capacity to manage funds and administer the OTI-supported projects, thereby questioning the need for the in-kind distribution system. Thus, an inventory or assessment of grantee’s financial and administrative capacity (a modified checklist used by auditors) would enable some CSO/NGOs to implement their own project. As a result, the PO/Logistics Officer’s time and efforts could be extended, refocused and/or better utilized, thereby allowing that person to focus on the CSO/NGOs who may require more capacity building assistance. · If a uniform in-kind distribution system must be utilized, the Implementing Partner should take the necessary time to explain the rationale to CSO/NGOs with strong financial and administrative capacity. This will help prevent sentiment among some grantees that OTI believes them untrustworthy, and it will demonstrate respect for their capacity – something that has a direct and positive effect on program impact. Merging of OTI and DG Management Roles Lesson Merging dual management responsibilities into one role is a workable approach to aligning OTI and DG programs, if constraints are addressed. OTI/DG integration will be more effective the more it is made part of a systemic approach to synchronizing mission, roles, and functions. Recommendations · Streamline the administrative load on the DG Program at the field level. · Conduct monthly information meetings to share vital program information. To facilitate the integration of OTI/DG programming, sharing information in a more structured fashion is recommended. Program staff should meet regularly (monthly) to update each other and share information (not to exert influence over the other’s program direction). This would ensure that all staff are well-informed on key aspects of the OTI/DG integration. · To streamline effectively, USAID offices need to look “upstream” as well to ensure lateral communication and coordination between DG and OTI management and to reduce duplication of effort in program administration. Social Impact, Inc. OTI/Angola Evaluation Report Page 30 of 76 APPENDICES Appendix 1 – Scope of Work Task Order #12 HAD–I–12–03–00124 –00 U.S. Agency for International Development (USAID) Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA) Office of Transition Initiatives (OTI) FINAL EVALUATION OF OTI’S PROGRAM IN ANGOLA (August 2004 – November 2004) I. OTI Background The USAID Administrator created OTI in the Bureau for Humanitarian Response (now the Bureau for Democracy, Conflict and Humanitarian Assistance) to assist priority countries to make successful transitions from crisis to recovery and stability. The volatile political and economic nature of transitioning countries requires fast, emergency-type political responses that show immediate, visible and positive effect. Countries experiencing complex crises resulting from internal conflict and civil war have special needs that are often not addressed by traditional emergency assistance programs. OTI enables USAID to capitalize on ‘windows of opportunity’ where quickly deployed aid can make a critical difference to a country’s transition to peaceful, democratic government. Interventions are tied to pivotal events, such as cease-fires, peace accords, or the advent of progressive leadership, often through key elections. OTI responds swiftly to these events with near-term, high-impact actions that support a country’s transitional needs. While operating in a country, OTI works to bring new groups into the transition process, tests new activities for advancing democratic governance, and provides fast and flexible support for immediate transition needs. OTI’s program options for transition responses include: 1) expanding democratic political process, 2) building citizen security, 3) promoting reconciliation, 4) support peace negotiations, and 5) cross-cutting themes, including community-based approaches and media activities. As appropriate and necessary, relationships and practices that prove productive may be handed off to the Social Impact, Inc. OTI/Angola Evaluation Report Page 31 of 76 USAID mission or other donors for further development when OTI phases out its assistance. II. Angola Country Background The situation in Angola changed dramatically during 2002, with UNITA leader Jonas Savimbi's unexpected death quickly followed by a cease-fire, a renewed commitment to the peace process, and the rapid demobilization of UNITA forces. After some 40 years of war, these developments presented a unique opportunity to invigorate Angola's stalled democratic transition. For the first time in the history of Angolan independence, the government no longer had a wartime or national security justification for its poor governance performance. In addition, a nascent political transformation appeared to be underway, with elections widely anticipated for 2004 or 2005. Accordingly, a fast, flexible small-grants program targeting civil society, media, and legislative strengthening over a two-year period was viewed as a means to help ensure that the elections will be as meaningful as possible. This, in turn, could help make the difference between a more equitable, democratic peace versus continued one-party dominance and poor governance. III. OTI Angola USAID/OTI's goal has been to use the increased stability of Angola’s transition from war to peace to promote a more open and participatory democratic society. Working closely with USAID/Angola's Democracy and Governance (DG) team, OTI/Angola has tried to spread and strengthen participatory democratic practices and promote greater political competition, accountability, and transparency. These efforts have taken place through activities focused on: strengthening media capacity; supporting citizen groups in improving their capacity to advocate for key reforms and increase participation in advocacy efforts; and, increasing local-level engagement between citizens and governmental authorities to effectively address community problems. IV. Objectives of the Evaluation There are three basic questions to be answered by the final evaluation: 1. To what extent did OTI/Angola’s program meet its stated goal and objectives? 2. How did the management and operation of the program contribute to or detract from achievement of the program goal and objectives? 3. Based on the evaluation findings, what are the lessons learned and ways OTI can improve its programs? These basic questions will be more clearly defined through discussions with OTI Washington and field staff during methodology and workplan development. Social Impact, Inc. OTI/Angola Evaluation Report Page 32 of 76 V. Methodology The evaluation team will be responsible for developing an evaluation strategy and methodologies that include a mix of qualitative and quantitative data collection and analyses approaches. Specific methods, and the appropriate instruments, will be developed in concert with OTI/Washington. VI. Evaluation Components and Deliverables 1. 10 work days, Washington, DC Conduct literature review and desk study including OTI/Angola grants data base Draft workplan Develop methodology and instruments Interview key Washington, DC stakeholders Finalize workplan 2. 18 work days, Angola Collect evaluation data from OTI/Luanda, Creative Associates, grantees, and beneficiaries Conduct initial analysis and develop initial findings Confer with field staff at evaluation mid-point De-brief OTI/Angola staff; present a 5-7 page report of key findings 3. 12 work days, USA and Washington Prepare draft report Debrief OTI/Washington; collect comments from Washington and the field Prepare final Report Final Report The outline for the final report shall comprise, but not be limited to the following: Executive summary Social Impact, Inc. OTI/Angola Evaluation Report Page 33 of 76 Table of contents Introduction and background Summary description of evaluation objectives Description of methodology and data sources, and limitations of the study Analysis and statement of findings Recommendations for future OTI programs A USAID-wide presentation on the evaluation will be scheduled upon receipt of the final report. Fifty bound copies of the final evaluation report and supporting documents will be provided to OTI, along with an electronic version of the report and an electronic copy of all data files used to conduct analyses. VII. Timeframe Activity Location Time Dates Selection of evaluation team; initial meeting with OTI/Washington staff Wash., D.C. ---- Aug. 16 Review OTI documents; discuss work plan and other needs with relevant OTI staff; begin interviews with OTI/Washington-based Africa team members and other relevant field partners with offices in the Washington area. Wash., D.C 10 work days Aug. 16 – Aug. 27 Review documents as needed/appropriate; adjust work plan as needed; interview OTI, USAID Mission, U.S. Embassy, partner and other staff as appropriate; conduct quantitative and qualitative research; analyze data; and, debrief/report on preliminary findings. Angola 18 work days Aug. 30 – Sep. 22 Write report; circulate for review/comments; revise/finalize report; debrief OTI Washing- ton staff and others. Wash., D.C 12 work days Sep. 27- Oct. 13 Social Impact, Inc. OTI/Angola Evaluation Report Page 34 of 76 The evaluation team will be responsible for making its own arrangements for translators, transportation, housing, and other logistics. The team is also responsible for its own work space, computers, and printers. VIII. Composition and Qualifications of the Evaluation Team The evaluation team will consist of a senior level evaluation analyst, who will also serve as the team leader, and a mid-level local (in-country national) evaluation analyst. The team leader should have extensive experience designing and conducting evaluations, and analyzing quantitative and qualitative data. Other qualifications include: Evaluation/research: experience in the social sciences evaluating programs – particularly ones involving community participation, media, and civil society organizations – in countries undergoing transitions. Rapid appraisal techniques: training and experience with rapid appraisal techniques (survey development, direct observation, focus group interviews, community interviews, and key informant interviews). Local knowledge: knowledge of Angola Afghanistan and/or Islamic culture. Language ability: Ideally, both members of the team will have a demonstrated knowledge of and/or fluency in Portuguese. Social Impact, Inc. OTI/Angola Evaluation Report Page 35 of 76 Appendix 2 – Evaluation Framework Evaluation Framework In an effort to elaborate the information included in the Scope of Work, I have developed an evaluation framework for review and feedback. The purpose of this exercise is to gain a clear understanding of the specific areas OTI would like to explore during the final evaluation process. Under each final evaluation objective, the data sources used to address each question are presented. In the case of the last objective, there is need for confirmation. In so doing, I have presented ideas and conceptual areas which I believe reflect OTI/Angola’s program objectives. To recall, the final evaluation objectives or basic questions to explore as articulated in the Scope of Work are: 1) To what extent did OTI/Angola’s program meet its stated goal and objectives? 2) How did the management and operation of the program contribute to or detract from the achievement of the program goal and objectives? 3) Based on evaluation findings, what are the lessons learned and ways OTI can improve its programs? To respond to the first objective, documentation and interviews with OTI/Angola and CAII will be used to review program components, how they were implemented, and obstacles and constraints that influenced obtaining program goals. From a discussion with Mary Stewart on September 7th in Washington, the second objective is not intended to have as much focus as originally conceived, since greater efforts will be spent on the last evaluation objective. Many OTI/Washington stakeholders are specifically interested in how the Country Representative’s dual role as the OTI and Democracy and Governance (DG) representative influenced her ability to manage OTI’s program. That is, as a result of integrating OTI and DG’s program to achieve “complementarity”, which is really progressive, how did this program approach affect the management of OTI’s program? The findings from this question may have implications on other programs, where an integrated approach is being considered. It is envisioned that documentation and reflections from OTI/Angola’s Country Representative will be used to address the second question. Feedback from the OTI representative in Angola and the Washington DC team indicates that understanding the lessons learned is important, but the interest primarily lies in exploring the impact that specific grant activities may have had. Having this particular focus in mind, there are two broad areas presented below – reflecting the objectives from the Strategic Plan, which require review. Please note that the time period available to conduct interviews in Angola is comprised of a three-week window, and the capacity of those available to assist with collecting and analyzing data is limited. Moreover, logistical constraints associated with travel and communications within Angola may present challenges, and so it is for these reasons that impact has been restricted to two thematic areas. Social Impact, Inc. OTI/Angola Evaluation Report Page 36 of 76 Area #1: Examining the extent to which CSO or/and NGO’s capacity to advocate on key issues has increased Angola has had tremendous humanitarian distributions for years, thereby distorting CSOs or/and NGOs capacity to be proactive entities. The OTI Washington team is interested in knowing if and how the inputs and resources offered enabled the grantees to manage their organizations, including [but not limited to] a) setting up strategic plans and implementing them (which would include having future plans to advocate in other areas), b) developing a mandate or a mission statement, c) engaging in fundraising activities, and d) having a better understanding about financial practices and/or administrative procedures. The lenses normally used to measure benefits in this area include: a. Acquiring knowledge from training and technical assistance, and b. Gaining skills from professional experiences (i.e., implementing a program and networking with others knowledgeable about organizational development). A mixed methodology would be used to cover this conceptual area. Apart from examining the organizational capacity, it may be of interest to observe their technical ability to develop and deliver a social message for the purpose of educating and, with the intention of, empowering people. Qualitative methods could be used to review the basic message(s) or themes communicated to educate and inspire individuals as well as the methods in which the CSOs and/or NGOs used to advocate (i.e., training, pamphlets, theater, open forums, and other methods). Examples of areas in which data would be collected include (but are not limited to) the following: a) A taxonomy of current issues being advocated (i.e., rights to a free education, rights of women in the market place, land rights, labor rights, formulation of a new government constitution); b) General media used to advocate certain issues/topics; and c) Future plans to advocate in other areas. Essentially, this area may address the following basic questions: Did the support extended make a difference? If so, in what way? If not, why not? And, what did the CSOs or/and NGOs do (in terms of activities undertaken) with their capacity? Area #2: Exploring the degree to which the CSO/NGO’s ability to promote participation between citizens and local authorities to address community problems Since the OTI supported activities in Angola concentrate on nurturing community engagement with local authorities, some key areas to investigate would include a) the CSO/NGO’s role in facilitating a) problem identification within communities, b) problem solving, c) people’s knowledge and understanding of laws and institutional mechanisms, d) the organization of citizens to present their voice about problems or issues of importance to the local government, e) the participation of citizens in meetings and public activities, and f) community engagement in detailed discussions related to logistics and organizational arrangements around a specific project. Social Impact, Inc. OTI/Angola Evaluation Report Page 37 of 76 Qualitative methods would be the most appropriate approach for this area. Part of the qualitative analysis would also include a review and analysis of factors which may shape or influence a grantee’s ability to work. Some of these factors could include but are not limited to: a) Region in which the grantee is working; b) Political tension levels of the region; c) Presence of DG activities taking place; d) Size of the organization; e) Demographic factors (i.e., ethnicity, experience, education levels) of those managing the CSO or NGO; f) Frequency or level of interaction with the Contractor; and g) Other relevant factors which may shape the effectiveness of grantees’ initiatives. To facilitate an understanding of how the above conceptual areas would be examined, I have developed a table which includes a) the two thematic areas, b) what methodologies would be used and with whom, and c) the possible topics of data to be covered. Table 1: Conceptual Areas to Explore14, Corresponding Methodologies, and Topics of Data to be Covered Main Impact Questions Methodologies Possible Data Collection Topics to Cover AREA #1: STRENGTHENED ORGANIZATIONAL CAPACITY OF CSOS/NGOS TO ADVOCATE ON KEY ISSUES a) OTI and CAII Staff Interviews and records; b) Open ended questions with grantees; c) Grantee survey. Participant (grantee and OTI/CAII Staff) attitudes about: a) The skills acquired from OTI supported activities (training, technical assistance, experiences from networking with others-if applicable). b) The type of advocacy in which they engaged (social issues and methodologies used to engage communities and individuals); c) What results took place afterward, if any? d) What factors or constraints exist when communicating social messages and/or inspiring participation, and why? AREA #2: EXPLORING THE DEGREE TO WHICH THE CSO/NGO’S ABILITY TO NURTURE LOCAL PARTICIPATION BETWEEN CITIZENS AND LOCAL AUTHORITIES TO ADDRESS COMMUNITY PROBLEMS a) Open ended questions with grantees; b) Open-questions with program staff (their ideas about which CSO/NGOs had the greatest successes and constraints in nurturing community engagement and why; c) Brief open-ended questions with local government officials to follow-up on data received Participant (grantee) attitudes about the following areas: a) What process(es) was used to engage communities with local authorities? b) How or what activities were used to identify and solve problems? c) Which approaches ensured the strongest participation in meetings and public activities among beneficiaries (also including local government stakeholders)? d) Which approaches resulted in the weakest participation among beneficiaries (also including local government stakeholders)? 14 This table reflects the conceptual areas, methodology, and possible topics to cover for impact areas exclusively. Social Impact, Inc. OTI/Angola Evaluation Report Page 38 of 76 Main Impact Questions Methodologies Possible Data Collection Topics to Cover PROBLEMS from grantees. including local government stakeholders)? Project Work Plan The evaluation consists of the following tasks: 1) Conduct a thorough review of OTI documents related to the Angola program; 2) Develop a sampling frame of all stakeholders based in Washington DC, including but not limited to: a. OTI/Angola headquarters staff; b. Africa Bureau headquarters staff; and c. CAII program staff. 3) Developing a sampling frame of all stakeholders who will participate in evaluation in the field, including but not limited to: a. OTI/Angola Country Representative; b. CSOs/NGOs in Huila, Benguela, Huambo, and Luanda, and those with regional programming (this makes a five categories in the sample); c. Relevant government stakeholders in Luanda; d. CAII program staff (Acting COP, PDOs, and Logistics Officers); and e. Relevant local government stakeholders based in Huila, Benguela, Huambo, and Luanda as they relate to grantee activities. 4) Draft question guides for each stakeholder group mentioned above as well as one survey; 5) Plan all logistics required to conduct site visits in the field (i.e., telephone calls with grantees to arrange appointments, car hire, hotel reservations in Huila, Huambo, and Benguela); 6) Conduct site visit to Huila, Benguela, and Huambo where OTI-supported activities are being undertaken; 7) Based on the sampling frame, organize a tentative schedule of all interviews, so that stakeholders have advanced notification about when to meet with evaluation team; 8) Formulate a training module for in-country evaluator, so that she is fully informed of a) the type of information to be collected, b) the evaluation methods to be used, c) who will be participating in the evaluation process, d) the instruments she will assist in translating, e) the interviews she will participate in collecting, and f) subsequent data analysis; 9) Analyze Data; 10) Present findings to OTI Country Representative prior to departure; and 11) Prepare final report. Please note that an outline of the report will be provided later. Sampling Frame - Grantees Social Impact, Inc. OTI/Angola Evaluation Report Page 39 of 76 An estimated 10 - 15 projects will be purposefully sampled using qualitative methods, and this figure depends on the logistics of in-country travel. The sampling criteria used to select the OTI’s grantees include the following characteristics: ß CSOs/NGOs by region based on the percent of the entire portfolio; ß CSOs/NGOs by objective; ß Proportion of the CSOs/NGOs working in DG areas (to test program dose); ß One CSO/NGO which did not receive any funding from OTI (to test negative case analysis), time permitting; ß Smaller CSOs/NGOs which would not normally have access to resources – those who OTI took a chance on; ß CSOs/NGOs which would normally have access to resources (of larger size); ß CSOs/NGOs funded in the early stages (to test program dose); ß CSOs/NGOs funded in the latter stages; ß Any others which Holly feels are important. Below, please find the list of the grantee projects which were practically randomly selected. The sampling technique used included, dividing the grant-making calendar into three periods to reflect those offered grants in the beginning, middle, and end of OTI’s program. The database enabled me to divide the time periods when projects were cleared and separated the grants by region. Each project’s number or code was then recorded onto a small piece of paper, and the random selection took place thereafter. This list is just a start, and should not be considered a fixed or final one. I am aware that the number of projects on this list does not reflect objective #3 adequately, and I will correct this later. Since I will be organizing all related logistics to visit grantees and stakeholders, it is likely that I will be cover a portion of those listed below. I also would appreciate any feedback from the Angola Country Representative and other stakeholders in Angola. Project Number Date Cleared from Database Title of Project and Name of Grantee Falls under which Program Objective Place or Region Amount of Grant First Round of Projects Sampled n=7 CR012 7/2003 Human Rights Protection Zones National Counseling Center (NCC) Objective #3 Participation Huambo $67,000 roughly CR013 6/20/03 Availability of Print Media Huambo Coalition Objective #2 Huambo $16,872 CR007 5/19/03 Regional Poverty Forum ALSSA – I selected purposefully Objective #2 Huila $26,837 CR006 5/30/03 National Strategic Planning Conference COIEPA Objective #2 All provinces $39,540 CR002 5/24/03 Market Women against Violence Bismas das Acacias Objective #2 Benguela $37,825 CR004 6/3/03 Terra: Land Advocacy Okutaka Objective #2 Benguela $67,888 CR018 8/22/03 Community Conflict Resolution Committees Objective #2 Luanda $20,425 Social Impact, Inc. OTI/Angola Evaluation Report Page 40 of 76 Project Number Date Cleared from Database Title of Project and Name of Grantee Falls under which Program Objective Place or Region Amount of Grant Acçao Crista para Infancia Second Round of Projects Sampled n=7 but one CSO is a repeat from first round CR030 11/10/03 Monthly Civic Society Issues Forum ACCORD Objective #2 Huila $45,613 LD001 11/21/03 Promoting Community Land Rights Maos Livres Objective #2 Huambo $26,448 CR017 9/4/03 Support for rehabilitation, reintegration, and subsistence agriculture OFDP Objective #3 Huambo $62,380 CR022 9/4/03 “Onjila”: Strengthening the Education Sector ADRA Objective #2 Huambo $63,868 CR027 10/20/03 Local Campaign to Promote Free Education Okufuka & Ensino Gratuito Objective #2 Huila Benguela $2,400 CR028 11/6/03 The Citizen’s Voice: Radio Program and Civic Education NCC Objective #2 Luanda $53,350 CR023 9/9/03 Path of Peace: National Conflict Resolution Network Caritas-Angola Objective #2 Huila $51,010 Third Round of Projects Sampled n=6 but one CSO/NGO is repeated CR040 2/9/04 Wind up radio for rural residents Freeplay Objective #2 – not sure Benguela $50,300 CR043 2/9/04 AJAD-H/Associacao Juvenil Angolana dos Direitos Humanos Community Education on Democracy Objective #2 Benguela $68,205 CR034 1/20/04 Labor Rights Advocacy Associacao dos Desempregados de Angola"Ponto de Encontro" ADA Objective #2 Luanda $50,652 CR031 2/2/04 Strengthening Farmers Associations Huambo Coalition Objective #3? Huambo $42,578 CR039 2/16/04 Creation of Palanca Youth Forum Grupo de Jovens Esperanca do Futuro Objective #2 Luanda $18,206 LD007 2/9/04 Civic Education Through Radio Associacao de Jovens para o Desenvolvimento Comunitario Objective #2 Huambo $29,300 Sampling Frame – Key Stakeholders in Angola Social Impact, Inc. OTI/Angola Evaluation Report Page 41 of 76 The list below has helped me to determine how much time I plan to spend with various stakeholders. ß CAII’s Program Staff (PDOs, Logistics Officers) – At least two days with them; ß USAID Mission individuals (not sure how many interviews here, Mary and Holly please guide); ß Government Persons in Luanda (not sure who is the person Holly interfaces with regarding this program, please guide); ß Interview with the OTI Country Representative (Roughly 3-4 hours); and ß In the different regions, the key local government contacts within each major town. (This is particularly important if Option #2 is selected). Guidelines for Plan of Action: It is preferable to devote one day to meet with no more than two grantees. This scheduling pace will enable the evaluation team to have plenty of time to conduct qualitative interviews and execute any survey with grantees, their beneficiaries, and community members. The schedule is also flexible in the event that there are delays due to logistics or power outages. For grantees with more than one OTI supported project, it is expected that they will need additional time during interviews. Those who have interfaced frequently with the OTI Country Representative and CAII Program Development Officers (PDOs) may also require more time. The grantees who have access to more resources may also require considerably more time, as they will have a different perspective about where and how OTI ‘fits’ with the other funders from whom they may secure funds. It has been my experience that government representatives do not necessarily need more than one hour for an interview, but there is sometimes a delay waiting to see them even with an appointment. Further, a portion of that hour is normally dedicated to protocol issues, whereby I explain the nature of the evaluation, the evaluation team’s focus, and our schedule (in vague terms) while on site. Social Impact, Inc. OTI/Angola Evaluation Report Page 42 of 76 Appendix 3 – List of Interviewees Persons Interviewed in Alphabetical Order Stakeholders based in Washington DC and in Angola Allia Afshar, Program Associate for Communities in Transition Division/Project Backstop Technical Assistance, Creative Associates International Inc. (CAII) Bronwyn Bruton, OTI Program Manager from 09/2003 to 04/2004 Elias Isaac, Democracy and Governance (DG) Program Specialist (USAID/Angola) Bob Leavitt, Africa Bureau Conflict Program Advisor Angela Martin, Africa Team Leader Richard McCall, Senior Vice President for Programs, Creative Associates International Inc. (CAII) Julie Nenon, Angola Project Director, Creative Associates International Inc. (CAII) Holly Flood, Angola Country Representative, +244-2-399-519 Nhelly Saleh -Ramirez, OTI Program Manager from 04/2004 to early 07/2004 Sandra Shuster, Creative Associates International Inc. Chief of Party 01/2003 to 09/2003 Larry Meserve, Special Projects Officer (assisted during program development stage) Josée Le Mieux, Creative Associates International Inc. Chief of Party 09/2003 to 04/2004 Mary Stewart, PhD, Team Leader Program Development Eveline de Lima Viegas, Democracy and Governance (DG) Program Specialist (USAID/Angola) Tjip Walker, PhD, Senior Conflict Advisor (assisted during program development stage) CREA Staff in Angola Carlos Jusitino, Program Development Officer (PDO) –Huambo José Manzumba da Silva, Program Development Officer (PDO) – Benguela Ernesto Quarta, Logistics Officer (LO) – Benguela and Huambo Social Impact, Inc. OTI/Angola Evaluation Report Page 43 of 76 Appendix 4 – List of Grantees and Projects Sampled Project Title Grantee Organization Names of Interviewees HUILA PROVINCE LD003 Project Name: Radio Program: “Our land our wealth” Radio 2000: Free Lancer Journalists Moisés Sachipangue C. Chinhama, Assistant Coordinator Morais Augusto da Silva, Coordinator Program Objective 3 CR007 Project Name: Regional Poverty Forum Associaçao Leonardo Sikufinde Shalom Angola (ALSSA) António Soma, Director of ALSSA Téresa Rangel, President of the Assembly of ALSSA Program Objective 2 CR038 Project Name: Independent Media workshop and study tour Cooperativa de Iniciativas Locais (COPIL) Kiala Manuel, Program Coordinator António Soma, Director of COPIL Téresa Rangel, COPIL’s Program Administrator Program Objective 2 CR011- CR025 switched Project Name: Bulletin: ‘Citizenship and Rights’ Grupo Jornalistas Estanislau Costa, Journalist for the Jornal de Angola and Revista Económica e Mercado Program Objective 3 CR030 Project Name: Monthly civil society issues Agencia de Cooperaçao e de Pesquisa para o Desenvolvimento (ACORD) Fátima Sendo, Program Coordinator Avelino Tyiteta, Program Director Program Objective 3 Project Title Grantee Organization Names of Interviewees BENGUELA PROVINCE CR009 – CR027 and CR004 switched Project Name: Campaign to Promote Free Public Education Okutiuka & Ensino Gratuito Coalition Muanga Ferreira Vietu, Project Administrator and Activist Bráulio Teixeira, Activist Program Objective 3 CR017 Project Name: Support for rehabilitation, reintegration, and subsistence agriculture Organizaçao para o Fortalecimento e Desenvolvimento de Povos (OFDP) José Manuel, Project Coordinator Program Objective 3 CR014 – LD001 switched Project Name:Rehabilitation of five rural primary schools Asociaçao para o Apoio ao Desenvolvimento Comunitário (AADC) Eurico Numa, Program Coordinator in Cubal Program Objective 3 CR002 Project Name: Market women against violence Bismas das Acacias José Mbalote, Project Coordinator Joao Marcela, Project Activist Helga Correia, Project Activiist Joaquina Pacavira, Project Activist Cristovao Kajibanga, Director of Bismas das Acacias Social Impact, Inc. OTI/Angola Evaluation Report Page 44 of 76 Project Title Grantee Organization Names of Interviewees Kátia Canjila, Project Activist Ámelia Armando, Project Secretary Program Objective 3 Project Title Grantee Organization Names of Interviewees Luanda Grantees operating in field CR008 – CR0012 switched Project Name: Human Rights Education and Awareness at the Community Level National Counseling Center (NCC) Reis Luís, Director of NCC Manuel Modesto, Administrator Program Objective 3 Project Title Grantee Organization Names of Interviewees LUANDA CR028 Project Name: The Citizen’s Voice: Radio Program and Civic Education National Counseling Center (NCC) Reis Luís, Director of NCC Manuel Modesto, Project Administrator Program Objective 2 CR039 Project Name:Creation of Palanca Youth Forum Grupo de Jovens do Palanca Esperança do Futuro Agustinho André, Project Activist Emicilia Barciso, Project Coordinator Manuel Domingos, Project Activist Pungua Manuel, Project Activist Program Objective 2 CR034 Project Name: “Ponto Encontro” Labor Rights Advocacy Associaçao dos Desempregados de Angola Nelson Paulo, Program Officer Vicent Albino Paulo, General Secretary Program Objective 2 Project Title Grantee Organization Names of Interviewees HUAMBO PROVINCE CR031 Project Name: Strengthening Farmers Associations Huambo Coalition Gilberto Alfredo, Program Coordinato Program Objective 2 CR022 Project Name: “Onjila”: Strengthening the Education Sector Accao para o Desenvolvimento Rural e Ambente (ADRA) Angelo Carlos Kapwatclia, Program Coordinator – Huambo Program Objective 3 LD007 Project Name:Civic Education Through Radio Associacao de Jovens para o Desenvolvimento Comunitario Wilson Alfredo, Director of Organization Alexandre Junior, Assistant Denis Roger, Project Coordinator Program Objective 2 Social Impact, Inc. OTI/Angola Evaluation Report Page 45 of 76 Social Impact, Inc. OTI/Angola Evaluation Report Page 46 of 76 Appendix 5 – References and Works Cited Austral Consoltória, Lda. Mid-Term Evaluation of the OTI Program in Angola (Luanda: Austral Consoltória, Lda. March.) Behling, O. and Law, K. 2000. Translating Questionnaires and Other Research Instruments: Problems and Solutions. SAGE Quantitative Applications in Social Sciences Series (QASS). Thousand Oaks, CA, Number 07-133. Blaire, H.,Blue, R., Popovski, M., Trajkovski, I. 2003. Assessment of Civil Society Sector in Macedonia for USAID/Macedonia, (Washington: Development Associates, Inc. (DAI). August.) Blaire, H., Dayao, L., and Salomo, R. 2002. Civil Society Strengthening Program: Mid-Term Evaluation for USAID/Jakarta, (Washington: Management Systems International (MSI). September.) Bradburn, N., Sudman, S, and Wansink, B. 2004. Asking Questions: The Definitive Guide to Questionnaire Design – For Market Research, Political Polls, and Social and Health Questionnaires. (San Francisco: Jossey-Bass.) Capacity Assessment Tool used by Family Health International (FHI). Creative Associates International, Inc. 2004. Support to Angola’s Democratic Transition: Bi￾Annual Performance Report. February through July 31, 2004 (Washington: CAII.) Greene, J. and Caracelli, V. 1997. Crafting Mixed-Method Evaluation Designs, New Directions for Program Evaluation, Number 74 [Sunmmer]. Lippman, H. and Duwan, G. 2004. Performance Monitoring in Post-conflict/Transition Programming: Brainstorming Session (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives. May.) Lippman, H. 2001. Linking Democracy and Development: An Idea for the Times, USAID Program Operations Assessment Report Number 29 (Washington: USAID Center for Development Information and Evaluation. June.) Millsap, W. and Brusset, E. 2003. OTI Macedonia Confidence Building Initiative Evaluation Report. (Washington: Social Impact, Inc. October USAID. US Mission to Angola: Mission Performance Plan, FY 2005 (Washington: USAID Center for Development Information and Evaluation.) Sensitive but not classified. USAID. 2003. USAID/ANGOLA Annual Report (Washington: USAID Center for Development Information and Evaluation. April.) Social Impact, Inc. OTI/Angola Evaluation Report Page 47 of 76 USAID. 2003. OTI/Angola Strategic Plan – DRAFT VERSION (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives, April.) – DOES NOT CIRCULATE. USAID. 2003. OTI Grant Worthiness Guide. (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives. May.) USAID. 2003. OTI/Angola Program Strategy – DRAFT VERSION (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives, August.) – DOES NOT CIRCULATE. USAID. 2002. Angola: Transition and Development Assessment, The War Is Over, Peace Is Here To Stay (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives. October.) USAID. 2002. Democratizing the Peace: The Case of an OTI in Angola (Washington: Bureau for Democracy, Conflict, and Humanitarian Assistance and Office of Transition Initiatives, September.) USAID. 2001. Supporting Civic Advocacy: Strategic Approaches for Donor-Supported Civic Advocacy Programs, DRAFT VERSION (USAID: Democracy and Governance.) Wholey, J., Hatry, H., and Newcomer, K. 1994. Handbook of Practical Program Evaluation (San Francisco: Jossey-Bass.) Noteworthy Websites: Http://www.unicef.org/about/execboard/files/04-PL1_Angola.pdf. Http://www.cia.gov/ http://hdr.undp.org/statistics/data/ Social Impact, Inc. OTI/Angola Evaluation Report Page 48 of 76 Appendix 6 – Survey Demographics & Results from Attitudinal Statements Demographic Data Categories Sex Male Female Percent 80% 20% Status Single Married 68.6% 31.4% Education Levels Middle school High school University 45.7% 5.7 % 48.6% Type Of Organization Association NGO Coalition Religious organization 11.4% 62.9% 14.3% 11.4% Organizational Affiliation International Domestic 94% 6% Region in Which Those Surveyed Are Working Huila Benguela Huambo Luanda Benguela and Huambo 25.7% 31.4% 14.3% 22.9% 5.7% Strategy, project design, and fundraising work well and need little improvement. Frequency Percent strongly agree 13 37.1 agree 19 54.3 disagree 1 2.9 strongly disagree 1 2.9 not applicable 1 2.9 Total 35 100.0 Social Impact, Inc. OTI/Angola Evaluation Report Page 49 of 76 Communication with community is frequent. Frequency Percent strongly agree 19 54.3 agree 12 34.3 disagree 3 8.6 not applicable 1 2.9 Total 35 100.0 I am committed even if it involves risk and difficult times – Commitment Statement #1 Frequency Percent strongly agree 21 60.0 agree 13 37.1 not applicable 1 2.9 Total 35 100.0 There is no need to change our message, as we understand community reality and mentality. Frequency Percent strongly agree 9 25.7 agree 12 34.3 disagree 6 17.1 strongly disagree 6 17.1 not applicable 2 5.7 Total 35 100.0 Including the community in projects is time consuming and problematic. Frequency Percent strongly agree 1 2.9 agree 2 5.7 disagree 6 17.1 strongly disagree 25 71.4 not applicable 1 2.9 Total 35 100.0 I believe in including peoples' opinions, but the organization's philosophy does not have to change. Frequency Percent strongly agree 2 5.7 agree 9 25.7 disagree 13 37.1 strongly disagree 11 31.4 Social Impact, Inc. OTI/Angola Evaluation Report Page 50 of 76 Total 35 100.0 I have a clear understanding about planning and budget formulation. Frequency Percent strongly agree 12 34.3 agree 14 40.0 disagree 4 11.4 not applicable 5 14.3 Total 35 100.0 The community deserves a better life, but the risks are too great.- Commitment Statement #2 Frequency Percent strongly agree 12 34.3 agree 10 28.6 disagree 8 22.9 strongly disagree 5 14.3 Total 35 100.0 Strategy, project design, and fundraising work well and do not need improvement. Frequency Percent strongly agree 2 5.7 agree 10 28.6 disagree 18 51.4 strongly disagree 4 11.4 not applicable 1 2.9 Total 35 100.0 Advocacy messages should change after understanding community reality and mentality. Frequency Percent strongly agree 16 45.7 agree 11 31.4 disagree 5 14.3 strongly disagree 2 5.7 not applicable 1 2.9 Total 35 100.0 Networking with local authorities is very important to achieving advocacy goals. Frequency Percent strongly agree 24 68.6 agree 10 28.6 Social Impact, Inc. OTI/Angola Evaluation Report Page 51 of 76 Frequency Percent strongly disagree 1 2.9 Total 35 100.0 Project design should be a democratic process. Frequency Percent strongly agree 18 51.4 agree 17 48.6 Total 35 100.0 Project design is done in a participatory manner. Frequency Percent strongly agree 22 62.9 agree 11 31.4 disagree 2 5.7 Total 35 100.0 Partnering with the community on projects is a smooth and simple process. Frequency Percent strongly agree 5 14.3 agree 12 34.3 disagree 12 34.3 strongly disagree 5 14.3 not applicable 1 2.9 Total 35 100.0 Social Impact, Inc. OTI/Angola Evaluation Report Page 52 of 76 Appendix 7 – Question Guide 1: CAII/OTI Stakeholders based in Washington, DC Please note that these questions are only intended to guide a purposeful discussion. Question Guide for OTI staff and stakeholders in Washington, DC Staff/stakeholder interviews are intended to understand the activities associated with each program component, detailing specifically what is involved in strengthening the capacity of civil society organizations and increasing citizen participation in advocacy and to address community problems (program objectives two and three). The guide is structured to elicit staff/stakeholder reflections about perceived role(s), descriptions of activities relating specifically to the Angola program, thoughts about difficulties encountered or factors which may have influenced work, and conceptions of program impact. Sex: Role: Please tell me about what role you served on this program or what is your connection to this program? For what duration of time did you work on this program? Activities: Now, I would like to ask you a few questions about OTI’s work in Angola. Please describe how OTI selected the contractor responsible for implementing the program? Please describe the types of activities you are/were involved in. What other offices/organizations (if any) did you work with? Influential Factors: Have you encountered any difficulties/obstacles in your work? If so, describe. Impact: What kind of impact do you think this program has had on promoting a more open and participatory democratic society in Angola? What kind of impact information would you be most interested in reading about in this final evaluation? Social Impact, Inc. OTI/Angola Evaluation Report Page 53 of 76 Appendix 8 – Question Guide 2: CREA/OTI Stakeholders based in Angola Collect demographic and background information: ß Where is your place of birth or where are you from in Angola? ß How long in Luanda? ß Where has this person since the last elections? [Probe for location during war] ß Education level; ß Gender; ß Age (if they wish to disclose this); ß Previous work experience two years before this job – Starting since the year 2000; and ß How did you find out about this job? ß How long working for OTI as Country Representative or CREA as PDO/Logistics Officer? o Interviewing process; o For CREA, explore the nature of OTI’s involvement during recruitment; o At that time, what was your understanding about the CREA-OTI partnership, how was this partnership explained to the respondent? Overall Program Approach and Implementation Process Regional Selection Process: How were the specific regions selected on which OTI decided to focus? I know that this is elaborated on in the OTI assessment, but I need to know what steps (if any) were taken to confirm that these geographic areas were indeed appropriate. CSO/NGO Selection Process: Describe how the groups were selected in the beginning? What methods were used to advertise or promote the program for the purpose of identifying grantees? What were the steps taken to: ß Nurture an idea a potential grantee would come with? ß How much interaction was required to establish details? (Explore inductive versus deductive approach here) ß What was the process to elaborate a project proposal? (Probe definition of PTG) ß What was the process to elaborate a plan of action? ß What was the process to elaborate a budget? ß Any problems encountered in this area? Social Impact, Inc. OTI/Angola Evaluation Report Page 54 of 76 Grant Administration: ß What are the final steps for approving the grants? ß How long does the process take? (confirm the difficulties encountered) ß So now, the grant is approved. What happens after that? (Probe: procurement themes) o What are the procurement systems in place to purchase all items grantees need? o For each step, look at the time interval; o Explore periods of time when procurement is difficult and comparably easier; o When was the system fully functional and operational? o What other problems exist in Angola related to procuring items? Specifically for Country Representative ONLY – this is for institutional memory purposes and gives the general overview Conceiving the partnership between CREA and OTI: What was it like and how was it done? Think back in the very beginning of this program, I would like to understand in a general sense how the CREA-OTI partnership began. How did OTI go about selecting those who formed part of this program (start with the COPs and then anyone else of relevance after that)? Were there any unforeseen difficulties experienced in identifying COPs? Collaboration with PDOs: How would you characterize the working relationship between you and the PDOs? How often would you meet with them? How long would conversations about grantees last? (Probe: explore how approvals and conflicts were communicated?) How participatory was the process of including relevant PDOs into decision-making processes? Overall Reflections about entire program portfolio: Background on grantees; ß Generally, how many individuals form part of one grantee? ß What is the average size grant? ß What is the average duration of the grant? ß How were the funding guidelines developed? (Probe: Who was involved, what was the nature of the projects reviewed, and roughly how long did it take from selection to grant approval and disbursement) ß Describe the method(s) OTI uses to encourage or assist the grantees. ß Describe the collaborative relations OTI has with CREA which you believe may have influenced the implementation of this project component. Describe the type of training the PDOs have received through OTI’s support? Describe any technical assistance you may have given or organized for PDOs? (probe for developing any educational materials) Social Impact, Inc. OTI/Angola Evaluation Report Page 55 of 76 Reflections about Impact: Do you feel like this program has influenced Angolan society to be more democratic and participatory? What kind of impact do you think this program had? Grantee Portfolio: ß How many projects do you have in your portfolio? What proportion of that would be very active? ß Did you ever have to cover for another person (due to illness, vacation time); ß Describe the types of projects you have in your portfolio (advocacy versus civic engagement proportion); ß In what region are your projects located? ß What is the process of monitoring your grantee projects? ß What kind of inputs do they receive (training, technical assistance, networks with others, workshops, forums)? ß If training offered, list the topics normally covered – this will go into the survey: monitoring, responsibilities to fulfill grant, finance and administration, strategy development, communications, etc) ß Describe the dialogue you had with grantees about achieving objectives (starting from formulating strategy and then planning and then any problems encountered). How much time was spent on talking about achieving objectives? ß What were your primary methods of communications with grantees (telephone, email, face to face)? ß Describe the process of how a grantee develops materials for mass distribution? (Probe time intervals for developing educational materials, the number of iterations it took for review, message development process, and any problems encountered); ß How does the program respond to changing needs the grantee may have during the project implementation phase? What kind of reports were grantees expected to provide? What are the steps taken for project close-out or completion? Did you have any grantees who re-applied for a new grant? If so, describe this process Reflections about Impact: Do you feel like this program has influenced Angolan society to be more democratic and participatory? What kind of impact do you think this program had? Other Program Component – Training of PDOs: How participatory was the process of including PDOs/Logistics Officer into decision￾making processes? ß What kind of training did you receive from OTI and when did training take place? ß Who topics covered and who provided? ß Participation Rates: How many (international and local) persons participated in training provided? Social Impact, Inc. OTI/Angola Evaluation Report Page 56 of 76 Appendix 9 – Question Guide 3: Interview Guides for Grantees in Angola ß Introduction and General interviewing protocol: ß Who am I? ß What am I doing in Angola? ß What kinds of questions do I plan to ask – there are no right or wrong answers? and ß Confidentiality clause. Collect demographic information: ß Where is your place of birth? ß Education level; ß Gender; ß Age (if they wish to disclose this); ß Mandate and history of the organization (if any) - Assess group formation process; and ß Previous experience two years before this project. Think back in the beginning when you first became involved in OTI…. ß How did you find out about OTI’s program? Assess how this program ‘fits’ into their life circumstances. ß What is the nature of the grantees’ activities (triangulate with reports too)? ß Characteristics: How many form part of the group? ß History: How much was their grant? Where are they in the disbursement or grant provision process? ß Problems encountered, if any? For grantees working on civic engagement with local authorities…. Collaboration with Local Government: How would you characterize the working relationship between you and the local authorities? How often would you meet with them? How long would conversations about your organization’s mission last? (Probe: explore how grantee presents the program, works with authorities, approvals and conflicts were communicated?) How participatory was the process of including relevant community stakeholders into the decision-making processes? -- fix this question What are some of the constraints or obstacles that influenced your relations with local government authorities? Collaboration with Community: How were the target communities selected? Social Impact, Inc. OTI/Angola Evaluation Report Page 57 of 76 How did you incite their participation? (Probe: explore how grantee presents the program, works with village key players, problems and solutions are communicated and agreed upon?) For instance, how or what activities were used to identify and solve community problems? How would you characterize the working relationship between you and the village stakeholders? How often did you meet with them? How long did the process of nurturing a relationship with the community take? What are some of the constraints or obstacles that influenced your project? What differences in attitudes (if any) among your program participants have you observed? Give example. Which approaches ensured the strongest participation in meetings and public activities among beneficiaries (also including local government stakeholders)? Which approaches resulted in the weakest participation among beneficiaries (also including local government stakeholders)? For grantees engaged in Advocacy: How did you develop the message(s) your organization communicated? What methods were used to communicate the message(s)? What was distributed? And, what results took place afterward, if any? What factors or constraints exist when communicating social messages and/or inspiring participation, and why? Or, did your education or awareness campaign go as planned? What are some of the constraints or obstacles that influenced your project? Where do you expect your organization to be in five years? What are your ambitions for doing advocacy in the future – probe the other area(s) of interest? What differences in attitudes (if any) among those your targeted have you observed? Give example. OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 58 of 76 Appendix 10 – Capacity-Building Survey International NGO 1 Domestic NGO 2 Date: ___________ Interview Number:_______________ Interview Name_________ Region Village _____________ I. Demographic Information 1. Male ___1___ Female ____2_____ (Circle One) 2. Age _________ (number of years) 3. Highest level of education: _________ 4. Marital Status (Circle One) Single 1 Married 2 Divorced 3 Widowed/er 4 5. Number of children: ___________ 6. What kind of organization is the grantee? __1__ Member-based CSO __2__ Community organization __3__ Non-governmental organization __4__ Coalition __5__ Other: Please specify___________________ 7. What is your position in the organization? __1__ CSO/NGO Board Member __2__ Director or Manager __3__ Program Officer __4__ Other: Please specify___________________ II. Inventory of Training Topics OTI Offered 8. Check the areas in which you have received OTI-sponsored training through a workshop or Technical Assistance Provider. (Go down the list and check all that apply) __1__ Accounting and bookkeeping __2__ Procurement and purchasing inputs __3__ Orientation about project and responsibilities of being a grantee __4__ Strategy Development __5__ Project Proposal Development __6__ Budget development __7__ Monitoring and evaluation __8__ Other: specify __________________________ 9. Name the areas in which you have received guidance or informal support from a CREA staff member. (Go down the list and check all that apply) __1__ Accounting and bookkeeping __2__ Procurement and purchasing inputs __3__ Orientation from about project and responsibilities of being a grantee __4__ Strategy development and planning __5__ Project design & proposal writing __6__ Budget development __7__ Program evaluation __8__ Program reporting __9__ Networking with local government __10_ Other: specify __________________________ OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 59 of 76 III. Effectiveness of Training and Skills Application Now, I would like to ask about how useful/important you think your formal training or informal support has been. How would you rank the training in the following categories: Not Important Helpful but not important Very important 10. Financial Management 1 2 3 11. Evaluation 1 2 3 12. OTI Rules/ Regulations 1 2 3 13. Proposal development 1 2 3 14. Project Monitoring 1 2 3 15. Strategy and Analysis 1 2 3 IV. Commitment to Cause and Nature of Cause 16. Did you vote in the last election? __1__ Yes __2__ No 17. If answered no, why not: __1__ I was not old enough __2__ I was outside of the country __3__ I did not participate, because I knew the elections would not result in anything meaningful __4__ I could not register early enough __5__ Other: specify __________________________ 18. Name all the areas in which your organization is presently doing advocacy (mark all that apply: __1__ Land rights __2__ Rights to a free education __3__ Rights of women in the market place __4__ Labor rights for farmers and other groups __5__ Formulation of a new government __6__ Other: specify __________________________ __7__ Other: specify __________________________ 19. Name all the areas in which your organization would like to do advocacy in the future (mark all that apply: __1__ Land rights __2__ Rights to a free education __3__ Rights of women in the market place __4__ Labor rights for farmers and other groups __5__ Formulation of a new government __6__ Other: specify __________________________ 20. Two years ago before participating in this program, do you consider your advocacy cause: __1__ Advanced – positive change observed __2__ Worse off – negative change observed __3__ No observed change in the people OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 60 of 76 V. Organizational Capacity and Capacity to Advocate I am going to read a several statements to you, and I would like for you to tell me if you agree or disagree. (Please circle one response only) 1 2 3 4 5 STRONGL Y Agree Agree Dis￾agree STRONG LY Dis￾agree Does Not Appl y 21 The system of doing needs assessment, getting funding, and project design works well but needs more improvement. 1 2 3 4 5 22 We communicate with the community frequently. 1 2 3 4 5 23 I am committed to helping my community get the rights they deserve, especially if it includes difficult times. 1 2 3 4 5 24 There is no need to change our message(s) since my organizations understands our target community’s mentality and reality. 1 2 3 4 5 25 Including the community on projects is time-consuming and problematic. 1 2 3 4 5 26 I believe that one must be receptive to others’ opinions, but the organizational philosophy does not need to shift. 1 2 3 4 5 27 I have a clear understanding about planning and preparing corresponding budgets. 1 2 3 4 5 28 The community deserves more rights and a better life, but the risks are too great. 1 2 3 4 5 29 The system of doing needs assessment, getting funding, project design, works well and needs little improvement. 1 2 3 4 5 30 The messages advocated should change after understanding better the audience’s mentality and reality. 1 2 3 4 5 31 We maintain new [and old] networks (with local authorities) to increase our ability to advocate. 1 2 3 4 5 OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 61 of 76 32 The system of project design should follow a democratic process inside the organization. 1 2 3 4 5 33 The system of project design is done in a participatory manner. 1 2 3 4 5 34 Partnering with the community is a simple and smooth process. 1 2 3 4 5 OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 62 of 76 Appendix 10, cont. – Capacity-Building Survey: Portuguese Version Data: ___________ N.º da entrevista:_______________ Nome do entrevistado_________ Aldeia, região _____________ I. Informação Demográfica 1. Masculino ___1___ Feminino ____2_____ (Faça um círculo) 2. Idade _________ (número de anos) 3. Nível mais alto das habilitações: _________ 5. Estado Civil (Faça um círculo) Solteiro/a 1 Casado/a 2 Divorciado/a 3 Viúvo/viúva 4 5. Número de filhos: ___________ 6. Que espécie de organização é a donatária? __1__ Membro de base da OSC __2__ Organização da Comunidade __3__ Organização Não Governamental __4__ Coligação __5__ Outra: Por favor especifique__________________ 7. Qual o seu posto na organização? __1__ Membro da Administração da OSC/ONG __2__ Director ou Gestor __3__ Oficial de Programas __4__ Outro: Por favor especifique:___________________ II. Inventário dos tópicos de formação ministrado pela OTI 7. Assinale as áreas em que recebeu formação financiada pela OTI através de workshop ou de Assistência técnica. (Assinale na lista que se segue) __1__ Contabilidade e escrituração __2__ Compras e mecanismos de aquisições __3__ Orientação do e acerca do projecto e responsabilidade em ser um donatário __4__ Desenvolvimento Estratégico __5__ Desenvolvimento de Proposta de Projecto __6__ Desenvolvimento do orçamento __7__ Monitoria e Avaliação __8__ Outro: Especifique __________________________ 8. Nome das áreas sobre as quais recebeu orientação ou apoio informal da equipa do CREA. (Assinale na lista que se segue ) __1__ Contabilidade e Escrituração __2__ Compras e Mecanismos de Aquisição __3__ Orientação acerca do projecto e responsabilidades de ser um donatário __4__ Estratégia de Desenvolvimento e Planificação __5__ Desenho do Projecto e formulação de propostas __6__ Desenvolvimento do orçamento __7__ Avaliação do Programa OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 63 of 76 __8__ Relatório do Programa __9__ Relacionamento com o governo local __10_ Outro: Especifique __________________________ III. Eficácia da formação e Aplicação das Habilidades Agora, gostaria de lhe perguntar como é que pensa quanto a sua formação/treinamento foi útil e importante… Responda tendo em conta também se a formação/treinamento recebido representou para si mais oportunidades formais ou informais. Como classificaria a formação dentro das seguintes categories: Não Importante Proveitosa mas não importante Muito importante 9. Gestão Financeira 1 2 3 10. Avaliação 1 2 3 11. Regras e Regulamento da OTI 1 2 3 12. Propostas de Desenvolvi￾mento 1 2 3 13. Monitoria de Projectos 1 2 3 14. Estratégia e Análise 1 2 3 IV. Comprometimento na Causa e natureza da Causa 14. Você votou nas últimas eleições? __1__ Sim __2__ Não 15. Se respondeu não, porquê: __1__ Não tinha idade suficiente __2__ Estava fora do País __3__ Não participei porque sabia que as eleições não iam resultar em nada significativo. __4__ Não me consegui registar a tempo __5__ Outra: especifique __________________________ 16. Assinale as áreas em que a sua organização está presentemente a fazer advocacia. __1__ Direitos da Terra __2__ Direito a educação gratuita __3__ Direitos da mulher nos mercados __4__ Direitos do Trabalho para camponeses e outros grupos __5__ Formulação de um novo governo __6__ Outro: especifique __________________________ __7__ Outro: especifique __________________________ 17. Quem meios utilizou para fazer a advocacia? (Por favor assinale com um círculo) 1 Palestras formais 2 Encontros informais 3 Afixação de cartazes 4 Rádio 5 Outro (especifique):__________________ 6 Outro(especifique):___________________ OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 64 of 76 18. A quem foi feita a advocacia? (Por favor assinale com um círculo) 1 Mulheres 2 Camponeses 3 Jovens do meio rural 4 Jovens do meio urbano 6 Outro (especifique)__________________ 7 Outro (especifique) 19.Assinale com um círculo as áreas em que a sua organização gostaria de trabalhar futuramente __1__ Direitos da Terra __2__ Direito à educação gratuita __3__ Direitos da mulher nos mercados __4__ Direitos do Trabalho para camponeses e outros grupos __5__ Formulação de um novo governo __6__ Outro: especifique __________________________ 20. Dois anos antes de participar neste programa, considerava que a sua advocacia causaria: __1__ Progresso/Avanço – iriam ser observadas mudanças positivas __2__ Deterioração/ Piorar – iriam ser observadas mudanças negativas __3__ Não se iriam observar mudanças nas pessoas OTI Survey for Capacity Building OTI/Angola Evaluation Report Page 65 of 76 V. Capacidade da Organização, Capacidade para Advogar, e Participação Vou ler uma série de afirmações e gostaria que me dissesse se concorda ou não. (Por favor, assinale com um círculo apenas ,cada afirmação) 1 2 3 4 Con-cordo MUITO Con-cordoDis-cordo Dis￾cordo MUITOApplica 21 O sistema de análise das necessidades, o desenho dos projectos, e a concessão de fundos trabalham bem mas necessitam de mais melhoramentos. 1 2 3 4 22 A comunicaçao com a comunidade é frequente. 1 2 3 4 23 Sou delegado para ajudar a minha comunidade a gozar dos seus direitos, especialmente se isso inclui , riscos e tempos difíceis. 1 2 3 4 24 Nao ha necessidade de mudar as nossas mensagens desde que a minha organizacao perceba bem a mentalidade e a realidade da populacao alvo. 1 2 3 4 25 Incluir a comunidade nos projectos cria confusão e perde-se muito tempo. 1 2 3 4 26 Creio que se deve ser receptivo as opinioes das pessoas mas a filosofia da organizacao nao precisa de mudar. 1 2 3 4 27 Tenho uma clara compreensão sobre planear as atividades e preparar orçamentos que correspondam aos planos. 1 2 3 4 28A comunidade merece mais direitos e uma melhor vida, mas o risco de ajudá-la é muito grande. 1 2 3 4 29 O sistema de análise das necessidades, desenho dos projectos, obtenção de fundos trabalha bem e nao necessita grandes melhoramentos. 1 2 3 4 30As mensagens que se transmitem nas accoes de advocacia devem mudar depois de percebermos melhor a realidade e a mentalidade da nossa populacao alvo. 1 2 3 4 31 As relações novas (e antigas) com autoridades locais sao muito importantes para melhorar a nossa habilidade na advocacia. 1 2 3 4 32 O desenho dos projectos deve seguir um processo democrático dentro das organizacoes. 1 2 3 4 33 O sistema de desenho dos projectos é feito numa maneira participative. 1 2 3 4 34 Fazer parceria com a comunidade nos projectos é um processo simples e tranquilo. 1 2 3 4 Social Impact, Inc. OTI/Angola Evaluation Report Page 66 of 76 Appendix 11 – Sampling The qualitative sample included a practical census of all those who worked on OTI/Angola program in DC, including (1) OTI/Washington DC staff (CTOs, Africa Bureau, and those involved in OTI assessment), and (2) CAII) in Washington (program assistance and support, first Chief of Party, and Vice￾President of Programs). Upon arrival in Angola, the evaluation consultant met on two separate occasions with the OTI/Angola Country Representative and/or DG Team Leader to discuss the nature of each program. Moreover, the evaluation team met with staff from CAII in Angola (for a brief three-hour period) otherwise known as CREA (Logistics Officer, Program Development Officer, and second Chief of Party), and program grantees. Essentially, the qualitative interviews with staff persons linked to the program could be characterized as convenient.15 The sampling for qualitative interviews with grantees followed a slightly different arrangement. Specifically, it was estimated that between ten to fifteen projects would be purposefully sampled, using the following criteria or characteristics: · CSOs/NGOs by region based on the percent of the entire portfolio; · CSOs/NGOs by objective; · Proportion of the CSOs/NGOs working in DG areas; · One CSO/NGO which did not receive any funding or a negligible amount of funding from OTI (to test negative case analysis), time permitting; · Smaller CSOs/NGOs which would not normally have access to resources – those who OTI took a chance on; · CSOs/NGOs which would normally have access to resources (of larger size); · CSOs/NGOs funded in the earlier grant-making stages; · CSOs/NGOs funded in the latter grant-making stages; and · Any others which the Country Representative felt may be of importance and would add to the analysis. All of the aforementioned criteria were met in the final grantee sample. With the aid of the database, the Evaluation Consultant generated a sample of twenty projects, of which two were not projects but rather funded activities (i.e., radio free play and strategic planning conference)16 . The Country Representative suggested three additional projects to the list which she believed were important. Thus, the qualitative sample with grantees could be described as stratified (according to the time period when grant was cleared), non-random. There were only three cases, whereby the evaluation team interviewed a grantee about a project which was not on the sample list, in which case that project replaced one already on the list. A total of seventeen projects and fourteen grantees comprised the entire sample. It should be noted that some grantees were funded multiple times, and so it is for this reason that the number of grantees and projects are not identical. (See table below) 15 Convenience sampling is a method of sampling used and recognized frequently. 16 Specifically, the grant-making calendar was divided into three periods (ranging from three to four months) to reflect projects supported in the beginning, middle, and end of OTI’s program. The database also separated the grants by region and program objective. Each project’s number or code was then recorded onto a small piece of paper, and the random selection took place thereafter. Social Impact, Inc. OTI/Angola Evaluation Report Page 67 of 76 The survey sample consists of 35 persons, all of whom are grantees. The procedure may be characterized as convenient based on the stratified, non-random sample. That is, all those available for grantee interviews were surveyed, and such persons ranged from support staff to project coordinators to directors. Prior to initiating data collection, the Evaluation Consultant provided orientation to the national evaluator who also formed part of the team. This brief but warranted preparation covered the following topics, a) introduction to program evaluation, b) purpose and objectives of the evaluation, c) the interview guide (including the nature of information to be solicited), d) the survey and question sequence, e) interviewing techniques, and f) data entry and coding. Survey translation also took place during this stage. Appendix 12 – Evaluation and Corresponding Data Sources/Methodologies Table 3: Evaluation and Corresponding Data Sources/Methodologies Data Sources/Methodology Evaluation Objectives Interview s Focus Groups Organizational Literature & Records/data base Survey To what extent did OTI/Angola’s program meet its stated goal and objectives? ¸ ¸ How did the management and operation of the program contribute to or detract from achievement of the program goal and objectives? ¸ Based on the evaluation findings, what are the lessons learned and ways OTI can improve its programs? Thematic Area #1: Examining the extent to which CSO or/and NGO’s capacity to advocate on key issues has increased. ¸ ¸ Thematic Area #2: Exploring the degree to which the CSO/NGO’s ability to promote participation between citizens and local authorities to address community problems. ¸ This evaluation embodies multiple methods, and so the following strategies were incorporated to assure data reliability and validity: 1. Two sampling procedures to draw two distinct samples: 2. Question guides for interviews and focus groups; Social Impact, Inc. OTI/Angola Evaluation Report Page 68 of 76 3. One survey; 4. Specific protocols that were aligned to varying methods, (i.e., interviews and focus groups); and 5. Divergent data analysis for results. The survey implemented included attitudinal statements, using a four-point Likert Scale ranging from Strongly Agree to Strongly Disagree. One should note that qualitative information was also used to develop some ‘case studies’ or vignettes delineating results the CSO/NGOs achieved. Table 3 illustrates which methodology was used to respond to evaluation objectives. Social Impact, Inc. OTI/Angola Evaluation Report Page 69 of 76 Appendix 13 – Instruments and Interviewing Protocol Two slightly different question guides were used to elicit information from stakeholders. For instance, for those based in Washington DC, the question guide obtained information about stakeholders’ perceived role(s) in the OTI program, the activities in which they were involved, thoughts about difficulties encountered or factors influencing the respondent’s work, conceptions of programs impact, and reflections of what type of evaluation information would be most informative. Those connected to the OTI program in Africa, however, were asked more detailed questions about the overall program approach and implementation process (i.e., region and grantee selection process, the grant-making cycle, grant administration, portfolio analysis, interaction between OTI/CREA). The Evaluation Consultant also asked the OTI Country Representative for detailed thoughts about managing the DG portfolio. Overall, the evaluation questions asked were very simple and process-oriented. As for grantees, the team asked how they heard about CREA, the grant application process, the type of interaction they had with CREA staff, their attitudes about the relevance/utilization of [informal] training (if applicable), their perspectives about the nature of CREA’s service delivery, their experience(s) with the OTI supported project, the perceived impact of their work performed, and future plans for advocacy work. All questions were open-ended, and interviews were in Portuguese. Discussions with a few CREA staff revealed that a grantee may have exposure to technical assistance or training, albeit informally, in the areas of proposal development, budgets, activities planning, report writing, evaluation, and networking with local officials. Moreover, CREA staff elucidated that most grantees had strong understanding of problem identification and how to go about eliciting community participation. Based on the information gathered the evaluation team developed the survey measuring the concept of “capacity”. Capacity under these circumstances included 1) managing the organization (i.e., mission statement development, strategic planning, and fundraising), but most importantly 2) advocating or transmitting a message [to a target population] with the hopes of inspiring action. The evaluation team also consulted literature and one source with established instruments measuring organizational capacity building.17 First, the evaluation consultant developed the survey in English, and then the instrument underwent translation. The translation process used is called a modified ‘simple direct translation’, whereby both evaluators versed in the Portuguese and English languages developed the instrument, taking into account the precise information sought from grantees.18 The survey had five substantive parts, 1. The first parts included questions requesting demographic information; 17 Capacity Assessment Tool used by Family Health International (FHI). 18 Behling, Orlando and Law, Kenneth. Translating Questionnaires and other Research Instruments: Problems and Solutions. SAGE Quantitative Applications in Social Sciences Series (QASS). Thousand Oaks, CA, Number 07- 133, pg. 18. Social Impact, Inc. OTI/Angola Evaluation Report Page 70 of 76 2. The second and third parts embodied questions with open and closed response formats, focusing on organizational attributes, position descriptions, and an inventory of [informal] training topics CREA [may have] offered; 3. The fourth section also presented six themes (which may have been selected in the previous part), and asked respondents to rank their relevance. Additional questions inquired about the current and future areas in which the grantee was advocating as well as corresponding target populations; and 4. The last section intended to measure six distinct attitudinal areas as well as three smaller themes intended for information purposes only. (See appendix 9 for the questionnaire.) Social Impact, Inc. OTI/Angola Evaluation Report Page 71 of 76 Appendix 14 – Evaluation Limitations Sampling: The CREA staff’s availability for in-depth interviews was limited and during the three hours, all three persons refused to be tape recorded which may have compromised the quality of data collected. Specifically, the evaluation team was unable to understand fully the program implementation process, and the implication of this information gap is such that when interviewing grantees, the team was only able to discern pieces of data about the grant administration system. That is, without the staff reflections, it was not possible to glue or link outcomes to activities. For instance, the CREA staff would normally offer the inner-workings of the in-kind grant administrative process, and without such detailed data it was not possible to link grantee complaint/ praises about distinct aspects of the in-kind methodology to a specific part of the administrative system. Thus, the complaint/praise (outcome) could not be associated with specifics within CREA’s system (activities). Instrument Development: The quantitative instrument has two limitations. First, the survey could not be legitimately pre-tested, but in every case the evaluation team closely observed closely in every case which questions presented respondents with the most difficulties (in terms of Portuguese language comprehension or question format). Therefore, each time the survey was administered, the evaluation team built on the experiences from the last respondents. Also, those surveyed were consistently asked if everything was clear or if any question required more explaining. Second, measuring the concept of “CSO/NGO capacity to advocate” in and outside of OTI is novel and experimental. At the beginning of this assignment, the evaluation consultant conducted a wide search of questionnaires assessing skills development within the realm of political belief system(s). Time restrictions combined with the need to develop a tailored survey prevented the consultant from locating surveys developed in the past, and it is for this reason that only fourteen attitudinal statements (in the last section of the survey) were used to measure so many thematic areas. It is recognized that the number of statements used is insufficient to measure accurately the six distinct attitudinal domains, and so any results should be interpreted with caution, observing the information as a plausible trend rather than a theory. While some of the attitudinal data is presented in this report, the results are unreliable and presented with caution – see section entitled “Measurement”. Given this initial attempt at designing an OTI - specific attitudinal assessment tool, the survey statements could be altered and used again in the future (see OTI’s toolbox for more details). Interviewing: Given the difficulties CREA experienced while administering grants, most of the grantees (14 out of 17) interviewed wanted to inform the team of their complaints and dissatisfaction(s). Although this aspect did not fall within the scope of the evaluation, the team had to listen to the information provided. This It should be noted that the time the grantee spent elaborating on his/her disapproval of CREA a) created a high volume of static, distracting the team from focusing on impact and b) consumed a considerable amount of time which could have been devoted to examining impact in greater depth. Social Impact, Inc. OTI/Angola Evaluation Report Page 72 of 76 The goal was to find coherence in impact data in as many places as possible, but due to data variability (from the grantee’s need to express frustration(s) about CREA’s performance); data quality is not as high as it could have been. Measurement: The question format was relatively new for some respondents and, after measuring the margin of error in attitudinal questions, it was discovered that some questions were invalid. A Chronbach alpha test was conducted – which measures the true score plus the margin of error in attitudinal statements – and the results from this test (far less than 0.60 which is an acceptable level in social sciences) indicated that the error margin was too high to consider the attitudinal questions valid. It is recognized that some of this error margin could be attributed to the small sample size (n=35), thereby contributing to the variability observed. In addition, the question format was relatively new for respondents, but great care was used during survey execution. It is also speculated that a proportion of the variability could be explained by a wavering sense of commitment among grantees as well as their (in)ability to work with communities. This possibility was also observed through triangulation in qualitative data. Section 7 presents more about this phenomenon. Program Financial Summary: The database used to record and document the amount(s) issued to grantees may not be considered an actual reflection of grants disbursed. Given these slight discrepancies, the figures presented in this report should be interpreted with caution. However, the timing when grants were cleared and complete d, however, were consistent with qualitative data collected. Program Management: Because management issues were examined in a separate external review, this evaluation focused on one particular aspect of program management: the Country Representative’s dual role as the OTI Representative and DG Team Leader.