FINAL REPORT FINAL EVALUATION OF OTI’S PROGRAM IN SERBIA — MONTENEGRO Prepared by: Thomas J. Cook, Ph.D. Ivo Spalatin Assisted by: Frank Schultz Submitted by: Development Associates, Inc. 1730 N. Lynn Street Arlington, VA 22209 Email: devassoc1@devassoc1.com December 31, 2002 Final Evaluation of OTI’s Program i December 31, 2002 in Serbia - Montenegro TABLE OF CONTENTS List of Acronyms Introduction..........................................................................................................................1 Section I. Political Backdrop of the Evaluation ..............................................................1 Section II: The Objectives, Issues and Methodology of the Evaluation.........................3 A. The Objective of the Evaluation ....................................................................3 B. Evaluation Issues ...........................................................................................4 C. Evaluation Methodology................................................................................5 Section III: The Nature and Extent of the OTI Program ...............................................6 A. OTI Country Program Management ..............................................................6 B. OTI Grant and Expenditure Profile................................................................7 Section IV. Findings on Issues Raised by the Scope of Work.........................................8 A. The ability of OTI/Serbia and Montenegro to meet stated program objectives.........................................................................................8 B. The ability of OTI to change its strategic objectives and activities in response to evolving political environments. ...........................................9 C. The applicability and appropriateness of OTI's Programmatic design for Serbia and Montenegro as a means to effect change ..............................13 D. OTI's community development activities, with particular attention to the conflict mitigation strategy in the contentious Presevo Valley. .........16 E. The impact of OTI's work with media groups ...............................................18 F. OTI's effectiveness in developing media campaigns to hasten the pace of reform...............................................................................................19 G. Election-related activities leading up to the overthrow of the Milosevic regime ...........................................................................................19 H. The impact of OTI's work With NGO groups. ..............................................20 Section V: Conclusions of The Evaluation........................................................................20 A. Local media....................................................................................................20 B. NGO groups...................................................................................................21 C. Community Development Projects (CDP).....................................................21 D. Conflict mitigation efforts..............................................................................22 E. Elections-Related Activities...........................................................................22 F. 'Handover Strategies' and relations with the USAID Mission, other donors and partner..........................................................................................23 G. OTI management and administration.............................................................25 Section VI. Recommendations...........................................................................................25 A. General Recommendations ............................................................................25 B. Recommendations for management and administrative structures ...............28 Section VII: Lessons Learned............................................................................................29 Development Associates, Inc. Final Evaluation of OTI’s Program ii December 31, 2002 in Serbia - Montenegro ANNEXES: Annex A People/Organizations Contacted .....................................................................A-1 Annex B List of References............................................................................................B-1 Annex C Data Collection Protocol .................................................................................C-1 Annex D Scope of Work.................................................................................................D-1 Annex E In-Country Data Sources .................................................................................E-1 Final Evaluation of OTI’s Program December 31, 2002 in Serbia - Montenegro LIST OF ACRONYMS AKCIJA A nationwide coalition of NGO’s, Independent Media (TV and radio), technical experts, and educators who work together to promote democratic and economic reform in Montenegro ANEM Association of Independent Media in Serbia CDP Community Development Project CDT Center for Democratic Transition CeSID Centar za Slobodne lzbore I Demokratija (party) CHF Cooperative Housing Foundation International CIP Community Improvement Program CR US/OTI Country Representatives CRDA Community Revitalization through Democratic Action Program DART OFDA’s Disaster Assistance Response Teams DOS Democratic Opposition of Serbia (party) DTI Democratic Transition Initiative GSZ Ground Safety Zone IREX Serbian Independent Media Company IRI International Republican Institute NDI National Democratic Institute for International Affairs OSCE Office for Security and Co-operation in Europe OTI USAID’s Office of Transition Initiatives OTPOR Serbian for “Resistance”, Student Resistance Organization PDO OTI Program Development Officer POR OTI Pace of Reform Program SOW Evaluation Scope of Work Final Evaluation of OTI’s Program 1 December 31, 2002 in Serbia - Montenegro FINAL EVALUATION OF OTI’S PROGRAM IN SERBIA - MONTENEGRO INTRODUCTION This is the final evaluation of the activities of USAID's Office of Transition Initiatives (OTI) in Serbia-Montenegro during the period 1997 to 2002. The evaluation is divided into seven sections:  Section I briefly introduces the political background of the events that occurred in the former Yugoslavia, including the advent of the OTI program.  Section II explains the objectives, issues and methodology of the evaluation by Development Associates.  Section III describes the nature and extent of the OTI program, including the profile of its grants and expenditures.  Section IV provides the findings of the evaluation team on the issues raised by the evaluation scope of work.  Section V narrates the conclusions reached in the evaluation.  Section VI contains the recommendations of the evaluation team on future activities.  Section VII suggests several "lessons learned" from the OTI experience in Serbia￾Montenegro. The body of the evaluation report is followed by an Annex that contains a list of the persons interviewed for the assessment. SECTION I. POLITICAL BACKDROP OF THE EVALUATION The 1980 death of Josip Tito, the charismatic leader and unifier of Yugoslavia, signaled the initial disintegration of the most liberal communist party and country in Europe. A variety of efforts within and outside of Yugoslavia sought to preserve the unifying force represented by nearly forty years of rule by Tito. All ultimately failed. The center of Yugoslavia was slowly breaking down, international and local factors were changing, some players disappearing, new ones emerging, except for one who remained active on the center stage, Slobodan Milosevic. For more than a decade from his rise to power in Serbia in 1987, Milosevic never ceased his consolidation and expansion of political power by internal repression and external use of some of the most destructive domination since WW II. Milosevic used the Yugoslav Army (consisting primarily of Serbian generals and soldiers), Serbian paramilitary forces, and his infamous "black shirt" guards to implement the creation of Development Associates, Inc. Final Evaluation of OTI’s Program 2 December 31, 2002 in Serbia - Montenegro "Greater Serbia". As he said: "The borders are always dictated by the strongest. We consider that it is a legitimate right for the Serb people to live in one country. That is the beginning and the end. And if we need to fight for that, God is my witness that we will!" Subsequent Serbian invasions from 1991 to 1999 of Slovenia, Croatia, Bosnia-Herzegovina, and even in Serbia proper (i.e. Kosovo) became the mechanism to move and separate the various ethnic groups in Yugoslavia. Milosevic and his regime unleashed concentration camps, the over three and a half year siege of Sarajevo- the longest siege in modern history – massacres, organized plundering and destruction of whole cities and villages, the raping of women, and the destruction of hundreds of cultural, historical, and religious sites; furthermore, the Milosevic regime initiated and conducted three wars in former Yugoslavia followed by the final act in 1999 of attacking nearly one million Albanians in Kosovo, Serbia. These Milosevic-led efforts culminated in over 3.8 million refugees, 300,000 deaths, and the use of such destructive and inhumane methods of violence that finally brought on a UN Protectorate established in Kosovo in 1999. In an attempt to capitalize on the Serbian nationalism generated by the 1999 NATO bombing in Serbia, Milosevic called for special elections for fall 2000, with every expectation that he would prevail again. The following three major factors, however, explain the surprising and extraordinary defeat of Milosevic in the fall 2000 elections: 1. Pent-up frustration by the Serbian people over the military losses of thousands of Serb soldiers within and outside of Serbia; 2. The economic deterioration and bankruptcy within Serbia caused by the failed wars of the 1990s; and 3. OTI assistance to action-orientated groups within Serbia, including to media, opposition groups and NGOs. Just six months later in June 2001, another “surprise” jolted Serbia. The newly elected democratic government transferred Milosevic to the War Crimes Tribunal in the Hague, where he is currently being tried for some 50 counts of ethnic cleansing and war crimes. USAID's Office of Transition Initiatives began operations in Serbia and Montenergro in July 1997. In its initial phase OTI established a central office in Belgrade with field offices in Nis, Podgorica, and Pristina. It began its work with local organizations that were not controlled by, and were often in opposition to, the existing regime, and provided them with certain types of technical support and financial assistance through a grants process. Against a wartime backdrop, it is obvious that the period 1997 to 2000 was a dangerous period for OTI staff and grantees. The Milosevic regime and its supporters viewed OTI as part of the ‘enemy opposition.’ OTI staff members were followed by security police, telephones were tapped and some were arrested. Police tried to hire staff as informants. As Ray Jennings, the first OTI Country Representative stated, “…it was a very hostile environment... the OTI national staff were outstanding....brave...courageous....fearless.”1 Routine threats and intimidation created an environment that demanded both staff competence and courage. 1 Ray Jennings, Interview, October 2002. Development Associates, Inc. Final Evaluation of OTI’s Program 3 December 31, 2002 in Serbia - Montenegro Grantees also had to be wary and discreet. Grantees, many were NGOs, were labeled “enemy collaborators” and their receipt of an OTI grant could easily incite police harassment. Danica Stefanovic, an OTI grantee in the Novi Sad area, was arrested five times in 2000 by the local police. Some potential grantees even declined OTI support because it was too dangerous and police would interfere. Program records and documents were closely guarded to protect grantee identity. Staff often met with grantees surreptitiously; telephone contact was infrequent and coded. The Regime’s hostility reportedly was most menacing in the areas outside of Belgrade, which it viewed as its heartland.2 According to program staff, OTI’s de facto goal during this hazardous time was obvious, namely, to do everything possible to support the opposition. This was the unequivocal focus of the OTI staff, and remained so, as a result of Milosevic’s increasingly visible cruelty. The program provided timely support to effective change agents such as NGOs, independent media, and opposition political parties engaged in nonviolent democratic methods, such as marches and rallys attracting over one million participants, and resulting in a dramatic change from authoritarianism to democracy. Commenting on OTI’s early years and the extraordinary bravery and courage of the local Serbian staff, Ray Jennings, OTI’s first Country Director from 1997 to 1999, described a ‘most hostile’ environment confronting those individuals, and noted with deep pride, respect and appreciation that “these local people who stayed with us from the beginning and who took most of the pressure and risk enabled us not to be exploited by Milosevic’s people.” SECTION II. THE OBJECTIVES, ISSUES AND METHODOLOGY OF THE EVALUATION This section provides a three-fold description of the evaluation assignment, as follows: l) the objective of the evaluation, 2) the issues addressed by the Development Associates' evaluation team and 3) the methodologies employed in gathering information. A. THE OBJECTIVE OF THE EVALUATION This evaluation is the first comprehensive external assessment of the OTI program in Serbia/Montenegro. It examined the full sweep of the program 1997 – 2002, with emphasis on the post-Milosevic period from 2000 to 2002. On direction from OTI/Washington3 , the team allocated approximately 25% of its effort to the 1997 – 2000 periods and 75% to the 2000 – 2002 periods. 2 Observers of Serbian Politics interviewed noted that, especially since the 1996 election, Milosevic viewed Belgrade as hostile territory. He lost the Belgrade vote in these elections and the opposition forces originated from there. 3 Sara Brewer, interview, September 9, 2002, OTI/Washington Office. “More people are now interested in what OTI is doing. (OTI) did not do a lot of monitoring in the early years. – Want to see if appropriate people on the ground had been assisted....” Sara Brewer, OTI Washington, September 10, 2002 Development Associates, Inc. Final Evaluation of OTI’s Program 4 December 31, 2002 in Serbia - Montenegro OTI/Washington charged the evaluation team to document the impacts of the program, to identify the lessons learned that might benefit other OTI programs, and to make programmatic recommendations for program improvement. This was an external evaluation conducted by consultants under contract to Development Associates, Inc.. B. EVALUATION ISSUES The Evaluation Scope of Work (SOW) specified an ambitious list of issues for the evaluation team. The SOW mandated an assessment of OTI’s past accomplishments and the identification of lessons learned and the submission of recommendations for the design and implementation of OTI programs in other countries. These issues guided the conduct of the evaluation and the preparation of this report. Below are the three categories of issues – Impacts, Programmatic Lessons, and Recommendations – that the Evaluation Team was asked to address: The Impacts of the OTI Program: 1. The ability of OTI/Serbia and Montenegro to meet stated program objectives. 2. The ability of OTI to change its strategic objectives in response to evolving political environments. 3. The ability of OTI to change its activities in response to evolving political environments. 4. The applicability and appropriateness of OTI's programmatic design for Serbia and Montenegro as a means to effect change. 5. OTI’s community development activities, with particular attention to the conflict mitigation strategy in the contentious Presevo Valley. 6. OTI’s effectiveness in developing media campaigns to hasten the pace of reform in several key sectors, following the overthrow of the Milosevic regime. 7. Harnessing NGO efforts to hasten the pace of reform in several key sectors, following the overthrow of the Milosevic regime. 8. Election-related activities leading up to the overthrow of the Milosevic regime. 9. The impact of OTI’s work with NGO groups leading up to the overthrow of the Milosevic regime. 10. The impact of OTI’s work with media groups leading up to the overthrow of the Milosevic regime. Programmatic Lessons from the OTI Program: 1. With local media; 2. With NGO groups; 3. In community development projects; 4. In conflict mitigation efforts; 5. In elections-related activities; 6. Regarding ‘handover strategies’; 7. Regarding relations with the USAID Mission, other donors and partnership; 8. Regarding OTI management and administrative lessons, particularly in relation to OTI’s; ability to meet its programmatic goal. Development Associates, Inc. Final Evaluation of OTI’s Program 5 December 31, 2002 in Serbia - Montenegro Recommendations from the OTI Program: 1. Programmatic recommendations for OTI’s on-going work in other countries in transition. 2. Recommendations for management and administrative structures: a. Procurement mechanisms b. Record-keeping c. Use of the OTI database C. EVALUATION METHODOLOGY Table I summarizes the variety of data sources used in the evaluation. The Team collected these data through individual interviews, small group interviews and data retrieval from OTI data bases. It developed and pre-tested a detailed data collection protocol (see Appendix) for the field work. The protocol covered all of the SOW issues. The Team also visited OTI grantee sites to observe examples of project results, such as a school renovation under the Community Improvement Program (CIP), or NGO development under the OTI Pace of Reform (POR) Program. These visits to villages and communities in and around the Ground SafetyZone (GSZ) were particularly useful for the Southern Serbia data collection. Team members were able to talk with Serbian and Albanian citizens, so as to get their views on the changes that have occurred in this region especially since 2000. Through these conversations they also were able to gain at least a partial sense of the tensions and hostility that OTI staff had to cope with in their attempts to promote a peaceful transition in the region. TABLE 1 OTI Evaluation Data Collection Number of Organizations/Offices Contacted Number of People Interviewed Other Data Sources Used in the Evaluation Serbia 39 65 Montenegro 11 14 Washington, DC 4 13 Totals 54 92 Data Bases: OTI/Belgrade Data Base, OTI/Belgrade Baseline Surveys, OTI/Belgrade Pace of Reform Data Base, Program Documents: OTI/DC; OTI/Belgrade Web Sites: OTI/DC; OTPOR, OTI Serbia/Montenegro, RONCO, Creative Associates Development Associates, Inc. Final Evaluation of OTI’s Program 6 December 31, 2002 in Serbia - Montenegro SECTION III. THE NATURE AND EXTENT OF THE OTI PROGRAM This section describes the OTI program in Serbia with emphasis on country program management and its grant and expenditure profile. A. OTI COUNTRY PROGRAM MANAGEMENT OTI has endured disruptions and discontinuity from 1997 – 2002. It has operated in different locations at different times and under different management leadership, as noted in Table 2. Names with * refer to ‘Acting OTI Country Representative.’ TABLE 2 Time Period Program Location OTI Country Representative Implementing Partner 9/97 – 3/99 Belgrade Ray Jennings 3/99 – 6/99 Budapest (NATO Bombing; Suspended OTI Activities) Ray Jennings* 7/99- 12/99 Macedonia Jeanne Bourgault* 12/99 – 11/00 Montenegro (4 mos.) Bosnia (4 mos.) Budapest (3 mos.) Patrick Wingate Jeanne Bourgault* RONCO 11/00 – 5/01 Belgrade Ray Jennings* David Costello* John Penn* Erin Miller* 5/01 – 10/02 Belgrade Paul Randolph Creative Associates The seventy day NATO bombing in 1999 triggered the evacuation of OTI expatiate staff to Budapest and to a suspension of normal program operations. The program subsequently relocated four more times, from July 1999 to November 2000, before returning to Belgrade. It is noteworthy that OTI’s local Serbian Program Development Officers (PDOs) and staff kept the program running during this seventeen month interval. According to OTI staff, they managed to continue the grant making process, so that there was not an interruption in support to the opposition. It should also be noted that OTI/Belgrade had three U.S. Country Representatives (CRs) and six acting CRs during its five year tenure. It lacked a permanent CR from November 2000 until May 2001, the seven month period immediately following Milosevic’s sacking, an opportune time to establish a foothold for reform. It also replaced its implementing partner – Creative Associates took over from RONCO - in November 2000. According to Albert Cevallos, former OTI Balkans Program Manager, there was considerable confusion and chaos during the transfer of responsibilities from RONCO to Creative Associates. The intimidating atmosphere of the RONCO tenure meant that record keeping was irregular and was shielded to minimize opportunities for Regime harassment. OTI staff reported that there was a limited and elusive paper trail documenting exactly what decisions were taken, an account of program expenditures and the completion status of project activities. As a result, Creative Development Associates, Inc. Final Evaluation of OTI’s Program 7 December 31, 2002 in Serbia - Montenegro Associates, in many respects, had to start from scratch in setting up to manage the program, while at the same time attempting to keep going what at the time was a somewhat vague collection of grantee activities. B. OTI GRANT AND EXPENDITURE PROFILE Table 3 provides an overview of the OTI expenditures from 1997 – 2002. The data were more complete for 2000 – 2002. Table 4 presents the 2000 – 2002 profile, with additional data on the matching contributions provided by the Serbia and Montenegro partners. TABLE 3 OTI Grant and Expenditure Profile: 1997 – 2002 ** Source: Paul Randolph, OTI/Belgrade office, September 2002 The most striking fact in Table 3 is that, while the number of grants remained about the same, there was a significant increase (62%) in the average size of grants in the post-Milosevic period. This is due, in part, to the onset of the CIP activity in 2001, with its relatively costly infrastructure investments; it also reflects the considerable effort expended to accelerate the pace of reform in OTI’s final year. TABLE 4 Total USAID/OTI Grant Activity in Serbia and Montenegro, October 2000 to October 2002 Program Category Activities Approved Activities Completed USAID/OTI Contribution Matching Contribution Total USD Civil Society Org. Support 36 36 $873,681 $181,302 $1,054,983 Community Impact Program (CIP) Activities 177 164 $5,084,125 $5,264,357 $10,348,482 Pre￾Milosevic period: July 1997 – October 2000 * Average Grant Size Pre￾Period Post￾Milosevic Period: October 2000 – September 2002 ** Average Grant Size Post￾Period % Change in Total Amounts between Pre and Post Milosevic Periods Totals OTI Expenditures 1997 – 2002 Number of Grants 631 633 .003% 1264 Total Expenditures $8,675,565 $13,749 $14,051,608 $22,198 62% $22,727,173 * Source: Nada Ilic – Creative Associates procurement staff member and former RONCO procurement staff member for the 1997 – 2000 time periods. Data from RONCO Montenegro and Serbia Completed PTG Report, January 24, 2001 Development Associates, Inc. Final Evaluation of OTI’s Program 8 December 31, 2002 in Serbia - Montenegro Program Category Activities Approved Activities Completed USAID/OTI Contribution Matching Contribution Total USD Election Processes 60 60 $1,768,633 $302,550 $2,071,183 Media 101 97 $1,714,9591,415 $379,973 $2,094,932 Pace of Reform 258 248 $4,607,389 $1,812,895 $6,420,284 Grand Total** 632 605 $14,048,788 $7,941,076 $21,989,864 ** Grand Total includes RONCO grants approved and delivered since October 2000. Table 4 shows that OTI grantees made significant matching contributions - approximately 36% of the grand total of the expenditures in October 2000 – October 2002. The bulk of the matching contributions (66%) came in the CIP category. OTI/Belgrade informed the Team that matching funds were also part of the 1997 – 2000 profile, but these figures were unavailable at the time of the evaluation. The largest category for USAID/OTI expenditures was the CIP activities (36%) followed next by Pace of Reform. (33%). This table also discloses that OTI’s biggest push came in its final two years – representing approximately 76% of the total 1997 – 2002 expenditures. SECTION IV. FINDINGS ON ISSUES RAISED BY THE SCOPE OF WORK4 The evaluation team was given an elaborate, specific set of questions to be answered in its inquiry. For simplicity and clarity of response, the team herein provides its findings to the issues raised by USAID-provided scope of work (SOW): A. THE ABILITY OF OTI/SERBIA AND MONTENEGRO TO MEET STATED PROGRAM OBJECTIVES In the final years of his dictatorship, Milosevic’s regime was becoming increasingly despotic and vicious. The people interviewed expressed a clear vision of OTI’s 1997 – 2000 objective in Serbia: Get rid of Milosevic.5 This may not have been OTI’s stated program 4 Final Evaluation of OTI’s Program in Serbia – Montenegro, Scope of Work, OTI Washington, internet transmission, August 2002. 5 A more ‘official’ version of the USG policy is suggested by the statement of David Costello, “Our mission in Serbia and Montenegro was largely an extension of our already existing program in Bosnia and Herzegovina, where we sought to aid in securing peace in the region by directly assisting indigenous organizations in their efforts to advance peaceful democratic change in Serbia and Montenegro (S/M). Serbia, in particular, was seen as a major impediment to regional peace and stability and we attempted to target our resources accordingly." David Costello, OTI Program: Serbia/Croatia Program Manager (May 1997 - October 1997) and OTI Balkans Region Team Leader (October 1997 - April 2001) Statement to evaluation team, October 2002 “Without OTI, Milosevic would not have been overthrown” Slobodan Homen, Member of the Executive Board, OTPOR Development Associates, Inc. Final Evaluation of OTI’s Program 9 December 31, 2002 in Serbia - Montenegro There are no set responses; rather, strategies are tailored to meet the unique needs of each transition country. Typically, they are tested on a small scale and applied more broadly, when it is clear that high impact is being achieved. Changing conditions are quickly reflected in new or modified strategies. OTI Website, October 2002 objective, but the grantees and the OTI PDOs charged with implementing the program on the ground in Serbia were unequivocal; he had to go.6 The situation in Montenegro mirrored the Serbia experience. In 1998, anti-Milosevic forces had a bare majority. Yet they still had to suffer acts of intimidation and the presence of the Yugoslav army, under Milosevic’s control, in bases in Montenegro. According to Nebojsa Cagorovic, “… OTI staff were effective "shock forces" helping the bare majority elements to withstand the Milosevic￾sponsored political, economic, and military attempts to regain power…”7 Moreover, when interviewees were asked about OTI’s impact, they invariably pointed to its role in helping to get rid of Milosevic. Grantees credit OTI with being early on the scene (among international donors) with timely and unfettered support.8 They said that the PDOs were particularly effective in working with them (grantees) to design and execute effective opposition action. The Serbian PDOs who continued the program in Serbia during the NATO bombing were singled out as representing solid evidence of a sustained commitment. OTI’s CIP program (2000-2001) proved to be a winning post-Milosevic strategy for rewarding communities for their reform progress, while at the same time providing needed infrastructure assistance. As Paul Rowland expressed it, CIP helped demonstrate “… the tangible results of peace.”9 The communities visited for this evaluation stated that CIP opened their eyes to a new way of thinking and dealing with community problems. In the past, political connections determined what streets were paved or what houses had access to potable water. Under the CIP process, the citizens determined. They decided community priorities and checked to make sure that the job was done properly. B. THE ABILITY OF OTI TO CHANGE ITS STRATEGIC OBJECTIVES AND ACTIVITIES IN RESPONSE TO EVOLVING POLITICAL ENVIRONMENTS It was noted that OTI/Serbia operated within a hostile and volatile political environment. These conditions shaped the evolution of the program, as it responded to changing events and reform 6 “OTI's initial objectives were to curb nationalistic attitudes and perceptions; maximize the availability of objective information; and promote peaceful and democratic political changes throughout the country.” FINAL EVALUATION OF OTI’S PROGRAM IN SERBIA – MONTENEGRO, SCOPE OF WORK 7 Former OTI PDO in Montenegro (from the end of 1997 to June 2000). 8 Others active in the 1997-2000 periods were the Soros Foundation, European Union, and USAID/E&E. 9 Paul Rowland, personal interview, National Democratic Institute, Belgrade, Serbia. "OTI was on the ground working quietly and quickly with disparate local opposition forces to Milosevic...OTI is a unique political force … no one else in the United States Government (USG) did as well in Serbia....OTI created peace out of chaos." Bertram Braun, Political Officer, US Embassy in Belgrade Development Associates, Inc. Final Evaluation of OTI’s Program 10 December 31, 2002 in Serbia - Montenegro possibilities. The evolution of the OTI program, 1997 – 2002, can be seen in the following timeline, Table 5. TABLE 5 Timeline Timeline : 1997 – 2002 OTI Phases of Operation 1997 1998 1999 2000 2001 2002 Phase I: July 1997 - March 1999 Initiate Support to Milosevic Opposition Phase II: March 1999 - August 1999 Sustain OTI During Serbia Evacuation Phase III: August 1999 - June 2000 Intensify Opposition Support Phase IV: June 2000 - October 2000 Milosevic Falls in October Phase V: November 2000 – December 2001 Shift to Community Improvement Projects(CIP) Phase VI: January 2002- November 2002 Shift to Pace of Reform The program focus and activities included in each of the phases are described in more detail below. PHASE ONE: July 1997 – March 1999 Program Focus: Supported opposition forces aimed at the overthrow of the Milosevic regime.10 Program Activities: Strengthened media institutions through support to print and electronic media; supported civil society and NGO efforts to open the political system to greater citizen participation. In Montenegro, OTI supported the establishment of NGOs, civic society, and Independent media that helped the anti-Milosevic forces to win a majority in 1998. 10 “We were unique and most effective when we concentrated our assistance on S/M’s most relevant and partisan political actors (i.e., its opposition media outlets, political parties and municipal governments, and politically engaged and aggressive civic organizations), in Montenegro in 1998 and 1999, and in Serbia in the lead up to the 2000 elections.” David Costello, October 2002 Development Associates, Inc. Final Evaluation of OTI’s Program 11 December 31, 2002 in Serbia - Montenegro PHASE TWO: March 1999 – August 1999 Program Focus: Sustained the continuity and visibility of the program during the NATO bombing of Serbia and the evacuation of OTI expatriate staff (from Belgrade, Nis and Pristina). Program Activities: In close cooperation with OFDA’s Disaster Assistance Response Teams (DART) and other humanitarian agencies, OTI worked to relieve the stress on Albanian and Macedonia communities resulting from the refugee crisis. Local staff continued to award grants and to work with partner organizations to continue the program in Serbia. The Program continued in Montenegro. PHASE THREE: August 1999- June 200011 Program focus: Provided direct grant assistance to pro-democracy (civic and media) organizations in Serbia. Supported the development of the independent media and political opposition to Milosevic. Program Activities: OTI grants supported the publishing and distribution of newspapers, magazines, pamphlets, and other written materials; also the production of informative radio and television broadcasts (i.e., objective public affairs and public information programs, investigative reports, PSA’s, and documentaries) and the organizing of public forums, debates, discussions, petition drives, and mass demonstrations. OTI also provided media outlets and urgently needed equipment, operations support and legal advice for NGOs -- and helped these groups to establish links and to share ideas and information with other opposition organizations on the local, national and regional levels. PHASE FOUR: June 2000 – October 2000 Program Focus: Intensified support to the democratic opposition with the aim of removing Milosevic. Program Activities: Provision of direct grants to democratically-oriented municipalities, political parties and civic groups promoting peaceful, democratic change. PHASE FIVE: November 2000 – December 2001 Program focus: The Democratic Transition Initiative (DTI) commenced in early 2001 to help consolidate the political gains achieved in 2000, by promoting citizen action that encouraged participation and government accountability. Program Activities: Community Improvement Projects constituted the bulk of OTI programming in this phase. These projects built community/citizen participation, in most cases for the first time, into priority setting for community infrastructure projects, and produced 11 “Our most effective political development initiatives during this period (post 1999 – Milosevic fall) centered on our support for the Ring Around Serbia (RAS) initiative and other objective information outlets; our implementation of the USG’s ALT-NET project; our funding of numerous indigenous GOTV initiatives; and our direct assistance to OTPOR and DOS.” David Costello, October 2002. "Changes in 1999 and 2000 were happening rapidly in Serbia...OTI was very flexible....OTI adapted well to B92 changing needs...Its timing was crucial." Marija Milosavljevic, B92 Project Manager Development Associates, Inc. Final Evaluation of OTI’s Program 12 December 31, 2002 in Serbia - Montenegro "OTI was quick, fast, easy, and uncomplicated....It played a key role in bringing political and economic reform to Montenegro." Milka Tadic, editor of a weekly, The Monitor tangible results (e.g., new roads, school renovation, and access to potable water). This was particularly effective in South Serbia. PHASE SIX: January 2002- November 2002 Program Focus: The CIP was passed on to CRDA and the OTI’s focus shifted to the Pace of Reform effort, by helping to improve the quantity and quality of public information about on￾going reform efforts. Program Activities: In its last year, OTI initiated the Pace of Reform media campaigns with five target sectors in both Serbia and Montenegro. Four out of five sectors targeted in the Pace of Reform – anti-corruption, economy, judiciary, local self-governance – were the same in Serbia and Montenegro, with the fifth being different: minority rights in Serbia and law and order in Montenegro. This timeline highlights the central point that OTI shifted gears rapidly in response to changing political circumstances in Serbia/Montenegro. As discussed, the 1997 to October 2000 program focus was unequivocal – to throw out Milosevic. Some people interviewed said that this target made it relatively straightforward to mobilize and sustain the opposition forces. After October 2000 the focus shifted to consolidating the election gains of 2000 through the Community Improvement Projects. These projects were potent interventions. Several PDOs referred to CIP infrastructure improvements as ‘positive reinforcements’ for communities that chose the reform path.12 They showed that the CIP process of widespread citizen participation in local political decision making produced desirable results. Even more important, CIP addressed long-neglected community needs, such as access to potable water. In past elections politicians promised, but never delivered. After 2000, the situation had changed. Communities set their priorities and, with OTI support, took action to produce tangible results (e.g., new roads, rebuilt schools, and canalization). In this sense, CIP was a ‘tool’ to establish communication between citizens and local authorities. It gave newly elected political local authorities a new found opportunity to show that they could deliver on their promises. This benefit was particularly important in the South Serbia region. It should also be recalled that the CIP program required that communities provide a substantial matching contribution – they kicked in about 36% of the total costs - which meant that it was a community investment and not just a ‘free gift’ from a donor.13 12 In general, communities were selected for a CIP grant if they voted for opposition party (DOS) candidates, or gave strong indications that they desired to move in a reform direction. In South Serbia, communities that were severely impacted by the conflict, or those that experienced the greatest insurgent control, were targeted for CIP. 13 Communities provided matching contributions equaling approximately 36% of the total OTI program expenditures, 1997-2002. Development Associates, Inc. Final Evaluation of OTI’s Program 13 December 31, 2002 in Serbia - Montenegro "Changes in 1999 and 2000 were happening rapidly in Serbia...OTI was very flexible....OTI adapted well to B92 changing needs...Its timing was crucial." Marija Milosavljevic, B92 Project Manager Towards the end of 2001, USAID/Belgrade launched the Community Revitalization through Democratic Action (CRDA) Program, a five year, $200 Million effort. CRDA largely absorbed the CIP program; about 70% of the CIP communities were rolled into CRDA. Also at about this time, OTI learned from its baseline surveys that the Serbian/Montenegro populations were disturbingly apathetic. They seemed to be drifting back to the old way of waiting for someone to do something for them, rather than proactively making it happen. OTI saw this political indolence as a reform slowdown that needed to be counteracted.14 Given these circumstances, OTI again changed its course to begin the Pace of Reform (POR) Program, which was designed to inform the citizenry about reform issues and to encourage their vigorous participation in the political process. This new track required both a change in OTI’s focus and an upgrading of NGO skills, since most of NGOs were unversed in the core media campaigns of the POR.15 The POR campaign strategy of focusing on one theme (e.g., corruption, judicial reform) per month added additional stress for the PDOs, NGOs and for the Creative Associates staff. As one theme ended, another one started, ad seriatim, until all five themes were completed. The appropriateness of this strategy is discussed more fully below. C. THE APPLICABILITY AND APPROPRIATENESS OF OTI'S PROGRAM- MATIC DESIGN FOR SERBIA AND MONTENEGRO, AS A MEANS TO EFFECT CHANGE The evaluation interviewees consistently commended the appropriateness of the OTI program during the 1997 – 2000 and the November 2000 – December 2001 (CIP) periods. OTI PDOs were very knowledgeable about and experienced in the local contexts of target areas, and effectively tailored OTI activities (e.g., voter education, election monitoring, CIP, NGO support) to these varied milieus. They worked effectively with local partners under especially difficult conditions, especially in Serbia. As one South Serbia Mayor expressed it, “The OTI process fits with the local situation… sometimes technology from west does not transfer well… this was not a problem with OTI.” 16 Similar flexibility and nimbleness by OTI was also demonstrated in its activities in Montenegro: It provided support to independent media, to election monitoring and to opponents of Milosevic; OTI kept its Montenegro office open and active during the 1999 NATO bombing of Serbia and, as of late 1998, successfully supported the fragile newly elected democratic government in Montenegro, in its resistance to Milosevic’s ethnic cleansing campaign in 1999 in Kosovo; it assisted the Montenegrin government to receive tens of thousands of Albanian refugees. For 14 Evidence of this indolence could be seen at the time of this report: less than the required 50% voted in the October 2002 Presidential Election runoff, thereby negating the runoff and requiring a second, December 2002 runoff vote. 15 Chris Dorval, Trip Report, December trip to Yugoslavia, January 9, 2002 16 Mayor of Bujanovac, South Serbia, interview, September 2002. Development Associates, Inc. Final Evaluation of OTI’s Program 14 December 31, 2002 in Serbia - Montenegro a. Timeline of events: 1997 – 2002 1997: Similar political contexts; both Serbia and Montenegro were pro-Milosevic, operating authoritarian, repressive governments. 1998: Serbia remained under Milosevic regime control, while anti-Milosevic parties won a majority in Montenegro. This change enabled OTI to work with the Montenegrin government, while OTI continued to work with anti-Milosevic opposition forces. 1999-2000: Similar political contexts return to Serbia and Montenegro; anti-Milosevic forces solidify their position in Montenegro, while the dramatic fall of Milosevic occurs in Serbia in October 2000. 2001 – 2002: While both Serbia and Montenegro pursue political and economic reforms in democratically elected governments, there is greater activity and enthusiasm in Serbia. In contrast, in Montenegro, the speed of reform is slowed by the national debate (which continues as of this writing) over the political future of the country - to seek either independence or a new federation with Serbia. b. OTI Response to events Serbia: OTI supported anti-Milosevic opposition forces through October 2000, and then shifted focus to consolidating election gains and to building a civil society foundation. The 1997 – 2000 opposition thrust promoted NGO growth and community organization development. Montenegro: The 1998 electoral results opened the way to working with the government on projects. The absence of an NGO sector in the mid-nineties prompted major drive to build a viable network of NGOs, community organizations and advocacy groups, on issues such as women’s rights and electoral monitoring and reform. c. Pace of Reform Implementation The Pace of Reform was similar in implementation strategy in both areas in the following four sectors: anti￾corruption, judicial reform, economic reform and local self-governance. The fifth sector for Serbia was the promotion of minority rights, and in Montenegro, the promotion of law and order reform. example, the city of Berane in Northern Montenegro, with a population of 48,000, took in some 50,000 refugees. The Team was advised by several observers to take into account the Serbia-Montenegro differences in our analysis. Table 6 summarizes these similarities and differences, as they relate to OTI, and shows how OTI responded to these two environments in its program activities. TABLE 6 Serbia – Montenegro: Similarities and Differences 1997 - 2002 The1998 Montenegro electoral victory signaled the declining authority of the Milosevic regime. This enabled an earlier transitioning there than in Serbia. OTI was able to initiate a beginning cooperation with government agencies in Montenegro, while Regime opposition continued to engage the Serbian OTI program. “Pace of Reform began with the CIP hand-off [to CRDA], but there was not enough field work and time to properly plan and implement it.” Paul Randolph, OTI Country Representative. Development Associates, Inc. Final Evaluation of OTI’s Program 15 December 31, 2002 in Serbia - Montenegro Questions were raised about the appropriateness of the Pace of Reform campaign launched in 2001-2002. Part of the concern centered on the POR campaign strategy – one theme per month for the five themes. According to Francic Abouzeid, Creative Associates Public Information Specialist, “Pace of Reform tried to do too much in too short a period of time with NGO grantees who, for the most part, lacked the capacity to do effective media campaigns.”17 Several of the PDOs expressed a similar assessment, stating that they felt that the strategy was ill-conceived and did not give much promise of being effective in reinvigorating the citizenry to step up their political participation. The time pressures of the strategy impacted on the OTI procurement staff as well. They reported working under very tight (some said inappropriate) time pressures to complete the necessary procurements for each theme, month after month during the campaign. A related aspect of the appropriateness issue is the question of how long OTI should carry on in Serbia/Montenegro. OTI is generally viewed as a short￾term (i.e., max. two year) intervention in a country to facilitate the transition from humanitarian to development assistance.18 OTI has been in Serbia/Montenegro since 1997. Some people interviewed suggested that OTI perhaps should have departed Serbia/Montenegro after CRDA took over the CIP activity at the end of 2001.19 By then OTI had purportedly fulfilled its ‘transition’ mission and it was time for the USAID Mission to carry on with a more conventional development agenda. It should be noted that on-going discussions and a field retreat among OTI DC and Field staff concluded that there were “…sufficient opportunities for it to positively push the democratic transition in Serbia and Montenegro, even with the emergence of the Mission programming.”20 The Ambassador apparently agreed. At the end of 2001, he wanted OTI to remain active.21 He viewed OTI as an effective means of responding to critical rebuilding needs, especially in the contentious South Serbia region, and felt that it should continue through 2002. OTI/Serbia Montenegro officially ends in November 2002. 17 This point was echoed by Albert Cevallos, former OTI Balkans Program Manager: POR “… was too ambitious over too short a period of time...the lesson learned is to do a pilot project on one issue...” Interview, October 2002. 18 According to Diana Ohlbaum, Deputy Director of OTI from Nov.1999 to Oct. 2001, “OTI was to fill the gap between providing humanitarian relief caused by human decisions (e.g., war, repressive regimes) and traditional economic development....often there was a one to two year gap that was not being addressed effectively in the past....OTI was to fill that gap....”, Interview, October 2002. 19 Interviews with James Stephenson, USAID/Belgrade Mission Director; Diana Ohlbaum, Deputy Director of OTI from Nov.1999 to Oct. 2001. 20 Sara Brewer, response to draft final report, OTI evaluation, December 5, 2002. 21 Interview with James Stephenson, USAID/Belgrade Mission Director “OTI should always be at the cutting edge and never stay too long....maybe they stayed one year too long in Serbia …OTI needs to learn to leave at the pinnacle of success, not after.” Robert Jenkins, interview, 2002 Development Associates, Inc. Final Evaluation of OTI’s Program 16 December 31, 2002 in Serbia - Montenegro D. OTI’S COMMUNITY DEVELOPMENT ACTIVITIES, WITH PARTICULAR ATTENTION TO THE CONFLICT MITIGATION STRATEGY IN THE CONTENTIOUS PRESEVO AND BUJANOVIC MUNICIPALITIES22 Shortly after the NATO bombing stopped in Serbia and Milosevic agreed to the establishment of a UN Protectorate in Kosovo in 1999, the USG became deeply concerned about the Albanian￾Serb disputes extending into South Serbia, in which there were a number of mixed ethnic communities, many adjacent to Kosovo in which the Albanians were the dominant majority. Because the US Government was concerned about the stability of power transition in Belgrade and potential ethnic insurgency following the ouster of Milosevic in the area, and as to the prospects for democratic development – it charged OTI to take the lead role in the Presevo and Bujanovac municipalities. The objective was to enable sustainable peace to continue and to become rooted, and to pursue democratic and economic development in one of the poorest regions of Serbia. Five key factors contributed to averting an outbreak of significant hostilities that could lead to war in the area in the period 2001 and 2002, and these factors have enabled the USG and others to pursue the objectives of democratic and economic development in South Serbia. These five factors included: 1. A pro-active engagement of OTI with other sectors of the USG. Under the direction and personal involvement of the U.S. Ambassador, OTI maintained a persistent presence and local contacts throughout South Serbia. The features of OTI’s strategy at this time were:  A locally based and engaged professional staff;  Identifying and delivering small tangible projects that had an almost immediate impact on the local community. Illustrative examples include: the asphalting of a road connecting the Preshevo and Zujince communities; road repairs in Bustranje; medical equipment for the only health clinic in Veliki Trnovac; the remodeling (providing heat and running water for the first time) of the only school in Turija;  OTI’s capacity and practice of acting speedily, transparently, and flexibly;  OTI’s working well with other USAID programs (i.e.,CHF and CRDA) and with international donors, such as UNDP in delegating projects; and  OTI’s development of a proactive public affairs campaign, with the active participation of the US Ambassador, with senior Serbian officials, and with the local media – of which a few were OTI grantees, like Radio Preshevo and Jehona Weekly. 22 An excellent synthesis of the OTI’s Southern Serbia activity is presented in “Southern Serbia Crises: January 2001 – March 2002”, Special Report, USAID/OTI, Belgrade, no date, mimeo. “OTI appeared in the village, and life changed forever” Mayor Kutzinetza, South Serbia Development Associates, Inc. Final Evaluation of OTI’s Program 17 December 31, 2002 in Serbia - Montenegro 2. First time involvement of a new democratically elected Serbian government. For the first time since the peace agreement was negotiated and signed in Kosovo, the new Serbian government cooperated with local Albanian majority villages and communities in South Serbia. Examples included:  Participation by Serbia Deputy Prime Minister Covic in public appearances with the US Ambassador in South Serbia;  Development of a multiethnic (vs. all Serb) police force in South Serbia23;  Meeting between Covic and local Albanian mayors; and  Serbian government contributions to local projects, such as the establishment of electricity and phone networks in Velika Trnovac, the OTI sponsored school renovation project in Turija, and the OTI road building projects in Preshevo and Zujince. 3. Active OTI cooperation with other international donors and organizations. The presence and contributions of international donors and organizations, such as UNDP and OSCE in South Serbia, afforded an opportunity to maximize OTI’s impact in South Serbia. For example, OTI completed road projects are scheduled to be expanded, in cooperation with the UNDP’s contributing to economic improvements and related developments in South Serbia. In the media area, Radio Presevo, with the financial help of OTI was able to double its listeners from 35,000 to 70,000. With OTI-type assistance expected in the future from UNDP, Radio Presevo should be able to enhance its programming and coverage, as the only Albanian radio station in the Presevo Valley. Behlul Nasufi, Director of Radio Presevo, noted that OTI was "the first and best donor and was the only donor with local representation." He was confident of the future, following OTI’s departure, because "UNDP now operates with local input due to OTI’s example." In the area of politics, local officials and residents contend that the presence and involvement of international organizations, such as OSCE will enhance the likelihood of continued Serb government participation and contribution to economic development and political reform in South Serbia. 4. The inclusion of moderate Albanian community leaders and the involvement of grass root community groups. The fourth key to success in South Serbia is the active involvement of local moderate Albanian mayors and other community leaders in the various villages. Combined with financial 23 The border police, however, reportedly remain heavily armed, totally Serbian and viewed warily by the Albanian population "There is a 70% improvement from two years ago....The future is dependent upon practicing democratic principles by us and the Serbians." Glap Beqiri, Mayor of Veliki Trnovac, South Serbia . "My people have turned away from conflict and are now looking for economic opportunity." Sclajden Mustafa, President of the Zujence community, South Serbia Development Associates, Inc. Final Evaluation of OTI’s Program 18 December 31, 2002 in Serbia - Montenegro contributions (i.e., money, labor, and/or materials) by the local villages and municipalities, these OTI partnership-type projects enhanced the political notion of ‘ownership’ by the local residents and accelerated the needed change, in the effort to combat the deep seated distrust and hopelessness that is prevalent in the area. 5. Transparency in the operations and availability of independent media sources. For decades, the local Albanian communities in South Serbia had no access to independent or to Albanian sources from the print or broadcast media, or to inclusion in the decision making process of the Serbian government. OTI’s funding of an Albanian monthly, “Jehona” and of “Radio Preshevo” (the first Albanian radio station in the area) combined with the emphasis OTI placed on organizing and convening meetings by the local residents to discuss community needs and priorities -- made notable contributions to redressing longstanding minimal, if not non-existent, public participation in, and awareness of local matters. These developments combined with a concerted public campaign by the Ambassador and by senior Serbian officials (often jointly) over the past two years, have significantly reduced the corrosive nature of the ‘rumor mill’ (always incomplete, often inaccurate) in the many sidewalk cafes in the region. In short, there are more conversations today about cooperation than about conflict. E. THE IMPACT OF OTI’S WORK WITH MEDIA GROUPS OTI was very effective in identifying key independent (change agent) media groups, such as Radio B92 and Radio Boom 93. It identified the independent media leaders and provided them with essential support just in time. The police were actively harasssing the independent media by shutting them down, taking their equipment, and in general by making it very difficult for them to operate in anything approaching a normal manner.24 OTI helped offset this harassment by supporting the purchase of new equipment, as well as with programming support. OTI was also instrumental in supporting the creation of media networks, such as the Association of Independent Media (ANEM) in Serbia, which for the first time provided a venue for mobilizing independent media against the Milosevic regime. OTI contributed to the establishment of AKCIJA, a nationwide coalition of NGOs, independent media (TV and radio), technical experts, and educators who worked together to promote democratic and economic reform in Montenegro. Today, it is the closest thing to a nationwide nonpartisan ‘watchdog’ group in Montenegro. Also, OTI support helped establish the first independent radio and TV station in Montenegro. 24 An excellent account of the role of the independent media in getting rid of Milosevic and the constant police harassment they endured is in Mathew Collin, This Is Serbia Calling (Serpent’s Tail, London, 2001) "The OTI process was smooth, speedy, and unique. No other donor was as good as OTI in Montenegro." Rade Bozovic, Coordinator of AKCIJA, Montenegro Development Associates, Inc. Final Evaluation of OTI’s Program 19 December 31, 2002 in Serbia - Montenegro OTI’s support in South Serbia of Radio Presevo expanded the coverage of radio transmission for Albanian language programs (a very important first at the time) in the region. The Municipality originally created this station with support from the Serbian government. The OTI grant sought to modernize the station by improving and expanding its transmission capacity/coverage. OTI purchased the necessary equipment that expanded the coverage from an estimated thirty-five thousand households to seventy thousand. F. OTI’S EFFECTIVENESS IN DEVELOPING MEDIA CAMPAIGNS TO HASTEN THE PACE OF REFORM From 1997 through the October 2000 elections, OTI provided hundreds of small but targeted and timely material assistance grants to independent TV producers, radio operators, investigative journalists, and media groups. Beyond key material support such as computers, cameras, recorders, and basic office equipment, OTI also provided other assistance, such as timely, short-term rental payments. Finally, grantees also benefited not only from this financial and equipment assistance, but also from regular contacts, exchange of ideas, moral support, and overall strategizing with OTI staff. Examples of media groups that benefited from an OTI relationship included B92 Radio in Belgrade, Radio 021 in Novi Sad, the ANEM network, Radio Boom 93 in Pozarevac, and the Independent Media Center in Belgrade. The multi-type of OTI assistance (financial, technical, and moral) created a high level of trust between OTI and the grantees that enabled them not only to survive during OTIs temporary relocation outside of Serbia during the l999 NATO bombing, but to actually flourish and provide independent and objective information that ultimately led to Milosevic’s electoral defeat in October 2000. OTI conducted several baseline surveys as a foundation to document the impact of the POR media campaign. These surveys were launched prior to the POR campaign and did not include post-campaign follow-up data. Moreover, the timing of the campaign precluded the collection of follow-up survey data (comparable to the baseline data) that would have been credible (i.e., too short of a time interval between baseline and follow-up) for measuring the impact of the campaign. The team concluded that it was too early to measure the cognitive (i.e., information gain, perceptions) impacts of the Pace of Reform media campaigns with the available data. One of the Report’s recommendations is to conduct follow-up surveys at a later date. G. ELECTION-RELATED ACTIVITIES LEADING UP TO THE OVERTHROW OF THE MILOSEVIC REGIME OTI implemented a potent election strategy comprised of three integrated components. First, OTI supported voter awareness campaigns designed to inform the citizenry of the importance of voting and to stimulate voter turnout. Second, OTI supported voter monitoring activities. The monitors “OTI is generally credited with playing an important role…OTI support for NGO opposition to Milosevic was key to his overthrow…OTI sent message that we care about Serbia” U. S. Ambassador William Montgomery, Serbia/Montenegro "OTI was crucial in making contributions (financial and ideas) on a wider front bringing together NGOs, media groups, political parties, and small projects that would never have survived otherwise." Veran Matic, Editor in Chief/Chairman, B92 Development Associates, Inc. Final Evaluation of OTI’s Program 20 December 31, 2002 in Serbia - Montenegro were trained to observe the voting process at polling stations, to report on any suspicious activities or possible irregularities and to document the election results. Finally, independent media mounted a campaign right after the 2000 elections to widely publicize the results to all citizens. The campaign sought to minimize the potential of a repeat of the 1996 election in which Milosevic stole the election by ignoring the valid results and claiming victory. H. THE IMPACT OF OTI’S WORK WITH NGO GROUPS OTI was very successful in identifying the most effective NGO change agents in the opposition movement and in getting them timely support. OTI often was first on the scene with timely support to opposition NGOs. OTI program development officers quickly established and maintained their credibility within the opposition NGO community. They monitored the activities of their NGO grantees, and gave them the necessary independence to take the action they (the NGOs) deemed necessary. They also developed the only national NGO network (AKCIJA) in Montenegro. OTI’s Pace of Reform initiative following the fall of Milosevic helped establish various NGO networks in South Serbia. The POR initiative help extend NGO development to villages and small towns. With varying degrees of enthusiasm, commitment and capacity, NGOs in Serbia and Montenegro participated in the Pace of Reform campaigns. Some evidence of NGO effectiveness is suggested in the following quote from a Montenegrin government official: “The basis of the Montenegro government’s reform program should be AKCIJA’s reform agenda, as contained in AKCIJA’s ‘Reforms for a Healthy Society’. Some interviewees raised the concern that the recent proliferation of NGOs in Serbia – some place the number at 20,000 + - may dilute the effectiveness of the sector and lead to the perception that NGOs are a donor-dependent employment program.. This report recommends that USAID in the future should seek to focus and sustain NGO support to those organizations that have proven their value in the OPI experience. SECTION V. CONCLUSIONS OF THE EVALUATION In this section the team presents the conclusions it has reached as a result of its information gathering and fact-finding. We have organized them around several programs implemented and beneficiaries targeted through OTI activities. A. LOCAL MEDIA As demonstrated in both Serbia and Montenegro, one of the basic elements of democratization taking hold, surviving, and ultimately flourishing is the development and the preservation of local independent media in both broadcast and print forms. We were less distinctive and effective when we focused resources and attention on the large number of already existent western oriented media outlets and NGOs. While we did a reasonably good job of coordinating our assistance with other donors (i.e., OSI, AID ENE, Swedes, Dutch, EU, etc.), many of the recipients of international donor assistance were politically insignificant. The vast majority of human rights, women’s, peace and civic groups, and western oriented media, commanded little or no following. David Costello, October 2002 Development Associates, Inc. Final Evaluation of OTI’s Program 21 December 31, 2002 in Serbia - Montenegro In preparation for the fall 2000 elections in Serbia, OTI worked with a number of established grantees – the Student’s Resistance Organization (OTPOR), the Democratic Opposition of Serbia (DOS), and the Centar za Slobodne Izbore i Demokratija (CeSID)—on local media efforts to develop and promote election￾related activities. OTI successfully leveraged funding to local media groups from other donors such as the National Democratic Institute for International Affairs (NDI), the International Republican Institute (IRI), and the Serbian Independent Media Company (IREX), emphasizing the need for greater coordination in regular meetings between local media groups, students, women’s groups, and opposition political parties. In Montenegro, OTI assistance to local media groups resulted in the establishment of the first TV station (Montana) outside the capital city of Podgorica and the preservation of the only independent political weekly “Monitor”. As a result of assistance from OTI and other donors over the past 5 years in Montenegro, the number of independent TV stations has grown from 1 to 10, and independent radio stations have increased from 2 to 20. Several conclusions from the Serbia/Montenegro experience are worth noting:  Support is best directed to those independent media that have a discernible track record of efficient and effective action. OTI did this with its support for B92, Boom 93, etc.;  Support to media should be ‘just in time’ to produce desirable results. OTI worked closely with its media grantees and effectively timed its support to match the media programming schedules;  The media should be selected carefully and then given the freedom of action to do their work as they see fit. OTI was very effective in working with their media grantees, on programming ideas, for example, and then letting them decide the details of campaign implementation. B. NGO GROUPS NGO groups should be selected to carry out activities for which they are qualified. This may seem like an obvious lesson. However, too often NGOs are pressed to sustain their funding and to take on projects that they may be only marginally qualified to do well. While the OTI NGOs were very effective in the 1997-2001 periods, the POR campaign (2001-2002) funded some NGOs that were inexperienced in mounting the type of media campaign required. This meant that time had to be spent in training them and in relying on on-the-job training. This was inappropriate in this campaign with its tight deadlines and monthly, rotating theme strategy. C. COMMUNITY DEVELOPMENT PROJECTS (CDP) The OTI experience demonstrated that Community Development Projects (CDPs) that include Community Improvement Program (CIP) resources have the greatest impact. These projects promote community involvement, because they contain a tangible benefit, some type of material gain for the community. The improvement is a lasting community benefit resulting from the “OTI responded to B92 needs quickly, but B92 was in charge. There was no micro￾management from Washington. They let B92 do what they knew how to do” Interview with B92 staff, September 2002 Development Associates, Inc. Final Evaluation of OTI’s Program 22 December 31, 2002 in Serbia - Montenegro CDP process that supplements its value. The CIP requirement of direct citizen participation contributes to community acceptance. Another lesson from the CDP project is that tangible community contributions build local ownership. Approximately 36% of the CIP expenditures represent community matching. In the team’s interviews, community members emphasized that they had made substantial matching input to the effort. They felt that these matches proved that the grant was not simply a gift from a donor, but instead was part of a collaborative endeavor between the donor and the community. The matching also helped expand the scope of the project to support more ambitious upgrading. D. CONFLICT MITIGATION EFFORTS One of the most promising signs of easing tensions in South Serbia is the growing Serbian￾Albanian cooperation on community improvement projects. OTI helped fuel this process through the CIP grants. Interviews revealed that these former disputants are exploring other opportunities for funding as well. This finding suggests that successful collaboration by former disputants on an actual community improvement project will not only help rebuild the community, but might also have the additional benefit of promoting conflict mitigation. This potential benefit was envisioned in the CIP process. Enduring conflict mitigation must also include political access and economic opportunity for all citizens. The team saw evidence that not all barriers to peaceful coexistence have been removed. For example, a post office in a town that has an 85% Albanian population had no Albanian employees. Another example involves the owner of a small shoe store in Bujanovac (60% Albanian majority residents) who is confronted with a “decision from Belgrade” to build a Serbian Orthodox church replacing his store and others in the center of town. In the past, the shop owner had no choice but to lose his business. Today, however, he is circulating a petition to protest the decision, and is receiving assistance from the Office for Security and Co-operation in Europe (OSCE) officials to help “reverse the decision.” This suggests that the goal of full and equal opportunity for all is still a long way off and its attainment will require continued skill, perseverance and patience and the presence of international donors. E. ELECTIONS-RELATED ACTIVITIES The availability of fair, open, and competitive elections, election monitoring, and objective reporting of election results are key to the replacement of authoritarian/repressive regimes with authentically democratically elected governments. That reality was graphically demonstrated in both Serbia and Montenegro with OTI providing timely, reliable, and crucial support to local NGOs and to citizen groups committed to a democratic electoral process. The major lessons learned by OTI election-related activities in Serbia and Montenegro included:  Election monitoring is essential to minimizing election fraud. The monitoring provided quick and unassailable evidence on the validity of the election results (e.g., Milosevic lost the 2000 Election);  Publicizing the valid election results promotes compliance with those results. Publicizing the results widely to citizens and government organizations, such as the army, counteracts the attempt to ignore the results (e.g., 1996 Election) and to retain power; Development Associates, Inc. Final Evaluation of OTI’s Program 23 December 31, 2002 in Serbia - Montenegro  Citizens are more likely to vote when they believe that elections are fair and honest. Non￾voting by Albanians in South Serbia resulted from their belief that elections were rigged;  Citizens are more likely to vote when they believe that elections are important to their lives. The CIP project sought successfully to demonstrate to disbelieving citizens (e.g., Albanians in South Serbia) that their vote was important and could lead to tangible benefits (e.g., a repaved road). For example, in Montenegro OTI provided funding for computer and office equipment to the Center for Democratic Transition (CDT) enabling this independent group to open in 2000, and to monitor some 1,000 polling places with over 3,000 independent observers covering over 90% of Montenegro. CDT has also brought about the acceptance by all political parties, for the first time in Montenegrin history, of a code-of-conduct for campaigns -- thereby enhancing the prospects for fair and democratic elections. F. ‘HANDOVER STRATEGIES’ AND RELATIONS WITH THE USAID MISSION, AND WITH OTHER DONORS AND PARTNER OTI/Belgrade senior management stated forcibly that it had taken the necessary steps in time to effect a timely and efficient handover of the suitable OTI portfolio to the USAID/Belgrade development offices. It gave the impression that it felt it had done all it could for a successful handover. The results of the handover are summarized in Table 7. This summary shows, however, that the only certain ‘handover’ was the CRDA pick up of the sixty-one CIP communities. CRDA is just going to use these communities as project sites and is not adopting the CIP methodology. Also it is obvious from our interviews with Mission staff that CRDA was going to be a big part of the Mission’s portfolio (i.e., $200 Million over 5 years), regardless of OTI and CIP. The rest of the OTI portfolio is very problematic for any substantial Mission handover. A partial exception, perhaps, is the Mission’s decision to fund the AKCIJA/Montenegro activity ($750,000 for one year) through 2003. TABLE 7 Handover Results These findings and other numerous conversations with OTI and with Mission staff led the team to conclude that OTI has not yet developed and executed a timely and promising handover strategy. We heard comments from Mission staff, for example, that they had little idea of exactly what OTI was doing. This could reflect either OTI’s failure to communicate, or the Mission staff’s failure to be interested. Regardless of the reason, the point remains that there is an information chasm that has hampered handover. As another example, in October 2002, one month before the official OTI closeout, some of the PDO staff met for the first time with Mission 4 CRDA picked up 70% of the 61 CIP communities; 4 Freedom House and IREX will have grants open to competition by former OTI grantees; 4 USAID will pick up AKCIJA/Montenegro for one year at $750,000; 4 A few OTI PDOs found jobs with other donors and with other USAID contractors; 4 A group of OTI PDOs formed an NGO, the Democratic Transition Initiative (DTI) and plan to seek funding to continue work in the sector. Development Associates, Inc. Final Evaluation of OTI’s Program 24 December 31, 2002 in Serbia - Montenegro programming staff to discuss handover possibilities. In their defense, they said they should have been used much earlier to ‘sell’ OTI to Mission programs, but that they were never given the ‘go- ahead’. Nonetheless, October 2002 is much too late to begin talking about handover options in light of a November 2002 closeout. Several conclusions have emerged from this experience:  Handover works best when it is planned from the start of OTI operations in a country. Given the political situation in Serbia, as discussed, this was perhaps not feasible or possible to any significant degree. Nonetheless, it is advisable to begin the process as soon as OTI has its portfolio underway and the Mission is up and running. The Team saw no evidence that this proactive approach was implemented in any systematic manner.  Handover works best when OTI has established a solid working relationship with US Embassy and USAID Mission management and staff. We saw little evidence that OTI and the Mission management and staff had established a solid working relationship. On the contrary, the communication seemed rather episodic and unplanned, and not a routine, continuous dialogue aimed at laying a foundation for handover. The comment that “OTI is a bunch of cowboys that follow their own rules”, seemed to capture the mood of the relationship.25 The working relationship with the Embassy, according to the Ambassador, on the other hand, was viewed as effective; OTI was credited with being responsive to Embassy needs.  OTI needs to develop a ‘constituency’ within the USAID Mission to support handover. This point follows from the last – OTI did not have a constituency within the Mission. That is, it lacked a specific group of Mission staff in relevant program areas (e.g., media, elections) that understood what OTI was doing and the results it was producing and, most importantly, could make the case for handover. For example, the only real ‘handover’26 was the AKCJIA network activity in Montenegro, and this was limited to one year with no guarantee of continuation. This happened only because the OTI CR personally took the Mission Director, who reportedly did not favor picking up AKCJIA, to Montenegro to visit the project, with the result that it was agreed to provide another year of funding. As another example, OTI identified and trained a cadre of very effective NGOs (about 35) that could be very effective in future Mission activities. A proactive, effective handover strategy would have made sure that these NGOs were very well known to the Mission and that some support would be provided to sustain their efforts, and thereby 25 The evaluation team found that OTI was very conscious of its operating requirement under its special funding authorization: “Special funding mechanisms contribute significantly to OTI's ability to respond quickly and flexibly. OTI draws from USAID's International Disaster Assistance account, and thus is able to react quickly to highly time￾sensitive transition opportunities. OTI follows standard USAID procurement procedures for routine matters, but uses expedited procedures when rapid response is required. In September 1998, OTI created an implementing mechanism known as SWIFT (Support Which Implements Fast Transitions) to speedily establish a presence for new country programs. SWIFT partners can provide communications, security, administrative and logistic support for rapid assessments; establish or enhance operational field offices; hire and manage local and expatriate personnel; set up regional operations; advise on program interventions; and implement small grants programs. It is presently in use in Indonesia, Nigeria and the Philippines. While OTI manages the contract, SWIFT services can be used by any USAID office or mission, and even by other government agencies and donors.” OTI Website, October 2002. 26 As discussed previously, the CRDA activity was nominal ‘handover’, since it was obvious from numerous discussions that the Mission was going to do a CRDA program, regardless of what OTI had done in the country. Development Associates, Inc. Final Evaluation of OTI’s Program 25 December 31, 2002 in Serbia - Montenegro capitalize on the prior development. This did not happen and, as reported to the Team, many of these NGOs will face a very difficult financial future.  OTI PDOs are very effective communicators of program activities and achievements. The PDOs know best the OTI program activities as implemented in the field (e.g., media, elections, civil society), and they also knew what these program activities have accomplished and could accomplish in the future. They knew how to sell the program. Yet, by all accounts, they were not involved, to any substantial degree, in any handover discussions. As noted, the only evidence that the Team saw of this happening was in October 2002, in a follow-on to the Team debriefing meeting with OTI and with one member of the Mission staff.  The Team saw considerable evidence that OTI worked effectively with other donors and with its implementing partners. This cooperation helped to maximize OTI’s impact through teaming with other donors to get projects (e.g., cooperating with UNDP to build a road) successfully completed. This was especially obvious in South Serbia. Interviewees noted that OTI worked well with other donors (e.g., UNDP, IBRD, Soros Foundation) -- in some cases helping to coordinate the effort and to promote timely communication among donors. One grantee commented that UNDP learned from OTI how to be more efficient and less bureaucratic, and subsequently adjusted its approach to operate more like OTI.27 G. OTI MANAGEMENT AND ADMINISTRATION The importance of PDO competence, commitment and inclusion in OTI’s success has been noted. The PDOs are the front line implementers, the ‘OTI gatekeepers’, who select the local partners (e.g., NGOs), support them with money and technical assistance and monitor their performance. Thus they, in effect, determine what the program actually does and how well it performs. Their careful selection, nurturing, support and inclusion in the program decision￾making is obviously essential. Teamwork among OTI management and administrative staff, OTI Procurement staff and PDOs is critical to effective project implementation. OTI’s strength lies in flexibility, simplicity, and swiftness. A rapid procurement system is essential to this rapid response capability. PDOs know the most about ‘OTI in the field’ and are an invaluable resource for program planning and design. Administrative staffs have to handle efficiently and effectively the day-to-day details of running the program and of communicating both internally within the country staff, with the implementing partner (e.g., Creative Associates), and with Embassy and Mission staff and, externally, with OTI/Washington. The OTI data base system has proven to be a very useful management tool. It has been essential for managing and monitoring program implementation and progress reporting. The addition of the POR data base system has expanded the coverage with timely information on grantee activity. 27 Interview with Radio Presevo, South Serbia, September 2002. Development Associates, Inc. Final Evaluation of OTI’s Program 26 December 31, 2002 in Serbia - Montenegro SECTION VI. RECOMMENDATIONS The following recommendations are based on the findings and conclusions by the Evaluation Team about the operations of the Serbia/Montenegro program from 1997 to 2002. A. GENERAL RECOMMENDATIONS 1. Develop an overall country strategy that pursues a ‘vital few’ objective that can be accomplished within OTI limits of two to three years. This was the de facto goal set for 1997-2000, as discussed, and it was cited as the zenith of OTI’s success in Serbia/Montenegro. 2. Preserve OTI practices that promote speed, flexibility, responsiveness, and reliance on local expertise in program implementation. This is the ‘soul’ of OTI – simple, direct, flexible, quick and grounded in the people. 28 3. Maintain a rapid response OTI-type capability in OTI/Washington that can be quickly deployed, on an emergency short term basis, to avert conflict breakout and to promote conflict resolution. This OTI rapid action team would be available for less than a two year deployment. 4. Keep OTI financially and organizationally lean and nimble. Any future OTI budget increases should be motivated either by OTI extension to new countries or reentry into prior program countries that are experiencing destabilizing internal conflicts. 5. The experience here strongly points to the importance of transition planning for both management and program. OTI and central USAID should include a formal assessment leading to a transition play, halfway through the estimated duration of OTI work in a country. This would allow both efforts to prepare for the transition to normal USAID programming and management. The following Serbia/Montenegro OTI Success Model identifies the recommended behaviors gleaned from this evaluation. 28 This was the universal consensus of the people interviewed by the team, ranging from OTI and AID personnel to PDOs, grantees and local government officials in Serbia and Montenegro. Development Associates, Inc. Final Evaluation of OTI’s Program 27 December 31, 2002 in Serbia - Montenegro TABLE 8 Serbia/Montenegro OTI Success Model The political tradition in Serbia was for politicians to promise before an election and not to deliver after the election. OTI used the CIP experience effectively to promote a new mindset for politicians: Promise only what you can deliver, then deliver on what you promised. Another OTI tactic was to take into account the politics of an area in planning an activity. PDOs were very knowledgeable about the traditions and political stakes in their areas, such as, for example, who were the key players and what was possible. They used this information to fit their activities as to what was possible and likely to have the most impact. As demonstrated in Montenegro, timely grants to small but enterprising local media groups can create lasting impacts. OTI was the first donor to provide assistance to “Montana”, originally a small TV production group outside of the capital city of Podgorica. Today it has grown to become a private station, and is even more popular than the state-owned TV station based in Podgorica. One of the deficits of the POR campaign was overstretching the capacities of NGOs to implement media campaigns under very tight time lines. Most were unprepared and, according to several PDOs, this limited their effectiveness in mounting effective campaigns. The obvious recommendation is to either fit the demands of an activity to the capacities of the implementing partners, or to provide the training necessary to do the activity well. OTI clearly showed the value of hiring PDOs that and have courage, good judgment and a strong commitment to improving the lives of the target population. Many of the people interviewed extolled OTI’s ability to get in the picture quickly and then sustain a visible presence. This was most evident in the opposition push, in which OTI staff provided ‘just in time’ support and then stayed visibly active, even during the NATO bombing hiatus. Unlike other donors, OTI emphasized routine, face to face contact with grantees. This was appreciated by the grantees as evidence that OTI actually cared about what they were doing, and 4 Deliver on Promises. 4 Tailor OTI objectives and activities to the political realities of the target area. 4 Match grant technical requirements to the capacities of local project implementers. 4 Hire local staffs that know the target area thoroughly and have courage, good judgment and a strong commitment to improving the lives of the target population. 4 Get into the target area early and quickly to establish a visible on-the-ground presence. 4 Maintain routine face-to-face contact with grantees to demonstrate an interest in and sustained commitment to their project. 4 Routinely monitor progress and provide timely feedback to grantees. 4 Minimize bureaucratic requirements, keep the process simple and direct. 4 Fast track and streamline the procurement process. 4 Maintain a flexible and responsive project implementation process. 4 Routinely publicize key project activities and achievements to a wider target population. Development Associates, Inc. Final Evaluation of OTI’s Program 28 December 31, 2002 in Serbia - Montenegro was not simply giving them money. Grantees interviewed stressed that the personal relationship and ultimate trust developed with OTI staff was, at times, more important that equipment and technical support. OTI should establish relationships with local grantees that show up OTI as being not only a reliable financial donor, but also a reliable source of information, support, strategy, and new ideas. Grantees appreciated the feedback they received from OTI on their progress. In fact, some of the grantees commented that this ‘feedback assistance’ was as helpful to them as was the money. It helped in effectiveness in other areas, as well as in the area of the grant support. Grantees were especially appreciative of OTI’s ‘non-bureaucratic’ method of doing business: simple and direct; fast track procurement; and flexible responsiveness. As one person expressed it, “OTI helped us when we needed it and didn’t get in our way”. Finally, OTI did a good job of publicizing the successes of its grantees, making sure that they got all of the credit. This was especially effective in South Serbia, where the goal was to boost the image of local leaders (via CIP results) as willing and able to deliver on promises. B. RECOMMENDATIONS FOR MANAGEMENT AND ADMINISTRATIVE STRUCTURES 1. Procurement Mechanisms The OTI procurement process was very efficient and was perceived by Grantees, in comparison with other donors, as being quicker and less procedurally complex. Grantees said that the equipment they received (e.g., cameras, computers, video recording equipment) was delivered when promised, and though perhaps not the ‘top of the line’ this equipment was fully adequate to their needs. Other OTI programs should adopt this process. The OTI procurement staff also implemented several control mechanism to guard against the misuse of OTI funds in the procurement process. They included: 1) Certificates of Delivery’ certifying that a procurement item met required specifications; 2) OTI senior management approval for all payments; 3) vendor monitoring by procurement officers, time permitting; 4) a ‘blacklist’ of poor performing vendors. Other OTI programs should adopt this process. 2. Record-keeping Record-keeping for the 1997 – 2000 periods is uneven and less complete than for the 2000 – 2002 periods. The 1997-2000 gaps are doubtless due to the exigencies of running the program during this dangerous period. Ideally OTI programs should maintain a consistent record keeping process, including budget and expenditure data, from the start of the program in a country. The program did this, especially from 2001-2002, to provide a thorough program documentation. OTI has developed and maintains, through the OTI data base and the Pace of Reform data base (discussed below), a complete and impressive record keeping and program data system that should be the standard for OTI programs in other countries. It has also sponsored a series of Baseline Surveys that should be the basis of follow-up analysis. Development Associates, Inc. Final Evaluation of OTI’s Program 29 December 31, 2002 in Serbia - Montenegro The OTI database is well organized, current, user friendly, interactive and well maintained by Creative Associates. OTI management and staff routinely use the database to manage and monitor the OTI program implementation process. The POR database developed by Creative Associates provides current information on grant activities and includes a very accessible geographic display feature. Similar systems should be standard requirement for all OTI programs in other countries. The baseline surveys contain random samples of citizens and public officials that were used by the program for program planning purposes. In their present form – data from 2000 and 2001 – the team decided that these were inappropriate for assessing program impact, because of the short time period between surveys. However, they provide a good comparative basis for follow￾up surveys in 2-3 years, to assess the change in cognitive perceptions in Pace of Reform communities. The team recommends that the follow-up surveys should be done. 3. Monitoring and Evaluation The OTI/Serbia Montenegro program lacked a systematic monitoring and evaluation system. The OTI data base and the POR data base were useful for program management (e.g., to track project activities) but were inadequate for measuring impacts. Baseline surveys have the potential, with subsequent follow-up data collection, to assess the impact of the POR activities. Lacking was a routine data collection process that tracked the immediate and intermediate results that can be traced to OTI activities. The evaluation team had to rely on interviews and reported events (CIP improvements). Focus group data were available, but lacked any documentation of the methodology used, and therefore were inadequate for measuring impacts. The recommendation is for evaluation from the start to be built into the initial design and into any adjustments to strategy. Having an evaluation specialist at the strategy design table would help in reaching this goal. The specialist would be able to suggest evaluation measures and data collection options that could be incorporated into the OTI data base. SECTION VII. LESSONS LEARNED As a result of the five years of experience with OTI in Serbia and Montenegro, the team has described in prior sections some of the experiences that may have application in other OTI and/or USAID programs in other places and at other times. Here we have distilled these "lessons learned" into several observations that have a more universal application: 1. In communities that express deep distrust of government born out of years of neglect, community development projects that contain tangible benefits for the citizenry are likely to have the greatest impact. This was precisely the case in South Serbia. Albanian citizens felt that traditionally the Serbian government did not care about them and did nothing to help them. They welcomed the OTI/CIP project because, for the first time, they were listened to and, most importantly, they could experience a tangible benefit (e.g., paved road, renovated school) resulting from government action. 2. Successful collaboration by former disputants on an actual community improvement project will not only help rebuild the community, but will also have the equally important benefit of promoting conflict mitigation. With OTI encouragement and support, in South Development Associates, Inc. Final Evaluation of OTI’s Program 30 December 31, 2002 in Serbia - Montenegro Serbia, former Albanian and Serbian antagonists for the first time worked together to bring donor funding into their communities to effect infrastructure improvements. This multi-ethnic collaboration reportedly eased tensions by demonstrating that cooperation was possible and yielded material community payback. 3. For OTI to be successful in a threatening and unstable political environment, it needs local project implementers (e.g., PDOs in Serbia/Montenegro) who are thoroughly in touch with the local political scene, are calm in view of the risks involved, are able to work under uncertainty, and are discreet. They are the front line implementers, the ‘OTI gatekeepers’ that select the local partners (e.g., NGOs), support them with money and technical assistance and monitor their performance. Their behavior determines what the program actually does and how well it performs. 4. If OTI is going to rely on NGOs as a key part of its implementation strategy, it needs to recruit NGOs that are fully qualified and fittingly experienced. OTI relied heavily on NGOs during its Serbia/Montenegro tenure. The NGOs were very effective in the 1997- 2001 periods, in doing tasks for which they were qualified and experienced. The POR campaign (2001-2002) funded some NGOs that were unqualified and inexperienced in conducting the type of media campaign required. Time had to be spent in training them and then in relying on more on-the-job training. This was inefficient and unproductive in this type of media campaign, with its tight deadlines and monthly, rotating theme strategy. 5. The OTI Success Model can and should have application in other man-made disaster situations, particularly where uncooperative or hostile governments are involved. While application of the model necessarily would have to be adapted to a particular situation, it provides useful guidelines and a checklist. Final Evaluation of OTI’s Program December 31, 2002 in Serbia - Montenegro ANNEXES Annex A People/Organizations Contacted .....................................................................A-1 Annex B List of References............................................................................................B-1 Annex C Data Collection Protocol .................................................................................C-1 Annex D Scope of Work.................................................................................................D-1 Annex E In Country Data Sources..................................................................................E-1 Final Evaluation of OTI’s Program in A-1 December 31, 2002 Serbia - Montenegro ANNEX A PEOPLE/ORGANIZATIONS CONTACTED September 4, 2002: Wednesday Sara Brewer, OTI Program Manager, DC Patrick Wingate, Former OTI Country Rep in Serbia Denise Dauphinais, Management Associates, Creative Associates Eric Meissner, Creative Associates September 5: Thursday Peter Davis, President, Development Associates Tyler McMillan, Financial Project Assistant, Development Associates Sara Brewer, OTI Program Manager, DC September 9 Monday Peter Davis, President, Development Associates Jack Sullivan, Vice President, Development Associates Tyler McMillan, Financial Project Assistant, Development Associates Sara Brewer, OTI Program Manager Sara Farnsworth, Serbia Desk Officer, AID September 11 Wednesday Paul Randolph, OTI Country Representative, Serbia Erin Miller, OTI Country Deputy Representative, Serbia Ed Reineur, Chief of Party, Creative Associates in B Belgrade September 12 Thursday Erin Miller, OTI Country Deputy Representative, Serbia Paul Randolph, OTI Country Representative Sandra Dobic, Program Development Officer, Novi Sad September 13 Friday Veran Matic, Editor in Chief, B92 Marija Milosavljevic, Project Manager, B92 Sasa Mirkovic, General Manager, B92 Paul Rowland, Resident Representative, National Democratic Institute Paul Randolph, OTI Country Representative, Serbia September 14 Saturday Danica Stefanovic, President of NGO, Pnonija in Novi Sad Marina Fratucan, founder of Urbans, TV in Novi Sad September 16 Monday Aleksandra Petrovic, Market and Research Director, Medium Index, Belgrade Dr. Sbrbobran Brankovic, Director, Medium Index, Belgrade Development Associates, Inc. Final Evaluation of OTI’s Program in A-2 December 31, 2002 Serbia - Montenegro September 17 Tuesday James Stephenson, USAID Mission Director, Belgrade U. S. Ambassador to Serbia/Montenegro, William Montgomery Art Flanagan, USAID/CRDA Paul Randolph, OTI Country Director, Serbia/Montenegro September 18 Wednesday Erin Miller, OTI Deputy Country Director, Serbia/Montenegro Adrianna Loainica, Program Officer, AID Gene Szeprey, Deputy Program Officer, AID Kathryn Stevens, Democracy and Governance Officer, AID Robert Norman, DCM, Belgrade Betram Braun, Political/Economic Counselor, Belgrade September 19 Thursday Milorad Tadic, Director of RadioBoom 93 in Pozarevac Radoslavka Despotovic, Director of NGO in Pozarevac September 20 Friday Milorad Duric, Mayor of Vrsac, Serbia Zoran Dekic, President of bourough next to Vrsac September 23 Monday Ana Drakic, OTI Project Coordinator, Podgorica, Montenegro Goran Kalezi, Administrative Assistant, OTI office in Podgorica Marko Canovic, Director of Center for Democratic Transition, Podgorica Howard Handler, Officer-in-Charge, AID Mission in Podgorica Dora Plavetic, Democracy Advisor, AID Mission in Podgorica Timothy Collins, Public Information Officer, AID Mission in Podgorica Rade Bojovic , Coordinator, AKCIJA Group in Podgorica Milka Tadic , Executive Director and Editor of Monitor, Podgorica Kocha Pavlovic , Director of TV Production Group-Obala in Podgorica Vladan Raznatovic, OTI Project Coordinator, AID Mission, Podgorica September 24 Tuesday Tufko Softic, Project Coordinator of Citizens’ Group in Berane, Montenegro Remzija Ramusovic, Coordinator of Flood Project in Berane, Montenegro Zoran Matic – Vice President Municipality of Sabac Milos Milosevic - Director of Public Company for City Development, Sabac Dragica Lovcevic - General Manager of technical service in Public Company for City Development, Sabac Slobodan Nikolic – Director, NGO Village Doorstep Gordana Mandic - Project Manager, NGO Village Doorstep Verica Trifunovic - Project Manager, NGO Village Doorstep September 26 Thursday Dilaverdi Beqiri, CDG member in Bustgranje, Serbia Selajdin Mustafu, President of Zunjinie , Serbia Development Associates, Inc. Final Evaluation of OTI’s Program in A-3 December 31, 2002 Serbia - Montenegro Ruzhadi Jozuzi, President of Presevo North, Serbia Behlul Nasufi, Director, Radio Station in Presevo, Serbia September 27 Friday Glap Beqiri, Mayor of Veliki Trnova, Serbia Suleimani Sabedin - President of Local Community, Turije, Serbia Fehmi Ebidi - teacher from primary school, Turije, Serbia Sverdail Hjpeni, Editor of Jehona, Brujanovac, Serbia Farush Islami, Project Coordinator, Human Rights Council, Brujanovac, Serbia September 28 Saturday Nebojsa Kovandjic, Representative of Aleksinac, Serbia Zvonko Radovanovic, President CDG, Rutevac, Serbia Dragor Videnovic, Director, NGO group Enter, Nis, Serbia Drago Djodjevic, President, Committee for Human Rights, Nis, Serbia Marija Peternel, Director, Committee for Civic Initiatives, Nis, Serbia October 2 Wednesday Sasa Brankovic, OTI PDO October 3 Thursday Goran Radojevic, OTI PDO October 7-10 Washington, DC Interviews Jeanne Bourgault, Former OTI Balkans Field Staff Nebojsa Cagorovic, former PDO in Montenegro Albert Cevallos, Former OTI Balkans Program Manager David Costello, Former Balkans Team Leader Tatiyna Crepulja, NGO organizer in Montenegro Robert Jenkins, OTI Europe and Eurasia Team Leader Ray Jennings, Former Serbia-Montenegro Country Director Diana Olhbaum, Former OTI Deputy Director Sean Moffatt, Former OTI Staffer in South Serbia Other OTI/Belgrade Staff contacted Jelena Nesic - PDO, OTI office, Belgrade Ivan Vukojevic - PDO, OTI office, Belgrade Miodrag Stojadinovic - PDO, OTI office, Nis Maja Miljkovic - PDO, OTI office, Nis Jovica Spasic - Administrative Assistant, OTI office, Nis Zoran Vitas - PDO, OTI office Kragujevac Jelena Matejic - Administrative Assistant, OTI office, Kragujevac Final Evaluation of OTI’s Program in B-1 December 31, 2002 Serbia - Montenegro ANNEX B LIST OF REFERENCES Democratic Transition Initiative - SWIFT/Serbia & Montenegro, Creative Associates International, Inc., Bi-Annual Reports 2000-2002 Documentation and Assessment of OTI/Serbia-Montenegro Activities Supporting the 9/00 Elections, Cressida Slote, July 2001 DTI Retreat, Zlatibor, Serbia, February 12-14, 2001 Don Krumm’s Close-Out Report, Don Krumm, 12/20/01 Final Report: Local Election in Montenegro, Center for Democratic Transition, Podgorica, June 2002 Montenegrin Public Opinion in 2002: Attitude Towards Reforms, Center for Democracy and Human Rights, February 2002 OTI Website, October 2002 Ousting Foreign Strongmen: Lessons from Serbia, Thomas Carothers, Carnegie Endowment, May 2001 Parliamentary Election - April 2001 - Final Report, Center for Democratic Transition, Podgorica, July 2001 Public Perceptions and Attitudes about the Pace of Reform in Serbia, USAID/OTI, June 2002 RANCO Montenegro and Serbia Completed PTG Report, January 24, 2001 Report of Visit by Diana Ohlbaum to Serbia 12/13-16/2000, Diana Ohlbaum Report on Media Transition Opportunities in Serbia/Montenegro, John Penn, April 2001 Reforms for Healthy Society: Reformer Creating New Political, Legal, and Economic System in Montenegro, AKCIJA -Network of Non-Governmental Organizations in Montenegro, January 2002 "Southern Serbia Crises: January 2001-March 2002", Special Report, USAID/OTI, Belgrade, no date, mimeo This is Sebia Calling, Mathew Collin, (Serpent’s Tail, London, 2001) Development Associates, Inc. Final Evaluation of OTI’s Program in B-2 December 31, 2002 Serbia - Montenegro Transition to Long-Term Development: An Evaluation of the USAID/OTI Program in Kosovo, Robert J. Morin, Jr. and Dana S. Stimson, November 2001 Trip Report: December Trip to Yugoslavia, Chris Dorval, January 9, 2002 USAID/OTI: Advancing Peaceful, Democratic Change, May 1999 USAID Balkan Program Notes: 1997-1999 USAID/OTI Briefing Paper: Post Election Period in Serbia, October 3, 2000 USAID/OTI: Discussion Guide: Community Participation Programming in Serbia/Montenegro, February 9, 2001 USAID/OTI: Documenation and Assessment of OTI/Serbia-Montenegro’s Activities Supporting the September 2000 Elections, Cressida Slote, July 2001 USAID/Office of Transition Initiatives: Guide to Program Options in Conflict-Prone Settings, September 2001 USAID/OTI: Guide to Performance Management, prepared by USAID/OTI Program Development Team, April 2002 USAID/OTI: Guide to Performance Management, prepared by Cressida Slote, June 2002 USAID/OTI: Hot Topics - DTI, 2001-2002 USAID/OTI: Monthly Reports, 1999-2002 USAID/OTI: Results Strategy Development Meeting, Belgrade, October 19, 2001 USAID/OTI Serbia/Montenegro: Draft Program Orientation Guide 27, January 2001 USAID/OTI Special Report: Public Perceptions and Attitudes about the Pace of Reform in Serbia, June 2002 USAID/OTI: Yugoslavia Initiatives Program (12/99 - 12/00) Final Evaluation of OTI’s Program in C-1 December 31, 2002 Serbia - Montenegro ANNEX C DATA COLLECTION PROTOCOL Important: Get the Date, name, location and role/function of every person interviewed, or in a focus group Be sure to ask every question for each person or focus group! A. Interview/Focus Group Guide: OTI Country Program Staff and USAID • What do you think should be the main goal(s)/objective(s) of this evaluation? • What do you see as the main objectives of the OTI program in Serbia/Montenegro? • What do you think were the major challenges (or barriers) that the program faced in its start-up phase? How did it deal with them and was it effective? • Are there any national or community-level factors, or conditions, we should be aware of because they affected OTI program implementation and limit its ultimate impact/effectiveness? o In what ways has OTI had to change its goals/objectives/activities since it started, how did it do it, and did it work well? • How do you know if the program is performing well? What do you look at (i.e., performance indicators) and how? How do you use this information to improve the program? • Where do you think OTI has had its most important and detectable impacts: local level, regional level and/or national level? • What have been the most important impacts of the OTI program? Evidence? o Has the OTI program met its stated objectives? Why or why not? • What have been the most/lease effective parts or components of the OTI program? Evidence? • Do you have any recommendations on how the implementation of the program could have been improved to make it more effective, have greater impact? • What have been the major “lessons learned” about what worked or didn’t work in the program? What should be done different in the future? • Do you have any comments/recommendations specifically on the management/administration of the program? On how it is organized? On program staffing and supervision? The Grant process? On the monitoring and evaluation of program implementation and results? What do you see as the main ways in which the OTI Program management/administration could have been improved? How have you used the OTI database? Is it useful? • What is the OTI handover strategy? Is it working? Why or why not? How could it be improved? • How would you describe the OTI program relationships with other USAID, other donors, and partners? Have they been effective/ineffective, and why? How could they be improved? • Are you aware of any data (e.g., surveys, evaluations, research reports, etc) that that evaluation team should be aware of? Where are the data? Development Associates, Inc. Final Evaluation of OTI’s Program in C-2 December 31, 2002 Serbia - Montenegro B. Interview/Focus Group Interview Guide: National Level Stakeholders • What has been your involvement with the program? • What do you see as the main goal(s) of the program? • Are their any factors regarding the implementation of the program (in Serbia/Montenegro) that we should be aware of because they might have affected the impact/effectiveness of the OTI program? Where do you think OTI has had its most important and detectable impacts: local level, regional level and/or national level? What have been the most important impacts of the OTI program? Evidence? o Has the OTI program met its objectives? Why or why not? Evidence? • What have been the most/lease effective parts or components of the OTI program? Evidence? • Do you have any recommendations on how the implementation of the program could have been improved to make it more effective, have greater impact? • What have been the major “lessons learned” about what worked or didn’t work in the program? What should be done different in the future? • Do you have any recommendations specifically on the management/administration of the program? On how it is organized? How it conducts its business? On the Grant process? On the relationships among the various stakeholders? Do you think OTI program management/administration should be improved? How? • Do you think that the program has had (or will have) a lasting impact? What do you think they will be? C. Interview/Focus Group Guide: Local Level Program Staff and Other Stakeholders • What has been your involvement with the program? • What do you see as the main goal(s) of the program? • Are their any local level factors (e.g., barriers or facilitators) regarding the implementation of OTI (in Serbia/Montenegro) that we should be aware of because they might have affected the impact/effectiveness of the Program? How were they dealt with and did it work? Why did they or didn’t they work? • What have been the most important impacts of the OTI program in Serbia/Montenegro? Evidence? o Has the OTI program met its stated objectives? Why or why not? Evidence? • What have been the most important components (or activities) of the program, in producing these impacts? Evidence? • In what ways do you think the program (as implemented in Serbia/Montenegro) has been most effective and/or least effective, and why? • What have been the major “lessons learned” regarding the Program (e.g., programmatic design, implementation strategy)? What has/has not worked and Why? • Do you have any recommendations on how the implementation of the program could be improved in order to make it more effective, to have greater impact? • Do you have comments/recommendations on the management/administration of the program? On how it is organized or on the relationships among the various stakeholders, such as the local government, program staff, community groups, other Development Associates, Inc. Final Evaluation of OTI’s Program in C-3 December 31, 2002 Serbia - Montenegro similar program stakeholders, etc.? Could program management/administration be improved? How? • Are you aware of any data on program implementation and/or impacts that the evaluation team should collect? Have you used the OTI database? Has it been useful? Note: think of “OTI impact” and “OTI effectiveness” as synonymous terms) *** Suggestion: Use “OTI Evaluation Issues” sheet for interview probes, as appropriate *** Write-up the information you get under one or more of the following headings: “OTI Impacts”; “Lessons Learned”; “Recommendations”. tjc/September 10, 2002 Final Evaluation of OTI’s Program in D-1 December 31, 2002 Serbia - Montenegro ANNEX D SCOPE OF WORK I. OTI BACKGROUND The USAID Administrator created the Office of Transition Initiatives (OTI) in the Bureau for Humanitarian Response (now the Bureau for Democracy, Conflict and Humanitarian Assistance) to assist priority countries to make successful transitions from crisis to recovery and stability. The volatile political and economic nature of transitioning countries requires fast, emergency￾type responses that show immediate, visible and positive effect. Countries experiencing complex crises resulting from internal conflict and civil war have special needs that are often not addressed by traditional emergency assistance programs. Fledgling governments in newly established democracies often need direct, targeted assistance to adequately identify and address the tremendous political and economic challenges facing them. Likewise, other sectors and segments of society within new democracies require positive engagement and managed assistance. OTI's strategic objectives are based on the premise that fast and direct assistance, which takes into consideration the political ramifications and potential leverage of such assistance, is needed as a catalyst to move countries beyond the threat of crisis to stability. OTI, among development agencies and International Organizations, is one of the first offices to specifically address the gap between relief and development. When a crisis occurs in a priority country, OTI designs a country program to address the fundamental constraints that inhibit governance and economic functioning. Each country program has a set of objectives, an implementation strategy for accomplishing the program objectives, and an exit strategy. For more information on OTI, please visit: http://www.usaid.gov/hum_response/oti II. PROGRAM BACKGROUND The Office of Transition Initiatives (OTI) began operations in Serbia and Montenegro in July 1997. The program has undergone several phases of programming, reflecting the changing political situation within Serbia and Montenegro in the last five years. OTI's initial objectives were to curb nationalistic attitudes and perceptions; maximize the availability of objective information; and promote peaceful and democratic political changes throughout the country. Unlike most OTI country programs implemented prior to 1997, OTI engaged in Serbia and Montenegro before a clear democratic transition was underway in an effort to support local groups in their efforts to accelerate democratization. PHASE I In this initial phase, OTI established a main office in Belgrade with field offices in Nis, Podgorica, and Pristina. OTI continued its efforts to work with local grantees throughout 1998, Development Associates, Inc. Final Evaluation of OTI’s Program in D-2 December 31, 2002 Serbia - Montenegro despite deteriorating conditions under the increasingly repressive Milosevic regime. During the 1999 NATO intervention, OTI evacuated its Belgrade, Nis and Pristina offices and suspended grants within Serbia. OTI's Podgorica, Montenegro office remained open throughout the intervention with increased grant activity. PHASE II The second phase of OTI’s programming in Serbia and Montenegro began at this time. Following the NATO bombing, USAID/OTI's program in Serbia and Montenegro provided direct, short-term, high-impact support to politically active civic action groups, independent media, opposition-controlled municipalities, and democratically oriented political parties. In the lead-up to the September 2000 Federal Elections, OTI, in coordination with E&E grantees, contractors and State Department personnel, funded the several critical elections-related activities, including supporting a coordinated democratic opposition campaign and providing election monitoring. PHASE III After the overthrow of the Milosevic regime in fall 2000, OTI significantly expanded its programming in Serbia and Montenegro to take advantage of the democratic transition. In November, OTI launched the Democratic Transition Initiative (DTI) to consolidate the political gains achieved in the elections of 2000 by promoting citizen action that encourages government accountability and peaceful democratic change. The DTI program in Serbia and Montenegro works with community groups, NGOs, media organizations, and municipal, Republic and Federal governments by providing grants and training for community development, media, and other political transition initiatives. In 2001, community improvement projects (CIPs) constituted the bulk of DTI programming. DTI targeted select municipalities, using community-identified projects such as the rehabilitation of schools and health clinics, and the repair of electrical and water systems, to promote citizen participation and give standing to democratically-elected local officials. Throughout the winter and spring of 2001, OTI engaged heavily in the contentious Presevo Valley in Southern Serbia to work with local communities identifying and implementing of community improvement projects to ease tensions and allow ethnic Serb and Albanian negotiators time to resolve the conflict peacefully. Since its start, DTI has initiated 177 community development projects. The USAID Mission’s program Community Rehabilitation through Democratic Action (CRDA) has assumed much of the community development work done in 2001 by OTI. In its final year, DTI’s strategy has been focused on working at both the local and Republic levels to engage citizens and promote activities to increase the pace of reform. DTI’s efforts target five key sectors of reform: the judiciary, the economy, local self-governance, anti-corruption issues, and minority rights promotion. DTI also works to increase awareness and knowledge of human rights issues, expand the dialogue on truth and responsibility and support electoral activities. As of May 31, 2002, DTI had approved 521 small grants valued at $ 12.2 million. DTI has leveraged an additional $6.5 million in contributions from other international donors and local communities. Development Associates, Inc. Final Evaluation of OTI’s Program in D-3 December 31, 2002 Serbia - Montenegro III. OBJECTIVES OF THE EVALUATION Since programming started in 1997, OTI's Serbia and Montenegro program has undergone several iterations. Despite the difficulty of reviewing such a lengthy and diverse portfolio of activities, the evaluation will examine the entirety of OTI’s experience in Serbia and Montenegro, including its work with local non-governmental organizations, local media outlets, municipal authorities, and the Republic and Federal governments. The evaluation will also review OTI’s management structure and its relationship to overall program implementation. The evaluation will focus more heavily on the impact of the program, particularly on OTI’s activities undertaken following the ouster of Milosevic and the expansion of OTI programming and the launch of the Democratic Transition Initiative in Fall 2000, but should address key management issues as well. Under each of the following three sections, the more critical issues have been listed first. 1) Impact of OTI Performance in Serbia and Montenegro over the life of programming, including: • The ability of OTI/Serbia and Montenegro to meet stated program objectives and to change its strategic objectives and activities in response to evolving political environments; • The applicability and appropriateness of OTI's programmatic design for Serbia and Montenegro as a means to effect change; • OTI’s community development activities, with particular attention to the conflict mitigation strategy in the contentious Presevo Valley; • OTI’s effectiveness in developing media campaigns and harnessing NGO efforts to hasten the pace of reform in several key sectors following the overthrow of the Milosevic regime; • OTI’s elections-related activities and the impact of OTI’s work with NGO and media groups leading up to the overthrow of the Milosevic regime. 2) Lessons Learned • Programmatic lessons from OTI’s work with local media and NGO groups, community development projects, conflict mitigation efforts and elections-related activities; • Handover strategies, relations with the USAID Mission, other donors and partnerships; and, • Management and administrative lessons, particularly in relation to OTI’s ability to meet its programmatic goal. 3) Recommendations • Programmatic recommendations for OTI’s on-going work in other countries in transition; and, • Recommendations for management and administrative structures, including procurement mechanisms, record keeping and use of the OTI database. Development Associates, Inc. Final Evaluation of OTI’s Program in D-4 December 31, 2002 Serbia - Montenegro IV. METHODOLOGY The evaluation team will be responsible for developing a coherent evaluation methodology, which includes both qualitative and quantitative approaches. The methodology should include, but not be limited to semi-structured interviews, surveys, a document review, and the OTI Serbia and Montenegro database. V. TIMELINE FOR EVALUATION A proposed timeline for the evaluation follows, but OTI will work with the evaluation team upon award to further define the evaluation timeline and to determine deliverable due dates. • Washington, DC: Review OTI documents, discuss work plan and other needs with relevant OTI staff; begin interviews with OTI Washington-based Europe and Eurasia team members, and USAID/ Serbia and Montenegro Mission staff, and other relevant field partners with offices in the Washington, DC area. The time in Washington DC should not exceed 10 business days; • Serbia and Montenegro: An estimated 25 business days of fieldwork to include meetings with DTI and OTI staff, local grantees including media outlets and non-governmental organizations, former Community Development Group members, municipal and government officials who have worked with OTI as well; • Washington, DC: Final report writing followed by a final debrief of OTI staff and others. This period should take no more than 20 working days. VI. DELIVERABLES/DEBRIEFINGS The selected evaluation team will be expected to produce the following four deliverables: 1) A proposed work plan will be finalized by the evaluation team leader prior to departure, and cleared by the OTI Europe and Eurasia team in Washington, DC and the OTI Serbia and Montenegro field staff; 2) A three to six-page draft evaluation report with major findings and recommendations will be prepared in the field with initial debriefings to include OTI Serbia and Montenegro-based staff, and possibly expanded to include USAID/Serbia and Montenegro Mission and US Embassy in Belgrade staff; 3) A report of no more than 12-pages summarizing the key programmatic successes and challenges of OTI’s work in Serbia and Montenegro; 4) An expanded final report, no more than 50 pages, as described in section VII below. In addition to the four deliverables, the evaluation team will be expected to provide three debriefs on its findings: 1) A preliminary briefing of the team’s findings and progress of the evaluation project to the OTI/Washington DC-based Europe and Eurasia Team upon return from Serbia and Montenegro; Development Associates, Inc. Final Evaluation of OTI’s Program in D-5 December 31, 2002 Serbia - Montenegro 2) Two final briefings, including a presentation to an OTI-only audience and one presentation to a larger USAID audience. VII. FINAL REPORT The final evaluation report should include quoted comments from those interviewed, including but not limited to beneficiaries, grantees, DTI local and expatriate staff, OTI Washington-based staff and USAID/Europe and Eurasia and USAID Serbia and Montenegro Mission and Washington-based staff. The written format and writing style of the final report should be engaging and reader-friendly. The evaluation team should make use of the several external evaluations and internal OTI Serbia and Montenegro writings about the program, which should be referenced in the evaluation team's final report. The final report, not to exceed 50 pages, will be comprised of at least the following sections: • Executive summary; • Introduction and background; • Summary of evaluation objectives and methodology; • Significant successes and challenges; • Programmatic and managerial lessons learned for Serbia and Montenegro and beyond; and, • Recommendations for future OTI programs. To make sure that the evaluation findings are available to USAID and its partners, a copy of the final report and supporting documents shall be sent to PPC/CDIE/DI. VIII. TEAM COMPOSITION The evaluation team shall consist of four individuals: a Senior Level Evaluation Analyst, who will also serve as the Team leader, two Mid-Level Evaluation/Democracy and Governance Analysts, and a Cooperating Country National (CCN) who will provide administrative, logistics, and translation support as appropriate. The team leader should have extensive experience designing and conducting evaluations. OTI local and expatriate staff may be available to help facilitate the work of the evaluation team when in Serbia and Montenegro, though the evaluation team should be prepared to work and move independently. The team will be expected to make a number of field visits, to be determined, within Serbia. One team member should plan on spending an extended period of time in Montenegro to evaluate OTI programming in that Republic as well as activities targeted for Serbia launched from Montenegro during the 1999 NATO bombing campaign. The team should demonstrate the following experience and skills: • Evaluation research: Academic experience in the social sciences evaluating programs, particularly with community participation, media and civil society organizations, in countries undergoing transitions; Development Associates, Inc. Final Evaluation of OTI’s Program in D-6 December 31, 2002 Serbia - Montenegro • Rapid appraisal techniques: Academic training and experience with rapid appraisal techniques (survey development, direct observation, focus group interviews, community interviews and key informant interviews); • Survey and statistical analysis: Academic preparation and experience in survey research methods (survey design, sampling techniques and statistical computer applications); • Local knowledge: General knowledge of the Balkans' unique political, social, economic, and cultural environment and specific knowledge of Serbia and Montenegro; and, • Language abilities: Ideally, some members of the team will have a demonstrated knowledge of Serbo-Croatian. IX. SELECTION CRITERIA The offerors are encouraged to submit a document of no more than 2-3 pages on the proposed approach to the evaluation as well as why the offeror is qualified to undertake this evaluation. Offerors will be evaluated on the following criteria: 20 Points Contractor’s Responsiveness: The contractor must demonstrate that the team is available to begin field research no later than September 6, 2002. 20 Points Quality and availability of proposed personnel: Previous evaluation experience, particularly with community participation, media and civil society organizations; demonstrated survey and statistical analysis skills; demonstrated knowledge of Serbia and Montenegro and the Balkans; Serbo-Croatian ability. 15 Points Methodology: Team’s demonstrated skills and experience in developing coherent evaluation methodology to include both qualitative and quantitative approaches. The methodology should include but not be limited to semi-structured interviews, surveys, focus group interviews, a document review, and the OTI Serbia and Montenegro database. 10 Points Past performance: Previous work in the Balkans and evaluating the impact of democracy and governance programs. 10 Points Technical approach: Responsiveness to the objectives outlined in the SOW. 25 Points Cost: Costs will be evaluated for reasonableness and allowability. Provide a budget proposal in accordance with Section F.7 (b) (D). Final Evaluation of OTI’s Program in E-1 December 31, 2002 Serbia - Montenegro ANNEX E IN-COUNTRY DATA SOURCES The evaluation team had access to a variety of in-country data sources. They are identified in the table below. Of the various sources, the OTI Data base and the POR data base were the most useful for the team. The other three sources either lacked sufficient documentation to assess their quality of completeness or, in the case of the baseline surveys, were inappropriate for measuring OTI program impact. Data were collected before the implementation of the POR activities. The assessment of the various sources and their use by the team are summarized in the table. TABLE 9 OTI Serbia/Montenegro Evaluation Data Sources Source Of The Data Use Of The Data By Oti Program Usefulness Of The Data For The Evaluation Data Limitations OTI Data Base OTI Serbia/Montenegro Program Used routinely by program management and staff to monitor program implementation and report on progress Data very useful for understanding program implementation and planning evaluation field work OTI Pace Of Reform (POR) Data Base OTI Serbia/Montenegro Program Used routinely by program management and staff to monitor program implementation and report on progress. Contains geographic display of program implementation. Data very useful for understanding POR program implementation and geographic distribution of program services OTI Baseline Surveys OTI Serbia/Montenegro Program Used a diagnostic tool to plan future program activities (e.g., POR) Data were somewhat useful to understand citizen perceptions of local government and value of political Timing of the survey data collection (i.e., prior to POR full implementation) precluded use for assessing POR program impact; Development Associates, Inc. Final Evaluation of OTI’s Program in E-2 December 31, 2002 Serbia - Montenegro Source Of The Data Use Of The Data By Oti Program Usefulness Of The Data For The Evaluation Data Limitations activities some supposed no-OTI program comparison sites became CRDA sites, thereby negating the value as comparison sites. OTI/Serbia Focus Group Data OTI Serbia/Montenegro Program Reportedly used by program staff as diagnostic tool Not useful for evaluation team due to lack of documentation on the focus group implementation protocol Unable to verify the quality and completeness of the data collection OTI/Montenegro Phone Survey OTI Serbia/Montenegro Program Reportedly used by program staff as diagnostic tool Not useful to the evaluation team due to lack of documentation on the survey protocol; comments from Montenegro respondents that survey was not useful. Unable to verify the quality and completeness of the data collection OTI, Serbia-Montenegro Eval Report-EvalIQC-13.doc