EVALUATION OF THE UNIVERSITY COLLABORATION AND PARLIAMENTARY ASSISTANCE ACTIVITY: IRIS IN INDONESIA Submitted to: USAIDflndonesia Under Contract No. AEP-1-00-00-00023-00 Task Order No. #814 Prepared by: Roger Paget, PhD, Team Leader Thomas Cook, PhD Samuel Taddesse, PhD Lia Jutiani Submitted by: Development Associates, Inc. 1730 North Lynn Street Arlington, VA 22209-2023 July 2002 ACRONYMS ..................................................................................................................... iii EXECUTIVE SUMMARY ................................................................................................ iv CHAPTER 1: INTRODUCTION ...................................................................................... I A . Context for the Evaluation ...................................................................................... 1 B . Purpose and SOW for the Evaluation ..................................................................... 2 C . Methodology ........................................................................................................... 3 CHAPTER 2: REGIONAL UNIVERSITIES .................................................................... 4 A . Introduction and Background ................................................................................. 4 . . B . Purpose and Objechves ........................................................................................... 5 ... C . Achvlhes ................................................................................................................. 5 . . D . Findings ................................................................................................................... 6 E . Relevance, Effectiveness, Impact and Sustainability ............................................ 9 F . Conclusions ............................................................................................................. 10 CHAPTER 3: PARLIAMENT .......................................................................................... 12 A . Introduction/Background ........................................................................................ 12 B . PurposeIObjectives of IRIS' DPR-RI Support Activities ....................................... 12 ... C . Achvlhes Implemented ........................................................................................... 13 D . Findings ................................................................................................................... 13 E . Relevance, Effectiveness. Impact and Sustainability ............................................ 17 F . Conclusion ........................................................................................................ 18 CHAPTER 4: IRIS LPEM PARTNERSHIP ..................................................................... 20 A . Program Purpose ..................................................................................................... 20 ... B . Achvlhes ................................................................................................................. 20 C . Findings and Conclusions ....................................................................................... 21 CHAPTER 5: TRAINING .................................................................................................. 26 CHAPTER 6: IRIS ManagementPrevious Evaluation ..................................................... 27 A . Introduction ............................................................................................................. 27 . . B . Fmdmngs ................................................................................................................... 27 C . Concluding Thought ............................................................................................... 29 D . IRIS Program Expenditure Analysis ....................................................................... 29 Ewlu~tion of fhe Univemiy I Jnl~ 2002 Collaboration and Parliamenfnry Assisfanee Aen'vily: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00, Task Order # 814 CHAPTER 7: OTHER ACTIVITIES AND ISSUES ........................................................ 36 A. Partnership with US-Indonesia Society (USNDO) .............................................. 36 1) Findings 36 2) Conclusion 36 B. " Incubator" of Programs, People and Ideas ............................................................ 36 1) Findings 36 2) Conclusions 37 C. Other Contributions in Response to Developmental Needs ................................... 38 1) Findings ..................................................................................................... 38 2) Conclusion .. ............................................................ 38 D. Cooperative Agreement vs. Contract ..... ................................ 39 CHAPTER 8: RECOMMENDATIONS 41 . . A. Regional Univers~t~es .............................................................................................. 4 1 B. Parliament 42 C. IRISLPEM Partnership 43 D Management ............................................................................................................ 45 APPENDICES Original Evaluation SOW ...... ................................................................................. A- 1 Evaluation Team Draft Proposed Evaluation Purpose Statement - Submitted to USAIDIJakarta on Tuesday, June 18, 2002 ........................................................... B-1 USAID Feedback on the Team's Proposed Strategy ............................................. C-1 USAIDIJakarta - Priorities and Weightings, June 26, 2002) ............................... D-1 USAIDlJakarta Response to Evaluation Team's Proposed Evaluation Purpose Statement - and Evaluation Team Comments ................................................. E-1 Contacts & People Interviewed ............................................................................ F-1 Field Data Collection Protocol ....................................................................... G-1 IRIS Evaluation Focus Group Protocol ................................................................. H-1 References and Documents Reviewed ............................................................. 1-1 Evuluation of the University . . 11 July 2002 Collaborm'on and Parliamenfury Assistance Activity: IRIS in Indonesia BAPPENAS BPK CD DG DLG DPR-RI ECG EKUIN ELIPS FEU1 GO1 ILO IRIS LPEM MOF MOHA NGO OTI PEG PPPI RAPBD SO SOW UI UNDP USAID USINDO National Economic Development Agency Supreme Audit Agency Compact Disk Democracy and Governance Decentralized Local Government House of Representative - Republic of Indonesia Economic Growth Economics and Finance Ministry (now Menkeu) Economic Law, Institutional & Professional Study Program Faculty of Economics, University of Indonesia Government of Indonesia International Labor Organization Institutional Reform and the Informal Sector Institute of Economic and Social Research (UI) Ministry of Finance Ministry of Home Affairs Non-Governmental Organization Office of Transition Initiatives Partnership for Economic Growth Secretariat General Center for Research and Information Regional Development Budget Strategic Objective Scope of Work University of Indonesia United Nations Development Program United States Agency for International Development United States-Indonesia Society Eva 9 vation of the Univenify Collaborolion and Parliamenra~y Assistance Anivi~: IRIS in Indonesia This evaluation, which was camed out in Indonesia by a four-person team, from June 17 through July 12, 2002 - assesses a collection of disparate activities included under a cooperative agreement between the United States Agency for International Development (USAID) and the Center for Institutional Reform and the Informal Sector (IRIS) of the University of Maryland. This project essentially sought to provide economic assistance related to the complex processes of democratization and decentralization in Indonesia during a time of rapid and radical change politically, economically and socially. The IRIS operations in Indonesia began modestly in 1997, but in 1999 - with a military dictatorship deposed and real nationwide elections looming for the first time in 44 years - IRIS proposed, and USAID funded, a much broader array of activities intended to help facilitate and channel political and economic discourse in directions beneficial to Indonesia and congenial to US foreign policy. Indeed, the "road shows" orchestrated by IRIS that traveled the length and breadth of the country in 1999 were a success by all accounts. And, this success led the next year to the additional project agreements and funding for activities, which are under scrutiny in the present evaluation. The funding level was just under 54,000,000 and covered a period from early 2000 through August 2002. Some aspects of the projects had begun earlier and simply continued under a new agreement; others began later and were designed to require funding well beyond the official contract period. Some of the earlier projects, or their functions, became appropriated by other USAID programs, and some of the new projects were abruptly re-oriented in content and purpose in response to exigencies in Washington and Indonesia. Policy changes in Washington, secessionist movements in Indonesia, religious conflict, major legislative initiatives in Jakarta, and other factors required nimbleness on the part of the IFUS management. The projectslactivities under review by this evaluation include: regional university capacity building to assist in the establishment of better practices in regional and district legislatures; up￾grading of legislative process and capacity building among support and research staff in the national Parliament in Jakarta; assistance to the Institute of Economic and Social Research of the University of Indonesia; the organization of conferences, workshops and haining sessions; and sundry other activities that have emerged along the way. IRIS, founded in 1990, is dedicated to a philosophy which views successful economic development as the product of integrally related political, administrative and fiscal capacities and synergies. Thus, the scope of work for the projects in Indonesia reflects a commitment to an integrated approach to economic governance. Questions of power, practicality and implementation, according to the documents, are viewed as closely related to matters of economic and fiscal theory. Capacity building and sustainability are expected to be outcomes of virtually all aspects of the projects. A fundamental conclusion of the present evaluation is that the central tenets of the IRIS philosophy, cited in the SOWS. generally have not guided the management and implementation of the project components. As a consequence, many worthy objectives have not been attained. and, in some cases, the activities actually may have been counterproductive. Evaluation of the Univemily iv July 2002 Collaboration ond Parlinmenfaty Asisfmce Acfivily: IRIS in Indonesia Developmenr A.~sociates, Inc. AEP-I-00-00-00023-00, Task Order # 814 USAID's highest priority was the establishment of a regional university consortium, which would also include the new special autonomous regions, Aceh and Papua; this consortium was expected to generate momentum nationally towards improved civil governance at the provincial and district levels. IRIS' efforts have helped to generate nationally an embryonic university consortium, expressed through conferences, workshops, trainings and selective intellectual ferment - as regions adapt to rapid movement toward fiscal decentralization. However, IRIS did not effectively use existing and potential channels of cooperation, carry out reinforcement1 follow-up effort or seek to leverage limited resources to build a foundation for the future. As a result, while a few participants remain enthusiastic about the program, the preponderant view gleaned from the evaluation interviews is that planning and implementation flaws have contributed to disappointing outcomes. This was fundamentally a top-down activity, largely confined to issues of fiscal decentralization, and a needs assessment was never conducted to determine what the actual programmatic preferences of target constituencies might be. Few participants expected the consortium to outlast the funding. Infra-structure development, or other arrangements which would serve so as to make the consortium self-sustaining were not pursued The second IRIS priority was the Parliament, which in 1999 was comprised of newly elected representatives who lacked experience in the actual process of legislation. Even worse, staff support, largely left over from an authoritarian past and protected by civil service provisions, had little or no training or education adequate to the new demands. IRIS undertook to provide expert advice to selected parliamentarians and training to the support staff. IRIS is proud of its influence on particular pieces of legislation and on a few key members of Parliament. And it believes that their briefing papers, research reports and other inputs have had dramatic impact. With its nominal partner, LPEM, the research institute at the University of Indonesia, IRIS can claim numerous instances of valuable conhibutions. However, the evaluation interviews - held almost entirely with individuals of IRIS' choosing - yield an almost uniformly contrary picture. At a high cost in the use of expatriate advisors, one committee chairman may benefit and a piece of legislation may be positively affected. Yet, when the chairman steps down, the net result in terms of capacity building will be marginal. Meanwhile, the resources available for strengthening the analytical skills of staff have been sharply diminished. In a similar vein, the supply of highly 'academic' papers generated under IRISLPEM auspices may will be to be valuable contributions to the Parliament, but the recipients regard them as abstruse, overly theoretical and not attuned to the needs of practical legislative tasks. Regrettably, the IRIS project in Parliament has failed to produce self-sustaining institutional advances consistent with the specified objectives. One finding that stands out in the Parliament project, namely, the reliance on expatriate experts, also characterizes much of the rest of IRIS activity, and perhaps serves as a metaphor of actual practice as opposed to stated objectives. Of the total funding package for IRIS project activities, 55% was expended on long-term technical assistance. And 84% of that funding went to foreign expatriates. Another 25% of IRIS resources went to short-term technical assistance, but none of the available data made the distinction between local and expatriate personnel. Only 7% of the Evaluation of the Universily v Jury 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia Development Associores, Inc. AEP-I-00-00-00023-00, Tak Order # 81d funding went to training activities. in other words, relatively little of the funds available found their way to Indonesians or to activities that might generate sustained energy in the direction of civil society. An oft-repeated rekain among our interviewees was a polite suggestion that Indonesians - in Parliament, in LPEM, in regional universities - need to be accorded a more integral role in determining what is needed and in implementing any resulting activity, if the (much appreciated) USAID dollar is to have a longer-range, sustained impact. Many interviewees also expressed the concern that so few women were included as participants. Finally, the evaluation team, while supportive of most of the stated project objectives. believes that none of these activities should continue to be funded, unless management practices are significantly improved. Programs lacking a strategic focus and implementation plan are drawn to ad hoc activities that may not contribute to strategic results. Needs assessment is essential if such projects are to have a base of support in the target beneficiary populations. A performance￾based budget with expenditure data that links resources expended and results achieved. and which enables the monitoring of progress to ensure continued congruity between assumptions and reality is also important. These essential components of strategic, results-based project management were missing in the IRIS project management practice, and should be an indispensable requirement for any future funding. Evaluation of the Universiry vi July 2002 CollaboraIion and Parliamentary Assirlance Adivi$: IRIS in Indonesia A. CONTEXT FOR THE EVALUATION The USAID Country Strategic Paper, "Transition to a Prospering and Democratic Indonesia," cites two foreign policy goals: 1) the institutionalization of democracy and the resumption of economic growth, and 2) the preservation of the territorial integrity and unity of Indonesia The Evaluation Scope of Work (SOW), and other documents of the IRIS program, acknowledge the formidable challenges that confront these aspirations for successful change. The Center for Institutional Reform and the Informal Sector (IRIS) at the University of Maryland is a development organization that has operated internationally since 1990. Through research and advisory assistance it focuses on a complex of institutional features that it believes are contributive to successful economic growth. Central to the IRIS approach is the view .'that market augmenting institutions are more secure in the long-run when political rights are imbedded in a democracy and that the intersection of economics, law and politics is critical to the wealth of nations." Indonesia's constitution, which has been in force continuously for more than forty years, provides an illustration of the conceptual and institutional complexity that attends development initiatives. Parliamentary democracy, autocratic democracy, military dictatorship, and variations of each, governed under this fundamental document. Free market, liberal. socialist, command and other economic orientations also found this constitution congenial. Behind each of the two U.S. foreign policy goals of political and economic development is played out a complex struggle to square Indonesia's historical reality with preferred instruments of governance and civil society. One of the principal IRIS documents in the present evaluation fittingly captures this daunting challenge, and specifically refers to the program issues of decentralization: "The three basic aspects of decentralization - political, fiscal and administrative - are intertwined. At all levels of decentralization, political processes and outcomes affect fiscal matters. Fiscal decentralization is shaped by political arrangements as much as by incentives designed to influence privatization and public finance. At their pinnacle these political arrangements involve constitutional questions: What are the rights and responsibilities of each level of government? How may these relationships be legitimately changed? Other key political decentralization questions are: What is the status and strength of plural political parties at regional and local levels? How transparent is the political process? To what degree and in what time frame does the political system allow politicians to be held accountable for their decisions and activities? How strong and transparent is the local legislature? What strength and voice does the political structure allow for civil society?"' ' IRISLPEM Decentralization Grant: Life of Grant and Year One Work Plan, November 29, 1999, Evaluation of the Univeniry 1 Jdy 2002 CoNaborntion and Parliomenrory Assistance Activity: IRIS in Indonesia Development Assoeiotes, Inc. AEP-I-00-00-00023-00, Tark Order # 814 While the IRIS program has evolved over a period of several years, and sub-programs and unanticipated dimensions of programs have been remarkably varied, each program activity examined for this evaluation has revealed a set of conditions consistent with the above paragraph written two and a half years ago. Questions of power, practicality and implementation are intimately close to matters of economic and fiscal theory. Understandings and assumptions about constitutionally derived authority and practice vie with customary relationships that may enjoy the privilege of compelling historical imperatives. Programmatic credibility and cultural sensitivity are indispensable to achieving capacity-building and sustainability. B. PURPOSE AND SOW FOR THE EVALUATION USAIDiIndonesia is entering a phase of assessing programming options for advancing its strategic goal of furthering the "transition to a prospering and democratic Indonesia." The present evaluation seeks credible information on the current relevance, effectiveness, impact, sustainability, performance measurement and financial monitoring of the IRIS program, and recommendations for improving the types of activities (e.g., assistance to Parliament, regional university network development) included in the program. Based on a reading of the SOW and consultation with USAID, the team states that the purpose of this evaluation is to assist this assessment process, by providing data and information that will help the Mission in its deliberations to weigh the strategic value of supporting different types of activities. In discussions with the Mission, it was decided that this evaluation would focus on the post￾October 2000 timeframe, which is the period following the midterm evaluation of IRIS (See Appendix C). General themes in this evaluation cut across specific program components and address issues, such as the coordination and the synergy/impact/role of IRIS within the Economic Growth (ECG) Office strategy and within the overall USAID mission strategy. The SOW lists seven specific components of IRIS program activities: Parliament, Regional Universities, IRIS-LPEM Partnership, Special Autonomy Regions, USINDO Partnership, IRIS as "incubator of programs, people and ideas," and Miscellaneous Development Responses. Of these, USAID set the highest priorities (see Appendix D) on: 1) Regional University and Special Autonomy, 2) LPEM Partnership components, and 3) Parliament. The remaining components were ranked according to priority at much lower levels, and will be discussed accordingly in this report. Within each of the IRIS program components, the SOW lists a number of questions that address the relevance, effectiveness, impact and sustainability of the IRIS activities. In our discussion of these questions for each IRIS program activity, we will present the specific SOW questions and present our findings, conclusions and recommendations. A secondary priority, but nonetheless an important area of SOW attention, concerns the previous, mid-term evaluation. The Mission wants to know if, and to what extent, IRIS management repaired the deficiencies identified in that evaluation. We shall address each of the deficiencies and discuss the extent to which IRIS has responded in a satisfactory manner. Evaluation of the Universily 2 July 2002 Collaboration und Parliamenta~y Assistance Activity: IRIS in Indonesia Development Assoeinres, Inc. AEP-I-00-00-00023-00, Tosk Order 81.1 The evaluation team used a variety of information sources to evaluate IRIS. The team began by reviewing the SOW and discussing it with the Mission, in order to ensure that there was a shared understanding of the purpose of the evaluation. The team prepared a draft statement of purpose for Mission comment; comments received from USAID were included in the purpose statement included in this report. As noted, the team also asked the Mission to assign a priority weighting to the SOW evaluation issues, which it did'. in order to help the Team allocate its resources to the priority Mission questions. The team collected information through personal interviews, site visits to selective regions, and through a focus group with program beneficiaries. The data collection protocols, the people interviewed, the sites visited and the focus group participants are presented in the appendices. Each of the four team members reported their individual findings, conclusions and recommendations for each of the SOW issues in a common evaluation analysis matrix. They then met and compared and contrasted their findings, etc.. in order to reach team agreement on the specific statements to be included in the report. This process weighed the evidence for each reported finding, conclusion and recommendation, so as to ensure that these were sufficiently supported by available data andlor personal observation. The writing of each of the report chapters also underwent a similar team review and revision process. It should be noted that "training" was treated as a cross-cutting issue and is discussed in those sections where it was appropriate. One nettlesome issue that was never satisfactorily resolved during the course of the evaluation revolved around the choice of words. The terms program, project and activity are not used consistently in any of the documentation we have examined. nor did interviews offer enlightenment. Although the favored term ostensibly was project, the team concluded that in most cases the activities were too diffuse to warrant such a characterization. Nonetheless, we tended to accept IRIS' choice of language in most conversations. In this evaluation report our reservations about word-choice are also evident. See Appendix D for the USAIDIJakarta weighting of the evaluation issues. Evaluation of the Universiry 3 July 2002 Collubor~~on ~d Parliamentary Assisfanee Activity: IRIS in lndonesia A. ~VTRODUCTION AND BACKGROUND The Dutch left Indonesia with virtually no institutions of higher education and with fewer than 100 college graduates in a nation of 68 million people. During fifty-three years of independence, Indonesia has built a university system throughout the nation, and dozens of major private and denominational institutions of higher education now augment this system. Four institutions dominate the system, two of them - the University of Indonesia, in Jakarta, and Gadjah Mada University, in Yogyakarta - are what might be classified as comprehensive universities. The other two are professionally oriented, one in technology, the other in agriculture. All four are located on the island of Java. Spread out across the archipelago, the many other universities vary substantially in quality of faculty, students and facilities. Some are close in quality to the flagship institutions and may have selected departments that are superior. Others struggle to maintain minimum academic credibility. Some, located in regions favored by a strong economic resource base, or by a politically influential status in national affairs, may have steadily advanced over recent decades. Others have remained isolated and impoverished, existing mainly because national regulations stipulate their survival. All the universities, even their leaders, however, must struggle to keep educational values alive while coping with a hand-to-mouth existence. Most faculty work in more than one job. often in three, four or five jobs, to maintain sufficient income for their families. Library holdings are meager, and books tend to disappear if they have commercial value. Although most departments now have computers, these are generally ancient and defective, and more often than not, there are only one or two computers for a department that may number dozens of regular and adjunct faculty. Few of the universities have local server capacities, and electricity supply in many parts of the country, even in Java, is highly erratic and costly. Internet use thus tends to be discouraged for financial reasons, even when, technically, it is available. . During the past few years of major economic decline and perpetual fiscal crisis. the central government has encouraged entrepreneurial, off-campus. remunerative activity throughout the university system. Such activity rapidly has become a major source of revenue, as a result of severely reduced national budgets for higher education. While a few departments have easily exploited parallel off-campus alignments that have opened up unprecedented sluices of alternative revenue, many departments have found themselves in ever more financially straitened circumstances. And, this has further entrenched an unfottunate traditional characteristic of Indonesian higher education, namely, the isolation of even closely allied disciplines &om one another. These and other, seemingly pedestrian, concerns represent crucial obstacles to the implementation of ambitious schemes for regional university invigoration and capacity-building - in the interest of decentralization, democratization and general national advancement. The IRIS regional university and the closely related special autonomous region progtam depended, Ewlualion of the Univeniv 4 July 2002 Collaboration and Parliamentary Assisfmce Activily: IRlS in Indonesia Development Associafes, Inc. AEP-I-00-00-00023-00, Task Order # 814 from the outset, upon multidisciplinary academic cooperation and multi-faceted inter￾relationships between regional universities and counterpart provincial, regional, and local governmental institutions. This challenge may have been insurmountable, given the prevailing circumstances in Indonesian higher education. B. PURPOSE AND OBJECTIVES IRIS, with its partner organization, LPEM, sought to provide the means for faculty members in the regional universities to update and expand their understanding of decentralization issues. Second, IRIS aimed to have members of political parties, civil society, the private sector and local government in the regions served by the regional universities, expand and deepen their understanding of decentralization issues. Third, IRIS was to encourage regional universities' faculty members to incorporate the additional understanding of decentralization provided by the IRISLPEM program into their work -teaching, research, consulting, speaking and civil society involvement. Fourth, IRIS provided mechanisms and incentives whereby members of political parties, civil society, the private sector and local government would use their enhanced understanding of decentralization and the additional expertise of specific faculty and the regional universities, in addressing decentralization and related policy issues. Finally, IRISLPEM endeavored to develop between the University of Indonesia and the regional universities a level of interaction that was consistent with regard to decentralization issues. IRIS sought to enhance the capacity of regional universities, especially their capacity to provide fiscal and economic advice to regional legislatures and other government agencies. IRIS facilitated the role of LPEM in taking the lead, among Indonesian university organizations, towards meeting IRIS' objectives in the regional universities. IRIS endeavored especially to work with university economists in the new special autonomy regions, Aceh and Papua, to expand their capacity to assist in the implementation of the Special Autonomy Law. Together with Aceh and Papua, four other regions - West Java, East Java, North Sulawesi, and East Kalimantan - constituted a six-region focus, under the USAID mission statement. IRIS thus focused primarily on Aceh and Papua, secondly on the other four focus regions, and thirdly, on the remainder of the provinces of Indonesia. (Two provinces, West Java and East Kalimantan, were eliminated as priority target regions as some unspecified point during the last year.) IRIS conducted workshops in Jakarta and in several other locations, on several subjects related to decentralization, especially fiscal decentralization. Evaluation of the Universiry 5 July 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia Development Associates, Inc AEP-I-OO-OBOOO23-00, Task Order if 81.I IRIS brought distinguished foreign specialists to workshops. conferences and other special events. IRIS provided some back-up support to regional university faculty, as they advised local decision-makers on decentralization issues. IRIS helped to disseminate some training and reading materials, papers produced from research and advisory assistance, and other decentralization information gathered by the program in electronic form (CD Ram) to interested individuals and organizations. IRIS conducted an approximately 18 month program of research sponsorship among Regional University Indonesia's universities. This program solicited ~onsordum Leader research proposals, made selections among them, awarded substantial research grants, and provided monitoring and support that led ultimately, in July 2002, to the production of substantial papers focused principally upon fiscal and economic decentralization issues. These papers were presented and discussed at a conference in Jakarta. The university consortium activities under IRIS auspices began in October, 2000, WS did fulfill the objective to orchestrate multiple regional university research programs, which, under designated principal researchers, produced substantial papers which were submitted toward the end of the program period and were presented at a national conference of representatives from consortium institutions. Interviews have raised questions, however, both about the objective and about the impact of this relatively expensive program: 1. Several informants in regional university settings and in focus groups in Jakarta noted that a number of the IRIS research programs were already in progress, or were planned prior to the advent of IRIS initiatives. [IRIS funded in some regional faculty and research centers what these had already planned to do.] 2. Many consortium participants and foreign observers could not see how the research findings could be used. Some observed, however, that the research programs helped to build the capacity of faculty members (and their departments) to become more valuable teachers, and perhaps eventually become more effective contributors to local stakeholders outside of the university. 3. Interviewees regularly expressed the view that IRIS was interested almost exclusively in matters of fiscal decentralization, and seemed generally ignorant of other important decentralization challenges that had been enumerated in the IRIS scope of work. Evaluation of the Universiry 6 JuIj 2002 CoII~orntion andParliamenfory Assistance Adldl~iry: IRIS in Indonesia Development Associates, Inc. AEP-I-00-00-00023-00, Task Order # 814 4. Some participants commented on the inadequacies of the researcher/topic selection process (in the IRIS competition for research funding), noting a lack of clarity about criteria, competition process, favored foci and other aspects. (See recommendations.) While the IRIS management insisted repeatedly that the setting up of a university network had never been part of its scope of work - no one outside of IRIS seemed to comprehend the distinction between the concepts of "network" and "consortium". In fact, IRIS' own documents clearly refer to a "network," and these documents commonly interchange the words "network" and "consortium". The distinction may have some resonance, because of the fact that no consortium appears to have been established, except in the loosest sense of a network. A major finding, indeed, is that no functional consortium has been established. Events involving regional university faculty have, by and large, been discrete occasions - i.e., conferences, workshops, training sessions, on specific subjects - and have not been designed in any way to yield a consortium structure, leadership, by-laws, or any other vehicle to assure a continuation and further development of regional university cooperation beyond the current funding. One indicator of this failure is that no directory, for consortium use, of participating faculty has been produced that might enable dialogue across the regions, or between the regions and Jakarta colleagues and agencies. Activities of this IRIS program largely focused upon issues directly related to fiscal decentralization. Estimates from participant institutions agreed that 80-90% of the activities centered on fiscal decentralization, while the remainder tended to deal with legally related matters, most often involving training in the area of legal drafting. Some individuals felt that the main benefit that they had gained was some off-campus identification with "expertise" on fiscal decentralization. Almost all of the IRIS activities in this program occurred within the confines of participant universities' economics departments, so far as the higher education aspect of the regional involvement was concerned. (Note that in Indonesia, economics departments are almost all self￾standing departments, more like independent schools or colleges; they are not part of social science faculties, and normally have no structural or functional relationship with departments, such as political science, public administration, business administration, sociology, anthropology, communication, or psychology.) Individual faculty from departments other than economics might in some cases become involved in the IRIS activities, but normally only as contributors in support of economics faculty efforts. Although many dozens of discrete activities are listed in IRIS records, specific representative activities included: the set-up and delivery of training courses, e.g., in legal drafting and in the use of Internet and e-mail; the organization and administration of workshops on subjects, such as legal and regulatory changes, fiscal decentralization, and special autonomy; the sponsorship and funding of round-table TV discussions of fiscal decentralization, and the production of CD Rams covering the discussions, for subsequent distribution; the funding and mentoring of the development of economic databases and analyses. University participants constituted the great majority of individuals attending workshops and most other IRIS functions. For many faculty it was already a common practice to be directly Evaluation of the Universily 7 JuIy 2002 Collaboration ondPorIiumentory Assistonce Activity: IRIS in Indonesia Development Asociares, Inc. AEP-I-00-OD-00023-00, Tmk Ordm d 8IJ involved in regional government agencies and programs. before the advent of the IRIS activities. In this sense, strategic alliances with which IRIS could work already existed in the regions. Focus group sessions, as well as numerous individual interviews, revealed that individual faculty members and some economics departments greatly valued and benefited from IRIS and IRIS￾related activities. This was hue even when gains could not directly be attributed to IRIS. as when the chance to attend a workshop or training session had merely facilitated the continuation of an academic relationship already in existence, long before the advent of IRIS. A key design flaw in this program was the absence of any needs assessment. Without this base. many assumptions were made about targeted constituencies and institutions. These assumptions are directly manifested in the basic program objectives and implementation strategies. Chief among them, perhaps, is the assumption that "one size fits all," albeit with minor modifications. mainly to satisfy local sensibilities. Many interviews included strongly expressed remarks about how IRIS and/or LPEM, however well-meaning, consistently failed to ask first what the needs of regional university faculty, provincial stakeholders and other regional interests might be - before announcing what the Jakarta-based offices were intending to provide them. An ostensibly related finding is that the IlUS program conveys to nearly everyone who has observed its evolution - a lack of identifiable focus. This problem has been cited frequently by USAID officials, other program personnel, other donor advisors and staff, and by regional university faculty. According to IRIS' chief of party and several other principal players, the university program idea emerged from a casual conversation during a social occasion. This reportedly was also hue of the Parliament program. . - forays, without serious planning and preparation. A view repeatedly expressed to the evaluation team is that the university consortium cannot survive the termination of hnding. This opinion is often registered amidst observations about focus, and about the very limited and episodic nature of contact between IRISILPEM and the regional universities. At the special behest of USAID, Aceh and Papua L. ~~~ Both of these programs are opportunistic 'The Decentralization Conference [at Hotel Borobudur in Jakarta in July, 20021 seems to be seen as a kind of farewell for the university network." Typical comment heard 6om foreign advisors, IRIS personnel, and consortium members. received special attention in the regional university program, including many times the funding provided to other regions and program leadership by individual expatriate offrcers of WS assigned specifically to the respective regional tasks. These efforts succeeded in advancing the capacities of universities' economics faculties (and perhaps a few other individual faculty members, particularly in Papua) and in effecting modest impacts in the universities' relations with local stake holders. Given the program's narrow focus and limited window of opportunity, only very modest accomplishments were acknowledged by Indonesian participants and observers. At the same time, other regional university faculty members questioned the inordinate attention and knds devoted to Aceh and Papua, and directed some criticism at program bonafides. Evaluation of the Universiq 8 July 2002 Collaboration and Parliamentary Assistance Activiw: IRIS in Indonesia Development Associntes, Inc. AEP-1-00-00-00023-00, Tark Order tl814 E. RELEVANCE, EFFECTIVENESS, LMPACT AND SUSTAINABILITY The main elements of the regional university consortium and special autonomy province programs are intrinsically worthwhile. Observer and participant comments support this evaluation team's view that the efforts to enhance university capacity vis-his decentralization and to advance regional stakeholders' use of university resources - are laudable endeavors. The basic features and implementation of this program, however, raise serious concerns about relevance, effectiveness, impact and sustainability. Interviewees have repeatedly included remarks about "ad hoc," opportunistic shifts in direction, lack of focus, absence of clear program objectives, and failures of team-building, delegation and management. Few respondents speak confidently about the purpose of the regional university consortium, about its directions, and about its relevance and sustainability. The most effective IRIS activities were those (i.e., road shows, workshops, and dialogues) that enabled academics from various regional universities and government institutions to be drawn together to discuss issues of fiscal decentralization from an academic point of view. Without external funding, such get-togethers would not have occurred. The least effective aspect of IRIS activity was the absence of follow-up. Workshops and other events were predominantly one-shot events, and not even a directory of participants followed. Focus groups and individual interviews, for instance, yielded many complaints about the fleeting nature of the IRIS activities. Many individuals offered suggestions for improvement. (See Recommendations.) A number of academic participants interviewed by the team stated that IRIS has been a catalyst for them, in that their faculties (e.g., economics, and political and social sciences) formed a new research institution (Lembaga Kajian) on decentralization issues. Their interactions with IRIS during workshops inspired them to actualize these institutions - which they had been discussing since the passage of the local autonomy law in 2000. However, the usefulness of these units is still to be determined, as they are, at most, only 6-12 months old. Moreover, some faculty members claimed that these units would have been formed regardless of IRIS activity, and that currently IRIS had no role in them. The failure to assess needs, both before the program began and during the program's various shifts in direction, has contributed to doubts about its purpose and direction. The failure to conduct an effective, and more comprehensive evaluation (beyond immediate and sporadically used checklist responses) of workshops, conferences and training sessions appears to have perpetuated poor practices and passive participation. The failure to carry through on initial experiences and contacts and also the failure to continue to work in a sustained manner with faculties and stakeholders, has frequently left target individuals, departments, universities and others, with the impression of "single-shot" encounters, quickly forgotten. Evaluation of the Universiy 9 July 2002 Collaboration and Parliamentary Assistance Activity: NUS in Indonesia Development Associates. Inc. AEP-I-00-00-00023-00. Tmk Order a 8N The failure to build in individual universities a durable foundation of multi-department participation and 'ownership', such as was clearly anticipated in the program SOW. has weakened the program by narrowing the stakeholders inside the university. These have been predominantly members of economics departments, or persons even more narrowly identified as interested in fiscal decentralization. While eager to take advantage of program opportunities to improve their skills and to hrther their careers - even they questioned a program strategy that was so narrowly focused. The team found little evidence of communication with the range of USAID offices. This failure to consult regularly with other USAID offices and pro,mms resulted in a ,-dual distancing of key USAID personnel from the IRIS program and from a sense of investment and ownership in its outcomes, thus reducing the 'purchase' of this program in the minds and hearts of IRIS' central sponsor. Similarly, little effort was made with other donors. This failure to seek cooperation with other donors, or even to learn what they were doing, adversely affected the view of most allied donors concerning the regional university consortium -even when they admire the program objectives. This failure is especially notable at the Jakarta level (except for activities in the Papua region, where coordination has been exceptional between OTI and IRIS). These failures all bear on issues of relevance. effectiveness and sustainability. Few of the many respondents registered optimism about this program's sustainability, and many questioned the effectiveness of resource deployment over such a broad and inchoate geographic area of pluralistic and regional variety. IRIS' efforts have helped to generate nationally an embryonic university consortium, operating through conferences. workshops, trainings and selective intellectual ferment - as Indonesia's regions adapt to rapid steps toward decentralization. However, existing and potential channels of cooperation, reinforcement methods, and leveraging of limited resources have been ineffectively utilized by program personnel. While a few participants remain enthusiastic about the program, the preponderant view gleaned from the evaluation data is that planning and implementation flaws have contributed to disappointing outcomes. The regional university consortium and the special autonomy region activities fulfilled a small portion of the objectives set forth in the scope of work. The hnding level may have been inadequate to the program parameters, but the available hnds were not effectively utilized to establish a base among the participating universities, with sufficient def nition and programmatic durability. Ironically, the preoccupation with fiscal decentralization amid the broad consortia1 efforts discouraged the kind of cooperation promoted in 1RIS' own philosophy of economic development. Without a broader disciplinary engagement with decentralization issues, it was difficult to stimulate discussion and research of benefit directly to regional stakeholders. Even in narrow economic terms, for instance, the avoidance of any attention to the roles of the military and of the Indonesian-Chinese meant that an immense portion of Indonesian economic life Evoluafion of the Universiry 10 JuIy 2002 Colluborarion and Pnrlinmentory Asn'stnnce Anivity: IRIS in Indonesin Development Associates, Inc. AEP-I-00-OQ00023-00, Tack Order tf 814 remained outside the purview of academic discussion. Fiscal decentralization, according to many observers interviewed, from the point of view of most stakeholders, cannot be treated in narrowly technical and theoretical terms. Regular review by project managers of regional university consortium activities in terms of their consistency with lRIS and USAID mission strategic assumptions might have led to more successful outcomes. Evaluation of the University I I July 2002 CI Collaboration and Parliamentarv Assistance Activity: IRIS in Indonesia Since 1999, the House of Representative (DPR) of the Republic of Indonesia (RI) has greatly expanded its role in setting national policy and economic priorities. Increasingly it is becoming a critical decision-making body. The DPR-RI's powers and duties include making laws, and determining the State Budget. Its roles also include the supervision of the implementation of laws; implementation of the State Budget and the management of the State Finance; and the supervision of Government Policy, in accordance with the spirit of the 1945 Constitution and Decrees of the People's Consultative Assembly (MPR) of the Republic of Indonesia. DPR-RI has powers to discuss the results of the State Finance audit, matters for ratification, andlor to approve the declaration of war, and to review peace agreements, or any agreement and treaty with other countries made by the President. DPR Operates through 9 Commissions Commission I: Security and Defense, Foreign Affairs, and Information Commission 11: Law. Human Rights, and Internal Affairs Commission 111: Agriculture, Forestry, and Maritime Commission fV. Transportation and tn6-ashucture Commission V: Industry, Trade, Cooperatives, and Tomism Commission VI: Religion and National Education Commission VII: Social Welfare Commission VIII: Energy, Mineral Resources and Environment Commission UL: Finance, Banking, and Development Plan DPR-RI functions through nine Commissions and an ad hoc Budget Committee. It is supported by the Secretariat General. The Secretariat is tasked with the provision of technical and administrative assistance to the DPR-RI Commissions, in order to ensure the smooth execution of the DPR-RI's duties and responsibilities to the nation. All member of the DPR-RI are also members of the People's Consultative Assembly, which is the state institution with the highest level of power in the nation. According to the Cooperative Agreement SOW, the purpose of IRIS' support to the Indonesian Parliament is to "help the legislative bodies more effectively fblfill their new and expanded role in the emerging democratic government". Within this broad overarching purpose, the SOW had identified the following four objectives: b Improving parliamentary staff experience and skills in economic, budgetary. financial. legal drafting, and other areas, supplemented by domestic and expamate expertise. Evaluation of the University 12 July 2002 Collaborofion UndParfiamentary Assistame AcIiviQ: IRIS in Indonesia Development Associares, Inc. AEP-1-00-00-00023-00, Task Order # 814 r Strengthening the Parliament's and the Assembly's technical, legal and economic expertise and analytical capabilities. b Enhancing the quality of parliamentary deliberations and hearings - by influencing the way issues are approached, specifically through dialogue, the relative transparent presentation of points of view, and the bipartisan consideration of options based on technical merits. t Enhancing the impact of USAID's and other donors' activities, in regards to the legislature, through IRIS'S donor coordination support to the Parliament. It was stressed during discussions with IRIS' management and staff that the aim of IRIS' support to Parliament was the strengthening of the legislative and policy making capacities of Parliamentary committees (commissions), and the development of long-term intellectual capacity within the DPR-RI Secretariat General's Center for Research & Information Services (PPPI). In addition, IRIS' stated exit strategy suggests that its direct support to Parliament will end when the following three results are achieved: 1. The staff of the PPPI are trained and functioning. 2. The DPR-RI members have full access to, and are able to utilize professional staff support. 3. Adequate funding is made available for the DPR-RI operation. According to the Cooperative Agreement, IRIS was to implement the following activities, so as to generate the above stated results: t Conduct bi-monthly issues education seminar & meetings for lawmakers and staff t Provide short-term expert analysis and advisory services, with 5 studies leading to policy dialogue. b Provide informational (i.e., in-depth workshops) and technical support (i.e., series of studies on particularly complex set of economic, budgetary or financial issues). t Provide graduate studies and internship programs for Parliamentary staff. r Provide training by US Congressional staff and other expatriates. b Provide training of Parliamentary staff and legislative aides. According to informants, Parliament has had since 1999, a broad rolelmandate in policy making - but is generally viewed as weak, compt, and lacking in the capacity to be effective. In addition, according to informants and to a recent survey reported in "Jakarta Daily," Parliament is viewed as being out of touch with the population and disregarded by the Government. According to the DPR-IRIS team leader and to IRIS documents reviewed by the team, IRIS was asked by USAID to meet with the DPR leadership and to assess how IRIS could be of assistance in strengthening the operation of the parliament, with a focus on the areas of finance and economics. Initially there was resistance to having USAID provide assistance to the DPR, since it was deemed to be too sensitive politically. After some internal changes within the Evuluution of the University 13 JuIy 2002 Collaboration andParliamentan, Assistance Activity: IRIS in Indonesiu Development As.wciates, Inc. AEP-I-00-00-00023-00, Tark Order fi 814 In addition to hiring internationally recognized expatriate experts, IRIS hired domestically professionals who were well known in their fields. For instance, IRIS Indonesia management indicated that they had engaged a prominent former minister of defense, who had served under the now-deposed Suharto military regime; this man was a key government official who originally, in the early 1980s, set up the parliament's staff secretariat to work on donor coordination tasks for IRIS. However, our interview with this individual revealed that. although his name was associated with IRIS, he did not have a contract with IRIS, and was not engaged in donor coordination. His only involvement was compiling activities carried out in support of Parliament by Partnership-UNDP, KONRAD, and by the Asia Foundation in 200112002, with the promise of future employment with IRIS. With regard to IRIS' Cooperative Agreement with LPEM to provide support to parliament, initially there were efforts to use university faculty research papers to brief Commission members, but this program was later abandoned, as the papers were too technical for non-experts to comprehend. Our LPEM informants expressed the opinion that the parliamentarians and their staffs were not on the same intellectual level as the university faculty - to be able to discuss government policy, or to understand the policy papers presented by LPEM. At the same time, PPPI researchers did not have the technical capacity and skills to summarize and present, in non￾technical terms, these technical papers to parliamentarians and to their staffs. Thus the link between the Indonesian universities and parliament appears to be very weak. In addition, IRIS provided expert economic and legal advice, as well as research assistance, through the provision of long term resident advisors. An expatriate expert and team leader (on the job for 15 months) was hired to work with the chairman of Commission LX. In addition, an Indonesian economist (on the job for 12 months) was hired to work with Commission IX and with the staff of PPPI; and an Indonesian lawyer (on the job for 9 months) was hired to provide legal advice and to assist in legislative drafting. This team has been assisting PPPI staff and members of the Budget Committee by developing a list of inquiries, researching data from the Ministry of Finance, conducting budget analysis, and participating in concluding meetings. According to documents reviewed and interviews with the DPR-IRIS team, the IRIS team has provided input to the Special Committee (Pansus) for the State Budget Bill, which discussed the State Finance Bill, State Treasury Bill and State Finance Audit Bill. The input was achieved through coordination with the Supreme Audit Agency (BPK), and other related departments. In addition to providing advice to the Chairman, the IRIS team helped set up the Budget Commission's agenda and also lobbied for the Commission's agenda. In setting up the Commission's agenda, the Indonesian economic advisor invited experts to discuss technical issues related to the Commission's work. These discussions were to take place during the Commission's scheduled meetings, so that the parliamentary members were obliged to attend. It should be noted that the Commission's agenda was set single-handedly by the Indonesian economic advisor. There is no system or process for tapping expert advice andlor a data base for technical data. Several sources suggested that the assistance provided by the IRIS Indonesian economic advisor to the Budget Committee had resulted in enhancing the quality of the budgetary dialogue between parliament and the executive branch and had reduced the time required to pass the Evaluation of the University 15 Jub 2002 Collaboration andPnrliamentary A.~sistanee Activity: IRIS in Indonesia Development Associates, Inc AEP-I-00-OD-00023-00. Tosk Order $811 Commission IX leadership -the DPR-IRIS team leader met with Dr. Benny Pasaribu. the new chairman of Commission IX3. It was only after the team leader explained to the chairman that "the nature of our (IRIS) support was technical human resource development and institutional capacity building" and that IRIS "has no intention to engage in or support any political agenda,'J that the chairman agreed to proceed in the design of IRIS' support program. Thus began IRIS' assistance to the Parliament and to Commission IX, which was responsible for the state budget. and for all finance and banking issues. IRIS' assistance to the parliament was initiated mainly on the basis of IRIS' credentials and due to the team leader's personal experience. According to a number of informants, IRIS' parliamentary support activities did not, however. begin with a systematic needs assessment. Activities undertaken or supported by IRIS are summarized in Table 3.1. With regard to training, IRIS has facilitated the overseas as well as the domestic training of selected individuals from the DPR-RI Commission IX and the DPR-RI Secretariat. For example. with regard to overseas training, IRIS conducted the selection and assisted the approval process for scholarships for post- ,gaduate studies and for short-term training in the United States. One individual from the PPPI and one person from the Budget Committee were selected and sent for a Master Degree program at Georgia State University in the United States. One PPPI researcher attended a three-week seminar on Fiscal Decentralization in Developing Countries & Transition Economies at Georgia State University. In addition, the Chairman of the Budget Committee went on a study tour to the United States to observe how the US Congress functioned. With regard to domestic training, IRIS paid for workshops, travel expenses, lodging and per diem of selected PPPI and Budget Committee staff to attend workshops and seminars within Indonesia. For example, 3 PPPI researchers attended a two-day conference on "Smuggling and Trade of Human Beings and Other Transnational Crimes"; 2 Budget Committee staff attended a one-day workshop on "Drafting of the Bill for Regional Development Budget (RAPBD); 6 PPPI economic researchers participated in an IRIS sponsored "Economic Model Trainine and 9 PPPl staff and 11 persons from the Ministry of Justice attended the Economic Law, Institutional & Professional Study (ELIPS) sponsored "Advanced Legal Drafting Training" in Jakarta In addition. 15 PPPI researchers were given English language training. From time to time, IRIS had engaged several recognized expatriate experts for short-term consultancies to address specific issues. For example. Chris Manning conducted a seminar on minimum wage. Professors Jorge Martinez-Vasqua and Roy Bahl worked with the Government of Indonesia (GOT) and LPEM and offered advice on fiscal decentralization and revenue sharing formulas. Dr. Anthony Lanyi and Anita Doraisami provided advice on national economic policy. Dr. Lanyi worked with EKUIN and LPEM to develop a macroeconomic policy analysis capacity, in order to provide high quality independent economic advice to the GOI. Dr. Patrick Meagher held high-level discussions with the Attorney General. members of the Anti-Cormption Commission, and with government and non-government lawyers. ' Gajewski, Peter, Program Development and Sustainability Assessment for IRIS' Support for the DPR and the DPR Secretariat, July 27,2002 ' ibid Evaluation of the University 14 July 2002 Collaborm.on and Parliamenta~ Assislonee Activity: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00 Tark Order # 8N budget bill. For example. the 2002 budget bill took only 3 weeks to pass. as compared to more than 3 months for the 2001 budget bill. In addition. the 2002 budget bill contained less error than did the 200 1 budget bill. The Indonesian economic advisor has earned hish respect among selected members of Parliament, and this has helped to change the 'mindset' of Commission staff. According to some informants, parliament is now more appreciative of the benefit of analysis for budget decision-making. The IRIS team. assisted by the Institute of Economics and Management Research Institute of the University of Indonesia, is in the process of developing and installing a budget database within PPPI. A simplified, Microsoft Excel-based budget simulation model has been developed for internal analysis, and as a staff training tool for PPPI and the Budget Committee. According to our sources, the Microsoft Excel-based budget simulation model is currently the only model being used by the DPR-RI staff. In addition, IRIS has donated a selection of books to the PPPI library. However, our review of the library checkout register revealed that these books were not being used by the DPR Commission staff or by the PPPI researchers. The library register showed that a few books were loaned out to students. The Indonesian economic advisor and several sources pointed out that IRIS' scope of work and programs were not explained to the PPPI staff, nor to the Indonesian resident advisors. Several sources also indicated that IRIS' program implementation techniques were ad hoc and not systematic and strategic. Selected PPPI research staff indicated that, except for the few workshops and seminars they attended, as well as the two scholarships to the United States - IRIS did not provide them with a focused and systematized training to enhance their research and policy analysis capabilities. While they appreciated the informal and occasional one-on-one discussions with the resident Indonesian economic advisor, the assistance they received was not sufficient to impart significant research and policy analysis skills. The resident Indonesian economic advisor did indicate that he had only devoted between 10 to 15 percent of his time to assist the staffs of PPPI and the Budget Committee. Several informants indicated the belief that IRIS did not coordinate its support to Parliament with other donors nor coordinate the support of other donors to Parliament. IRIS points out that it had provided technical assistance for the preparation and implementation of workshops and seminars funded by the Asia Foundation and KONRAD. That assertion is borne out by IRIS documents. Several sources have suggested that IRIS did not collaborate with other USAID-funded programs. There is some evidence, however, that IRIS has collaborated with ELPS. IRIS provided technical assistance for the preparation of selected conferences, workshops and seminars, and funded selected PPPI researchers and Budget Committee staff to attend these ELIF'S conferences and workshops. The DPR-RI has made an in-kind contribution to WS by providing furniture and office space in the parliament complex to house the resident advisors and their research assistants. Evaluation of the University 16 Juiy 2002 Collabor&'on end Parliamentary Assistance Adldlvity: IRIS in Indonesia Deve/opmentAssociates, Inc. AEP-I-00-00-00023-00, Tark Order # 814 E. RELEVANCE, EFFECTIVENESS, IMPACT AND SUSTAINABILITY Relevance: IRIS' support to the DPR-RI is relevant to the extent that it sought to address and strengthen DPR-RI's analytical and technical capacity in legislation and policy-making, at a time when DPR-RI was becoming increasingly responsible for the nation's fiscal and economic policies and laws. Effectiveness: The effectiveness of IRIS' assistance towards enhancing the long term intellectual capacity of the DPR-RI to understand the economic, political and social implications of the pieces of legislation being considered or reviewed, as well as the policy analysis capacity of PPPI - are questionable. IRIS' technical assistance personnel have largely functioned as integral components within the contingent of parliamentary personnel, and not just as supplementary enhancements to parliamentary commission work. There is very little evidence to suggest that the legislative and policy making capacity of Parliamentary committees (commissions) has been strengthened, andlor that long-term intellectual capacity has been developed within the Center for Research & Information Services. For example, while the assistance provided by the resident tndonesian economic advisor to the Budget Committee resulted in enhancing the quality of budgetary dialogue between parliament and the executive branch, and reduced the time required to pass the budget bill - this success is attributed mainly to the network of friends that this particular individual had maintained in the ministry of finance and in other government entitiess. Currently there are no individuals, within the PPPI or the Budget Committee staff, who have similar access. And IRIS is not engaged with the Ministry of Finance. In addition, there is no evidence to indicate that the training that the PPPI researchers have received through the IRIS program has enhanced their research and policy analysis skills. IRIS' training implementation techniques were ad hoc. Out of 42 PPPI researchers, less than 15 percent were exposed to limited economic policy analysis training. Except for the one individual who was sent to Georgia State University for a Master Degree, the training received by the other participants was episodic and inadequate to sufficiently prepare them to conduct quality policy analysis and research. Furthermore, the training and expert advice was exclusively provided by US andlor Australian citizens. This has potentially limited the intellectual dialogue, knowledge and skills to US and Australian experiences. IRIS has failed to develop the capacities of LPEM counterparts to deliver policy analysis, technical assistance and training to Parliament. Impact: The work of the resident Indonesian economic advisor has facilitated and shortened the time required for the passage of the national budget. This economist interacts with many people in the DPR-RI, and has created a good understanding between the DPR-RI and the ministry of finance on the national budget. He has been instrumental in influencing parliamentarians' views 5 IRlSAndonesia management did indicate that this Indonesian economic advisor was hied partly due to his strong ties with senior officials at the Ministry of Finance. Evulaafion of the Universi* 17 July 2002 Collaboration and Pnrliamenlary Assismnce Activity: IRIS in Indonesia Development Anocinfes, Inc AEP-I-OaOO-00023-00. Tad Order f 81J on the law. On the other hand, IRIS' capacity-building activities at the PPPI have had little or no impact on the PPPI researchers' capacity to conduct quality policy analysis and research. Sustainability: Parliament has made an in-kind contribution by providing ofice space for the IRIS consultants. In addition, it has obtained additional funds to hire or contract Indonesian professionals to help the parliamentary commissions. The amount of funding it obtained from the ministry of Finance is, however, too small to attract good talent. Furthermore, IRIS' projection of the centrality and indispensability of its parliamentary function appears to have diminished parliament's incentive to develop its own, fully independent parliamentary staff infrastructure (beyond the current capabilities universally recognized to be negligible and counter-productive). Thus the gains made to-date cannot be maintained without continued IRIS involvement. Even so, IRIS' successes with the budget bill will end when the resident Indonesian economist leaves his post andlor when and if the budget committee leadership changes. With regard to capacity-building, IRIS has provided training to selected individuals from the DPR-RI Budget Commission and from PPPI. It has also assigned long-term resident experts to work directly with DPR-RI. However, IRIS has not successfully met its objective of strengthening the legislative and policy-making capacities of parliamentary committees, and of building long-term intellectual capacity within DPR-RI Secretariat General's Center for Research & Information Services. In addition, there is no evidence to suggest that IRIS has coordinated effectively other donors' support to Parliament. However, IRIS has successfully supported Commission IX - Budget Committee with regard to the 2002 budget bill. Evoluotion of the Universip 18 July 2002 Collaboration and Porliomenfary Assistance Aen'vify: IRIS in Indoneria Development Associates, Ine. AEP-I-00-00-00023-00, Task Order & 814 TABLE 3.1 IRIS'S Parliamentary Support Activities (2001 - 2002) Training /TffhniealAssistaaec Phce & Date - I IRIS conducted selection and assisted the I ~anuary 200 1 approval process for a training program on ~enny Pasaribu lRIS technical assistance to the PPPI for 1 Jakarta. preparation and implementation of discussion I 0ctobe; 18,2002 on the Bill for Sports Development English language training. IRIS provided I Jakarta, teaching staff for TOEFL course I July - Nov 2001 * IRIS SDO~SO~~~ "Economic Model Training" - I Jakarta I July - NOV 2001 ELIPS sponsored training on Pre-LL.M I Bali, ~rogra~ & Introduction to American Law I Feb -April 2002 ELIPS s~onsored discussion reeardine the 1 Jakarta. possibil& to form a Bill ~eveg~men; Unit led by Professors Ann Seidman and Bob Seidman Training on English Language and Law at the I person from the Budget Committee I PPPI researcher Feb 2002 Jakarta, - - ~niversjty of ~idonesia ELIPS sponsored "Advanced Level Legislative Drafting Training" Legislative drafting training workshop and course. Chairman of Budget Committee 3 PPPI Researchers April - May 2002 Jakarta, May 2001 Jakarta, Oct - Nov 2001 2 Budget Committee staff PPPI Research Staff Chairman, Commission IX PPPI Research Staff Members of Commission vi . - 15 PPPI Staff 6 PPPI Economic Researchers 2 Setjen Staff Chairman & ASSES I Staff Chairman & PPPI Researchers 3 PPPI Researchers 9 persons from DPR-RI I I persons from the Ministry of Justice 6 persons from SEKNEG Secretariat Staff Evafrrotion of the University 19 July 2002 Collaboration and Parliamentary Assistance Aetiviry: IRIS in Indonesia CHAPTER 4: IRIS LPEM PARTNERSHIP A. PROGRAM PURPOSE The IRIS-LPEM partnership was established through a USAID ,pnt to IRIS-LPEM. under the Partnership for Economic Growth (PEG). IRIS was the prime grant recipient and received a total grant amount of S750,OOO. LPEM was IRIS' 'prime partner' and a sub- IRIS-LPEM Grant Objectives grantee. From January 2001 to May 2002, LpEM received $198,548 (approximately 1. Update and expand regional university facult?. 26% of the total grant am~unt).~ members' understanding of decentralization issues 2. Update and deepen the understandig of decentralization issues by political parties. civil The stated purpose of the IRIS-LPEM society, the private sector and government in Grant Promam is to "...increase the regions served by the regional universities capacity of a select set of (15 to 20) 3. Encourage regional university faculty members to incorporate the additional decentralization regional universities to analyze and understanding in their work understand the issues and challenges of 4. Provide mechanisms and incentives for oo~itica~ ~~~~ ~~~~ ~~~-~~~- -~ political, administrative and fiscal parties, civil society. the private sector and decentralization."' The premise underlying government to use their enhanced understanding the grant is that regional universities are a and regional faculty expertise in their decentralization policy making that can be tapped to 5. Develop productive relationships between the Indonesia's movement towards University of Indonesia and regional universities for decentralization. Through their increased their work on decentralization issues. capacity, regional universities will be able to provide higher quality, more useful advice to a r&e of stakeholders - political parties, government officials. civil society organizations, and the private sector - on the important and complex issues surrounding the devolution of political and economic power. These regional universities are located in Sumatra. Kalimantan, Sulawesi, Irian Jaya, Bali and Java. INS-LPEM activities carried out under the grant, focused mainly on training sessions (i.e., training of trainers approach) and workshops. The grant included five main activities: Activity One: In-Depth Training for Selected Regional University Faculty. - Two workshops, 5-10 days for 50 faculty members; one (optional) additional workshop for faculty needing more instruction. Workshop topics: Decentralization in Indonesia Framework for Fiscal Decentralization in Indonesia (optional) Implementing Decentralization in Indonesia 6 IRIS Budget and Expenditure Data provided by IRIShdonesia Office, June 2002. IRlSLPEM Decentralization Grant: Life of Grant and Year One Work Plan, n.d., mimeo (provided to evaluation team by IRIS) Evaluarion of the University 20 July 2002 CoII&or&'on md Parliamenlary Assirranee Auiviq: IRIS in Indon& Development Associates, Inc. AEP-1-00-00-00023-00, Tuqk Order # 814 Activity Two: Ten Regional Two-Day Workshops - Regional universities will hold workshops for members of political parties, civil society, private sector and government officials - Workshop material drawn from Activity One in-depth training Activity Three: Ten Jakarta Workshops for Political Party, Civil Society, Private Sector and Government Officials - Conducted by LPEM faculty - Focus on topics of particular importance to Jakarta Activity Four: Eleven Joint Regional University, LPEM and IRIS Research Programs - Request research proposals from regional university faculty participating in Grant on a decentralization issue of particular importance to their region - Provide paid time (approximately three months) to prepare proposal, research assistant, IRIS and LPEM assistance and IRIS oversight Activity Five: Information Dissemination - Access to training materials, research and advisory assistance papers, related reading materials on decentralization FINDINGS AND CONCLUSIONS IRIS-LPEM PARTNERSHIP-PROPOSED MONITORING AND EVALUATION PLAN IRIS proposed in the grant work plan an ambitious performance monitoring agenda that would monitor and evaluate the grant activities in a variety of ways: * Monitor all workshops in terms of number of participants, groups and interests represented by them, gender of participants, and geo-political factors represented by participants; + Implement participant questionnaires to identify strengths, weaknesses, successes and failures; + Collect baseline data on 'in-depth workshop' participants (their initial state of knowledge and understanding) and document their ending state of knowledge and understanding; * Conduct tracer study participants of all workshops (near the end of the grant period) to identify how they have used the workshop knowledge and information; * Survey to document the ways research results and advisory assistance affected policy dialogue and decision makers; * Collect media coverage of grant activities; and Evaluation of the University 21 July 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia Development Associorer. Inc. AEP-I-O&OO-00023-00. Tmk Order ;! 831 * Conduct a variety of surveys of regional faculty, political parties, civil society, the private sector and local governments, in order to assess their expanded understanding of decentralization issues and how they are using this information in their professional work and in official deliberations. Finding: The pledged information from these diverse efforts was to be reported in the USAID PEG Quarterly Reports. The team reviewed the Quarterly Reports for 2001 (none for 2002 were available to the team), the IRIS-LPEM Annual Report and the weekly progress reports - and found information only on the list of workshop participants and media coverage. The information in the quarterly and weekly progress reports was limited to 'transaction' (i.e.. output) data - the number of workshops delivered, who attended the workshops, who provided technical assistance, and so forth. Also of note was the absence of any information on the actual use of the knowledge and information presumably conveyed in the training and workshops. s Conclusion: IRIS did not implement the proposed monitoring and evaluation system that would generate credible performance data and information on the IRIS-LPEM partnership. In the absence of the proposed monitoring and evaluation information. the team relied on comments and assessments of IRIS results, that had been provided by several people interviewed by the team for the evaluation and the results from a focus group of IRIS participants (see protocol and list of focus group participants in Appendix). 2) RELEVANCE OF THE IRIS-LPEM PARTNERSHIP Finding: The reported relevance of the IRIS program. noted by the LPEM staff, pertains to the program's 'road shows.' The activity was seen as effective in identifying economic issues and in communicating to people in regions their importance (i.e., a form economic education). LPEM staff also saw their IRIS involvement as increasing their visibility within Indonesia as a center of decentralization expertise. They remarked that government officials were increasingly seeking their advice on this issue. The focus group discussion produced a similar finding. The participants felt that the program was very relevant to needs of their respective regions. Faculty at universities in remote and poor regions felt that the program was particularly relevant, since it helped them gain access to information and research support ordinarily unavailable to their faculty. 8 One seeming exception to this conclusion was a 'mail survey' of parlicipants in three fiscal decenmlization workshops coiducted by IRIS. The survey was undated. therefore it is not obvious when thesurvey wasconducted, or if the survev immediatelv followed workshoo . oarticioation. or was a follow-UD insuirv. Also. the samole frame . . .. for the survey was not discussed, and therefore the representativeness of the results is unknown. The survey instrument was not included: therefore it was not possible to examine its methodological appropriateness. The survey revealed that the majority of the participants - while they felt that the workshops were useful - were already participating in "... various decentralization activities" before their participation in the IRIS workshops. The survey asked only if respondents were involved in decentralization activities: it did not examine exactly how the IRIS workshop experience improved their capacity to advise or consult on fiscal decentralization. The evaluation team. therefore, concludes that this survey, in addition to being methodologically flawed, does not shed light on the follow-up issue. Djafaar, Isfandiary, 'Evaluation of the Questionnaire of the IRISLPEM program on shen,*ening regional universities within the perspective of fiscal decentralization' (IRISIJakarta Program Office, Mimeo, n.d.). Evaluation of fhe Univemiy 22 July 2002 ColI~borafion and Parliomenfary Assistance Acfivily: IRIS in Indonesia Development Associaks, lnc. AEP-1-00-00-00023-00, Ta~k Order # 814 The discussants said that their IRIS participation was useful in increasing their knowledge and understanding of decentralization issues, which enhanced their capacity to provide advice to their regional government. They also felt that the regional university network put them in contact with university faculty from other regions, which was relevant to keeping up with new funding opportunities and research materials. Conclusion. The IRIS-LPEM partnership activities were viewed as relevant to the need of regional universities to upgrade their skills and knowledge in the area of fiscal decentralization. 3) EFFECTIVENESS OF THE IRIS-LPEM PARTNERSHIP Finding. IRIS was credited (by LPEM) with helping LPEM gain access to USAID funding, through its role as the sub-grantee 'partner' to IRIS. LPEM also asserts that IRIS support (see Text Box) enabled it to increase its capacity in the fiscal decentralization area, which, in turn, heightened its visibility as a source of useful expertise on the important issue for Indonesia kal decentralization. The Govenunent of Indonesia recognizes this increased capacity by expanded requests for assistance. Serendipitous Effect LPEM was in competition with other Indonesian universities to develop an 'inter-governmental transfer formula'. IRIS arranged for an international advisor to provide technical assistance to LPEM on the development of a transfer formula. This enabled LPEM to win the competition and, thereby, to establish its 'profile' as a source of expertise on fiscal decentralization. Source: LPEM Evidence of IRIS' effectiveness is limited to activities involving the regional university faculty, and to a limited extent, government officials in one region (Papua). Little if any information was available on the effectiveness of IRIS activities involving political parties, the private sector, civil society or local governments. The team interviewed one local government official from Papua who reported that IRIS activities were very useful in his region in preparing university faculty to provide budgetary preparation assistance. Although university faculty had advised the local government prior to IRIS, he said that he felt IRIS had increased their capacity, especially on the topic of fiscal decentralization. The impact of IRIS activities on political parties, the private sector or civil society in regions was not discussed in any appreciable detail in the IRIS program documentation. In the area of computer support, IRIS provided some computer hardware to LPEM. It also supported, to some degree, the further development of an existing LPEM budget simulation model and associated database. The model reportedly is still in development. Many of those interviewed associate IRIS with the operation of the regional university network - and this is mentioned as the most visible indicator of IRIS' effectiveness. Although IRIS states emphatically that development of the regional university was not part of its original work plan, it nonetheless cites the creation of the regional university network as a valuable result of its activities. Evaluation of the Universify 23 July 2002 Collaboration and Purliamentary Assisfanee Activify: NUS in Indonesia Development Associates, Inc. AEP-I-00-00-00023-00, Task Order Y 81.i Interviews with LPEM staff revealed that the network came out of an IRIS-LPEM brainstorming session held at LPEM. Apparently (according to LPEM), LPEM identified a list of regional universities that potentially could be a resource for providing advice to governments on decentralization issues. IRIS then used this initial list to identify a group specific regional university faculty that IRIS could work with within its training-of-trainers approach (i.e.. wo~kshop series) to develop a network for training on decentralization. One interviewee stated that the network would not have existed without IRIS' effort and support.9 The network is intended to provide ready access to a 'regionally-based resource' that can be used to facilitate the decentralization process throughout Indonesia, through the provision of sound and useful advice to various decentralization stakeholders (eg, political parties, civic society). The team's interviews produced mixed reviews on the effectiveness of the network. One person noted that, prior to the network, Indonesia had no history of inter-governmental relations. The network helped to meet this need, to a certain extent, by being a potential venue for cooperation. The Ministry of Home Affairs (MOHA), for example. used the nenvork workshops to launch Law 22 (local decentralization) in the regions. The network workshops provided a useful forum for the MOHA "...to get its message out to the regions on Law 22."" Another interviewee, on the other hand, downplayed the significance of the network, stating that he doubted Source: ~iswy of ~ome that the network had much of an impact. It should be noted that the network activities pertain to relationships between individual faculty and other organizations, such as government and other universities. The network is not based on inter-institutional relationships, and no formal agreements among network "institutions" (e.g., regional universities) exist. The team found no institutional or organizational vestige from the network activities. Conclusion. IRIS activities helped to expand the capacity of the participants in the IRIS￾LPEM grant, in the area of fiscal decentralization. This result pertains mainly to LPEM as an institution, and to regional faculty members as individuals. The impact of the IRIS-LPEM grant on government operations and decision-making is suggested but not fully demonstrated. Absent is any convincing evidence of the impact of IRIS-LPEM ,pnt activities on the private sector, civil society or political parties. The regional university network is viewed as a useful but underdeveloped resource. It may be the most important result, but it exists as an informal, unorganized, unofficial collection of individual relationships. 4) THE SUSTAINABILITY OF THE IRIS-LPEM PARTNERSHIP Finding. The team found no evidence to suggest that the IRIS-LPEM partnership (i.e., the activities subsumed under the proposed partnership) is sustainable. LPEM has been successful Interview with World Bank staff member 'O Interview with advisor to Minishy of Home Affairs on local decenwalization. Evaluation of the Universiry 24 July 2002 Collaboration and Parliamentary Assistance Aetiviry: IRIS in Indonesia DevelopmentAs.~ociates, Inc. AEP-1-00-00-00023-00, Tmk O~der # 814 in attracting donor funding. The IRIS-LPEM grant was but one of several LPEM grants underway. The IRIS-LPEM grant work plan did not contain a specific sustainability strategy, and the actual grant activities appeared to be implemented, for the most part, as discrete actions involving mainly individual capacity-building and network relationships. The workshops and training were discussed by participants in terms of building the capacity of individual faculty members in the area of fiscal decentralization This enabled faculty members to expand their repertoire of advisory activity to include that topic. A common lament of the training, however, was the absence of follow-up. It seems that faculty members were approached about training opportunities, recruited for the training, trained, and then contact with IRIS ended. There was no follow-up to trace their use of the training in their work, and no attempt just to maintain contact. The fact that the training was limited to fiscal decentralization was seen as a limitation by some of the participants. Some felt that an equally important topic was local government accounting since, with the passage of Law 22, local governments were tasked with the responsibility of properly managing the additional monies that they would be receiving. Those desiring local government accounting felt that the decentralization workshops were too general to meet their needs. In a few instances, faculty members mentioned that interactions with IRIS staff stimulated them to think about creating some type of institutional base (e.g., a research center) for their work, but that IRIS was not involved in its actual planning or creation. This was left to the individual faculty members. Site visits suggests that several of the 'centers' existed on paper prior to IRIS, and only became activated to some degree in response to IRIS' input. The 'regional university network' was cited by several interviewees as the most demonstrable and lasting indicator of sustainability. Supposedly, it would continue after the grant to be a source of credible advice and assistance to decentralization stakeholders. The team's review of the network's modus operand;, however, suggests that the network is basically an informal group of individuals that have been identified as potential sources of advice and consultation. The group lacks any legal status or organizational structure. It surfaces intermittently in response to a workshop call and then withdraws to its normal dormancy. This observation does not gainsay the fact that regional university faculties occasionally provide advice to, for example, local governments. Rather the team found that they generally provide their advice as individual experts, and not under the auspices of a so-called regional university network. Conclusion. The network is a potentially valuable resource that needs to be organized and supported, so that it can continue to be useful to local governments and to other relevant stakeholders. Evaluation ofthe University 25 July 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia Please note that this report treats training as a cross-cutting issue relevant to several IRIS program activities. The SOW-related issues have been addressed in several chapters of the report. The specific training issue and the section of the report where it is discussed are as follows: 1. What has been the mix of IRIS training, policy development and reform activities in Parliament and in the university consortium? Parliament: Findingssection andTable 3.1 2. To what extent has this mix of activities under the IRIS program been appropriate in the Parliament and in the university consortium? Has it been inappropriate and in what way? Evaluation of the University 26 JYly 2002 Collaborarion and Parliamenfary Assisfance Ammvig: IRIS in Indonerio Parliament: Findingssection 3. Have these activities contributed to the long-term sustainability of Indonesian policy development capacity? Why or why not? Parliament: Findingssection Expenditure Analysis: Conclusions section Certain potential and actual management deficiencies were noted in a previous evaluation of part of the IRIS activity (i.e., four PEG grants) that was completed in November 2000." In the SOW for this evaluation, USAID asked the team to assess the extent to which these various deficiencies have been resolved, or if they were still a concern? The team was also tasked with assessing the current IWS management capacity and performance. The following discussion will examine how IRIS has responded, in the period following the mid￾term evaluation (November 2000 - June 2002) to the deficiencies cited. The findings and conclusions are based on a review of IRIS documents (i.e., weekly progress reports. quarterly reports and final reports), interviews with IRIS staff, interviews with USAID staff and management, and interviews with other IRIS stakeholders in Jakarta, and during field visits to various regions. The deficiencies noted in the mid-tenn evaluation. the findings from the Team's assessment and - the Team's conclusions are summarized in the following matrix: 'IRIS-LPEM -has no ~erformance monitoring Lystem whatsoever .... idditional emphasis needs to >e placed on tracking nrennediate results and wentual impact, rather than ust program outputs." Financial Monitoring: "IRIS ... could not orovide my expenditure data in the ield ... Atter more than a month from [he ~nitial requesS IRIS' home office ncver responded to the Evaluation Team's information request ... to hire a full time writerleditor to prepare IRIS reporting materials. * The team was told that the weekly progress reports were the IRIS' mechanism for performance monitoring reporting. * IRIS response was to hire a full time financial administration person to handle budget and expenditure activities. to "han&ction data" that lists program activities and associated outputs. 2. IRIS repotting is not results-based. does not repon on either intermediate or ultimate impacts. or implemented a performance monitoring system in the accepted meaning of the term. as used by USAID. It has failed to respond to thc mid-term deficiency. I. The WS financial -, IRlS has responded to management system , the deficiency and is able is a basic line-item 1 to provide basic he-item budget and : budget and expenditure expenditure system. ! data in the field. 2. The IRIS System is +The IRIS fmancial not a performance- management scheme is not based budgeting set up to prov~de system. performance-based budget " Insert reference Evaluation of the Universily 27 July 2002 Collaborafion and Pndiamenlary Assistance Acfiviry: IRIS in Indonesia ---- - Development Associates, Inc. AEP-I-00-00-00023-00, Tnsk Order # 814 Impact: he mid-term evaluation ". .. uestions whether the initial rant impact can be sustained rough an attempt to utilize niversities as a platform to ffect reform of economic ovemance. ... Without sntinued external fmancing ich advocacy efforts are nlikely to survive. Iniversities, by their very ature, do not build ~nstituencies for refo rm..." ; the university consortium esigned to ffect/iifluenceihelp shape :form of economic overnance? Is this strategy uitahle? iustainability: he previous evaluation ummary states that "the Jniversity-based advocacy :fforts do not appear to be ustainable institutionally ~ecause they lack natural :onstituencies for reform" and ilso recommends that the exit ;uategy needs to he carefully pelled out. * IRIS helped develop the consortium (with LPEM) and has used it to access regional university faculty for training sessions, workshops and conferences. *INS states that there was no expectation of sustainability for the current funding period nor any expectation of an exit strategy. 3. The financial manager was very cooperative in providing fmancial data to the team. I. People interviewed for the evaluation saw the consortium as a very useful activity that should he continued. 2. Some regional university faculty stated that their consortium contact was related to their being asked for their advice by governments. 3. IRIS did not systematically collect results data on the impact of consortium activities 4. The consortium lacks any legal status 5. It has been organized and utilized as an informal, loosely linked collection of individuals 6. It lacks any discernible organizational or institutional t7amework. 7. No plan for the sustainability of the consortium has bee] developed. 1. Ad hoc initiatives have characterized the advocacy efforts, and the bulk of activity has been in the special autonomy regions￾areas of maximum political, economic anc social turmoil. nd expenditure data that inks program :xpenditures to ntermediate and ultimate esults (i.e., impacts). i The consortium has the ~otential to be a :ontinuously useful nstrument for regional lniversity capacity nilding The consortium should ~e formally organized and unded to permit its full ievelopment and enable ts full potential + The consortium should )e equipped with the iecessary IT equipment ind software to enable :fficient and timely :ommunication among ietwork members. + Little, if any, serious effort was expended to establish training and infrastructure essential to any sustained regional university advocacy capability. Svnlrrotion of the University 28 July 2002 Collabororion and Parliamentary Assistance tcfivity: IRIS in Indonesia Development Arsocinres, Inc. AEP-I-00-00-00023-00. Tmk Or&r # 81.1 I Other Observations on Overall IRIS Program Deficiency Cifed in Mid￾Term F;vatuaiion Management IRLS Aciion to Respond te Deficiency I. Participants stated that IRIS lack, focus 2. IRIS activities (e.g., workshops) are not followed-up to assess impacts 3. IRIS did not routinely use needs assessments (e.g., workshops) to plan program activities 4. IRIS program activities were mainly opponunistic rather that strategic 5. IRIS financial management system was not performance based IRIS stated in interviews with the evaluation team that it was too early in the evolution of the program to measure intermediate and final results. This may be true. But this point does not negate the necessity or possibility of having in place a strategic program management structure and processes that permit the measurement of program results, when the time is appropriate. Unfortunately, IRIS has not set in place a management system that will enable this performance assessment to occur. + The pro-mm was not designed. implemented or managed in a strategic, results-based manner so that the evaluation team could readily examine the linkages between program objectives -program expenditures - program activities - intermediate results - and fmal results. Source: USND Rogram D. IRIS PROGRAM EXPENDITURE ANALYSIS 1) INTRODUCTION This section examines IRIS program expenditure data, under the cooperative agreement, to provide an overview of how the progam resources were used. The financial data used in this analysis were provided by the IRISlJakarta ofice and are presented in the Appendix. This analysis should be read in close concert with the earlier discussion. for the findings bear importantly on many of the perspectives that inform this entire evaluation. 2) FINDINGS a. Program Adminisiration versus Program Activity Evaluation of the Universiry 29 July 2002 Colluborotion ondPorliomentary Assistmce Activity: IRIS in Indonesia Development Assoeiotes, Inc. AEP-I-00-00-00023-00, Tark Order iY 8I4 The first analysis reviews the relative cost of actual promam activities versus the cost of 1= all staff in Jakxta and College Park, Maryland that support the program (e.g. staff who cannot be allocated specifically to one of the activity categories) and office expenses (telephone, internet, photocopying, supplies etc). 2= Ten IRIS program activities (e.g., Regional Universities, Special Autonomy, Parliament) Table 6.1 shows that approximately 18% of the program expenditures go for administration. The financial data provided by IRIS does not specify exactly what the administration support staff do, except to note that they are not attached to specific program activities. b. Program Technical Assistance Expenditures IRIS spent approximately 55% of its total resources on long-term technical assistance. Table 2 shows how this assistance was provided. The data pertain to the period from January 2001 to May 2002 (data for the January 2000 -December 2000 period were not provided by IRIS to the evaluation team). Table 6.2 indicates that approximately 84% of the long-term technical assistance was provided by expatriates in Jakarta, or from the University of Maryland, College Park. Short-term technical assistance represented 25% of the IRIS resources expended; the IRIS financial data provided to the team did not distinguish between local versus expatriate sources of the short term technical assistance. c. Other Program Activity Costs The remaining (approximately) 20% of the program expenditures are summarized in Table 3 Evaluation of the Universitj 30 July 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia rrl r3 Y @d Y hil ell Development Associates, In". AEP-I-OO-OO-00023-00 TmL. Order t 811 1 evaluation team by the 1~1~1~akarta program office.' The main program (non-technical assistance) expenditures went for "Socializations," followed in amount by workshops and conferences. Training courses represented only 7% of the expenditures. The 12% for equipment is not broken down by type of equipment in the expenditure-data provided to the team. d Expenditure Evolution IRIS expenditures have varied over time. They peaked during the January - June 2001 period. and have been declining since - but still remain at about 200% above the program's starting point. The average six-month expenditure has been approximately S868.006. Evaluation ofthe Chivemi?, 51 July 2002 Collaboration andParliamentary AssQtance Aen'viry: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00, Ta~k Order # 814 The pattern shows a very rapid build-up to the peak IRIS expenditure level in the January - June 2001 oeriod. In the first six months. . vromam - exoenditures increased aovroximatelv 226%. In . . the first twelve months, they increased by approximately 389%. e. Regional Expenditure Pattern Expenditures Expenditures Aceh I Special; 499,751 26 Autonomy North Sumatra Regional 80,484 4 University West Sumatra Regional 64,887 3 ~niversity Riau Regional 51,578 3 University South Sumatra Regional 58,499 3 Evaluation of the Universiry 32 July 2002 Collaboration and Parliamentary Assistance Activity: IRlS in Indonesia 16L'SZ leuopaa V'leW Lqsia~!un 029'95 [euo@aa LNM Development Associales, Inc. AEP-I-00-00-00023-00, Task Order # 814 J: Program Emphasis Program Activity % of Program Expenditures Legional Universities 25 Jational Parliament 23 $ecial Autonomy 17 qational Economies 16 ?seal Decentralization 11 ;overnance 8 :onflict Resolution 1 1 reminology Regional Universities Strengthen local universities and research institutes to provide sound economic analysis. Special Autonomy Focus on special autonomy issues facing Papua and Aceh. National Parliament Provide technical assistance to the members and training for the staff of the DPR-RI. National Economics Provide technical assistance to the National Economic Council and other national agencies Fiscal Decentralization Provide economic overview of fiscal policies and options pertaining to fiscal decentralizations. Governance Increase governance capacity in East and West Java and in North Sulawesi. Conflict Resolution No definition provided in INS budget material provided evaluation team The regional university and parliament activities received 48% of the expenditures. 3) CONCLUSIONS: a. Almost 20% of program resources were expended on items nor specijkally allocated to program activities. 6. Over 80% of program resources went to support long-term expatriate technical assistance, suggesting a general management strategy of optimizing expatriate participation involvement in the program rather than using local (ie., Indonesian) capacity to achieve program objectives. c. The program relied on workshops, conferences and (undefined) 'Socialization" activities as its primary "methodology" for reaching program objectives. Training represented only 7% of the expenditures. Evuluation of the Univmsity 34 July 2002 Collaboration and Purliomentory Assistance Activity: IRIS in indonesiu A. PARTNERSHIP WITH US-INDONESW SOCIETY (USINDO) 1) FINDINGS USMDO, according to the chief-of-party and to another IRIS official, was critically important in "getting us into Parliament." The founder and evidently moving force of USMDO. a former US Ambassador to Indonesia, arranged for IRIS to meet key Indonesian officials with whom IRIS then worked to construct the initial agreement to perform certain services to Parliament. IRIS officials stated that, even if USINDO did nothing else for IRIS, the initial contact had proved the value of the partnership. Except for the two IRIS officials who mentioned USMDO, no one alluded to this partnership during the many interviews. Nor was any interview suggested or arranged with any personnel associated with this organization. 2) CONCLUSION The evaluation SOW asks whether this partnership has been a "fertile" one. No persuasive information has emerged relative to this question. However, we note that IRIS appears to have had very little activity relative to the private sector, which reportedly has provided much of the basis for USINDO's development. The IRISAJSINDO relationship perhaps could serve as a significant channel for alternative fundraising. Also, USINDO'S growing prominence in educational exchange would seem to offer a natural affinity to the central IRIS pro-mm objectives. IRIS credits USINDO with enabling the program to work in Parliament. B. "INCUBATOR" OF PROGRAMS, PEOPLE AND IDEAS 1) FINDINGS IRIS has conducted numerous conferences and seminars on a wide range of topics in Jakarta and around Indonesia. Conference participants came from DPR-RI, national and regional universities, local and national govemment minishies and organizations, including the National Economic Development Agency (BAPPENAS), the Ministry of Finance, the ministry of Home Affairs, the Ministry of Foreign Affairs, Bank of Indonesia, the National Economic Council. Competition Policy Commission, the Ministry of Manpower, and the Ministry of Justice. IRIS has also facilitated USAID scholarships for master degrees, short-term courses at Georgia State University in Atlanta, and study tours in the United States. Participants were selected from national and regional universities. local governments, DPR-RI staff and other govemment agencies. Evuluution of the Universiry 36 July 2002 Collaboration ondPorliamenfory Assisfonce Acn'vily: IRIS in Indoneria Development Associates, Ino AEP-I-00-00-00023-00, Trrrk Order # 814 Some informants from regional universities have indicated that the workshops that they attended had inspired them to be more actively involved in the provision of sound economic and policy analysis and advice to local government officials. The establishment of the Research Center for Economic Policy at Syiah Kula University, Aceh, the Public Policy Laboratory at Nomenssen University, North Sumatra, and the Decentralization and Autonomy Laboratory in the Faculty of Social and Political Sciences, University of Papua, Manokwari - have benefited from these conferences and from participation in the University Network program. IRIS' conflict management and related economic development activities in Aceh and Papua have been absorbed by USAID'S Office of Transition Initiative (OTI). IRIS' non-fiscal decentralization governance activities, which aimed to increase the capacity of regional universities and civil society actors in East and West Java and in North Sulawesi to carry out analysis, information dissemination, and advocacy in order to achieve greater local government accountability - have been delegated to the USAID/Decentralized Local Government (DLG) Office. USAID sources indicated that many of the IRIS activities were either redundant or overlapped with other USAID-supported programs. IRIS has also provided short-term technical assistance to the Ministry of Manpower in regards to the needed laws and regulations which would enable the Ministry to meet the International Labor Organization's (ILO) standards. IRIS claims that its workshops and short-term technical assistance on competition contributes to the establishment of an effective, fully functioning, self￾sustaining system of laws, institutions and advocacy groups - to ensure a steady improvement in competition and consumer protection in key markets. 2) CONCLUSIONS IRIS has incubated, or in one way or another contributed to numerous initiatives. These have included overseas educational and training activities, US study tours, forays into government ministries of finance, home affairs, industry and trade, and the like. The information disseminated through IRIS workshops and seminars has enhanced the quality of the national and regional dialogue on various issues, including fiscal decentralization. The master degree program, short-term training and study tours in the United States should be expected to enhance the quality of Indonesia's pool of human capital. Short term expatriate consultants, engaged by IRIS, have discussed their views and opinions with a wide range of policy makers. These intellectual exchanges have enriched policy makers' understanding of the relevant issues. The team found no evidence, other than IRIS self-report, that allow the team to attribute these developments directly to IRIS' program strategy. The evidence suggests that what occurred was a byproduct of IRIS' ad hoc and opportunistic program response rather than from an articulated program strategy. It appears that these opporhrnistic forays, however interesting, may have had the effect of siphoning resources away from IRIS' core activities. Evaluation of the University 37 Juh 2002 Collaboration and Purliamentary Assisfonce Activity: IRIS in Indonesia Development Asweiotes, Inc. AEP-I-00-00-00023-00, Tmk Order L! 8IJ C. 0THER CONTRIBUTIONS IN RESPONSE TO DEVELOPMENTAL NEEDS 1) FINDINGS As indicated elsewhere, IRIS engaged itself in various activities. It had hired a number of expatriate experts to come to Indonesia and provide short-term technical assistance to selected Indonesian government organizations and universities, as well as to speak at selected conferences and workshops. Document reviews and informant interviews suggest that IRIS had worked in various areas that had the potential to influence Indonesia's economic development. For example, IRIS documents indicate that IRIS had: Developed guidelines for the enforcement of the substantive provisions of Indonesia's Competition Law (Law NOS) on the Ban on Monopolistic Practices and Unfair Business Competition; Developed a plan to build EKUIN's macroeconomic analysis capacity to provide high quality independent economic advice to the Government; Conducted high-level discussions with the Attorney General. members of the Anti￾Conuption Commission, and lawyers - on issues related to corruption and anti-corruption measures; Developed and implemented local revenue sharing formulas; Provided short-term technical assistance to the Ministry of Manpower, Facilitated closed hearings and opened discussions on current legislative issues, including a Zday seminar on money laundering; Set up a 10-day workshop in Jakarta for participants from 22 universities, PPPI. the Ministry of Finance, BAPPENAS and the Secretariat for Regional Development - to discuss fiscal decentralization; and Set up a seminar on domestic trade and decentralization in Jakarta. where standard international practices for local government taxation, and how they might be adapted to the Indonesia context, were discussed. In addition, the IRIS staff produced more than 20 white papers and discussion notes on a range of issues, including the need for radical reform of the banking sector and an assessment of proposed adjustments to the 2001 national budget. A series of economic education and national dialogue workshops were also conducted in different cities across Indonesia. Many informants have indicated that many of these seminars and workshops were useful and good. Some informants have also indicated that the information disseminated through these workshops and seminars have enhanced the quality of the dialogue on national and regional economic development issues, and on decentralized governance in particular. However, many of our informants emphasized that the absence of follow-up by IRIS may have dampened the use of the information and insights gained from these workshops and seminars. 2) CONCLUSION There is some evidence that the IRIS supported short-term technical assistance and information seminars and workshops have shaped a number of people's views on fiscal decentralization. The local revenue sharing formula has also been adapted. However, many of the suggestions Evaluation of the Universiry 38 July 2002 Collaboration mdPnrliamenmry Asn'stanc~ Ach'vity: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00, Task Order # 814 made by the IRIS expatriate experts appear to have fallen off the wayside, because of lack of adequate follow-up. Lack of follow-up and reinforcement is endemic to all IRIS activities. In some cases, the subject matter covered by the workshops does not cover all the relevant dimensions of an issue, thereby leaving a knowledge and skill gap. For example, a fiscal decentralization workshop should also provide a discussion on local government fiscal accounting methods. The IRIS program operated initially under a grant mechanism, switched to a cooperative agreement in 2000, and was operating under this procurement mechanism at the time of this evaluation. The SOW raised several questions regarding the efficiency and effectiveness of the different procurement mechanisms. In the following table, we present our findings and conclusions for each question. SOW issue Conclusion . Should IRIS continue to be implemented as a cooperative agreement, or is it in the best interest of the US government to transform all or part of its current activities into one or more contracts? 2. What would be the main differences between the two approaches? Evaiuatbn Finding Evaluation of the University 39 July ZOO; Collaboration and Parliamentnry Assistance Activity: IRIS in Indonesia * was implemented under both a grant and coo~erarive agreement. J IRIS was cited in the mid￾term evaluation for its management deficiencies under a grant. * IRIS was cited for essentially the same management deficiencies in this evaluation, when it was under a cooperative agreement. * USAID staff stated that grants and cooperative agreements can be, and often are, managed the same way (i.e., as strategic, performance￾based) as contracts. * IRIS claims that grants and cooperative agreements permit the necessary 'flexibility' to respond to crises and/or opportunities. * USAID staff state that contracts can be managed -+ It would be in the best interest of the government to transfer all of the current IRIS activities to one or more contracts, j ~h~ main difference between the two approaches appears to be more in the performance of the individuals managing the program than in the procurement mechanisms used. Development Associaes, Inc. AEP-I-OO-OO-00023-00. Task Order + 81d SOW Issue I. What would be the main advantagesldisadvantages to a contract approach versus a cooperative agreement? to provide the necessary flexibility to respond to crises andlor opportunities. * IRIS claims that cooperative agreements are less costly, due to cost-sharing (e.g.. in-kind contributions) requirements. IRIS did not present data to support this claim. * The team did not discover any notable advantages/disadvantages based on the IRIS experience. Both approaches offer the same degree of flexibility, and it is unclear from the IRIS txperience that one approach is more cost￾effective. -+ Both approaches can be effective. if properly managed. Evnluarion of the University 40 Jury 2002 Collo6orotion ondPorliomenrary Assistance Activity: IRIS in Indonesia A. REGIONAL UNIVERSITIES The regional university consortium concept has unquestioned merit and should be sustained. This will require a needs assessment, careful construction of a strategy of programmatic components and implementation procedures, clear confinement to modest objectives. and do￾able tasks, projection of realistic time dimensions, sufficient funding to assure the accomplishment of program objectives, the establishment of process mechanisms to encourage local buy-in, cross-disciplinary ownership and institutional investment, and, finally. program management consistent with Indonesian cultures. as well as with program details and with USAID practices and expectations. Needs assessment Key features of precepts set forth in this program's 1999 SOW", and quoted on the first page of this evaluation, ought to be reflected in all future programs. Because most Indonesian regions, and many sub-regions, already have strong identities socially, culturally and economically, a first-principle should be to solicit solid information on what a target area (and, in this case. the target universities) says it needs. Programmatic components and implementation procedures. In view of the undeveloped and vulnerable state of present economic conditions and most other governance in the regions - cooperation among outside aid givers seems essential. One way to support this principle might be to make complimentarity or synergy among program activities a central feature of programmatic planning. Time, money and buy-in. IRIS - or any other agency - should carefully consider the dangers of over-extension and of unreasonable promises in a program that seeks to impact thirty￾plus institutions and as many regions. An Indonesian regional university consortiuminetwork surely must quickly rely on itself, given the limited resources of outside funds. Yet, to achieve this end, IRIS needs to use a far higher proportion of program funds on Indonesians and Indonesian institutions, rather than on expatriate experts and advisors. Focus. A few well-chosen provincial and kabupaten (district) locations, geographically representative, and--even more important-suitable to cultivate as compelling future models for other parts of the country - are likely to have maximum impact. Part of any new pro,gam mandate should be to "market" the program and its successes. as an example of an IRIS program to be clearly identified in kind and activity. Follow-up. Whatever IRIS decides to do in regional capacity building, an integral dimension should include follow-up and responsiveness. Focus groups and individual interviews have made the following suggestions: Maintain an up-to-date calendar of events on the IRIS web site; include reports and evaluations on the web site, with facility to provide feedback from participantslmembers of the network; encourage and facilitate networking among members to " IRlSLPEM Decentralization Grant: Life of Grant and Year One Work Plan, Xovember 29, 1999 Evaluation of the Universiry 41 July 2002 Collo6orntion and P~lrlinmentnryA~sist(~nee Aen'viry.' IRIS m Indonesia Develnpment Associates, Inc. AEP-I-OO-OO-00023-00, Task Order # 814 improve tools and methodology; channel technical assistance directly to regionaVlocal universities, instead of through LPEM. Currently, IRIS identifies itself, and is identified by others, as being involved with one, or perhaps two parliamentary committees. Moreover, the team's interviews brought out repeated references to only one or two, or perhaps three members of parliament. Thus, a change in committee leadership, illness, or failure to hold an elective seat can instantly adversely affect IRIS' essential working relationship. In addition, IRIS' success in parliament rests on the continued employment of the resident Indonesian economic advisor. To have a wider institutional impact in Parliament, IRIS should broaden its parliamentary working relationships to enhance potential for sustainable capacity building. And it should target the enhancement of those kinds of professional capacities that are easily transferable across different committees, assignments and staff needs. The enhancement of professionalism among existing parliamentary staff should be a primary focus, and not a secondary consideration, as at present -- while direct involvement in policy analysis should be secondary. To this end, PPPI and/or the committee staff should be encouraged to provide direct information and technical assistance to committee leaders. Instead of doing the work themselves directly, the IRIS resident advisors should assist, encourage and advance the work of selected PPPI and Budget committee staff. In addition, local NGOs and universities should be encouraged to provide information and technical assistance to the relevant DPR committee. In addition, to enhance the professionalism among existing parliamentary staff, a thorough inventory of skills and needs assessment for developing quality research studies and sound policy analysis should be conducted. Then, a series of training seminars and hands-on exercises designed to impart the most essential skills to the staff of the Budget Commission (Panja - Panitia Keja) and their researchers (PPPI) should be developed and implemented, in collaboration with other donors. On-site training could be carried out within the parliament complex to optimize attendance. The number of participants should be kept to a manageable size. The courses should be designed as a series of two-day seminars (over a period of 6-12 months, scheduled on a monthly basis). The workshop contents should be directly linked to the relevant ongoing and current policy dialogue, in order to ensure the maximum usability of the training (i.e., leaming-by-doing). In addition, the training should he closely monitored and evaluated to ascertain grasp of the subject matter and use of the knowledge and skills learnt by course participants in their daily work. Results of these evaluations should then be used to remedy shortcomings in the training program. The training design could be developed, in collaboration with other regional organizations, to provide trainerslfacilitators from neighboring countries, such as the Philippines, Malaysia, Thailand, so that the knowledge and skill are broadened to include regional experience. At the same time, a series of short in-country information seminars directed at enhancing the intellectual capacity of parliamentarians and their staff should be designed and offered in collaboration with other donors. These information seminars should address the most current Evaluation of the University 42 July 2002 Collaboralion and Parliamentary Assistance Activity: IRIS in Indonesia w Development Asroeiotes, Ine. AEP-I-00-00-00023-00, Tacli Order t 811 and critical issues faced by Indonesia. The current "IRIS agenda" should be reinforced through coordination with other donor agencies, so that their experts can be included as speakers in the Commission technical talkslmeetings. Within USAID, there should be better collaboration andlor coordination among the relevant technical offices, i.e., Democracy and Governance @G). Decentralized Local Government (DLG), ECG, to ensure a more comprehensive (mainstreaming) approach of the issues, to effectively assist the parliament, and to optimize the tapping of expert-resources from USAID partners andlor contractors. The evaluation identified several recommendations that could bolster the impact and sustainability of the LPEM-IRIS grant activities: 1. Establish an institutional base for the regional university network. Participants agreed that the network is a good idea, but that it exists in name only. It should be set up as a legal entity (e.g., as an NGO), provided with seed funding to develop its own resource base, and be organized and staffed, so that it can continuously support regional university faculty capacity development. It should also develop the IT infrastructure to link faculty members within the network. This would be especially beneficial to the more isolated faculty working in poorly funded colleges and universities. 2. Workshops and training should be explicitly tailored to the needs of regions and their university faculties. A 'cookie cutter' training approach, such as that being used in the IRIS-LPEM program - which assumes that all participants need exactly the same training - too often fails to meet the needs of various sub-groups of participants (e.g., those needing local accounting training). This is especially likely to occur in a heterogeneous country such as Indonesia. 3. Workshops and trainings should be followed-up, to determine if they really expanded capacity and if participants were using the training in their work to produce better results. A short questionnaire filled out immediately after the training is an unsatisfactory substitute for an in-depth follow-up - 4-8 months after the training - with an assessment of the on-the-job use of the training and the impact on performance of a sample of participants. Reasons for non-use can also be identified and addressed in future mining planning. 4. The IRIS-LPEM grant work plan correctly noted that a variety of stakeholders (i.e., political parties, private sector, government, civil society) needed to be brought into the capacity-building process, if decentralization was going to work Future efforts, such as the IRIS-LPEM grant, should ensure that all key stakeholders are fully involved, or perhaps should re-think the advisability of doing a watered-down version of the program at all. Evaluation of the Univenig 43 Jv5 200.2 Collaboration and Parliomenfary Assistance Activity: IRIS in Indonesia Development Associotes,ocie, lnc. AEP-I-00-00-00023-00, Task Order # 814 Management practices must be substantially improved to justify continuation of the IRIS activities, and the team has serious question about the advisability of allowing the present activities to continue under IRIS management. We must add, however, that deficiencies in IRIS management clearly were exacerbated by USAID's failure appropriately to monitor the IRIS activities and to insist upon regular review, adjustment and revamping of practices. The team believes that programs lacking a strategic focus and implementation plan are drawn to ad hoe activities that may not contribute to strategic results. Needs assessment is essential if such projects are to have a base of support in the target beneficiary populations. A performance￾based budget with expenditure data that links resources expended and results achieved, enables the monitoring of progress to ensure continued congruity between assumptions and reality. These essential components of strategic, resubbased project management, which were missing in the IRIS project management practice, should be an indispensable requirement for any future funding. Any management program for continuation of the present IRIS activities should also have a public relations dimension. When neither fellow development agencies nor the recipients of IRIS assistance can generally identity what IRIS does or what its projects' objectives are, the need to communicate more effectively seems imperative. The capacity to project such identity will depend, in turn, on IRIS' clearer understanding of its own focus. Finally, management of assistance programs across multiple constituencies in a place like Indonesia demands acute cultural sensitivity. Moreover, efforts toward capacity building and sustainable institutional change require genuine reliance upon indigenous participation and ownership. Indonesians must be integral players, if the kinds of programs WS has undertaken are to succeed. Evaluation of the University 44 July 2002 Collaboration and Parliamentary Assistance Activiry: IRIS in Indonesia APPENDICES Original Evaluation SOW ................................................................................... A- 1 Evaluation Team Draft Proposed Evaluation Purpose Statement - Submitted to USAIDIJakarta on Tuesday, June 18, 2002 ............................................................ B-1 USAIDIJakarta Feedback on the Team's Proposed Strategy ................................. C-l USAIDlJakarta - Priorities and Weightings, June 26, 2002) ............................... D-1 Mission Response to Draft Executive Summary, IRIS Project Evaluation. and Evaluation Team Comments ............................................................................ E-1 Contacts & People Interviewed .............................................................................. F-i Field Data Collection Protocol ................................................................................ G-l IRIS Evaluation Focus Group Protocol .................................................................. H-1 References and Documents Reviewed .................................................................... 1-1 Evaluation of the Universiry July ZOO2 Collaborolion andPartiamenlary Assistace A&."@: IRIS in Indonesia ORIGINAL EVALUATION SOW EVALUATION OF IRIS COOPERATIVE AGREEIMENT REGARDING ECONOMIC ASSISTANCE TO IMPROVED UNIVERSITY COLLABORATION AND TO PARLIAMENTARY ASSISTANCE I. SUMMARY: An evaluation is required for the USAIDAndonesia Economic Growth Team economic activity of University Collaboration and Parliamentary Assistance being carried out under a cooperative agreement by the University of Maryland (IRIS). This activity consists of three components: b Technical Assistance b Short-term Training r Capacity Building Purpose: An evaluation of the IRIS activity is required. The evaluation must address the effectiveness, impact, sustainability, performance and financial monitoring of the activity, links to the intermediate results of the USAIDIIndonesia Economic Growth strategic objective, and the question of whether the activity should be continued at all, in its present form as a cooperative agreement, or transformed into a contract. The purpose of the evaluation also includes a determination of the extent to which the activity is achieving its purposes, to compile success stories and lessons learned and to make recommendations for improving the activity during any follow-on work that might be determined to be appropriate by the USAID mission in Jakarta. Contract Mechanism: The evaluation will be performed under the Development Information Evaluation Services IQC. The intention is that the chosen contractor will field a four-person team consisting of three expatriate experts supported by an Indonesian expert They will undertake the evaluation and submit their findings within 45 days of receipt of the task order. 11. ACTIVITIES AND ISSUES: ***** The activities under the IRIS cooperative agreement and issues that are to be considered by the evaluation team are set forth in this section. The activities or component areas that should be evaluated include parliament, the regional university consortium, work in the special autonomy regions, the partnership with LPEM-FEUI, facilitating the involvement of the US-Indonesia society in economic and institutional development, IRIS' role as an incubator of programs and individuals, other activities undertaken in response to perceived development needs in Indonesia, and IRIS management. Interviews conducted regarding the IRIS activities must be carried out with a sufficient number of knowledgeable individuals to assure the judgments made by Evaluation of the Universi(y A- l July 2002 Collabormion and Porliamentnry Assistonee Aeriviryt IRIS in Indonesia Develnpment Asrocioles, Inc. AEP-I-00-00-00023-00, Tmk Order # 814 evaluators are sufficiently well based. Specifically the evaluation team will want to look carefully at the following aspects of IRIS work under the cooperative agreement. 1. Parliament: IRIS has been providing support to the DPR-RI on finance, economic. budgetary policies and law development. This support has been principally directed to Commission IX (Economics and Finance), the Budget Committee, and the Parliamentary Secretariat which provides staff support to the entire parliament. The nature of this support has included technical assistance and policy advice, training and development of personnel, and the design and establishment of management procedures and institutional systems with the ultimate goal of enabling the DPR-IU to eventually become a fully functional contributing branch of government in a democratic Indonesia. Have these goals been met? What additional actions, if any, are appropriate to meeting these goals? How successfully has coordination with other donors been camed out? How successfully has coordination with other USAIDfindonesia off~ces providing technical assistance to Parliament been carried out, e.g., the democracy office? 2. Regional Universities: Another part of the program is the development of a university consortium to build the capacity of regional universities to provide economic policy making expertise and technical support to local governments, parliaments, civil society organizations, businesses, media outlets and other stakeholders in the regions. One critical question includes the current effectiveness of the university consortium and its members in providing leadership, counsel, and advice on economic issues to local governments, local parliaments, local NGOs and other stakeholders within the geographic areas in which the universities work. The relevant regions vary greatly, with the University of Indonesia working across many more geographic areas than local universities. In addition to effectiveness, the evaluation team should also consider the potential future relevance and sustainability of the evolving university consortium. Another critical question is the extent of appropriate additional strengthening, particularly given the augmentation of capacity expected of certain faculties of economics after the year-long training to a masters degree in economics is completed for many faculty members at Georgia State University. Is there a demand for the research and analysis that is and will be offered by members of the university consortium? How successfully has coordination with other USAID offices and other donors been carried out? 3. Special Autonomy Regions: Within the regional universities special focus has been given to those universities dealing with the enhanced challenges and responsibilities inherent in the special autonomy status being considered for and lately conferred on Aceh and Papua. Included in this work has been close coordination and cooperation with both the political and economic sections of the US embassy in these especially sensitive areas. Critical questions include the extent to which IRIS has been able to work effectively within special autonomy regions and the need for and/or relevance of additional such work, and willingness and success of IRIS coordination with other relevant USAID offices active in Aceh and Papua and with other donors. 4. Partnership with Institute of Economic and Social Studies (LPEM) of the Faculty of Economics of the University of Indonesia (FEUr): IRIS has worked directly with LPEM on all aspects of the program with the aim of making both institutions more Evaluation of the University A-2 July 2002 Collaboration and Parliamentary Assistance Activity: IRIS in Indonesia Development Associates. Inc. AEP-1-00-00-00023-00, Taclr Order f 8M effective. Under the program, the LPEM has had a central role to play in providing training, advice and experienced policy makers to guide the nation's economic policy and to serve as the leading institution within the university consortium. Within an environment of constrained public resources since the financial crisis. LPEM has been receiving broad support from IRIS in fulfilling its commitment to the Indonesian society. A critical question relates to an assessment of the success or failure of the LPEM in providing economic advice and counsel to the Government of Indonesia and the Parliament. Two other critical questions to be addressed include whether this LPEM leadership role to the university network has been effective, and how to make such a role sustainable in the longer term. 5. Parfnership with US-Indonesia Society: This relationship has also been a productive partnership, with USMDO providing assistance to the program in key areas where they have expertise and the program providing a means for USINDO to be more involved across a broad spectrum of economic and parliamentary activities in Indonesia. This IRIS-USINDO involvement is an interesting mix of two organizations. IRIS is one organization and USMDO is a non-profit organization principally designed to obtain financial support from US and other corporations working in Indonesia to explain Indonesia and the nature of the US continuing interest in Indonesia to Washington policymakers. Has this mix of organizations been a fertile one? Have any conflicts of interest or other problems been identified? Are there lessons to be drawn for development purposes elsewhere from the collaboration that has occurred? 6. Incubator of Programs, People and Ideas: The IRIS program has spun off a large number of activities and individuals to other programs managed by the Economic Growth office, elsewhere within USAID, to other donors, and to the Government of Indonesia. Although not part of its current terms of reference, these past activities have in IRIS'S view nonetheless been an important contribution of the program. Is this IRIS assessment correct? Identify and assess these spin-offs. To what extent might these "spin-offs" have inappropriately distracted IRIS or caused a redirection of resources away from activities that are in the terms of reference? Assess based on this experience whether university-based incubator programs of this sort are useful and whether they should be additionally encouraged. 7. Other Contributions in Response to Developmental Needs: During the course of its activities in Indonesia, IRIS has been willing and able to respond quickly to perceived technical assistance and development policy needs within the economic sphere. including those raised by USAID economic team staff. IRIS has reported to USAD that it believes its activities have resulted in several critical interventions in the policy making process in the areas of competition policy, specification of revenue formulae, and perhaps most importantly in the 1999 pre-election period when the IRISLPEM regional economic road-shows preempted consideration of several ill-advised economic agendas and set the tone for the predominantly market-oriented program which has prevailed since then. In each case IRIS believes it utilized the best possible people and was able to make a positive demonstrable impact on the policy-making process in a timely and efficient manner. Are these IRIS judgments correct? Where do they lead, if anywhere, Evaluation of the University A-3 July 2002 CoIlaborotion and Parliamentary Assistmee Activity: IRIS in Indonesia Development Associates, Inc. AEP-I-00-00-00023-00, Tosk Older # 814 in terms of additionaVrevised/reduced activities for the current IRIS endeavor in the future? 8. IRIS Managernent/Previous Evaluation. Certain potential and actual management deficiencies were noted in a previous evaluation of part of the IRIS activity that was completed in November 2000. Have these various deficiencies been resolved, or are they still of concern? Assess the current IRIS management capacity and performance. Specifically, the previous evaluation raised the following issues that must be addressed by the current evaluation. Performance Monitoring. The previous evaluation states that "IRIS-LPEM -has no performance monitoring system whatsoever .... ...[ Aldditional emphasis needs to be placed on tracking intermediate results and eventual impact, rather than just program outputs." The current evaluation team will assess and determine whether IRIS has been sufficiently responsive to this issue. Financial Monitoring: "IRIS ... could not provide any expenditure data in the field ... After more than a month from the initial request, IRIS' home office never responded to the Evaluation Team's information request . . . It was hard for the Evaluation Team to endorse this as an efficient financial management approach". The current evaluation team will assess and determine whether IRIS has been sufficiently responsive to this issue. Sustainabiity: Page iv of the previous evaluation summary states that "the University- U based advocacy efforts do not appear to be sustainable institutionally because they lack natural constituencies for reform" and also recommends that the exit strategy needs to be .. carefully spelled out. It is notable that a majority of the participants interviewed stated W they expected the institutional linkages created by the IRIS-LPEM activity to disappear without several more years of grant support. This outlook, combined with the perception of a lack of a natural constituency for advocacy, raises questions about putting extensive, funding into a university network under a new activity. The evaluation team W will consider and report upon these views together with the views of relevant Indonesian and international economistslexperts on the appropriate sources for ideas/characteristics for economic reform. M Impact: The original program description obviously changed in response to the economic and political crisis. The assessment notes IRIS-LPEM's accomplishments in &I changing the nature of the economic dialogue via the "Road Shows" and bringing a new generation of policy makers and advisors forward as leaders and media spokespersons (e.g., Dr. Sri Mulyani). Indonesia is now well into a post-crisis phase. Page 24 of the w Assessment questions whether the initial grant impact can be sustained through an attempt to utilize universities as a platform to effect reform of economic governance. "...[W]ithout continued external financing such advocacy efforts are unlikely to survive. k Universities, by their very nature, do not build constituencies for ref0 rm..." Is the university consortium designed to effecthfluenceihelp shape reform of economic governance? Is this strategy suitable? The evaluation team will consider and report w upon these views together with the views of relevant Indonesian and international 9 Evaluation of the University w Cnllaboratinn and Parliamentary Assistance Activity: IRIS in Indonesia Development Assoeinres, Inr. AEP-1-00-00-00023-00, Task Order $814 economists/experts on the appropriate sources for ideadcharacteristics for economic reform. Link to IR: (Page 1) The IRIS-LPEM target area of reform is stated as "Improving economic governance by strengthening private sector capacity for analysis. policy design. and promotion of sound economic policy". How does the university network and parliamentary strengthening fit into a cooperative agreement with this as the target? Are there other statements/rationales within the current set of ECG IRs under which the IRIS activities appropriately fit? Cooperative Agreement vs. Confract Assess whether this activity should continue to be camed out as a cooperative agreement or whether it is in the best interest of the US government to transform all or part of the current activities into one or more contracts. Training. In addition to the long-term advisory services mentioned above, the transfer of intellectual capital is key to successful policy development and reform. Short-term training and workshops, primarily in Indonesia and the United States, are encouraged in order to make the overall program a success. To what extent has the mixture of training and policy development and reform under the IRIS program been appropriate in the Parliament and in the university consortium. Have these activities contributed to long￾term sustainability of Indonesian policy development capacity? EVALUATION REQUIREMENTS GeneralRequirements: The contractor should undertake a broad-based assessment to determine to what extent the Parliamentary Assistance and University Capacity Building Assistance activity is achieving its purposes. This will include examining relevance, performance. effectiveness, efficiency, impact, sustainability, and financial monitoring. This will also include looking at the activity concept, implementation and management. Among other tasks, the contractor should compile success stories and lessons learned. In carrying out the evaluation. the contractor shall klly consider the critical issues and questions raised in section II of this evaluation SOW. Specific Task Elements. The following points should be addressed. although not necessarily in the order or structure below. Effiiency. Is the cooperative agreement proving to be a cost-effective means of addressing the desired objectives? In the opinion of the evaluation team, could the progress in evidence have been achieved more efficiently with a different mix of activities? Assess the quality of the administration of the activity. Effectiveness. Assess the extent to which expected output and results have been or are being produced and achieved by the activity. Are the objectives spelled out clearly enough to allow for an objective assessment? Has progress been made? What concrete accomplishments can be traced to the activity? Are the activities on track to achieve the design results? If not, what changes are required for that end? Ewluation of the Universiq A- 5 July 2002 Collaboration andParfiameniary Assistance Aetivirv: IRIS in Indonesia Development Associates, Ine. AEP-I-00-00-00023-00, Task Ordcr # 814 In addition to the above, the evaluation team is expected to look at the dissemination of analyses and findings from the IRIS activity to the Government of Indonesia, the university network, the Parliament, and the broader public. Have best practices and lessons been effectively spread outside the program participants? What techniques worked best and why? What is recommended for future programs? Relevance. Assess the concept and fit of the grant in meeting the USAIDECG strategic objective (Sol) and sub-objectives (IRs). Is there consistency with the underlying strategy and support to other activities within the overall ECG strategy? Are the activities that have been planned or undertaken relevant to the purposes of the contract? To what extent does the program address problems of high priority, as viewed by its stakeholders? Coordination. Has the cooperating agency pursued effective pro-active coordination with USAID-funded programs, both within and outside the ECG portfolio? With other donors? PracticalityAdequacy of Funding. Assess the attainability of goals and objectives under the cooperative agreement. Are the goals reasonable in terms of the level of resources and the activity design? Are there good prospects for success? Are the resources allocated to the cooperative agreement consistent with the resource requirements foreseen in the design process? Management of the Cooperative Agreement. Has the cooperating agency effectively structured and exercised management control over the activity and the commitment and disbursement of resources? Substantial involvement of USAID. Given the limitations of USAID involvement in "assistance" activities, has USAID provided appropriate and timely input to program management decisions? Personnel: Have the individuals funded under the cooperative agreement, both Indonesian and expatriate, been appropriate to the tasks at hand? Counterpart: Has the cooperating agency, in accordance with the cooperative agreement terms, made appropriate and meaningful counterpart funding available, including in-kind support? Sustainabilify: Are program activities likely to be sustained in the absence of further USAID funding? Have enduring institutional andlor personal bonds been formed between technical experts and US institutions with Indonesian counterparts, and are those bonds focused on the objectives of the cooperating agreement. Will the positive changes induced by this program be maintainedlsustained after termination of this program? Impact. Has the program had substantial beneficial impacts on the Indonesian policy decision￾making process? Success stories should be compiled and documented. Outcomes. The evaluation should also make note of lessons learned. The evaluation team should understand that USAID may determine to discontinue any of the IRIS activities or might redefine its needs and issue a solicitation rather than extend the IRIS cooperative agreement. Evaluation of the University A- 6 July 2002 Collaboration and Parliamentary Assistome Activity: IRIS in Indonesia Development Assoeiofer, Inc. AEP-I-00-00-00023-00. Tmk Order d 81.1 TV. TEAM COMPOSITION The evaluation will be performed under the Development Information Evaluation Services IQC. The intention is that the chosen contractor will field a four-person team that has a suggested mix of three expatriates and one Indonesian expert. The team should have substantial expertise in economics. relevant expertise in institutional capacity development. and substantial relevant field experience with economic issues in the context of parliamentary relations and university capacity building. Country andlor region-specific knowledge and experience with economic policy support is highly desirable. Previous experience with an international development agency is also highly desirable. The team leader should have thorough knowledge and understanding of current Indonesian political-economic situation and be fully conversant with the major policy issues and strategic choices confronting the nation. A second team member has the option of an economic or business professional background. Helshe should have also thorough knowledge and understanding of current Indonesian political-economic situation and be fully conversant with the major policy issues and strategic choices con6onting the nation. The economists should have advanced technical training (PhDiMBA) and practical experience in the relevant fields (economic, commercial, trade. agriculture, etc.). A third team member should have the right mix of technical skills together with results managementlprogram management skills as to be able to make sophisticated relevant judgments about the issues that arise because USAD plans to make a determination about whether to manage this activity as a contract or as a cooperative agreement. These matters should considered in the context of program effectiveness, impact, and sustainability. These matters will be relevant should USAID decide to continue this activity. A fourth team member should have very substantial local knowledge of Indonesia and excellent bahasa Indonesia. Technical skills should be complementary among the members of the team, so that the needed mix of skills is appropriately available. V. TECHNICAL APPROACH The team will carry out the evaluation through: k Review of program documentation (the cooperkve agreement, the previous evaluation, Annual Work Plans, Annual Reports, Weekly Progress Reports, the strategic objective and intermediate results of the USAID Economic Growth Office, etc.); b Interviews with relevant USAID staff and Indonesian counterpart personnel; with beneficiaries both direct and indirect) such as those 6om relevant committees and staff in the Parliament; the participating universities; other donor organizations such as the World Bank; relevant officials of the Indonesian government, such as those from Bappenas, the Ministry of Finance, the Ministry of Home Affairs; local or provincial governments, local or provincial parliaments. non-governmental organizations; and with knowledgeable individuals in academia. USAD staff or their cont~actorsl~ntees who might be interviewed as appropriate include those dealing with issues critical to this activity and Indonesian economic success might include relevant contractors of the Partnership for Economic Growth, the Office of Civic Participation and Transition or their contractors/grantees, the Office of Rural Environmental Management or their Evaluation of the Universi@ A-7 July ZOO2 Coll&orafion and Parliamentary Assistance Anivity: IRIS in Indonesia Development Associates, Inc. AEP-I-00-00-00023-00, Tark Order # 814 contractorslgrantees, and the Office of Decentralized Local Government or their contractors1grantees. Both the political and economic sections of the American embassy in Jakarta should also be consulted. b Review and assessment of a sampling of studies, reports, and analyses funded under the cooperative agreement. VI. DELIVERABLES The evaluation team will be expected to: 1. Present the initial evaluation plan to USAID. Upon arrival, the evaluation team will meet with the USAID CTO, the USAID Program Officer, and the USAID Economic Growth Team Leader or his designee to provide a proposed evaluation plan and methodology. 2. After this initial meeting, the evaluation team will meet with the USAID Deputy Mission Director andlor her designees. 3. Brief the USAID CTO, the USAID Program Officer, and the USAID Econom~c Growth Team Leader andlor his designees prior to departure, as appropriate, to present the team's major findings, conclusions and recommendations. The departure meeting or a separate meeting is to include a briefing for the Mission Director and the Mission Deputy Director or their designees and may include the Contract Officer or his designee. 4. Provide a draft Executive Summary, including key findings and recommendations, and an outline of the report at the pre-departure briefing. 5. Provide a final written report, which details the team's findings, to USAIDIJakarta within 10 working days following completion of the in-country work. Ten hard copies should be sent to the USAID CTO, James M. Hope, USAIDAndonesia Program Office. At the same time an electronic copy in Microsoft Word format shall be emailed to jahope@usaid.gov. The report will include all specified requirements of the SOW, including success stories, lessons learned and recommendations for present and future activities. A copy of the final written report should also be sent to PPCICDIEIDI, USAIDIWashington to be put in the USAID library and database. Report studieslproceedings of this evaluation should be properly marked in accordance with AIDAR, Contract Clause 752.7034, Acknowledgement and Disclaimer. VII. PERIOD OF PERFORMANCE d Four weeks, to start as soon as possible. USAIDIJakarta expects that the team will arrive in Jakarta by mid-May 2002. Evaluation of the University A- 8 July 2002 Collaboration and Parliamentarv Assistants Activity: IRIS in Indonesia Development Associafes, Inc. AEP-I-00-00-00023-00. Tnsk Order '! 8N VIII. REPORTING AND SUPERVISION The team will meet with the USAID CTO and the Program Office designee once per week (or as otherwise agreed to by those officers) and provide a briefing on activities and progress. In-country travel plans must be approved by USAID, in advance. Travel to insecure areas is discouraged and unlikely to be approved. The team will be responsive to USAID suggestions, and will observe any guidance given as to political sensitivities. progress reporting, and in-country travel restrictions. ORGANIZATIONAL CONFLICTS OF INTEREST: PRECLUSION FROM FURNISHING CERTAIN SERVICES AND RESTRICTION ON USE OF INFORMATION This task order calls for the Contractor to furnish important services in support of evaluation of IRIS. In accordance with the principles of FAR Subpart 9.5 and USAID policy, THE CONTRACTOR SHALL BE INELIGIBLE TO FURNISH, AS A PM OR SUBCONTRACTOR OR OTHERWISE, IMPLEMENTATION SERVICES UNDER ANY CONTRACT OR TASK ORDER THAT RESULTS IN RESPONSE TO FINDINGS, PROPOSALS, OR RECOMMENDATIONS IN AN EVALUATION REPORT WRITTEN BY THE CONTRACTOR. THIS PRECLUSION WILL APPLY TO ANY SUCH AWARDS MADE WITHIN 18 MONTHS OF USAID ACCEPTING THE REPORT, unless the Head of the Contracting Activity, in consultation with USAID's Competition Advocate, authorizes a waiver (in accordance with FAR 9.503) determining that preclusion of the Contractor from the implementation work would not be in the Govenunent's interest." X. AUTHORIZED WORK WEEK Contractor is authorized a six-day work-week without premium pay. Evaluation of the Universiry Collabornrion and Parliamentary Assistance Adivih.: IRIS in Indonesia ON TUESDAY, JUNE 18,2002 USAIDAndonesia is entering a phase of assessing programming options for advancing its strategic goal of furthering the "transition to a prospering and democratic Indonesia." The present evaluation seeks credible information on the current relevance, effectiveness, impact, sustainability, performance measurement and financial monitoring of IRIS and recommendations for improving the program. Based on a reading of the SOW and conversations with USAID, the team has determined that the purpose for this evaluation should be to assist this process by providing data and information that will help the Mission in its decision making about future IRIS support. A mid-term evaluation of IRIS was completed in October 2000. Monday's discussions with Mission staff have led to the team's concurrence to review the program's evolution from its beginning in Indonesia, but to emphasize the period from October 2000 to the present for the more intensive analysis and reporting purposes. In addition to assessing IRIS in the terms noted above, the Mission is seeking information on procurement options. IRIS has operated under both a grant mechanism and a cooperative agreement mechanism. The evaluation will assess how well these procedures have worked and explore the potential value of a contract approach for continued IRIS support. Finally, while the evaluation will center on the IRIS program, it is important to understand how the program operates within the overall Mission Country Strategy. The team will therefore assess the synergy between IRIS and other Mission program activities and weigh its contribution. We have been invited also, both in Monday's meeting and in the SOW. to address larger issues of prioritization and planning relative to the multiple kinds of assistance that have been associated with the several years' legacy of IRIS activity. What are some of the basic alternative strategies that might be considered based on the findings of the present evaluation? Are there substantially different approaches that may be worth considering, for IRIS-type programs, within the USAID strategic planning process currently underway? This type of information clearly must be secondary to the primary purpose of the evaluation and will be based on insights gained in the course of conducting the evaluation. Evaluation of fhe Universiry B- I July 2002 Collaborutian mdParli4menfa~ Asdstmce Activity: IRIS in Indonecia USAID FEEDBACK ON THE TEAM'S PROPOSED STRATEGY Timing and Purpose. Agree that the evaluation should focus on the post-October 2000 timeframe. Purpose captures discussions from Monday. Procurement Mechanisms. The key question to be answered here is whether a grant/cooperative agreement is the best way for USAID to provide the kind of economic technical assistance offered through IRIS. Or, should this kind of assistance be provided via a contract. This issue was also discussed, I believe, in the mid-tern evaluation. Either way, discussion and analysis on this topic should not be done at the expense of the IRIS components. General Themes. There are a number of general themes to be evaluated that cut across the specific components of the IRIS program-addressing issues such as coordination, and as you note in the proposal-the synergy/impact/role of IRIS within the ECG office strategy and with the overall Mission strategy. For each of the components and for the program as a whole, how successful was coordination with other donors, with other SO teams in the mission? Another general theme relates to lessons learned or success stories from the IRIS program. IRIS Components. The SOW notes 7 specific components in the RIS program￾Parliament, Regional Universities, Special Autonomy Regions, LPEM Partnership, US￾MDO Partnership, IRIS as Incubator, and Misc. Development Responses. Of these. priority should be placed on the Regional University - LPEM Partnership components and Parliament. Within each of these components, the SOW lists a number of questions that get at issues related to the effectiveness. impact and sustainability of the IRIS components. Specifically, the questions for each, in my mind, are: Have the goals been met? Has the economic researchlanalysis generated through the university consortium been used (was there a demand), was it effective (in the opinion of those who used it￾parliamentary commission, local governments, local parliaments)? Is the university consortium effective-does it play a role in local governance, is it seen as a resource by decision-makers? Is the consortium sustainable? In Aceh and Papua, have the IRIS programs had an impact? What impact has IRIS training had on economic decision making (in the opinion for example of participants)? Has IRIS served as an incubator to grow and then hive off programs implemented by ECG or other ofices in USAID, especially in the post Oct 2000 timeframe-how have these activities contributed to IRIS success or failure? Similar questions with the ad hoc contributions IRIS has made as outlined in point 7, page 3 of the SOW. Previous Evaluation. As presented in the SOW, the mission would like to basically close the loop on the previous evaluation. Have the issues from Oct 2000 been addressed, resolved, or are still outstanding [sic]-financial monitoring, performance monitoring, sustainability, impact/effectiveness, USAlD and IRIS management. Evaluation of the Univemiry C-I July 2002 Collaboration andParliamenmy Assistance Adviry: IRLS Indonesia APPENDIX D USAID JAKARTA - PRIOIUTIES AND WEIGHTINGS Our suggested overall weighting (using 100 point scale) for each of the evaluation components is below. Regional Universities and Special Autonomy Regions. (20 points). This is an area that has high relevance for future mission programming. The evaluation team should consider this component in conjunction with item 2 below, the LPEM partnership - i.e., the changing interaction between the early strong role of LPEM in dealing with regional universities and LPEM's later policy roles with the GOI. There has been some legitimate controversy as some regional universities have expressed concern about LPEM dominating the use of expatriate resources - which is part of the reason the mission moved our focus partially outside of LPEM. In addition to the focus on universities in Aceh, West Papua, and to a lesser extent North Sulawesi, there are other universities (West Java and East Java and South Sulawesi (particularly younger faculty members in SS) where there have also been real benefits from the IRIS activity, particularly the decentralization trainings, the regional economic seminars, and the interuniversity competition for limited IRIS funds to support good technical economic work at regional universities. Partnership with LPEM of the Faculty of Economies of the University of Indonesia (15 points). As noted above. IRIS management Previous Evaluation (15 points) The evaluation should assess how the issues raised in the mid-term evaluation have been addressed. Parliament: (15points). This is an area where the Mission is continuing activities. As we go fonvard with other parliamentary activities we want to squeeze all the juice we can out of what IRIS has learned to help our other programs be successful. Training (I5 points). The evaluation should address the training component question posed in the SOW. Parfnership with USINDO. (5 points). Should be considered in conjunction with Parliament component and in the regional university activity. Incubator of Programs. (5 points). Has IRIS been able to move economic policy outside the government and into civil society through the University system. Should be considered in conjunction with item 8 below. Development policy needs. (Spoints). Questions as posed in SOW Cooperative Agreement vs. Contract. (5 points). As discussed last Friday. Eualrtation of the Universiy Dl July002 Collabornrion andParliomentory Assirronce AehehviCy: IRIS indonesin APPENDIX E USAID/JAKARTA RESPONSE TO DRAFT EXECUTIVE SUMMARY, IRIS PROJECT EVALUATION --AND GENERAL COMNENTS It is extremely essential that the findings and conclusions are strongly supported by the evidence - upon what and in consideration of what was a certain finding or conclusion reached. Impressions or general feelings are not sufficient. The final evaluation report should clearly differentiate between how well IRIS performed against the parameters and requirements of the cooperative agreement versus the evaluation team's recommendations to the Mission for future pro,gam development Comments on alternatives or better ways that the program could or should have been implemented should be presented in a way that makes clear what are evaluation "findings" regarding IRIS performance and what are "no fault" recommendations for the future to the Mission. In a similar vein, IRIS needs to be judged as against its SOW, action plans, budgets and length of time of operation for the various activities- what they were approved to do or try to do in the specific, not what would have been good or better nor a generalized goal or objective that serves as an introduction; what they had resources to do, not what might have been done with more funds; what was reasonable at the moment, not what would have been good with hindsight; and. what could be reasonably expected under the circumstances as an activity in process, especially for new initiatives or foci. not the desired finished products. Reasonable time must be allowed for activities to bear hit. Substantial consideration needs to be given to the circumstances: the times and the need for urgent responses - economic crisis, historical political transition. widespread violence and social unrest, and extremely rapid decentralization; major changes in USAID strategy (a new strategy was adopted in 2000 with its geographical emphasis areas); superceding [sic] US foreign policy priorities, such as Aceh and Papua or anti-terrorism actions; unexpected considerable shortages in funding availability; the experimental, evolving or pilot nature of many of the activities; the lack of expectation or requirement for sustainability in many of activities aimed at dealing with immediate problems rather than building for the future; and the realities under which the activities were implemented, such as the security concerns, time required to build trusting relationships and the sheer physical distances involved. Generally it is expected that the "glass is half full as well as half empty." Team Response: In general a program/activity is evaluated against its stated objectives and scope of work. IRIS' program objectives and SOW were provided to the team by the IRIS Indonesia Team Leader via numerous reports. The evaluation team's data collecrion lcas Evaluation of the Universily E-1 JuIy 2002 Cotlaboralion and Parliamentnry Assistance Adivity: IRIS in Indoneria Development Associates, Inc. AEP-I-00-ODOOODZ-00, Task Order # 814 geared to both the evaluation scope of work and IRIS' stated objectives and claims. The team's conclusions were based on findings sijied from document reviews, selected informant interviews and focus groups. In some instances the findings were clear cut, such as the failure of IRIS to respond to the deJkiencies cited in the mid-term evaluation. In other situations, such as IRIS's failure to implement a systematic training/workshop follow-up process, which was promised in the IRIS￾LPEM grant, the evaluation findings were based on interviews with training/workshop participants (that reported no follow-up) as well as the absence of any systematic follow-up data. The one 'jollow-up survey' uncovered by the team was methodologically flawed and did not examine the actual use of training/workshop materials in job settings. The evaluation team is fully aware of the somewhat trying circumstances under which IRIS operated and the relatively short period of time under which IRIS was funded leading up to the time of this evaluation. Thus, the team did not hold IRIS to unrealistic expectations. Rather, the team took IRIS at its word and looked for evidence that IRIS had achieved the results it claimed it had actually achieved Where we found these results, we reported them. Where these results were not achieved, we reported that as well. Equally important, where we discovered the absence of any results data, we reported that too. In general, the team's inability to verz3 IRIS results was due to the fact that IRIS failed to provide credible performance data beyond subjective IRISstafftestimonials and 'success stories. ' Finally, the team found IRIS'S attitude towards the issue of sustainability puzzling. IRIS seemed to be saying that sustainability was either irrelevant to certain issues (e.g., dealing with immediate problems) or too early in the IRIS program implementation process to be a concern. IRIS never explained what was an 'immediate problem' that obviated the need for sustainability. For example, was there no interest in the sustainability of the process applied to deal with the problem? On the question of timing, when is the appropriate time to think about the sustainability of an IRIS activity? IRlS did not address this question. SPECIFIC COMMENTS I. Conclusions and Recommendations Regarding the Regional University Consortium: 1. The results must be judged against the time and resources available. The regional university consortium is basically the focus for the future, rather than the expected outcome of the past. It is our understanding that there was only a full time IRIS expatriate staffer addressing the university consortium issues starting roughly in January 2001. Previously the university consortium focus had been only at the University of Indonesia and in Aceh. The only very modest resources available were designed to encourage understanding and research on fiscal decentralization and this was done. 2. Since Aceh and Papua were the areas of the greatest attention, their accomplishments offer examples of what might be done under an extended and expanded regional university emphasis. In addition, the IRIS university conference showcased more than a half a dozen sustainable individual university programs undenvay supporting fiscal decentralization and regional economic development. Evaluation of the University E-2 July 2002 Collaboration and Parliamentarv Asslstanee Activity: IRIS in Indonesia Development Amcimes, Inc. AEP-I-OD-OD-00023-00, Tmk Order 81d Team Response: In actuality, the development of a regional universiry consortium wns a central task of the IRIS project under review. "One critical question," !he evaluation SOW reads, "includes the current effectiveness of the university consortium and its members in providing leadership ..." To hold that the consortium was essentially a matter for firture development and not an expected outcome of the present IRIS project is to contradict the mandate of the evaluation SOW (see Appendix A), the first-year SOW for the project, and the claims ofproject success by those in charge of it. Findings and conclzrsions in the chapter on the regional university consortium address this and related issues in detail. Similarly, the dceh and Papua activities, and their utility as models elsewhere throughout rhe consortium, are addressed in the same chapter. 11. Conclusions and Recommendations Regarding Parliament 1. The circumstances and length of time for this activity need to be considered. Circumstances were highly complex during the initial phases of the IRIS parliamentary activities. Initial judgments and approaches had been made by ECG and IRIS staffto the Parliament. There were responsive chords from several members of Commission IX and the Budget Committee. However, just as Mr. Gajewski was hired to follow up on these chords, a transfer in the Chairmanship of the Commission and the Committee was made. The new chairperson took his time in taking up the USAID offer of technical assistance, but eventually did so enthusiastically. The funds involved were entirely from the ECG team, and intended therefore for economic and legal matters. The approach was deemed exploratory, and ECG determined that it wanted to start small, and not tackle too much initially. ECG certainly would like to be more responsive to the ferment of parliamentary concern with economic and legal issues from various Komissi that are now coming to the fore. IRIS has in progress an effort to design and seek funding for such a broader approach. 2. The IRIS activity in the Parliament did not begin in earnest until the long-term advisor. Mr. Peter Gajewski, was hired roughly 18 months ago. Given the need to present an Indonesian face in the highly sensitive Parliamentary environment, it probably makes sense to begin to think about measuring performance from 15 months ago, when the first Indonesian staffer was hired. Training has been camed out for members of the multi￾commission Budget Commission, Commission lX on economics, and the Parliamentary secretariat. 3. It is not realistic to expect sustainability f?om work on one year of budgeting. 4. It also takes time to develop an institution such as the Secretariat General's Office of Research and Information Services. For 15 months in which relations and trust had to be built, a lot has been accomplished. 5. ECG and IRIS have engaged in substantial donor coordination with the World Bank. Ausaid, New Zealand AID, the UNDP Partnership for Governance Reform, as well as within the ECG team with such activities as the ELIPS law improvement pro,Qm. The Parliamentary secretariat sponsored formal meetings for coordination. Many informal Evaluation of the Univenig E-3 July 2002 Collabornrion ond Parliamentary Assistance Aetiviry: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00, Tmk Older #814 W interactions were also held. AUSaid, New Zealand Aid and the World Bank have all provided collaborative support for IRIS activities here. ri 6. To build the institutional capacity, for more than a year among other activities IRIS has provided weekly seminars for the professionals on the secretariat staff. The University of Maryland is providing certificates of completion for these courses, and three certificates U have already been awarded. 7. IRIS from the beginning of its activities in the parliament 18 months ago has concentrated on strengthening the Secretariat's ability to assist the members of Parliament. More than 25 reports have been improved and supported and delivered to the Members in Indonesian by the Secretariat with substantial IRIS help. 8. While transferable skills have been a focus of IRIS, ECG does not agree that there should be an exclusive focus on such transferable skills. Certain economic issues and techniques highly relevant to judgments about economic and social policy are simply not so easily transferable, but need to be done to deal with current critical issues of the highest priority. Team Response: As iterated in detail in our full report, the team's conclusions and recommendations are based on testimonies of IRISDPR personnel and the recipients of IRIS' assistance. It should be noted that the informants we interviewed from the DPR-RI Secretariat General's Center fo'r Research & Information Services (PPPI) and Commission IX-Budget Committee staff were hand picked by IRIS. According to these informants, IRIS' effort to enhance policy analysis skills of the staffwas minimal. IRIS had not conducted capacity building needs assessment with regard to PPPI and the Budget Committee stafi The training strategy was ad hoc with no follow-up. According to these informants, except for the occasional help and comments they received from the resident Indonesian economic advisor, "there was no formalized process for reviewing and enhancing their research reports." The IRIS resident Indonesian economic advisor also had indicated to the evaluation team that "he had no clue as to what IRIS' objectives and mandates were and he was not given a job description. He provided comments and assistance when approached by the PPPI and Budget Committee staff out of hisftee will. " Mr. Juwono, an ex-government official, who IRIS identiJied as its employee, charged with donor coordination, told the team that he had no employment contract with IRIS. However, he had inventoried the parliamentaiy support activities implemented by the three major donors - Partnership-UNDP, the Asia Foundation and KONRAD '>ro bono" with a promise of future employment. As many informants have attested, the long-term resident Indonesian economic advisor has had considerable input to the 2002 National Budget. However, he functions as one employee of Commission LX. At the end of the day, when he walks away ftom the job, there will be no capacity left behind to continue the work where he left ofi Evaluation of the University E-4 July 2002 w Collaboration and Parliamentary Assktanee Activity: IRIS in Indonesia Development Associofes, Inc. AEP-I-O(1-00-00023-00, Tmk Order d 81.1 Finally, there is the question of time. There is no question that things have happened, papers were produced, and workshops were conducted. However, there was no needs assessment, no systematic plan and no documented results beyondprocess. In the team 's view these largely "ad hoc" efforts have produced no measurable results which would lead to the achievement of the SOW objectives. III. Conclusions and Recommendations regarding LPEM 1. IRIS already does substantial reporting. Please explain what specific additional information would be required and or systems that need to be put into place beyond the detailed IRISECG project reporting system that is already in place for LPEM. Response: The results-reporting that IRIS promised in the original IRIS-LPEM Grant - but did not do - would have met this need. 2. In accord with the work plan, IRIS and LPEM activities were designed to build capacities jointly through regional workshops and policy research papers and to address macroeconomic and international policies such as their joint contribution to the recent conference sponsored by ECG on international terrorism. Capacity and training have been provided to LPEM to improve their technical computer capacity and macroeconomic modeling capacity. Impacts on the private sector, civil society or political parties were not the focus, although impacts were of course provided, for example, in the road show. With respect to role of LPEM with political parties, ECG judged that the best strategy for IRIS for working with the private sector. including the media, was through senior WS people, particularly in the DPR rather than through LPEM. Response: This is not a request for infonnation~ciarification, but rather an IRIS testimonial. 3. Please make specific suggestions as to what should be done to the current reporting systems with modest resources as present reporting is already tasking. Response: See answer to # I. 4. ECG agreed with WS early on that for dealing with the media. press, and private sector IRIS would concentrate their limited resources available to their DPR activities, rather than LPEM. Response: Acknowledged 5. ECG hopes that the regional university consortium will eventually be transformed into a network of universities, although the extent to which that may be able to be realized depends on the resource envelope available. The development of a nehvork was proposed for a three year IRIS extension. IRIS prepared a work plan for such a task. but it has not been accepted by the Mission. Evaluation of the Universiy E-5 Juty 2002 CoNoboration ondParliamentary AssiFfonce Acn'uiy: IRIS in Indonesia Development Associates, Inc. AEP-1-00-00-00023-00, Tavk Order # 814 Response: The words consortium and network (as explained in Chapter 2), are used interchangeably throughout the relevant documentation concerning this project. Neither the project SOW nor the Evaluation SOW suggests that this consortium was not expected to become an operative consortia1 network during the period under evaluation. IK Conclusions and Recommendations Regarding Management Substantial reporting already goes on. Please explain what specific additional information would be required and or systems that need to be put into place beyond the detailed IRISECG project reporting system that is already in place in order to meet the standard of implementing a monitoring system. Response: IRIS did not respond to the mid-term evaluation 2 cited deficiencies by putting in place a monitoring and evaluation system to generate the type ofpevfonnance data called for in that evaluation. The IRIS 'monitoring system' reported on 'transactions' (e.g., number of workshops and who attended) and did not report on either intermediate or ultimate results. The type of monitoring and evaluation system called for in the mid-term evaluation, and the type of system IRIS pledged in its IRIS-LPEM Grant, would have met the need identified in this evaluation. 1. IRIS reports to ECG that it provided budgetary information in all the categories that the evaluation team requested, including spatial location, program, time frame, and functional organization. IRIS says that the evaluation team did not ask for a breakdown by performance based budgeting that links project expenditures to intermediate results and ultimate objectives. IRIS claims its system is able to cany out such a request. ECG has not had in recent months any problems with the IRIS reporting system. Response: IRIS provided data on spatial location, etc., to the evaluation team as noted in the question. There is no evidence, however, that IRIS routinely used budget data with this type of break out - so as to track the use of resources by activity and geographic dispersion - in its project management. The team did not specifically ask for performance-based budget data, because IRISLndonesia staff made abundantly clear early in the evaluation period that they did not have performance-based budget data. So why ask for it? Moreover, the team saw no evidence that IRIS/ndonesia was able to provide such data. The fact that they do not produce such information and routinely use it, is a serious management defciency. 2. The work plans give specificity to purpose and strategy, including incubation and exploration of new program possibilities. Over time as activities were spun off and resources diminished, focus on parliament, special autonomy, and regional development emerged. Any criticism regarding focus should be directed to the Mission. Among spun off activities were labor, competition, conflict resolution, work with faculties, and fiscal decentralization. For example, USAIDIOTI took over two programs initially incubated by IRIS (conflict resolution activity and work with the economics faculty at Syiah Kuala University). Roy Bahl has been hired by the World Bank to continue essential work started under IRIS on fiscal decentralization. Evaluarion of the University E-6 July 2002 Colla6ororion and Parliamentary Assicfnnce Activity: IRIS in Indonesia Development Assoeiafes. Inc. AEP-I-00-00-00023-00, Tmk Order if 811 Response: The comment regarding IRIS'S lack of 'jocus' is directed to IRIS because many of the people interviewed for this evaluation specifically criticized IRIS for its lack of foc~lcr That criticism was bolstered by the comments of other informants who stated that, while they knew about IRIS, they did not know what it was supposed to do or was trying to do. 3. IRIS depends very heavily on Indonesian staff and faculty members for its program activities. For example, the Parliament work or the road shows. It does not optimize expatriate participation at the expense of using local capacity. For management, the hiring of expatriate staff for accounting and management systems was done up at ECG's demand in response to the recommendations of the previous evaluation. The three expatriates provide needed skill sets that are not otherwise available in Indonesia These highly qualified people are now departing the IRIS project, and, as the management systems are now in place, will be replaced by local persons who will mn the systems. Response: The evaluation observation rhat IRIS has optimized the me of expatriate staff is based on the actual budget eqenditures up to the time of the evaluation. The team smv no evidence of a 'succession 'plan for the eventual replacement of expatriate staff by local persons. 4. The expendime patterns in the first half of 2001 were based on specific requests made of the activity by the CTO and ECG staff. based on mission and embassy concern. As resources diminished, ECG and IRIS determined together to focus on critical areas and activities. The focusing on relatively few project activities in a limited number of regions is considered a success. K Conclusions and Recommendations Regarding Cooperative Agreement versus Con fract The team should make a clearer statement on the pros and cons of using assistance or acquisition mechanisms for components of the IRIS program. Issues to consider include 1) cost effectiveness for the USG in light of 25 percent matching and low overhead rate under cooperative agreement versus no matching and a three times higher overhead rates under contracts, 2) what sort of organization would be appropriate for working with universities and how it is normally reached, and 3) whether USAID purpose is to support (e.g., through a ,mnt or cooperative agreement) or to order (e.g.. contract) the development of a network Response: The comment on 'cost-effectiveness' assumes that IRIS has credible effectiveness (i.e., results) data. We have alrea& commented on the absence of such data, which means that this issue is moot as regards the IRIS program. Neither USAID nor IRIS chose to operate the IRIS program in Indonesia within a results-basedji-amework. According to interviews with USAID/Indonesia st& this could have been done with IRIS under either the grant or cooperative agreement scheme. The team has recommended that any future work of this type should be funded under a results-based procurement mechanism and that perhaps the contract procurement mechanism is the most appropriate means to achieve rhat end. Evaluation of the University E-7 July 2002 Collnborotion and Parli(~mentnry Assistance Activity: IRIS in Indonesia Development Associates, Inc. AEP-I-00-00-00023-00, Tmk Order X 814 w VZ. Conclusions and Recommendations Regarding USZNDO USINDO has been and continues to be involved on an ad hoc basis. USINDO was particularly Lri helpful in establishing relationships with the Parliament and advising on how to restructure the complaints office in the DPRD in early 2002. ul VII. Conclusions and Recommendations Regarding Lessons Learned: 1. ECG firmly believes IRIS activities did contribute to ECG strategic results, especially to w IR 4 on Knowledgeable Public Participation in Economic Decision Making and to the Objective of Setting Foundations for Rapid, Sustainable and Equitable Econom~c Growth. u 2. ECG believes there are a lot of lessons, positive and negative, to be learned from the IRlS activities. Please look again and remember the urgency of the times to respond to w economic crisis, the emerging democracy, independence movements and social violence, and decentralization. For example, ECG is convinced that the gains it sees might not have been obtained in the Parliament were a needs assessment to have been canied out in w advance of starting work. Response to # 1 and 2: The team has responded to this point in earlier comments regarding the W absence of a credible IRIS monitoring and evaluation system. ECGS conviction of IRIS effectiveness needs to be supported by inter-subjectively transmissible results data. We did not find these data in this evaluation despite repeated requests for them. bm~ 3. Testimonials by knowledgeable persons are certainly valid sources of information. U Response: The team acknowledges that 'testimonials'Ji.om informed people are useful data, and it used them to support its findings and recommendations. These testimonials gain credibility when they come ffom people who do not have a vested interest in the program evaluated. U 4. If the activity were to be extended, more follow up would certainly be on the agenda. U Team Response: The key lesson learned is that programs that do not have explicit and realistic objectives and a systematic plan to accomplish them always fail to address the problems they were intended to solve. Programs with grandiose objectives and no strategic road map with ki resources to back it up generally gravitate to ad hoc program approaches and no impact. Unfortunately, this happened here. 'd Evaluation ofthe University E-8 July 2002 b, Collaboration and Porliamenrary Assistance Aetivify: IRIS in Indonesia APPENDIX F INS EVALUATION TASK ORDER Meeting Sche(lu1es & Contact Inforn~ntiorr ...... IRIS ~ - -~ ~~. - ~~ ...... -~-.~Lp DPK Secretarial ..... -. Jim Ho~~~.- -- Firmian Aji - Bob Alcn Richard Houxh - Muhamad Rum Ali .. ..... .- Su!!!a!i w:Rosf.. Tuli Kusumas~uli ... -. - Neil tvlcMullcn ..... .............. Bruce II.?rker ........ Beny Pasnnbu Ihvid 1)cOovimi ~~ ~ ~ Moryil~~ llnll . ~ ~ William Ilala -. .... - IRIS ....... IRIS ... . -- ........... .. -- .....- I Commissio!i 1X - Budget Cot?mission ..... IRIS -~ ~ ...... 1iv~ht~li011 of 11,~ lJniwrsify 1:- I July 2002 Coli~hrulio~~ un11 I'urIi~t~~~s~~~~ry A.w~,~I~I~cc - Aerivily: INIS ht 111rl~mesiu Develnpntenl Assoeiales, Inc. AEP-1-00-00-00023-00, Task Order # 814 I I I I I Nardi Hardjo I I Research Staff, Budget I I I I I I 6/24/02 1 9:00 am I USAID/DLG I Jessica McKema 1 DLGDeputy Director 1 3435-9324 I jmckenila@,~!soid.~ov I I I I I Robert Aten Richard Hough Jim Hope 6/21/02 1:30pm 3:00 pm I I I I I 6/25/02 1 9:30 am I I USAID/DO I Mike Calavan I Democracy Team Leader 1 3435-9368 I mcalavan~i.~~sni~i.qoi 9:00 am Papua Provincial Assembly LPEM-FEU1 1 10:OO am I I PERFORMIMOF I Blane Lewis I I I Fiscal Decentralization I USAID I I I I I I I Sam Resaeboem Moharnad l!&san 6/26/02 Chairman of Committee "B (Economy) Director I I I I I I 9:00 am 967-521439 311-3177 GTZ sam resem-feui.orz Rainer Rohdewohld Local Government Advisor, MOHA 351-2609 Eain~~~r~Me?~~!!~!.~Sci~I~.nr!:~e.m. Develt,pmenr Assnciales, Inc. AEY-1-00-00-00023-00, Tork Order # 814 1 I Aksara I ASEAN I I - I I30pm 1 PEG I I World Bank -l=7--- .......... - I .. .... I ........... 430 pnr I MOHA I .... ..................... illk?~ .. IO:IJO am DPR-RI - PPPI I I I Belt Hofman Finance Law. 5299-3062 bl~ofmanric;worlclb'd~~k,or~ Dcccntralizatiun and ...... - .... Christine , ,. Flctchcr Dcccntralization Advisor 345-3079 dflc!cher?001f~vi~bao.co1n ..... .... .... Grctu Morris ~ounsc'lod~ub~ic Affairs 344221 pnniorris(@pd.stalo.~ov I -~- - ... I Advisor I Chris Mt~nning I Labor Advisor1 Regional DevelopnrenlA.~.v:soei~Ie.s, Ine AEP-1-00-00-00023-00, ?bsk Order # 814 I I I I I .- 7/9/02 I IO:30am I USAID Paul Dcuster I Economic Dcvclopn~ent 1 3435-9418 1 pdcustcr@useid.gov I I2:OOpm I USAID I Economic Development 1 3435-9230 I ratcn(4usaid.gov I I I I I .. .... " ................ .... . . ... . . . . . . . . . . . . . . . . . . . . . . - .. . - - - . . . . . . . -- . - Attendocs . .._ . Ndmr Svbjcct Area Phone 1 E-Mail I I DaLo Time 1 .-. O*gatrizallon SOURCE: SOW SECTION 11. ACTIVITIES AND ISSUES* PARLIAMENT What have been the main IRIS objectives in working with Parliament? What have been the main IRIS activities in working with Parliament (for each goal)? How long have they been implemented? How were they implemented? Who implemented them? In what ways has IRIS been most effective in working with Parliament? Least effective? How could the IRIS program activities working with Parliament be improved? Are there any other actions that could be taken to increase the positive impact of IRIS in Parliament? Should the IRIS activity in Parliament be continued? At what level (more, less) of USAID support? REGIONAL UNIVERSITIES AND SPECIAL AUTONOMY REGIONS What have been the main objectives of the IRIS program in working with regional universities (Special Autonomy Regions)? What have been the main IRIS activities in working with regional universities (for each goal)? How long have they been implemented? How were they implemented? Who implemented them? In what ways, if any, is the University Consortium relevant as a positive activity in its region? In what ways has the University Consortium been most effective? Least effective? Does the Consortium need additional strengthening? What would the most appropriate type of strengthening? Is there a demand for the research and analysis that is and will be offered by members of the university consortium? How successfully has coordination with other USAID offices and other donors been camed out? Is the Consortium sustainable, after the end of IRIS activity? Why or why not? PARTNERSHIP WITH INSTITUTE OF ECONOMlC AND SOCIAL STUDIES (LPEM) OF THE FACULTY OF ECONOMICS OF THE UNIVERSITY OF INDONESIA (FEUI) When and how was the Partnership established? * Includes actual questionslissues from SOW, as well as leod-in questions and follow-up probes. All nine issues areas are covered. Evaluation of the University G- 1 Ju@ 2WZ Colloborotion and PorLiornenlory Assistmee Activity: IRTS in Indonesia Development Associates, Inc AEP-I-00-00-00023-00, Tmk Order # 814 2. 3. 4. 5. IV. 1. 2. 3. 4. 5. v. 1. 2. 3. 4. VI. 1. 2. 3. VII. How has the partnership operated? Specific activities? To what extent has the Partnership been successful, or failed to provide sound economic advice and counsel to the Government of Indonesia? To the Parliament? Has the LPEM leadership role to the university network has been effective? If yes, in what ways? Is this role sustainable? If no, why not? If yes, what evidence supports this conclusion? How could such a role be made sustainable in the longer term. PARTNERSHIP WITH US-INDONESIA SOCIETY What is the purpose of the partnership? When and how was it formed? Has this mix of organizations been a "fertile" one? In what ways? Have any conflicts of interest or other problems been identified? Are there "lessons" to be drawn for development purposes elsewhere from the collaboration that has occurred? INCUBATOR OF PROGRAMS, PEOPLE AND IDEAS To what extent has lRIS spun off activities and individuals to other programs managed by the Economic Growth office, elsewhere within USAID, to other donors, and to the Government of Indonesia? To what extent might these "spin-offs" have inappropriately distracted IRIS or caused a redirection of resources away from activities that are in the terms of reference? To what extent are these types of university-based incubator programs useful? Should they be expanded? OTHER CONTRIBUTIONS IN RESPONSE TO DEVELOPMENTAL NEEDS IRIS has been willing and able to respond quickly to perceived technical assistance and development policy needs within the economic sphere, including those raised by USAID economic team staff. How has it responded? What have been the main positive impacts of these activities? How have these impacts advanced IRIS objectives? IRIS MANAGEMENT/PREVIOUS EVALUATION Certain potential and actual management deficiencies were noted in a previous evaluation, of part of the IRIS activity that was completed in November 2000. Have these various deficiencies been resolved in a llly appropriate manner, or are they still of concern? Areas of concern: 1. Performance Monitoring: has IRIS installed a credible performance monitoring system? Evidence? 2. Financial Monitoring: has IRIS installed a trustworthy financial monitoring system? Evidence? 3. Sustainability: has IRIS established a sound basis of support/ownership necessary to ensure sustainability? Evidence? Evaluation of the University G-2 July 2002 Coilaboration and Parliamentary Assistance Activity: IRIS in Indonesia Development Associurer. Ine. AEP-I-OD-00-00023-00, Tak Order 81d d. The overall pattern of expenditures also reveals a practice of spending the bulk of program resources on a relatively few program activities in a limited number of regions. This raises the question of the "narional" impact of the program in a corrntty as diverse and populated as Indonesia. Evuluution of fhe University 35 Jub 2002 Coltdorolion and Purlinmentury Assistance Activity: IRIS m Indonesia Development Associates, Ine AEP-Id0-00-00023-00, Td Order "14 4. Impact: Is the university consortium designedloperated to effectlinfluenceihelp shape the reform of economic governance? Is this (university-based) strategy suitable? What is expert opinion (e.g.. views of relevant Indonesian and international economistdexperts) on this question? Are there other, appropriate sources for ideadcharacteristics on how to advance economic reform? 5. Link to Intermediate Results: how (well) do the IRIS university network and parliamentary strengthening activities fit into a cooperative agreement with "Improving economic governance by strengthening private sector capacity for analysis. policy design, and promotion of sound economic policy" as the target? Are there other statementshationales within the current set of ECG IRs under which the IRIS activities (more) appropriately fit? VIII. COOPERATIVE AGREEMENT VS. CONTRACT Should IRIS continue to be implemented as a cooperative agreement or is it in the best interest of the US government to transform all or part of its current activities into one or more contracts? What would be the main differences between the two approaches? What would be the main advantagesldisadvantages to a contract approach versus as cooperative agreement? TRAINING What has been the mix of IRIS training, policy development and reform activities in Partiament and in the university consortium? Specific activities? To what extent has this mix of activities under the IRIS program been appropriate in the Parliament and in the university consortium? Has it been inappropriate and in what way? Have these activities contributed to the long-term sustainability of Indonesian policy development capacity? Why or why not? Evoluntion ofthe Universiry G-3 JuIy 2002 Colldorotion and Partiamentory Assistmee Acfiviry.. IRIS in Indonesia IRIS EVALUATION FOCUS GROUP PROTOCOL Welcome -Thank you very much for providing your time for this discussion. Purpose of the Evaluation and this discussion: to leam about the IRIS program - e.g.. what was most effective? -so we can provide advice to USAID for its future programs. The discussion will take about 50 minutes - please feel free to get some of the refreshment available during the discussion. Please try to be as brief as possible in your comments so that everybody has a chance to speak. Thank you. Please put your name and affiliation on the sheet of paper being passed around. How did you first become in contact with the IRIS program? What specific IRIS program activities were you involved in? When were you involved in them? And who else from your organization was involved? What specifically did you (or other members of your organization) do in your IRIS participation? What do you see as the main goal or purpose of the IRIS program? Was the IRIS goaYpurpose clear to you during your participation? Based on your experience with IRIS. do you think it was a usefuWeffective program? Please be specific - how was it useful or effective? What specific changes have happened because of IRIS? (If you mentioned specific changes) Do you think these changes will last after IRIS has ended its activities in Indonesia? Why or why not? Do you have any suggestions about how the IRIS program. or its activities, could have been more effective or successful or effective? Does anyone have any final thoughts or comments you would like to share with us? Thank you for your time and comments. They have been very helpful Euol~otion of the Universig H- l 2002 Collnborufion and ParEamentary Assirronce Anivig: IRIS in Indonesin APPENDIX I REFERENCES AND DOCUMENTS REVIEWED IRIS PROGRAM DOCUMENTATION (IRISIJAKARTA PROGRAM OFFICE, JAKARTA, INDONESIA, JUNE -JULY 2002) IRIS Indonesian Partnership for Economic Growth Cooperative Agreement - Draft Program of Work, (undated) Budget and Financial Enpenditure Data, IRIS/Jakarta Program Ofjice (June - July 2002) IRIS-Indonesia Partnership for Economic Growth, Cooperative Agreement Progress Reports: Weekly, Quarterly and Annual (1 999-2002) IRIS Advancing Democracy through Strengthening the Indonesian Legislature (ADIL): A Technical Assistance and Support Program (Draji Dec. 3, 2001) Program Development and Sustainability Assessment for IRIS' Support for the DPR and the DPR Secretariat, July 27, 2002 Regional Workshop Participants and Workshop Agenda Economic Education and Discussion Program. Final Report for the Pre-Election Activity (Januaty through May 1999) Memorandum of Understanding Between IRIS and the DPR-RI Secretariat Generak Advisoy Assistance to the DPR Secretariat General (undated) IRIS Cooperative Agreement Workplan for Strategic Framework, Fiscal Decentralization, IRIS Program under USAID Cooperative Agreement (Draft, May 2001) Mid-Term Evaluation Report for USAID PEG Program (October 2000) IRIS Program Success Stories (July 2002) Evaluation Questionnaire of the IRISILPEM Program on Strengthening Regional Universities within the Perspective of Fiscal Decenhalization, Isfandia~y Djafaar. (Mimeo, n.d.) Scope of Work for the Enhancement of Economic Governance During Indonesia's Political Transition (Spring 2000) Opportunities for Eqansion of the IRIS Economic Governance Program for Indonesia, Cooperative Agreement, Workplan and Budget, October 2001 - December 2002 Opportunities & Challenges: IRIS Economic Governance Program for Indonesia Proposal for Cooperative Agreement Extension (January 2002 - December 2004) Original and Current Workplan for PEG Grant, IRNLPEM Decentralizarion Grant, Life of the Grant and Year One Work Plan (November 1999) IRIS Activity Report at DPR-RI (2001 - 2002) USAIDIJAKARTA DOCUMENTS USAID/Indonesia Strategic Objective 10: Decentralization and Participatoy Local Government, Statement of Work in respect of Decentralization Policy (Mimeo, n.d) Evaluation ofthe Universip 1- 1 July 2oD2 Collaborolion and Parliamentnry Assistance Activih.: IRIS in Indonesia Development Associates, Inc AEP-I-00-00-00023-00, Tosk Order # 814 2. USAID/lndonesia County Strategy ... C. OTHER DOCUMENTS I. LPEM-FEUI Institute for Economic and Social Research, Faculty of University of Indonesia (Brochure, n.d) 2. Partnership-UNDP Activity Report at DPR-RI 3. Asia Foundation Activity Report at DPR-RI 4 KONRAD Activity Report at DPR-RI Final Iodoneria INS Report-EvalQC-10 Evaluation of the University 1-2 July 2002 CoNaboration and Parliamentary Assistance Activity: IRIS in Indonesia