Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 1 of 30 INDONESIA CIVIL SOCIETY SUPPORT AND STRENGTHENING PROGRAM Internal Evaluation Submitted by: Chemonics International Inc. USAID Contract No. 497-C-00-99-00053-00 February 2001 Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 2 of 30 Internal Evaluation Indonesian Civil Society Support and Strengthening Program (CSSP) Contract No. 497-C-00-99-00053-00 For the Period 1 January 2000 – 14 February 2001 Presented to CSSP and USAID/CPT Conducted by: Kris Merschrod Consultant to CSSP February 2001 Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 3 of 30 Internal Evaluation of CSSP – January-February 2001 EXECUTIVE SUMMARY As per the terms of its contract with USAID, the Chemonics group is called upon to evaluate CSSP’s performance and its capacity to meet the needs and undertake the tasks specified in the contract on an annual basis. The terms of reference called for the presentation of findings, analysis and recommendations. The evaluation was conducted by interviewing all the personnel involved at CSSP and CPT (the Civic Participation and Transition team at USAID), and also the directors of partner organizations. A training workshop was monitored and directors from 8 CPT and CSSP grantees were interviewed to assess the grant making and financial disbursement process. All the periodic reports and contract amendments were reviewed. Key informants and reports by experts and periodicals were used to assess the changing political context. The data management systems were examined and data used to analyze the purposes of the present grant portfolio as well as potential projects with regard to their balance in terms of the Country Strategy principles of advocacy and enabling. The accomplishments during the first quarter of FY (fiscal year) 2001 have been a substantial culmination of the first year startup activities, and it is fortunate that this internal review could be conducted at this time to document the accomplishments of the first 13 months in the field as well as to make timely recommendations for the remainder of the project. As usual, the start-up year had its shake down period with personnel and teambuilding trials. It should be noted that in this case there were two teams to integrate and bring to a consensus – CPT and CSSP. This was not an easy chore and it is generally recognized that it took a great deal of patience to accomplish. Nevertheless by the end of the fiscal year the teams were producing and the accomplishments of the first 13 months were found to be on target as per the contract. Both Advocacy and Financial Sustainability (PO [Performance Objective] 1 and PO3) were involved with international events and publications and ended the year with workshops for grantees. The financial sustainability effort led to the formation of a committee that is preparing to do an international financial sustainability workshop in Indonesia this year. Administrative strengthening of CSOs (PO2) was also on target as per the contract; a local service provider was also identified to train CSOs (civil society organizations) this coming year. One organizational strengthening aspect which is not noted in the reporting process, but which is praised by the CSOs when discussing the grant making process, is the considerable (3-4 days) time invested by the grant managers from CPT and CSSP, plus the finance Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 4 of 30 personnel, to help grantees understand how to make a better proposal, do better work plans, budget more accurately and assess their strengths and weaknesses. They note this strength in comparison with the work of other grant-making agencies or PVOs (private voluntary organizations) funded by USAID and other country programs. The joint CPT/CSSP grant screening, preparation and selection process (PO4) started immediately. Although the process was slow with only four CSSP grants totaling $680,847 (a little less than half the goal) approved by the end of the fiscal year (really only 9 months in the field), by the end of the first four months of FY2001 an additional 9 grants totaling $874,617 had been approved. Thus the grants program is also “up to speed.” The Special Activities Fund (PO5) was much more active than anticipated, with over 30 activities costing $724,976 (50% greater than planned for an average year). It is recommended that the use of these funds be reviewed so that their focus is on supporting events and organizations in geographical focus areas and more directly related to ongoing activities and potential grantees. Strategic steps recommended to assure that the required results can be achieved over the next 1½ to 3 years are: 1) Decide on exercising the option period for years 4 and 5. 2) Complete the concept papers and strategic plans for the 6 geographical focus areas. 3) Carefully analyze each region when planning grants with the idea of balancing the principles of enabling and advocacy, as stated in the USAID Country Strategy. 4) Use a proactive stance to identify potential grantees in each of the geographic focus areas. 5) Negotiate a modification in the contract to include a 90-day advance from USAID for grant disbursements rather than using the present system of requests for disbursement based on immediate, 30-day needs. Recommendations made to increase the efficiency and effectiveness of the program are: 1) Integrate the various databases into one system across sections for ease of planning and reporting. 2) Use longer advance periods for grant disbursements, especially grants to umbrella organizations. 3) Ensure that grant managers are geographically focused for fieldwork and the grants that they manage, but that proposal reviews in the office include them so as to take advantage of their technical specialties. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 5 of 30 Programmatic recommendations based on CSSP partner experience and other CSSP-type programs include: 1) The city forums work now being done by CARE, 2) The regional television work with local government, CSOs and stations of TVRI, 3) An internet, information-age strategy and training module drawing on the financial sustainability training program from Ecuador. Some recommendations are team-building approaches that should also increase efficiency, e.g., the CPT and CSSP members should list the steps in making grants and disbursing funds and then decide which steps are really necessary for the responsible administration of the grants. Perhaps a series of facilitated workshops of a day would be a good way to quickly bring closure to these issues. The joint CSSP/Mission fieldwork being done on geographical focus area concept papers is noted as a joint strategic planning consensus builder. One overarching observation is that the time frame for SOs (Strategic Objectives) involving transitions from highly centralized societies to decentralized, democratic societies probably takes a generation. Accordingly, these types of Strategic Objective should be 10-year commitments, not unlike commitments made for natural resource SOs. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 6 of 30 Internal Evaluation of CSSP – January-February 2001 Table of Contents I. BACKGROUND ........................................................................................................7 II.ASSESSMENT OF THE RELATIONSHIP TO DATE BETWEEN CSSP AND USAID/CPT...................................................................................................................7 III. EVALUATION OF THE PROCEDURES AND PRACTICES FOR GRANTS...8 A. BACKGROUND:.....................................................................................................8 B. PLANNING GRANTS ..............................................................................................8 C. APPROVING GRANTS – THE PRC MEETINGS ........................................................ 10 D. FINANCIAL MANAGEMENT .................................................................................. 10 E. MONITORING GRANTS (AND OTHER PO ACTIVITIES) ........................................... 12 IV. AN ASSESSMENT OF CSSP’S PROGRESS TO DATE IN MEETING ITS PERFORMANCE OBJECTIVES.............................................................................. 14 A. BACKGROUND: NEW RESULTS FRAMEWORK IRS AND CONTRACTUAL OBLIGATIONS. 14 B. SUMMARY OF THE ACCOMPLISHMENTS THROUGH JANUARY, 2001. ...................... 17 C. PERFORMANCE OBJECTIVE 1 “IMPROVED CSO ANALYSIS AND ADVOCACY FOR POLICY REFORM AND IMPLEMENTATION” ................................................................... 20 D. PERFORMANCE OBJECTIVE 2 “EFFECTIVE CSO ADMINISTRATIVE MANAGEMENT AND PLANNING”......................................................................................................... 20 E. PERFORMANCE OBJECTIVE 3 “INCREASED CSO CAPACITY TO OBTAIN FUNDING” 21 F. PERFORMANCE OBJECTIVE 4 “GRANTS TO CSOS AWARDED AND MANAGED EFFECTIVELY” ........................................................................................................... 23 G. PO-5 SPECIAL ACTIVITIES FUND ........................................................................ 27 VI. EXTENT TO WHICH CSSP IS MEETING OTHER ARTICULATED NEEDS AND REQUIREMENTS OF USAID ........................................................... 28 A. BACKGROUND:................................................................................................... 28 B. THE REDESIGN OF THE SO-7 RESULTS FRAME WORK, INDICATORS AND PMP:....... 29 C. THE R-4 PROCESS:.............................................................................................. 29 D. AREA PROFILES AND STRATEGIES:...................................................................... 29 E. EVALUATIONS OF COOPERATIVE AGREEMENTS: ............................................... 29 F. PUBLIC OPINION SURVEYS:.............................................................................. 29 CONCLUSION:.......................................................................................................... 30 Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 7 of 30 Internal Evaluation of CSSP – January-February 2001 I. Background Under the terms of its contract with USAID, the Chemonics group is called upon to evaluate annually CSSP’s performance and its capacity to meet the needs and undertake the tasks specified in the contract. The contents of the Scope of Work for the evaluation include numeral II through V plus annexes which are presented in a “Findings, Analysis and Recommendation” format. Comments in parenthesis [ ] are notes indicating on-going work by CPT and/or CSSP, as noted by participants in the joint CPT-CSSP roundtable discussion of the draft version of this self-evaluation on February 26, 2001. II. Assessment of the relationship to date between CSSP and USAID/CPT. Finding: As would be expected of most projects in a dynamic context where two teams are brought together for a single purpose, the first year, really only 9 months, was dedicated to establishing a consensus on implementing the concepts of the contract. Analysis: The major task was reconciling CSSP’s team view based on the original proposal, which envisioned a massive, nationwide program of grants and training, with the contract’s more specific focus of working with existing CPT grantees and with new grantees coming from the program. The other area of adjustment was in the management area. Some CSSP members envisioned a more opened role not unlike their previous experience with Cooperative Agreements. But the contract called for and required a tighter integration of all team members in both CSSP and CPT, with CPT taking the oversight and leadership role on many details. Because of the above, agreement on certain details such as criteria for project selection, training targets, and work plans took more time than anticipated. Nevertheless, as is shown later in the report, by the end of the 5th quarter a productive working relationship had been established. Its efficiency can be further improved and recommendations are later made for this purpose. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 8 of 30 III. Evaluation of the procedures and practices for Grants This should cover planning, approving, implementing and monitoring grant applications and agreements, in the context of the CPT-CSSP Project Review Committee. A. Background: In the response to the RFP it was proposed that a close relationship between CPT and CSSP be established and that within the first months of CSSP instructions for submitting proposals, criteria for selecting grantees, procedures for reviewing proposals, a customer service plan, and processes for awarding grants would be established (Activity 4.2, p.1-19). CSSP was told not to use the RFA procedure called for in the contract but rather to respond to unsolicited proposals. The process took longer than anticipated, but nevertheless by the beginning of FY2001 the number of grants made (13) was within the contract target of 10-15 grantees. [Toward the end of the evaluation CPT and CSSP staff members began a review of the current grant making process with the goal of improving procedures.]1 B. Planning Grants Finding: The new geographic focus and the concept papers are excellent steps forward from the general, nation-wide approach taken at the beginning of the project. Analysis: At the beginning of the first fiscal year there was not a geographic focus. The grant program was broadly opened to CSOs from all over the country yet focusing on democracy issues such as human rights, advocacy, and truth and reconciliation. During the latter part of the first year USAID announced a geographical focus on 6 areas. Accordingly CPT and CSSP began to shift to area focuses. Profiles of programmatic focus were prepared for 2 of the 6 areas, East Java and West Papua. The CSSP team worked with the USAID team and the new approach was immediately incorporated into CSSP’s 2001 work plan (PO 4.3 & 4.4, East Java and West Papua, respectively). The draft outline “Local Governance in East Java: A possible program for CSSP” is an example for this approach. The more comprehensive concept paper entitled “USAID’s DG support to West Papua, July 2000” is the type of work needed in each of the remaining focus areas so that CSSP activities will be part of the overall USAID strategy in each area. The joint effort represented by the East Java concept paper is also an excellent way for the USAID and CSSP teams to jointly analyze a region and come to a 1 Memo of February 26, 2001 from Mimy Santika to Michael Calavan, et al. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 9 of 30 consensus for CSSP activities and their coordination with other USAID programs. The completion of these profiles and draft activity sheets for the remainder of the areas is an urgent need in order for CSSP to have: a proactive stance; a guide for grant managers when screening and preparing synopses; and a guide for the selection of proposals by PRC. They will allow the targeting of grants to CSOs in a way that establishes comprehensive programs in each of the six areas. The phasing in of the other four regions may take more than the remainder of this (2001) fiscal year. This work should therefore also be included in the CSSP work plan for 2002. The decision on extending CSSP through a 5-year period should be taken as soon as possible because if the project is not extended, grants to the other areas would probably not be possible, the reason being that there will not be enough time for other than very brief grants to be completed during FY 2002. Recommendations: 1) Complete the geographic concept papers with CPT as soon as possible. [This was an ongoing activity at the time of the evaluation.] 2) Based on the strategy papers, take a more proactive stance by inviting CSOs to make proposals based on their experience and location so that in each of the 6 focus areas there will be a comprehensive CSO initiative to have a strategic impact on the area. 3) Based on the limits of the program, and the strategic needs of each area, write selection criteria to guide CSOs in making grants and also grant managers in reviewing proposals [already being discussed by the PRC in the concluding stage of this evaluation]. 4) Decide on the extension of the contract on CSSP before the end of this fiscal year to facilitate planning. Finding: Assignment of grant managers to both geographic area and specialist areas needs refinement. Analysis: Prior to the geographic focus, proposals were distributed among grant managers according to their specialization. With the geographic focus grant managers are assigned to specific areas. Nevertheless, some grant managers travel outside of their geographic areas to review proposals within their specialty. Considering that the role of the grants managers is not organizational strengthening in the specialist areas, it is not necessary for them to do field visits to all CSOs in their area of specialization. Their specialist knowledge can be applied more efficiently by advising the grant management team and the PRC when reviewing synopses. Recommendations: Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 10 of 30 1) Continue the practice of grant managers being geographically focused for fieldwork and the grants they manage. 2) To take advantage of their specialties, include grant managers with specialties in the review of proposals in the office. Finding: CSSP was asked not to use RFA procedures to identify potential grantees. Analysis: CPT felt that the procedure would raise too many expectations. Accordingly it advised that unsolicited proposals were preferred. During the first part of 2000 geographical focus was not established, but now that there is an geographical focus, and there should be strategies for each area, it is appropriate to reconsider the use of RFAs. Recommendations; : 1) If RFAs are not going to be used, amend the contract. 2) If RFAs are going to be used, base them on the specifics of the area strategies. [Discussion and agreement on the use of RFAs in selected geographical focus areas was under way in the PRC as this evaluation was being written.] C. Approving Grants – Project Review Committee (PRC) meetings Finding: It would be more efficient if the grant managers who have been teamed up to review a given grant were to meet prior to the PRC meeting to arrive at a consensus to present at the meeting. Analysis: Grant managers from CPT and CSSP are paired to review proposals to be discussed by the PRC. This is a good combination of their diverse skills and experience. Synopses for proposals are written and circulated the week before each PRC meeting. Thus everybody has had a chance to review the proposal synopses before the meeting. Nevertheless, it was observed that grant managers read synopses aloud in PRC meetings even though the synopses had been distributed prior to the meeting. Recommendation: Arrange for the grant manager making the presentation of a synopsis at a PRC meeting to start by simply stating the recommendation - rejection or acceptance – and the reasons for it. If the other members of the PRC agree, then the discussion can quickly shift to the type of response that should be sent to the CSO. D. Financial management Finding: The initial advances to CSO grantees and subsequent liquidations are not timely, and USAID should delegate more financial responsibility to CSSP. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 11 of 30 Analysis: The original proposal underestimated the extent of the approval process needed under a contract.2 Accordingly a few false steps were taken which have complicated the financial management of CSSP grants. It was envisioned that the first advance for a grant would be for the first two months of the grantee’s approved work plan, and that the advance would be delivered at the time of signing the Grant Agreement. Most Grant Agreements have provided for grantees to begin work at the time the Grant Agreement is signed. However, because the transfer of funds from USAID to Chemonics does not take place until after the Grant Agreement has been signed, the funds do not arrive until a month or so later. CSOs do not usually have the working capital they need and must borrow money or ask for an amendment to postpone the starting date of the grant. In the worst case interviewed, by the time the CSO received its first 2- month advance it had spent all the money it had borrowed to tide it over and was always without advance funds from then on. An additional complication is that some grantees are umbrella groups that advance the grant funds they receive to their member organizations. The member organizations have to liquidate their advances before the umbrella organization can liquidate its own grant advance. Grantees and sub-grantees are allowed only 10 days at the beginning of each month to close their books and file for the liquidation of advances: this is simply not long enough. Thus the whole system falls behind. Through the contract very little responsibility is delegated to the contractor that is not double-checked by the Mission prior to the transfer of payments to both the contractor and to the grantees. This means that the paper work has to pass through many hands. It also means that the pressure is on Mission personnel to meet a tight deadline for a small window of opportunity each month to request transfers from Arkansas to Chemonics in Washington. The responsibility to manage an advance or a letter of credit is not delegated to Chemonics. Recommendations: A few steps will help improve the situation notably, e.g., 1. Provide an advance when the Grant Agreement is signed, or make the Grant Agreement state that work will start when the first advance is received. 2. Make the first advance for 90 days instead of 60 days. 3. Arrange for the CPT/CSSP team to meet; list the steps that are presently taken; and then decide objectively just which steps are absolutely necessary for disbursing grants. This would be a good way to synchronize the team effort, and could perhaps best be done at a one-day workshop, using a facilitator. 2 Many of the CSSP staff had worked on similar projects under a Cooperative Agreement and so they made reference to that type of structure. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 12 of 30 But to really take the pressure off and relieve the administrative burden more substantial steps are needed, e.g., 1. Amend the contract so as to delegate to Chemonics the responsibility for managing a 90 day advance fund which it will use for grant-making and training based on the annual work plan and budget. Given such a fund, CSSP would report the monthly disbursements to CPT as a liquidation, and CPT, in its oversight role, would approve the report and a corresponding request to reimburse the fund. Area of Inquiry: There was not enough time to do a comparative cost benefit analysis of (a) the fieldwork done by CSSP financial management specialists in preparing grantees for grants and following up on them – doing what CPT calls ‘grant worthiness assessments’ - and (b) hiring a service provider to do the same thing. The idea behind hiring a service provider is that once a CSO is considered to be a potential grantee, a local firm is hired to review its systems, teach the basic ADS 303 requirements, and make the grant. The service provider follows up after the CSO has liquidated an advance to review the systems. A third review is then usually carried out as an annual review. This constitutes organizational development, too, which helps with the financial sustainability effort (PO3). Thus the cost is not wholly attributed to financial control. CPT has used a service provider in this way. One of the direct benefits of this approach could be to reduce the time that CSSP financial management specialists have to spend in the field so that they can attend to liquidations and advances. E. Monitoring Grants (and other PO Activities) Finding: Standard checklists in a consolidated database should be used to assess CSO capability, establish base lines and assess training needs. Analysis: Financial management specialists use an extensive grant worthiness checklist to assess the strengths and weaknesses of CSO capabilities. Advocacy and Financial Sustainability (PO1 & PO3) staff also have checklists for assessment. Grant managers (PO4) use a variety of approaches to assess the capacity of CSOs - OCA, IDF and SWOT. Thus CSSP has been gathering the necessary information, but can improve it by standardizing it, especially with respect to grants. The information is stored in various formats. The grant management team (PO4) generates synopses of CSO proposals for use at the PRC, and these are stored under each CSO in MS Word. This information is pulled from the text into an MS Access system by cut and paste, and this system is used for generating some reports. The Special Activity Fund (PO5) has a tracking file in MS Excel. Training has a MS Access database to track events and trainees. Finance has Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 13 of 30 MS Word files on the grant worthiness of each CSO, stored by CSO, and also maintains a quick book system and an Excel database arranged by CSO that focuses on amounts obligated, disbursements and the timing of reports and liquidations. Thus there are many different databases or files which can be consolidated so that the information can be easily analyzed for (1) identifying training needs, (2) planning training programs, 3 3) measuring the impact of training efforts, and 4) preparing periodic reports using process indicators. Recommendations: 1) Ensure that the financial management specialists code the findings of their complete 5-area check lists in a standard form for the consolidated database. 2) Ensure that grant managers and training specialists establish appropriate checklists for the areas of organizational capacity relevant to CSSP POs, and use a simple scale with ample classifications for each item on the check list (for example, 0= non-existent, 1= exists but needs work, and 3= exists). 3) Store and accumulate this information in an integrated database and access program that is ß comprehensive, so that each CSO can be tracked through granting, training, finance and impacts, ß on the “Y-drive” of the CSSP network so that all CSSP staff members can input and access data, ß done with MS Access, the best choice for the system. Staff who do not have MS Access experience can be brought up to speed not as programmers per se but as users, by means of information entry screens which use the same format as they are using now, for example the synopsis format. There are a few relevant Access systems which should be shown to CSSP staff so that they can see the utility and need. These are: (1) the USAID Mali “managing for results” system which the M&E specialist has; (2) the ”OTI PTG Database” which Zulli Saida at OTI will demonstrate. (Be sure to see the manual “OTI PTG Database for Dummies” as an example of what should be prepared for the CSSP system); 3) the grants tracking system currently used by CSSP; and (4) the training tracking Access program. These examples show how each of the CSSP POs can be part of a whole. A system like this will increase the efficiency of all involved, and provide a needed management system. 3 The term program is underlined here to emphasize the recommendation that a massive approach to CSO strengthening needs to be used rather than the one-on-one approach which seems to have been USAID’s preference for strengthening. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 14 of 30 An aside on GIS: Although the programmatic activities of CSSP are now largely geographically specific, unless these activities are related to smaller geographic units such as districts, sub-districts and villages, and it is necessary to show coverage in these terms, there is no need for a GIS link as envisioned during the design of CSSP, when the original intent was nationwide coverage. An aside on the organization of the CSSP “Y-drive.” At the present time the “Y￾drive” is organized by office, e.g., grant secretariat, and the content of folders determined by that office, e.g. letters. In those folders the next subdirectory may be by CSO or simply a file such as a Grant Agreement file. While this structure may replicate the structure of CSSP, it does not keep the focus on the objective of CSSP, i.e. the CSOs. It is the traditional structure used in most organizations, but a radical departure should be considered. Turn the structure around so that the first branch of the tree is the CSO. The next branch could be the PO. This way anyone in CSSP can review the raw material for the consolidated database without having to know how other POs have structured their files. If the system were designed with these two approaches, all CSSP team members could easily access all the written material on each CSO, and by means of the consolidated database generate reports, do needs assessments to design training, have baselines against which to show progress and, hence, impact, etc. IV. An assessment of CSSP’s progress to date in meeting its Performance Objectives This assessment should be in terms of the Intermediate Results of the new USAID Results Framework, and of CSSP’s contractual obligations and benchmarks. A. Background: New Results Framework IRs and contractual obligations. The New Results Framework: When the RFP and corresponding proposal were drafted, the response was in terms of Special Objective 1 “Democratic Transition Strengthened,” IR 2 “Effective CSO Participation in Political Processes,” This then became Strategic Objective 5, “Increased effectiveness of selected institutions which promote democracy” with 3 IRs. These 3 IRs were focused on “NGOs” – their organizational development and advocacy - plus the “rule of law.” CSSP was designed to respond to the advocacy and organizational strengthening needs of NGOs. The new Strategic Objective 7 is “Democratic Reforms Sustained and Deepened” and it is broader than SO5. Nevertheless, IR 7.2 encompasses the advocacy Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 15 of 30 and organizational strengthening IRs from SO5 as well as the performance objectives proposed. SO7 fully reflects the guiding principles of advocacy and enabling found in the new USAID Country Strategy. From this perspective, the new results framework does not eliminate or add to the contractual obligations of CSSP. The major change, considered a positive and strategic one, has been to the geographical focus areas noted elsewhere in this report. Although the CSSP contract has not been amended to reflect the changes in SO number and structure noted above, these changes have been included in the current work plan and indicators. The close relationship between CSSP staff and DG on the PMP and the new results framework made for a smooth transition in this regard. In conclusion, changes are not needed in the program objectives of CSSP because the goals of the original Intermediate Results still hold. Contractual Obligations: To date there have been two amendments to the contract (Sep 00 and Feb 01) neither of which changed the programmatic obligations (“targets”) of the contractor. The best basis for evaluating CSSP’s progress and accomplishments during the first 13 months (through January 2001) are the “targets” in Section C, pp. 11-17 of the CSSP contract. A summary table for the POs is followed by a specific analysis of each area. In general, given that: 1) the focus of CSSP’s organizational strengthening activities (PO1, PO2 and PO3) during the first year was on CSOs receiving grants or already in receipt of grants, 2) grant approvals started to take place in the last quarter of the first FY and the first quarter of the 2nd FY, and 3) the first year was required to identify appropriate service providers, the activities under PO1 to PO3 accomplished all that they were supposed to under CSSP’s work plan. In the case of grant making (PO4) the number of grants approved through the end of January 2001 also puts this activity on target, while the number of grant proposals on hand bodes well for maintaining this rhythm throughout the rest of FY2. Special Activity Funds (PO5) was well ahead of the original expectations both in number and dollar value. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 16 of 30 Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 17 of 30 B. Summary of accomplishments through January, 2001. First Year 2000 Second Year through January 2001 Objectives Accomplishments Objectives Accomplishments PO1 – Advocacy Identify a service provider 10 – 25 NGOs as per Contract · Found suitable Service Provider for training CSOs · TA provided directly to 3 CSOs · 16 CSOs supported in advocacy of new Foundation Law 4 10 – 25 NGOs as per contract · 10 CSOs of Sawarug attended advocacy training course · Manuals written and circulated to 500 CSOs · Case studies on advocacy begun. PO2 – Organizational Development · 10 – 25 NGOs as per Contract · 15 service providers reviewed for organizational development (OD) · 23 CSOs trained in financial management, reporting and gender. · 3 CSOs received management assessments. · 12 CSOs received one￾on-one advice on compliance with USAID regulations, strategic planning and reporting. 10 – 25 NGOs as per contract · Satunama selected as OD service provider · 7 CSOs introduction to capacity building · 3 CSOs one-on-one strategic planning and organizational assessment 4 CSSP, under this PO, funded Yappika, working with a coalition of 16 NGOs in supporting the new Foundation Law. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 18 of 30 Summary of accomplishments through January, 2001 (Continued) Objectives Accomplishments Objectives Accomplishments PO3 – Financial Sustainability Years 1 & 2 are “Phase I” in which “designated NGOs” will receive USAID funds (PO4). In phase II, year 3, NGOs will secure non-USAID funds. Nevertheless work began to prepare for phase II during year one. · 5 leading CSOs sent to workshop & 10 others linked to Manila fundraising workshop · 1 fundraising working group formed · fundraising book edited and translated by the project, sent to 300 CSOs. · Working group for Bali 2001 workshop started. Years 1 & 2 are “Phase I” in which “designated NGOs” will receive USAID funds (PO4). In phase II, year 3, NGOs will secure non-USAID funds. Preparation for phase II continued in year 2. · Case Studies on fund raising begun · Copies of “Philanthropy & Law” given to Advocacy Forum’s core group of 15 CSOs · 10 Sawarung CSOs attended the resource mobilization training course in Bandung. PO4 Grants Number screened not a target 10-15 Grants per year $10 million during the life of the project · 325 screened, 188 went to review committee · 6/45 · $680,847 Number screened not a target 10-15 grants per year. · 48 screened and went to PRC · 6/86 · $999,916 5 By the end of the period 6 grants had made it through the CPT/CSSP proposal review committee (PRC), but only 4 had the contract approval of USAID. (See next ¼ column.) 6 During the 1st quarter of FY 2001 6 grants went through the PRC and 8 were approved by USAID Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 19 of 30 Summary of accomplishments through January, 2001 (continued) Objectives Accomplishments Objectives Accomplishments Evaluation of CPTcoop-agreements and CPT grants Not anticipated but potentially 40 or more than 10 per year. No SAF evaluation targets · 2(YPMD & LPD3ES) · Follow-up on SAFs (34) Needs to be planned as noted in PO-4 review. CSSP grant evaluations also need planning. No SAF evaluation targets · 2 in process · Too early for CSSP grant evaluations. · Follow-up on SAFs (2) PO5 SAF Number not targeted $ 2 million during the life of the project · 32 · $643,109 Number not targeted $ 2 million during the life of the project Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 20 of 30 C. Performance Objective 1 “Improved CSO Analysis and Advocacy For Policy Reform and Implementation” Finding: This PO got off to a quick start Analysis: The target of this PO is advocacy training for relevant, existing CPT grantees and new grantees. It was estimated that this would include 10-15 CSOs per year. Existing literature/manuals were on hand from program personnel experience; this material was translated and published for distribution to CSOs. A trial workshop on advocacy benefiting some 10 CSOs was conducted by CSSP team members with Sawarung. The first year being a start-up year included the screening and identification of a service provider to do the training. INSIST has begun preparing case studies of successful advocacy to strengthen the training material for this activity. This effort has been undertaken in cooperation with PACT and the Ford Foundation. Thus this PO started well and accomplished the contract target for the year. In addition this PO was linked to PO5 and was supported by SAF funded activities. In addition, under the terms of this PO SAF funds were used to bring a significant group of national-level civilian and military leaders to South Africa to examine the truth and reconciliation process there. The coordinator of this PO resigned at the end of the first year and a new coordinator will be named shortly. Recommendations: 1) As soon as the new PO-1 Coordinator arrives the first task should be to prepare a list of CSOs to be trained from actual grantees. 2) With the new strategic plans for each geographical focus area the type of advocacy training and the CSOs selected for the training should be tailor-made for each area, as per the analysis and suggestion on balancing the enabling and advocacy needs of each area. D. Performance Objective 2 “Effective CSO Administrative Management and Planning” Finding: This PO made a substantial start and has had support from a broad range of project personnel. Analysis: PO2 is the organizational development component of CSSP to strengthen grantees administratively. These activities will assure that CSOs can manage grants, as well as prepare them, in order to be better candidates for other funding sources (PO3). This important effort also involved CSSP financial Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 21 of 30 management specialists as they worked with each grantee in preparation for grants. M&E staff and grant staff from USAID also participated in workshops for grantees on reporting. Grantees’ reports indicate that the work with CSSP grant managers when these managers were showing them how to make better plans and proposals constituted important technical assistance (TA). One group of 23 CSOs was trained during the first year in finance, reporting and gender. A second, more in-depth reporting workshop was conducted for the same CSOs during the 2nd quarter of FY 2001. PO2 has also provided for a network of local organizations under WALHI to conduct MIS training for members. It is important to note that under CSSP coordination, specific OAs (organizational assessments) were undertaken for 9 CPT grantees, and 7 CPT grantees were trained in strategic planning. CSSP also completed a review of service providers and identified a service provider (Satunama) to offer specific TA to CSOs and conduct workshops in areas such as strategic planning. In addition, CSSP has included funds in some grants so that grantees can contract for technical assistance directly in the case of flexible, short-term and timely needs. Thus this program objective was an excellent example of integration across CSSP sections (PO4, PO2 and the finance management specialists) and also included CPT in the effort. This bodes well for the future management of the grants as well as the quality of future grantee proposals and reporting. Recommendations: 1) Treat the baseline data generated by the grant management team and advocacy and financial sustainability specialists as OA inputs for planning TA and workshops. 2) Encourage team members working for this PO to (a) coordinate training as planned by other components to assure that CSOs are not overloaded with events and TA, (b) contract service providers to evaluate the impact of all training against baseline data 2 years after training. E. Performance Objective 3 “Increased CSO Capacity to Obtain Funding” Finding: This PO got off to a very quick and ample start, but needs to be restarted. Analysis: The success of this PO during CSSP’s first 2 years depends upon PO4 and financial management specialists because as per the contract funding counted toward this goal will come from CSSP grants to CSOs. Accordingly, it is Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 22 of 30 the effort grants managers and finance personnel make with CSOs in preparing their proposals that help obtain funds. As concluded in the next section on PO4, the granting process is up to speed and on target. Nevertheless, in order for CSOs to prepare to diversify their funding base so that the targets for years 3 and onward can be met, training was begun during the period reviewed and steps taken to continue this effort during the 2nd year. With this goal in mind this PO got off to a very rapid start by facilitating workshops; linking the CSOs to the Manila regional fundraising workshop of the IFRG (now the Resource Alliance); starting a fundraising roundtable; and helping plan for the second year a replica of the Manila workshop here in Indonesia. Also, materials were translated and printed. Before departure team members working on PO3 conducted a training workshop for about 10 CSO members of Sawarung in Bandung. It was based on the booklet called “Toward Financial Sustainability” This could be improved as a training resource if it had a manual for facilitators or trainers which would include case studies to illustrate successful Indonesian CSO fundraising activity and experiences, as is being done in PO1 for advocacy. Information-age styles of fundraising using the internet as a communications tool, web site, and means to searching for sources, etc. should become a PO3 training module. In 1999 a successful information-age fundraising training program in Ecuador managed by Fundación Esquel, Boris Cornejo (an NGO effort by USAID/E) was reviewed. The description and findings were, unfortunately, prepared in Spanish, but once the new coordinator of PO3 is on board it would be useful to consider consulting Boris Cornejo (English speaking). Copies of the report will be left in CSSP in case the new coordinator reads Spanish. Locally, the PAKTA Foundation, www.pakta.or.id, may be a useful means to achieving the goal of diversified funding for CSOs. LP3ES, with CSSP support, is preparing a source book of donors who fund Indonesian non-profit activity. This will be a useful and important contribution7 . Recommendations: 1) Make a training module of information-age-style fundraising using the internet as a communications tool. 2) See the report on Fundación Esquel. 3) Gather base-line data on the number and amount of funding of each CSO in the program at the time of contact so that the impact of both USAID funds and, later, diversified funding can be measured. 4) Consider creating a training manual, case studies and a donor source book for this PO. 7 48.6% of the 179 CSOs responding to the letter sent by CSSP to 500 Indonesian CSO had email. Thus only some CSOs are capable at this time. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 23 of 30 F. Performance Objective 4 “Grants to CSOs Awarded and Managed Effectively” Introduction: Because the grant making process is analyzed, step by step, in “III. Evaluation of the procedures and practices for grants” this section will comment on the accomplishment and content of the grants. Finding: Although a late starter, grant making is on track. Analysis: This activity took longer than proposed to start. Originally it was planned that in 90 days there would be a package of proposals with which to convene the Project Review Committee, but it took longer than anticipated to sort the large number of proposals that came in. Nevertheless, by the end of the first year the CPT/CSSP team work for screening and preparing proposals was picking up, and during the first four months of FY2001, 9 grants totaling $874,617 were approved, adding to the four grants totaling $680,847 made towards the end of FY2000. It should be kept in mind that the grants approved during the 1st quarter of FY2001 were really the product of considerable field and office work during FY2000. The average goal for each year was to fund 10-15 grants for a yearly total of $1.5 million. At the present time the proposals in “the pipeline” are as follows: Proposals in the Pipeline Category Number $ Sum P1 Ready for Approval 10 826,570 P2 needs edit/review 18 6,726,783 P3 needs much info 32 7,919,801 Clearly there are ample proposals to make a 2-year total of 20-30 grants by the end of this second year. Thus, quantitatively, PO-4 is on track. However, one should not conclude that these proposals are all of grant-level quality, nor that these grants are the only ones to be reviewed. With the new area focus strategies many of these proposals may not be acceptable. In addition, the amounts in the proposals are usually much greater than is actually merited or negotiated. Thus the prospective budget given above should treated with caution. The following table, based on previous experience, suggests that there will be ample funding for proposals on hand: Estimate of value of final portfolio from proposals on hand Category Number $ SumProbable approval level % Probable final amount P1 ready for approval 10 826,570 0.6 495,942 P2 needs edit/review 18 6,726,783 0.53,363,392 Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 24 of 30 P3 needs further work 32 7,919,801 0.1 791,980 Sum 15,473,154 4,651,314 already committed 1,454,700 Total 6,106,014 One problem is that the area strategies are not completed, and it is doubtful whether the PRC will be able to select projects for all 6 geographical focus areas quickly enough for funds to be completely committed and used before the end of FY 2002. This being the case, as mentioned elsewhere it is important to decide on the contract extension through September 2004 in order to make appropriate obligations for the six area strategies. Recommendation: 1) See next section on balancing the portfolio. 2) Keep in mind the geographic strategies for future grants. Finding: The balance of the grants portfolio between enabling and advocacy needs to be monitored and continued coordination with other SO activities is important. Analysis: The USAID country strategy presented two guiding principles - advocacy and enabling – which should be balanced to help maintain stability as well as to support democracy and decentralization reform efforts. Although the number and dispersion of present grants does not make a region￾by-region analysis possible, a brief review from the enabling/advocacy perspective is worthwhile to illustrate the two principles guidelines. The present portfolio of 13 grantees approved by the CSSP/CPT process through January 2001 can be classified into four categories of purpose, as shown in the following table: (n.b. not all the grants can be neatly classified into either enabling or advocacy.) Classification of Portfolio as of 12 January 2001 Purpose Average Number Sum $ Percent 1 - Enables government to respond 100,093.67 3 300,281 21 2 - Promoting advocacy 120,346.64 7 842,426 58 3 - Ethnic mediation efforts 84,137.00 1 84,137 6 4 - Study or Investigation 115,439.43 2 230,879 15 Total 112,132.56 13 1,457,72 3 As can be seen, most of the budgetary effort (58%) has been on the advocacy side and comparatively little (21%) on the side of preparing the system to be able to be more responsive. These two efforts should be balanced in each Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 25 of 30 geographic area so that the “system” can move toward the goal of increased citizen participation and be more responsive to diverse interests. To assess this balance in terms of the potential portfolio pending proposals8 can be classified9 into four categories of purpose as shown in the following table: Classification of Portfolio (P1 thru P3) as of 12 January 20015 Purpose Average Number Sum $ Percent 1 - Enables government to respond 411,101 10 4,111,011 28 2 - Promoting advocacy 261,662 38 9,943,157 69 3 - Ethnic mediation efforts 64,498 5 322,490 2 4 - Study or Investigation 22,826 3 68,477 1 Total 56 14,445,135 Again, note that more than twice the funding effort goes to advocacy (68%) vs. enabling (28%). This is only a rough approximation of the purpose of the potential portfolio, but it brings to light meaning of the country strategy using these two principles – advocacy balanced with enabling. A type of project that would combine the development background of NGOs with the civil society focus would be to team NGOs with local government to do participatory planning and to establish development and budgetary priorities. If this could be done in conjunction with the new decentralization budgets, then local priorities and citizen participation would strengthen local governments instead of overburdening them with rising and, perhaps, unrealistic expectations. Projects of this nature could combine the technical expertise of the development NGOs with the political legitimacy of the newly elected officials. Two excellent examples of this type of work can already be found in work CSSP is undertaking with its partners: 1) the “City Forum” scope of work being developed by CSSP with CARE in 12 kabupaten in East Java with the purpose of empowering CSO groups in clean governance while developing their partnership with local government; and 2) the CSSP project being developed this year with IFES to bring together community, local government, CSOs and local TV stations in six target areas. Coordination with SO-10.2, “Local Government Capacity strengthened to deliver effective services” will be crucial to implement the aforementioned guiding principles for the country strategy. The “Participatory Development Planning” (PDPP) started under SO-10 and now encouraged by the Home Ministry is the type of enabling activity to which CSSP work can be attached for increased synergy. It should be noted that this work is of interest to the Asian Development Bank and other groups such as the National Democratic Institute (NDI), so 8 4 of the 60 proposals were not classifiable for a lack of information on the evaluator’s part. 9 The classification was based on the summary description of the purpose in the grant-tracking database. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 26 of 30 coordination is extremely important. Fortunately the COP of CSSP has been actively networking in this area and the aforementioned areas strategies have CSSP personnel on board. Recommendations: 1) Balance the effort in each geographic area to assure that advocacy is matched with a corresponding enabling effort. 2) Make this type of analysis part of each of CSSP’s area strategies. Finding: The grants need to form a program of considerable duration Analysis: A review of grants made so far reveals that they are scattered geographically. While these grants are no doubt important for the overall effort, future grants should be part of a long-range program in given geographic focus areas so that there is continuity and synergy. The coordination of enabling and advocacy will lead to changes in the perceptions and expectations of society as structural changes take place and people interact with them. However, as mentioned in the country strategy paper, these shifts will take a generation before they evolve to be the new social and political organization. To put this sort of transition into context, note that most efforts to create an environmental movement in a country take at least 10 years to consolidate. It usually takes 4-5 years to create a foundation or umbrella organization, even though board members are formed from people sharing a common vision and mission. Therefore the transition envisioned in the socio￾political context of Indonesia will take longer than the present project timeframe. The present grants are mostly less than 2 years in duration. Grant proposals funded should take into consideration the fact that at least 3 to 5 years will be needed, either of constant effort with particular local governments or of intermittent efforts with many local organizations or governments over a large area. Recommendations: 1) Make the time frame for SO7 10 years and for projects like CSSP 5-year periods with the expectation that they will probably continue for more than one 5-year period. 2) Make grants for CSO projects 3 years long with the possibility for renewal. An aside relating the time frame to the grants processing burden on CPT: If CSSP is not extended to a 5 year time frame, then it will require more grants of shorter duration to support the work in each area so that they can be closed by September of 2002. Of course a few larger grants could reduce the grant- Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 27 of 30 processing burden, but, given the need to incorporate new CSOs in the new target areas, with less track records and small staffs, this strategy would probably lead to over extended CSOs. It is a known problem that CSOs can easily be funded to expand beyond their programmatic and financial managerial capacity. POs 1,2 and 3 can be a great help to these CSOs, but not within the time frame of the present agreement. Finding: Under PO4 there are, potentially, a large number of evaluations of the Cooperative Agreements held in CPT’s portfolio. Analysis: One of the Tasks (4.1.11) of this PO took on added significance during FY 2001, that is, CSSP was asked to include the evaluation of cooperative agreements that are in CPT’s portfolio. The original logic of this task was the evaluation of grants generated through the PO4 process. This is another example of good service provided by CSSP to CPT that relieves the managerial burden on the CPT staff by either doing the work directly with CSSP staff or by contracting consultants to do the work. Otherwise CPT would have the arduous task of the using IQCs or conducting bids. To date one evaluation was done by a CSSP member and a consultant, another was subcontracted (LP3ES) and a third is being sub-contracted as TA at this time. There are approximately 40 agreements (between agreements, extensions and projected to be signed) in CPT’s portfolio at the present time which is, potentially, a substantial number of evaluations to be conducted each year which can be planned in advance because the mid-term and final closing dates are established in the agreements. Recommendation: CPT should provide CSSP with a calendar of evaluations for each year and, once the full extent is established, plan and budget the staffing needs or sub-contracting for the length of the project. G. PO-5 Special Activities Fund Finding: The SAF has been very active and has supported other POs and it should reinforce and complement the main grants and training. Analysis: This fund was very active and it is easy to underestimate the amount of time and effort by CPT and CSSP personnel to manage this fund. Most of the requests came from an inventory of requests at CPT, and this allowed the activity to have a quick and substantial start. To date 33 special activities, totaling $724,976, have been funded. In the contract, as amended, the target was to allocate $ 1.122 million to an unspecified number of CSOs. Clearly in terms of quantity this PO is on schedule. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 28 of 30 One of the drawbacks in this activity has been that CSOs have not been holding up their side of the agreement to file final reports. Only 9 of 33 special activities had been closed out by the time of this review, even though none was pending. A review of the special activities reveals an ad hoc use of special activity funds that does not seem to add to the “program” except for a few (about 30%) notable cases. It is however, difficult to judge the appropriateness of these activities because there has not been a concise program with a geographic and programmatic focus. Nevertheless, the notable ones are worth mentioning because without this fund important activities could not have been undertaken: 1) When the truth and reconciliation process was being considered at the national level, SAF funds provided the means for civilian and military officials to travel to South Africa to review the process of reconciliation there. This was related to PO1 2) In the face of inter-ethnic conflict, groups were brought to training sessions. 3) A “critical mass” of CSOs was built by funding travel to the Manila philanthropy conference in 2000 and this helped for the group behind the conference to be held in Bali during 2001 for training and networking for fund raising. This reinforces PO3. 4) The fund allowed the program to work with a few potential grantees to know them better. Recommendation: The fund should be used for activities to get to know potential grantees by funding relevant events in key areas, to support events for ongoing programs, and for events which reinforce the work which POs are funding, in other words, short term events related to longer, ongoing grants. [On-going discussions within CSSP and with CPT are refining the criteria for the use of these funds.] VI. Extent to which CSSP is meeting other articulated needs and requirements of USAID A. Background: During this first period, USAID has articulated other substantial needs that CSSP has been able to address with quality. It should be noted that the joint CPT/CSSP work on the area strategies and the results framework has helped CSSP and CPT begin to have a shared vision. The CSSP work plan for 2001 reflects this shared vision and understanding and shows a smooth planning transition from old SO-5 to SO-7. Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 29 of 30 B. The redesign of the SO-7 results frame work, indicators and PMP: The transition from SO-5 to SO-7 results framework, appropriate indicators and corresponding PMP were accomplished by assigning an experienced CSSP staff member to work with the SO-7 team as a facilitator and complier of the consensus of the CPT staff. Not only was the result of the effort important help to the USAID; it helped transfer the content of the new results framework to CSSP which will be important to adjust the current CSSP work plan and, also, for planning the activities and focus for 2002. C. The R-4 process: DG also asked for assistance in preparing the R-4 in 2000. As in the case of the new results frame, an experienced person from CSSP was assigned to work with the DG team to complete this important Mission task. The background material was gathered, ordered it and drafts of the R-4 prepared for review. This same person is presently preparing for the R-4 that will be due in March of 2001. D. Area Profiles and Strategies: CSSP has assigned experienced staff to the fieldwork for area strategies or concept papers for the geographic area foci. This not only strengthens DG and, indirectly, other SOs, it helps build a common vision and team effort for CSSP and CPT. E. Evaluations of cooperative agreements: CSSP has begun to assume the responsibility for the evaluations (3 to date) of Cooperative Agreements held by CPT either by directly assigned CSSP staff to the evaluation or by subcontracting consultants. The SOWs and selection of consultants is a joint CPT/CSSP effort, but the task of identifying and supervising the evaluations is a CSSP effort which normally would be a Mission or SO chore under IQCs or contracts. This sub-contracting is an administrative relief for DG F. Public opinion surveys: The most important impact indicator for SO-7 is the public opinion survey; CSSP has assumed the responsibility for contracting the service provider for the survey. G. Justice Sector Assistance: CSSP carried out a few significant activities in support of the Mission’s work on the justice sector. It supported a review, through the CSO CINLES, of the commercial courts, participated in inter-agency roundtable discussions on justice Internal Evaluation of CSSP January/February 2001 kris@ merschrod.net 30 of 30 sector reform, enabled the University of Indonesia Human Rights Center to provide technical assistance to the DPR for its work on drafting a human rights court law, supported the National Law Commission to organize fourt national symposia on key aspects of law reform, and commissioned two reports for USAID on justice sector reform by a leading international consultant. Conclusion: The contractor and USAID have made a substantial first-year effort to bring together two teams to strengthen civil society organizations. The coordinated effort of grant managers from CPT and CSSP has provided grantees with technical assistance in making proposals and in reporting on the substance and financial resources of the grants. Both teams have been able to process a substantial number of special activity grants. By these means CSSP has shown that it is “up and running” and can manage and achieve the targets as per the contract. The recommendations made can be used to strengthen the joint CPT/CSSP effort so that they will be more effective and efficient. The major recommendation is that an early decision be made on the extension of the project for years 4 and 5 so that this year’s decisions and activities can prepare the way for greater impact in the six geographic areas.