SAMBA Project Final Evaluation Report May 2004 OD and Management Consulting Jl. Tunjung III/10, Tomang, Jakarta Tel/Fax (021) 568-0359 fcpage@bigfoot.com Frank C. Page TABLE OF CONTENTS 1. EXECUTIVE SUMMARY.........................................................................................................................................I 1.1. PROJECT PURPOSE..................................................................................................................................................I 1.2. EVALUATION OBJECTIVES.....................................................................................................................................I 1.3. SUMMARY OF EVALUATION ISSUES.....................................................................................................................I 1.4. CONTEXTUAL ANALYSIS.......................................................................................................................................I 1.5. PROJECT DESIGN....................................................................................................................................................II 1.6. PROJECT HISTORY .................................................................................................................................................II 1.7. PROJECT IMPLEMENTATION...............................................................................................................................III 1.8. RESULTS AND ACHIEVEMENTS............................................................................................................................V 1.9. SUSTAINABILITY....................................................................................................................................................V 1.10. RECOMMENDATIONS............................................................................................................................................VI 1. INTRODUCTION........................................................................................................................................................1 1.1. EVALUATION OBJECTIVE...................................................................................................................................... 1 1.2. EVALUATION TEAM.............................................................................................................................................. 1 1.3. BRIEF BACKGROUND ............................................................................................................................................ 1 2. METHODOLOGY.......................................................................................................................................................2 2.1. DESCRIPTION OF EVALUATION TOOLS AND METHODS..................................................................................... 2 2.2. LIST OF INFORMATION SOURCES......................................................................................................................... 3 2.3. EVALUATION SCHEDULE....................................................................................................................................... 3 3. CONTEXTUAL ANALYS IS ....................................................................................................................................3 4. FINDINGS ......................................................................................................................................................................4 4.1. PROJECT DESIGN.................................................................................................................................................... 4 4.1.1. Proposal and Logframe...................................................................................................................................4 4.2. PROJECT IMPLEMENTATION................................................................................................................................. 8 4.2.1. History................................................................................................................................................................8 4.2.2. Purpose..............................................................................................................................................................9 4.2.3. Structure..........................................................................................................................................................10 4.2.4. Helping Mechanisms.....................................................................................................................................13 4.2.5. Rewards and Motivation...............................................................................................................................15 4.2.6. Internal Relationships...................................................................................................................................16 4.2.7. Leadership.......................................................................................................................................................16 4.2.8. Skills.................................................................................................................................................................17 4.2.9. Gender..............................................................................................................................................................17 4.2.10. External Relationships.............................................................................................................................17 4.3. RESULTS/ ACHIEVEMENTS................................................................................................................................. 19 4.3.1. Achievements Relating to Objective 1 ........................................................................................................20 4.3.2. Achievements Relating to Objective 2 ........................................................................................................23 4.3.3. Conclusion.......................................................................................................................................................26 4.4. SUSTAINABILITY.................................................................................................................................................. 26 4.4.1. Exit Strategy....................................................................................................................................................27 4.5. PROJECT STRENGTHS AND WEAKNESSES AND RECOMMENDATIONS.......................................................... 28 4.5.1. Summary of Project Strengths and Weaknesses.......................................................................................28 4.5.2. Recommendations and Suggestions............................................................................................................29 List of Tables Table 1: Evaluation Workshop Evaluation of Goals and Objectives................................................. 8 Table 2: SAMBA Project Outputs by Activity for Objective 1........................................................20 Table 3: Results Observed for Objective 1 ....................................................................................22 Table 4: SAMBA Project Outputs by Activity for Objective 2........................................................23 Table 5: Results Observed for Objective 2 ....................................................................................25 List of Figures Figure 1: Project Structure - Year One ..........................................................................................11 Figure 2: Project Structure – Year Two.........................................................................................12 SAMBA Project Final Evaluation Page i 1. Executive Summary 1.1. Project Purpose The purpose of the SAMBA project was to improve the capacity of Business Development Service Providers (BDSPs) to support and develop Micro-Entrepreneurs capacity in silk production and marketing. The SAMBA Project office in Bandung, West Java implemented the SAMBA Project for 27 months from February 2002 through April 2003. Over this period, the final budget for the project was US$ 928,374. Of that amount, US$ 699,999 was provided by a grant from USAID and matches from the grant recipients provided US$ 228,375. The official goal of the project was: In West Java, 2,000 silk related Micro Entrepreneurs achieve access to 90 improved Finance and Non Financial Service Providers through a competitive and market oriented approach. In order to achieve this goal, the project proposed to achieve two objectives: ß Objective 1: 50 Business Development Service Providers deliver improved services to increased number of sericulture Micro Entrepreneurs; and ß Objective 2: 40 Micro-Finance Organizations deliver improved financial services with an increased lending base to an increased number of Micro Enterprises served. Half way through the project, the second objective was modified with the knowledge and agreement of the donor USAID. With this modification objective 3 became: Strengthen the capacities of 5 to 10 MFIs to provide quality services with market oriented approach [sic] to SMEs with special focus on silk businesses in West Java This project hierarchy reflects each partners (UNBAR, CARE, and USAID) individual desires for the project. UNBAR wished for a project that would use its R&D to assist silk farmers. USAID wanted a project that supported business development in West Java. CARE wanted a project that would utilize and develop its experience in micro-credit. The compromises made to include each of these goals are clear in the project proposal. UNBAR and CARE agreed to work through the BDSPs to reach the farmers and USAID agreed to accept the financial/credit portion of the project. 1.2. Evaluation Objectives The primary objectives of this evaluation is to assess the level of attainment of project results in relation to the SAMBA project objectives and to identify lessons learned from the project. This evaluation is underpinned by a capacity building ethos that sets out to build the knowledge and skills of both CARE-UNBAR project staff and CARE International Indonesia in order to increase their effectiveness in programming and implementation, particularly in the area the Small Economic Activity Development (SEAD). 1.3. Summary of Evaluation Issues The scope of work lays out three key evaluation issues. They are: 1. Project Design & Implementation: Assess the project design and reflect critically on the focus and overall quality of the management of implementation. 2. Results/Achievements: Assess the level of attainment in project results (project outcomes and activities). 3. Sustainability: Present an analysis of the potential for sustainability of the project. These issues provide the outline structure for this executive summary and evaluation report. 1.4. Contextual Analysis When the project was being designed in late 2001 and implemented during 2002, a key aspect of the silk production chain in West Java was PT Indo Jado, a large silk spinning company. This company dominated the silk weaving segment of the industry, and provided what was essentially a monopsony (single buyer) to the cocoon producers of West Java. In the third quarter of 2003, PT Indo Jado was forced to close down, as it could SAMBA Project Final Evaluation Page ii not procure enough cocoons to support its spinning machine. Almost overnight, cocoon producers lost their major buyer and the region’s capacity to produce cocoons became greater than the region’s capacity to process them into silk yarn. Silk weavers lost a major local supplier of silk yarn and many of them then turned to importing yarn in order to continue their weaving business. The closing of PT Indo Jado resulted in a fundamental change in West Java’s silk industry changing a key silk industry constraint from a low local production of silk cocoons to a limited market for silk cocoons and low local production levels of silk yarn. 1.5. Project Design The background and preliminary assessment sections of the proposal describe the production chain of silk industry with the emphasis on the production of mulberry, cocoons, and silk yarn. The analysis, in a round about way, points out that at the time of project initiation, a key constraint to the silk industry was cocoon production. However, the analysis does not do a complete analysis of each aspect of the production chain, nor does it clearly point out opportunities and threats to the industry. Thus the initial design and implementation focussed mainly on cocoon production. The program did work to support other aspects of the industry, however it was the closing of PT Indo Jado and the need to respond to the loss of a key market and supplier that forced the project to increase its focus on developing silk weavers and supporting cloth producers. The project’s logframe demonstrated some weaknesses. First, it was not a logical hierarchy. The goal was a reiteration/combination of the two objectives. But a goal should be a higher order achievement that derives from the achievement of the objectives. Thus the goal as stated is not such a higher order achievement and therefore does not help project implementers understand the underlying purpose of the project nor the relationship between the two objectives. Second, the logframe did not provide any performance indicator for the goal, but did provide performance indicators for the objectives. The performance indicators are that were given for the objectives were the minimum indicators required for such a program. The evaluation found the first objective to be specific, measurable, achievable, realistic, and time-bound, the second objective, as originally planned, could not be considered achievable and realistic within the two-year time frame of the project. As this became apparent during implementation, along with the recognition that the project had not budgeted appropriately for 40 MFIs, the project readjusted their approach to working with MFIs as described above. Overall, the planning was ambitious. To set up a program from scratch to develop an industries BDSP, even a relatively small industry such as the silk industry in West Java, requires a period longer than two years to become both effective and efficient and to achieve high levels of observable impacts. 1.6. Project History The SAMBA Project was started in February 2002 and completed in April 2004. The history of the project can be broken up into two distinct periods. The first year and a quarter represents project start up, socialization, and the struggle to develop its approach, identify BDSPs and MFIs, and develop the training programs it hoped to deliver. However, progress was not as fast as required for a two-year project and in the second quarter of 2003, the Project Director was replaced. The new Project Director reorganized and restructured the project, most notably by adding 10 field staff. With the addition of the new staff stationed in the field, the quantity of project activities (trainings, workshops, and consultations) increased greatly. This period is referred to as “year two” or the “second year” although it does include the work done up until the project closed down in April 2003. Two trends are clear from the project’s history. The first trend was toward a greater numbers of activities implemented more efficiently during the second year. The second trend was that the project activities increased their breadth of assistance to the silk industry by promoting yarn production, disseminating both technical and market information on the silk industry, developing new markets, and promoting networking among the key actors of the West Java silk industry. SAMBA Project Final Evaluation Page iii 1.7. Project Implementation 1.7.1. Staff Understanding of Project Goal and Approach For a project team to come together, it is important that they share a common understanding of the project’s goal and how the strategy the project will use to achieve that goal. As noted above, the logframe does not provide the project with an explicit higher order goal. Therefore, the project staff developed their own working which was to increase the incomes and welfare of silk farmers; this goal became the actual driving objective of the project. This objective clearly reflects two influences. First, UNBAR’s intention to help silk worm farmers embodied in its original proposal to USAID, and second, a strong feeling within the silk industry that silk farmers are under producing. Nevertheless, all the staff interviewed clearly understood that the project’s approach was to develop the capacity of the BDSPs to provide silk farmers with markets and/or technical training/extension. 1.7.1.1. Project Structure During the first year, it became clear that the originally planned staffing levels could not effectively carry out all of the activities planned by the SAMBA project. The small staff could not cover the large project area and maintain the SAMBA presence in the field necessary to assist in the logistics of planning and implementing a large number of workshops, seminars, trainings, and follow up. Thus, during the first year, the project was essentially centered in Bandung with weak efforts to ensure a field presence in other areas of West Java. The net result was slow implementation. This situation was recognized and improved upon in year 2 with the reorganization of the project. The reorganization added 10 field supervisors, 2 financial specialists, and a media person. The UNBAR staff were then organized into a specialist group that would provide support for the field supervisors such as conducting trainings and providing advice. These approaches made more effective use of the time UNBAR staff were available to the project. The addition of full time field staff who worked full time in different regions of West Java was an important reason for the project’s ability to implement its programs and meet its output targets during the second year. The one area that the structure did not support the objectives of the project was by separating the non-financial and financial aspects of the program. The financial specialists developed economic models for the silk industry and were sharing that information with financial institutions in West Java. However, the field supervisors did not have this information and therefore was not sharing it with the BDSPs. The net result of this division was that the project did not coordinate its efforts to promote credit for the silk industry. The interviews of bankers and BDSPs clearly indicate that there is a gap between what the BDSPs want and need for credit terms and what the banks are able or willing to offer. In such a case, it is possible that both sides might benefit from learning more about the other in order to increase the possibility of negotiating mutually acceptable credit agreements. To bridge this gap, both silk producers and banks need to have good understanding of each other’s business in order to build the necessary relationships and develop innovative approaches. 1.7.2. Monitoring and Evaluation For the monitoring system, the reports and data available from the first year are relatively sparse and not very descriptive. The quarterly reports attempt to describe activities undertaken as laid out in the logframe and work plan. However, the reports are often not clear and are padded with largely unnecessary annexes. The planning, monitoring, and evaluation were greatly improved during the second year. The improved monitoring system was based on the original logframe and work plans updated in January 2003 and again in October 2003 by the project management team. These plans were structured by the major activities being implemented by the project and provided the outline for the monitoring and reporting system for year two. This system included weekly and monthly meetings, monthly reports, quarterly reports, and a baseline and final evaluation survey of BDSPs and MEs. This planning and monitoring system implemented by the new Project Director greatly improved the day to day management of the project and contributed to the substantial increase in completed activities the project achieved in year two. The projects monitoring system instituted in the second year focused on reporting and measuring project activities and outputs. The weakness of the system is that, other than the final evaluation survey, it did not seek to collect information on higher order results, such as outcomes and impacts. However, this oversight is SAMBA Project Final Evaluation Page iv understandable given the short time period remaining in the project (less than a year) and the pressure to improve implementation when the project’s M&E systems were upgraded. 1.7.3. Staffing and Teambuilding This hiring process in the second year followed a couple of practices unusual in NGOs. The new Project Director used a hiring firm, which applied psychological tests to prospective employees, and, once hiring was completed, arranged for a jungle survival team building exercise for the new employees. The use of hiring firms and psychological testing is a relatively common practice among Indonesian businesses. If the tests have been demonstrated to be statistically reliable and valid (consistently and actually measure future employee performance), then such test are very useful. “Outward Bound” type team building exercise are also more common in business than with NGOs. The purpose of such activities is to help staff to learn about group development and to understand and work with each other’s idiosyncrasies. The key to such team building exercises is to bring the learnings back from the field into the office, and that apparently took place in this situation. It is difficult to diagnose internal relationship issues that have occurred in the past among people who are no longer present; therefore, this evaluation did not attempt to evaluate the project’s internal relations during its first year. However, during the second year, it was clear from the evaluation interviews that the current Director and his team had managed team building and internal relationships well. All staff interviewed indicated that they worked well with their colleagues both within their unit and outside their unit, and there were no indications of conflict. Another indication of the success of the project’s teambuilding was that staff motivation remained high even during the project close down phase. 1.7.4. Gender The SAMBA project did not implement any efforts to take into account gender in its planning, implementation and monitoring. Internally, the office staffing reflected traditional gender roles with women in administrative and financial positions and men in technical and field positions. 1.7.5. Relationships with Stakeholders The structure of the partnership with UNBAR made true partnering difficult. The partners did agree that Care would be responsible for overall management of the project and that the Care staff would be full time and the UNBAR staff generally half time (including the Deputy Director). Nevertheless, the unequal relationships for responsibility, participation, and pay levels lead to feelings on the part of UNBAR that the collaboration was not a true partnership, but that UNBAR worked for Care. Given the partnership structure, this should not be a surprising outcome as true partnering means shared responsibility, and officially this was not an option under USAID requirements. Given this official disparity in responsibility, it was difficult to “unofficially” practice true partnership. Aside from the issues described above, the University did not feel that the project had supported their research and development efforts as much as they had hoped it would. However, it is important to note that this was not an explicit goal of the project or partnership. Upon reflection, it does not seem that partnering with an international NGO to implement a short-term project was the right tool for developing the university's research and development. The SAMBA Project’s relationship with BDSPs was excellent. The BDSPs high regard for the project was reflected in the unanimous recommendation from all the BDSPs involved in the evaluation for the SAMBA project to continue operation. The BDSPs acknowledge the assistance of the project in improving cocoon production, helping with management during difficult times, and providing opportunities to expand markets and networks. It was clear from the reactions of BDSPs that the project has developed a high level of credibility with the BDSPs and is in a good position to continue working with them. The MFIs that participated in SAMBA activities included BPRs, LPKs, BMTs, Cooperatives, and general banks. As with the BDSPs, the project’s relationship with MFIs appears good. Respondents from MFIs recognized the potential benefits of the SAMBA project, and appreciated the information they received about the silk industry from SAMBA. However, the MFIs met during the evaluation did not see a real fit between their business and the businesses SAMBA was promoting. The mulberry and silk worm farmers were not a primary target group for the SAMBA project, and this was born out in the interviews. The farmers interviewed during the evaluation were generally aware of SAMBA’s SAMBA Project Final Evaluation Page v existence but were not clear on what type of organization SAMBA was. Given that the project was not designed to work directly with the farmers, this result was expected. 1.8. Results and Achievements In its work with the BDSPs and Silk Industry (objective 1), the SAMBA project was able to effectively implement the activities and achieve the outputs planned in the project paper. Due to the slow start in year one, much of the work was accomplished in year two and with the help of a short (two month) extension; nevertheless, with this help the project met its work plan for the two years. As a result of the project’s activities and outputs, those interviewed during the evaluation indicated the following trends: · Farmers had adopted improved moriculture and sericulture practices and some had benefited financially from the improvements. · BDSPs are able to provide better technical advice/training to farmers; · BDSPs have somewhat improved abilities in financial analysis and administration; · Silk industry business and marketing networks have developed. · Increased interest and mutual support within the West Java Silk Industry Unfortunately, the loss of markets due to the closing of Indo Jado and recent experiences with poor quality worm eggs, have reduced the expected impacts of outcomes designed to increase silk cocoon production. The SAMBA project did help some BDSPs develop skills and ability in silk spinning and one BDSP leased spinning machines from the project. However, the installed base of spinning machinery in West Java is not enough to replace what was lost when PT Indo Jado closed down. Making up for this will require more than the SAMBA project was designed to offer in both access to capital and time to develop. In its work with Micro-finance institutes (objective 2), the project found that the original objective and indicators of success were unachievable, and therefore changed its objective and approach. The project reduced the number of MFI receiving funding and expanded working with MFIs to increase their understanding of, and interest in, the silk industry. To achieve this objective, the project: · Provided financial support totaling US$ 124,977 to five Bank Perkreditan Rakyat (BPR) through CARE’s partner BISMA; · Held seminars and workshops to introduce MFIs to the Silk Industry and train them in evaluating silk related businesses; and · Promoted market linkages for the silk industry The financial support for the 5 MFIs has helped the institutions to increase their lending portfolio and total number of borrowers. Most of these ME borrowers are in micro-trading and micro-services. The promotional activities conducted by SAMBA have resulted in MFIs having a greater understanding of the silk industry and in cases where MFIs were already lending in the agricultural sector an increase in interest in lending to the silk industry. The project reports that five MFIs have provided financing to silk related businesses. However, it must be said that from the BDSPs perspective, this effort has not generally created suitable opportunities for financing. Overall, the project met the goals and objectives it set for itself (after the midterm adjustments for the second objective for MFIs). Given the slow start in the first year of the project, praise should be given to the project management and staff for the second year for their efforts to implement and complete the activities laid out in the work plans. With this effort, the project was able to implement its activities and achieve the outputs planned. Thus they were able to achieve their objectives, farmers’ skills and knowledge increased, production increased in places, BDSPs ability to provide technical support increased, MFIs gained knowledge and interest in the silk industry, and the industry strengthened its own networks and reached out to explore new markets. 1.9. Sustainability The level of support to BDSPs and the silk industry provided by the SAMBA project was clearly not sustainable, nor was it designed to be sustainable. Without the outside funding, or funding from the industry itself, the level of trainings, extension, consultations, workshops, and seminars implemented by the project can not be maintained. SAMBA Project Final Evaluation Page vi The key to the sustainability of the project’s activities rests in the skill and knowledge gained and put to use by the BDSPs and farmers. The large training programs arranged and supported by SAMBA do not appear to be replicable without outside funding. However, the technical skills gained by the BDSPs will be maintained in their extension programs and the BDSPs reported that they would like to continue local training they do with their farmers, although in a scaled down manner. The sustainability of the project’s networking and marketing activities now depends on the industry members themselves. Such activities such as business meeting seminars, marketing seminars, etc. can be arranged by the industry itself. The BDSPs as a group can easily donate the funds for such activities if they were willing and organized. However, it is not clear at this point if the industry recognizes this and/or if there will emerge a leader to organize such activities. The exit strategy developed by the project was a combination of development project logframe describing the development work of the Yayasan and a set of business plans describing potential businesses that would generate income for the Yayasan. The log frame demonstrated the same weakness as the original logframe in that the goal basically restated the combination of the objectives. The Yayasan SAMBA should explore the many possible ways to describe a higher order result for its goal that would infer a stronger, more sustainable, and possibly self managed silk industry in West Java. The business plans put together to describe potential businesses the Yayasan could develop to support itself is essentially incomplete in two ways. The first is that the numbers presented in the plans are unsupported. The second is that the business plans received by the evaluator do not show cash flow or income projections or break-even points. It is very likely that many of these businesses are viable, but they do not yet layout: · The need for start up capital for both businesses and Yayasan and how that need will be covered; · When the incomes from the businesses will start supporting the work of the Yayasan; · Action planned to be implemented; · Organizational structure; and · How much financial support these businesses will provide the Yayasan once they are up and running. As the project is now completed and the Yayasan is an independent organization, the level of planning the Yayasan management feels is necessary is their decision. However, if it has plans to seek outside funding from either other donors or financial institutions, a more detailed business plan modeled on a good project proposal outline or traditional business will be necessary. 1.10. Recommendations · Consider the inherent power differences in partnering with local institutions and formally or informally manage the project in such a way to share, as much as possible, power and control. This may require CARE to place a greater emphasis on working with partners to develop their capacity and CARE taking responsibility for partner’s weaknesses in front of the donor; · In projects designed to support an industry, base the project analysis on a strategic study of the industry that defines strengths and weaknesses of the industry and identifies key leverage points for the project’s interventions; · Clearly define the target groups for the project describing their economic status, support needed, and impact that support will have on developing the industry; · Base analysis of agricultural products at the farm level on farming systems not only on the crop or livestock itself; · Ensure all logframes provide a logical hierarchy where each level is a logical higher order result based on the achievements of lower levels; · Focus on establishing a team that clearly understands the goals and approaches of the project from the beginning and learns how to work together. The team building exercise in year two of this project was effective and helped develop the team’s ability to work as a team. · Ensure that monitoring and evaluation systems are “results based” and collect data not only on activities/outputs but also expected outcomes and impacts as described in the logframe; SAMBA Project Final Evaluation Page vii · Integrate gender into project planning, implementation, and monitoring. Provide gender training to all project staff. · Hold annual internal evaluation meetings to measure progress and results and to re-evaluate directions if necessary; · If similar projects are designed, integrate the non-financial and financial aspects of the project in order to develop synergy; · Integrate technical and management training into the longer term planning of BDSPs. If they do not have long term plans, assist them to develop them as part of the management support. · If project activities are to be sustainable after outside funding is comp leted, these activities must be designed from the beginning to be within the financial capabilities of the target group so that the target group can implement/maintain the activities without outside funding assistance. · If possible, design exit strategy into the initial planning of the project. For projects that plan to become Yayasans, provide training in business planning and develop exit integrated strategy plans that show both the Yayasan's plans for continuing the project work and business financial analysis showing how the Yayasan will fund itself and its work. SAMBA Project Final Evaluation Page 1 2. Introduction This report is the final external evaluation of USAID funded CARE-UNBAR Project Silk & Micro enterprise Development in Bandung Raya (SAMBA). The SAMBA Project office in Bandung, West Java implemented the SAMBA Project for 27 months1 from February 2002 through April 2003. Over this period, the final budget for the project was US$ 928,374. Of that amount, US$ 699,999 was provided by a grant from USAID and US$ 228,375 was provided by matches from the grant recipients2 . The ultimate objective of the SAMBA project was to improve the capacity of Business Development Service Providers (BDSPs) to support and develop Micro-Entrepreneurs capacity in silk production and marketing. The current funding was completed 30 April 2004 and the evaluation visit took place in mid April 2004. The evaluation was conducted by an external evaluator assisted by CARE. 2.1. Evaluation Objective The primary objectives of this evaluation is to assess the level of attainment of project results in relation to the SAMBA project objectives and to identify lessons learned from the project. This evaluation is underpinned by a capacity building ethos that sets out to build the knowledge and skills of both CARE-UNBAR project staff and CARE International Indonesia in order to increase their effectiveness in programming and implementation, particularly in the area the Small Economic Activity Development (SEAD). To this end, the evaluation will focus on strengths and weaknesses of the project and document key lessons learnt. 2.2. Evaluation Team The SAMBA final evaluation was conducted by one consultant, Frank Page, working under contract to CARE/Indonesia. Frank Page is an Organizational Development Consultant with 10 years experience working with both local and international NGOs in Indonesia providing consulting services in strategic planning, program planning, monitoring and evaluation, and developing participatory processes and skills. 2.3. Brief Background CARE Indonesia, in cooperation with the University of Bandung Raya (UNBAR) implemented this USAID grant through the Partnership for Economic Growth (PEG) Project from February 2002 through April 2004. The partnership was based on UNBAR’s long experience in the sericulture sector and CARE’s experience and expertise in micro enterprise development and project management. The purpose of the SAMBA project was to improve the capacity and effectiveness of Service Providers providing financial and non-financial services to silk-related small micro enterprises. The project planned to target 90 Service Providers in West Java (50 Business Development Service Providers and 40 Micro Finance Organizations). Through its interventions, the SAMBA Project sought to strengthen the technical, management and financial aspects of the Service Providers in order to improve their support and services to their micro-entrepreneurs. 1 Includes an approved three month no cost extension. 2 Summary Budget from the SAMBA Project proposal SAMBA Project Final Evaluation Page 2 The goal of the project was: In West Java, 2,000 silk related Micro Entrepreneurs achieve access to 90 improved Finance and Non Financial Service Providers through a competitive and market oriented approach. In order to achieve this goal, the project proposed to achieve two objectives: ß Objective 1: 50 Business Development Service Providers deliver improved services to increased number of sericulture Micro Entrepreneurs; and ß Objective 2: 40 Micro-Finance Organizations deliver improved financial services with an increased lending base to an increased number of Micro Enterprises served. Half way through the project, the second objective was modified with the knowledge and agreement of the donor USAID. With this modification objective 3 became: Strengthen the capacities of 5 to 10 MFIs to provide quality services with market oriented approach [sic] to SMEs with special focus on silk businesses in West Java3 These changes were made as it became clear that the original budget of approximately US$ 100,000 would provide little actual support for expanding lending and customers for 40 financial institutions. Instead, the project decided to increase the impact of using that funding by focusing on five institutions and managing the revolving fund program through CARE’s partner, PT BISMA. The project focused on intermediary Service Providers and through training and support worked to improve their effectiveness in management, business services and business linkages in order to enhance their capacity to better support silk related Micro Entrepreneurs. Key activities of the project included: ß Research and development for improved production and quality in the areas of moriculture, sericulture, and filature; ß Training of Trainers for BDSPs in sericulture, moriculture, and filature; ß Training of BDPs in management, financial analysis, and book keeping; ß Supporting trainings by BDSPs for micro entrepreneurs through a voucher based training program; ß The production of media that promoted the silk industry (VCD), provided technical information to silk producers (books and a calendar), and promoted networking and market development (address listing and website); ß Workshops and seminars to promote the silk industry, develop networks, and assist in marketing; and ß Through CARE’s partner organization PT. BISMA provide funding support for 5 micro finance institutions. 3. Methodology 3.1. Description of evaluation tools and methods The Scope of Work calls for the collection of data on Samba Project results; program development, activities, and management; stakeholder perceptions; program constituents; and the inclusion of 3 Samba Work Plan for the period of January 2003 - 2004 SAMBA Project Final Evaluation Page 3 gender into program development and implementation. In order to collect this data, the evaluation team adopted a number of approaches. These approaches included: · Review of project documents including quarterly reports, training reports, project plans, and program descriptions to determine project history, planning, implementation, and results; · Interviews with SAMBA staff in West Java to review project achievements and implementation; · Site visits which included interviews and focused discussions with BDSPs, MFIs, and MEs in order to verify: o Project results including achievements and management practices of the forums; and o Effectiveness of SAMBA activities (such as training and consulting visits), · A final participatory evaluation workshop with SAMBA Project Staff and invited members of the silk industry (BDSPs) review achievements, strengths and weakness of both CCFP program and forums and develop suggestions for future follow up; and The questions for the discussions and agenda for the workshop are included in the attachments to this report. 3.2. List of information sources Data was collected from project documents including the proposal, quarterly reports, event reports, logframes, and project plans, as well as interview with key stakeholders and constituents of the project. These informants included: · Care Staff; · BDSP and MFI managers including managers of weaving factories, silk cooperatives, farmers groups, Bank Shariyah and Bank Perkreditan Rakyat (BPR); and · MEs (farmers) receiving services from cooperatives and farmers groups. The complete list of informants interviewed and participants of the evaluation Workshops is presented in the attachments. 3.3. Evaluation schedule The evaluation took place between April 12 and April 30, 2004 April 12 – 13 Review documents and interview staff in Care’s Jakarta Office; April 14 – 15 Review documents and interview staff at the SAMBA office in Bandung; April 16 - 17 Interview BDSPs in Tasik/ area; April 19 – 20 Interview BDSPs in Sukabumi area; April 21 – 22 Interview staff at SAMBA office and prepare for workshop; April 23 Hold evaluation workshop; and April 26 – 30 Report drafting 4. Contextual Analysis When the project was being designed in late 2001 and implemented during 2002, a key aspect of the silk production chain in West Java was PT Indo Jado, a large silk spinning company. This company dominated the silk weaving segment of the industry, and provided what was essentially a monopsony (single buyer) to the cocoon producers of West Java. Moreover, at the time of project design and SAMBA Project Final Evaluation Page 4 initiation, the machinery of PT Indo Jado was greatly under-utilized; thus, PT Indo Jado essentially guaranteed markets for farmers, cooperatives, and other BDSPs that could improve the production of higher quality cocoons. By all reports, PT Indo Jado never managed to procure more than ten percent of the cocoons it spinning machine was able to process. By the third quarter of 2003, the company could no longer afford to carry this excess capacity4 and PT Indo Jado closed down. The closing of PT Indo Jado in September of 2003 was an external shock that had a great impact on both the SAMBA project and the silk industry. Almost over night, those selling silk cocoons lost their major buyer and the region’s capacity to produce cocoons became greater than the region’s capacity to process them into silk yarn. In addition, silk weavers lost a major local supplier of silk yarn and many of them then turned to importing yarn in order to continue their weaving business. The closing of PT Indo Jado resulted in a fundamental change in West Java’s silk industry changing a key silk industry constraint from a low local production of silk cocoons to a limited market for silk cocoons and low local production levels of silk yarn. 5. Findings 5.1. Project Design The original proposal for the project came from the University of Bandung Raya (UNBAR) which submitted a proposal to USAID after attending a USAID seminar. UNBAR’s original proposal was a project designed to increase the capacity and production cocoon farmers based on UNBAR research and development. USAID indicated its willingness to consider the proposal provided UNBAR refine the project to be more a business development project and partner with an USAID registered international NGO. For this project, USAID suggested partnering with CARE and UNBAR agreed. With the partnership set, the CARE/UNBAR partnership developed a final proposal that was acceptable to both partners and USAID. As is usual in all partnerships, each party had its own desires for the project. UNBAR wished for a project that would use its R&D to assist silk farmers. USAID wanted a project that supported business development in West Java. CARE wanted a project that would utilize and develop its experience in micro-credit. After a number of negotiation rounds, CARE/UNBAR and USAID came to an agreement on a final project proposal. The compromises made to include each of these goals are clear in the project proposal. UNBAR and CARE agreed to work through the BDSPs to reach the farmers and USAID agreed to accept the financial/credit portion of the project. 5.1.1. Proposal and Logframe The project proposal starts out with an analysis of the Silk Industry in West Java, attempts to describe Business Development Service Providers and Micro Finance Institutes in West Java, and then sets out the Log Frame, program description, management description, and monitoring and evaluation systems. 5.1.1.1. Project Analysis The background and preliminary assessment sections of the proposal describe the production chain of silk industry with the emphasis on the production of mulberry, cocoons, and silk yarn. The analysis, 4 PT Indo Jado ran one automated spinning machine with a capacity of 30 or 40 tons per month. Therefore, it could not reduce its capacity in order to bring its productive capacity in line with the area’s supply. Nor, for a number of reasons, could PT Indo Jado develop contracts to import cocoons from other regions of Indonesia or other countries in the region. SAMBA Project Final Evaluation Page 5 in a round about, way points out that at the time of project initiation, a key constraint to the silk industry was cocoon production. However, the proposal does not do a complete analysis of each aspect of the production chain, nor does it clearly point out opportunities and threats to the industry. In other words, the analysis did not take a holistic approach to analyzing the silk industry in West Java and designing interventions. Some questions that were not answered and have been raised during the course of the project include: ß What where the key strategic weaknesses in the silk industry in West Java? The analysis did described in detail the weaknesses in cocoon production and the potential for improvement; however it did not do this sort of in depth analysis for other segments of the production chain, including silk yarn producers, silk weavers, and silk clothing manufacturers. In hindsight, such an analysis that included cost/income estimates; potential for improving quality; potential markets (for supply, machines, and production); and detailed credit needs (such as amounts, interest, and loan terms) for of the silk industry may have lead to additional interventions and/or provided stronger justification for the key areas for project intervention. ß In terms of interventions for farmers, what are the returns of mulberry and silk worm production for the farmer compared to other options for using their land? During the project, CARE itself raised questions regarding the competitiveness of mulberry/silk worm farming. If silk farming alone can not compete with other options, what system of mulberry farming maximizes return to land and labor and can compete? This question was not addressed in the proposal but is an important question when considering interventions in rural farming systems. According to CARE’s resident agriculturalist, at this time, the farmers themselves appear to be doing their own experiments in intercropping and multiple land use in order to maximize the returns to their land and labor. ß Who/what were the BDSPs in the silk industry in West Java? After the descriptive analysis of the silk industry, the proposal attempts to describe the Business Development Service Providers (BDSPs) and Micro Financial Institutions (MFIs). Unfortunately, the section entitled Business Service Development Providers in actuality describes cocoon production systems and the potential increases in productivity in cocoon production. Despite a list of potential BDSPs in the annex, the proposal does not define the type of institutions that might be BDSPs. This was lack was recognized in the proposal and the project planned to implement activities to identify BDSPs, their strengths and weaknesses, and project interventions; however, as noted below, the lack of definition caused some confusion among project staff early in the project. ß What are the financial needs of the each segment of the silk industry? The micro finance analysis section of the proposal was a general analysis of the micro finance system in the area and did not relate the financing needs of the silk industry to the lending patterns of the financial industry. This general analysis of the financial industry can be explained by the fact that USAID regulations prohibit the use of USAID funds for targeted lending. Therefore the proposal writers did not want to intimate that funds for supporting MFIs were only for silk industry businesses. Nevertheless, the an analysis of the borrowing needs of the silk industry compared to the lending practices of the various types of institutions considered might have helped the project target its MFI assistance to financial institutions that would be in a position to assist the silk industry. In conclusion, the analysis in the project proposal focused heavily on issues and weaknesses of farmer production of silk worms and silk yarn. Similar analyses were not made of other businesses in the silk industry such as large scale yarn producers, weavers, and clothing makers. Thus the analysis was not a holistic review of the silk industry. Nor was the analysis of financial institutes connected with the needs of the silk industry. Although USAID regulations prohibit directed the use of AID funds for SAMBA Project Final Evaluation Page 6 industry specific loans, a more holistic analysis could have helped the project target financial institutes whose rules, regulations, and practices would permit them to support appropriate silk related businesses as well as other businesses. 5.1.1.2. Project Logframe The compromises made between UNBAR, CARE, and USAID to accommodate each organization’s goals was reflected in the logframe and project planning, which combined helping farmers, building BDSP capacity and supporting MFIs. In addition, the proposal presents two Logframes, one in the body and one in the appendices. In the outline below, the Performance Indicators are from the logframe in the proposal’s body5 and the Outputs are from the logframe in the appendices6 . The outline below arranges the outputs and performance indicators to show the logical progression and how the indicators measure the outputs. The projects log frame consisted of one goal supported by two objectives. The project’s goal was: In West Java, 2000 silk related Micro Entrepreneurs achieve access to 90 improved Financial and Non Financial Service Providers through a competitive and market oriented approach. Outputs: Participating SP [Service Providers] incomes increase by up to 20% by the end of the project. Performance Indicators: None To achieve this goal, the logframe proposed two objectives: Objective 1: 50 Business Development Service Providers deliver improved services to increased numbers of sericulture Micro Entrepreneurs; Outputs ß At least 50 BDSPs in various fields are able to expand their market and services to silk farmers Performance Indicators: ß Service providers’ customers (sericulture Micro Entrepreneurs) accomplish increased cocoon and handicraft productivity and quality; ß Project partner Service Providers attain an increased customer base and augmented income. Objective 2: 40 Micro Finance Institution deliver improved financial services with an increased lending base to an increased number of Micro Entrepreneurs served; Outputs ß 40 selected MFIs apply project standards in providing Micro Finance services to micro￾enterprises 5 SAMBA Project: Silk and Micro-enterprise Development in Bandung Raya, page 26 6 SAMBA Project: Silk and Micro-enterprise Development in Bandung Raya, Appendix 3: Logical Framework Analysis: Samba Project, page 43 SAMBA Project Final Evaluation Page 7 Performance Indicators: ß Micro Finance Organizations achieve an increase of (minimal) 20% in micro credit portfolio; ß Micro Finance Organizations attain an improved repayment rate during the project period. The project’s key strategy to achieve its goal was to assist intermediary BDSPs and MFIs to improve service to Micro Entrepreneurs (MEs) and hence increase MEs’ productivity and incomes. According to the proposal, the “selected service providers [BDSPs] will receive technical and management assistance to increase outreach and scale, greater cost effectiveness, improved cost recovery, and higher quality of administration and financial accountability. Participating Micro Finance partners are expected to increase there financial resource base and improve their lending management and accounting systems.” In order to implement its strategies, the project planned the following general activities: For non-financial institutions: ß Introduce and socialize the Project through workshops, seminars, and meetings ß Select BDSP partners and conduct baseline survey for Monitoring and Evaluation ß Demonstrate and promote appropriate technology in moriculture, sericulture, and filature to the BDSPs ß Train the BDSPs in administration, management, bookkeeping, production, marketing, quality control, and providing service to their clients ß Advocate related government sectors to remove hindering factors For financial institutions: ß Identify and select appropriate MFIs ß Introduce and promote basic banking practices and standards ß Increase access of MFIs to alternative financial resources include SAMBA project revolving fund ß Develop and apply institutional development framework and participatory institutional assessment tools ß Provide need based training, technical assistance, consulting services to partner MFIs to assist them to improve recording, accounting, MIS, and service delivery This particular logframe demonstrates some important weaknesses. First, it is not a logical hierarchy. The goal is a reiteration/combination of the two objectives. But a goal should be a higher order achievement that derives from the achievement of the objectives. Thus the goal as stated is not such a higher order achievement and therefore does not help project implementers understand the underlying purpose of the project nor the relationship between the two objectives. SAMBA Project Final Evaluation Page 8 Second, while the evaluation found Objective One to be specific, measurable, achievable, realistic, and time-bound, Objective Two as planned could not be considered achievable and realistic within the two year time frame of the project (Table 1). Steve Gilbert reported that in a project with a similar goal, that project had effectively helped approximately 35 financial institutions in four years. As this became apparent, along with the recognition that the project had not budgeted appropriately for 40 MFIs, the project readjusted their approach to working with MFIs as described above. Finally, the logframe does not provide any performance indicator for the goal, but does provide performance indicators for the objectives. Activities were written with associated outputs. The performance indicators are that are given are, at the best, the bare minimum indicators required for such a program. 5.2. Project Implementation To order the discussion on project implementation, this section will be structured along the lines of Marvin Wiesbord’s Six Box Model for diagnosing an organization with additional sections as required by the scope of work. The key components of the 6 Box Model are purpose, structure, helping mechanisms, rewards, relationships, and leadership. Key points of the projects implementation under each of these headings will be discussed. In addition, project history, staff skills, gender, and external relations will be covered. Often, given the change in project management in year two, the discussions will compare the first year with the second year. 5.2.1. History The SAMBA Project was started in February 2002 and completed in April 2004. The history of the project can be broken up into two distinct periods. The first year and a quarter represents project start up, socialization, and the struggle to develop its approach, identify BDSPs and MFIs, and develop the training programs it hoped to deliver. However, progress was not as fast as required in a short, two year project and in the second quarter of 2003, the Project Director was replaced. The new Project Director reorganized and restructured the project, most notably by adding 10 field staff. With the addition of the new staff stationed in the field, the quantity of project activities (trainings, workshops, consultations) increased greatly. This period is referred to as “year two” or the “second year” although it does include the work done up until the project closed down in April 2003. The consensus among those interviewed during the evaluation was that the project got of to a slow start and struggled through its first year. Accomplishments in first year (reported in Evaluation Workshop and gleaned from the quarterly reports) included: ß Socialization of the SAMBA project to the Silk Industry; ß Identification of partner BDSPs; ß Development of training modules and first 3 trainings in moriculture, sericulture, and filature; ß Socialization of MFI program and identification of MFIs in the project area; and Table 1: Evaluation Workshop Evaluation of Goals and Objectives Criteria Objective 1 Objective 2 Goal Specific Yes Yes Yes Measurable Yes Yes Yes Achievable Yes No Yes/? Realistic Yes No Partially Time bound Yes Yes Yes SAMBA Project Final Evaluation Page 9 ß The beginning of bringing the members of the West Java silk industry together in meetings and seminars. Although the project appears to have struggled during this period, the work done did lay the groundwork for great improvements in implementation during the second year. Year 2 was characterized by a change in management that focused on achieving the activities and objectives the project had committed to in the project proposal. Key changes included adding additional staff, particularly field staff to the project; providing them with the transportation necessary to cover West Java; and refining the planning and implementation of trainings, workshops and seminars conducted by the project. With the change in management the documented accomplishments increased dramatically when compared to the first year of the project. Thos accomplishments included: ß 4 TOT in moriculture, sericulture, and filature with BDSPs; ß 3 demonstration plots established; ß 6 voucher trainings in moriculture and sericulture for MEs; 6 voucher trainings in filature for MEs; 1 voucher training in design and batik for MEs; and 1 voucher training in weaving for ME’s; ß The production of media materials including the website www.silkybusiness.com; a VCD on the silk industry; a calendar showing proper management of mulberry and silk worms; a pictorial training module for use by BDSPs; and a directory of silk related businesses for West Java; ß 2 Management trainings for BDSPs; ß Arranging for two BDSP to lease silk spinning machines; ß Workshops and seminars that included a Silk BDSPs Business Gathering; 2 Seminars for MFI on the silk industry; a seminar for presenting research results and a workshop on BDSP development; and ß Routine visits to BDSPs and MFIs by Field Staff Two trends are clear from this history. The first trend was toward a greater numbers of activities implemented more efficiently during the second year. The second trend was that, particularly in response to the closing of PT Indo Jado, the project activities increased their breadth of assistance to the silk industry by promoting yarn production, disseminating both technical and market information on the silk industry, developing new markets, and promoting networking among the key actors of the West Java silk industry. 5.2.2. Purpose For a project team to come together, it is important that they share a common understanding of the project’s goal and how the strategy the project will use to achieve that goal. As noted above, the logframe does not provide the project with an explicit higher order goal. Therefore, the project staff developed their own working goal, which, according to a majority of the staff interviewed during the evaluation, was to increase the incomes and welfare of silk farmers; increasing farmer welfare was in essence the driving objective of the project. This objective clearly reflected two influences. First, UNBAR’s intention to help silk worm farmers embodied in its original proposal to USAID, and second, a strong feeling within the silk industry that silk farmers are under producing. However, all the staff interviewed clearly understood that the mechanism for achieving this goal was to develop the capacity of the BDSPs to provide silk farmers with markets and/or technical SAMBA Project Final Evaluation Page 10 training/extension. This strategy adopted by the project reflects the compromise with USAID in order to provide business development assistance in order to strengthen the overall silk industry. Developing this consensus on project process took some time. Interviews with the few staff that worked with the project from the beginning indicated that in the early period of the project the project’s purpose, goals, and methods were not clearly defined nor agreed upon by the staff. Issues regarding the purpose of the project mentioned in the evaluation included: ß Early on, the definition of Business Development Service Provider was unclear, and it took the project some time to define which BDSPs were the appropriate partners for the group; and ß Staff members remaining from the first year report that management and staff were not clear on what was wanted or how to do it and did not take a positive learning approach. This resulted in recriminations when things went wrong or were not up to management expectations, expectations that often had not been made clear. However, these issues were resolved, particularly by the new project director who concentrated on developing a common understanding of the project’s goal and process. This effort on the part of the new project manager contributed to the well developed team observed by the evaluator and the higher level of accomplishment in year two. As noted above, for most of the staff the focus of the project was to increase farmer welfare. But with the closing of PT Indo Jado and the resulting need to develop other segments of the West Java silk industry, the focus of the project has shifted to a degree to consider the whole silk industry, including yarn producers and weavers, while maintaining a strong concern for the farmer. 5.2.3. Structure A project’s structure influences implementation in a number of ways. First, it allocates the work of the project among the staff. Second, it determines a trade off between depth of expertise and flexibility to respond quickly to external changes. Finally, the structure influences cross-departmental coordination needs within the project. An effective structure assigns all the work of the organization to its members (at reasonable levels) ensuring no work “slips between the cracks” and has a level of flexibility consistent with the external changes faced by the project; no structure can avoid coordination issues which require helping mechanisms to manage. In the SAMBA project proposal, the description of the structure and staffing budget are not consistent. The staffing description and organizational chart show the staff made up of a Director and Deputy Director, Financial and Non-financial coordinators and specialists, field facilitators, and support staff. However, the budget description did not provide a budget for the field facilitators, and it appears that the first Project Director followed the budget when staffing the project. In addition, staff secunded from UNBAR to the project were assigned to the project on a part time, usually half time, basis. This was to fulfill UNBAR’s role as partner and also allow UNBAR staff to continue their work at the university. Although, it achieved this purpose, in the first year it also effectively reduced the staff available to the project. The organizational chart below recreates the staffing patterns during the first year of the project. SAMBA Project Final Evaluation Page 11 Figure 1: Project Structure - Year One Project Director Project Deputy Director (half time) Moriculture (2 full time) Financial Specialists (1 full time) Support Staff Finance Accounting Secretary Drivers Guards Office Boy Sericulture (1 half time) Filature (1 half time) R&D Assistants (2 full time) Research & Development (1 full time) This staffing pattern reflects the staffing budgeted in the proposal. The project did not directly hire field facilitators. In practice, it appears that for the first year the project hoped to train and develop a group of facilitators that were direct staff of BDSPS. These “imbedded” facilitators would then provide both financial and non-financial support services to their own and other BDSPs. However, the quarterly reports from the first year indicate that the project had difficulty developing such field facilitators and this approach was abandoned in year two. During the first year, it became clear that this staffing pattern could not effectively carry out all of the activities planned by the SAMBA project. This small group of staff could not cover the large project area and maintain the SAMBA presence in the field necessary to assist in the logistics of planning and implementing a large number of workshops, seminars, trainings, and follow up. The consensus among staff from the first year was that the program was strongly centered in Bandung with weak efforts to ensure a field presence in other areas of West Java. The net result was slow implementation. This situation was recognized and improved upon in year 2 with the reorganization of the project. The reorganization added 10 field supervisors, 2 financial specialists, and a media person. The UNBAR staff were then organized into a specialist group that would provide support for the field supervisors such as conducting trainings and providing advice. These approaches made more effective use of time UNBAR staff were available to the project. The final project structure is shown in the diagram below: SAMBA Project Final Evaluation Page 12 Figure 2: Project Structure – Year 2 Project Director Project Deputy Director Field and Admin. Coordinator Media Non Financial Specialists Moriculture Siriculture Filatur R&D Field Supervisors (10) Financial Specialists (2) Support Staff Finance Accounting Procurement Senior Secretary Secretary Drivers Guards Office Boy The addition of full time field staff who worked full time in different regions of West Java was an important reason for the project's ability to implement its programs and meet its output targets during the second year. As can be seen from the diagram above, project structure is organized by functions - media specialist, financial specialists, non-financial specialists, and field supervisors. Such structures tend to be inherently slow to respond to unexpected external changes. In part this is due to the inherent difficulty in coordinating between functions and in part it is due to the need to add functions to respond to change. However, the project was able to respond to the key external shock experienced during the life of the project – the closing of PT Indo Jado in the third quarter of 2003. When this company closed down and the market for cocoons dried up, the project responded by increasing its focus on filature training and adjusted the credit program to be able to lease reeling machines in order to rebuild the market for cocoons (13 BDSPs were offered this opportunity, however only 1 accepted the deal). The project was able to respond this way because filature was one of the areas of expertise incorporated in the project. However, this problem of lack of buyers for cocoons is not as easily addressed as the traditionally low production of cocoons, and will take the industry some time to work out7 . The one area that the functional structure did not support the achievement of the project’s goals was by separating the non-financial and financial aspects of the program. The financial specialists developed economic models for the silk industry and were sharing that information with financial institutions in West Java. However, the field supervisors did not have this information and therefore was not sharing it with the BDSPs. Conversely, the MFI staff did not have the intimate knowledge of the locations of silk and cocoon producers the field staff had. 7 Key issues for silk yarn spinning businesses include: a) investment capital and costs for manual or automatic spinners, b) quality control, especially for manual/non-automatic spinning machines, c) matching inflow of cocoons with potential yarn production rates, and d) basic spinning and business skills. SAMBA Project Final Evaluation Page 13 The net result of this division was that the project did not coordinate its efforts to promote credit for the silk industry. On one hand, financial institutes were learning about the industry through seminars and visits; on the other hand, the BDSPs were not learning about the banking industry in the same way. The interviews of bankers and BDSPs clearly indicate that there is a gap8 between what the BDSPs want and need for credit terms and what the banks are able or willing to offer. In such a case, it is possible that both sides might benefit from learning more about the other in order to increase the possibility of negotiating credit agreements acceptable to both sides. As it stands now, there is an appreciable difference. If this gap is to be bridged, both silk producers and banks have to have good understanding of each others business in order to build the necessary relationships and develop innovative approaches. 5.2.4. Helping Mechanisms Helping mechanisms are those mechanisms, both informal and formal, that help (or hinder) a project to get its work done. The formal helping mechanisms that will be briefly discussed below include the project’s planning, monitoring and evaluation system, its hiring and team building practices, and its financial/procurement systems. 5.2.4.1. Planning, Monitoring, Evaluation, and Reporting Planning, monitoring, evaluation and reporting ideally should be an integrated whole, with the monitoring system based on and feeding back into the planning system. A basic PM&E system was described in the proposal9 . This description included meeting schedules, monitoring/evaluation reports, and evaluation tools. The monitoring and evaluation plan was fairly complete, but not all of it was implemented. Key parts of the plan that were not implemented were the annual reports and annual assessments and evaluation of the capacity and improvement of BDSPS. In addition, the quality of planning and designs of monitoring and evaluation could have been improved. The SAMBA Project’s plans were based on the log frame described above, with one set of activities designed to improve the capacity of BDSPs and another set of activities designed to support micro￾finance institutions. The monitoring system that was initiated, based on quarterly reports and a final evaluation, was based on the outline from the logframe. In terms of planning, the timeline for establishing the office and staffing the project appears very ambitious, with only one month to staff the program and no planned orientation or team building activities. The evaluator recognizes the importance of starting quickly on relatively short term projects like the SAMBA project, but as the experience in the second year demonstrates, the time it takes for team building, orientation, and developing a consensus on goals and methods with the staff pays of in more efficient implementation later in the project. For the monitoring system, the reports and data available from the first year are relatively sparse and not very descriptive. The quarterly reports attempt to describe activities undertaken as laid out in the logframe and work plan. However, the reports are often not clear and are padded with largely unnecessary annexes. 8 BDSPs appear to require/desire larger loans with longer terms and longer grace periods than most banks (such as BPRs) are willing to offer. 9 SAMBA Project Proposal, pages 32-34 SAMBA Project Final Evaluation Page 14 The planning, monitoring, and evaluation were greatly improved during the second year. The improved monitoring system was based on the original logframe and work plans updated in January 2003 and again in October 2003 by the project management team. These plans were structured by the major activities being implemented by the project. These plans laid the basis for the monitoring and reporting system for year two. This system consisted of: ß Monthly meetings of all staff to cover progress during the past month, issues and solutions, and priorities and plans for the next month; ß Weekly meetings of the Field Staff to review progress, address issues, and plan for upcoming week; ß Ad-hoc meetings of appropriate people to resolve issues and overcome problems; ß Monthly Reports; ß Bi-Monthly Reports; ß Quarterly Reports describing progress of each of the major activities laid out in the project plans. ß A baseline and final BDSP and ME evaluation survey. The projects monitoring system in the second year focused on reporting and measuring project activities and outputs. This planning and monitoring system implemented by the new Project Director greatly improved the day to day management of the project and contributed to the substantial increase in completed activities the project achieved in year two. The weakness of the system is that, other than the final evaluation survey, it did not seek to collect information on higher order results, such as outcomes and impacts. However, this oversight is understandable given the short time period remaining in the project (less than a year) and the pressure to improve implementation when the project’s M&E systems were upgraded. Nevertheless, this meant that the performance indicators were not officially monitored during the life of the project and not reported on in the projects quarterly reports; the project’s M&E system was not designed to systematically collect and report information on improved BDSP performance and services, increased membership/clients of BDSPs, nor information on increased ME productivity and incomes. The project was certainly aware of its impact in an informal sense, and did act on that knowledge; however, such results were not reported or explicitly analyzed by the project’s M&E system, which was geared toward managing activities. The project did implement its plans to conduct a baseline and final evaluation survey. The final evaluation survey was being completed at the time of this evaluation and the evaluator has not yet seen the results. However, it appears that the survey was not designed directly around the performance indicators laid out in the log frame. According to the survey forms provided to the evaluator, the survey collected the following data: ß Impact on farmers’ practices in moriculture, sericulture, and filature; ß Changes in farmer income from mulberry and cocoons; ß Benefits farmers received from BDSPs; ß Samba activities BDSP participated in by BDSPs; and ß BSDP developments including, improved networks within industry, new business relationships and transactions, new knowledge and technology received, and benefits from such new knowledge and technology stemming from SAMBA activities. SAMBA Project Final Evaluation Page 15 As can be seen from the information being collected on these surveys, the data only directly measured performance indicator number one. The other three performance indicators were not directly measured by the surveys, as there were no direct questions asked about BDSPs client base or incomes. 5.2.4.2. Hiring and Team Building A key part of the restructuring of the project at the start of the second year was the hiring of new staff, particularly field facilitators and MFI specialists. This hiring process followed a couple of practices unusual in NGOs. The new Project Director used a hiring firm which applied psychological tests to prospective employees and, once hiring was completed, arranged for a jungle survival team building exercise for the new employees. The use of hiring firms and psychological testing is a relatively common practice among Indonesian businesses. The practice may or may not be useful depending on the tests given. If the tests have been demonstrated to be statistically reliable and valid (consistently and actually measure future employee performance), then such test could be very useful. However, before trusting such tests, it would be best to have their reliability and validity confirmed by an independent human resources expert. “Outward Bound” type team building exercise are also more common in business than with NGOs. The purpose of such activities is to help staff to learn about group development and to understand and work with each others idiosyncrasies. The key to such team building exercises is to bring the learnings back from the field into the office, and that apparently took place in this situation. Regarding the team building exercise, the staff interviewed (even longer term staff that did not attend the team building) unanimously reported that the experience was extremely useful and helped them develop the capacity to understand each other and to work effectively together. 5.2.4.3. Finance and Procurement Systems The financial and procurement systems were based on CARE/Indonesia standards. In general they were effective. The staff came to understand and follow the systems, which were managed smoothly. The evaluation turned up only two comments on the financial systems. The first was that the documentation for purchases of around Rp. 500,000 (US$ 60) that was required was often more advanced than many shops were able to provide, particularly the many shops that still use manual accounting systems. This was not a great impediment, but tended to limit purchasing to shops that were more expensive and/or less convenient to deal with. In addition, the disbursement time from CARE/Jakarta for expenses over Rp. 10,000,000 was too long to support the some of the activities implemented by the project. This was particularly the case for the voucher training programs. In these programs, the BDSPs would pay the costs of trainings (Rp. 100,000 per participant) and receive reimbursement from CARE after the trainings. Often, these BDSPs could not carry that expense for long resulting in friction between the project and BDSPs if voucher reimbursement took too much time. To be sure to avoid this issue, the SAMBA project management would occasionally scale back programs so that the costs would be under the Rp. 10,000,000 limit in order to ensure that SAMBA could reimburse the vouchers without undue delay. It is important to note that neither of these issues had a great impact on the program. They are issues only in regard to considerations on further improving what was for the SAMBA project a smooth, well managed (from an implementation perspective) financial system. 5.2.5. Rewards and Motivation Rewards and motivators can be broken into two categories. The first is Hygiene Factors, which influence staff morale (such as wages, policies, working conditions). The second is Motivating SAMBA Project Final Evaluation Page 16 Factors, which influence motivation to work harder and succeed (such as responsibility, achievement, and recognition). Within the SAMBA project, the partnership with UNBAR combined with rules on salary levels created wide disparities in salary levels between CARE staff and UNBAR Staff. CARE salary levels were often five to ten times higher than equivalent UNBAR salaries levels. These disparities were known in the project and, despite that UNBAR proposed the salaries for its staff, resulted in some feelings of being undervalued on the part of the UNBA staff10 . In addition, the structure of the partnership made true partnering difficult. Although it was agreed that Care would be responsible for overall management of the project and this fact combined with the Care staff being full time the UNBAR staff being half time (including the Deputy Director) lead to feelings that the collaboration was not a true partnership, but that UNBAR worked for Care. Given the partnership structure, this should not be a surprising outcome as true partnering means shared responsibility, and officially this was not an option under USAID requirements. Given this official disparity in responsibility, it was difficult to “unofficially” practice true partnership. These issues probably contributed somewhat to the slow development of the project during the first year. However, in the second year, after the project restructuring and with a limited time left in the project, the two partners appeared to have grown more accepting of their roles worked together well to complete the project. The main indicator of success of how the reward system was managed during the second year of the project was the fact that staff were still motivated and working hard to complete their work and meet their targets up to the very last days of the project. 5.2.6. Internal Relationships There are three main types of internal relationships that need to be managed – 1) between people; 2) between units; and 3) between people and technology. Problems in these relationships generally manifest themselves as conflict and indicate that the project team has difficulty in managing differences. It is difficult to diagnose internal relationship issues that have occurred in the past among people who are no longer present; therefore, this evaluation did not attempt to evaluate the project’s internal relations during its first year. However, during the second year, it was clear from the evaluation interviews that the current Director and his team had managed the internal relationships well. All staff interviewed indicated that they worked well with their colleagues both within their unit and outside their unit, and there were no indications of conflict. 5.2.7. Leadership Under this model, leadership is measured by the extent the leader or leadership team monitors and keeps each of these the categories “in balance”. During the first year, it appears that the leadership had difficulty identifying which categories needed improvement and then making those improvements. From the evaluation as well as the corrections made during the second year of the project, it is clear that during the first year two key issues were: ß A lack of a clear and shared understanding of the goals and methods of the project; and ß Structural issues affecting the project’s ability to maintain an effective field presence; 10 Interview with Pak Wibowo SAMBA Project Final Evaluation Page 17 The inability of the project management at that time to correct these key issues appears to have contributed to the slow pace of project implementation during the first year. At the end of the first year, the CARE/Jakarta management stepped in and took the decision to replace the project director and restructure the project in order to ensure that the project would complete its activities and achieve its outputs. This is not an excessive period of time for making such a decision as it is natural to seriously try and solve issues with current management before taking the drastic step of replacing them. The move by CARE/Jakarta was successful. The new project director was able to bring the categories described above back into balance by creating a common agreement and understanding of the project’s goals and methods, restructuring and re-staffing, and using effective working and monitoring systems to ensure that the work moved ahead quickly. 5.2.8. Skills The skill make up the SAMBA project staff focused on technical skills in moriculture, sericulture, filature, and finance. The non-financial specialists had skills in silk worm production and yarn spinning as well as teaching. The fie ld supervisors had backgrounds in agriculture. The financial specialists had backgrounds in economics. These skills were put into place to in order to improve BDSP’s ability to support farmers and therefore increase farmers’ welfare. This skill set reflected the working goal of the project. However, as the focus of the project broadened to provide more non-technical support to the industry, the project has begun to feel some limits on the skills available within the project. At the final evaluation the non-specialist staff indicated, either directly or indirectly, the desire to improve skills and knowledge in training (TOT) and general management (structure, planning, monitoring and reporting) and organizational development (how to help associations become effective). 5.2.9. Gender The SAMBA project did not implement any efforts to take into account gender in its planning, implementation and monitoring. Internally, the office staffing reflected traditional gender roles with women in administrative and financial positions and men in technical and field positions. 5.2.10. External Relationships The SAMBA Project’s key external relationships were with its partner and the groups the project was providing services too, the project’s target group/customers. These included the BDSPs, MFIs, and indirectly, SMEs or farmers. 5.2.10.1. UNBAR The relationship with UNBAR was colored by the partnership structure and salary issues described above. However, by the end of the project, the two sides had come to a common understanding of their roles and were able to work together effectively. On the whole, the UNBAR representatives interviewed during the evaluation appeared a bit disappointed with how the partnership turned out. Aside from the issues described above, the University did not feel that the project had supported their research and development efforts as much as they had hoped it would. However, it is important to note, that this was not an explicit goal of the project or partnership. Upon reflection, it does not seem that partnering with an international NGO to implement a short term project was the right tool for developing the universities research and development. SAMBA Project Final Evaluation Page 18 5.2.10.2. BDSPs The project worked with a variety of BDSPs made up of: · Farmers groups; · Self supported cooperatives; · Business supported cooperatives; · Silk spinning and weaving businesses; · Batik and clothing makers. Some of these BDSPs were not well funded, particularly the farmers groups and some cooperatives, and others were quite wealthy, such as some of the cloth producers. The cooperatives tended to be cooperatives only in that they directly serviced and bought from farmers and were established as cooperatives under Indonesian regulations. None of the cooperatives visited during the evaluation were supervised or funded by their farmer members, and none had plans or systems to distribute profits back to their members. The SAMBA Project’s relationship with BDSP’s appears excellent. The unanimous recommendation from all the BDSP’s visited during the evaluation and those that attended the final workshops were for the SAMBA project to continue operation. As will be described below, the BDSPs acknowledge the assistance of the project in improving cocoon production, helping with management during difficult times, and providing opportunities to expand markets and network both within West Java and outside of the region. It was clear from the reactions of BDSPs that the project has developed a high level of credibility with the BDSP’s and is in a good position to continue working with them. 5.2.10.3. MFIs The MFIs that participated in SAMBA activities included BPRs, LPKs, BMTs, Cooperatives, and general banks. As with the BDSPs, the project’s relationship with MFIs appears good. Respondents from MFIs recognized the potential benefits of the SAMBA project, and appreciated the information they received about the silk industry from SAMBA. However, the MFIs met during the evaluation did not see a real fit between their business and the businesses SAMBA was promoting. 5.2.10.4. Farmers and others The farmers that were members of the farmers groups and cooperatives that received support from the project tended to cultivate mulberry and silk worms as a side activity from their main activities as rice or vegetable farmers. Almost all of these farmers had land for mulberry. The farmer’s group and cooperatives interviewed during the evaluation reported that on average the farmers they serviced had a half hectare of land. Each respondent classified their members as lower middle class; none of the respondents classified their members as poor. Mulberry and silk worm farmers were not a primary target group for the SAMBA project, and this was born out in the interviews. The farmers interviewed during the evaluation are generally are aware of SAMBA’s existence. They had no direct contact with SAMBA and were not clear on what type of organization SAMBA was nor did they understand how SAMBA operated. A number of the farmers interviewed in the evaluation thought SAMBA was a company (PT) much like PT Indo Jado. SAMBA Project Final Evaluation Page 19 5.3. Results/ Achievements In its work with the BDSPs and Silk Industry (objective 1), the SAMBA project was able to effectively implement the activities and achieve the outputs planned in the project paper. Due to the slow start in year one, much of the work was accomplished in year two and with the help of a short (two month) extension; nevertheless, with this help the project was met its work plan for the two years. As a result of the project’s activities and outputs, those interviewed during the evaluation indicated the following trends: · BDSPs are able to provide better technical advice/training to farmers; · BDSPs have somewhat improved abilities in financial analysis and administration; and · Silk industry business and marketing networks have developed. As a result of these outputs, farmers groups and cooperatives reported that farmers have adopted improved moriculture and sericulture practices and some have benefited financially from the improvements. However, the changes in the silk industry in West Java, including the loss of markets due to the closing of Indo Jado and recent experiences with poor quality worm eggs, have reduced the expected impacts of these outcomes. A number of farmers have halted silk worm production and are waiting for either the market conditions for cocoons to improve and/or waiting for the quality of silk worm stock to return to normal11 . These changes have also affected the development of BDSPs that market cocoons. Those BDSPs that marketed cocoons to PT Indo Jado have lost a major market. Thus, two cooperatives visited reported to be reluctant to expand their farmer base as they are unable to market or process additional cocoons. When these two issues are resolved, the impact of the project should increase as farmers restart silk worm rearing using improved techniques. The SAMBA project has helped some BDSPs develop skills and ability in silk spinning and two BDSPs were able to lease spinning machines from the project. However, the installed base of spinning machinery in West Java is not enough to replace what was lost when Indo Jado closed down. Making up for this will require more than the SAMBA project was designed to offer in both access to capital and time to develop. In its work with Micro-finance institutes (objective 2), the project found that the original objective and indicators of success were unachievable, and therefore changed its objective and approach by reducing the number of MFI receiving funding and expanding working with MFI to increase their understanding of and interest in the silk industry. To achieve this objective, the project: · Provided financial support totaling US$ 124,977 to five Bank Perkreditan Rakyat (BPR) through CARE’s partner BISMA; · Held seminars and workshops to introduce MFIs to the Silk Industry and train them in evaluating silk related businesses; and · Promoted market linkages for the silk industry The financial support for the five MFIs has helped each institution to increase their lending portfolio to MEs in their areas. Most of these MEs are in micro-trading and micro-services. 11 The end of project survey conducted by the SAMBA project found that 29% of the farmers surveyed had stopped mulberry and/or silk production due to changes in market conditions. SAMBA Project Final Evaluation Page 20 The promotional activities conducted by SAMBA have resulted in MFIs having a greater understanding of the silk industry and in cases where MFIs were already lending in the agricultural sector an increase in interest in lending to the silk industry. The project reports that five MFIs have provided financing to silk related businesses. However, it must be said that from the BDSPs perspective, this effort has not generally created suitable opportunities for financing. 5.3.1. Achievements Relating to Objective 1 For the final year of the program, the project was guided by a work plan that laid out five major activities. Table two shows the outputs for each of these activities12 : Table 2: SAMBA Project Outputs by Activity for Objective 1 Activity Outputs Conduct training of trainers for BDSPs in silk industry techniques 154 participants from 65 BDSPs received TOT trainings in Moriculture, Sericulture, and Filature Training modules on Moriculture, Sericulture, and Filature which were given benchmark status for by the Ministry of Forestry for evaluating trainings. The following media materials were produced and distributed: · VCD on the silk industry; · Calendar showing proper management of mulberry and silk worms; · A pictorial training module for use by BDSPs; and Provide Management Training for BDSPs Participants from 64 BDSPs attended management trainings Market Survey and Network Study A market survey was conducted and presented to BDSPs at a seminar A directory of silk related businesses for West Java The website www.silkybusiness.com has been established to provide market information, contact information, and sales venue for the silk industry. Research and Development 5 Research papers were prepared by UNBAR on topics including: · Silk worm and clothe production in West Java; · Mulberry disease and treatment; · Silkworm disease and treatment; · Feed supplement/fertilizing mulberry; and · Silkworm feeding methods 3 Demonstration Plots established Voucher training program 498 participants trained in voucher training programs (229 in moriculture and sericulture; 232 in filature; 21 in design and batik; and 16 in weaving) General/Other 94 BDSPs participating in at least one SAMBA activity 2 BDSP leased reeling machinery from BISMA 12 These results are compilation of data from SAMBA Quarterly reports through December 2003. The final quarterly report for Jan-Mar 2004 had was not yet available at the time of the evaluation. SAMBA Project Final Evaluation Page 21 The project was able to directly train 295 BDSP members in training of trainers and/or in management. After the training of trainers to BDSPs, the project support 6 BDSPs train 410 silk farmers in improved techniques in moriculture, sericulture, and filature through the voucher training program. Respondents that had experienced the trainings unanimously rated the trainings useful and effective. In all cases the respondents were able to describe new techniques adopted after the training, such as improved feeding schedules, use of natural fertilizers, use of milk for feeding mulberries, etc. Some of these new techniques (such as use of natural disinfectants and feeding supplements for mulberry fertilizer) were proven through the research supported by the project. The evaluation found that there were strong indications that farmers had gained useful/practical practices from the trainings and had or were ready to adopt them. These respondents also indicated there was an increase in farmer interest in silk worms due to the trainings. To support the trainings, the SAMBA project developed a number of publications and a CD. The publications included Buku Pintar Sutera Alam (The Smart Book on Silk), Modul Peltatihan Bergambar Budidaya Sutera Alam (Picture Training Modules for Managing Silk), and a calendar showing best practices in mulberry and silk worm production. The project also produced two CDs, Merajut Sutera Impian (Knitting the Silk Dream) and video recording of a workshop/training on developing silk business in West Java held in October, 2002. The CD, Merajut Sutera Impian, is a good introduction to the silk industry from mulberry production to batik. The training materials are complete and easy to understand. Both BDSPs and farmers reported finding them useful during and after the trainings. However, due to two changes in the silk industry outside the control of the project (the closing of PT Indo Jado and decrease in quality of worm eggs), incomes and productivity for BDSPs and farmers may or may not have increased. These two changes in the silk industry appear to have offset, or delayed potential positive impacts on both production and incomes of BDSPs and SMEs (silk cocoon farmers) and increases in BDSPs’ membership. Therefore, despite the improvements in technical training and extension from the BDSPs, the measurements of the performance indicators of increased farmer and BDSPs incomes and clients serviced may or may not actually measure improved services from the BDSPs. For farmers that still had a market for cocoons and access to good starter crop, the trainings have helped them increase quality and/or quantity, and thus income from their cocoons. But many farmers are waiting for improvements in either cocoon demand and/or worm egg quality before spending resources to increase production, and some cooperatives are waiting for the same improvements before they focus on expanding their membership base. The projects impact on improving production capacity and quality of yarn up to this point appears to be less than for cocoon production. The evaluator visited 3 cooperatives that were spinning yarn, two of which had received direct assistance from the project. Each of these reported some improvement due to the project. The trainings supported by the project have helped them improve output and quality. At one cooperative, the management has taken the initiative to innovate and further adopt a UNBAR design for electric spinning machines in order to decrease purchase, operating, and maintenance costs of the machine. Although the results of the trainings in this area appear to have been positive, low skills in yarn spinning is only one of the blocks to increasing West Java’s productivity in producing yarn. The other key impediment is related to the cost of purchasing silk spinning machines and the capital and cash flows needed to purchase cocoons. To counteract these impediments, the project also promoted an offer to rent spinning machines to thirteen BDSPs. Of these, five were deemed eligible and only one decided to accept the offer13. The others were interested but lacked confidence in the market and their ability to repay the rent. Therefore, the achievements made by the trainings are still first steps in addressing a fairly large and complex issue. 13 Interview with Project Director SAMBA Project Final Evaluation Page 22 A consistent theme when meeting with BDSP’s was the need for capital. This was less so for the more professionally run businesses such as PT Prima that have adopted a strategy of self re￾investment to grow. The more farmer based BDSPs tended to appear to be more concerned about getting outside support. These groups also did not have any formal long term growth plans. Therefore, the project could have supported its efforts to promote reeling and spinning by helping these organizations develop long term plans and understand the strategy of growth by re-investment. However, it must be noted that before the closing of Indo Jado, improving farm or coop level silk spinning was not a priority of the silk industry; in fact, it was a direction that PT Indo Jado actively opposed14. Thus the project had a very short time to improve an area that needs not only technical support, but business and financial support in order to establish the machinery and business. The table three relates the results observed in the evaluation to Objective 1 and its performance indicators as laid out in the project proposal. Table 3: Results Observed for Objective 1 Objectives Results Objective 1: 50 Business Development Service Providers deliver improved services to increased numbers of sericulture Micro Entrepreneurs; Output: At least 50 BDSPs in various fields are able to expand their market and services to silk farmers 7 BDSPs developed the capacity to arrange and provide voucher training programs ß 6 BDSPs provided voucher training in moriculture and sericulture ß 6 BDSPs provided voucher training in filature ß 1 BDSPs provided voucher training in design and batik 65 BDSPs received trainings to increase their capacity to provide technical information to mulberry and cocoon farmers by increasing their capacity to train farmers and incorporating the results of research into their training programs Performance Indicator 1: Service providers’ customers (sericulture Micro Entrepreneurs) accomplish increased cocoon and handicraft productivity and quality; The cooperatives and farmers visited during this evaluation indicated that the technical trainings were useful and that farmers have adopted techniques taught in the trainings. Examples include the use of bore fertilization for mulberry plants and improved feeding practices for silk worms. Respondents interviewed during the evaluation indicated that production, quality, and incomes from cocoons could have increased due to the trainings (TOT and Vouchers) provided good worm eggs were available and there were assured markets for cocoons. However, this evaluation was not detailed enough to measure the total impact of the project. Performance Indicator 2: Project partner Service Providers attain an increased customer base and augmented income. Some BDSPs visited were not trying to increase customer base due to the difficulty in selling/processing cocoons; Interest in moriculture, sericulture, and filature have increased among farmers; BDSPs report that the management trainings were useful and that they have been able to improve their financial analysis, networks/markets, and to some extent other management systems; Other One cooperative has, on its own, designed and developed a reeling machine that a) uses less power and locally available parts and b) less expensive to build than previous models. This cooperative is also developing a single reeling system that allow for single or double reeling depending on the temperature conditions (if the temperature is high enough, double reeling is not necessary) Although the performance indicators were influenced by other changes in the silk industry and may not show improvement, the evaluator feels that the project was able to meet its objective improving the technical services provided by 50 BDSPs to silk farmers. The trainings and training materials and 14 Discussion with the Project Director SAMBA Project Final Evaluation Page 23 process were consistently reported to be effective in providing farmers with improved techniques which increased quality and or quantity of cocoons produced and which were adopted. 5.3.2. Achievements Relating to Objective 2 For the reasons described above in the planning section, the second objective was found not to be feasible given the time and resources of the project. Therefore, with the agreement of USAID, this objective was changed to Strengthen the Capacity of 5- 10 MFIs to provide quality services with market oriented approach to SMEs with special focus on silk business in West Java. With the change of this objective, five major activities were adopted. These five activities are listed in Table 4 along with the corresponding outputs as reported in the quarterly reports up to December 2003. Table 4: SAMBA Project Outputs by Activity for Objective 2 Activity Outputs Provide revolving loan funds to 5 to 10 selected MFIs in the target region for re￾lending to SMEs Provided loan support to 5 financial institutions (Bank Perkreditan Rakyat) to provide credit (loans under Rp. 5,000,000) to micro-entrepreneurs. Promote the silk industry to financial institutions The development of a Silk Business Analysis Tool 66 MFI (32 BPR, 15 LPK, 8 BMT, 6 Cooperatives, 3 Public Banks, and 2 other LKM) attending at least one SAMBA workshop/seminar on financing the silk sector including: · 2 Introduction to the silk industry seminars · Seminar on best practices in micro credit and silk industry development · 3 regional workshops for MFI on providing credit to the silk industry · Moriculture/Sericulture trainings (a few MFI representatives did attend these trainings) 37 MFI (26 BPR, 5 LPK, 8 BMT, and 2 Cooperatives) receiving consultation from SAMBA MFI staff Promote market linkages for the silk industry A Silk Industry Business Gathering attended by 49 BDSPs and 9 Govt. agencies held by SAMBA, PARASILK, and the Forestry Department Attendance at the National Silk Congress A second Business Gathering to present the results of the marketing survey and distribute the directory of Silk Industry Businesses Develop an MOU with markets in Medan Overall, the program provided five MFIs with funding designed to increase their lending to micro entrepreneurs. Criteria for choosing MFIs in this program included past record and interest in lending to MEs and both BISMA and government measures of bank health. The loans, which were provided as Time Deposits with a periodic draw down over a year (with the possibility of new funds at a later date), were able to increase the MFIs lending to MEs, particularly micro-traders and micro-services. Progress reports from three BPRs show that the BPRs' loan portfolios and number of borrowers increased. The SAMBA project’s efforts to promote the silk industry appear to have succeeded in informing MFIs about the silk industry and raising interest in the industry. All respondents from the three MFIs visited during the evaluation reported that they had learned about the silk industry from SAMBA SAMBA Project Final Evaluation Page 24 programs. Two of them had also become interested in silk. One of these three then implemented a survey of silk farmers in his area, indicating interest although he found there were no potential customers. Another, noting that his Bank Shariyah limited its lending to a maximum Rp. 1,000,000 and therefore was not in a position to finance silk businesses, became interested enough to investigate going into mulberry and worm production as a personal activity. However, despite this positive outcome, the impact was not great. By the end of the project, after working reaching out to 66 MFIs, the project reported that 5 had actually provided financing to silk related businesses. This low response rate may have been due to the divergent needs of the silk industry and the type of lending MFIs are comfortable with. The list bellows describes some observations from my evaluation visits: ß Given the Plasma-Inti system between cocoon buyer and producer, there is very little opportunity for financial institutions to provide cocoon producers with operating capital (usually around Rp. 250,000 per box of cocoon seeds). Although, this type of lending would be consistent with the common practices and requirements (small loans quickly repaid) of many MFIs. ß Farmers often need a higher level of investment to develop land for mulberry trees and building the housing for cocoon production. An estimated figure was around 20 – 25 million to develop one hectare. Moreover, cocoon producers would not start earning from this investment for at least 10 to 11 months resulting in a desire for long grace periods. ß Investments for silk yarn production and weaving start at around Rp. 10 million for hand operated machines. For example, the machinery rented out by BISMA was worth Rp. 40 million. In addition, yarn producers report that they would desire grace periods and repayment schedules that correspond with the cash flow of the weaving business. Aside from the running production costs covered by the plasma-inti system, the amounts needed by the silk industry are both greater and for a longer term than what are normally considered micro￾finance. For example, Moslem Banks do not lend this amount, often maintaining maximum loans of Rp. 1,000,000 repayable within 3 to 4 months. Bank Perkreditan Rakyats' average loan size is around 2 – 3 million, although they do give out loans up to 20 – 30 million, but do not provide grace periods and require monthly repayment on all loans. Such an analysis in the planning could have helped the project target banks that would provide financial services that meet the needs of the silk industry as well as other industry and avoid working with banks that could not or would not provide services to the silk industry. The project was greatly more successful in building networks and linkages among the Silk Industry in West Java. All respondents reported that the linkages that had made within West Java and outside the province due to participating in all types of SAMBA activities was extremely positive. These outcomes came about because of the support/trainings the project provided members of the silk industry in West Java and from the workshops and seminars the project held to support the silk industry. This work resulted in developing15 directly or indirectly: · Silk industry associations including PARASIL, the West Java Silk Association established by the silk industry from ideas generated informally through industry members meeting at SAMBA project activities · Informal networks among Silk Producers; · Increased sharing of market information and connections to new markets including Bali and Medan 15 Results of Identifying Achievements session by SAMBA Staff and Industry Participants at the final evaluation workshop SAMBA Project Final Evaluation Page 25 · Increased motivation within the industry and the reputation of the West Java Silk Industry · Increased the interest by government agencies in supporting the silk industry The value the BDSPs placed on these connections is reflected in the unanimous response requesting the continuation of the SAMBA project. It was clear from the seminar observed and workshop held by the evaluator, that the industry credited SAMBA with helping it develop internal and external marketing and support networks, increasing reputation and motivation of the industry, and providing valuable technical/training services. The table below relates the results observed in the evaluation to Objective 2 and its performance indicators as laid out in the project proposal. Table 5: Results Observed for Objective 2 Objective 2: 40 Micro Finance Institution deliver improved financial services with an increased lending base to an increased number of Micro Entrepreneurs served; The target of reaching 40 MFIs was met by having 66 MFIs involved in SAMBA’s activities, including a) the promotion of silk industry to financial institutions, and b) dissemination of silk industry feasibility analysis to MFIs16 Performance Indicators Micro Finance Organizations achieve an increase of (minimal) 20% in micro credit portfolio 5 banks receiving revolving loan funding support from PT Bisma supporting an increase in lending to micro entrepreneurs mostly consisting of micro traders and micro services. Progress reports from three of five banks (2 progress reports were not yet due at the time of the evaluation) show that total loan portfolios increased an average of 51% and number of borrowers increased an average of 30%. Performance Indicators Micro Finance Organizations attain an improved repayment rate during the project period Repayment records during 2003 were good. Repayment rates remained high in three banks reporting, with lowest being 92.76% and the highest 98.99%. Informal Objective: Local MFI are motivated to provide financing to silk industry businesses The financial institutions interviewed during the evaluation indicated that they have gained a greater understanding of the silk industry; The SAMBA activities appear to have increased the more agriculturally oriented banks interest in the silk industry. One banker interviewed was motivated to conduct a survey of silk farmers in his area (Sukabumi). However, he found that there were very few silk farmers active in the area. Project reports indicate that 5 MFIs extended loans to silk businesses. Other Indicators Formal and informal networks among silk businesses at all levels have been developed creating new marketing opportunities. For example, the connection made between PT Prima and Kumas Batik that lead to Kumas purchasing silk cloth from PT Prima; Improved motivation, reputation, and information flows within the West Java silk industry. Officially, the project met Objective 2 after correcting for the overly ambitious plans set out in the original project proposal. The project did provide funds for five MFIs to increase their lending to MEs and the project’s promotion of the Silk Industry to MFIs succeeded in increasing awareness and to some degree interest, but was not successful in generating finance for the silk industry, particularly 16 This conclusion is based on minutes of a meeting held on January 21, 2004 between Mr. Firman Aji of USAID and Mr. Sihol Aritonang of CARE in which both parties agreed that this target would be met “by having at least 40 MFIs involved in SAMBA activities...” SAMBA Project Final Evaluation Page 26 to yarn and cloth producers. On the other, hand the project appears to have been extremely successful in promoting linkages within the industry and in re-motivating the industry. 5.3.3. Conclusion Overall, the project met the goals and objectives it set for itself (after the midterm adjustments for the second objective for MFIs). Given the slow start in the first year of the project, praise should be given to the project management and staff for the second year for their efforts to implement and complete the activities laid out in the work plans. With this effort, the project was able to implement its activities and achieve the outputs planned. Thus they were able to achieve their objectives, farmers’ skills and knowledge increased, production increased in places, BDSPs ability to provide technical support increased, MFIs gained knowledge and interest in the silk industry, and the industry strengthened its own networks and reached out to explore new markets. The down side to the project was that changes in the structure of silk industry near the end of the project combined with the need to revise the approach to the financial institutes during the implementation of the project reduced the impacts the project hoped to achieve on farmer incomes, BSDP cash flows and memberships, and financing accessed by the silk industry in West Java. The project’s first objective to increase the technical services offered by BDSPs was designed before the closing to PT Indo Jado, and required the market provided by that company in order to achieve maximum impact. With the closing of PT Indo Jado, the positive impacts of the technical training at the BDSP and farm level were reduced until the market for cocoons increases again. The closing of PT Indo Jado in the latter stages of the project also presented the project with a large challenge in terms of supporting the Silk Industry and a very short time to address that challenge. The project made appropriate adjustments and attempted innovative approaches (renting reeling machines, for example), but was not, understandably, able to have a large immediate impact. The project’s second objective was problematic from the design phase, setting an ambitious goal combined with under-budgeting for that goal. The adjustments made were able to make the effective use of the money budgeted by providing enough supplemental funding to five BPRs to have a significant increase on their lending to MEs. The adjustment to promote the silk industry succeeded in part, but was not successful in generating financing from MFIs for the silk Industry. Here again, the short period (year two of the project) of implementing this program was not enough to allow the project team to learn from the experience and make adjustments that would increase the project’s ability to bring financers and silk businesses together. Finally, the project’s efforts to develop networks and develop markets, combined with its training services have helped generate interest and mutual support within the West Java Silk Industry. This interest and support of the silk industry will provide the key basis for the continuation of the project after funding is complete. For, over the past two years, the project has supported the industry and earned its trust, and is now in the position, if it can sustain itself financially, to develop and expand the support it has given the industry during the past two years. 5.4. Sustainability The key to the sustainability of the project’s activities rests in the skill and knowledge gained and put to use by the BDSPs and farmers. The large training programs arranged and supported by SAMBA do not appear to be replicable without outside funding. The SAMBA project was financing trainings at approximately Rp. 100,000 per trainee through its voucher system. It is unlikely that either the BDSPs or farmers themselves can afford such costs; therefore, large scale trainings such as those held under the project will not be repeated by BDSPs. However, the technical skills gained by the BDSPs will be maintained in their extension programs and the BDSPs reported that they would like to continue local training they do with their farmers, although in a scaled down manner. SAMBA Project Final Evaluation Page 27 The level of support to BDSPs and the silk industry provided by the SAMBA project is cle arly not sustainable, nor was it designed to be sustainable. Without the outside funding, or funding from the industry itself, the level of trainings, extension, consultations, workshops, and seminars implemented by the project can not be maintained. The key to the sustainability of the project’s networking and marketing activities lies with the industry members themselves. Such activities such as business meeting seminars, marketing seminars, etc. can be arranged by the industry itself. There are BDSPs that as a group can easily donate the funds for such activities if they were willing and organized. However, it is not clear at this point if the industry recognizes this and/or if there will emerge a leader to organize such activities. Essentially, the skills, knowledge, and attitudes developed by those who participated in SAMBA activities will remain after the project. The activities implemented and funded by the project will cease with the closing down of the project. The new Yayasan and industry may come together to continue some of those activities, but this is not a sure thing and depends on the new Yayasan’s ability to finance itself and survive and the industries ability to work together and fund mutually benefiting activities. 5.4.1. Exit Strategy The exit strategy was a combination of development project logframe describing the development work of the Yayasan and a set of business plans describing potential businesses that would generate income for the Yayasan. The basic logframe is presented below: Objective: Promote flows of information within the silk industry in West Java Output 1: Increased linkages to best practices in the silk industry; Output 2: Improved linkages at market level; Output 3: Improved access to industry to provide input to government policy This log frame demonstrates the same weakness as the original one in that the objective only restates a combination of the outputs. The objective should describe a desired end state resulting from the achievement of all the outputs. Looking at outputs 1 – 3, there are many possible ways to describe a higher order result that would infer a stronger, more sustainable, and possibly self managed silk industry in West Java is. The business plans put together to demonstrate the potential businesses the Yayasan could develop to support itself is essentially incomplete in two ways. The first is that the numbers presented in the plans are unsupported. The business plans received by the evaluator do not show cash flow projection income projections, or break even points. It is very likely that many of these businesses are viable, but it is unclear what it will take to develop them, how long until they start with a positive cash flow and/or become profitable. Moreover, the plans have not been integrated on paper. In other words, there is no integrated plan or cash flow chart showing · The need for start up capital for both businesses and Yayasan and how that need will be covered; · When the incomes from the businesses will start supporting the work of the Yayasan; · Action planned to be implemented; · Organizational structure; and · How much financial support these businesses will provide the Yayasan once they are up and running. SAMBA Project Final Evaluation Page 28 As the project is now completed and the Yayasan is an independent organization, the level of planning the Yayasan management feels is necessary is their decision. However, if there are plans to seek outside funding from either other donors or financial institutions, a more detailed business plan modeled on a good project proposal outline or traditional business plan (outlines of which can easily be found on the internet) will be necessary. 5.5. Project Strengths and Weaknesses and Recommendations 5.5.1. Summary of Project Strengths and Weaknesses The weaknesses of the project included: · An initial analysis which focused on farmers and cocoon production and not on the industry as a whole; · A proposal that did not clearly define the goal or the intended target group leading to early confusion as to who the BDSPs were and informal goal which focused on farmer welfare; · Weak and unclear management and management systems during the first year resulting a slow start and the need to reorganize; o Unclear goals and approach; o A structure/staffing pattern that was not sufficient to cover the region effectively; and o Leadership unable to balance the organizational needs of the project. · A partnership structure (salaries and responsibility structure) that did not support true partnership; · A monitoring and evaluation system that, while effective for monitoring activities and outputs, did not monitor for larger order results; · The lack of integration and separation between non-finance and finance; · The original plans for MFIs and credit support were too ambitious for the project; · The exit strategy lacked financial details and action plans for moving forward · The time period of the project was too short to see real impact during the life of the project. The strengths of the project were by the second year were: · CARE’s ability to recognize management issues that resulted in a slow start and reorganize/restructure to make up for it; · A dedicated and well functioning team with good leadership; o Clear goals and approach o Team building o Strong planning and implementing systems · The ability to provide useful and practical technical training to BDSPs based on applied research; · The ability to provide some basic management training to community level BDSPs (the project offered little to BDSPs that were private, on-going businesses); · The ability to adjust to changing situations in the silk industry; · The ability to support the silk industry by: SAMBA Project Final Evaluation Page 29 o Creating opportunities to promote the industry; o Extending information about the industry and industry business opportunities to financial institutions and industry members; and o Assisting member industries develop internal and external networks and business partnerships; · The ability to listen and respond to the situation in the silk industry in West Java · High levels of trust and support from the Silk Industry in West Java 5.5.2. Recommendations and Suggestions · Consider the inherent power differences in partnering with local institutions and formally or informally manage the project in such a way to share, as much as possible, power and control. This may require CARE to place a greater emphasis on working with partners to develop their capacity and CARE taking responsibility for partner’s weaknesses in front of the donor; · In projects designed to support an industry, base the project analysis on a strategic study of the industry that defines strengths and weaknesses of the industry and identifies key leverage points for the project’s interventions; · Clearly define the target groups for the project describing their economic status, support needed, and impact that support will have on developing the industry; · Base analysis of agricultural products at the farm level on farming systems not only on the crop or livestock itself; · Ensure all logframes provide a logical hierarchy where each level is a logical higher order result based on the achievements of lower levels; · Focus on establishing a team that clearly understands the goals and approaches of the project from the beginning and learns how to work together. The team building exercise in year two of this project was effective and helped develop the team’s ability to work as a team. · Ensure that monitoring and evaluation systems are “results based” and collect data not only on activities/outputs but also expected outcomes and impacts as described in the logframe; · Integrate gender into project planning, implementation, and monitoring. Provide gender training to all project staff. · Hold annual internal evaluation meetings to measure progress and results and to re-evaluate directions if necessary; · If similar projects are designed, integrate the non-financial and financial aspects of the project in order to develop synergy; · Integrate technical and management training into the longer term planning of BDSPs. If they do not have long term plans, assist them to develop them as part of the management support. · If project activities are to be sustainable after outside funding is completed, these activities must be designed from the beginning to be within the financial capabilities of the target group so that the target group can implement/maintain the activities without outside funding assistance. SAMBA Project Final Evaluation Page 30 · If possible, design exit strategy into the initial planning of the project. For projects that plan to become Yayasans, provide training in business planning and develop exit integrated strategy plans that show both the Yayasan's plans for continuing the project work and business financial analysis showing how the Yayasan will fund itself and its work. Attachments: Attachment 1. Scope of Work...................................................................................................... 2 Attachment 2. CV for Frank Page ................................................................................................ 7 Attachment 3. List of Informants................................................................................................. 13 Attachment 4. Care Samba Budget............................................................................................... 14 Attachment 5. Revolving Loan Fund Data.................................................................................... 14 Attachment 1. Scope of Work SCOPE OF WORK (draft 3) EVALUATION OF SILK & MICROENTERPRISE DEVELOPMENT IN BANDUNG RAYA (SAMBA PROJECT) USAID COOPERATIVE AGREEMENT 497-A-00-02-00007-00/ ECONOMIC GROWTH PROGRAM A. Summary This is the Scope of Work (SOW) for the evaluation of USAID funded CARE-UNBAR Project Silk & Microenterprise Development in Bandung Raya (SAMBA). The current funding is slated to end on 30 April 2004 and the evaluation will take place in April 2004. The evaluation will be conducted by an external evaluator who will be assisted by CARE throughout the process. The end product will be an evaluation report that includes an executive summary, introduction, methodology, contextual analysis, findings, conclusions, recommendations, lessons learned and unresolved issues. B. Donor/Grantee Guidelines USAID Cooperative Agreement with 497-A-00-02-00007-00/ (page 16) states that when an evaluation is performed, it should be conducted by a person or organization not associated with the Grantee (CARE). The final project narrative report prepared by CARE (as opposed to the evaluation report) should include a brief description of how the evaluation was performed and by whom, as well as results of the evaluation and CARE’s own assessment. This is the only known reference in the Cooperative Agreement to evaluations. In accordance with CARE Indonesia’s procurement guidelines, consultancy services over IRP10m must be received as sealed bids and over IRP50m must be managed by a bidding committee. C. Evaluation Objective The primary objective of this evaluation is to assess the level of attainment in project results in relation to the SAMBA project objectives. The evaluation is further underpinned by a capacity building ethos that sets out to build the knowledge and skills of both CARE-UNBAR project staff and CARE International Indonesia towards increased effectiveness in its programming, particularly in the Small Economic Activity Development (SEAD) area. To this end, the evaluation will also focus on key lessons learnt from the project and document these. The final evaluation report is therefore expected to provide detailed recommendations on key evaluation areas. These key areas will form the basis for the outline of the final evaluation report. Brief Background CARE Indonesia, in cooperation with the University of Bandung Raya (UNBAR) has been implementing a USAID grant through the Partnership for Economic Growth (PEG) Project since February 2002. UNBAR has long experience in the sericulture sector and CARE brings experience and expertise in micro enterprise development. The purpose of the SAMBA project is to improve the capacity and effectiveness of Service Providers providing financial and non-financial services to silk-related small micro enterprises. The project planned to target 90 Service Providers in West Java (50 Business Development Service Providers and 40 Micro Finance Organizations). The project’s interventions sought to strengthen the technical, management and financial aspects of the Service Providers for a duration of 27 months1 . The SAMBA goal is: In West Java, 2,000 silk related Micro Entrepreneurs achieve access to 90 improved Finance and Non Financial Service Providers through a competitive and market oriented approach. The project will focus on intermediary Service Providers and through training and support improve their effectiveness in; management, business services and business linkages in order to enhance their capacity to better support silk related Micro Entrepreneurs. CARE-UNBAR has agreed to focus the intervention on moriculture-sericulture-filature related micro enterprises. D. Information Sources Secondary Data Sources: project proposal, grant agreement; donor reports & correspondence; training/workshop reports; financial reports; visit reports; media produced; etc. ß Primary Data Sources: interviews with key informants and other stakeholders as appropriate; meetings; focus group discussions if appropriate; direct observation etc. E. Key Evaluation Issues 1. Project Design & Implementation: Assess the project design and reflect critically on the focus and management of implementation. ß How clear is the logical, cause-effect linkage among the different levels in the logframe? ß What are the risks and assumptions involved in the project and how did the project monitor and respond to them? ß Have indicators been stated in the logframe & are these used to measure achievements? ß Was the design based on an holistic analysis of the silk industry/rela ted market conditions? ß Describe the involvement and level of influence of the target group and partner organizations in the analysis, design, implementation, monitoring and/or evaluation of this project. ß What is the general reputation of this project among target group and partner organizations? ß Assess the overall quality of the focus and management of implementation. 2. Results/Achievements: Assess the level of attainment in project results (project outcomes and activities). ß Were activities carried out as per plan, on time, scale and with the intended target group? If not, what where the reasons? ß Did these activities lead to intended outcomes? ß Who/how many are the ultimate beneficiaries and what is the impact on these people’s lives? ß Were there any unintended outcomes (positive and negative)? 1 Includes an approved three month no cost extension. ß What are the most significant achievements (according to the project, its participants and other stakeholders)? ß Which target group and/or partner organizations have yielded more results than others. Why? 3. Sustainability: Present an analysis of the potential for sustainability of the project. ß What explicit exit strategies have been implemented to improve the sustainability of the project and what is the likelihood of these being effective? ß Are there sufficient mechanisms in place to maintain sustainable activities once CARE exits? ß How is the project building the capacity of beneficiaries and project partners to be in a stronger position to address their own needs? G. Conclusions & Recommendations Conclusions, based on the evaluation findings, will include effectiveness of the project in relation to its goal/objectives and key strengths, weaknesses, opportunities and threats. Recommendations, following from the conclusions of the evaluation. The evaluation is underpinned by a capacity building ethos than sets out to build the knowledge and skills of CARE and its partners towards increased programming effectiveness. To this end, the evaluation report is expected to provide detailed recommendations on: ß Each of the 3 key evaluation issues (see section E). ß The lessons that can be learned from this project experience. ß The practical steps that can be taken to overcome the existing weaknesses and obstacles identified. ß The particular organisational systems/processes that require attention and/or strengthening. ß How these lessons learned could be incorporated into CARE Indonesia’s broader SEAD program. H. Methodology ß Evaluation schedule preparation ß Review of secondary data and documentation ß Fieldwork: discussions, interviews, meetings, workshops, focus group discussions, etc. ß Feedback on preliminary findings to CARE and SAMBA ß Report compilation ß Presentation of findings. I. Consultant Tasks & Support from CARE The tasks of the Evaluation Consultant are as follows: ß Review and input into this Scope of Work; ß Draft evaluation schedule and budget; ß Conduct the evaluation according to schedule in the work plan; ß Write up the first evaluation report draft; ß Discuss and receive feedback on the draft report with team members; ß Revise the evaluation report based on input from CARE; ß Submit final evaluation report to CARE. CARE will support and assist the Evaluation Consultant as required, including the following tasks: Program Leader ß Review and comment on this Scope of Work; ß Support the evaluation process as required; ß Provide approval for Evaluator’s contract, budget, travel arrangements etc; ß Participate in the review meetings of evaluation report draft; ß Final acceptance of completed evaluation report on behalf of SAMBA; ß Dissemination of evaluation findings on behalf of CARE. Project Director ß Review and comment on this Scope of Work; ß Overall coordination of field travel including accessing necessary stakeholders; ß Support the evaluation process as required; ß Participate in the review meetings of evaluation report draft; ß Dissemination of evaluation findings within SAMBA. Contracts & Grants Officer ß Draft this Scope of Work; ß Discuss and negotiate Scope of Work with Evaluator; ß Provide information and documentation required for the evaluation; ß Manage the work of contracted Evaluator; ß Support the evaluation process as required; ß Participate in the review meetings of evaluation report draft. J. Outputs 1. Evaluation Schedule 2. Evaluation Budget 3. Evaluation Report (outline attached) K. Schedule Guideline # Phase Date (by) Participants 1. Objectives, TOR, schedule planning, contracts * CARE, SAMBA 2. Discussions on project/review data CARE, SAMBA 3. Briefing & discussions with UNBAR CARE & SAMBA if required 4. Fieldwork (to various sites) CARE, SAMBA, other stakeholders 5. Verbally present initial findings CARE, SAMBA 6. Draft and circulate report Evaluation Consultant 7. Feedback on draft report CARE, SAMBA 8. Submission of final evaluation report Evaluation Consultant *Exact dates and billable days will be determined by agreed schedule. L. Evaluation Report Format The evaluation report should be no more than 12 pages (not including essential attachments) and in English. The format to be used for the report follows. Topic Sub sections Executive Summary Project purpose Evaluation objective Summary of evaluation issues Key findings Recommendations Explanation of report lay-out Introduction Project details – donor, period, amount, etc Evaluation objective Evaluation team introduction: teams, positions, roles in evaluation Project purpose: goal, objectives, activities and outcomes (and any modifications) Methodology Description of evaluation tools and methods List of information sources Evaluation schedule Evaluation limitations/shortcomings Contextual Analysis What significant external events have taken place that have a direct bearing on the project? What did any internal organizational changes pose for the project? Findings 1. Project Design & Implementation 2. Results/Achievements 3. Sustainability Conclusions & Recommendations Overall effectiveness – Strengths, Weaknesses, Opportunities, Threats Recommendations on key findings Unresolved issues Lessons Learnt With reference to experiences that could prove useful to CARE, other organizations and USAID. Number the lessons learnt. Attachments Only essentials please Attachment 2. CV for Frank Page PROFILE Ten years experience consulting to strengthen local NGOs in Indonesia. Six years of international experience with US Private Voluntary Agencies in managing grassroots community development. Skilled in designing, monitoring, and evaluating projects. Trained in assessing organizational needs and planning for change. Able to communicate in Bahasa Indonesia. MS. in Human Resource Development/Organizational Development and an MS. in International Agricultural Development with an emphasis in Agricultural Economics. Particular experience and interest in: · Developing effective participatory organizations · Assessing organizational needs and managing change · Designing programs that meet specific needs of organizations and communities · Developing partnerships with NGOs & communities · Strategic Planning · Organizational Diagnosis and Profiles · Project Evaluation and Planning · Planning, Monitoring & Evaluation Systems (PMES) · Organizational Structure · Training & Coaching - Management/Leadership Skills, OD & Consulting Skills, TOT & Participatory Methodologies, Team Building & Myers-Briggs Type Indicators WORK EXPERIENCE Organizational Development Consultant, December 1993 - Present Jakarta, Indonesia Established one person consulting business under the auspices of PT UMS. As a consultant I have provided organizational development consulting, training, and project evaluation services to both local and international NGOs in Indonesia and Cambodia. My key strengths are in the areas of a) Strategic and Project Planning, b) Organizational Diagnosis, c) Management, Monitoring, and Evaluation, d) Building Collaborative Relationships, and e) Managing Change. PARTIAL LIST OF CLIENTS AND WORK COMPLETED Heifer Project International/Indonesia, Bukit Tinggi, Indonesia May - August, 2003 Design and establish Planning, Monitoring, Evaluation and Reporting (PMER) Systems. Worked with Management and Staff of Heifer Project International to collaboratively design and implement results based PMER by facilitating choice of indicators based on the programs vision and strategic plan, developing collections systems and schedules, collaboratively designing semi-annual workshops incorporating their Conerstone Model methodology, and designing workshop sessions to introduce results based PMER to their NGO partners and community groups. BP/Chemonics International, Jakarta, Indonesia September, 2001 – October, 2002 BP/Chemonics Resettlement Advisory Panel Manager: Managed interactions between a Resettlement Advisory Panel made up of two world class experts based in Washington DC and the BP Tangguh Project Resettlement Team based in Jakarta and Irian Jaya. Worked closely with Resettlement Managers to implement Panel Advice and develop medium and short term action plans. With the Resettlement Advisory Panel, planned, organized, wrote, and edited the Land Acquisition and Resettlement Action Plan for the Tangguh Project’s Environmental Impact Statement and BP’s internal review process. Civil Society Support and Strengthening Program (CSSP), Jakarta, Indonesia July – August, 2001 Reporting and Monitoring Training Workshops: Designed and conducted two follow up workshops that a) reinforced the results of the first workshop, b) provided feedback to the participants on first round of reports submitted in order to improve report accuracy and quality, and c) facilitated the participants to evaluate their own NGO’s monitoring system and develop plans for improvement. Cooperating with PAKTA, developed a simple monitoring database program (MonEv) that assists projects and NGOs to records and report outputs, outcomes, and impacts. This program was introduced and distributed during these workshops. March, 2001 Manual Writing: Wrote manual describing the reporting the requirements for CSSP grantees and explaining how to organize and complete the CSSP program monitoring forms, data tables, and narrative reports. January, 2001 Report Training Workshop: Designed and conducted the first in a series of 3 workshops to develop CSSP grantees' capability in writing monitoring reports. Facilitated the group through refining the monitoring report outline, understanding the data tables, required, and practicing writing draft reports. October, 2000 Project Evaluation: Designed framework for and conducted evaluation of a national public opinion survey conducted by the Consortium for Public Opinion Polling. The evaluation showed that the Consortium was able to conduct a statistically valid national public opinion poll and deliver a useful product to the MPR/DPR. Médecins Sans Frontiéres (MSF)/Yayasan Tanpa Batas, Kupang Indonesian March – April, 2001 Program Evaluation: Conducted participatory end of project evaluation for MSF’s AIDS prevention program in Kupang as well as designed and implemented organizational diagnostic and planning workshops local staff and new board of directors to develop action plans for converting the MSF project into the local NGO, Yayasan Tanpa Batas. August, 2001 Strategic Planning: Designed and facilitated strategic planning workshop for the newly formed Yayasan Tanpa Batas. PACT, Jakarta, Indonesia December, 2000 Training of Trainers: Co-designed and facilitated a Training of Trainers Workshop for NGO staff members who will train their colleagues in facilitation skills under Pact's Discuss project August, 2000 Project Evaluation: Lead a three member team to evaluate Pact's Indonesian NGO Partnership Initiative (INPI). Designed evaluation framework and conducted staff interviews and a three day collaborative workshop to collect and analyze data on the project. May, 2000 Annual Retreat: Facilitated PACT staff in redefining their core technologies and country mission. Assisted group in developing plans to implement and market updated office mission. November, 1999 Project Evaluation & Planning: Facilitated a three day workshop for members of Jaringnet to evaluate accomplishments and lessons learned over the two years of the Indonesian Rapid Response Initiative (IRRI) project and develop plans for new IRRI proposal. August, 1999 Training: Co-designed and facilitated Evaluation Training Workshop for a team of evaluators that PACT will hire to evaluate their INPI project grantees. April, 1999 Training: With a team from Business Dynamics, designed and facilitated a Training for Facilitators in Participatory Methods for members of PACT's Jaringnet Project November - December, 1995 Project Evaluation: Represented PACT on final evaluation team of the USAID funded EPOCH AIDS Prevention Project. Evaluated the project's design, process, and impact of delivering technical and institutional development assistance to seven Indonesian NGOs working with high risk groups in Jakarta, Surabaya, and Bali. The evaluation documented the effectiveness of combining TA and funding, especially to new and developing NGOs. March - September, 1995 Organizational Structure: Assisted PACT Staff review and develop their organizational structure. Training: Conducted needs assessment of Institutional Development tools and skills within PACT. Designed and conducted trainings on Diagnosing Organizational Health, Training of Trainers, Consulting Skills, and Team Building. Developed organizational diagnostic tool for use by PACT Project Directors. October - November, 1994 Strategic Planning: Facilitated PACT/Indonesia develop five-year strategic plan. Monitoring & Evaluation: Developed project mo nitoring system for three PACT Institutional Development projects and facilitated PACT/Indonesia's Strategic Planning process which developed PACT/Indonesia's vision, strategic directions, quarterly plan, and 90 day work plan. Business Dynamics, Jakarta Indonesia May, 2000 Organizational Diagnosis: Conducted four day Institutional Development Framework Workshop with Ikatan Ahli Kesehatan Masyarakat Indonesia (IAKMI) August, 1999 Organizational Diagnosis: With a team from Business Dynamics, designed workshop and facilitated MSI's Institutional Development Framework Workshop for the HIV/Aids Prevention Project (HAPP) Secretariat. December, 1998 Organizational Diagnosis: With a team from Business Dynamics, designed and facilitated MSI's Institutional Development Framework Workshop for the Natural Resource Management (NRM) Program Facilitation/Secretariat section. Project Concern International (PCI), Jakarta, Indonesia May, 2000 Myers Briggs Type Indicators: Administered and scored MBTI instruments (some in English, some in Bahasa Indonesia) and conducted one day workshop to confirm results from the instruments and introduce the MBTI to the organization. January - February, 2000 Strategic Planning: Facilitated PCI staff collect data on the organizations customers, competitors, strengths, and weaknesses and develop a five year vision for the organization and plans to achieve that vision. Facilitated half-day discussions on managing conflict and developing PCI structure. The Nature Conservancy (TNC), Jakarta, Indonesia February, 2000 Conservation Strategic Planning: Facilitated TNC senior staff in developing TNC/Indonesia's conservation strategies for upcoming five years. December, 1999 - May, 2000 Grant System Development: Working with TNC senior staff, develop a grant proposal and financing system consisting of proposal formats, financial formats, and handbooks describing how to use the systems. October, 1999 Strategic Planning: Facilitated TNC's Senior Management team to evaluate external opportunities and develop a vision, strategic aims, and program plans for the Indonesian Country Program World Wide Fund for Nature, Jakarta, Indonesia July, 2001 & June, 2000 Project Planning: Facilitated WWF's Nusa Tenggara program review and evaluate its achievements for the previous year and develop program plans for the next three years based on the evaluation and strategic plan. April, 2000 Ecoregion Based Conservation (ECB): Facilitated WWF/Wallacea and partner NGOs in developing plans to undertake and ECB program for Nusa Tenggara. October, 1999 Program Planning: Facilitated week and a half workshop to develop strategies (based on WWF/Netherlands's strategy of Magnification) and work plans for WWF/Wallacea Program Campaigns. July, 1999 Project Planning: Facilitated WWF's Nusa Tenggara program develop strategies and program plans for the next three years. May, 1999 Strategic Planning: Facilitated Strategic/Program Planning for WWF's National Initiatives Division with Division members and the senior management team March, 1999 Program Planning: Facilitated Team Building and Program Planning Workshop with WWF's Wallacea Program December, 1998 Ecoregion Based Conservation: Facilitated the introduction of WWF/US's Ecoregional Based Conservation System to senior members of the Irian Jaya Region Team and Jakarta Technical Support Unit. October - November, 1998 Project Evaluation and Planning: Facilitated an Internal Review of the WWF Kayan Mentarang National Park Project. The Internal Review identified work accomplished to date, sharpened plans to complete the current phase of the project, and developed the vision and goals for the next phase of the project. October, 1998 Facilitated the WWF/Indonesia's Annual Meeting. May, 1998 Organizational Structure: Facilitated WWF/Indonesia's senior management team develop a new structure for the organization based on Bioregional Conservation. April, 1998 Strategic Planning: Facilitated Senior Staff of WWF to further define and finalize the Strategic Plan for WWF/Indonesia. In the workshop, the group developed a clear vision for WWF's work in Indonesia and broad strategies to achieve that vision. March, 1997 Project Evaluation: Designed and conducted a three day workshop to evaluate WWF's Wasur National Park Project and develop plans for the continuation and eventual phase down of the project. Wrote draft proposal that was funded by WWF/Netherlands. Yayasan Lembaga Bantuan Hukum Indonesia (YLBHI), Jakarta, Indonesia .April, 1998 Monitoring & Evaluation Systems: Designed and conducted three-day Planning, Monitoring, and Evaluation System (PMES) Workshop for YLBHI senior management and LBH local office Directors. Workshop resulted in the design of an integrated PM&E system. NOVIB, Jakarta, Indonesia ... December, 1997 -February, 1998 Organizational Profiles: Researched and wrote organizational profiles of Wahana Lingkungan Hidup Indonesia (WALHI) and Yayasan Lembaga Bantuan Hukum Indonesia (YLBHI) for the Dutch funder NOVIB. Profiles describe history, organizational structure, work processes of these organizations, and documented the results of organizational development programs. Wahana Lingkungan Hidup Indonesia (WALHI), Jakarta, Indonesia May, 1997 - May, 1998 Management Systems: Consulting with Badan Executive of WALHI to improve internal planning and monitoring systems, develop recruiting and hiring systems, increase effective internal communications and management mechanisms, and increase effectiveness of network wide meetings. Facilitated two annual evaluation and planning workshops for WALHI Badan Executive. Conducted short training sessions on delegation and conflict management. January - February, 1995 Project Evaluation: Managed three-man team to prepare the final evaluation of the USAID funded Promoting Environmental Advocacy and Policy Dialogue Project completed by WALHI. Facilitated the staff of the WALHI Secretariat through a strategic planning process, which developed the Secretariat's vision, strategic directions, strategic benchmarks, quarterly plan, and 90-day work plan. MANAGEMENT EXPERIENCE OEF International, Acting Latin America and Asia Director, Washington March, 1991 - July, 1991 Developed networks and marketing materials necessary for developing partnerships with organizations in Latin America and Asia. Wrote proposal for OEF's partner organization in Honduras and received funding. OEF International, Project Manager, Somalia July, 1989 - March, 1991 Managed establishment of Micro Enterprise Credit Program for Women. Developed implementation plans. Hired local staff of five field workers and two accountants. Developed credit procedures and policies with credit consultant. Represented OEF to local and national government officials and USAID. Prepared project budget and co-wrote project amendment for USAID. Planned and implemented organized close out of project before civil war forced evacuation of Somalia. Save the Children, Field Manager, Sri Lanka November, 1986 - July, 1988 Managed establishment of new Impact Area. Developed five-year plan and annual implementation plans. Designed and implemented community based revolving loan programs. Developed with Deputy Director planning, financial, and reporting systems for Sri Lanka Field Office Impact Areas. EDUCATION MS. Human Resource Development, American University/NTL Institute, Washington, DC. April 1993. MS. in International Agricultural Development with an emphasis on Agricultural Economics, University of California, Davis. December 1984. BA. in Economics, Carleton College, Northfield, Minnesota, June 1979. LANGUAGES Bahasa Indonesia and Cebuano (Philippine Dialect) OTHER TRAININGS ATTENDED Facilitating Strategic Planning, NTL Institute, Alexandria, Virginia. This program covered both a systematic process for developing strategic plans and methods for facilitating a group through the process. Managing Conflict: Organizational and Interpersonal, NTL Institute, Alexandria, Virginia. Covered nature of conflicts and tools for diagnosing and positively confronting conflict situations. (1/97) Winning Through Participation, Institute for Cultural Affairs, Fairfax, Virginia. Covered processes for participatory discussion, consensus building, and action planning. (3/93) and Strategic Planning (5/93) Open Space Technology, McNair and Associates with Harrison Owen, Springfield, Virginia. Covered the process for conducting Open Space Technology conferences. (2/93) Myers-Briggs Type Indicator Qualifying Workshop, Otto Kroeger Associates, Fairfax, Virginia. Received qualification to purchase, administer, and score Myers-Briggs Type Indicator test. (7/92) Training of Trainers, US Save the Children, Westport, Connecticut. (10/85) Attachment 3. List of Informants Name(s) of Informant Organization Steve Gilbert CARE/Jakarta Johan CARE/Jakarta Pak Thamrin SAMBA Project Director Pak Wibowo UNBAR – SAMBA Project Deputy Director Pak Gatot SAMBA Project - Finance Ibu Ferry SAMBA Project – Field and Program Administration Coordinator Ibu Wita SAMBA Project Procurement Pak Juarto SAMBA Project Field Supervisor Pak Mifta SAMBA Project Field Supervisor Pak Wahyu SAMBA Project Field Supervisor Pak Eddy SAMBA Project Field Supervisor Pak Davi SAMBA Project Field Supervisor Pak Majalenka SAMBA Project Field Supervisor Pak Irfam SAMBA Project Financial Specialist Pak Dedi SAMBA Project Financial Specialist Pak Dudi Santosa & Budi Negraha CV Ahmad Maksum Silk Art (weavers) Pak Mubaroka & Hajil Burhan Kelompok Tanih Genung Tokeh (a farmers group) Pak Sole Silk Spinning and Weaving Business Pak Endang Dahlan BMT Ar-Ridwan Pak Eddy Budiman BPR LPK Garut Pak Amsuhara Farmer Pak Diskinafud Farmer Pak Aan Supriganta Kooperasi Usaha Konservasi Genung Harimun Pak UU, Pak Nasrul, Pak Munawan, & Pak Ayep Kooperasi Sabiluhunang Pak Uus Somantri PT Hijau Murbai (mulberry tea factory) Pak Sumako PT Hijau Murbai Pak Budi Sunandi BPR Sukabumi PT Prima Expindo Utama Pak H. Sansudin Farmer Pak Oke UNBAR – SAMBA Technical Specialist Attachment 4. Care Samba Budget Samba Project Budget Items USAID (Old) US $ USAID (Ammended)* Partner Total Personnel 257,752 210,612 210,612 Fringe Benefit 48,723 45,684 45,684 Travel and Per Diem 85,932 61,789 61,789 Equipment 60,405 29,302 34,100 63,402 Supplies 60,228 48,460 48,406 Contract Services 133,050 247,221 194,275 441,496 Other Direct Costs 3,730 3,624 3,624 Total Direct Charges $649,820 $646,692 $875,067 Indirect Charges 50,179 53,307 53,307 Totals $699,999 $699,99 $928,374 * On December 9, 2003, USADI approved an amendment to the original budget Attachment 5. Revolving Loan Fund Data Recipients of Revolving Loan Fund MFI Partner Fund Placement Date Fund Placement Amount BPR Cisurupan 26-Jun-03 $16,179.13 BPR Sadayana Artha 27-Jun-03 $32,358.25 BPR Panjawan Mitra Usaha 5-Aug-03 $31,230.50 BPR Tanjungsari 23-Dec-03 $20,035.05 BPR Hayura Artholola 3-Feb-04 $20,260.81 Total 124,977.74 Progress Reported by 3 BPRs MFI Partner Cut-off Date # of Borrowers Loan Portfolio (Rp Million) Repayment Rate BPR Cisurupan Apr-03 Dec-03 294 355 1,372 1,983 95.66% 96.85% BPR Sadayana Artha Apr-03 Dec-03 456 805 1,814 3,622 97.19% 98.99% BPR Panjawan Mitra Usaha Jul-03 Mar-04 837 781 4,614 5,028 93.17% 92.7%