End-of-Program Evaluation of the Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru Prepared for Strategies for International Development By Loyd C. Brown Haydée Delgado de Brown La Paz, Bolivia and Cusco, Peru December 2003 Matching Grant Program End-of-Program Evaluation i Presentation and Acknowledgements A team of consultants was contracted by the International Non-governmental Organization, Strategies for International Development, to conduct this external evaluation coinciding with the scheduled termination of the Matching Grant Program to Provide Technical Assistance to NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru. The team was lead by Dr. Loyd C. Brown, an agricultural economist with more that two decades of experience in international consulting work in the area of rural development. The team members who assisted Dr. Brown include Lic. Haydée Delgado de Brown, economist, who participated in the data collection phase in Bolivia and in the results analysis and report preparation. Staff members of the SID Country Offices in Perú and Bolivia participated in the data collection activities and data preparation in the two countries. The present document reflects of the results of all these efforts. The evaluation team wishes to express its gratitude to all of those persons who contributed their time and energy to the preparation and presentation of materials that went into the research behind this evaluation effort. Special thanks are offered to Dr. Karen Kraft, Country Director of Peru, Mr. Evan Meyer, Program Director of Bolivia, and Mr. Abraham Borda, Country Director of Bolivia for their valuable technical contributions, their administrative support and their untiring efforts to support the team. Finally, our thanks to Mr. Charles Patterson, Executive Director of Strategies for International Development, for his continuing confidence in the Team and to him and his staff for the support provided throughout the evaluation. Matching Grant Program End-of-Program Evaluation ii Contents Page Presentation and Acknowledgements i Contents ii List of Tables iv List of Figures v Executive Summary 1 1. Program Background 5 1.1 Brief History of the Matching Grant Program 5 1.2 Rationale for the Program 5 1.3 Situation at the Beginning of the Program 6 1.4 What the Program Seeks to Achieve 6 1.4.1 Bolivia 7 1.4.2 Peru 8 1.4.3 The SID Central Office 8 1.5 The Phases of Technical Assistance 9 1.6 Principal Partners 10 2. Program Effectiveness 12 2.1 Program Model and Approach 12 2.1.1 Unique Features of the Program 12 2.1.2 Program Justification 12 2.2 Soundness of the Program Hypothesis and Assumptions 13 3. Achievement of Objectives 13 3.1. Results of the Bolivian Program 15 3.1.1. Overall Results 15 3.1.2. Reactions to the Midterm Evaluation 15 3.1.3 Results Indicators 16 3.1.4 Inter-institutional Evaluation 28 Matching Grant Program End-of-Program Evaluation iii 3.1.5 SID-Bolivia Staff Input 29 3.2 Results of the Peruvian Program 31 3.2.1. Overall Results 31 3.2.2 Reactions to Mid-term Evaluation 34 3.2.3 Results Indicators 35 3.2.4 Inter-institutional Evaluation 51 3.2.5 SID-Peru Staff Input 52 3.3 Results of the SID Central Office 54 3.3.1. Overall Results 54 3.3.2. Reactions to the Mid-term Evaluation 54 3.3.3. Results Indicators 54 4. Expressed Client Needs 59 5. Program Strengths 61 6. Conclusions 64 7. Evaluation Methodology 66 7.1 Evaluation Objectives 66 7.2 The Methodology Employed 66 7.3 Contacts 67 8. Concluding Comments 67 Bibliography 68 Annex A - Logical Framework for the Matching Grant Technical Assistance Program 70 Matching Grant Program End-of-Program Evaluation iv List of Tables Page Table B-1. Number of Events conducted by the SID-Bolivia Staff during the Life of the Program, by Department 15 Table B-2. Organizations and Families that Benefit from the Bolivian National Program 19 Table B-3. Program Progress of Bolivian NGOs and Grassroot Organizations 20 Table B-4. Results for Indicator 6 of Objective 2 for the Bolivian Country Office 26 Table B-5. Utility of the SID-Bolivia Technical Assistance 28 Table P-1. Participation in General Meetings, by Department, Accumulated through September 2003. 31 Table P-2. Institutional Workshops – Number of Participants, Number of Events and Level of Effort, by Department and Institution, Accumulated through August 31, 2003. 32 Table P-3. Number of Events conducted by the SID-Peru Staff during the Life of the Program, by Department 33 Table P-4. Organizations and Families that Benefit from the National Program of Peru 39 Table P-5. Program Progress of Peruvian NGOs and Grassroot Organizations 40 Table P-6. Requests for Technical Assistance Received from NGOs and Grassroots Organizations by the Peruvian Country Office 41 Table P-7. Results for Indicator 6 of Objective 2 for the Peruvian Country Office 48 Table P-8. Utility of the SID-Peru Technical Assistance 51 Table C-1. Results Indicators for the Central Office 55 Table C-2. Results Indicators for the Central Office (Continued) 57 Table C-3. Results Indicators for the Central Office (Continued) 59 Table A-1. Logical Framework for the Matching Grant Technical Assistance Program, SID-Bolivia Planning Matrix 71 Table A-2. Logical Framework for the Matching Grant Technical Assistance Program, SID-Peru Planning Matrix 72 Table A-3. Logical Framework for the Matching Grant Technical Assistance Program, SID-Central Office Planning Matrix 73 Matching Grant Program End-of-Program Evaluation v List of Figures Page Figure B-1. Increase in Average Farm Family Income 17 Figure B-2. Increase in the Hectares of Productive Crop and Pastureland, per Farm Family 18 Figure B-3. Project Monitoring 22 Figure B-4. Project Evaluation 23 Figure B-5. Percentage of Institutions that Use Planning Techniques and Instruments 24 Figure B-6. Planning Aspects of Major Utility to Participating Institutions 24 Figure B-7. Percentage of Institutions that Use Planning Techniques and Instruments 25 Figure P-1. Increase in the Average Farm Family Income 36 Figure P-2. Increase in the Hectares of Productive Cropland and Pasture per Farm Family 37 Figure P-3. Yield Increases for Principal Crops 38 Figure P-4. Project Monitoring 43 Figure P-5. Project Evaluation 44 Figure P-6. Percentage of Institutions that Use Planning Techniques and Instruments 44 Figure P-7. Planning Aspects of Major Utility to Participating Institutions 45 Figure P-8. Institutional Strengthening Aspects of Major Utility to Participating Institutions 46 Figure P-9. Project Design Adjustments 47 Figure P-10. Increase in the Average Number of Participating Families per Institution 49 Figure P-11. Increase in the Average Number of Participating Families per Community 49 Figure P-12. Increase in the Average Participation of Women in Projects 50 Matching Grant Program End-of-Program Evaluation 1 Executive Summary The Matching Grant Program of Strategies for International Development (SID) addresses two problems - rural poverty and the erosion of soils and pastures in rural communities. The ultimate goal of the Program is to reclaim the soils and pastures and to increase rural income in the Altiplano and high valleys of Bolivia and Peru. Since Non-Governmental Organizations (NGOs) and Grassroot Organizations provide the services that help farmers address both these problems, the Program helps these organizations improve the coverage and impact of the projects that concentrate on this goal. In Bolivia, the objective of the Program is to reclaim the soils and pastures and to achieve a sustainable increase in the production and income of 9,600 farm families in 320 communities served by 32 NGOs. In Peru, the objective is to reclaim the soils and pastures and to achieve a sustainable increase the production and income of 6,600 farm families in 220 communities served by 22 NGOs. In addition, SID also proposes to establish effective and sustainable training and technical assistance services to farmers in reclaiming soils and pastures and increasing production and income by the 32 NGOs in Bolivia and the 22 NGOs in Peru. Finally, SID has set overall objectives to improve the ability of all staff to provide technical assistance to NGOs in order to achieve the objectives in Bolivia and Peru. The strategy for achieving these objectives is a combination of three key elements: (1) provide technical assistance to NGOs in accordance with the major phases of the Project cycle for all organizations - learning new ideas and opportunities; planning how to incorporate them into existing projects or planning new projects; designing better monitoring and evaluation systems; implementing new or revised projects; evaluating, reporting and assessing performance; and improving the projects for the coming year; (2) create some competition as well as collaboration among NGOs to find better methods and to improve performance; and (3) partner with NGO federations and networks in Bolivia and Peru to carry out the technical assistance and help institutionalize it. The three organizational units of the Program include the Bolivian Country Office in La Paz, Bolivia, the Peruvian Country Office in Cusco, Peru, and the Central Office of SID in Arlington, Virginia in the United States. The evaluation of the Program is organized in a manner that permits viewing of each of these elements. Conclusions ÿ Large Numbers of Beneficiaries - The Matching Grant Program is reaching a very large number of final beneficiaries through its work with the NGOs and Grassroot Organizations in Bolivia and Peru. A total of 23 NGOs and Grassroot Organizations participated in the Program in Peru. They managed projects in agriculture and natural resources (as well as in other thematic areas) for a total of 217 communities and 10,987 rural Matching Grant Program End-of-Program Evaluation 2 families. These numbers indicate that SID-Peru exceeded the target number of organizations and, when considered the two organizations for which information is not available, far surpassed the end-of-program goals for the number of communities and rural families. On the other hand, a total of 35 NGOs and Grassroot Organizations participated in the Program in Bolivia. These organizations managed projects in agriculture and natural resources (as well as in other thematic areas) for a total of 1,184 communities and 30,246 rural families. These numbers indicate that SID Bolivia also surpassed the target number of participating organization and went far beyond the end-of-Program goals for the number of communities and rural families. These numbers are indicative of the highly successful work of the Country Offices in their application of the selection procedures for participant organization and in the provision of technical assistance in the different phases of the six-phased, two￾tiered plan on which the Program is based. The other facet of this element deals with the numbers of professionals and technicians that the Program has trained and made available to the sector. In SID-Peru alone, a total of 379 NGO staff persons attended one or more workshops. The numbers for Bolivia are somewhat lower. These trained professionals will have a continuing impact on rural development￾related activities in Cusco, Apurimac and Puno for some time to come. ÿ Highly Qualified and Experienced Program Base - The Matching Grant Program is a one-of-a-kind design that builds on the extensive and rich experience of SID and its Executive Director. Their trajectory extends to a large number of related Programs in Latin America and other parts of the world, each of which contribute in some way to the ideas and methodologies used in the design and development of this Program. Personnel who worked in the Country Offices in both Bolivia and Peru were well qualified, highly motivated professionals. Personnel of the NGOs and Grassroot Organizations termed SID personnel as being collaborators. They indicated that SID Personnel listened to the ideas of others and were disposed to help encounter solutions to the problems of the NGOs. Technically, their teachings, critical analysis and identification of project weaknesses contributed to the improvement of the work done by the participating organizations, permitting them to make needed adjustments in their projects and technical assistance programs. This resulted in a high degree of confidence and esteem on the part of the participating organizations, generating demand for services and permitting an open and frank relationship between these organizations and SID. The directors of the two Country Offices demonstrated a high degree of dedication and professionalism in their work. Their abilities to direct and coordinate the work of the offices and the teams resulted in positive impacts beyond the targets of the Program. ÿ Improved NGO Projects - The institutional strengthening methodologies employed by the Program proved very effective in providing the personnel of the participating organizations with the tools required to expand and fortify their projects and to bolster the impact among the communities and the rural families that they were serving. The techniques and procedures taught were extremely useful to the organizations in upgrading the projects Matching Grant Program End-of-Program Evaluation 3 they were managing, in the design and development of new projects that were effective and efficient, and in monitoring and evaluating both previously existing and as well as newly developed projects. The format employed by SID in their technical assistance activities will have a lasting impact on the capability of the NGOs and Grassroot Organizations in the development and implementation of their future projects and has already began to be multiplied in other areas and organizations as the trained professionals begin to spread their influence. ÿ Impacting Final Objectives – The final objectives of the Program are to reclaim the soils and pastures and to achieve a sustainable increase in production and income for the rural families in the Altiplano and high valleys of Bolivia and Peru. Data provided by the Bolivian Country Office based on their interviews with 13 of the participating organizations during the Internal Evaluation in August and September 2003 indicate an increase of 23.3 percent in the average farm-family income. Results of the same study indicate an average increase of 0.93 hectares (58.9%) of productive crop and pastureland for communities in seven reporting organizations. Similar data for Peru reflects an increase in the average farm-family income of 25 percent, while the average amount of productive crop and pastureland increased by 0.43 hectares (21.5 %). The reactions of the NGOs and Grassroot Organizations show that the new focus of their projects and the newly learned methods and procedures have permitting them to design and implement activities that are more effective in targeting and achieving the desire results. ÿ Sound Program Foundation - The Matching Grant Program was built around the idea of providing technical assistance for NGOs, consisting of new and improved ways of broadening and strengthening the impact of their projects in the target populations. This focus is different from that of many traditional rural development and natural resource projects that utilize available funding to try to reach as many rural families as possible with specific technical assistance activities. The Matching Grant Program has been successful in combining a limited budget and a small amount of human resources and infrastructure to leverage the human and financial resources available to the participating organizations and, at the same time, strengthen those organizations. This has resulted in a significant multiplier effect in reaching current beneficiaries and in setting the stage to reach even larger numbers in the future. ÿ External Influences – Both Bolivia and Peru experienced some degree of problem related to their identified associates. The political turmoil in Peru during the year 2000 reduced the funding available to its Program associate, La Coordinadora Rural, and to the participating NGOs during the early stages of the Program. During the succeeding years this problem was overcome and La Coordinadora Rural became a stalwart force in the overall Program. In the case of Bolivia, the lack of true associate in AIPE limited the proposed effect that this relationship was intended to have on the Program. Matching Grant Program End-of-Program Evaluation 4 ÿ Valuable Training – The SID workshops provided the participating organizations with valuable elements that have permitted them to accomplish a critical analysis and the identification of deficiencies within their projects. These continuing activities were part of the strategy that promoted a learning environment within the organizations. ÿ Women´s Participation Improved – All of the Peruvian organizations and 19 of the 20 Bolivian organizations interviewed in the Internal Evaluations reported advances in the level of participation of women in the projects in their areas. In all of these cases the percentage of women participating in community and project activities increased substantially – in the range of 15 to 40 percent in most project areas. The participation of women in family and community activities is more often observed in tasks related to the home and to agricultural and livestock production. In the case in the training workshops the participation of women was initially observed to be limited. However, adjustments to the SID approach and programming of workshops helped to increase their presence and active role. ÿ Increase in Income vs. Increase in Productive Land – In the case of both the Bolivia and Peru, the percentage increase in family income reported in Results Indicator 1, was slightly lower than the targeted goal of 30 percent (23% for Bolivia and 25% for Peru). However, considering the advance achieved in the percentage increase in the average hectares of productive crop and pastureland compared to the targeted figure of 20 percent (Bolivia 58% and Peru 21.5%) farm family incomes will surely improve in the short-run as the effect of increased productive land is manifested in increased production. ÿ Utility of SID Technical Assistance – The inter-institutional evaluation workshops conducted by the Country Offices in September 2003, with the participation of the cooperating institutions and a large number of NGOs and Grassroot Organizations resulted in numerous valuable insights into the design, development, implementation and impact of the Program. (See Tables B-5 and P-8.) Specific comments relate to improvement in use of participatory methodologies, an increased importance of monitoring and evaluation activities, a better attitude toward change and teamwork spirit, improved reporting procedures and timing, expanded use of advance techniques in project design and development, more local population involvement in all project phases, as well as many others. Closing Statement The evaluation team anticipates that the focus and content of this evaluation will prove to be useful to all parties involved in the management and implementation of the matching grant programs and to those final beneficiaries of program activities – the NGOs and the rural farm families they serve. It is our desire that the sharing of experiences, knowledge and lessons learned during the field visits will lend to the expanded collaboration and cooperation among the participating parties, resulting in the continuous improvement of the quality and effectiveness of this type of program in each of these countries as well as around the world. Matching Grant Program End-of-Program Evaluation 5 Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru 1. Program Background1 1.1 Brief History of the Matching Grant Program The “Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and to Reduce Rural Poverty in Bolivia and Peru” was conceived as a matching grant activity by “Strategies for International Development” (SID), under a contract signed with AID/Washington. The Program began in September 2000 and, including extensions, terminates in December 2003. The ultimate goal of the Program, as indicated in the title, is to reclaim the soils and pastures and to increase rural income in the Altiplano and high valleys of Bolivia and Peru. Since Non-Governmental Organizations (NGOs) and Grassroot Organizations of these areas provide the services that help farmers address both these problems, the Program helps these organizations improve the coverage and impact of the projects that concentrate on this goal. The stated mission of SID is to help organizations use the most effective methods for achieving the goals of international development. SID works in rural development because there is a lack of successful strategies and methods. Market-driven approaches are gradually replacing the failed integrated rural development strategies of the past, but they are not the whole solution. 1.2 Rationale for the Program The development philosophy of SID consists of three fundamental components. First is that no single organization nor small group of organizations can effectively help farmers reclaim soils and pastures and increase income. The regions are too large, and the management burden and need for resources is too great. A consortium of organizations is needed, and that consortium is the NGOs working in rural development. Second is that farmers must take the initiative and make the decisions on how to change their practices, if these changes are to be 1 Extracted from: Brown, Loyd C., Haydée Delgado de Brown and Nohemí Villena, Mid-term Evaluation – Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru, prepared for Strategies for International Development, La Paz, Bolivia and Cusco, Peru, December 2002, p. 9-15 Matching Grant Program End-of-Program Evaluation 6 permanent. The best role for the NGO is to provide information which helps farmers make better decisions, facilitate the decision-making process, and establish structures which encourage and energize farmers to find their own solutions, implement them and improve upon them. Finally, is that members of the NGO staff need to become full knowledge workers who collaborate and compete in developing and applying the best knowledge for reclaiming soils and pastures and increasing income. The task is to help NGOs learn better methods, adapt them to their own projects, implement the projects and then evaluate and improve upon them. This also includes the creation of a consortium of NGOs that collaborate and compete in devising better methods, applying them, evaluating performance and improving on the methods. With these considerations, the NGOs and Grassroot Organizations that work in the selected Program areas have become elements of the delivery system for activities aimed at increasing farm family income as well as reclaiming natural resources and protecting the environment. However, it was determined that these organizations required strengthening in order to improve the focus of their targets and the methods used in project design, implementation, monitoring and evaluation. Only in this way would they be able to broaden their impact. 1.3 Situation at the Beginning of the Program The natural resources of the Altiplano and Sierra regions of Bolivia anal Peru are already seriously degraded as evidenced by the generalized soil erosion. The erosion levels in the areas of interest in both countries are in the range from 50 to 80 percent. In Bolivia, almost 1.25 million people of the rural areas of the Altiplano (40 percent of the total population of the country) live under conditions of extreme poverty. Also, around 70 percent of the Peruvian Sierra and valleys population are considered in this category. The erosion is caused by overgrazing, removal of vegetation and poor agricultural practices. The dramatic population growth that began in the 1930´s and 40´s put extreme pressure on the available land. The farmers reduced the agricultural cycles from 10 to 15 years, to 5 to 7 years. Family holdings decreased in size every year, and the land was farmed and grazed even more intensively. Soil erosion led to lower yields, abandonment of some land, more overworking of the land, low incomes and further pressure on the land. The erosion problems and poverty have an inter-relationship that requires that both be solved together, in the same context. 1.4 What the Program Seeks to Achieve The strategy for achieving the objectives of the Program is a combination of three key elements: (1) provide technical assistance to NGOs in accordance with the major phases of the Project cycle for all organizations - learning new ideas and opportunities; planning how to incorporate them into existing projects or planning new projects; designing better monitoring and evaluation systems; implementing new or revised projects; evaluating, reporting and Matching Grant Program End-of-Program Evaluation 7 assessing performance; and improving the projects for the coming year; (2) create some competition as well as collaboration among NGOs to find better methods and to improve performance; and (3) partner with NGO federations and networks in Bolivia and Peru to carry out the technical assistance and help institutionalize it. The goals and objectives of the Program are specified for three groups based on the level within the Program and the place of action. These are as follows: 1.4.1 Bolivia o Objective 1 – Soils and pastures reclaimed, and production and income of 9,600 farm families increased in 320 communities served by 32 NGOs. ß Indicator 1 – Percentage increase in the average income of farm families in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 30 percent. ß Indicator 2 – Percentage increase in the average number of hectares of land per family used for crops and pasture in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 20 percent. o Objective 2 – Effective training and technical assistance services to farmers in reclaiming soils and pastures, and in increasing production and income established and sustained by the NGOs working with the Program. ß Indicator 3 – Change in average Program score of participant NGOs on a SID developed capacity development self-assessment instrument. Target: 25 percent. ß Indicator 4 – Percentage of NGOs that report results in a timely manner. Target: 75 percent. ß Indicator 5 – Percentage of NGOs that diversify their sources of funding. Target: 25 percent. ß Indicator 6 – Percentage increase in the number of farmers participating in NGO technical assistance activities (disaggregated by sex). Target: 25 percent (40 percent women). ß Indicator 7 – The Asociación de Instituciones de Promoción y Educación (AIPE), the SID Associate in Bolivia, assumes an effective technical assistance role for NGOs by conducting workshops. Target: Yes. Matching Grant Program End-of-Program Evaluation 8 1.4.2 Peru o Objective 1 – Soils and pastures reclaimed, and production and income of 6,600 farm families increased in 220 communities served by 22 NGOs. ß Indicator 1 – Percentage increase in the average income of farm families in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 30 percent. ß Indicator 2 – Percentage increase in the average number of hectares of land per family used for crops and pasture in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 20 percent. o Objective 2 – Effective training and technical assistance services to farmers in reclaiming soils and pastures, and in increasing production and income established and sustained by the NGOs working with the Program. ß Indicator 3 – Change in average Program score of participant NGOs on a SID developed capacity development self-assessment instrument. Target: 25 percent. ß Indicator 4 – Percentage of NGOs that report results in a timely manner. Target: 75 percent. ß Indicator 5 – Percentage of NGOs that diversify their sources of funding. Target: 25 percent. ß Indicator 6 – Percentage increase in the number of farmers participating in NGO technical assistance activities (disaggregated by sex). Target: 25 percent (40 percent women). ß Indicator 7 – The Coordinadora de Organizaciones Campesinas e Instituciones Agrarias del Perú (La Coordinadora Rural), the SID Associate in Peru, assumes an effective technical assistance role for NGOs by conducting workshops. Target: Yes. 1.4.3 The SID Central Office o Objective 1 - Increase SID´s capacity to provide technical assistance for NGOs in methods for helping farmers reclaim soils and pastures and increase production and income. ß Indicator 1 – Percent of NGOs requesting all phases of technical assistance after participating in planning workshops (indicator is expressed as the annual ratio of the Matching Grant Program End-of-Program Evaluation 9 number of NGOs requesting four phases of technical assistance / number of NGOs at the initial planning workshop.) Target: 66 percent. ß Indicator 2 – Percent of time SID Country Office Staff time spent on technical assistance for NGOs. Target: 50 percent. o Objective 2 – Increase SID´s capacity to help NGOs monitor, evaluate and report results, and to improve performance. ß Indicator 3 – Percent of NGOs requesting help in monitoring, evaluation and reporting after participating in planning workshop (indicator is expressed as the annual ratio of the number of NGOs requesting technical assistance for Monitoring and Evaluation / the number of NGOs at the initial workshop.) Target: 66 percent. ß Indicator 4 – Percent of SID Country Office Staff time spent helping NGOs conduct baseline surveys and analyze results, and in monitoring, evaluating and reporting their results. Target: 50 percent. o Objective 3 – Improve the capacity of SID to supervise and support field programs. ß Indicator 5 – Percent of HQ Staff time spent on field support activities. Target: 60 percent. ß Indicator 6 – Percent of monthly and annual reports submitted on time, and that are of an acceptable quality. Target: 75 percent. ß Indicator 7 – Percent of responses of HQ to field staff requests. Target: 90 percent. o Objective 4 – Improve the capacity of SID in Strategic Planning. ß Indicator 8 – Five-year strategic plan adopted by SID Board of Directors. (yes/no) Target: Yes. 1.5 The Phases of Technical Assistance To help achieve these objectives, SID and its affiliates have divided the technical assistance into six phases. These coincide with the six principal phases of the Program and are destined to support the NGOs to: o Learn more effective methods for helping farmers reclaim soils and pastures and increase incomes, using as a base the demonstration areas in Patacamaya in Bolivia and Marenass en Peru. Matching Grant Program End-of-Program Evaluation 10 o Re-plan existing projects and design new projects in a way to permit the use of the more effective methodologies and, according to the needs identified, improve the logic of their goals, objectives and results indicators. o Design better systems for project monitoring and evaluation. o Implement the new or revised projects. o Implement the improved systems for evaluating, reporting and assessing performance. o Jointly with the NGOs and Grassroot Organizations in each region, conduct annual activities aimed at the evaluation of results and problem identification, and the improvement of projects in the coming year. 1.6 Principal Partners The Program was designed to implement actions for the establishment of a technical assistance plan that incorporates new techniques of planning, implementation, monitoring and evaluation of projects, combined with community participation, for the NGOs and Grassroot Organizations so that they can better execute projects in the areas of production, food security and natural resource management in the participating rural populations in the target areas of each country. The Program strategy is a combination of four key elements: 1) A two-tiered program of technical assistance, including: ß training and low-level assistance to all interested NGOs and Grassroot Organizations in the area served by the Program, and ß intensive technical assistance in all phases of Program management to NGOs and Grassroot Organizations that currently are implementing agrarian or natural resource management project in the more limited area of concentration. 2) Provide technical assistance for NGOs and Grassroot Organizations in accordance with the six major phases of the Program cycle: ß Learning new ideas and opportunities ß Planning how to incorporate them into existing projects or planning new projects ß Designing better monitoring and evaluation systems ß Implementing the new or revised projects ß Evaluating, reporting and assessing performance ß Working together to improve projects for the coming year 3) Creating some competition as well as collaboration among NGOs to find better methods and improve performance. Matching Grant Program End-of-Program Evaluation 11 4) Partnering with NGO federations, the Asociación de Instituciones de Promoción y Educación (AIPE) in Bolivia and the Coordinadora Rural in Peru, which have the express mission of helping improve the performance of their affiliates as well as other NGOs. The global goal of the Program, as specified in the Detailed Implementation Plan (DIP), is to support activities aimed at: a) the reclamation of soils and pastures in these regions and b) the increase in incomes among the rural families that reside in the Altiplano and high valleys of Bolivia and Peru. The Program is centered on a strong commitment by SID and its Central Office, as well as that of the staffs of the Country Offices, to achieve positive impacts in the communities of the regions where they work, and to accomplish this by working with NGOs and Grassroot Organizations. The activities undertaken include a mix of various workshops and follow-up assistance efforts in the offices of the NGOs and at field sites. The work is based on the need to encounter or construct solutions to project design and implementation problems, working with the NGO staffs. Matching Grant Program End-of-Program Evaluation 12 2. Program Effectiveness 2.1 Program Model and Approach 2.1.1 Unique Features of the Program The Program is characterized by several features that are unique in the design and implementation of this type of activity. These elements lend to the high degree user demand for the services provided by the Program and to the overall great success the Program has achieved, both in Peru and Bolivia. The uniqueness of the Program is exemplified by: ß Continuity – The technical assistance and training activities of the Program are designed and programmed to accompany the users throughout the six cycles. ß Practicality – The technical assistance and training is practical in its conception, building on what the projects of the NGOs are currently doing. ß Multi-tiered – Technical assistance is provided by means of group and internal workshops with all interested parties, and individual advising to NGOs and grassroots organizations in the geographic areas of concentration. ß Learning organizations – The technical assistance promotes learning organizations, including a permanent analysis by the organization as to where is currently stands. ß Exchanges of experiences – The overall Program promotes the interchange of experiences and knowledge within and among the participating organizations. ß Voluntary process – The Program is based on the voluntary participation of each organization and a system of incremental change to meet the pace determined by that organization. ß Flexibility – The technical assistance provided by the Program is flexible in response to the needs of the organization and its projects. ß Shared costs and responsibilities – The cost of all technical assistance activities and the responsibility for their programming and coordination is shared between the SID and the partner organizations. 2.1.2 Program Justification En general, the focus of the NGOs and the Grassroot Organizations that work in rural development leads to projects that are aimed at the management of natural resources, agricultural production and food security. These constitute valid factors for the motivation and orientation of rural families to increase production, reclaim soils and pastures, and strengthen human resources – factors that establish the basis for sustainable increases in family incomes. Matching Grant Program End-of-Program Evaluation 13 In one way or another, the organizations capitalize on their experiences for the implementation of new projects based on the knowledge obtained working in the communities and the execution of other projects in this area. SID offers an interchange of experiences obtained from a variety of projects that include elements related to the management of natural resources in high-altitude ecosystems. Through a process of inter-institutional relationships, SID is achieving its objective of helping the NGOs to improve their interventions in favor of the rural farm families. 2.2 Soundness of the Program Hypothesis and Assumptions The basic hypothesis and assumptions of the Program are well founded. The idea is to reach large numbers of farm families and assist them to increase the productivity of their croplands and pastures. The Program does this by means of providing technical assistance to the NGOs and Grassroot Organizations in the various regions covered in order that they can improved the design, development, implementation, and monitoring & evaluation activities of their projects. This is meant to improve the quality of their technical assistance, the quality of the project designs, the number of projects presented to, and financed by, the various sources, and the overall results achieved by the local populations associated with the projects. The evaluation team feels that the manner in which the results indicators were defined could have been more closely tied to the hypothesis and assumptions of the Program. Several of the indicators indicate expected variations in specific variables (increases in income levels, increases en the area of cropland and pasture available for use by farm families, etc.), at the local population level. These types of results, rather than being a direct consequence of actions on the part of the technical staff of the Program, are more likely affected by variables exogenous on which the Program has no effect, or are at best affected through activities at a secondary level – actions by the NGOs and Grassroot Organizations as affected by actions of the Program staff. It is true that one of the ultimate effects expected from the actions of the Program is an improvement in the well-being of the farm families. However, to tie the evaluation of the success of a Program directly to these variables may not be the best indicators of success. 3. Achievement of Objectives The present report contains the results of an end-of-program evaluation of the “Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru”, executed by the International NGO, “Strategies for International Development (SID)”. The information presented herein is the result of the revision of the documentation provided by the SID offices, the results of data collection efforts by staff members of the two Country Offices during the period September through November of 2003 (as well as information gleaned from the Program records) and the Matching Grant Program End-of-Program Evaluation 14 interviews with directors and coordinators of the selected organizations in the two countries and of the SID Country Offices. As pointed out in the mid-term evaluation, the elements of the delivery system for technical assistance to the final beneficiaries of the Matching Grant Program are the NGOs and the grassroots organizations. SID works through these organizations in order to impact the level of natural resource recovery and the level of reduction of rural poverty among the residents of the rural communities involved. The gamut of SID activities are aimed at the NGOs and the grassroots organizations, and at the improvement of their ability to plan, organize, execute, monitor and evaluate projects at the community level. These activities are specifically targeted to those organizations that currently have funded projects in the areas of agriculture, livestock and/or natural resources. A large number of NGOs and grassroots organizations in both Peru and Bolivia expressed their interest in participation during the life of the Matching Grant Program. SID actively pursued the inclusion of as many as possible of these organizations in the Program as they completed qualification under the established criteria and as Program resources permitted. Matching Grant Program End-of-Program Evaluation 15 3.1. Results of the Bolivian Program 3.1.1.Overall Results In general the Bolivian Country Office has reached a very large number of NGOs and Grassroot Organizations during the life of the Program. Through this, it has affected the lives and levels of living of large numbers of farm families in the Altiplano and high valleys of Bolivia. The number of general meetings conducted by the SID-Bolivia staff during this period, by department, is shown in Table B-1. These meeting were held to orient and inform the candidate institutions as to the scope and purpose of the Matching Grant Program and to explore the needs and desires of the organizations to take part in the activities. As results of these meetings a total of 35 organizations participated in the Bolivia Matching Grant Program. The Program also conducted events specifically targeted to individual institutions. The second column of Table B-1 contains data showing the numbers of this type of events, by department, for the life of the Program. It was by means of these events that the SID-Bolivia staff provided topic-specific technical assistance to the individual NGOs and Grassroot Organizations, based on their stated needs. Table B-1. Number of Events conducted by the SID-Bolivia Staff during the Life of the Program, by Department Department General Meetings Institutional Events Total La Paz 9 32 41 Oruro 7 35 42 Potosí 5 18 23 Chuquisaca 3 10 13 Cochabamba 2 9 11 Total 26 104 130 Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. 3.1.2.Reactions to the Midterm Evaluation A mid-term evaluation of this Program was conducted in August-September 2002 with the preliminary results presented to the staff members of the two Country Offices in Copacabana, Bolivia on September 26 of that year. A final, revised report document was presented to SID in December 2002. Based on the information received during this presentation, SID-Bolivia effected several adjustments in the direction of their segment of the Program. These included the following reformulation of specific objectives and strategies: Matching Grant Program End-of-Program Evaluation 16 ß Objectives o Improve the quality of the technical assistance that SID-Bolivia offers to the participating institutions. o Expand the coverage of the SID-Bolivia Program to institutions that are implementing projects related to economic-productive and environmental themes and that accept the procedural work that should be developed in these projects. ß Strategies o The workshops presented should be considered as high quality by both the institutions working with SID and the invited institutions. o The promotion and sale of projects should be much more aggressive and constant in relation to other areas. o The Pachamana Urupa Project should continually support the natural resources component of the Program. o Contract an additional technician if the demand for technical assistance merits. o Expand the client coverage by means of general workshops in La Paz, Oruro, Potosí, the Northern Potosí Region, Chuquisaca and Cochabamba, during the months of January and February 2003. o Plan actions to continue with the 13 client institutions, taking advantage of the next internal workshop to concentrate on the desired procedural focus. o Generate an austerity policy within the participating institutions that will help them to finance the travel costs of the SID-Bolivia technical assistance personnel. 3.1.3 Results Indicators The following section contains information related to the progress made by the SID-Bolivia Program Office towards the achievement of each of the results indicators that were established at the outset of the Program. Table A-1 of Annex A contains the Program Logical Framework in which are listed each of the objectives, indicators, goals, measurement specifications and principal activities for the Bolivian Country Office. Objective 1 – Soils and pastures reclaimed, and production and income of 9,600 farm families increased in 320 communities served by 32 NGOs. The indicators for this objective are measures of indirect results of the activities of the SID offices, reflecting the impact of the specific activities of the NGOs and Grassroot Organizations as related to their work with the communities and the farm families in their areas of influence. Indicator 1 –Percentage increase in the average income of farm families in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 30 percent. Matching Grant Program End-of-Program Evaluation 17 Data on growth of farm family incomes during the period of the Program is available for twelve of the participating institutions. As reflected in Figure B-1, the average increase through September 2003 is Bs 86. This is a 23 percent increase, slightly lower than the 30 percent target set in the indicator. Figure B-1. Increase in Average Farm Family Income 367 453 86 0 50 100 150 200 250 300 350 400 450 500 Baseline September 2003 Increase Bolivianos Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Indicator 2 - Percentage increase in the average number of hectares of land per family used for crops and pasture in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 20 percent. Baseline data was obtained for this indicator for 13 of the 35 participating NGOs and Grassroot Organizations. See Figure B-2. This data indicates that the average farm family in the communities of the reporting organizations had approximately 1.58 hectares of productive cropland and pastures prior to the beginning of the Program. By September of 2003, this figure had increased to 2.29 hectares. This is an absolute increase of 0.93 hectare – approximately 58 percent above the baseline figure and well beyond the targeted increase of 20 percent. An associated number that was derived from the September 2003 effort shows that for nine reporting institutions, the productivity also was improved on an average of 106 hectares of communal lands. Matching Grant Program End-of-Program Evaluation 18 Considering these results, the slightly less than targeted increase in family incomes will surely improve in the short-run as the effect of the increased amount of productive land is manifested in increased production. Figure B-2. Increase in the Hectares of Productive Crop and Pastureland, per Farm Family 1,58 2,29 0,93 0 1 2 3 Baseline September 2003 Increase Hectares Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Thirty-five NGOs and Grassroot Organizations are listed in Tables B-2 and B-3, and were taken into consideration in the current evaluation. These include nine institutions in the department of Oruro, eight in the department of Potosí, five in the department of Chuquisaca, four in the department of Cochabamba and nine in the department of La Paz. This number exceeds by three the 32 institutions targeted in the Objective. The data in Table B-2 provides an indication of the scope of the work done by the SID Bolivian Office. Therein are listed the names of the NGOs and Grassroot Organizations that have worked with the Bolivian Office of SID during the life of the Program, and the numbers of participating communities and rural families that have benefited from the projects managed by the NGOs and Grassroot Organizations during their participation in cooperative agreements signed with the Program. This is the level of indirect coverage achieved by the Program. The progress made by each institution is directly related to the interest of the technical personnel and the directors of the institution, the status of the project(s) at the time of Program intervention, the duration of the project(s) and the Program, and the number of new projects for which the institution received financing during the life of the Program. These numbers of communities and participant families are results of highly motivated technical Matching Grant Program End-of-Program Evaluation 19 teams at the institutional level that have been able to introduce attitudinal changes. This also reflects that the pace of change is gradual in the type of participants prevalent in the target populations of the projects. Table B-2. Organizations and Families that Benefit from the Bolivian National Program No. Name Department Communities Families 1 PCI Oruro 44 1200 2 UMAG Oruro 25 700 3 PDA Wiñaypaj Oruro 27 1700 4 Caritas Oruro-Project 1 Oruro 6 480 5 CASE Oruro 52 270 6 CIPDA Oruro 3 240 7 SEMAC Oruro 3 68 8 CETHA Socamani Oruro 8 300 9 PDA Soracachi Oruro 25 875 10 PDA Chayantaca Potosí 55 2800 11 PDA Aymaya Potosí 35 1225 12 PDA Layme Potosí 52 1165 13 CAD Potosí 12 380 14 ATEC * Potosí n.a. n.a. 15 Caritas – Potosí * Potosí 80 2,880 (e) 16 SEPAS * Potosí n.a. n.a. 17 ISALP * Potosí 47 1,700 (e) 18 ETAPAS * Chuquisaca n.a. n.a. 19 IPTK * Chuquisaca 18 390 (e) 20 FIA * Chuquisaca 23 1,100 21 Fundación Treveris * Chuquisaca 120 2,000 22 Red UNISUR * Chuquisaca 23 499 (e) 23 PDA Bolívar Cochabamba 45 778 24 Radio Esperanza Cochabamba 11 219 25 PDA Los Vecinos * Cochabamba 196 (e) 3,500 26 PDCR * Cochabamba 34 595 (e) 27 Qhana La Paz 48 1,120 28 Caritas LP-Project 1 La Paz 30 70 Caritas LP-Project 2 La Paz 16 500 Caritas Corocoro * La Paz 18 390 29 Kurmi La Paz 32 1,500 30 Ayuda en Acción-Caranavi La Paz 32 550 Ayuda en Acción-Coroico La Paz 38 600 31 CIPCA La Paz La Paz 16 106 32 CEPROMU La Paz 10 346 33 CEPRODA La Paz n.a. n.a. 34 CIEC * La Paz n.a. n.a. 35 OECA * La Paz n.a. n.a. Totals 1,184 30,246 Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. n.a. – Not available. (e) – Estimado. * - Organizations added in the final months of the Program. Matching Grant Program End-of-Program Evaluation 20 As a further indication of the level of commitment and the progress made by the staff of the SID-Bolivia Office, Table B-3 contains information related to the advancement of the organizations during the life of the Program in the six phases of technical assistance offered by the Program. Table B-3. Program Progress of Bolivian NGOs and Grassroot Organizations Phases of Technical Assistance No Organization Department 1 2 3 4 5 6 1 PCI Oruro X X X X X 2 UMAG Oruro X X X X 3 PDA Wiñaypaj Oruro X X X 4 Caritas Oruro-Project 1 Oruro X X X X X 5 CASE Oruro X X X 6 CIPDA Oruro X X X X 7 SEMAC Oruro X X X X 8 CETHA Socamani Oruro X X X X 9 PDA Soracachi Oruro X X X 10 PDA Chayantaca Potosí X X 11 PDA Aymaya Potosí X 12 PDA Layme Potosí X X X 13 CAD Potosí X X X X X 14 ATEC * Potosí X X 15 Caritas – Potosí * Potosí X X 16 SEPAS * Potosí X 17 ISALP * Potosí X X 18 ETAPAS * Chuquisaca X X 19 IPTK * Chuquisaca X X 20 FIA * Chuquisaca X X 21 Fundación Treveris * Chuquisaca X X 22 Red UNISUR * Chuquisaca X X 23 PDA Bolívar Cochabamba X X 24 Radio Esperanza Cochabamba X X X 25 PDA Los Vecinos * Cochabamba X X 26 PDCR * Cochabamba X X 27 Qhana La Paz X X X 28 Caritas LP-Project 1 La Paz X X X Caritas LP-Project 2 La Paz X X X Caritas Corocoro * La Paz X X 29 Kurmi La Paz X X X X 30 Ayuda en Acción-Caranavi La Paz X X Ayuda en Acción-Coroico La Paz X X 31 CIPCA La Paz La Paz X X X X 32 CEPROMU La Paz X X X X 33 CEPRODA La Paz X X X 34 CIEC * La Paz X 35 OECA * La Paz X Number of Institutions 16 35 28 14 4 0 Percent of Total 45.7 100.0 80.0 40.0 11.4 0.0 Matching Grant Program End-of-Program Evaluation 21 Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. * - Organizations added in the final months of the Program. All of the NGOs and Grassroot Organizations participating in the Program manage projects dealing with the central topics of the Matching Grant Program – agriculture and natural resources. Based on the interviews with the selected organizations during the mid-term evaluation, it was determined that 88.9 percent of the organization also managed projects that were not directly related to SID activities – that is, projects that did not involve activities related to the recovery of soils and pastures nor to the increase of production and incomes. Nevertheless, these were projects that included elements directed at the reduction of poverty in the broad sense of the word – economic, social and political development, democratization, and gender and generation issues. Realizing that the technical assistance provided by SID and the resultant changes in the client organizations bring about positive impacts in the design, implementation and evaluation of these projects, the communities and rural families that gain from these projects are often reported by the organizations as being beneficiaries of the Matching Grant Program. Of the 35 Bolivian organizations included in Table B-3, a total 45.7 percent participated in general and/or internal workshops of Phase 1 of the Program –“learning new methods.” All of these organizations participated in one or more workshops related to the planning of better projects – Phase 2. As for Phase 3 –“the design of better evaluation systems” – 80.0 percent participated one or more workshops. At the same time, 40.0 percent of the organizations participated in activities related to the “execution of new projects” - Phase 4. It is also observed that four of the organizations of the Program advanced to the Phase 5 activities, “implementation of improved systems for evaluating, reporting and assessing performance”. None of the institutions reached Phase 6 of the Program. At this point it is appropriate to mention that the numbers of NGOs and Grassroot Organizations included in the tabulation for SID-Bolivia encompasses only those that were selected for assistance under the established selection criteria. It is estimated that at least half of the organizations that requested assistance were not eligible to participate in the Program because at that time they did not have agriculture or natural resource projects, or because they were not NGOs. This is to point out the great potential of this Program to reach many more clients and beneficiaries should the effort be continued. The combined result of the impressive number of NGOs and Grassroot Organizations with whom SID worked in Bolivia and the fact that these organizations advanced so rapidly in the six-phase program of technical assistance, is evidence that the Country Office was well focused in their application of the selection criteria for organizations and that their program for working with individual organizations was functioning well. Time limitations were critical in not being able to achieve more. Matching Grant Program End-of-Program Evaluation 22 Objective 2 – Effective training and technical assistance services to farmers in reclaiming soils and pastures, and in increasing production and income established and sustained by the NGOs working with the Program. Indicator 3 – Change in average Program score of participant NGOs on a SID￾designed instrument for the self-assessment of capacity development. Target: 25 percent. The data utilized by the evaluation team consists of that gathered by the SID Country Office and information provided by the Central Office. Much of this data is based on interviews and inter-institutional workshops conducted in 2002 and 2003. Additionally, the evaluation team conducted interviews with the remaining Country Office Staff members during visits in late November and early December 2003. Final confirmation and adjustments to the data were effected by means of internet and telephonic communication between members of the Evaluation Team and the staffs of the Country Offices during December 2003. Data collected from the participating institutions indicate major advancements in all areas of project design, implementation, monitoring and evaluation. Eighty-five percent of the NGOs and Grassroot Organizations reporting indicated increases in the use of monitoring and evaluation techniques and in the timely production of project results. In the case of the 20 institutions interviewed in the internal evaluation, 100 percent reported that they had improved their methods of monitoring and evaluation and that their reports are being delivered to the cooperating organizations in a timely manner. Information displayed in Figure B-3, Project Monitoring, shows that 35 percent of the institutions that participated in the SID-Bolivia Technical Assistance Program indicate that their current monitoring activities involve the technical team and the target population – a positive tendency supporting the sustainability of the projects involved. Figure B-3. Project Monitoring Technical Team 30% Director/Project Manager 35% Team/local Population 35% Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 23 An equal proportion (35%) reported that the director of the institution carries out monitoring activities in coordination with the project manager, while 30 percent reported that the project technical team performs project monitoring. The data for Project Evaluation, as shown in Figure B-4, reflects a somewhat better result. Forty-five percent of the institutions are performing evaluations with the participation of the target populations. Figure B-4. Project Evaluation Directors 10% Technical Team 25% Team/local Population 45% Consultant/Team 20% Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. This is a sign of marked improvement in the tendency to achieve real participation of the local population in the decision making process, increasing project sustainability. A consultant with the support of the project technical team performs the project evaluations of another 25 percent of the institutions, while only 20 percent reported that external consultants working alone conduct their evaluations. Ten percent of the institutions reported that the directors of the organization perform their evaluations. The data in Figures B-5 and B-6 reflect the perceived utility of the technical assistance received by the NGOs and the Grassroot Organizations. The data in Figure B-5 depicts the percentage of the participating institutions that report the use of specific planning techniques and instruments promoted by the Program. Among the principal variables mentioned by the institutions are the Logical Framework (80%), Work Plan (65%) and Strategic Planning (50%). According to the opinions expressed by the technical teams of the participating institutions, the planning aspects of major utility include: 1) participatory methodologies (85%), project Matching Grant Program End-of-Program Evaluation 24 planning (35%), monitoring and evaluation (25%), and field demonstrations (20%). The full range of these aspects is depicted in Figure B-6. Figure B-5. Percentage of Institutions that Use Planning Techniques and Instruments 80% 65% 50% 0% 20% 40% 60% 80% 100% Logical Frame Work Plan Strategic Planning Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Figure B-6. Planning Aspects of Major Utility to Participating Institutions 85% 5% 5% 35% 25% 5% 10% 10% 10% 20% 0% 20% 40% 60% 80% 100% Partipatory Methodologies Interpersonal Relations Logical Frame Project Planning Monitoring & Evaluation FODA SAAR Competition Learning by Doing Field Demonstrations Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Indicator 4 – Percentage of NGOs that report results in a timely manner. Target: 75 percent. Institutions that participate in the Matching Grant Program do not provide regular reports on their activities to the SID Country Office. However, they do prepare and submit reports to Matching Grant Program End-of-Program Evaluation 25 their cooperating institutions and to the organizations that provide financing for their projects. Of the twenty-three cases analyzed in the internal evaluation, 52 percent of the institutions described improvements in the analysis contained in their reports. On the other hand, 70 percent indicated progress in the timely presentation of reports to their cooperating institutions. Indicator 5 – Percentage of NGOs that diversify their sources of funding. Target: 25 percent. Data from the September 2003 interviews with 23 of the participating organizations reflected that 14 had increased the number of active projects. For these fourteen, the number of projects being implemented increased from an average of two to three – a total of 21 more projects than before the Program began. Comments made by the institutions during interviews during the mid-term evaluation include: ÿ As a result of the technical assistance provided by SID, we are now able to produce saleable project proposals. ÿ The NGOs can now participate successfully in response to requests for proposals (RFPs). ÿ The project planning and design methodology taught by SID has helped NGOs approach financial institution with more confidence. ÿ The assistance provided by SID is useful in training personnel in the planning and design of projects, and in negotiating the funding of projects with financing organizations. These results can be further analyzed by means of the number of institutions that reported adjustments in the design and implementation their projects, incorporating guidance provided through the technical assistance received. See table B-7. Figure B-7. Percentage of Institutions that Use Planning Techniques and Instruments 60% 40% 40% 35% 65% 0% 20% 40% 60% 80% 100% Indicators Objectives Monitoring Strategies & Methods Workplan Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 26 All of the twenty institutions analyzed in this case indicated that they had made adjustments in one or more areas to improve the definition of project indicators, the specification of project activities and the definition of strategies and methodologies. Indicator 6 – Percentage increase in the number of farmers participating in NGO technical assistance activities (disaggregated by sex). Target: 30 percent (40 percent women). This indicator is used to measure the quality of service being provided, under the assumption that the delivery of good quality services will generate demand for additional services. The numeric values presented in Table B-4 indicate the targets for the Country Office and the results achieved, related to the number of NGOs, number of participating communities, number of participating rural families and the percentage participation of women. Table B-4. Results for Indicator 6 of Objective 2 for the Bolivian Country Office Indicator Target Progress￾to-date Percent of Target Number of participating NGOs 32 35 109% Number of participating communities 330 1,184 358% Number of participating rural families 9,600 30,246 315% Percentage of women participants 40% 44% 110% Source: Prepared from data provided by SID-Bolivia in the form of unpublished information tabulated from their internal evaluation conducted in September 2003, La Paz, Bolivia, and additional information collected during the preparation of this report. The number of organizations in Bolivia that have received technical assistance of some type from the Program reached 35 by September 2003. At the same time, the number of participating communities was 1,184, and the estimated number of families benefiting from the Program activities of these organizations grew to 30,246. These numbers, when compared to the 320 communities and 9,600 rural families targeted in the PID, are evidence that SID Bolivia has by far bested the expected level of effort. The Bolivian Country Office is reaching three and one-half times the targeted number of communities and over three times the goal of the number of rural families. At this point it is necessary to reemphasize one of the unique features of this Program – that of the multiplier effect of the technical assistance provided by SID through the NGOs. It is this detail that permits SID to impact such a large number of final beneficiaries. On the other hand, nineteen of the twenty interviewed organizations described visible advances in women´s participation in community and Program activities over the last few years. The participation of the women in the projects managed by the interviewed organizations of the Program is extremely variable ranging from 20 to 80 percent – up from a range of 10 to 60 percent prior to the initiation of Program activities. With a target of 40 Matching Grant Program End-of-Program Evaluation 27 percent, the data in Table B-4 shows an average increase of 44 percent in women´s participation for Program components of the Bolivian Country Office. These overall values indicate that the Program activities have achieved the desired positive impacts in this area. All of the organizations agree that the activities of the SID Program, working through their projects, are having a positive impact. The following are among the more important comments made by the Bolivian NGOs and Grassroot Organizations during the mid-term evaluation: ÿ The traditional culture of the communities is changing; women are taking a more active role in meetings and decisions; they are active in the irrigation and seed organizations, cooperatives and associations. ÿ Many of the men in this region are immigrating in search of work; there are more women than men in the workshops; our projects include components for women; we use women as promoters, especially in the formation of leaders and in the “rural schools”. ÿ The participation of women is very important in activities related to agriculture and livestock production; however, their participation is hindered by the fact that they are very busy; it is difficult for them to participate (to compete) in meetings with men. ÿ Activities related to women are being strengthened, especially in themes concerning project planning from a female perspective. ÿ Workshops are conducted at night and on weekends so that women can participate. Indicator 7 – The Asociación de Instituciones de Promoción y Educación (AIPE) assumes an effective technical assistance role for NGOs by conducting workshops. Target: Yes. The Asociación de Instituciones de Promoción y Educación (AIPE) was originally identified as the associate for the Bolivian Country Office. This organization consists of more than 32 NGOs in the several regions where it operates. The mission of AIPE is stated as to, “strengthen its associates as agents of development and change; coordinate actions for the promotion, organization and participation in a global society, in search of important solutions that contribute to integral social development in a manner that is will be sustainable, participative and just”. The thematic thrust of AIPE includes the environment, gender, culture and human rights. This organization has never been an active partner in the Program. The Matching Grant Program began attempting to coordinate with AIPE during the first semester of the year 2001. Although personnel of the Bolivian Country Office have made continued efforts to involve AIPE in the Program and to obtain its expanded support, it is the opinion of the Country Office director that the configuration and interests of AIPE are not fully compatible with those of the Program. AIPE´s regional concentration in the Bolivian eastern lowlands and a somewhat non-agricultural focus tends to limit the effectiveness of this organization as an associate. Matching Grant Program End-of-Program Evaluation 28 On the other hand, SID Bolivia has worked with other strategic partners who finance numerous NGOs and projects: the Unidad de Seguridad Alimentaria de la Comunidad Europea (USACE) in the department of Oruro, the Plataforma de Suelos and Ayuda en Acción in the department of La Paz, and World Vision in both of these departments. All have had strong relationships with SID Bolivia. These organizations have participated in general planning workshops and in internal planning workshops for their programs. In the context of partners, SID Bolivia has not only working with programs in the field, but with the national directors for the different programs. This represents the potential for helping to influence change on a national level as well as to transfer SID responsibilities to different institution. 3.1.4 Inter-institutional Evaluation The inter-institutional evaluation conducted by SID-Bolivia in September 2003, resulted in a number of valuable points related to the design, development, implementation and impact of the Program. The most important of these are listed in Table B-5. Table B-5. Utility of the SID-Bolivia Technical Assistance Institutional Management Project Management Strengthening Institutional Capacities Impact on Local Populations Utility 1) Periodic reports more precise, clear, technical and standardized. 2) Reports to financial institutions based on results indicators. The monitoring system contributes to this, generating more interest and feedback. 3) Improved coordination between institution directors and the affiliate organizations. 4) Works with communities to identify problems, to insure that actions focus on realistic solutions, means of measurement. 1) More participation in project planning - includes local population, municipal government, institutions and technicians. 2) Utilization of improved log frames, strategies and M & E system. 3) SID technical assistance served to improve the writing quality – clear and objective. 4) New projects employ tools such as the problem tree. Makes the formulation of indicators easier. 5) Methodology born in the community result in instrument for community. Creates harmony. 1) Technical team more committed to the task, serious about work. Members work toward the common team goal 2) Self-diagnostic results in documentation of lessons learned. Contributes to team knowledge and commitment. Technical team no longer waits for external evaluation. 3) Evaluations aimed at quality - not quantity. Look at benefits produced, not just at results achieved. 4) M&E training permits technical team to improve work, fine tune activities, differentiate between follow-up and M&E, clarify concepts and manipulate tools. 5) TA helps technical teams overcome methodology fears. 1) Increased participation (leadership) of women in all community processes. 2) Self-diagnostics demonstrate changes in attitudes and greater community participation. 3) Incomes increased 5% or more in many communities. Projects justified more on a social than economic basis. 4) Project activities result in soil and pasture recovery, harvesting of water and conservation methods. 5) Working with community generates high level of confidence. Live, eat and work together - all as equals. A meticulous task, person￾to-person. Changes not always perceptible. Ways to Improve Usefulness 1) Planning and monitoring software. 2) “Tesistas” in investigations. 3) More documentation in M&E and marketing. 4) “Management by Results” methods. 1) TA for programs and projects to include increased follow-up in search for common objectives. 2) Increased participatory M&E at community level. 3) “Objective methods” in income calculations. 1) Reinforce strategic project planning, M&E, log frame and documentation of experiences. 2) Project implementation: soil & pasture recovery, product processing, baseline studies and results indicators. 3) Conflict resolution, participatory assemblies, interchange of experiences. 1) Work with Andean planning for illiterate persons 2) Use tools and techniques appropriate for local populations. 3) Expand technical assistance to include more Grassroot Organizations, not just NGOs. Source: Prepared from information contained in the documentation of interviews conducted with participating institutions during the internal evaluation in the period August-September 2003. Matching Grant Program End-of-Program Evaluation 29 3.1.5 SID-Bolivia Staff Input The Director of the SID-Bolivia Matching Grant Program provided important insight related to a number of important factors in evaluating its results and impact. The following points are considered to be of primary interest: ß The Matching Grant Program provided technical assistance to the participating institutions that helped them to better define the objectives of their projects, clarify the concepts involved in project design and development, and think through the planning activities by means of the use of improved techniques and tools. ß The Program placed emphasis on project planning and on monitoring & evaluation. The institutions that joined in the late stages of the Program did not have sufficient time to take advantage of the full scope of the Program. The staff time was insufficient to bring these institutions up to speed in all of the Program areas – staff was not willing to accelerate the process because this would have resulted in a loss of Program quality. Each member of the team worked at a different pace. Overall, the Program activities were about 6 to 12 months behind schedule. ß An active counterpart institution for the Program never developed. AIPE never fully understood the Program and, since they did not need technical assistance for their own activities, the several meetings with them never developed an interest. The relationship between SID-Bolivia and AIPE was not one that would have helped the Program. Relationships with other partners (USAGE, Platforma de Suelos, Ayuda en Acción and World Vision) proved to be fruitful. ß Following the adjustment made during year two of the Bolivian Matching Grant Program, the emphasis was changed from one of recuperating crop and pasturelands to activities more closely related to the overall objectives and goals of the Program – that of providing technical assistance to participating NGOs and Grassroot Organizations in the areas of project planning, monitoring & evaluation, etc. Cost sharing was emphasized with those participating institutions that were considered to be able to afford it. ß The SID-Bolivia staff observed substantial evidence of professional mobility among the employees and management of the participating institutions. This is a positive aspect in that the new concepts and skills acquired by these participants are being proliferated in other institutions and geographical areas. ß The concepts and skills learned by the professionals of the participating institutions are being applied to activities of a broad range of projects in their portfolio – not only in the projects related to the Program’s scope of work. Matching Grant Program End-of-Program Evaluation 30 ß The professional staffs of the participating institutions, as well as the staff of SID-Bolivia, have become over time more motivated and focused, permitting them to attack problems, analyze alternative solutions and affect project adjustments more rapidly and effectively. ß A more effective field organization for the bi-national Program may have been to employ a highly experienced professional as the Regional Director with less experienced country directors. ß An additional one and one-half years would be required for the Program to more nearly reach its overall goals. During this period the Program should do more institutional appraisal; additional institutions should be required to contribute to activity costs, while other need assistance with their budgets. This also would permit more of the participating organizations to further advance in phases 5 and 6 of the Program. Matching Grant Program End-of-Program Evaluation 31 3.2 Results of the Peruvian Program 3.2.1.Overall Results The technical assistance team of SID-Peru prioritized the strengthening of the capabilities of the technical teams of the participating NGO and Grassroot Organizations in order that they, through changes in their own attitudes, were able to awaken in the target population the creativity necessary to discover solutions to the problems presented in each community. During the implementation of the Program, the SID-Peru Country Office provided technical assistance under two different modalities. First, working under an umbrella agreement with the Peruvian Rural Coordinator, personnel of this Office conducted General Meetings in six departmental capitals (Arequipa, Puno, Cusco, Apurimac, Ayacucho and Junín); see Table P￾1. The regional offices of the Coordinator were responsible for the organization of these events, and the costs were shared with SID-Peru. Participants in the events included NGOs (both members and non-members of the Coordinator), public institutions, as well as professionals and non-professional from other areas. The purpose of these events was to motivate institutions with interest in participating in intensive technical assistance at the institutional level, to initiate coordination activities leading to the signing of cooperative agreements with SID-Peru. Table P-1. Participation in General Meetings, by Department, Accumulated through September 2003. Department Number of Participants Number of Workshops Level of Effort (Person-days) Apurimac 123 3 342 Cusco 139 4 396 Puno 189 3 434 Arequipa 86 2 258 Junín 118 3 230 Ayacucho 115 3 319 Total 770 18 1,979 Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que ejecutan proyectos agropecuarios y medio ambientales”, Cusco, Perú, Noviembre 2003., and additional information collected during the preparation of this report. Additionally, SID-Peru provided assistance on a cost-share basis to the Arequipa Regional Office of the Ministry of Agriculture in a general workshop on project planning. A total of 76 participants attended this event. Sixty-four participants from public and private institutions attended two events on participatory project planning and budgeting during the “Roundtable for the Fight Against Poverty” held in Puno. Finally the Manco Capac-Puno Municipal Consortium, including six Districts, signed an agreement for technical assistance in three events related to strategic planning in the development of district development plans. Matching Grant Program End-of-Program Evaluation 32 Teams from each of the six districts attended these training events. They then directed the formulation of community plans with the input of the population and local associations in each planning area, culminating in strategic development plans for each district. On the other hand, the staff of SID-Peru conducted intensive institutional workshops in the headquarters of the NGOs and Grassroot Organizations. Table P-2 contains a summary of the number of events, number of participants and the level of effort expended in these workshops. Table P-2. Institutional Workshops – Number of Participants, Number of Events and Level of Effort, by Department and Institution, Accumulated through August 31, 2003. Department and Level of Effort (Person-days) Institution Number of Participants Number of Events Phase 2 Phase 3 Phase 4 Phase 5 Total Apurimac 98 39 143 171 135 115 564 CEDES 15 7 28 16 22 15 81 IDMA-Abancay 10 6 21 18 17 19 75 IIDA 16 9 23 14 24 29 90 IRD 16 5 8 12 27 24 71 CEPRODER 26 7 26 84 45 4 159 CICCA 15 5 37 27 - 24 88 Cusco 130 49 221 162 158 197 738 IAC 19 10 21 22 58 37 138 PAC 24 8 28 33 76 85 222 PEJ 24 2 36 30 - - 66 RUNAMAQUI 10 3 12 18 14 7 51 IMAGEN 13 10 10 12 4 21 47 CESA 11 10 25 23 6 35 89 INCA 13 1 26 - - - 26 FARTAC 14 2 33 - - - 33 Plan International 11 3 3 24 - 12 66 Puno 98 20 13 41 71 38 280 Caritas – Juli 29 6 39 20 26 19 104 AMUDA 21 5 39 - - - 39 WIÑAY 14 5 15 7 19 19 60 SI 8 1 21 - - - 21 ADRUPMA 12 1 16 - - - 16 FEDICAPU 14 2 - 14 26 - 40 Arequipa 32 2 79 - - - 79 Ayuda en Acción 20 1 52 - - - 52 El Taller 12 1 27 - - - 27 Tacna 12 2 30 30 - - 60 MALLKU 12 2 30 30 - - 60 TOTAL 379 112 603 404 364 350 1,721 Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, Noviembre 2003., and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 33 These workshops were initiated once the organization signed the cooperative agreement. The purpose of the workshops was to provide technical assistance in the areas of: 1) analysis and adjustments to project planning; 2) design and adjustments to the monitoring and evaluation plan; 3) direct technical assistance in project implementation; and 4) direct technical assistance in implementation of the monitoring and evaluation plan. Of the total of 112 events in which the SID-Peru staff participated, 46 percent were held in Cusco, 35 percent in Apurimac and 18 percent in Puno. The data related to the level of effort reflects that 44 percent of the events were dedicated to themes related to monitoring and evaluation, while 35 percent were in the area of project planning and 21 percent to technical assistance in project implementation. The latter of these numbers reflects the short period of time available for dedication to this activity – not a lack of experience of the technical assistance team in providing support for problem solving in the diverse situations. The stated strategy of the SID-Peru Country Office is “Analysis and Reflection”, and the methodology employed is “Learn by Doing”. In this context, the staff of SID-Peru has worked under the idea of flexibility in the implementation of their activities, responding to the existing situation of each institution, to the projects being addressed, and to the requirements of the technical staffs. The SID-Peru staff has always utilized its experience in achieving the goals of each event that it facilitates, and to the maximum extent possible, in permitting the technical staffs of the participating institutions to apply the lessons learned in practical work with the populations in their areas. Table P-3 contains summary data that reflects the total number of events conducted by the SID-Peru staff during the life of the Program. In phases 2 and 3 of the Program general meetings were held in six departments. On the other hand, institutional workshops were conducted in phases 2 through 5 of the Program for those institutions that signed cooperative agreements with SID. Table P-3. Number of Events conducted by the SID-Peru Staff during the Life of the Program, by Department Department General Meetings Institutional Events Total Apurimac 3 39 42 Cusco 4 48 52 Puno 3 20 23 Arequipa 2 2 4 Tacna - 2 2 Junín 3 - 3 Ayacucho 3 - 3 Total 18 111 129 Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 34 During the life of the Program, SID-Peru has conducted two Inter-institutional Evaluation Workshops – in December 2002 and September 2003. The purpose of these activities was to analyze the progress made to date and the quality of the technical assistance provided by the SID-Peru Staff. The action plan for 2003 was developed taking as a point of departure the results of the Inter-institutional Evaluation of December 2002 and the conclusions and recommendations of the external mid-term evaluation conducted in mid-2002. This plan was oriented toward overcoming some of the weaknesses identified in these evaluations and to reinforce some of the successes achieved to that pointing time. The philosophy of the SID-Peru office during the implementation of the Program has been centered on the following elements: 1) exemplary professionalism and commitment on the part of the SID-Peru technical staff, permitting the provision of high-quality technical assistance, practical solutions to problems, and full dedication to the task at hand without reservation; 2) execution of events in response to requests of the interested parties, attending to each client according to the expressed needs; 3) implementation of events on a cost-share basis – the requesting institution paying the costs of the SID-Peru staff (transportation, food and board) for institutional workshops in the Program area and, in the case of the general meetings, cost-sharing with the Regional Offices of the Rural Coordinator on an average of 50 percent each; 4) a high degree of commitment on the part of the requesting institution in order to guarantee de participation of the technical personnel involved in the project to which technical assistance is being provided and the respective practical application of the concepts in the target population; and 5) a positive attitude toward change on the part of the technical staffs and in their activities with the target population is a key factor in the success of the projects. The overall success of the Program can be attributed to a good combination of these elements. The failure to achieve an even greater level of success in sustainable changes can be ascribed to the lack of time available to the Program staff. 3.2.2 Reactions to Mid-term Evaluation A mid-term evaluation of this Program was conducted in August-September 2002 with the preliminary results presented to the staff members of the two Country Offices in Copacabana, Bolivia on September 26 of that year. A final, revised report document was presented to SID in December 2002. As a result of this evaluation SID-Peru effected several adjustments to their Program. Among these fine-tuning modifications were: o Increased communications with participating organizations, including meetings at the director level each 3 or 4 months and consultations between technical teams, permitting the clarification of doubts and the orientation of individualized assistance for each NGO or Grassroot Organization. o Improved feedback from the implementing teams by means of: ß Manuals and information sheets dealing with Project Planning, Strategies and Methodologies, Monitoring and Evaluation, and Documentation; Matching Grant Program End-of-Program Evaluation 35 ß Evaluation sheets provided to the participating organization at the end of each institutional workshop; and ß Written comments related to the Program advancement provided to the SID technical team. o Increased field trips by the SID technical staff to accompany the implementation teams to promote the application and adoption of lessons learned and the resolution of bottlenecks in the field. o Increased depth of project analysis by the technical teams and local population to promote the implementation strategy and the search for new creative solutions. o The incorporation of Andean culture components into the training and technical assistance activities. o Improved coordination within the SID-Peru technical team, including training space, interchange and discussion of ideas, and the follow-up of technical assistance progress. o Better follow-up for projects by means of detailed reviews of reports and related materials on previous events, study of the work atmosphere, and discussions among the advisors. o Monthly update of data, review of baseline information and SID-Peru Program monitoring. o Incorporation in the SID-Peru multi-disciplinary team of a new professional to attend to the demand for technical assistance in the area of agricultural projects in the Southern Highlands. o Programming of two workshops related to rural development topics (priority themes for each region) – one each year in Puno, Cusco, Arequipa, Abancay, Ayacucho and Huancayo. o Demand the provision of information and the programming of required time for the successful implementation of technical assistance events. o Promote the interchange of experiences among participating projects and related specific themes. o Documentation of technical assistance experiences and their effects and impact, based on the information provided by the participating organizations and on the SID monitoring activities. o Improved coordination with thematic networks, related groups and governmental entities permitting SID-Peru contributions to themes and efforts in the promotion of rural development. 3.2.3 Results Indicators The following section contains information related to the progress made by the SID-Peru Program Office towards the achievement of each of the results indicators that were established at the outset of the Program. Table A-2 of Annex A contains the Program Logical Framework in which are listed each of the objectives, indicators, goals, measurement specifications and principal activities for the Peruvian Country Office. Matching Grant Program End-of-Program Evaluation 36 Objective 1 – Soils and pastures reclaimed, and production and income of 6,600 farm families increased in 220 communities served by 22 NGOs. The indicators for this objective are measures of indirect results of the activities of the SID offices, reflecting the impact of the specific activities of the NGOs and Grassroot Organizations as related to their work with the communities and the farm families in their areas of influence. Indicator 1 – Percentage increase in the average income of farm families in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 30 percent. Data on growth of farm family incomes during the period of the Program is available for five of the participating institutions in the departments of Puno, Cusco and Apurimac. As reflected in Figure P-1, the average increase through September 2003 is S/. 237. This is a 25 percent increase, near the 30 percent target set in the indicator. Figure P-1. Increase in the Average Farm Family Income 948 1.185 237 0 200 400 600 800 1.000 1.200 1.400 Baseline September 2003 Increase Nuevos Soles Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. Indicator 2 – Percentage increase in the average number of hectares of land per family used for crops and pasture in the target areas of NGOs that go through the first four phases of technical assistance and remain with the methodology for the life of the Program. Target: 20 percent. Matching Grant Program End-of-Program Evaluation 37 According to secondary information acquired by the SID-Peru staff, the average tenancy for the High Sierra of the country is 2.0 hectares per family. On the other hand, the 4,446 farm families in the Program area of the eleven institutions with information reported an increase of 810 hectares in cultivated cropland and 1,087 hectares in pastures. This total increase of 1,897 hectares results in an average total of 2.43 hectares per farm family – an estimated average increase of 0.43 hectares. See Figure P-2. This is an average increase of 21.5 percent, slightly higher than the targeted 20 percent. The same eleven institutions reported on the levels of increase in productivity for the principal crops. See Figure P-3. The positive results in the percentage changes in yields for these major agricultural enterprises are products of a combination of Program practices and other factors that have permitted the recuperation of soil fertility. Major improvements in productivity were observed in wheat and alfalfa with substantial increases in crops such as corn, barley and potatoes (amargos). Lesser expansion was experienced in crops such as chirimoya, tuna, beans, broad beans and other potatoes. In addition to managing agriculture and natural resource projects, it was determined that 71.4 percent of the organizations also manage projects that are not directly related to SID activities – that is, projects that do not involve activities related to the recovery of soils and pastures nor to the increase of production and incomes. Nevertheless, these are projects that include elements directed at the reduction of poverty in the broad sense of the word – economic, social and political development, democratization, and gender and generation issues. Figure P-2. Increase in the Hectares of Productive Cropland and Pasture per Farm Family 2,00 2,43 0,43 0,00 0,50 1,00 1,50 2,00 2,50 3,00 Baseline September 2003 Increase Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. Matching Grant Program End-of-Program Evaluation 38 Figure P-3. Yield Increases for Principal Crops 45 10 54 58 100 14 22 29 30 100 0 20 40 60 80 100 120 Potatoes, Amargo Potatoes, Other Corn Barley Wheat Broad Beans Beans Tuna Chirimoya Alfalfa Percent Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. Realizing that the technical assistance provided by SID and the resultant changes in the client organizations bring about positive impacts on the design, implementation and evaluation of these projects, the communities and rural families that gain from these projects often have been reported by the organizations as being beneficiaries of the Matching Grant Program. As an example of the inter-institutional cooperation that exists between SID Peru and the participating organizations, representatives of FADEP attended group workshops, however their participation in more intensive technical assistance was via AMUDA´s project as this farmer federation did not have its own project and AMUDA was a federation member. FARTAC is a special case of a Farmer’s Federation that signed a cooperative agreement at the beginning of the Program and was provided assistance in Strategic and Project Planning and in Project Monitoring and Evaluation during the early stages in anticipation of an agricultural project that never materialized. Some assistance also was provided with agricultural practices based on the expressed interest on the part of some of the agrarian leagues in providing informal farmer extension to their members. When the Program Country Office determined that the interests of FARTAC were more political than technical, technical assistance to the organization was suspended. As a result, the inflated numbers for FARTAC that were included in the mid-term evaluation are not shown in the current summary. Matching Grant Program End-of-Program Evaluation 39 UNCA, an organization included in the mid-term evaluation in 2002, has been eliminated from consideration in the current evaluation due to the fact that their 6-month agricultural project terminated before the Program was able to provide more than cursory assistance. Another modification in the data presented for this evaluation is the combination of FADEP and AMUDA. These two organizations received technical assistance together. (AMUDA is a member of the Federation.) On the other hand, IDIR data continues to be included in the tables even though the organization never signed a technical assistance agreement with the Program. IDIR did, however, participate in technical assistance activities in phases 2 and 3 (project planning and M & E) as part of the CHA project. The consortium of NGOs implement the CHA project as three distinct projects with different objectives in different zones. As a result, 23 NGOs and Grassroot Organizations are listed in Tables P-4 and P-5, and are being taken into consideration in the current evaluation. These include nine organizations in the department of Cusco, six in the department of Apurimac, six in the department of Puno, and one each in the departments of Arequipa and Tacna. The baseline data in Table P-4 reflects the numbers of communities and rural families that were participating in the projects managed by the NGOs and grassroots organizations at the time they signed cooperative agreements to work with the SID Peru Program. These numbers are compared to the number of participant communities and rural families in September 2003. This is the level of indirect coverage achieved by the Program. The progress made by each institution is directly related to the interest of the technical personnel and the directors of the institution, the status of the project at the time of Program intervention, the duration of the project and the Program, and the number of new projects for which the institution received financing during the life of the Program. The changes observed in the numbers of communities and participant families are results of highly motivated technical teams at the institutional level that have been able to introduce attitudinal changes. This also reflects that the pace of change is gradual in the type of participants prevalent in the target populations of the projects. Table P-4. Organizations and Families that Benefit from the National Program of Peru Beneficiaries Baseline September 2003 No. Name Department Communities Families Communities Families 1 IAC Cusco Proyecto 1 8 250 8 400 Proyecto 2 9 300 9 600 2 PAC Cusco 19 600 22 700 3 PEJ Cusco 14 200 14 200 4 CESA Cusco 4 42 4 48 Matching Grant Program End-of-Program Evaluation 40 5 IMAGEN Cusco 2 40 4 80 6 FARTAC Cusco n.a. n.a. n.a. n.a. 7 RUNAMAQUI Cusco 5 240 5 240 8 INKA Cusco 8 568 8 568 9 Plan International Cusco 10 718 10 718 10 CEDES Apurimac 3 300 3 300 11 CEPRODER Apurimac Proyecto 1 4 200 4 500 Proyecto 2 4 275 4 275 Proyecto 3 5 450 5 450 Proyecto 4 6 250 6 250 12 IDMA Apurimac 9 350 9 350 13 IIDA Apurimac Proyecto 1 7 210 7 210 Proyecto 2 11 365 11 365 14 IRD Apurimac 6 200 6 200 15 CICCA Apurimac Proyecto 1 10 360 10 360 Proyecto 2 7 173 9 220 16 Caritas-Juli Puno 8 480 22 1,723 17 Wiñay Puno 2 100 2 540 18 FADEP-AMUDA Puno 9 700 12 900 19 FEDICAPU Puno 4 90 4 90 20 ADRUPMA Puno 7 300 7 300 21 Servicios Integrales Puno 5 300 5 300 22 Ayuda en Acción Arequipa n.a. n.a. n.a. n.a. 23 MALLKU Tacna 7 100 7 100 Totals 193 8,161 217 10,987 Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. n.a. – Not available. On the other hand, of the 23 Peruvian organizations included in Table P-5, a total 47.7 percent participated in general and/or internal workshops of Phase 1 of the Program – “learning new methods.” Twenty-one of the organizations (91.3%) participated in one or more workshops related to the planning of better projects – Phase 2. As for Phase 3 –“the design of better evaluation systems” – 82.6 percent participated one or more workshops. At the same time, 60.9 percent of the organizations participated in activities related to the “execution of new projects” - Phase 4. An equal number of organizations participated in technical assistance activities related to Phase 5 of the Program. Finally, a total of nineteen organizations (82.6%), participated in activities of Phase 6 of the Program during either the year 2002 or 2003. Table P-5. Program Progress of Peruvian NGOs and Grassroot Organizations Phases and Sub-phases of Technical Assistance Matching Grant Program End-of-Program Evaluation 41 No Organization Depart 4 5 6 1 2 3 1 2 3 4 1 2 3 4 5 6 2002 2003 1 IAC Cusco X X X X X X X X X X X X X 2 PAC Cusco X X X X X X X X X X 3 PEJ Cusco X X X Project ended X X 4 CESA Cusco X X X X X X X X X X X X X 5 IMAGEN Cusco X X X X X X X X X X 6 FARTAC Cusco X 7 RUNAMAQUI Cusco X X X X X X 8 INKA Cusco X X 9 Plan Interna. Cusco X X X X 10 CEDES Apurimac X X X X X X X X 11 CEPRODER Apurimac X X X X X X X X X 12 IDMA Apurimac X X X X X X Project ended X X 13 IIDA Apurimac X X X X X X X X X X X 14 IRD Apurimac X X X X X X X 15 CICCA Apurimac X X X X X X 16 Caritas-Juli Puno X X X X X X X 17 Wiñay Puno X X X X X Project ended 18 FADEP-AMUDA Puno X X X X X X Project ended X 19 FEDICAPU Puno X X X 20 ADRUPMA Puno X X Project ended 21 Servicios Integrales Puno X X Project ended 22 Ayuda en Acción Arequipa X X 23 MALLKU Tacna X X X Guide to the Phases of Technical Assistance: 1 =Learning new methods; 2 = Planning better projects; 3 = Designing better evaluation systems; 4 = Implementing the new projects; 5= Implementing the better evaluation systems; 6 = the NGOs and other organizations working together to evaluate results and determine how to improve their projects. Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. At this point it is appropriate to mention that the numbers of NGOs and Grassroot Organizations included in the tabulation for SID-Peru encompass only those that were selected for assistance under the established selection criteria. About 50 percent of the organizations that requested assistance were not eligible to participate in the Program because they did not have currently funded agriculture or natural resource projects, or because they were public entities. The ratio is even higher for Puno. (See Table P-6.) Overall, of the 39 organizations that participated in the training workshops, 24 (61%) presented requests for technical assistance. This is to point out the great potential of this Program to reach many more clients and beneficiaries should the Program receive renewed support and financing. Table P-6. Requests for Technical Assistance Received from NGOs and Grassroots Organizations by the Peruvian Country Office Department Training Workshop T.A. Request Cusco 14 organizations 9 organizations Puno 16 organizations 7 organizations Matching Grant Program End-of-Program Evaluation 42 Apurimac 8 organizations 7 organizations Arequipa 1 organization 1 organization Total 39 organizations 24 organizations Percentage 61% with qualifying projects Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. The combined result of the impressive number of NGOs and Grassroot Organizations with whom SID has worked in Peru and the fact that these organizations have advanced so rapidly in the six-phase Program of technical assistance, is evidence that the Country Office has maintained good focus in their application of the selection criteria for organizations and that their program for working with individual organizations has functioned well. At the same time, the fact that the organizations have advanced through the phases of the Program can be interpreted to mean that they were well please with the type and quality of the technical assistance received in the early stages and, therefore, were eager to continue with the more advanced elements. It is obvious that that the personnel of SID-Peru progressed well in their work related to these phases of the Program. Their work shows a good organization and purpose. Objective 2 – Effective training and technical assistance services to farmers in reclaiming soils and pastures, and in increasing production and income established and sustained by the NGOs working with the Program. Indicator 3 – Change in average Program score of participant NGOs on a SID￾designed instrument for the self-assessment of capacity development. Target: 25 percent. The data utilized by the evaluation team consisted of that gathered by the two SID Country Office and information provided by the Central Office. Much of this data is based on interviews and the inter-institutional workshops conducted in 2002 and 2003. Additionally, the evaluation team conducted interviews with several of the Country Office staff members during visits in late November and early December 2003. Final confirmation and adjustments to the data were effected by means of internet and telephonic communication between members of the Evaluation Team and the staff of the Country Office during December 2003. Data collected from the participating institutions indicate major advancements in all areas of project design, implementation, monitoring and evaluation. Seventy-five percent of the NGOs and Grassroot Organizations reporting indicated increases in the use of monitoring and evaluation techniques and in the timely production of project results. In the case of the 11 institutions interviewed in the internal evaluation, 100 percent reported that they had Matching Grant Program End-of-Program Evaluation 43 improved their methods of monitoring and evaluation and that their reports are being delivered to the cooperating organizations in a timely manner. Information displayed in Figure P-4, Project Monitoring, shows that 46 percent of the institutions that participated in the SID-Peru Technical Assistance Program indicate that their current monitoring activities involve the technical team and the target population – a positive tendency supporting the sustainability of the projects involved. On the other hand, the director of the institution in coordination with the project manager carried out monitoring activities in 27 percent of the cases, and an equal proportion (27%) reported that the project technical team performs monitoring activities. While these data show that more than half of the institutions require additional assistance to reach the desired level of participatory project monitoring, all of them are now involved in this type of activity. Figure P-4. Project Monitoring Technical Team 27% Director and Project Manager 27% Team and Population 46% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. The data for Project Evaluation, as shown in Figure P-5, reflects a somewhat better result. Fifty-four percent of the institutions are performing evaluations with the participation of the target populations. This is a sign of significant improvement in the tendency to achieve real participation of the local population in the decision making process, increasing project sustainability. A consultant with the support of the project technical team performs the project evaluations of another 31 percent of the institutions, while only 15 percent reported that external consultants working alone conduct their evaluations. Matching Grant Program End-of-Program Evaluation 44 Figure P-5. Project Evaluation Consultant 15% Consultant and Team 31% Technical Team and Population 54% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. The perceived utility of the technical assistance received by the NGOs and the Grassroot Organizations is shown by the data in Figures P-6 and P-7. The data in Figure P-6 depicts the percentage of the participating institutions that report the use of specific planning techniques and instruments promoted by the Program. Among the principal variables mentioned by the institutions are the Problem Tree (26%), Logical Framework (26%), Work Plan (21%) and Task Assignment (16%). Eleven percent of the institutions mentioned other techniques and instruments. Figure P-6. Percentage of Institutions that Use Planning Techniques and Instruments Problem Tree 26% Logical Frame 26% Work Plan 21% Task Assignment 16% Others 11% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 45 According to the opinions expressed by the technical teams of the participating institutions, the planning aspects of major utility include: 1) logical frame (27%), project planning (18%), monitoring and evaluation (13%), analytical frame (13%), and participatory methodologies (9%). The full range of these aspects is depicted in Figure P-7. Figure P-7. Planning Aspects of Major Utility to Participating Institutions Logical Frame 27% Analytical Frame Project Planning 13% 18% Baseline Studies 4% Monitoring & Evaluation 13% Data Collection Methods 4% Partipatory Methodologies 9% Activity Adjustment 4% Institutional Self￾evaluation 4% Interpersonal Relations 4% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. Grouping these aspects by category, we find that 49 percent of the institutions indicated that aspects related to Project Planning (logical frame, project planning and activity adjustments) were of most utility to them. On the other hand, thirty-eight percent of the institutions placed Monitoring and Evaluation aspects (monitoring and evaluation, analytical frame, baseline studies, data collection methods, and self-evaluation) at the top of their list. Only 9 percent picked participatory methodologies as being of major utility, while 4 percent tapped interpersonal relations. Obviously if these numbers had been tabulated in a different manner (utilizing multiple response techniques) some of the resulting combinations of aspects would have been interesting. Another important variable to be analyzed in this area is extent to which the technical teams of the participating institutions have been strengthened. The data depicted in Figure P-8 results from information reported by twelve participating institutions. Matching Grant Program End-of-Program Evaluation 46 Figure P-8. Institutional Strengthening Aspects of Major Utility to Participating Institutions Analytical Capacity 25% Team Participation 8% Use of Implementation Methodologies 17% Report Quailty & Cooperartion 17% Less Activist 8% Focus and Target Definition 25% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. It shows that 75 percent of the organizations report an improved capacity for project planning and project monitoring and evaluation. This includes 25 percent that experienced improvements in their analytical capacity; 25 percent indicated that they have an improved focus and target definition; and 17 percent that have enhanced their project implementation through application of improved methodologies. Another 17 percent indicated that they have experienced an improvement in the quality of their reports and their cooperation, while 8 percent reported being less activist. Indicator 4 – Percentage of NGOs that report results in a timely manner. Target: 75 percent. Although the institutions that participate in the SID programs do not provide regular reports on their activities to the SID Country Office, they do prepare and submit reports to their cooperating institutions and to the organizations that provide financing for the project. Of the twelve cases analyzed in the internal evaluation, 58 percent of the institutions described improvements in the analysis contained in their reports. On the other hand, 42 percent indicated progress in the timely presentation of reports to their cooperating institutions. Indicator 5 – Percentage of NGOs that diversify their sources of funding. Target: 25 percent. Matching Grant Program End-of-Program Evaluation 47 There are two results that lend to the analysis of the results of this indicator. The first is related to the number of new projects presented for financing. Of the twelve institutions interviewed, project proposals were presented for 36 new projects. Ten institutions (83%) presented between one and five projects, while only 2 institutions (17%) failed to present proposals during the life of Program. The other aspect of this indicator is the number of projects that received financing. Twenty￾five percent of the institutions reported an increase in number of approved proposals and the diversification of their sources of financing. Six of the 10 institutions that presented proposals (60%) were successful in getting one or more of their projects financed – a total of 18 new projects or 50 percent of those presented. The institutions attribute this success rate to the improvement in the quality of their proposals based on the technical assistance received in the area of project design. These results can be further analyzed by means of the number of institutions that reported adjustments in the design and implementation their projects, incorporating guidance provided through the technical assistance received. All of the eleven institutions analyzed in this case indicated that they had made such adjustments to improve the definition of project indicators, the specification of project activities and the definition of strategies and methodologies. The data in Figure P-9 show numeric results of these variables. Figure P-9. Project Design Adjustments Indicators 36% Objectives 28% Activities 24% Strategies and Methodologies 12% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 48 Indicator 6 – Percentage increase in the number of farmers participating in NGO technical assistance activities (disaggregated by sex). Target: 25 percent (40 percent women). This indicator is used to measure the quality of service being provided, under the assumption that the delivery of good quality services will generate demand for additional services. The numeric values presented in Table P-7 indicate the targets for the Country Office related to the number of NGOs, number of participating communities, number of participating rural families and the percentage participation of women. Table P-7. Results for Indicator 6 of Objective 2 for the Peruvian Country Office Indicator Baseline Target Progress￾to-date Number of participating institutions - 22 23 Number of participating communities - 220 217 Number of participating rural families - 6,600 10,987 Percentage of women participants - 40% 42% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. The number of participating institutions for SID-Peru indicates that it has surpassed the Program-end goal. When the analysis of this target is considered jointly with that of the number of participating communities and the number of rural families, it is obvious that the values for the NGOs and Grassroot Organizations, and therefore for SID, are within the targeted ranges. The 23 participating NGOs and Grassroot Organizations of the Peruvian Country Office are reaching 217 communities – 98.6 percent of the targeted number. This value is lower than that reported in the mid-term evaluation based on the fact that the numbers of FARTAC (a very large number in that evaluation) are not being carried forward. Also, there are two organizations for which numbers were not available at the time of this evaluation. Nevertheless, the results indicate that the SID office is doing an excellent job in applying the criteria for the selection of the organizations to participate in the Program and in providing technical assistance for application by these organizations in their areas of influence. The numbers of participating rural families, the actual theme of this indicator, exceed the targeted values. The NGOs and Grassroot Organizations that work with the Peruvian Office are reaching an estimated 10,987 rural families in the Peruvian Sierra (166% of the targeted number). At this point it is necessary to reemphasize one of the unique features of this Program – that of the multiplier effect of the technical assistance provided by SID through the NGOs. It is this detail that permits SID to impact such a large number of final beneficiaries. Matching Grant Program End-of-Program Evaluation 49 The other aspect of this variable is the increase in the average number of participating families per institution. (See Figure P-10.) According to the data reported in Table P-4, there was an average of 388 families benefiting from the activities of the 21 reporting institutions. This number had increased to 523 by the time of the internal evaluation in September 2003 - an increase of 135 families, or 34.8 percent. At the same time the number of participating families per community increased from 42 at the beginning of the Program to 51 by September 2003 – an increase of 19 percent. (See Figure P-11.) Figure P-10. Increase in the Average Number of Participating Families per Institution 388 523 135 0 100 200 300 400 500 600 Baseline September 2003 Increase Number of Families Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. Figure P-11. Increase in the Average Number of Participating Families per Community 42 51 9 0 10 20 30 40 50 60 Baseline September 2003 Increase Number of Families Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. Matching Grant Program End-of-Program Evaluation 50 Similarly, the all of NGOs and Grassroot Organizations reported increases in the participation of women in their projects and in overall community activities. These institutions reported that at the beginning of the Program an average of only 24 percent of the women in the project areas actively participated, while at the time of the internal evaluation they indicated that the proportion had increased to 42 percent - an increase of 19 percent during the life of the Program. See Figure P-12. Figure P-12. Increase in the Average Participation of Women in Projects 24 42 19 0 5 10 15 20 25 30 35 40 45 Baseline September 2003 Increase Percent Participation Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003. The participating NGOs and Grassroot Organizations attributed these increases to the application of the “campesino-a-campesino” methodology by 37 percent of the institutions and the use of participatory workshops by a full 63 percent of these organizations. Indicator 7 – The Coordinadora Rural assumes an effective technical assistance role for NGOs by conducting workshops. Target: Yes. The associate for the Peruvian Country Office was the Coordinadora de Organizaciones Campesinas e Instituciones Agrarias del Perú, known as the Coordinadora Rural. This organization assumed an active role in the Program. As a private, non-profit consortium, it promotes rural development in the Andean region of Peru. The affiliates include social organizations, organizations of campesinos and small agricultural producers, and non￾government development organizations that promote and stimulate rural development in the high Andean region of the country. Matching Grant Program End-of-Program Evaluation 51 The mission of the Coordinadora Rural is, “to promote joint endeavors, contributing to the expansion of the capabilities and rights of rural men and women, and institutionalizing the best experiences; to achieve and maintain an important and widely recognized mandate to speak on behalf of the rural Andean region in matters before the country and the State; and, in this manner, contribute to human development in a way that is sustainable and just for the rural families and the Andean Sierra.” This mission is very similar in many ways to that of SID and the Matching Grant Program in that it targets the rural family and elements of rural development the contribute to their human, social and economic development. The Coordinadora Rural and its many regional offices work through the NGOs and Grassroot Organizations to reach large numbers of final beneficiaries. That role extends to the coordination of, and the participation in, the programming and implementation of specific training activities at the local and regional level. Under the supervision of the national office, the regional offices provide a positive input to Program activities and encourage affiliate NGOs and Grassroot Organizations to participate. 3.2.4 Inter-institutional Evaluation The inter-institutional evaluation workshop conducted by SID-Peru in September 2003, with the participation of the cooperating institutions and a large number of the participating NGOs and Grassroot Organizations, resulted in many valuable insights into the design, development, implementation and impact of the Program. Foremost among these are the items listed in Table P-8, the utility of the Program from the points of view the institutions, the projects, the technical teams and the local populations, and the perspectives of each of these as to how the Program could be more useful. Table P-8. Utility of the SID-Peru Technical Assistance Institutional Management Project Management Strengthening Institutional Capacities Impact on Local Populations Utility 1) Improved application of participatory methodologies in the decision-making process of the local population 2) Varying level of advance in the development of baseline studies 3) Institutionalization of Monitoring & Evaluation 1) Less activism in 2) technical teams. 3) Construction and/or reformulation of baseline information. 4) Improved level of local population participation in project implementation. 5) Increased importance of Monitoring and Evaluation activities. 6) Improvement in the quality of reports. 1) Improved understanding of design and importance of 2) project indicators. 3) Improved knowledge, 4) ability and agility in the elaboration of baseline studies and analytical framework. 5) Improved knowledge and practice related to participatory methodologies. 6) Better “attitude toward change” and teamwork spirit among personnel. 1) Improved community and family participation in project management. 2) Predisposition of the population de change. 3) Use of traditional practices in conservation of Natural Resources. 4) The population validates the services provided by the Program. Ways to Improve Usefulness 1) More practical application of concepts. 2) Constant advisory feedback to T.A. teams 1) Field validation of instruments. 2) Implementation of project documentation. 5) Improve relation between technical team and administration in Monitoring and Evaluation. 1) Reinforce project-planning training. 2) Technical assistance en participatory training methodologies. 3) Reinforce Phase 1 technical assistance. 1) Continued monitoring during the life of the Program. 2) Documentation of experiences. 3) Practical application of the concepts of impact, sustainability, replicability, gender, coverage and competitively Matching Grant Program End-of-Program Evaluation 52 Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. 3.2.5 SID-Peru Staff Input The observations and perception of the SID-Peru staff is also an important factor in evaluating the results and impact of the Program. The evaluation team considers the following points to be of primary interest in this light: 1) The SID-Peru Technical Assistance Program has been highly useful to the NGOs in: a) helping them to increase the number of projects being implemented; b) expanding the coverage area to various river basins; and c) increases in annual budgets and number of participants. 2) One of the key elements in the SID-Peru Technical Assistance Program has been to attempt changes in the attitudes of the directors and technical staff of the participating NGOs and Grassroot Organizations with the purpose of producing an increased level of creativity in their search for sustainable solutions to community development problems. The fact that these professionals now speak of producing baseline studies, of the ease of implementing process evaluations, and of the sustainability effects and impacts of their projects, are signs of the utility of the Program actions. 3) The NGOs and Grassroot Organizations now show increased preoccupation with the sustainability of their projects. These institutions are continually improving their policies related to the participation of the local population in project planning activities (the definition of goals and objectives, the contribution of local resources, etc.), and in the decision-making processes connected with project monitoring and evaluation. 4) Institutional talent flight is a two-edged topic in this Program. It is negative in the sense that the Program frequently receives request to provide assistance to train new staff members that were hired to replace those leaving. This is a drain on the limited time and resources of the SID-Peru Staff. However, it can be seen to be a feather in the hat of the Program. Personal statements made by technicians participating in SID training events reflect that they have improved their professional capacity to the extent that they have been able to accede to positions of greater responsibility and higher salaries. 5) The NGOs and Grassroot Organizations have experienced a 100 percent improvement in their Monitoring and Evaluation capabilities. Prior to their participation in the Program they were implementation oriented. They now are on tract with the understanding that Monitoring and Evaluation of overall project impact is more important than measuring increases in income. 6) Following the SID work during this year, the NGOs and Grassroot Organizations more frequently are doing participatory planning and less contracting of project design. 7) The local population is now more involved in project planning and design. The NGOs now understand that this is necessary in order that the community members consider a project to be “theirs” and fully support project implementation. Matching Grant Program End-of-Program Evaluation 53 8) NGOs and Grassroot Organizations now do baseline studies to provide the information required for the evaluation of project results and impact. 9) It may have been more logical to start the Program with “Monitoring and Evaluation” activities. But the question remains whether the NGOs would have been willing to go in this direction. Doing things the way they were done, probably helped gain clients and build confidence through project design and the development of project proposals. 10) It is necessary that the technical staff have faith in the local population that they are capable of doing things well – even those that are illiterate. 11) Project design and development must be done “for the local population – with their participation”. They are the ones who are going to live in the area and the ones who must survive based on the results of the project. 12) Communication in a language understood by the local population is of utmost importance. They must be able to understand the objectives of a project and decide on their participation in the project according to their own possibilities and capacities. Matching Grant Program End-of-Program Evaluation 54 3.3 Results of the SID Central Office 3.3.1.Overall Results As indicated in the overview to the Program, SID divided the technical assistance to NGOs into six phases: (1) learning new methods; (2) planning better projects; (3) designing better evaluation systems; (4) implementing the new projects; (5) implementing the better evaluation systems; and (6) working together to evaluate results and improve future projects. The eight indicators and four objectives for the SID Central Office were established in order to measure the responsiveness, quality and impact of SID´s technical assistance to the participating NGOs and Grassroot Organizations, and its support to the SID Country Offices. The indicators use the demand for services as a proxy for the quality of service being delivered, under the assumption that the delivery of good quality services will generate demand for additional services. The amount of time that personnel of the SID Country Office staffs spend on direct technical assistance for NGOs is used as a measure of responsiveness to the needs of clients, and the time SID Headquarters Staff dedicate to its Country Offices is meant to represent its responsiveness to their needs. Objectives 3 and 4 are congruent with BHR/PVC’s institutional strengthening strategic objectives. 3.3.2.Reactions to the Mid-term Evaluation The mid-term evaluation did not contain specific recommendations related to the Central Office operation. However, the Program director and staff did react positively to the several recommendations made for the country offices and the manner in which they could positively influence and affect Program activities in order to support the fieldwork. Communications were improved and reaction time to field requests for support was reduced. In general, the Central Office has done a excellent job in supporting the field offices and in its specific implementation tasks. 3.3.3.Results Indicators The following section contains information related to the progress made by the SID Central office towards the achievement of each of its results indicators. Table A-3 of Annex A contains the Program Logical Framework in which are listed each of the objectives, indicators, goals, measurement specifications and principal activities for the SID Central Office. Objective 1 - Increase capacity of SID to provide technical assistance to NGOs in methods for helping farmers reclaim soils and pastures and increase production and income. Indicator 1 – Percent of NGOs requesting all remaining phases of technical assistance after participating in planning workshops. (The indicator is expressed as a ratio of the number of NGOs requesting technical assistance in the remaining phases of the Matching Grant Program End-of-Program Evaluation 55 Program as a percent of the number of NGOs at the initial planning workshop. The indicator will be reported annually.) Target: 66 percent. Indicator 2 - Percent of time SID Country Office Staff spent on technical assistance for NGOs. Target: 50 percent. The aim of the first indicator is to measure the demand generated during the Program for continued technical assistance (phases 4 to 6) after the NGOs received technical assistance in learning new methods and planning better projects (phases 1 and 2). On the other hand, there was thought to be a danger in that the technical assistance staff would spend most of their time in their offices planning, preparing materials, and organizing the technical assistance; monitoring, evaluating and reporting on their work; and attending to other administrative tasks. In that sense, the aim of the second indicator was to measure and insure that the technical assistance staff spend at least half their time in the offices and communities of the NGOs, providing technical assistance. Table C-1 contains information related to the progress made by Bolivia and Peru during the life of the Program as to the percentage of the NGOs and Grassroot Organizations that participated in planning workshops and subsequently requested additional technical assistance in the first four phases of the Program. Table C-1. Results Indicators for the Central Office Indicator Baseline Target Level Achieved 1. Percent of NGOs requesting all remaining phases of technical assistance after participating in planning workshops. Bolivia 0 66% 59% Peru 0 66% 61% 2. Percent of time SID Country Office Staff spent on technical assistance for NGOs. Bolivia 0 50% 45% Peru 0 50% 75% Source: Prepared from data provided by the Central Office and the Country Offices of SID, and additional information collected during the preparation of this report. As is indicated therein, the baseline for each of the indicators is zero. The target for Indicator 1 is 66 percent, while that for Indicator 2 is 50 percent. The results of Indicator 1 for the country programs are 59 percent for Bolivia and 61 percent for Peru. The evaluation team considers this result to be exceptional. On the other hand, the progress made for Indicator 2 is 45 percent for Bolivia and 70 percent for Peru. The result for the Peruvian Office is well beyond the targeted value and reflects a real accomplishment for its organization and purpose. On the other hand, the Bolivia Country Office, having Matching Grant Program End-of-Program Evaluation 56 improved substantially since the mid-term evaluation, barely missed the target levels for these indicators. Figure C-1 contains data for an overall analysis of the Peruvian Country Office staff time dedicated to different activities during the period 2002-03. This chart shows that 29.7 percent of all staff time was devoted to the provision of technical assistance to the participating organizations. Another 17.1 percent of their time was spent in activities related to the preparation for these workshops, and another 29 percent was spent on related activities. Figure C-1. Level of Effort of the Peruvian Country Office Staff, by Activity, 2002 - 2003 Workshop Preparation 17% Technical Assistance 29% Report Preparation 11% SID Follow-up 10% Travel 8% Others 25% Source: Prepared from data contained in: SID-Perú, “Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios, que Ejecutan Proyectos Agropecuarios y Medio Ambientales”, Cusco, Perú, November 2003., and additional information collected during the preparation of this report. Objective 2 – Increased capacity of SID to help NGOs monitor, evaluate and report results, and to improve performance. Indicator 3 – Percent of NGOs requesting help in monitoring, evaluation and reporting after participating in planning workshop. (The indicator is expressed as the ratio of the number of NGOs requesting technical assistance for Monitoring and Evaluation as a percent of the number of NGOs at the initial planning workshop. The indicator will be reported annually.) Target: 66 percent. Indicator 4 – Percent of SID Country Office Staff time spent helping NGOs conduct baseline surveys and analyze results, and in monitoring, evaluating and reporting their results. Target: 50 percent. Matching Grant Program End-of-Program Evaluation 57 One of the premises of SID's Matching Grant Program is that NGOs cannot steadily increase the impact and coverage of their projects if they do not periodically monitor and evaluate the results of their work. For this reason, technical assistance in improving the monitoring and evaluation of projects was included as a key feature of SID's technical assistance and as a principal component of assistance provided during phases 3, 5, and 6. SID also set increasing its capacity to provide technical assistance to NGOs in monitoring, evaluation and reporting as a separate Headquarters objective. The aim of the third indicator was to measure the demand for technical assistance in monitoring, evaluation and reporting after the NGOs had received technical assistance in learning new methods and planning better projects. The aim of the fourth indicator was to measure the percentage of SID's Country Office staff time spent helping NGOs conduct baseline surveys and analyzing results and in monitoring, evaluating and reporting their results. Table C-2 contains information related to the progress made during the life of the Program by the Bolivian and Peruvian Country Offices as to the percentage of the NGOs and Grassroot Organizations that participated in planning workshops and who requested additional technical assistance in the phases of the Program related to monitoring, evaluation and report preparation. As is indicated in the table, the baseline for each of the indicators is zero. The target for Indicator 3 is 66 percent for each of the Country Offices. The results reflect that 80 percent of the participating organizations in Bolivia and 91 percent of those in Peru requested technical assistance related to monitoring, evaluation and report preparation. Both SID Bolivia and SID Peru have surpassed the targeted levels. Utilizing the new methods and procedures provided through the technical assistance in these areas, the NGOs and Grassroot Organizations are now better able to evaluate the progress and impact of their projects and effect the required adjustments to better serve the communities and rural families in their regions. Table C-2. Results Indicators for the Central Office (Continued) Indicator Baseline Target Level Achieved 3. Percent of NGOs requesting help in monitoring, evaluation and reporting after participating in planning workshop Bolivia 0 66% 80% Peru 0 66% 91% 4. Percent of SID Country Office Staff time spent helping NGOs conduct baseline surveys and analyze results, and in monitoring, evaluating and reporting their results. Bolivia 0 50% 20% Peru 0 50% 30% Source: Prepared from data provided by the Central Office and the Country Offices of SID, and additional information collected during the preparation of this report. Matching Grant Program End-of-Program Evaluation 58 The target for Indicator 4 was set at 50 percent for each of the Country Offices. Actually this target represents the sum of the time spent providing technical assistance for these types of activities, plus the time spent helping the NGOs to improve the monitoring, evaluation and reporting activities for specific projects. Obviously, the text of the Indicator, and the data reported during the mid-term evaluation of the Program, reflect only that portion of the staff time dedicated to helping with specific Program activities. This was the true intent of the indicator. Therefore, the target should be reflected as 25 percent of the staff time for each of the Country offices. The results for Indicator 4 show that 20 percent of the staff time in the Bolivian Country Office and 30 percent of staff time in Peru was spent on activities related to monitoring, evaluation and report preparation. These numbers reflect that the NGOs and Grassroot Organizations advanced to those Program activities involving workshops and field events related to monitoring and evaluation. Objective 3 – Improve the capacity of SID to supervise and support field programs. Indicator 5 – Percent of HQ Staff time spent on field support activities. Target: 60 percent. Indicator 6 – Number of months per year when reports are submitted on time and are of acceptable quality. Target: 75 percent. Indicator 7 – Percent of responses of HQ to field staff requests. Target: 90 percent. Table C-3 shows data related to the progress made during the life of the Program by the Central Office related to the percentage of HQ Staff time dedicated to the support of field office activities, the percentage of monthly and annual reports delivered on time and of an acceptable quality, and the percentage of responses by the HQ staff to field office requests. The progress made for Indicator 5 shows that 63 percent of Headquarters Staff time was dedicated to the support of the Country Offices, easily surpassing the target of 60 percent. Indicator 6 reflects that 80 percent of the monthly and annual reports were delivered on time and were of an acceptable quality, a full five percent above the established target of 75 percent. The progress-to-date for Indicator 7 (94%) is shown to be four points higher than the end of Program target. All indications are that the support provided by the Central Office has been excellent in the sense that it avails the Country Offices with the backstopping and technical support necessary for them to do their job. This is accomplished without overburdening the Country Offices with unnecessary administrative tasks. Matching Grant Program End-of-Program Evaluation 59 Table C-3. Results Indicators for the Central Office (Continued) Indicator Baseline Target Level Achieved 5. Percent of HQ Staff time spent on field support activities 0 60% 63% 6. Number of months per year reports submitted on time and are of acceptable quality 0 75% 80% 7. Percent of responses of HQ staff to field staff requests 0 90% 94% Source: Prepared from data provided by the Central Office and the Country Offices of SID, and additional information collected during the preparation of this report. The Matching Grant Program made an important contribution to SID's capacity to supervise and support field programs. The amount of Headquarters staff time dedicated to supporting field activities exceeded the target. The quality and timeliness of field reports gradually improved throughout the Program, as did the response of Headquarters staff to field requests. SID's Headquarters staff also provided onsite assistance in carrying out key phases of the Program. For example, Headquarters staff made field visits to the Program every two months during the last eight months of the Program to ensure completion of all activities and adequate data collection and preparation for the final evaluation. Objective 4 – Improved capacity of SID in Strategic Planning. Indicator 8 – Five-year strategic plan adopted by SID Board of Directors. (yes/no) Target: Yes. The SID Board of Directors adopted the five-year Strategic Plan at their board meeting in November 2002. In addition, the plan was updated and revised at the end of the first year in the board meeting in December 2002. 4. Expressed Client Needs Even though it was known that the Program was in its final stages, many of the organizations interviewed during the September 2003 evaluation expressed the desire for additional assistance in some areas aimed at achieving a greater impact. A listing of the principal topics follows. ß To attain greater impact in the management of natural resources: o More practice in the utilization of participative methodologies; o More community participation in the determination of agreements and mechanisms for the preparation and approval of proposals in the area of natural resource conservation; Matching Grant Program End-of-Program Evaluation 60 o Development and strengthening of local community capabilities; and o Stimulus of strategic agreements and mutual accords. ß To attain greater impact in the improvements of productivity and market development: o Institutional policies related to market development; o Product quality and productivity; o Generate value added in agriculture production; and o Development of strategic alliances for market entry, achievement of economies of scale in production, and attention to large volumes. ß To attain greater impact in self-development: o Strengthening the local culture without using extreme measures; o User participation in the decision-making process at all levels of project development; o Project should place a high level of importance on self-esteem and popular participation; o Organizational strengthening and development of local leaders; o Gradually overcome paternalistic and assistentialistic attitudes; and o Promote decision-making at the local population level through attitudinal changes. Further suggestions were made to improve the sustainability of the changes achieved to this point: ß Implement specialization programs in the areas of: o Baseline studies o Project planning o Project monitoring and evaluation o Documentation ß Develop a program of o Design of quality projects o Quality in the implementation of projects o Participative methodologies. Matching Grant Program End-of-Program Evaluation 61 5. Program Strengths This section presents some of the principal strong points of the Matching Grant Program as observed by the evaluation team during the two evaluative exercises. The fact that the SID team has been able to build on these points has resulted in a unique Program with impacts reaching far beyond those visualized by the designers. The team believes that the use of these design concepts in future endeavours of this type will result in a significant increase in the impact of rural development and natural resource projects. Large Numbers of Beneficiaries - The Matching Grant Program has managed to reach a very large number of final beneficiaries through its work with the NGOs and Grassroot Organizations in Bolivia and Peru. A total of the 23 NGOs and Grassroot Organizations participated in the Peruvian effort. They managed projects in agriculture and natural resources (as well as in other thematic areas) for a total of 217 communities and 10,987 rural families. These numbers indicate that SID-Peru reached more than the target number of organizations and communities, and far surpassed the end-of-program goals for the number of rural families. On the other hand, a total of the 35 NGOs and Grassroot Organizations participated in the Bolivian Program. They managed projects in agriculture and natural resources (as well as in other thematic areas) for a total of 1,184 communities and 30,246 rural families. These numbers indicate that SID Bolivia exceeded the target number of participating organization and far surpassed the end-of-program goals for the number of communities and rural families. These results are indicative of the highly successful work of the Country Offices in their application of the procedures for the selection of participant organizations and in the provision of technical assistance in the different stages of the six￾phased, two-tiered plan on which the Program was based. The other facet of this element deals with the numbers of professionals and technicians that the Program has trained and made available to the sector. In SID-Peru alone, a total of 770 NGO and Grassroot Organization staff persons attended 18 workshops. It is known that many of these trained technicians have already moved to positions of equal or greater responsibility in other development organizations. Further, it is anticipated that these persons will have a continuing impact on rural development-related activities in Bolivia and Peru for some time to come. Strong Organizational Trajectory - The Matching Grant Program is a one-of-a-kind design that builds on the extensive and rich experience of the International Non-Governmental Organization, Strategies for International Development, and its Executive Director, Charles Patterson. The trajectories of Mr. Patterson and the NGO extend to a large number of related projects in Latin America and other parts of the world, each of which contribute in some way to the ideas and methodologies used in the design, development and implementation of this Program. Matching Grant Program End-of-Program Evaluation 62 Sound Program Foundation - This Program was built around the idea of providing technical assistance to local NGOs, to local offices of international NGOs and to Grassroot Organizations. This was accomplished through regional or thematic groupings, providing the institutions with new and improved ways of broadening and strengthening their projects and the impact that they produce in the target populations. This two-tiered approach is different from the focus of many traditional rural development and natural resource projects that utilize available funding to try to reach as many rural families as possible with specific technical assistance in these areas. The Matching Grant Program, on the other hand, strives to combine a somewhat limited budget and a small amount of human resources and infrastructure to leverage the human and financial resources available to the participating organizations and, at the same time, strengthen those organizations operationally and functionally to better assist the communities and rural families that they serve. The technical assistance provided by the Matching Grant Program has gone beyond the conduct of workshops – accompanying the field staff of these organizations to assure that alternative solutions are successfully placed into practice. This means a greater bang for the buck for each dollar available to the Matching Grant Program. High Level of Professionalism and Motivation - The evaluation team observed during this and the mid-term evaluation effort that the personnel of the Country Offices in both Bolivia and Peru are well qualified, highly motivated professionals. This resulted in the development of a high degree of confidence and esteem on the part of the participating organizations, generating a high level of demand for the services of the Program, and an open and frank relationship between the personnel of the organizations and personnel of SID. The comments of the large majority of the directors and coordinators of these organizations indicated that they have been extremely pleased with the quality of the services delivered and desired to continue their participation. The directors of the two Country Offices demonstrated a high degree of dedication and professionalism in their work. Their abilities to direct and coordinate the work of the offices and the teams resulted in positive impacts beyond the targets of the Program design. Effective Methodologies and Techniques - The institutional strengthening methodologies employed by the Program were very effective in providing the personnel of the participating organizations with the tools required to expand and fortify their projects and to bolster the impact among the communities and the rural families that they serve. The techniques and procedures taught continue to be extremely useful to the organizations in upgrading their on￾going projects, in the design and development of new projects that are effective and efficient, and in monitoring and evaluating both previously existing as well as newly developed projects. Seventy-five percent of the organizations interviewed indicated that the SID technical assistance activities have aided them to make adjustments in the methodologies utilized in their work. The format employed by SID will have a lasting impact on the capability of the NGOs and Grassroot Organizations in the development and implementation of their future projects and also will be multiplied in other organizations as the trained professionals begin to spread their influence. Matching Grant Program End-of-Program Evaluation 63 Matching Grant Program End-of-Program Evaluation 64 6. Conclusions The evaluation team has extracted several conclusions deemed important for the consideration of the SID Central Office and others involved in the design, development and implementation of similar Programs in the future. These conclusions are presented below. 6.1. Institutional Sustainability o Various participating institutions in both Peru and Bolivia have captured, and are actively practicing, the criteria and concepts required for continuing improvements in their projects and in the development of a higher level of creativity in their new project proposals. o The fact that the institutions have achieved a level of 50 percent financing of the new projects presented, is evidence of the professional capacity attained and the confidence that the cooperating agencies now have in their ability. o The constant movement of personnel debilitates team cohesion and the formulation of creative and sustainable project proposals. o The technical teams have improved the professional capacity which, complemented by the vast experience, permits them to be creative while confronting the multiple challenges presented by rural development. o The interest of the institutions in including monitoring and evaluation activities in their projects permits them to be analytical in their approach and to constantly improve their strategies and methodologies. 6.2. Sustainability of the effects and impacts of Program activities in the local communities. o The local populations need to interiorize the objectives of the Program – to make them their own. o The local populations (both men and women) should take an active role in the decision-making process related to project planning, implementation, monitoring and evaluation. o The projects should develop the creative capacities of each and every man and woman in the local community in order that their forces can be utilized to find solutions to the problems confronted. Matching Grant Program End-of-Program Evaluation 65 o The development of a decision-making and creative capacity of the men and women involved in a project is the basis for the sustainability of the impact of that project. 6.3. Future Improvements in Technical Assistance o Technical assistance should be practical, aimed at solving problems of sustainability. It should give more attention to coverage, a higher-level of local population participation in the decision-making process related to the techniques used, the organization of the project, the innovative ideas employed, the use of local resources, and project replicability. o Practical technical assistance in the area of monitoring and evaluation, permitting a process of “learning by doing” in activities aimed at analysis and reflection, in the use of the results of M&E and in the manner in which the population should participate in project adjustments. 6.4. Program Funding The agreements signed with NGOs and Grassroot Organizations calling for them to pay for a portion of the costs of providing technical assistance has had a positive impact in that the participating organizations learned to place a monetary value on the services they received. In this manner, the services were valued more highly in the minds of the recipients. 6.4. Learning Environment and Strengthening Organizations The SID workshops provided participating organizations with valuable elements permitting critical analysis and the identification of deficiencies within their projects. These activities were part of the strategy to promote a learning environment within the organizations. Personnel of the NGOs and Grassroot Organizations viewed SID personnel as being collaborators who listened to the ideas of others and were disposed to help encounter solutions to the problems of the organizations. Technically, their teachings, critical analysis and the identification of project weaknesses contributed to the improvement of the work done by the participating organizations, permitting them to make needed adjustments in their technical assistance programs and projects. 6.5. Measurement System for Indicators and Objectives As mentioned in the mid-term evaluation, the planning matrix of the Program contains certain elements that are difficult to evaluate due to their definitions and directions. Examples are listed in the mid-term evaluation. Certain results indicators that require the measurement of long-term impacts should be reformulated in order to be more realistic in terms of the objectives established for the Program. Matching Grant Program End-of-Program Evaluation 66 6.6. Foment Women´s Participation Major advances were made in both Bolivia and Peru in the promotion of women’s participation in project activities and in community-level endeavors. Nevertheless, this major untapped source of intelligence, agility and work power still requires development and recognition. 6.7. Active Associate for Bolivia The lack of true associate in Bolivia, as mentioned in the mid-term evaluation limited the ability of that Country Office to achieve one of its stated objectives. The Country team overcame this deficiency by incorporating the participation of other organizational groupings. The evaluation team has determined that this did not cause any significant negative effects in the attainment of other Program goals. 7. Evaluation Methodology 7.1 Evaluation Objectives The purpose of this evaluation is to assess the impact of the Program activities in its three components: the Bolivian Country Office, the Peruvian Country Office and the Central Office in the United States. The objectives of the evaluation include the assessment of the quantitative and qualitative impacts achieved by these components as reflected in the progress-to-date of activities related to each of the results indicators. The terms of reference of the evaluation team cover the extent to which the SID interventions have had a positive impact on the ability of the NGOs and Grassroot Organizations to fulfill their objectives in the recovery of soils and pastures and the reduction of poverty among the rural families of the communities where they work. 7.2 The Methodology Employed The general methodology used in the evaluation consisted of the following: ß Preparatory meetings were conducted with both the representative of the Program Central Office and the personnel of the Country Offices of Bolivia and Peru. ß The Executive Director of SID, Mr. Charles Patterson, and personnel of the Country Offices of both Bolivia and Peru shared information with the Evaluation Team by means of personal interviews, telephone conversations and internet transmissions during all Matching Grant Program End-of-Program Evaluation 67 phases of the evaluation. Initial consultations with the Mr. Patterson set the terms of reference for the evaluation and determined the administrative arrangements for the tasks. ß The country teams of Peru and Bolivia designed a series of instruments for use in data collection activities related to the internal evaluations conducted during August and September of 2003. These instruments, the base data and summarized information are available in the Country Offices. The Evaluation Team relied heavily on these information sources during the present exercise. ß The evaluation team conducted a review of the Program documentation of each of the Country Offices, including the periodic reports made to the Central Office, information related to the participating NGOs and Grassroot Organizations, database information related to activities of the participating organization, and information related to their project activities and funding sources. ß Personal interviews were conducted with each available staff member of the two Country Offices. 7.3 Contacts The contacts made during the data collection phase of the evaluation include: Peru: Karen Kraft, Director Antenor Vásquez Ramírez, Agronomist Bolivia: Evan Meyer, Director 8. Concluding Comments The evaluation team anticipates that the focus and content of this evaluation will prove to be useful to all parties involved in the management and implementation of matching grant programs and to those final beneficiaries of program activities – the NGOs and the rural farm families they serve. It is our desire that the sharing of experiences, knowledge and lessons learned during the evaluation exercise will lend to the expanded collaboration and cooperation among the participating parties, resulting in the continuous improvement of the quality and effectiveness of this type of program in each of the countries involved in this Program, as well as around the world. Matching Grant Program End-of-Program Evaluation 68 Bibliography Brown, Loyd C., Haydée Delgado de Brown and Nohemí Villena, Mid-term Evaluation, Matching Grant Program to Provide Technical Assistance for NGOs to Reclaim Soils and Pastures and Reduce Rural Poverty in Bolivia and Peru, prepared for Strategies for International Development, La Paz Bolivia and Cusco Peru, December 2002. Central Office Staff of SID, Matching Grant Evaluation, Memo to Abraham Borda, Karen Kraft and Evan Meyer, Arlington Virginia, June 30, 2003. Centro de Servicios Agropecuarios, Proyecto de Conservación In Situ de Cultivos Nativos y Parientes Silvestres, CESA, Lima, Perú, 2001. Centro de Estudios y Desarrollo Social, Memoria de Proceso de Capacitación, Resumen Secuencial y Anexos, CEDES, Abancay, Perú 2002. Coordinador de Organizaciones Campesinas e Instituciones Agrarias, Siembra, Boletín de la Coordinador Rural, No. 42, Mayo-Junio 2002. Coordinador de Organizaciones Campesinas e Instituciones Agrarias, Memoria 2000, Coordinador Rural, Lima, Perú, 2001. Coordinador de Organizaciones Campesinas e Instituciones Agrarias, Programa General de Acción, Fase 3 (Años 2000-2002), Coordinador Rural, Lima, Perú, enero 2000. Kürmi, Tríptico Informativo de la ONG Kürmi, La Paz, Bolivia, sin fecha. Strategies for International Development, A Matching Grant Proposal to Reclaim soils and Pastures and Reduce rural Poverty in Bolivia and Peru, Project Proposal, Arlington, Va., 2000. Strategies for International Development, A Matching Grant Proposal to Reclaim soils and Pastures and Reduce rural Poverty in Bolivia and Peru, detailed Implementation Plan (DIP), Arlington, Va., 2000. Strategies for International Development, Overview of Year 1 and Changes subsequent to DIP, Arlington, Va., 2000. Strategies for International Development, Balance de la Gestión 2001 y Estrategias y Plan Operativo 2002, Cusco, Perú., enero 2002. Matching Grant Program End-of-Program Evaluation 69 Strategies for International Development, Taller Plan de Asistencia Técnica a ONGs, Cusco, Perú., enero-febrero 2002. Strategies for International Development, Asistencia Técnica al Desarrollo Rural, Tríptico Informativo de SID Perú, 2002. Strategies for International Development, Informe de SID-Perú para 2001, Cusco, Perú, enero 2002. Strategies for International Development, An Informative Bulletin for Friends of ..., Spring 2002, Arlington, Virginia, 2002. Strategies for International Develpment, Proyecto Matching Grant Facilitación Técnica Procesal a Proyectos Agropecuarios y de RR.NN. de ONGs y OCBs, SID, La Paz, Bolivia, 2002. Strategies for International Development, Informes Mensuales de la Oficina Nacional de SID￾Bolivia, La Paz, Bolivia, marzo 2002 al agosto 2003. Strategies for International Development, Base de Datos de las ONGs que Trabajan con SID￾Perú, Cusco, Perú, 2002-03. Strategies for International Development, Evaluación Final Interna, Programa de Asesoramiento Técnico a ONGs y Gremios que Ejecutan Proyectos Agropecuarios y Medio Ambientales, Cusco, Perú, November 2003. Strategies for International Development, Resultados del Taller de Auto-evaluación del Proyecto de Asesoramiento Técnico de SID-Perú, 25-26 de septiembre de 2003, Programa de Asesoramiento Técnico a ONGs y Gremios que Ejecutan Proyectos Agropecuarios y Medio Ambientales, Cusco, Perú, Septiembre 2003. USAID, Bureau of Democracy, Conflict and Humanitarian Assistance, Office of Private and Voluntary Cooperation, Program Development and Management Division, Evaluation Guidelines for Matching Grants to be Evaluated in 2003, Washington, D.C., May 2003 (Updated). Matching Grant Program End-of-Program Evaluation 70 Annex A - Logical Framework for the Matching Grant Technical Assistance Program Planning Matrix Final Objective: To recover soil and pasture productivity and increase farm family incomes in the Bolivian and Peruvian highlands. Matching Grant Program End-of-Program Evaluation 71 Table A-1. Logical Framework for the Matching Grant Technical Assistance Program, SID-Bolivia Planning Matrix Final Objective: To recover soil and pasture productivity and increase farm family incomes in the Bolivian Altiplano and high valleys. PROGRAM OBJECTIVE INDICATOR BASELINE AND GOALS MEASUREMENT METHODS, SOURCE AND FREQUENCY OF DATA COLLECTION PRINCIPAL ACTIVITIES 1. Soils and pastures reclaimed, and production and income of 9,600 farm families increased in 320 communities served by 32 NGOs. a) Percent increase in average incomes of farm families in the areas attended by the NGOs that participate in the 4 phases of technical assistance and that faithfully follow the new methodologies for the duration of the Program. b) Percent increase in the average number of hectares of land per family utilized for crops and pastures en the areas attended by the NGOs that participate in the 4 phases of technical assistance and that faithfully follow the new methodologies for the duration of the Program. a) Goal: 30% increase in average farm family income. b) Goal: 20% increase in the number of hectares. Results reported by NGOs during the annual evaluation meetings – sample of participating communities. NGO evaluations Annual evaluation meetings Surveys accomplished by SID and the Asociación de Instituciones de Promoción y Educación (AIPE) Project Evaluation NGOs provide training and technical assistance to farmers for the recuperation of soils and pastures, and for the increase in agricultural production and incomes. Preparation of technical assistance materials. 2. Effective training and technical assistance for farmers en the recovery of soils and pastures and in the increase of production and incomes, established in a sustainable form. a) Positive changes observed in the results of the self-evaluation instruments. b) Percent of NGOs that report results in a timely manner. c) Percent of NGOs that diversify their sources of financing. d) Percent increase in the number of farmers that participate en the technical assistance offered by the NGOs (disaggregated by gender). e) The Asociación de Instituciones de Promoción y Educación (AIPE), the SID Associate in Bolivia, assumes an effective technical assistance role for NGOs by conducting workshops. a) Goal: 25% average improvement. b) Goal: 75% c) Goal: 25% d) Goal: 25% increase in the number of participants e) The Asociación de Instituciones de Promoción y Educación (AIPE) provides information to its associates and provides technical assistance directly to the member organizations. a) Data resulting from the use of diagnostics with NGO participants. Baseline data available from NGOs when they begin the technical assistance agreement with SID. Information from annual reports of participating NGOs. b) Data from annual self-evaluations. c) Data from annual self-evaluations. d) Annual and quarterly reports from NGOs; SID technical assistance reports e) SID and Asociación de Instituciones de Promoción y Educación (AIPE) annual evaluations and shared responsibilities. External evaluation Final Program Evaluation Training and technical assistance provided by NGOs. Application of diagnostic instruments at the beginning and end of the TA activities. Implementation of Program with the AIPE and its gradual acceptance of responsibilities and is responsible for technical assistance in the third year. Matching Grant Program End-of-Program Evaluation 72 Table A-2. Logical Framework for the Matching Grant Technical Assistance Program, SID-Peru Planning Matrix Final Objective: To recover soil and pasture productivity and increase farm family incomes in the Peruvian highlands. PROGRAM OBJECTIVE INDICATOR BASELINE AND GOALS MEASUREMENT METHODS, SOURCE AND FREQUENCY OF DATA COLLECTION PRINCIPAL ACTIVITIES 1. Soils and pastures recovered; increase in production and incomes for 6,600 farm families in 220 communities attended by 22 NGOs. a) Percent increase in average incomes of farm families in the areas attended by the NGOs that participate in the 4 phases of technical assistance and that faithfully follow the new methodologies for the duration of the Program. b) Percent increase in the average number of hectares of land per family utilized for crops and pastures en the areas attended by the NGOs that participate in the 4 phases of technical assistance and that faithfully follow the new methodologies for the duration of the Program. a) Goal: 30% increase in average farm family income. b) Goal: 20% increase in the number of hectares. Results reported by NGOs during the annual evaluation meetings – sample of participating communities. NGO evaluations Annual evaluation meetings Surveys accomplished by SID and the Coordinador Rural Project Evaluation NGOs provide training and technical assistance to farmers for the recuperation of soils and pastures, and for the increase in agricultural production and incomes. Preparation of technical assistance materials. 2. Effective training and technical assistance for farmers en the recovery of soils and pastures and in the increase of production and incomes, established in a sustainable form. a) Positive changes observed in the results of the self-evaluation instruments. b) Percent of NGOs that report results in a timely manner. c) Percent of NGOs that diversify their sources of financing. d) Percent increase in the number of farmers that participate en the technical assistance offered by the NGOs (disaggregated by gender). e) The Coordinator Rural assumes the leadership in the NGO workshops with the objective of providing more effective technical assistance. a) Goal: 25% average improvement. b) Goal: 75% c) Goal: 25% d) Goal: 25% increase in the number of participants e) The Coordinador Rural provides information to its associates and provides technical assistance directly to the member organizations. a) Data resulting from the use of diagnostics with NGO participants. Baseline data available from NGOs when they begin the technical assistance agreement with SID. Information from annual reports of participating NGOs. b) Data from annual self-evaluations. c) Data from annual self-evaluations. d) Annual and quarterly reports from NGOs; SID technical assistance reports e) SID and Coordinador Rural annual evaluations and shared responsibilities. External evaluation f) Final Program Evaluation Training and technical assistance provided by NGOs. Application of diagnostic instruments at the beginning and end of the TA activities. Implementation of Program with the Coordinador Rural and its gradual acceptance of responsibilities. The Coordinador Rural is responsible for technical assistance in the third year. Matching Grant Program End-of-Program Evaluation 73 Table A-3. Logical Framework for the Matching Grant Technical Assistance Program, SID-Central Office Planning Matrix Final Objective: To recover soil and pasture productivity and increase farm family incomes in the Bolivian and Peruvian highlands. PROGRAM OBJECTIVE INDICATOR BASELINE AND TARGETS MEASUREMENT METHOD, DATA SOURCE AND FREQUENCY OF DATA COLLECTION MAJOR PLANNED ACTIVITIES 1) SID’s capacity to provide technical assistance to NGOs in methods for helping farmers reclaim soils and pastures and increase production and income. a) Percent of NGOs requesting all 4 phases of technical assistance after participating in planning workshop b) Percent of SID Country Office Staff time spent on technical assistance to NGOS a) Baseline: 0 Target: 66% Year 1:10/14 (Bolivia) Year 1: 6/9(Peru) Year 2: 12/17 (Bolivia) Year 2: 8/12 (Peru) Year 3: 12/18 (Bolivia) Year 3: 8/12 (Peru) EOP: 54/ 82 (Peru+Bolivia) b) Baseline: 0 Target: 50% (average over 3 years) a) Lists: NGO participants at annual evaluation workshop/NGO participants at planning workshop (reported yearly) b) Timesheets and monthly reports (data collected monthly and reported yearly) a) Planning workshop, implementation, technical assistance, evaluation workshop b) Technical assistance and strategic planning activities with NGOs 2) SID’s capacity to help NGOs monitor, evaluate, and report results and improve performance. a) Percent of NGOs requesting help in monitoring, evaluation, and reporting after participating in planning workshop. b) Percent of SID County Office staff time spent helping NGOS conduct baseline surveys, analyze results and in monitoring, evaluating and reporting their results. a) Baseline: 0 Target: 66% Year 1:10/14 (Bolivia) Year 1: 6/9(Peru) Year 2: 12/17 (Bolivia) Year 2: 8/12 (Peru) Year 3: 12/18 (Bolivia) Year 3: 8/12 (Peru) EOP: 54/ 82 (Peru+Bolivia) b) Baseline: 0 Target: 50% (average over 3 years) a) Lists: NGOs requesting technical assistance for M&E/NGO participants at planning workshop (reported yearly). b) Timesheets and monthly reports (data collected monthly and reported yearly) Technical assistance to NGOs at workshops, in the field, and in their offices on M&E design, data collection, analysis, and reporting. Matching Grant Program End-of-Program Evaluation 74 Table A-3. Logical Framework for the Matching Grant Technical Assistance Program, SID-Central Office Planning Matrix Continuation PROGRAM OBJECTIVE INDICATOR BASELINE AND TARGETS MEASUREMENT METHOD, DATA SOURCE AND FREQUENCY OF DATA COLLECTION MAJOR PLANNED ACTIVITIES 3. Improved capacity of SID to supervise and support field programs a) Percent of HQ staff time spent on field support activities. b) Number of months/year when reports are submitted on time and are of acceptable quality c) Percent of responses of HQ to field staff requests a) Baseline: 0 Target: 60% b) Baseline: 0 Target: 9/12 (75%) c) Baseline: 0 Target: 90% (# of responses/ # of requests) a) Data come from timesheets (collected monthly and reported to USAID by semester) b) Information collected monthly and reported by semester. c) Data from log of phone and email requests and responses (data tallied monthly and reported by semester) Operational support activities to the field include: assistance in the design of planning and evaluation workshops; field supervision visits; responses to requests for information; assistance with and review of reports, manuals, and publications; timely provision of financial and human resources. 4. Improved capacity of SID in Strategic Planning Five-year strategic plan adopted by SID Board of Directors Baseline: no Target: yes Vote of acceptance of plan by Board of Directors