IMPACT EVALUATION OF THE PEACE CORPS SMALL PROJECT ASSISTANCE (SPA) PROJECT Conducted by Assia Alexieva (USAID/Bulgaria) Katia Alexieva (USAID/Bulgaria) Ivica Vasev (USAID/Macedonia) August 2003 2 Table of Contents Table of Contents......................................................................................................................... 2 Acknowledgments........................................................................................................................ 3 Executive Summary..................................................................................................................... 4 Background ................................................................................................................................... 6 Purpose ......................................................................................................................................... 7 Evaluation Questions................................................................................................................... 7 Methodology ................................................................................................................................. 8 Impact ............................................................................................................................................ 9 Sustainability...............................................................................................................................12 Capacity Building .......................................................................................................................17 Monitoring and Evaluation........................................................................................................20 Other Findings ............................................................................................................................21 Other Recommendations ..........................................................................................................22 Collaboration with Other USAID Programs............................................................................22 General Observations ...............................................................................................................23 Suggested Joint Steps ..............................................................................................................26 Annex 1: Scope of Work ..........................................................................................................27 Annex 2: Map of Site Visits .....................................................................................................32 Annex 3: Project Description Matrix.......................................................................................33 Annex 4: List of Interviewees ..................................................................................................41 Annex 5: Questionnaires .........................................................................................................42 Annex 6: Notes and Quotes from Interviews ........................................................................45 3 Acknowledgments This evaluation is a joint effort of evaluators from USAID/Bulgaria and USAID/Macedonia. The team would like to express its gratitude to Debra McFarland, USAID/Bulgaria Mission Director, and Dick Goldman, USAID/Macedonia Mission Director, for their support and trust in this pioneering effort. The evaluation would not have been possible without the continued support and advice of Ivanka Tzankova, USAID/Bulgaria Program Officer. The team also acknowledges the support of Alfreda Brewer, USAID/Macedonia Program Officer, who allowed Ivica Vasev’s participation in one of the busiest periods for every program office. The EXO offices of both missions were really effective in providing logistic support to the team. Last but not least, the evaluation members would also like to acknowledge the responsiveness and support of Valentina Dobreva, SPA Coordinator, and Carl Hammerdorfer, Peace Corps/Bulgaria Country Director. 4 Executive Summary Purpose and Methodology. This evaluation aims at assessing the impact and sustainability of SPA grassroots activities implemented across Bulgaria in the past five years (1998-2003). The evaluation took the form of a rapid appraisal, using a sample of 16 projects out of the 141 activities implemented. The selected activities included projects from all fiscal years reviewed and scattered around five different locations in Central and Northeastern Bulgaria (Apriltzi, Gabrovo, Veliko Tarnovo, Varna, and Balchik). Impact. The Evaluation Team has identified eight factors for achieving long-term impact: 1) a well-defined project proposal; 2) participation; 3) a small to medium-sized community; 4) mid-term planning; 5) continuity; 6) preparedness for the local context; 7) PCV-host fit; and 8) personality. Focusing on these conditions, the Evaluation Team found that: ß SPA activities are generally not community-driven. ß The wider community is not sufficiently involved in the SPA process, and projects do not reflect community priorities as identified in a participatory needs assessment. ß While SPA has greater impact in smaller to medium-sized cities, effect is largely contingent upon the participation of and cooperation among various community actors. ß Impact is greater when a comprehensive sustainability plan has been developed and advanced after project completion. However, this requirement is not very well￾observed, and largely varies in level of detail and overall strategic planning required. ß Impact seems greater when 2-3 PCVs have worked in a row in a certain community. ß Some general factors also impact SPA activities, such as PCV preparedness for the local context; the fit between PCVs and host organizations; and, as subjective as it may seem, personality. To respond to these deficiencies, the Evaluation Team recommends that: ß Stricter requirements are introduced for conducting comprehensive needs assessments that incorporate input from all community actors. ß Uniform rules and guidance are introduced with regard to the sustainability plan section of SPA project proposals. ß PCVs do not base their agenda using only their own judgment or relying on feedback received from a single organization. Sustainability. The Team distinguished the following three factors as conducive to sustainability: 1) proactiveness and capacity of the host organization; 2) activity diversification; and 3) partnership building. It further ascertained that: ß Sustainability and the amount of the SPA grant are not proportionally related, with the former being largely subject to the proactiveness and know-how of the host organization. ß Proactive host organizations with developed institutional capacity have managed to sustain the SPA products/processes by diversifying their activities and introducing innovative income generation techniques. 5 ß The likelihood of making SPA initiatives sustainable increases with the existence of a working partnership among the various local community players. In this regard, the role of local government is central. ß Project results seem much more visible when used as a mechanism for complementing a certain community’s concerted efforts. At the same time, a number of limiting factors prevent certain SPA activities from reaching sustainability. These could be summarized as follows: o Non-implementation of projects’ sustainability plans; o of post-project accountability (about equipment and intellectual property); o Need for fundraising mechanisms established; o Migration of SPA Trainees and NGO donor dependence (both beyond PC’s manageable interest). These limitations stem from deficiencies such as: ß Although project proposal forms require the development of a sustainability plan, the relatively short duration of SPA activities and the limited funds available do not allow for their actual implementation. ß There is a lack of established procedures for the transfer of property to recipient organizations, and consequently, transfer of accountability. ß None of the host organizations visited had an established fundraising mechanism as a way of sustaining the SPA results and processes. The Team recommends that PC: ß Instruct PCVs assigned to cities with completed SPA activities to check on their status and, if necessary, provide additional assistance. ß Invest in NGOs with a proven record; ß Involve local government as much as possible. ß During the last 1-2 months of project implementation, focus on helping host organizations put sustainability plans into action. ß Assist local communities in developing fundraising mechanisms. Capacity Building. ß The SPA Program has positively impacted the wider community in terms of providing key training to target community groups in computer skills and environmental education. ß However, the 16 SPA activities reviewed have had limited impact in terms of building the capacity of local counterparts to write project proposals, manage programs, and conduct fundraising activities. ß The Team found that all project proposals have been written by PCVs themselves, with limited participation of local counterparts. Furthermore, 35% of the organizations contacted thought PCVs need more training in proposal writing. ß Host organizations have also not been sufficiently involved in the performance process as PCVs have been the leading agents of project implementation. Equipment vs. Capacity Building. Various communities view differently the “trade-off” between equipment and capacity building. Young organizations need both equipment 6 and capacity building, with equipment being their primary need as it makes them more competitive. The need for equipment is much lower for experienced organizations which need capacity building in fundraising, lobbying and advocacy building. To better match the capacity building skills of host organizations, the Evaluation Team recommends that PCVs: ß Focus their assistance to municipalities or experienced NGOs on building their fundraising, lobbying, advocacy building, and public relations skills. ß Make their role in proposal writing and project development more advisory. ß Encourage host organizations to write the first project proposal draft. ß Involve host organizations more in the development of public information materials and awareness campaigns. Monitoring. Generally, project proposals do not contain any baseline data on the selected Quality of Life indicators; completion reports do not report against them; and the majority of monitoring plans simply follow the sequence of events. It is therefore difficult to quantify SPA’s impact and the attitudinal change it has caused. The development of a system of quantitative and qualitative indicators is recommended. Other Findings and Recommendations. ß No stickers on SPA-funded equipment; ß Logo requirement not universally observed; ß False belief that host organizations somehow own PCVs; ß Communities highly value the simplified application process. The Team recommends that PC introduce the use of USG stickers, and better enforce the use of Peace Corps and USAID logos on all publication materials. Collaboration with Other USAID Programs. Also, general observations and suggested next steps, p. 23 and 26, respectively. 1. Background For the past ten years, USAID has funded the Small Project Assistance (SPA) Program implemented by Peace Corps/Bulgaria. The program involves community development projects in support of the three Peace Corps assistance sectors: environmental protection, community economic development, and English language education. In addition, many PCVs and their communities develop SPA projects that are important to the community but not necessarily linked to the PCV primary area or assignment. The SPA Program aims at encouraging self-help efforts on behalf of local communities, in collaboration with Peace Corps Volunteers, to identify common concerns, plan small￾scale activities, and develop strategies to address those issues. It is based on the principle of “help people help themselves” and focuses on facilitating sustainable development at the local community level. The intent is to provide an efficient 7 mechanism that can respond quickly and effectively to community initiatives with an immediate and lasting impact at the local level. SPA grants provide community organizations with small amounts of financial support ($3,000 on average: the lowest being $160, the highest: $9,803) to carry out activities they have identified. Funds can support training, technical services, and materials development – all designed to strengthen community-based development efforts. Activities address a variety of areas, including upgrading infrastructure, computer instruction, environmental education, biodiversity conservation, income-generating activities, civil society development, youth issues, etc. PCVs play an important role in facilitating the planning process, assisting with implementation, building local capacity, and monitoring project success. In an attempt to appraise the impact of these grassroots activities and their effect on the capacity and well-being of local communities, USAID has planned an impact evaluation to be conducted by a team of internal experts and a visiting evaluator from USAID/Macedonia. 1.1 Purpose The overall purpose of this evaluation is to assess the impact and sustainability of SPA grassroots activities implemented across Bulgaria within the past five years (1998- 2003). The study seeks to ascertain whether the overarching goals of the program have been achieved and what its long-term effect on local communities is. Special emphasis has been put on identifying some of the lessons learned and the existing challenges to “making a difference” and achieving long-term sustainability. To respond to the interest expressed by the Peace Corps Mission in Sofia, the Evaluation Team has put special emphasis on exploring the value that local communities attribute to the purchase of equipment and building their capacity, as well as on seeking the optimal balance between these major SPA components. 1.2 Evaluation Questions This report seeks to respond to the following questions, as defined in the initial Evaluation Scope of Work: ß Have activities brought any change to local communities? ß Are the SPA products/processes established with SPA assistance still operational and taken care of? What is their current status? ß What is the quality of the existing monitoring, evaluation and reporting systems? ß Which type of activities has been most or least effective in terms of achieving lasting impact and sustainability? 8 Table 2 Fiscal Year Number of Projects Reviewed 1998 5 1999 4 2000 3 2001 3 2002 1 Table 1 City Population Size Apriltzi 5,000 Balchik 13,500 Gabrovo 71,386 Veliko Tarnovo 301,284 Varna 346,000 Table 3 Project Type Number of Project Reviewed Environmental Education 5 Business Development 4 Education 3 Environment 3 NGO 1 ß Has local capacity been built through the SPA Program? ß What is the project’s effect on the relationships among the various community actors (municipality, businesses, schools, etc)? ß Have there been any collaboration efforts with other USAID activities implemented in this city/region? ß What are the lessons learned? ß What are the challenges/obstacles to achieving sustainability and/or having long-term impact? ß Recommend program performance measures. 1.3 Methodology This evaluation took the form of a rapid appraisal, which allowed for a quick, inexpensive, and relatively reliable way of grasping the essence, impact, and sustainability of the Peace Corps SPA Program. Following a comprehensive review of the 141 activities implemented through SPA between FY1998 and FY2002, a sample of 16 projects was selected in five different locations in Central and Northeastern Bulgaria (Apriltzi, Gabrovo, Veliko Tarnovo, Varna, and Balchik). (See Annex 2: Map of Site Visits and Annex 3: Project Description Matrix). Given the evaluation’s particular focus on impact and sustainability, the Evaluation Team chose to review completed projects only. The following criteria were used for the sample selection: o Two or more SPA activities completed in a particular city; o Diverse community size (large, medium, and small cities); o Projects representing the five fiscal years reviewed (FY98-FY02); o Proportional representation of projects in all three major sectors; o Maximum variety of implemented activities; o Feasibility in terms of time and effort. At the outset of program evaluation, a comprehensive review of key documentation was conducted, including the SPA Proposal Development Guide for Volunteers and Counterparts, the 2002 Country Level SPA Program Evaluation, selected project proposals and their respective final reports, if and when available. To get a better understanding of the Program’s historical perspective and review process, key informant interviews were conducted with the Peace Corps Country Director and the Peace Corps SPA Coordinator. The Planning Phase was followed by a one-week field trip to the 16 SPA activities selected. Table 1, 2 and 3 summarize the geographic distribution, program type, and implementation year of the SPA projects reviewed. In the course of the evaluation, the 9 Team conducted 11 individual and 8 group interviews, meeting 31 project counterparts, partners, and/or beneficiaries (see Annex 4: List of Interviewees). The interviews were conducted using a set of Guiding Questionnaires, specifically tailored for the three groups of interviewees (Annex 5). Free conversation style was, however, the Team’s primary interview technique, as a way of better grasping the uniqueness and specificity of each community’s approach and on-the-ground activities. In all five cities, the Team met with local government representatives (two mayors and three senior municipal officials) as critical actors in all aspects of community development. Constraints: o Except for the two volunteers in Gabrovo, the Evaluation Team did not have the opportunity to meet with PCVs who have actually participated in the development and implementation of a specific SPA project in the five selected cities. This drawback is due to the specific timing of the evaluation, which coincided with the departure of many volunteers. o The project sample included activities implemented in Central and Northeastern Bulgaria only as a country-wide coverage would not have been cost-efficient. Nevertheless, the Team feels completely confident about the representativeness and validity of the evaluation findings. o No interviews or surveys were conducted among the direct/indirect program beneficiaries (citizens, school children, etc.), due to the lack of time and the vacation season that coincided with the timing of the field trip (August 11-15). o The timing of the evaluation prevented the team from including a member of the Peace Corps. Such involvement could have been beneficial to the evaluation process, especially in terms of getting a better grasp of the SPA internal rules and procedures, and learning about any improvements introduced to the SPA process in the past five years. 2. IMPACT This section seeks to respond to the following evaluation questions: ß Have activities brought any change to local communities? ß Which type of activities has been most/least effective in terms of achieving long￾lasting impact? ß What is the project’s effect on the relationships among the various community actors (municipality, businesses, schools, etc)? 2.1 Findings ß Activities Are Not Community-Driven. Although SPA activities are supposed to be locally initiated, they are generally not community-driven. In the majority of 10 the cases, projects respond to the needs of the first organization that has approached the PCV and reflect the priorities of a very limited number of beneficiaries. Except for the three Balchik activities and one of the projects in Varna, all activities have been selected by the PCV him/herself following limited consultations with the respective host organization. ß Inadequate Involvement & Participation of the Wider Community. Projects that do not reflect the community priorities as identified in a participatory needs assessment do not go beyond the PCV’s term of service. They simply die out with the PCV’s departure. For example, the Apriltzi Tourist Association Project and the Support to Apriltzi Craftsmen Project have brought temporary change and short-term impact to the host community in terms of developing, furnishing and equipping the local Tourist Office and producing a nice set of information materials. Mechanisms to promote local crafts have also been introduced, such as a craftsmen referral system and a showcase displayed at the Tourist Office. Nevertheless, the Evaluation Team found out that a year and a half after project completion, the craftsmen initiative had entirely fallen apart; the equipment provided to the Tourist Office was missing; and people did not feel motivated to move forward and build on the results achieved. There were personal conflicts among the main counterparts, which existed at the time the PCV worked there, which did not reach any resolution. Generally, there was limited community mobilization to advance these initiatives as the two projects were not selected by the wider Apriltzi community (the Tourist Association Project was suggested unilaterally by the former Manager of the Tourism Office). Conversely, the community of Balchik directed the PCV to the local needs and potential use of SPA funding. It actively participated in the planning phase and subsequent project development and, consequently, managed to expand the activities initiated through the first SPA grant in 1999. A similar guiding approach has been utilized by the community of Veliko Tarnovo where SPA activities have also expanded and persisted in time. ß Community Size Is Not A Major Factor. Intra-Community Relationships Are. The partnership among the Municipality of Balchik, the Children’s Complex, local schools, businesses, media, and other institutions that joined forces in the Child and Nature Project survived in time in this relatively small town of 13,500 citizens. This was largely due to the personal proactiveness and resourcefulness of the local actors. In contrast, the Tourist Association Project and the Support to Apriltzi Craftsmen Project in a small town like Apriltzi (5,000 citizens) has hardly achieved any long￾term impact. A major hindrance to this end was the total lack of trust and collaboration among the municipality, the Tourist Office, and local craftsmen. 11 Factors for Having Long-Term Impact ß A Well-Defined Project Proposal ß Participation ß Small to Medium-Sized Community ß Mid-Term Planning ß Continuity ß Preparedness for the Local Context ß PCV-Host Fit ß Personality In a medium-sized city like Veliko Tarnovo, the spirit of cooperation that existed between the municipality and the PCV’s host organizations proved crucial for SPA’s impact and sustainability. While this positive relationship was not generated by the SPA Program, the projects implemented contributed to strengthening this existing partnership. Notably, decisions related to project implementation were taken by consensus among the various players, and without any separation of local government from civil society. In a large city like Varna, the effect varies from project to project. In some cases there was no collaboration with other community actors (for example, the English Lending Library Modernization Project). In other cases, SPA proved semi￾successful in terms of setting the basis for inter-agency cooperation after project completion (the Black Sea Model in Varna). ß Continuity Has A Positive Effect. Major change has occurred in the town of Balchik through the three SPA activities implemented with the help of three consecutive PCVs. Today, the host organizations feel capable of developing project proposals and raising additional funds for the Municipal Children’s Complex. ß No Mid-Term Planning. While no mid-term planning was required under the old SPA Proposal Regulations and Procedures, the latter has become obligatory in the current uniform SPA proposal form. However, this requirement is not well￾observed, and largely varies in terms of level of detail and overall strategic planning required. (Note: this evaluation finding is closely related and applicable to the Sustainability Section as well). Some general factors stemming from the overall Peace Corps Program in Bulgaria have certain impact on SPA’s process and end results. While these do not have direct relationship to the SPA Program, the Evaluation Team considered it important to point some of these out: ß PCVs Not Well-Prepared for the Local Context; ß PCV-Host Fit; ß Personality. See Section 9: General Observations. 2.2 Conclusions To achieve long-term impact, it is essential for short-term projects like SPA to reflect local needs as identified by the wider local community. The involvement and participation of various community actors is crucial as the 12 less their participation, the lesser the project’s impact and sustainability. While the SPA Program seems to have greater impact in smaller to medium-sized cities, effect is largely contingent upon the participation of and cooperation among various community actors. However, it is generally not possible to conclude that relationships among different stakeholders have improved or worsened as a result of SPA activities. The potential for “making a difference” or “having impact” is much greater when a comprehensive sustainability plan has been developed and advanced after project completion. Impact in terms of increased capacity and viability of local organizations seems greater when 2-3 PCVs have worked in a row in a certain community, building upon each other’s achievements, helping the community to make the products of previous SPA grants sustainable, and expanding the skills of local organizations. It should be noted, however, that this approach is somewhat doubled-edged as it may foster dependency on behalf of the host organization on PCV support. The larger community will also have fewer opportunities to benefit from the SPA Program, if PCVs are based in only one organization. 2.3 Recommendations ß Peace Corps should introduce and enforce stricter requirements for conducting comprehensive needs assessments that incorporate input from all community actors. ß In identifying local needs, PCVs should strive to involve the wider community. They should be more proactive in establishing contacts with all main actors and prioritize their needs based on the multi-faceted feedback acquired. ß Stricter and uniform rules and guidance should be introduced with regard to the sustainability plan/mid-term planning section of the SPA project proposals. ß At the time of workplan approval, Peace Corps officers should ensure that PCVs have not based their agenda using only their own judgment or relying only on feedback received from the host organization. 3. SUSTAINABILITY This section encompasses the Evaluation Team’s findings with regard to the following evaluation questions: ß Are the SPA products/processes established with SPA assistance still operational and taken care of? What is their current status? 13 ß Which type of activities has been most or least effective in terms of achieving sustainability? ß What are the challenges/obstacles to achieving sustainability and/or having long￾term impact? 3.1 Findings ß Little Correlation between Grant Size and Success. Although the Environmental Film Festival in Veliko Tarnovo was supported with only $690 back in FY1998, the event continues to take place annually, and has even been expanded to a Week of Ecology organized by a municipality-supported eco-club. Today, it involves a variety of public events, such as cleanups, exhibitions, drawing contests, discussions, etc. Similarly, the anti-litter activities initiated through the Trash Container Environmental Art Project ($1,555) took a life of their own, supported by the municipality. The practice of paining trash containers also lived through the years. New activities were introduced such as an exhibition of broken municipality property and trash cans, in an effort to demonstrate the cost of broken items and teach good citizen practices. Conversely, the Black Sea Model in the Varna Sea Garden was repaired only two years ago, but the SPA investment that adds up to $4,170 is generally lost due to the lack of consistent follow-up and regular maintenance. The $3,307 awarded to the Apriltzi Tourist Association has also not ensured the desired impact. ß Proactiveness and Institutional Capacity of Host Organizations Are Key to Sustainability. Proactive host organizations with developed institutional capacity have managed to sustain the SPA products/processes by diversifying their activities and introducing innovative income generation techniques. Notably, the Call Center established under the Open Doors Project in Veliko Tarnovo is still operational and provides very well advertised consulting services to start-up businesses. Furthermore, the idea has been replicated to the establishment of a Domestic Violence Hotline. Similarly, the equipment provided to YMCA Gabrovo has been upgraded through another project, and is being used for the production of a wide range of informational materials. Training to local citizens continues. While the computer workstation is really not self-sustainable, it adds value to the local community as it has not commercialized in time and sticks to its initial educational purposes. As another example of the sustained results that proactive organizations can accomplish, the Public Environmental Center for Sustainable Development (PECSD) in Varna has been creative enough to utilize the equipment and resources provided under SPA to keep up the Center’s services and expand its catalogue and activities. All materials were provided for free at the beginning. Gradually, the Center introduced fees and subscriptions for using its existing resources. Currently, PECSD has 40-50 individual subscribers as well as seven schools and kindergartens. They have managed to expand their facilities through other projects, 14 thus purchasing a TV, a VCR, and a video library. The monthly eco-bulletin developed under SPA is still being published. The eco-clubs established in all Gabrovo schools under the Ecological Education – One Step Forward Project resulted in two permanently functioning eco-clubs and one NGO at the university level. ß Local Government Participation Is Important. The two SPA activities in Balchik significantly strengthened the existing partnership among the municipality, the Children’s Complex, and local businesses. Today, the Complex is still capable of offering for free the computers and ceramics kiln acquired through SPA as the Municipality entirely supports the Center’s operations. While the local administration in Gabrovo still needs some encouragement to cooperate, it participates in all public events organized by the Youth Cultural Information Center, some of which are implemented under the aegis of the municipality. This year’s Earth Day was jointly organized between the Municipality and the Central Balkan National Park (celebrations that have initially been supported through SPA). Reversely, the Municipality of Apriltzi did not feel itself involved in both the planning and implementation phases of the SPA projects. The latter did not contribute to building increased trust and cooperation among the local players. Consequently, none of the SPA products are still operational. ß No Sustainability Plan Implementation. Although the project proposal forms require the development of a sustainability plan, the relatively short duration of SPA activities and the limited funds available do not allow for the actual implementation of these plans. For example, the sustainability objectives of the English Lending Library Modernization Project in Varna have completely not been met. Although the equipment provided was supposed to generate a modest income for the facility from user fees and subscriptions, the computer and copier provided under SPA are used by the library staff only. There is no indication that the equipment can be used by outside visitors, with or without payment. The library’s inventory has not been computerized (which was another improvement envisioned under SPA); neither does the library function as the resource center it was supposed to turn into. For the past year, not a single book has been purchased to expand the library’s assets. Instead of increasing its membership, the number of subscribers went down four times following project completion. In a similar fashion, the Craftsmen Initiative in Apriltzi anticipated reaching sustainability through the establishment of a craftsmen association and the maintenance of the SPA-funded Internet website. Today, the webpage is not operational and all contacts with the company that had developed the website seem lost. Although the registration fee for establishing the craftsmen NGO was disbursed 15 by SPA, the organization is still not registered, and there is no prospect for further development. In contrast, the environmental training of school kids supported by SPA and implemented by the Golden Sands National Park and the Public Environmental Center for Sustainable Development in Varna persisted over time and was replicated in other schools, including specialized schools for vulnerable children, by using the methodology introduced by the PCV. ß Lack of Post-Project Accountability. There is a lack of established procedures for the transfer of property to recipient organizations, and consequently, transfer of accountability. For example, the digital camera and laser printer purchased with SPA funding for the Tourist Association in Apriltzi are currently not within the inventory of the Tourist Center, and the current manager has no idea of their whereabouts. Despite the gravity of this situation, nobody seems accountable for the missing equipment. The same applies to the transfer of products of intellectual nature. All of the organizations that have developed some kind of information materials under the SPA Program, do not hold the printing layout for updating and reproducing these materials. The Apriltzi Tourist Association also does not own the copyrights for the map produced under SPA. ß No Fundraising Mechanisms Established. None of the host organizations visited had an established fundraising mechanism as a way of sustaining the SPA results and processes through monetary or in-kind community contributions. PCVs have not provided any training to host organizations in fundraising and advocacy building. Even though the English Lending Library Modernization Project in Varna originally aimed at “generating income to ensure sustainability,” no such planning or follow-on steps had been undertaken during the time of the project or after its completion. While fundraising has generally not been part of the majority of the PCV’s workplans and SPA project objectives, some of the counterparts contacted thought some guidance in how to use alternative financing mechanisms could have been very useful for sustaining and diversifying their activities. As the Library’s Manager has put it: “Bulgarians tend to have a psychological barrier for getting out and asking for money. However, once they see the results of their fundraising campaign, they become motivated to further use this approach to advance their goals and activities.” ß Donor Dependence. A lot of the organizations visited stick to their educational, not￾for-profit principles and do not charge fees to allow for wider access. While this is absolutely praiseworthy, it also impacts community partners adversely in terms of keeping them heavily donor dependent. ß Migration Is A Hindering Factor. Local counterparts point out that a major hindrance to sustainability is the flow of SPA trainees to the big cities. According to 16 Factors for Achieving Sustainability ß Proactiveness and Capacity of the Host Organization ß Activities Diversification ß Partnership Building Limitations Sustaining SPA Products ß No Sustainability Plan Implementation ß Lack of Post-Project Accountability ß No Fundraising Mechanisms ß Migration of Trainees ß Donor Dependence local estimates, only 10% of the volunteers associated with YMCA and the Youth Cultural and Information Center have stayed in Gabrovo. Of those who have migrated, less than 1% has come back permanently. While the Team recognizes that this factor is beyond the Peace Corps’ manageable interest, the evaluation officers considered it worthwhile to point out this shortcoming. 3.2 Conclusions Sustainability and the amount of the SPA grant are not proportionally related. It is largely subject to the proactiveness and know-how of the host organization on how to further develop the products and processes introduced under SPA. Furthermore, with no implementation of the project proposal sustainability plan, sustainability becomes entirely dependant on the resourcefulness and commitment of the recipient organization. SPA projects seem to have greater sustainability when followed by activities that diversify and continuously develop the scope of the SPA projects. The likelihood of making SPA initiatives sustainable increases with the existence of a working partnership among the various local community players. Given the limited resources available under the SPA Program, project results seem much more visible when used as a mechanism for complementing a certain community’s concerted efforts. Also, whatever the nature of the SPA projects (infrastructure, civil society capacity building, income generation), the role of local government is central to reaching the desired objectives and making the outcomes sustainable. 3.3 Recommendations ß Peace Corps should instruct PCVs assigned to cities with completed SPA activities to check on their status and, if necessary, provide additional assistance that can ultimately contribute to achieving sustainability. ß Keeping in mind that the NGO sector in Bulgaria is still donor dependent and the big donors are gradually phasing out, it seems reasonable to invest in NGOs with a proven record of successful/sustainable projects (ex. Veliko Turnovo and Balchik). The need for shortlisting and filtering host organizations is further strengthened by 17 the short-term nature of the SPA projects, thus implying that investing in newly established organizations will often not bring the desired sustainability effect. ß PCVs should strive to involve the local government as much as possible. Municipalities are the primary institutions that can make the SPA initiatives sustainable by supporting them financially and administratively. ß Peace Corps should seek to observe more strictly the implementation of the sustainability plans that are part of each project proposal. Moreover, during the last 1-2 months of project implementation, PCVs should focus on helping host organizations to put these plans into action. If not, they will merely stay on paper. ß PCVs should particularly assist local communities in developing fundraising mechanisms as a way of opening avenues for the launch of new initiatives and acquiring resources for SPA product support and maintenance. 4. CAPACITY BUILDING This section outlines the Team’s findings on the following evaluation questions: ß Has local capacity been built through the SPA Program? ß What do communities need and value more: equipment or capacity building? 4.1 Findings Limited Capacity Building Impact. Of the 17 organizations visited, only four believe PCVs have contributed to building their project development capacity. Those who have managed to support their activities through alternative financing are experienced organizations with a proven track record of donor grants and past experience in proposal writing (YMCA Gabrovo, the Open Doors Project, municipalities). This limiting effect is largely due to a couple of deficiencies: 1) No Proposal Writing Skills Developed. With the exception of the Renaissance Woman Project in Varna and SPA activities in Balchik, all project proposals have been written by the Peace Corps Volunteers themselves, with limited participation of local counterparts. Although host organizations have taken part in the conceptualization process, the specific know-how of grant application development and proposal writing has not been transferred to the local level. The Children’s Complex in Balchik is one such positive exception, assisting the local kindergarten and other institutions in proposal writing, applying its experience from the three SPA activities. However, 35% of the organizations contacted thought PCVs needed more training in proposal writing. A few of them shared with the Evaluation Team that 18 some of the volunteers lacked well-developed project development skills. Proposal writing has often been novel to the PCVs, thus turning the SPA process into a learning experience for them too. 2) Insufficient Involvement of Local Counterparts in Product Development. PCVs have been the leading agents of project implementation, thus leaving host organizations not sufficiently involved in the performance process. As a result, although the Central Balkan National Park has greatly benefited from the work of the PCV in terms of launching Earth Day public events and raising public awareness through the development of a great number of posters and brochures, the Park Directorate does not currently have the internal capacity to develop these information materials. Work has entirely been done by the PCV without the direct involvement of the Parks’ Interpretative Education Specialist. Today, the Central Balkan Directorate procures such contracts to outside implementers. As mentioned earlier, it does not own the matrices for reproducing the SPA￾funded posters and brochures; neither have park specialists acquired the capacity to develop such resources by themselves. Enhanced Computer and Environmental Education. Although SPA has had limited impact on the managerial and public outreach capacity of host organizations, citizens of SPA-supported cities continue to benefit from the computer trainings and environmental classes introduced under SPA. Hundreds of children from the cities of Varna, Balchik and Gabrovo continuously expand their knowledge of the environment and the need for nature protection through the environmental training activities conducted under SPA. Using the computer workstation developed with SPA funding, YMCA Gabrovo attracted numerous young people to the Center’s facilities with the creation of the Y-Net Studio which provides access to the Internet and basic computer training. More than 15 active volunteers maintain the workstation’s equipment, use it for the design the published materials, and update the organization’s website. In addition, target community groups, including youth, parents and unemployed persons, have also received basic computer training initiated under the SPA Project. The same formula was also used by the Open Doors Project in Veliko Tarnovo. Equipment vs. Capacity Building. Various communities view differently the “trade-off” between equipment and capacity building. Some recognize a pressing need for equipment; others value the PCV’s bright ideas and capacity building skills more. The Mayor of Apriltzi sees “the truth as lying in the middle.” He believes it is important for people to see something tangible and visible resulting from the SPA, which can in turn motivate citizens to further work together and enhance their well-being. On the one hand, the equipment provided under SPA to Local Agenda 21 proved critical for the organization’s survival and the massive expansion of the Open Doors Project in Veliko Tarnovo. After getting its first grant ever from SPA, the organization used the equipment as in-kind contribution for other grant applications and further fundraising. Today, the amount acquired for the Social and Recreational Center in Veliko Tarnovo exceeds EUR 150,000. Similarly, the books and equipment provided to the NGO 19 Environmental and Development Center in Varna was also essential for starting up the resource center’s operations. Conversely, SPA’s non-equipment component is also highly valued, especially by mature NGOs and municipalities. The Mayor of Veliko Tarnovo sees the value of PCVs work mainly in “the introduction of avant-garde ideas that are later built upon.” Similarly, Local Agenda 21 also sees the role of the volunteer as “a generator of bright ideas” and “a resource of valuable know-how”. However, the Evaluation Team has found that PCVs have generally put little emphasis on sharing their American experience in fundraising activities and PR techniques (skills that could have been extremely beneficial to the income generation objectives of the English Lending Library in Varna, the Municipal Children’s Complex in Balchik, and the Black Sea Model Renovation Initiative). 4.2 Conclusions Generally, the 16 SPA activities reviewed have had limited impact in terms of building the capacity of local counterparts to write project proposals, manage programs, and conduct fundraising activities. PCV’s unilateral SPA proposal writing, as well as local counterparts’ limited involvement in product development, have been the two major limiting factors to exposing local communities to an intensive learning experience. Despite the limited project development capacity developed under SPA, the Program has impacted positively the wider community in terms of providing key training to target community groups in computer skills and environmental education. Depending on the organization with which the volunteer works, both equipment and capacity building can play a significant role in the organizational development of the NGO or other institution involved. ß Young organizations need both equipment and capacity building, with equipment being their primary need as it enables them to start up certain activities, organize themselves and, most importantly, use the equipment as in￾kind contribution for future grant applications. In this way, SPA equipment not only provides them with an equipped facility for proposal writing, but also boosts their organizational competitiveness. ß Experienced organizations need capacity building in sophisticated types of skills like fundraising, lobbying and advocacy building. They also greatly benefit from PCVs’ bright ideas and pioneering endeavors. The need for equipment at such organizations is much lower, when compared to young NGOs. 20 4.3 Recommendations ß Volunteers working with municipalities or experienced NGOs should focus their assistance on fundraising, lobbying, advocacy building, and public relations. The Evaluation Team believes that such training will greatly increase SPA’s prospects for sustainability and long-term impact. ß PCV’s role in proposal writing and project development should become more advisory. Although the Team takes into account the language barrier that PCVs and host organizations face in the proposal writing process, the Evaluation Team strongly believes that volunteers should encourage more host organizations in taking the first step and writing the first project proposal draft. They can then take the lead and finalize the papers in a format appropriate for submission, while pointing out the document’s weaknesses and possible means for improvement. ß PCVs should strive to involve host organizations more in the development of public information materials, awareness campaigns, and other SPA products, so that SPA organizations acquire the know-how before the PCV’s departure. 5. MONITORING AND EVALUATION This section seeks to respond to the following evaluation question: ß What is the quality of the existing monitoring, evaluation and reporting systems? 5.1 Findings No Baseline Data. Weak Monitoring Plans. In the majority of the project proposals reviewed, the Quality of Life indicators contained no baseline data. Neither did the completion reports contain figures or facts reporting against these indicators. The monitoring plans of SPA proposals generally look like workplans (i.e. describe the sequence of events and processes that is supposed to take place) rather than outlining a specific set of measurable and reportable indicators. No preliminary needs assessment or survey has been done to explore the need for and expectations from the business consulting services launched with SPA support at the Open Doors Project in Veliko Tarnovo. After 1.5 years of operation, it turned out that there is not great interest for such type of services. Even with extremely successful projects such as the ones in Balchik, nobody had thought of studying the situation before project implementation and reporting on the differences between the previous and current data. 21 5.2 Conclusions Without baseline data, it is difficult to measure the improvements that came as a result of SPA over a certain period of time. Without having such preliminary information available, quantifying impact or taking hold of the attitudinal changes and shifts in mentality is virtually not possible. Only visible events and equipment purchases seem to be monitored. Evaluation generally does not make part of the project proposals and completion reports. 5.3 Recommendations Peace Corps/Bulgaria needs to reconsider its monitoring and evaluation practices. The SPA program will only benefit from a system of quantitative and qualitative indicators that report against results in an organized and sequential manner. Mechanisms for tracking progress and certain evaluation tools need to be developed in order to “quantify” the change that SPA projects bring to local communities. 6. OTHER FINDINGS ß No Stickers on SPA-Funded Equipment. None of the equipment provided through SPA has stickers acknowledging U.S. Government funding. ß Logo Requirement Not Universally Observed. According to the Peace Corps regulations and procedures, all materials published with SPA funding should have the Peace Corps and USAID logo. While the Team was unable to see all the information materials developed through SPA, a lot of the brochures and leaflets that the Team saw did not have either of the two logos. Positive exceptions were the brochures and tourist map developed in Apriltzi as well as the leaflet published under the NGO Environmental and Development Center Project in Varna. The booklets published under the Friends of the Park Project did not have any of the U.S. Government agencies logo. Though displaying the Peace Corps logo, the information tablets installed at the Black Sea Model in Varna also did not acknowledge USAID’s contribution for the repair and reconstruction of this recreational facility (see picture). ß False Belief That Host Organizations Somehow Own PCVs. There is a broad misunderstanding of the role that PCVs are supposed to play within communities. At least four of the organizations visited thought PCVs solely worked for the institution that hosts the PCV and provides for his/her lodging and workplace. Local communities are generally unaware of the fact that they can approach the PCV even though he/she is not based within their organization, and jointly develop a SPA proposal. 22 ß Value of Simplified Application Process. Six out of the 17 organizations contacted greatly appreciate the simple and straightforward SPA application format, procedures, and requirements. They believe other donor prerequisites are too cumbersome and complicated, as compared to the SPA process. 7. OTHER RECOMMENDATIONS ß Peace Corps should introduce the use of USG stickers as an explicitly stated requirement in the SPA Regulations and Procedures. This should also be accompanied by appropriate enforcement mechanisms that ensure adherence to the new prerequisite. ß Similar enforcement mechanisms should also be developed with regard to the existing SPA requirement of putting Peace Corps and USAID logos on all publication materials. ß As suggested by six out of the 17 local counterparts visited, Peace Corps should put greater emphasis on training PCVs in proposal writing. These local communities believe that volunteers are not sufficiently prepared and lack adequate experience in project development. 8. COLLABORATION WITH OTHER USAID PROGRAMS No past or ongoing complementary and/or cooperative efforts have been reported between SPA counterparts and USAID partner organizations operating in these regions. While most host organizations interviewed have heard or are slightly aware of a few USAID programs like DemNet, the Technical Twinning Program, and the Foundation for Local Government Reform, they are generally not aware of the Agency’s activities and have not been in contact with either of our implementing partners. Recommendations ß PCVs should seek to expand their linkages to USAID and other donor projects. This collective approach can open up some existing capacity, expand the resources available, and develop into a successful partnership that can ultimately compensate for the limited timeframe and resources of SPA activities. ß Peace Corps and USAID should exchange information on a more regular basis. Specifically, PC should provide USAID with a list of all PCVs working in Bulgaria and their locations. USAID will further share this information on its behalf with partner organizations working outside Sofia, and encourage contacts and collaborative efforts between contractors and volunteers. USAID will also provide PC w/ a detailed list of ongoing activities and partner information. 23 9. GENERAL OBSERVATIONS The SPA Program is inextricably linked to the general Peace Corps Program in Bulgaria. Naturally, recipient organizations do not separate SPA results from the overall activities of the respective PCV. Therefore, during the interviews, the Evaluation Team received some on-the-side information about local communities’ impressions of and recommendations to the general PC Program. In keeping with the structure of this report, the following are the Team’s additional findings and recommendations: 9.1 Findings ß PCVs Not Well-Prepared for the Local Context. Five out of the 17 host organizations visited expressed concern about the insufficient preparedness of the PCV upon his/her arrival. It is widely believed that volunteers need intensive preliminary training on the local context, needs and interests of the local community. In cases when the PCV had not been properly prepared, he/she started to lose direction and motivation “to help communities help themselves.” This lowered sense of feeling useful, coupled with the lack of well-defined tasks and objectives, made the volunteers disengaged and disinterested from the local issues. To deal with this invading sense of helplessness, they often decided to draft a SPA proposal based on a single person’s or their own personal judgment only. Such projects generally have lowest impact. Two-thirds of the organizations visited emphasized the need for increased pre￾service PCV training and more intensive preparation on the local context. Several of the interviewees felt the need for introducing trainings for host organizations that should precede the PCV’s arrival. ß PCV-Host Fit. Three of the host communities visited have had a volunteer working with them more than once. They were so satisfied with the work of the first volunteer that they immediately requested another one upon his/her departure. However, the newcomers did not respond to the needs and skills required by the local organization which ended up with an environmental volunteer instead of the English teacher requested, or with an entrepreneur rather than the social worker needed. ß Personality Does Matter. All counterparts contacted underscored the inter￾relationship between a PCV’s personality and the aftermath of his/her activities. Some of the contacted partners spoke highly of PCVs support, creativity and enthusiasm; others shared striking examples of certain volunteers’ inadaptability and non-performance. “It all depends on personalities, motivation and desire,” commented the Director of the Golden Sands Nature Park on this truly subjective factor. 24 A number of SPA grantees think that PCVs have considerably contributed to their first steps in organizational development and public outreach. The Executive Director of the Preservation of Women Association in Varna believes that the first PCV’s work proved indispensable for “giving them the first thrust” and “confidence to move forward.” The Central Balkan National Park Directorate also highly values the work that the PCV did for their public relations activities. “We learned a lot from the models and experience that the American volunteer brought to our organization,” revealed the Director of the Golden Sands Nature Park. Markedly, the Mayor of Veliko Tarnovo further expressed his appraisal of PCV’s performance, describing them as “a resource of innovative ideas.” At the same time, some of the partners visited unveiled the bitter experience that they had with the volunteers assigned to their organizations. Most strikingly, the officials interviewed from the English Lending Library in Varna contend that upon her departure to the United States, the PCV based at their organization at the time took $300 which was earmarked for periodical subscriptions under the project. They said they never heard back from her again nor have they gotten back the SPA money. Other Varna-based organizations also shared their difficult experience with particular PCVs. Some believed “conflicts emerged because the volunteer tried to impose his ‘American way”; others had conflicting views on the volunteer’s priorities and specific activities. On a number of occasions, Varna PCVs have switched from one host organization to another either due to incompatibility with their counterparts or “because he/she never really got tuned in to the type of work he had chosen to dedicate himself to for two years.” ß Differing Positions on Who Should Host PCVs. Judging from his experience with the PCV hosted by Local Agenda 21, the Mayor of Veliko Tarnovo believes project implementation is much easier if the volunteer is based within a local NGO. Other counterparts think that local needs are much better satisfied if support is focused on any organization as long as it is the only channel of assistance. Younger organizations and NGOs, though willing to work with a PCV and learn from his/her experience, feel incapable of covering his/her lodging and providing him with a proper workplace. For this reason, they are more prone to ‘sharing’ a volunteer with the municipality or another organization. ß Sometimes PCVs Are Viewed as Cash Cows. Sadly, some local communities demonstrate minor appreciation of PCV’s expertise and experience. Rather, they view volunteers as a guaranteed channel to USG funding and mere cash cows. This is partly due to the above finding that not all PCVs match the needs of local organizations. In such cases, the search for equipment becomes the only driving force behind communities’ application to SPA. 25 9.1 Conclusions It is important for the PCV and the host organization to “fit” and complement each other. Project impact is higher when the skills and knowledge of the PCV respond to the needs of the host organization, as expressed in its request for a volunteer. While the personality factor may seem too subjective and discrete, the Evaluation Team feels the need of emphasizing the fact that personality does matter. Moreover, it is among the leading factors for achieving impact and laying out the groundwork for future sustainability. 9.1 Recommendations ß In cases when Peace Corps/Bulgaria seems unable to respond to a certain community’s request for a volunteer with specific skills and professional background (i.e. social worker, teacher, manager, or fundraiser), a one-year deferral would be better rather than assigning a PCV that does not match the requestor’s needs. ß Peace Corps should put greater emphasis on introducing PCVs to the local context and community expectations in the course of the PCV Pre-Service Training. ß The Evaluation Team believes that the Peace Corps assignment system can be significantly improved with the development of a PCV-host matching system. The latter will not only contain information on the type of PCV assistance requested, but also provide for a “psychological fit” between the volunteer and the host actor (i.e. personality traits, work style, interests, etc). While the development of such a system will demand certain costs, the Team believes it will prove cost-effective in the long term by ensuring higher “customer satisfaction” and the improved quality of the PCV service. ß As suggested by several counterparts, Peace Corps should consider initiating specialized trainings for host organizations where local organizations can spell out their expectations and receive some guidance on how to better integrate a volunteer into their community; what kind of assistance they can rely upon; and some of the cultural differences that they may encounter. A training needs assessment has to be conducted among PCV host organizations prior to conducting such seminars to define the training curriculum and learn more about local interests and issues. ß One of the host organizations that have experienced difficulties in responding to a certain volunteer’s expectations proposed that Peace Corps conduct a preliminary workplace assessment, as part of the matching system used for volunteer assignments. ß Several of the organizations visited suggest “joint hosting” as a way of cutting the cost for local communities on accommodating a Peace Corps Volunteer, while at the 26 same time increasing the number of direct beneficiaries from his/her presence in town. 10. SUGGESTED JOINT STEPS ß Peace Corps and USAID officials dealing with the SPA Program should sit together and check out any recent improvements that have been made to the Program’s rules and procedures that respond to some of the Team’s concerns and community sentiments. Since some of the projects have been implemented back in 1998 and 1999, some steps could already have been taken. An Action Plan for responding to the rest of the findings can be developed. ß As an organization with vast experience in program monitoring, USAID can assist PC in enhancing its system and replicating some relevant mechanisms and techniques. ß USAID can provide regular training to PCVs on how to develop monitoring plans, choose appropriate indicators, set targets, and collect relevant baseline data. ß In light of graduation from USG assistance, USAID has established some sustainability mechanisms and legacy techniques from its in-country presence. It can provide its “know-how” to PC and work with volunteers on comprehending these concepts and developing sustainability methods that match the specific nature and duration of SPA activities. ß PC can introduce stricter requirements and enforcement mechanisms with regard to: o Conducting needs assessments; o Developing and initiating the implementation of sustainability plans; and o Use of logos on publication materials and, possibly, stickers to USG￾funded equipment. ß To better respond to community needs and enhance the prospects for sustainability, PC can introduce specialized trainings for PCVs on: o Lobbying and advocacy building; o Fundraising; o Generating community participation; and o Proposal writing. ß PC and USAID can exchange information on a regular basis on the location of PCVs and agency projects, thus encouraging collaborative efforts between USAID implementing organizations and volunteers. 27 Annex 1: Scope of Work IMPACT EVALUATION OF THE PEACE CORPS SMALL PROJECT ASSISTANCE (SPA) PROJECT I. Introduction For the past ten years, USAID has funded the Small Project Assistance (SPA) Program implemented by Peace Corps/Bulgaria. The program involves community development projects in support of the three Peace Corps assistance sectors: environmental protection, community economic development, and English language education. In addition, many PCVs and their communities develop SPA projects that are important to the community but not necessarily linked to the PCV primary area or assignment. In an attempt to appraise the impact of these grassroots activities and their effect on the capacity and well-being of local communities, USAID has planned an impact evaluation to be conducted by a team of internal experts and a visiting evaluator from USAID/Macedonia. II. Purpose of Evaluation The overall purpose of this evaluation is to assess the impact and sustainability of SPA grassroots activities implemented across Bulgaria within the past five years (1998-2003). The study seeks to ascertain whether the overarching goals of the project have been achieved and what its long-term effect is on local communities. The evaluation will specifically identify any lessons learned and existing challenges to producing the expected effect and achieving long-term sustainability. III. Background The SPA Program aims at encouraging self-help efforts on behalf of local communities, in collaboration with Peace Corps Volunteers, to identify common concerns, plan small-scale activities, and develop strategies to address those issues. It is based on the principle of “help people help themselves” and focuses on facilitating sustainable development at the local community level. The intent is to provide an efficient mechanism that can respond quickly and effectively to community initiatives with an immediate and lasting impact at the local level. SPA grants provide community organizations with small amounts of financial support ($3,000 on average: the lowest being $160, the highest: $9,803)) to carry out activities they have identified. Funds can support training, technical services, and materials development – all designed to strengthen community-based development efforts. Activities address a variety of areas, including upgrading infrastructure, computer instruction, environmental education, biodiversity conservation, income-generating activities, civil society development, youth 28 issues, etc. PCVs play an important role in facilitating the planning process, assisting with implementation, building local capacity, and monitoring project success. IV. Evaluation Questions ß Have activities brought any change to local communities? ß Are the SPA products/processes established with SPA assistance still operational and taken care of? What is their current status? ß What is the quality of the existing monitoring, evaluation and reporting systems? ß Which type of activities has been most or least effective in terms of achieving lasting impact and sustainability? ß Has local capacity been built through the SPA Program? ß What is the project’s effect on the relationships among the various community actors (municipality, businesses, schools, etc)? ß Have there been any collaboration efforts with other USAID activities implemented in this city/region? ß What are the lessons learned? ß What are the challenges/obstacles to achieving sustainability and/or having long-term impact? ß Recommend program performance measures. V. Evaluation Methods ß Rapid Appraisal ß Non-probability sampling ß Site Selection Criteria: - Diverse community size (large, medium, small) - Activities implemented in all the fiscal years reviewed (FY98- FY02) - Proportional representation of projects in all three major sectors - Maximum variety of implemented activities - Feasibility in terms of time and effort VI. Team Composition The evaluation team shall comprise of three USAID evaluation officers: Ms. Assia Alexieva (SPA Activity Manager, USAID/Bulgaria); Ms. Katia Alexieva (Deputy Program Officer); and Mr. Ivica Vasev (Development Specialist, USAID/Macedonia). VII. Schedule The evaluation is expected to last four weeks, with one week in the field (August 11-15, 2003). Week 1: Preparatory Work: The team shall spend the first week in planning (methodology, agenda, evaluation work plan). During this period, it is also 29 supposed to settle on a list of interviewees (focus groups), locations to visit, and duration of the site visits. A set of interview questionnaires shall also be developed. The remaining time will be spent on collecting data and identifying data sources. Week 2: Field Work: The team shall conduct site visits to completed SPA activities. This phase will include field interviews with customers and stakeholders as well as informational interviews with the Peace Corps Director, the SPA Coordinator, and PCVs with prior SPA experience. Upon completion of field work, the evaluation team shall debrief relevant USAID officials. Week 3 and 4: Report Writing: The team shall produce a first draft of its evaluation report within seven days of completing Phase Two of the evaluation. Following internal USAID review, a final report shall be developed within another week, reflecting the Mission’s comments and recommendations. VIII. Logistics Travel and per diem will be covered by USAID/Bulgaria for the whole evaluation team. The Mission will also provide the team with a driver and a vehicle for the field trips, as well as furnish the evaluator from Macedonia with office space and access to printer. No translator will be necessary. The Peace Corps/Bulgaria will facilitate contacts with stakeholders and provide some logistical support on a limited basis. IX. Existing Performance Information Sources A Country Level SPA Program Evaluation was conducted in the first quarter of 2002 by a Peace Corps Evaluation Planning Team comprised of 14 participants. The final evaluation report is available with USAID/Bulgaria and will be used as a reference material by the evaluation team. Other relevant documents include the SPA Program Handbook, the SPA Guide for Volunteers and Counterparts, project proposals, reports, and other relevant documents. X. Reporting Requirements The team shall produce an evaluation report with findings, conclusions, and recommendations in English. The final report shall be submitted to the USAID Mission by September 8, 2003. XI. Attachments Attachments A and B summarize the Peace Corps definitions of important aspects of the SPA Program as well as illustrate their goals, objectives, and perceived impact. 30 ATTACHMENT A PEACE CORPS SPA GOALS & OBJECTIVES GOALS ß Encourage and support community self-help efforts. ß Support small-scale community development projects with an immediate impact at the community level. ß Join the PCV human resource capabilities and knowledge of local conditions with the financial resources of USAID to help community groups in their self-help efforts. ß Provide seed money to communities to address community concerns. ß Demonstrate the benefits a community can accrue from using its own talents, skills and ideas to help itself. ß Enhance leadership skills, transfer technical skills and promote self-reliance. OBJECTIVES ß Increase the capabilities of local communities to conduct low-cost, grassroots, sustainable development; ß Provide community organizations with small amounts of financial support to carry out activities they have identified to address their basic needs and those of their communities; ß Support activities that provide “hands-on” training and help increase employment and income; ß Through the support of educational activities, improve access to formal and non￾formal education for students and community residents, upgrade infrastructure, and provide instruction in subjects ranging from literacy to computer instruction, to vocational and employment training; ß To support activities that focus exclusively on community-level training, organized by Peace Corps Volunteers and community members. IMPACT & LEGACY The Peace Corps is a process-oriented rather than a goal-oriented organization. Ultimate success lies in the community's learning from the project development and proposal writing process and applying successfully the acquired skills. 31 ATTACHMENT B PEACE CORPS DEFINITIONS AND TERMINOLOGY · Community: the village or neighborhood in which the volunteer works. In many SPA proposals, the community is the organization with which the volunteer works, whether a school, an NGO, or a municipality. While community projects may benefit a school or an NGO, they should also involve and benefit the community as a whole, since broad-based participation helps ensure continuity and the equitable distribution of benefits. Therefore, a community is more or less a geographic locale. · Community development: In its broadest sense, development is any process that promotes the dignity of people and their capacity to improve their own lives. Peace Corps volunteers become catalysts for facilitating such change. · Development: The Peace Corps uses “development” in human, people-to￾people terms: helping people develop the capacity to improve their own lives. The focus is on the development of people, not things. · Capacity building approach focuses on helping people learn to identify what they would like to see changed, use their own strengths, and learn new skills to achieve what they believe is most important; a village or organization realizes its own potential to accomplish self-defined goals. To be effective, capacity building needs to happen at multiple levels: individual members, project counterparts (students, clients); professionals, service providers (school teachers, NGO leaders); organizations (schools, NGOs, cooperatives). · Sustainable development: Development work is said to be “sustainable” when the community is able to continue on its own without outside support. The Peace Corps sees sustainable development as a process whereby people learn to build on their own strengths to take charge of their lives, and to address their expressed needs. · Using SPA to promote sustainable development: Development is a long-term process wherein communities learn, through an iterative process of trial and error, how to improve their lives. 32 Annex 2: Map of Site Visits 33 Annex 3: Project Description Matrix # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary 1 Apriltzi Tourist Association NGO: Tourist Map, Brochure and Website Apriltzi Apriltzi Tourist Association NGO 03/29/01 – 03/29/02 $3,307 To promote tourism in Apriltzi by creating a map, a tourist brochure, and a website for the local Tourist Bureau. Create a brochure, a map, and a website containing information about local attractions and landmarks, bike and hiking routes, key events, shops, hotels, restaurants, and local craftsmen. Revenue will be generated from advertisements and membership and service fees. Equipment: digital camera and a printer. 2 Support for Apriltzi Craftsmen Apriltzi Apriltzi Craftsmen Community 03/23/01 – 05/03/02 $2,577 To increase the success of craftsmen and artisans in Apriltzi and to stimulate growth of crafts in the community. Assist with the establishment of a craftsmen community through the development of a brochure, website, a store front for displaying finished products, and training for national trade shows. 3 Computer Workstation to Support NGO Development Gabrov o YMCA￾Gabrovo 04/30/00 – 02/28/01 $3,087 To develop the computer and Internet skills within the NGO community in Gabrovo. Purchase a computer workstation (consisting of 3 computers, 1 printer, and a modem) for training NGOs in basic computer skills. The workstation will become part of the Community Development Center. A Reference Manual will also be developed. 4 Ecological Education – One Step Forward Gabrov o Youth Cultural 10/98 – 06/99 $1,463 To improve environmental Activities will include a series of seminars (3 x 2 days), the formation 34 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary Information Center education and empower youth through the development of a network of eco￾clubs and the creation of a resource center. of eco-clubs (10 expected), and the construction of an eco-center. The letter will serve as a central meeting place for the clubs, and will contain environmental resources like textbooks, a directory of environmental foundations, fundraising info, etc. Brochures and posters will publicize the Center. 5 Central Balkan Environmental and Earth Day Awareness Campaign Gabrov o Central Balkan National Park Directorate 03/25/98 – 03/20/99 $1,668 To raise environmental awareness through a series of Earth Day events. Activities include art and essay contest; park clean-ups; the production of an anti-litter video; printing of 2,000 anti-litter and 1,500 environmental education posters; 2 environmental textbooks (30 copies each). 6 Emerging Business Support Center Veliko Tarnov o Local Agenda 21 08/01/01 – 05/31/02 $4,395 To strengthen the economic situation of working Bulgarians by providing access to both technical resources (computer and fax machine) and human resources (comprehensive business consulting and training) in one, easily accessible 1) Provide a physical space for people to gather and learn about business and management; 2) provide full-time, affordable consulting in general business practices and access to more specialized consultants (a center staff person refers clients to various specialists); 3) free training courses (from Writing Business Plans to How to Officially Register a Business in Bulgaria); 4) develop an information￾sharing network among business people in the region. Center clients 35 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary business center. The project is part of a larger initiative called Open Doors. will pay 10 leva per year to have access to all hardware and classes. Equipment funded under SPA: Xerox machine, fax machine, overhead projector, screen, file cabinet, erase board, training materials, etc. 7 Trash Container Environmental Art Veliko Tarnov o Municipality of Veliko Tarnovo and Sts. Cyril and Methodius University 05/01/98 – 02/10/98 $1,555 To increase public environmental awareness and draw attention to waste management efforts; increase community action to place trash in proper containers; involve the community; draw attention to the work of the university art students. Replace damaged trash containers and organize an environmental art contest and a Trash Container Art Exhibit. University art students will paint their works on new trash containers provided by the Municipality. The containers will be placed in service around the city drawing the public’s attention to environmental and waste management efforts. 8 Environmental Film Festival Veliko Tarnov o Veliko Tarnovo Youth House 09/01/98 – 10/31/98 $690 Promote environmental awareness and education; Encourage creativity towards the environment; Provide opportunities for SPA funds will be used to secure the use of local cinemas to show films on environmental topics; purchase materials for the river and town clean-up; pay transportation and lodging expenses for guest teachers and speakers; purchase materials for the Fill a Bag of Trash Program. 36 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary local citizens to get involved. 9 NGO Environmental and Development Center Varna Public Environment al Center for Sustainable Developmen t 06/98 – 01/99 $3,450 Develop and provide environmental information to the Varna community; Transfer of information to and from NGOs; Offer environmental literature to young people. Involves the creation of an Info Center; development of an informational leaflet; and provision of information to the media and other consultations. Equipment purchased: computer, printer, copier, videotapes, books, National Geographic, office supplies. 10 Renaissance Woman Varna Preservation of Women Association 01/15/99 – 05/30/99 $3,045 To have a fully developed informational and educational center for prevention and preservation of women. 1. Educate women on health issues, early diagnosis, and prevention; 2. Raise awareness on how to solve social, health, legal, and economic problems; Provide the Center with informational and educational materials for public access. Produce and publish 350 booklets with appr. 50 pages of information about health, social, legal and business development, to be distributed at the Center’s Grand Opening and Health Fair. Involves the purchase of the following equipment: computer, printer, scanner, cassette player, earphones, office supplies. 37 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary 3. Stimulate women participation in community development 4. Increase women’s standard of living in the region 5. Provide public awareness of the Center. 11 English Lending Library Modernization Varna English Lending Library 12/02/98 – 06/14/99 $2,930 1. To provide access to Internet 2. To allow patrons facilities for copying and faxing; 3. To organize the library via computerized cataloguing; 4. To generate income to ensure sustainability 5. To update periodical subscriptions. Funding will be used for the purchase of a computer, printer, fax machine, copier, phone system, and periodicals subscription. In order to ensure revenue and sustainability, fees will be charged for using the equipment, and membership dues will be introduced. 12 Friends of the Park Club Varna Golden Sands 11/01/98 – $2,009 Make students understand the Activities include: teachers workshop; class visits/lessons to 38 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary National Park Directorate 06/30/99 meaning of protected areas and the necessity of their protection. 1. Create a Friends of the Park Club at the middle and high school level; 2. Create a student workbook on local protected areas; 3. Facilitate in-class and hands-on lessons; 4. Create a network of local professionals interested in environmental education sixth- and ninth-graders from 7 Varna schools; one site visit per class to the Golden Sands National Park; contracting local media to cover the Club’s activities. 200 t￾shirts, 3,200 club membership cards, and 3,500 booklets were printed with SPA funding. 13 Community Efforts for Black Sea Model Restoration Varna Black Sea NGO Network 01/09/00 – 04/05/01 $4,170 To clean, repair, and provide regular maintenance to the Black Sea Model within the Varna Sea Garden. To provide visitors with environmental information about 1) Build community support by involving NGOs, youth organizations and local students in a community clean-up; 2) create info tablets and plaques; 3) work with the municipality and local clean-up firms to provide regular maintenance of the model. SPA funding will be used for the fence construction, repairing 39 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary Black Sea issues through info tablets. the drainage system, labor costs for painting the model, surface reconstruction costs, and thank-you party costs. 14 Child and Nature Balchik Municipal Center for Children 04/01/00 – 06/15/00 $2,095 To broaden and enrich knowledge of the environment for children’s groups from schools of Balchik and the sanatorium school through interdisciplinary means, including art, journalism, visual media. Two new eco-clubs will be set up in the Strajitsa School and the sanatorium school; A planting of trees and bushes at Antim I Primary School; Help with weeding and cleaning the flower gardens at the Palace Architectural Complex; Produce artwork (pottery, drawings, photos, video film); Publish the Eco Telegraph newspaper; Have more people in partnership with BCC. SPA funding will be used for the purchase of art supplies, a kiln for pottery, a video camera, and lodging and travel expenses for teacher, students, and visiting artists. 15 Equipment for Computer Lab in Balchik Balchik Municipal Department of Education 05/99 – 06/99 $2,708 To make Bulgarian children more employable and competitive through the provision of free computer education to children and teachers. To create a computer laboratory at the Municipal Children’s Center that will provide formal computer classes to children. A computer teacher will be hired to develop the curriculum. Four computers will be purchased for the purposes of this project. 16 Primary Foreign Language Center Balchik Antim I Primary 04/20/02 – $2,768 Increase the effectiveness of Provide Antim I School with two functional interactive foreign 40 # Project City Partner Impl. Period SPA Fundin g Purpose / Objectives Project Summary School 01/05/03 computer and foreign language instruction at Antim I School. language classrooms (a resource room and a video room) equipped with a TV, VCR, and a computer with Internet connection. Computer classes will be available to staff, incl. Internet training and how to use the equipment. 41 Annex 4: List of Interviewees 1. Valentina Dobreva, SPA Coordinator, U.S. Peace Corps, Bulgaria 2. Carl Hammerdorfer, Country Director, U.S. Peace Corps, Bulgaria 3. Ivan Pirov, Mayor, Municipality of Apriltzi 4. Doncho Pachnikov, Former Manager, Apriltzi Tourist Association 5. Teodor Nikolov, Manager, Apriltzi Tourist Association 6. Ivailo Ivanov, craftsman, Apriltzi 7. Irina Vachkova, craftsman, Apriltzi 8. Ann Jones, Peace Corps Volunteer, Gabrovo 9. Antoaneta Yankabakova, Executive Director, YMCA/Gabrovo 10.Youth Volunteer 11.Emilia Marinova, Youth Cultural Information Center, Gabrovo 12.Vladimir Marinov, Youth Cultural Information Center, Gabrovo 13.Thomas Jones, Peace Corps Volunteer, Gabrovo 14.Pavlina Nedelcheva, Expert, Municipality of Gabrovo 15.Diana Terzieva, Chief Expert, Interpretative Educational Programs, Central Balkan National Park 16.Rumen Rashev, Mayor, Municipality of Veliko Tarnovo 17.Teodora Kaleynska, Director, Local Agenda 21, Veliko Tarnovo 18.Yulia Tumbarkova, Director, Golden Sands National Park 19.Alexander Stoychev, English Language Library, Varna 20.Rumiana Tultukova, Library Manager, English Language Library, Varna 21.Ralitza Zhekova, Coordinator, Black Sea NGO Network, Varna 22.Alexander Shivarov, Coordinator, Black Sea NGO Network, Varna 23.Kosta Bazitov, Director, Education, Youth Activities and Sport Directorate, Municipality of Varna 24.Ilian Iliev, Public Environmental Center for Sustainable Development, Varna 25.Elena Patronova, Executive Director, Preservation of Women Association, Varna 26.Maria Chankova, Executive Director, Agency for Social Assistance; Member of the Managing Board, Preservation of Women Association, Varna; 27.Vessela Fazheva, Psychologist, Preservation of Women Association, Varna 28. Maria Pavlova, Volunteer, Preservation of Women Association, Varna 29.Bobadelieva, Summer Volunteer, Preservation of Women Association, Varna 30.Galina Nedelcheva, Head, Education, Health and Culture Department, Municipality of Balchik 31.Tinka Sivrieva, Director, Municipal Children’s Complex, Balchik 42 Annex 5: Questionnaires DRAFT INTERVIEW QUESTIONS FOR SPA COUNTERPARTS 1. How did you get involved in SPA? Who approached you? 2. How were the needs and priorities of the community identified? Would you characterize the decision-making process as transparent and inclusive? 3. What was your role? What is your role today? 4. Did you involve other partners (municipality, businesses, etc)? What was their role? What is their role today? Are you still working together? 5. What was the PCV’s role? What was your relationship w/ him/her? 6. Who provided the matching funds (in-kind contribution)? 7. Are the resources/materials/equipment acquired through this project still used and maintained? By whom? Are there sufficient resources to maintain them? 8. Who takes care of the project outputs now? Do you currently have an established mechanism for resource mobilization? 9. Have you made any upgrades/improvements since? 10.What do you think this project managed accomplished? Do these hold true today? Can you identify any best practices that came out as a result of this project and that you would like to share? 11.What did it not manage to accomplish? 12.How did you manage to take care of the project outputs after SPA assistance had ended? 13.Do you think SPA has brought any change to your organizations/community? What was the biggest change in your opinion? 14.What would you have done differently? 15.Did you have any kind of a monitoring/evaluation system during project implementation? Do you maintain such a system today? 43 16.What did you learn from the SPA process? What knowledge and skills do you think you gained as a result of this project? Do you still apply these? 17.What was the most difficult part throughout the process? 18.Are you aware of any other USAID projects in your city/region? Have you or do you work w/ them for the implementation/maintenance of this project? 19.Did you or your community initiate any follow-on activity later on? Did your community initiate any other projects? 20.Do you think SPA has had any impact? On what? DRAFT INTERVIEW QUESTIONS FOR MUNICIPAL OFFICIALS 1. Do you see any benefits to your municipality from SPA? If yes, please explain or describe them. 2. What was the role of the municipal administration, if any? What is its role today? 3. Do you see any changes in the amount of involvement of the community/business/NGOs/other parts of the community in their participation in community life? Please describe or provide examples. 4. Any other noticeable change? 5. What do you think was SPA’s impact? 6. Do you think it has had any impact on encouraging various community actors to work together? 7. Was there a follow-on activity after SPA has ended? Or any other project inspired by the SPA process? 8. What do you know about the project outcomes today? Who takes care of them? Is the municipality involved in any way in the maintenance/sustainability of project outcomes? 9. Issues? 10. Recommendations? 11. Best practices? 12. Other USAID activities in your city/region? Are you aware of any cooperation/coordination efforts in implementation of this project? 44 DRAFT INTERVIEW QUESTIONS FOR PEACE CORPS SPA COORDINATOR AND COUNTRY DIRECTOR 1. Who writes the project proposals? 2. Criteria for project selection? 3. Do you think the project ceiling is enough? 4. Do you manage to absorb funds? 5. What is the chain of funds transfer? What documentation do you require? 6. Do you think the demand for SPA funding has increased for the past 5 years? 7. Do you have an established monitoring system and how do you monitor projects that have already been completed? Who’s included? Just the PCV or the community as well? 8. Which project would you define as successful? Please characterize. 9. How would you define an “innovative project”? What % would you define as such? 10.What gives the whole process a long lasting impact? 11.Which type of projects do you consider most/least effective? 12.What would you have done differently? Improvements? 13.Any best practices? 14.SPA weaknesses? 15.What are the factors for sustainability? What are the obstacles to achieving sustainability? 16.Do you know of any community that has managed to receive funding after SPA? 17.Are PCVs informed about other AID projects in the region? By whom? 18.What do you think USAID’s role should be? 45 Annex 6: Notes and Quotes from Interviews “Thanks to SPA, the Center has become a place where young people can develop.” “We have not considered the PCV as a cash cow but as a resource of innovative ideas.” “SPA/PCV work introduces avant-garde ideas which are later built upon.” “Thanks to SPA, we got equipment and resources that no one had access to at the time.” “The PCV has not written any project before. It was a learning experience for her.” “Volunteers are extremely helpful in brainstorming what to do with the equipment. They have a role in encouraging young people.” “We thought we could receive a SPA grant only if we host a volunteer.” “There is a gap between a PCV’s expectations and the existing environment.” “If the PCV thinks and decides that infrastructure is needed, then we let them propose the purchase of equipment in the SPA proposal. We let them decide entirely.” “I think I liked the Peace Corps Program better when there were less PCVs.” “If it weren’t the computer provided with SPA funding, our NGO would not have been what it is today.” “The very purpose of volunteerism is to leave enduring traces behind. If a certain PCV does not have the … to serve as a volunteer, he will never achieve a lasting impact.” “Information reaches us a bit late. If the volunteer is not active enough, we do not get it.”