31 May 2008 This publication was produced for review by the United States Agency for International Development under contract with Development Alternatives, Inc. Final Evaluation Report Timor-Leste Small Grants Program Contract No. DOT-I-00-03-00004-00 Task Order #801 Final Evaluation Report Timor-Leste Small Grants Program Contract No. DOT-I-00-03-00004-00 Task Order #801 Prepared by: Nancy E. Horn, Ph.D., Team Leader Jana Hertz Nancy Sheehan Submitted by: Nancy E. Horn, Ph.D., Team Leader 31 May 2008 Development Alternatives, Inc. 7600 Wisconsin Avenue, Suite 200 Bethesda, Maryland 20814 Fax: 301 771-7777 ii The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii ACKNOWLEDGEMENTS The research undertaken for this end-term evaluation would not have been possible without the levels of coordinated support the team was provided by the SGP staff, our Timorese consultant colleague, and our administrative assistants. The organizations and individuals contacted for interviews were extremely gracious in providing us the time we needed to reach the understanding we did. We also wish to acknowledge the input provided by the USAID SO teams to our understanding of the parameters of the SOs and the environment in which USAID is operating. We especially wish to acknowledge further our Timorese consultant who accompanied us on field visits, translated, was available at all hours for consultation, and who provided us with an excellent orientation to the country, its history and the problems it currently faces. His output is presented in Section 2 of this document. As a team, we struggled with ideas, with the appropriate way of putting those ideas forward, and how we could make this report beneficial to future SGPs. For identifying the members of the team, we thank DAI’s leadership. The team hopes that we have provided valuable insight to DAI on its operations, and to future SGP donors. We all have become very enamored with the people of and country called Timor-Leste and look forward to making significant contributions to its development. iv ACRONYMS AF Alola Foundation AMFITL Association of Microfinance Institutions in Timor-Leste AMTL Associaciao dos Medicos Timor-Leste ARKTL Asosiasaun Radio Komunidade Timor-Leste BB Bibi Bulak BDS Business Development Services BELE Building Empowerment, Leadership, and Engagement BPC Bairro Pite Clinic CAMS-TL Centre Audiovisual Max Sthal Timor Leste CBO Community-Based Organization CCF Christian Children’s Fund CCT Café Timor Co-operatives CD Community Development CFJP Centro Formacao Joao Paulo II – Comoro CHC Community Health Centre CHV Community Health Volunteers CLC Community Learning Center CRS Catholic Relief Services CSO Civil Society Organization CTID Centro Treino Integral e Desenvolvimento CTO Cognizant Technical Officer CV Curriculum Vitae DAI Development Alternatives, Inc. DBAC Don Bosco Agricultural College DG/D&G Democracy & Governance DHO District Health Officials DHS District Health Services DIT Dili Institute of Technology DMO District Medical Officer DNAT Diresaun Nasional da Administrasaun do Territorio ECHO European Comission, Humanitarian Area Office ECM Fundasaun Educasaun Comunidade Matebian EG Economic Growth ETADEP Fundacao Ema Mata Dalan Ba Progreso ETAM Escola Tecnica Agricola Maliana ETAN Escola Tecnica Agricola Natarbora ETDA East Timor Development Agency ETIS East Timor Insight FAO Food and Agriculture Organization FBM Fundacao Buka Manenek F-FDTL Falintil-Forcas de Defesa de Timor Leste (Timor-Leste Defense Force) FC Fundacao Cristal FH Fundasaun Haburas FHP Family Health Promoter FO Functional Objective FONGTIL Forum ONG Timor-Leste FY Fiscal Year v GFFTL Grupo Feto Foinsa’e Timor Lorosa’e GIS/GPS Geographic Information System/Global Positioning System GIO Government Information Office GOTL Government of Timor-Leste GSI Grupo Suporta Inan GTZ Deutsche Gesellschaft fur Technische Zusammenarbeit (German development agency) HAI Health Alliance International HAIM Hamutuk Ita Ajuda Malu HDI Human Development Index HIV/AIDS Human Immuno-Deficiency Virus/ Acquired Immune-Deficiency Syndrome HRM Hotel and Restaurant Management ICfJ International Center for Journalist ICS Instituto de Ciencias de Saude IDP Internally Displaced Person IFES International Foundation for Election Systems IGA Income Generating Activity IMCI Integrated Management of Childhood Illness INAP Instituto Nacional da Administracao Publica INGO International Non Government Organization INFUSE Inclusive Finance for the Under-Served Economy ITN Insecticide Treated nets JA Junior Achievement JL Jornal Labarik LF lymphatic Filariasis L&N Literacy and Numeracy LOL Land O’Lakes MAF Ministry of Agriculture and Fisheries MDG Millennium Development Goal(s) M&E Monitoring and Evaluation MFI Microfinance Organization MOE Ministry of Education MOED Ministry of Economy and Development MOH Ministry of Health MOU Memorandum of Understanding MR Moris Rasik NCHET National Centre for Health Education and Training NDNFE National Directorate of Non-Formal Education NGO Non-governmental Organization OJT On-the-Job Training O&M Operations and Maintenance OTI Office of Transition Initiatives PAR Portfolio-at-Risk PDC Pastoral das Criancas PDS Program Development Specialist PMP Performance Monitoring/Management Plan PNTL Policia Nacional Timor-Leste PRR Putri Reina Rosario (a religious order from Indonesia in T-L) PSD Private Sector Development vi PSI Program Support Initiative RFTL Rede Feto Timor-Leste RO Resource Organization RR Radio Rakambia SFA Strategic Focus Areas SGP Small Grants Program SIDA Swedish International Development Agency SISCA Servisu Integradu Saude Communitaria SHG Self Help Group SO Strategic Objective STAE Technical Secretariat for Electoral Administration SWIFT Support Which Implements Fast Transition SWO Scope of Work TA Timor Aid TAIS Timor-Leste Asistencia Integradu Saude TB Tuberculosis TEPS East Timor Community Stabilization Programs Support TLMDC Timor-Leste Media Development Center TOR Terms of Reference TOT Training of Trainers TRM Tuba Rai Metin TYEI Timorese Young Entrepreneurship Initiative UH University of Hawaii UI University of Iowa UMR Uma Media Regional UNCDF United Nations Capital Development Fund UNDP United Nations Development Program UNFPA United Nations Population Fund UNESCO United Nations Education, Science and Cultural Organization UNICEF United Nations Children’s Fund UNTAET United Nations Transitional Administration in East Timor USAID United States Agency for International Development WFP World Food Program ZS Zero Star vii CONTENTS Page No. 1. INTRODUCTION AND SGP BACKGROUND ………………………………. 1 1.1 Status of USAID SGP Award ………………………………………….. 1 1.2 OTI Background to SGP Award ………………………………………. 2 1.3 Changing USAID Objectives …………………………………………… 2 1.4 Operational Plan ………………………………………………………... 3 2. BACKGROUND …………………………………………………………………. 4 2.1 Timor-Leste Since Independence ……………………………………… 4 2.2 The Birth and Growth of Local NGOs ………………………………... 5 3. METHODOLOGY ………………………………………………………………. 7 4. GRANTEE FINDINGS ………………………………………………………… 7 4.1 Governing Justly and Democratically ………………………………… 9 4.1.1 What Worked Well ……………………………………… 10 4.1.2 What Did Not Work So Well …………………………… 14 4.1.3 Recommendations …………………………………………. 16 4.2 Accelerating Economic Growth ………………………………………. 17 4.2.1 What Worked Well …………………………………………… 18 4.2.2 What Did Not Work So Well ………………………………… 24 4.2.3 Recommendations ……………………………………………. 27 4.3 Improved Health of the Timorese People ……………………………... 27 4.3.1 What Worked Well …………………………………………… 29 4.3.2 What Did Not Work So Well ………………………………… 36 4.3.3 Recommendations ……………………………………………. 35 4.4 Summary and Conclusions: Addressing the SOW Questions ……… 37 5. MANAGEMENT STRUCTURE AND PROCESS …………………………….. 38 5.1 Introduction ………………………………………………………………. 38 5.2 Internal Operations ……………………………………………………… 39 5.3 Staff Reflections …………………………………………………………... 42 6. LESSONS LEARNED AND RECOMMENDATIONS ………………………… 43 6.1 Small Grants Project Cycle Design .…………………………………….. 43 6.2 Grant Management Processes and Procedures ………………………… 49 6.3 Creating Synergies for Greater Impact ……............................................. 52 6.4 Final Recommendation to DAI …………………………………………… 52 ________________________ APPENDICES: A – SOW B – MEETINGS HELD C – DRAFT METHODOLOGY D - BIBLIOGRAPHY viii EXECUTIVE SUMMARY Introduction The Timor-Leste Small Grants Program (SGP) (SWIFT II, DOT-I-00-03-00004, Task Order No. 801), was established in October 2004 to support USAID’s Strategic Objectives (SOs) of accelerating economic growth, improving good governance practices, and improving the health of the Timorese people, as outlined in the USAID/Timor-Leste1 Mission Strategy for 2004-2009. Through small grants, the program has supported communities, organizations, and government in their efforts to build a stable, economically robust, democratic country. As of April 15, 2008, SGP had awarded 219 grants and disbursed more than $6.5 million to 94 local institutions, 21 government agencies, and 12 international NGOs. It is estimated that during the project, 89 awards were made in Governing Justly and Democratically (DG) for a total of $2,245,811 (34% of award amounts); 78 awards were made in Economic Growth (EG) for a total of $2,692,826 (40% of award amounts); 50 awards were made in Investing in People (including health and L&N) for a total of $1,516,522 (24% of award amounts) and 4 awards were made for special projects totaling $99,512 (2% of award amounts).2 This end-term evaluation documents what actually worked well, and what did not work so well under each SO. It also makes recommendations related to each SO and then makes broader recommendations to provide guidance and direction for the development of future SGPs. Background DAI implemented the Office of Transition Initiatives (OTI) SWIFT contract for 18 months - 2000-2001. The work was transferred from OTI to the Mission in 2002, one year into the 2001-2004 Program Support Initiative (PSI). In many ways, the OTI and PSI programs served as a model for developing SGP. While launched on the basis of the above three SOs, changes in mission administration, USAID directorship, CTOs, and the SOs themselves presented challenges to DAI in an environment that was already sufficiently challenging by the history of the country, changing political, economic and social environments, and several waves of civil disturbance. Consequently DAI had to modify their operations several times. Most NGOs DAI sought to fund to pilot or extend activities emerged just before or since independence in 2002. Those who established the NGOs had experience in the independence struggle, in working for organizations or going to school in the diaspora, working for international NGOs (INGOs), as alumni from specific colleges and universities, or formed to take advantage of a funding opportunity. In most cases, NGO leadership lacked management skills, especially those in financial management. Consequently, when proposal solicitations produced inquiries, DAI staff often had to nurture the creation of the idea, guide potential grantees through the proposal process, or had to explain the entire funding process. Methodology DAI exhibited a great deal of flexibility in providing grants to NGOs and INGOs. When designing the evaluation methodology, it was considered important to have both an SO and a 1 East Timor was the name used officially by the U.S. State Department and USAID until 2007, when all internal and external naming changed to Timor-Leste, the official name of the country under the Constitution of the Democratic Republic of Timor-Leste. 2 DAI. 2008. USAID Timor-Leste Small Grants Program Semi-Annual Report October 2007 – March 2008. Dili: DAI/USAID, April 15, pp. v; 3-4. ix geographic diversity of grantees represented. When conducting the first interviews with DAI, the team learned that most of the grants were made to Dili-based organizations that delivered their programs in multiple districts. Consequently, our research design changed to focus on these organizations, with data collection occurring in Dili, Baucau, Liquica, Manatuto and Aileu. Summary of Findings In this section we have customized our findings to answer the questions: What worked well? And, what did not work so well? Consequently, we do not present our findings for each NGO or government agency interviewed, but in accordance with several parameters that emerged as major factors in contributing to either project success or design/implementation challenges (either in concept or execution). We state that projects that do well under the SGP mechanism are, among others, pilots, responsive to local need, embedded in local networks for sustainability, create new materials in demand by different populations, target the appropriate audience, etc. Projects that do not do as well under the SGP mechanism are those, among others, that do not understand the needs of the target audience, do not communicate or coordinate with other projects and so are a duplication of effort, cannot overcome challenges that were not a part of the design, etc. We divide this section into three parts to reflect the three SOs under which grants were made: 1) Governing Justly and Democratically (DG); 2) Accelerating Economic Growth (EG); and 3) Improving the Health of the Timorese People (Health). Under DG, interviews were held with 14 grantees and with several ministries. Data were collected in five categories: 1) public information; 2) government service delivery; 3) civic education; 4) legal education; and 5) elections and local governance. Grants that worked well under the SGP mechanism included grants that responded to need and piloted new ideas such as the Radio Rakambia radio drama for voter education during the first ever suco and aldeia elections; grants that fund new components of existing programs such as the addition of a civic education page and additional distribution of the children’s newspaper Jornal Labarik (JL); grants showing synergies across sectors such as JL focus on literacy (EG) through distribution in schools and inclusion of articles on basic nutrition (Health); measurement of outcomes through inclusion of a simple readership survey to JL readers; grants that encouraged central-local collaboration and government buy-in from the start such as the GIO/Council of Ministers training program for effective two-way communication between ministries and districts and the grants to INAP-DNAT-Local NGOs for the curriculum development, TOT, and delivery of village administration and management training for aldeia leaders; grants that encourage collaboration between partners for collective impact such as the SGP “civic education consortium” of grants to RR, ETIS, and BB for delivery of a nationwide multi-media civic education campaign; and grants that respond quickly to local needs and fund delivery of services by local organizations such as the legal education and legal internship program delivered by ECM, a district-based NGO, to help communities to resolve disputes and understand different legal processes for handling cases. Grants that did not work well under the SGP mechanism included grants that require longer term support (more than 6 months) or include support for the establishment of new institutions such as Uma Media Regional, a media center set up to serve journalists in three districts (also needed were feasibility studies, building local ownership, raising long-term support); grants that are for one-off programming and not linked to further outputs or x outcomes such as funding for the ARTKL community radio association congress without funding for follow-up activities to implement congress recommendations; and funding pilots without conducting evaluations to determine replicability, such as funding of an innovative civic education program for youth and martial arts group in one sub-district without follow￾up to determine replicability in other sub-districts. The latter two represent “missed opportunities for greater impact” more than poorly executed programs, but highlight the need to focus on strategic (although small in scale) interventions in SGP. Recommendations for future SGPs in DG include: 1. Include listener/viewer surveys in grants to media organizations to find out to what extent media programming led to absorption of key messages by the public. Measuring the extent to which listener/viewer knowledge increased and/or opinions changed is crucial toward developing effective media programming for public information campaigns in the future. 2. Continue to support media associations through meetings to share experiences, network, and develop joint programming and advocacy, but make sure support is phased to ensure follow- up action by the association. 3. Continue to fund new public information components to existing media programs (additional pages to newspapers, air time to regular show, etc.) to ensure sustainability and immediate impact. 4. Support central government communication programs to underline the importance of public information for conflict mitigation and national development. 5. Include an evaluation of training provided in grants with government materials to find out to what extent participants absorb the information and in what ways they act on the information. This evaluation is needed to inform materials development and methodology for future government training. 6. Future government service delivery initiatives should prioritize programs with strong central government leadership and willingness to cooperate with local government and NGOS at the district level. 7. District NGOs that have strong relationships with local government should be prioritized as implementing agencies over Dili-based agencies to execute local level training on government service delivery. This will promote contextualized training and decentralize capacity building service provision. 8. Future programming should include replication and/or scale up of successful civic education programs targeted at the village level, such as the human rights and democracy for youth program. These programs are crucial for conflict mitigation and effectively utilize civic education programs to relay peace and democracy messages to the audience. 9. Continue to use multiple media options in the implementation of civic education programs including radio, video/film, and drama. Use the civic education initiatives to leverage other programs (i.e., sharing facilitators and materials) and use community groups to effectively facilitate interactive discussion and monitor impact of the programs. Large civic education campaigns should be combined with smaller, facilitated discussions. Curriculum guides or discussion aids should also be developed for use with each type of civic education medium used. 10. Disseminate civic education messages through the school system, churches, youth groups, women’s groups, local government networks, and other existing structures as much as possible to take advantage of regular communication channels rather than relying only on outside facilitators and externally imposed community meetings for more lasting impact. Experiment with different ways of meeting people’s information needs by tailoring civic xi education materials to local issues and concerns (i.e., link food security or flooding to government structures and citizen participation). Under EG, 22 grantees were interviewed, along with several ministries and NGOs implementing larger projects that included grantees in some way. Grantees fall into the following three categories: 1) literacy and numeracy; 2) private sector development (microfinance and business development); and 3) youth programs/pilots/entrepreneurship education. Grants that worked well under the SGP mechanism address local needs (literacy and numeracy classes); improve the income of agricultural and craft producers by creating stable markets; create synergies in a value chain; promote local products to expand markets; empower beneficiary populations with an understanding of finances in a market-based economy; train populations on how to develop a small business/microenterprise; creates jobs or linkages to jobs for the unemployed; prepare the unemployed to access employment; provides internships for students to give them practical experience; and links sectors together (agriculture and environment) to produce synergies. Grants that did not work so well under the SGP mechanism include those awarded in microfinance, largely because the enabling environment does not exist; youth pilot projects in horticulture targeted to the wrong populations or had design and execution problems; and certain aspects of HRM internships. Recommendations for future SGPs in EG include: 1. Continue to fund the expansion of L&N projects, especially if the time frame for implementation is at least a year to allow for the development of greater Tetum fluency by facilitators and the further development of more innovative adult teaching methodologies that take into account the difficulties of acquiring literacy skills in a second language. 2. Continue to fund innovative linkages between L&N and private sector development in the form of economic literacy/entrepreneurship education, but also fund the development of appropriate materials and a facilitator TOT in their use. Provide consultancy services to guide the development of these materials so that they address the needs of neo￾literates. Evaluate the use and implementation of these materials and their impact in terms of equipping economically active participants with skills needed to improve their businesses. 3. Consider funding intra- and cross-sector linkages to utilize and/or adapt materials already developed in each sector to expand their use (e.g., JA materials adapted for economically active adults). 4. Do not provide further funding to MFIs for expansion until the enabling environment has improved and MFIs have secured commercial funding to on-lend to their clients. 5. Continue to support feasibility studies, pilots (including funding and time for the evaluation of each pilot), purchase of equipment, provision of training, and similar endeavors to promote long-term private sector development, but link these projects to more longer-term development endeavors so that the projects can be nurtured and leaders can be mentored.3 3 In conjunction with a larger project (such as DSP), support the pilot creation of a district-based business “incubator” to meet the advisory and learning needs of microentrepreneurs and business people at all levels. The incubator should be a place where ideas of all types are discussed, where business plans can be developed, where entrepreneurs can learn how to conduct a simplified value chain analysis, and can be trained on financial xii 6. Continue to support aspects of broader, longer term projects in private sector development that these projects are not equipped to provide. By so doing, require accountability mechanisms in the larger project for the creation of impact (i.e., provide whatever services are necessary to leverage greater impact of the SGP grant). 7. Continue to support opportunities to promote the sale of local goods, e.g., craft fairs. (This could also be included in a more longer-term project in private sector development.) 8. Continue to support job/career fairs that link potential employers to the unemployed; consider the feasibility of funding the implementation of fairs in other districts. 9. Continue to support pilot and capacity building programs that would position youth to be more competitive in the employment market, monitor such programs closely, and evaluate them to determine if they are appropriate for scaling up in a larger project. 10. Continue to support pilot HRM and other internships to provide work experience for participants, but closely monitor and evaluate each pilot to determine the ability of interns to find employment afterwards. 11. Support short-term and/or pilot collaborative endeavors between GOTL and NGOs in economic development activities that spearhead food security, export promotion, agricultural innovation, appropriate technology, cooperative formation, and other similar pursuits. Evaluate these rigorously to determine if they should be leading elements in a broader, longer-term project. Under Health, 11 grantees and members of several community based supporting organizations and ministries were interviewed in the following two categories: 1) workforce development, and 2) program support. Grants that worked well under the SGP mechanism include those that were consciously aligned to the needs of MOH and its USAID partners, which provided more depth and scale for impact. Most of the bilateral projects funded for health were for larger interventions that the MOH was not able to fund at the outset. Consequently, SGP provided funding for many of the needed inputs for the interventions to roll out. These have since become part of MOH’s basic services package. The grants that worked well were used for a response in 1) Leveraging for Greater Impact - through co-funding and collaboration, 2) Strengthening the Service Logic to CBOs - for future campaigns, 3) Innovative Programs – psychosocial healing, and 4) Emergency Health Responses. Grants that did not work so well under the SGP mechanism were those that required longer-term funding and stronger policy frameworks, i.e., workforce development, or had significant funding or logistical problems in implementation. Recommendations for future SGPs in health include: 1. Continue to support professional associations if the grants are provided for discrete outputs and can be leveraged off other contributions. Look for longer term grant impact in the cohesiveness of the association, the leadership, human resource capacity and sector support. 2. Continue to provide funding for inputs, e.g., training or materials, to support larger health intervention campaigns particularly when leveraging off other donors for greater impact and when there are good central-local connections. management. The center should be launched with a grant to last long enough to create customers who might, at some time, be charged a small fee for services. Technical assistance could be provided by DSP so that managers will know how to develop and sustain the institution. xiii 3. Continue to support opportunities to strengthen the service logic of CBOs (MOH- >INGO’s->NGOs->CBOs->community) by funding inputs at different points on the logic chain, e.g., CRS and ITNs or CBO and training inputs. This will continue to provide support to CBOs to enable the high levels of personal motivation the coordinators and their volunteers contribute to make the health strategies maintain effective outreach. 4. Continue to support innovative health service projects that meet a need or can be scaled up for future consideration. Look for projects where the lead agency has good evaluation capacity and networks for referrals or specifically request these structures be developed as a part of the grant. 5. Do not fund projects through an SGP that have long time horizons and require sector support within unclear policy frameworks. These are best funded by other longer term mechanisms. 6. Consider the opportunities for replication and/or scale up of health education programs that are within the absorptive capacity of the NGOs but will ensure ongoing commitment and regular service delivery. 7. Consider funding more integrated projects that support good health outcomes, e.g., perma-culture and income generation activities that link food security, effective food utilization, better healthcare and sanitation. 8. Consider including innovative media components when funding NGOs to support better information transfer that will lead to long-term behavior change. The Summary and Conclusions to this section respond to the specific questions posed in the SOW. 4 Overall the evidence for the efficacy of SGP as a funding mechanism at the specific time in Timor-Leste’s history indicates that there are many more cases that worked well than those that did not. The strategy utilized by DAI in selecting grantees produced many success stories in terms of projects meeting their goals (normative) than those whose ideas were inappropriate or whose implementation was entirely off the mark. SGP is a good mechanism where 1) there is an emergency/ humanitarian crisis that needs a funding source to meet needs quickly, 2) support for a project is embedded in other, larger projects for sustainability, and 3) a good idea should be piloted. Did SGP investments have a sustainable impact? Yes, we believe they did. Training programs have imparted much-needed skills, health campaigns have prevented the spread of disease, society has knowledge of how elections work and the rights of citizens, incomes have increased, and parents have the knowledge of how to better care for their children nutritionally (among other long-term results). SGP funded a number of pilot projects in each sector that could possibly be scaled up into larger projects. However, since pilots were not systematically evaluated, it is unclear which was sufficiently successful to be considered for expansion. In terms of building capacity, SGP did a good job in helping grantees in developing proposals, identifying line items for the in-kind budget, in visiting projects and providing advice to grantees to improve practice, and, in general, providing project support. SGP also identified more developed grantees to expand their capacity through more rigorous managerial training (the development of ROs). While the development of ROs is a good idea, development of managerial capacity for all grantees could have been undertaken in a 4 The second question is answered differently for each SO reported on in Section 4. The third and fourth questions could not be fully answered due to the changes in methodology and in the time constraints for conducting sub-studies to focus specifically on the media, private sector development and nutrition education. xiv different way. This way is presented in Section 6.2, beginning with the grant solicitation process and continuing through the grant proposal review process, managerial training, and monitoring and evaluation. Could USAID have achieved the same or better results more cost effectively? Probably not. The changing security situation, the lack of managerial capacity in many NGOs/CBOs, a government not yet focused on long-term goals, and a high population of IDPs all pointed to the need to be financially and programmatically conservative. There was a need to pilot different ideas, to respond to humanitarian needs, and help develop capacity. All of this was accomplished, to one degree or another, by SGP. Since Timor-Leste is now moving into a new phase of development, however, it is now time to consider designing and implementing longer-term projects, especially in education, health, and agriculture, with support for the development of civil society across these sectors. It also time for greater donor collaboration to achieve GOTL goals. SGP Management As with most development projects, hindsight produces a lot of lessons learned and instruction for how projects could be managed more effectively in future. SGP is no different. In the face of 1) an extremely fragile socio-political environment in which there were several civil disruptions, 2) a donor environment that shifted focus as well as personnel several times, and 3) an initial shortfall in skilled project management personnel, DAI performed extremely well. It created internal systems to facilitate proposal development, acceptance, implementation, monitoring and evaluation; it identified a number of very well￾qualified, loyal and committed staff; it devised strategies to link grantees with larger projects; and it developed a type of grantee capacity building that would serve them well in dealing with the donor community in future. That being said, some of the processes and procedures developed to identify, process, monitor and evaluate grantees and their projects could have been more results oriented and involved grantees in the process more directly so that they could learn from their own mistakes. We point out where the systems could have been improved in our discussion. We do not make any formal recommendations for SGP management as the project will end in just a few months. Rather, we take our comments and transform them into lessons learned and recommendations for future USAID funding and for other donors considering launching an SGP. Lessons Learned and Recommendations In this final section, we present the rationale and make recommendations for future SGPs based on what we learned from the research. A condensed version of these recommendations is as follows: Small Grants Project Cycle Design • Define parameters of SGPs carefully, including: flexibility/open vs. strategic/focused, ceiling amounts, time frames, stand alones vs. embeddedness in other projects, in￾kind vs. cash, and pilots vs. expansion of existing project. Table 6.1 on pp. 46-48 sets forth the advantages and disadvantages of one choice over another in each category. • Conduct background research/needs assessments/situation analysis for each district in which a grant will be implemented to ensure that local needs are being met. Have xv potential grantees conduct a needs assessment using rapid appraisal methodologies to identify specific proposal-related needs and target populations. • Demonstrate alignment with GOTL priorities at all levels. • Involve GOTL in coordinating donors and granting agencies so as to produce collaborations and synergies among grantees and produce greater impact. • Encourage formalized stakeholder involvement to develop ownership. • Favor NGOs/CBOs that work within existing systems so that when the project ends, group activities will continue. • Evaluate pilot programs to determine if they can be replicated or scaled up. • Have grantees self-monitor their projects by providing them with simple tools so that they can learn from their own mistakes and make project adjustments accordingly. • Build the capacity of grantees to take on larger responsibilities. Grant Management Processes and Procedures In this section, we recommend a process for managing grants that includes the following components: • Proposal Solicitations • Initial Workshop • Follow-on Consultations • Composition of the Proposal Review Team • Proposal Review • Successful Proposals • Grant Monitoring and Evaluation • Grantee Capacity Building for Implementation Creating Synergies for Greater Impact • Continue to strengthen professional associations through support of annual meetings and occasional operation support, but conduct an evaluation of this support in terms of how the organizations have become more effective and efficient as a result. • Use media across SOs to enhance public information campaigns for health, EG, and D&G Final Recommendation to DAI • If funding allows, invite all grantees to a three-day workshop to share implementation strategies, lessons learned, materials produced and to create synergies for future operations xvi 1. INTRODUCTION AND SGP BACKGROUND This end-term evaluation assesses the USAID-funded Timor-Leste Small Grants Program (SGP). It documents the program’s accomplishments and identifies lessons learned. In particular, it identifies what worked well and what did not work so well in implementing a small grants mechanism. Particular attention is paid to 1) the capacity building processes SGP undertook with grantees, 2) the new ideas that have been piloted by grantees that can inform future USAID programming, 3) the ways in which the flexibility of SGP has promoted the development of effective implementing partners, and 4) how selected grants have been linked to larger USAID-funded projects. Assessment was undertaken under the three Strategic/Functional Objectives that have guided USAID activities: Governing Justly and Democratically (DG), Accelerating Economic Growth (EG), and Investing in People - Health (see Appendix A – Scope of Work). The report is structured as follows: the remainder of Section 1 addresses the changes in USAID priorities over the course of the grant period to orient the reader to the flexibility that SGP had to exhibit. In Section 2, Background, we provide a brief historical overview of activities in Timor-Leste that have a direct bearing on SGP and of the development of civil society organizations. Section 3 presents the Methodology, a condensed version of the broader methodology developed prior to arrival in Timor-Leste. Section 4 presents our findings on DG (Jana Hertz), EG (Nancy Horn), and Health (Nancy Sheehan).1 In this Section we analyze our findings in terms of what worked well and what did not work so well, and make recommendations applicable to each sub-section. The Summary and Conclusions sub-section directly addresses several of the questions posed in the SOW. Section 5 discusses the Management Structure and Process of DAI in implementing SGP, identifying good practices and areas where improvement could be made. In Section 6 we present key lessons learned in implementing an SGP and make recommendations for the development of synergies and future USAID or other donor-funded SGPs. 1.1 Status of USAID SGP Award The Timor-Leste Small Grants Program (SWIFT II, DOT-I-00-03-00004, Task Order No. 801), was launched in October 2004 to support USAID’s Strategic Objectives (SOs) of improving good governance practices, accelerating economic growth, and improving the health of the Timorese people, as outlined in the USAID/Timor-Leste2 Mission Strategy for 2004-2009. Through small grants, the program has supported communities, organizations, and government in their efforts to build a stable, economically robust, democratic country. As of April 15, 2008, SGP had awarded 219 grants and disbursed more than $6.5 million to 94 local institutions, 21 government agencies, and 12 international NGOs (INGOs).3 It is estimated that during the project, 87 awards were made in DG for a total of $2,245,811 (34% of award amounts);4 78 awards were made in EG for a total of $2,692,826 (40% of award amounts); 50 awards were made in Investing in People (including health and L&N) for a total 1 Literacy and Numeracy (L&N) was originally under EG, but since the changes in 2006 has been under “Investing in People.” For the purposes of this report, we have included L&N under EG. 2 East Timor was the name used officially by the US State Department and USAID until 2007, when all internal and external naming changed to Timor-Leste, the official name of the country under the Constitution of the Democratic Republic of Timor-Leste. 3 By the time this report was submitted, May 31, 221 grants had been awarded. 4 In Section 4.1, we report that there were 89 DG grants awarded. One occurred during the time that the research was underway and will be reported on in the next SGP semi-annual report, and one was not classified on the DAI database in any of the categories discussed. This brings the total to 89. of $1,516,522 (24% of award amounts), and 4 awards were made for special projects totaling $99,512 (2% of award amounts).5 1.2 OTI Background to SGP Award Prior to SGP, DAI implemented the Office of Transition Initiatives (OTI) – Building Empowerment, Leadership, and Engagement (BELE) and Transition Engagement for Population Support II (TEPS II) Programs – under the East Timor Community Stabilization Programs (AOT-I-01-98-00199-00). The latter sought to overcome the devastation brought about by clashes between Indonesian troops and local militias during the people’s vote for independence.6 Picking up where the two OTI initiatives left off, DAI also implemented the Program Support Initiative (PSI) (2001-2004), which, in retrospect, served as a bridge in concept and function to the current SGP. In many ways, the current SGP has served as a more in-depth, finely focused follow-on to the OTI and PSI activities in that it sought to rehabilitate, reconstruct and lay the foundations for supporting the three USAID SOs. However, in identifying potential grantees for the SGP, DAI learned that many community-based organizations (CBOs), non-governmental organizations (NGOs), and government agencies did not yet have the skills to compete for a small grant. Hence, much effort has been expended by DAI to build the capacity of local organizations so that they could successfully compete. 1.3 Changing USAID Objectives Over the contract period, SGP implementation, in general, moved from an underlying flexible strategy to a more strategic programmatic focus. Proposals had to fall under at least one SO category and one Strategic Focus Area (SFA):7 SO1, Accelerated Economic Growth: 1) Agricultural Production and Youth Programs 2) Foundations for Entrepreneurial Education 3) Financial Services 4) Investment Promotion and Economic Policy. SO2, Key Foundations of Good Governance Strengthened: 1) Rule of Law 2) Media, Oversight and Improved Service Delivery 3) Local Elections. SO3, Improved Health for the People of Timor-Leste: 1) Improved Health Service Delivery and Disease Prevention 2) Maternal and Child Health. For the first year of the project, grants were implemented under these three SOs and nine SFAs. 5 DAI. 2008. USAID Timor-Leste Small Grants Program Semi-Annual Report October 2007 – March 2008. Dili: DAI/USAID, April 15, pp. v; 3-4. 6 John P. Mason., et. Al. 2002. End-of-Project Participatory Evaluation. Building Empowerment, Leadership, and Engagement (BELE) & Transition Engagement for Population Support II (TEPS II) Programs. East Timor. Washington, DC: USAID/OTI/DAI 7 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report: April to September 2005. Washington, DC: USAID 2 Beginning in October 2005 the number and type of SFAs increased, and the type of grants awarded was more finely tuned. Projects reported on were still under the three SOs, but the SFAs changed.8 During the October 2006-March 2007 and April-September 2007 reporting periods, there was an increase in social disturbance, culminating in presidential and parliamentary elections. Dissatisfaction with results was demonstrated in outbreaks of violence in many locations. The violence and the promulgation of a new USAID/STATE Foreign Assistance Framework and Operational Plan for Timor-Leste created the need for further adjustment in SOs and SFAs:9 The Operational Plan Focused Objectives (FOs) for economic growth and governing justly and democratically were more finely tuned, and L&N was moved from EG to Investing in People.10 1.4 Operational Plan USAID’s Operational Plan (developed 2006/7) reduced the number of outputs it tracked from the original Performance Management Plan (PMP) that had been developed in accordance with the Mission’s Strategic Plan: DG, 17 to 4 indicators; EG, 10 to 2 indicators; Health, 7 to 2 indicators. Hence, SGP had to reorganize its database to accommodate the reduction and blending of indicators as follows: Governing Justly & Democratically • Number of individuals who received training in local government and or decentralization • Number of people who have completed Civic Education Programs • Number of Non-State News Outlets Assisted • Number of Journalists Trained Economic Growth • Total number of micro-enterprise clients of USG-assisted business development services • Number of Persons receiving training in USG-supported micro-entrepreneurship training sessions Investing in People • Number of adults enrolled in literacy programs • Number of people trained in maternal and/or newborn health and nutrition care through USG-supported programs It should be noted that the indicators under the Operational Plan are NOT results indicators, but output indicators that measure only participation in training and/or technical assistance programs. These vary considerably from the results orientation of the PMP.11 8 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report: October 2005 to March 2006. Washington, DC: USAID, 9 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report October 2006 – March 2007. Washington, DC: USAID, April 16, 2007. 10 Ibid., pp. 3-4; for a full explanation of these changes, please see Appendix C – Methodology. 11 The team understands that a new PMP is being developed, but it was not ready at the time of writing this report. 3 DAI reported on the Operational Plan indicators in their semi-annual reports. Output reporting presented some problems to the evaluators because we did not have sufficient outcome criteria against which to measure success or the lack of it within SGP. 2. BACKGROUND 2.1 Timor-Leste Since Independence Since independence in May 2002, Timor-Leste has adopted a constitution that provides for a democratic and pluralistic society and basic fundamental rights and freedoms. The second presidential elections were conducted in April 2007 and parliamentary elections in June 2007. The new government was sworn in on 30 August 2007. The new government faces many challenges on several fronts. Peace is still elusive and civil strife remains a harsh reality. Internally displaced persons (IDPs) live in constant fear of reprisal if they return to their original places of residence. Poverty is endemic and worsening – particularly in rural areas where 46% of the population lives below the official poverty line of US$ 0.55 per capita per day. About 20% of the population is food-insecure and a further 23% is highly vulnerable to becoming food-insecure. According to a recent FAO/WFP Crop and Food Supply Assessment Mission Report, there has been a significant decline in production of maize (30%) and rice (20%). The recent increase in the cost of rice has exacerbated food insecurity. Youth unemployment is high and is also a major contributory factor to violence and social unrest. Agriculture is at its early stages of development and industry almost non-existent.12 Timor-Leste’s Human Development Index (HDI) is the lowest within the Southeast Asian region: 150 out of 177 countries. Tuberculosis and malaria are endemic. The status of women and children is generally low. With an average fertility rate of 7.8 births per woman (highest in the world)13 and with the likelihood of the population doubling in 15 years, the country is passing through a demographic transition of unprecedented proportions. Under￾five mortality is as high as 130 (per 1000 live births) and maternal mortality 660 (per 100,000 live births). There are huge gaps in educational and health attainments between men and women even though life expectancy among women is higher. UNICEF reports that life expectancy at birth in 2005 was 56 (up from 45 in 1990, and 40 in 1970).14 Gender-based violence in different forms is another area of concern. The mass media, as a conduit of all sorts of information and as an independent oversight entity, remains fragile and underdeveloped and therefore requires significant capacity development efforts to be able to reach out to more people in the rural areas. The majority of the population faces a multitude of insecurities such as food insecurity, especially among the rural population, for most of the year; physical insecurity, because of the prevalence of generalized violence in society; and insecurity emanating from the absence of rule of law manifest in the lack of property rights, among others. Additionally, rural communities also face livelihood insecurities resulting from dependence on a fast-degrading environmental resource base and climate change. The cumulative effect of all these insecurities is the heightened vulnerability of an average Timorese. 12 UNICEF reports that in 2005 only 8% of the population was urban and growing at an average rate for the period 1990-2005 at 1.6%. www.unicef.org/infobycountry/Timorleste_statistics.html?q. 13 www.unicef.org/infobycountry/Timorleste_statistics.html?q 14 Ibid. 4 In a scenario like this, survival, livelihoods and dignity are under serious threat and pose immediate danger to people’s fundamental freedoms. The SGP sought to address these by building trust among people, assisting them to rebuild their lives and livelihoods, and create enabling conditions where they can live without fear and with freedom and dignity. 2.2 The Birth and Growth of Local NGOs It is very important to understand some of the factors surrounding the formation of the many Timorese NGOs and some of the constraints that have impeded their growth. The following features may vary from one organization to the other, depending on whether the organization has a strong international network, whether it has some basic income generating activities to keep it going, whether it benefits from special governmental attention, or it is merely a group of people with a common interest to implement a project based on funding availability. The United Nations Transitional Administration in East Timor (UNTAET 1999-2002) served as an interim and the first ever government of the newly born country. It had huge international support to rebuild and rehabilitate over 80% of the new country’s infrastructure and facilities that were either destroyed or rendered inoperable during the 1999 violence and to build the government’s institutions. For the first time in the history of the UN, it sought to rebuild a country from scratch; it required assistance to launch such a huge international effort. At the time of the emergency everything was new and everything became a priority. The influx of financial resources to the country was in billions of dollars. International organizations were relatively new to Timor-Leste and needed local organizations to implement some of their programs. During these early days of independence, Timorese – many exiled during the Indonesian occupation and/or fresh from internal struggle - felt the need to establish their own enterprises, political parties and civil society organizations. It was against this backdrop that a range of local organizations was formed. Many, if not all, were formed on the basis of the common goals and ideas shared during the period of resistance. Most of the founders of these organizations were former activists whose ideas were either oppressed or could not be realized during 24 years of occupation. Many of these activists wanted to see real societal change but were not well prepared to make it happen. Consequently, when SGP was launched, it found huge deficits in capacity among NGOS as most had been formed only to take advantage of funding opportunities to bring about the societal change they sought. The Timorese NGOs will indeed take time to develop due to some structural impediments that are beyond their control. One of the major impediments is the poor and underdeveloped private sector as a key driving force in economic growth. Another is the cost of doing business, which is exacerbated by monopolies providing overpriced goods and services preventing many Timor-Leste’s businesses from expanding. As a logical consequence, the economy remains stagnant, foreign direct investment is not flourishing, and funding from multinationals that can be tapped for corporate social responsibility is minimal. Most NGOs have to rely on donor and government funding for their survival. Compounding development concerns is the lack of laws and regulations to protect and promote NGO participation in development activities. During “lean” financial seasons, some NGOs dissolve and staff obtain employment with other agencies. NGO ability to manage larger projects is challenged by human resource limitations. Educated Timorese, especially those with management skills in development assistance, are 5 limited in number. This dearth in skilled person power is not being fully addressed by donors and government in joint education and training efforts to produce a cadre of individuals who can successfully manage economic development and who can become long term, sustainable partners in project design and implementation. FONGTIL, an umbrella organization of NGOs (local and international), has tried to improve development skills by providing technical assistance and advocacy for its members. FONGTIL was formed in 1998 beginning with 14 organizations and now has 487 NGOs on its membership roster, including 394 local NGOs and 93 INGOs.15 Other organizations such as Belun, which works with 150 NGOs, is also active in NGO organizing and capacity building. SGP has partnered with many organizations registered through FONGTIL and has sought to build their capacity. At the time of the evaluation, we ascertained that NGOs and Timorese organizations could be categorized in different ways: • Several NGOs were actually spin-offs from international NGOs. Having been exposed to and participated in different levels of management, leaders of new NGOs had already established a more business-oriented culture in which results were to be achieved (ETADEP, TRM) • A few NGOs were established by members of the Timor-Leste diaspora who had been in Australia or other countries for an extended period of time either in school or working in different jobs where a work culture was established. When these same people returned to Timor-Leste, they brought not only expertise, but also relationships with influential others in the diaspora and nationals of the country of temporary residence. These networks were then called upon for a range of support (ETDA, Timor Aid). • A few NGOs were established by very highly placed government officials or their relatives who had the power and the relationships to move things along. In such elevated positions, these NGOs could more easily access funding from different sources, gain more experience in obtaining and managing grants, and in providing needed services throughout the country (Timor Aid, Alola Foundation). • A few NGOs were established by those who were part of the resistance and lived in economically challenging areas strategizing how independence could be achieved. The survival and strategizing experiences, as well as the relationships established in the jungles, all served to build capacity in strategic thinking, planning, and implementation (Haburas Foundation). • Some NGOs and businesses had long-term experience in the sector in which they sought grants and so their technical expertise was well developed, although some of the management skills may have been lacking (Fundacao Cristal, Zero Star, FBM, Acelda). • Some NGOs employ foreigners as advisors or leaders, and build capacity internally as a part of their jobs (Alola, Moris Rasik, Rede Feto, JA). • Some NGOs are led by or employ individuals who have had significant overseas experience, either in school or for work. The exposure to and participation in different management practices may have had a positive influence in the management of projects (TYEI, FBM). We present this typology to point out 1) the personnel and skills variation in organizations, 2) why different organizations might be better prepared for SGP or other grants, 3) why some NGOs might have been more successful than others in implementing SGP grants, and 4) what 15 Numbers reported from the FONGTIL database, March –April 2008. 6 relationships have to do in establishing, developing and maintaining an NGO. Many other NGOs that could not claim any of these relationships, it appears, did not fare as well as those that did. That being said, when SGP provided a grant, a seed was planted. In some cases, the harvest can be documented, but in others, the seed is still germinating. The legacy of SGP will be seen in the NGOs harvesting what was planted in very turbulent, fragile times. 3. METHODOLOGY The methodology originally developed by the Team Leader for this research was conceptualized, reviewed by DAI and USAID, and revised on the basis of documents provided by DAI and USAID (see Appendix C – Draft Evaluation Methodology). However, after the evaluation team’s initial interviews with DAI staff, we learned that the preponderance of grantees are headquartered in Dili, implemented their programs in the districts, but had no long-term presence there.16 Hence, we limited our geographic scope for this research to Dili, Baucau, Liquica, Manatuto and Aileu. We then learned that synergies among grantees had not been cultivated and that each grantee had little, if any, knowledge of other grantees. Consequently, the “mosaic” approach to gathering data had to be dropped. During the course of the evaluation, 14 DG grantees, 23 EG grantees, and 11 Health grantees were interviewed (many grantees had been awarded several grants) (see Appendix B – Meetings Held). As we went from interview to interview, we utilized a modified “snowball” approach, i.e., when one grantee recommended we talk to a particular agency to get a clearer picture of project development and implementation, we interviewed them. When we were encouraged to have a conversation with a senior government or church official, we also did that. In conducting interviews we utilized the logic model approach: inputs -> activities-> outputs-> outcomes-> impact. Much of the information we were provided by DAI on each grant informed us of inputs, a bit about the activities and outputs, but very little on outcomes and impact. Consequently, we focused our attention on activity implementation and learned as much as we could about outcomes and impacts although, in most instances, we did not have time to interview ultimate beneficiaries. Accompanying us was a local consultant who worked primarily with Nancy Horn on EG, but who also provided valuable assistance in understanding the local context, writing portions of this report, reviewing all our writing for accuracy and designing our feedback presentation template. He also translated when necessary. We also had two assistants to help us set up appointments and to translate when needed: Maria Maia Boavida and Dulce Maria Gusmao. . 4. GRANTEE FINDINGS In this section we present our findings in terms of what worked well and what did not work so well in the design and implementation of grantee projects (according to each SO area). In the analysis of what worked well or not, we consider whether the idea itself was good or not (summative discussion), or if the problems were those of execution/implementation 16 The Team Leader assumed, based on the information DAI provided, that grants were awarded to organizations in the districts in which they were implementing. With several offices and projects in one district, it was also assumed that synergies among grantees could be created. When we learned in the field that grants were awarded to Dili-based organizations who set up grant-based implementation mechanisms in different locations (not more permanent facilities), the mosaic/synergies approach had to be dropped. 7 (formative discussion). We conclude each subsection with recommendations relevant to the particular collection of projects. In the “what worked well” sections below, some of the parameters that emerged to guide us in our analysis included: • Pilot programs (provided evaluations are conducted to determine the possibility of replication) • Response to local/expressed need • Working closely with the community/mobilization of the commuity • Embeddedness in local networks for sustainability • Endorsement of local officials for sustainability • New components of existing projects/programs • Leverage other grants or participation • Create cross-sector synergies • Create central-local collaboration • Delivery of services by local organizations (CBOs) • Create GOTL-NGO/CBO linkages • Create collaborative relationships among grantees • Create an enabling environment • Ability to respond quickly in emergency situations • Ability to phase funding based on achievements • In line with government priorities • Increased income for participants • Laying the foundation for long-term change • Creating new materials in local languages • Targeting a program to the right audience • Contributing to the development of a larger project/program • Building local and national networks for sustainability • Suitability to the Timorese context In the “what did not work so well” sections below, several parameters also emerged: • Failure to understand the needs of target beneficiaries • Failure to coordinate with others and maintain good communications • GOTL shortfalls in providing the appropriate policy environment • Inability to communicate • Failure to follow-up on or evaluate informational campaigns • Establishment of new institutions • Funding one-off projects not linked to other activities • Total reliance on a “temporary” population (i.e., students) for implementation • Reliance on only one person’s expertise • Expansion into a totally unknown environment without conducting a feasibility study • Lack of supervision by more experienced professionals • Failure to conduct a simple feasibility study thus leading to erroneous assumptions The recommendations we make are directed to any donor who is establishing an SGP in one of the areas discussed. We have extrapolated the lessons learned from implementing the current SGP to inform the design and implementation of future SGPs. 8 4.1 Governing Justly and Democratically The democracy and governance portfolio within SGP consists of approximately 89 grants17 divided into public information (43), government service delivery (15), civic education (14), legal education (7), elections and local governance (6), and oversight institutions (3). These grants reflect the strategic and functional objectives of USAID in the 2005-2009 strategic plan and revisions in 2006. The portfolio is diverse and includes funding NGOs and CBOs to meet the needs of both government and the public. The organizations interviewed for the evaluation in the various grant categories are listed below.18 Grants are classified according to the various SGP database focus codes for the DG SO, although grants in practice may bridge several different focus codes. The number in parenthesis indicates the number of grants to the organization in the grant category. For D&G, 14 grantees who had received 29 grants were interviewed in Dili, Baucau, and Liquica. Public Information • Radio Rakambia (RR) - radio drama and voter education for suco and aldeia elections (5) • Asosiasaun Radio Komunidade Timor Leste (ARKTL) - capacity building for the association of community radio stations and national community radio congress (2) • Timor-Leste Media Development Center (TLMDC) - newspaper distribution and video production for youth (2) • Uma Media Regional (UMR) - regional media center for eastern districts (1) • Jornal Labarik (JL) - civic education component in children’s newspaper (1) • Ministry for the Presidency and the Council of Ministers - Government Information Office (GIO) training program (2) Government Service Delivery • Cailalo - leadership and administration training for aldeia chiefs (1) Civic Education • Cailalo - civic education for youth on human rights and democracy (1) • Center Audiovisual Max Sthal Timor-Leste (CAMS-TL) - film on presidential and parliamentary elections (1) • Radio Rakambia (RR) - nationwide civic education (eastern districts) and radio drama on parliament (3) • East Timor Insight (ETIS) - nationwide civic education (western districts), (1) • Bibi Bulak (BB) - nationwide civic education (central districts), (1) • Forum ONG Timor-Leste (FONGTIL) - parliamentary film distribution, (1) Legal Education • Fundasaun Educasaun Comunidade Matebian (ECM) - legal education and legal internship program (2) Elections and Local Governance 17 One grant was not classified on the DAI database in any of the categories discussed. 18 Oversight institutions (consisting of two equipment grants and one seminar) and are not represented in the interview list. 9 • Instituo Nacional da Administracao Publica (INAP) - equipment and materials for village council training sessions (2) • Ministry of State Adminstration/Technical Secretariat for the Electoral Administration (STAE) - equipment and voter education for suco and aldeia elections (3) The interviews focused on organizations that received multiple grants or that highlighted an aspect of the SGP the evaluation sought to investigate in further detail, such as capacity building, piloting of new ideas, and links to larger government or donor programs. 4.1.1 What Worked Well Public Information With a flexible SGP designed to respond to needs and pilot new ideas, RR was funded to produce a radio drama for voter education prior to the suco and aldeia elections. This grant is an example of a successful pilot. It was a response to an expressed need for widespread voter education, particularly for illiterate populations, for the first ever direct elections at the village level. RR was able to reach a large part of the voting population through radio broadcasts and was able to reach remote locations with its mobile broadcasting equipment. STAE, one of the central government ministries with which RR worked closely, expressed satisfaction with RR’s production. This grant also highlighted RR’s ability to work closely with community radio stations (through distribution of radio drama CDs) throughout the country. By doing so, RR activated local networks to provide information; this can be replicated to broadcast varying content. RR also demonstrated its ability to mobilize at the community level through the endorsement of local government officials and police for their radio programming. Local government officials announced the radio broadcasting schedule in regular meetings and police officers announced the schedule during routine village patrols. To measure the impact of this grant and to decide whether the pilot should be replicated, it is important for RR or STAE to conduct a listener evaluation or voter survey to find out to what extent voters used the information obtained through the radio drama to educate themselves on the voting process. Simple evaluations and surveys should be included in future grants that serve to pilot a new idea or a new medium to make decisions regarding scale-up and improve the intervention itself. One of the challenges of funding large quantities of small grants is to measure the impact or sustainability of individual grants. One way to address this issue is to fund new components of existing programs. The SGP grant to JL illustrated this best practice by introducing a civic education section and increasing distribution of this previously established children’s newspaper. JL is a good example of the way in which an SGP grant leveraged another program (thus ensuring its continuation to some extent) and used existing media as a channel for public information. JL, primarily funded through UNICEF, is also a good example of synergies across sectors. The newspaper is used for literacy (EG objective) and is distributed through the elementary and middle schools across the country. The newspaper also covers nutrition and good health practices reflecting “Investing in People” priorities. The printed interviews with young people in different districts and the pen pal section of the newspaper also fostered communication among young people in Timor-Leste thus integrating the youth focus of USAID’s objectives. 10 This grant also included a reader survey thus building an outcome measurement aspect into the grant. After several months of distribution with different features, JL conducted a survey of 325 children and 155 teachers to find out how they were using the newspaper and give feedback on the content. A readership survey is a best practice and should be incorporated in future small grants to measure whether the public information being provided is received in the way it was intended and make adjustments accordingly. Government Service Delivery and Local Governance In the selection process for small grants, funding activities that encourage central-local collaboration were shown to be effective in implementation. In addition, grants that have government buy-in from the start have a much higher possibility of sustainability. The SGP grant to the GIO communication training program based in the Ministry for the Presidency and the Council of Ministers illustrates both of these best practices. It was set up to provide capacity building for ministries to share information more effectively with the public and incorporate public feedback into ministerial planning and decision making. SGP funded two grants to the GIO and the program is now fully funded by the GOTL national budget. The SGP grants to GIO consisted of communication training for key members of ministerial staff to increase capacity to manage communication and outreach. In developing the program, the GIO approached the ministers to request trainee nominations. The trainees were required to hold a position level 4 or above (this is stated in the MOU and TOR). The course is taught intensely for four hours a week for nine months. This mandate and support from the various ministries was crucial to the success of the training program. The trainees were assured access to information, logistical support, and approvals for their work. Another set of SGP grants that highlights central-local collaboration and government buy-in is the cooperation between INAP, DNAT, and local NGOs to conduct training sessions for aldeia leaders on village administration and management in nine sub-districts in Manatuto, Baucau, and Lautem. In this series of grants, SGP funded INAP for materials development and training and local NGOs to deliver the INAP course on administration, management, and leadership for village leaders. INAP is an interesting example of a central government agency willing to work closely with NGOs to implement programs. INAP trained 65 sub-district officials and 45 NGO/university graduates in their TOT. INAP had positive working relationships with district level NGOs, such as Guntur, Cailalo, and Aifunan, that conducted the training at the village level. The director believes that capacity building must be done collaboratively and that no one agency can do it alone.19 INAP was pleased with the results and stated that local NGOs were crucial to the program because they have the relationships and local knowledge to contextualize and make the training happen. This grant was exemplary due to synergies formed between local NGOs, local government, and central government. However, Cailalo (one of the local NGO grantees) talked about one of the weaknesses in execution. They were critical of the oversight and monitoring of INAP, and complained of mishaps in the administration of the grant (partly due to coordination issues between INAP and DNAT/DNAL) that caused problems for the NGOs in the field. They reported that SGP monitoring was more regular, although SGP was unable to address some of the administrative issues because of contractual arrangements with INAP. In spite of this, Cailalo reported that 19 Interview with INAP, April 22, 2008. 11 the village leaders requested more training to cover other courses/modules available from INAP. Another area for improvement of this grant is to include follow-up by INAP or the NGOs to evaluate to what extent village leaders have implemented what they learned. This information will be very valuable in revising the national curriculum. INAP plans to work with four “sample sucos” and continue the training program. These sucos can become models for others. INAP suggests that future programming for community leaders or district level capacity building should focus on the sustainability and impact of the program. The director stated that changes in behavior will not happen in one-off training; there must be follow-up and monitoring. Thus, future grants should include monitoring and evaluation components to measure to what extent materials and capacity building were successful in changing behavior and improving governance. Civic Education SGP grants that encouraged collaboration between partners led to a noticeable collective impact. Encouraging associations, consortiums, or a collaboration of partners in small grants means that inputs from individual grants complement each other and create a result that is bigger than the sum of its parts. Collaboration also provides the opportunity for partners to bring different skills and thus create a “conglomerate” of products and services. One good example of an SGP grant that illustrated this success characteristic is the “civic education consortium” of RR, ETIS, and BB, three well known NGOs with previous civic education and/or media experience that collaborated to cover all 13 districts in a nationwide civic education campaign. This consortium also took advantage of the “conglomerate” of products and services created from their collaboration to develop a multi-media event that they took on the road. The civic education campaign included a number of different films on various topics (justice, environment, gender), a drama in the central districts where BB performed, and an audience quiz with prizes to test the participants retention of various points during the campaign. All campaigns included a local band performing music to attract a crowd and entertain the participants. To build on the success of this type of grant in the future, a viewer survey should be implemented to find out to what extent participants absorbed the information and messages they were exposed to during the course of the multi-media event. A school teacher in Liquica commented that the civic education campaign was well attended, and it introduced key messages about responsible civic behavior and discouragement of recklessness. He said the disadvantages of a multi-media event are that viewers may not understand the messages since they do not have adequate opportunity for discussion. The informant recommended a smaller event with better facilitation and a carefully chosen target group with materials specifically tailored to this group.20 The recommendations from the school teacher were echoed by other INGOs and UN agencies conducting civic education. The need to have focused materials and a carefully selected audience with skilled facilitation is important to ensure transfer of knowledge and retention of information in addition to allowing an opportunity for questions and further probing. IFES has a civic education program addressing the changes in parliament using a flip chart methodology and TOT. The UN CSO Working Group also has four facilitators in 20 Interview in Liquica, April 21, 2008. 12 each district tasked with small group discussions of civic education issues. The SGP￾supported civic education coalition of RR, BB, and ETIS could work with IFES and the UN CSO Working Group to leverage their programs and use some of their facilitated, small group discussions as follow-up to larger media events. RR, ETIS, and BB all agreed that the civic education campaign was useful because it reached remote populations and catered to the illiterate. However, the weaknesses were the lack of facilitation and interactive discussion and the distraction of too many different messages. They recommend a large community event once every year or six months with one topic and smaller group civic education session conducted in the intervals on specific topics. The civic education sessions, whether in small or large groups, must be coordinated well with local government agencies, officials, and communities. In several cases during implementation this coordination was neglected and had a negative impact on participation and receipt of civic education messages. In addition, the SGP grantees should solicit input on civic education issues and information needs to tailor the content and delivery mechanisms appropriately. Finally, schools, churches, and other community groups should be targeted and encouraged to conduct civic education sessions to ensure penetration throughout the population. Overall this grant shows the positive results of NGOs working together and the creative use of media for public consumption, especially for illiterate populations. The SGP was an effective vehicle for testing the concept of a nationwide multi-media campaign and it can now be used in combination with other smaller, more targeted civic education sessions that are conducive to implementation in a local context. Legal Education SGPs have the ability to respond quickly to local needs and fund delivery of services by local organizations. One good example of an organization meeting local needs funded through SGP is ECM based in Baucau. ECM received SGP grants for two programs: a legal education program and an internship program. Legal education was conducted in three sub-districts in Baucau. Participants learned the difference between civil and criminal cases and legal advocacy. They also learned about reporting processes and responsibilities of various law enforcement officials (police, local government administrators). ECM received requests to repeat the legal education program not only for leaders (suco chiefs and traditional leaders) but also for the general public due to the importance of the information thus illustrating the responsiveness of the community. Based on their experience, ECM was also able to recommend more regular legal education sessions targeting women due to the number of criminal cases filed related to violence against women, and legal education targeting youth, a group that ECM reports misinterpret their roles within the context of the law. The other part of the SGP grant, the legal internship program, is in its second phase. The first phase involved eight interns that identified legal cases and made recommendations for handling of the cases. A number of interns (4-6) from the first phase are still in communication with ECM and they have become a part of a mobile team in the sub-district. ECM decided to repeat the legal internship program on their own but this time increasing the 13 age limitations to 40 (the previous program included youth up to age 25). The new interns are required to identify five cases each month and resolve one case per month, so the results expected from interns has been quantified. Overall, the grants to NGOs to provide legal education in the communities are meeting an identified need for greater legal information at the community level. Based on monitoring reports the legal education provided informative and clear legal material. Through the internship program, the legal program provided transfer of knowledge and resulted in roaming legal advisors in addition to improving the chance for successful mediation since the public understands the various channels for resolving legal issues. Finally, ECM continues to identify and resolve cases on a regular basis. Many of the legal education NGOs that received grants from SGP have been integrated into The Asia Foundation’s Access to Justice Program to increase rural citizens’ access to legal information and legal aid and improve the quality of informal dispute resolution or mediation. This “scale up” demonstrates the way in which piloting of small scale legal outreach programs contributed to the development of a larger scale program including multiple organizations and broad geographical coverage.21 4.1.2 What Did Not Work So Well Public Information One type of grant that is less suitable to an SGP and more suitable for a longer term support program is the establishment of new institutions. Launching a new institution and building it up to be effective can take several years. Small grants programs can contribute to parts of an institution building process; however it is not advisable for small grants to be the main funding mechanism in this area. The SGP grant to establish UMR, a media center located in Baucau to serve journalists in Baucau, Viqueque, and Lautem, is an example of an ambitious grant to set up a new center in a six-month period (the grant was awarded in December 2007 and must be completed by June 2008 at SGP close-out). The center was envisioned to be a state-of-the art facility for journalists to send and receive stories from Dili and provide journalism training in three districts. It was established by a board consisting of the five major journalist associations based in Dili and linked to ICfJ, who provided TA and attempted to fundraise for a trust fund to continue the center. The idea for this grant made sense, to serve journalists outside of Dili and create a resource for people to obtain information in the districts. However, there are a number of problems in implementation that are linked to sustainability and the time needed to develop. For example, although UMR has strong backing from journalists’ associations in Dili, the board for UMR does not include representatives from the media or other key stakeholders who are based in the three priority districts. For UMR as an institution to take root and serve these districts, influential and dedicated board members from these districts who will build the relationships and promote the center are needed. UMR decided to co-locate with an Australian-funded local NGO that helped in advertising the center, but due to limited space UMR is planning to move into its own office nearby. Although UMR will maintain connections, it will not benefit directly from the walk-in traffic generated from this NGO. In addition, although ICfJ has pledged support and the center is linked to this INGO ensuring some degree of sustainability, 21 In a feedback session with SGP on an earlier draft of this report, SGP cited another example of a pilot that contributed to larger programs: the internship program absorbed by The Asia Foundation inspired the Provedor’s Office to start its own legal internship program. 14 alternative sources of funding (such as income from an internet café) need more time to be developed. In the future, funding for the establishment and building of new institutions can more reasonably achieve results when part of a larger USAID capacity building program. An SGP can pay for the equipment and maintenance, provide TA for outreach activities, or fund a series of journalism trainings in the districts (similar to the kind of deliverables in the current grant). Although the UMR was initially linked to the USAID funded ICfJ Strengthening Independent Media Program in Timor-Leste, this program was scheduled to end in September 2008, just three months after SGP close-out in June 2008. If USAID approves an extension of the ICfJ program for an additional two years, it is a good opportunity to discuss technical assistance and financing for UMR as part of the extension and address some of the fundamental issues of ownership discussed earlier. Another type of grant that makes sense in the SGP context – but must be designed very carefully to demonstrate impact - is one-off programmatic support. One example is SGP support for the association of community radio stations (ARKTL). An ARKTL grant in 2006 contributed to the cost of a three-day congress of media organizations. Some of the results from the congress included plans for a scale up of a Dili-based news packet information service (initially piloted through ICfJ) to all districts. The congress also addressed the need for advocacy on the draft media law to ensure media freedom and protection for journalists. This grant was a good example of an effort to fund activities that link local organizations to one another to encourage joint programming and advocacy. However, because the grant was designed to provide funding for a single event and not for the follow-up activities, it is difficult to judge to what extent the input led to concrete outputs or outcomes. This one-off grant did not ensure the association would have the human or financial resources to implement the recommendations from the congress, namely the scale up in programming and the media advocacy. Thus, for small grants to ensure some level of outcome from the funded activities, it is advisable to organize phased funding (provide funding for congress and additional funds for completion of congress recommendations). This way, unsolicited proposals (such as ARKTL’s request for meeting funds) can still be processed but serve as the starting point for further engagement and more results-based activities. Civic Education Other types of activities funded by SGP with mixed results are well conceived and successful pilots lacking evaluation or follow-up and links to larger programs or existing structures. For instance, the NGO Cailalo received a grant for a civic education program originally targeting village leaders and elders. At the request of the local communities, it was delivered specifically to young people and those involved in martial arts groups. The pilot provided training on human rights and democracy to reduce the potential for youth-initiated conflict. This grant experimented with civic education as a tool for conflict resolution and succeeded in targeting youth at the request of the community. Unfortunately, this grant was not replicated or linked to a larger government or donor program and Cailalo covered only Vemasse sub-district in Baucau. Given the feedback from the community and the willingness of Cailalo to respond to the expressed need, this civic education program could have been replicated in other sub-districts and an invitation 15 extended to encourage cooperation with other local NGOs for delivery in Baucau and surrounding areas. In this case, SGP provided a small grant for a pilot activity in one sub-district that was very successful due to the innovativeness of the local NGO that received the grant. Similar to the previous example regarding one-off funding, it would have been advantageous for SGP to encourage Cailalo to submit a proposal for further development of civic education for youth as a conflict mitigation tool and test it in other sub-districts. If the pilot is judged successful in other sub-districts under different circumstances it could then be recommended to USAID, other donors, or the GOTL for replication and scale-up at a larger level. 4.1.3 Recommendations Overall the DG portfolio of small grants was innovative and addressed information needs in a number of different sub-areas of governance. The majority of SGP grants was appropriate for a small grants mechanism and showed the strengths of the mechanism. A small number of grants that were creative and developed to meet an important need in the community were not as well suited for the small grants mechanism or did not include enough phased funding or follow-up to maximize results. The following summary of recommendations can be used when selecting organizations and projects for funding in future SGPs based on the lessons learned from the DG grants. With a few small interventions at the program design and evaluation stages, the collective impact of an SGP can be greatly increased. 1. Include listener/viewer surveys in grants to media organizations to find out to what extent media programming led to absorption of key messages by the public. Measuring the extent to which listener/viewer knowledge increased and/or opinions changed is crucial toward developing effective media programming for public information campaigns in the future. 2. Continue to support media associations through meetings to share experiences, network, and develop joint programming and advocacy, but make sure support is phased to ensure follow- up action by the association. 3. Continue to fund new public information components to existing media programs (additional pages to newspapers, air time to regular show, etc.) to ensure sustainability and immediate impact. 4. Support central government communication programs to underline the importance of public information for conflict mitigation and national development. 5. Include an evaluation of training provided in grants with government materials to find out to what extent participants absorb the information and in what ways they act on the information. This evaluation is needed to inform materials development and methodology for future government training. 6. Future government service delivery initiatives should prioritize programs with strong central government leadership and willingness to cooperate with local government and NGOS at the district level. 16 7. District NGOs that have strong relationships with local government should be prioritized as implementing agencies over Dili-based agencies to execute local level training on government service delivery. This will promote contextualized training and decentralize capacity building service provision. 8. Future programming should include replication and/or scale up of successful civic education programs targeted at the village level, such as the human rights and democracy for youth program. These programs are crucial for conflict mitigation and effectively utilize civic education programs to relay peace and democracy messages to the audience. 9. Continue to use multiple media options in the implementation of civic education programs including radio, video/film, and drama. Use the civic education initiatives to leverage other programs (i.e., sharing facilitators and materials) and use community groups to effectively facilitate interactive discussion and monitor impact of the programs. Large civic education campaigns should be combined with smaller, facilitated discussions. Curriculum guides or discussion aids should also be developed for use with each type of civic education medium used. 10. Disseminate civic education messages through the school system, churches, youth groups, women’s groups, local government networks, and other existing structures as much as possible to take advantage of regular communication channels rather than relying only on outside facilitators and externally imposed community meetings for more lasting impact. Experiment with different ways of meeting people’s information needs by tailoring civic education materials to local issues and concerns (i.e., link food security or flooding to government structures and citizen participation). 4.2 Accelerating Economic Growth The EG portfolio within SGP includes approximately 78 grants.22 Due to the changes made in 2006, the current focus of grants is on entrepreneurship and enterprise development. For the purposes of this evaluation, however, we have investigated the areas in which SGP provided EG grants, i.e., Literacy & Numeracy, Private Sector Development, and Youth Pilot projects. Approximately 25 EG grantees who had received 40 grants, three government ministries, USAID SO teams, and several support organizations were interviewed in Dili, Baucau, and Aileu to gather data for this section. The following analysis presents some of the reasons why SGP was an appropriate mechanism to fund different endeavors in most cases but inappropriate in others. Literacy & Numeracy 1) Fundacao Cristal (FC) – expansion, development of a video, RO (5) 2) Timor Aid (TA) – expansion (1) 3) Grupo Feto Foinsa’e Timor Lorosa’e (GFFTL) – expansion (1) 4) Fundacao Buka Matenek (FBM) – expansion, development of a facilitator assessment tool (2) 5) National Directorate of Non-Formal Education (NDNFE) (background information) Private Sector Development Microfinance 22 Because L&N grants were moved to “Investing in People,” the number of EG grants does not currently include L&N. 17 1) AMFITL – offer training to members (2) 2) Moris Rasik (MR) – expansion and database development (3) 3) Tuba Rai Metin (TRM) – operational support (2) Business Development 1) Alola Foundation (AF) – equipment for a Christmas fair (1) 2) University of Hawaii (UH), Candlenut Feasibility Study (resulting in a candlenut oil processing and marketing operation led by Acelda) (1) 3) Fundacao ETADEP – support farmer rental of tractors, agricultural diversification (soya beans), RO (3) 4) Zero Star (ZS) – purchase a cold truck, cold storage facilities and greenhouses to improve marketing of perishable crops (1) 5) Rede Feto Timor Leste (RFTL) – conduct a nationwide campaign to promote local products (1) Youth Programs (Pilots)/Entrepreneurship Education 1) Rotaract Club of Dili - attend a conference in Malaysia and write curriculum, to deliver financial literacy classes to high schools, to deliver same training to street vendors (3) 2) Dili Institute of Technology (DIT) – career fair (1) 3) Fundacao ETADEP – youth training program in horticultural production (1) 4) Timorese Young Entrepreneurship Initiative (TYEI) – training in improved agricultural methods, establish a youth rural exchange program (2) 5) Junior Achievement (JA) managed by the Asia Regional Pacific Operating Center - conduct a feasibility study and to implement the JA program in primary schools (2) 6) Centro Treino Integral e Desenvolvimento (CTID) in Baucau – to implement an OJT program in hotel and restaurant management (HRM) (1) 7) Centro Formacao Joao Paulo II – Comoro (CFJPII) – establish an internship program in HRM (1) 8) East Timor Development Agency (ETDA) – to provide improved business services and link unemployed youth with employers, including database upgrade (3) 9) PB Mobiliario (PBM) – training program and internship in carpentry in Baucau (1) 10) Haburas Foundation – environmental education program for teachers (1) 11) Don Bosco Agricultural College Fuiloro (DBAC) - (co-funded by LOL) for soil analysis kits and GIS/GPA technology (1) 12) Escola Tecnica Agricola de Natarbora (ETAN) - (co-funded by LOL) for soil analysis kits and GIS/GPA technology (1) 4.2.1 What Worked Well Literacy and Numeracy (L&N) 23 The expansion of several different L&N organizations to meet local needs in different locations was a good use of SGP funds at the time because ministries were just formulating policies, many L&N NGOs sought funding to expand, language policies were changing from Portuguese to Tetum as the official and national language, and the country was beginning to grow. Because each of the organizations follow a similar approach (i.e., identify and train local facilitators) and use the same materials, we have grouped them for our analysis. 23 This part of the analysis supplements that of Katy Anis. 2007. Assessment of the Effectiveness of Literacy and Numeracy Programs in Timor-Leste. Bethesda: DAI. 18 All implementing agencies use the materials developed by NDNFE for basic literacy and numeracy. The first set of books was too difficult and confusing to participants, and over the course of several consultations, NDNFE developed a set of four books - Hakat Ba Oin – that begin with the alphabet and numbers and end with creating three- or four-sentence stories based on pictures. The course is designed to be implemented 10 hours a week (normally two hours a day) with participants attending consistently. Realistically, however, TA indicated that the course generally takes 10 months to complete as participants are forced to choose between competing economic activities and are often either late or absent. The use of NDNFE materials by all L&N implementing agencies means that L&N training is somewhat uniform throughout the country. The difficulties in teaching L&N in a second language by teachers who, themselves, are not fluent in the language of the literacy materials24 was analyzed in the DAI-commissioned evaluation in 2007. Having learned that the materials were too difficult for facilitators to use, NDNFE convened several consultations to develop and test new materials. On one level, the new books offer participants a solid learning system, but on the other, participants learn in a book-centered environment. Research on literacy programs in other countries has shown that adults learn more quickly when they can develop and read stories about their own lives (the Freirian model of teaching literacy). Given that facilitators may not be very well educated themselves, NDNFE believes that the new books will create the appropriate results. NDNFE, FC, TA, GFFTL, and FBM have provided basic L&N training. FC (with SGP funding), TA and GFFTL added a second course for economically active women in rural areas (TA and GFFTL did this with their own funding) for those who have completed the first course. The materials used are the “Mathematics” book developed for the primary equivalency program offered by NDNFE, supplemented with teacher aids and handouts developed by each implementing organization. The materials are varied and so is the quality of instruction and results. Skills taught generally fall into two categories: record keeping and separating consumption from business activities so as to understand profit and savings. It is not clear whether the materials for this second course are adequate or whether the course is meeting its goal of “literacy for special purposes.” The book is not designed especially for entrepreneurs and concentrates more on general numeracy skills development. An evaluation of the materials used in these classes and the outcomes of the teaching would provide needed feedback to course implementers. It appears necessary that other, more relevant materials for economically active neo-literates should be developed. Each grantee has exhibited strengths and weaknesses. The oldest (FC) was formed by school alumni that constructed several formal sector schools and then branched out into NFE. Facilitators had a lot of experience working together in one domain (formal schooling) and transferred that experience to another (NFE). Another L&N grantee was established with external funding from Australia and with funds from President Horta’s Nobel Peace Prize (TA). It is involved with education, community mobilization, book publication (first Tetum dictionary), women’s health, malaria and HIV/AIDS. A third was established as a local women’s NGO and is run entirely by very enthusiastic university student volunteers (many of who leave after finishing their studies). GFFTL has joined Rede Feto (see below) for further 24 The same problems existed when literacy was taught in Portuguese. With the development and implementation of the second set of materials in Tetum, the difficulty existed as not all Timorese speak Tetum, and several speak different dialects of Tetum. 19 capacity building.25 FBM also teaches L&N classes, but its grant was to develop a facilitator assessment tool. Its leader has close ties to the University of Iowa stemming back several years when he attended the school for a short time. The idea of expanding L&N courses is a good one for SGPs, because L&N projects respond to a local need. However, there are problems in execution. Many facilitators are not fluent in Tetum and there are very few, if any, reading materials in Tetum for neo-literates. The lack of supplementary reading materials has created confusion in whether or not literacy skills have been acquired because students have memorized the textbooks from going over them so often. When given new materials to read, participants have difficulty in decoding. Ongoing field monitoring and/or discussions of difficulties in implementing L&N programs with grantees would have revealed these difficulties earlier when SGP might have funded an organization (perhaps TA or NDNFE) to produce appropriate materials in Tetum for practice reading. While SGPs can fund projects to enhance the understanding and usage of Tetum, a more appropriate mechanism for L&N is a long-term project that addresses the range of literacy needs (materials development, facilitator training, publishing reading materials in Tetum, etc.) of different populations throughout the country. Private Sector Development Some of the “star” success stories of SGP within EG have come from grants in this category. Three supported efforts in agriculture (UH, ETADEP and ZS) and two supported the promotion of the sale of locally-made products (Rede Feto and AF). Demonstrable benefits in the form of increased income accrued to a significant number of participants. By supporting a feasibility study to UH on candlenut oil processing, SGP laid the foundations for the further development of this industry. The feasibility study recommendations included a review of possible ownership/management entities in Timor-Leste that could take over the production of candlenut oil and the amount of oil needed to fulfill the needs of Oils of Aloha (a Hawaiian cosmetics manufacturer who was contracted to purchase the oil). Acelda emerged as the ownership/management entity. It launched its production of oil to meet Oils of Aloha needs, and has expanded its candlenut sales to Indonesia (through collecting more nuts from women mobilized by CRS). After the initial study, GTZ continued funding Acelda to develop their oil press operations. Through working with DSP, Acelda has also started processing and selling coconut oil locally. This is a particularly good example of how SGP funding of a feasibility study leveraged the development of an export industry, leveraged other funding to support the development of that industry, developed the capacity to expand operations and created linkages to other projects. Its sustainability depends on its ability to identify other markets for the oil once the Oils of Aloha contract expires. ZS is also a good example of the use of an SGP grant because it illustrates how a larger project – DSP – can work together with SGP to produce good results. DSP provided technical assistance to ZS’s pre-existing horticultural market operation by helping the owner to launch a cold value chain. SGP provided the funding for the purchase of cold trucks, the construction of a cold storage facility, and the purchase of greenhouses for producers. SGP also sponsored several months’ training for producers on continuous production so that ZS 25 A best practice of GFFTL that was introduced by DAI’s PDS can be found in the development of teacher aids. GFFTL uses adult learning methods and locally-produced materials to make their classes very practical. 20 could meet its customer demands throughout the year. ZS continues to work with an ever￾increasing number of farmers who supply this marketing operation with a steady stream of products in increasing demand in Dili. This project illustrates several key success factors: expansion of existing operation to include the development of new products, the development of the foundation for long-term change, the development of value chains, and the creation of new markets for private sector operators. Both the technical assistance and financial support were needed to develop these private sector enterprises. The first grant to ETADEP funded tractor hire for local farmers, who, after the first harvest, were able to hire tractors to till their fields themselves. By supporting ETADEP in this way, farmers could plant more, harvest more, and become sustainable in their farming operations. The ETADEP grant was a good use of SGP funds because it leveraged expanded income generating activities to reduce food insecurity and create sustainable livelihoods. The promotion of the sale of local products was supported by one-off grants to AF and RFTL, but each grant was also a type of pilot to determine if such a promotion produced increased income. The support to AF funded the purchase of tables and chairs to be used in a Christmas fair promoting crafts developed by women (the fair earned participants $10,000).26 The support to RFTL was to conduct a nationwide campaign to promote local women-created products. RFTL, a women’s network organization, also had its organization participate in AF’s Christmas fair. Both grants produced successful results, according to grantees, and plans are underway to self-finance future fairs. Although one-off pilot grants, these two organizations were able to leverage sufficient income to begin a process that has long-term consequences. These examples demonstrate synergies among granting organizations, a concentrated, focused effort to produce a targeted outcome (increased income), and building local and national networks for sustainability. The two women’s promotion organizations have benefited from external advisors, and one was launched by the wife of the former president. These are also factors that promote success, as discussed in Section 2.2 above. Youth Pilot Projects Unemployment and unrest are endemic to the youth of Timor-Leste. To address the unemployment problem, SGP provided a number of grants to organizations for pilot projects to test their feasibility in income generation and their potential to be incorporated into or serve as the basis of larger, more long-term projects. Core to each project was the understanding of economics and finance, development of enterprises/small businesses, skill building to obtain employment, and/or linking youth to potential employment opportunities. Several projects in this category have experienced significant success. JA, a US-based organization that teaches enterprise education and business development skills to primary and high school students, launched its program - on the basis of a feasibility study funded by SGP - with five private, Roman Catholic schools. The program utilizes very innovative, hands-on, practical teaching techniques to promote the understanding of economics, business, and the needs of private sector development at every level. After piloting at the five private schools, JA conducted a very rigorous evaluation and learned it needed to make several adjustments: 1) train teachers on how to facilitate the JA curriculum 26 The fair was so successful that AF plans to hold one every year so the equipment provided by SGP will be used in successive years to leverage increased income for women. 21 (visitors are not allowed in schools so business people cannot teach the modules); 2) provide separate training for different primary grade levels; and 3) make certain JA materials available to non-school organizations because they are so useful. SGP funding to JA included an evaluation of this pilot program, a best practice, so that specific, contextual adjustments could be made before rolling out the program into different schools. JA is a good example of how SGP funding of a feasibility study leveraged a larger project (also SGP funded) to prepare youth for an economically active future. It also addresses local needs not only through the content of the courses provided but also by helping children and young people to think critically. Because of the success it experienced in the first five schools, considerable interest in JA has been generated by the MOE, who is considering adopting the program nationally.27 Another good use of SGP funds was in supporting ETDA in the creation of a database that links employers with the unemployed (including youth) – a best practice. ETDA’s business center also serves as a job search center. As a training center, it offers courses in management, construction, retail, administration and finance to potential jobseekers and then places the names of the participants (if unemployed) in their database. ETDA’s full-service business center has an internet café at which the unemployed can access information on jobs. Because of ETDA’s growing repertoire of courses, SGP has linked them with other grantees for training in economic literacy, basic bookkeeping, and financial management. The organization was launched by a Timorese woman who had lived in Australia during the Indonesian occupation. In Australia she created relationships which she later leveraged to establish ETDA. ETDA demonstrates a number of SGP success factors: it responds to a local need in creating relationships between employers and those seeking employment; it has leveraged support from other donors and networks to advance the mission of the organization; it has created linkages with other grantees by providing its services to their beneficiary populations; and it has laid the foundation for long-term change among the unemployed and those who take their courses. The SGP grant provided to DIT was a one-off grant, but the grant activity was embedded in an already-existing institution that sought to extend a different type of service. DIT is a technical college in Dili that offers courses in two streams – business and engineering. It has just launched a program in HRM that will train youth in all aspects of the hotel and restaurant business (including language and computer skills) and will shortly be adding an internship program. With an SGP grant, DIT held a career fair that was attended by some 4,000 people. Several employers were represented, as were different training organizations. Unemployed youth could take a series of “work readiness” classes (on how to write a CV, a letter of application, ask for a reference, and how to interview well) – a best practice. The Secretary of State for Professional Training and Labor has indicated he would like this fair to be held annually but that it should be extended over several days and include the promotion of locally-produced crafts. DIT’s activities also illustrate a number of SGP success factors: responding to local needs, creating linkages with GOTL, creating the possibility of increased incomes for participants, and learning from pilots (although a proper evaluation to determine how many people were employed as a result of attendance was not conducted after the event). The one-off funding to HF for the extension of activities of a long-standing NGO in environmental education teacher training is an example of how SGP funding can set the ball 27 The Minister has himself participated in JA training. 22 rolling for other, related activities. It is also an example of how an SGP grant can leverage greater innovation and other funding as HF is constructing an environmental education center (funded by other donors). While ultimately a project to benefit youth, the environmental education project implemented by HF taught school teachers about conservation using several books and magazines HF developed. Teachers were able to incorporate what they had learned into their own science classes. HF was established by a number of people who fought in the resistance and lived in the forests for several years. That experience was instrumental in giving HF’s founders a lot of ideas about conservation, which they used in launching HF and designing projects that link environmental conservation and agricultural production. HF has developed several best practices in working with traditional mechanisms to improve conservation. HF helps local entrepreneurs to form cooperatives for income generation and shore communities have been taught how to monitor the marine ecology so that there is no over fishing. They focus on empowering local populations to address conservation issues on their own. To demonstrate their commitment to community conservation, HF identified a community, filmed people throughout the day to document their behaviors, and then showed them the film at night. It was extremely successful in pointing out how much waste the community created. The showing of the film, according to the Director, created changes in behavior. The overall impact of this pilot has not been evaluated, although the Director reported that teachers were using the materials in their classes. The principal success factors demonstrated in the implementation of this SGP grant were embedding teaching in local, sustainable practices, trying out new components of existing activities, and creating linkages with other grantees working in the agricultural sector. Because of its strong linkages to the National Directorate of Forestry in the Ministry of Agriculture and with the Department of energy, HF also illustrates another success factor: working closely with GOTL. Another successful SGP-funded pilot has created a cadre of skilled carpenters in Baucau. PBM, owned by the Diocese of Baucau and led by a Portuguese manager, identifies youth who have had some experience in carpentry or who have graduated from technical school and gives them OJT in the furniture factory (three-twelve months’ duration). Of the 20 who have been trained, nine have been retained by the factory in full-time employment, two have established their own carpentry business, and others have found piece work. The two trainees interviewed are very satisfied with what they have learned as they now know how to use the latest equipment and are earning a salary. This project was embedded in local operations and tried out a new component of an existing operation as well as created employment opportunities for participants. This Diocese operation will continue and grow as the security issues of the country are resolved and people invest more in housing. SGP also funded a pilot internship program to create job opportunities for young people who had participated in HRM training. The internship/OJT programs have not been in operation for a very long time and will be evaluated separately before SGP closes. However, a best practice (on the basis of SGP recommendations) was established in the internship project reviewed (CFJPII), the six-month training and internship placement of CTID, and in one not reviewed (a three-year training program in Venilale): trainees keep a Skills Logbook in which they document all the skills they acquire in classroom and OJT activities. The instructor reviews what has been written and makes adjustments in her curriculum to reflect the local context. This is the first such program in Timor-Leste that uses this approach. The success factor highlighted in this practice is the ongoing evaluation of what is being learned so as to make program adjustments. Other success factors include creating job opportunities for youth, expanding the activities of local institutions, and laying the foundations for future 23 income-generating activities (CFJPII is now establishing a bakery to provide training and job experience to participants). Although not evaluated as it has just been launched, the development of six Resource Organizations through the use of the SGP mechanism is also a good idea conceptually. ROs will be positioned to lead various activities in their respective sectors, be of service in broader, longer-term projects, be more efficient and effective in their own domains of activity, and be able to respond to different needs as they arise. Given the limited managerial capacity of many grantees, this institutional strengthening activity – conducted over a limited time period – will help to build the capacity of participating organizations. Because capacity building is such a critical issue in Timor-Leste, we will make a broader recommendation on that subject in Section 6. 4.2.2 What Did Not Work So Well Microfinance The operation of MFIs has been problematic since they were launched. In this case, the problem was the enabling environment: Small and micro-business owners have been particularly badly affected as many have been displaced from their homes and/or place of business. Transportation of goods in and out of the capital, the country’s only significant market, became exorbitant for most individuals due to increased risk to transporters and fears of traveling. This has affected secondary markets, as many farmers rely on selling their goods to Dili to earn income. The consequences for microfinance providers have been quite dire, with the third largest provider, Opportunity Timor-Leste, closing operations. As [MFIs] operate in the markets most affected by the conflict . . ., the displacement of clients and stoppage of economic activity has resulted in very high arrears and resignation rates.28 The political and socio-economic environment in which many MFIs were launched created many challenges. In addition to the enabling environment issues, MFIs must work with clients who do not save money regularly and who do not understand market forces in the development of their businesses. The two MFIs that survived the 2006 crisis and are still operating are MR and TRM. As a member of the Grameen family, MR (headed by an expatriate) can access external funds; TRM cannot because it is not situated in an international network and because it has not achieved operational sustainability. TRM was once a part of CRS and as a stand-alone operation, it has reduced its operating costs from $8,500 to $6,800 per month. Each organization has its own geographical domain of clients, although when both of them expanded there was some overlap in selected districts. The work of MFIs is very hazardous. In their efforts to reach clients operating microenterprises and small businesses, credit officers must travel to very remote locations. Most use motorcycles to reach their clients (which is a disincentive for women). Because there are no financial facilities in most locations where the MFIs work, credit officers must travel with large amounts of cash either to deliver loans to solidarity groups or as a result of making collections at group meetings. In Timor-Leste’s insecure environment, this practice puts credit officers at risk. 28 Jason Belanger. 2007. Final Report for DAI033 Extension. Institutional Strengthening of Tuba Rai Metin to Provide Microcredit Services for Women Entrepreneurs. Dili: CRS, pp. 3-4. 24 In principle, supporting the expansion of MFIs is a good idea because they satisfy the need for small amounts of capital and enhance the growth of microenterprises. However, the lack of legal controls and enhancing policy environment, as well as the political instability of the country, makes investing in these operations at this time very risky. The grant to AMFITL, the network membership organization established to provide capacity building services to its members, enabled it to provide training to its members, most of whom closed down after the 2006 crisis.29 The grants to MR enabled it to expand operations and to buy a database system (which proved useless as it was intended for use at branch offices, whereas MR financial accounts are centralized). The grants to TRM allowed it to expand operations and covered operational costs. Both organizations continue to struggle for capital for onlending, and local banks (which are actually international) do not provide this type of loan to MFIs. 30 SGP support of MFIs should be weighed very carefully in light of the changing policy and security environment. There is a high local demand for MFI services which MR and TRM are trying to meet, but they will be unable to do so effectively without changes in government and private sector policy Youth Pilot Projects Several of these projects are in the range of faulty-to-good conceptually, but most suffered significantly in execution. Rotaract is an example of both of these shortcomings, although it received three SGP grants. With the first, Rotaract sent its director to Malaysia to attend a conference on economic literacy and small business development. With the second grant, he worked with some consultants to adapt the materials he had received at the conference into a training curriculum to be implemented at high schools in Dili (a best practice for developing curriculum materials). The members of Rotaract, all university students (largely majoring in English), were provided a TOT on how to deliver the program. The third grant had the students deliver the same workshops to street vendors, but this was problematic because facilitators did not have any real-life experience in this area. No evaluation was conducted either in the high schools or with the street vendors to determine how useful the project was and what, if any, impacts resulted. This pilot suffered from a number of design challenges: 1) reliance on students to implement (students leave when they finish their university education and accept full-time jobs); 2) reliance on one person’s (the director’s) insights and experience in driving the whole project (he is also graduating and will step down as director); 3) expansion into an unfamiliar domain in which students had no experience; and 4) no professional oversight exercised over the course of the pilot . The director had a vision to expand further either into high schools in the districts or to other economically active populations, but acknowledged this would be difficult given the high turnover in student facilitators and the need for cash for transportation 29 There were also concerns about AMFITL management because the expatriate leader of MR was also the leader of AMFITL. As she does not live in Timor-Leste continuously, the demands of the job at MR made it very difficult to also manage AMFITL. Consequently, she hired a manager and took the position of secretary so that there would be a continuous presence in AMFITL’s leadership to address the needs of its members. 30 For a further analysis of MFIs, see Lene Hansen and Novanto Agus. 2005. Financial Services Sector Assessment and Seminar/Workshop on “The Millennium Development Goals and the Eradication of Poverty – the Role of Microfinance in Timor-Leste”. Final Report. Dili, Timor-Leste: Association of Microfinance Institutions in Timor-Leste (AMFITIL) (January 10); UNDP and UNCDF. 2008. Joint Programme. INFUSE – Inclusive Finance for the Under-Served Economy (2008 – 2012). Proposal submitted to the Ministry of Economy and Development. Dili: April 2008. 25 and meals. The project may have done well in imparting lessons to high school students who, themselves, were not yet economically active, but we do not know this because no evaluation of that part of the pilot was conducted. No evaluation of the street vendor pilot occurred either. Because of the design and implementation challenges, the different facets of this project discussed should serve as a warning to future SGPs. Had this project been embedded or linked to the adult Rotary Club, a different outcome might have been achieved. ETADEP’s youth pilot also suffered from both shortcomings. The pilot on horticultural development implemented by ETADEP targeted young people with primary school, high school and technical school diplomas. ETADEP sought to train youth on innovative techniques in horticultural production to stem the tide of urban migration by providing the opportunity to generate an income rurally. Although enthusiastic at first, many youth participants dropped out because they believed they had job opportunities elsewhere that would take them out of a sector in which their mainly uneducated parents worked.31 Only primary school graduates remained in the project because they were not as convinced that they would find jobs elsewhere. ETADEP “salvaged” the project by bringing older, more experienced farmers into the groups.32 This project, suggested by DAI (without conducting a rapid appraisal or simple feasibility study), suffered from a failure to understand the needs of the target audience and made erroneous assumptions about high school educated youth (i.e., that they would be willing to participate in agricultural production as an income-generating or livelihood activity). No evaluation was conducted on this project, although ETADEP’s and DAI’s monitoring allowed for adaptation during the course of the grant so that participants could experience an income from their work. The two grants to TYEI were also problematic. The goal of the project was to provide training on modern agricultural techniques and marketing to youth in Liquica and Baucau to improve crop production and generate income. One of the delivery vehicles used was a district-to-district exchange program to expose youth to different crop production techniques. While those who participated were enthusiastic at first – especially about the exchange experience - the groups failed to thrive once they returned to their homes. The Director informed us that many youth left for employment elsewhere, either in temporary $2/day cash￾for-work programs or more long-term opportunities. The excitement generated by going to live in another location for a short period took youth out of situations that might have appeared hopeless. Although they received technical and marketing training, youth returned to their homes where local constraints did not always allow them to put into practice what they had learned. The number of groups formed during grant implementation declined and are in full operation only in Baucau. The TYEI grants are an example of a good idea conceptually but had difficulties in design and in understanding the needs of rural youth. That the pilot was not evaluated means it is unclear as to what has the potential to be successful in a larger endeavor and what would need to be revised. The pilot HRM training and internship projects reviewed are still being implemented (CTID and CFJPII). While internships are generally a very good idea because they provide youth with work experience, the two projects reviewed were experiencing execution problems. Although agreements were reached about the range of experiences an intern should have (front desk, food and beverage, and housekeeping), we found evidence of hotels focusing only on one – housekeeping. The six months of combined classroom and OJT is appropriate 31 Imparted by Gilman Dos Santos, Director, ETADEP, in an interview held April 14, 2008, Dili, Timor-Leste. 32 Ibid. 26 for young people to have work experience and acquire entry-level jobs, but if there is a desire to train Timorese young people for more management-level positions, the six months of training is insufficient. The evaluation of the HRM projects to take place before SGP close￾out will reveal far more than the team was able to discern. 4.2.3 Recommendations Overall, the SGP mechanism produced some excellent results and gave different organizations an opportunity to be innovative in addressing different issues. Each of the projects had excellent conceptual elements to help solve the economic growth problems facing Timor-Leste. Assuming that Timor-Leste is entering a more stable political and economic phase, L&N and private sector development should both be considered for more longer term, comprehensive projects that address a range of needs. SGP support could then be utilized, as it was in the current case, to pilot new ideas, provide certain types of training and/or equipment, create linkages between different institutions, and create income￾generation activities. For instance, in an SGP Working Group meeting, the substantive issue of donors developing a more comprehensive (non-SGP) L&N project should be raised. In addition to expansion of L&N activities throughout the country, such a long-term project could address the further development of Tetum language reading materials, working with media organizations to create radio literacy programs, linking L&N projects to private sector development (e.g., in publishing Tetum reading materials for neo-literates) and other strategies. If such an L&N project is not developed, then continued support for discrete L&N projects could be continued, but with the provisos noted below. Private sector development, already the purpose of one USAID-funded longer-term project (DSP), requires a range of initiatives. For instance, the pilot creation of a district-based business “incubator” to meet the advisory and learning needs of microentrepreneurs and business people at all levels could be supported by DSP or other entrepreneurial support project. The incubator should be a place where ideas of all types are discussed, where business plans can be developed, where entrepreneurs can learn how to conduct a simplified value chain analysis, and where they can be trained on financial management. Because this pilot is in the nature of launching and building a new institution, which can take a considerable amount of time, it would not be appropriate for an SGP mechanism. Whereas SGP provided complementary support to DSP in funding pilots and feasibility studies or purchasing equipment, no formal evaluation of these projects has taken place. However, anecdotal information on lessons learned and information on increased incomes generated point to the largely untapped potential in cultivating entrepreneurial development. A longer￾term project should examine these lessons and construct a project that will address the needs of entrepreneurs, including economic literacy, creating markets, wholesaling and retailing, investment of income generated, and the like. If such a project is not developed then, as in L&N, SGPs could continue to support discrete projects, also with the provisos noted below. 1. Continue to fund the expansion of L&N projects, especially if the time frame for implementation is at least a year to allow for the development of greater Tetum fluency by facilitators and the further development of more innovative adult teaching methodologies that take into account the difficulties of acquiring literacy skills in a second language. 27 2. Continue to fund innovative linkages between L&N and private sector development in the form of economic literacy/entrepreneurship education, but also fund the development of appropriate materials and a facilitator TOT in their use. Provide consultancy services to guide the development of these materials so that they address the needs of neo￾literates. Evaluate the use and implementation of these materials and their impact in terms of equipping economically active participants with skills needed to improve their businesses. 3. Consider funding intra- and cross-sector linkages to utilize and/or adapt materials already developed in each sector to expand their use (e.g., JA materials adapted for economically active adults). 4. Do not provide further funding to MFIs for expansion until the enabling environment has improved and MFIs have secured commercial funding to on-lend to their clients. 5. Continue to support feasibility studies, pilots (including funding and time for the evaluation of each pilot), purchase of equipment, provision of training, and similar endeavors to promote long-term private sector development, but link these projects to more longer-term development endeavors so that the projects can be nurtured and leaders can be mentored. 6. Continue to support aspects of broader, longer term projects in private sector development that these projects are not equipped to provide. By so doing, require accountability mechanisms in the larger project for the creation of impact (i.e., provide whatever services are necessary to leverage greater impact of the SGP grant). 7. Continue to support opportunities to promote the sale of local goods, e.g., craft fairs. (This could also be included in a more longer-term project in private sector development.) 8. Continue to support job/career fairs that link potential employers to the unemployed; consider the feasibility of funding the implementation of fairs in other districts. 9. Continue to support pilot and capacity building programs that would position youth to be more competitive in the employment market, monitor such programs closely, and evaluate them to determine if they are appropriate for scaling up in a larger project. 10. Continue to support pilot HRM and other internships to provide work experience for participants, but closely monitor and evaluate each pilot to determine the ability of interns to find employment afterwards. 11. Support short-term and/or pilot collaborative endeavors between GOTL and NGOs in economic development activities that spearhead food security, export promotion, agricultural innovation, appropriate technology, cooperative formation, and other similar pursuits. Evaluate these rigorously to determine if they should be leading elements in a broader, longer-term project. 4.3 Improved Health of Timorese People The following analysis identifies which interventions are more appropriate for an SGP mechanism and which are best funded through other mechanisms. The majority of the grants in health were for larger interventions that the MOH did not have the resources to fund. Most 28 of the grants in this portfolio answer the summative question affirmatively: “yes, this was a good idea;” problems were found largely in implementation. The evaluation of the grants under health Special Objective 3 (redefined after 2006 changes) sought to investigate the effectiveness of the SGP grant mechanism for the projects that were most likely to provide programmatic insights - challenges, constraints and achievements – and the appropriateness of the SGP mechanism. As MOH received grants to meet a range of needs, a number of different units within the MOH were interviewed. As most MOH grants were implemented in multiple districts or sites through the DHS, the district health officials were also interviewed. Of the approximately 30 grants awarded in the health and nutrition area, 11 grantees who had been awarded 21 grants were interviewed in Dili, Liquica, Baucau and Managuto as follows: Workforce Development • World Health Organisation (WHO) – Clinical Nurse Training (2) • Health Associates International (HAI) – Conference Attendance (1) • Associaciao dos Medicos Timor-Leste (AMTL) – Professional Association (1) Program Support • CARE International – Nutrition and Humanitarian Assistance (IDP) (5) • CRS – ITNs and Humanitarian Assistance (IDP) (3) • Ministry of Health (MOH) – Large population based interventions (3) • Alola Foundation (AF) – Pyschosocial healing and Emergency Relief (IDP) (2) • Bairo Pite Clinic (BPC) – Medical Support (IDP) (1) • WHO – Population based intervention (1) • Pastoral de Crisas (PC) – mother support group (nutrition strategy) (1) • Grupo Supporta Inan (GSI) – mother support group (nutrition strategy) (1) 4.3.1 What Worked Well SGP has been used very effectively to leverage other donors or community support for greater impact. Impact here can be measured by geographical coverage, immunization rates, access and penetration into target groups. Flexibility/responsiveness of the SGP mechanism should be measured by timeliness and in the different ways the SGP was used – leveraging for greater impact, strengthening the services logic, innovative programs, and emergency health responses. For each of these examples the SGP model was used effectively: it filled funding gaps, and enabled inputs to be funded and services to be delivered for both emergent and emergency needs. In a post-conflict situation such as Timor-Leste where there are pervasive capacity issues in politically tenuous situations the inability of services to reach populations only exacerbates tensions. SGP’s are very useful as they can respond more quickly than other mechanisms. Determining the more effective use of SGP between 1) the investment of SGP grants in MOH programs for a range of large health campaigns at the beginning of the period compared to 2) the focused use of SGP sitting alongside the USAID partners is arbitrary. SGP investment in large multi-donor funded campaigns for a new SpO early in the funding period ensured highly visible USAID support for a nascent government agency in Timor-Leste. This changed as the USAID partners came online with their services, which facilitated a change of direction and a more efficient conduit for the USAID/SGP. This was because funding was then able to better leverage their programs, further strengthening USAID support and mutually reinforcing visibility in the health sector. 29 Program Support Leveraging for Greater Impact The SGP grant awarded to AMTL33 for its first Congress was co-funded by the MOH. The grant is a good example of an award made not only on the basis of a good idea, but also on the basis of the leadership, human resource capacity and multi-donor support. The 2005 AMTL Congress focused on how AMTL could support GOTL initiatives34 and provided a venue to elect an executive body. The AMTL President noted that the association has an MOU with MOH, is now a registered association with an office, has received other support from international donors – UNFPA, WHO, UNDP, UNICEF – and has developed a number of strategies to become more self-sustaining. The association has progressed slowly from the initial Congress mainly due to resource constraints. AMTL, however, has achieved a number of the tasks it set itself at the first Congress, including collaboration with medical associations from other countries – Singapore, Indonesia, Fiji, PNG and Australia. AMTL plans to hold the next Congress in August 2008 to discuss plans to establish a Medical Board and Medical Council to develop codes of practitioner conduct for all doctors practicing in Timor-Leste. This grant led to a more sustainable impact than the original input and investment from SGP because AMTL demonstrated the leadership, human resource capacity and sector support leveraged off multi-donor support needed to continue AMTL activities. The majority of SGP program support grants for health were funded for national population￾based interventions for major diseases such as Dengue, polio, lymphatic filariasis, malaria, and malnutrition. These SGP grants worked well because they were responsive to outbreaks of disease and leveraged support for USAID partners and co-funders. In addition, SGP program support grants supported CBOs linked into existing structures that effectively delivered health services to target populations and included support for innovative medical practices such as mental health services crucial to serving a population in a conflict situation. Finally, the flexibility of the SGP mechanism allowed for a quick response to emergency needs of IDPs during the 2006 crisis. A number of SGP grants were awarded to respond to the health needs of the country. SGP often provided additional inputs to a number of health interventions for a nascent MOH with little capacity and financial resources. 2005 Dengue Outbreak: Two grants were awarded to control this disease. The first grant was awarded in early 2005 to WHO to support the immediate vector control measures in Dili (co-funded by UNFPA). As the outbreak escalated, producing a significant number of deaths in children under 15, it coincided with the second SGP grant awarded to MOH for community mobilization in seven districts - Bobonaro, Liquica, Baucau, Lautem, Manatuto, Covalima and Ermera. Interviews with Environmental Health officials in the MOH confirmed that at the time of the outbreak the operational capacity, resources and budget within MOH was low. SGP grants were used to purchase inputs, repair spraying and fogging 33 AMTL was established in 1999. During Indonesian times it was known as Ikatan Dokter Timor-Timur. In 1999 a new committee – the East Timor Health Professional Working Group - was formed to directly deal with the emergency situation in the country. In 2002 AMTL elected an executive committee and has actively participated in health sector development and policy formation in the National Health System. 34 At the time, this included raising awareness about children’s health issues in support of GOTL’s commitment to reducing child mortality. 30 equipment, spraying infected areas and printing brochures. MOH now has an annual cycle for Dengue containment.35 Polio: The SGP grant for the polio intervention in 2005 supplemented the MOH social mobilization budget and substantial support from UNICEF. The grant funded training for facilitators and drug administrators. The evaluation of the campaign showed coverage achieved to be between 82-102%.36 The polio campaign was implemented in two rounds between August-September 2005. Polio drops are now part of the immunization schedule for children under 5 as a part of the basic services provided by MOH. For this campaign, the main problem in execution was logistics. A number of deficiencies in the grant budget delayed training in two sub-districts (Dom Alexio and Vera Cruz). MOH had only one vehicle to cover seven departments, which proved problematic when supervising training at different venues. USAID and WHO vehicles were utilized when visiting sub-districts. There was also confusion about trainee numbers and level of remuneration in a few sub-districts. The issues in implementing this campaign identified a number of lessons learned that were then applied to the roll-out to other districts. Lymphatic Filariasis:37 Another SGP grant was awarded to fund the training component of the LF campaign38 in Dili and Liquica. The program was a population-based drug administration campaign. The SGP grant funded a cascaded training for 1,110 drug distributors39 and facilitators (685 men, 425 women). In 13 courses conducted over four days in Dili, trainers from the NCHET were initially trained by WHO staff and were then responsible for training the facilitators. The facilitators then trained the distributors in each of the nine sub-districts over two additional days. The training made possible by the SGP grant was successful in that a large number of people were trained over a short period of time. In this campaign there were not enough vehicles available which meant vaccine delivery was late. Getting information out to remote communities was problematic, which had an impact on attendance at the immunization post. There was also some confusion in the first round about locations because they were not clearly identified (the production of banners was not listed in the budget). However, these constraints were quickly overcome and the second round ran smoothly. Malaria: This grant, awarded to CRS, was made in partnership with the MOH and TAIS. CRS provided logistical and procurement support for a campaign to increase the use of ITNs 35 At the start of each rainy season the DHOs hold community forums and request local leaders to mobilize their communities to support clean-up campaigns in common areas. Health information is disseminated at these meetings. Technical assistance is now provided by the Australian Government. 36 Some of the district results show more than 100% coverage against the target population. This was because it was MOH policy to vaccinate children even if they were over 5. Many parents took advantage of this opportunity to have all their children vaccinated, no matter the age. 37 These two campaigns are part of the National Immunization Strategy, an integral component of the Basic Package of Services of the MOH. The Package is implemented through the DHS, whose emphasis is on strengthening routine immunization services. Their work is based on: 1) creating equity of access to interventions and services; 2) promoting health through behaviour change; and 3) strengthening management of service and interventions. 38 The intestinal parasite is one of the leading causes of disability in the world. A survey conducted in 2002 identified that this disease was prevalent in every District in Timor-Leste and that 80-90% of children and 8- 10% of adults had LF. 39 The role of drug distributors and facilitators was to conduct the Mass Drug Administration in the districts of Liquica and Dili. 31 by pregnant women and young children to reduce the deaths caused by malaria in Baucau. The grant included a behavior change program to stimulate demand for ITNs and improve knowledge of the determinants of malaria. This was the first malaria prevention campaign to focus on children under 5. By the completion of the project 14,735 ITNs had been distributed in six sub-districts. In follow-up visits to communities to pick up signed distribution lists, many additional requests were submitted for potential beneficiaries who had been excluded from the list. CRS responded to every request by establishing two distribution points resulting in coverage of up to 90% of the population. CRS is now a partner in the IMCI project co￾ordinated by TAIS. Nutrition: Three SGP grants were awarded to CARE in this category. The first grant funded a rapid nutrition assessment of children 6-59 months old in Covalima in 2005. CARE had carried out similar studies in Covalima previously and nutritional surveys in Bobonaro and Liquica. CARE worked with the DHS in Covalima. CARE made an in-kind contribution to this effort by supporting the services of an international nutritionist. The assessment, conducted by a team of at least 20, found that a majority of target group children in Covalima were moderately wasted40 and more than 50% were stunted41 and underweight. Contributing factors were food insecurity, ineffective food utilization, poor health care and poor sanitation. The first grant was followed up with a second to support CARE’s growth monitoring and targeted supplementary feeding program. This was a significant grant by SGP for a large multi-donor strategy that included AusAID and ECHO to the value of nearly US$2 million. CARE had previous experience in the area of nutrition and this grant provided an extension of CARE’s existing nutritional and supplementary feeding programs.42 The third grant provided technical and logistical support to the MOH Supplementary Feeding Program, which included provision of food for vulnerable populations and health and nutrition education. This initiative was co-funded by AusAID and focused on training CHC staff on commodity management and appropriate food storage, conducting growth monitoring to ensure national protocols are understood and applied, logistical support to remote locations, and upgrading CHC facilities to enable proper food storage. CARE is now an active member of the Nutrition Working Group at MOH. The longer term benefit of this funding is that the interventions funded are now structured into routine services of MOH – thereby ensuring more sustained impact. The grants also illustrate how relationships can be strengthened between MOH and NGOs, namely CARE and CRS. These projects that support USAID partners demonstrate how SGP funding can be embedded in larger partner program grants to achieve even greater impact and longer term sustainability and a deepening of the working relationships as they are a part of a of the larger IMCI project. To be even more effective, based on the lessons learned from these examples, a comprehensive economic development program with an emphasis on integrated nutritional support services using a combination of SGP and bilateral funding could be developed between MOH and CARE. This could be undertaken in districts that are not in crisis but are ready for a longer term development program. An SGP mechanism could fund innovative programs like perma-culture or build new relationships in new locations or with new CBO 40 Insufficient weight for age and size 41 Low height for weight 42 These programs included anthropometric measurements to assess the nutritional status of children and pregnant and lactating mothers in 4 sub-districts in Covalima. The children identified as malnourished received supplemental feeding and parents were provided with health education support. 32 partners. In each situation – regardless of whether one uses an SGP or other funding mechanism - the key design elements should link income generation to food security, effective food utilization, better health care and sanitation. Strengthening CBO Health Services Logic Two SGP grants are examples of how the flexibility of SGP funding can fit in with USAID bilateral partner projects to consolidate the MOH/TAIS->CSO services logic to CBOs. The benefit of providing such a responsive mechanism is not only that it brings the IMCI strategy closer to the beneficiary level, but that it also provides an important function of reinforcing social legitimacy of the CBO partners and thereby strengthens the services logic of future campaigns. In the first example, an SGP grant was awarded to PDC, a local church-based organization, to provide training materials and supplies to help PDC deliver a nutrition education training program to pregnant women and new mothers in two sucos in Laleia, Manatuto. PDC also promotes breastfeeding and the production of healthy, locally available food to supplement children’s diets. The training was delivered in coordination with the MOH DHS and the TAIS project. PDC’s health education materials were developed in coordination with the MOH’s nutrition education program. PDC training is delivered in sucos where the MOH and other health partners are not working. The two lead trainers at PDC have been delivering similar health education programs since 2001. PDC also has 33 active volunteer trainers, and much of the work is conducted by community volunteers (who pay for their own support activities). The SGP grant paid for basic inputs – tables, chairs, stationery, cooking equipment, balances and reproduction of training modules - to facilitate nutrition education activities. The PDC coordinator in Laleia shared that the group enjoys a high level of social legitimacy and said that, according to their experience, information must be imparted to the community several times if there are high levels of illiteracy and fewer times if the audience is literate. This means that transaction costs for these groups are high but necessary. In the second example, an SGP grant was awarded to GSI in Baucau, a local women’s group, to deliver nutrition education to pregnant women and new mothers in the six sucos of Baucau. GSI has received support from AF and MOH. GSI provides ongoing support to pregnant women and new mothers in Baucau Hospital and through door-to-door monitoring of homes in Baucau town and surrounding rural areas. GSI has built a good relationship with the DHS, particularly the midwives, and encourages pregnant women to have their children at the hospital rather than at home. GSI also provides nutrition information and delivers maternity packs to new mothers.43 GSI’s health education programs use training materials developed by the MOH’s nutrition education program, HAI and UNICEF. All activities are coordinated with the DHS in Baucau, AF and local leaders. The grant was used to pay facilitator fees, transportation costs, training materials and refreshments. The ongoing impact of these two grants could be enhanced by funding each organization to undertake follow-up activities to determine the effectiveness of the training in producing longer-term behavior change. Expansion or scale up could also be considered to include a wider integrated development project that reduces the risk factors for malnutrition. Funding additional components that add value to the service offering of CBOs – as long as they are in their capacity to absorb – would also enhance longer term impact. 43 The interview also identified that the motivation to produce the required change in behaviour was greatly encouraged by inducement of the receipt of the maternity kits from AF. 33 Innovative Programs An SGP grant was awarded to AF to conduct psychosocial counselling sessions and provide assistance to the widows of the F-FDTL and PNTL officials during the 2006 crisis. The grant highlights the often undervalued area of psychosocial healing from trauma. The project was developed in collaboration with Fokpers and PRADET (members of the Rede Feto network). AF contributed half the funding for this project from their own funds and gained additional support from IMOG and the MILK Fund from Singapore. AF has considerable experience in designing, implementing and evaluating programs. The project highlights the collective impact of structured group development as AF and its CSO partners continue to support and meet the changing needs of the widows. This is achieved through its close working relationships with other members of Rede Feto and relevant government agencies. Although the grant has had a greater positive impact than the funding paid for, AF did not ask for additional funding to provide follow-up support as AF staff indicated the compliance burden for such a small grant outweighed its benefits. However, the grant does highlight the flexibility of SGP in funding innovative projects. The ongoing impact of this grant could have been enhanced by including an evaluation to determine the effectiveness of the training in making longer-term behaviour change. An interview with AF revealed that they did conduct a small evaluation with its own resources, which led to development of other programs. Emergency Health Responses All six of the SGP grants in this category were awarded to respond to IDP needs as a result of the 2006 crisis. They were awarded over a period of 7 months from May – December 2006, as follows: • May: CARE provided supplementary feeding for IDPs in Dili and Liquica, specifically for pregnant and lactating mothers and children under 5. CARE also provided the support to distribute the supplemental food, personal hygiene materials and drinking water. The distribution was expected to happen only once as the assumption was that most IDP’s would return home within one week. • May: CRS for hygiene kits to 5,000 families in IDP camps in Dili, Aileu, Emera, Liquica and Baucau. The kits were to raise awareness about good hygiene practice to improve sanitary conditions in the future. • May: CARE to provide emergency services to IDP’s. • June: Bairo Pite Clinic Mobile Medical Teams to extend and maintain reach into the other districts. • June: CRS to provide 10,000 ITN’s for IDP’s • July: Alola Foundation for the provision and distribution of re-useable feminine hygiene products and other sanitary products to women in IDP camps. The ability of SGP to respond immediately to the emergency needs of IDPs reinforces the importance of the flexibility of the mechanism. The mosaic of support offerings identified during the IDP situation highlight the adaptive strength in the NGO sector - in particular the INGOs - who have the expertise to mobilize quickly and to move from humanitarian assistance to development programs in a seamless manner. For the INGOs CARE and CRS, who also had site liaison in addition to support and management roles for some camps, the challenges changed in a very fluid environment where peace building, reintegration activities and site security in a highly tenuous situation were needed. This seamlessness provides a 34 much needed depth and a number of links for local NGO’s and CBO’s to work alongside. Interviews with ex-IDP camp coordinators identified the need for these interventions to be more sustainable than single, one-off assistance. 4.3.2 What Did Not Work So Well Workforce Development In a country with such new systems and a multitude of different interventions to build capacity quickly, there is a need to ensure consolidation and sustainability of workforce development schemes. The use of the SGP mechanism to support the capacity development needs of the health sector over the longer-term becomes problematic only if the assumptions of the program design change. The problem is that the clinical nurse training would have been more appropriately funded under a longer-term project as this would have necessitated clear policy guidelines. Locking in assumptions and ensuring sector support for a strategy by ensuring longer-term funding reduces uncertainty and strengthens the resolve of the sector actors to play out a strategy. This should not be confused with questioning the usefulness of a mechanism. What is of paramount importance is that the policy framework and guiding principles are clearly understood, that there is sufficient sector support, and that the guiding principles do not change, i.e., capacity development of existing resources vs. capacity gap￾filling. In 2005 two SGP grants were awarded to support clinical nurse training.44 The initial strategy was premised on good levels of collaboration between WHO and MOH’s district health structure. WHO, the recipient of the two grants in 2005 (for competency-based training and follow-up support), provided technical assistance for a program implemented by the MOH through ICS. The objective of the training was to up-skill 71 nurses (17 females and 54 males) in the CHCs to provide comprehensive basic health services. The first grant supported a six-month competency-based training program in clinical skills in prevention and curative services. The second grant funded a comprehensive evaluation and supervision program conducted by the DMOs, a national workshop to review the future role of clinical nurses, and an evaluation of the training program itself.45 As a workforce development strategy it reinforced the importance for stakeholder commitment and good communication channels. Although the idea for clinical nurse training was sound, the problems occurred when the second grant was implemented. The level of professional support required of DMOs was not specified and they were not adequately informed about their supervisory roles. Training difficulties were exacerbated by language barriers, poor management, and lack of co￾ordination between the different levels of MOH. A number of changes in the sector also contributed to a dilution of the perceived benefits of this program: • The change in national strategy from capacity building of nurses to increasing the number of doctors • The establishment of the new medical school and increased capacity to train more Timorese medical students 44 After 1999, the exodus of the largely Indonesian health professionals and the destruction of many of the health facilities, the country was left with a much weakened health workforce. Independent clinics supported by the church or Cafe Cooperative Timor health clinics were sought after for basic health care.   45 The sustainability and quality technology transfer was planned to be structured into the program by training the best clinical nurses as District Clinical Facilitators that could then act as supervisors to other clinical nurses and thereby enhance the capacity of other health workers after the training. 35 • The introduction of a large number of Cuban doctors to make up for the lack of Timorese doctors Interviews with Senior WHO officials indicated the need for continuous supervision for nursing staff. DHOs in Liquica and Baucau knew of the training and that, in their opinion, the skills of the clinical nurses in their areas had been improved. The continued impact of this training will be in evidence for the long term. Unfortunately, the SGP mechanism cannot mitigate against policy changes, problems in stakeholder communications, and the more systemic divisive issue of language. Implementation Challenges in Program Support A number of the interventions experienced implementation challenges. Although the idea to fund informational and immunization campaigns was sound, the projects faced a number of challenges in logistics. These issues could have been easily overcome by ensuring the budget had been developed comprehensively by line item and the logistical support was available to reduce implementation constraints. The regularity in which these issues appeared in each of the larger health campaigns signals a weakness in program design, budgeting and logistical capacity in MOH. As these systems in MOH are still being strengthened, it would be sensible for any donor wishing to fund health programs - regardless of the funding mechanisms - to not assume MOH has taken this into consideration when the budgets are submitted. 4.3.3 Recommendations The following recommendations can be used when designing health projects best funded under an SGP mechanism: 1. Continue to support professional associations if the grants are provided for discrete outputs and can be leveraged off other contributions. Look for longer term grant impact in the cohesiveness of the association, the leadership, human resource capacity and sector support. 2. Continue to provide funding for inputs, e.g., training or materials, to support larger health intervention campaigns particularly when leveraging off other donors for greater impact and when there are good central-local connections. 3. Continue to support opportunities to strengthen the service logic of CBOs (MOH- >INGO’s->NGOs->CBOs->community) by funding inputs at different points on the logic chain, e.g., CRS and ITNs or CBO and training inputs. This will continue to provide support to CBOs to enable the high levels of personal motivation the coordinators and their volunteers contribute to make the health strategies maintain effective outreach. 4. Continue to support innovative health service projects that meet a need or can be scaled up for future consideration. Look for projects where the lead agency has good evaluation capacity and networks for referrals or specifically request these structures be developed as a part of the grant. 5. Do not fund projects through an SGP that have long time horizons and require sector support within unclear policy frameworks. These are best funded by other longer term mechanisms. 36 6. Consider the opportunities for replication and/or scale up of health education programs that are within the absorptive capacity of the NGOs but will ensure ongoing commitment and regular service delivery. 7. Consider funding more integrated projects that support good health outcomes, e.g., perma-culture and income generation activities that link food security, effective food utilization, better health care and sanitation. 8. Consider including innovative media components when funding NGOs to support better information transfer that will lead to long-term behavior change. 4.4 Summary and Conclusions: Addressing the SOW Questions46 The evidence for the efficacy of SGP as a funding mechanism at the specific time in Timor￾Leste’s history indicates that there are many more cases that worked well than those that did not. The strategy utilized by DAI in selecting grantees produced many success stories in terms of projects meeting their goals (normative) than those whose ideas were inappropriate or whose implementation was entirely off the mark. The team agreed that SGP is a good mechanism where 1) there is an emergency/ humanitarian crisis that needs a funding source to meet needs quickly, 2) support for a project is embedded in other, larger projects for sustainability, and 3) a good idea should be piloted. We also agreed that as Timor-Leste enters a new phase of development, more longer-term, comprehensive projects should be designed in establishing new institutions, L&N, private sector development and workforce development (and capacity development in all of these endeavors) – all areas that require long-term investment. That being said, however, because of the more rigid nature of budgets for long-term projects, a certain amount of funding should be made available to respond to humanitarian needs, to pilot and evaluate innovative ideas, and to adapt to changing circumstances and the policy environment. Going back to the points we raised in Section 2, Background, in the end we have to acknowledge that Timor-Leste is extremely fragile, that humanitarian needs continue, that larger projects USAID funds are definitely in order because of the challenges GOTL faces in state and nation building, and that reconstruction and the expansion of projects that work in DG, EG and health are all very much a part of Timor-Leste’s landscape. Did the SGP project accomplish what it set out to do? We believe it did. Was it the right mechanism at the time? We believe it was, for all the reasons we set out in this report. Could it have done better? In hindsight, yes, and the recommendations for improvement can be found in each of our subsections above and (for future SGPs) in Section 6. Do we recommend to USAID that another such fund be established? No, we do not. SGP was a good mechanism to help people overcome a part of their history that required quick funding responses and a vision that began with reconstruction and moved to more strategic, targeted grant making that launched more long-term development. Did SGP investments have a sustainable impact? Yes, we believe they did. Training programs have imparted much-needed skills, health campaigns have prevented the spread of disease, society has knowledge of how elections work and the rights of citizens, incomes 46 We do not answer all of the questions in the SOW because we were unable to implement the original research design and time did not permit us to write any sub-studies. 37 have increased, and parents have the knowledge of how to better care for their children nutritionally (among other long-term results). SGP funded a number of pilot projects in each sector that could possibly be scaled up into larger projects, but it is unclear as to which pilots should be so considered because there was no systematic evaluation of each pilot. Hence, we have made recommendations in this Section and Section 6 to build in funding and time for evaluation of each pilot. In terms of building capacity, SGP did a good job in helping grantees in developing proposals, identifying line items for the in-kind budget, in visiting projects and providing advice to grantees to improve practice, and, in general, providing project support. SGP also identified more developed grantees to expand their capacity through more rigorous managerial training (the development of ROs). While the development of ROs is a good idea, development of managerial capacity for all grantees could have been undertaken in a different way. This way is presented in Section 6. As political and socio-economic stability are achieved and Timor-Leste moves from a crisis situation to one in which long-term goals are established, a more long-term development assistance vision should be implemented in tandem with GOTL priorities. It is appropriate, given the deficiency of well-trained professionals in many sectors, to focus on capacity building in all USAID programming. Building the education (both formal and non-formal), health and agricultural sectors are a “must” to move the economy forward. In-kind SGPs are costly to implement. Now that Timor-Leste is emerging “out from under” the crisis challenges of the past, USAID should focus its attention on the three sectors noted in the last paragraph – education, health, and agriculture - while, at the same time, integrating the use of media and the building of civil society under each sector. Since there are several donors implementing different types of SGPs already, it is more appropriate for USAID to go on to more long-term development to walk alongside GOTL in building the nation. 5. MANAGEMENT STRUCTURE AND PROCESS 5.1 Introduction DAI had considerable skills in supporting small grant activities in Timor-Leste through the implementation of the OTI and PSI awards noted in the Introduction. The key word guiding SGP activity was flexibility. DAI had to accommodate changing CTOs, changing USAID structures and priorities, a country in crisis, the internal displacement of approximately 70,000 people, an underdeveloped infrastructure, and human capacity challenges in finding top-notch employees to identify, administer and evaluate grants. Because it was unclear who would respond to a proposal announcement, DAI had to be prepared for any type of submission (within the domain of the three SOs). This strategy created several management concerns and made certain changes necessary in DAI operations. In this section we address these management issues, highlighting where DAI has done incredibly well under very trying circumstances and where grant giving, monitoring, and evaluation might have produced slightly different results. We begin with the changing CTOs and SOs/FOs of USAID and the resulting shifting priorities, and then move on to internal practices that might have produced more impact information and greater synergies among grantees. 38 5.2 Internal Operations Despite several changes DAI had to make to accommodate new Mission directors, CTOs, and changes in SOs, DAI has followed a specific set of procedures in identifying grantees for in￾kind (local organizations) and cash awards (international organizations), reviewing proposals, making awards, monitoring and evaluating. In this Section, we point out some of the inconsistencies the team found in management of the process. The strategy DAI used initially for proposal solicitations was to identify the different NGOs involved in a target domain and share the guidelines with these organizations. As a member of the NGO Forum and Small Grants Network, DAI learned about the organizations working successfully in specific areas and advertised the solicitation among these groups. As news of SGP was circulated, many potential grantees went to USAID for information, and were then referred to SGP. DAI did not advertise on radio or TV but used informal networks to circulate information. In retrospect, the COP determined that that might have been a good strategy initially, but as priorities changed and more NGOs became aware of SGP, they could have made more effective use of the media (radio and newspapers). The team agrees. A representative of an interested NGO would often come to the DAI office to learn more. Program staff discussed the concept/idea with the NGO representative and, if the idea fell within the solicitation parameters, the NGO would be given the proposal writing guidelines and asked to submit a proposal. Oftentimes, the NGO did not have any experience in writing proposals and so sought DAI’s assistance in developing its ideas in appropriate proposal format. Once the proposal was submitted, it was reviewed by the PDS and a checklist filled out. Critical questions included: 1) Does this proposal fit any of USAID’s Program Sub-elements (eight are listed)? 2) Does this proposal fit one of the Small Grants Program Focus Areas (ten are listed)? If the answers were yes to these two questions, then other questions became significant: 3) Is this project cost effective? If not, why? 4) Does the implementing organization have the capacity and experience to manage this project? 5) Is the timeframe for the implementation of this project realistic? Listed on the form is the following legend: “If the proposal meets the criteria listed above, please prepare a grant form and present it to the Proposal Review Teams; otherwise, provide reasons for rejection of the proposal.” Once a grantee was informed of its successful proposal, a Grantee Handbook was distributed – one for In-kind Grants (local organizations) and one for Cash Grants (international organizations). Most important for in-kind grants were sections on the budget, required documentation (nine administrative forms), and activity reporting (a mid-term and final report). For cash grants, the most important sections deal with the budget, and financial (end of grant) and activity reporting (a mid-term and final report). The manuals are available in Bahasa Indonesia and English, as is the contract. Grants management and monitoring was guided by a manual rather than a Performance Monitoring/Management Plan (PMP). While this document exists for USAID reporting, it was not possible to construct one for SGP due to the variability in projects, outputs and outcomes, the “rolling” way in which grants were awarded, and the shifts in emphasis on SOs. The Manual outlines grant receipt procedures and monthly action plans, the life of an SGP grant, monitoring – impact and output, and additional guidance. The document also includes nine annexes: 1) proposal and grant flow chart; 2) weekly grants finance review tracker; 3) weekly grants action tracker; 4) weekly procurement tracker; 5) grants workload; 39 6) grants audit checklist; 7) initial environmental examination; 8) proposal competition record; and 9) grant negotiation memo. Under monitoring – impact and output, there is a discussion of USAID’s changing SOs/FOs and SGP’s response in creating SFAs. Indicators used are output indicators developed for USAID monitoring. These are utilized in the grant document created once an award has been made. When implementing an in-kind SGP project, it is common practice that grantees are not provided any cash directly. This creates difficulties when a grantee provides a training and must pay per diem to participants. Staff reported that when such payments must be made, DAI staff must carry a significant amount of money on their person to distribute directly to and obtain signatures from participants. This practice puts DAI staff at risk and leads to issues of mistrust among grantees.47 In discussing the grant process with grantees, many appreciated the input the PDSs provided not only in helping them to prepare their proposals but also in the feedback they shared when monitoring project activities. In a few cases, the time frame for completion of specific projects was unrealistic (e.g., an L&N project was given only six months to implement due to SGP closing dates, but would actually take ten months). In some cases, especially those that were granted toward the end of the project, the timeframe could not be negotiated.48 Although a type of capacity building was provided to grantees in the development of their proposals and in writing reports, the deeper capacity building required for a grantee to become sustainably successful was not provided as it was not a contracted deliverable. Hence, there was no mechanism to provide the management training needed by grantees for sustainability. When potential grantees were assessed on capacity (noted above), it is not clear how a decision was made. Given that education in Timor-Leste is very traditional (not learner centered to produce critical thinkers), having a diploma or a degree does not attest to the types of practical know-how needed to implement a successful project. Without a formal organizational capacity assessment, including questions relating to the specifics of the project, capacity may have been deemed to be suitable. It was only in some of the monitoring exercises that the PDSs (and other members of DAI’s leadership team) realized that the capacity, especially in financial management, was not there. Had this been known, specific arrangements could have been made for organizations to engage in developing improved practices either through attending workshops, being mentored by DAI staff (if appropriate), or to have someone with the desired skills “walk along” to assist the organization. In some cases, a grant was directed toward the procurement of the services of an “expert” consultant to advise and build the capacity of the grantee, but in those cases, the grant itself was for capacity building in technical and other areas. The proposal solicitation form from the outset (2004) requested that potential grantees describe their expected results (short and long term). This changed with the first solicitation of 2006 that requested a “description of expected results and impact (short and long term).” In examining the proposal forms for the grantees interviewed in EG, 15 of 31 (slightly less than 50%) identified the expected results. In only 2 of the 15 cases were actual results identified; the remainder identified outputs. 47 A particular problem arose when Cailalo (see DG discussion above) was accused of corruption when training participants did not receive their per diem because Cailalo was not allowed to make disbursements themselves. 48 One grantee wanted to conduct an impact assessment of his project, but was unable to do so because the SGP contract had not yet been extended. 40 In the monitoring “Field Trip Report,” more normative questions were posed of grantees and beneficiaries. The key question posed to ultimate beneficiaries on impact is: “How have you used what you’ve learned from participating in this activity?” It is from this type of question that results can be monitored, but not all monitoring visits addressed beneficiaries and the entire monitoring form was not filled out for each monitoring visit. Hence, results were not fully incorporated into the feedback process and the lessons learned were minimal in many instances. To their credit, however, those who monitored grantee activities did provide feedback to grantees on different ways the project might be improved. For the most part, grantees adjusted implementation accordingly. On the “Final Grant Evaluation Form,” filled out at the end of a grant activity, a normative approach is used to inquire into whether the project goals were met, whether there were any outputs developed by the project, and whether any part of the project will continue. No questions are asked about impact. For example, in agricultural development projects, participants were not asked how much they increased their yields and/or their income, whether their families eat better, or whether any assets were purchased for the family as a result of participation. Whether it is just assumed that participants are better off or whether impact was not really a serious part of the project is not clear.49 Because many of the projects funded under SGP were in the nature of a “pilot” project in which new ideas were being tested out, the team finds it very difficult to understand why specific results/impact questions were not posed in monitoring or in end-of-project evaluation. It is critical for pilots to be evaluated properly to determine if the idea, the implementer, and the implementation produced desired outcomes (not outputs). Some type of formative evaluation must be conducted that addresses not only the normative reaching of objectives but also documents the results/impacts the project has had. This is the way to determine if a pilot has really been successful and whether it is worthy of replication or expansion.50 Also noted on the proposal solicitation is the need for potential grantees to identify other sources of funding they currently have or will seek once their SGP support has ended. In many cases, this information is also missing. DAI provided us with an “SGP Grant Co-fund Commitment and Grantee Cost Sharing” table. By extrapolating information under “Coordination,” DAI was able to determine of the (now) 221 grants awarded, 49 (22%) of grantees made either a cash or in-kind contribution, and 60 (27%) received other donor funds or financial support. Relationships between SGP and national or local government were more directly addressed after 2006. It was necessary for DAI staff to review the security situation in areas where potential grantees wanted to implement their programs. In determining the security situation, DAI consulted with suco and aldeia leaders. Grantees were also encouraged to visit with local leaders before launching their projects so that leaders were informed of the activities 49 We were informed that more impact information is now being collected by program staff at the end of each grant through “Program Impact Statements.” 50 A similar point could be made when DAI identified ROs that could be of service in Timor￾Leste to other, like organizations once SGP closes down. In Becker’s needs assessments of the six selected ROs, he found a shortage in assets as well as significant shortfalls in capacity, especially in financial management and planning (see Gerald Becker, 2007, Needs Assessment of Resource Organizations in Timor-Leste. Dili: DAI, Inc.). Had the capacity question been investigated in the proposal review process, the catch-up capacity building activities now underway might not have been necessary. 41 taking place in their areas. Consulting with local leaders is a best practice even in more stable conditions. 5.3 Staff Reflections In conducting focus group discussions with staff, we learned that there are several areas in which they feel DAI could make improvements. In general, staff has been very satisfied with their employment; however, they did highlight dissatisfaction with the following. Many of the professional staff indicated that while they have benefited somewhat from office-based training provided internally, by external DAI staff and others, they would prefer offshore professional development, more specifically in aspects of management and leadership.51 In discussing the $450 stipend made available in the last few months for professional development, staff indicated that the amount would only partially cover expenses for a workshop; they had the expectation that DAI would pay the whole cost because of previous experience working for DAI under another project. Support and professional Timorese staff would appreciate a deeper understanding of USAID processes and procedures. They would like to understand more about large project development to build skills in this area. Staff also reported that they felt their loyalty and commitment during the crisis period was not seriously acknowledged. A non-financial reward or recognition for a job well done over and above what was expected would have been in order. While salary increases are granted after performance reviews, staff feels that the increasing cost of living has not been addressed adequately. When staff spends extra hours in the field or in completing documents, they said they are not given comp time or overtime to acknowledge the extra time spent at work. Staff would like to be able to take the extra hours in comp time and come into the office later than normal reporting time when this occurs. Staff shared with the evaluation team that the monthly large-group discussions do not lead to the changes sought. Several staff viewed these meetings as ineffective because staff is reluctant to raise issues in such a format and language issues sometimes presented problems. A few staff who had been promoted from within did not feel good about the process because external applicants were sought at the same time as the internal candidate was being considered. One staff member indicated that the Timorese are required to follow DAI rules and regulations, but that the expatriate staff is not held to the same standard of accountability. Another staff member observed that some Timorese staff wear multiple “hats” and still remain local support rather than professional staff. Some Timorese felt they were not appropriately trained to assume new jobs when they were promoted. Some staff feel that OJT sometimes is not enough and that attendance at a course would prepare them more effectively. 51 Professional staff do not believe they can obtain the types of training they seek in Timor-Leste. 42 The staff working in procurement would like the ceiling for direct purchases without obtaining quotes to be raised from $50 as suppliers complain about giving quotes for relatively small items. There is a general feeling among staff that they represent the image of the US Government to potential grantees and so deserve more professional treatment. They believe that modeling democracy begins with them and they should be recognized especially for the work they do with grantees because it is furthering the cause of USAID. The team does not believe that any formal recommendations should be made in this area because SGP is closing. We do consider management issues in the following section and make recommendations for future SGPs. We did make a couple of suggestions to staff: • To make sure the views of local staff are addressed in a more systematic manner, staff might create a staff council and choose their own representatives (1-2 people) to discuss their collective concerns with senior management. This would allow for staff to share their concerns with a trusted colleague in a non-threatening environment and ensure that human resource issues are negotiated at the management level. • Place a “suggestion box” in the office so staff can make anonymous suggestions, raise issues or voice their concerns in a confidential manner. 6. LESSONS LEARNED AND RECOMMENDATIONS Since we have already made SO-specific recommendations in Section 4, we take the opportunity now to consider the cross-cutting themes and issues that might be addressed in future SGPs developed by USAID or other donors. We further suggest the different types of synergies that could be developed among grantees or between grantees and larger projects awarded to different partners. We begin with recommendations on different aspects of the SGP project cycle, then present a sequenced set of recommendations on grant management issues, address the need to create more synergies, and end with a specific recommendation to DAI. 6.1 Small Grants Project Cycle Design In this sub-section we identify the rationale - in the form of lessons learned – for making the specific recommendation. 6.1.1 Define Grant Parameters Rationale: Small grants programs have been devised by donors to meet certain objectives. Many are designed to be flexible while others are designed to be strategic or focused. Some have ceiling amounts to grants while others are willing to transcend limitations in special cases. Some have specific time frames for funded projects to be completed while others are more open ended. Some programs are created to fund stand-alone projects, while others call for small grants to be linked to larger donor-funded projects. Some are willing to fund pilot programs to test what will work in a post-conflict (or other) environment, while others want to fund the expansion of projects that have already proven their ability to meet goals and objectives. Some grants are for in-kind expenditures for grantee operational costs, supplies, equipment, etc., while other grants are made in cash. According to the definition of the program parameters, donors develop reporting, monitoring and evaluation systems, some of which are the same as those for much larger, multi-year grants, and some of which are very simple and/or call for a period of reflection on the part of the grantee as to what the lessons learned have been. 43 Recommendation: Any organization wishing to design a small grants program must define its parameters in the following categories: 1) Domains of activity addressed (on a continuum from highly flexible/open to strategic/focused) 2) Grant ceiling amounts and categories (perhaps including a projected number of grants in each amount category) 3) Time frame allowed for the implementation of each grant 4) Stand-alone, linked to or part of other projects 5) In-kind or cash grants 6) Pilot or expansion of an existing endeavor. Also, will the granting organization have the same reporting requirements for a $5,000 grant as one for $100,000?52 What will be the difference? How often should each be monitored? If the SGP grant is embedded in a larger grant, should personnel employed by the larger grant be made responsible to monitor and evaluate the effectiveness of the small grant and then report the results to the SGP granting agency? All of these decisions should be made in setting up a small grants program, then appropriate staff hired to administer the program, and appropriate monitoring and evaluation tools developed. We are unable to recommend any one specific approach because small grants programs need to be contextualized to donor requirements, country stability, limits on staffing (both human and financial resources), and whether the program is stand alone or meant to be implemented in concert with other, larger and more long-term grants. There is no “one size fits all” recommendation. Rather, we set forth below a table of advantages and disadvantages in choosing one approach over another. Table 6.1 Choosing a Small Grants Approach Parameter Advantages Disadvantages 1a) Flexible/Open • Appropriate in post-conflict societies • Can address funding issues very quickly in a crisis • Can address many sectors at the same time • Can pilot different activities • Can be linked to any longer term projects • Can be abused by donor to fund items not covered in larger grants • Is costly to implement due to the number of grants possible • Is costly to manage in terms of staff time 1b) Strategic/Focused • Appropriate in more stable environments • Can focus on selected areas to create synergies among like-minded grantees • Can build a sector in discrete steps • Can create greater managerial expertise in granting organization in the domains of activity selected • In volatile situations, grants awarded in only one sector may create greater risk to the donor • May not be able to respond easily to changing environments affecting the focus area 52 For USAID, the same paperwork is required of all grants, no matter the size. 44 2) Grant Ceilings • Higher number of grants possible • Similar accounting procedures • Can exercise budgetary control and make projections • Cannot take advantage of more costly, short-term opportunities • Limited number of grants possible if ceiling open￾ended 3) Time Frames • Can distribute funding cyclically in a pattern • Can monitor grants in patterns • Some pilots or other projects cannot be completed in stated time frames • Limited time frames may not allow for a rapid appraisal or feasibility study, if needed • Limited time frames may not allow for an evaluation to take place 4a) Stand Alone • Can fund one-off reconstruction projects if needed • Can take advantage of a specific opportunity that has future potential • Might be needed in an environment were a lot of reconstruction is needed • Can be a pilot idea that should be tested • Can leverage other ideas and other funding • Less impact is likely • Implementing agency must exercise diligent oversight • Must be evaluated to determine impact 4b) Embedded in or linked to larger, longer-term projects • Can share responsibility with larger project for oversight • Enhanced impact possible as a “piece of the whole” project • Can leverage other activities within a project • Potential for manipulation by larger project • May limit range of funding choices if this is only strategy 5a) In-kind grants • Less costly to grantee to implement • Grantee can easily track items and services purchased • Works well with agencies that have financial management deficiencies • More costly in human and financial resources to implement • Require more policies and procedures for implementing agency • Require greater know-how and skills in implementing agency 5b) Cash grants • Less costly for donor to implement in terms of financial management and personnel • Require grantees to have significant grants, financial management and budgeting experience and systems • Requires grantees to have an 45 infrastructure that can procure items, create contracts, monitor and evaluate activities • Implementing agency must closely monitor expenditures possibly requiring increased reporting by the grantee 6a) Pilot projects • Opportunity to test new ideas with a limited time horizon • Opportunity to venture into areas other, larger grants cannot • Opportunity for potential grantees to be innovative • Can create opportunities for longer term, more comprehensive projects • Must be evaluated managerially and for impact to determine suitableness and broader applicability • Must be monitored closely to create a project-based “roadmap” for success • Creates expectations among grantees • Starts something among beneficiaries that might/will end at the end of the pilot • Must also test sustainability factors 6b) Expansion of projects • Ideas have been tested and found suitable and sustainable • Gives grantee opportunity to expand operations • May take project in a different direction that may be more effective than the original project • Project to be expanded must be evaluated to determine feasibility • Areas of project weakness must be strengthened before expansion • “Parent” project must be willing to make adaptations in light of what is being proposed by the SGP project 6.1.2 Conduct Background Research/Needs Assessment/Situation Analysis Rationale: Many SGPs do not conduct location-based needs assessments or conduct a country-wide situational analysis that outlines the projects/programs in operation in different districts. The value of conducting an overall needs assessment is in the SGP’s ability to determine whether a proposal would be suitable in a particular location. Moreover, the value of a potential grantee conducting a very specific, project-based needs assessment is to help identify the very specifics that should be included in the proposal to achieve outcomes and to obtain buy-in from local stakeholders. Recommendation: In preparation for funding a range of small grants, the granting organization itself must undertake a type of needs assessment/situation analysis in different locations. A community-based data collection methodology should be devised in which key community members are asked to identify their program needs, whether it be education, jobs, health, an understanding of their government, etc. This background analysis should serve as the basis for developing a request for proposals for the delivery of projects in certain domains and in selected locations. The research need not be undertaken throughout the country at one time, but could be undertaken as a “rolling” process in which more parts of the country are 46 included over time. It is possible that the first grant awarded by the SGP is to conduct these needs assessments/situational analysis. Alternatively, if more than one donor is implementing an SGP, then money could be pooled to have one needs assessment/situation analysis undertaken (this could serve as the groundwork for greater donor coordination later). The assessment may take place more than once during the life of the program (depending on how long the SGP will last). For instance, if it is to go for four years, then it might be useful to conduct a mid-term needs assessment along with a mid-term evaluation to determine how well grantees are meeting their objectives and creating impact, or to determine if other, more effective strategies should be developed. Potential grantees should also undertake a needs assessment (using a rapid assessment tool) as part of the grant proposal process, but this one must be location and project specific to set the parameters for their projects, including identifying the appropriate target population, methodology of implementation to fit the particular context, and what participants will be able to do once the project is over (impact). The SGP can develop simple tools and distributed to grantees during the initial workshop offered by SGP to explain how SGP works (see Grant Management Process and Procedures below). 6.1.3 Demonstrate Alignment with Government at All Levels Rationale: Some of the most effective grants are those that are integrated into GOTL’s programs at all levels. Recommendation: Future grant solicitations should direct CBOs and NGOs to align their projects with GOTL programs and priorities, both at the central and local levels, and show evidence of collaboration with local government officials. 6.1.4 Involve Government in Coordinating Donors and Granting Agencies Rationale: GOTL, as the main stakeholder in any development assistance activity, partners with many donor organizations to achieve its goals. It is also responsible for setting policies and regulations to be implemented at district and local levels. The same principle guiding donor coordination for large grants could be utilized for coordinating SGPs. Some donors/grantee organizations have a comparative advantage in certain domains and so might be better able to provide support for grants in, for example, media, health, microfinance in support of GOTL priorities. Recommendation: GOTL should convene an SGP working group in which all granting organizations and/or donors participate. The working group should hold regular meetings to discuss the different types of small grants on offer by each program, the NGOs being funded in each area (and at what dollar level), the activities of these NGOs, any lessons learned in implementation, and how SGPs can be more fully aligned with different levels of government. Collaboration could be developed among granting organizations to achieve a larger goal. If SGPs are very similar, collaboration could also be extended to the types of documents used for monitoring and evaluation so that grantees are not overburdened by or confused with the paperwork requirements of each granting agency/donor. If the working group is chaired by GOTL (either central or other delegated government official), the discussions held by the working group might leverage greater alignment within GOTL itself. 6.1.5 Encourage Stakeholder Involvement to Develop Ownership Rationale: Stakeholder buy-in to SGPs is important for understanding how small initiatives can be transformed over time into larger initiatives. Acting as a body to provide inputs to grant decisions, stakeholders could learn from how a project develops through all its phases so that successful endeavors could be replicated. 47 Recommendation: Donors wishing to establish an SGP should consider who its stakeholders are and the role they can play in project design, implementation, and evaluation. A determination should be made on how government buy-in can be achieved, e.g., make some type of in-kind or budgetary contribution, or to play a role in monitoring grants. Whatever is most appropriate for the donor organization and implementing agency should be determined though negotiation. Regardless of the strategy chosen, specific roles and responsibilities should be formalized. 6.1.6 Favor CBOs that Work within Existing Systems Rationale: It is common for SGPs to award grants to organizations that want to create new endeavors, new methods of service delivery, new materials, etc. To create more sustainable results, however, it would be wise to work within already-existing systems. By building on location-specific institutions and traditional groups and networks, the intervention proposed can become more sustainable because the projects are “joining in” an already-existing institution.53 Recommendation: Whenever feasible, fund district-based CBOs and supporting organizations (e.g., Rede Feto, AF, PDC) directly for sustainability so that, when the project ends, the community groups and activities can continue. Such groups might include women’s associations, church- or activity-based groups, young mothers groups, and the like. Suco chiefs might help identify these groups for the needs assessment and an inventory of CBOs could be developed and serve as the basis for community mobilization. 6.1.7 Evaluate Pilot Programs to Determine whether They Should be Scaled Up Rationale: Since SGPs often fund pilot projects, it is necessary to conduct a more in-depth evaluation of such projects to determine 1) if the idea itself was useful and produced the desired results, 2) if it might be useful if implemented in other locations, 3) the specific aspects of the project that worked well or not, 4) whether specific inputs were deemed critical, 5) if specific social structures enhanced or prevented project goal achievement, 6) whether the amount of time allowed was sufficient, and 7) if there were any extraneous factors that facilitated or prevented success. If the data collected point to a successful project, then it might be wise to consider a second grant to try out the idea in a different location. Recommendation: Pilot projects funded through an SGP should be evaluated to determine if the project might be scaled up or replicated in other parts of the country. Identifying outcomes should be a part of this process. If a project has done well in one location, grants to expand that project into other locations should also be evaluated to determine if there are any differences in implementing the project in separate locations. If a project is found to work in several locations, and the management of the project is acceptable, then it is possible that increased funding in the form of a larger project could be provided to carry on the same type of work throughout the country. 6.1.8 Have Grantees Self-Monitor Their Projects Rationale: Many grantees do not employ a monitoring and evaluation system to measure the short- or long-term impacts of their activities. The grant process requires that the granting agency staff conduct a final evaluation, which is more a normative evaluation than one that inquires into results or impacts. Very simple methodologies are available for grantees to measure their own impact, to document and share lessons learned, and to gauge whether their projects have been useful to participants/beneficiaries. 53 The SGP that is the subject of this evaluation funded 42 NGOs or government agencies with headquarters or permanent offices in the districts, and 91 NGOs or government agencies based in Dili. This recommendation encourages the continued expansion of funding for district-based organization. 48 Recommendation: Each grantee should be responsible for monitoring its own projects, identifying areas where they could be strengthened, and making appropriate adjustments. As part of the potential grantee orientation, participants should be introduced to the contractual needs in M&E. Once awarded a grant, grantees should be trained in and/or provided with simple participatory evaluation techniques so that they can ascertain the impact their projects have had on beneficiary populations. The evaluation mechanism should be a part of the grant contract. 6.1.9 Build the Capacity of Grantees to Take on Larger Responsibilities Rationale: Capacity development in the area of project design, implementation, management, and M&E of local NGOs is critical to their own ability to accept increasing responsibility in delivering well thought out, planned projects. Hence, capacity building should not only address how to write a proposal for a particular donor, but should target behavioral outcomes that would enable grantees to assume higher levels of responsibility. A system of capacity building should be devised and implemented over a given period of grant implementation so that, if there is to be a follow-on grant, or if the grantee wishes to seek larger funding from an external or governmental source, the grantee is more capable of doing so on its own. Recommendation: In line with the Management recommendations in 6.2 (Proposal Review), granting organizations (or implementers for donor organizations) should undertake an organizational capacity assessment as part of the grant proposal and review process in which NGO professional capabilities are assessed to manage a proposed grant. Once the areas needing strengthening have been identified, a range of workshops/modules should be provided. For those organizations that have been successful in managing grants previously, the granting organization should alert the donor that the specific NGO might be ready for larger, more complicated projects either standing alone, in a consortium, or as a “junior” partner to INGOs. Monitoring the capacity building exercise would help the granting agency to determine when an NGO is ready to assume greater responsibilities. 6.2 Grant Management Processes and Procedures In this sub-section, we have taken the lessons learned from Section 5 and transformed them into recommendations on how to improve the small grants mechanism. We have organized this sub-section in steps, which differs from the previous sub-section. Proposal Solicitations An SGP should develop a strategy for soliciting proposals that includes use of the media, professional associations, networks and word-of mouth. The strategy should also include the number of times solicitations will be announced annually and where workshops will be held. The solicitation should begin with an invitation to attend an informational workshop by the granting agency during which the parameters of the program are spelled out and proposal guidelines/ forms distributed. Initial Workshop The workshop facilitators should go through every line of the proposal, giving examples of the type of information that is required. It should be made perfectly clear that proposals must be submitted either on the form (if it is so constructed) or following the guidelines directly, 49 and that proposals that do not conform to these requirements will be rejected.54 It should also be made clear that attendance at the workshop does not guarantee proposal acceptance, and that consultations will not be held at that time to discuss (at length) the substance of a likely proposal. Workshop dates should be set to follow the announcement by approximately one month so that there is an opportunity for word to circulate and groups to more finely tune their ideas. Workshop locations might focus on Dili, say, twice a year, and a district once a year to encourage district-based organizations to develop proposals. The proposal packet should include: • Proposal Instructions • Proposal Form • Capacity Assessment Form • Local Needs (Rapid) Assessment Form The Proposal Form, aside from the standard information required, should also include questions on: • How was the local need for your project determined? What is the range of this need in the location of implementation? • How will this project relate to local government organizations? • How will this project relate to other, similar projects in the area? • What type of internal support will be provided to this project (in-kind, cash contributions)? • What type of other external support will this project have? • What is the anticipated outcome that this project will create? What will participants be able to do once the project is finished? • What is the strategy you will use to evaluate the effectiveness of your project? • How might your project become sustainable? Follow-on Consultations If those who attended the initial workshop still have questions or are unsure of how to proceed, potential grantees should make an appointment to discuss their issues with SGP program development officers (or their equivalent). As many CBOs and NGOs have not prepared grant proposal forms before, they may need this additional capacity building. Composition of the Proposal Review Team Members of this team should be senior leaders of the SGP as well as other stakeholders, e.g., a member of GOTL. Proposal Review Once a proposal is submitted and logged in, the granting agency should review the proposal by answering a number of questions. The following should be included in the review form (extrapolated from SGP forms with an additional point on local needs assessment): • Does the project demonstrate response to a local need? • Does the proposal fit into the donor’s priorities? • Does the proposal fit into one or more of the strategic focus areas of the SGP? • Has the local need for the project been established? What evidence is provided? 54 If the SGP grant strategy includes “emergency” funding, then this should also be made clear in the workshop. In this case, what is required for an emergency (the word must be contextually defined) may be different from standard grant proposals. The facilitator should spell out what the difference is. 50 • Is the project cost effective? Why? Why not? • Does the implementing organization have the capacity and experience to manage the proposed project? • Is the time frame for implementation realistic The first question requires the potential grantee to determine the need for the project in the proposed areas of implementation. If the proposal does not meet this need, it could either be rejected or returned to the organization for further information. The second two questions can be answered fairly easily while the remaining questions require a broad understanding of the country and skills in certain technical areas. To demonstrate local need, the proposing organization should include the Local Needs (Rapid Assessment) Form, and inquire as to whether any other organization is working in the location on similar programs. The signature and concurrence of the suco or aldiya leader should appear on this document. The document could then be reviewed against the overall needs assessment that had been conducted by the SGP at the outset of the program. To determine cost effectiveness, the review team should have an understanding of the cost of different inputs, salaries, travel costs, etc., to know 1) whether the budget designed will be sufficient to meet the costs identified, 2) whether other items should be added to the budget that have been forgotten (e.g., maintenance contracts for copy machines), and 3) whether inflation during the implementation period will be sufficiently high to affect the cost of the project. To determine organizational capacity, a simple form should be included in the proposal packet distributed at the initial workshop. That form should inquire into the structure of the organization, how long the organization has been in existence, the number and expertise of staff who will be involved in implementation, the equipment availability for implementation, and the different languages spoken by implementers. This form should be submitted as part of the proposal packet and reviewed by the Review Team. If there are shortfalls in capacity, the proposal might be rejected. If the project is well thought out and looks very do-able with the SGP mechanism, then the Review Team will have to decide whether the shortfall in capacity can be overcome in some way so as not to affect project implementation. In considering the time frame, attention should be paid to differentiating between an “ideal” and an actual time for project completion. Local staff on the Review Team should know if a time frame is realistic. Projects that may potentially entail several phases should be launched earlier rather than later. Successful Proposals Following the practice of SGP, once an organization receives notice that its proposal has been awarded a grant, the SGP should distribute a grantee handbook that includes all the forms needed to move the project along. The handbook should include simple self-monitoring and evaluation tools so that the grantee can monitor its own progress in implementing the project, track its impacts, and consider the outcomes the projects is having. These tools are especially important when the project is a pilot or involves some sort of information dissemination or training. Grant Monitoring & Evaluation 51 Grantees should be able to monitor and reflect on their own progress by using the simple monitoring tools mentioned above. Regular telephone conversations should also be held between the SGP and the grantee and a field visit should be scheduled to observe on-the￾ground activities. Key in monitoring and evaluation are interactions with the ultimate beneficiaries to determine if the project is accomplishing what it set out to do and whether the beneficiaries believe they are learning something useful. Questions on impact should be posed of ultimate beneficiaries, i.e., how have you changed since participating in this project? Grantee Capacity Building for Implementation Grantee capacity building on how to implement a project successfully, including monitoring and evaluating, should be undertaken throughout the project. SGP program development specialists might hold periodic workshops – in the same manner that the orientation workshop is held – to train grantees on how to manage the implementation process, how to monitor activities, and how to evaluate to understand impact. The tools included in the Grantee Proposal Packet and Grantee Handbook would serve as the basis for these workshops. They could be given a few times a year so that all grantees could take advantage. 6.3 Creating Synergies for Greater Impact In this sub-section we present two cross-cutting recommendations designed to create synergies among grantees. 6.3.1 Strengthen Professional Associations Rationale: Several different associations have been funded by small grants either to hold special meetings or for operational costs. However, there has been very little follow-up to determine the impact of these activities. Recommendation: SGPs should continue to support associations to contribute to a more professional and active civil society. However, a specific evaluation of these projects should identify how the meetings and operational support have actually strengthened the associations and led to greater effectiveness and efficiency. 6.3.2 Use Media Across SOs Rationale: In a number of grants in DG, EG and Health, the use of radio and video have proven to be very effective in reaching a large part of the population, especially illiterate populations. Recommendation: SGP grant solicitation should encourage including the use of media in any project that involves information dissemination (e.g., information campaigns on prevention of disease and good health practices, community conservation and waste management, or the role of parliament) thus cutting across strategic objectives. 6.4 Final Recommendation to DAI – Hold an All-grantee Workshop Rationale: Building relationships among grantees pursuing like projects is important for the development of future collaborations and sustainability. Recommendation: If DAI has sufficient funds, an all-grantee workshop should be organized to take place over three days. Grantees in like categories should come together to share implementation practices, materials produced, lessons learned, and to chart a course for working together in the future to develop materials, write joint funding proposals, and other productive activities. DAI should ask groups for written lessons learned and insights into future collaboration. The expected outcome of this workshop would be increased 52 professionalism, increased understanding of the field of endeavor, professional networks, and the ability to collaborate for mutual support. 53 USAID Small Grants Program Rua Humberto da Cruz Samping SD 3 Bidau, Acadiruhun Dili, Timor-Leste Tel: (+670) 3324 107/8 Fax: (+670) 3321 705 APPENDIX A SCOPE OF WORK Timor-Leste Small Grants Program (SGP) Scope of Work, Program Evaluation Team Contract Name: SWIFT II – Timor-Leste Small Grants Program (SGP) Contract Number: DOT-I-00-03-00004, Task Order No. 801 BACKGROUND The Timor-Leste Small Grants Program (SGP) was established in October 2004 to support USAID’s strategic objectives of accelerating economic growth, improving good governance practices, and improving the health of the Timorese people, as outlined in the USAID/Timor-Leste1 Mission Strategy for 2004-2009. Through small grants, the program supports communities, organizations, and government in their efforts to build a stable, economically robust, democratic country. The Small Grants Program was envisioned as a flexible program with the abilities to adapt to the social and political dynamics of Timor-Leste’s post-conflict development and to assist in identifying, testing, and opening up new focus areas for USAID. The scope of work in section 3.2.1 of the Task Order contract reads: “The objective of this Task Order is to support the implementation of the goals and objectives of the FY 2005 - FY 2009 Country Strategy under the technical direction of the Cognizant Technical Officer (CTO) in coordination with the East Timor Country Representative and other Mission units (as requested) and according to the terms of the IQC and applicable Task Order.” Section C.2 also further delineates objectives of the Country Strategy: The Mission’s goal of the 2005-2009 Country Strategy is to support efforts to build a socially stable, economic viable democratic country. Measurable results will be achieved in three mutually reinforcing specific areas: • Economic growth, which will focus on accelerated economic growth with a special focus on increasing employment and income opportunities, improving food security and product service delivery, and developing the capacity and skills of East Timorese to be achieved while promoting a sound enabling policy/legislation environment for agricultural diversification, private sector development, and access to local and regional markets; • Good governance, which will support an effective and responsive judicial system, citizen participation in local government and improved service delivery reaching all citizens while strengthening selected national and local institutions; and • Improved health, which will contribute to improving the health status of Timorese people at greatest risk, mothers and young children. Accordingly, SGP has implemented grant activities that have generated new opportunities for USAID’s long-term investment in several areas, including job 1 East Timor was the name used officially by the U.S. State Department and USAID until late 2005, when all internal and external naming changed to Timor-Leste, the official name of the country under the Constitution of the Democratic Republic of Timor-Leste. Therefore, all references in the contract have not been changed although all reporting since late 2005 now uses Timor-Leste officially. creation programs, microfinance, land tenure and agricultural school linkages to agribusinesses, and internship programs for young people. During its first two years of implementation, SGP responded to needs identified by USAID by implementing activities in a range of agreed upon focus areas under the strategic objectives as detailed in the USAID Country Strategy 2005-20092 . In September 2006, USAID/Timor-Leste introduced the new USAID/State Foreign Assistance Framework and Operational Plan. As demonstrated in the graphic below, the program’s strategic objectives were aligned with the USAID/State framework. Additionally, SGP used this opportunity to further refine its focus areas, providing more targeted assistance. USAID/TL Strategic Objective SGP Strategic Focus Area SGP Strategic Focus Area USAID/State Objective Government Service Delivery Elections & Local Government Oversight Institutions Legal Education Civic Education Public Information (Media) Public Information (Government) Private Sector Development Microfinance Business & Entrepreneurship Education Youth Pilot Programs Agriculture Agriculture Education Environmental Educaton Literacy & Numeracy Literacy & Numeracy Nutrition Education Other Health N/A Special Projects Special Projects N/A Improved Health of the Timoresee People Health Public Information Key Foundations of Good Governance Strengthened Accelerated Economic Growth Governing Justly and Democratically Economic Growth Investing in People Government Service Delivery Old Timor-Leste Small Grants Program Strategic Framework (Based on USAID/TL Mission's Strategic Plan) New Timor-Leste Small Grants Program Strategic Framework (Based on USAID/State Operational Plan) Effective Date for New Operational Plan: 30 September 2006 Private Sector Development Youth Programs, Entrepreneurship Education DAI implements this $13.4 million contract, which runs through September 2008. As of February 2008, SGP has awarded 213 grants and disbursed more than $5.7 million in grant funds. It is estimated that during the period of this evaluation approximately 45% of grant funding will have been programmed in economic growth, approximately 33% in democratic governance and approximately 23% in health. Additional information on the Timor-Leste Small Grants Program including program highlights, success stories and fact sheets can be found on http://timor￾leste.usaid.gov/. PURPOSE USAID Timor-Leste seeks an evaluation of the Small Grants Program to document the Program’s accomplishments and identify lessons learned. This report will be used to inform the final report for SGP, a deliverable of contract DOT-I-00-03-00004 TO 801.3 The evaluation should address the following questions: 2 Report available on USAID Timor-Leste website. See page 2. 3 Page 18, Section C.6.3 of SGP Task Order Contract: “The contractor shall submit…a Final Report summarizing all activities and accomplishments of the contract, assess lessons learned 2 • How effective was the SGP model given the Timor-Leste context? • What types of synergies were developed among and between SGP grantees and other externally- and internally-funded projects? • How did the selection and sequencing of grants and the awarding of multiple grants to specific grantees contribute to the emergence (retrospectively) of an analytical “roadmap” in the achievement of the three strategic objectives? • What is the specific impact of SGP awards on media, private sector development, and nutrition education on targeted populations? • Did SGP investments have a sustainable impact? • Did SGP help pilot programs for USAID? Which programs were expanded after a successful pilot? • Did SGP assist in building the capacity of local grantee partners? • Could USAID have achieved the same or better results more cost effectively? • Was SGP well-managed by USAID? TASKS • Develop an appropriate methodology for the evaluation, including a list of sub-questions for the central evaluation questions above. • Develop appropriate data collection instruments and interview schedules for sub-studies on the impact of the media, private sector development and nutrition education. • Evaluate the performance and impact of the Timor-Leste Small Grants program through a literature review and interviews with staff of the USAID Mission in Timor-Leste, implementing partner staff, Government of Timor￾Leste personnel, grantee partners and beneficiaries. • Provide a briefing to USAID Timor-Leste before departure. • In a draft evaluation for USAID distribution and comment, document findings, conclusions and lessons learned from the program as well as recommendations for the future. • Provide a final evaluation report for public distribution. DELIVERABLES • Brief outline of methodological approach for assessments, including sub￾questions, proposed itinerary, schedule for interviews, and identification of all logistical support needs. • Draft evaluation report, not to exceed 50 pages, plus additional annexes. The report should include the following sections: Table of Contents, Acronyms, Executive Summary, Background (USAID Mission’s strategy, country context, evaluation objectives and methodology, overall observations), Conclusions, and Recommendations. • Final evaluation report. TEAM COMPOSITION The evaluation team should include a Team Leader, one expatriate technical expert, two Timorese technical experts and a mid-level evaluator/coordinator. and provide comments and recommendations on the continued need, direction and benefits of a USAID Small Grants Program in East Timor.” 3 The Team Leader/senior-level evaluator will have extensive experience designing and conducting evaluations and some experience with small grants evaluation. The Team Leader will be responsible for the review, interviews, briefing, draft report and final evaluation report. S/He should have technical expertise in Economic Growth, Democratic Governance and/or Health. The expatriate technical specialist with experience conducting program evaluations and two Timorese technical experts should complement each others’ technical and evaluation experience and background. All technical specialists will also have extensive technical experience in either Economic Growth, Democratic Governance and/or Health depending on the technical expertise of the Team Leader, so that the evaluation team includes experts in all areas. The mid-level evaluator/coordinator will have relevant technical experience in one or more of the key areas and experience coordinating evaluations and assisting in the completion of evaluations. GENERAL TIME FRAME Desk Review and Draft Methodology: January 14 – March 31, 2008 Submission of Outline of Methodological Approach: February 29, 2008 Travel to Timor-Leste: April 1-4, 2008 Field Visit (Dili, Baucau, and Liquica; travel to selected other districts for sub￾studies): April 4 - 22, 2008 Submission of Draft Report: April 25, 2008 (by close of business. USAID will review and provide comments for incorporation into final draft by close of business on April 29, 2008.) Briefing on Findings: April 28, 2008 Submission of Final Report: May 3, 2008 Return Travel: May 4-6, 2008 4 1 APPENDIX B MEETINGS HELD Date Team Member Organization Title of & Person Interviewed Purpose 4/4 Nancy S, Jana, Alex & Nancy H DAI Kate Heuisler, COP; Jose , M&E; and Emre , COO Introductory Orientation 4/7 Whole Team DAI 1) Kate Huisler, COP 2) Grants Managers (3) 3) Selected PDOs 1) Overall project discussion 2) Discussion of roles and responsibilities 3) Obtain project contact numbers 4/8 Whole Team USAID 1) Brian Frantz, CTO 2) Dr. Teodolo de Ximenes, Project Management, Health SO 1) Discuss project, USAID’s role, linkage to other USAID’s projects, sub-studies 2) Discuss health projects, sub-study 4/9 Whole Team DAI USAID 1) Program Development Specialists (5) 2) D&G Team – Maria Isabel Ximenes and Ana Lorenco Guterres 3) EG Team – Dorvin Stockdale, Candido da Conceicao, Angela Rodrigues 1) Discuss roles and responsibilities; obtain recommendations for changes 2) Discuss D&G projects, SGP projects working with D&G projects; sub study 3) Discuss EG projects, SGP projects working with EG projects, sub study 4/10 Nancy H and Alex 1) Fundacao Cristal 2) Timor Aid 3) Grupo Feto Fonsa’e Timor Lorosa’e – GFFTL 1) Augustine Gomez, Program Manager 2) Alexander Gusmao, CEO 3) Filomena Fuca, Director; Abilita Da Silva, Ed & Trng; Discuss Literacy & Numeracy classes for business women; IGA classes; related training with each partner 4/10 Jana Hertz and Nancy S (Jornal Labrik only) 1) Jornal Labarik 2) Radio Rakambia and 1) Otelio Othe, Director 2)Eurico Pereira, 1) Discuss Jornal Labarik’s civic education pages and distribution 2) Discuss SGP grants to 2 Asosiasaun Radio Komunidade Timor￾Leste (ARKTL) 3) East Timor Insight Director 3)Antonio Eduardo, Director Radio Rakambia in particular nationwide civic education campaign, radio drama on parliament; ARKTL Congress 3) Discuss the nationwide civic education campaign in the eastern districts 4/11 Nancy Sheehan USAID Dr. Teodolo de Ximenes, Project Management, Health SO (ex-WHO) Discuss the Clinical Nurse Training grants and Dengue Fever Outbreak in 2005 4/11 Nancy H and Alex 1) National Directorate of Non-Formal Education 2) Haburas Foundation 3) Rede Foto 4) Rotary Club of Dili 1) Francisco Amaral, Assistant Director 2) Deometrio, Executive Director 3) Edna Desauro, Advisor; Sonya Bwavida, admin. 4) Jose Masado, President 1) Discuss Ministry of Education’s NFE in L&N activities; production of first and second curriculum 2) Discuss environmental ed program support 3) Discuss teaching of Level II L&N curriculum to economically active women 4) Discuss business education in high school and for street vendors projects 4/11 Jana Hertz (and Nancy S, UN Civil Society Working Group only) 1) Centro Audiovisual Max Sthal Timor￾Leste 2) Forum ONG Timor￾Leste 3) UN Civil Society Working Group 1) Max Sthal, Director, Julio Aporicio, Helder da Silva, Program Managers 2) Mario da Silva, District Liaison Officer 3) Aoao Jorge B. Pinto (CNJTL), Irene Maria da Costa (FONGTIL), 1) Discuss production of film on government and roles of parliament and audiovisual media as a tool for CE 2) Discuss FONGTIL distribution of film on role of parliament and FONGTIL as an umbrella NGO organization for information dissemination and advocacy 3) Discuss the UN Civil Society Working Group purpose, 3 Crisanto de A. Estevers (KSUTL), Frediano Piadade (CNJTL) organization, and program and its coordination with SGP civic education programs 4/14 Nancy H and Alex 1) East Timor Development Agency (ETDA) 2) Fundacao ETADEP 3) Zero Star 1) Palmira Pires, Director 2) Jilman, Director; Norberto, Program Coordinator 3) Comar Mendosa, Director 1) Discuss Youth Pilot in bookkeeping, business center development and other projects in which ETDA participates 2) Discuss youth horticulture training, soybean production, tractor rental and other related projects 3) Discuss horticultural marketing project (link to DSP) 4/14 Jana Hertz 1) Bibi Bulak 2) Government Information Office (GIO) 3) Former USAID D&G Program Manager 1) Irem Tolentino, Project Manager 2) Joaquim Santos, Director 3) Nicole Siebel 1) Discuss nationwide civic education campaign in central district 2) Discuss GIO communications training program for Council of Ministers 3) Discuss D&G program development and management and SGP contributions 4/14 Nancy Sheehan 1)Bairro Pite Clinic 2)Health Alliance International (HAI) 1) Dr. Danniel Murphy 2) Nadiane Hoekman, Director 1) Discuss support of building maternal clinic and IDP medical services 2) Discuss support for conference attendance and overview of HAI and obtain contacts for MOH interviews 4/15 Nancy H and Alex 1) Fundacao Buka Matanek 2) Tuba Rai Metin 3) Dili Institute of Technology 1) Hipolito Aparicio, Director 2) Angelo Soares, Executive Director 3) Estanislau Saldanha, Director 1) Discuss project to design an assessment tool for all L&N facilitators of NFE department and DAI partners 2) Discuss MFI operations and former relationship with CRS 3) Discuss career fair, DIT program 4 4) Asia Regional Pacific Operating Center 4) Karis Cooper, Executive Director 4) Discuss Junior Achievement program in primary schools and future plans 4/15 Jana Hertz 1) Timor Leste Media Development Centre (TLMDC) 2) IFES 1) Fransisco da Silva, Director, Nysan, PLAN advisor to TLMDC, 4 TLMDC staff (video production trainees) 2) Mary Lou Scharam, Chief of Party. Bernardo Cardoso, Program Officer 1)Discussed newspaper distribution, youth video production grants, and media development in Timor-Lest 2) Discussed IFES role in the USAID CEPPS program and connections with SGP; discussed IFES civic education program, methodology, and evaluation 4/15 Nancy Sheehan (and Jana for IDP Obrigado Barracks) 1) Alola Foundation 2) TAIS 3) CARE International 4) IDP Camp: Obrigado Barracks) 1) Ms. Meredith Budge, Advisor 2) Dr. Aoturo Sanabria, Director Lauri Winter, Technical Director 3) Diane Francisco, Director, Herminio M. Lelan, Project Manager REINHA Program. 4) Maria Filomena Martins, Camp Coordinator for distribution of hygiene kits, Agustinha Pompeia, Camp Coordinator, Celestino, Vice Coordinator, Liborio, 2nd Camp Coordinator 1) Discuss support for Psychosocial healing project and IDP project services 2) Discuss TAIS project and links into districts and with CSOs 3) Discuss CARE nutrition projects and IDP project services 4) Beneficiaries interview to triangulate for effectiveness 4/16 Nancy H and Alex 1) Private Sector Development Project (DSP) 2) Alola Foundation 3) MAF 1) Lendell Heuisler, Deputy COP 2) Meredith Budge, Communications 3) Adelino do Rego, 1) Discuss linkages between DSP and SGP￾funded projects 2) Discuss Christmas Faire and other activities 3) Discuss MAF priorities 5 Director of Agribusiness in agricultural development 4/16 Jana Hertz BELUN Antonio Conceicao, Director Discussed NGO Sector Strengthening Program and links to SGP 4/16 Nancy Sheehan 1) Nutrition Dept., MOH 2) CRS 3) Alola Foundation 4) Community Disease Control and Environmental Health Dept., MoH 1) Cecily Duignan, Advisor, Dirce Araujo, Director, Elias, Program Officer 2) Dr. Richard Marcowki, Program Manager, 3) Teresa Verdial Araujo, Advocacy and Education Program Manager 4) Tomasia de Sousa, Director 1) Discuss Nutrition Strategy, Nutrition working group and NGO and CSO links into strategy 2) Discuss CRS grants for ITNs and IDP services 3) Discuss Psychosocial Healing project and collaboration with to Rede Feto members 4) Discuss Dengue Fever outbreak in 2005 and MOH Environmental Strategies for containment 4/17 Nancy H and Alex 1) Candlenut Oil Enterprise, Acelda, Baucau 2) CTID Training Center, Baucau 3) PB Mobiliario 4) Timorese Youth Entrepreneurship Initiative 1) Higino Freitas, Director 2) Sister Candida, Director, and 3 teachers 3) Jaime, Carpenter 4) Talik Reis, Director 1) Discuss Candlenut project and export￾based income generation 2) Discuss hospitality training project 3) Discuss TA project for furniture production 4) Discuss current status of volunteer groups in agricultural development 4/17 Jana Hertz 1) CAILALO 2) Uma Media Regional 3) Yayasan Educasaun Comunidade Matebian 1) Augusto Pires, Director 2) Francisco Suni, Project Coordinator, Manuel Silva, Project Officer, Marcelino Pereira, Project Leader 3) Juliao Antonio de Jesus, Director, Lino Lopes, Program Manager 1) Discuss village administration and management training for aldeia leaders 2) Tour UMR and discuss its establishment and program 3) Discuss legal education and legal internship program in Baucau 6 4)Stakeholders/Viewers Nationwide of Civic Education Campaign 4) Sabino da Silva, Laga Sub-district Administrator, Augusto Pereira, Chief of Suco Atelari, Chief of Suco Soba. 4) Discussed implementation and reception of nationwide civic education campaign in Laga 4/17 Nancy Sheehan 1) WHO 2) IDP Camp: Airport 3) CARE International, Program Manager, REINHA Program. 4) CRS 1) Dr. Arun Thapa 2) Bendita, Camp Coordinator 3) Herminio M. Lelan, Project Manager REINHA Program 4) Manuel Viegas, Head of Program Support 1) Discuss WHO grants for Clinical Nurses Training, existing capacity of WHO and MOH 2) 2nd Beneficiary interview for IDP grants 3) Discuss effectiveness of Nutrition working group and make contacts with field staff in Liquica and Covalima 4) Discuss program support to CRS grants for ITNs and IDP services 4/18 Nancy H and Alex 1) Moris Rasik, Aileo 2) Ministry of Economic Development 1) Helen Todd, CEO 2) Colin Hartwell, Advisor 1) Discuss MFI Operations; visit two clients at their businesses 2) Discuss government policy on MFIs and related economic development 4/18 Nancy Sheehan 1) Liquica District Health Service (DHS) 2) CARE International 1) Filomeno dos Santos, Director 2) Joao Bosco, Program Officer for Nutrition Program, Joao Tedeana, Based Administrator, Luis Gonzaga, Program Office for LIFT Program, Maria Gorreti, Medical Field Supervisor, REINHA Program, Agustinho da Silva, 1) Discuss implementation issues of MOH and WHO grant interventions, district capacity issues and links with NGOs/CBOs 2) Discuss scope and synergies of district office and implementation support for CARE field staff 7 Medical Field Assistant, REINHA Program 3) Ms. Augustina da Silva and Mr Martinho Galocho from Loidahar suco Liquica 3) Discuss usefulness and impact of supplementary food strategy to two beneficiaries 4/21 Nancy H and Alex 1) Land O’Lakes 2) Timor Aid 3) Centro Formacao Joao Paulo II Comoro 1) Rodrigo Brenes, Director 2) Rosalia Suarez, Program Manager; Linda Alba, Manager L&N Programs 3) Sister Paula Batagliola, Director 1) Discuss cooperative education programs in agricultural schools 2) Discuss L&N course expansion with SGP grants 3) Discuss hospitality internship program 4/21 Jana Hertz Government Information Office field trip to Liquica Bazartete Sub-District Administrator, all chief of Suco in Liquica District To observe GIO trainee from the Ministry for Promotion of Equality lead a stakeholder meeting regarding information needs in Liquica district 4/21 Nancy Sheehan 1) National Hospital 2) Maternal Child Health, MOH 1) Dr. Odete Viegas, Director 2) Isabel Gomes, Director 1) Discuss about the result of the first congress of the Timor-Leste Medical Association (AMTL), what is the challenges and constrains 2) Discuss the Maternal and Child Health Program 4/22 Jana Hertz 1) STAE 2) ICFJ 3) INAP 1) Tomas Cabral, Director, 10 staff, and UN advisor 2) Dave Bloss, Director 3) Agustinho Letencio de Jesus, Director and Jose Vicente, Head of Local Governance Unit 1) Discuss polling staff training, equipment grant, and elections assistance 2) Discuss ICfJ Strengthening Independent Media Program in Timor Leste and UMR 3) Discuss materials development and oversight for village administration and management training for aldeia leaders 4/22 Nancy 1) Baucau District 1) Gelazio da Costa, 1) Discuss implementation 8 Sheehan Health Service 2) Baucau Hospital 3) Grupo Suporta Inan 4) Pastoral da Criancas Director 2) Cicilio Belo, Nurse Coordinator 3) Marciana Freitas, Coordinator 4) Ana Ximenes, Coordinator for Pastoral da Criancas issues of MOH and WHO grant interventions, district capacity issues and links with NGOs/CBOs 2) Discuss Clinical Nurse grant strategy effectiveness and follow￾up 3) Discuss implementation and support structures for GSI grant 4) Discuss implementation issues and support structures for PDC grant 4/23 Nancy H and Alex Catholic Diocese of Dili Fr. Augustinho Soares, Director of Commission on Social Affairs Discuss role of Church in development in T-L 4/23 Nancy Sheehan TAIS Dr. Aoturo Sanabria Follow-up information and check strengthen of strategy based on issues highlighted by district discussions 4/24 Nancy H and Alex Catholic Relief Services (CRS) Jason Belanger, Director Discuss CRS MFI and related activities 4/24 Jana Hertz FONGTIL Annual General Meeting Audience Attended the NGO Forum AGM and discussed the statutes, commission findings, and other AGM business with AGM participants and FONGTIL organizers 4/24 Nancy Sheehan 1) CRS 2) Health Net 1) Manuel Viegas, Head of Program Support 2) Francisco Vieira, Director, Simeon Coates, Consultant, Leao Pinto, Program Manager, Carlos Belo General Coordinator 1) Follow-up information 2) Discuss about the volunteers to support MoH operation for community health strategy related to the nutrition program 4/25 Jana Hertz 1) Foundation Hirondelle 2) NDI 1) Grey Dai, Project Manager 2) Telibert Laoc, Director 1) Discussed National Media Survey Timor￾Leste 2) Discussed NDI role in CEPP and interaction with 9 SGP and NDI civic education programs Nancy Sheehan UNDP Civil Society Project Simon Poppelwell Discussed volunteer support and remuneration issues in civil society sector 4/26 Team Meeting Discovery Hotel Nancy S., Jana H., Alex, and Nancy H. Integrated pieces of report; discuss 4/28 Team Meeting Discovery Hotel Nancy s., Jana H., Alex and Nancy H Integrated pieces of report discussed, prepare power point discussion 4/29 Whole Team Beach Restaurant Kate Heuisler, COP SGP and Brian Frantz, CTO, USAID Discuss panel discussion to be held 4/30 with other small grants providers 4/30 Whole Team Casa Minha SGP Stakeholders Hold panel discussions with other small grant providers, USAID, et al. 5/1 Whole Team DAI Office All SGP Staff Present evaluation research findings to staff APPENDIX C SGP EVALUATION METHODOLOGY Draft – 2/22/08 Nancy E. Horn, Ph.D., Team Leader 1. BACKGROUND Since DAI has awarded at least 210 grants, a key issue to an evaluator is determining, with the limitations of time and budget, how to understand the many results achieved and how these results lead or contribute to the achievement of USAID’s Strategic Obective (SO) results. The Timor-Leste context for the USAID-funded SGP project was a brand-new country emerging from struggle and launching its own state-nationhood. USAID identified two integral SOs to focus on – Economic Growth (EG) and Democracy and Governance (D&G) – to launch the state’s initiatives, and added a Special Objective – Health – because of the debilitating effects of disease and poor health care, especially on women and children. Within these SOs, several results indicators were developed (see below). Also developed were Strategic Focus Areas (SFA) within each SO (see below). Although some of these changed with the State Department’s new regulations, Timor-Leste’s new Operational Plan, and USAID’s targeted funding to international NGOs for certain areas, the core of the project remained almost the same. The methodology developed in this paper has several goals: 1) to identify an ex poste facto “roadmap” of grants, each of which contributed – sometimes obscurely – to each of the SOs; 2) on the basis of geographic location, to identify a “mosaic” of grants such that, for a given location, the overall result was larger than the sum of the individual grants awarded; 3) to ascertain whether the awarding of multiple grants to one organization in certain sectors increased impact; 4) to ascertain the impact of grants awarded to media organizations, especially on the training of journalists, the management of the media enterprise, the diversity of programming, and the development of an independent media; in addition, the impact of the media development awards on actions taken by listeners/viewers, thus contributing to a more transparent society and government; 5) to ascertain the economic growth achieved through support for private sector development (including microfinance, youth entrepreneurship, financial management training and direct support to businesses); and 6) to determine the impact of nutrition education grants. All of the goals will require the team to interview the grantees as well their beneficiaries. For the first three goals, this will largely occur in three geographic locations – Dili (the locus of a major portion of the grants awarded), and two other locations to be determined. For the final three goals, the team will be divided with each of the expatriates addressing one goal to conduct supplemental interviewing in several additional locations to provide greater depth and breadth to the evaluation in the areas of media development (D&G), economic growth achieved as a result of funding in four types of projects (EG), and nutrition education (Health) The primary strategy for data collection will be based on the Logic Model – inputs, activities, outputs, outcomes, results. By implementing this strategy, the team will be able to document: 1) capacity building of local organizations; 2) experimentation/piloting new ideas before USAID makes a larger investments; 3) DAI flexibility in being able to adapt to rapidly changing circumstances; 4) how SGP contributed to USAID’s strategy; 5) whether USAID could have achieved the same or better results more cost effectively; and 6) if USAID managed SGP well. 1.1 Link to Prior OTI Grant Development Alternatives, Inc. (DAI), was awarded the SWIFT II Timor-Leste Small Grants Program contract DOT-1-00-03-00004, Task Order 801, on September 29, 2004. In 2006, the contract was extended for a further two years, with additional funding. The contract has been running from October 1, 2004, and will end September 30, 2008. Prior to being awarded this grant, DAI had been awarded a contract by the Office of Transition Initiatives (OTI) – Building Empowerment, Leadership, and Engagement (BELE) & Transition Engagement for Population Support II (TEPS II) Programs – under the East Timor Community Stabilization Programs (AOT-I-01-98-00199-00). The latter were launched to overcome the devastation brought about by clashes between Indonesian troops and local pro-Indonesia militia during the people’s vote for independence in a government sanctioned referendum. The BELE and TEPS II programs were implemented over a 15-month period through a small grants mechanism (as is the case of the current project). The focus of TEPS II was on community participation in identifying, implementing and completing sub-projects, the majority of which were in community building restoration, road repair, irrigation systems, small-scale clean water systems, and in-kind support for income-producing cooperatives or groupings. BELE’s focus was to restore and enhance the social and productive asset base of rural men, women and youth and strengthen local governance structures through community engagement.1 In many ways, the current SGP has served as a more in-depth, finely focused follow-on to the OTI contract in that the OTI contract sought to rehabilitate, reconstruct and lay the foundations for Democracy & Governance (D&G), Economic Growth (EG) and Health activities to be awarded through USAID funding (although that was not a part of the project design). However, in identifying potential grantees for the SGP, DAI learned that many community-based organizations (CBOs), non-governmental organizations (NGOs), and government agencies did not yet have the skills to compete for a small grant. Hence, much effort has been extended by DAI to build the capacity of local organizations so that they could successfully compete. The need for capacity building of organizations and agencies cannot be understated. As the reader will see from the presentation of the tables below, a significant number of grants ultimately had capacity building as their goal. Whether a foundational, contributory, or programmatic grant, each had an element of capacity building within it in order for the organization/agency to implement its project. While the OTI contract sought to rehabilitate different buildings, facilities, roadways, etc., a significant number of grants under the current SGP also went to the rehabilitation of buildings, refurbishing of offices, and purchasing office equipment. Because USAID provided funding under three SOs - D&G, EG, and Health - the rehabilitation/ refurbishment of an office would 1 John P. Mason., et. Al. 2002. End-of-Project Participatory Evaluation. Building Empowerment, Leadership, and Engagement (BELE) & Transition Engagement for Population Support II (TEPS II) Programs. East Timor. Washington, DC: USAID/OTI/DAI 2 fall under these categories in accordance with the office’s domain, e.g., most government offices and media outlets fell under D&G; and health clinics under Health. While, as a team, we acknowledge that the rehabilitation/refurbishment was necessary for grantees to undertake this step before being able to move on to more programmatic endeavors, we will not evaluate these projects directly. We will, however, discuss the effect these projects had on the ability of the grantees to achieve the goals and objectives of their individual proposals. In general, we believe that the types of grants provided for rehabilitation/ refurbishment are an extension of the activities completed under the prior OTI grant. While they contribute to the ability of the grantee to successfully complete their projects, they are not programmatic in nature. On the “roadmap” (discussed below), after all physical necessities for operations were provided, further developing the human capacity for programmatic concerns took priority. Hence, capacity building to create quality performance was a major undertaking funded in grants across the three SOs. To address future USAID funding, DAI undertook an assessment of its grantees to determine which ones were in a position to manage more advanced programming in the future without the support it provided to each of the grantees. That assessment identified six organizations as ready to move on. DAI also launched a number of pilot programs that could serve as the basis for an expansion of activities. This evaluation will consider both the managerial status of each organization as well as whether the pilot programs launched are ready for scaling up. 1.2 Flexibility and Focus Over Time Over the grant period, the flexibility of SGP allowed for the development of an ever-deepening focus. At the outset, very general parameters surrounding D&G, EG and Health were a part of the call for proposals. Projects reported on were SFA under the three SOs:2 SO1, Accelerated Economic Growth: 1) Agricultural Production and Youth Programs 2) Foundations for Entrepreneurial Education 3) Financial Services 4) Investment Promotion and Economic Policy. SO2, Key Foundations of Good Governance Strengthened: 5) Rule of Law 6) Media, Oversight and Improved Service Delivery 7) Local Elections. SO3, Improved Health for the People of Timor-Leste: 8) Improved Health Service Delivery and Disease Prevention 9) Maternal and Child Health. Hence for the first year of the project, grants were implemented under these three SOs and these nine SFAs. 2 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report: April to September 2005. Washington, DC: USAID 3 Beginning in October 2005 the number and type of SFAs increased, and the type of grants awarded were more finely tuned. Projects reported on were still under the three SOs, but the SFAs were as follows:3 SO1, Accelerated Economic Growth: 1) Literacy and Numeracy 2) Environmental Education 3) Agricultural Education 4) Agriculture 5) Youth Pilot Programs 6) Business and Entrepreneurship Education 7) Microfinance 8) Private Sector Development. SO2, Key Foundations of Good Governance Strengthened: 9) Public Information 10) Oversight Institutions 11) Government Service Delivery 12) Legal Aid and Education 13) Elections and Local Government. Special Objective 3: Improved Health of the Timorese People: No SFAs specified. From semi-annual reports, we can conclude that the programmatic focus of SGP were further refined as some of the earlier reconstruction/refurbishment projects were completed, and as the political and economic capacities were strengthened. In the selection and implementation of many grants, it became clear that specific sectors needed to be addressed. These became the new SFAs. 1.3 Changes Due to New USAID/State Foreign Assistance Program Framework and Operational Plan During the October 2006-March 2007 reporting period, two events occurred that led to an adjustment in USAID strategy: increase in social disturbance, and the holding of elections. Both of these were very disruptive and entailed outbreaks of violence in many locations. Consequently, a study was commissioned by USAID on The Crisis in Timor-Leste: Causes, Consequences and Options for Conflict Management and Mitigation (Washington, DC: USAID, November 2006). The combination of the recommendations made in this report to enhance security and the promulgation of a new USAID/STATE Foreign Assistance Framework and Operational Plan for Timor-Leste created a further adjustment in SOs and SFAs:4 SO1, Economic Growth: 3 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report: October 2005 to March 2006. Washington, DC: USAID, 4 DAI. USAID Timor-Leste Small Grants Program. Semi-Annual Report October 2006 – March 2007. Washington, DC: USAID, April 16, 2007. 4 1) Private Sector Development (under the Program Sub-Element Microenterprise Business Development Services) 2) Youth Programs, Entrepreneurship Education (under Program Sub-Element Entrepreneurship Development) SO2, Governing Justly and Democratically: 3) Government Service Delivery (under Program Sub-Element Delivery of Local Goods and Services) 4) Civic Education (under Program Sub-Element Civic Education and Democratic Culture) 5) Public Information (Media)(under Program Sub-Element Media Outlets and Infrastructure) 6) Public Information (Government)(under the same Sub-Element). SO3, Investing in People: 7) Nutrition Education (under Program Sub-Element Maternal and Young Child Nutrition, Including Micronutrients) 8) Literacy and Numeracy for Economically Active Adults (under Program Sub￾Element Basic Education). While DAI was on a trajectory of deepening and refining the SFAs under each SO, the Operational Plan entailed a reduction in and combination of SFAs, including the recategorization and refocusing of literacy and numeracy training to meet the needs of youth in their entrepreneurship education, and reduced health to nutrition education and “other.” DAI characterized the changes as follows:5 The most significant changes are the merging of data from multiple SFAs and the introduction of new SFAs. Further explanation of how DAI-SGP will present data under each new SFA follows: • Data reported under the new Government Service Delivery SFA is the summation of all data previously reported under Government Service Delivery, Elections & Local Government, and Oversight Institutions. Future grants under this SFA will have a local government focus. • The Civic Education data includes grants previously categorized as Legal Education. DAI-SGP has discontinued awarding grants for Legal Education since this area of focus in being covered by USAID partner The Asia Foundation. SGP is currently working with USAID to refine its strategy for funding civic education activities. • The old Public Information SFA has been divided into two new SFAs: o Public Information (Media): targeting independent media-outlets, and; o Public Information (Government): targeting government communications Grants awarded to the Government Information Office (GIO) fall under the latter SFA. 5 Ibid., pp. 3-4. 5 • Data reported under the new Private Sector Development SFA is the summation of all data previously reported under Private Sector Devleopment and Microfinance. • Five old SFAs have been merged into the new Youth Programs, Entrepreneurship Education SFA. These are: Business & Entrepreneurship Education, Youth Pilot Programs, Agriculture, Agriculture Education, and Environmental Education. The decision to merge this data was based on DAI￾SGP’s experience piloting a number of youth-centric econojic development grant programs. Lessons learned from these activities have yielded DAI-SGP’s new strategy for targeted practical entrepreneurship development for children, youth and adults. • Data for Literacy and Numeracy was previously reported under the USAID/TL “Accelerated Economic Growth” Strategic Objectives. It is now reported under USAID/State objective of “Investing in People.” • Health activities have been disaggregated into two categories. Data under the new Nutrition Education SFA includes previous grants that dealt specifically with education for Maternal & Young Child Nutrition. Grants that did nto fit within this new SFA have been re-categorized as “Other Health” for the purposes of this report. DAI has charted these changes as follows: USAID/TL Strategic Objective SGP Strategic Focus Area SGP Strategic Focus Area USAID/State Objective Government Service Delivery Elections & Local Government Oversight Institutions Legal Education Civic Education Public Information (Media) Public Information (Government) Private Sector Development Microfinance Business & Entrepreneurship Education Youth Pilot Programs Agriculture Agriculture Education Environmental Educaton Literacy & Numeracy Literacy & Numeracy Nutrition Education Other Health N/A Special Projects Special Projects N/A Improved Health of the Timoresee People Health Public Information Key Foundations of Good Governance Strengthened Accelerated Economic Growth Governing Justly and Democratically Economic Growth Investing in People Government Service Delivery Old Timor-Leste Small Grants Program Strategic Framework (Based on USAID/TL Mission's Strategic Plan) New Timor-Leste Small Grants Program Strategic Framework (Based on USAID/State Operational Plan) Effective Date for New Operational Plan: 30 September 2006 Private Sector Development Youth Programs, Entrepreneurship Education 6 1.4 Performance Management Plan The Performance Management Plan (PMP) also underwent a number of changes, although not all have been documented officially in a new PMP. It is important that the team and the reader of this Methodology understand that the PMP went from the following: EG, 10 indicators; D& G, 17 indicators; Health, 7 indicators; to EG, 2 indicators; D&G, 4 indicators; and Health/Investing in People, 2 indicators. The reader can consult the 2005 PMP for a full description of the 34 indicators,6 which follow USAID’s Strategic Plan for East Timor 2005-2009.7 The indicators developed under the USAID’s new operational plan are as follows: Economic Growth • Total number of micro-enterprise clients of USG-assisted business development services • Number of Persons receiving training in USG-supported micro-entrepreneurship training sessions Governing Justly & Democratically • Number of individuals who received training in local government and or decentralization • Number of people who have completed Civic Education Programs • Number of Non-State News Outlets Assisted • Number of Journalists Trained Investing in People • Number of adults enrolled in literacy programs • Number of people trained in maternal and/or newborn health and nutrition care through USG-supported programs It should be noted that the indicators under the operational plan are NOT results indicators, but output indicators that measure only participation in training and/or technical assistance programs. These vary considerably from the results orientation of the PMP.8 1.5 SO, Geographic and Organizational Diversity 1.5.1 Overall Grant Diversity - According to the last Semi-Annual Report provided to the team by DAI (April – September 2007), as of October 15, 2007, 174 grants had been awarded: EG – 57 (32.8%); D&G – 73 (42%); Health/Investment in People – 40 (23%); and Special Projects – 4 (2%). The value of the grants was: EG -$2,255,411; D&G - $1,714,687; Health/Investing in People - $1,365,812; and Special Projects - $99,864 [insert updated information when received in April]. [Info for SOW provided 2/08, $15.7 million in 213 grants, 45% in EG, 33% in D&G and 23% on H – estimated.] Geographically, grants were awarded to organizations in each of the 13 districts: 6 USAID. East Timor. Performance Management Plan. USAID/East Timor. FY2005 – FY2009. USAID: Dili, East Timor, July, 2005. 7 USAID. East Timor. USAID Strategic Plan for East Timor. A New National Moving Forward. 2005-2009. Dili, East Timor. 8 The team understands that a new PMP is being developed, but it was not ready at the time of writing this draft methodology. 7 Table 1: Grants Awarded by SO in Each District*9 DISTRICT NO. OF GRANTS AWARDED Economic Growth (includes Literacy & Numeracy)10 Governing Justly & Democratically Health, Investment in People Special Projects TOTAL Dili 38 52 22 2 114 Aileu 9 10 6 1 26 Ainaro 7 12 2 1 22 Baucau 14 19 8 41 Bobonaro 9 12 5 26 Covalima 7 14 6 27 Ermera 8 10 6 24 Lautem 10 17 5 32 Liquica 12 9 10 31 Manatuto 7 13 4 24 Manufahi 7 11 3 21 Oecussi 10 17 3 1 31 Viqueque 6 13 2 21 TOTAL 144 209 82 5 440* *The number of grants awarded and implemented per district exceeds the total number of grants (210) awarded because many grants were implemented in several districts. The largest number of grants was awarded to organizations and agencies in Dili (112), Baucau (41), Lautem (32), Liquica (31) and Oecussi (31), with most other districts receiving between 21 and 27 grants. However, when we review the different types of grants provided, we learn that many grants were awarded to reconstruct and/or refurbish buildings, purchase supplies and equipment, and translate and produce documents. These “grant categories” are foundational to the ability of different agencies and institutions to conduct their programmatic work, especially in capacity building and training, and we acknowledge that the types of programs made possible by providing grants in these categories facilitated later programmatic undertakings. We consider this type of grant “foundational” as without them, the activities undertaken would not have been possible. Other grants deemed integral to the ability of an agency or institution to conduct its work include operational support, payment for the advice of external experts, conducting feasibility studies, and sponsoring of international trips to attend conferences (for capacity building or promotional purposes). These are not necessarily programmatic either, although they make possible the efficient and effective functioning of the agency or institution. These grant categories are of a higher order on the “roadmap” although are still in the nature of support mechanisms rather than 9 DAI. “USAID Small Grants Program – Grant Information Sheet for Final Evaluation Team,” February 1, 2008. 10 After the new Operational Plan was introduced, Literacy & Numeracy was moved to “Investment in People.” In the evaluation we shall consider all grants awarded before the 2007 literacy & numeracy study in the EG category, and the final three awarded on the basis of the recommendations made in the study as part of “Investment in People.” 8 direct project delivery. The important point being made in awarding such grants, however, is that without these contributory inputs, the projects awarded in the programmatic areas might not have been possible. We shall address this point in our analysis of the impact of projects, although we will not examine each project in each of these four categories for its individual impact. We take it as a given that such contributions were deemed necessary by DAI, and our analysis of actual program outputs, outcomes and results will demonstrate how these items contributed to the success of each project evaluated in the mosaic. We consider these types of grants “contributory” to program outputs. When we remove the foundational grants from the numbers of grants awarded under each SO, the results are as follows: Table 2: Grants Awarded by SO in Each District, Minus Foundational Grants11 DISTRICT NO. OF GRANTS AWARDED Economic Growth Governing Justly & Democratically Health, Investment in People Special Projects TOTAL Dili 24 31 16 1 72 Aileu 5 7 4 1 17 Ainaro 5 6 2 13 Baucau 11 14 5 30 Bobonaro 5 5 5 15 Covalima 5 8 6 19 Ermera 5 4 4 13 Lautem 7 10 5 22 Liquica 6 4 9 19 Manatuto 4 8 4 16 Manufahi 5 4 3 12 Oecussi 7 9 3 1 20 Viqueque 5 8 2 15 TOTAL 94 118 68 3 283 The 36% drop in the number of grants attributable to foundational grants is highly significant because so many agencies and institutions required refurbishment, supplies and equipment, and assistance with the production of materials for dissemination in workshops or in conducting informational campaigns. We shall return to this point when we address the methodology of the “roadmap.” From these totals, we remove the total number of Operational Support Grants (EG – 7; D&G – 38; and H – 14), 59; grants for external consultants (EG – 12; D&G – 5), 17; grants for feasibility studies (EG – 6; H – 1), 7; and international trips to attend conferences or promotions (EG – 3), 3, for a total of 86. These contributory grants were also integral to the successful 11 DAI. “USAID Small Grants Program – Grant Information Sheet for Final Evaluation Team,” February 1, 2008. 9 operation of the particular agency/organization that received the grant. This leaves the following programmatic grants awarded under each SO in each district:12 Table 3: Grants Awarded by SO in Each District, Minus Foundational Grants and Contributory Grants13 DISTRICT NO. OF GRANTS AWARDED Economic Growth Governing Justly & Democratically Health, Investment in People Special Projects TOTAL Dili 10 12 13 1 36 Aileu 4 2 3 1 10 Ainaro 4 3 2 9 Baucau 4 9 3 16 Bobonaro 3 3 3 9 Covalima 3 4 3 10 Ermera 3 2 2 7 Lautem 5 7 4 16 Liquica 5 3 6 14 Manatuto 3 5 3 11 Manufahi 4 2 3 9 Oecussi 6 7 2 15 Viqueque 1 4 2 7 TOTAL 55 63 49 2 169 These grants fall into the following categories: capacity building, holding of local informational sharing conferences, launching pilot programs, providing direct health care, conducting disease prevention activities (including nutrition education), and conducting informational campaigns on various governance- and health related topics. It cannot be understated that the foundational and contributory grants made possible the more programmatic grants. The specific Strategic Focus Areas targeted by these 169 grants (to help us further in our selection of research sites), are represented in Table 4: 14 12 The numbers do not agree in all cases because some grants fell into more than one category. 13 DAI. “USAID Small Grants Program – Grant Information Sheet for Final Evaluation Team,” February 1, 2008. 14 The SFAs (a combination of two of the sets noted above) under each SO are as follows: • EG – Private Sector Development, Microfinance, Agricultural Education, Agriculture, Literacy & Numeracy, Youth Pilot Programs, Environmental Education, Business & Entrepreneurship Education, Special Projects • D&G – Public Information, Government Service Delivery, Oversight Institutions, Elections & Local Government, Legal Education, Civic Education, Special Projects • Health – Health, Nutrition 10 Table 4: Grants Awarded in SFAs Under Each SO by District DISTRICT ECONOMIC GROWTH GOVERNING JUSTLY AND DEMOCRATICALLY HEALTH Dili Microfinance – 1 Environmental Ed. – 1 Literacy & Numeracy – 3 Youth Pilot – 1 Bus. & Entrep. Ed. – 4 Oversight Insts. – 1 Public Info. – 4 Govt. Serv. Del. – 2 Civic Ed. - 5 Direct Svc. – 3 Pilot – 1 Nutrition Ed. – 2 Info. Campaign – 2 Disease Prevent. – 2 Cap. Build. – 3 Aileu MFI – 1 Literacy & Numeracy – 2 Bus. & Entrep. Ed. – 1 Public Info. – 1 Civic Ed. – 1 Cap. Build – 2 Info. Campaign – 1 Ainaro MFI – 1 Literacy & Numeracy – 2 Bus. & Entrep. Ed. – 1 Legal Ed. – 1 Public Info. – 1 Civic Ed. – 1 Cap. Build - 1 Info. Campaign – 1 Baucau Private Sect. Dev. – 1 Environmental Ed. – 1 Youth Pilot – 1 Literacy & Numeracy – 1 Public Info. – 4 Legal Ed. – 2 Civic Ed. – 2 Govt. Svce. Del. – 1 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed. - 1 Bobonaro MFI – 1 Environmental Ed. – 1 Literacy & Numeracy – 1 Govt. Svce. Del. – 2 Civic Ed. -1 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed. -1 Covalima MFI – 1 Literacy & Numeracy – 2 Govt. Svce. Del. – 1 Public Info. – 1 Civic Education – 2 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed. - 1 Ermera MFI -1 Bus. & Entrep. Ed. – 2 Govt. Svce. Del – 1 Civic Ed. – 1 Cap. Build - 1 Info. Campaign – 1 Lautem Environmental Ed. – 1 Agricult. Ed. – 1 Literacy & Numeracy – 2 MFI – 1 Public Info. – 4 Legal Ed. – 2 Civic Ed. – 1 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed – 2 Liquica MFI – 1 Literacy & Numeracy – 2 Bus. & Entrep. Ed. – 2 Legal Ed. – 1 Civic Ed. – 1 Public Info. – 1 Cap. Build - 1 Info. Campaign – 2 Direct – 1 Nutrition Ed. – 1 Disease Prev. – 1 Manatuto Literacy & Numeracy – 2 Youth Pilot – 1 Public Info. – 1 Legal Ed. – 2 Civic Ed. – 2 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed – 1 Manufahi MFI – 1 Youth Pilot – 1 Private Sect. Dev. – 1 Literacy & Numeracy – 1 Public Info. – 1 Civic Ed. - 1 Cap. Build - 1 Info. Campaign – 1 Nutrition Ed – 1 Oecussi Agric. Ed. – 1 Literacy & Numeracy – 4 Public Info. – 1 Govt. Svce. Del. – 2 Cap. Build - 1 Info. Campaign – 1 11 Bus. & Entrep. – 1 Civic Ed. – 3 Legal Ed. – 1 Viqueque Literacy & Numeracy – 1 Public Info. – 2 Civic Ed. - 2 Cap. Build - 1 Info. Campaign – 1 There is significant diversity in the types of organizations that were provided SGP awards, although many received a multiple of grants. It should be noted, however, that many grants were funded to be implemented in more than one district. This means, in selecting site locations for interviews, the team must identify the headquarters of the organization that received the particular award, interview the staff there, and then go to selected other sites to compare/contrast delivery methods in each location. Table 5 identifies local grantees and the number of awards they received (if more than one) within each of the three SOs: Table 5: Organizations Receiving Multiple Grants Organization No. of Grants Awarded Economic Growth Governing Justly & Democratically Health, Investment in People 1) Agriculture Development Organization 2 2) Aifunan 2 3) Alola Foundation 3 4) Asia Pacific Regional Operating Center 3 5) Asosiasaun Hak 2 6) Asosiasaun Radio Komunidade Timor Leste 2 7) Association of Microfinance Institutions in Timor Leste 4 8) Bairo Pite Clinic 2 9) CAILALO NGO 2 10) CARE International Timor Leste 5 11) Catholic Relief Services Timor-Leste 3 3 12) Comissao Nacional de Eleicoes 2 13) Crystal Foundation/Fundacao Cristal 5 14) East Timor Development Agency 2 15) Fatumaca Technical School 2 16) Forum Peduli Wanita Oecusse 5 17) Fundacao ETADeP 4 18) Fundacao Fatu Sinai Oecusse 3 19) Fundacao Xanana Gusmao 2 1 12 20) Fundasaun Educasaun Comunidade Matebian 3 21) Haburas Moris 2 22) Health Alliance International 2 23) Internews 3 24) Judicial System Monitoring Program 3 25) Ministry for the Presidency and the council of Ministers/Government Information Office 2 26) Ministry of Agriculture, Forestry & Fisheries 1 1 27) Ministry of Development 2 28) Ministry of Foreign Affairs & Cooperation 4 29) Ministry of Health 4 30) Ministry of Justice 3 31) Ministry of State Administration, Technical Secretariat for the Electoral Administration 3 32) Moris Rasik 3 33) Office of Prosecutor General 2 34) Office of the Prime Minister 4 35) Office of the Prime Minister, Human Rights Advisor 2 36) Radio e Televisao Timor￾Leste 2 37) Radio Comunidade Lospalos 3 38) Radio Rakambia 8 39)Rotaract Club of Dili 3 40) Timor-Leste Media Development Centre 2 41) Timorese Young Entrepreneurship Initiative 2 42) University of Hawaii 2 43) World Health Organization 3 If we delete the international organizations (CARE International, Catholic Relief Services, Health Alliance International, Internews, the University of Hawaii, and the World Health Organization), we find that DAI provided more than one grant to 37 different government agencies and local NGOs. If we remove grants to government agencies (whose main focus was to improve the delivery of government services) – approximately 11 – then we would expect to see more improvement in the 26 remaining organizations delivering SGP-funded activities. 13 In reviewing the Resource Organization Assessment conducted by DAI,15 we learn that all of the six organizations chosen as the strongest have all received multiple grants under SGP (between 2 and 5): • Association Hak - 2 • Crystal Foundation/Fundacao - 5 • Fundacao ETADeP - 4 • Fundacao Fatu Sinai Oecusse - 3 • Fundasaun Educasaun Comunidade Matebian - 3 • Forum Peduli Wanita Oecusse - 5 It is the intention of this research to examine a selection of these organizations – as opposed to organizations that received only one grant to ascertain the effect of increased support over time in the ability of the organization to perform well. The theory to be tested is that those organizations who received more grants over a longer period of time benefited from the unstated capacity building given to each organization by DAI in the development of ideas, proposals, budgets, M&E plans, creating results, and similar activities. 1.5.2 Grants in Media Development - The diversity of grants awarded also includes a relatively large number of grants to radio and other media to broadcast information on elections, changes in government and laws, economic information, and related items. The following media organizations were provided these types of grants; where more than one grant was awarded, it is so noted: Table 6: Organizations Receiving Media Improvement Program Grants Organization District Purpose of Grant Period of Grant PBS Timor-Leste 1) Dili 2) Aileu Ainaro Bauicau Bobonaro Covalima Dili Ermera Lautem Liquica 1)Provision of transmitter and renovation of electrical grounding system 2) Expansion of national radio broadcatsing area to underserved areas 1)November 19, 2004 – May 30, 2005 2) May 5 – November 30, 2006 Radio Rakambia 1) Oecussi 2) Baucau Manatuto 3) Covalima 4) Liquica 5) Aileu Ainaro Manufahi 1) Campaign on village chief and councils elections 2) Campaign on suku and aldeia elections 3) Campaign on suko and aldeia elections 4) Campaign on suku 1) December 1- 31, 2004 2) February 11 – March 31, 2005 3) April 7 – May 31, 2005 4) August 5 – September 30, 2005 15 Gerald Becker. Needs Assessments of Resource Organizations in Timor-Leste. Bethesda, MD: DAI, December 2007, pp.5-6, 14-26, and appendices. 14 6) Covalima Manatuto Oecussi Viqueque 7) Dili 8) Baucau Lautem Manatuto Viqueque and aldeia elections 5) Mobile broadcast during crisis 6) Mobil broadcasts on parliamentary elections in underserved areas 7) Development of a radio drama on parliament 8) Conduct national civic education campaign in Eastern districts 5) May 5 – July 28, 2006 6) June 1 – August 1, 2007 7) September 20 – November 30, 2007 8) December 12, 2007 – March 12, 2008 Radio Comunidade Lospalos 1) Lautem 2) Lautem 3) Lautem 1) Campaign on suko and aideia elections 2) Establishment of a production studio 3) Purchase of generator 1) March 9 – April 15, 2005 2) August 6, 2005 – January 27, 2006 3) February 12 – May 31, 2007 Internews 1) Dili 2) Dili 3) Dili 1) Journalism training for balanced political reporting 2) Assist Timor-Leste Media Development Center to become preeminent media development center 3) Assist Government Information Office to develop sustainable communications programs 1) June 1 – November 30, 2005 2) October 1, 2005 – March 31, 2006 3) November 1, 2005 – March 31, 2006 Asosiasaun Radio Komunidade Timor Leste 1) Dili 2) Dili 1) Operational support; strengthen management capacity 2) To help implement a national congress of community radio stations 1) June 30, 2005 – June 30, 2006 2) October 18 – December 31, 2007 Lifau Pos 1) Oecussi 2) Oecussi 1) Equipment and transportation for weekly newspaper 2) Business training and operational support to continue 1) July 20, 2005 – April 30, 2006 2) August 7 – November 7, 2007 15 publishing newspaper Government Information Office 1) Dili 2) Dili 1) Improve communication with residents by 5 ministries (see Internews 3) 2) Assist government ministries to communicate with remote populations 1) April 3, 2006 – January 2, 2007 2) November 24, 2006 – February 1, 2008 Timor-Leste Media Development Center 1) Dili 2) Baucau Lautem Viqueque 1) Support for transition to independent media organization (see Internews 2) 2) Distribution of newspapers during crisis 1) April 3 – October 31, 2006 2) May 9 – November 10, 2006 Suara Timor Lorosae Dili Support for printing and distribution of newspapers during crisis June 1 – August 31, 2006 Radio e Televisao Timor-Leste 1) Dili 2) Dili 1) Renovation of toilets and kitchen for use by IDPs living on RTTL compound 2) National media coverage of parliamentary elections 1) July 18 – December 15, 2006 2) June 11 – June 30, 2007 Diario Nacional Dili Restart and continue printing the newspaper during and after crisis 1) July 27 – October 27, 2006 Centro Audiovisual Max Sthal Timor Leste Dili Film production and distribution on presidential and parliamentary elections 1) June 1 – July 13, 2007 Plan USA Dili Establish video production unit and provide training for the TLDC on youth 1) November 1, 2007 – May 30, 2008 Uma Media Regional Baucau Establish an independent regional media house to serve 1) December 11, 2007 – June 30, 2008 16 media outlets and journalists in the eastern districts East Timor Insight Bobonaro Covalima Ermera Liquica Oecussi Conduct nationwide civic education campaign on changes in government using multiple media sources December 6, 2007 – March 6, 2008 Bibi Bulak Group Dili Aileu Ainaro Manufahi Conduct nationwide civic education campaign on changes in government using multiple media sources (central districts) 1) December 12, 2007 – March 17, 2008 Forum ONG Timor Leste Dili Create civic education film on roles and responsibilities of parliament (see CAMS) 1) December 17, 2007 – June 13, 2008 Jornal Labarik Dili Aileu Ainaro Baucau Bobonaro Covalima Ermera Lautem Liquica Manatuto Manufahi Oecussi Viqueque Increase printing volume of children’s newspaper 1) January 2 – April 30, 2008 A stand-alone study funded by USAID on the effectiveness of media outreach was undertaken by foundation Hirondelle (Media for Peace and Human Dignity) in October-November 2006, and was finalized May 2007.16 Relevant information will be extrapolated from this report, and an evaluation will be undertaken on how DAI SGP directed its grants in light of the information presented in this report. A separate data collection instrument and target beneficiaries will be developed to focus on changes in media grantees and the impact these changes have had on Timorese.17 16 Eduardo Soares and Graham Mytton. Timor-Leste National Media Survey. Final Report. Timor-Leste: Fondation Hirondelle, USAID, May 2007. 17 The media study was focused primarily on listenership (for radio), viewership (for TV), and readership (for newspapers), although most questions were directed toward radio and TV, especially RTTL – the government￾17 The report of a workshop convened by USAID’s Bureau for Policy and Program Coordination in 2003 entitled Media Assistance: Best Practices and Priorities by Ellen Hume (November 2003) identified how media related to peace-building, a key issue in Timor-Leste. Hume identifies media development needs in three categories: 1) vulnerable countries; 2) countries at war; and 3) post-conflict media development. Because Timor-Leste, over the period of SGP implementation, has gone through each of these categories, we find it useful to consider how SGP undertook funding of local media development proposals under the D&G SO so as to improve governance. We offer a brief analysis of these three categories as a preface to the type of data collection strategy we will use in the evaluation research to demonstrate a linkage between media development and the improvement of governance. In vulnerable countries, it is important to build “local and regional journalism capacity . . . to offset the power of targeted hate messages” (Hume 2003, p. 10). The issue to get across to journalists is that they must “think beyond ethnic lines and include divergent viewpoints” (ibid.). The focus is on the development of pluralistic media. In countries at war, citizens “need news and special programs, particularly broadcast, that support peace-building, dialogue, and information about available humanitarian assistance” (ibid., p. 11). In post-conflict countries, four issues are key: 1) promoting independent media (and weaned off donor support); 2) bridging ethnic divisions (by supporting a large number of independent media outlets); 3) sustaining development efforts over the long term (addressing plurality, professionalism, and media independence); and 4) strengthening donor cooperation (through periodic meetings to share best practices, form coalitions, and address common regional or thematic media development issues). By utilizing this framework to develop data collection instruments, we believe that information on the results of the implementation of SGP grants will be focused both on the development of the media organization and the impact media development has on its constituent listeners/viewers. 1.5.3 Grants to Enhance Private Sector Development – Within this EG area, there are four types of grants that were provided: microfinance, youth entrepreneurship, financial management training, and direct support to businesses. The organizations involved in implementing these grants are as follow. owned, but independently operated, radio and TV stations. It also addressed access. It did not disaggregate all its findings by gender, and it did not go deeply into the types of programming in demand by listeners beyond news, music, sports and related broadcast items. The study addressed only indirectly the radio and TV programming on local elections, changes in government, civic education, and the like when the study asked the types of programming listeners preferred. The study also did not address the possibility of utilizing radio and other media for educational purposes in the areas of economic growth and literacy/ numeracy. The current study will address these types of programming as that is what the SGP sought to improve. 18 Microfinance Approximately 80% of Timorese are rural, unemployment is very high, and a very large percentage of Timorese earn an income through the operation of a microenterprise. The first grant that DAI provided to understand more about microfinance in Timor-Leste was to Catholic Releif Services to study the extent, capacity and operations of the MFI sector. In response to this study as well as an understanding of the need for microentrepreneurs to access loans, DAI provided several grants both for the extension of microfinance services and to the umbrella organization responsible for oversight. They are as follows: Table 7: Organizations Receiving Microfinance Grants Organization District Purpose of Grant Period of Grant Catholic Relief Services Timor-Leste 1) Dili Baucau Viqueque 2) Baucau Viqueque 1) Conduct microfinance sector assessment 2) Expansion of MFI activities and improvement of loan portfolio 1) November 23, 2004 – February 15, 2005 2) April 1, 2005 – April 30, 2007 Moris Rasik 1) Aileu Ainaro Bobonaro Covalima Ermera Liquica Maufahi 2) Aileu Ainaro Bobonaro Covalima Ermera Manufahi Liquica Oecussi 3) Baucau Lautem Manatutu Viqueque 1) Expansion to two districts; capacity building 2) Installation of new IT system to track clients 3) Expansion of services to Eastern districts 1) March 18 – September 19, 2005 2) January 16 – May 30, 2006 3) March 1, 2007 – March 1, 2008 Association of Microfinance Institutions in Timor￾Leste 1) Dili 2) Covalima 3) Dili 4) Dili 1) Managerial capacity building 2) Rapid appraisal study of how to reach more poor clients 3) Operational support and capacity building on use of common performance 1) June 15 – November 15, 2005 2) August 12 – September 5, 2005 3) February 20 – September 30, 2006 4) December 18, 2006 – June 12, 2008 19 monitoring system 4) Improve advocacy services, training, standards development and coordination across MFIs Ministry of Development 1) Dili 1) Training in microfinance for government employees (by CGAP and UNCDF) 1) January 19 – February 19, 2006 Although only two organizations received programmatic support, the grant provided to the Ministry of Development attests to the importance of microfinance to the economic development of the country. In this part of the research, we will build upon the study conducted by CRS and ascertain the impact of the expansion of Moris Rasik’s operations as well as of the increasing professionalization of the industry by AMITL Youth Entrepreneurship (Pilot Projects) With so many youth out of work and vulnerable to factional disputes, SGP awarded a number of organizations funding to launch pilot programs to help train youth in various types of income generation activities. Table 8 details those grants: Table 8: Organizations Receiving Youth Pilot Project Program Grants18 Organization District Purpose of Grant Period of Grant Timorese Young Entrepreneurship Initiative Baucau Liquica Training in Horticulture and Trade Skills March 16 – July 15, 2006 Fundacao ETADeP Manufahi Manatuto Training in Agriculture August 17 – October 30, 2007 Rotaract Club of Dili Dili Business Skills Training for Young Street Vendors October 15, 2007 – February 2, 2008 Financial Management and Training (Business & Entrepreneurship Training)19 Pilot training for youth has been augmented by grants in this category, although other beneficiaries (i.e., women) are included in these grants also: 18 Excludes foundational and contributory grants. 19 It should be noted that Literacy and Numeracy training awards were revised in the last year to include economically active participants who, after the first six months, would participate in business and entrepreneurship training. For this purpose, one grant that was implemented in Oecusse is included in this table. The other literacy and numeracy awards will be considered in another study. 20 Table 9: Organizations Receiving Business & Entrepreneurship Program Grants20 Organization District Purpose of Grant Period of Grant Rede Feto Dili Aileu Ainaro Lautem Liquica Rural Women’s Economic Product Promotion October 12 – November 12, 2007 East Timor Development Agency Dili Basic Bookkeeping TOT for NGOs February 15 – October 31, 2007 Rotaract Club of Dili Ermera Liquica Financial Education April 1 – December 16, 2006 Alola Foundation Dili Cap. Dev. For Handicraft Production and Marketing October 10, 2007 – January 31, 2008 Dili Institute of Technology Dili Dili Job Fair: Connecting the Unemployed to Potential Employers October 25 – December 31, 2007 Centro Feto Enclave Oecussi Basic Bookkeeping & Money Management for Women’s Groups in Pante Makassar November 16, 2007 – May 16, 2008 Fundacao Caminho Manufahi HH Money Management and Bookkeeping Training for Small Businesses May 14 – July 30, 2007 Forum Peduli Wanita Oecussi Business & Entrepreneurship follow-on from L&N, Sequence II February 4 – June 4, 2008 Direct Support to Businesses (Private Sector Development) Under this heading, DAI made several awards to individual and/or groups of businesses to improve their operations: Table 10: Organizations Receiving Private Sector Development Program Grants21 Organization District Purpose of Grant Period of Grant University of Hawaii Baucau Training in Candlenut Oil Processing Enterprises (COPE) February 15 – October 31, 2006 PB Moliliario Baucau TA for Production Enterprise May 4, 2007 – March 3, 2008 20 Excludes foundational and contributory grants. 21 Excludes foundational and contributory grants. 21 1.5.4 Nutrition Education Grants While each of the grantees noted in Table 11 had a different set of beneficiaries, it is of interest to know which approach had the best results in terms of participants adopting the behavior changes inherent in the training. Table 11: Organizations Receiving Nutrition Education Program Grants22 Organization District Purpose of Grant Period of Grant HIAM Health Dili Follow-up Care for Children with Severe Malnutrition Deceber 20, 2004 – March 18, 2005 Alola Foundation Dili Seminar Series on Infant and Young Child Feeding Practices January 17 – March 17, 2008 Grupo Suporta Inan Baucau Nutrition Education for Pregnant Women & Women with Children Under 5 January 7 – May ;7, 2008 Care International Timor Leste Bobonaro Covalima Liquica Nutrition Education Support to MOH project June 1 – October 31, 2007 Masino Foundation Timor Lorosa’e Lautem Nutrition Education June 27, 2007 – January 11, 2008 Medicos Do Mundo Lautem Nutrition Program September 10, 2007 – February 10, 2008 Pastoral Da Crianca Manatuto Nutrition Education for Mothers in Laleia December 13, 2007 – May 13, 2008 Sharis Haburas Communidade Manufahi Nutrition Education for School Children January 28 – June 28, 2008 1.6 Research Site Choices and Summary - On the basis of the diversity of SGP awards in a given district and the geographic variability of each district, the team will collect data for the “mosaic” approach discussed below with grantees in Dili, Baucau, and a third district to be chosen with input from DAI and USAID.23 Data will also be collected for the three sub-studies in selected and/or representative districts in which the team can learn the most within the research time frame. To conduct the “mosaic” research, the team will travel to the districts together. To collect data for each of the sub-studies, the team will split to go to each of their respective locations, although members of the team will go to a district together if the site selection for each study allows for it. Site selection is very difficult in the SGP case because of the widespread nature of awards, travel difficulties, and the fact that many grants were completed some time ago. While the latter point 22 Excludes foundational and contributory grants. 23 There are many similarities in the type and number of grants provided to other districts; hence, the team needs further input to help them select one other site. 22 is most interesting in conducting impact assessments, personnel change and institutional memory may be a challenge. Moreover, recent violence may preclude our visiting certain districts. Hence, we will rely on insights from DAI and USAID to determine the specific organizations in each district we will interview. 2. EVALUATION RESEARCH APPROACH In this section we attempt to find “logical” grouping into which the grants fall so as to reduce the amount of time needed for data collection and to meet the terms of the evaluation budget. More importantly, however, the groupings seek to consider SGP grants more holistically so that a roadmap can be developed and synergies ascertained. 2.1 Conceptual Frameworks Several factors must be taken into account in the design of the research to be conducted in this evaluation: • Changing of and Diversity within Strategic Focus Areas • Changing Strategic Objectives • Changing Performance Management Plan and Indicators • Geographic Diversity • Grantee Diversity Concepts are put forward below as the basis of several foci of research. These include: 1) a conceptual “roadmap” in which project contributions to the achievement of USAID’s Strategic Objectives are determined (moving from the results of OTI to the implementation of USAID’s operational plan); 2) a geographically-based “mosaic” of projects in which the synergies developed through the implementation of projects in selected locations under all three SOs are examined (specific districts to be purposefully selected as a cluster sample); and 3) a ranking of impact each organization has had in accordance with the number and amount of grants it has received and the duration of the sum of the grants (testing the hypothesis that the longer support is received, the stronger the impact may be). This point is particularly significant because of the low level of capacity most Timorese organizations are said to have; i.e., the more grants received by one organization, hypothetically, the more capacity building they received from DAI to write, implement and manage those grants. In the case of grants provided to media organizations, those supporting private sector development (i.e., microfinance organizations, youth entrepreneurship (pilots), financial management training, and direct support to businesses), and nutrition education, separate data collection strategies will be developed although all activities within each of the conceptual framework will be considered. 23 2.1.1 Conceptual Roadmap – Moving Toward Satisfying Strategic Objectives (Capacity Building and Beyond) To move USAID’s D&G, EG and Health/Investment in People agendas forward in such a manner as to achieve the respective results of Key Foundations of Good Governance Strengthened, Accelerated Economic Growth, and Improved Health of the Timorese People (and, later, Investment in People), Especially Women and Children at Risk,24 a number of foundational activities had to take place. For the most part, the foundation for the development of these SOs was laid during the OTI contract managed by DAI. However, the 15 months of implementation of this contract was insufficient to carry out all the needed reconstruction and refurbishing activities necessary for many government institutions, NGOs and other agencies to be able to launch programs that could be successful. Moreover, many agencies were ill-prepared to undertake a range of activities due to shortfalls in office equipment and furnishings, a lack of understanding of how to prepare project documents, and the like. Hence a number of foundational grants were awarded in several non-programmatic categories. Awards were also made to provide operational support to bridge budgetary gaps, provide technical assistants by external consultants to improve organizational efficiency and effectiveness, to conduct feasibility studies with a potential for increased private sector development, and to pay for international trips by leaders of government agencies and NGOs in which capacity building and promotion were the focus. These “contributory” grants were all in the nature of supporting the development of more programmatic undertakings, some of which were funded by DAI/SGP. Programmatic grants included, among many others, developing media campaigns in preparation for the suku elections, holding literacy and numeracy classes, implementing anti-malarial campaigns, supporting different types of initiatives to enhance youth employment, and the like. An integral, if not unsaid, part of this grant process was the capacity building provided by DAI/SGP to assist grantees in being successful. The ongoing support provided by DAI/SGP in the form of monitoring and evaluating further built the capacity of grantees. It is the intention of this research to ascertain the extent of this capacity building support to determine the type of impact DAI itself had on grantees’ ability to deliver project results. The conceptual roadmap to be constructed in this research is an ex poste facto roadmap in which the team will consider the many steps that must be taken in building the society the USAID strategic plan envisions. To construct this roadmap, however, the work not only of DAI/SGP must be considered, but also of other USAID projects and hose of other donors. 2.1.2 Geographic Mosaic – Defining Synergies When reviewing the grant locations, grantees, and projects funded, it became clear that: 1) there were more grants awarded in some districts (a “cluster”) than others; and 2) several organizations received multiple grants. It is of interest to the project to determine if and what type of synergies were created between and among grantees, and between and among other donors and programs in selected locations. The choice of USAID’s SOs implied, and sometimes made explicit, the linkage between the SOs, especially D&G and EG, and that, the mission goal of “A New Nation 24 USAID East Timor. Performance Management Plan. USAID/East Timor. FY 2005-FY 2009. Dili, East Timor: USAID, July 2005, p. 4. 24 Moving Forward” could best be accomplished if synergies were developed among and between projects under D&G and EG SOs, other USAID-funded projects, and projects funded by other donors. To test this hypothesis, and to determine the nature of any synergies developed, the methodology employed in this evaluation will lead the team to selecting certain “clusters” of grants (in specific locations) to be evaluated not only for their individual results, but also for the type of composite picture created as a result of several grants being awarded in the same location. Moreover, the team will consider how SGP grantees interact with other donors providing funding for related projects in the given locations. By taking this approach, the type of synergies created can be determined and whether there were any “gaps” in funding in a given location. Our working assumption is that the best synergistic results could be obtained when institutions and agencies working on each SO learn from each other in the field and build upon each other’s successes.25 Absent this type of integration, the results of all grant efforts will not amount to more than the sum of each of the parts. Constructing what this mosaic looks like will assist USAID in being more intentional in implementing a small grants program in the future because it will be able to capitalize on lessons learned. When selecting mosaic locations, attention will be paid primarily to programmatic grants awarded rather than those that might be foundational (reconstruction and refurbishment; purchase of equipment and supplies; printing and copying) or contributory (providing operational support, paying for an external consultant conducting research and/or feasibility studies, and sponsoring international trips). The mosaic locations where these types of grants predominate will not be selected. It should be borne in mind, however, that those organizations receiving foundational and/or contributory grants will be included in the research to develop both the roadmap and the mosaic; the foundational and contributory grants will simply not be used as site selection criteria. 2.1.3 Impact Ranking – Duration and Funding Many grantees received more than one grant, sometimes to continue the same activity launched with the first grant and sometimes to undertake a different activity. Grants were awarded in different amounts according to the objectives of each grant. It is the intention of this evaluation to address the impact of the programs implemented over longer vs. shorter periods of time, and with lesser vs. greater amounts of resources to determine which of these variables makes a difference. Impact ranking will be undertaken within the cluster sampling of the mosaic methodology and within each of the sub-studies, if applicable. Hence, in a given location where a grantee has been implementing its projects over successive grant cycles, a grantee might be able to create greater impact than a grantee implementing its project over one grant cycle. It is assumed that with each successive grant, the implementing agency becomes more adept, knowledgeable, and capable, thus creating the potential to have a greater impact. When successive grants are awarded to a single organization, that succession, in and of itself, is a capacity building mechanism that creates greater skill in the design, implementation, and achievement of outcomes. 25 In inquiring into the management of SGP by DAI, the team will determine to what extent DAI fostered inter￾grantee, and inter-project communication to create synergies. 25 2.1.4 Media Sub-Study – What Do People Know? What Did People Do? Since media capacity building and programming played such a central role in guiding the awarding of grants under the D&G SO, a separate sub-evaluation will be designed (following on the Timor-Leste National Media Survey undertaken by Fondation Hirondelle (Media for Peace and Human Dignity) in May 2007. The question guiding this sub-evaluation is: What types of action did people take as a result of various media, especially radio, programming? A separate interview schedule will be developed to be implemented with representatives of media organizations and with listeners to determine the outcomes achieved from the programming offered. This sub-evaluation will also take place among the clusters chosen for the mosaic approach as well as in selected other locations (to obtain a deeper understanding of the impact of media projects). As many of the grants to media created a capability of broadcasting in ever-widening circles – to the extent that many grants produced country-wide broadcasting – site selection will favor those areas where media efforts were strong and broadcasting targeted different purposes (e.g., voting, explanation of changes in the law, etc.). 2.1.5 Private Sector Development Sub-Study This sub-study has four parts: microfinance, youth entrepreneurship, financial management training, and direct support to business. Each part will utilize a slightly different interview schedule and/or data collection process. Microfinance Data on the depth and breadth of MFIs as well as their relation to economic growth are well presented in the SGP grant awarded to CRS to undertake a microfinance sector assessment. Finalized in January 2005,26 the report states (p. 19): • Poverty increases from East to West in Timor-Leste. Oecussi, Bobonaro and Covalima in the West are home to 20% of the total population, but 25% of the poor. Baucau, Lautem and Viqueque in the East account for 25% of the total population, but less that 20% of the poor. • Poverty increases with altitude – the central highlands are home to the highest number of poor. • Poverty is higher in rural areas than in urban areas. While 80% of the population live in rural areas, so do 75% of the poor. The rural central Timor-Leste is the poorest area, followed by the rural west and urban centers other than Dili and Baucau. Annex 5 to the report presents several maps showing the geographic outreach of different types of financial institutions and the concentration of these institutions more in the western than the eastern part of the country (with the exception of Baucau which has a large urban population). The report, with the accompanying annex, also correlates the locations of poverty and where income generation is most problematic because of total reliance on subsistence agriculture. Hence, the more alternative means (beyond agriculture) a household has to increase its income, 26 Lene Hansen and Novanto Agus. Financial Services Sector Assessment and Seminar/Workshop on “The Millennium Development Goals and the Eradication of Poverty – the Role of Microfinance in Timor-Leste.” Timor￾Leste: USAID, January 10, 2005. 26 either through a microenterprise or employment, the greater the demand for financial services. Financial service providers merely followed the demand. In the interests of expanding MFI coverage, DAI provided three grants to Moris Rasik to expand their operations to Liquiqua and Ermera before the 2006 disturbances, and after the disturbances in 2007 to the four Eastern Districts of Baucau, Lautem, Manatutu, and Viqueque. It is the operation of these branch offices that are the subject of our questioning. The operations of microfinance organizations (MFI) are unique in that they provide opportunities for the poor to gain access to and manage their finances more effectively. The sustainable operation of an MFI is dependent on various internal rules, regulations and controls, as well as external factors such as the domain of in which enterprise clients operate, their degree of poverty (including education, income, health and other factors), and the appropriate usage of funds borrowed. One grantee received a succession of grants for expansion and operational sustainability. The sub-study will address the effective management of this MFI as well as the impact the MFI has had on its clients (including satisfaction, loan use, and recommendations for future activities). Hence, data will be collected from both MFI management and clients. Youth Entrepreneurship (Pilots) Youth entrepreneurship projects have largely focused on horticulture/agriculture production, trade skills, and bookkeeping (crossing with Business & Entrepreneurship training grants). The intention of these projects was to provide skills training to unemployed youth who could then either get a job in the appropriate sector or establish a small business. The sub-study will select, as time permits, the organizations from which to collect data and then identify the trainees to determine if they are using what they learned and have since obtained employment or launched their own businesses. Business & Entrepreneurship Training Business & Entrepreneurship Training was provided by at least seven organizations in at least eight districts. Those organizations that provided specific business skills training will be selected for evaluation (as time permits). In each case, data will be collected from project management on the curriculum and support provided and from the trainees to determine if their enterprises have improved as a result of the training. Private Sector Development It appears that only one private company received a private sector development grant, and the grant paid for technical assistance provided by a consultant. However, the University of Hawaii, after conducting a feasibility study on candlenut oil processing (under an SGP grant), provided training on this topic. It might be more beneficial to collect data from trainees to determine how they are managing their oil processing businesses. The selection of grantees under this category will be discussed further with DAI and USAID as USAID has funded a separate private sector development project. 2.1.6 Nutrition Education Sub-Study At least eight organizations received funding for different types of nutrition education in eight districts. While some of these grantees will be covered in the “mosaic” site selection, others will 27 be selected (as time permits) for further data collection. Both the grantee and trainees will be interviewed to determine the structure/content of the project as well as the results achieved. 2.1.7 Summary of Approach It is envisioned that no less than three locations will be chosen – Dili, Baucau and one other – where the mosaic approach will be implemented. Other sites will be selected for the sub-study. In the analysis of both of these types of study, both the roadmap and the impact ranking will emerge. 2.2 RESEARCH STRATEGY AND DATA COLLECTION 2.2.1 Research Strategy - Owing to the many different types of grants and SFAs, the research strategy will follow the conceptual frameworks noted above, namely: roadmap, geographic mosaic, impact ranking; and looking specifically at media, private sector development and nutrition education. The mosaic strategy entails choosing three locations - Dili, Baucau, and one other site - to collect data from all grantees on both the management of their activities and the impact on their intended beneficiaries. As each organization is interviewed, it will be determined whether any synergies with other grantees, other USAID-funded projects, and other donor-funded projects – especially those with similar goals/objectives – have been created. In this way, a mosaic of interrelationships can be created, if the linkages were formed. The criteria for site selection include: 1) diversity of programmatic grants awarded in the district (including some under each SO); 2) the number of programmatic grants awarded in each district; 3) physical accessibility (combined with time available for data collection); 4) unique issues to address (to be determined); 5) availability of grantees and beneficiaries to be interviewed (many grants were awarded and completed a few years ago); and 6) DAI/USAID advice. Within each location, each team member will be deployed to collect data at a specific set of grantees. For instance, one expatriate team member will collect data on health/nutrition, one on D&G and one on EG. The two Timorese team members will assist each expatriate team member in accordance with his expertise. By so doing, each team member will develop expertise in the grants awarded under a specific SO, and will then be able to discuss the impact of each SO more broadly. Each expatriate team member will also be assigned one area of supplemental research – media, private sector development, and nutrition education, and be supported by the Timorese team members. The entire team will go to each district and collect data at their respective organizations. At the end of each day, the team will gather to debrief and to begin constructing the mosaic of relationships. Each team member will be responsible for transcribing field notes, for delivering findings in their respective SO in the USAID/DAI debriefing session, and for writing their respective sections of the report. The team will decide, collectively, the structure of presentation (within the guidelines provided in the SOW). Each team member may also be assigned to construct appendices, as needed. The Team Leader will be responsible for assembling all the pieces of the report and finalizing it for submission. 28 2.2 Categories of Data Collection Instruments – A number of instruments will be developed to collect data from different types of organizations and their beneficiaries: 2.2.1 Grantees – Several concerns inform the type of instruments to be utilized in data collection with grantees: 1) management of the project; 2) capacity building; 3) specific organizational issues; 4) achievement of goals and objectives; and 5) linkages to/synergies with other grantees. Several data collection instruments will be developed and utilized: 1) individual and focus group interviews with selected grantee personnel; and 2) implementation of a SWOCA (modified form of Strengths, Weaknesses, Opportunities, Threats, now transformed into Strengths, Weaknesses, Opportunities, Constraints, and Achievements), paying particular attention in asking follow-up probing questions on how these might have changed over the course of the grant. Specific interview schedules will be developed for the interviews in keeping with the goals and objectives of each project. A set of specific questions will be posed of all grantees, while supplemental questions will be designed for particular types of organizations (e.g., media, literacy/numeracy, and microfinance). 2.2.2 Intended Beneficiaries – Those who benefited from the actions of a particular organization will be organized into focus groups for a group interview. In general, they will be asked as to their satisfaction, how they would change the intervention if they could, and what they did with what they learned, i.e., how their behaviors (lives) have changed as a result of the particular intervention. Where appropriate, a picture-drawing methodology will be employed in which beneficiaries will be asked to draw themselves before and after the intervention. (Where time permits, and it is appropriate, beneficiaries are asked to draw a third picture that depicts themselves some time in the future in response to the question: What would you like your life to be like [concerning the particular type of intervention being discussed] in a year, two years, or 5 years from now?) In debriefing, beneficiaries will be asked to describe the different aspects of their lives that have changed as a result of the intervention.27 As with the grantees, a set of questions will be common to all beneficiaries, with supplemental questions posed to reflect the specific type of intervention. 2.2.3 Government Employees – While government departments and employees can be either grantees or beneficiaries, a specific type of questioning needs to be pursued that targets their particular responsibilities for policy-making and implementation. Hence, the set of standardized questions will be posed (for comparability of findings) but they will be supplemented with the specific impact questions directed at what they did with what they learned, what types of initiatives were launched, and how each participant – whether grantee or beneficiary – shared what they learned with other members of government. 2.3 Specific Data Collection Instruments 2.3.1 CORE QUESTIONS OF ALL GRANTEES Questions posed to all grantees will follow the outline of a logic model: 1) What were the goals and objectives, and who were the participants in the grants you received? 27 This data collection technique has proven to be very successful, especially among people who are illiterate. By drawing a self-portrait, beneficiaries can choose to draw themselves as they actually are or by using different symbols. All of this is debriefed after the drawings have been completed. 29 2) What kinds of inputs did you provide to promote the success of your projects? 3) What kinds of activities did you undertake to achieve the goals and objectives of your projects? 4) What kind of outputs were produced by the participants? 5) Do you believe your project goals and objectives were reached? How do you think they fit in the overall USAID goal of expending good governance? Explain 6) What are the current Strengths, Weaknesses, Opportunities, Constraints and Achievements of your organization? 7) If you were to undertake this type of project again, what would you change? 8) Do you receive external funding from any other organization? Which one? Do you need other resources to continue your work? What further is needed? To do what? 9) How do you interact in this location with other SGP grantees? With other USAID￾funded projects? With other donor-funded projects? 10) What types of capacity building did DAI provide you in the process of writing your grant proposal, implementing your project, and monitoring/evaluating your project? 11) Is there anything else you would like to tell me about this project? Other questions will be developed by the team in accordance with the type of project funded. 2.3.2. MEDIA DATA COLLECTION Grantees - The following specific questions are to be posed of media grantees in the specific mosaic locations as well as others (to be identified) (a total of 18 organizations have received grants, 9 of which received more than one grant; some focused on one district while others sought to create an impact throughout the country). Programmatic grants generally fall into two categories: 1) capacity building/institutional strengthening; and 2) expansion and broadcasting of a wider range of programs to a wider audience. Capacity Building/Institutional Strengthening 1) Describe the state of journalistic reporting that you wanted to improve with the SGP award(s) you received. What did you want to change/improve? 2) How did the training you provide improve communications with constituencies? What challenges were the most difficult to overcome in this area? 3) How long was the training provided? Who provided it? Who and how many participants? What were the goals and objectives of the training? What was the curriculum of the training? What type of training evaluation was undertaken? What were some of the participants’ comments on the evaluation? 4) How did the participants use what they learned? What results were achieved when they did so? 5) Did participants need any follow-on support to be able to implement what they learned? If yes, what was it? Was the support provided? By whom? For how long? 6) What type of information network has been developed as a result of the training you have provided? Expansion and Delivery of Programs 1) How did you decide into which districts to expand? Did you conduct any type of market study? 30 2) How did you decide on the types of programming to offer in each district? 3) How did you develop local capacity to develop and deliver the types of programming you offered? 4) On what days did you broadcast these programs? At what times did you broadcast these programs? How long did each program last? Did you repeat the program? When and how often? 5) In what language did you broadcast these programs? 6) Explain the content and the goal of the programming delivered on the 1) suku and aldeia elections, 2) crisis, 3) presidential and parliamentary elections, 4) civic education (choose according to grantee). 7) What kinds of challenges did you face in offering these types of programming? 8) Did you conduct any type of listenership study after programs were broadcast? If so, please discuss your findings. 9) Did you provide any type of interactive format for listeners when you broadcast the programs? If yes, what was the response? 10) What other types of programming do you broadcast? How frequently? 11) What do you believe is the impact of your SGP-funded programming on the listening audience? That is, how does your programming make a difference? Listenership – A sample of listeners to radio programs offered will be identified to answer the following questions: 1) At what times of the day do you generally listen to radio? 2) With whom do you generally listen? Do you discuss what you have heard with them after the broadcast? 3) Have you listened to the programming on: 1) suku and aldeia elections, 2) crisis, 3) presidential and parliamentary elections, 4) civic education (choose according to grantee)? What is your opinion about the programming? (prompt: one-sided or fair; provided information you needed or needed more information; proper length of broadcast, etc.). 4) Did you take any action after listening to the broadcast? If yes, what did you do? If no, did you need more information to take action? What did you need? 5) What types of programming do you want this radio station to offer? For what purpose? 2.3.3 PRIVATE SECTOR DEVELOPMENT DATA COLLECTION TOOLS Microfinance Grantees Only two MFIs and one MFI association received SGP support. The former two were to expand services, and the latter one was for operational support and a number of activities on behalf of MFIs. DAI selected several organizations to become Resource Organizations. In the process of selection, the major MFI may have been weeded out. The team will consider the reasons for this and determine, with the advice of DAI, whether this organization should be interviewed. Clients Client satisfaction will be measured in focus groups where the following matrix will be filled in: 31 Current Practice What Clients Like About It What Clients Dislike About It Recommendations for Improvement Why Number Voting for Change Youth Entrepreneurship Grantees In addition to the organizational questions asked of all grantees, the following questions will be posed: • Tell me about the curriculum used in providing the training to youth – what type of learning did it foster (i.e., hands-on, memorization, etc.)? • What types of difficulties did you encounter in delivering the curriculum? • What type of placement support did you provide the youth who participated in the training? • What level of skill did the participants achieve by the end of the training? Were participants “ready” for employment? • What type of follow-up did you undertake with each participant? • What would you change in the training if you had a chance to offer it again? Participants • Tell me about the training you receive in [horticulture, agriculture, business skills]? What did you learn? • Were you able to put to use what you learned? What of the training, in particular, helped you to use what you learned? What prevented you from using what you learned? • What type of job were you able to find after the training? Did the people who provided you the training help you find this job? If yes, what did they do? • Are you satisfied with the training you received? What other types of training do you think you need to be employed? To start your own business? • What would you change about the whole training experience to help you to be successful in getting a job or starting your own business? Financial Management Training Grantees In addition to the organizational questions asked of all grantees, the following questions will be posed: • Tell me about the support/curriculum you provided – what type of learning did it foster (i.e., hands-on, memorization, etc.)? • What types of difficulties did you encounter in delivering the support/curriculum? • What level of skill did the participants achieve by the end of the training? Were participants “ready” to use what they had learned? • What type of follow-up did you undertake with each participant? • What would you change in the training if you had a chance to offer it again? 32 Recipients/Participants • Tell me about the support and/or training you received to help you in your business. • Were you able to use what you learned in your business? What, in particular, were you able to use? How did this help you in your business? • Has your income increased as a result of using what you have learned? How? • If another training or support could be provided to you, what would you need? • If you could change anything about the support/training you were provided, what would it be? Direct Support to Businesses Since there was only one grant to a business listed on the chart prepared by DAI, a field decision will have to be made as to whether or not it would be worthwhile to collect data from this business. 2.3.4 NUTRITION EDUCATION Data collection instruments and/or questions to be developed by Nancy Sheehan. 3. SUGGESTED SCHEDULE OF ACTIVITIES April 4-5 – Meeting with DAI and Team Members; finalization of methodology and data collection instruments April 7-8 – Meetings with DAI, USAID CTO and SO Team Managers April 9-12 – Team Data Collection in Dili April 14 – 16 – Team Data Collection in Baucau April 17 – 19 – Team Data Collection in 3rd Location April 21-22 – Data Collection for Sub-Studies in Various Locations April 23-25 – Writing of Draft Report April 26 – Writing of Appendices and Supporting Documents; preparation for Briefing on Findings April 28 – Briefing on Findings April 29 – May 3 – Finalization of Report; Submission of Final Report The following types of support will be needed: • Vehicles for all data collection trips • Interpreters for all data collection trips • Hotel reservations for all nights to be spent in Dili and in other data collection points • Satellite phones for each vehicle to remain in touch with the head office of DAI • Office space in which to work together and/or meet as a team 33 APPENDIX D BIBLIOGRAPHY Anis, Katy. 2007. Assessment of the Effectiveness of Literacy and Numeracy Programs in Timor-Leste. Bethesda, MD: Development Alternatives, Inc. (December 26, 2007). Aube, Thierry, Fabrizio Cesaretti, Filippo Fossi, and Yvonne Forsen. 2007. FAO/WFP Crop and Food Supply Assessment Mission to Timor-Leste. Rome: Food and Agriculture Organization of the United Nations, and World Food Programme (June 21). Becker, Gerald. 2007. Needs Assessments of Resource Organizations in Timor-Leste. Dili, Timor-Leste: Development Alternatives Inc. (December). Brady, Cynthia, Laurentina “Mica” Barreto Soares, Antonio da Conceicao, Steve Darvill, Sidonio Freitas, Ana Guterres, David G. Timberman, and Gavin Williams. 2006. A Report for USAID Timor-Leste Based on a Joint USAID-AUSAID Conflict Vulnerability Assessment: The Crisis in Timor-Leste: Causes, Consequences and Options for Conflict Management and Mitigation. Dili, Timor-Leste: USAID (November). DAI. 2005a. USAID East Timor Small Grants Program Semi-annual Report: October 2004- March 2005. Dili, Timor-Leste: Development Alternatives Inc. (April). DAI. 2005b. USAID Timor-Leste Small Grants Program Semi-annual Report: April-September 2005. Dili, Timor-Leste: Development Alternatives Inc. (October). DAI. 2005c. Report on Options for Financial Service Provision to Second-tier Microfinance Clients in East Timor. Dili, Timor-Leste: Development Alternatives Inc. DAI. 2006a. USAID Timor-Leste Small Grants Program Semi-annual Report: October 2005- March 2006. Dili, Timor-Leste: Development Alternatives Inc. (April). DAI. 2006b. USAID Timor-Leste Small Grants Program Semi-annual Report: April 2006 - September 2006. Dili, Timor-Leste: Development Alternatives Inc. (October 15). DAI. 2007a. USAID Timor-Leste Small Grants Program Semi-annual Report: October 2006- March 2007. Dili, Timor-Leste: Development Alternatives Inc. (April 16). DAI. 2007b. USAID Timor-Leste Small Grants Program Semi-annual Report: April 2007 – September 2007.. Dili, Timor-Leste: Development Alternatives Inc. (October 16). DAI. 2008. USAID Timor-Leste Small Grants Program Semi-annual Report: October 2007- March 2008. Dili, Timor-Leste: Development Alternatives Inc. (April). DAI. 2007. USAID Small Grants Program – Grant Information Sheet for Final Evaluation Team. Dili, Timor-Leste: Development Alternatives Inc. (February). 1 Day, Melissa. 2007. Financial Services in Timor-Leste: An Update (Draft). Dili: USAID. East Timor Development Agency. 2007. Fatumaca and Becora Technical Schools Graduates Survey (2004-2006) Final Report. Dili: ETDA/USAID. Engel, Rebecca E. 2007. The Building of Timor-Leste: International Contributions to a Fragile State. Dili, Timor-Leste: Center for International Conflict Resolution (CICR), School of International and Public Affairs, Columbia University, BELUN, and Ford Foundation (April). FONGTIL. 2007. Annual Report. Dili: Tija Tchai & Assoicates (January-December). FONGTIL. 2008. Kriteriu Membership FONGTIL, Code Etika NGO non-Partisan ba Membru FONGTIL, Papel No Fungsaun Bod Iha Secretariadu Fongtil, Definisaun Sosiadade Sivil, ONG nacional, ONG Lokal, ONG Internacional. Dili (April). Government of Timor-Leste. 2008. Working Together to Build the Foundations for Peace and Stability and Improve Livelihoods of Timorese Citizens. 2008 National Priorities. Dili: GOTL. Hansen, Lene,and Novanto Agus. 2005. Financial Services Sector Assessment and Seminar/Workshop on “The Millennium Development Goals and the Eradication of Poverty – the Role of Microfinance in Timor-Leste”. Final Report. Dili, Timor-Leste: Association of Microfinance Institutions in Timor-Leste (AMFITIL) (January 10). Hoekman, N., S. Thompson, and M. Mercer. Health Alliance International: PVO Child Survival and Health Grants Program, 2008. Third Annual Report for Improving Maternal and Newborn Health in Timor-Leste. Dili: HAI. (October) Junior Achievement Timor-Leste. Primary School Program. Preparing the Young People of Timor-Leste for a Financially Stable Future. 2007. Pilot Program Report. Dili: Junior Achievement, USAID. Mason, John P., John Doyle, Joao Noronha, Luisa Cardoso, Filipe do Costa, and Margarith Araujo. 2002. End-of-Project – Participatory Evaluation. Building Empowerment, Leadership, and Engagement (BELE) & Transition Engagement for Population Support II (TEPS II) Programs. East Timor, for U.S. Agency for International Development Office of Transition Initiatives and Development Alternatives, Inc. (AOT-I-01-98-00199-00). Dili, East Timor: USAID/OTI/DAI (January 28). Ministerio da Administracao Estatal, Instituto Nacional da Adminstracao Publica (INAP). 2006. Kursus Administrasi dan Manajemen Kepemimpinan dalam Rangka Meningkatkan Kemampuna Para Lider-Lider Komunictario di 13 District Timor Loro Sa’e. Dili: INAP (February). Ministry of Health, Timor-Leste. 2004. National Nutrition Strategy. Dili: MOH, 2007 (July) Ministry of Health, Timor-Leste. 2004, National Strategy for Health Promotion: 2004-2010. Dili: MOH (August). 2 Ministry of Health, Timor-Leste. 2007a. “Community Consultation” on Child Health Practices in Timor-Leste. Dili: TAIS/USAID (September). Ministry of Health Timor-Leste. 2007b. Mid-term Report, July 2005 – July 2007. Dili: TAIS/USAID (July) Mize, Lucy S., Thompson, S., Hulme, J., Hoekman, N., Mercer, M., Bucens, I., Health Alliance International: Improving Maternal and Newborn Health in Timor-Leste. 2006. Mid-Term Evaluation Report. Dili: HAI. Jornal Labarik. 2008. Relatoriu: Evaluasiun Distribuisauan Jornal Labarik. Dili: Marsu. NGO Forum, National Youth Council, Student Solidarity Forum, and Rede Feto. Civic Education Needs Assessment Report. Dili: UNDP. Report Third Congress, Timor Leste Community Radio Association (TLCRA/ARKTL). 2007. Dili: Radio Rakambia, Rala, TLMDC, CRC, UNICEF, USAID, and ICfJ (18-20 October). Republica Democratica de Timor-Leste Ministerio da Adminstracao Estatal E Ordenamento do Territorio. 2008a. Policy Orientation Guidelines for Decentralization and Local Government in Timor-Leste. Dili (March). Republic Democratica de Timor-Leste. Ministry of State Administration and Territorial Management. 2008b. Timor-Leste Decentralization Strategic Framework: What needs to be done and how do we get there? Part 1. Dili (March). Scambary, James. A Survey of Gangs and Youth Groups in Dili, Timor L’Est. Dili: AusAid, 2006. Soares, Eduardo and Graham Mytton. 2007. Timor-Leste National Media Survey Final Report. Dili, Timor-Leste: Fondation Hirondelle, Media for Peace and Human Dignity (May). Tambun, Wayan. 2008. Peran NGO Internasional dalam Memperkuat Kapasitas NGO Nasional. Dili (April). Tuba Rai Metin. 2008. Financial Report as of February 2008. Dili: TRM. UNDP and UNCDF. 2008. Joint Programme. INFUSE – Inclusive Finance for the Under￾Served Economy (2008 – 2012). Proposal submitted to the Ministry of Economy and Development. Dili: April 2008. USAID. 2005. Fragile States Strategy. Washington, DC: USAID. USAID East Timor. 2005a. Performance Management Plan. USAID/East Timor FY 2005- FY2009. Dili, East Timor: USAID (July). 3 USAID East Timor. 2005b. USAID Strategic Plan for East Timor. A New Nation Moving Forward 2005-2009. Dili, East Timor: USAID USAID. 2008. U.S. Assistance to Timor-Leste: Nine Years of Partnership. PowerPoint presentation to the Timor-Leste Development Partners’ Meeting, 27-29 March 2008. Dili, Timor-Leste. University of Hawaii-Manoa, Oils of Aloha, Hawaii, USA. 2005. Feasibility Study for the Development of a Candlenut Micro-Enterprise in Timor-Leste. Dili, Timor-Leste: USAID Small Grants Program (February). W.K. Kellogg Foundation. 2004. Logic Model Development Guide. Using Logic Models to Bring Together Planning, Evaluation and Action. Battle Creek, MI: W. K. Kellogg Foundation. 4