PHOTO: USAID/SRI LANKA EVALUATION REPORT MID-TERM PERFORMANCE EVALUATION FOR THE DEVELOPMENT GRANTS PROGRAM PORTFOLIO IN SRI LANKA August 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Peter Malvicini, Upali Sedere, Radha Pathmanathan, and Pulendran Tharmendra for Social Impact, Inc. USAID Contract AID-486-I-14-00001; Task Order AID-383-TO-16-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Development Grants Program Mid-term Evaluation ii CONTENTS Executive Summary .......................................................................................................................................................... vi Project Background ..................................................................................................................................................... vi Evaluation Objectives .................................................................................................................................................. vi Evaluation Questions ................................................................................................................................................... vi Evaluation Methodology ............................................................................................................................................. vi Findings, Conclusions, and Lessons Learned ........................................................................................................ vii Background .......................................................................................................................................................................... 1 The Development Problem ........................................................................................................................................ 1 The United States Agency for International Development’s Response........................................................... 1 Evaluation Purpose and Questions ................................................................................................................................ 2 Evaluation Purpose ........................................................................................................................................................ 2 Evaluation Questions .................................................................................................................................................... 2 Evaluation Methodology ................................................................................................................................................... 3 Data Collection Methods ............................................................................................................................................ 3 Limitations....................................................................................................................................................................... 4 Team Composition ....................................................................................................................................................... 4 Findings, Conclusions, and Lessons Learned ............................................................................................................... 4 EQ 1 ................................................................................................................................................................................. 4 EQ 2 ................................................................................................................................................................................. 6 EQ 3 ............................................................................................................................................................................... 11 EQ 4 ............................................................................................................................................................................... 16 Recommendations ........................................................................................................................................................... 18 EQ 1 ...............................................................................................................................................................................18 EQ 2 ...............................................................................................................................................................................18 EQ 3 ...............................................................................................................................................................................19 EQ 4 ...............................................................................................................................................................................20 Annexes ............................................................................................................................................................................. 22 Annex I: Evaluation Statement of Work ................................................................................................................ 23 Annex II: List of DGP Grantees ............................................................................................................................... 31 Annex III: Data Collection Methods, Tools, Analysis, and Data Quality Assurance................................... 32 Annex IV: Desk Study Findings and Observations from Inbrief and Utilization Workshop ..................... 36 Annex V: Methodology and Limitations ................................................................................................................ 37 Annex VI: Summary of Findings from Desk Study, Internal Consultations, and Fieldwork ...................... 41 Annex VII: Documents Reviewed ........................................................................................................................... 44 Annex VIII: Data Collection Instruments .............................................................................................................. 47 Development Grants Program Mid-term Evaluation iii Annex IX: Key Informant Statistics by Data Collection Method ..................................................................... 57 Annex X: Persons Interviewed ................................................................................................................................ 59 Annex XI: Data Collection Schedule ...................................................................................................................... 72 Annex XII: Mini Survey Summary Report ............................................................................................................. 75 Annex XIII: Open System Characteristics ............................................................................................................ 77 Annex XIV: Success Factors in Designing Capacity Development Assistance ............................................. 78 Annex XV: Disclosure of any Conflicts of Interest ............................................................................................ 79 Development Grants Program Mid-term Evaluation iv ACRONYMS ADB Asian Development Bank AOR Agreement Officer’s Representative CBO Community-based Organization CD Capacity Development COR Contracting Officer’s Representative CPI Counterpart International CSO Civil Society Organization DGP Development Grants Program DPO Disabled People Organization E3 USAID Bureau for Economic Growth, Education, and Environment EG Economic Growth EQ Evaluation Question ET Evaluation Team FGD Focus Group Discussion FRC Family Rehabilitation Center GMSL Green Movement Sri Lanka GOSL Government of Sri Lanka GVP Governance and Vulnerable Populations HQ Headquarters HR Human Resources ICES International Center for Ethic Studies IDP Internally Displaced Person IP Implementing Partner JSAC Jaffna Social Action Center KII Key Informant Interview LKR Sri Lankan Rupee LRWHF Lanka Rain Water Harvesting Forum M&E Monitoring and Evaluation MSI Management Systems International NGO Non-governmental Organization OAA Office of Acquisition and Assistance OCA Organizational Capacity Assessment OFM Office of Financial Management PAD Project Appraisal Document PALTRA Partners in Alternative Training PPS Program and Policy Support Office PWD Persons with Disabilities RF Results Framework RFA Request for Application SLCDF Sri Lanka Center for Development Facilitation SPICE Support for Professional and Institutional Capacity Enhancement UNDP United Nations Development Program USAID United States Agency for International Development Development Grants Program Mid-term Evaluation v USD United States Dollars WASH Water, Sanitation, and Hygiene Development Grants Program Mid-term Evaluation vi EXECUTIVE SUMMARY PROJECT BACKGROUND Sri Lanka suffered from a 26-year civil war—displacing more than one million people, destroying infrastructure, and damaging social networks in the Northern and Eastern Provinces. Against this backdrop, the United States Agency for International Development’s Mission in Sri Lanka (USAID/Sri Lanka) manages the Development Grants Program (DGP), which seeks to: strengthen the capacity of service providers to provide access to quality services for Persons with Disabilities (PWDs) and better livelihood opportunities; support the acceleration of sustainable economic recovery and growth opportunities in economically-lagging and former conflict areas; and strengthen selected Sri Lankan non￾governmental organizations (NGOs). A total of 13 DGP awards were implemented from 2012 to 2016, with a budget of 11.3 million United States Dollars. These implementing partners (IPs) typically form or work through community-based organizations (CBOs) to gather or reach beneficiaries. CBOs often have a more permanent presence in villages beyond the timespan of individual projects. EVALUATION OBJECTIVES Social Impact, Inc. conducted a mid-term performance evaluation of the DGP from May to July 2016 for the period of performance from 2012 to May 2016. The purpose of this evaluation is to: 1) determine the extent to which the DGP is on track to achieving “improved management capacity and achieving technical objectives” and to evaluate the strengths and weaknesses of the Mission’s management of the DGP portfolio; and 2) recommend corrective actions needed and/or areas for improvement to achieve programmatic effectiveness and impact. The audience for this evaluation includes USAID/Sri Lanka and other bilateral and multilateral donors working for civil society capacity building and economic growth (EG) as well as other concerned development partners. EVALUATION QUESTIONS Evaluation Question (EQ) 1: To what extent has the management capacity and sustainability of the 13 IPs improved? At this point, is the project on track to meet the expected end results? EQ 2: The Mission is using a three-prong methodology to provide capacity building. To what extent have the capacity building efforts and model been effective? What has and has not worked well and why? EQ 3: How effective are DGPs in delivering planned results in their technical fields? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. EQ 4: With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and local organizations that are feasible to achieve in the remaining period of performance. EVALUATION METHODOLOGY The evaluation used a multi-level, mixed methods approach, focusing on qualitative data collection and analysis. Primary data collection methods included: 1) a desk review; 2) key informant and group interviews; 3) brief participatory workshops; 4) a mini survey for IPs to gather data on technical results; and 5) site visits among IPs. The evaluation involved four weeks of field data collection in Sri Lanka, traveling to ten cities and towns and interviewing a total of 436 respondents. Development Grants Program Mid-term Evaluation vii FINDINGS, CONCLUSIONS, AND LESSONS LEARNED EQ 1 Findings  13 IPs reached more than 44,000 direct beneficiaries. All 13 met or exceeded their proposed quantitative targets and quality of services proposed. 12 of the 13 IPs made good progress in improving their organizational sustainability.  For most IPs, the three-year time period of the DGP was inadequate to complete the capacity development (CD) work while undertaking projects.  12 of the 13 IPs demonstrated incremental increases in effectiveness and efficiency of service delivery. This is for two reasons: 1) USAID selected high performing organizations, and 2) field￾level effects of CD would not be felt for two to three years. Conclusions  Solid IP selection yielded strong results in technical areas. Investing in helping high-performing IPs with organizational capacity made good sense in terms of efficiency and effectiveness considering the length of the DGP.  The EQ anticipated results demonstrating improved effectiveness in the field. These were found in only one IP, and the duration of the DGP was too brief to cascade CD results to the field level. However, the progression of capacity building activities from more foundational (an emphasis on the board, senior leaders, and mission/vision and leadership emphases) to a focus on organizational dimensions was sound. Lessons Learned  Creative operational partnerships and linkages to expertise beyond their mandate will help IPs expand their effectiveness without diluting their core expertise.  Although not identified as a specific outcome, IPs were beginning to display a “culture of capacity building” as capacity in the IP accumulated over time. If it persists, this culture will sustain positive change and move the IPs forward to continue a strategic approach to capacity building. Projects with local partners should create this outcome. EQ 2 Findings  USAID/Sri Lanka staff created a DGP that was more innovative than usual.  Organizational Capacity Assessment (OCA) results during the DGP produced ambitious plans for each IP. Support for Professional and Institutional Capacity Enhancement (SPICE) helped each IP to narrow and prioritize its capacity building activities.  Although they were aware and planned to meet cost sharing requirements, six of the 13 IPs struggled to demonstrate sufficient cost sharing toward the end of their DGP.  All IPs selected and revisited several of the following: strategic plans, finance management, human resources policies, management guidelines, and transport policies, etc. With SPICE’s and/or USAID’s technical help, they upgraded select systems. Conclusions  The three-prong methodology (involving the IP, SPICE, and USAID) worked well with all 13 IPs in enhancing organizational capacity.  The multiple CD activities each IP was involved in showed a cumulative effect. Therefore, a CD Development Grants Program Mid-term Evaluation viii “culture” was beginning to form. However, the sustainability of some of their CD interventions was uncertain. Lessons Learned  Future work with new IPs must emphasize and continue to create a “capacity development culture” that focuses on long-term organizational development and change. EQ 3 Findings  IPs reported that the DGP enabled 1,074 PWDs and victims of trauma to earn income. Further, 1,073 PWDs participated in capacity building activities (such as business plan development, leadership, and bookkeeping); 903 PWDs established constructive linkages with private sector organizations; 1,146 participated in microenterprises value chains; and 1,458 individuals used credit for EG input to strengthen their livelihoods. All targets were met or exceeded.  For some beneficiaries, the DGP provided the primary or only services they received, while others received complementary services from other projects or Government of Sri Lanka (GOSL). As many beneficiaries were displaced or in need of services for many years, they encountered other projects.  Women beneficiaries of the DGP were empowered by obtaining civil documents asserting legal rights, ownership of assets, access to credit, marital rights, and rights to conduct business.  The changes experienced by individuals, families, and communities were more transformational than incremental. Transformational change was brought about, for example, by owning livestock, having reliable income/livelihood, a reliable market for fish, a second crop of rice, a reasonable home and toilet, nearby water, etc. For DGP beneficiaries these widespread results indicated they were significantly reducing extreme poverty and restoring many households who had lost small businesses during the civil war. Conclusions  Across the findings, the evaluation team (ET) found three principal factors to empower PWDs: 1) counseling services; 2) livelihood activities; and, 3) the expertise of IPs working with PWDs and trauma victims.  The ET observed livelihood efforts in nine IPs. Effective livelihood activities were characterized by: 1) technical capacity; 2) diversification in livelihoods; and 3) increased marketing for products and services. Lessons Learned  A longer time period for the DGP would increase the likelihood of sustaining IPs’ work.  CBOs functioning with IP involvement need to be strengthened and continued for sustainability. EQ 4 Development Grants Program Mid-term Evaluation ix  USAID/Sri Lanka staff play a dual role as decision-makers and advisors for the IPs. IPs find it difficult to disclose their implementation challenges.  The DGP did not focus on the development of capacity systems,1 instead it addressed the capacities of individual IPs. Although the DGP created a strong temporary internal learning system to meet the needs of the IPs, it was project-based and the interventions of USAID, SPICE, and CD partners were not intended to extend beyond project life. To build the capacity of small and medium size direct grantees (IPs) required a significant investment of time by USAID staff.  The lesson of selecting strong IPs cannot be under-emphasized. Depending on the situation, particularly with shorter duration projects, this will be effective. Recommendations 1. Consider longer CD timeframes (from three to five years), for projects working particularly with IPs having lower capacity. 2. USAID/Sri Lanka should continue flexible adaptive approaches to capacity building that optimize choices for IPs who are used to “canned” standardized programs. 3. Increase IPs’ annual funding when they are able to comply with USAID’s system requirements. 4. Require IPs to hold internal gender training for field workers. 5. USAID/Sri Lanka should strengthen its approach to the OCA tool by asking IPs to retake the OCA self-assessment at the end of the funding or CD period to compare progress to date. 6. Beyond technical services, strengthen the capacity of IPs to support and form solid community￾based groups and for those groups to become self-reliant. 7. USAID/Sri Lanka should continue to fund more holistic approaches to development by addressing psychosocial and livelihood needs together. 8. USAID/Sri Lanka should discuss its dual role as advisor and donor, commitments to capacity building, and options for future programs. 9. USAID/Sri Lanka should build a longer-term approach to build the capacity of future providers learning systems into future CD efforts. Use the IPs to build the capacity of service providers based more closely to beneficiaries or larger CBOs in their technical focus area (a “training of trainers” model).2 1 Systems thinking has been central to good international development practice for over a decade. USAID’s Global Excellence in Management initiative was an early adapter. USAID’s Learning Lab and its emphasis on adaptive management and the collaborate, learn, and adapt program demonstrate its institutional commitment in this area. A new generation of USAID funding has replaced the DGP with systems-oriented programming. Similarly, CD is now conceived as development of a “capacity system” rather than a collection of individual and organizational capacities. 2 The ET was informed that a professional association of organizational development facilitators was established in Sri Lanka. An active body such as this would be a significant step to achieving this recommendation. A systems approach would include participants from across NGOs, CBOs, private sector, GOSL, and academia. Development Grants Program Mid-term Evaluation 1 BACKGROUND THE DEVELOPMENT PROBLEM Sri Lanka suffered from a 26-year civil war that displaced more than one million people, destroyed infrastructure, and damaged social networks in the Northern and Eastern Provinces. Despite significant aid interventions, extreme poverty, food insecurity, and sub-standard living conditions remain the norm across many provinces. During the war, Internally Displaced Persons (IDPs) lived in temporary shelters in safe locations. As families return to destroyed living spaces, their resettlement experience has been inadequate. Water, sanitation, and hygiene (WASH) facilities are poor, especially for those in transitional shelters. Currently, IDPs are a highly vulnerable group with limited economic prospects and many victims are still suffering from trauma. Persons with Disabilities (PWDs) have limited access to healthcare, suffer from social stigma, and struggle with coming to terms with their disabilities and limitations. As a result of the war, many IDPs cannot access public services as they have lost their most important documents, including birth certificates, national identity cards, and property title deeds. Because the destruction of lives and human settlements is greater than the visible physical destruction of property, the Government of Sri Lanka (GOSL), international development partners, and non￾governmental organizations (NGOs)/civil society organizations (CSOs) are focusing their efforts on rehabilitating civilian casualties. THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT’S RESPONSE In 2012, the United States Agency for International Development (USAID)/Sri Lanka Development Grants Program (DGP) began to increase and diversify the number and quality of NGO/CSO implementing partners (IPs). USAID's publication “2014 CSO Sustainability Index for Asia” discusses the organizational capacity in Sri Lanka in the context of external constraints to the operating environment which have since been lifted.3 The DGP resources develop IPs’ organizational and technical capabilities to meet the needs of their own communities. USAID expects that these IPs will become stronger, more flexible, and increasingly sustainable in the process. The DGP portfolio advocates for issues of common concern and establishes alliances with local government and/or private sector partners to deliver more effective services. These awards maintain a focus on interventions and services that respond to the needs of youth, women-headed households, ex-combatants, IDPs, and other vulnerable populations. A total of 13 DGP awards were finalized for implementation between 2012 and 2016 with a budget of 11.3 million United States Dollars (USD).4 The 13 awards in the DGP portfolio seek to: 1) strengthen the capacity of service providers to provide access to quality services for people with disabilities and better livelihood opportunities; 2) support the acceleration of sustainable economic recovery and growth opportunities in economically-lagging and former conflict areas; and 3) support the strengthening of selected Sri Lankan NGOs. The Mission also extended the Support for Professional and Institutional Capacity Enhancement (SPICE) program’s contribution to DGP capacity building efforts. 3 USAID Bureau for Democracy, Conflict, and Humanitarian Assistance Center of Excellence on Democracy, Human Rights, and Governance Bureau for Asia Office of the Assistant Administrator, “The 2014 CSO Sustainability Index for Asia” (2014). 4 See Annex II for complete list of 13 DGP grantees. Development Grants Program Mid-term Evaluation 2 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this evaluation is two-fold: 1) to determine the extent to which DGP is on track to achieving “improved management capacity and achieving technical objectives” and evaluate the strengths and weaknesses of Mission’s management of the DGP portfolio; and 2) recommend corrective actions needed and/or areas for improvement to achieve programmatic effectiveness and impact. The scope of this mid-term performance evaluation required assessing 13 DGP grantees, the SPICE program’s contribution to DGP capacity building efforts, USAID’s contribution to DGP capacity building efforts, and the achievement of the DGP’s overall goal and objectives. The findings and recommendations of the evaluation are to provide lessons learned and recommendations for future programming adjustments for local IPs. This evaluation covers the DGP implementation period of performance from 2012 through May 2016. The audience for this evaluation includes USAID/Sri Lanka, particularly the Program and Policy Support Office (PPS), the Governance and Vulnerable Populations (GVP) team, and the Economic Growth (EG) team; USAID’s Bureau for Economic Growth, Education, and Environment (E3); DGP IPs; SPICE implementers; other bilateral and multilateral donors working for civil society capacity building and EG; and other concerned development partners. Hopefully, lessons drawn from this evaluation can be shared with the donor community and organizations/networks committed to strengthening NGO and CSO capacity in Sri Lanka and USAID capacity development (CD) efforts in other countries. EVALUATION QUESTIONS5 This evaluation focuses on four core evaluation questions (EQs)6: EQ 1: To what extent has the management capacity and sustainability of the 13 IPs improved? At this point, is the project on track to meet the expected end results? EQ 2: The Mission is using a three-prong methodology to provide capacity building. To what extent have the capacity building efforts and model been effective? What has and has not worked well and why? EQ 3: How effective are DGPs in delivering planned results in their technical fields? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. EQ 4: With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and local organizations that are feasible to achieve in the remaining period of performance. 5 The original statement of work contained two questions. These were expanded due to USAID’s interests in the results of IP service delivery and overall lessons learned. 6 Sub-questions are included in Annex III: Data Collection Methods, Tools, Analysis, and Data Quality Assurance. Development Grants Program Mid-term Evaluation 3 EVALUATION METHODOLOGY DATA COLLECTION METHODS The evaluation used a multi-level, mixed-methods approach, with a focus on qualitative data collection and analysis. The evaluation was conducted by a four-member team from May to July 2016, consisting of three phases: 1) a comprehensive desk study, internal of all relevant documents; 2) approximately four weeks of field data collection in Sri Lanka, involving qualitative data collection in ten cities and a total of 436 respondents; and 3) analysis of all data collected, report writing, and discussion of preliminary findings (while in the field) and an out-brief presentation of draft findings (at the USAID/Sri Lanka Mission). The primary methods of data collection were (1) desk review and analysis; (2) key informant interviews (KIIs) to explore issues in-depth with individual stakeholders; (3) participatory workshops and small group interviews to discuss open-ended questions; (4) a mini survey to gather data on results in IPs’ technical fields; and (5) site visits among IPs to examine issues on service delivery to beneficiaries. Desk Review and Analysis: The desk review largely consisted of official program documents provided by USAID and DGP staff working on the management and design of the DGP. The evaluation team (ET) also reviewed individual DGP IP documents and other relevant data (see Annex VII for the initial desk review bibliography). Primary sources of data for the document review included: overview documents relevant to the DGP activities; monitoring, evaluation, and reporting documents specific to the DGP; and secondary data and analysis relevant to DGP component areas. Mini Survey of DGPs: A mini survey was sent to the IPs to collect key information, excluding three IPs as their projects did not directly relate to the survey questions (see Annex XII). The survey provided a recent consolidated picture about results in the IP’s technical fields in terms of the number of beneficiaries with access to services. This enabled the ET to use fresh data as a basis for discussion about service delivery and programming. Field Visits: The ET spent two and a half weeks visiting IPs—dividing into two sub-teams to maximize geographic and focus area coverage throughout data collection. The sub-teams spent on average two days with each IP—including beneficiaries and other key stakeholders. The team conducted 146 KIIs (56 females and 90 males) and 31 small group interviews in the North and 22 in the East and Nuwera Eliya with 288 beneficiaries (176 females and 112 males). The ET also conducted appreciative evaluation workshops, which were three-hour participatory workshops to promote IP staff discussion on changes in IP capacity and effectiveness of service delivery (the design of these workshops is included in Annex VIII). Interview guides and key questions to elicit more focused discussion on some of the key evaluation sub-questions were utilized by all ET members (see Annex VIII). Internal Consultations and Fieldwork: The ET held a conference call with USAID/E3 in Washington D.C. to understand the DGP framework. In addition, the ET met with staff from Counterpart International (CPI) and Management Systems International (MSI) for information on SPICE’s role. In Sri Lanka, the ET met with key USAID staff and conducted interviews with project IPs, SPICE and its CD partners, selected project beneficiaries, donors with similar projects in the field, and other stakeholders, thereby allowing a range of perspectives and depth to be incorporated into the evaluation. Annex IX presents key informant statistics by data collection methods. The ET held informal group interviews with GOSL officials and beneficiaries, who were cooperative and open to discussion with the ET. The ET was accompanied to the field by volunteers and IP field staff. Annex III presents the evaluation design matrix and illustrates the relationship between the EQs, data sources, methods used for data collection and analysis, and data quality assurance. Development Grants Program Mid-term Evaluation 4 LIMITATIONS Time Constraints: The ET divided into two teams to observe and interview beneficiaries. In certain locations, the ET had to travel more than 80 km from the IP’s office to reach field sites. This was unavoidable because disabled persons and war victims are scattered over working areas of the interior. Extensive travel limited time for other evaluation activities, such as daily analysis conferences. Nevertheless, the ET was able to complete data collection according to schedule, covering 13 IPs. Biases: The ET worked with USAID and other stakeholders to identify potential respondents, based on their programmatic experiences. IPs selected participants for small group interviews among beneficiaries, indicating selection and response biases. The team sought to overcome these biases by clearly communicating the purpose of the evaluation with all respondents and highlighting the ET’s role as evaluators and the contribution of honest responses to the evaluation. Triangulation was another technique used to counter the effect of bias. The ET triangulated the sample by involving diverse respondents. The team triangulated data collection methods by using KIIs, group interviews, and participatory workshops. The ET triangulated analysis by using multiple analysts in the team and by exploring emerging themes as well as those suggested by the EQs. Respondents were encouraged to speak openly with the reassurance that their responses would be kept anonymous. But in two cases, the ET overheard IP staff telling community members to “only say good things about the project.” The Use of Ratios to Express Findings: Wherever possible, ratios are given to roughly indicate the extent of a particular finding. The ET based these ratios on evidence gathered during interviews, not quantitative surveys. These are not conclusive, and should not be used to generalize findings across IPs. TEAM COMPOSITION 1. The ET Leader, Dr. Peter Malvicini, is an international consultant based abroad and external to USAID 2. The Assistant Team Leader, Dr. Upali Sedere, is a national consultant with international training and extensive evaluation experience 3. Pulendran Tharmendra and Radha Pathmanathan are local ET members FINDINGS, CONCLUSIONS, AND LESSONS LEARNED EQ 1 To what extent has the management capacity and sustainability of the 13 IPs improved? At this point, is the project on track to meet the expected end results? FINDINGS Discussions with leadership and senior staff across 12 of the 13 IPs7 indicated that they made good progress in improving the sustainability of their organizations (see Organizational Capacity Assessment 7 Some findings are expressed as a ratio, for example three of the 13 IPs demonstrate a finding. This ratio shows the ET has evidence to support the presence of this finding in three of the 13 IPs. If the finding only applies to a subset, the denominator is lower, e.g. three out of seven. This ratio only provides a rough idea of the extent of the finding. It should not be used to generalize across the IPs, indicate the intensity of the finding (as this may vary Development Grants Program Mid-term Evaluation 5 [OCA] discussion below). Generally, this included strengthening second-tier leaders, discussing or doing succession planning, and/or longer term planning for funding the organization. However, two of the 13 IPs struggled with these issues toward the end of the grant. They will require further support if they are to make a successful transition from their DGP to a portfolio of work without USAID funding. Though organizations did not mention dependence on USAID specifically, 10 of the 13 IPs noted their DGP was their biggest grant to date. The DGP created considerable IP capacity. However, seven of the 13 IPs noted that the three-year time period of the DGP (and the limited duration of SPICE’s engagement in CD with the IPs) was inadequate to complete CD work. Only one out of 10 USAID/Sri Lanka staff members brought up the need for a longer timeframe to build IP capacity. Sub-Question 1.1: Are the 13 organizations showing tangible outcomes to maximize organizational performance and carry out high impact programs? Before the DGP, all of the 13 IPs were high performers in their technical areas—this was a key factor in their selection by USAID/Sri Lanka. Each IP showed different signs of significantly strengthening their organizational performance. The OCA revealed each IP started from a very different baseline—while one IP was establishing basic organizational systems such as human resources (HR), another worked on designing an advanced professional development program for staff. Nine of the 13 IPs began to create a “culture of capacity development”—though activities sponsored by the DGP have been useful, it is this organizational change that will serve them well in the future. This culture means the orientation of the organization (and individuals) views capacity building as a system intervention rather than a string of activities. Effectively, CD has been institutionalized and valued as an essential function of the organization. At this stage, it is no longer project-based, but the organization will create ways to support this culture in the long-term. Nine of the 13 IPs regularly set aside funds within projects funds for capacity or professional development. Sub-Question 1.2: Have the 13 organizations demonstrated improved effectiveness and efficiency of their service delivery? Are their practices sustainable? 12 of the 13 IPs demonstrated incremental increases in effectiveness and efficiency of service delivery in the field, such as better personnel practices (recruitment and job descriptions and timely payroll) and clear financial disbursement protocol requiring regular submission of expenses. This is for two reasons: 1) USAID selected organizations that were already high performers in their technical areas—successful practices honed by experience with past projects; and 2) while the effects of capacity building efforts were immediately felt at the organizational/systems level, those in the field would not realize the impact for two to three years. Furthermore, capacity building gains were not reported on qualitatively by IPs in quarterly reports, making them more difficult to track. During selection, USAID/Sri Lanka followed the expertise of the IPs in their flagship areas, from water resources, to psychosocial services, resettlement, and livelihoods. For 12 of the 13 IPs, the organizations and the models they employed, such as cattle raising, psychological counseling, construction of housing and water supplies, and livelihood generation, represented national best practice according to USAID, widely across IPs), or surmise the finding is absent in all others—the ET worked from information volunteered during semi-structured interviews based on the EQs rather than quantitative surveys. Development Grants Program Mid-term Evaluation 6 the IPs, and sector experts on the ET. This was a conscious and important feature of the DGP, given the limited duration of the grants and the need to show results quickly. However, eight of the 13 IPs expanded to new technical areas or geographical areas, thereby requiring more time. IPs who worked with a partner organization(s) possessing complementary expertise did better than IPs who worked independently. For example, an IP effective in resettlement and housing was able to link upstream to an organization providing psychosocial services to trauma victims and downstream to an organization working on livelihood development and marketing. It would have taken years for the resettlement IP to gain its own expertise in psychosocial services and marketing, so linking with existing organizations made more sense. The IP’s focus on sustainability and continuity of service delivery after the DGP was handled differently by each IP. IPs with a long-term presence in the district would have direct access to beneficiaries and at least help them troubleshoot and move the process forward, versus project offices that would shut down and/or relocate after the DGP was completed. Three of 13 IPs had sustainable funding strategies for their DGP projects or ways to fold the work into future projects. These three IPs, with three others (a total of six out of 13), maintained strong groups or revolving funds and were in a better place to sustain benefits, as were those who linked project beneficiaries with local stakeholders, such as government and the private sector. Sub-Question 1.3: Are the 13 organizations on track to meet the expected end results? The 13 IPs reached more than 44,000 direct beneficiaries (24,000 women and 20,000 men). According to both USAID/Sri Lanka reports and self-reporting, all 13 IPs met or exceeded their proposed quantitative targets and quality of services proposed (see Findings for Question 3 below for more detail). While the evaluation focuses on the capacity building results for the IPs, the sustainability of beneficiary results is an important consideration and indicator of IP effectiveness (see discussion under Question 3). CONCLUSIONS  Solid IP selection yielded strong results in technical areas. Investing in helping high performing (technically) IPs with organizational capacity made good sense.  The EQ anticipated results demonstrating improved effectiveness in the field. These were found in only one IP, and the duration of the DGP was too brief to cascade CD results to the field level. However, the progression of capacity building activities from more foundational (an emphasis on the board, senior leaders, and mission/vision and leadership) to a focus on organizational dimensions was sound. LESSONS LEARNED  When used, creative operational partnerships and linkages to expertise outside the IPs mandate helped expand their effectiveness. These partnerships worked because they did not need to dilute their core effectiveness.  Over time, IPs that displayed a “culture of capacity building” were able to better sustain positive change and keep on track with internal capacity building activities. This works because it creates a shift in the organizations thinking from “quality service delivery” to improving service delivery through stronger organizational capacity. This shift started with senior staff and spread downward. EQ 2 The Mission is using a three-prong methodology to provide capacity building. To what extent have the capacity building efforts and model been effective? What has and has not worked well and why? Development Grants Program Mid-term Evaluation 7 FINDINGS Though not formally tested, the USAID-funded SPICE and DGP initiatives were the largest and most widespread coordinated efforts encountered during the evaluation (according to the knowledge of Sri Lanka-based ET members) providing capacity building services to development-oriented NGOs and CSOs. 12 of the 13 IPs never systematically analyzed their organizational capacity before. Initial results from the use of OCA during the DGP produced an ambitious set of plans for each IP. The engagement of SPICE as a capacity building service provider mitigated this by allowing each IP to further narrow and prioritize its set of capacity building activities. The evaluation’s focus on capacity building was to understand what worked and did not work for the 13 IPs. Because USAID/Sri Lanka is interested in OCA results, the ET organized a brief section using the seven OCA domains8 in order to discuss common findings without attempting to cover all OCA categories. Besides providing a framework for DGP CD, the IP’s CD goals which emerged from the OCA exercise are continuing to guide future CD plans and activities and these results should be regularly revisited and adjusted as the IP’s situation changes. If desired, the IP can repeat or reconsider the self-assessment results as they move forward. Governance: To understand the size of the IPs, 11 of the 13 IPs had 25 or fewer core staff (without counting project staff) and 10 IPs spread these staff across satellite offices in different regions. Seven out of 13 IPs with weak organizational structures were able to improve primarily by clearly defining roles and responsibilities. SPICE proceeded with a foundational approach, which included interventions at the board level early in the capacity building process. Eight of the 13 IPs mentioned capacity building involving development of their board members—reporting that it strengthened the engagement of board members and helped focus their role in service. Six of the 13 IPs stated that succession planning was an important undertaking, particularly for those dependent on a top leader who was also the founder of the organization. The OCA created the space for these conversations about succession to take place. Training for second-tier managers further supported these conversations. The CSO Sustainability Index for Asia supported findings of many rural NGO projects being run from their headquarters (HQ) in Colombo. Administration: 12 of the 13 IPs lacked functional policies or administrative manuals before the DGP. Each organization expressed that developing the policies and manuals was a lot of work and that the DGP gave them a strong head start by providing samples and feedback. Through evaluation workshops or group interviews 12 of the 13 IPs reported they had improved their information and information technology systems; four of these IPs migrated from manual to computerized systems and are generating reports with them. Many IPs lacked sound procedures for either travel, procurement, or asset control to comply with USAID requirements. Seven of the 13 IPs initially lacked staff with the capacity to develop or implement the policies, procedures, and systems they created. At least nine of the 13 IPs used external consultants or contractual employees to supplement their capacity in this area. While some organizations plan to strengthen staff capacity to run new systems, others feel capacity substitution through external consultants is an efficient approach. 8 As there are approximately 40 OCA categories, a checklist was not used with IPs but the design employed semi￾structured interviews based on the EQs. Development Grants Program Mid-term Evaluation 8 HR: Like administrative procedures, 10 of the 13 IPs lacked useful and clear job descriptions before the DGP. Staff in 10 of the 13 IPs that created job descriptions under the grant, stated they helped create a better division of responsibilities and set of individual deliverables. Also, the IPs stated that the discipline of creating personnel policies and plans made staff think more deliberately about their current practice and what it would take to move the organization forward. The strategic plan and other documents required stronger basic policies and procedures. The fact that several procedures were required by USAID further motivated most of the IPs (12 of the 13) to explain that other donors did not require such procedures or “were not so strict.” The short-term nature of NGO employees, cited in the 2014 CSO Sustainability Index for Asia, was also found to be true among the IPs. Financial Management: While the pre-award survey assessed the soundness of basic financial systems it did not specifically test IPs’ potential to comply with USAID’s financial reporting requirements. Seven of the 13 IPs found USAID requirements for financial reporting, audits, and cost-sharing the most arduous among all capacity areas. 12 of the 13 IPs perceived that receiving funding from USAID and complying with its requirements was the gold standard of performance in Sri Lanka, believing other donors would think, “if they can meet USAID standards, they can meet ours.” 12 of the 13 IPs were able to adapt some existing systems to work with USAID. This adaptation was easier for those who previously worked with international donors. 12 of the 13 IPs developed financial reports on a project￾by-project basis for donors. USAID staff reported that different IPs required varying levels of support. The IPs appreciated the discipline of creating financial policies and manuals, even if the process was challenging and time-consuming. Although they were aware and had planned to meet cost-sharing requirements early on in the grant, six of the 13 IPs struggled in the end to demonstrate sufficient cost￾sharing. The question was raised whether simpler procedures, particularly for procurement and financial reporting would be more practical for these organizations. These would require USAID to use a different modality when working with them. Organizational Management: Strategic planning9 exercises were a stated highlight of the DGP’s capacity building activities. There were a number of capacity building implementers who provided these services and nine IPs were given freedom to choose whom to work with. For eight of the nine IPs, strategic planning helped them proactively focus when it might otherwise simply move from project to project. The processes used in the CD activities involved more staff and stakeholders actively in the programming process. However, beneficiary involvement seemed to be driven by the IPs’ own operational culture—this ranged from a conventional service delivery approach in the field to strong group formation and control over program elements in villages. SPICE’s strategy was to lay a foundation with the board, management, senior leaders, and key organizational systems. Extending this to change in field practice would require a longer engagement in CD activities or longer time for the IPs to extend their learning to operational practice. When strategic planning was conducted outside Colombo, more field staff were able to participate. However, field staff are generally less involved in organizational matters as they are normally tied to a project-based contract. Program Management: The capacity of IPs to comply with donor requirements was directly related to prior experience working with international donors. Six of the 13 IPs struggled with compliance in this area. These IPs reported the direct coaching and assistance provided by USAID in this area helped 9 Although the DGP included specific strategic planning processes, IPs also viewed their work on succession planning, sustainability planning, and resource development as forms of strategic planning leading to action/implementation plans. Development Grants Program Mid-term Evaluation 9 considerably. Only eight of the 13 IPs were strongly involved in NGO networks in their sector. Nine of the 13 IPs operated within a silo mentality by being focused on their own projects or limited to their particular area of operation. Early on in the grant, through USAID’s facilitation, IP leadership met with each other regularly while staff participated in some CD activities together. From participation in capacity building activities, gender sensitivity at the organizational level for all IPs was strengthened by their integration of gender equality within project design and activities. However, the field staff’s understanding of gender equality and development issues appeared to be more uneven—even though inclusion and engagement of women seemed to be natural in field activities. This unevenness is because not all concepts and approaches from gender sensitivity and inclusion training were passed on to field workers. Project Performance Management: 12 of the 13 IPs operated with strong field oversight coming from regional project offices with senior field staff. Even fresh secondary school graduates were mentored to become effective workers in local villages. In one village, a middle-aged women’s group laughed as they reported that, although young, the field worker was strict and would “scold them” if they did not follow the guidance they received. All IPs agreed that the seven OCA categories they initially self-assessed were a useful tool to guide to action planning for capacity building, a process which USAID led. Though priority and capacity building activities were different for every IP, they started from a common point. Within the DGP, the use of the OCA helped work with IPs holistically. Neither USAID/Sri Lanka nor the IPs can expect another DGP-style program moving forward. USAID initiated good efforts to bring together IPs early in the grant period. In addition, IPs met each other informally at various capacity building activities. As IPs reported and USAID confirmed, monitoring procedures with quarterly reports were followed faithfully by all IPs. However, they were limited to counting outputs and little qualitative reporting was reported. 12 of the 13 IPs perceived the DGP as a grants program with capacity building activities. The accountability was for program deliverables, not capacity building. However, quality assurance was created by the caliber of the IPs themselves. IPs were able to maximize opportunities in most settings because staff and managers were experienced and communicated well in the field. This factor goes back to the initial strength of the IPs selected. Sub-Question 2.1: To what extent have the capacity building efforts and models implemented by the 13 organizations themselves been effective? The three-prong method—relying on the IPs themselves, USAID, and SPICE—in providing capacity building worked well with all IPs. The OCA helped most of the IPs to understand their own capacity needs and start to address them. The overall efforts made to develop the institutional capacities, management, and finance administration were well received by IP staff and all IPs trained their staff in different thematic areas of management and service delivery with the support of SPICE and USAID. Many commented on the strong fit of the capacity building activities to their needs. IPs were assisted to develop and implement policies and procedures in practical ways. OCA helped all IPs to understand their organizational strengths and limitations in administrative, financial, and monitoring and evaluation (M&E) systems. The process contributed to the IPs being viewed by outsiders as organizations with stronger capacity. One IP even took some of their CD experience and helped sub-grantees to develop their own strategic plans. Sub-Question 2.2: To what extent have the capacity building efforts of USAID been effective? USAID staff distinguished between the capacity to comply with USAID requirements and the creation of broader capacity for the IPs. For example, financial management was considered a broad capacity area relevant to all of the IPs’ future undertakings—while understanding USAID systems was only a specific application of that capacity. The broader view was that capacity building is a long-term undertaking and the DGP efforts were effectively helping IPs take a great step forward in the process. DGP funds were disbursed before the IPs had the capacity to comply with all of USAIDs policy, procedural, and reporting Development Grants Program Mid-term Evaluation 10 requirements (with the exception of one IP). Within USAID/Sri Lanka, staff and management maintain divergent views on this. Some believe USAID was setting certain IPs up to fail—these IPs would not be able to comply, particularly with various finance-related requirements. Others believed that passing the pre-award survey was an indication that the IPs’ capacity could be strengthened to meet USAID’s requirements. IPs themselves echoed this lack of readiness, describing a steep learning curve involved in early reporting. USAID flexibility in understanding the ground situation and operational issues during project implementation also helped the IPs. IPs often described USAID staff, particularly the Agreement Officer’s Representative (AORs), as service people who provided constructive input and troubleshooting with individual attention. USAID workshops for IPs succeeded in building capacity and were often backstopped with on-site coaching. Strong CD features included customization to IP needs (focused through OCA goals). For example, all 13 IPs reported that their standard operational procedures were strengthened; that USAID’s help revising and updating their regular financial review procedures helped the IPs enhance their internal control systems and reporting; and they translate strategic plans into annual operational work plans, which includes project management, budgets as well as staff CD. Three IPs were immediately able to write stronger proposals and secure new funding. Sub-Question 2.3: To what extent has the SPICE contracting role of the provision of training services been effective? SPICE helped all IPs revisit their organizational systems and engaged many IPs in different forms of strategic planning through various capacity building partners. SPICE carried this out under a specific set of deliverables to support the DGP in agreement with USAID. Besides USAID’s AORs, SPICE played a key role in tracking and nurturing the IP’s CD progress. Furthermore, SPICE worked across a continuum from strategy, sustainability, and fundraising to management and organizational system CD activities. 12 of the 13 IPs established detailed performance monitoring systems. SPICE helped them identify key result areas and indicators. SPICE was not able to fully exploit the potential of M&E CD (using real data) due to the limited duration of CD activities. 11 out of 13 IPs adopted some coordination for implementation through government and other related stakeholders. However, these systems generally were underdeveloped and did not reach their full potential. IPs indicated that SPICE supported the OCA by helping all IPs focus and select a range of CD activities. IPs also indicated that SPICE enabled two IPs to pass on the CD they received to sub-grantees, other IPs, and local organizations. Moreover, IPs regularly mentioned SPICE’s support in the process of developing their capacity in planning and M&E. Each organization related that the processes used for capacity building workshops were strong (participatory, client-driven, practical, and with sufficient variety). However, four IPs mentioned they found long training modules with many handouts difficult to read and study over the short time frame of particular workshops. Others complained that as capacity was built in IPs, some was lost due to staff turnover. This was particularly true for staff whose employment contracts were directly linked to term￾based project funding. Two of the 13 IPs had core funding to carry staff over to the next project or recruitment strategies to retain their best talent. These IPs were larger, more established, and with shorter or no gaps between projects. Moreover, these organizations had a longer-term vision for addressing social issues. Such organizations were likewise able to develop systems, such as volunteer￾based work, and thus able to keep operational costs to a minimum. They also have relatively diverse sources of funding, such as donations from charities and individuals. CONCLUSIONS  The three-prong methodology adopted by USAID with OCA, USAID direct inputs, and SPICE training worked well and in different ways with all 13 IPs enhancing organizational capacity. Development Grants Program Mid-term Evaluation 11  All IPs revisited their strategic plans, financial management, HR policies, management guidelines, and travel policies, and with SPICE and/or USAID technical help have upgraded these systems.  Although many capacity building interventions were introduced, the sustainability of some IPs as organizations and the sustainability of their development interventions remain areas of concern.  Evidence from interviews with USAID staff, IP managers, and capacity building partners showed that USAID/Sri Lanka staff acted outside the box to create a DGP that was more effective. Without this extra effort, the effectiveness of the DGP would be limited. For USAID/Sri Lanka’s DGP, the knowledge of the Sri Lanka context held by AORs (and the IPs themselves) was used to adapt the program design and guide IP projects. LESSONS LEARNED  Creating a “capacity development culture” within IPs was effective because many operate on a project-by-project basis while focusing on long-term organizational development and change would be part of more strategic and sustainable approach to their work. EQ 3 How effective are DGPs in delivering planned results in their technical fields? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. FINDINGS The DGP enabled the 13 IPs to reach a total of 44,380 direct beneficiaries (24,192 females and 20,196 males). All 13 IPs met and exceeded the targets in terms of quantity as well as proposed standards of quality of services. Instead of small changes, the results of the DGP were transformative for individuals, families, and communities. In other words, the DGP helped a majority of its beneficiaries reduce extreme poverty significantly. The examples below demonstrate how many people actually moved from situations of desperate poverty to an improved level of well-being. Directly resulting from the work of IPs, for example, at a meeting with a group of four male farmers, they explained restoring the canal meant an additional harvest of paddy each year. One farmer stated: “Now we eat three meals a day instead of one, and now we can send our children to school.” In addition, raising cattle had become the primary income source for a woman and her husband, who was an unemployed casual laborer. With only three cows, they became self-sufficient. Families talked about how moving from an improvised home to a semi-permanent one with a metal roof, adequate space, and block walls was the foundation for all productive activities of the home—a basis for more improvement. Beneficiaries attributed these results directly to the IP interventions under the DGP. The 2014 CSO Sustainability Index for Asia discusses high levels of community participation in rural areas which were found to be true. IPs have established community-based organization (CBO) networks at the community level and built the capacity of these organizations—two IPs in the Eastern Province established CBOs and Disabled People Organizations (DPOs) and trained their officers; seven IPs, particularly those working on WASH, helped strengthen CBO capacity in strategic planning, implementation, simple operational guidelines, and basic policy development; and a cattle-raising scheme in the north invested significant training in its village groups to ensure the model was sustainable after project completion (they also established strong linkages with the local government veterinary department). Some CBOs were already addressing new issues, beyond the project scope, on their own. Focus 1: Enhancing Lives of PWDs and Trauma Survivors includes challenges and achievements in treating, providing functional support, and enhancing the lives of those affected (sub-questions 3.1 and 3.3). All four IPs working with PWDs and trauma survivors helped reduce the stigma among their families and Development Grants Program Mid-term Evaluation 12 communities through awareness campaigns: mobile health screening services to the community, inclusive schooling, livelihood activities, and access to services (medical, livelihood, mental health, and government services). Their experience in these areas was extensive and their capacity was very strong due to their staff’s skills. The services provided, according to survivors themselves, have enhanced their lives. They noted high quality of counselors and IP services that included an integrated approach to their needs. Psychosocial counseling sessions helped people affected by war to recover and engage in productive activities. To illustrate, one IP was able to provide household-based counseling services, visiting each household (selected) periodically and providing counseling support. They also networked with government mental health units to refer cases when necessary. Additional assistance for medical treatment (cash grant) was provided to needed families. They also sought service of experts in the field to help them better deliver services. Another advantage was most workers lived in the same or neighboring communities. One Colombo-based IP received a DGP for work that did not directly address any of the technical sectors supported in the general portfolio. Instead, they provided support research and media-based activities focusing on reconciliation through the use of creative media with groups of young people from mixed ethnicities. Together they learned art and visual medium then produced and published work for distribution. Their visual works were surprising and unusual as their subjects were a product of their personal experiences of ethnic difference in their communities, schools, and country. The strategy was a significant breakthrough and the work is being spread through schools and other venues in Sri Lanka as group members present their work to others. The academic research represented topics of great interest to the donor community as they provided a background and direction for development interventions focused on ethnic reconciliation. DPOs provided micro credit loans, starting at 1,300 USD, to their members for livelihood activities or improvement of their living situation; PWDs run the DPOs. A revolving credit system was implemented through the sub-grantees of one IP, in six different districts. To be eligible for government support and for Official Development Assistance, DPOs must register with the Social Service Department. Also, for two out of seven IPs, the IPs’ support effectively guided the DPOs in the process of registration and obtaining grant assistance. For example, two IPs also provided regular counseling services, including transport, for identified trauma victims. In four small group interviews, 18 clients (beneficiaries) reported they were received and treated well at the counseling units and that the counselors maintained their privacy and confidentiality. One hospital reported that due to the IPs’ awareness campaigns and mobile health screening services, patients are now attending the mental health clinic. This dramatic increase was from only three to four patients a month to 300 to 400 patients a month. Only one or two disabled patients used to attend physiotherapy treatment, but this increased—now the physiotherapy unit needs at least three more therapists to meet the demand. Because of the rapid increase in demand, several issues have arisen, such as lack of space in the hospital, lack of equipment, and lack of staff. The government has limited capacity and government procedures for recruitment and purchasing take a long time. There was no mention of decrease in service quality. There are clear indicators that DGPs enhanced PWDs’ lives: 1,170 PWDs (644 males and 526 females) accessed prosthetics and orthotics services; 1,328 received locally manufactured prosthetic and orthotic supports10; 12,165 PWDs (6,560 males and 5,605 females) accessed physical and rehabilitation support; and 927 PWDs (424 males and 503 females) re-entered normal inclusive schools. The counseling sessions were effective in mitigating 10 This number is higher than the total number of persons, as some beneficiaries required more than one device. Development Grants Program Mid-term Evaluation 13 mental health issues in target communities, helping people refocus on productive day-to-day activities. These beneficiaries stated they previously lacked opportunities to share their feelings and experiences, as they have not had anyone to trust these with for a long time. However, both the beneficiaries and the professional counselors found the number of counseling sessions was too limited. It was not possible for beneficiaries to share the accumulated feelings and pains they endured across 20 to 25 years. IPs faced many challenges in reaching PWDs: the high degree of poverty in communities, especially in families of PWDs and trauma survivors; high cost of medical treatment, health, and social infrastructure; and the lack of professional caregivers. Transport support from IPs helped PWDs and trauma victims attend counseling sessions, visit clinics, and attend organizational meetings. IPs felt the timeframe was inadequate for DGP projects to be successful in terms of support for PWD and trauma victims. The reduced time for trauma rehabilitation was a challenge, based on direct responses from 35 out of 5011 beneficiaries total in group interviews (the average small group interview involved three to five people who were familiar with each other). Trauma and PWD victims need lifelong support, therefore a long￾term approach. Focus II: Economic Recovery and Livelihoods includes challenges and achievements in providing accelerated economic recovery and growth opportunities in the former conflict zone and increasing access to livelihood opportunities (Sub-questions 3.2 and 3.6). For 11 out of 13 IPs, livelihood support was a crosscutting issue for beneficiaries. Without a focus on stable income from livelihoods, the sustainability of other development interventions is difficult. For example, though the DGPs livelihood support was sometimes minimal, through partnerships with GOSL, IPs managed to increase the number of beneficiaries. The addition of income-generating activities enhanced livelihoods. Further, to be eligible for GOSL livelihood assistance, a PWD requires acceptable documentation; the IPs and the DPO provided the necessary support. GOSL assistance ranged from 5,000 Sri Lankan Rupees (LKR) to 20,000 LKR, and field visits witnessed PWDs’ active participation in monitoring livelihood support through the DPO. Strong market linkages were vital for sustainability of livelihood activities, which livelihood efforts struggled to develop. Two out of nine IPs had particularly weak linkages, which limited results (potential income) for beneficiaries. In some cases, the IPs provided marketing linkages that did not result in new sales. Some IPs brought beneficiaries to marketing events where buyers afterward proved unreliable. IPs reported that, as a result of the DGP, 1,074 PWDs and victims of trauma are employed or have access to other economic opportunities (564 males and 462 females). Further, 1,073 PWDs participated in capacity building activities, such as skills trainings on business plan development, leadership, and bookkeeping; 903 PWDs established constructive linkages with private sector organizations to gain additional funding support; 1,146 participated in microenterprises value chains; and 1,458 individuals used credit for EG input to strengthen their livelihoods. IPs went beyond their targets by supporting PWDs in their process of overcoming social stigma, depression, regaining hope in life, addressing dependency, and supporting their families with their income. The ET also found several PWDs engaged in more non-traditional livelihood activities resulting from the training and capacity building interventions of the IPs, such as cellphone repairs, fixing emergency lamps, and assembling light-emitting diode bulbs, 11 Typical group size was three to five PWDs. In two cases, the group was as large as 10-12 people. In a separate organizational meeting, 20 people participated from a single DPO. Development Grants Program Mid-term Evaluation 14 although these beneficiaries were predominantly male. Many female beneficiaries were in conventional livelihood activities. For example, one widow received financial assistance to get material to weave containers, bags, and mats from palmyrah leaves, which are traditionally used in rural homes. She has expanded to poultry raising and a fruit plant nursery. Conservation and organic farming activities were effective in creating awareness among target beneficiaries on the harmful effects of pesticides, and in promoting local seed production and integrated farming. Increased use of locally available organic material was found in almost all target communities trained in conservation or organic farming. However, both farmers and the agricultural service providers noted that the practices are difficult to scale up and apply beyond home gardens. The primary agricultural activity of one IP was helping beneficiaries raise dairy cows. Findings from this experience were a model for others work in rural livelihood. The IP used well-designed skill training on cow￾rearing. Close monitoring and including technical guidance from field staff helped beneficiaries apply proper feeding practices, identify diseases on time, and claim insurance for deceased dairy cows. A strong market link was found to sell milk at a good price, stabilizing income and encouraging further investment. Farmers had strong networking with the Department of Animal Production and Health, the district veterinary office, and an NGO that provided health services as well as advice on cattle rearing through “barefoot vets.” Livelihood beneficiaries participate across a continuum of activities including: simple household-based activities, such as sewing or canning; skills development to allow them to work for others as wage earners in their communities; and entrepreneurs establishing micro and small businesses and medium￾size business improvements (these entrepreneurial undertakings generate jobs, but were rare). The DGP’s approach to invest at different levels of livelihood allowed more diverse participants and potentially stronger ties in the local economy. Beneficiaries participated from household-based livelihood activities, such as organic and conservation farming, dairy farming to vocational training, and micro enterprise promotion. The diversity in livelihoods enabled the participation of men, women, young persons, and PWDs. Small and micro enterprises are now listed in the Department of Industries database and receive information about training, funding opportunities as well as advice and guidance from development officers based in each divisional secretariat. Focus III: Shelter; Water, Sanitation, and Hygiene, and; Civil Documentation includes challenges and achievements in providing support for vulnerable populations. The shelter and WASH areas faced severe challenges, including: the remote location of communities; lack of livelihoods; lack of land for cultivation; lack of access to markets; and lack of access to water (in the North due to physical limitations like groundwater levels and nitrate infiltration). Without civil documents, some services cannot be given by the government, which in turn affects livelihood. For example, water is necessary for agriculture and registration is important to operate a business. PWDs in particular face the challenge of limited knowledge of and access to offices and officials to obtain GOSL documents (national identity cards, birth certificates, marriage certificates, ownership deeds, and titles) from the Registrar General’s Office. Civil documents are essential in the process of resettlement to prove their claims and access services and resources. The IP addressed this challenge through awareness programs, capacity enhancement programs, and the establishment of the DPOs. IP interventions enabled internally displaced individuals and families to obtain essential documents to access public services. Five of the 13 IPs worked closely with GOSL officers, who began to understand that many clients have no documents to claim GOSL services. The process of working with IPs and rural clients and clients forming CBOs created a positive situation for PWDs to obtain essential documents to re-establish their lives; find new settlements, or claim their original lands. CBOs are run by volunteers. As such, they do not have problems relating to funding and salaries, and they are the beneficiaries themselves. This helped speed up implementation, as well as build community ownership. This process Development Grants Program Mid-term Evaluation 15 also helped PWDs claim GOSL assistance for livelihood, shelter, and services. IPs reported that 521 citizens returning to conflict-affected areas received support with civil documentation; 405 families received transitional shelters; 3,103 households now have access to safe and reliable water supply; and 974 households have access to sanitation facilities. According to USAID, there are up to one million total IDPs due to the war. Focus IV (cross cutting) – Effectiveness in Integrating Women’s Issues and Women’s Leadership Role in Civic Life Results from an analysis on gender equality from a female empowerment perspective are consolidated below to demonstrate the larger impact the DGP had in this area, although they were found across all technical focus areas and almost all IPs. The DGPs focused on the needs of women-headed households who were direct beneficiaries of livelihood, shelter, and WASH programs. Reliable and clean water (well water or connections to utilities) and household toilets enhanced women’s safety, dignity, and hygiene. With access to water, women do not have to travel far to fetch water, beg neighbors to draw water from their wells, or go to other people’s homes to ask for water. Women with new water supplies emphasized how it saved time and labor. This in turn created more time to pursue economic opportunities or activities to increase the family’s well-being. The ET found that women were empowered by obtaining civil documents asserting their legal rights and giving them ownership of assets, access to credit, marital rights, rights to conduct business outside of their homes, access government livelihood assistance, and obtaining toilets. In one project, a group of 20 women brought their civil documents with them explaining how this documentation was required to connect their homes to the water utility. IPs organized beneficiaries through women’s CBOs or membership groups. By meeting regularly and giving them decision-making authority, IPs built their capacity to work together across issues. For instance, groups benefited from a project that would provide latrines to beneficiary homes. They helped their members obtain documents, by participating in training, sharing their experience with each other, and accompanying women to various offices. Similarly, the movement from displacement to permanent shelter was transformational for families. A widow with two daughters and a son living in the transitional shelters said now they have a sense of safety and security for the family, especially with two other women in the household. Another widow over 60 years old said the transitional shelter and toilet provided is timely, thereby helping her resettle. Women holding leadership positions in these CBOs were empowered to make decisions. The improvement of women’s lives and in some cases significant empowerment was demonstrated in communities across projects. In places where women appear to be empowered, the ET discovered children have also returned to schools. The ET found, based on discussions with women, the challenge of alcohol use among men persists, which is linked to gender-based violence. Stable household incomes, where men are newly engaged in economic activity, only slightly mitigated this. Women still face a possible stigma when they increase their public exposure due to stereotypes of cultural and gender roles. For example, a “strong woman” is sometimes not appreciated and may also reflect poorly on her husband/family. Integrating gender equality across livelihood activities for PWDs also helped overcome this stigma. CONCLUSIONS The ET noted three principal interventions for the empowerment of the PWDs:  Counseling services provided by the IPs have been the entry point for healing and livelihood productivity. The number of sessions was inadequate for healing for some who experienced Development Grants Program Mid-term Evaluation 16 severe trauma. Each individual is unique, and trauma levels vary. However, IPs have had to limit sessions due to financial constraints, especially as transportation support was costly.  Livelihood activities reinforce healing of trauma victims and PWDs. The IPs and DPOs have done significant mobilization and capacity building and GOSL officials have positively contributed to reconstruct PWDs’ lives.  IPs’ success in addressing the rehabilitation needs of PWDs was due to the experience and expertise of IPs working with PWDs and trauma victims, and their experience in the project locations. The IPs’ implementation staff came from the same communities or were highly familiar with the local context, including cultural complexities. Livelihood activities that were effective were characterized by three factors—although sometimes only one or two of these factors were present in a single IP:  Livelihoods were effective when beneficiaries were able to increase their technical capacity and apply to productive activities, such as dairy farming or product and market enhancement through small and micro enterprise activities.  Diversification in household livelihoods, such as crop diversification adopted by farmers. Dairy farming allowed beneficiaries to reduce risks and ensure additional revenue streams. Incorporation of several “sideline” livelihoods protected a family if demand for one failed.  Increase in marketing for products and services. However, there were exceptions, particularly due to market fluctuations. Access to steady markets allowed beneficiaries to sell products and increase their income through milk collection centers, trade fairs, and links with the private sector. LESSONS LEARNED  When IPs enhance their technical capacity and stay informed of good current practice, they help clients find market-oriented and innovative livelihood opportunities. This depended on resources to do so and/or IP capacity for innovation being strong when a project begins.  Diversifying their livelihoods and linking them to markets have worked for PWDs, as PWDs require approaches that take into account their particular circumstances and needs, which are different from other beneficiaries.  For some efforts, such as livelihood and counseling, program support is too short to ensure sustainability. These areas require a longer-term perspective when planning interventions. Some projects seem to be designed around length of funding rather than time required to address problem.  CBOs (DPOs, Women’s Community Groups, etc.) functioning with IP involvement need to be strengthened and continued for sustainability. The best examples, such as cattle raising, started the product with a strong model and plan to become sustainable. EQ 4 With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and the local organizations that are feasible to achieve in the remaining period of performance. By the time of the evaluation, most grants concluded or were close to completion. As such, it was not feasible for them to make significant substantive changes to program implementation. The overall success of the IPs at achieving their service delivery targets indicate that only minor refinements are needed. However, there remains time for some IPs to take final measures to sure up sustainability of community-based groups—this can be done for IPs with time remaining in their grants and for IPs with an ongoing presence through another project or local office near their beneficiary communities. There is Development Grants Program Mid-term Evaluation 17 time to ensure the IPs themselves begin to implement their sustainability plans and their ongoing plans for organizational CD. Since SPICE will continue to offer services to both active and expired grantees until the end of 2016, they are the most promising mechanism to conduct final follow up activities or consultations. The ability to offer services to expired grantees was a distinct advantage of the arrangement.12 Sub-Question 4: What are the lessons learned?  Mission staff, particularly AORs, play a dual role as decision-makers over the project and advisors helping to troubleshoot and strengthen the capacity of the IPs. IPs are faced with a dilemma about how much of their implementation and systems struggles to disclose. USAID staff must decide where to draw the line between assistance with capacity and action on poor performance, which could jeopardize project outcomes. USAID staff appear aware of this and deal with it informally. In some ways, USAID staff encounter a similar dilemma with all projects, but in the case of the DGP, the direct funding with the intent of building capacity seems to accentuate this dynamic.  Working with small and medium size direct grantees required a significant investment of time by USAID. This varied according to the IP’s capacity, the complexity of the project, available staff time, and interest in the particular approach to the work. In some cases, AOR turnover disrupted continuity given a learning curve to establish trust and approve IP requests.  USAID used outside-the-box and flexible approaches to capacity building, including the three￾prong approach (involving IPs, USAID, and SPICE). Willingness to creatively address the DGP seems to be a major source of its effectiveness. Other programs could learn from this approach and seek creativity in design and flexibility in implementation versus a standardized one-size-fits￾all approach.  The DGP focused on the capacities of individual IPs, not a capacity system, which could have included a wider range of providers and participants. A different approach might be to look at the range of stakeholders that are critical to long-term healing of trauma survivors and investing in the capacity of different types of organizations (NGOs, local governments, academies/institutes, and hospitals) to address the needs in concert. Similarly, a learning systems approach was not used to its full potential, as capacity building activities were left primarily to USAID and SPICE.  Future projects could look to draw from and enhance the range of knowledge and experience available to address a particular development need. Initially, this is more difficult to arrange and tap, but provides for greater system capacity and sustainability in the long term.  The lesson of selecting strong IPs with prior successful implementation in their focus area for relatively short-term DGP projects cannot be over emphasized. However, a more strategic long term approach would be to use the IPs to build the capacity of local service providers or larger CBOs in their technical focus area (a “training of trainers” model). While this would not be as efficient as direct service delivery by the IPs, capacity gaps were glaringly evident in some parts of the country where the IP was the sole expert in over 160 km from the area being served. 12 This was only possible because the SPICE program of capacity development was operating under a separate contract with a later end date. Therefore, to expand services to the DGP only required a contract modification. Development Grants Program Mid-term Evaluation 18 RECOMMENDATIONS The ET recommends these actions proceeding from the findings and conclusions above. Due to the timing of the evaluation, the recommendations are primarily for future USAID/Sri Lanka projects engaging IPs and local partner organizations. Several recommendations from Question 3 have implications for improving IP practice. EQ 1 To what extent has the management capacity and sustainability of the 13 IPs improved? At this point, is the project on track to meet the expected end results? 1. Extend the CD timeframe for USAID/Sri Lanka projects to three to five years, particularly for weaker IPs. Even when USAID support is for shorter periods, CD efforts may be needed after funding ends. This should be anticipated and planned for during the OCA (it was in some cases), identifying resources beyond USAID. Seven out of 13 IPs lack solid succession plans and could be assisted before the SPICE project ends.  USAID/Sri Lanka should include IP reporting on capacity building progress alongside project outputs, as reports did not provide a narrative of progress against OCA plans. The DGP reporting mechanisms limited this in practice as USAID staff reported the IPs were not able to include a qualitative narrative on CD progress in their DGP quarterly reports, but a simple informal system could have generated this information.  USAID/Sri Lanka should develop a mechanism to challenge IPs to produce more results once agreed upon targets are met. For the DGP, all IPs reached or exceeded implementation targets (as self-reported by IPs and confirmed by USAID staff). Additional challenges could result in enhancing performance, scaling up to more beneficiaries, or providing more in-depth services.  The Mission should discuss how important IP capacity for scaling up is to its strategic objectives and whether it prefers to scale up geographically (through smaller CBOs and organizations based in multiple communities) and/or scale the range of services provided to well-known beneficiaries (through national/subnational organizations). The ET sees this as strategic. EQ 2 The Mission is using a three-prong methodology to provide capacity building. To what extent have the capacity building efforts and model been effective? What has and has not worked well and why? 2. USAID/Sri Lanka should continue flexible adaptive approaches to capacity building that optimize choices for IPs. The approach emphasizing choice (with selected coaching) was unique and appreciated by the IPs who are used to standardized programs. Without customization and flexibility, capacity building programs add less value and, in the worst case, waste the time of managers. 3. Increase IPs’ annual funding when they are able to comply with USAID’s system requirements. When working with weaker-capacity IPs in a multi-year project, limit first-year project budgets until the IP has demonstrated capacity to apply required USAID procurement, financial management, and human resource systems. Since capacity building is a USAID/Sri Lanka priority, the ET recommends the Mission pay for these activities in future programs directly. If future programs do not provide capacity building activities, ask IPs to include specific components for building organizational and technical capacity within their project proposal. IPs should keep the line item flexible so that it can be guided by an assessment exercise and allows for a choice of providers. Development Grants Program Mid-term Evaluation 19 4. Require IPs to pass on capacity building and hold internal gender training for field workers. Even though many field workers have good skills and instincts, this is less common among men who work with women beneficiaries. IP discussion of how women are affected by project activities rarely includes broader questions of whether they are empowered in the process. However, after staff leadership participate in gender training, most IPs can conduct their own training internally. To enable this, gender workshop facilitators (capacity building partners) should provide materials to IP participants and include a planning session on “spreading gender integration” throughout the organization.  Similarly, IPs should pass capacity building learning on to field workers, few of whom benefited from the capacity building programs provided. Even though training was mostly management-oriented, discussing issues with field workers increases their sense of ownership and understanding of the big picture affecting their IP. 5. USAID/Sri Lanka should strengthen its approach to the OCA tool by encouraging IPs to retake the self-assessment at the end of the funding or CD period to compare the initial assessment against progress to date. This can be done directly by the Mission or performed by a capacity building IP, such as SPICE. The purpose should be less quantitative and more to gauge if the IP’s internal conversation about capacity is creating results leading to change. Other organizational development tools could be used to accomplish this.  Before the SPICE project ends, work with each IP to sustain their CD process (and include next-level needs) beyond the project life. Perhaps the recommended final OCA update would identify specific CD service providers, mechanisms, and timetables (there are indications this is underway). If possible, extend the duration of SPICE to allow this adjusting deliverables as needed. EQ 3 How effective are DGPs in delivering planned results in their technical fields? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. 6. To effectively deliver technical services, IPs must combine strong technical competence with solid community-based groups. Some IPs were effective in technical service delivery in their focus area without forming sustainable groups. These are required for communities to address future development issues more independently. The ET discovered many IPs lack skills in group formation from the start, USAID can help them build capacity through training activities, or IPs can link community organizations to existing groups in their area. The latter is more sustainable.  IPs should then help community-based groups link to support networks. Whether from national, local government, other organizations, or projects, it is likely that village or CBOs will need linkages to support their ongoing technical work. IPs should be aware of or locate these networks in the broader area of the community geographically and in the area of technical focus (agriculture, livelihood, psychosocial services, and water). These activities are implied in the sustainability work IPs already do with CBOs. USAID should emphasize these activities in an IP’s future terms of reference. 7. USAID/Sri Lanka should continue to fund efforts holistically, addressing psychosocial and livelihood needs in a post-conflict environment. Few other donors support this subsector, as resources are limited. Without a holistic approach, livelihood and basic services have limited impact particularly given the challenge of recovery. Much current effort follows a traditional one-on-one individual client model—future efforts could integrate more family, group, and community support to follow up one-on-one counseling. Explore integrating livelihood activities through active partnerships with other local organizations with Development Grants Program Mid-term Evaluation 20 experience and a strong track record of effectiveness. DGP experience proved the effectiveness of livelihood work alongside other interventions. EQ 4 Mission-level Recommendations 8. USAID/Sri Lanka staff and management should discuss capacity development issues emerging from the DGP evaluation. Because the DGP was initiated during a downturn in funding and there was less funding available for other new programs, staff (particularly AORs) had more time available to work with the OCA and support a complex program with IPs in the field. The current context is the opposite, as the Mission increases staff numbers anticipating program expansion over the next several years. CD (organizational or systemic) should remain an ongoing priority. The Mission should determine its approach(es) through a structured internal dialogue initiated by Fall 2016 to engage the support of newly assigned staff and aligned so the results inform its programming cycle. Participants must include program and contract officers as well as management. Discussion points should include:  What is the extent of USAID/Sri Lanka’s commitment to CD? Is it the long-term legacy USAID/Sri Lanka’s development assistance will leave to the country and key organizations? If generally, yes:  What options are effective and efficient to integrate capacity building into programs involving Sri Lankan IPs? What USAID modalities best support this?  To what extent is it effective to work directly with small and medium-size direct grantees versus through international or national IPs? Risks? Transaction and opportunity costs versus international partners? 9. USAID/Sri Lanka should build learning systems13 into future CD efforts. According to Nils Boesen,14 “capacity of one or more organizations can be seen as an element in a wider system”—a capacity development system.15 For Boesen and the CD community at large, CD is no longer supply-driven, but rather controlled by governments and organizations and embedded in the results chain leading to development impact. The implications for USAID/Sri Lanka operations are multiple.  The practical capacity to work systemically is new to both the Mission and its local partners. However, systems approaches are already embedded in USAID Forward through emphases 13 The terms “learning” or “capacity systems” fall under the broader concept of “open systems theory.” Open systems (in the context of organizations) simply says effective organizations and people constantly “actively adapt” to changes in their external environment. As opposed to working in isolated silos or attempting to control people and information, open systems thrive under different assumptions. For nearly 75 years, researchers and practitioners—including those emerging from appreciative inquiry and Peter Senge’s Fifth Discipline—have based their work on open systems theory. Annex XIII contains a comparison of core assumptions driving conventional bureaucratic approaches versus open systems. In our context, we can easily make the connection from these to “open learning systems” or “open capacity development systems.” 14 Nils Boesen is currently a United Nations Development Program (UNDP) group leader for CD. The quote is from his Practical Guide to Capacity Development in a Sector Context. Asian Development Bank. August 2011. http://www.adb.org/sites/default/files/institutional-document/33285/files/cd-guide-sector-context.pdf. 15 Annex XIV contains a list of success factors in designing capacity development assistance developed by the Asian Development Bank (ADB). These would be a useful reference to consider when programming CD. Development Grants Program Mid-term Evaluation 21 on local systems, scaling for impact, and the Learning Lab’s support for collaborating, learning, and adapting—yet they are not well exploited in USAID/Sri Lanka.  Therefore, management and senior staff at USAID/Sri Lanka must take the lead16 by helping all staff integrate systems approaches across their action and work plans, particularly by supporting and incentivizing capacity development systems. Plans can be peer reviewed internally, while managers guide staff on how to make this work throughout operations—to succeed, mutual accountability must be created.17 In the future, the Mission will need to strengthen its capacity to develop, support, and share good CD practice as a key co￾contributor across like-minded groups and sectors in Sri Lanka. A lingering silo mentality fights against this systems approach.  Beyond plans, USAID/Sri Lanka should require this systems approach to CD be incorporated in program designs and have a mechanism to strengthen weak designs. System￾oriented tools, readily available through many sources (particularly the learning lab) should be applied prior to, or early on in, the project design process. 16 USAID has plenty of resources, tools, and support for a systems approach to CD. However, the limitation to a shift like this will be the mindset of management and staff in country missions who already have heavy workloads and are skeptical of investing time exploring new approaches that, in their thinking, may have little practical value and absorb precious time. To operationalize changes, they will need to experience the benefits themselves—and be given some room to experiment. 17 A strong approach would be to assign an internal “CD systems” working group with deadlines and deliverables to ensure the process fits USAID/Sri Lanka. However, it may be useful to have experts from either the Learning Lab or Regional Development for Asia conduct a workshop for several half days in a row on-site in Colombo to jumpstart this process. This could be combined with or following the internal discussion proposed in Recommendation 8. Development Grants Program Mid-term Evaluation 22 ANNEXES Development Grants Program Mid-term Evaluation 23 ANNEX I: EVALUATION STATEMENT OF WORK C.1 BACKGROUND INFORMATION USAID/Sri Lanka manages the Agency’s largest Development Grants Program (DGP) portfolio in the world in terms of number of awards, rather than total value. In support of President Obama’s 2013 “Stand with Civil Society Initiative” to strengthen civil society organizations, the Sri Lanka DGP portfolio builds on past successes and supports economic regeneration and community recovery activities in economically-lagging and former-conflict areas. More specifically, the awards in the DGP portfolio: (1) Strengthen the capacity of service providers to provide both access to quality services for people with disabilities and better livelihood opportunities; (2) Support the acceleration of sustainable economic recovery and growth opportunities in economically-lagging and former conflict areas; and (3) Support the strengthening of selected Sri Lankan non-governmental organizations (NGOs). The DGP portfolio advocates for issues of common concern and establishes alliances with local government and/or private sector partners to deliver more effective services. Consistent with USAID/Sri Lanka’s overall strategy, these awards maintain a focus on interventions/services that respond to the needs of youth, women-headed households, ex-combatants, Internally Displaces Persons (IDPs), and other vulnerable populations. See Attachment One for the DGP PAD and Attachment Two for the logical framework of the DGP portfolio. A total of 13 DGP awards were finalized for implementation between 2012 and 2016. As of February 2016, there are 10 remaining active DGP awards which are managed by two Development Objective (DO) teams in the Mission: the Governance and Vulnerable Populations team (GVP) and the Economic Growth (EG) team. At this point in the implementation of the DGP portfolio, the Mission would like to assess the approach, methodology, and results of the active DGP awards to date. The mid-term evaluation findings and recommendations will provide both lessons learned to date and recommendations for future programming adjustments for local Implementing Partners (IPs). The primary objective of the evaluation is to determine the extent to which the project is on track to achieving the stated purpose of “improved management capacity and achieving technical objectives;” evaluate the strengths and weaknesses of Mission’s management of the DGP portfolio and the secondary objective is recommending corrective actions needed and/or areas for improvement to achieve programmatic effectiveness and impact. The scope of this mid-term performance evaluation will require assessing the ten remaining DGP grantees, the ‘Support for Professional and Institutional Capacity Enhancement’ (SPICE) program’s contribution to DGP capacity building efforts, USAID’s contribution to DGP capacity building efforts, and the achievement of the DGP’s overall goal and objectives. C.2 PROGRAM INFORMATION 1. Name: Development Grants Program (DGP) 2. Funding: $11,303,570.00 transferred in three tranches as follows: (1) $8,944,742.00/DV 2009/2010/August 2010, (2) $1,843,828.00/DV 2010/2011/Oct 2010, and (3) $515,000/Program Support/June 2013. 3. Number of Awards: Ten active awards 4. Awards Managers: Different AORs Development Grants Program Mid-term Evaluation 24 Following is a list of the 10 remaining active DGP activities that are included in this mid-term evaluation: No. Award End Date Organization Name Award TEC (USD) Geographic Region in Sri Lanka Focus Area(s) Mgmt. Unit 1 30 Mar 2016 Sevalanka Foundation 2,211,253 North IDP returnee assistance GVP 2 2 May 2016 Navajeevana Rehabilitation Tangalle 905,721 South and East People with disabilities GVP 3 7 May 2016 Peragamana Guild 249,540 East Livelihoods EG 4 31 May 2016 Green Movement Sri Lanka (GMSL) 414,082 North and East Livelihoods EG 5 9 July 2016 Nucleus Foundation 1,028,404 North and East Livelihoods EG 6 14 Aug 2016 Mencafep 452,464 Central and East People with disabilities GVP 7 14 Sept 2016 Family Rehabilitation Center (FRC) 869,219 North and East Psychosocial services GVP 8 31 Oct 2016 Sri Lanka Center for Development Facilitation (SLCDF) 836,680 North Livelihoods EG 9 7 Nov 2016 International Center for Ethic Studies (ICES) 420,000 National Interethnic￾religious/ research GVP 10 31 Jan 2017 Jaffna Social Action Center (JSAC) 1,047,733 (and 500,000 in other funding) North Livelihoods GVP C.3 DEVELOPMENT CONTEXT The 26-year conflict in Sri Lanka displaced more than 300,000 people, destroyed infrastructure, and damaged social networks in the Northern and Eastern Provinces. Even after significant aid interventions, extreme poverty, food insecurity, and sub-standard living conditions remain the norm across many provinces. As families returned to destroyed living spaces, their resettlement experience has been inadequate with little-to-no economic prospects. Consequently, many areas that were opened for resettlement under the previous government remain in need of significant recovery assistance. While the former government had taken steps toward post-conflict reconstruction and recovery, it did not substantively address reconciliation, human rights, and good governance. In 2012, many donors began Development Grants Program Mid-term Evaluation 25 winding down emergency assistance programs in the Northern and Eastern Provinces to focus more on Sri Lanka’s longer-term development challenges. This backdrop and a dwindling annual budget prompted the Mission to tap into the Agency’s DGP funding mechanism starting in 2012. Around the world, the DGP resources fund local NGOs to develop their organizational and technical capabilities to meet the needs of their own communities. As a result of the Mission’s commitment to the DGP portfolio, USAID has increased funding for and diversity of the number of local NGO and CSO we support. USAID also expects these local organizations to become stronger and more flexible and sustainable in the process of delivering much-needed services. Despite the end of the civil war, Sri Lankan society remains disturbed by ethnic tensions. In 2014, anti￾Muslim sentiment and attacks on Muslim communities increased and were a growing source of tension. The Tamils largely concentrated in the North and East of Sri Lanka, harbored distrust and anger towards the former government. This distrust was especially related to allegations of war crimes at the end of the conflict, a large number of unresolved disappearance cases, and allegations of sexual and gender￾based violence at the hands of the security forces. The Sinhalese, in turn, viewed the Tamils with suspicion and believed that they could restart an insurgency, if not tightly controlled. The government was resistant to engaging on issues of accountability and believed its activities during the war were fully justified. Under the previous government restrictions on civil liberties were imposed, human rights activists were targeted by government security forces and journalists were threatened and attacked. Activities funded by USAID experienced various forms of interference, from slow approvals by the GSL to varying levels of harassment and intimidation in the field by GSL authorities. This impacted the implementers’ ability to operate in a consistent, sustained manner. USAID’s assistance was a critical element and the lifeblood of many of the implementing organizations we currently fund. U.S. Government support played an important role in advocating for increased space, human rights, and increased tolerance of vulnerable communities. The operational environment has been changing for the positive since the January 2015 elections. Still, marginalized groups in Sri Lanka face significant barriers to full participation in social, educational, and political processes. USAID plays an important role in advocating for increased space and tolerance and believes now more than ever it is critical to sustain and emphasize capacity building support for CSOs. It is in this ever-changing environment that the mid-term evaluation team shall assess to what extent organizational capacity has improved and technical objectives have been met, in addition to making recommendations for the remaining implementation period. C.4 DGP INTENDED RESULTS The DGP Project Appraisal Document (PAD) sets out its principal objective as ‘increasing and diversifying the number and quality of NGO/CSO implementing partners to meet the needs of the communities they serve.’ The development hypothesis poses that through improved capacity and operational efficiency, local organizations would be more empowered to fulfill their goal of responding to the social and economic recovery and reintegration needs in economically-lagging and conflict￾affected areas. The PAD’s stated DGP sustainability goal is: ‘by the end of the project in 2016, Sri Lanka’s development would be led by local NGOs and individuals who would remain in the country and transform into stronger organizations in the long-term.’ The four indicators for the DGP portfolio are: 1. Number of service providers trained who serve vulnerable persons (3.3.2-13) 2. Number of USG assisted organizations and/or service delivery systems strengthened who serve vulnerable populations (3.3.2-15) 3. Number of vulnerable people benefitting from USG supported social services (3.3.2-15) 4. Number of microenterprises supported by USG enterprise assistance (4.7.3-6) A draft results framework is found in Attachment three. Development Grants Program Mid-term Evaluation 26 The anticipated long-term outcomes, as stated in the PAD, are:  Families, female-headed households and persons with disabilities establishing and benefiting from more sustainable livelihoods;  Communities accessing proper water/sanitation facilities; and  Ex-combatants able to function and lead a prosperous life within their communities.” The PAD envisioned that sustainability of the project will be shown by the ability of returnees to maintain productive livelihoods; by vulnerable populations and communities to successfully integrate into their societies; and by the ability of these groups to better advocate with the government and other organizations for common interests. One of the key elements of long-term sustainability of these activities will be the relationships that recipients have built and established with communities as well as with government officials to foster and maintain public goods during the project. It was expected that such relationships will continue to develop, strengthen, and evolve post-USAID involvement. However, that said, the evaluation team will have to take into consideration the current transitional political environment; relations at the national level were stymied, to say the least, under the previous government. Under the Sirisena government, we are starting to see some changes, some relationships built, and some connections resulting in positive change for civil society. C.5 APPROACH AND IMPLEMENTATION The capacity building approach identified as, "increase and diversify the number and quality of NGO/CSO implementing partners who could meet the needs of the communities,” is the essence of the DGP model and of direct grants to local organizations. These grants qualified under the Request for Application (RFA) that passed a pre-award survey conducted by the Mission or by KPMG/Sri Lanka. A core group of USAID staff members formed an Organizational Capacity Assessment (OCA) team in January 2013; over a six-month timeframe, the team conducted OCAs for each DGP organization. The OCA findings were then incorporated into the awards and are the primary tool on which action plans were based. The Mission then used a three-prong approach to provide capacity building support for the recipient organizations: 1. The organization itself was required to execute a plan to improve their capacity. 2. USAID staff was required to provide capacity building/training/mentoring efforts. The weaknesses identified in the OCA process were to be addressed by hands-on training from the USAID Office of Financial Management (OFM), the AORs, the Program and Policy Support Office (PPS), and the Office of Acquisition and Assistance (OAA). 3. SPICE18 was contracted to provide capacity building training and consultations. SPICE’s role is limited to contracting for the provision of training services. 18 SPICE is a 36-month activity with a total budget of just over $15 million. The SPICE activity is a civil society strengthening program and provides grants to national and regional level civil society and community based organizations. In addition to the grants, SPICE provides organizational development and capacity building support for these organizations in order to improve their management capabilities which will, among other objectives, Development Grants Program Mid-term Evaluation 27 Solicitation and Award Background 2009: The Mission participated in a USAID/E3/DGP RFA. A USAID/E3/DGP Technical Evaluation Committee selected ten applicants and passed those ten applications to the Mission. 2010: KPMG/Sri Lanka was contracted by the Mission to conduct pre-award surveys. Three of ten organizations passed the pre-award survey (Palm Foundation, LRWHF, and a third organization that dropped out later). Therefore, two awards were made in the first round. 2010: Based upon pre-award surveys, awards were made to Palm and LRWHF; no further DGP awards were made. 2011: The new Mission Director decided to revive the DGP project, and the PAD was approved on October 18, 2012. A phased approach was planned for the award of grants and startup of activities. The initial group of awardees (Palm and LRWHF) was to begin implementation of activities in late 2012, while the second group of awardees would begin implementation by the second quarter of 2013. 2012: The Mission, through USAID/IDEA/DGP office in Washington DC, issued another RFA with the remaining DGP funds. 125 applications were received, 13 were selected but two were dropped. Finally, 11 awards were completed (for a total of 13 DGP awards). The Technical Evaluation Committee that evaluated this round of proposals consisted of Mission staff with representation from the Offices of Transition Initiatives (OTI), EG, GVP PPS. Thirteen DGP grants were awarded to 13 local organizations. Three have already closed and therefore this mid-term evaluation will assess the remaining 10. Technical Implementation DGP activities focus on interventions/services that respond to the needs of youth, women-headed households, ex-combatants, Internally Displaced Persons (IDPs), and other vulnerable populations. Two modes of implementation are followed in the DGP portfolio: (1) the local organization as the service provider, and (2) the local organization initially serving as the service provider while helping to build capacity of a government entity and ultimately handing over the services. Descriptions of the DGP activities are found in Attachment four. Response must include but not be limited to the following areas: a. the key results and effectiveness and/or limitations of each approach and methodology. b. describe strengths and weaknesses of the OCA tool and its implementation in the Mission’s DGP context. c. analysis of the local service providers’ role (the SPICE sub grantees) and contribution. d. efforts to ensure sustainability of programmatic results. allow them to be better able to win grants from other donors and to develop alternate means of sustainability as donor dollars dwindle in Sri Lanka. Approximately a year after the start-up of SPICE, an amendment was prepared that incorporated the provision of SPICE-contracted capacity building services to the Mission’s DGP implementing partners. Development Grants Program Mid-term Evaluation 28 1. How effective are the DGPs in delivering the planned results in their technical field? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. Response must include, but not be limited to, the following areas: a. effectiveness in integrating women's issues and women’s leadership role in civic life. b. use of the results framework (RF) – are there results achieved that are not in the RF? Do the evaluation findings differ from the RF? If yes, please describe. 2. With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and the local organizations that are feasible to achieve in the remaining period of performance. Recommendations shall address improvements for: a. capacity building performance results. b. technical performance results. The contractor must prepare responses in the form of specific and actionable recommendations. C.6 EVALUATION DESIGN AND METHODOLOGY C.6.1 EVALUATION DESIGN AND PROCESS This mid-term performance evaluation is intended to answer the evaluation questions presented above. The suggested conceptual approach to be used to answer these questions will focus on, but not be limited to, the following: desk study, key informant interviews, site visits, focus group discussions, and consultations with relevant stakeholders. Other applicable methods will be welcomed as appropriate. An evaluation team comprised of independent external consultants, with support from USAID staff, will examine the performance of the DGP project. While the evaluation should address past performance, USAID/Sri Lanka is keenly interested in forward-looking recommendations on strategies and corrections to achieve planned results during the duration of the project. The independent external consultants are expected to work in conjunction with USAID/Sri Lanka’s PPS Office to plan and implement the proposed evaluation. The consultants are expected to provide significant overall leadership and direction, and exercise a degree of autonomy, as well as have the final responsibility for conducting the evaluation and completing evaluation deliverables. C.6.2 DATA COLLECTION AND ANALYSIS METHODS The evaluation team will be required to evaluate this multi-faceted project in a timely manner. Data requirements, collection methods, and required analyses will be determined collaboratively with USAID/Sri Lanka, under the direction of an independent evaluation team leader. Details on final data collection methods (including evaluation instruments and key informants or respondents), and analytical framework(s) will be approved by USAID/Sri Lanka (the PPS office specifically) as part of the initial work plan approval. Data, where applicable, are expected to be disaggregated by sex and level of intervention (region). As summarized below, the data collection and analysis process will comprise of three phases. All questions stated in section C.5 must be addressed, to the extent practical, in all three phases. The desk study and internal consultations will support planning for external interviews and focus group discussions. Development Grants Program Mid-term Evaluation 29  Desk study: The evaluation team shall review existing documents and information listed in section 1.5 above, and work with USAID/Sri Lanka to acquire additional documents and information as needed, and prioritize primary data collection where gaps remain.  Internal Consultations: The evaluation team shall meet with key stakeholders. Meetings with USAID/E3 in Washington DC to understand the DGP framework may be conducted via conference calls. In Sri Lanka, the evaluation team will hold a group meeting with members of the OCA team, a meeting with the EG and GVP AORs who manage DGPs including OAA, PPS, OFM.  External interviews and focus group discussions: The evaluation team will conduct in￾person interviews and focus group discussions with project implementing partners, collaborating partners, selected project beneficiaries, donors with similar projects, and other key stakeholders to allow for a range of perspectives and give depth to the evaluation. The team will conduct meetings via conference calls with CPI and MSI to become grounded in the role that the SPICE activity plays. The Contracting Officer’s Representative (COR) of this Task Order will provide lists of potential informants/respondents once the task order is awarded. C.6.3 METHODOLOGICAL STRENGTHS AND LIMITATIONS Methods Strengths Limitations Desk Study  Provides background on the program to be evaluated and valuable information on substantive issues and assists in generating a list of questions that can be used in primary data collection.  Helps to focus efforts and prioritize issues and gaps  Depends on resource availability Consultations with E3 and IP HQ  Provides valuable information on substantive issues and generates a list of questions including key stakeholders that can be used for other evaluation methods.  Provides greater depth and insights and general information.  Depends on availability of key stakeholders. Individual face-to-face interviews with DGP partners and beneficiaries in the field  Potentially data rich and detailed answers  Might require translation, thus, possible loss of meaning and data richness.  With security concerns, selection of informants might be limited to certain areas that might result in selection bias. Focus group discussions  Can generate a broader range of ideas and responses.  Can include a greater number  Might require translation, thus, possible loss of meaning and data richness. Development Grants Program Mid-term Evaluation 30 Methods Strengths Limitations of participants in less time and result in rich discussion, if facilitated well.  Some respondents may dominate the discussion, if not facilitated well.  Same security issue as above. C.6.4 LOGISTICS USAID/Sri Lanka will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID/Sri Lanka will assist in identifying key stakeholders. The evaluation team is responsible for arranging meetings, as identified during the course of this evaluation and advising USAID/Sri Lanka prior to each of these meetings. The evaluation team is also responsible for arranging their international and domestic travels, accommodations, and vehicle rental and drivers as needed for site visits in Sri Lanka. The evaluation team will be responsible for procuring its own work/office space, computers, internet access, printing, and photocopying. Evaluation team members will be required to make their own payments. USAID/Sri Lanka personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Team members should have the necessary language skills for working in Sri Lanka, or engage local language interpreters to support interviews and reviews of local language documents and records when necessary. Development Grants Program Mid-term Evaluation 31 ANNEX II: LIST OF DGP GRANTEES  FRC  GMSL  ICES  JSAC  Lanka Rain Water Harvesting Forum (LRWHF)  Mencafep  Navajeevana Rehabilitation Tangalle o Ladder of Hope (sub-grantee) o Deaf Link (sub-grantee) o Lanka Evangelical Alliance and Development Services (sub-grantee)  Nucleus Foundation o Rural Humanity Services Foundation (partner) o Right Brain Network (sub-contractor)  Palm Foundation  Peragamana Guild  Sevalanka Foundation  Shantiham  SLCDF SPICE had the following CD partners (sub-grantees) working with DGP IPs:  Partners in Alternative Training (PALTRA)  Business Consultancy Services  Management Frontiers  Creators’ Forum  Strategic Inspirations Development Grants Program Mid-term Evaluation 32 ANNEX III: DATA COLLECTION METHODS, TOOLS, ANALYSIS, AND DATA QUALITY ASSURANCE Questions and Sub￾Questions Sub-Questions Sample Methodology and Instruments Data Analysis and Deliverables Data Quality Assurance EQ 1: To what extent have the management capacity and sustainability of the 10 local organizations improved? At this point, is the project on track to meet the expected end results? Sub-Question 1.1 Are the 13 organizations showing tangible outcomes to maximize organizational performance and carry out high impact programs? Board Members, Management Staff, Field Staff, GOSL staff, INGOs KIIs Appreciative evaluation workshops Review of program documents and reports/research/ evaluations Qualitative content analysis Written reports Data collection methods follow the same procedures for each evaluator and over the course of field work. Each ET member shared the same meaning of the instruments. Discussed and confirmed methods choices with USAID for optimal utility. Combined self￾reporting of IPs, quarterly reports, USAID staff information, and direct observation in the field. Sub-Question 1.2 Have the 13 organizations demonstrated improved effectiveness and efficiency of their service delivery? Are their practices sustainable? Sub-Question 1.3 Are the 13 organizations on track to meet the expected end results? EQ 2: The Mission is using a three￾prong methodology in the provision of capacity building. To what extent have the capacity building efforts and model been effective? What has and has Sub-Question 2.1 To what extent have the capacity building efforts and models implemented by the 13 organizations themselves been effective? Board Members, Management Staff, Field Staff, GOSL staff, INGOs KIIs Review of program documents and other related reports/ research/ evaluations Appreciative Descriptive qualitative content analysis Final written report Biases were anticipated and mitigated by the ET before field work (cf. Methodology section). Findings were compared and prioritized. Development Grants Program Mid-term Evaluation 33 Questions and Sub￾Questions Sub-Questions Sample Methodology and Instruments Data Analysis and Deliverables Data Quality Assurance not worked well and why? Sub-Question 2.2 To what extent have the capacity building efforts of USAID been effective? evaluation workshops Weaker findings were excluded from reports. Cross-checking data from multiple stakeholders, including USAID, SPICE, and IPs (as participants). Sub-Question 2.3 To what extent has the SPICE contracting role of the provision of training services been effective? EQ 3: How effective are the DGPs in delivering the planned results in their technical field? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors. Sub-Question 3.1 How effective are the DGP IPs in treating and providing functional support for those affected by trauma and PWD? Management Staff, Field Staff, GOSL staff, INGOs Beneficiaries KIIs Review of program documents and other related reports/ research/ evaluations Appreciative evaluation workshops Small group interviews Descriptive qualitative content analysis Final written report Analytic conferences established face validity that findings fairly represented what we saw in the field and can be attributed to the DGP. Triangulation of findings from beneficiaries among the four-person ET. Reviewed interview protocols in utilization workshop with USAID staff. Balanced practical Sub-Question 3.2 How effective is the DGP IPs in providing accelerated sustainable economic recovery and growth opportunities in the former conflict zone? Sub-Question 3.3 How effective are the DGP IPs in enhancing the lives of PWDs and trauma survivors? Development Grants Program Mid-term Evaluation 34 Questions and Sub￾Questions Sub-Questions Sample Methodology and Instruments Data Analysis and Deliverables Data Quality Assurance Sub-Question 3.4 How effective are the DGP IPs in increasing access to shelter; WASH; and processing civil documentation of vulnerable populations? considerations of data collection to optimize the number of data collection activities. Sub-Question 3.5 How effective are the DGP IPs in improving access to livelihoods opportunities? EQ 4: With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and the local organizations that are feasible to achieve in the remaining period of performance. Sub-Question 4.1 What are the lessons learned? Management Staff, Field Staff, GOSL staff, INGOs Beneficiaries KIIs Review of program documents and other related reports/ research/ evaluations Appreciative evaluation workshops Small group interviews Descriptive qualitative content analysis; Final written report Rich descriptions were available from ET member notes with enough detail to compare with others and produce consistent findings. Use of constant comparative methods to draw out primary lessons across EQ1- EQ3 and synthesize findings across questions. Sought to ensure consistency across findings, conclusions, Development Grants Program Mid-term Evaluation 35 Questions and Sub￾Questions Sub-Questions Sample Methodology and Instruments Data Analysis and Deliverables Data Quality Assurance lessons learned, and recommendations. Cross Cutting Issue: Gender Management Staff, Field Staff, GOSL staff, INGOs Beneficiaries KIIs Review of program documents and other related reports/ research/ evaluations Appreciative evaluation workshops Small group interviews Descriptive qualitative content analysis; Final written report Safeguards were in place to ensure stakeholder confidentiality, especially for women and vulnerable groups. Use of sensitive data gathering involving female ET member, woman field workers, and leaders of women’s groups. Emphasized learning not judgment. Development Grants Program Mid-term Evaluation 36 ANNEX IV: DESK STUDY FINDINGS AND OBSERVATIONS FROM INBRIEF AND UTILIZATION WORKSHOP19 Desk Study Findings (May 17) Observations  Effectiveness in service delivery is not linked back to improved organizational capacity.  Effectiveness in service delivery is not linked back to improved organizational capacity.  Internal systems seem stronger (based on CD activity) and records.  To what extent are the system improvements driven by USAID requirements? If a fair extent, does this make sense for the future?  Results of CD activities and following their OCA plan are not mentioned. Trainings are mentioned or listed in budgets OCD lacks an explicit strategic approach.  If a key DGP focus is CD, do the IPs own and track progress against OCD plans? Application? Reporting?  IPs differ widely in organizational capacity.  How well can diverse IPs absorb and apply learning from CD action. Are we looking at relative or absolute improvement?  CD activities appear to align with the IP’s goals from OCA etc.  Is the whole (organizational development) greater than the sum of its parts (one-off trainings, etc.)? Expect differentiation across IPs.  13 IPs reached more than 44,000 direct beneficiaries. All 13 met or exceeded their proposed quantitative targets and quality of services. 12 of the13 IPs made good progress in improving their organizational sustainability. However, almost all IPs manage for outputs rather than outcomes.  IP reports focus mainly on outputs and (with few exceptions) not outcomes. M&E stats are complete but lack depth.  Is effectiveness limited when an outcome (results) focus is missing?  Quality of gender integration is unclear in IP docs (but gender disaggregated data on beneficiaries receiving services is present)  Is there gender empowerment in the field? Are women better off (or simply recipients?)?  Many packages of services are thoughtfully designed and well integrated.  Does it indicate a holistic view? Are they building on their strengths/core expertise?  From a quantitative standpoint, most IPs appear on track to completion (reaching targets).  Do we know much about quality of services (appropriate, met expectations, lead to better wellbeing)? How to understand this better? 19 Staff actively discussed and did not agree with all of these initial findings—one purpose of the workshop was to explore these with staff. The observations were not conclusions but questions for further exploration during internal consultation and fieldwork. Development Grants Program Mid-term Evaluation 37 ANNEX V: METHODOLOGY AND LIMITATIONS20 The evaluation used a multi-level, mixed methods approach, with a focus on qualitative data collection and analysis. The evaluation was conducted by a four-member team from May to July 2016, consisting of three phases: 1) a comprehensive desk study of all relevant documents; 2) approximately four weeks of field data collection in Sri Lanka, involving qualitative data collection in ten cities and a total of 436 respondents; and 3) analysis of all data collected, report writing, and discussion of preliminary findings (while in the field) and an out-brief presentation of draft findings (at the USAID/Sri Lanka Mission). The primary methods of data collection were (1) desk review and analysis; (2) KIIs to explore issues in￾depth with individual stakeholders; (3) participatory workshops and small group interviews to discuss open-ended questions; (4) a mini survey to gather data on results in IPs’ technical fields; and (5) site visits among IPs to examine issues on service delivery to beneficiaries. USAID/Sri Lanka, SPICE and the IPs themselves helped the ET access a strong and diverse sample in Colombo and across the data collection in the field. DOCUMENTS For the desk study, the ET reviewed program documents from USAID and SPICE Sri Lanka. These documents included: PAD for the DGP; DGP Logical Framework; Scope of Work for the provision of Organizational Development Training to the Capacity Building Component of the USAID-Funded SPICE Project; and SPICE Mid-Term Review. The team reviewed a range of documents and other relevant data from IPs, including those whose awards had already expired. These were: program descriptions; OCA action plans; CD action plans; annual work plans, quarterly reports, and M&E plans. In addition, for expired awards, the team reviewed final reports (see Annex IV for a full list). KIIS, SMALL GROUP INTERVIEWS, AND APPRECIATIVE EVALUATION WORKSHOPS Prior to fieldwork, the ET met with key stakeholders—including a conference call with USAID/E3 in Washington D.C. to understand the DGP framework as well as staff from CPI and MSI for information on SPICE’s role. In Sri Lanka, the ET met with key USAID staff and conducted interviews with project IPs, SPICE and its CD partners. IP staff, selected project beneficiaries, community mobilizers, CBO representatives, government agency representatives, donors with similar projects in the field, and other stakeholders were also interviewed, allowing for breadth and depth in this evaluation. Small group interviews and appreciative evaluation workshops—three-hour participatory workshops to promote IP staff discussion on changes in IP capacity and effectiveness of service delivery engaged IP staff in the evaluation. In total, the ET conducted 146 KIIs (56 females and 90 males) during field visits as well as in Colombo, and 11 evaluation workshops with IPs. The ET also conducted a total of 53 small group interviews: 31 in the North, and 22 in the East and Nuwera Eliya, with 288 beneficiaries (176 females and 112 males). The table below presents the statistics by data collection method. 20 This Annex is an expansion of the content in the main body of the report, providing greater details. Development Grants Program Mid-term Evaluation 38 Organization Gender KIIs Small group interviews Gender-wise Sub-total Total Mini Survey Family Rehabilitation Center Male 5 3 8 Female 30 1 4 18 22 Navajeevana Male 10 55 65 124 1 Female 10 49 59 Peragamana Guild Male 6 16 22 30 1 Female 4 4 8 Palm Foundation Male 8 10 18 36 1 Female 1 17 18 Mencafep Male 6 2 8 33 1 Female 10 15 25 ICES Male 3 3 6 8 1 Female 2 0 2 GMSL Male 8 8 16 35 1 Female 3 16 19 Nucleus Foundation Male 4 2 6 21 1 Female 1 14 15 SLCDF Male 5 0 5 28 1 Female 4 19 23 Shanthiham Male 8 0 8 20 1 Female 3 9 12 JSAC Male 7 13 20 35 1 Female 5 10 15 Sevalanka Male 10 0 10 17 1 Female 1 6 7 LRWHF, Colombo Male 1 0 1 3 1 Female 2 0 2 SPICE Male 3 0 3 4 Female 1 0 1 USAID Mission Male 7 0 7 12 Female 5 0 5 Total Male 90 112 203 436 13 Female 56 176 233 Total 146 288 436 Development Grants Program Mid-term Evaluation 39 FIELD VISITS The team visited several areas: Jaffna, Kilinochchi, Vavuniya, Mullaitivu, Trincomalee, Batticaloa, Ampara, Nuwari Eliya, and Tangalle. The ET spent two and a half weeks visiting IPs—dividing into two sub-teams to maximize geographic and focus area coverage throughout data collection. The sub-teams spent on average two days with each IP—including beneficiaries and other key stakeholders. See map below. Development Grants Program Mid-term Evaluation 40 MINI SURVEY OF DGPs A mini survey was sent to the IPs to collect key information, though data from three IPs was limited as their projects did not directly relate to the survey questions. The survey provided a recent consolidated picture about results in the IP’s technical fields in terms of the number of beneficiaries with access to services, such as the number of PWDs that accessed prosthetics and orthotics services; the number of PWDs and victims of trauma that have access to employment or economic opportunities; the number of citizens returning to conflict-affected areas who received support with civil documentation; the number of households using rainwater harvesting systems; and the number of transitional shelters established for families that did not receive permanent housing. The survey enabled the ET to use fresh data as a basis for discussion about service delivery and programming. LIMITATIONS Time constraints: The ET divided into two teams. Each sub-team travelled great distances to observe and interview beneficiaries. In certain locations, the ET had to travel more than 80 km from the IP’s office to reach field sites. This was unavoidable because disabled persons and war victims are scattered over working areas of the interior. Extensive travel limited time for other evaluation activities, such as daily analysis conferences. Nevertheless, the ET was able to complete data collection according to schedule, covering 13 IP organizations. Biases: The ET worked with USAID and other stakeholders to identify potential respondents, based on their programmatic experiences. IPs selected participants for small group interviews among beneficiaries, indicating selection and response biases. The team sought to overcome these biases by clearly communicating the purpose of the evaluation with all respondents, highlighting their role as evaluators, and the contribution of honest responses to the evaluation. Triangulation was another technique used to counter the effect of bias. The ET triangulated sample by involving diverse respondents. The ET triangulated data collection methods by using KIIs, group interviews, and participatory workshops. The team triangulated analysis by using multiple analysts in the team and by exploring emerging themes as well as those suggested by the EQs. Respondents were encouraged to speak openly with the reassurance that their responses would be kept anonymous. The use of ratios to express findings: Wherever possible, ratios are given to roughly indicate the extent of a particular finding. The ET based these ratios on evidence gathered during interviews, not quantitative surveys. These are not conclusive, and should not be used to generalize findings across IPs. Development Grants Program Mid-term Evaluation 41 ANNEX VI: SUMMARY OF FINDINGS FROM DESK STUDY, INTERNAL CONSULTATIONS, AND FIELDWORK Questions and Findings Desk Study Internal Consults Field Work EQ 1: To what extent has the management capacity and sustainability of the 10 local organizations improved? At this point, is the project on track to meet the expected end results?  13 IPs reached more than 44,000 direct beneficiaries. All 13 met or exceeded their proposed quantitative targets and quality of services. 12 of the13 IPs made good progress in improving their organizational sustainability. However, almost all IPs manage for outputs rather than outcomes. X X X  For most IPs, the three-year time period of the DGP was inadequate to complete the CD work while undertaking projects. X X  12 of the 13 IPs demonstrated incremental increases in effectiveness and efficiency of service delivery— for two reasons: (a) USAID selected high performing organizations, and (b) field level effects of CD would not be felt for two to three years. X X EQ 2: The Mission is using a three-prong methodology in the provision of capacity building. To what extent have the capacity building efforts and model been effective? What has and has not worked well and why?  USAID/Sri Lanka staff took an innovative approach to the DGP (OCA implementation, degree of staff engagement and flexibility). The three-prong method in providing capacity building worked well for all IPs. X X  OCA results during the DGP produced ambitious plans for each IP. SPICE helped each IP to narrow and prioritize its capacity building activities. X X  Unique features, such as choice of content and provider as well as customization to the IPs current context and challenges increased the effectiveness of CD activities. X X EQ 3: How effective are DGPs in delivering planned results in their technical field? What have been the biggest challenges to their success? Describe achievements made through USAID funding in the technical sectors.  IPs reported 1,074 PWDs and victims of trauma are earning income and participated in capacity building activities; 903 PWDs established constructive linkages with private sector organizations; 1,146 participated in microenterprise value chains; and 1,458 individuals accessed credit for livelihood X X Development Grants Program Mid-term Evaluation 42 Questions and Findings Desk Study Internal Consults Field Work activities.  Particular IPs implemented awareness programs to reduce the stigma (and shame) of PWDs. X X  11 out of 12 IPs engaged in livelihood generation activities, without which trauma and WASH components would not sustain. Activities suited very poor to middle income returnees who had lost businesses. X  IPs reported that 521 citizens returning to conflict-affected areas received support with civil documentation; 405 families received transitional shelters; 3,103 households now have access to safe and reliable water supply; and 974 households have access to sanitation facilities. Communities are limited by remote locations. X X  Women included in the DGP were empowered by obtaining civil documents asserting legal rights, ownership of assets, access to credit, marital rights, and rights to conduct business. X X  One IPs innovative approach to research and use of popular communication medium effectively supported dialog around ethnic reconciliation based on first-hand experience supported by rigorous research. X X X  Some IPs operate in geographic or technical silos with little contact with outside organizations where they could share and learn good technical and organizational practice and collaborate on project implementation. X EQ 4: With several months to a year or more remaining in the implementation period for the awards, provide recommendations/corrective actions to improve performance results by USAID, the service providers, and the local organizations that are feasible to achieve in the remaining period of performance.  USAID/Sri Lanka staff play a dual role as decision-makers and advisors for the IPs. IPs find it difficult to disclose their implementation challenges. X X  System capacities were not the focus of the DGP, instead it was the capacities of individual IPs. Similarly, a learning systems approach was not used to its full potential as capacity building activities were left primarily to USAID and SPICE. Working with small and medium size direct grantees required a significant investment of time by USAID staff. X X X Development Grants Program Mid-term Evaluation 43 Questions and Findings Desk Study Internal Consults Field Work  The lesson of selecting strong IPs cannot be under emphasized. But, a more strategic long-term approach would be to use the IPs to build the capacity of local service providers or larger CBOs in their technical focus area (a “training of trainers” model). X X X Development Grants Program Mid-term Evaluation 44 ANNEX VII: DOCUMENTS REVIEWED USAID and SPICE Sri Lanka Documents 1. USAID/Sri Lanka. Project Appraisal Document (PAD) For The Development Grants Program (DGP). Sri Lanka, USAID, 2012. 2. USAID/Sri Lanka. DGP Logical Framework. Sri Lanka, USAID. 3. USAID/Sri Lanka. Scope of Work for the provision of Organizational Development Training to the Capacity Building Component of the USAID-Funded SPICE Project. Sri Lanka, USAID. 4. USAID/Sri Lanka. Action Memorandum- Project Authorization for the Civil Society Strengthening Project under Development Objective (D.O.) # I-Strengthened Partnership between the State and Citizen. Sri Lanka, USAID, 2012. 5. SPICE/Sri Lanka Sri Lanka. Program Description MSI Proposed SPICE Project Program Description. Sri Lanka, SPICE, 2013. 6. SPICE/Sri Lanka. Support for Professional and Institutional Capacity Enhancement (SPICE) Mid-Term Review. Sri Lanka, SPICE, 2015. Expired DGP Awards 1. LRHF. Final Report. Water Security for Resettlement Areas in the North of Sri Lanka through Rainwater Harvesting. Sri Lanka, 2013. 2. PALM Foundation. Project Description. Sri Lanka. 3. Project Description. Shantiham, Sri Lanka. 4. Project End Report. Enhancing Resilience and Recovery Via Integrated Mental Health and Psychosocial Health Services in The Post Conflict Setting. Shantiham, Sri Lanka. FRC Documents 1. FRC. OCA Action Plan. 2. FRC. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment 2012. 4. FRC. Programme Description. 5. FRC. Quarterly Reports. Sri Lanka, 2013, 2014, 2015. 6. FRC. Annual Work Plan. 2014. 7. FRC. M&E Plan. Sri Lanka, 2015. GMSL Documents 1. GMSL. OCA Action Plan. 2. GMSL. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment, 2012 (4). 4. GMSL. Programme Description. 5. GMSL. Quarterly Reports. Sri Lanka, 2015. 6. GMSL. Annual Work Plan & Performance Monitoring Plan, Sri Lanka, 2015, 2016. ICES Documents 1. ICES. OCA Action Plan. 2. ICES. Capacity Development Action Plan. Development Grants Program Mid-term Evaluation 45 3. USAID. Organizational Capacity Assessment. 2012. 4. ICES. Programme Description. Sri Lanka. 2013. 5. ICES. Quarterly Reports, Sri Lanka. 2015, 2015. 6. ICES. Annual Work Plan. 2014. 7. ICES. Annual Work Plan and Gantt Chart. 2014. 8. ICES. M&E Plan. Sri Lanka, 2014. JSAC Documents 1. JSAC. OCA Action Plan. 2. JSAC. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. JSAC. Programme Description. Sri Lanka, 2013 5. JSAC. Quarterly Reports. Sri Lanka, 2013, 2014, 2015. 6. USAID/JSAC. Modification of Assistance. Mencafep Documents 1. Mencafep. OCA Action Plan. 2. Mencafep. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment, 2012. 4. Mencafep. Programme Description. 2013. 5. Mencafep. Quarterly Reports. Sri Lanka, 2013, 2014, 2015. 6. Mencafep. Annual Work Plan. 2015 7. Mencafep. M&E Plan. Sri Lanka, 2013. Navajeevana Documents 1. OCA Action Plan-Navajeevana. 2. Navajeevana. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. Navajeevana. Programme Description. 2014. 5. Navajeevana. Quarterly Reports. Sri Lanka, 2013, 2014, 2015. 6. Navajeevana. Annual Work Plan. 2015, 2016. 7. Navajeevana. M&E Plan, 2013-2016. Sri Lanka, 2013. Nucleus Documents 1. Nucleus. OCA Action Plan. 2. Nucleus. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. Nucleus. Programme Description. 2016. 5. Nucleus. Quarterly Reports. Sri Lanka, 2015, 2016. 6. Nucleus. Annual Work Plan. 2015. 7. Nucleus. M&E Plan. Sri Lanka, 2015. Peragamana Documents Development Grants Program Mid-term Evaluation 46 1. Peragamana. OCA Action Plan. 2. Peragamana. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. Peragamana. Programme Description. 2015. 5. Peragamana. Quarterly Reports. Sri Lanka, 2015. 6. Peragamana. Annual Work Plan. 2013, 2014. 7. Peragamana. M&E Plan. Sri Lanka, 2014. Sevalanka Documents 1. Sevalanka. OCA Action Plan. 2. Sevalanka. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. Sevalanka. Program Description. 2015. 5. Sevalanka. Quarterly Reports. Sri Lanka. 2013, 2014, 2015. 6. Sevalanka. Annual Work Plan. 2015. 7. Sevalanka. Performance Management Plan. Sri Lanka, 2013. SLCDF Documents 1. SLCDF. OCA Action Plan. 2. SLCDF. Capacity Development Action Plan. 3. USAID. Organizational Capacity Assessment. 2012. 4. SLCDF. Program Description. 2015. 5. SLCDF. Quarterly Reports Sri Lanka. 2015, 2016. 6. SLCDF. Annual Work Plan. 2015. 7. SLCDF. M&E Plan. Sri Lanka. 2014. Development Grants Program Mid-term Evaluation 47 ANNEX VIII: DATA COLLECTION INSTRUMENTS INTERVIEW PROTOCOLS Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are currently working with USAID to gain a better understanding of the DGP. This is a final evaluation and will be used by USAID and IPs to inform results and impact, and to also look at any reasonable improvements that can be made to implementation and planning moving forward. Our team has had the opportunity to review documents provided by USAID to get a better sense of the design and implementation of the DGP. However, such documents can only tell us so much. We would like to speak with you today to hear about your opinion about the project implemented by the DGP IP, in order to help us better understand how these projects look and function “on the ground.” We have planned this interview to last no longer than 30 minutes. During this time, we have several questions that we would like to cover. Please note that, at any time, you may terminate the interview or decline to answer a specific question. You may also decline participation in this interview. Confidentiality Protocol We will collect information on individuals’ names, organizations, and positions. A list of KIs will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless the respondent grants express written consent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? A. DONOR/OTHER NGOS: INDIRECT STAKEHOLDERS (GOSL, INTERNATIONAL NGOS, DONOR PROJECTS) Context 1. How did you know the DGP project? a. How long has your project been in operation here? b. What are your interventions in this location? c. How long you had known the IP? Description 2. Please describe your experiences with the project. (Please be specific) a. Do you see the differences when you compare the situation before the project and after the project? 3. Who receives services under this project a. To what extend the project is inclusive of vulnerable and marginalized population? b. Are both women and men involved? How? Development Grants Program Mid-term Evaluation 48 4. What has worked well with the project? (Please be specific) a. Are there factors, actions, or conditions that contributed to this? b. How did the IP coordinate with you and similar actors? c. Any best practices that you could observe? 5. What struggles or challenges did the project experience? a) What do you think of the quality of the project? b) How do you feel the community has responded to the project? c) Did the project face any challenges if so what were they? d) How did the organization overcome those challenges? Recommendations 6. How could a donor strengthen this type of grants project? 7. Is there anything else you wish to share about your experience with the DGP project and the organization which supports you? B. USAID, SPICE (INDICATIVE INTERVIEW PROTOCOL) INTRO: Fast forward to the June 10 evaluation out-brief. Pretend you have been part of our ET in the field. What are your initial findings and conclusions for each of your IPs, one-by-one? Context 1. How did you become involved with the DGP project? a. What do you see as the project’s purpose? b. What is your role with the project? c. Role of E3 partnering with the Sri Lanka USAID Mission? Description 2. Please describe your experiences with the project. (Please be specific) a. SPICE for capacity building to your IPs? b. USAID capacity building to your IPs? c. How do feel about the capacity building dimension? 3. What has worked well with the project with your IPs? (technical areas; please be specific) a. What things helped the IP succeed in their programs? b. Are there other factors, actions, or conditions that contributed to this? c. What did you / the mission learn from this? 4. What struggles or challenges did the project experience? a. Are there factors, actions, or conditions that hindered the IP? b. Were they able to overcome this/these? (If so, how?) c. What did you / the mission learn from this? Analysis 5. Were there any gaps in the project? (Missing elements?) a. Were the results what you expected? b. Did your IPs improve access to services for the most vulnerable and poor? c. Describe what specifically was done to reach and engage women in the project? 6. Were there any changes in your IPs since the project began? a. Are they mostly the same? Have they become stronger (If so, how?) b. Who participated in (project-related) capacity building programs? Are you aware if they applied what they learned to the DGP? Recommendations Development Grants Program Mid-term Evaluation 49 7. What message do you think the IPs would give to USAID about the DGP approach? a. What should they keep? b. What should they change or remove? c. What should they add? 8. Is there anything else you wish to share about your experience with the DGP project and USAID or your IPs? C. IP LEADERS AND OFFICE Context 1. How did you become involved with the DGP project? a. What do you see as the project’s purpose? b. What is your role with the project? c. Have you worked on similar projects before? (If so, please tell me a little about that.) Description 2. Please describe your experiences with the project. (Please be specific) a. Describe a typical day with your organization? b. How do feel about your work? 3. Were there any changes in your organization since the project began? a. Is it mostly the same? Has it become stronger (If so, how?) b. Were you able to participate in a (project-related) capacity building program with your organization? (describe in detail) c. Were you able to apply learning from CD in your work? How, specifically? 4. What has worked well with the project? (Please be specific) a. What things did staff do to help this project succeed? b. Are there other factors, actions, or conditions that contributed to this? c. What did you learn from this? Were you able to use this learning in other settings? 5. What has worked well with the project? (Please be specific) 6. What struggles or challenges did the project experience? Analysis 7. Apart from more funding, were there any gaps in the project? (Missing elements?) a. Were the results what you expected? b. Describe how your work has improved access to services for the most vulnerable and poor in the community where you work? c. Describe what specifically was done to reach and engage women in the project? Recommendations 8. Apart from more funding, what does your organization need to work more effectively? 9. What message would you give to a donor about how this type of grant project could be more effective if they tried it again in the future? a. What should they keep? b. What should they change or remove? c. What should they add? 10. Is there anything else you wish to share about your experience with the DGP project and your organization? D. IP KEY STAFF AND MANAGERS Development Grants Program Mid-term Evaluation 50 Context 1. How did you become involved with the DGP project? a. What do you see as the project’s purpose? b. What is your role with the project? c. Have you worked on similar projects before? (If so, please tell me a little about that.) Description 2. Please describe your experiences with the project in the community. (Please be specific) a. How do feel about your work? b. How did the staff in your organization coordinate with you regarding this project and how often they share information? 3. What type of people received services under this project? a. What motivates them to approach this service? b. Do the beneficiaries have access to similar services in the community? 4. What has worked well with the project? (Please be specific) a. What things did staff do to help this part of the project succeed? b. Are there other factors, actions, or conditions that contributed to this? c. What did you learn from this? Were you able to use this learning in other settings? d. Did this project help for sustainability of future implementation? 5. What struggles or challenges did the project experience? a. Are there factors, actions, or conditions that hindered your work? b. Was the project able to overcome this/these? (If so, how?) c. What did you learn from this? Were you able to use this learning in other settings? d. How is the organizational M&E structure aligning with this project? Analysis 6. Apart from more funding, were there any gaps in the project? (Missing elements?) a. Were the results what you expected? b. Describe how your work has improved access to services for the most vulnerable and poor in the community where you work? c. Describe what specifically was done to reach and engage women in the project? d. Have you developed the policies for any services (HR, Admin, travel) and how far is it in practice? e. Have you come across any overlapping activities by other stakeholders? If so how do you justify your services? 7. Were there any changes in your organization since the project began? a. Is it mostly the same? Has it become stronger (If so, how?) b. Were you able to participate in a (project-related) capacity building program with your organization? If so, how were you able to apply what you learned to the project? c. Could you explain how far the staff have been strengthened their capacity? d. Can you describe what you see would be the ideal standards of development for your organization 8. What percentage of government counter parts/ stakeholders meetings that you/ your organization? Recommendations 9. Apart from more funding, what does your organization need to work more effectively in the community? Development Grants Program Mid-term Evaluation 51 10. What message would you give to a donor about how this type of grant project could be more effective if they tried it again in the future? a. What should they keep? b. What should they change or remove? c. What should they add? 11. Is there anything else you wish to share about your experience with the DGP project and your organization? E. IP BENEFICIARIES / INDIVIDUAL / FOCUS GROUP: BENEFICIARIES PROTOCOL Context 1. How did you become involved with this project? a. How long have you been in this location? b. What were you doing for ________ before this assistance? c. How long have you been doing this activity? Description 2. Please describe your experiences with the project. (Please be specific) a. Who participates in this project? b. How do you feel about the assistance provided to you by this organization? c. What changes do you see before and after the project? 3. What has worked well with the project? (Please be specific) a. Overall improvement in wellbeing of you or your family? Sustainable? b. How far your lifestyle improved after the service received by you? c. Are there other factors, actions, or conditions that contributed to projects activity? 4. What struggles or challenges did the project experience? e) What do you think of the quality of the services provided by this project? f) How do you feel the community has responded to the project? g) If you have any problems are you comfortable to contact the project for assistance? Analysis 5. Apart from more funding, were there any gaps in the project? (Missing elements?) a. Were the results what you hoped for this project? b. Describe what specifically was done to reach and engage women in the project? c. Have you come across any overlapping activities by other projects? d. Do you think there are more people in your community who could benefit from such a project? Recommendations 6. How could the organization improve their work in your community (quality)? a. If they did the project over, what should they change? 7. Is there anything else you wish to share about your experience with the DGP project and the organization which supports you? Interview Topics to Probe MANAGEMENT CAPACITY AND SUSTAINABILITY EFFECTIVENESS Strategic plan Trauma and PWDs (IP services) Development Grants Program Mid-term Evaluation 52 Organizational governance Access to prosthetics/orthotics Organizational management Access to physical rehabilitation support Program delivery Timing / quality of support Financial management Local prosthetics/orthotics Systems for quality Social inclusion Accountability and sustainability School participation Networks and alliances Community participation Resource mobilization Acceptance of survivors’ needs Women’s leadership Economic recovery and growth Government partnerships Employment/economic activities (especially PWDs and trauma victims) Protection issues Household indebtedness Socio-economic recovery Farmer/fisher associations support Economic opportunities Building capacity of associations Food security Participation in value chains Scope and outreach Agricultural extension CAPACITY BUILDING Access to markets Capacity areas (IP) Access to credit Implementation (IP, USAID, SPICE) Technologies introduced to microenterprises M&E/Results (IP, USAID, SPICE) Linkages to large-scale enterprises Comparison of Methods (IP, USAID, SPICE) Shelter, WASH, and civil documents Roles (SPICE) Access Impact: org, field (IP, USAID, SPICE) Transitional shelter Satisfaction with services (USAID, SPICE) Program for WASH Coordination: successes, challenges Rainwater harvesting systems LESSONS Civil documents Selection process OCA process Development Grants Program Mid-term Evaluation 53 Action planning process Capacity building delivery Participation SPICE’s services Best practices through providers Institutionalizing cap building services Relevance, effectiveness, efficiency of cap building services Lessons in linking with IPs Services of IP Airing grievances Meaningful participation Understanding budget allocation Development Grants Program Mid-term Evaluation 54 APPRECIATIVE EVALUATION WORKSHOP: DESIGN Three-hour workshop for IPs (field workers and office staff) The evaluation workshop replaces focus group discussions (FGDs) as a more useful and appropriate way to engage IPs around EQs. The workshops generate data paralleling core EQs in a way that is directly useful to the IPs. The participatory approach will resonate with NGOs and CSOs and allow evaluators to go into greater depth during later key informant interviews and discussions with beneficiaries. Welcome, purpose of learning about project, the workshop, introductions, agenda I. Service delivery to beneficiaries (appreciative interviews and force-field analysis)  Trauma and PWDs  Economic recovery and growth  Shelter, WASH, and civil documents What worked Share a specific incident in the past year when you felt most effective working with DGP beneficiaries. Who? What? (interviews in groups of 3):  Debrief, group analysis of stories and factors supporting program effectiveness What challenges or struggles did you face working with your beneficiaries? (table groups w/flash cards 2 colors; orange/yellow for challenges & blue/green for what works)  Reasons for and common factors around struggles?  Actions? Any overcome?  Role of the organization in addressing challenges? II. DGP timeline (Changes in your organization’s capacity) How has your organization changed from the time of the organizational capacity assessment to now? What’s different?  New skills, results, processes, achievements, setbacks, milestones, people, etc.  As a result of CD efforts, what do you do differently? III.Lessons learned and recommendations Lessons for the organization: What have you learned about your organization during this DGP (processes and OD lessons)? (plenary brainstorm)  How was your OCA implemented? What happened? Sustainability scenarios: How would you like to see the organization two years after the DGP? (Table groups 2 – 3 scenarios / table on flipchart pad)  What will it take to get you there? Recommendations: Based on your experience, how could other grant programs be stronger— help LOs more? (Tables 2 – 3 recommendations / table on flipchart pad) (Debrief section-by-section, include: What capacities do you still need and how will you get them?) Development Grants Program Mid-term Evaluation 55 MINI SURVEY PROTOCOL Social Impact is conducting a mini survey of 13 IPs with the DGP funded by USAID (DGPs). Please complete these questions to the best of your ability. If the question is not relevant, please check “NA” for not applicable. If you do not track this information, please check “ND” for no data. We will collect and review your survey personally when we visit your organization during the evaluation. Responses are confidential; your name will not be used in the evaluation report. NAME OF ORGANIZATION: Name of person completing the survey: Contact Number: Date of completion: Estimated total number of direct project beneficiaries: Please disaggregate data by gender. # Question NA ND 1 # of PWDs that accessed prosthetics and orthotics services _____________ 2 # of PWDs that accessed physical and rehabilitation support _____________ 3 # of PWDs using locally manufactured prosthetic and orthotic supports _____________ 4 # of PWDs participating in mainstream society _____________ 5 # of children with disabilities going to school (Please provide enrolment and completion rates) _____________ 6 # of PWDs and victims of trauma have livelihoods (employment or economic opportunities) _____________ 7 # of former victims of trauma that participated in community activities _____________ 8 # of families have less debt _____________ 9 # of economic societies (agrarian/fisherman/other) participated in your capacity building efforts _____________ 10 # of economic societies (agrarian/fisherman/other) have constructive linkages with the private sector _____________ 11 # of microenterprises participating in value chains _____________ 12 # of individuals using credit for EG input _____________ 13 # of microenterprises are accessing new technology (What are these technologies) ________________________ _____________ Development Grants Program Mid-term Evaluation 56 # Question NA ND 14 # of microenterprises now linked to large-scale enterprises _____________ 15 #of farmers in microenterprises linked to large-scale enterprises _____________ 16 # of farmers who accessed agricultural extension services _____________ 17 # of farmers who were able to access markets _____________ 18 # of households and / or farmer/fisher associations that received economic support _____________ 19 # of households and / or farmer/fisher associations that received capacity-building support _____________ 20 # of households and / or farmer/fisher associations now make profit _____________ 21 # of transitional shelters were established for families that will not receive permanent housing 22 # of households now have access to safe and reliable water supply (meeting public health standards) _____________ 23 # of households with access to sanitation facilities _____________ 24 # of citizens returning to conflict-affected areas who received support with civil documentation _____________ 25 Household prevalence of rainwater harvesting systems _____________ 26 # of households using rainwater harvesting systems _____________ Development Grants Program Mid-term Evaluation 57 ANNEX IX: KEY INFORMANT STATISTICS BY DATA COLLECTION METHOD IPs Gender KIIs Small group interviews Gender-wise Sub￾total Total Mini Survey FRC Male 5 3 8 30 1 Female 4 18 22 Navajeevana Male 10 55 65 124 1 Female 10 49 59 Peragamana Guild Male 6 16 22 30 1 Female 4 4 8 Palm Foundation Male 8 10 18 36 1 Female 1 17 18 Mencafep Male 6 2 8 33 1 Female 10 15 25 ICES Male 3 3 6 8 1 Female 2 0 2 GMSL Male 8 8 16 35 1 Female 3 16 19 Nucleus Foundation Male 4 2 6 21 1 Female 1 14 15 SLCDF Male 5 0 5 28 1 Female 4 19 23 Shanthiham Male 8 0 8 20 1 Female 3 9 12 JSAC Male 7 13 20 35 1 Female 5 10 15 Sevalanka Male 10 0 10 17 1 Female 1 6 7 LRWHF, Colombo Male 1 0 1 3 1 Female 2 0 2 SPICE Male 3 0 3 4 Female 1 0 1 Development Grants Program Mid-term Evaluation 58 IPs Gender KIIs Small group interviews Gender-wise Sub￾total Total Mini Survey USAID Mission Male 7 0 7 12 Female 5 0 5 Total Male 90 112 203 436 13 Female 56 176 233 Total 146 288 436 Development Grants Program Mid-term Evaluation 59 ANNEX X: PERSONS INTERVIEWED Family Rehabilitation Centre, Colombo and Trincomalee Number of People Gender Position Affiliation 1 Male Executive Director FRC, Colombo 1 Female Program Coordinator FRC, Colombo 1 Female Program Coordinator FRC, Colombo 1 Male Program Coordinator FRC, Trincomalee 1 Male Physiotherapist FRC, Trincomalee 1 Male Doctor FRC, Trincomalee 1 Female Counsellor FRC, Trincomalee 6 Female Community Mobilizers (field staff workshop) FRC, Trincomalee 1 Male Community Mobilizers FRC, Trincomalee 1 Female Social Service Officer FRC, Trincomalee 8 Female Beneficiaries (group interview) FRC, Kumpurupittiya, Trincomalee 3 Male Beneficiaries FRC, Veruhal, Trincomalee 4 Female Beneficiaries FRC, Podiveddtai, Trincomalee 8 Male 22 Female 30 Total Navajeevana, Tangalle Number of People Gender Position Affiliation 1 Female Chair Person Navajeewana 1 Female Director Navajeewana 1 Female Head of Education Unit Navajeewana 1 Male Head of Finance Unit Navajeewana Development Grants Program Mid-term Evaluation 60 Navajeevana, Tangalle Number of People Gender Position Affiliation 1 Male Head of Therapy and Rehabilitation Unit Navajeewana 1 Male Head of Prostheses Navajeewana 3 Female Early Childhood Care and Development Teachers and Parents Navajeewana 1 Female Beneficiary Navajeewana, Pethegama, Tangalle 2 Male Beneficiary Navajeewana, Pollgahamulla, Tangalle 1 Male Social Service Officer Navajeewana, Dickwella, Tangalle 1 Female District Social Service Officer Navajeewana, Tangalle 12 Male DPO Members, Rural Development Societies, Women Rural Development Societies (group interview) Navajeewana, Dickwella, Tangalle 4 Female DPO Volunteers (group interview) Navajeewana, Dickwella, Tangalle 15 Female Community Mobilizers, Teachers, Head of Units (staff work shop) Navajeewana, Tangalle 6 Male Community Mobilizers, Teachers, Head of Units (staff work shop) Navajeewana, Tangalle 1 Male Director INDICOS (NGO), Matara 1 Male Director Ladder of Hope, Batticaloa 1 Female Community Mobilizer Ladder of Hope, Batticaloa 1 Male Community Mobilizer Ladder of Hope, Batticaloa 1 Female Beneficiary Ladder of Hope, Batticaloa 1 Female Beneficiary Ladder of Hope, Batticaloa 3 Female Beneficiaries/Teachers (group interview) Ladder of Hope, Batticaloa (Saint Mary’s International School) 6 Female Beneficiaries/Teachers (group interview) Ladder of Hope, Batticaloa (Ozanam Children Home) 20 Male DPO Members (group interview) Ladder of Hope, Batticaloa Development Grants Program Mid-term Evaluation 61 Navajeevana, Tangalle Number of People Gender Position Affiliation 7 Female DPO Members (group interview) Ladder of Hope, Batticaloa 1 Female Beneficiary Ladder of Hope, Batticaloa 2 Female Beneficiary Ladder of Hope, Batticaloa 1 Male Beneficiary Ladder of Hope, Batticaloa 1 Male Director Center for Accessibility, Monitoring, and Information on Disability (NGO), Batticaloa 1 Male Social Service Officer Divisional Secretariat Office, Manmunai North, Batticaloa 3 Male Doctor, Head Nurses (group interview) Valaichchenai Government Hospital, Batticaloa 1 Female Physiotherapist (group interview) Valaichchenai Government Hospital, Batticaloa 12 Male DPO Members (group interview) Ladder of Hope, Batticaloa 9 Female DPO Members (group interview) Ladder of Hope, Batticaloa 65 Male 59 Female 124 Total Peragamana Guild, Ampara Number of People Gender Position Affiliation 1 Female Director Peragamana Guild, Ampara 2 Male Finance Officer, Finance Consultant Peragamana Guild, Colombo 1 Female Administrative Officer Peragamana Guild, Colombo 2 Male Career Guidance Officers Peragamana Guild, Ampara 1 Female Beneficiary Peragamana Guild, Navithan Veli, Ampara Development Grants Program Mid-term Evaluation 62 Peragamana Guild, Ampara 1 Male Beneficiary Peragamana Guild, Navithan Veli, Ampara 1 Male Beneficiary Peragamana Guild, Navithan Veli, Ampara 1 Female Beneficiary Peragamana Guild, Navithan Veli, Ampara 3 Male Agricultural Officers Agriculture Department, Navithan Veli, Ampara 1 Male Government Agent Kachcheri, Ampara 1 Male Divisional Engineer Irrigation Department, Ampara 1 Male Additional Divisional Engineer Irrigation Department, Ampara 6 Male Farmers Navithan Veli, Ampara 1 Male President Farmer Organization, Navithan Veli, Ampara 3 Male Community Workers (staff workshop) Peragamana Guild, Navithan Veli, Ampara 4 Female Community Workers (staff workshop) Peragamana Guild, Ampara 22 Male 8 Female 30 Total Palm Foundation, Nuwara Eliya, and Batticaloa Number of People Gender Position Affiliation 1 Male Director Head Office, Palm Foundation, Nuwara Eliya 1 Male Project Coordinator Head Office, Palm Foundation, Nuwara Eliya 1 Male Accountant Head Office, Palm Foundation, Nuwara Eliya 1 Male Project Director Palm Foundation, Batticaloa 1 Male Project Officer Palm Foundation, Batticaloa 4 Female Community Mobilizers (staff Palm Foundation, Batticaloa Development Grants Program Mid-term Evaluation 63 Palm Foundation, Nuwara Eliya, and Batticaloa Number of People Gender Position Affiliation workshop) 1 Male Project Accountant Palm Foundation, Batticaloa 2 Female Community Mobilizers (group interview) Palm Foundation, Kokatticholai, Batticaloa 1 Male Community Mobilizer Palm Foundation, Chengaladdy, Batticaloa 1 Male Divisional Secretary Chengaladdy, Batticaloa 1 Female Divisional Secretary Eravur Pattu, Batticaloa 7 Male Beneficiaries (group interview) Chengaladdy, Batticaloa 5 Female Beneficiaries (group interview) Chengaladdy, Batticaloa 6 Female Beneficiaries (group interview) Kokattichcholai, Batticaloa 3 Male Beneficiaries (group interview) Kokattichcholai, Batticaloa 18 Male 18 Female 36 Total Mancafep Nuwara Eliya Number of People Gender Position Affiliation 1 Male Director Mancafep, Nuwara Eliya 1 Female Directress Mancafep, Nuwara Eliya 1 Female Head Teacher Mancafep, Nuwara Eliya 1 Female Finance Officer Mancafep, Nuwara Eliya 1 Female HR Officer Mancafep, Nuwara Eliya 1 Female Warden Mancafep, Nuwara Eliya 1 Male Teacher Mancafep, Nuwara Eliya Development Grants Program Mid-term Evaluation 64 1 Female Volunteer Nurse Mancafep, Nuwara Eliya 1 Female Pediatrician Mancafep, Nuwara Eliya 1 Male Social Service Officer Divisional Secretariats Office, Nuwara Eliya 1 Male Director SOS Children Home, Nuwara Eliya 2 Male Staff (PWD) Mancafep, Nuwara Eliya 1 Female Staff (PWD) Mancafep, Nuwara Eliya 4 Female Vocational Training Students (group interview) Mancafep, Nuwara Eliya 4 Female Parents of Mencafep School Children Mancafep, Nuwara Eliya 2 Male Parents of Mencafep School Children Mancafep, Nuwara Eliya 1 Female Beneficiary Mancafep, Nuwara Eliya 1 Female Beneficiary Mancafep, Nuwara Eliya 7 Female Teachers, Administrative officer, Volunteers (staff workshop) Mancafep, Nuwara Eliya 8 Male 25 Female 33 Total ICES Colombo Number of People Gender Position Affiliation 1 Male Director ICES, Colombo 2 Male Program Officers ICES, Colombo 2 Female M&E Officer, Program Officer ICES, Colombo 3 Male Field Officers ICES, Colombo 6 Male 2 Female Development Grants Program Mid-term Evaluation 65 ICES Colombo Number of People Gender Position Affiliation 8 Total GMSL, Vavuniya Number of People Gender Position Affiliation 1 Male Chief Executive Officer GMSL, Colombo 3 Male Field Oversight Officers GMSL, Vavuniya 1 Male M&E Officer GMSL, Colombo 1 Male Project Manager GMSL, Colombo 3 Female Volunteer GMSL, Vavuniya 1 Male Former Farm Manager Department of Agriculture, Vavuniya 1 Male Extension Officer Department of Agriculture, Vavuniya 3 Male Beneficiary Tharanikulam, Vavuniya 10 Female Beneficiary Tharanikulam, Vavuniya 1 Male Beneficiary Karipaddamurippu, Mullaitivu 4 Male Beneficiary Puthiya Kulam, Mullaitivu 2 Female Beneficiary Puthiya Kulam, Mullaitivu 4 Female Beneficiary Mankulam, Mullaitivu 16 Male 19 Female 35 Total Nucleus, Mullaitivu Number of People Gender Position Affiliation Development Grants Program Mid-term Evaluation 66 Nucleus, Mullaitivu Number of People Gender Position Affiliation 1 Female Director, M&E Communications Nucleus, Kandy 3 Male Field/ District Coordinators Nucleus, Vavuniya, Mannar, Jaffna 1 Male Senior Coordinator/ Consultant Nucleus, Colombo 1 Female Beneficiary Poonthoddam, Vavuniya 1 Female Beneficiary Thirunavalkulam, Vavuniya 4 Female Beneficiary Udayarkaddu, Mullaitivu 1 Male Beneficiary Kaiveli, Mullaitivu 1 Female Beneficiary Puthukudiyiruppu, Mullaitivu 1 Female Beneficiary Puthukudiyiruppu, Mullaitivu 3 Female Beneficiary Thevipuram, Mullaitivu 3 Female Beneficiary Puthukudiyiruppu, Mullaitivu 1 Male Development Officer Department of Industries, Mullaitivu 6 Male 15 Female 21 Total SLCDF, Kilinochchi Number of People Gender Position Affiliation 1 Male Chairmen SLCDF, Colombo 1 Female Senior Program Officer SLCDF, Colombo 1 Male Consultant SLCDF, Colombo 1 Male Project Manager SLCDF, Kilinochchi 3 Female Field Coordinators SLCDF, Kilinochchi Development Grants Program Mid-term Evaluation 67 SLCDF, Kilinochchi 4 Female Volunteers SLCDF, Kilinochchi 1 Male Coordinator World Vision, Kilinochchi 1 Male District Veterinary Surgeon Department of Animal Production and Health, Kilinochchi 3 Female Beneficiary Mulankavil, Kilinochchi 6 Female Beneficiary Piramanthanaaru, Kilinochchi 4 Female Beneficiary Punnaineeravi, Kilinochchi 1 Female Beneficiary Kilinochchi 1 Female Beneficiary Kilinochchi 5 Male 23 Female 28 Total Shanthiham, Jaffna Number of People Gender Position Affiliation 1 Male Deputy Director Shanthiham, Jaffna 1 Female M&E Officer Shanthiham, Jaffna 3 Male Master Trainer Shanthiham, Jaffna 1 Male Director Finance Shanthiham, Jaffna 1 Female Director Projects Shanthiham, Jaffna 2 Male Project Coordinators Shanthiham, Jaffna 1 Female Volunteer/Field Counselors Shanthiham, Jaffna 1 Male Volunteer/Field Counselors Shanthiham, Jaffna 2 Female Beneficiary Jeyapuram South, Kilinochchi 2 Female Beneficiary Jeyapuram North, Kilinochchi 5 Female Beneficiary Mudkompan, Kilinochchi Development Grants Program Mid-term Evaluation 68 Shanthiham, Jaffna Number of People Gender Position Affiliation 8 Male 12 Female 20 Total JSAC, Jaffna Number of People Gender Position Affiliation 1 Male Coordinator JSAC, Jaffna 1 Female Administrative Manager JSAC, Jaffna 1 Female Program Manager JSAC, Jaffna 3 Female Program Assistant JSAC, Jaffna 1 Male Program Assistant JSAC, Jaffna 1 Male Technical Officer JSAC, Jaffna 1 Male Procurement and Logistic Officer JSAC, Jaffna 1 Male Finance Officer JSAC, Jaffna 1 Male Divisional Secretary Tellipalai, Jaffna 1 Male UNDP, Jaffna 6 Male Beneficiary Mathagal West, Jaffna 2 Male Beneficiary Vithakapuram, Jaffna 1 Female Beneficiary Vithakapuram, Jaffna 2 Female Beneficiary Palai Veemankamam North, Jaffna 1 Male Beneficiary Palai Veemankamam North, Jaffna 4 Male Beneficiary Kakaithivu, Jaffna 4 Female Beneficiary Mathagal East, Jaffna 3 Female Beneficiary Ponnalai, Jaffna Development Grants Program Mid-term Evaluation 69 JSAC, Jaffna Number of People Gender Position Affiliation 20 Male 15 Female 35 Total Sevalanka, Jaffna Number of People Gender Position Affiliation 1 Male Vice Chair Person Sevalanka, Jaffna 1 Female Project Coordinator Sevalanka, Jaffna 8 Male Project Coordinator Sevalanka, Jaffna 1 Male Divisional Secretary Tellipalai, Jaffna 2 Female Beneficiary Thanthai Selvapuram, Jaffna 2 Female Beneficiary Vithakapuram, Jaffna 1 Female Beneficiary Jaffna 1 Female Beneficiary Jaffna 10 Male 7 Female 17 Total Lanka Rainwater Harvesting Forum, Colombo Number of People Gender Position Affiliation 1 Female Director LRWHF, Colombo 1 Male Accountant LRWHF, Colombo 1 Female Communication Officer LRWHF, Colombo 1 Male Development Grants Program Mid-term Evaluation 70 Lanka Rainwater Harvesting Forum, Colombo 2 Female 3 Total SPICE, Colombo Number of People Gender Position Affiliation 1 Female Deputy Chief of Party SPICE, Colombo 1 Male Chief of Party SPICE, Colombo 2 Male Sub-grantees of SPICE PALTRA, Business consultancy Service 3 Male 1 Female 4 Total USAID Number of People Gender Position Affiliation 1 Female Development Budget Specialist USAID, Sri Lanka 3 Female AOR USAID, Sri Lanka 3 Male AOR USAID, Sri Lanka 1 Male Finance Specialist USAID, Sri Lanka 1 Male Chief of Mission USAID, Sri Lanka 3 1 Female 2 Male E3 Team USAID HQ 7 Male 5 Female 12 Total Development Grants Program Mid-term Evaluation 71 Asia Foundation/CPI/MSI Number of People Gender Position Affiliation 2 Male Executive Officers Asia Foundation, Colombo 1 Female Executive Officer Asia Foundation, Colombo 1 Female Technical Director for Gender and Civil Society MSI 3 2 Male 1 Female Senior Program Officer; Program Officer for Asia/Eurasia; Director CPI 4 Male 3 Female 7 Total Development Grants Program Mid-term Evaluation 72 ANNEX XI: DATA COLLECTION SCHEDULE Data collection was carried out from May 14 to June 10 2016. The team worked together in Colombo and then split in two teams of two ET members. Date Activity Location May 9 - 12 Consultations with USAID/E3/DGP Washington, DC Consultations with USAID/E3/DGP Washington, DC Consultations with CPI Washington, DC Consultations with CPI Washington, DC Consultations with MSI Washington, DC Consultations with MSI Washington, DC May 13 Team Leader travels to Sri Lanka May 14 Afternoon team meeting to discuss and finalize in-brief and kickoff May 15 Day off May 16 One-on-one meetings with CORs/AORs Colombo May 17 USAID/Sri Lanka in-brief Kick-off meeting with DGP COR/AOR Colombo May 18 Team Planning Meeting and evaluation work plan finalization with revised instrumentation; meetings with FRC and Peragamana Team 2: late afternoon travel to Tangalle May 19 Team 1: Meetings with USAID officials, OCA team, Ford Foundation Colombo Team 2: Meeting with Navajeewana staff/government stakeholders Tangalle May 20 Meetings with SPICE: MSI program manager and partner Business Consultancy Services Colombo Team 2: FGD with Navajeewana beneficiaries/returns back to Colombo Tangalle May 21 Debrief to assess first IP visit and discuss lessons learned May 22 Meeting with SPICE partner PALTRA; off day May 23 Holiday: Team 1 and Team 2 travel May 24 Team 1: KIIs with GMSL staff/government/other stakeholders Vavuniya Team 2: KIIs with Peragamana staff/government/other stakeholders Ampara May 25 Team 1: Small group interviews with GMSL beneficiaries Vavuniya Team 2: Small group interviews with Peragamana beneficiaries Ampara May 26 Team 1: KIIs with Nucleus staff/government/other stakeholders Mullaitivu Team 2: KIIs with Navajeewana staff in Batticaloa/government/other Batticaloa Development Grants Program Mid-term Evaluation 73 Date Activity Location stakeholders May 27 Mid-point briefing with USAID/Sri Lanka via teleconference call Team 1: Small group interviews with Nucleus beneficiaries Mullaitivu Team 2: Small group interviews with Navajeewana Batticaloa beneficiaries Batticaloa May 28 Team 1: KIIs with SLCDF staff/government/other stakeholders Team 2: FGD with Palm beneficiaries and travels to Trinco Kilinochchi Batticaloa May 29 Off day May 30 Team 1: Small group interviews with SLCDF beneficiaries Kilinochchi Team 2: KIIs with FRC Trincomalee staff Trincomalee May 31 Team 1: KII with Shanthiham staff and FGD with Shanthiham beneficiaries Jaffna Team 2: FGD with FRC Trincomalee beneficiaries Trincomalee June 1 Team 1: KIIs with JSAC staff/government/other stakeholders Jaffna Team 2: Travels to Nuwera Eliya stakeholders Nuwera Eliya June 2 Team 1: Small group interviews with JSAC beneficiaries Jaffna Team 2: KIIs with Mencafep staff/government/other stakeholders Nuwera Eliya June 3 Team 1: KIIs Sevalanka staff/government/other stakeholders Jaffna Team 2: Small group interviews with Mencafep beneficiaries Nuwera Eliya June 4 Team 1: Small group interviews with Sevalanka beneficiaries, returns to Colombo Team 2: KII with Palm staff and stakeholders (morning), returns to Colombo Jaffna Nuwera Eliya June 5 Teams return to Colombo Off day June 6 ET begins data analysis and conducts internal preliminary findings, conclusions, and recommendations workshop Colombo Out-brief outline submitted by ET to Social Impact, Inc. HQ for comment Colombo June 7 Sub-grantees (PALTRA, Management Frontiers, Creators Forum, LRWHF) (1/2 day) Colombo Team 2: Small group interviews and KIIs ICES (full day) Colombo June 8 GMSL, Sevalanka; Peregamana Out-brief presentation submitted by Social Impact, Inc. to USAID for comment Colombo June 9 USAID sends feedback on PowerPoint, team revises and submits by close of Development Grants Program Mid-term Evaluation 74 Date Activity Location business Organize materials for final analysis and draft report preparation June 10 Out-brief with USAID (morning if possible); ET debrief Colombo Travel from Sri Lanka (evening) Development Grants Program Mid-term Evaluation 75 ANNEX XII: MINI SURVEY SUMMARY REPORT Summary Findings of the Mini Survey NUMBER OF IPs 13 Date of completion: June 16 Total number of direct project beneficiaries reported: 44380 Men: 20186 Women: 24192 Survey Question Total Male Female # of PWDs that accessed prosthetics and orthotics services 1170 644 526 # of PWDs that accessed physical and rehabilitation support 12165 6560 5605 # of PWDs using locally manufactured prosthetic and orthotic supports 1328 764 564 # of prosthetic and orthotic devices locally fabricated 1979 # of special shoes locally manufactured 87 # of PWDs participating in mainstream society 2582 1357 1225 # of children with disabilities going to school 967 464 503 Enrolment and completion rates 300% 377% 253% # of PWDs and victims of trauma have livelihoods (employment or economic opportunities) 1074 564 462 # of former victims of trauma that participated in community activities 414 248 154 # of families have less debt 1030 # of economic societies (agrarian/fisherman/other)/Disabled People's Organizations participated in capacity building efforts 1073 # of economic societies (agrarian/fisherman/other)/Disabled People's Organizations have constructive linkages with the private sector 903 # of microenterprises participating in value chains 1146 251 895 # of individuals using credit for EG input 1458 148 1310 # of microenterprises are accessing new technology 1005 What are these technologies? _____________________ Development Grants Program Mid-term Evaluation 76 # of microenterprises now linked to large-scale enterprises 10 #of farmers in microenterprises linked to large-scale enterprises 530 296 204 # of farmers who were able to access markets 1763 296 204 Survey Question Total Male Female # of households and / or farmer/fisher associations that received economic support 6251 # of households and / or farmer/fisher associations/Disabled People's Organizations that received capacity-building support 4816 # of households and / or farmer/fisher associations now make profit 3277 # of transitional shelters were established for families that will not receive permanent housing 405 # of households now have access to safe and reliable water supply (meeting public health standards) 3103 # of households with access to sanitation facilities 974 # of citizens returning to conflict-affected areas who received support with civil documentation 521 236 285 Household prevalence of rainwater harvesting systems 50 # of households using rainwater harvesting systems 50 Development Grants Program Mid-term Evaluation 77 ANNEX XIII: OPEN SYSTEM CHARACTERISTICS BUSINESS AS USUAL Bureaucratic WHAT OUR ORGANIZATION COULD BE Active Adaptive System21 Supervision Responsibility Individuals work under supervisors who are responsible to others, “it’s not my job” Self-managing work teams sharing leadership; people who do the work are responsible for their work Procedures Staff follow procedures mechanically, complying blindly with rules Work processes add value, improve quality and productivity, consistency improves Incentives Staff are rewarded for being on-time and on￾budget with their inputs Quality in outcome is rewarded as much as efficiency System Information Closed system, work environment is largely ignored; gathers knowledge into reports Open (active adaptive) system monitors work environment; learns from experience and innovates based on reflection Timing Feedback received too late to make a difference Timely feedback avoids mistakes Work People outside office design systems The people who do the work customize it to be most effective Staff Morale & Will Low. Cynicism, apathy, short-term changes, low motivation Greater empowerment; energy and creativity for innovation, action and sustaining change Ownership Primary responsibility for control and goals stay with management; position determines “what and how” Staff have primary responsibility for control, coordination, and goals; hierarchy of functions among peers with strategy, resourcing, and operations; supervisors / managers as leaders 21 This Open Systems model is adapted from forty years of effective practice and theory, much of which is built on ideas coming from Emery, F. (1981). Open systems thinking. Volumes I & II. Penguin. Emery, F. & Trist, E. (1965). The causal texture of organizational environments. Human Relations, 18, 21-32. This work is the foundation of the majority of contemporary approaches to change in the workplace. Development Grants Program Mid-term Evaluation 78 ANNEX XIV: SUCCESS FACTORS IN DESIGNING CAPACITY DEVELOPMENT ASSISTANCE A special evaluation study by the Operations Evaluation Department of the ADB found that the following factors were driving the successful design of CD support:  Presence of a clear RF or capability of being evaluated for CD to be measured and monitored;  Strategic direction with realistic CD objectives;  Adequate diagnostic baseline assessments at all CD levels (individual, organizational, network, and contextual);  Long-term continuity to institutionalize CD, careful phasing and/or sequencing, and exit strategy;  Appropriate mix of modalities;  Mainstreaming of project implementation and management units’ activities into target agencies’ normal operations;  Adequate staff time and skills, and financial resources;  Inclusive participatory approach, with strong commitment of and ownership by target agencies; and  Cooperation and harmonization with other development partners. Source: ADB. 2008c. Special Evaluation Study on Effectiveness of ADB’s Capacity Development Assistance. How to Get Institutions Right. Manila. Development Grants Program Mid-term Evaluation 79 ANNEX XV: DISCLOSURE OF ANY CONFLICTS OF INTEREST Development Grants Program Mid-term Evaluation 80 Development Grants Program Mid-term Evaluation 81 Development Grants Program Mid-term Evaluation 82 Development Grants Program Mid-term Evaluation 83 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523