Laurie Zivetz, MPH, PhD June, 2018 ACA No. DFD-A-00-08-00260-00 Special Programs to Address the Needs of Survivors (SPANS), Grant Solicitation and Management (GSM) Project Final Evaluation Thanks to Luisa Angelsmith, Lauren Godfrey and Grace Longwe from World Learning for their help and responsiveness in this evaluation process. A number of staff of current and former grantees and members of USAID, including technical advisors, past and present took time to share perspectives and insights and thanks go to them. While they remain anonymous in this report I hope they see their views coming through. The evaluation of the SPANS/GSM project and this report were made possible by the generous support and contribution of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the author and do not necessarily reflect the views of World Learning, USAID or the United States Government. Table of Contents Executive Summary......................................................................................................................... 1 I. Introduction............................................................................................................................. 5 II. Methodology ........................................................................................................................... 6 III. Findings and Recommendations............................................................................................. 9 1. Overview of achievements ................................................................................................ 10 2. Effectiveness...................................................................................................................... 15 3. Capacity Strengthening...................................................................................................... 30 4. Non Grant Activities........................................................................................................... 34 5. Efficiencies ......................................................................................................................... 36 6. The Mechanism.................................................................................................................. 41 IV. Summary and conclusions..................................................................................................... 45 Annexes A. Evaluation terms of reference B. Inception report, including interview and survey instruments C. List of interview respondents D. List of grants List of Tables Table 1. Participants in the final evaluation ................................................................................... 7 Table 2. Snapshot of SPANS/GSM budget to various congressional Funds (2009-Q2/FY18)....... 11 Table 3. SPANS/GSM solicitations, grants, and expenditures...................................................... 12 Table 4. Number of people served under SPANS/GSM (FY10-FY17, by gender and Fund) ......... 14 Table 5. Technical Assistance on Grants Management and Compliance Issues .......................... 29 Table 6. Non Grant Activities supported under SPANS/GSM (2008-2018).................................. 34 Table 7. Average length of time for preaward steps, SPANS/GSM .............................................. 37 Table 8. Solicitation process breakdown by RFA.......................................................................... 37 Table 9. TEC Selection efficiencies................................................................................................ 39 List of Figures Figure 1. Geographic spread of grantees and numbers interviewed in the evaluation................. 8 Figure 2. Steps in the preaward process under the SPANS/GSM................................................. 15 Figure 3. Responses to on line survey about elements of SPANS/GSM (April, 2018).................. 16 Figure 4. Average time required for preaward processing and USAID approval over the life of SPANS/GSM................................................................................................................................... 38 Figure 5. Awards compared to applications, by source*.............................................................. 43 List of Acronyms AOR Agreement Officer Representative APC Advancing Partners and Communities (JSI) BAFO Best and Final Offer CA Cooperative Agreement CECA Center for Children in Adversity CLASP II Consolidating Logistics for Assistive Technology Supply and Provision COPE Cooperative Orthotic & Prosthetic Enterprise DAC Development Assistance Committee DCHA Bureau of Democracy, Conflict and Humanitarian Assistance (in the Empowerment and Inclusion) DCOF Displaced Children and Orphan’s Fund DF Disability Fund INGO International Non-Government Organization JSI John Snow, Inc LwAA Leader with Associate Award LWVF Leahy War Victims Fund M&E Monitoring and Evaluation MRF Motivation Romania Foundation NGO Non-Government Organization OAA Office of Acquisition and Assistance PIA Participatory Institutional Analysis PVO Private Voluntary Organization Q&A Question and Answer RfA Request for Applications RfP Request for Proposals RUK Roda Untuk Kemanusiaan SPANS/GSM SPANS Special Programs to Address the Needs of Survivors/Grant Solicitation and Management TEC Technical Evaluation Committee TOR Terms of Reference UCP-WfH UCP-Wheels for Humanity USAID United States Agency for International Development USG United States Government VT Victims of Torture WL World Learning WP Wheelchair Program SPANS/GSM Final Evaluation June, 2018 1 Executive Summary Over the course of a decade (2008-2018), World Learning (WL) implemented the Special Programs to Address the Needs of Survivors (SPANS) Grant Solicitation and Management (GSM) project on behalf of the United States Agency for International Development (USAID). SPANS/GSM is a grant making mechanism that supports service delivery, training and related activities aimed at assisting vulnerable populations in the Global South. Resources come through five congressionally mandated funds: Displaced Children and Orphans Fund (DCOF), Leahy War Victims Fund (LWVF), Victims of Torture (VT), the Wheelchair Program (WP), and the Disability Fund (DF). SPANS/GSM was a Cooperative Agreement under Grant Solicitation and Management, Leader with Associate Award, held by World Learning. World Learning executed sub-awards and managed compliance with USAID rules and regulations. It provided formal capacity building support to two local NGOs and a number of contracts. Direction and technical decision making rested entirely with USAID. The final evaluation of the SPANS/GSM sought to assess the extent to which the SPANS/GSM project has met its intended goal to effectively serve as an umbrella grants making mechanism on behalf of USAID (ToR, 2/2018). Evaluation findings are based on a review of documentation and data gathered by World Learning over the life of the program (including an efficiency analysis and a survey administered to all grantees at close out of their project); an on-line survey sent to grantees, USAID and World Learning stakeholders; and in-depth interviews with 34 respondents (24 grantees). Some highlights of the portfolio: • 36 awards were granted to 31 non-governmental organizations (including 3 US universities) as well as a number of small contracts for an overall total portfolio of $53.3 million. • An estimated 197,868 people (81,608 male; 93,194 female) benefited from the decade of grantmaking. . • Seven (7) of the 31 participating grantees (23%) were new to USAID funding. • Nearly half (45%) of all applicants were local NGOs; 19% of awards went to local NGOs. SPANS/GSM Final Evaluation June, 2018 2 Findings Effectiveness and efficiencies. Findings from this evaluation present a very positive report card of World Learning’s performance from both grantees and USAID. One local NGO summed up the sentiments that were widely echoed, especially amongst grantees new to USAID: We feel lucky to have a donor like USAID. We see them as a partner and a collaborator to help us grow. When they give advice, it matters the way the advice is given. This distinguishes USAID and World Learning from the others. Nonetheless, grantees also acknowledged the intensive nature of reporting requirements under SPANS/GSM. While grantees who were new to USAID seemed inclined to credit the process of gearing up to comply, with stronger organizational project management systems, those with more experience with USAID compared compliance requirements unfavorably to other USG agencies. As the efficiency analysis conducted by World Learning shows, the rigor of the review process often took the better part of a year. Average length of time for pre-award steps, SPANS/GSM Grantees complained that they didn’t have enough time to write proposals, and some felt the review process was too lengthy and too prescriptive. Requirements for shortlisted applicants to obtain USAID mission approval letters were problematic for many who had little entre to local missions. Waiver requests, required under a now outdated rule—was common for grantees providing wheelchairs through their programs, and also delayed implementation. Quarterly reporting was similarly demanding. Feedback from World Learning on financial reports was said to be thorough and timely. At the same time, a number of grantees wondered whether anyone was reading their activity reports, as comments would be delayed or very sparse. For much of the decade of grantmaking, World Learning essentially served as a pass through of the activity reports. In reflecting on this, some at USAID felt this could have been more effective had World Learning at least done a cursory review for completeness. Steps in preapproval process Average Time Solicitation Release with Q&A 35 days TEC Selection 68 days Pre-Award 98 days Approval Requested 37 days Overall Process, on average 237 days (8 months) SPANS/GSM Final Evaluation June, 2018 3 Efficiencies in terms of the days consumed for the two longest activities (TEC Selection and preaward systems screening which included responding to TEC questions, and proposal revisions) did not improve over the life of SPANS/GSM. M&E was a weakness throughout SPANS/GSM with USAID acknowledging this capacity as an unaddressed aspiration on the technical or management team. Financial reporting, governance and compliance received significantly more attention than understanding project impact or sustainability, leaving an impression of an outsize emphasis on compliance over demonstrable programmatic results. The absence of a coherent strategy in the design of SPANS/GSM for each Fund (and the suite of Funds overall) reflected a missed opportunity to foster communities of practice and contribute to learning more broadly. Over the life of the program, the SPANS/GSM appears to have been a high functioning, team effort. World Learning played an effective go-between role, respecting the requirements and decisions of its donor client while also responding to the varied needs of its sub-awardee partners with tools and individualized support to optimize compliance. Good working relationships and continuity in some key positions characterized the management team. Capacity strengthening. Many grantees reflected that the partnership with World Learning contributed to organizational capacity building and systems strengthening as noted above. Towards the end of the program, World Learning provided more formalized capacity building support to two local grantees. The mechanism. Although World Learning’s role in SPANS/GSM did not fully align with the usual application of a Cooperative Agreement (in terms of collaborative decision making and cost share), it nonetheless offered a flexible mechanism for USAID to engage a range of non￾government organizations to deliver programs under the five Congressional Funds. Recommendations: 1. A two stage approach starting with concept notes should be considered to lighten the load for applicants and reduce the ratio of solicitations reviewed in the more rigorous TEC and GSM screening review. 2. Forecasting RFA release would extend planning time if the formal solicitation period cannot be extended. 3. Bundle all components requiring USAID sign off if possible to preclude delays. 4. Take the onus of requesting mission approval off of the (shortlisted) applicants’ shoulders and assign it to USAID or the managing agency during the preaward stage. 5. Conduct a rapid review of applicant compliance systems before the Technical Evaluation Committee convenes, to assess whether capacity might be a barrier to implementation. 6. Provide shortlisted agencies the option of a conversation with the USAID technical advisor to minimize the number of BAFO iterations, boost collaboration. SPANS/GSM Final Evaluation June, 2018 4 7. Hold preaward processes for weaker agencies in person and include the development of light touch capacity building plan consistent with project cycle requirements. 8. In a future program, if USAID wishes to broaden the net of participating organizations: • Establish a pool of funding for small, pilot initiatives to encourage grantees to trial new approaches and encourage smaller and more local organizations to participate. Smaller grant size would lower intensity of reporting requirements. Provide appropriate technical and M&E support. • In high priority countries, award a country-based grant making mechanism for local organizations. Participating, high performing organizations could then transition to larger grants under the broader program. 9. Consider longer awards for larger grants so that sustainability gains are more likely, and extensions become less necessary. 10. Develop a Strategic Plan, including a Theory of Change, that lays out a broad conceptual framework for each of the Funds (and highlights synergic opportunities amongst them). 11. Include M&E expertise on the technical team. 12. Promote a learning network amongst grantees to empower cross fertilization of ideas and innovation. SPANS/GSM Final Evaluation June, 2018 5 I. Introduction Over the course of a decade (2008-2018), World Learning implemented the Special Programs to Address the Needs of Survivors (SPANS) Grant Solicitation and Management (GSM) project on behalf of the United States Agency for International Development (USAID)1 . SPANS/GSM is a grant making mechanism that supports service delivery, training and related activities aimed at assisting vulnerable populations in the Global South. Resources come through five congressionally mandated funds: Displaced Children and Orphan’s Fund (DCOF), Leahy War Victims Fund (LWVF), Victims of Torture (VT), the Wheelchair Program (WP), and the Disability Fund (DF). To date 36 awards have been granted to 31 non-governmental organizations totaling $47 million. In addition to grant making, World Learning also provided conference and training logistics and issued RFPs for a number of activities related to SPANS, for an overall total portfolio of $53.3 million. In more recent years, World Learning also provided formal capacity building support to two local grantees. This is the report of the final evaluation of the SPANS/GSM. The evaluation sought to assess the extent to which the SPANS/GSM project has met its intended goal to effectively serve as an umbrella grants making mechanism on behalf of USAID. It posed the following questions: 1. To what extent did SPANS/GSM meet its intended goal to effectively manage grants in support of USAID’s efforts to fulfill the five congressional mandated funds? 2. How did SPANS/GSM perform in implementing non-grant activities in support of USAID’s efforts to fulfill the five congressional mandated funds? 3. Has the timelines and costs of managing the SPANS/GSM mechanism changed throughout the life of the project and if so, how? 4. What lessons and recommendations can be drawn from the SPANS/GSM project that can contribute to the improvement of similar future programs? (SPANS/GSM End of Program Evaluation Terms of Reference, 2/2018) The evaluation was conducted in the April-June, 2018 period by an independent evaluator as part of the project’s close out. It included document and data review and interviews with informants from USAID, World Learning and grantee organizations. Since this is an end of project evaluation, the main audience for this report is USAID. It is hoped that findings and recommendations will help inform strategies that enable continued inclusion of NGOs and universities in delivering programs under the five Congressional Funds. World Learning was the key subject of the report and thus is also an 1 When the SPANS/GSM award started, SPANS stood for “Special Programs to Aid the Needs of Survivors”. At some point in the process, Aid was changed to Address, and Survivors are now referred to as Vulnerable Populations. SPANS/GSM Final Evaluation June, 2018 6 important audience. Organizations that participated in SPANS/GSM as grantees or contractors may also find this evaluation of interest. A note on references in this report. The priorities of SPANS/GSM are established by a team of technical experts including five technical staff seconded to USAID, and an Agreement Officer’s Representative (AOR) from USAID based in the Empowerment and Inclusion Division of the Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA). While the AOR had ultimate sign off, the technical team and AOR functioned as a unit when it came to providing technical guidance to World Learning’s work. In this report, USAID refers generically to this unit as USAID, unless otherwise noted. Grantees are referred to as a cohort of organizations unless it is appropriate to distinguish, for instance between international and local organizations. International NGOs include US as well as a number of other international organizations. “Local” references organizations based in countries that are recipients of USAID support. In some cases, grantees implemented their program through sub-awards to other local partners. This evaluation does not address these arrangements but confines itself to the relationship between World Learning and the SPANS/GSM grantees, though in at least one case the sub￾awardee in the country of implementation was interviewed. Contractors include individuals or organizations engaged to deliver specific trainings, manuals, or evaluations. Individuals who agreed to be interviewed for this evaluation did so with an understanding of confidentiality. References to viewpoints have insofar as possible be attributed generically—to USAID, World Learning, INGOs or local NGOs. II. Methodology The methodology for this end of project evaluation was largely qualitative. Findings reflect: 1. A review of documentation, including World Learning’s quarterly reports, the project agreement and modifications, RFAs and RFPs released under this project, as well as close out reports for each of the grantees that were interviewed. 2. Data gathered over the life of the project. Of particular interest was an efficiency report compiled by the World Learning team which chronicled the turnaround time for each stage of the solicitation and award process for the 10 RFAs. World Learning also shared detailed financial information, including the aggregated financial information presented in this report. Data from comprehensive surveys completed by each grantee during the close out process was also used and is referred to as the “close out survey” in this report. 3. Interviews with a cross section of 34 purposively sampled stakeholders from World Learning, USAID and grantees. Most of the interviews with World Learning and USAID were conducted in person, whereas grantee interviews were conducted by Skype. The interview guidelines are included in the Inception Report in Annex B. SPANS/GSM Final Evaluation June, 2018 7 4. A short on-line survey was created to gather perspectives from all stakeholders about SPANS/GSM performance on a number of dimensions. It included a satisfaction scale and a list of 10 key solicitation and reporting processes, as well as a number of open ended questions. The survey instrument is included in the Inception Report. The interviews and on-line survey were designed to capture candid reflections on the performance and support provided by World Learning over the course of the project, as well as insights into the broader structure of the grant making mechanism. The breakdown of respondents is presented in Table 1. Table 1. Participants in the final evaluation Respondent group Interviewed First award from USAID? Responded to on-line survey Org’s P eople Grantees 17 24 4 17 INGOs 1 14 US Universities 1 1 Local NGOs 5 9 World Learning 5 4 USAID 5 3 Total 34 24 *More than one person participated in a number of calls with local NGOs and two separate interviews were conducted with one organization that had two different awards. SPANS/GSM Final Evaluation June, 2018 8 Figure 1 presents the broad geographic distribution of grants. Numbers appearing alongside the pie pieces are the number interviewed from each geographic area (one grantee was global). Figure 1. Geographic spread of grantees and numbers interviewed in the evaluation The distribution was more or less representative geographically, and also in terms of the type of grantee (NGO, INGO, University) and agencies that were new to USAID. Data from the close out survey conducted by World Learning was quite granular in scope, asking participants to provide ratings and comments for many of the forms and every step in the grant cycle.2 While picking up on some of the same items, the on-line survey carried out in April 2018 for this evaluation also offered a level of anonymity which was not possible in the World Learning survey. Neither survey offers a full picture from all grantees. Sixteen (16) respondents filled in the on-line survey; all 31 grantees are represented in the close out survey. Nonetheless response rates in the close out survey is uneven; grantees rarely filled in every question, and there are many questions that are left unanswered, with as low as 7 responses to some questions and an average of 23 grantees responding per question overall. Confidentiality. The on-line survey was anonymous and confidentiality was assured for interviewees. Because of the very small number of people interviewed and the easily recognizable detail in their reflections on the SPANS/GSM process, interview notes have not been shared with World Learning (by agreement). Limitations. The Terms of Reference anticipated interviews with 60% of grantees (23), with representation of international and local organizations, and organizations familiar with and new to USAID respectively. At the outset, a random sample of grantees in each of the categories of interest was drawn by the evaluator. However, because of the 2 35 items Asia MENA SSA Eur/C.Asia LAC Chart Title 6 1 2 3 4 SPANS/GSM Final Evaluation June, 2018 9 duration of SPANS/GSM, quite a number of people involved in the early grants were no longer traceable. The invitation to participate in the on line survey went only to those on the existing list (45 people) and only 38% responded, for instance. A few of those who were reached for interviews were not appropriate because their role was entirely technical and they had had little contact with World Learning. This low response rate was disappointing but not surprising. While a few replacements to individuals who could not be traced from the original sampled shortlist were made, a second round of outreach to the grantee agencies to try to track individuals involved in SPANS/GSM was not undertaken. Sampling was thus largely purposive and opportunistic. Nonetheless, as noted above, the range of respondents ended up being representative, at least in terms of the criteria from the ToR. For some of those who were interviewed, detailed recall was a challenge because the project of interest had closed years ago. Nonetheless, the 24 grantee representatives who were interviewed do represent a cross section of grantees. USAID-based Technical Leads (past and present) for each of the Funds ended up being particularly hard to contact, and only three were available for an interview (two current; one past). Only three of the 10 USAID representatives who were sent the on-line survey responded (30%). III. Findings and Recommendations The Leahy War Victims Fund (LWVF) was launched in 1989 by Vermont Senator Patrick Leahy through a Congressional earmark. Initially established to help victims of landmines, over time the Fund embraced other conditions that affect mobility or physical function. Similar Congressional Funds targeting other marginalized populations because of circumstance or physical disability followed over the years. Today, the combined Funds total $53 million/year, which is disbursed by USAID through a number of mechanisms. 3 This final evaluation looked closely at SPANS/GSM, a mechanism set up to deliver grants and related support from these Funds to international and local NGOs and Universities. This chapter is organized broadly around the Evaluation Questions put forward by World Learning. It considers: 1. Overview of achievements 2. Effectiveness 3. Capacity strengthening 4. Non grant activities 5. Efficiencies 6. The mechanism Each of these sections is retrospective—drawing on the wealth of existing data within World Learning, as well as data analysis and in-depth interviews carried out over the 3 In FY2017 there were 112 awards supported through these Funds. SPANS/GSM Final Evaluation June, 2018 10 course of this evaluation. It is also forward looking, considering how a next generation program might benefit from lessons from the last decade. Recommendations are presented at the end of each section. 1. Overview of achievements In a clear description of World Learning’s role in the SPANS/GSM, the initial Agreement notes: The purpose of the GSM Cooperative Agreement-SPANS Associate Award, managed by World Learning, is to establish a mechanism whereby USAID can support the programs of selected PO/NGOs and partners around the world. In collaboration with USAID, World Learning conducts solicitations, participates in application reviews, executes sub-awards. In addition, the GSM may be asked to review and execute sub￾awards for unsolicited proposals. Sub-awards will support service delivery, training and treated actives for war victims, displaced children and orphans, victims of torture, persons with disabilities. Associate Award No, DFD-A-00-08-00260-00 under Leader Award No. GPO-A-00-04- 00021-00, Attachment C The original Agreement (signed August 2008) covered the period August 2008-August 2013 with a total of $43.9 million from four Funds (WC, DCOF, WVF, VoT). It anticipated awarding 12 solicitations and making 50 awards during the initial four year period. Modification to the Agreement brought the dollar amount up to $47.8 million and extended the program through 2015. This enabled 3-year projects awarded in 2012 to complete. From 2008 to 2015, nine solicitations were released and 31 grants funded. A series of modifications in a second phase (2015-2018) brought the total amount to $53.3 million. This allowed World Learning to release an additional RFA (#10) and issue two additional awards. During the second phase, three grants were issued under the Wheelchair Program. Two were follow on awards to Motivation Romania Foundation (MRF) and UCP-Wheels for Humanity (UCP-WFH) and one was a sole sourced award to the University of Pittsburgh received bridging support to enable it to continue the foundational activities to establish the International Society of Wheelchair Professionals. SPANS/GSM Final Evaluation June, 2018 11 Table 2. provides a snapshot of the level of funding disbursed through SPANS/GSM over the course of the mechanism, by Fund. Details about each of the grants is provided in Annex D. Table 2. Snapshot of SPANS/GSM budget to various congressional Funds (2009-Q2/FY18) Fund Grants issued Obligated by USAID to World Learning Projected Grants & non-grants through FY18 Percentage by Fund Wheelchair Program (WP) 13 $17,703,980 $16,890,448 32% Leahy War Victims Fund (LWVF) 13 16,329,631 14,878,250 28% Displaced Children and Orphans Fund (DCOF) 8 15,222,133 14,358,200 27% Disability Fund (DF) 1 1,791,147 1,783,093 3% Victims of Torture (VoT) 1 1,292,183 1,027,242 2% Center for Children in Adversity (CECA)* 455,121 291,901 1% World Learning Administration (projected thru FY18) 3,066,780 6% TOTALS 36 $52,794,195 $52,295,915 100% *This is essentially part of DCOF; USAID requested that it be tracked separately. It was supported with two contracts, no grants. The remaining obligated amount at the end of the program will be used mostly to procure wheelchairs. In addition to the grants to non-government organizations and universities for their projects, more than two dozen contracts were issued at USAID’s request (See Section 4. Below). The CECA Fund for Children in Adversity: a workshop and an evaluation—both non grant activities. Ninety percent (90%) of the overall SPANS/GSM budget was devoted to Grants, 4% to contracts (non grants), and 6% for management. SPANS/GSM Final Evaluation June, 2018 12 The program funded projects across 40 countries, though for a number of multi-country initiatives the contact was entirely remote (and those individuals would not have been counted). (see Figure 1 for a broad geographic breakdown). Table 3 breaks down the solicitations, applications received, numbers of awards issued and overall project expended amounts by RFA over the life of the program. Table 3. SPANS/GSM solicitations, grants, and expenditures Year RFA Number Fund No. of Applications No. of Awards Issued Expended Amount ($) Phase I 2008 1 WP 34 8 $ 13,150,752 2009 2 DCOF 96 1 3,130,780 3 LWVF 55 4 4,211,741 4 DF 3 1 1,599.287 2010 5 DCOF 40 4 8,077,618 6 LWVF 29 4 5,827,870 7 WP 50 2 2,978,213 2012 8 LWVF 19 4 3,089,293 9 DCOF 17 3 2,991,042 Phase II 2015 10 LWVF 4 2 2,136,191 2015- 2017 Follow on & sole source WP 3 (included above) Total 10 347 36 $ 47,192,787 Other notable facts about the grants. • Four grantee organizations received multiple awards. • Over the course of SPANS/GSM, seven grantees were provided with cost extensions, which allowed additional time and resources to expand their program. 4 Some of these extensions were competed under RFA #7; others were awarded on a sole source basis at the discretion of USAID. 4 These include: 001/034 UCP, 004 WID, 006 ADRF, 007 MRF, 009 IRC, 0016 Every Child SPANS/GSM Final Evaluation June, 2018 13 • Two thirds of grantees (23) received no cost extensions (including about half of those who received cost extensions). • Seven (7) of the 31 participating grantees (23%) were new to USAID funding (3 of them local NGOs). • The ratio of submissions to awards averaged about 10:1, though there was significant variation from one RFA to the other. Observers familiar with other grant making programs consider this overall ratio reasonable. Nonetheless, the cost and opportunity cost of compiling a submission is not negligible (noted especially by a number of respondents from smaller organizations). 5 There is also a cost associated with the review process. One hundred and forty five (145) local NGOs submitted proposals but were not successful (see Figure 5.). • Of particular note is the large number of applications received in response to RFA #2 in 2009 (see Figure 5.). Seventeen (17) of the applications for RFA#2 were deemed ineligible for review. Even after removing these, the number of applications (79) was high and in this instance the TEC adopted a two phase process, winnowing the 79 to 11, shortlisting four and ultimately awarding only one. (The RFA anticipates awarding 3-5 grants of up to $2.5million each). • RFA #2 is also responsible for the 10:1 ratio. A closer look at the ratio of applications to awards under the LWVF, the largest of the Funds, dropped significantly over time. DCOF demonstrates the same trend. Language in the RFAs is characterized by increasing specificity about the types of programs that were eligible, which may explain this trend. • One of the grants issued under RFA #8 was funded through the Victims of Torture Fund (SPANS 027) at USAID’s request. • In at least one case, a proposal submitted to SPANS/GSM was referred to the relevant USAID mission and ended up being managed directly by the mission using a different mechanism. Programmatic reach. • grantee monitoring data estimates that 197,868 people (81,608 male; 93,194 female) benefited from the decade of grantmaking. 6 Table 4 breaks this down by Fund. The proportion of beneficiaries is broadly consistent with the relative expenditure by Fund. 5 . Unsuccessful organizations were offered feedback on their applications upon request and within a 30-day window (this as anticipated in the RFAs). In total 103 applicants requested and received such feedback. No unsuccessful applicants were interviewed for this evaluation 6 This might be an underestimate as early data in particular was quite sparse. World Learning has concerns about data quality overall, as discussed later in the report. SPANS/GSM Final Evaluation June, 2018 14 Table 4. Number of people served under SPANS/GSM (FY10-FY17, by gender and Fund) 7 People Served (FY10-FY17) Fund Male Female Total LVWF 25,302 30,213 68,049 DCOF 22,456 36,242 63,116 WC 33,598 26,258 65,970 VoT 222 452 674 DF 30 29 59 TOTAL 81,608 93,194 197,868 • Most of the RFA’s were open ended in this regard though two prescribed a geographic focus (RFAs #5 and #9) and the final RFA encouraged multi-country submissions. • Overall 40 countries were reached in some way through the program. Though the majority of grants provided direct implementation to specific populations, some also worked remotely, offering generic services, and in at least one case the grant supported a global network. As noted above, USAID manages approximately $53 million/year from the five Congressionally mandated Funds. In allocating the Funds, USAID prioritizes direct support to missions that express an interest, but there were reportedly few such instances. Additional resources are allocated to multilateral organizations like UNICEF and WHO, as well as other to US Government agencies like the CDC that have broad reach and absorptive capacity. SPANS/GSM is one of a number of mechanisms available to disburse 7 Number of people served was a core indicator that grantees were required to report on. These figures are drawn from annual consolidation reports shared by SPANS/GSM. It is very likely that there is double counting as the same people may have been served in multiple or recurrent ways over the span of a multiyear program (for instance receiving a wheelchair in the first instance, with follow up training on how to use it). There is no way to estimate the extent of the double counting or whether it was the same across grantees. SPANS/GSM Final Evaluation June, 2018 15 these Funds to non-government organizations. 8 Over the course of the program, SPANS/GSM managed about 10% of the resources from the five Funds. 2. Effectiveness Evaluation Question: 1) To what extent did SPANS/GSM meet its intended goal to effectively manage grants in support of USAID’s efforts to fulfill the five congressional mandated funds? a) To what extent did SPANS/GSM achieve its intended results per the agreement? The Development Assistance Committee (DAC) of the Organization for Economic Co— operation and Development (OECD) defines effectiveness broadly as A measure of the extent to which an aid activity attains its objectives. In the case of SPANS/GSM, support provided to USAID and grantees to ensure compliance for grantmaking for the five Funds, was the broad objectives for World Learning. This section considers the effectiveness of World Learning in managing (and managing compliance) at each step in the program cycle for SPANS/GSM. A number of issues associated with effectiveness, including team dynamics and portfolio effectiveness are also covered at the end of this section. Figure 2. shows the steps in the preaward process under SPANS/GSM. World Learning drafted the compliance elements of RFAs and RFPs, and did initial screening of grant application for completeness. World Learning convened and facilitated the Technical Evaluation Committee (TEC). Once projects were shortlisted by the TEC, a preaward process during which time TEC questions were being addressed by shortlisted organizations, World Learning also undertook an intensive screening process to ensure grantees had systems and staff capable of maintaining compliance standards. Figure 2. Steps in the preaward process under the SPANS/GSM Figure Courtesy World Learning 8 An analogous mechanism, Advancing Partners and Communities (APC,), sits under the Global Health Bureau, is held by John Snow, Inc, has also been used to disburse these five funds. While not used exclusively for the five Funds (as SPANS/GSM is), it is broader, larger, and has more flexibility in terms of AOR sign off. • RFA is released and applications are due within 30 days • Includes Q&A period Solicitation Release • TEC evaluates applications • Applicants are short￾listed • Short-listed applicants respond to technical questions and feedback from the TEC TEC Selection • Final TEC negotiation • Applicant completes pre-award Survey • World Learning reviews budget and prepares award package Pre-award • Award package is sent to USAID for approval • USAID approves the issuance of the award Request for Approval SPANS/GSM Final Evaluation June, 2018 16 SPANS/GSM required World Learning to manage compliance for grants from solicitation to close out. World Learning brought considerable experience from other grantmaking programs to the job, including under the LwAA umbrella. Over the course of the grant, World Learning received quarterly reports from grantees, passed narrative components to USAID and reviewed quarterly financial reports. World Learning managed waiver, travel pre-approval and other compliance related issues, which required USAID approval. World Learning also updated USAID through quarterly reports and convened quarterly meetings with technical advisors to review grantee progress and ensure administrative and technical advice was aligned. During the grant period, World Learning provided reporting tools and templates as well as on-going feedback and what became a kind of hands on mentoring to grantees as needed (mostly done remotely). In more recent years, World Learning initiated two formal organizational capacity development assessments— for COPE in Laos and UCP in Indonesia. Finally, World Learning undertook a thorough close out at the end of each grant to ensure disposal of assets and other requirements consistent with USAID policy. The effectiveness of each of these steps is considered in the following section. World Learning also managed competitive bidding and contract execution and management for a number of related contracts. (Details are included in Section 3 below). Results of the on-line survey carried out under this end of project evaluation presents a very positive picture of World Learning’s performance from both grantees and USAID, as shown in Figure 3. Figure 3. Responses to on line survey about elements of SPANS/GSM (April, 2018) SPANS/GSM Final Evaluation June, 2018 17 The data suggests high levels of satisfaction from USAID overall (bars in dark blue)— though the number of respondents was quite small. Grantees expressed particularly high levels of satisfaction about responsiveness from and communications with World Learning (bars in light blue). This resonates with the adjectives heard from grantees and USAID alike in the course of in-depth interviews to describe World Learning’s delivery of the SPANS/GSM: effective, responsive, knowledgeable, patient, committed, gracious, transparent. World Learning took their compliance mandate seriously and the rigor was universally noted. Many grantees appreciated the attention to their individual needs. Noted one local NGO: We feel lucky to have a donor like USAID. We see them as a partner and a collaborator to help us grow. When they give advice, it matters the way the advice is given. This distinguishes USAID and World Learning from the others. Communications and responsiveness from World Learning were also scored highest in the close out survey (4.7 out of 5.0). Nonetheless, as shown in Figure 3., there was some variation in satisfaction with specific steps of the project cycle. These scores are also largely consistent with the interviews conducted in the course of the evaluation. A notable exception is the overall scores in the close out survey related to Technical assistance from World Learning on grants management and compliance, which was notably ranked the lowest amongst all the domains (3.2 out of 5.0). This and other results from the close out survey are discussed in the specific aspects of the project cycle that follows. Consideration of these issues as they relate to efficiencies are included in Section 5. Solicitation RFAs were posted on World Learning’s website, broadcast through its Twitter account, posted on the SPANS/GSM Facebook page, and on some of the development sites like Devex. A number of PVOs said they monitored the grants.gov site. Some heard about the opportunity through their networks. In the early days of SPANS/GSM, the resources available for programming in the disability and child protection space were quite scarce, and grantees remember being motivated to bid. RFA requirements were streamlined somewhat over time. The required level of documentation and required length of submissions were reduced. This is one of the clearest examples of adaptation based on learning. USAID and World Learning recognized that the added value of excessive detail in the documentation was low and could be deferred to the BAFO stage. RFA guidelines were scored higher (4.0 out of 5.0) than issues related to timing (3.5 out of 5.0) or solicitation feedback (3.7 out of 5.0) by grantees at close out. SPANS/GSM Final Evaluation June, 2018 18 Time. Most RFAs allowed a month to respond to RFAs9 , though this was extended in a number of cases with an average time of 35 days (see Table 7.). Grantees were united in their sense that the time allocated was too limited, in particular when consultation and co-creation with local partners was required. A number suggested that the timeframe may have cut short a process leading to a better design. USAID considered the timescale appropriate (see Figure 3. above), and a number of grantees who were veterans of USAID funding acknowledged this to be the norm. This was also an area where a number of PVOs and US University informants compared SPANS/GSM unfavorably to other US Government agencies, noting that other USG donors routinely provided more time for proposal development. Recommendation: Forecasting RFA release insofar as possible, would allow interested international organizations more time to talk through design, staffing and budget issues with local partners and for organizations with less capacity to manage the deployment of their human resource to prepare the bid. In order to level the playing field and preclude wasted time from organizations that are unlikely to qualify, forecasting information should include the potential scope and reporting requirements anticipated for the solicitation. Solicitation review World Learning reviewed submissions for completeness and timeliness before sharing with the Technical Evaluation Committee (TEC). World Learning prepared materials and scoring sheets for individual TEC reviewers, and convened the TEC. World Learning facilitated the TEC deliberations, but did not vote. The TEC focused exclusively on design issues. The question of capacity to implement was undertaken subsequently, at the preaward stage. World Learning managed communications with the grantees on behalf of the TEC, playing a largely pass through role. The intensity of this process varied considerably with some grantees responding to multiple rounds of questions from the TEC before their grant was approved. Recommended one grantee in the on-line survey: Consolidate all of the reviews in one set of comments, if at all possible. Multiple reviews with multiple sets of comments (some of which are redundant) is a lot of workload for a relatively small team on a small award. In addition to the number of iterations and the time consumed, some grantees were frustrated by the prescriptive nature of some of the requests. In a number of cases these 9 The ADS (Automated Directive System) of that guides USAID stipulates that An AO should not make an RFA available for less than 30 days unless there are exigent circumstances supported by a written determination by the AO (see 2 CFR 200.203(b)). In setting the timeline for proposal submission, World Learning defaulted to the minimum length of time. SPANS/GSM Final Evaluation June, 2018 19 memories pivot around the removal of a component of the program that the applicant considered essential (e.g. travel for supervision; a baseline survey; additional countries in a regional bid), or a significant reduction in the budget. One grantee recalled 7-8 rounds of comments (which may have also included comments from the TEC as well as the preaward screening), noting that they felt a little “uncoordinated” and stood out as more work than any other solicitation that grantee had ever been involved with.10 On the other hand, one of the grantees who was new to USG funding remembered that the feedback served to significantly strengthen their design (this organization ended up hiring a grant writer to redraft their submission). The open ended nature of the RFAs offers opportunities for innovative ideas to surface. Nonetheless, the directive and time consuming nature of feedback during the TEC review—reported by multiple grantees—seemed to mitigate this invitation somewhat. Recommendations: 1. When the TEC is recommending significant changes to a shortlisted application, offer the applicant the given the option of a conversation with the USAID technical advisor to promote clarity on both sides. This would minimize the number of BAFO iterations (and is more consistent with the collaborative spirit of the Cooperative Agreement). 2. An explicit strategy for each Fund would provide a shared point of reference for applicants and contribute to coherence in programming under each Fund, as well as efficiencies in the technical review process. This issue is returned to in more detail below. Preaward screening At the same time as TEC clarifications and proposal revisions were being undertaken, World Learning proceeded with a preaward screening process. A detailed preaward survey was administered to each shortlisted organization, and World Learning held follow up calls with each organization. Surveys covered financial management, human resources, reporting and governance systems. This process sometimes resulted in recommendations for various kinds of management or support adjustments in the staffing structure of a project (in particular when financial management was under￾budgeted or understaffed in the proposal). In some senses, this process put in place an informal capacity building plan, although it may not have been recognized as such by either World Learning or the grantee. World Learning was scored high in the close out survey for responding in a timely manner to issues arising in the preaward process (4.5 out of 5.0), and the majority of grantees seemed fine with the grant negotiation process in the close out survey, which received a score of 4.4 out of 5.0. 10 This respondent also noted that the multiple layers of comments may have been due to the fact that some of the countries in the proposal were conflict countries and input from Office of Foreign Disaster Assistance (OFDA) may have been sought—this was a conjecture as the source of comments were not provided. SPANS/GSM Final Evaluation June, 2018 20 Newer grantees were typically required to provide more financial detail in their reports than organizations with a track record with USAID. Organizations that had never received a grant from USAID before were provided support to attend a USAID rules and regulations workshop. Several remarked that the workshop significantly eased the reporting process, while one complained that a delay of 10 months hindered their ability to understand the requirements and made reporting a frustrating experience in the interim. Grantees who received cost extensions had to resubmit the paperwork for the preaward screening, though one organization said they just updated the documentation, saving themselves time. Recommendations: 1. Some of the preaward screening could take place ahead of the technical review of the proposal to reassure USAID that proposals they are reviewing are from applicants capable (or potentially capable) of responding to the required reporting requirements. This screening should be based on investigating information provided in the proposal. 2. Flagging weaker organizations ahead of the TEC would also allow good ideas to be retained, particularly if there were alternative, less demanding funding options available, or opportunities for capacity building support. USG approval Thirty-four (35) of the 36 grants that were submitted for award required OAA approval since they were over $350,000.11 USAID approval turnaround varied considerably from an average of 4.5 days (RFA#10) to 77 days (RFA#7). A number of grantees noted that significant lag time in USG approval affected their ability to launch their programs on schedule and deliver on objectives.12 This will be explored in greater detail in Section 5. (Efficiencies) below. Grantees raised a number of other approval-related issues that are worth considering: • Approvals and waivers. Overall OAA approval typically preceded initiation of other types of government sign off’s which were particularly important for many of the SPANS/GSM grantees. Waivers on procurement and tax issues contributed to delays in project implementation in a sizable number of cases and received notably lower 11 SPANS 013 HI was cancelled before the grant was issued in 2009, leaving 36 grants, only one of which was under $350,000 (for Bethany Christian Services International in China—which was also cancelled before completion). 12 Scores from the close out survey suggest a substantial number of grantees experienced implementation delays for this reason. Ten (10) of the 28 respondents who responded to the statement about delays based on waivers from USAID marked this 1-3 on the 5 point scale in the close out survey (average 4.0 overall). SPANS/GSM Final Evaluation June, 2018 21 marks from grantees in the close out survey (3.4 points out of a possible 5.0; n = 25)13 . Waivers to allow wheelchair purchase from third countries was particularly common and held up on time implementation for a number of organizations. This was particularly critical in the first half of SPANS/GSM, when the program was restricted under a now abolished requirement (Geo Code 000, then 937). USAID approvals were required for all international travel under SPANS/GSM. While this regulation changed in 201414, it was grandfathered in to SPANS/GSM, requiring pre-approval for all international travel for all grantees up to the very end of SPANS/GSM in 2018. This created headaches for grantees that for some reason could not plan their travel well in advance—for instance when there was an emergency in the field, or partner decisions did not allow enough lead time to get travel approval. It also created more work when travel reimbursement for unauthorized trips needed retroactive approval (though this was less common and mostly an issue with a couple of grantees). Recommendation: Insofar as possible, bundle all components requiring USAID sign off at the initial submission stage so that approvals for the grant and associated waivers, travel requests and tax issues can be resolved simultaneously and at the outset. 15 • Mission approval. Shortlisted applicants were required to obtain a letter from the local USAID mission approving their proposed project. 16 This presented a problem for a number of organizations. In some cases, organizations struggled just to figure out who to talk with at the mission, and many said it took multiple requests to get a response. One grantee remembered: The US Mission…provided a general ‘wishing you well’ letter, and said it is not their practice to provide letters of endorsement. World Learning notes that USAID/Washington went to great lengths to get mission approval. This appears to have been unknown to grantees interviewed for this evaluation who were struggling to get the letters on their own. Issues related to mission engagement in SPANS/GSM-supported projects continued into the implementation stage. In most cases it was described as something between hands off and indifferent. One local NGO who had traveled thousands of miles to the 13 Most RFAs advised applicants about this in general terms and encouraged procurement of non-US items through matched funding to avoid the cumbersome waiver process. World Learning also notes that grantees were advised about this during the preaward and implementation period. 14 Uniformed Guidance (2 CFR 200) 15 World Learning notes that this was done where possible, but a more proactive plan that staged information requirements and aligned them with approval cycles may enhance turnaround time. 16 This is spelled out in RFA’s#3-10. SPANS/GSM Final Evaluation June, 2018 22 country’s capital city to present the project concept at the mission, successfully received their letter, and followed up with reports and updates but received no feedback or communication from the mission. Noted another local NGO: We invited them to everything, but our project wasn’t a priority, they weren’t interested. Several grantees remembered coordination challenges between DC-based USAID representatives and local mission personnel, including at grantee-sponsored events. Those close to USAID recognize that the relatively small size of the grants and already stretched human resource in some missions may account for this situation. Missions in conflict affected countries took a greater interest in proposed projects, and missions like Albania and Laos with presumably less pressures were reported to have contributed to the SPANS/GSM- supported projects in a number of ways. Recommendation: Recognizing that SPANS/GSM was positioned for countries where missions could not accept resources from one of the Congressional Funds for whatever reason, putting the onus on the grantee to solicit mission sign off seems unduly onerous, particularly at the BAFO stage. Mission buy in and awareness of all USG programming in their country is important, of course, but under the circumstances it may be more appropriate for USAID/W to make this request, or for the managing agency to make it on USAID DC’s behalf. This would be most straightforward under an RFA which anticipated funding in a limited number of countries. When geographic focus is not a criterion, mission approval could be sought at the vetting stage. • Cancelled projects. Over the life of SPANS/GSM one project was selected by the TEC but did not ultimately go ahead—in Afghanistan—and one was launched but had to be cancelled early in the project cycle—in China. Both cancellations had to do with partnership arrangements that were either unworkable for the grantee (both projects) or USAID (China, where a partnership with the GoC was proposed). In the case of the Afghanistan project, the original implementing partner was also deemed too weak to manage the size of the proposed grant. Whether these small number of cancellations could have been avoided with more lead time and discussion at the preaward stage is not known, and these issues were not investigated in this evaluation. Reporting Grantees were required to submit quarterly reports to World Learning which included project highlights, output monitoring data, and financial information. USAID credits World Learning with streamlining efficiencies through the introduction of user friendly forms and appreciated the organization of information in quarterly reports. Grantees were divided, with more concerns related to the reporting forms than the preaward survey or SPANS/GSM Final Evaluation June, 2018 23 the Grant Agreement. The quarterly financial forms got particularly high marks in the close out survey (4.5 out of 5.0), compared to the quarterly activity report forms (4.0 out of 5.0). Local NGOs found reporting particularly daunting. One said the explanatory guidance had so many components and links they got overwhelmed and stopped reading them. Another estimated it took a full week of staff time to compile every report because the requirements were so demanding, and because the reports they received from the field had to be synthesized and translated into English. Some of the INGOs balked at the forms, with a number suggesting the form was too confining (“tick box”), constraining a full presentation of their program and implementation realities. A number also noted that they have their own reporting systems, and that the forms added work when they required different information or information in different formats. On the other hand, a number of grantees—in particular grantees new to USAID— appreciated the forms so much they adapted them into their organization-wide monitoring and financial systems. Some saw the forms (and the requirements behind them) as a turning point in their systems development. Even some who complained about the demanding nature of the forms credited the requirements with strengthening their thinking and practice. Noted one: My colleagues would say ‘it’s too hard to manage these grants’. But World Learning boiled it down to the essentials. Their templates and advice were so helpful. Many grantees said the knowledge and experience they gained helped them to continue to engage with USG and other funders. Most of the grantees appreciated reporting reminders sent out by World Learning, though a number of the larger organizations noted that they have their own reminder system. Consistent with the on-line survey (Figure 3.), grantees had more to say about feedback on the quarterly reports, than the forms or format. There was broad agreement that to receive feedback on both narrative and financial reports was empowering, confirming the donor’s interest in their program. Many commented on the rigor of World Learning’s review of financial reports, with a number noting this kept them on their toes (and strengthened their reporting procedures in the process). World Learning reportedly went back to every grantee every quarter with at least one query—sometimes many more. SPANS/GSM team members also worked closely with grantees that were struggling to meet the reporting and other compliance requirements and this back and forth was SPANS/GSM Final Evaluation June, 2018 24 largely welcomed in the spirit of learning and improvement. 17 One grantee that struggled with the reporting requirements throughout had a different impression, remarking we always felt like we were on probation. By contrast, a substantial number of grantees said they wondered at times if anyone was reading their quarterly activity reports as no questions or comments would come back, or comments would come very late. This element received somewhat lower scores on both surveys (4.1 out of 5.0 in the close out survey). Anecdotally, there appears to have been an uptick in feedback towards the end of SPANS/GSM. The division of labor in report review mirrors the respective roles of USAID and World Learning. Grantees submitted their reports to World Learning. Review of financial quarterly reports was undertaken by World Learning. However, the activity component was passed directly to USAID without review. This point was made in each of World Learning’s own quarterly reports to USAID and appears to be based on a precedent or instruction established early in the program. Preferences from the USAID team also drove the review process with some asking for report summaries and others preferring to review every report in full. Although the request for activity reports to be submitted directly to USAID, a number of USAID observers reflected that this pass through was not optimal. Looking back there was a feeling that at minimum a review for completeness would have saved a lot of back and forth. Comparisons were made with the APC mechanism18 which USAID also uses to disburse support from the same Congressional Funds. Informants familiar with both mechanisms favored the lens that JSI (the APC holder) applied in prescreening reports before sending to USAID; and providing some technical feedback to grantees. Whether this vetting was something that was formally or informally requested by USAID is unknown. M&E reporting. The original Agreement states that World Learning would monitor program performance at three levels: i) grant solicitation and award; ii) performance of sub-awards; iii) program level performance (all sub-awards put together). USAID established a set of common indicators (otherwise referred to as “F” indicators) that all grantees we’re required to report on. These included information about number of people served; number of people trained; number of organizations strengthen and 17 It should be noted that most of the interviews carried out were with program or director-level staff who commented on the benefits to the grantee. Some of them also commented that their finance colleagues complained about the reporting demands at the time. 18 Towards the end of the first Phase of SPANS/GSM, USAID began channeling funds through this mechanism based in the Global Health Bureau, and managed by John Snow, Inc, a public health consulting firm. This mechanism has a similar ceiling to SPANS/GSM ($53.8m) and also largely caters to non-government and university implementing partners. The ceiling for this mechanism is almost identical to SPANS/GSM. The period APC under the five Funds is 5/5/2013-12/4/2018. SPANS/GSM Final Evaluation June, 2018 25 other output information, some of which was Fund-specific. USAID changed some of the indicators during the life of the program. Data was submitted to World Learning, then aggregated by Fund and submitted annually to USAID. No analysis was carried out. One USAID informant said that the information was useful for learning. A number of grantees noted that aggregating data from multiple field sites network was a challenge, particularly when they had weak systems for gathering or organizing monitoring data. The requirement to do so appears to have stimulated internal systems strengthening amongst a number of local NGOs. In 2015 World Learning developed a performance monitoring plan tracker to help one agency track its own objectives. When this was found to be effective, it was rolled out to all current grantees. Another innovation was the addition of learning questions in the quarterly reports at round the same time. Although this had always been a part of DCOF grants, it was added in as a requirement across the SPANS/GSM portfolio. According to World Learning appears to have had the desired effect of helping grantees reflect on and document change in their programs. All of the RFA’s request bidders to refer to prior evaluations in supporting proposed strategies. However, it was not until 2012 (RFA#8) that mid-term and final project evaluations were required under SPANS/GSM. Grantees who were interviewed who had conducted an evaluation referred to the value added in informing their learning and practice, and in providing a touchstone for sharing more widely. One regretted not having done an end of project evaluation. While these findings suggest some innovations in M&E reporting, they also point to the lack of a coherent M&E strategy more broadly. Although not directly connected to World Learning’s performance, this finding is included to provide learning for future Congressional Fund programming and is discussed further below. Financial management World Learning received funding annually for SPANS/GSM and grant awards mirrored the commitment to annual funding. However, shortfalls were never an issue with SPANS/GSM or with grantees. Most of the grantees opted to receive advances based on budget projections, rather than submit invoices based on expenditures. Both options were available and procedures were agreed during the due diligence phase. There were no complaints about the fund transfer process in the interviews, and it received high scores from grantees on the both surveys (4.6 out of 5.0 in the close out survey). Audits are mandated for all agencies receiving USG funding. Non-US based grantees that spent over $300,000 of USG funding during a given fiscal year were required to undergo a Recipient Contracted Audit (RCA). US-based organizations that spent over $500,000 of USG funding during a given fiscal year were required to undergo an A-133 audit. SPANS/GSM Final Evaluation June, 2018 26 In the decade long program, there were two cases of fraud--one involving funds from two SPANS/GSM grants under the same agency, the other within a grantee agency but involving resources from a different source. Both were resolved. Project extensions Of the 36 awards under SPANS/GSM, 23 (62%) were granted no cost extensions. Seven (7) grantees were awarded additional funding to allow them to continue their work for an additional 1-3 years (four were competed under RFA#7; three were awarded sole source). Three projects which were initially awarded in the first cycle, under RFA#1 received both a no cost extension and a cost extension (009 IRC/Burundi; 001 UCP-WFH; and 034 UCP￾WFH for RUK Indonesia). World Learning had developed forms for the process, which appears to have been smooth. The number of extensions is notable. Most of the grants started as 3 year projects. Delays in project start due to approval and other non-programmatic issues, cannot be discounted in terms of slower than anticipated burn rates. But grantees who received both cost and no cost extensions noted that in addition to catching up, the additional time also allowed them to consolidate gains and put in place programmatic elements that would contribute to sustainability. Motivation Romania Foundation (MRF) was one of a few agencies that was granted a five year project (and then received an additional one year cost extension). Over the course of the program, MRF expanded services to wheelchair users across 17 regions of the country, translated WHO service training modules into Romanian, mapped accessible buildings in Bucharest, and did a study on social inclusion of wheelchair users which has been cited internationally. They were reportedly able to achieve all of this because of the time available under the project to launch and consolidate services and support initiatives. Recommendation: Consider longer awards for larger grants so that sustainability gains are more likely, and extensions become less necessary. Funding streams from Congress appear to have been predictable for decades. Given the clear evidence that most of the types of programs SPANS/GSM funded over the last decade required longer time horizons, some flexibility in the allowable timescales might make programs more effective and improve efficiencies by reducing repeated applications and approvals for extensions. Close out Close out processes were no less rigorous and required significant inputs from World Learning and individual grantees. Close out calls have become a standard practice under SPANS/GSM but reportedly focus mostly on programmatic issues, including projected sustainability. SPANS/GSM Final Evaluation June, 2018 27 The Team Each Fund was supposed to have a dedicated technical advisor who was responsible for drafting RFAs and RFPs’, contributing to solicitation review and selection, reviewing grantee reports, and providing on-going technical feedback and support to grantees. 19 Changes and gaps in the advisory team did not appear to affect World Learning’s ability to play its role. Grantees signed agreements with World Learning under the Cooperative Agreement LWA held with USAID. As such, the fiduciary risk and liability for compliance with US Government rules and regulations was held by World Learning. World Learning already had a track record in managing analogous grant making schemes. Robust mechanisms, strong designs and conducive contexts contribute to good development programming. But projects can rise and fall based on the people who deliver them. Over the life of the program, the SPANS/GSM appears to have been a high functioning, collaborative team effort. This relied in important ways on each member of the team and each partner playing their respective role well, respecting the boundaries of that role, and respecting the contributions and decisions of others. Good working relationships appear to have characterized the team throughout, with shared commitment to the goal. The current Directors of the SPANS/GSM and the technical team lead at USAID have been with the program since its inception (in different roles), and many others enjoyed long tenures. Continuity within teams on both the USAID and World Learning sides certainly contributed to this constructive relationship. USAID has had three AORs over the course of the program—a relatively low turnover rate. One was with the program for five years. A few grantees mentioned that changes in the Agreement Officer at World Learning did create some disruption, as the relationship had been built. Portfolio effectiveness/impact At the portfolio level, it would be hard to assess whether project selection and Fund investments resulted in an effective program. The disparate nature of the grants, the apparent absence of an overall strategy, guiding theory of change or set of outcome indicators, even at the Fund level, would make this challenging. It appears that little effort has been made to promote or to consolidate learning within the portfolio, or to promote networks and cross pollination amongst grantees. While this issue is somewhat beyond the remit of this evaluation, it did underpin a number of structural and staffing issues that emerged in the course of the evaluation. The absence of a strategy or learning agenda showed up in terms of limited M&E capacity across the SPANS team, as noted above. USAID recognized this limitation even amongst its own technical team, but it remained a nascent aspiration over a decade-long program. World Learning’s mandate was limited to an annual aggregation of output indicators from grantee reports. Financial reporting, governance and compliance systems strengthening 19 In fact for some years one technical advisor supported several funds, and at present two of the five Funds have no advisor. SPANS/GSM Final Evaluation June, 2018 28 received significantly more attention than systems for measuring project impact or sustainability, leaving an impression of an outsize emphasis on compliance over demonstrable programmatic results. Little if anything was done to document or consolidate implementation outcomes or share lessons amongst or beyond the cohort of grantee organizations, even within a single Fund. As noted above, while a number of grantees opted to include midterm and final evaluations in their projects, neither were required. Noted one grantee: We have no idea what’s been learned in SPANS. Once money is given, it’s like a dark hole. They don’t publish final reports. So, the field doesn’t know what’s going on. We are informed informally, but it is more like gossip. This is particularly striking given the relative youth of the disability field, the call for innovation in each of the SPANS/GSM RFAs, and some of the groundbreaking initiatives that SPANS/GSM supported in a number of countries. 20 Another surprising indication of weak programmatic coherence was that most grantees were unaware of who else was receiving support under SPANS/GSM. This represents a missed opportunity to enrich the conversation, stimulate new ideas and build supportive affiliations. SPANS/GSM did support two conferences that included some but not all of the Wheelchair Program grantees, though only the first (in 2012) appears to have been designed explicitly to draw on practical learning from the SPANS/GSM projects. Documentation and sharing of learning is certainly something USAID has prioritized in other programs, and given the emphasis on innovation at the solicitation stage, this perspective certainly bears consideration. Summary USAID respondents gave World Learning consistently high marks for being knowledgeable, thorough and effective in the preaward and compliance monitoring stages of grants management. Grantees also appreciated World Learning’s knowledgeable advice on compliance issues that arose throughout the project cycle. Quite a few grantees (and particularly those new to USAID) also came to appreciate the rigor of the requirements (and the support provided by World Learning), as noted above. Curiously, however, this domain scored lowest of all of the domains on the close out survey at 3.2 out of 5.0 as shown in Table 5. 20 Quite a number of grantees stated that SPANS/GSM had enabled them to change government policy, build new service delivery cadre, implement new WHO standards, and pioneer or extend community-based approaches. SPANS/GSM Final Evaluation June, 2018 29 Table 5. Technical Assistance on Grants Management and Compliance Issues Type ofsupport to Grantees Score out of 5.0 Number of Grantees who responded World Learning provided TA on grants management and compliance area was useful. 3.8 23 World Learning shared tools and examples from other countries/programs were useful. 2.3 12 World Learning assisted to standardize grantee policies, procedures and internal controlsystems. 2.9 14 World Learning’s supportresulted the grantee’s organizational capacity building (specify area) 3.4 15 Source: Close out survey, World Learning It may be that respondents were reflecting more on the compliance requirements (which was not queried in the survey) and which were universally felt to be demanding. Indeed, a number of INGOs wondered in the interviews whether the onerous requirements were proportional to the size of the grants. Some PVOs compared requirements under SPANS/GSM unfavorably with other USG and Foundation funding and felt that the reporting requirements under SPANS/GSM were overly demanding, and counterproductive to actually doing the work intended by the grant. It may also be that because the data in Table 5. was collected at the immediate end of the project cycle, the compliance challenges were fresh in grantees’ minds. World Learning observers suggest it may be no more than the language in the question which provoked a number of respondents to leave the question blank and those who responded to score World Learning low. Given the informal and as-needed nature of assistance provided by World Learning, the use of “Technical Assistance” may have been misconstrued. More broadly, it is fair to say that World Learning effectively discharged its contractual responsibility under SPANS/GSM to support compliance with USAID rules and regulations amongst grantees. By outsourcing this role to World Learning, USAID gained an experienced partner with the necessary expertise to carry out the compliance function. This allowed USAID and its advisors to focus on the technical aspects of the program. World Learning was also effective in its support to grantees, who received tools and individualized guidance to help them comply with the programs’ rules and regulations. World Learning played an effective go-between role by respecting the requirements and SPANS/GSM Final Evaluation June, 2018 30 decisions of its donor client while also responding to the varied needs of its sub-awardee partners with tools and individualized support to optimize compliance. 3. Capacity Strengthening Evaluation Questions: 1c) Beyond the agreement, did SPANS/GSM contribute to other achievements? If so, what and how? 1d) How and to what extent did SPANS/GSM mechanism contribute to strengthening grantee’s capacity? At the time that SPANS/GSM launched, international funding for disability-related programming was quite scarce. For a number of grantees SPANS/GSM represented an exciting opportunity to extend and expand their work, and also the first time they engaged with USAID. Seven of the 31 grantees (23%) had never worked with a US Government agency before and for several others this was their first USAID award. The stringent rules and regulations under SPANS/GSM were unfamiliar to these organizations, and more often than not, their systems were not equipped to respond. Recognizing this possibility, the original Agreement between USAID and World Learning states that: As needed, World Learning can provide capacity building assistance to grantees. (Pg 12) Capacity considerations were only addressed beginning in the preaward stage, and were largely related to helping to assess, diagnose and figure out how to support each grantee to successfully meet USAID’s reporting and compliance requirements. Nonetheless, the diagnostic element appears to have set the stage for tailored, and sometimes intensive capacity strengthening for individual grantees. Bringing its considerable expertise in grants management, World Learning approached this largely through a learning-by-doing approach. World Learning provided grantees with guidelines, tools and templates to help them organize information, and enable them to respond to all of the compliance points at each stage of the grant cycle. World Learning was reportedly meticulous in reviewing budgets and asking questions related to waivers, approvals and allowable and non-allowable expenses. As noted, many of the grantees also sent staff to a USAID rules and regs workshop. World Learning visited a few grantees to provide face to face coaching, but this was not the norm, and visits were mostly when there were significant issues related to understanding of requirements. The rigorous reporting and compliance requirements were naturally easier for organizations familiar with USAID. But those without a lot of prior exposure struggled at first. The process was described variously as demanding, overwhelming, tedious, time consuming, frustrating, and like walking through fire. Echoing others, one grantee noted: World Learning enforced a discipline that we were not used to. SPANS/GSM Final Evaluation June, 2018 31 But the end result appears to have been transformative for many grantees. Most of the organizations that were interviewed who were new to USAID credited SPANS/GSM with improvements to one or more of their management, budgeting, financial reporting or monitoring systems. Systems strengthening extended to partner or branch offices which grantees then mentored by sharing standardized expenditure and project monitoring templates and requirements. Said one PVO representative: The experience introduced a management approach into our organization. Now we have a more logical way of thinking, including management by objective and using numbers to tell our story. And another: I couldn’t have offered better training to my grants management team than to work with World Learning. And from a grantee new to USAID: Due to the diversity of activities implemented within SPANS and the decentralization of service implementation at the local level, our organization developed a communication and coordination system and specific procedures used for implementation, monitoring and reporting purposes. This system proved to be very useful and it is still used in our organization, beyond SPANS. For a number of grantees, the SPANS/GSM award signaled the beginning of broader engagement with USG donors, and reportedly gave them tools and confidence to pursue other funding as well. (Several were already receiving funding from other donors, but SPANS/GSM was reportedly more demanding). World Learning’s thoroughness was matched by a willingness to engage constructively with grantees, meeting each organization where they were in their process. There was widespread appreciation for the responsive, supportive approach and guidance from the World Learning team, as mentioned above. Grantee organizations that describe positive changes share a number of characteristics: • A continuity of leadership with the confidence of staff and partners, who could shepherd the change through. • A receptivity to new approaches and making changes to compliance and M&E systems. • The in-house capacity to implement the changes or ability to effectively hire in added capacity (a number of organizations hired proposal writers and at least one mentioned contracting IT support to help build their regional M&E and financial system). The in person contact bears noting in this regard. Although grantee visits were not a regular feature of the preaward or overall management approach to the project SPANS/GSM Final Evaluation June, 2018 32 (presumably to keep costs down), in cases where compliance issues were diverting significant attention from implementation, in person visits from World Learning appear to have helped move things forward. In some of the most problematic cases reviewed in this evaluation, these visits were reported by both the grantee and World Learning as building personal relationships (and promoting trust) which paid off in terms of capacitating the grantee to meet the requirements of the grants. Since some of the organizational weaknesses were apparent from the outset, it is possible that visits earlier in the piece would have contributed to a smoother process over the project cycle. Recommendation: Consider conducting the bulk of the pre-award process in person, especially for agencies with somewhat weaker capacity. For such agencies the in person visit could include development of a light touch capacity strengthening plan. This process should meet requirements, but not be overly protracted. Institutional capacity building beyond simply strengthening compliance capacity never really featured in SPANS/GSM. Nonetheless, towards the end of the program, World Learning received requests from USAID to provide formal capacity building support to two local NGOs: 1) the Cooperative Orthotic and Prosthetic Enterprise (COPE) in Laos, a national NGO with little history with USAID; 2) UCP-RUK (UCP-Indonesia), a local NGO with historical ties to UCP-Wheels for Humanity, a US PVO which received multiple awards through SPANS/GSM. This capacity building support drew on expertise within World Learning which carried out a Participatory Institutional Analysis (PIA) with each agency between 2014-2018. UCP-RUK is registered in Indonesia but engagement with the mother organization (UCP￾WfH) continues through joint programming and a UCP-WFH seat on the RUK board. UCPRUK received a sub-award through UCP-WfH to carry out a training and policy advocacy program in Indonesia, (the second award to UCP-WfH under SPANS/GSM). UCP￾RUK also received capacity building support directly through World Learning. 21 While these two initiatives were implemented simultaneously and with the same grantee, they ran largely in parallel and under quite tight timelines. Achievement of the capacity building component was challenged by internal conflict followed by staff turnover including at leadership levels as well as competing demands to deliver the ambitious objectives of the program in just two years. Although both UCP-RUK and COPE Directors reflected that they fell short of meeting the various targets agreed at the PIA baseline, there was broad appreciation amongst the Directors and the staff who were interviewed about the capacity building initiative. The new Director of UCP-RUK said inheriting the Capacity Development Plan offered a roadmap that helped him meet his own institution building goals. Interestingly, he used 21 COPE was also in line to become a direct recipient of USAID funds, the case for capacity strengthening was compelling. SPANS/GSM Final Evaluation June, 2018 33 almost the same words as many of the other grantees to reflect on changes to organizational capacities: We built staff capacity by implementing the ISAP [the UCP-WfH SPANS/GSM-funded project]. Summary For many grantees, the partnership with World Learning contributed to organizational and systems strengthening. The compliance requirements provided a strong incentive, and support from World Learning offered pathways and encouragement. For UCP-RUK and COPE, the Capacity Development Plan (CDP) was considerably more holistic and detailed, but it was the learning-by-doing approach in both cases that contributed to stronger institutions. Although it would be easy to wonder whether genuine “capacity building” has been too closely conflated with “USAID compliance” in this project, it appears that for many grantees the compliance demands contributed significantly to building individual and organizational capacity. The combination of contractual obligations, the role of World Learning as a compassionate task master, and of course receptive grantee partners resulted in more durable systems and constructive management practices, at least as reported by grantees who were new to USAID. The outcomes of the more intentional efforts at capacity building are clear, but whether the investment in standalone exercises is necessary or good value for money in every case is less certain. Lessons from the embedded approach adopted by SPANS/GSM is that effective capacity strengthening is possible through an iterative, somewhat less formalized process. Recommendations: 1. Carry out a rapid assessment of applicants’ capacity before the TEC to ensure basic capacity is in place to manage the requirements of the grant, as well as a willingness and capacity to work on systems (as noted above). For weaker organizations with strong technical proposals, consider sequencing systems strengthening and program implementation to get the foundational human resource and systems in place before launching the program. The foundation laying phase could be in-person but should not be overly protracted, keeping in mind that systems can be built while programs are implemented (so: diagnose, design, do, mentor, refine) 2. In a future SPANS, ensure adequate skills and level of effort within the GSM to review, respond and mentor grantees. Include M&E expertise. 3. When standalone capacity building is included, provide more continuity of mentoring following the PIA, particularly to support areas where organizations lack specific type of expertise. Ensure organization-wide engagement and documentation of new standards or practice in anticipation of staff turnover. 4. For formalized processes, draw on local capacity building expertise where possible to allow for context-aware, nearby mentoring opportunities. SPANS/GSM Final Evaluation June, 2018 34 4. Non Grant Activities Evaluation Question: 2. How did SPANS/GSM perform in implementing non-grant activities in support of USAID’s efforts to fulfill the fie congressional mandated funds? a) How well did SPANS/GSM implement non-grant activities? Over the course of the SPANS/GSM World Learning managed 24 non grant activities, as shown in Table 6. Non grant awards represented 4% of the overall GSM budget. Table 6. Non Grant Activities supported under SPANS/GSM (2008-2018) Activity Number Evaluation (Cambodia) 1 Purchase Orders (Haiti; CLASP II*) 2 Consultancies 4 Contracts 1) 2011 Development of E-Learning Course for USAID (Carney) 2) 2017 Provision of WSTP Training for AHS (Mobility India) 3) 2017 E-Learning Course for USAID-active (prototype stage) 4) 2017 DCOF Development of care guidance document consultancy (Partners for Every Child-P4EC) 5) 2018 ISWP Consultancy-support ISWP develop governance structure (AMCES) 5 Meetings 1) 2012 Wheelchair Program (WP) Partners Meeting, Washington DC 2) 2014 National Disability Forum/Conference, Burma, Myanmar 3) 2015 DCOF - Child Care Reform Exchange and Training Workshop, Kigali, Rwanda 4) 2016 Africa Regional Education Workshop (AREW) Inclusive Education training for USAID Education Officers, Addis Ababa, Ethiopia 5) 2018 Wheelchair Stakeholders Meeting, Bangalore, India 6) 2018 Assistive Technology Market Shaping Workshop, Bethesda, Maryland 7) 2018 Changing the Way We Care Global Development Alliance (GDA) Co￾Creation Workshop, Washington, DC 8) 2018 Mental Health and Psychosocial Support (MHPSS) Meeting, Washington, DC 9) 2018 DCOF Family Care Guidance Workshop, , Washington, DC 9 Capacity Development Activities 1) MCPD; 2) MI; 3) COPE PIA; 4) UCP-RUK Indonesia PIAx2; 5) AHSx2 7 Total Non-Grant activities: 28 Table courtesy World Learning *Consolidated Logistics for Assistive Technology Supply and Provision, a cooperative agreement with UCP Wheels which has warehoused equipment in China and will distribute wheelchairs and other assistive technology globally. SPANS/GSM Final Evaluation June, 2018 35 Non Grant activities included five contracts competed through an RfP process. Contracts were let by World Learning under SPANS/GSM for specific deliverables or technical assistance. Wheelchairs provided after the Haiti Earthquake were funded under this mechanism. Conferences and Meetings At USAID’s request, World Learning provided support to the organization of nine conferences and workshops, with the majority taking place during Phase II. USAID informants had very positive things to say about World Learning’s logistical and administrative support to launching the overseas conferences, sometimes on tight timelines. The Myanmar conference, for instance, brought together more than 500 participants from all over the country, many of them people using wheelchairs. World Learning was able to manage this complex effort in part because they had their own staff on the ground. At USAID’s direction, World Learning also issued a sole source award to a consultancy that provided technical input to the conference. For the workshop in Rwanda, World Learning was able to call on in-house facilitation expertise when a consultant dropped out at the last minute. In India they worked closely with a local grantee. These conferences and workshops had quite different constituencies and purposes. The Myanmar conference was national event designed to contribute to a national coalition and national level awareness about people with living disabilities; the Ethiopia conference/training workshop was internal, designed to expose USAID Education Officers working to issues of disability and inclusion; the India conference was a regular event, bringing thought leaders and practitioners working with people using wheelchairs from various countries together. The last four workshops (6)-9) in Table 6. a) coincided with the final evaluation period, and so were not explored in the interviews with grantees. Amongst the conferences and meetings held before that, it appears that only the two wheelchair conferences (1) and 5) in the Table) were opportunities for SPANS/GSM grantees to share learning from their projects (though this was a byproduct rather than the purpose of the conference which was reportedly more focused on USAID strategy). Consultancies The team of technical advisors for SPANS/GSM provided the bulk of technical support to the grantees. There were only a few cases in which external consultants were recruited, and this was done through a competitive process. Most came in the last two years of the program. Notably, one consultancy was carried by one of the grantees (Mobility India) to provide technical training and mentoring to another grantee (AHS)—a successful consultancy from all accounts and a good example of cross pollination. The other consultancies were for deliverables designed for broader application in the sector. World Learning was largely involved in procurement and compliance management. No issues came to light in this regard. SPANS/GSM Final Evaluation June, 2018 36 Publications The original Agreement anticipated that World Learning would support preparation of publications related to the five Funds--originally conceived of with a Congressional audience in mind. Two e-learning courses have been supported, as noted in the Table above, and World Learning has assisted in the contracting. For other more outward facing results and learning publications, USAID reportedly found alternative ways of getting this done. Summary World Learning managed two dozen non grant activities under SPANS/GSM. The SPANS/GSM mechanism turned out to be a convenient way for USAID to contract for services, deliver hardware in a hurry, and organize conferences and workshops relevant to SPANS. In addition to managing the solicitation for contracted services, World Learning contributed technical and logistical support to a number of these efforts. The ability to contract for services under SPANS/GSM added flexibility to the mechanism. 5. Efficiencies Evaluation Question: 3. Has the timeliness and costs of managing the SPANS/GSM mechanism changed throughout the life of the project, and if so, how? a) What were the major factors that contributed to the increase/decrease in time and costs of managing the SPANS/GSM activities? SPANS/GSM required World Learning to provide continuity and at the same time flexibility from its team in order to respond to the varied requirements of the grants management cycle. The current Director estimates, that on average two Full Time Equivalent (FTE) professional equivalent were required to manage SPANS/GSM. At its height, towards the end of Phase I, World Learning was managing 20 Grants and had to call other staff into the team. The demands from non-grant activities--and in particular conference management and formal capacity building—also increased the overall Level of Effort of World Learning staff time to SPANS/GSM at various points in time. In 2016, at USAID’s request, World Learning undertook an analysis of the average time it took for each phase in the steps leading up to award approval. The data from this study highlights the protracted nature of the process, and where bottlenecks occur. SPANS/GSM Final Evaluation June, 2018 37 Table 7. Average length of time for preaward steps, SPANS/GSM Steps in preapproval process Average Length of Time Solicitation Release with Q&A 35 days TEC Selection 68 days Pre-Award 98 days Approval Requested 37 days Overall Process, on average 237 days (8 months) Data courtesy World Learning As shown in this Table, while applicants had about a month to prepare their submissions, they waited an average of seven months to start their work. A breakdown by RFA shows the turnaround time for each stage of the preaward process in more detail. Table 8. Solicitation process breakdown by RFA RFA No. of days RFA release to RFA closing date Average no. of days: TEC review/ shortlisting Average no. of days: Pre￾award review Average no. of days: USAID approval Average no. days: RFA release to USAID approval 1 WC 31 70 85 39 226 2 DCOF 30 76 56 67 228 3 LWVF 31 35 64 47 177 4 DF 31 15 33 16 95 5 DCOF 49 98 65 22 235 6 LWVF 31 86 119 38 274 7 WC 38 92 210 77 417 8 LWVF 37 69 102 53 261 9 DCOF 30 103 83 9 225 10 LWVF 40 33 161 5 239 Average 35 68 98 37 237 Data courtesy World Learning (FY17Q4 report, Annex #8) In a number of cases these last two phases cut months, sometimes a year off of a three year (or less) project, sometimes with no opportunity to extend at the back end because of the impending close date of the SPANS/GSM mechanism. (A few grantees received a no cost extension when the SPANS/GSM mechanism was extended). A number of variables contributed to the long turnaround time: SPANS/GSM Final Evaluation June, 2018 38 1. The number of iterations of review and input from the TEC—which appears to have tended to be higher for submissions under the DCOF and lower for the LWVF reviews. 2. The nature of TEC questions and the time for Applicants to respond—typically a month at first, but sometimes spread over multiple rounds of questions. 3. Requests for significant design or budget changes which required applicants to revise their submissions, which varied from complete rewrites to minor edits. In some cases, in order to accommodate resource availability, the TEC recommended significant budget changes (in one case halving the proposed budget) which then required revisions to the proposal. In some cases, objectives or activities were revised or removed which also required bidders to make narrative and budget adjustments. 4. The timeliness and completeness of organizational response to the preaward survey, which also sometimes required multiple rounds of review, and revisions to staffing or compliance agreements. 5. Mission approval, which was required at the proposal stage (by the grantee) and then requested by the SPANS USAID team. Both reportedly could take considerable time and multiple requests to confirm. (At least one application was denied by the mission at the TEC and preaward stage). 6. Speed of AOR and OAA approval (which varied significantly, from an average 5 days for Awards under RFA#10 to 67 days for awards under RFA#2) as shown in Table 8., above. Despite what some grantees perceived, the preaward stage was consistently longer than the USAID approval stage. As shown in Figure 4., both of these last phases fluctuated considerably over the life of SPANS/GSM neither improving or lengthening over time. Neither did efficiencies improve over time, as shown in this Figure. Figure 4. Average time required for preaward processing and USAID approval over the life of SPANS/GSM Data courtesy World Learning (FY17Q4 report, Annex #8) 0 50 100 150 200 250 300 350 #1 (2008) #2 ( 2009) #3 (2009) #4 (2009) #5 (2010) #6 (2010) #7 (2010) #8 (2012) #9 (2012) #10 (2015) Number of Days Average days for preaward Average days for USAID approval RfA SPANS/GSM Final Evaluation June, 2018 39 From a purely statistical point of view, the number of awards being reviewed or processed by the TEC at any one time seems to have had little to do with turnaround efficiencies either, as shown in Table 9 below. Table 9. TEC Selection efficiencies RFA# No. of eligible applications reviewed by TEC No. days for TEC review Average no. of days per application 1 17 70 4.1 2 80 76 1.0 3 50 35 0.7 4 3 15 5.0 5 30 98 3.3 6 22 86 3.9 7 38 92 2.4 8 12 69 5.8 9 14 103 7.4 10 4 33 8.3 Total 270 677 2.5 Data courtesy World Learning (FY17Q4 report, Annex #8) Nonetheless, the number of proposals under review (including OAA review) is blamed for the lengthy process in RfA #7. Many of these proposals were for cost extensions. Lengthy vetting and approvals did not just delay project award. In a number of cases, even after the award was made, waivers and travel approvals delayed project start up or scheduled activities as noted above. An implementing partner working in volatile areas in N. Pakistan waited a year to get a purchase waiver that would allow him to buy a non￾American made vehicle that would blend in to the local environment (and not make him a target). Work went ahead in the meantime, but a third of the project was finished by the time this grantee was able to get around efficiently. USAID and World Learning recognize that the delays are anxiety producing for implementing partners that may have made commitments to beneficiaries and staff without a clear idea of when they will be able to make good on those promises. While many of the grantees new to working with USAID were surprised and unsettled by this, those more familiar appeared to be resigned. USAID is never quick, said one. Work Planning Throughout the decade of SPANS/GSM, World Learning’s role was largely responsive. In addition to the routine grants management activities, World Learning kept up with the SPANS/GSM Final Evaluation June, 2018 40 ebb and flow of new solicitations and one-off activities like conferences, sometimes determined at short notice. Annual Work Plans were prepared but could not always predict these major activities which could be added into the pipeline by USAID. This required a good bit of adept management on the World Learning side, and the ability to pull in additional capacity at times. Management fee Neither the Health Grant Solicitation Management Leader with Associate Award or SPANS/GSM (which was under the LwAA) explicitly stated an administrative multiplier cost. It seems that 10% was informally agreed under SPANS/GSM, though this appears first in reports to USAID. Because it is a Cooperative Agreement, there was more flexibility in how World Learning calculated its fee for managing the SPANS/GSM. Ultimately, rather than using the 10%, World Learning calculated costs based on actuals, which of course fluctuated over the years. It remained fairly consistently under the 10%, ending up closer to 6% overall. A proportion of the costs were assigned to each of the Funds based on grant expenditures over the period. Summary One leaves this time analysis and many of the stories related by grantees with a strong impression of the intensely demanding nature of reporting under SPANS/GSM. In the heroic effort to achieve compliance with the complexity of requirements, it seems at times that the purpose of the exercise—to deliver robust programs to people in need— got lost in the paperwork. The disproportionality in time allocated for proposal preparation, for instance, compared with the protracted approval process is not unusual for USAID. Nonetheless, it does raise a number of questions about priorities and the compliance process as it has evolved into business as usual. Recommendations: 1. Based on its internal efficiency review, World Learning put forward a number of recommendations relevant for a follow on SPANS/GSM-type program: a) Set turnaround targets for various phases in the process. With this, World Learning anticipates that with more efficient turnaround at each stage, 6 months is a reasonable target from release of solicitation to award. b) Include presolicitation workshops and a concept paper stage to help applicants better prepare for the solicitation. This could reduce the review time by providing guidance earlier and having better prepared proposals. c) Allow 60 days as an upper limit for TEC review, assuming many questions can be addressed in the concept paper stage d) Reduce the preaward stage to 60 days by providing bespoke, in person support to new grantees thus circumventing the protracted back and forth via email that delayed awards in a number of cases e) Reduce the USAID approval time to 30 days (which is just 7 days less than what it was under SPANS/GSM). SPANS/GSM Final Evaluation June, 2018 41 Findings from this evaluation resonate with these recommendations if the format of the mechanism remains the same in the future. 2. Setting grantee expectations about reporting and approval processes in advance may not completely mitigate legitimate frustrations with delays to implementation but could help new grantees manage more effectively. 3. Consider whether grantees with a track record with USAID (or even demonstrated capacity under the mechanism) need to go through the entire preaward process or can be prequalified in some way based on past experience22 . 6. The Mechanism The GSM under SPANS provided a mechanism for efficiently disbursing resources from the five Congressional Funds to non-government entities. SPANS/GSM managed a very small piece of the annual $53 million which USAID received from these Funds. It reached a group of implementing partners many of which required a greater level of management support than, for instance, block grants to international multilaterals or programming managed by USAID missions. The stretched capacity inside of USAID led to contracting a team of technical advisors to support the grantees. 23 The use of World Learning’s Leader with Associates (LwAA) Agreement was deployed to manage the program, including executing a series of sub-awards grantees selected through a competitive process by USAID and its technical advisors. This tripartite arrangement had a number of precedents, and in the case of SPANS/GSM functioned smoothly. Observers agree that separating the fiduciary management from the technical direction responsibilities avoided a conflict of interest. USAID appreciated having the technical team close by, and both USAID and grantees had high levels of confidence in World Learning’s ability to manage the compliance element. Most of the grantees that had worked with USAID in the past appreciated having World Learning as a buffer to manage approval and other processes on their behalf. 24 However, a number of respondents also commented on the lack of transparencies in terms of what to expect vis a vis timing and decision making. World Learning may have been the face of the program to grantees (and applicants), but with little decision-making authority, World Learning was itself sometimes uncertain about when decisions or approvals would be forthcoming. 22 Although this was a USAID practice under other mechanisms in the past, no current precedents exist. Further World Learning expressed skepticism about whether a grantmaking agency would (or should) agree to assume liability for an agency it had not vetted itself for the specific project under consideration. 23 The current team of advisors are currently provided under a contract with New Editions, Inc 24 Nonetheless, one respondent familiar with USAID noted that having World Learning in the middle didn’t really contribute to efficiencies, as USG approvals still slowed things down. SPANS/GSM Final Evaluation June, 2018 42 As a funding mechanism, the Cooperative Agreement was designed by the US Government to empower substantive engagement in the design and delivery of a program, including shared fiscal investment, on the part of an implementing non￾government partner. Using this mechanism for SPANS/GSM, met USAID’s preeminent priority of including of non-government recipients in the pool of implementing partners under the five Congressional Funds. It essentially served to outsource grants management and fiduciary risk. But under SPANS/GSM, World Learning had virtually no say in the dimensions of the program, and a cost share was not anticipated in the terms of the Agreement. 25 For these reasons, many note, that it is somewhat misaligned with the original concept of the CA mechanism. At his point, it is not clear that other mechanisms would provide the flexibility and accommodate the size and diversity of grant recipients. What is clear is that a determination has been made that the Cooperative Agreement will not meet USAID regulatory requirements going forward. Unsolicited proposals were not a part of the original Agreement for SPANS/GSM. However, towards the end of the program period USAID requested World Learning to facilitate funding for two grantees that had directly approached them with projects relevant to the SPANS/GSM remit and of interest to USAID. This may have been the sole instance in which World Learning was not able to comply, as the process ran aground on the rules surrounding the vetting process, and the applicant organizations’ own ability to provide sufficient information. Nonetheless, there appears to be general agreement that unsolicited proposals have the potential to add value to meeting the aims of a SPANS/GSM type program. Further, World Learning also reports that going through the process of trying to accommodate such proposals helped them determine how they would go about receiving and responding to such requests in future. Inclusion considerations. Although the mechanism itself did not drive the size or number of solicitations or awards, the intensive management investment and compliance requirements inclined somewhat towards more established organizations and relatively larger projects. Broader USAID policy during this period promoted a diversification amongst implementing partners, and in particular inclusion of local NGOs. This was reportedly a tacit aspiration of SPANS/GSM as well. All RFAs were open to bids from US and non-US international NGOs as well as registered non-government organizations who met the other geographic or technical criteria. As shown in Figure 5., there were more applications from local NGOs than other sources over the life of SPANS/GSM—45% of all applicants--suggesting considerable interest and greater sustainability potentials in support of the goals of the five Funds. Under SPANS/GSM, 28% of awards went to local NGOs. 26 25 grantees were also not required to include cost share, but a number did both to signal their commitment to the projects and also to cover unallowable costs or items that might require lengthy waiver requirements. 26 From Albania, Cambodia, India, Laos, Romania, a regional initiative in the Middle East. SPANS/GSM Final Evaluation June, 2018 43 Figure 5. Awards compared to applications, by source* Most participating international organizations implemented their programs through local offices of course, and in at least one case the grantee executed sub-awards to other implementing organizations. In Indonesia, the SPANS/GSM program (and direct assistance from World Learning) helped a PVO branch office move closer to becoming independent. Notably, RFA#3 encouraged applicants to include country-based grantmaking mechanisms, and a good number of the grants to both international and local NGOs included a component that reached more local partners. This is a positive step towards greater inclusion. 145 46 95 30 7 8 16 2 0 20 40 60 80 100 120 140 160 180 Local NGOs INGOs USPVOs Other Applications Source of Applications and Awards Applications Awards Source: World Learning, 2018 NB. “Other” includes universities and UN Agencies *Awards represent actual grants; grantees that received more than one award are counted more than once SPANS/GSM Final Evaluation June, 2018 44 Summary Although World Learning’s role in SPANS/GSM did not fully align with USAID’s intent or usual application of a Cooperative Agreement, it nonetheless offered a workable mechanism for engaging with a range of non-government organizations, including organizations that had never worked with USAID before. Alternative mechanisms with the same goal would need to retain the flexibility of SPANS/GSM, be able to maintain compliance, and ideally be able to accommodate a diversity of grantees, including potentially organizations only able to absorb and report on smaller amounts of funding. The costs associated with grants management and implementation efficiencies would incline selection of larger grants to more established organizations, but it is worth considering what is lost—in terms of innovation or even impact—in the process. 27 This may be particularly important in the emerging disability and child protection fields where innovation in a variety of contexts could represent an important contribution. Some options to address this are put forward in the final Section. Recommendations 1. Given the success of SPANS/GSM as a mechanism, and the concerns about whether it qualifies as a Cooperative Agreement, near cousin alternatives should be explored. An award to an American NGO to deliver grants on behalf of USAID seems optimal, since these are USG funds, and an NGO would be more programmatically aligned with the intended grantees than, for instance a consulting firm with no on the ground experience. Candidates for this would need to have the requisite technical, management and compliance capacity (which could also be achieved through a partnership of more than one organization). By assuming this role, the organization would take itself out of the running for receiving implementation grants. Other options, including mechanisms within mechanisms that would broaden the scope and diversify participation in a future fund could also be considered, and are presented in the next Section. 2. Having two mechanisms for funding non-government organizations from the same Congressional Funds may provide bureaucratic flexibility, but could also compromise portfolio coherence, particularly if the next generation program aims to build more evidence and more dialogue amongst grantees. Centralizing the delivery mechanism for the five Funds within one organization would contribute to programmatic effectiveness and management efficiencies. 3. Joint annual work planning would contribute to efficiencies in a future SPANS/GSM type program. A more complex program that included formal capacity building support and a more explicit learning agenda, and/or some of the additional grant making mechanisms put forward in the next section would benefit from clear milestone planning to allow the GSM holder to anticipate human resource and budget requirements in advance. 27 One observer also notes that in the Wheelchair field, at least, the dominance of a few large organizations over a long period of time has limited new ideas and innovation. SPANS/GSM Final Evaluation June, 2018 45 IV. Summary and conclusions Evaluation Question: 4. What lessons and recommendations can be drawn from SPANS/GSM project that can contribute to the improvement of similar future projects? a) Did the SPANS/GSM project adapt based on the learning derived from its implementation and management, and if so, how? SPANS/GSM was a high functioning grant making program that enabled USAID to channel Congressional funding designed to address the specialized issues of disability, child protection and other forms of vulnerability to a diverse group of 31 non-government organizations with projects across some three dozen countries. World Learning held fiduciary responsibility for the entire portfolio and invested its time and expertise in helping grantees adhere to USAID rules and regulations in order to avoid issues of non￾compliance. With some very minor exceptions, this appears to have paid off. World Learning provided exceptional client support to both USAID and the grantees, and is credited with building capacity in the process. The GSM was established for SPANS as a way of boosting efficiencies and reducing turnaround time for grant making. While overall the mechanism appears to have streamlined elements of the process and lifted the erstwhile burden from stretched compliance divisions inside of USAID, efficiency stumbled on the very requirements SPANS/GSM was created to uphold. Long delays during the preaward and grant approval process, waiting for waivers, and even travel consent were common and sometimes affected the quality of programming in the field. The burden of this bureaucracy was felt most acutely at the front line by those delivering the program, not just in terms of the demanding paperwork but in the ambiguities around when approvals would be forthcoming. These realities appear to be baked into the system which determined the design and requirements of the program. Indeed, they represent an important raison d’être for SPANS/GSM. They do, however, give an impression of the preeminence of compliance over programming. At the end of the day it is comforting to know that this relatively modest pot of money was spent accountability. But was it spent well? This question appears to have received less attention, and SPANS/GSM as a program28 missed an opportunity to foster a community of practice and deepen evidence sharing. Most informants (including PVO informants) said they didn’t know which other organizations were receiving funding under SPANS/GSM. The time allowed for grantees to prepare proposals compared with the time consumed by vetting and approvals seems disproportionate. Some of the bidding organizations have a lean staff with low overheads. 28 This is not a reflection on World Learning, which was delivering a program that was designed and driven by USAID. SPANS/GSM Final Evaluation June, 2018 46 Some of the most effective projects funded under SPANS/GSM seem to be those that received an extension. Strengthened service delivery often led to advocacy to government, boosting potentials for sustainability. On the other hand, even putting the approval delays aside, a number of grantees with three years or less to implement their programs felt it was quite compressed. The concept of the Cooperative Agreement is that there is a shared ownership, stake and risk in the project or program under question. The structure of SPANS/GSM put a clear delineation between the technical and the fiduciary, with World Learning managing the fiduciary risk, but having no say over technical issues or even budget allocation. USAID got a high quality grants management service, which was, by design, never meant to engage with the development goals of the program. By making World Learning the face of the program to grantees, the emphasis of SPANS/GSM leaned towards compliance considerations. In a future program, USAID may wish to consider whether an expanded mechanism (or mechanisms within a mechanism) might broaden the net of participating organizations which are able to participate. A broader net would also require adjustments for capacity and scale, and inclusion of management and potentially capacity strengthening resources. Suggestions in this regard anticipate the option of smaller awards with lighter compliance and approval demands (and delays). 1. A two stage approach whereby organizations were asked to submit concept notes should be considered. 29 This would lighten the load for applicants and could reduce the ratio of solicitations reviewed in a more rigorous TEC and GSM screening review. Technical and compliance-related comments and guidance on concept notes invited to move to a full blown proposal might also strengthen submissions and reduce the TEC and preaward time in subsequent stages. (USAID observers note that it could also lengthen the process, by adding a step, however). 2. To encourage greater diversity amongst grantees, establish a pool of funding for small, pilot initiatives. This would allow grantees to trial new approaches and potentially encourage inclusion of smaller organizations and organizations working with harder to reach populations. Design and reporting for such a scheme should emphasize programmatic results measurement, with lower intensity, less frequent reporting on compliance issues. Appropriate technical support should be available. 3. In high priority countries, a country-based grant making mechanism should be considered, potentially as part of a larger program. The Family Care First mechanism in Cambodia, for instance set its strategy at the outset through a co-creation workshop. The workshop prioritized specific initiatives which were then competed out via a centrally managed mechanism (in this case managed by Save the Children/Cambodia). In addition 29 This recommendation was welcomed in interviews with grantees and put forward by World Learning in its efficiency analysis (Appendix #8 to July 1,2016-Sept 30, 2016 Quarterly Report, Fourth Quarter, FY, 2016). SPANS/GSM Final Evaluation June, 2018 47 to opening up opportunities for leveraging local expertise and engaging a range of local actors around a specific issue, this approach promotes collaboration. It also reduces the compliance burden by keeping grants small. High performing organizations could then transition to larger grants under the broader program. In this example, Save/Cambodia could function as a grantee, though other country-based mechanisms would be more desirable, reducing the number of vetting levels. 4. Irrespective of what kind of mechanism is adopted, a Strategic Plan that lays out a broad conceptual framework for each of the Funds (and highlights synergic opportunities amongst them) would be desirable. This plan could then inform scopes and RFAs. It would provide a shared roadmap for the range of interested stakeholders --implementing partners, missions, the technical and management team(s), and other parts of USAID. Embedded in such a Plan a learning agenda (potentially including a Theory of Change), should provide further direction to funding decisions. A Fund-based Theory of Change would also help inform a more nuanced set of core M&E indicators that moved beyond output measures to look also at impact and sustainability outcomes. 5. Inclusion of a general technical understanding on a future SPANS/GSM team was suggested by a number of stakeholders as desirable for enhancing the managers’ ability to complement (not replace) the technical team’s role. Specifically, an understanding of health systems, disability issues, and development programming would contribute to a more hands on role in document review. 6. In a future GSM inclusion of expertise in M&E would broaden the GSM’s contribution to learning. This may be best situated within the technical advisory team (assuming the same configuration of responsibilities). Part of the mandate of this position could be the synthesis of learning from across each Fund. Resources should be provided for sharing opportunities amongst grantees—at minimum in webinars or through a common website. 7. Promote a learning network amongst grantees to empower cross fertilization of ideas and innovation. Responsibility for this should sit within the management organization and could include anything from a website to a newsletter to virtual or in person fora. Sharing of lessons learned from projects should be central to such networking. Special Programs to Address the Needs of Survivors (SPANS), Grant Solicitation and Management (GSM) Project Final Evaluation ACA No. DFD-A-00-08-00260-00 Laurie Zivetz, MPH, PhD lzivetz@gmail.com June, 2018 Annexes Table of Contents Annex A ...........................................................................................Evaluation terms of reference Annex B .........................................Inception report, including interview and survey instruments Annex C ............................................................................................ List of interview respondents Annex D ......................................................................................................................List of grants Annex A Evaluation Terms of Reference SPANS/GSM TOR for End of program Evaluation February 2018 1 TERMS OF REFERENCE FOR END OF PROJECT EVALUATION SPECIAL PROGRAMS TO ADDRESS THE NEEDS OF SURVIVORS (SPANS) GRANT SOLICITATION AND MANAGEMENT (GSM) PROJECT Position: Evaluation Consultant Project Title: Special Programs to Address the Needs of Survivors (SPANS) Grant Solicitation and Management (GSM) Award No.: Associate Cooperative Agreement No. DFD-A-00-08-00260-00 under Leader with Associate No. GPO-A-00-04-00021-00 Period of Performance: August 8, 2008 to June 29, 2018 Evaluation Timeframe: March to May 2018 SPANS/GSM TOR for End of program Evaluation February 2018 2 Table of Contents 1. BACKGROUND ...................................................................................................................................... 3 2. SCOPE AND FOCUS OF THE EVALUATION .................................................................................. 4 3. COMPOSITION OF EVALUATION TEAM....................................................................................... 8 4. ACTIVITIES AND TIMEFRAME ...................................................................................................... 10 5. EXPECTED OUTPUTS ........................................................................................................................ 10 6. DISSEMINATION OF FINDINGS...................................................................................................... 11 7. SUBMISSION OF APPLICATION ..................................................................................................... 11 SPANS/GSM TOR for End of program Evaluation February 2018 3 1. BACKGROUND Since 2008, World Learning has implemented the Special Programs to Address the Needs of Survivors (SPANS) Grant Solicitation and Management (GSM) project on behalf of the United States Agency for International Development (USAID). SPANS/GSM is an umbrella grant making mechanism that supports service delivery, training and related activities aimed at assisting vulnerable populations. World Learning has implemented the SPANS/GSM project through a Cooperative Agreement from USAID in support of USAID's goal of improving the health and wellbeing of specific vulnerable populations, including displaced children and orphans, war victims, persons with disabilities, and victims of torture. The purpose of World Learning’s SPANS/GSM project is to release solicitations on behalf of USAID, award grants and provide individualized management support throughout implementation of the grants to assist recipients to comply with terms of the agreement and meet the intended goals of their projects. On behalf of USAID’s Bureau for Democracy, Conflict, and Humanitarian Assistance, World Learning has issued 36 grants under the SPANS/GSM project under five congressionally mandated funds: Displaced Children and Orphan’s Fund (DCOF), Leahy War Victims Fund (LWVF), Victims of Torture (VT), Wheelchair Program, and Disability Fund. World Learning administers the SPANS/GSM grant making process from identifying potential grantees, soliciting applications, monitoring grant implementation, to close-out. As necessary, World Learning also provides targeted capacity development support to grantees to improve performance. Additionally, World Learning implements other non-grant activities such as coordinating and implementing partner meetings and workshops, managing service contracts, and conducting capacity development activities, in accordance with USAID programmatic priorities. Since October 2008, the SPANS/GSM project has released ten Requests for Applications (RFA) for projects funded under the congressionally-mandated funds referenced above. As a result, over $47 million has been committed to 36 awards. The first phase of SPANS/GSM was from 2008 to 2013 during which nine Requests for Applications (RFA) were released. As a result, 31 grants were awarded to US and Non-US non-governmental organizations implementing projects in more than 40 countries. The average length of awarded grants was 3 years. The second phase of SPANS/GSM was a cost extension of three years, 2015-2018. In this phase, one RFA was released yielding two grants. Three additional grants were awarded through non-competitive processes. Table 1 below is a breakdown of grants awarded by corresponding fund, the RFA number and year of solicitation: SPANS/GSM TOR for End of program Evaluation February 2018 4 Table 1: Solicitations by year and number of grants awarded Year RFA Number Corresponding Fund Number of Awards Issued Phase I 2008 RFA #1 WP- Wheelchair Program 8 2009 RFA #2 DCOF- Displaced Children and Orphans Fund 1 RFA #3 LWVF- Leahy War Victims Fund 4 RFA #4 Disability Fund (DF) 1 2010 RFA #5 DCOF- Displaced Children and Orphans Fund 4 Year RFA Number Corresponding Fund Number of Awards Issued Phase I RFA #6 LWVF- Leahy War Victims Fund 4 RFA #7 WP- Wheelchair Program 2 2012 RFA #8 LWVF- Leahy War Victims Fund 4 RFA #9 DCOF- Displaced Children and Orphans Fund 3 Phase II 2015 RFA #10 LWVF- Leahy War Victims Fund 2 2015 - 2017 Follow-on and sole source awards WP – Wheelchair Program 3 Total 10 36 Detailed information of SPANS/GSM awards will be made available to the identified contractor. 2. SCOPE AND FOCUS OF THE EVALUATION 2.1. Evaluation Purpose The overall purpose of the evaluation is to assess the extent to which the SPANS/GSM project has met its intended goal to effectively serve as an umbrella grants making mechanism on behalf of USAID. As such, key questions of this evaluation will focus on the overall performance of the SPANS/GSM grants making mechanism. The evaluation questions are aimed to achieve the following objectives: SPANS/GSM TOR for End of program Evaluation February 2018 5 1. Determine the degree to which SPANS/GSM has achieved its intended results 2. Assess SPANS/GSM’s performance in implementing non-grant activities 3. Assess how the timeliness and costs of managing the grantmaking mechanism increased/decreased throughout the life span of the project 4. Recommend how similar future projects can be designed and implemented better for increased quality and effectiveness The evaluation will provide World Learning and USAID a clear report which examines achieved results, challenges and how those were addressed, and distill learning. 2.2. Evaluation Questions The evaluation is expected to answer the following main questions: 1. To what extent did SPANS/GSM meet its intended goal to effectively manage grants in support of USAID’s efforts to fullfill the five congressional mandated funds? 2. How did SPANS/GSM perform in implementing non-grant activities in support of USAID’s efforts to fullfill the five congressional mandated funds? 3. Has the timeliness and costs of managing the SPANS/GSM mechanism changed throughout the life of the project and if so, how? 4. What lessons and recommendations can be drawn from the SPANS/GSM project that can contribute to the improvement of similar future programs? 2.3. Evaluation Criteria Based on the purpose and objectives, this evaluation is expected to explore the results achieved by SPANS/GSM and how those results were achieved. To do so, the evaluation will explore issues of effectiveness and relevance of the SPANS/GSM project1 . Effectiveness is a measure of the extent to which the project has attained its objectives or results. Relevance is the extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In view of this, Table 2 below is a breakdown of each main evaluation question into sub-questions that will help explore the results achieved by the SPANS/GSM project, how the results were achieved, and what lessons can be drawn to inform similar future programming: Table 2: Sub-set of questions under each main evaluation question Key and sub-set questions to guide the evaluation 1. To what extent did SPANS/GSM meet its intended goal to effectively manage grants in support of USAID’s efforts to fullfill the five congressional mandated funds? a. To what extent did SPANS/GSM achieve its intended results per the agreement? 1 The DAC Principles for the Evaluation of Development Assistance, OECD (1991), Glossary of Terms Used in Evaluation, in 'Methods and Procedures in Aid Evaluation', OECD (1986), and the Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000). SPANS/GSM TOR for End of program Evaluation February 2018 6 b. What were the major factors that influenced the achievement or non-achievement of SPANS/GSM intended results? c. Beyond the agreement, did SPANS/GSM contribute to other achievements? If so, what and how? d. How and to what extent did SPANS/GSM mechanism contribute to strengthening grantees’ capacity? 2. How did SPANS/GSM perform in implementing non-grant activities in support of USAID’s efforts to fullfill the five congressional mandated funds? a. How well did SPANS/GSM implement non-grant activities? 3. Has the timeliness and costs of managing the SPANS/GSM mechanism changed throughout the life of the project and if so, how? a. What were the major factors that contributed to the increase/decrease in time and costs of managing the SPANS/GSM activities? 4. What lessons and recommendations can be drawn from SPANS/GSM project that can contribute to the improvement of similar future projects? a. Did the SPANS/GSM project adapt based on the learning derived from its implementation and management and if so, how? 2.4. Evaluation Methodology The evaluation will mostly use qualitative data collection and analysis methods, and will be supplemented by quantitative methods. Qualitative data collection methods will mainly include desk review (review of program documents), and semi-structured interviews with sample respondents from key stakeholder groups. Quantitative data collection methods will mainly include trend analysis on the timeliness and costs of implemented activities under SPANS/GSM mechanism to determine if these increased or decreased overtime. The contractor may propose data collection methods to be considered by World Learning and USAID. Desk Review The evaluator will be expected to conduct a desk review and data analysis to help answer the evaluation questions referenced above. The desk review will include, but may not be limited to, the documents referenced below: • SPANS/GSM project overview • SPANS/GSM award agreement documents including modifications • Quarterly Reports submitted by World Learning to USAID • Grantees’ close-out activity reports and completed feedback forms SPANS/GSM TOR for End of program Evaluation February 2018 7 Semi-Structured Interviews Interviews with relevant SPANS/GSM project staff will be conducted to explore perceptions around SPANS/GSM’s effectiveness. The interviews will be conducted with selected key representatives of SPANS/GSM grantees, USAID and World Learning staff. Table 3 below is a breakdown of key informants to be included in the interviews: Table 3: Number of key informants for interviews Stakeholder group Tentative No. of interviews per stakeholder group SPANS/GSM Grantees 23 (approximately 60% of grantees) World Learning At least 5 staff USAID At least 4 staff Sampling of SPANS/GSM Grantees The contractor will be expected to conduct semi-structed interviews with at least 60% of the total number of SPANS/GSM grantees. A purposive sampling approach will be used to determine which grantees will participate in the evaluation. This is to ensure fair representation from the various funding portfolios and the different types of organizations that have received funding through the SPANS/GSM mechanism. The following selection criteria will be used to sample the number of grantees to participate in the evaluation: 1. At least 60% of total grantees of each funding portfolio (DCOF, DF, LWVF, VOT and WC) will be selected to participate in the evaluation; a. Whereby both local and INGO organizations are present, the 60% is split between the different types of organization b. Whereby and as applicable, the 60% is split between local NGOs and INGOs within the funding portfolio. c. Whereby and as applicable, the 60% is split between first time recipients of US Government (USG) funding and recipients with USG funding experiences. Table 4: Sample size of grantees for semi-structured interviews FUND Total Grantees Total Sample Size Type of Organization* Experience with USAID** 100% At least 60% of total grantees INGOs Local NGOs Other (university, etc.) First time recipients of USAID funding DCOF 8 5 5 0 0 0 DF 1 1 0 0 1 0 LWVF 13 8 5 3 0 3 (all local NGOs) VOT 1 1 1 0 0 0 WC 13 8 3 5 0 2 (both INGOs) TOTAL 36 23 14 8 1 5 * Breakdown of ‘type of organization’ adds up to the total sample size. **Experience with USAID is a sub-category under ‘Type of Organization’ SPANS/GSM TOR for End of program Evaluation February 2018 8 Interviews with grantees will be conducted virtually (via Skype). Please note that grantees are in various geographical locations around the globe. As necessary, the contractor will be expected to be flexible and willing to schedule calls outside of the typical working day hours. Interviews with USAID, World Learning, and grantees based in the Washington, DC area will be conducted in person. Travel costs for the in-person interviews must be included in the evaluation budget. Data Analysis For qualitative data analysis, the contractor may propose the most appropriate and efficient methods. For quantitative data, it is expected that the analysis can be done in Microsoft Excel; however, the contractor may propose other analysis tools deemed appropriate. 3. COMPOSITION OF EVALUATION TEAM The evaluation will be conducted by an individual evaluator and/or two-person evaluation team. World Learning will provide oversight and coordinate logistical arrangements for the evaluation, as necessary. 3.1. Evaluator’s Qualifications The ideal evaluator(s) will have the following qualifications • A Master's degree in International Development, or any other in Social Sciences field; • At least 5 years of experience conducting and managing evaluations of USAID￾funded projects; experience with grant making mechanisms is desirable; • Prior experience managing a wide variety of stakeholders with varied interests and needs; • Experience conducting qualitative and quantitative (especially financial document review) data collection, cleaning, processing and analysis; • Must be able to obtain, analyze and evaluate a variety of information; organize, interpret and present it in verbal or written form for varied audiences and provide solid analysis leading to sound decision-making; • Must have excellent writing skills; • International experience is required. 3.2. Evaluator’s Roles and Responsibilities World Learning’s approach to grant management is of collaboration, mutual respect and is aimed to enhance learning. As such, World Learning expects the evaluation to uphold the same principles. Additionally, World Learning takes seriously the ethical responsibilities of evaluation studies. As such, it is expected that the evaluation will follow these standards: • Analytic rigor – The evaluation will be conducted systematically and utilize appropriate methodologies. Evaluation deliverables will require approval from USAID and World Learning to ensure rigor. World Learning will require periodic updates from the evaluator or team of evaluators. • Ethical - The evaluation is not to reflect personal or sectoral interests. As such, the evaluator(s) must have professional integrity, respect the rights of the institutions SPANS/GSM TOR for End of program Evaluation February 2018 9 and individuals to provide information in confidence, and be sensitive to the beliefs and customs of local social and cultural environments. • Independent – USAID and World Learning must not impose restrictions on the scope, content, comments and recommendations of evaluation reports. Evaluators must be free of conflict of interest. • Respect for people – Evaluators must respect the security, dignity and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact. • Informed Consent and Confidentiality – World Learning and anyone it hires takes on the ethical responsibility of obtaining consent to participate in the evaluation. Additionally, the evaluator(s) need(s) to keep the identities of the respondents safe, and be able to present recommendations specific enough to be addressed. The specific recommendations coming from a particular individual or group need to be safely documented. • Ownership of data - The data used, manipulated, and generated in the evaluation belongs to USAID and World Learning. The evaluator or team of evaluators will be required to submit any datasets manipulated and/or created. The evaluator will not use data or findings of the evaluation without prior approval from USAID and World Learning. Apart from upholding the standards indicated above, it is expected that the evaluator or team of evaluators will carry out the following duties and roles: i. Produce an inception report detailing the agreed upon evaluation methodology, tools for data collection, and initial findings from desk review; ii. Develop and obtain feedback from USAID and World Learning on the data collection tools; iii. Test data collection tools (semi-structured interview guide, etc.); iv. In coordination with World Learning, sample and select key informants to be interviewed; v. Conduct data collection, cleaning and analysis; vi. Write draft evaluation report; vii. Finalize evaluation report; viii. Present summary findings of the evaluation to USAID and World Learning. 3.3. Oversight and Involvement World Learning will provide overall oversight on the implementation of the evaluation through an evaluation committee. The committee will comprise of the following personnel: • Associate Director, SPANS/GSM • Agreement Officer, SPANS/GSM • Program Officer, SPANS/GSM • Monitoring Evaluation Research and Learning Specialist Specifically, the evaluation committee’s involvement includes, but is not limited to: SPANS/GSM TOR for End of program Evaluation February 2018 10 • Draft and oversee the evaluator’s contractual agreement; • Provide the evaluator with the necessary project documents to review and list of people to interview; • Review and provide feedback at every stage of the evaluation process (inception report, tool development, findings from data analysis, draft evaluation report, etc.); • Obtain approval from USAID on the proposed evaluation methodology and other relevant issues requiring USAID’s approval (e.g. travel authorization, final report, etc.); • Coordinate logistical arrangements such as scheduling of interviews • Conduct periodic check-ins with evaluator on progress 4. ACTIVITIES AND TIMEFRAME The evaluation will run approximately from March to May 2018 (subject to change as necessary), with an estimated total of 28 billable days. Table 5 below outlines evaluation timeline indicating key activities and deadlines: Table 5: Activity workplan with key deadlines Activity Period/ Deadlines Est. Number of billable days Hold kick-off meeting with WL Week of March 12 0.5 Review program documents and write inception report Week of March 19 3 Deliverable #1: Inception report submitted March 23 Meeting with WL to discuss inception report, logistics Week of March 26 0.5 Design and finalize data collection tools Week of March 26 3 Collect and analyzed data April 2-April 27 15 Write evaluation report Week of April 30 4 Deliverable #2: First draft evaluation report submitted for USAID and WL to review and provide feedback May 7 Revise draft report, addressing feedback and comments from USAID and WL Week of May 21 2 Deliverable #3: Final evaluation report submitted May 28 Deliverable #4: Hold debrief meeting with WL/USAID Week of May 28 Total billable days 28 days 5. EXPECTED OUTPUTS The following outputs will be expected during and after the evaluation: a. A signed agreement between the evaluator and World Learning disallowing non￾authorized use of the project data beyond the scope of this evaluation, an inception report clarifying the agreed methodology and tools, any preliminary findings from readings, agreed to by World Learning b. Data collection tools c. All organized and compiled raw data SPANS/GSM TOR for End of program Evaluation February 2018 11 d. Draft evaluation report e. Final evaluation report (data set for quantitative and qualitative data attached) f. Debrief with USAID and World Learning 6. DISSEMINATION OF FINDINGS The contractor will be expected to deliver the final report to World Learning upon completion. It is noted that this evaluation is the intellectual property of USAID and may be shared or disseminated only with USAID’s official approval. 7. SUBMISSION OF APPLICATION Applications to conduct this consultancy must be submitted to gsm@worldlearning.org. Application package must include the following: 1. Cover letter 2. Curriculum Vitae (CV) 3. One professional writing sample (maximum 3 pages) This opportunity will be open until filled, but no later than February 28, 2018. Annex B Inception report, including interview and survey instruments Inception report, Final Evaluation, SPANS/GSM April 12, 2018 1 Laurie Zivetz, MPH, PhD lzivetz@gmail.com 2314 Blaine Dr. Chevy Chase, Md 20815 (301) 221- 0536 Inception Report Final Evaluation: Special Programs to Address the Needs of Survivors (SPANS), Grant Solicitation and Management (GSM) Project Contents 1. Introduction 2. Methodology 3. Final Report Outline 4. Schedule 5. Guiding questions for USAID and WL 6. Guiding questions for interviews with grantees 7. Rapid survey questions 1. Introduction World Learning has managed the Special Programs to Address the Needs of Survivors (SPANS) Grant Solicitation and Management (GSM) project on behalf of USAID for a decade (2008-2018). SPANS includes five congressionally mandated funds: Displaced Children and Orphans Fund (DCOF); Leahy War Victims Fund (LWVF), Victims of Torture (VT), Wheelchair Program and Disability Fund (WP). SPANS/GSM has awarded more than $47m for more than 36 grants and has implemented other related activities including conferences, workshops, and evaluations. World Learning has commissioned a final evaluation to assess the extent to which the SPANS/GSM project has met its intended goal to effectively serve as an umbrella grants making mechanism on behalf of USAID (RfP, 2/2018). Specific evaluation questions include: 1. To what extent did SPANS/GSM meet its intended goal to effectively manage grants in support of USAID’s efforts to fulfill the five congressional mandated funds? 2. How did SPANS/GSM perform in implementing non-grant activities in support of USAID’s efforts to fulfill the five congressional mandated funds? 3. Has the timelines and costs of managing the SPANS/GSM mechanism changed throughout the life of the project and if so, how? 4. What lessons and recommendations can be drawn from the SPANS/GSM project that can contribute to the improvement of similar future programs? This report lays out the methodology for addressing these questions. Instruments for a rapid on￾line survey and 1:1 interviews are presented in the report. A proposed format for the final report and a timeline of activities have also been included. Inception report, Final Evaluation, SPANS/GSM April 12, 2018 2 2. Methodology The evaluation will rely on four main sources of information: 1) Document review, including quarterly reports, RfPs and RFA’s, close out reports (for sampled grantees). The time efficiency analysis and grant close out feedback gathered by World Learning also provide valuable sources of information for this evaluation. 2) Interviews with World Learning, USAID and SPANS technical advisors. Insofar as possible current and prior members of the WL and technical teams will be interviewed. Respondents based in the DC-area will be interviewed in person. Guiding questions are presented in Section 5. 3) Interviews with a sample of grantees. A cross section of approximately 60% of projects (some grantees have received more than one award) have been sampled purposively to ensure representation from each of the congressionally mandated funds, a mix of INGOs and local NGOs, and a mix of organizations that have received USAID funding in the past and those that received funding for the first time. The evaluator also selected a few that received a lot of attention in the quarterly reports, and USAID requested the inclusion of one grantee. Interviews will be done by Skype or phone. The interview questions are presented in Section 6. These are designed to guide a conversation and allow for probing, not to be delivered as a structured interview. Responses will be recorded as close to verbatim as possible through notes taken during the interviews. Notes will be organized by topic in a spreadsheet. Analysis of interviews will be done by the evaluator. The analysis itself will remain confidential. Main findings will be incorporated in the final report. The spreadsheet with grantee responses will be shared with World Learning in an annex to the final report. Names and all other identifiers will be removed. 4) An on-line survey will be emailed to all grantees, former WL and USAID stakeholders. The survey will include identical questions to allow for comparison in perspective. The survey will be created on a Google platform and responses will come directly to the evaluator. The survey will be intentionally short to encourage responses. Aggregate results will be presented in the report and shared with participating respondents if they express an interest in seeing them. Depending on the results, it is likely that presentation of findings will be stratified by respondent group. The online survey is an additional data source that was not part of the initial Terms of Reference for this evaluation. The evaluator suggested its inclusion and through discussions with World Learning during kick-off meeting, it was agreed that a simple survey should be included to generate quantitative data that will complement the interviews. World Learning will send an email out to all potential respondents (grantees, WL and USAID stakeholders past and current) alerting them to the final evaluation and encouraging participation in interviews and the survey. All respondents will be assured of anonymity and data will be anonymized before sharing back with WL and USAID in the final report. Inception report, Final Evaluation, SPANS/GSM April 12, 2018 3 These four sources of data will provide a comprehensive view of the GSM in order to address the evaluation questions, including recommendations for the future 3. Final Report Outline Unless World Learning or USAID have a preferred format, the following format is proposed for the Final Report: Executive Summary • Key findings and recommendations I. Introduction • Background of the project II. Evaluation methodology • Details on how primary and secondary data was sourced and used • Issues and limitations III. Findings and recommendations • This may be organized around the evaluation questions, or thematically based on key issues arising. • Recommendations would follow the discussion of an issue. IV. Summary and conclusions • Overview of findings and recommended next steps Annexes 1. Terms of Reference 2. Inception report 3. Instruments 4. People interviewed 5. List of documents reviewed 6. Interview notes (in .xlsx), anonymized Survey data, anonymized (in .xlsx) 4. Schedule Task Week commencing…. 26-Mar 2-Apr 9-Apr 16-Apr 23-Apr 30-Apr 7-May 14-May 21-May 28-May 4-Jun 11-Jun Document review Briefing with WL and USAID Key Informant interviews On line survey Data analysis report writing Debrief on initial findings (remote) USAID and WL review of draft report Final evaluation report Debrief with WL and USAID Inception report, Final Evaluation, SPANS/GSM April 12, 2018 4 5. Guiding questions for USAID and WL The questions presented in this section will be used in individual interviews with USAID and World Learning (including the technical advisors for SPANS who are based in USAID). Some questions may be more or less relevant for individual respondents, so they will be used as appropriate. In this presentation, the questions move from the granular to the general, but the ordering may also be adjusted depending on the respondent. To encourage candid, honest responses, these interviews will be conducted in confidence and given the small number of respondents, it may be easy to identify speakers. Therefore, transcripts will not be shared in the final report. About the solicitation process 1. Where did you advertise? Did this change over time? What were the most effective sites/mechanisms? 2. Why are the award cycles different (e.g. in FY12Q4 report it says the LWVF RfP was for 30 months; DCOF for 27 months)? 3. The Agreement notes that WL will keep a log of proposals and funded projects and analyze where there are gaps (geographic or sectoral). Was this done? If so, what was the result and how was this data used? 4. Did you use a checklist for reviewing proposals ahead of the TEC? During the TEC? Can you share whatever doc’s you used and asked TEC members to use? 5. At the end of the project, there was a brief window during which unsolicited proposals were accepted (and then this window appears to have closed). What happened? In your view what would be the benefits and disadvantages of such a window for a future GSM? About the pre-award process 1. What are the pre-award steps? What were some common issues that arose? Were any unique to agencies that have never received funding from USAID before? 2. How many agencies offered a cost share? 3. Does USAID have to approve key personnel for every grant? Monitoring and reporting 1. Were there core lists of indicators common to all grantees or other types of minimum required monitoring data? 2. The Agreement suggests WL will review monitoring systems, provide feedback/input, and conduct a data quality assessment in Yr2. Can you provide more insights into what WL did? 3. How did you determine when a site visit was needed to a grantee/project? How many were undertaken over the life of the program? 4. How did you determine which agencies received an audit? Was this paid for under the project? Did this include subgrantees (as relevant)? Were there any issues arising from any of the audits? Was SPANS audited? 5. Were projects required to have final evaluations? If the agency conducted a final evaluation, what happened to the eval results—who reviewed and were any consolidation of results/lessons undertaken? 6. There was one evaluation funded under the mechanism—it seems this was at USAID’s request and an anomaly. Should I interview the evaluators? Do you think more should have been supported under the GSM? 7. On the report review process: Why does USAID review grantee reports before WL? How has that worked? Does WL receive feedback from USAID? Inception report, Final Evaluation, SPANS/GSM April 12, 2018 5 Capacity assessment and capacity development 1. It appears the capacity assessment piece gained momentum in the second half of the program? 2. How and at what point in the process were agencies identified for capacity assessment? Was this a requirement or an option? 3. When was the PIA tool developed? What do you see as the main benefits of doing this assessment? 4. What kind of follow up was undertaken after the formal in person assessment? 5. In your view, should this be a standard part of future programs? Why? For whom? Conferences and publications 1. How did the ideas for conferences originate? 2. Was there sufficient capacity under the GSM to manage (logistics and technical)? 3. How were the grantees and grants linked with the conferences in terms of attendance, learning, presentations etc.? 4. A post workshop survey was carried out for the Inclusive Education workshop. Why only for this one? 5. The Agreement anticipates 7 short publications (4 6pg brochures; 3 x 30 pg. publications), with the content provided by USAID. How many were prepared and on what topics? 6. Grace: It will be good to add some question (s) around other non-grant activities since one of the evaluation questions is about the non-grant activities Close out 1. Were there close out calls with all grantees and/or contractors, or only those that had issues during their projects? Who attended the calls? Do you have notes from those calls? (for sampled agencies)? 2. FY15Q4 has data from a short survey which I think is compiled from close out comments from grantees? Who are the 226 respondents and how was this data gathered? 3. There was a survey done in 2016 that only reached out to 20 grantees. Why not all? Only 7 responded. Why do you think the response rate was so low? Financials 1. What is the difference b/w ceiling and obligated amounts (columns in financial statements)? 2. How were disbursement amounts determined? Is this prescribed by USAID? Did it meet the needs of grantees? 3. How were subawards monitored? It sounds like in some cases WL was right in the weeds of financial management for subgrantees, but in other cases not. Can you clarify? 4. How many agencies were granted a no cost extension over the life of the program? How many agencies received additional funding for the same project? What about follow on projects (e.g. MRF, see FY14Q4)? How were these extensions/additions reviewed and decisions arrived at? 5. The administrative costs for WL were initially anticipated at 10%. They fluctuated throughout but seem to have remained under the 10%. How was this calculated (not clear to me from the reporting)—seems to be on the basis of disbursements, not actuals? How was this reconciled? About the efficiency of the GSM efficiency analysis 1. Generally, how efficient do you think the SPANS/GSM was? Which aspects do you think worked best? Which could be improved? How? (probe on institutional arrangements, communications, staffing, structures). 2. What do you think are the benefits and disbenefits of having the financial/admin management separate from technical for managing a portfolio program like SPANS? Inception report, Final Evaluation, SPANS/GSM April 12, 2018 6 3. How well did the coordination between WL and the technical advisors work? What contributed to smooth coordination? Limitations? 4. In a future program how could this be improved? 5. Where did the initiative to do an efficiency analysis come from? 6. It would be helpful to hear your thoughts on the results of this analysis in terms of: a. Submission to funded grant ratio (about 10:1) b. Implications of review turnaround time on project implementation 7. How do these findings compare with other GSM’s that WL is managing or USAID has contracted? 8. There are a number of recommendations in the various iterations of the report for streamlining the GSM process. Were these discussed with USAID? Were any changes in approach adopted within the current SPANS/GSM? These include: a. A ceiling turnaround time for review segments b. Limiting TEC comments to one round c. Introducing a concept paper stage 9. Some of the participating agencies are well known to USAID. Was a prequalification process ever considered? What you are curious about In addition to the evaluation Q’s already laid out, what else would you like to learn from this evaluation? 5. Guiding questions for interviews with grantees Introduction Thank you for agreeing to talk to me. I am an intendent evaluator and have been asked to conduct a final evaluation of the SPANS/GSM program. The program has spanned 10 years and offered grants and other types of assistance to a broad range of agencies in a number of countries. The funding comes from congressionally mandated funds administered by USAID. Your feedback is therefore valuable in ensuring that the resources are managed and spent as efficiently and effectively as possible. This interview is completely confidential. I will be making notes about what you tell me, but you will not be quoted by name or organization without your permission. So please feel free to be honest and open. Background 1. Can you tell me a little bit about your organization and the project that was funded? (cite name) 2. What has been the main source of support for your agency to date? 3. What was your role in the SPANS-supported project? 4. How long were you with the agency before the project started? How long have you worked on the project? Bidding and pre award 1. How did you first hear about this opportunity? 2. Were you involved in preparing the bid? If so, do you recall how much time it took for your agency to put it together? Does your agency participate in this kind of bid on a regular basis? 3. How did you find the proposal guidelines? Timeframe? 4. When did you first hear back about your proposal? What kind of feedback did you receive? Was it clear? helpful? 5. Can you tell me about any steps you had to take before you were able to begin your project? Probe about types of questions, clarity of communications (and with whom), timing. Inception report, Final Evaluation, SPANS/GSM April 12, 2018 7 6. Looking back, how would you describe this pre award process in one word? Finances and compliance 1. How did you find the funding procedures for this grant? (probe on requisition processes, forms, timing) 2. Was the disbursement/funding process timely in terms of your project implementation? 3. Were there any issues with US Government regulations that you needed help on before or during your grant was awarded? What kind of help did you receive from World Learning? How would you describe the help you received (probe on communications, timing, commitment to resolve problem)? Reporting 1. How did you find the reporting requirements for this project? (probe for examples of helpful and challenging elements of financial and narrative reporting requirements; ask about frequency, level of information required, forms). 2. Did you get feedback on your reports? Was it helpful? (probe for example) Was it timely? 3. Was the feedback shared and used inside your organization? How? Capacity development support 1. Aside from the grant you received, did your agency participate in an organizational assessment or other types of agency-wide support? Was this the first time your agency participated in such a process? 2. Looking back, how helpful was the process? What have been the most important changes in your organization as a result? Positive or not so positive. Close out (as relevant) 1. How did you find the close out process (probe on requirements, forms, payment processing, tax refunds, if relevant, timing)? 2. Do you have any recommendations for making this process more efficient or more useful to your agency? General/satisfaction 1. Starting from the beginning of the grant process how would you describe your working relationship with World Learning? 2. How would you describe the communication with World Learning over the course of your grant? Suggested areas of improvement? 3. Looking back on the process, if you had to recommend a management and reporting process and relationship that would support your program best, what would you keep the same as it was? what would you change about the process? (probe for specifics) Encourage respondent to complete the on-line survey and thanks. 6. Rapid survey This is how the survey appears on line. The Grantee survey is presented here. The survey sent to USAID and WL included slightly adjusted language in Q’s 12 and 13 related to known examples from amongst Grantees. Q’s 14-17 were not relevant and not included. Inception report, Final Evaluation, SPANS/GSM April 12, 2018 8 Inception report, Final Evaluation, SPANS/GSM April 12, 2018 9 Inception report, Final Evaluation, SPANS/GSM April 12, 2018 10 Inception report, Final Evaluation, SPANS/GSM April 12, 2018 11 Annex C List of interview respondents Grantees (To maintain anonymity of individual respondents, organizational designations have been provided only. In a number of cases more than one person from an individual agency was interviewed) SPANS Project Number Albania Disability Rights Foundation 006 Al Hassan Society, Jordan 036 Buckner International 032 Cooperative Orthotic & Prosthetic Enterprise, Laos 020 COPE, Laos 020 Director, Handicap International 003, 035 Friends International, Cambodia 031 Health Volunteers Overseas 028 International Society of Wheelchair Professionals, University of Pittsburgh 037 Leonard Cheshire Disability 015 Mobility India 029 Motivation Romania 033 San Francisco State University/Whirlwind Wheelchair International 024 UCP-Wheels for Humanity 001, 034 World Institute on Disability 004 World Learning Luisa Angelsmith Associate Director, GSM, Institutional Strengthening Grace Longwe Program Associate, Institutional Strengthening Lauren Godfrey Program Offices, Institutional Strengthening Stephen Katz Former GSM Agreement Officer Thomas Bruey Senior Compliance Specialist USAID Cara Thanassi Agreement Officer Representative Cathy Savino Project Director, DCOF, and VoT Funds John Williamson Snr Technical Advisor, Children in Adversity Michael Allen Snr Advisor, Assistive Health Technologies Sue Eitel Former Tech Advisor, WP, LWVF Annex D List of grants SPANS/GSM First Quarter of FY18, Appendix #1 Status of Grants & Non-grant Activities as of December 31, 2017 Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Wheelchair Program UCP - Wheels for Humanity (UCP-WFH) SPANS 001 CLOSED 3,290,056 3,290,056 3,290,056 - Reach Out and Care Wheels (ROC Wheels) SPANS 002 CLOSED 749,803 749,803 749,803 - Handicap International (HI-WAMI) SPANS 003 CLOSED 999,250 999,250 999,250 - World Institute on Disability (WID) SPANS 004 CLOSING 2,659,224 2,663,000 2,649,021 13,979 International Nepal Fellowship (INF) SPANS 005 CLOSED 650,000 650,000 650,000 - Albania Disability Rights Foundation (ADRF) SPANS 006 CLOSED 900,000 900,000 900,000 - Motivation Romania Foundation (MRF) SPANS 007 CLOSED 2,146,704 2,146,704 2,146,704 - Association for the Physically Disabled in Kenya (APDK) SPANS 008 CLOSED 598,765 598,765 598,765 - Handicap International (HI) SPANS 013 CANCELLED - - - - SFSU/WWI SPANS 024 CLOSED 982,433 982,433 982,433 - CBM-Zimbabwe SPANS 025 CLOSED 1,995,780 1,995,780 1,995,780 - Motivation Romania Foundation (MRF) SPANS 033 CLOSED 346,425 346,425 346,425 - UCP - Wheels for Humanity (UCP-WFH) Follow-on SPANS 034 OPEN 814,217 814,217 597,422 216,795 University of Pittsburgh/International Society for Wheelchair Professionals (UPitt/ISWP) SPANS 037 OPEN 349,808 349,808 - 349,808 Sub-total Grants 16,482,465 16,486,241 15,905,659 580,582 Handicap International (HI) PO 001 CLOSED 10,882 10,882 10,882 - Invacare PO 002 CLOSED 63,945 63,945 63,945 - Aug 2012 WC Partners Meeting 87,992 87,992 87,992 - UCP RUK PIA 2016 8,152 8,152 8,152 - Jan 2018 Wheelchair Program Partners Meeting**** 167,414 167,414 - 167,414 UCP-RUK Follow-on PIA (2017-18) 12,088 ISWP Capacity Development** 21,947 Sub-total Other Activities 372,420 338,385 170,971 167,414 Projected WCF Totals 16,854,885 16,824,626 16,076,630 747,996 Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Displaced Children & Orphans Fund (DCOF) International Rescue Committee (IRC)/Burundi SPANS 009 CLOSED 3,130,780 3,130,780 3,130,780 - EveryChild (EvC) SPANS 016 CLOSED 2,993,839 2,993,839 2,993,839 - International Rescue Committee (IRC)/Thailand SPANS 017 CLOSED 2,494,883 2,494,883 2,494,883 - World Vision SPANS 018 CLOSED 2,498,702 2,498,702 2,498,702 - Bethany Christian Services International (BCSI) SPANS 019 CLOSED 90,194 90,194 90,194 - CHF International SPANS 030 CLOSED 998,371 998,371 998,371 - Friends International (FI) SPANS 031 CLOSED 992,915 992,915 992,915 - Buckner International (BI) SPANS 032 CLOSING 999,947 999,947 995,133 4,814 Sub-total Grants 14,199,631 14,199,631 14,194,817 4,814 Care Guidance Consultancy 100,000 - - - Cambodia Multimedia Success Story** 44,072 Sub-total Other Activities 144,072 - - - Projected DCOF Totals 14,343,703 14,199,631 14,194,817 4,814 Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation CECA Fund (Children in Adversity) Sub-total Grants - - - - CCR Workshop Rwanda (Mar 2015) 170,174 170,174 170,174 - EMC Cambodia Evaluation - Part I, II, III 121,727 121,727 121,727 Sub-total Other Activities 291,901 291,901 291,901 - Projected CECA Totals 291,901 291,901 291,901 - Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Leahy War Victims Fund (LWVF) Catholic Relief Services (CRS) SPANS 010 CLOSED 757,653 757,653 757,653 - Motivation Charitable Trust - Multi Country (MCT/MC) SPANS 011 CLOSED 1,977,238 1,977,238 1,977,238 - Motivation Charitable Trust - Sri Lanka (MCT/SL) SPANS 012 CLOSED 539,459 539,459 539,459 - Leonard Cheshire Disability (LCD) SPANS 015 CLOSED 937,390 937,390 937,390 - Cooperative Orthotic & Prosthetic Enterprise (COPE) SPANS 020 CLOSED 1,421,997 1,421,997 1,421,997 - St. Boniface Haiti Foundation (SBHF) SPANS 021 CLOSED 1,629,753 1,629,753 1,629,753 - CURE International SPANS 022 CLOSED 783,959 783,959 783,959 - Christian Blind Mission SPANS 023 CLOSED 1,992,161 1,992,161 1,992,161 - UCP - Wheels for Humanity (UCP-WFH) SPANS 026 CLOSED 811,423 811,423 811,423 - Health Volunteers Overseas SPANS 028 CLOSED 736,345 736,345 736,345 - Mobility International (MI) SPANS 029 CLOSED 602,061 602,061 602,061 - Handicap International (HI) SPANS 035 OPEN 1,200,000 1,200,000 705,265 494,735 Al Hussein Society (AHS) SPANS 036 OPEN 757,740 757,740 567,605 190,135 Sub-total Grants 14,147,180 14,147,179 13,462,310 684,870 C.Zimbalist MI Visit - Other Activities 4,834 4,834 4,834 - COPE PIA 2014 29,059 29,059 29,059 - NDF Burma June 2014 281,484 281,484 281,484 - MCPD Activity 323,745 323,745 323,745 - AHS Grant Management Training 5,926 5,926 5,926 - Mobility India WSTP Amman 216,210 216,210 153,027 63,183 L.Godfrey R.Sun AHS Monitoring Visit 14,556 14,556 14,556 - L. Godfrey AHS Monitoring Visit 7,288 7,288 7,288 - Sub-total Other Activities 883,102 883,102 819,919 63,183 Projected LWVF Totals 15,030,282 15,030,281 14,282,229 748,053 Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Disabilities Fund (DF) University of Iowa (U of I) SPANS 014 CLOSED 1,599,287 1,599,287 1,599,287 - Sub-total Grant 1,599,287 1,599,287 1,599,287 E-Learning Contract 91,919 91,919 91,919 - AREW - Inclusive Education Training 42,384 42,384 42,384 - E-Learning 102 Course*** 50,147 35,599 - 35,599 Sub-total Other Activities 184,451 169,902 134,303 - Projected DF Totals 1,783,738 1,769,189 1,733,590 - Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Victims of Torture Fund Center for Victims of Torture SPANS 027 CLOSED 939,464 939,464 939,464 - Sub-total Grants 939,464 939,464 939,464 - JW Consultancies (July-Aug 2009; July-Sept 2010) 15,257 15,257 15,257 - KC Consultancy (June-July 2013) 25,531 25,531 25,531 - KC Consultancy (Dec '14-Apr '15) 46,990 46,990 46,990 (0) Global Mental Health Meeting 165,000 - - Sub-total Other Activities 252,778 87,778 87,778 (0) Projected VOTF Totals 1,192,242 1,027,242 1,027,242 (0) Grantee Grant # Status Grant Amount Actual/Projected Amount Obligated by WL Amount Disbursed To Date Balance Based on Obligation Total Grants 47,368,027 47,371,802 46,101,535 1,270,266 Total Non-grant Activities 2,128,724 1,771,069 1,504,873 230,597 Grand Total 49,496,751 49,142,871 47,606,410 1,500,863 ** Activity is illustrative and has not yet been approved by USAID *** Includes anticipated increased contract costs due to additional rounds of storyboard edits; not yet obligated because the budget is under review by WL and will seek AOR approval in due course