GBI ALLIANCE PERFORMANCE EVALUATION PROJECT PERFORMANCE, LESSONS LEARNED, AND RECOMMENDATIONS FOR FUTURE USE February 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Beeta Tahmassebi, Kelsey Simmons, and Elisa Knebel of EnCompass LLC. Prepared for the United States Agency for International Development Contract Number AID-OAA-1-15-00021; Order Number AID-OAA-TO-16-00031 February 2017 Implemented by: EnCompass LLC 1451 Rockville Pike, Suite 600 Rockville, MD 20852 Phone: +1 301-287-8700 Fax: +1 301-685-3720 www.encompassworld.com GBI ALLIANCE PERFORMANCE EVALUATION PROJECT PERFORMANCE, LESSONS LEARNED, AND RECOMMENDATIONS FOR FUTURE USE DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Policy, Planning, and Learning – Learning, Evaluation, and Research This task order is implemented through the Policy Planning, and Learning – Learning, Evaluation, and Research (PPL-LER) IDIQ, funded by the U.S. Agency for International Development (USAID) under Contract No. AID-OAA-1-15-00021. USAID’s Bureau for Policy, Planning and Learning (PPL) awarded EnCompass LLC the 5-year PPL-LER contract to provide technical and advisory services for performance and impact evaluations, evaluation and performance monitoring capacity building, and performance monitoring activities at the mission, bureau, and agency-wide levels. PPL-LER task orders design and implementing quantitative and qualitative evaluation studies and assessments, develop and deliver evaluation and performance monitoring training, and provide technical assistance in performance monitoring for USAID development programs worldwide Recommended Citation Tahmassebi, B., K. Simmons, and E. Knebel. 2017. GBI Alliance Performance Evaluation: Project Performance, Lessons Learned, and Recommendations for Future Use. Prepared for USAID. Rockville, MD: EnCompass LLC. February 2017 | GBI Alliance Performance Evaluation i CONTENTS ACRONYMS.......................................................................................................................II EXECUTIVE SUMMARY ................................................................................................ III EVALUATION PURPOSE AND QUESTIONS ............................................................. 7 Evaluation Purpose ........................................................................................................................................................ 7 Evaluation Questions .................................................................................................................................................... 7 PROGRAM BACKGROUND............................................................................................ 9 Inception .......................................................................................................................................................................... 9 Evolution ....................................................................................................................................................................... 10 EVALUATION METHODS AND LIMITATIONS .......................................................12 Methods ......................................................................................................................................................................... 12 Limitations..................................................................................................................................................................... 13 FINDINGS ........................................................................................................................14 Appropriateness of Award Mechanism and Technical Scope........................................................................... 14 Program Performance, Achievements, Challenges, and Lessons Learned .................................................... 17 Feedback on Lessons Learned for Future Mechanisms ..................................................................................... 23 Lessons Learned in the Management and Structure of the GBI Alliance ...................................................... 25 CONCLUSIONS ..............................................................................................................27 ANNEX 1. EVALUATION STATEMENT OF WORK ................................................29 ANNEX 2. EVALUATION METHODS AND LIMITATIONS ...................................31 ANNEX 3. DATA COLLECTION INSTRUMENTS ....................................................34 ANNEX 4. SOURCES OF INFORMATION ................................................................. 39 ANNEX 5. DISCLOSURE OF CONFLICTS OF INTEREST ......................................40 February 2017 | GBI Alliance Performance Evaluation ii ACRONYMS AOR agreement officer’s representative COR contracting officer’s representative GBI Alliance Global Broadband and Innovations Alliance ICT information and communication technology ICT4D information and communication technology for development Lab/CDD Global Development Lab’s Center for Digital Development M&E monitoring and evaluation NGO nongovernmental organization SOP standard operating procedure USAID United States Agency for International Development February 2017 | GBI Alliance Performance Evaluation iii EXECUTIVE SUMMARY The Global Development Lab’s Center for Digital Development (Lab/CDD) team catalyzes programs and partnerships in digital finance, inclusive access to digital and mobile-based information and service delivery, and the use of data collected by digital devices to improve decision making. As part of this effort, the Lab/CDD supports the Global Broadband and Innovations Alliance (GBI Alliance). The GBI Alliance was conceived to achieve two goals:  Extend the reach of broadband, including enhanced mobile networks, into more remote rural areas as key infrastructure for extending the reach of socioeconomic services through USAID’s nongovernmental organization (NGO) and contractor partners.  Leverage the extension of these networks for the delivery of network-enabled, value-added application support across USAID’s development portfolio. The GBI Alliance was designed to support expansion and use of broadband Internet (mobile and fixed) to provide rural connectivity in the developing world, while expanding the adoption and deployment of innovative, value-added, and scalable information and communication technology (ICT) solutions. The Lab/CDD contracted EnCompass to conduct a performance evaluation of the GBI Alliance from its inception period in 2010 until February 2017. The EnCompass evaluation team’s tasks were to:  Determine the appropriateness of the program’s original technical scope and award mechanism  Assess program performance, achievements, and challenges  Provide recommendations for future mechanisms  Compile knowledge of lessons learned and areas for improvement. EVALUATION QUESTIONS In a series of meetings in September 2016, EnCompass worked with the Lab/CDD and NetHope to craft a list of evaluation questions, provided below, that would generate the most useful and relevant data for future planning for USAID. The evaluation questions are grouped in three categories. 1. Appropriateness of the Program’s Original Technical Scope and Award Mechanism a) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for the GBI Alliance at the time of inception? b) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? c) Has the prime/sub-partner structure originally envisioned to implement GBI Alliance activities been effective? February 2017 | GBI Alliance Performance Evaluation iv d) Has USAID provided adequate guidance to the GBI Alliance regarding desired outcomes? Of the mechanism overall? Of particular projects? 2. Program Performance, Achievements, Challenges, and Lessons Learned a) How has the prime partner worked with project sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? b) What have been the actions taken (by the project) to mitigate risks and course-correct during project design and implementation? c) How satisfied have various stakeholders (e.g., USAID, sub-partners) been with the work done by and assistance from the GBI Alliance? d) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate, and timely? e) What have been the program’s achievements and challenges? 3. Recommendations for Future Mechanisms a) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from the GBI Alliance need to be continued and what anticipated services are likely to be needed? b) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? METHODS Given the evaluation’s focus and the limited resources available, EnCompass and USAID determined that interviews with key informants and an accompanying document review were the best data collection methods to obtain the information needed to understand the GBI Alliance’s performance. Key informants were sampled purposively and in concert with the evaluation reference group, which comprised USAID and NetHope staff. EnCompass designed structured interview guides, employing an Appreciative Inquiry approach to explore exceptional experiences, values, successes, challenges, and wishes for the future of the GBI Alliance. The evaluation team conducted 30 interviews from 4 key stakeholder groups: USAID/Washington, USAID missions, GBI Alliance staff, and partners that worked with NetHope on GBI Alliance projects. FINDINGS AWARD MECHANISM AND TECHNICAL SCOPE The GBI Alliance was funded through a cooperative agreement for two reasons: to enable USAID to engage implementing partners as thought partners in ICT for development (ICT4D), a field that is more nascent at USAID, and partially to allow more flexibility in working in the technology sector, which changes more rapidly than other sectors. USAID and NetHope staff both appreciated the flexible nature of the cooperative agreement, which enabled the program to adapt scopes of work based on the most February 2017 | GBI Alliance Performance Evaluation v appropriate technology and information available. However, NetHope reported that USAID occasionally mandated changes in scopes of work or consultants without NetHope’s input, hampering the collaborative relationship and resulting in inefficiencies and missed opportunities for innovation. Another reason for choosing a cooperative agreement, and for choosing NetHope to lead the agreement, was to give USAID access to a large pool of partners, including partners from the private partners, which have been traditionally more difficult to engage. Although USAID was satisfied with NetHope’s ability to reach technical experts, there was a desire to reach a broader range of partners in the future, especially those in the private sector. It was widely reported that NetHope effectively managed its GBI Alliance partners. However, key informants suggested that clarity in lines of communication and reporting among the partners, the GBI Alliance, and USAID/Washington could have improved. PROGRAM PERFORMANCE, ACHIEVEMENTS, AND CHALLENGES In general, USAID/Washington and mission staff were highly satisfied with the GBI Alliance’s overall performance and achievements. USAID and NetHope reported success from specific projects focused on sustainably increasing broadband connectivity. GBI Alliance was also recognized for their influence on increasing attention to connectivity issues, and USAID’s presence in the ICT4D space, inside and outside the agency. However, USAID’s narrow requirements for project performance monitoring and the GBI Alliance’s limited mechanisms for doing so limited the ability to measure this success. NetHope and USAID both faced challenges with the program. For NetHope, a challenge was its limited operational capacity, which translated to an inability to provide timely and complete financial reporting to the agency. USAID was challenged by changing leadership in the agency, which resulted in shifting priorities for the mechanism and made it difficult for NetHope to gain a clear vision of the goals. Limited marketing of the mechanism to missions and other bureaus and offices resulted in lower-than-expected buy-in from the missions, where staff are less familiar with the value of ICT4D and how it fits into their broader portfolios. FEEDBACK ON LESSONS LEARNED FOR FUTURE MECHANISMS GBI Alliance partners, USAID/Washington and mission staff, and partners suggested lessons learned for the GBI Alliance and for future mechanisms in three key areas:  Making more effective use of implementing partners to engage more offices inside USAID and more partners in the GBI Alliance consortium  Increasing USAID/Washington’s capacity to manage mechanisms, which would result in improved systems for communication, oversight, and accountability  Increasing advertisement of the mechanisms and engagement with USAID missions. February 2017 | GBI Alliance Performance Evaluation vi CONCLUSIONS The evaluation draws six main conclusions, listed below and mapped to findings in the conclusions section:  Conclusion 1: Although there are opportunities to improve collaboration and innovation between USAID and NetHope, the flexibility afforded by the cooperative agreement mechanism was appropriate for the GBI Alliance.  Conclusion 2: Although USAID’s financial support for the GBI Alliance was important, systems for management and communication could be improved.  Conclusion 3: USAID and partners were satisfied with NetHope’s overall performance, but the ability to assess NetHope’s level of success was limited by a lack of monitoring and evaluation (M&E) frameworks.  Conclusion 4: NetHope and USAID faced challenges related to management capacity, and the program overall faced challenges in lower-than-anticipated buy-in from missions. In the end, however, the GBI Alliance was recognized as having fulfilled its original goal of increasing connectivity throughout the world.  Conclusion 5: Looking forward, it will be important to continue building on initial successes, specifically NetHope’s technical expertise, engagement with the private sector, and engagement with missions.  Conclusion 6: USAID, GBI Alliance staff, and partners agreed that improved communication is the most important improvement for the future. February 2017 | GBI Alliance Performance Evaluation 7 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The Lab/CDD team catalyzes programs and partnerships in three areas: digital finance, inclusive access to digital and mobile-based information and service delivery, and the use of data collected by digital devices to improve decision making. As part of this effort, the Lab/CDD supports the GBI Alliance, which provides technical assistance to a range of engagements related to ICT to advance communication, relief, and payment systems, and immediate assessments—as required by the Ebola response effort at USAID. The Bureau for Economic Growth, Education, and Environment (E3) launched the GBI program (later renamed the “GBI Alliance” by NetHope) in 2010. In 2014, due to changes within the Agency, it was moved to the U.S. Global Development Lab. Originally the program was supposed to end in 2015, but it was extended until 2020. The Lab/CDD contracted EnCompass to conduct a performance evaluation of the GBI Alliance for the period of inception until February 2017. The aim of the evaluation was to identify accomplishments, performance issues, lessons learned, and constraints in implementation. The EnCompass evaluation team’s tasks were to:  Determine the appropriateness of the program’s original technical scope and award mechanism  Assess program performance, achievements, and challenges  Provide recommendations for future mechanisms  Compile knowledge of lessons learned and areas for improvement. EVALUATION QUESTIONS In a series of meetings in September 2016, EnCompass worked with the Lab/CDD and NetHope to craft a list of evaluation questions ( February 2017 | GBI Alliance Performance Evaluation 8 Exhibit 1) that would provide the most useful, relevant data for future planning for USAID. February 2017 | GBI Alliance Performance Evaluation 9 Exhibit 1: Evaluation Questions Category Questions 1. Appropriateness of the program’s original technical scope and award mechanism a) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for the GBI Alliance at the time of inception? b) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? c) Has the prime/sub-partner structure originally envisioned to implement the GBI Alliance activities been effective? d) Has USAID provided adequate guidance to the GBI Alliance regarding desired outcomes? Of the mechanism overall? Of particular projects? e) Program performance, achievements, challenges, and lessons learned a) How has the prime partner worked with project sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? b) What have been the actions taken (by the project) to mitigate risks and course-correct during project design and implementation? c) How satisfied have various stakeholders (e.g., USAID, sub-partners) been with the work done by and assistance from the GBI Alliance? d) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate, and timely? e) What have been the program’s achievements and challenges? f) Recommendations for future mechanisms a) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from the GBI Alliance need to be continued and what anticipated services are likely to be needed? b) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? February 2017 | GBI Alliance Performance Evaluation 10 PROGRAM BACKGROUND INCEPTION USAID granted the GBI Alliance as a cooperative agreement to NetHope in 2010. The GBI Alliance was conceived to achieve two goals:  Extend the reach of broadband, including enhanced mobile networks, into more remote rural areas as key infrastructure for extending the reach of socioeconomic services through USAID’s NGO and contractor partners.  Leverage the extension of these networks for delivering network-enabled value-added application support across USAID’s development portfolio. The goal of the GBI Alliance was to support expansion and use of broadband Internet (both mobile and fixed) to provide rural connectivity in the developing world while expanding the adoption and deployment of innovative, value-added, scalable ICT solutions. It was hoped that the GBI Alliance would not only leverage and scale proven solutions from pilots, but also work with USAID and its missions to build their capacity for public-private partnerships in ICT. Another hope was that the alliance would identify leverage points with the USAID overall strategy for incorporation of ICT with its programs and country strategies. This would be achieved through technical assistance, convening working groups, knowledge resources, and program pilots in USAID projects (USAID 2010). The total estimated amount of the agreement was $15,096,467. The agreement envisioned that NetHope would work with USAID to identify and secure resources—both cash and in-kind—from external partners at a ratio of at least 2:1. In 2014, after 3 years of successful implementation, the Alliance’s ceiling was increased; it was increased again in 2016. On Sept. 26, 2014, NetHope received a 5-year extension (to September 2020) and a $5 million ceiling increase. In April 2016, it received a second increase, to $25,056,366, to help with post-Ebola work in Liberia, Sierra Leone, and Guinea (USAID 2010). The Alliance was structured under a cooperative agreement to allow USAID to tap the expertise of NetHope’s consortium partners. NetHope facilitates dialogue, debate, and joint projects among its 50 leading international NGO members and public-private partners to find ways to best apply technology for connectivity in developing countries. It was envisioned that the GBI Alliance leadership would collaborate with NetHope’s members and USAID, sharing ICT knowledge, solving common problems through shared ICT-related solutions, and implement ICT value-add solutions to extend the impact, reach, and scale of USAID programs (USAID 2010). At inception, Alliance activities were split into two broad categories: access and applications (NetHope 2012a): February 2017 | GBI Alliance Performance Evaluation 11  Access: bringing affordable, reliable, competitively serviced broadband (voice and data) to rural areas in USAID priority countries  Applications: capitalizing on efficiencies introduced through technology to achieve USAID’s core objectives in health, democracy and governance, economic growth, gender equality, and new approaches to achieving development through mobile technology (e.g., mobile money) Projects initially focuses on education, health, geographic information systems, emergency preparedness, and rural connectivity. An additional project was proposed for gaming. A theory of change was developed for each project, but not for the Alliance overall. EVOLUTION NetHope had start-up challenges, including having to terminate a chief of party in mid-2012. A staff member served as interim chief of party for 5 months until Jonathan Metzger was hired in late 2012 (NetHope 2012a). The Alliance acknowledged this as a setback during the first 2 years of the mechanism. After Chief of Party Metzger was hired, the Alliance drafted a workplan that articulated its strategic objectives (see Exhibit 2). Exhibit 2: GBI Alliance Strategic Objectives Objective Description Strategic Objective 1 Piloting for Scale: The Alliance will leverage NetHope’s Life Cycle piloting methodology for identifying, developing, and deploying innovative ICT-enabled solutions for use with the global USAID development portfolio. Strategic Objective 2 Promoting Innovation: Leverage NetHope’s existing and future partnerships with private sector ICT companies to facilitate partnerships with USAID that enable the deployment of value-added ICT solution sets across USAID’s development portfolio. Strategic Objective 3 Driving Implementation: Leverage the ICT experience of NetHope’s member organizations across USAID’s portfolio to improve efficiency and effectiveness of field operations via technology support solutions that significantly lower costs and improve organizational effectiveness. Strategic Objective 4 Reuse-Replication-Refinement: The GBI Alliance will facilitate more cost-effective and efficient leveraging of ICT methods, tools, and partnerships through systematic introduction of commonly-shared applications and knowledge of best practices to the broader international development community. Strategic Objective 5 Multisectoral Alliances: Leverage financial contributions to NetHope from privately funded organizations for the GBI Alliance Source: NetHope (2012b) February 2017 | GBI Alliance Performance Evaluation 12 Exhibit 3 shows examples of GBI Alliance projects that targeted access and applications around the globe. Exhibit 3: NetHope Projects Access Application  Indonesia Rural Broadband Pilot  Jamaica TV White Space Project  Kenya Dadaab Connectivity Project  Kenya Mawingu Laikipia County TVWS and Solar Deployment  Lebanon Rural Broadband  Ebola Emergency Response Activities  Africa Broadband Partnership  Combating the Trafficking in Persons Program  e-MITRA  Innovations for Youth Capacity and Engagement  Payment Innovations  NetHope Solutions Center  Women and the Web Alliance Each project has its own M&E framework. In 2011, the Alliance developed a framework to track three aggregated performance indicators across the portfolio: funds leveraged; number of men and women receiving improved ICT services; and person hours completed of training in communications services (NetHope 2012b). For indicator 1, NetHope began reporting in 2013, providing the data shown in Exhibit 4. Exhibit 4: Data for Indicator 1, Amount Leveraged by the GBI Alliance 2013 2014 2015 2016 $13,297,834 $28,117,130 $14,661,842 $494,7341 NetHope provided data for indicator 2, “number of men and women receiving improved ICT services,” in 2014 only, showing that 3,400 directly received connectivity and the lives of more than 520,000 were improved through broadband enhancements to essential services (NetHope 2013). In 2015, NetHope said it was working on a new M&E framework to capture data for this indicator, but it was still not defined in 2016 (NetHope 2014). Although NetHope reported on Indicator 3, “person hours completed of training in communications services,” in Year 3, it stopped data collection in 2014. In that year’s Annual Report, NetHope stated, “as the primary goal of the GBI program is to leverage private sector investment and innovation to achieve development impact, the training indicator was no longer deemed a valid measure of GBI performance” (NetHope 2012b). 1 The Year 6 Annual Report noted that this number might raise to $4,197,734 pending private sector funds from the Overseas Private Investment Corporation. February 2017 | GBI Alliance Performance Evaluation 13 EVALUATION METHODS AND LIMITATIONS METHODS On October 10, 2016, EnCompass met with USAID staff to discuss the evaluation design and plan. USAID stressed the importance of acknowledging that this was a performance evaluation, not an impact evaluation, the implication being that EnCompass should not focus on what the Alliance achieved per se, but how it functioned and the related success factors, challenges, and lessons learned to inform the design of a follow-on mechanism. Given the evaluation’s focus and the limited resources available, EnCompass and USAID determined that interviews with key informants and an accompanying document review were the best data collection methods to obtain the information needed to understand the GBI Alliance’s performance. The purpose of the document review was to understand the evolution of the Alliance; to this end, the team focused on documents that provided information about the Alliance as whole, not individual projects. Although the team discussed conducting a survey, it determined that semi-structured interviews with USAID mission and headquarters staff, NetHope management staff, and Alliance partner organizations would yield more in-depth information. Interviewees were sampled purposively and in concert with the evaluation reference group, which comprised USAID and NetHope staff. Respondents were interviewed from four key stakeholder groups: USAID/Washington, USAID missions, Alliance staff, and partners who worked with NetHope on Alliance projects. USAID determined that emphasizing USAID/Washington and NetHope staff perspectives was appropriate, given limited resources and because interviewees from these organizations were most likely to have insights into the evaluation questions. EnCompass designed structured interview guides employing an appreciative inquiry approach to explore exceptional experiences, values, successes, challenges, and wishes for the future of the GBI Alliance. Draft tools were submitted to the reference group to ensure that suggestions were incorporated before data collection began. The evaluation team conducted 30 interviews. Exhibit 5 shows the demographic breakdown. February 2017 | GBI Alliance Performance Evaluation 14 Exhibit 5: Evaluation Interviewees by Stakeholder Group and Sex (N=30) Annex 2 provides more background on the determination of sample size and successes and limitations of methods. Annex 3 provides the structured interview tools used for each stakeholder. Annex 4 provides the list of documents reviewed. LIMITATIONS Although the evaluation approach was deemed appropriate given time and funding constraints, the team recognized the following limitations:  A lack of quantitative data to triangulate with qualitative data and produce more robust findings: o Quantitative analysis of performance indicators was not possible due to the lack of a program-level theory of change, M&E framework, and related monitoring data. o The three performance indicators could not be analyzed because they underwent various modifications and data collection was inconsistent during implementation. The performance indicators were not clearly defined, and primarily measured output-level performance.  The limited size and scope of the evaluation: o The limited budget meant EnCompass could not examine the performance of individual projects. o Although attempts were made to reach different stakeholders and to include as many perspectives as possible, these groups were small, especially at partner organizations and USAID missions. This should be kept in mind when interpreting evaluation findings.  Turnover at NetHope and the rate at which USAID staff rotate in and out of departments made it difficult to find interviewees who could speak to the full program lifecycle, what it had accomplished, and how it functioned. February 2017 | GBI Alliance Performance Evaluation 15 FINDINGS APPROPRIATENESS OF AWARD MECHANISM AND TECHNICAL SCOPE When the GBI Alliance was granted as a cooperative agreement by USAID to NetHope in 2010. USAID often uses cooperative agreements to tap the expertise of consortium partners. For service providers, agreements are structured to allow organizations to work with USAID as thought partners and to co￾create programs with greater flexibility. The cooperative agreement mechanism, however, has implications for how USAID can access sub-partner services and how the overall mechanism is managed. This section explores the ways in which the structure and technical scope of the cooperative agreement mechanism enabled USAID to achieve its desired outcomes. It explores the mechanism’s strengths and weaknesses, how the original prime/sub-partner relationship was utilized, and how USAID/Washington managed the agreement. AWARD MECHANISM AND TECHNICAL SCOPE The structure of a cooperative agreement was appropriate for the GBI Alliance’s technical scope because it allowed for collaboration and innovation to achieve desired outcomes. USAID/Washington staff, the only respondents with the institutional knowledge to comment on the appropriateness of the award, reported that the cooperative agreement structure was appropriate for the GBI Alliance. Respondents reported that it allowed for greater collaboration, innovation, and flexibility than other mechanisms, such as contracts. USAID staff explained that because ICT4D is a more nascent field for the Agency, they wanted a mechanism that would engage implementing partners as thought partners and collaborators, which is what transpired with GBI Alliance. Staff also stated that because the technology sector changes more rapidly than other sectors, more flexibility to change scopes of work of various projects throughout the agreement was needed. Although some staff noted that contracts were traditionally easier to manage than cooperative agreements, they viewed NetHope as a very cooperative partner that made the agreement easy to manage. A few USAID staff stressed the importance of recognizing that the type of mechanism was not as important as how the mechanism was managed within USAID. It enabled both sides to give input on the work. There was healthy exchange on the work and feedback. —USAID/Washington I don’t think it could have been a contract. We needed involvement where we were collaboratively working together as a partner on these activities. —USAID/Washington I like contracts personally because you can be more involved, but I think NetHope has been very cooperative. —USAID/Washington February 2017 | GBI Alliance Performance Evaluation 16 GBI Alliance staff appreciated the flexible nature of the cooperative agreement but anticipated a greater degree of collaboration and innovation with USAID. Alliance staff said they appreciated the flexible nature of the cooperative agreement, which allowed the program to adapt scopes of work based on the most appropriate technology and information available. Respondents noted that a contract would have made their project less innovative and able to adapt to new information coming from the technology sector. I think [the cooperative agreement] is appropriate in technology-driven development because it might be shifting so often as the technology shifts—so you need an agile instrument in order to change. —GBI Alliance However, Alliance staff also noted that they envisioned a collaborative relationship in which they could have substantial influence over the program. They said they were not engaged as equal thought partners with USAID staff, and that USAID occasionally mandated changes in scopes of work or consultants without their input, resulting in inefficiencies and missed opportunities for innovation. I think it was the right mechanism. I don’t think it was used the right way. As I understand, cooperative agreements are supposed to be just that, cooperative. —GBI Alliance We didn’t have the ability or will to say “no,” so we weren’t able to deliver as much as we could have. —GBI Alliance Alliance staff mentioned that USAID was at times overly directive in the selection of sub-partners and projects, noting that selections were sometimes made to pursue projects because a sub-partner had approached USAID or was already working in a country where the Alliance was also working. Such directives, although well-intentioned, not only raised questions about the appropriate selection of projects and partners—it seemed to be contrary to the collaborative nature of the cooperative agreement. It is often suggested who the partner might be … usually somebody decides they want to do a project with Microsoft in a particular country … or Microsoft might approach someone in the [USAID] office. —GBI Alliance CONSORTIUM The GBI Alliance effectively utilized technical experts and the private sector. However, consortium members were not fully utilized to the extent originally envisioned. USAID staff reported that one of the main reasons for choosing NetHope as the prime partner was its unique structure. NetHope is a consortium of more 50 NGOs and public-private partners, a model that encourages cross-sector collaboration among these partners to deliver information technology solutions to the developing world. USAID staff reported that NetHope provided an opportunity for USAID to utilize a large pool of partners by going through one agency. Specifically, staff mentioned a unique opportunity to work with private sector partners, who have traditionally been more difficult to engage. February 2017 | GBI Alliance Performance Evaluation 17 USAID staff said NetHope was effective at engaging key private sector partners, such as Microsoft and Intel, as was originally envisioned. NetHope also used contractors who were technical experts and provided a high degree of utility to GBI Alliance projects. On occasion, NetHope also used Integra LLC, which carried out a separate contract as a sub-partner to implement projects in Universal Service Funds and TV White Space. (See box, right, for more information.) This was a way for USAID to fund one implementer and get an NGO participating, multiple NGOs, multiple private sector partners, so [with] one instrument you can get to the source of all those partners. —USAID/Washington Here was an international NGO that had at that time in the 30ish international members, focused only on ICT, focused on 170 countries, and total budgets of $50 million. We could go to a cooperative agreement with one organization and could leverage a footprint. —USAID/Washington However, USAID/Washington and mission staff also reported a missed opportunity in utilizing the broad array of member organizations in NetHope’s consortium, as was originally envisioned. Respondents pointed out that the core NetHope staff and a small group of technical experts, not a broader group of consortium members, were used most often in GBI Alliance projects. Specifically, staff mentioned more limited engagement with private sector partners than was originally envisioned. Respondents also reported missed opportunities to collaborate with NetHope consortium members who were working in the same countries as GBI Alliance projects. I think the Alliance is great in that they have public and private sector members buying into it. I thought we were getting access to public-private partners. The Alliance itself did not bring a whole a lot to USAID besides NetHope staff. It was liked a regular project where you work with one prime. —USAID mission In the end, part of the reason for going with GBI was to have access to the consortium. In the end, we had our own array of experts but they were not a part of NetHope. In the end, I felt like we did not benefit from the broader NetHope network, either from NGOs or [the] private sector. In the end, really what made it possible was having good individuals. —USAID/Washington USAID staff noted this gap has been discussed with NetHope and that there is a concerted effort underway to broaden the members engaged in the next phase of the program. Integra LLC, an international development firm with expertise in ICT4D, held a separate contract with USAID under the GBI program. It focused on projects to better utilize and leverage Universal Service Funds. Integra also served as a subcontractor to NetHope on a project on Universal Service Funds in Kenya, Tanzania, Nigeria, and Ghana, and a second project on TV White Space in Indonesia. An independent performance evaluation of this project in 2014 reported overall high levels of success in each of the four countries where Integra worked. February 2017 | GBI Alliance Performance Evaluation 18 USAID SUPPORT FOR AWARD MECHANISM While USAID/Washington supported the GBI Alliance with increased funding and quick approval processes, it struggled to provide clear communication regarding reporting and management of the program overall. When asked about support provided to the GBI Alliance, USAID/Washington staff commented on the funding provided by the Agency, which increased twice during the project life cycle. Staff also pointed out support in encouraging and attaining buy-in from other USAID programs. Alliance staff reported that they appreciated the quick approvals from their counterparts at USAID. However, they also noted challenges in clear lines of communication and reporting. They spoke of shifting of the focus of projects, multiple USAID counterparts with whom they were told to report, and last-minute requests for deliverables from USAID/Washington. Alliance staff speculated that the changes during the program may have been due to changes in leadership at USAID. Alliance staff and partners expand on these issues when discussing challenges in their ability to provide timely and complete financial reporting. I don’t think we are a terribly supported effort across the board. There is a lot of random asking for things. Better support would be a time frame of when you want certain types of data and when. —GBI Alliance When you have 6-7 people at any given time that are communicating … different versions of communication … information tends to be compartmentalized and doesn’t always flow in a transparent way ... It can make it a little muddled sometimes in terms of what it is that we are trying to do. —GBI Alliance PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED As defined by USAID, a performance evaluation is designed to identify accomplishments, performance issues, and constraints in the implementation of a project. This section explores USAID/Washington staff and partners’ thoughts about NetHope’s performance and management, as well as achievements and challenges implementing the program. PRIME-SUB PARTNER MANAGEMENT While NetHope effectively managed the Alliance partners, communication and lines of reporting among partners, GBI Alliance, and USAID/Washington staff could have improved. The GBI Alliance collaborated with private companies, public institutions, NGOs, and independent consultants to serve as thought leaders and to provide additional cash and in-kind support for Alliance activities. February 2017 | GBI Alliance Performance Evaluation 19 Most partners interviewed for this evaluation were independent consultants and private sector companies working with the GBI Alliance. They reported that NetHope was easy to work with, and had positive experiences in communication, contracting, and overall management. Communication practices included weekly meetings, regular phone check-ins, and personal communication between NetHope staff and partners. Respondents said management practices improved after the new chief of party was brought on in late 2012, opening lines of communication and collaboration between partners and Alliance staff. In addition, many of the GBI Alliance projects were designed to include NetHope staff working on a team with partners, and communication was reported to flow easily and transparently between individuals from both sides within a project. We met on a weekly basis and so forth, and that was helpful for us to understand what they were doing. —Partner Partners reported that sometimes it was unclear with which NetHope and USAID/Washington colleagues they should communicate. At times, partners did not know whether report requests to NetHope, USAID, or both. This lack of clarity sometimes meant partners had to communicate across the two entities, and other situations where they reported having multiple leaders on a single project. Part of my role was to be a bridge between the two organizations to ensure there was communication. —Partner My time was paid for by NetHope, but I was effectively answering to the Global Development Lab in terms of any operational work, so that is the part I found confusing. I kind of had two masters there in a way. But to NetHope’s credit, they were very flexible and it took a little bit of additional communication to manage to ensure everyone was in the loop. —Partner Due to NetHope’s experience in the ICT4D sector, many partners had worked with the organization or its staff, making the process of finding and awarding subcontracts much easier than some traditional prime contractor-subcontractor partnerships. Both Alliance staff and partners reported no difficulty in the awarding and contract process. USAID requested limited information regarding project performance and the alliance had limited mechanisms in place to monitor performance of the program overall. The GBI Alliance did not have an overall program-level theory of change or M&E framework. Some projects had their own theories of change with indicators related to performance.2 However, annual and semi-annual reporting to USAID contained updates only on specific project outputs, and aggregate data on three performance indicators (funds leveraged; number of men and women receiving improved ICT services; and person hours completed of training in communication services) varied. Only the first 2 Project documents sent by NetHope staff provided evidence of project-level theories of change but limited evidence of measuring performance against these frameworks. February 2017 | GBI Alliance Performance Evaluation 20 indicator was tracked consistently. Longer-term results, measured by outcome or impact level indicators, were minimal. GBI Alliance staff and partners confirmed that M&E requirements from USAID for the program overall were minimal. The Alliance relied on its own monthly reports, weekly meetings, and review of deliverables to monitor partner performance. However, partners reported appreciating this informal monitoring, which they said allowed them more flexibility to accomplish their work. We don’t really [have mechanisms in place to monitor partner performance]. I think we use the same people again and again and USAID seems happy with the work so their project is somewhat successful and they are contractors we are going to keep working with. —GBI Alliance [Controls to monitor my performance were] pretty light-handed. [NetHope] wanted to receive the information but it was pretty light … We worked really well together, so if [they] had a question, [NetHope] would call me up and so forth, but we pretty much were given flexibility to do what we needed to do. —Partner SATISFACTION WITH GBI ALLIANCE PERFORMANCE Overall, USAID/Washington and mission staff were very satisfied with the overall performance of the GBI Alliance. USAID/Washington staff commented that they were very satisfied with the GBI Alliance for multiple reasons. They noted that the Alliance played a key role as a thought partner and someone they could rely on to sit alongside on working groups, as well as provide their technical expertise on materials in a timely manner. USAID mission staff also said it was easy to work with the Alliance, noting their satisfaction with the level of technical assistance provided to their projects. USAID/Washington staff pointed to their satisfaction with the Alliance’s ability to leverage private sector partnerships. In total, the Alliance leveraged more than $56 million in support over the life of the program. 3 Support has come from both private and public entities, including companies such as Microsoft, VISA, and Intel, and public sector entities such as host-country government ministries and USAID missions. They have been very responsive—I can say they are part of working groups or meetings and generally have been very responsive and very easy to work with. —USAID/Washington I think they have accomplished quite a bit in terms of leveraging private sector investment in broadband infrastructure and build-out and getting policy reforms. —USAID/Washington It was very easy to access the mechanism … it gave us strong possibilities to bridge the infrastructure gap. —USAID mission 3 GBI Alliance Year 6 Annual Report. February 2017 | GBI Alliance Performance Evaluation 21 USAID/Washington staff reported that the Alliance played a key role in liaising between the NGO community, USAID missions, and the private sector. PROJECT ACHIEVEMENTS The Alliance’s main achievements included their projects focused on increasing broadband connectivity sustainability, as well as increasing attention to connectivity issues within and outside USAID, and increasing USAID’s presence in the ICT4D space overall. All stakeholders commented positively on the Alliance’s achievements, noting that the main achievement was the successful expansion of broadband connectivity to people throughout the world. Respondents highlighted projects such as DadaabNet in Kenya (see box on the next page), a Universal Service Funds project in Indonesia, a TV White Space project in Jamaica, and post-Ebola work in Liberia, all of which have resulted in hundreds of thousands of people getting access to broadband connectivity. We did a build-out of connectivity in the worlds’ largest refugee camp in Northwest Kenya called Dadaab. We had corporate partners who were interested in this work and we were able to marry USAID with tech sector partners and deliver dramatically improved connectivity to refugees in Kenya. Here we are 5-6 years later and it is a self-sustaining service operated by Kenya’s enterprises. —GBI Alliance The Indonesia Broadband Plan gave the architecture of where [Universal Service Funds] should go—and not just Universal Service Funds, but funds out of other parts of government. The funds were frozen for 3-4 years and now they have been released. I think the release of $500 million in Indonesia alone. The Universal Service Funds collects on $150 million a year. If you look at [the] entire USAID budget of Indonesia Mission, it is less than that annually. —GBI Alliance USAID/Washington staff also reported that a key achievement of the GBI Alliance was that its broader consortium brought the discussion around connectivity to the fore and expanded USAID’s role in ICT4D more broadly. The Alliance served as thought partners with USAID staff and increased technical expertise inside and outside of the Agency. A project that demonstrates this new role for USAID is the Better than Cash Alliance, which works toward a more transparent, efficient, inclusive, cash-light society. Now housed in with the United Nations, the alliance is a global public-private partnership dedicated to accelerating the use of electronic payments in place of physical cash. The GBI Alliance partnered with the Bill & Melinda Gates Foundation, Citi, the Ford Foundation, Omidyar Network, VISA, and the UN Capital Development Fund to create the Better than Cash Alliance. February 2017 | GBI Alliance Performance Evaluation 22 [The] main achievement [was] getting connectivity discussion out on the table. —USAID/Washington They [GBI Alliance] have become recognized in this community beyond USAID with experts that work on this in private foundations, World Bank, and different global alliances and that is valuable for us. They are force multipliers. —USAID/Washington They have been a lot of the technical expertise and better at being partners and sitting around the table and participating in working groups, but they don’t have the logistics support. —USAID/Washington USAID/Washington staff pointed to GBI Alliance’s leadership as especially influential in expanding digital financial services in the humanitarian sector. Staff reported that the ability to bridge the gap between traditional humanitarian sector actors and actors working in ICT4D was important and effective. They have become champions and thought leaders on [the] issue of how we can ensure that women are included [in] digital finance services. Humanitarian services work, trying to bridge the worlds of humanitarian aid and financial inclusion. —USAID/Washington GBI Alliance staff and partners said making projects sustainable was a key achievement. Sustainability was made possible by bringing connectivity into the policy space in the countries where they worked, as well as leveraging funds from the public and private sectors. Aside from the $56 million the project was able to leverage from the private and public sectors, the Alliance was also successful in transferring project DadaabNet In 2012, the GBI Alliance launched a project in Dadaab Refugee camp. One of the largest refugee camps in the world, Dadaab hosts more than 200,000 people fleeing famine, drought, and civil war. NetHope collaborated with Cisco, Microsoft, and Inveneo to implement a large-scale, high￾speed broadband network. DadaabNet connects refugees to each other and the outside world. Its far-reaching results included:  Became the established tool for communication among the 43 NGO agencies in the camp  Saved hundreds of thousands of dollars per year in more efficient Internet technology  Created local jobs related to the ownership, support, and development of broadband  Increased education opportunities, including five community centers that now offer online programs for high school and college diplomas  Prepared refugees for the future, including vocational and life skills training to ease transition from the camp  Transitioned ownership of DadaabNet to a Kenyan company, which now runs the network. February 2017 | GBI Alliance Performance Evaluation 23 ownership to host-country governments. In the most recent annual report, Jamaica and Indonesia broadband projects were fully transitioned from the GBI Alliance to their respective ministries.4 Good work together on policy reform, good thought leadership on role of ICTs can play in international development. —GBI Alliance Policy space—the perspective of having impact in country—with Ministries and regulators, whether on universal access policy or spectrum policy, we are starting to see GBI has done more to universal access and broadband planning, and in the last year they have gotten more involved in spectrum policy and I would like to see that continue. —Partner PROJECT CHALLENGES NetHope experienced challenges in their ability to provide timely and complete financial reporting. As a virtual organization, NetHope has fewer operations staff and lower overhead than most subcontractors.5 Although USAID staff identified lower overhead as a benefit, it also resulted in a key challenge throughout the program: limited capacity for detailed financial accountability. USAID/Washington staff reported difficulty getting detailed financial reports from NetHope that would enable them to trace project funds to each activity. They also reported challenges in getting the level of detailed reporting required for USAID auditing requirements. Although this reportedly improved slightly during implementation, USAID staff acknowledged it remains a challenge because NetHope only has one staff member dedicated to financial accountability. I think internal management and reporting, budget and activity reporting, could have been improved. —USAID/Washington [Another] big challenge was administrative. Struggled to produce clear budgets. We have made very clear our frustration with that, but I don’t think NetHope is well positioned to manage that better. They have one main financial person for all of their work, so there are still concerns there. —USAID/Washington Changing leadership and the resulting shifting priorities within USAID challenged alliance implementation. Since inception of the GBI Alliance, there has been a lot of USAID staff turnover, as well as movement of the program from one office to another within the Agency. These changes resulted in differing priorities depending on who was in charge of the program at the time and what the priorities within the Agency were. Alliance staff and partners reported that the changes at the top of the Agency disrupted implementation. Both Alliance staff and partners reported that projects were stopped at the last minute 4 GBI Alliance Year 6 Annual Report, November 2016. 5 GBI Alliance NICRA started at 13.97% and increased to 26.4% according to the Year 6 Annual Report. February 2017 | GBI Alliance Performance Evaluation 24 due to changing priorities. Respondents reported this caused delays in implementation and wasted resources. It does seem that at times the change in leadership inside of the Lab and perhaps in other parts of USAID have created an environment where we are start/stop, start/stop, so I am not sure we have always been running at full steam towards a consistent vision, in part because of the leadership changes that have taken place. —GBI Alliance Sometimes during a project things would stop and we were told this was being re-thought and there is fresh thinking on that so we need to hold on and maybe not go forward, we will see. Sometimes we were about to go somewhere and then told “no,” the top-level decision is not supporting that. —Partner The lack of marketing for the GBI Alliance and awareness of the value of ICT4D throughout the Agency resulted in lower-than-expected buy-in from USAID missions. All informants acknowledged the missed opportunity of not engaging missions from project design through implementation. ICT4D is a newer sector for USAID (i.e., compared to more traditional sectors such as health, education, and agriculture) and is less known at missions. Because of the nascent nature of this crosscutting sector and the lack of marketing, many missions reported not knowing the initiative existed. There is very little buy-in from missions, and that is from a lack of advertising about the program and a lack of experienced USAID personnel to go out and convince USAID missions and unwillingness to let partner[s] go out and generate buy-in from missions in an undirected manner. —GBI Alliance [The] biggest challenge is trying to sell the missions and agency to understand how development can have a greater impact in their sectors by having greater connectivity in the rural areas where USAID works. —USAID/Washington FEEDBACK ON LESSONS LEARNED FOR FUTURE MECHANISMS This section reports on feedback from USAID/Washington, missions, the GBI Alliance, and partners on what they would keep, enhance, and add to future mechanisms. It also addresses lessons learned about project management and implementation. The GBI Alliance’s technical expertise should continue to be utilized and expanded throughout the Agency, taking a more systems-level approach to ICT4D and expanding to engage more offices throughout the Agency. All informants agreed that NetHope’s technical expertise is valuable and should continue to be utilized. USAID staff spoke of engaging the GBI Alliance within working groups, at international conferences, and as thought partners throughout the Agency. Alliance staff and partners also reported a need to add a more systems-level approach to ICT4D within the Agency, including expanding into more technical February 2017 | GBI Alliance Performance Evaluation 25 sectors. Staff reported an opportunity to build on their success in the humanitarian and disaster response sectors by broadening their scope to additional USAID sectors (e.g., health, education, and agriculture). I think just providing technical support … They do a lot of the legwork on the technical stuff that we can’t do, so going into missions and going into detail with partners, writing toolkits, etc. —USAID/Washington Also because it is a membership [organization] we have the ability to tap into the members. For example, I go to NetHope annual summit and we can present on the work we are doing. It brings value beyond a single organization. —GBI Alliance You just need to look at the disaster response space to see that NetHope has become a lead across the humanitarian community, in terms of responding to crisis. If we could bring that same paradigm [to] where it isn’t an emergency. —GBI Alliance The GBI Alliance should continue to expand its engagement with private sector partners, including leveraging more resources. All parties recognized the value in NetHope’s consortium, both in the broad array of actors engaged and the technical expertise they can provide to USAID. Interviewees noted the need to continue to build on their engagement with the private sector and increase the amount of funds leveraged, thus enhancing sustainability in the future. USAID/Washington staff said the GBI Alliance was well-positioned to be an objective partner in ensuring that private sector incentives are aligned with those of USAID missions and USAID more broadly. I think that for them to organize proof of concept for private sector, expand and using the network of private sector. —GBI Alliance I think overall the public/private partnerships and being willing to partner with the private sector to achieve development goals. There is a way to find alignment between goals private companies and goals public sector are trying to achieve —Partner Someone who is objective and can say [to USAID] this is the most cost effective technology OR this might be a great idea, but the other two technologies might be just as cost-efficient in that type of environment. —USAID/Washington USAID should better market the GBI Alliance to missions and engage more directly with missions at the start of projects. All informants identified a need to engage missions up front in the design of projects, in identifying local partners, and in project implementation. Really linking at the missions, showing cost, showing value, which means there has to be more people on the job doing that and not just two or three with access in the past. You have to spend money. —USAID/Washington February 2017 | GBI Alliance Performance Evaluation 26 I don’t have a great understanding of everything the Alliance offers. The missions don’t necessarily know what the Alliance is. There are a limited number of missions that are aware of Lab’s services. —USAID mission LESSONS LEARNED IN THE MANAGEMENT AND STRUCTURE OF THE GBI ALLIANCE USAID staff identified a need to improve its staff’s capacity to serve as activity managers, as well as improve systems for communication, oversight, and accountability, specifically in regard to M&E. USAID/Washington staff reported that lessons learned from this experience revolve around better management and communication inside the Agency and between the Agency and the implementing partners. Staff recognized the unique make-up of the Lab and lack of management experience by many technical experts in the Lab. Staff also reported a need to build in stronger systems for financial accountability from the implementing partners and M&E of program outputs and outcomes. USAID/Washington staff also reflected on the challenges in transferring this project between multiple offices (i.e., the Bureau for Economic Growth, Education, and Environment and the Lab) and the implications it had on fragmented communication and management challenges. We need to rethink our systems of oversight and financial visibility. (USAID/Washington) First, [we] need a management team within USAID for the mechanism and the management team needs to view it as their role to advertise the mechanism and be the customer service hub for the mechanism. —USAID/Washington GBI Alliance staff and partners suggested a need for clearer objectives and expectations for the program overall and specific details on roles and responsibilities throughout each project. GBI Alliance staff’s suggestions aligned with USAID staff in terms of better management and communication. Alliance staff suggested investing early in convening with USAID/Washington to set clear objectives and expectations for the program overall. Throughout a project, they suggested having clearer lines of communication for reporting and management. Clarifying the systematic part of the relationship. Establishing [standard operating procedures]—we have made it work and building the airplane as we fly but systematizing the process and levels of communication … Getting some additional clarifications of what the expectations are would be important. —GBI Alliance Spending more time up front in identifying what those common objectives will be and how we will measure and reach them … I would argue the investment you make in front in terms of being VERY specific of what we are measuring, how we will get there, who will we report to … often identifies the differences of expectations or opinions among the folks sitting around the table. —GBI Alliance February 2017 | GBI Alliance Performance Evaluation 27 A clear set of objectives and priorities that translate into specific milestones and outcomes that are to be achieved and focus of responsibility on different people, different experts and what their role is to be, and clear lines of communication. Everyone getting on the same page and whose job is to do what and when and who is in charge, clearly self-evident. —Partner USAID mission staff agreed with USAID/Washington in suggesting better advertisement of and engagement with missions about initiatives coming from the Lab. Although USAID missions reported positive engagement with GBI Alliance projects overall, they also recommended a few improvements. They suggested better communication with the missions that projects such as the GBI Alliance exist. Once the missions did know about the projects, many reported advertising it to their colleagues in other missions. Missions also suggested a need to engage as early as possible in order to ensure projects are appropriate and align with country priorities. I think its intent is very admirable, and I have promoted it to other units and introduced them so just to say that its intent is good and I would like to see it continued they just need to think strategically about how best to support whatever countries moving forward … they need someone in the mission who can be their advocate and cheerleader and use it as part of their core responsibly. —USAID mission February 2017 | GBI Alliance Performance Evaluation 28 CONCLUSIONS Although there are opportunities to improve collaboration and innovation between USAID and NetHope, the flexibility afforded by the cooperative agreement mechanism was appropriate for the GBI Alliance. (Based on findings 1, 2, and 3) The structure of a cooperative agreement was appropriate for the GBI Alliance, as its structure allowed for collaboration and innovation between USAID and Alliance staff. However, certain expectations that existed at the time of the agreement were not fully realized. In particular, USAID expected to have greater access to the NetHope consortium, NetHope expected to have more control over the selection of projects and partners. Although USAID’s financial support for the GBI Alliance was important, systems for management and communication could be improved. (Based on findings 4 and 5) The increased funding support for the GBI Alliance was important in allowing the program to continue to build on its successes over the last 6 years. Yet, lack of clarity in lines of communication and reporting between USAID, NetHope, and partners continued to be a problem throughout the program. USAID staff did not identify clear expectations and objectives for the program overall, or provide direction on specific roles and responsibilities within it. Therefore, GBI Alliance staff and partners often did not have a clear picture of the objectives of the program overall or their role within it. USAID and partners were satisfied with NetHope’s overall performance, yet the ability to assess NetHope’s level of success was limited by a lack of M&E frameworks. (Based on findings 6 and 7) The program did not have a portfolio-level theory of change and a subsequent M&E framework, which limited the ability to map and measure the aggregate success of all projects in the GBI Alliance. The Alliance had multiple projects with M&E frameworks, but these varied in their capacity to measure long￾term success. Although partners appreciated the limited requirements for reporting, it resulted in the inability to rigorously and continuously monitor partner and project performance throughout the Alliance. NetHope and USAID faced challenges related to management capacity, and the program overall faced challenges in less-than-anticipated buy-in from missions. In the end, however the GBI Alliance was recognized as having fulfilled its original goal of increasing connectivity throughout the world. (Based on findings 7, 8, 9, 10, and 11) Overall, the Alliance was recognized for its achievements in increasing broadband connectivity across multiple projects in a sustainable way while increasing attention to USAID’s role within the ICT4D space. February 2017 | GBI Alliance Performance Evaluation 29 NetHope experienced challenges due to its limited operational capacity and its ability to provide timely and complete financial reporting. In addition, changing leadership and the resulting shifting priorities within USAID challenged Alliance implementation. The lack of marketing about the GBI Alliance and awareness of the value of ICT4D throughout the Agency also resulted in lower-than-expected buy-in from USAID missions. Looking forward, it will be important to continue building on initial successes, specifically NetHope’s technical expertise, engagement with the private sector, and engagement with missions. (Based on findings 12, 13, and 14) The GBI Alliance’s technical expertise should continue to be expanded throughout the Agency, taking a more systems-level approach to ICT4D. Such an approach will allow USAID to expand its work in ICT4 to more offices such as health, education, and agriculture. The GBI Alliance should also continue to utilize NetHope’s relationship with the private sector partners, specifically leveraging more resources for increased sustainability. NetHope’s initial work with USAID missions should also be leveraged for buy-in from other missions. USAID, GBI Alliance staff, and partners agreed that improved communication is the most important improvement for the future. (Based on findings 15, 16, and 17) In terms of program management, USAID staff identified a need to improve capacity to serve as activity managers, thus improving their ability to provide clear communication and oversight of the program. GBI Alliance staff and partners also identified a need for clear communication about the program’s objectives and expectations from the beginning of implementation. This would include clarity of the program overall, as well as specific details on roles and responsibilities throughout the program. All respondents agreed that, moving forward, there is a need for better advertisement of and engagement with missions in regard to initiatives coming from the Lab, how they could benefit missions across multiple technical areas, and how missions could provide the needed country buy-in for all projects. February 2017 | GBI Alliance Performance Evaluation 30 ANNEX 1. EVALUATION STATEMENT OF WORK As stated in the award, the performance evaluation will work to: (1) determine the appropriateness of the programs’ original technical scope and award mechanism; (2) assess program performance, achievements and challenges; (3) provide recommendations for future mechanisms; and (4) compile knowledge of lessons learned and areas for improvement. The evaluation team will interview key informants NetHope project management, NGO, implementing partner organizations, and private sector partners, and USAID managers at missions and in Washington. The team will also review key project documents such as the award, work-plans, and project reports. Information gathered from key informant interviews and reports will be analyzed and synthesized into an evaluation report which covers three main areas: NetHope’s achievements and challenges, lessons learned, and recommendations for future programming. A list of illustrative evaluation questions was included, then refined by the evaluation team, in consultation with USAID, in order to meet the evaluation’s objectives (Exhibit 6). Exhibit 6: Original and Revised Evaluation Questions Original Evaluation Question in RFP Revised Evaluation Question 1. Appropriateness of the programs’ original technical scope and award mechanism 1a. To what extent did the programs’ design influence the award mechanism? 1b. What were the pros and cons of using the award mechanism? 1c. Was the choice of mechanism appropriate for the activities’ intended outcomes? 1d. Were the “Prime” and “Sub” partners (and its operational arrangement) appropriate? 1e. Did GBI Alliance staff feel they were given adequate guidance from USAID on what to accomplish? 1a. Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for the GBI Alliance at the time of inception? 1b. What have been the pros and cons of using the cooperative agreement versus other award mechanisms? N/A (removed the original 1c and embedded in 1a) 1c. Has the prime/sub-partner structure originally envisioned to implement the GBI Alliance activities been effective? 1d. Has USAID provided adequate guidance to the GBI Alliance regarding desired outcomes? Of the mechanism overall? Of particular projects? February 2017 | GBI Alliance Performance Evaluation 31 Original Evaluation Question in RFP Revised Evaluation Question 2. Program performance, achievements, challenges, and lessons learned 2a. How did the prime partner work with project sub-partners? 2b. What were the actions taken (by the program) to mitigate risks and course-correct during project design and implementation? 2c. How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from GBI Alliance? 2d. How useful, appropriate, and timely was the programs’ performance? 2e. What were the programs’ achievements and challenges? 2a. How has the prime partner worked with project sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? 2b. What have been the actions taken (by the program) to mitigate risks and course-correct during project design and implementation? 2c. How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from GBI Alliance? 2d. Do USAID stakeholders feel the support they receive from these programs is useful, appropriate and timely? 2e. What have been the program’s’ achievements and challenges? 3. Recommendations for future mechanisms 3a. What existing gaps and future technical directions/issues would need to be addressed in a follow-on mechanism that is not currently being addressed by the GBI Alliance? 3b. What kinds of inputs are no longer needed? 3c. What additional partners need to be brought on? 3a. What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from the GBI Alliance need to be continued and what anticipated services are likely to be needed? 3b. What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? February 2017 | GBI Alliance Performance Evaluation 32 ANNEX 2. EVALUATION METHODS AND LIMITATIONS The EnCompass team’s evaluation approach honored the guidance put forth in the USAID Evaluation Policy and drew from our own experience and values, exemplified by the following:  Evidence based: EnCompass conducted an evaluation that was supported by quantitative and qualitative information that is reliable, valid and generalizable  Systems thinking: EnCompass recognized the complexity of both projects and the environment in the countries in which GBI Alliance projects were implemented and the need to incorporate this context in the evaluation design.  Utilization-focused and participatory: EnCompass actively engaged target beneficiaries and key stakeholders in articulating an evaluation frameworks that was both meaningful and feasible, and in line with USAID’s desire that the evaluation directly inform future programming decisions  Appreciative: The evaluation team recognized and build upon the strengths of key actors and others (i.e., prior evaluations, different partners, established reporting systems). Our appreciative approach discovered examples of success, analyzed best practices, solicited innovative ideas, and ensured that the evaluation was contributing toward a shared vision of success.  Integration of gender equality and social inclusion: In line with USAID’s Gender Equality and Women’s Empowerment Policy, our approach engaged a diversity of stakeholders and voices in the evaluation process. EnCompass employed structured interviews and document review as the two main data collection methods (Exhibit 7). The evaluation team, along with USAID, deemed that these methods corresponded directly to the questions being asked and would generate the highest quality and most credible evidence possible, taking into consideration time, budget and other practical considerations. Exhibit 7: Data Collection Matrix Evaluation Question Data Collection Method 1. Appropriateness of the programs’ original technical scope and award mechanism 1a. Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for the GBI Alliance at the time of inception? 1b. What have been the pros and cons of using the cooperative agreement versus other award  Document review  Interviews with USAID staff, implementing partner and sub-partners on technical scope, award mechanism, and partnership roles/management February 2017 | GBI Alliance Performance Evaluation 33 Evaluation Question Data Collection Method mechanisms? 1c. Has the prime/sub-partner structure originally envisioned to implement GBI Alliance activities been effective? 1d. Has USAID provided adequate guidance to the GBI Alliance regarding desired outcomes? Of the program overall? Of particular projects? 2. Program performance, achievements, challenges, and lessons learned 2a. How has the prime partner worked with program sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? 2b. What have been the actions taken (by the program) to mitigate risks and course-correct during project design and implementation? 2c. How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from the GBI Alliance? 2d. Do USAID stakeholders feel the support they receive from these programs is useful, appropriate and timely? 2e. What have been the program’s achievements and challenges?  Interviews with USAID staff, implementing partners and sub-partners on working arrangements and lessons learned  Document review on results 3. Recommendations for future mechanisms 3a. What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from the GBI Alliance need to be continued and what anticipated services are likely to be needed? 3b. What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms?  Interviews with USAID staff, implementing partners, and sub-partners on recommendations February 2017 | GBI Alliance Performance Evaluation 34 The sampling frame for the interviews was decided upon with USAID, keeping in mind time and funding limitations, geographic reach, and desire for as much representation as possible. While the EnCompass team wanted to provide more data on the overall performance of the alliance in quantitative terms, the lack of any common theory of change, M&E framework, and related monitoring data made that impossible. The evaluation budget did not afford the opportunity for EnCompass to look at the performance of each project on its own. The rate at which USAID staff rotate in and out of departments made it difficult to find interviewees who could speak to the full life cycle of the alliance and what it had accomplished over time and how it functioned. While EnCompass made every effort to balance the perspectives of different stakeholders, over 70 percent of the interviews were conducted with USAID/Washington and NetHope staff. This was in part due the list that was provided by USAID and NetHope. While attempts will be made to reach a variety of stakeholders and to include as many perspectives as possible, the number for each stakeholder was small; thus, care should be taken in interpreting evaluation findings. February 2017 | GBI Alliance Performance Evaluation 35 ANNEX 3. DATA COLLECTION INSTRUMENTS To be completed before the interview: Name of person interviewed: Title: Organization and unit: Interviewer: Date: INTRODUCTION Thank you very much for setting aside time to talk with me/us today. I work for EnCompass LLC, a consulting company based in the United States that has been contracted by USAID to conduct a performance evaluation of GBI Alliance. The purpose of this evaluation is to determine the appropriateness of the program’s original technical scope and award mechanism; assess program’s performance, achievements and challenges; provide recommendations for the future; and compile knowledge of lessons learned and areas for improvement. This is not an evaluation of your organization or your work. For this evaluation, we are conducting interviews with USAID in DC and in missions, GBI Alliance team members, and implementing partners. Before I/we begin, I/we want to let you know that any information or examples we discuss during this interview will not be attributed to any specific person or institution. All quotes used in the evaluation report will be attributed to a general stakeholder group, not by individual, and all identifying information will be removed. The interview will take 45 minutes to 1 hour. Before I begin, do you have any questions about this interview? February 2017 | GBI Alliance Performance Evaluation 36 GBI ALLIANCE INTERVIEW PROTOCOL – USAID/WASHINGTON A. APPROPRIATENESS OF THE PROGRAMS’ ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. What do you do at USAID? 2. What has been your involvement with GBI Alliance? When did this start? 3. How would you describe the vision of GBI Alliance when it was conceived and awarded? Has that vision changed over time? If so, how? 4. How did the choice of setting up GBI Alliance as a Cooperative Agreement come about (as opposed to a Contract or Leader with an Associate Award (LWA)? What was the rationale? 5. How has being in a Cooperative Agreement helped or hindered the achievement of GBI Alliance’s vision? Looking back, could there have been an alternative set up? 6. How did the choice of a consortium for GBI Alliance come about? What was the rationale? 7. How has that structure helped or hindered GBI Alliance’s vision? Looking back, could there have been alternatives to this structure? 8. What has been the support given by USAID over time to GBI Alliance? (Probe for monitoring requirements, vision, reporting, etc.) How could this have improved? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. How satisfied has USAID been with the work done by and assistance from GBI Alliance? (probe for utility, appropriateness, and timeliness) 2. In your view, what have been the program’s main achievements? 3. In your view, what have been the program’s main challenges? 4. Please think of a time when you were really pleased with the support you received from GBI Alliance. Tell me the story of that time. Who was involved? What happened? What enabled that interaction/project/support to be a success? C. FUTURE DIRECTIONS 1. What are the top 3-5 services/supports that GBI Alliance offers to USAID that will continue to be needed in the future? 2. If USAID were to develop a follow-on mechanism to pick up where GBI Alliance leaves off, are there any relevant services which are not currently offered by GBI Alliance but that USAID should be sure to include in future programming? 3. Overall, what lessons have been learned regarding the structure, management, and results of this program that USAID should apply in creating a future mechanism? February 2017 | GBI Alliance Performance Evaluation 37 D. CONCLUSION 1. Is there anything that you’d like to add or to discuss that we have not covered in our conversation today? 2. Do you have any questions for me? GBI ALLIANCE INTERVIEW PROTOCOL – NETHOPE TEAM A. APPROPRIATENESS OF THE PROGRAMS’ ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. What do you do at GBI Alliance? When did you start? NOTE: Only ask remaining questions in Section A to senior leaders of GBI Alliance or those that have been at the program awhile. Otherwise, go to Section B. 2. How would you describe the vision of GBI Alliance when it was conceived and awarded? Has that vision changed over time? If so, how? 3. How has being in a Cooperative Agreement helped or hindered the achievement of the GBI Alliance’s vision? Looking back, could there have been alternatives to a Cooperative Agreement? (probe for contract, associate with a leader award)? 4. How did the choice of a consortium for GBI Alliance come about? What was the rationale? 5. How has that structure helped or hindered GBI Alliance’s vision? Looking back, could there have been alternatives to this structure? 6. What has been the support given by USAID over time? (Probe for monitoring requirements, vision, reporting, etc.) How could this have improved? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. Think about a time that you were really proud of your involvement with GBI Alliance and tell me that story. What was happening? Who was involved? What made this experience a peak experience for you? 2. In your view, what have been the program’s main achievements? 3. In your view, what have been the program’s main challenges? 4. What have been the actions taken by GBI Alliance to mitigate risks and course-correct to address these challenges? 5. How does NetHope decide whether or not to engage a consortium partner or other sub￾contractor? In what circumstances does NetHope decide to do the work themselves? 6. What is the process for engaging partners? 7. What controls does NetHope have in place to monitor partner performance and manage quality? 8. What lessons has NetHope learned about how to best utilize partners/subs for the GBI Alliance work. February 2017 | GBI Alliance Performance Evaluation 38 C. FUTURE DIRECTIONS 1. Looking forward, what is the value added of GBI Alliance for USAID? What could be added that is not there already? 2. Overall, what lessons have been learned regarding the structure, management, and results of this program that USAID should apply to future mechanisms? D. CONCLUSION 1. Is there anything that you’d like to add or to discuss that we have not covered in our conversation today? 2. Do you have any questions for me? GBI ALLIANCE INTERVIEW PROTOCOL – USAID MISSION A. APPROPRIATENESS OF THE PROGRAMS’ ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. Tell me about the process of accessing GBI Alliance for the mission from USAID/Washington? (Probe: what was the impetus, how did you hear about it, was this particularly easier or harder in comparison to other mechanisms, if so, how?) B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. What service has GBI Alliance provided to your mission? (probe for how that came about, when it began, what was the activity, who implemented it). 2. How satisfied has USAID been with the work done by and assistance from GBI Alliance? (probe for ease of doing business, utility, appropriateness, and timeliness) 3. In your view, what was GBI Alliance’s main achievement? 4. What wishes might you have to improve the service/support you were provided? C. FUTURE DIRECTIONS 1. Looking forward, what types of existing support from GBI Alliance would your mission need? 2. Is there any service that GBI Alliance is not providing that you wish you could access? D. CONCLUSION 1. Is there anything that you’d like to add or to discuss that we have not covered in our conversation today? 2. Do you have any questions for me? February 2017 | GBI Alliance Performance Evaluation 39 GBI ALLIANCE INTERVIEW PROTOCOL – NETHOPE PARTNER A. APPROPRIATENESS OF THE PROGRAMS’ ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. Was your organization part of the original consortium of GBI Alliance partners? 2. How have you been notified and awarded work from GBI Alliance? What has this process been like? Has it been what you expected as a partner? 3. What have been the strengths and weaknesses of this award process? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. Please tell me a bit about a project you did with GBI Alliance. (Probe for how project was awarded, what it was, how long it lasted) 2. How did NetHope communicate with you during project implementation? (probe for methods, frequency) 3. What did NetHope expect from you in terms of performance? (probe for deliverables, reporting) 4. What technical or managerial support did you get from NetHope during project design and implementation? 5. Overall, what were the strengths of NetHope’s communication and management? 6. How could NetHope’s communication and management of your work been improved? C. FUTURE DIRECTIONS 1. Looking forward, what are 3–5 ways that you think GBI Alliance could be improved? D. CONCLUSION 1. Is there anything that you’d like to add or to discuss that we have not covered in our conversation today? 2. Do you have any questions for me? 3. Do you have any questions for me? February 2017 | GBI Alliance Performance Evaluation 40 ANNEX 4. SOURCES OF INFORMATION DOCUMENT LIST GBI Rural Models, Measurement Workbook Overview. IYCE. 2015. Innovations for Youth Capacity Engagement (IYCE) Final Report (September). NetHope. 2012a. Annual Report Year 2. Fairfax, VA: NetHope. NetHope. 2012b. Master Work Plan. Fairfax, VA: NetHope. NetHope. 2012c. Mission Measurement Amendment, D-RuralModels (July 12). Fairfax, VA: NetHope. NetHope. 2013. Annual Report Year 3. Fairfax, VA: NetHope. NetHope. 2014. Annual Report Year 4. Fairfax, VA: NetHope. NetHope. 2015. Annual Report Year 5. Fairfax, VA: NetHope. NetHope. 2016. Annual Report Year 6. Fairfax, VA: NetHope. NetHope. Dadaab Project Report. March 2014. NetHope. 2013. Tanzania Workshop Introduction: What Could Be Better than Cash? (February 5). NetHope. Theory of Change: Digital Payments Ebola Response. NetHope. Theory of Change: Portfolio Acceleration Humanitarian Assistance. NetHope. 2016. Universal Services Funds Evaluation Report (June 24). NetHope Results Framework.Women and the Web Alliance. USAID. 2010. GBI Cooperative Agreement. Washington, D.C. United States Agency for International Development. February 2017 | GBI Alliance Performance Evaluation 41 ANNEX 5. DISCLOSURE OF CONFLICTS OF INTEREST Name Beeta Tahmassebi Title Director Organization EnCompass LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract Number AID-OAA-1-15- 00021; Order Number AID-OAA￾TO-16-00031 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) GBI Alliance, NetHope I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 31, 2017 February 2017 | GBI Alliance Performance Evaluation 42 Name Elisa Knebel Title Senior Evaluation and Learning Advisor Organization EnCompass LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Contract Number AID-OAA-1-15- 00021; Order Number AID-OAA-TO￾16-00031 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) GBI Alliance, NetHope I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 31, 2017 Name Kelsey Simmons Title Evaluation Specialist Organization EnCompass LLC Evaluation Position? Team Leader Team member X X February 2017 | GBI Alliance Performance Evaluation 43 Evaluation Award Number (contract or other instrument) Contract Number AID-OAA-1-15- 00021; Order Number AID-OAA-TO￾16-00031 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) GBI Alliance, NetHope I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 31, 2017 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov