Performance Evaluation: Integrating Science, Technology, Innovation & Partnerships at USAID FINAL REPORT September 2017 Prepared by Dexis Consulting Group Submitted October 2017 The LEARN mechanism for the USAID Learning, Evaluation and Research (LER) Office in the Bureau for Policy, Planning, and Learning (PPL) produced this publication for review by the Global Development Lab (the “Lab”). LEARN is managed by Dexis Consulting Group. STIP Performance Evaluation Final Report: September 2017 | i CONTENTS ACRONYMS LIST ii EXECUTIVE SUMMARY v INTRODUCTION 1 BACKGROUND 3 EVALUATION QUESTIONS 7 METHODOLOGY 8 FINDINGS 9 CONCLUSIONS AND RECOMMENDATIONS 30 ANNEX I. Evaluation Scope of Work 35 ANNEX 2. Methodology 40 ANNEX 3.a. Data Collection and Analysis Tools: Key Informant Interview Protocol for Lab Leadership 43 ANNEX 3.c. Data Collection and Analysis Tools: Key Informant Interview Protocol for Mission Staff Involved in STIP Activities 48 ANNEX 3.d. Data Collection and Analysis Tools: Key Informant Interview Protocol for Mission Implementing Partner Organizations 52 ANNEX 4.a. Sources of Information: Evaluation Participants 55 ANNEX 4.b. Sources of Information: Documents 58 ANNEX 5. Targeted Engagement 63 ANNEX 6. Description of Missions’ Partners 71 ANNEX 7. Evaluation Team Members: Qualifications, Experience, and Role on the Team 74 ANNEX 8. Disclosures of Conflict of Interest 75 STIP Performance Evaluation Final Report: September 2017 | ii ACRONYMS LIST AAAS American Association for the Advancement of Science ADS Automated Directives System AGRHYMET Agrometeorology, Hydrology, Meteorology Institute AOR Agreement Officer ASSESS Analytical Support Services and Evaluations for Sustainable Systems ASIWA Alliance for Seed Industry in West Africa CAI Center for Agency Integration CDCS Country Development Cooperation Strategy CDD Center for Digital Development CDI Center for Development Innovation CDR Center for Development Research CILSS Comité Permanent Inter-Etats de Lutte contre la Sécheresse dans le Sahel (Permanent Interstate Committee for Drought Control in the Sahel) CITE Comprehensive Initiative on Technology Evaluation CLA Collaborating, Learning, and Adapting CoP Chief of Party COR Contract Officer CORAF Conseil Ouest et Centre Africain pour la Recherche et le Développement Agricoles (West and Central African Council for Agricultural Research and Development) CTP Center for Transformational Partnerships CVE Countering Violent Extremism DIV Development Innovation Ventures DO Development Objective DQA Data Quality Assessment DRGO Democracy, Human Rights and Governance Office E3 Bureau for Economic Growth, Education, and Environment E4D Evaluation for Development ECOWAS Economic Community of West African States EGO Economic Growth Office EIA Evaluation and Impact Assessment FP Family Planning FtF Feed the Future FSN Foreign Service National GAPP Governance, Accountability, Participation and Performance GBV Gender-based Violence GDA Global Development Alliance GII Group Information Interview GIS Geographic Information System HESN Higher Education Solutions Network STIP Performance Evaluation Final Report: September 2017 | iii HIM Health Information Management HIS Health Information Systems ICT Information and Communication Technologies IP Implementing Partner IRD Information and Research Department KII Key Informant Interview LER Learning, Evaluation and Research LOI Letter of Intent LWP Lab-wide Priority MD Mission Director M&E Monitoring and Evaluation MECap Expanding Monitoring and Evaluation Capacities MEEPP II Monitoring & Evaluation of Emergency Plan Project MEL Monitoring, Evaluating and Learning MERLIN Monitoring, Evaluation, Research, and Learning Innovations Program MSH Management Sciences for Health OEC Office of Engagement and Communications OU Operating Unit PACTE-VIH Regional HIV/AIDS Prevention and Care Project in West Africa PAD Project Appraisal Document PDO Project Development Officer PEER Partnerships for Enhanced Engagement in Research PEPFAR President’s Emergency Plan for Aids Relief PMP Performance Management Plan PO Program Office PoC Point of Contact PPL USAID Bureau of Policy, Planning, and Learning PPP Public Private Partnership RAN Resilient Africa Network RCO Regional Contracts Office RDCS Regional Development Cooperation Strategy REGO Regional Economic Growth Office RF Results Framework RFO Regional Front Office RHO Regional Health Office RM Relationship Manager RPO Regional Program Office RPGO Regional Peace and Governance Office SAFE Supporting Access to Justice, Fostering Equity and Peace SOW Statement of Work SPACES Strategic Program for Analyzing Complexity and Evaluating Systems STIP Science, Technology, Innovation, and Partnerships TRACK TB Track Tuberculosis Activity TB Tuberculosis STIP Performance Evaluation Final Report: September 2017 | iv TDY Temporary Duty USAID United States Agency for International Development WAAPP West Africa Agricultural Productivity Program WAHIT West Africa Health Informatics Team WAHO West African Health Organization WALK West Africa Learning and Knowledge Agenda YALI Young African Leaders Initiative STIP Performance Evaluation Final Report: September 2017 | v EXECUTIVE SUMMARY The United States Agency for International Development (USAID) established the Global Development Lab (“the Lab”) in April 2014 with a two-part mission to: 1) produce breakthrough development innovations by sourcing, testing, and scaling proven solutions; and 2) accelerate the transformation of the development enterprise through modern tools and approaches. Toward this effort, the Lab works across USAID and the international development community using science, technology, innovation, and partnerships (STIP) to help USAID and its partner countries accelerate the achievement of their development objectives (DOs). Evaluation Purpose To evaluate progress against its strategy, the Lab’s Center for Agency Integration (CAI) commissioned a performance evaluation through the Learning and Knowledge Management mechanism (LEARN) funded by USAID’s Bureau for Policy, Planning, and Learning (PPL). This evaluation aims to understand how, through priority engagements with the West Africa Regional and Uganda Missions, the Lab is increasing the application of STIP throughout the Program Cycle and building STIP capacity within the Agency to increase STIP integration. The Lab plans to use the findings from this evaluation to better understand its role in enabling Missions to integrate STIP, and help hone the Lab’s strategies toward achieving the second part of its mission. Evaluation Questions The evaluation is guided by one main question and related sub-questions, as follows: ● Question 1: To what extent has the Lab’s targeted engagement with the West Africa Regional and Uganda Missions, through Next Generation partnerships and Lab-wide Priorities (LWPs),1 increased their understanding of STIP, capacity to use STIP, and application of STIP throughout the Program Cycle? ● Question 2: How has STIP been integrated into the Mission at the organizational, project, activity, and strategic levels? What strategies and methods have been most effective in integrating STIP? ■ Which of the Lab’s services have the most value in integrating STIP? ■ What is the role of the enabling environment (budget, policy, resources, leadership) in integrating STIP? ● Question 3: What additional STIP services (current or new) do Missions want from the Lab? ● Question 4: How have the Missions Implementing Partners’ (IPs’) projects and activities integrated STIP? 1 There are three Lab-wide priority (LWP) areas: LWP 1 supports recovery and improved resilience in the Ebola￾affected countries in West Africa. LWP 2 integrates a suite of coordinated digital tools and technologies into Feed the Future activities. LWP 3 accelerates access to off-grid energy with a focus on solar solutions. STIP Performance Evaluation Final Report: September 2017 | vi Methodology The evaluation employed a qualitative methodology, and data sources included documentation provided by the Lab and the Missions as well as key informant- and group information interviews (KIIs and GIIs) with staff (i.e., 96 from the Lab, both Missions, and from some of the Missions’ partner organizations). A purposive sample of interviewees was compiled for the evaluation to selectively interview key staff engaged in STIP activities and those in leadership positions. There were no major limitations in the evaluation; however, of note is that more Lab services have been implemented in the Uganda Mission, so most Lab interviewees had engagements with the Uganda Mission and fewer had engaged with the West Africa Regional Mission. Findings Q1: Role of Lab in Increasing Understanding of, Capacity to Use, and Application of STIP There are mixed views on the Lab’s role in increasing understanding of and capacity to use STIP. Some staff felt that the Lab had played a role in their understanding of and capacity to use STIP. Staff in both Missions who had taken part in the Lab’s Talent Exchange Fellowship reported an increased understanding of and capacity to use STIP as a direct result of the fellowship. Similarly, Mission specialists, such as those utilizing GIS, also benefitted from the support and training received from the Lab. Additionally, senior leadership and several staff in both Missions recognized the role of the Lab in contributing to this increase. However, they also emphasized that the Mission context — including Mission leadership and office directors’ support of STIP, along with the STIP work already implemented by the technical teams — enabled an increase in understanding and capacity to use STIP. On the other hand, a handful of staff members in both Missions did not believe the Lab’s work had an influence on STIP at their Missions, or thought it was too early to ascertain the Lab’s role. The Lab’s role in increasing the application of STIP in the Missions’ Program Cycles also varied. At the strategy level, the Lab had more influence in Uganda relative to West Africa Regional. As the West Africa Regional Mission’s Regional Development Cooperation Strategy (RDCS) was released prior to its formal engagement with the Lab, the Lab did not have an influence on the Mission’s strategy. On the other hand, the vulnerability mapping conducted by the GeoCenter at the Lab and the Futures analysis done by the Pardee Center at the University of Denver with support from the Lab, contributed directly to the development of USAID/Uganda’s Country Development Cooperation Strategy (CDCS). Both Missions saw an increase in project-level STIP application especially as a result of working with the Lab’s GeoCenter. The GeoCenter helped the Regional Health Office (RHO) at the West Africa Regional Mission in the design of their Project Appraisal Document (PAD).2 Through the catalytic Digital Development Advisors Fellowship, the Regional Mission’s newly hired GIS specialist will also provide visual mapping and geospatial analysis services to support all projects of the Mission’s technical offices. In Uganda, the GIS specialist has received various trainings from the GeoCenter and is currently providing visual mapping and geospatial analysis services for the development of all five of the Mission’s PADs. 2 A PAD describes the project design and the supporting evidence upon which it is based. STIP Performance Evaluation Final Report: September 2017 | vii The Lab’s targeted engagement with the two Missions has also led to various services and resources being employed at the activity level. For example, LWP 1 for Ebola has set up the West Africa Health Informatics Team (WAHIT) in West Africa, and the Environment Office at West Africa Regional obtained funding from the Lab to develop the Land Use Land Cover atlas. Under LWP 2 for Digital Development for Feed the Future (FtF), digital finance and agriculture workshops and financial service locations mapping, among other activities, have been conducted for the Uganda Mission. The Lab’s Digital Inclusion team also works with the Uganda Mission and its partners to strategize and design solutions for connectivity, mobile, and Internet access issues. Additionally, the Lab conducted Digital Development trainings and iDesign workshops for staff in both Missions. The application of STIP in monitoring and evaluation (M&E) has been limited. While the Health Office at the Uganda Mission has bought into one of the Lab’s Monitoring, Evaluation, Research, and Learning Innovations program (MERLIN) mechanisms, no Lab services related to M&E have been employed at West Africa Regional Mission. Other components of the Missions’ Program Cycles experience some increased STIP application. The two Missions (and all operating units across the Agency) do not have separate or specific budget line items for STIP, but this is not a barrier for the application of STIP in the Missions’ projects and activities. The position of the STIP Advisor at the West Africa Regional Mission and the STIP Fellow at the Uganda Mission (who work closely with the Lab), however, do serve as key resources for the Missions to promote STIP integration across their portfolios. USAID’s broader development policies, including the Agency policy framework, USAID Forward, and the Automated Directives System (ADS), also address and encourage the integration of STIP, but this is not Mission-specific. In addition, neither Mission is conducting any Mission-wide STIP learning, and hence, it is difficult to demonstrate any examples of learning and adapting relative to STIP. Q2: STIP Integration in the Missions, Lab Services with the Most Value, and the Enabling Environment STIP is integrated at the strategic level more holistically when it is a guiding principle in the Country Development Cooperation Strategy/Regional Development Cooperation Strategy . STIP is not integrated in the West Africa Regional Mission’s current RDCS, but is well integrated in the Uganda Mission’s CDCS. STIP is also a guiding principle for how the Uganda Mission will achieve its strategic goals, as Mission staff are directed to address development challenges throughout their work with STIP approaches as appropriate. Lab services contribute to STIP integration at the project- and activity levels. Various services and resources provided by the GeoCenter, through LWPs and research grants, are the most utilized and have contributed to STIP integration at the project- and activity levels across both Missions. When Missions’ PADs incorporate STIP, it is integrated in the subsequent activities. In addition, both Missions’ staff and implementing partners play a pivotal role in STIP integration regardless of the engagements with the Lab. These individuals and organizations have been incorporating STIP approaches into their projects and activities for decades. STIP Performance Evaluation Final Report: September 2017 | viii The Lab has had some influence on the ability of certain key staff to influence STIP integration at their respective missions. These staff include the STIP Advisor/Fellow, GIS specialists, and foreign service nationals who took part in fellowships at the Lab, along with the Project Development Officer (PDO) at the Uganda Mission. These staff have helped integrate STIP at the organizational level in their Missions. For example, the STIP Advisor/Fellow provides inputs to different technical teams on how STIP can be integrated in their projects and activities. Digital development trainings and iDesign workshops have also enabled targeted individuals to more effectively integrate STIP into their work. Lab services with the most value for STIP integration were not specificed. Staff at the West Africa Regional Mission did not identify any particular Lab services that had the most value in integrating STIP in the Mission’s work. In the Uganda Mission, the leadership and multiple staff members asserted the value of the Lab’s services through LWP 2 in supporting activities under the Economic Growth Office (EGO), and through the GeoCenter in helping the Mission develop its CDCS. A supportive enabling environment in combination with targeted Lab engagement increased STIP integration across the Missions. Leaders at both Missions proactively promote and emphasize STIP, which is a key enabler of STIP integration. The interviewees believe that STIP integration also occurs when the Lab aligns their targeted engagement with Mission strategy and the technical teams’ projects and activities. Effective engagement also happens when done in person via temporary duty assignments (TDYs). STIP points of contact in both the Mission and the Lab have also been instrumental in promoting STIP integration through their coordination role. Q3: Additional STIP Services Wanted from the Lab Missions request more demand-driven services. Many Mission staff addressed the aspect of unfocused offerings coming from the Lab to the Missions, though they noted that this challenge has improved over time. However, Program Office (PO) staff in both Missions expressed frustration that the Lab continues to offer services that do not align with the Missions’ strategies. Mission staff want to increase their STIP knowledge. Several staff in both Missions expressed a desire for more learning around STIP. The Lab Resource Portal (currently in development) will be critical to this effort; however, the portal is intended to house all of the Lab’s knowledge products on one website, while additional synthesis and dissemination of this learning is beyond its current scope. Mission staff also suggested a number of effective ways to collect and share STIP learning, including surveys, periodic reports, forums, and open platforms for ongoing collaboration. Mission staff want more specific Lab services and some new offerings. Staff in both Missions also requested continued assistance from the Lab with design and analytics training and other services that teams found particularly useful, especially mapping and analytical services from the GeoCenter. Some new services were also especially desired by Mission staff, including Lab assistance to Support Offices in employing STIP in management functions such as procurement, budget planning and execution, human resources, and information technology operations; sourcing technical experts who could offer individualized solutions for their work (e.g., applying STIP in programs on countering violent extremism, CVE); conducting Grand Challenges in the field; and, establishing satellite development labs. STIP Performance Evaluation Final Report: September 2017 | ix Q4: STIP Integration for Missions Partners STIP integration depends on many factors for implementing partners. Staff from both Missions’ IPs were clear that integrating STIP in their projects and activities depends on the technical scope of the work and the pertinence of STIP approaches and tools for their activities and target areas. The West Africa Regional and Uganda Missions also encourage their IPs to incorporate STIP in their projects’ activities, and the IPs praised the Missions for providing them the space to incorporate innovations in their activities. West Africa Regional has a specific mandate to build regional partners’ capacity. The regional partners of the West Africa Regional Mission integrate STIP in their projects and activities as their objectives and scope of work require them to do so. The West Africa Regional Mission has a specific objective to support and build the capacity of its regional partners, many of whom have STIP-specific mandates. Therefore, the Mission’s leadership and multiple staff asked for additional support from the Lab in building the STIP capacity of their regional partners. Recommendations To increase understanding of, capacity to use, and the application of STIP at Missions and to improve STIP integration in Missions, the evaluation team recommends that the Lab: ● Create an online module of the STIP Integration training to make it easily accessible for Missions around the world. Because some Mission staff are not clear about the different components of STIP and how they can incorporate these in their work, the Lab should also request Missions’ Front Offices to make the STIP integration course compulsory for all staff. ● Solicit feedback from Mission staff regarding additional specific STIP knowledge gaps. Specifically, the STIP Advisor/Fellow and relationship managers (RMs) could help gather feedback across both Missions. This information could help inform relevant Lab staff responsible for developing Mission-specific trainings so that topics highest in demand are addressed. ● Examine where GIS expertise might be needed in the Missions to provide additional support given the high demand for this type of service. ● Continue hosting a STIP forum annually or bi-annually to share lessons learned and best practices around the application of STIP in strategies, projects, and activities. This forum could be held in key regional missions (e.g., Southern Africa Regional) with an open invitation to bilateral missions in the region. ● Explore the possibility of working with relevant Mission teams to encourage STIP learning via a learning agenda or other mechanisms, such as an internal STIP integration competition, brown bags, etc. ● Explore how the PDO in USAID Missions can obtain a better grasp of the Lab’s services and resources, as well as comprehend where they might provide the most value in their Mission’s Program Cycle. ● Adjust the assessment approach for identifying openings for targeted assistance based on Missions’ needs. Specific Lab teams can then determine which Lab resources might address this request, and offer the Mission options for services based on the Lab team’s resources, bandwidth, and expertise, as well as the Mission’s time frame. STIP Performance Evaluation Final Report: September 2017 | x ● Establish a plan to capture knowledge before the end of the STIP Advisor/Fellow’s term in order to understand how this position has supported the engagement between the Mission and the Lab. ● Have Lab leadership engage with Mission directors to explain how Missions can benefit from Lab services and resources to address their needs and priorities. ● Provide a concise memo or brief report to Mission irectors and deputy Mission directors that highlights how Missions, such as Uganda, have integrated STIP in their CDCS, and which Lab services have contributed to this integration. ● Engage with Missions with strong leadership support and buy-in for STIP, as this is a key enabler for STIP integration. Lab leadership should consider this a key criterion when deciding to engage with future Missions. To provide services that are aligned with Missions’ priorities, the evaluation team recommends that the Lab: ● Conduct a strategic review to examine the extent to which its services and resources align with the strategies of the Missions. ● Discontinue submitting requests for concurrence on activities that are not clearly aligned with a Mission’s strategy or objectives. Lab leadership must ensure that Lab staff have a solid understanding of a Mission’s priorities and needs, and also confirm that the Lab’s initiatives are in alignment with the Missions’ strategies. ● Disseminate STIP learning products to Missions that highlight specific examples of STIP successes and challenges, with tips that cut across various types of work, development sectors, and countries. Consider how to better utilize the Lab field notes and other existing dissemination channels for this purpose. To increase STIP integration for Mission partners, the evaluation team recommends that the Lab: ● Identify specific Lab services, jointly with the Missions, that can contribute to addressing Missions partners’ STIP mandates. ● Explore whether Lab staff, in conjunction with relevant Mission staff, can conduct TDYs to work with Regional Missions’ regional partners. ● Conduct STIP assessments of Missions partners’ work to identify opportunities for increased collaboration with the partners. STIP Performance Evaluation Final Report: September 2017 | 1 INTRODUCTION The Global Development Lab was established in April 2014 by USAID, building on its 50 years of experience in applying science, technology, innovation, and partnerships to accelerate development impact. Working collaboratively within the Agency and with external partners, the Lab’s mission is to 1) produce breakthrough development innovations by sourcing, testing, and scaling proven solutions; and 2) accelerate the transformation of the development enterprise through modern tools and approaches. Today, the global development landscape faces unprecedented change. Driven by technological breakthroughs, information, and connectivity, this change presents a tremendous opportunity to help lift millions out of poverty. The Lab works across USAID and the international development community to catalyze a new model of development using STIP to discover, develop, and apply tools, innovations, and approaches that will enable USAID and developing countries to reach DOs better, faster, cheaper, and more sustainably. The Lab’s approach to applying STIP to international development efforts focuses on: 1) science to harness the power of research and academic communities; 2) technology that can empower underserved communities and provide data to inform decision-making; 3) innovation to identify transformative solutions and bring them to scale; and, 4) partnerships to drive collaboration and collective action to achieve sustainable impact. The Lab also operates under a set of guiding principles: ● Open and inclusive: Draw upon the ingenuity of people from around the world. ● Evidence-based: Invest based on strong evidence of impact. ● Catalytic: Attract the support of others to enable sustainable development solutions that reach massive scale. ● Agile: Create fast feedback loops that enable continuous learning and performance improvement. The Lab consists of five centers and two main offices: ● Center for Development Research (CDR): Increases the use of scientific research to improve development outcomes. ● Center for Digital Development (CDD): Advances the use of enabling technologies and data￾driven approaches to empower underserved communities and improve development effectiveness. ● Center for Development Innovation (CDI): Develops new breakthroughs by harnessing the best in American and global scientific and innovation communities. ● Center for Transformational Partnerships (CTP): Develops global partnerships with a wide range of public and private sector stakeholders. ● CAI: Promotes STIP integration and delivers Lab support for field Missions and Bureaus in Washington, DC. STIP Performance Evaluation Final Report: September 2017 | 2 ● Office of Evaluation and Impact Assessment (EIA): Conducts rigorous, evidence-based analysis to guide investment decisions. ● Office of Engagement and Communications (OEC): Supports the goals of the Lab through effective engagement and communication. Purpose of the Performance Evaluation CAI commissioned a performance evaluation through the Learning and Knowledge Management mechanism (LEARN), funded by USAID’s PPL. Through this assessment, the Lab intends to evaluate the effectiveness of priority engagements against its strategy to 1) increase application of STIP throughout the Program Cycle, and 2) build STIP capacity within the Agency. When considering where to conduct the evaluation, the West Africa Regional and Uganda Missions were clear choices, thanks to the length, depth, and breadth of the Lab’s engagement there. This evaluation sought to better understand how these Missions are moving forward on STIP integration on their own and with the support and targeted engagement provided by the Lab, as well as how their IPs integrate STIP in their work. Additionally, understanding how the Lab’s service model can be adapted and improved, by exploring what enabled and inhibited STIP integration efforts, is important to both Lab- and Mission-driven efforts. The evaluation is not intended to judge the success or failure of the Next Generation partnership or LWP efforts, but will offer insights into the extent to which these approaches met their objectives, and how they could be adapted to be more effective. The primary audience of the evaluation is CAI. The secondary audience includes the Lab’s leadership, centers and technical teams, as well as the Missions leadership and offices. The results of the evaluation will be used by the CAI team to better understand the Lab’s role in enabling Missions to integrate STIP. Understanding what works in integrating STIP, and the value of Lab services for this purpose, will help the Lab hone its strategies to catalyze and modernize the development enterprise. The senior leadership of the Lab will also look at the evidence from the evaluation to inform critical decisions for the Lab as a whole. STIP Performance Evaluation Final Report: September 2017 | 3 BACKGROUND A key component of the Lab’s overall strategy is its plan for Agency engagement, which aims to support USAID Missions and Bureaus in their effort to accelerate development outcomes in key countries, regions, and sectors through STIP integration. The plan identifies Next Generation Operating Units (OUs) that have demonstrated a clear commitment to capitalize on STIP opportunities across their portfolios. The Lab is also focused on integrating the best of its innovations, tools, and approaches across Agency priority areas and presidential initiatives, and has three LWP areas: Ebola (LWP 1); Digital Development for FtF (LWP 2); and, Beyond the Gridx (LWP 3). The Lab has Next Generation partnerships with both the West Africa Regional and the Uganda Missions; and the West Africa Regional Mission engages with LWP 1, while the Uganda Mission engages with LWPs 2 and 3. West Africa Regional Mission West Africa Regional leverages strategically targeted programs across 21 countries and works to solve regional problems primarily by working with and through regional partners. The Mission leadership sees its role as multifaceted: to serve as a learning platform, train Mission staff, learn from and leverage their experience, and mainstream the utilization of STIP approaches and tools across the region. The Mission’s RDCS for 2015-2019 states that West Africa Regional’s goal is “social and economic well-being advanced by West Africans, and a commitment to support African-led regional development.” Three DOs and one support objective help to achieve this goal: ● DO 1: Systems of non-violent conflict management strengthened in West Africa ● DO 2: Broad-based economic growth and resilience advanced through West African partners ● DO 3: Utilization of quality health services increased through West African partners ● Support Objective: USAID’s capacity to advance U.S. Government objectives in West Africa enhanced The Mission Director (MD) sought a partnership with the Lab as a natural ally in achieving the Mission’s mandate as a regional learning platform. To serve as this platform, the MD recognized that the Mission would need to access innovative approaches that were well-suited to the region, and the Lab provides networks and services from which West Africa Regional could benefit. A partnership with the Lab could also help the Mission stay “on top of its game.” It was with this context that the MD met Lab leadership and had initial discussions about a partnership in August 2015 in Washington, DC. The current director of the Regional Program Office (RPO) at the Mission then oversaw the development of the relationship with the Lab. Subsequently, in March 2016, a Letter of Intent (LOI) for cooperation was signed between the Mission’s MD and the Lab’s Executive Director, which expresses a common interest in cooperation to integrate STIP in the Mission’s strategy, activities, and operations to advance shared DOs. It remains in effect until the end of the current RDCS, in 2019. STIP Performance Evaluation Final Report: September 2017 | 4 Uganda Mission The Uganda Mission’s engagement with the Lab commenced prior to their current CDCS, which covers 2016-2021. The Uganda Mission’s CDCS has three integrated DOs: ● DO 1: Community and household resilience in select areas and target population increased ● DO 2: Demographic drivers affected to contribute to long-term trend shift ● DO 3: Key systems more accountable and responsive to Uganda’s development needs During the development of the CDCS (2014-2015), Mission leadership acknowledged that they could not continue to do things the way they had done them in the past and expect different results. At that point, the Mission was also aware that the Lab was implementing activities in Uganda, but there was little clarity or coordination around them. To address this issue, a member of the Mission’s EGO was sent to Washington, DC, to compile a list of all activities the Lab was conducting in Uganda. With this information, the Mission also took stock of its own STIP activities, and Mission leadership concluded that they could benefit from the ideas, services, and tools the Lab offers. Also in 2015, the Mission’s MD and Deputy MD (now the current MD) attended a STIP conference in Peru and met with the Lab’s Executive Director to discuss how STIP could be further incorporated in the Mission’s programs. Additional meetings and consultations were held leading to the PDO in the PO becoming the Lab’s STIP point of contact (PoC) in the Mission. Subsequently, in December 2015, a LOI was signed between the Mission’s MD and the Lab’s Executive Director, and is in effect for the duration of the Mission’s CDCS (through 2021). Targeted Engagement The Lab’s targeted engagement with the West Africa Regional and Uganda Missions included a STIP assessment for Uganda, integrated Work Plans for both Missions, and the creation of key coordinating positions. See Annex 5 for detailed findings on the state of the coordination between the Lab and the Missions. STIP Assessment and Integrated Work Plans The Lab did not conduct a STIP assessment in the West Africa Regional Mission, but following the signing of the LOI in 2016, the RPO director and a consultant researched and wrote a report about the Mission’s STIP framework and how specific Lab services could be used in the Mission’s programs. The Lab conducted a formal STIP assessment in the Uganda Mission over several visits from 2015-2016. The final STIP assessment report included discussions of the STIP enabling environment in Uganda, opportunities for USAID to engage in and develop this environment, and recommendations for how to do so. The Lab also developed an annual integrated Work Plan with each Mission, which is jointly agreed upon. STIP Performance Evaluation Final Report: September 2017 | 5 STIP Points of Contacts and Lab Services Information Both Missions’ LOIs with the Lab specified that a STIP PoC would be created in the Lab and at the Mission. In the Lab, the PoC for each Mission is based in CAI and is an RM. Initially, the STIP PoC in the West Africa Regional Mission was a consultant based in Ghana, but the position is now held by a STIP Advisor who was hired in February 2017 for a period of two years. The STIP PoC in the Uganda Mission was initially a staff member (PDO) in the PO, but this role has since been assigned to a STIP Fellow who was recruited in March 2017 for a period of one year. The two RMs, the STIP Advisor, and STIP Fellow coordinate the work between the Missions and the Lab. This relationship management involves communicating and engaging with the various offices in the Missions to discuss their STIP needs and the feasibility of employing Lab’s services in their work. The RMs also engage with teams across the different centers in the Lab, which provide services to the two Missions. Each RM works regularly and closely with their respective STIP Advisor/Fellow and their supervisors. The LOI with West Africa Regional also stipulated that three other positions would be created in addition to the STIP Advisor. A full-time Geographic Information System (GIS) specialist was hired in April 2017 through the catalytic Digital Development Advisors Fellowship for a period of three years, and an American Association for the Advancement of Science (AAAS) Fellow completed a six-week assignment at the Mission in June and July 2017. A Data Visualization specialist was sought but not hired, as a suitable candidate could not be found. The West Africa Regional Mission funds the STIP Advisor’s position, and the Lab provided partial funding for the GIS specialist and full funding for the AAAS Fellow. The RPO director supervises the STIP Advisor and the GIS specialist. Managed by PPL’s Office of Learning, Evaluation and Research (LER), the Expanding Monitoring and Evaluation Capacities (MECap) task order supports USAID in reinforcing and expanding M&E capacities across the Agency. A MECap fellow was recruited by the Uganda Mission who, apart from her M&E responsibilities in the Mission, also serves as a STIP Fellow and currently sits in the PO. A Development Program Assistant in the PO, who is a Foreign Service National (FSN) and took part in the Talent Exchange Fellowship at the Lab in 2016, also contributes to coordination efforts with the Lab. The PDO in the PO supervises the STIP Advisor and the FSN. These three staff are considered the Mission’s “STIP Team.” The Lab’s services are also introduced to Missions in different ways, including through: Interactions that the RMs, STIP Advisor and supervisor, and the STIP Team (in Uganda) have with Missions staff; Dissemination of literature on Lab services to Missions staff; TDYs done by Lab staff to the Missions for various activities, trainings, or workshops; The annual Lab STIP summit and annual Digital Development conference held by the GeoCenter; Individual connections between Lab staff and Mission staff; and the CTP network, which has built and nurtured relationships with the two Missions. STIP Performance Evaluation Final Report: September 2017 | 6 Coordination The current RM for West Africa Regional, who has been in the position for a year, communicates with the STIP Advisor and her supervisor, the RPO director, on a regular basis to make collective decisions about the direction of Lab support. When the RM visits the Mission for TDYs, she meets with staff across the different offices and technical teams. Notably, the Lab has had three RMs for West Africa Regional since the commencement of their targeted engagement, and it was frustrating for the Mission to have to keep reorienting and getting to know a new RM. However, the current coordination between the Lab and the West Africa Regional Mission is seen as beneficial by both sides. The engagement between the Lab and the Uganda Mission did not commence in an optimal manner as noted previously because the Lab was conducting many activities in Uganda of which the Mission was unaware. Furthermore, as the partnership commenced, the Mission leadership had some concerns about the Lab’s approach which did not include enough co-creation. In the first year of the engagement, motivated staff in the Lab and Mission also generated many opportunities for collaboration, but there was no system in place to track this work. To enable better organization of activities and relationship between the Lab and Mission, the RM and STIP team commenced their roles. The coordination between the Lab and Mission has now improved significantly because of the coordinating positions that have been created, the Lab’s increased understanding of the Mission’s needs, and request for concurrence of activities, now codified in the revised ADS 201. STIP Performance Evaluation Final Report: September 2017 | 7 EVALUATION QUESTIONS The evaluation is guided by one main evaluation question and related sub-questions, as follows: ● To what extent has the Lab’s targeted engagement with the West Africa Regional and Uganda Missions, through Next Generation partnerships and LWPs increased their understanding of STIP, capacity to use STIP, and application of STIP throughout the Program Cycle? ○ How has STIP been integrated into the Mission at the organizational, project, activity, and strategic levels? What strategies and methods have been most effective in integrating STIP? ■ Which Lab’s services have the most value in integrating STIP? ■ What is the role of the enabling environment (budget, policy, resources, leadership) in integrating STIP? ○ What additional STIP services (current or new) do Missions want from the Lab? ○ How have the Missions IPs projects and activities integrated STIP? STIP Performance Evaluation Final Report: September 2017 | 8 METHODOLOGY The evaluation employed a qualitative methodology, and data sources included documentation provided by the Lab and Missions, including KIIs and GIIs with 96 staff from the Lab, Missions, and from some of the Missions’ partner organizations. A purposive sample of interviewees was compiled by the evaluation team to selectively interview staff in leadership positions and key staff engaged in STIP activities at both Missions and some of the Missions’ partner organizations. Interview protocols for the KIIs and GIIs, as presented in Annexes 3.a-3.d, were semi-structured to allow for discussion flexibility and a natural flow to the conversation. Detailed notes were taken for all KIIs and GIIs, and the evaluation team analyzed this data by deductively coding themes and querying the data using Nvivo, a qualitative data analysis software. There were no major limitations in the evaluation; however, of note is that more Lab services have been implemented in the Uganda Mission, so most Lab interviewees had engagements with the Uganda Mission and fewer had engaged with the West Africa Regional Mission. The performance evaluation team consisted of two members: a senior evaluator and a Monitoring, Evaluation, Research, and Learning specialist. The evaluation team members’ roles and responsibilities are included in Annex 7. For a full description of the methodology, refer to Annex 2. STIP Performance Evaluation Final Report: September 2017 | 9 FINDINGS Understanding of STIP, Capacity to Use STIP, and Application of STIP in the Program Cycle This chapter provides the findings to address the first evaluation question: To what extent has the Lab’s targeted engagement with the West Africa Regional and Uganda Missions, through Next Generation partnerships and LWPs, increased their understanding of STIP, capacity to use STIP, and application of STIP throughout the Program Cycle? The evaluation found that the Lab has contributed most significantly to the application of STIP in the project and acitivity design and implementation phases of both Missions’ Program Cycles. Lab trainings and support have also improved the understanding and capacity to use STIP for several key staff in both Missions, such as the STIP Advisor/Fellow, GIS specialists, and FSNs who take part in fellowships at the Lab. West Africa Regional This section addresses the Lab’s contribution to the West Africa Regional Mission in: 1) increasing the understanding of and capacity to use STIP; and, 2) the application of STIP throughout the Mission’s Program Cycle. 1. Increase in Understanding and Capacity to Use STIP Some evidence suggests the Lab played a role in improving the Mission’s understanding, capacity, and/or application of STIP: ● One staff person gained understanding and capacity to use STIP through a Lab Fellowship. A Program Management specialist in the RHO, who took part in the Lab’s Talent Exchange Fellowship, was clear that the fellowship had made a big difference: “When I went to the Lab for my fellowship, that opened my eyes to many things I didn’t know before about STIP. My understanding of technology before the fellowship was very narrow. After the fellowship, I realized it’s not about reinventing the wheel. Technology exists but we’re not applying it. With innovation, the same thing.” ● Mission staff noted unique contributions from the Lab. A RHO staff member highlighted a few of the Lab’s key contributions to improving Mission staffs’ understanding and capacity to use STIP: “What is new is the idea of innovating and using new technologies that are less adapted to health because we get stuck in ruts. Also thinking of non-traditional STIP Performance Evaluation Final Report: September 2017 | 10 partners, because we get stuck in comfort zones. Discussions with the Lab have inspired this, to help us think through how to apply [STIP].” Additionally, the RPO’s PDO mentioned that there was now a formal (though unwritten) requirement that STIP be included in project design and planning. In her opinion, this intentional consideration of STIP during project design would not have been instituted without the relationship with the Lab. Other interviewees believed that factors beyond targeted engagement with the Lab contributed to an increase in STIP understanding, capacity, and application: ● Supportive mission leadership and staff technical capacity play a role in STIP application. The West Africa Regional Mission’s MD and RPO director acknowledged the role of the Lab in increasing the understanding and capacity to use STIP in the Mission. However, they also felt that the focus on STIP by the Mission’s leadership and office directors played an important enabling role in this increase. Additionally, technical teams had been implementing STIP work prior to their engagement with the Lab. For example, a staff member of the RHO said that STIP is not new, their team has a good understanding of STIP and has been employing it for a long time. ● The Mission’s understanding and capacity to use STIP is linked to their regional partners. Several Mission staff explained that they had to work closely with their regional partners in order to achieve the Mission’s objectives. Therefore, they suggested that both the Mission as well as their regional partners needed to increase their understanding of and capacity to use STIP. Last, some staff from both the Mission and Lab were unsure about the contribution of the Lab to increase understanding, capacity, and application of STIP at the Mission: ● Some Mission staff said it was difficult to gauge the role of the Lab. The Deputy MD suggested that it was premature to determine the Lab’s role as some Mission staff were still trying to better understand STIP and its value add. The PDO with the RPO did not have clear answers as to the Lab’s role in increasing the understanding and capacity to use STIP. The STIP Advisor also said that increased understanding and capacity to use STIP was difficult to gauge because there was so much non-Lab-led STIP work going on in the Mission. Further, the Advisor explained that some staff had a better understanding of certain components of STIP than others due to their prior work experience and current scope of work; for example, some staff were clear about what scientific research entailed, while others were more familiar with incorporating technology in their activities. ● A couple of Mission staff asserted that the Lab did not contribute. Staff of the Environment Office of the Regional Economic Growth office (REGO) who had received STIP resources and technical expertise from the Agency’s Bureau for Economic Growth, Education, and STIP Performance Evaluation Final Report: September 2017 | 11 Environment (E3) credited their knowledge and understanding of STIP to the E3 Bureau rather than the Lab. ● Most Lab staff felt uncertain of their contributions, though the Lab Relationship Manager believed the Lab contributed to an increase in STIP understanding, capacity, and application. Most Lab staff who had engaged with the West Africa Regional Mission were uncertain as to whether the Lab’s targeted engagement had led to an increase in understanding and capacity to use STIP in the Mission. Nonetheless, they believed that the FSN who sits at the RHO built understanding of STIP through the Fellowship experience with the Lab. The RM for West Africa Regional, on the other hand, asserted that the Lab’s targeted engagement had increased some Mission staff’s understanding of STIP, and that overall, Mission staff were eager to learn more about STIP. The RM also affirmed that the Lab’s engagement with the Mission had increased the capacity of some staff to use STIP, and felt that as the Mission collaborates on more work with the Lab, this capacity would further increase. She noted that some components of STIP were better understood and applied than others: “Understanding where science, data, and evidence influences their work — they get it, it’s just figuring out the nuance. Same for integrating technology; they understand where there’s capacity to utilize it, the modernized approaches for innovating. Where there’s a real gap is in the partnership construct. At the bilateral level it’s easy to partner with the private sector, government, etc. — that’s tougher to think through with the regional Mission because their partners are multilateral organizations. I don’t see in many of the portfolios where we have significant partnerships.” 2. Application of STIP in the Program Cycle3 a. Country/Regional Strategic Planning The Lab could not contribute to the Regional Development Cooperation Strategy due to the timing of engagement. As the West Africa Regional Mission’s RDCS was released prior to its formal engagement with the Lab, the Lab did not have the opportunity to contribute to the strategic planning phase of the Mission’s Program Cycle. Notably, the RDCS does not discuss STIP in any detail, but the Front Office felt that the absence of STIP from the current RDCS was more of a labeling issue, and that the Mission’s strategy does in fact encompass STIP. Front Office staff also stated that the next RDCS would be more explicit about the use of STIP. b. Project and Activity Design and Implementation The Lab made several contributions to the Mission’s project design and implementation. At this phase of the Program Cycle, the Lab has contributed to increased application of STIP in the following ways: 3 https://usaidlearninglab.org/program-cycle-overview-page STIP Performance Evaluation Final Report: September 2017 | 12 ● The RHO commissioned the GeoCenter in CDD to determine which of the 17 countries they cover had the greatest needs. The GeoCenter provided results in the form of maps and visual data, and these results were used by RHO in the design of their PAD). The RHO director was very complimentary of the GeoCenter’s work, and said that it had helped determine the strategic direction of the health PAD. Additionally, the RM strongly felt that the GeoCenter’s support for the health PAD led to increased awareness across the Mission of the utility of GIS mapping, particularly as other technical teams design new projects. ● Through the Lab’s catalytic Digital Development Advisors Fellowship, the newly hired GIS specialist provides visual, mapping, and geospatial analysis services to support project design and results analysis for the technical offices at the Mission. The Regional Peace and Governance Office (RPGO) director emphasized her team’s support for the GIS specialist position, anticipating it to help their program map violent extremism in the region. The Lab has clearly contributed to the Mission’s activity design and implementation. The STIP Advisor noted that, since the beginning of the engagement, the Lab’s services had played the most influential role in applying STIP in the activity design and implementation phase of the Mission’s Program Cycle. Various Lab services and resources have been employed in this phase. For example: ● Supported through LWP 1, WAHIT, a team of software developers based at the West African Health Organization’s (WAHO) Health Information System (HIS) office, will conduct regional data analysis of health information. ● A research contract under LWP 1, Evaluation for Development (E4D), will study the effectiveness of different technologies (such as SMS, mobile apps, etc.) in Ebola-stricken countries. ● Also under LWP 1, a company named Real Impact Analytics is developing a disease surveillance tool for decision-making using telecom-operated user data, to help understand the movement of individuals when there are disease outbreaks. ● The Environment Office obtained $250,000 funding from the Lab and provided $750,000 of its funds to develop the Land Use Land Cover atlas. ● The Lab provided funding to the REGO to beta-test a regional e-platform for a trade zone called the Borderless Alliance. ● Three Digital Development trainings were conducted for Mission staff and IPs. ● Various Higher Education Solutions Network (HESN), Partnerships for Enhanced Engagement in Research (PEER), and Development Innovation Ventures (DIV) awards and grants are ongoing in the countries covered by the Mission. ● The Lab was instrumental in helping the Mission define its West Africa Learning and Knowledge (WALK) agenda, narrow down its learning objectives, and socialize it across the Mission, including getting buy-in from technical teams and Mission leadership. Additionally, a Lab-supported AAAS Fellow, who recently completed a six-week assignment at the Mission, supported the WALK agenda. STIP Performance Evaluation Final Report: September 2017 | 13 c. Monitoring & Evaluation As of yet, no Lab services have been employed to support the monitoring and evaluation phase of the Mission’s Program Cycle. d. Budget, Resources, and Development Policy Missions do not have a separate budget for STIP. Budgets are key inputs to the Program Cycle, but West Africa Regional (and all OUs across the Agency) does not have a separate or specific budget line item for STIP. Offices in the Mission utilize their budgets for projects and activities, and incorporate STIP without having specific budgetary allocations for STIP. Rather than seeing this as detrimental, the Division Chief for Strategy and Learning in CAI explained that this lack of designated funding for STIP is deliberate: “Budget-wise there is no STIP earmark and there never will be. We think that [would be] a bad idea because the whole idea is to have STIP integrated in program funding. Having separate funding would discourage integration and lead to doing STIP for STIP’s sake.” The STIP Advisor position is a key resource for promoting STIP integration in the Mission. The position of the STIP Advisor serves as a key resource for the Mission to promote STIP integration across its portfolio; the STIP Advisor acts as a catalyzer by identifying, promoting, and capitalizing on opportunities for STIP in the Mission. The STIP Advisor meets with the various technical teams and offices in the Mission to understand their work. As part of this dialogue, the STIP Advisor provides input to teams regarding what Lab services might be incorporated into their work, and helps them identify how to incorporate STIP in their work. While STIP integration is stipulated in Agency development policy, guidance on its application at the Mission is not Mission-specific. The Agency policy framework lists seven operational principles, two of which apply to STIP: “Apply science, technology, and innovation strategically,” and “Leverage solution holders and partners strategically.” USAID Forward, a reform agenda for the Agency, also highlights STIP by calling for the promotion of sustainable development through high-impact partnerships and local solutions, and identifying and scaling up innovative, breakthrough solutions to address intractable development challenges. In addition, the Lab was involved in drafting the new Agency ADS regulation and worked closely with PPL to ensure that adaptive management and STIP principles were reflected in this key regulation. e. Learning and Adapting4 There are no Mission-wide STIP learning or adaptation platforms. With no Mission-wide STIP learning taking place, consequently there is no adaptation or decision-making based on learning. The MD affirmed that the Mission sees itself as a learning organization, and their Support Objective reinforces this emphasis. To that end, the newly developed WALK agenda is central to this effort, but it has yet to be formalized, structured, and integrated into the Mission’s procedures and processes. Nonetheless, the RM and the STIP Advisor believed the WALK agenda, along with the Collaborating, Learning, and Adapting (CLA) Advisor who will be 4 For more information, see: https://usaidlearninglab.org/sites/default/files/resource/files/keyconcepts_twopager_8.5x11_v7_20160907.pdf. STIP Performance Evaluation Final Report: September 2017 | 14 joining the Mission in the near future, will both contribute to Mission-wide STIP learning and adaptation efforts going forward. To promote learning around STIP at the Mission, the STIP Advisor started a STIP newsletter and hosted brown bags to make staff aware of STIP projects and activities at West Africa Regional and other Missions. Different offices also have their own events in which they showcase STIP-related learning. For example, REGO was involved in an innovative technology best practices conference, and some Mission staff spoke on a panel about new technological developments with seeds. Despite these positive efforts, the STIP Advisor acknowledged that not all technical teams are proactive and self-motivated in generating and disseminating STIP learning. Thus, while STIP learning occurred at the activity level, no Mission-wide, STIP-based CLA was taking place. Uganda Mission This section addresses findings regarding the Lab’s contribution to: 1) increasing the understanding and capacity of the Uganda Mission to use STIP; and, 2) the application of STIP throughout the Mission’s Program Cycle. 1. Increase in Understanding and Capacity to Use STIP The findings as to whether the Lab’s targeted engagement contributed to an increase in the understanding of and capacity to use STIP in the Uganda Mission also varied. Staff from both the Mission and Lab highlighted specific contributions the Lab made to increasing STIP capacity at the Mission. ● Staff who participated in the Lab’s trainings had improved understanding of STIP. The Development Program Assistant and GIS specialist, both members of the STIP team in the PO, traced the increase in their understanding and capacity to use STIP to the Lab. The Program Assistant took part in the Lab’s Talent Exchange Fellowship, and the GIS specialist benefited from GeoCenter trainings. ● Many other Mission staff attribute some increase in understanding and capacity to use STIP to the Lab. The Private Sector Unit Leader in the EGO stated that increased understanding and capacity to use STIP of staff on his team was mainly due to their own efforts and experiences with STIP. However, he noted the Lab’s role, saying: “I think you could say yes — that [the Lab] has increased understanding and capacity . . . especially with regard to digital payments.” Another EGO staff member reiterated the Lab’s support in a particular area of the team’s work: “The Lab has helped us to see partnerships, where opportunities might be, and how to engage partners.” A member of the Health Office also spoke of the team’s engagements with the Lab, and how this contributed to their increased understanding of STIP: “Now with regular engagement, explanations of STIP, iDesign training . . . the health team understands what STIP is. Of course it’s at different levels.” A member of the STIP Team in the PO also reported that, since joining the STIP Performance Evaluation Final Report: September 2017 | 15 Mission, she has noticed that many staff have increased their understanding of STIP, and believed that the Lab played a role in this increase by raising awareness of STIP: “Each year there’s more and more understanding. We’ve had Lab folks help with project design, research, digital training for AORs/CORs [Agreement Officers/Contract Officers]. We’ve had a lot of attention from the Lab, so that contributed to people understanding [STIP] more.” ● Lab staff also offer concrete examples of increased STIP understanding and capacity at the Mission. The RM also contended that targeted engagement with the Lab had led to an increase in the understanding of STIP in the Uganda Mission: “Since we started this relationship, STIP knowledge has increased exponentially. This makes sense because it’s related to our development as well. Our relationship with [the Uganda Mission] has grown so their knowledge has as well. More of the Mission has been able to learn about STIP. A lot of times in Missions one person will know about PEER, one about DIV, but they won’t know about each other. Uganda has been able to see a lot of different parts of the Lab.” This suggests that without the relationship with the Lab, the Mission would not have had as many robust activities around STIP, though they would still have conducted STIP work. The vulnerability mapping analysis the GeoCenter provided for the Mission’s CDCS helped increase Mission staff’s understanding and capacity to use STIP. A Senior GIS specialist noted that the GeoCenter built STIP capacity in the Mission by providing training for the GIS specialist. A CTP staff person confirmed this, noting that Mission staff gained increased understanding and capacity to engage in partnerships through its work with their Center: “When I visit and talk to staff there who are working on partnerships with us I feel that these key officials have increased their understanding of partnerships.” ● A key factor in increased STIP understanding and capacity is supportive mission leadership. Encouragement from Mission leadership complements the role of Lab. The MD and Deputy MD affirmed the role of the Lab in contributing to increased understanding and capacity to use STIP at the Mission. However, they explained the catalytic role of Mission leadership who put STIP forward as a Mission priority. PO staff members asserted that Mission leadership’s prioritization of STIP significantly contributed toward increasing staffs understanding and capacity to use STIP, along with the knowledge, inputs, and resources the Lab had provided that helped raise the profile of STIP in the Mission. While interviewees provided many specific examples of greater STIP understanding and capacity as a result of working with the Lab, these improvements are not uniform throughout the Mission. ● Mission staff’s understanding and capacity to use STIP varied among individuals and offices. The EGO, for example, tended to be stronger in their capacity to use STIP, because they had already incorporated digital tools into their projects. Their activities around markets STIP Performance Evaluation Final Report: September 2017 | 16 and youth also lend themselves to the use of technology. The PDP also noted that there are various levels of understanding about STIP integration among Mission staff; STIP meant different things to different teams and individuals. Some understood STIP in terms of science and technology, some in terms of partnership, and some view it in a more integrated way. Generally, there is a largely nuanced understanding of the aspects of STIP as staff view it based on their roles. Finally Mission staff noted that in other to further increase understanding and capacity to use STIP, they would need more hands-on experience with STIP. ● Some staff felt unsure how to integrate STIP into their work. A few Mission staff recognized that some of their colleagues struggle with how to incorporate STIP into their activities. They emphasized that a clear understanding and capacity to use STIP will not be realized until STIP becomes integral to staffs projects and activities. A member of the iDesign team in CDI at the Lab, who spent two months as a Fellow at the Uganda Mission, articulated this view: “A lot of learning is by doing. You can build capacity through training, but engaging with a real SOW [Statement of Work], activity design, etc. is where it becomes more real to people.” 2. Application of STIP in the Program Cycle a. Country/Regional Strategic Planning The Lab contributed directly to incorporting STIP in the CDCS. The Lab’s targeted engagement with the Uganda Mission has increased the application of STIP in the strategic planning phase in multiple ways: ● The vulnerability mapping, conducted by the GeoCenter, looked at household survey data to better understand the drivers of vulnerability across the country at sub-national levels. Findings from this analysis directly contributed to the development of the CDCS and to the Mission’s DOs. ● The Futures analysis by the Pardee Center at the University of Denver, supported by the CDD team at the Lab, directly contributed to the CDCS by providing a forecasting model. The MD explained the importance of the Futures analysis for the Mission, stating: “The Lab helped us get to a narrative, which is that the average Ugandan is a 14-year-old girl, and that message is starting to resonate across the country.” The PDO in the PO said that the Futures analysis was a “game-changer” for the Mission’s strategy, and that Uganda was the first USAID Mission to employ it. ● The RM for Uganda highlighted the integral support the Lab provided to the Mission in its strategic planning phase, stating: “Our relationship started because [the Uganda Mission] wanted to integrate STIP strategically, around their CDCS in June 2015. We’re heavily involved in that CDCS, which was just approved in December 2016. They have 15 guiding principles . . . two are directly integrated with STIP. You can search the word ‘innovation’ and see it throughout [the CDCS]. This whole thing is based off of a Futures analysis . . . they got the idea from us.” STIP Performance Evaluation Final Report: September 2017 | 17 b. Project and Activity Design and Implementation The Lab contributed to increased application of STIP in project design and implementation. This happened in the phase of the Mission’s Program Cycle through the inputs from the Lab-trained GIS specialist who received various trainings from the GeoCenter, participated in the annual workshop for GIS specialists at the Lab, and attended a STIP workshop in India. This specialist has provided mapping, visualization, and geospatial analysis support for all five of the Mission’s PADs. The Lab has contributed extensively to activity design and implementation. The Lab’s targeted engagement with the Uganda Mission has led to a number of Lab services and resources being applied at the activity design and implementation phase of the Mission’s Program Cycle, for example: ● LWP 2 initiatives at the Mission focused on the application of digital financial services, affordable connectivity, and digital inclusion and development in agriculture to improve farming systems. Various activities that have been conducted under LWP 2 include: digital finance and agriculture workshops, Digital Development training, publication of the Guide for the Use of Digital Finances in Agriculture, Internet connectivity for agriculture services, and mapping financial service locations across Uganda. ● CDD’s Digital Inclusion team works with the Mission and its IPs to strategize and design solutions for connectivity, mobile, and Internet access issues. ● CDI held iDesign workshops and equipped Mission staff with techniques and tools to identify opportunities for innovation to drive change. The PDO reported applying more inclusive design practices to her work based on learning from iDesign. ● A total of 15 DIV grants have been awarded in Uganda across various sectors, including health, solar energy, eco-fuels, safe water, and irrigation. Six PEER awards address health, agriculture, water storage, and crop yield issues across Uganda. HESN awards, such as Resilient Africa Network (RAN), find innovative local solutions to overcome the shocks and stresses of poverty, conflict, displacement, disease, and drought, while the Comprehensive Initiative on Technology Evaluation (CITE) helps create system maps of actors in Uganda for the FtF-value chain process. ● The GeoCenter supports a Youth Mappers project to help understand where communities are located and connected for providing HIV/AIDS services. ● A cost-benefit analysis of solar companies in Uganda was carried out through LWP 3. ● The Lab has held innovation marketplaces where IPs, nonprofits, NGOs, and businesses in Uganda network around innovative approaches for development challenges. Lab staff also shared examples of their engagements with the Uganda Mission in this phase of the Program Cycle. A Digital Inclusion Advisor in CDD said that Mission staff were now more aware of the possibilities for Internet connectivity beyond the traditional purposes previously explored by the Mission. An AOR for the HESN award for CITE believed that that activity had helped the Mission better assess their interventions in the agriculture- and value chain sectors. STIP Performance Evaluation Final Report: September 2017 | 18 A LWP 2 Activity Manager noted that their relationship has deepened with EGO, and the Private Sector Unit Leader now participates in donor meetings on digital finance. The role of the Lab is valued in applying STIP at the activity level. The PDO and STIP Fellow agreed that Lab has played a prominent role in the application of STIP in the activity design and implementation phase of the Mission’s Program Cycle. The Private Sector Unit Leader at EGO affirmed the beneficial role of the Lab in EGO’s activities: “We’ve taken our bilateral programming and are layering Lab investments on top of that to support activities where we see opportunities.” Reiterating the critical role of the Lab, another member of the EGO team commented: “The Lab has enabled us to have cross-linkages, like health and education. With financial services there’s still a lot we can do. The solar company can provide loans, health centers come up with an app where they can request help at a certain address. We need the Lab more than ever to help pull these tools together in context and layer them. We rely on them for impact analyses. There’s no doubt we’ll need their help to look across sectors we’re working in and see how we can leverage one tool across sectors.” c. Monitoring & Evaluation Some Lab services have been used for the monitoring and evaluation phase of the Program Cycle. MERLIN, led by the Lab in partnership with PPL and the Bureau of Global Health, offers a unique approach to program design that allows users to source, co-create, and co-design development solutions that innovate on traditional approaches to monitoring, evaluation, research, and learning. EIA staff held a workshop on MERLIN and its five buy-in mechanisms for staff at the Uganda Mission. As a result of this workshop, the Health Office bought into one of the MERLIN mechanisms, Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES). d. Budget, Resources, and Development Policy As previously noted for West Africa, Agency OUs do not have a separate budget for STIP. STIP is integrated into USAID’s broader development policy, but that policy is not Mission-specific. It is clear in both Missions that having a staff position focused on STIP is key to the Mission’s ability to promote STIP integration across its portfolios. In the case of Uganda, the STIP Fellow catalyzes technical team’s work in the Mission by identifying opportunities for STIP and connecting the teams to the relevant Lab services. e. Learning and Adapting Currently no STIP-specific, Mission-wide learning or adaptation is conducted. The PDO and STIP Fellow in the PO asserted that the Mission does not delineate STIP learning separately from non-STIP learning, and said that there was no specific agenda for STIP learning in the Mission. The STIP Fellow also reported that STIP learning in the Mission was based more at the activity level; however, with the new CDCS and PADs, STIP comes through at the strategy- and project levels, which should lead to more STIP learning opportunities in the future. The STIP Performance Evaluation Final Report: September 2017 | 19 Mission’s Learning Fellow said that the Mission is currently developing its Learning Agenda, but noted that they are not looking specifically at STIP. The evaluation did not surface any Mission￾wide adaptation or decision-making taking place based on evidence from Mission-wide STIP learning. Thus, while STIP learning occurred at the activity level, no Mission-wide STIP-based CLA was taking place. Understanding STIP Integration in Missions This section addresses the second evaluation question and its sub-questions: How has STIP been integrated into the Mission at the organizational, project, activity, and strategic levels? What strategies and methods have been most effective in integrating STIP? ■ Which Lab’s services have the most value in integrating STIP? ■ What is the role of the enabling environment (budget, policy, resources, leadership) in integrating STIP? The Lab has most notably contributed to STIP integration at the project and activity levels in both Missions, although other factors (such as an emphasis on STIP by the Missions’ leadership and a history of STIP work in both Missions) also played a role. STIP integration at the strategic and organizational levels in the Missions varied and was more context-specific. 1. How STIP Integration Happens in Missions a. Strategic Level Across both Missions, strategy-level STIP integration occurs when Mission leadership emphasizes and promotes STIP. STIP is also integrated at the strategic level more holistically when it is a guiding principle in the CDCS/RDCS. ● STIP is not integrated in West Africa Regional’s Regional Development Cooperation Strategy but is well integrated in Uganda Mission’s Country Development Cooperation Strategy. While STIP is not integrated in the West Africa Regional Mission’s current RDCS, the West Africa Regional MD affirmed that they emphasize STIP across the Mission at a strategic level, even though it is not prominently expressed in the RDCS. He added that the Mission adhered to STIP principles, and there are many examples of the Mission promoting the use of STIP that preceded their relationship with the Lab. In the Uganda Mission, STIP is well integrated in their CDCS, which emphasizes STIP throughout the Mission’s portfolio of work, and also includes it as a Guiding Principle shaping how the Mission will achieve its strategic goals. Certain Lab services, including the Futures analysis and vulnerability mapping done by the GeoCenter, directly contributed to the development of the CDCS. In addition, leadership in both Missions is very proactive in promoting and emphasizing STIP across their Mission’s portfolio of work. STIP Performance Evaluation Final Report: September 2017 | 20 b. Project and Activity Levels STIP is integrated at the project and activity levels as a result of the Lab’s support, emphasis by Mission leadership, and understanding that STIP is a best practice. As such, both Lab-led and non-Lab-led STIP efforts contribute to this integration. ● Multiple factors support STIP integration in Project Appraisal Documents and subsequently, in projects and activities. Various factors have contributed to the incorporation of STIP in PADs at both Missions. The GeoCenter and both Missions’ GIS specialists provided mapping, visualization, and analysis services that helped technical teams incorporate STIP approaches in their PADs. Office directors, technical teams, and the PDOs also emphasized STIP in the design of PADs. The West Africa Regional MD reported that the Mission’s PADs contribute to the integration of STIP, and observed that every approved PAD has elements of STIP. In Uganda, the technical teams play a major role in integrating STIP into PADs; currently, all of the five PADs in development are integrating STIP. This language then translates into implementation in projects and activities. ● Missions’ leadership emphasizes STIP. The RPO director in the West Africa Regional Mission stated that the RFO (Regional Front Office) emphasizes STIP, which furthers the integration of STIP in projects and activities. She added: “We have strong support from senior leadership, and they are going to continue to nudge forward institutionlizing STIP across the Mission’s work at the strategic, programmatic, and operational levels.” A member of the Uganda Mission’s PO also asserted that the Mission’s leadership is very interested in being innovative and injecting STIP into the Mission’s approaches, which catalyzes the integration of STIP into the Mission’s projects and activities. ● Both Missions have a history of integrating STIP. Technical teams at both the West Africa Regional and Uganda Missions have been integrating STIP in their projects and activities for decades. Several staff members in both Missions noted that STIP is part of good project design, and hence technical teams routinely integrate STIP in their projects and activities. In addition, the Missions’ IPs play a role in integrating STIP, as the projects they implement include various STIP-based activities. ● The Lab has played a role in integrating STIP. The Lab’s services have been applied in the project design and implementation phase of the Missions’ Program Cycles, which has contributed to STIP integration. The GeoCenter has provided services for the development of the RHO’s PAD in the West Africa Regional Mission. The GIS specialist at the Uganda Mission, trained by the Lab, also performed mapping and analytical services that supported STIP integration in all five PADs currently in development at the Mission. In addition, as noted in the previous chapter, many Lab services have been applied in the activity design and implementation phase of the Missions’ Program Cycles. STIP Performance Evaluation Final Report: September 2017 | 21 c. Organizational Level STIP is integrated at the organizational level as a result of the Missions’ context, influence, and expertise of certain staff, and the Lab’s support. The context at the West Africa Regional and Uganda Missions enables STIP integration. The MD of West Africa Regional explained that the creation of cross-cutting teams supported STIP integration at the organizational level, even though this effort was not specific to STIP. He further noted that these teams (also called Local Solutions teams) focus on technical issues as well as capacity building. Therefore, even if one individual on the Local Solutions team is not up to speed, other team members have the required expertise to help the team accomplish its STIP-related work. At the Uganda Mission, the Deputy MD explained that increasing STIP integration at the organizational level requires a Mission-wide cultural shift to help staff gain a better understanding of STIP: “You need a critical mass here who understand why it’s important to look at STIP. I say we’re trying because we have a certain level — people who naturally gravitate to change and new things — that’s kind of spotty. You start with those people but you want to go deeper. That takes time and a deliberate approach, and that’s where you need leadership. With know-how, I bring it back to gender. Everyone in the Mission might be committed to creating gender-equitable activities, but if you don’t know how to do that, it remains a couple of paragraphs on paper. So what does it mean to do STIP? That’s a different thing, and it will take time.” Certain Mission staff are integral to integrating STIP. Staff in both Missions who took part in fellowships at the Lab (i.e., STIP Advisor/Fellow, GIS specialists, and FSNs) contribute to integrating STIP at the organizational level. The STIP Advisor/Fellow help the technical teams in their respective Missions identify areas of technical work where Lab services might be of value. The Director of the Regional Contracts Office (RCO) at the West Africa Regional Mission noted that as the STIP Advisor continues to encourage staff to think about STIP in their work, this focus will contribute to STIP integration over time. Additionally, in both Missions, the GIS specialists support STIP integration by providing mapping and visualization services to technical offices, and the Lab-trained FSNs help technical offices apply a STIP lens to their work. PDOs with STIP capacity play a role in integrating STIP in the project-design phase. The PDO in Uganda served as the initial STIP PoC for the Mission, and thus knows the Lab and its services well. In her current role, which requires involvement in the design of projects, she brings a STIP focus to the Mission’s work across technical teams. A Health Office staff member affirmed this, noting that the PDO offers the Lab as a resource and facilitates introductions to other pertinent staff members at the Lab. Concurring, a technical office staff member explained: “The Project Development Officer is also a STIP PoC — she doubles, and that’s a plus. She gets to be in each and every design conversation, and can make sure that each design walks the talk with STIP, and can help identify elements of STIP that should be included in each different activity design.” STIP Performance Evaluation Final Report: September 2017 | 22 The Lab has played a role in integrating STIP. The RPO director at the West Africa Regional Mission said that the Lab has played a role in integrating STIP at the Mission’s organizational level by providing Digital Development trainings, as staff applied their learning from these trainings in their work Mission-wide. He added that Mission staff would also benefit from getting the combined Lab and Office of Acquisitions and Assistance (OAA) iDesign workshop, as well as STIP Integration training from the Lab in the fall (2017). The Lab RM for the Uganda Mission held a similar view, stating that the Lab’s services have contributed to STIP integration at the organizational level as the Lab helps Mission staff view their activities through a STIP lens. 2. Effective Strategies, Methods, and Enabling Environment for STIP Integration The evaluation found several effective strategies for STIP integration in both Missions, as well as the factors in Missions’ operating environments that contributed to this integration. Supportive Mission leadership is especially effective for STIP integration. Having engaged leadership that buys into and promotes STIP was the most important factor for integrating STIP in the Missions. The Uganda Mission’s MD clearly articulated the effectiveness of this approach for STIP integration: “The most important thing, as in many organizations, is enlightened senior leadership. The previous MD and I talked about it, felt it was important, and started pushing. That trickles down to office directors and their staff. We still have a long way to go, but we keep pushing the envelope from our level and then there’s a responsiveness.” The MD of West Africa Regional also stressed that Mission leadership and technical teams really enable STIP integration. He observed that there were directors in the regional offices who were playing a prominent role in integrating STIP in their work. Other staff members in both Missions emphasized the role of leadership in integrating STIP. The RPO director at the West Africa Regional Mission said that the MD has a keen vision for STIP. The RHO director concurred, noting that STIP is front and center because it is emphasized by Mission leadership who want to cast the Mission as a center for STIP for the region. The RPGO director agreed that Mission leadership is supportive and proactive in thinking about how to advance the Mission’s work using STIP. The RCO director also noted that the RFO had made STIP a clear priority, and senior management across the Mission embraces it. Echoing these views, a member of the RHO team asserted: “[Support for STIP integration] comes from leadership. In my opinion, it’s been successful because of leadership — right from the MD’s level. The MD, my Director [in the RHO], and my colleagues in the Health Office are very interested in STIP. That’s the main motivating factor.” Staff at the Uganda Mission also stressed the importance of Mission leadership as well as an open culture in promoting STIP. One staff member believed that the openness at the Mission enabled technical teams to try out new approaches in their work, and explained how the Mission provides its IPs the space to STIP Performance Evaluation Final Report: September 2017 | 23 test new technologies. Another staff member in the Health Office affirmed that the Health Office director and the Mission leadership offer strong support for STIP integration; however, there is a need for staff to present evidence prior to implementing a new STIP approach. Lab staff complement the STIP focus of Missions’ leadership. The Lab RMs for both Missions also highlighted leadership as vital to STIP integration. The RM for the West Africa Regional Mission emphasized that Mission leadership, in coordination with the STIP Advisor and the supervisor, the RPO director, were very important factors in STIP integration. The RM for the Uganda Mission succinctly stated: “We’re only there as much as we are because the leadership is so in favor of it.” She also suggested that, while it is essential to have the MD on board to get things done, the Mission’s office directors can really help move STIP forward within their teams. Other Lab staff spoke of the role of Mission leadership in integrating STIP in the Missions. A GeoCenter staff member who engages with the West Africa Regional Mission said that having the support of the MD and the RPO made it easier to provide Lab services and integrate STIP in the Mission. The director of the GeoCenter reported that the Uganda Mission has benefited from successive leadership teams who have been willing to try something new. She said: “I don’t know if that culture was started by a particular leader who continued or what . . . but something about that Mission . . . they’re willing to try things differently. By the time the Lab was created, there was great momentum already in that Mission. There seems to be a culture in that Mission that it’s willing to take risks and try new things.” A LWP 2 team member who had worked with the Uganda Mission agreed that leadership at all levels is critical: “Leadership in the Mission believes in STIP, so that has helped shape the entire structure of the role of digital inclusion in their strategy, and then in their work in different sectors. The MD has captured a unique understanding of the opportunity in Uganda. The leadership buy-in is a pre￾criteria but also a lesson learned. Leadership is also not just the MD, but also technical leads and office directors need to be involved and buy in.” Building relationships, aligning services to Mission needs, and TDYs contribute to STIP integration. As noted in the Background chapter, the coordination and relationship between the Lab and the West Africa Regional and Uganda Missions have improved. Some Lab staff stressed the importance of building relationships with the Missions so that the Lab is more cognizant of Missions’ priorities and needs, and can continue collaborating with Mission staff to contribute to STIP integration. Several Lab staff recognized that understanding Missions’ needs, and aligning the Lab’s services to these needs, would contribute further to STIP integration. An Activity Manager for LWP 1 who engages with the West Africa Regional Mission noted the importance of “developing a good relationship with the PoC that champions the work [that the Lab does]. Visiting and meeting the Mission staff is also helpful to build personal relationships. Aligning LWP 1 objectives with the Mission’s objectives is also key.” Other Lab staff, such as the Senior GIS analyst in CDD, explained how additional, initially unplanned TDYs played an important role in STIP integration: “You need more presence beyond the needs assessment and the two TDYs we did. Our Director went on the second TDY and extended her trip to be part of the CDCS planning STIP Performance Evaluation Final Report: September 2017 | 24 during a retreat. That was a key learning — sometimes it’s hard to implement things from Washington, D.C.” Challenges remain in integrating STIP. Not all staff in the two Missions felt that STIP was being integrated as much as it should be in their offices. Both the director and deputy director in the RPGO at the West Africa Regional Mission said that they try to keep abreast of innovations, partnership opportunities, and technology in their field, and while everyone on the team supports STIP, they need support employing STIP approaches in very specific contexts, such as programs on CVE. An EGO staff member in the Uganda Mission stated that even with leadership support, STIP integration can be challenged by the time limits of Mission staffs. She explained: “Honestly, though, it is a question of time if we integrate STIP in our work or not. If staff don’t have time, they cannot integrate STIP in their work. STIP is thinking outside of the box, and when staff have time and the Mission leadership (including office directors) cares about it, then we do think about it and include it in our work.” 3. Lab Services with the Most Value in Integrating STIP West Africa Regional Mission staff did not specify any particular Lab services that had the most value in integrating STIP in the Mission’s work. Nonetheless, as was discussed earlier, Mission staff recognized a variety of Lab services that contribute to integrating STIP at the project and activity levels. In the Uganda Mission, multiple Mission staff asserted that the vulnerability mapping and Futures analysis led by the GeoCenter were valuable in helping the Mission develop its CDCS. Mission staff also acknowledged their GIS specialist, trained by the GeoCenter, who is helping in the development of all five of the Missions’ PADs, LWP 2 activities (which have made a big contribution to the work of EGO), iDesign workshops that have imparted useful skills to Mission staff, and the M&E buy-in SPACES, which is supporting the Health Office. Additional STIP Services Wanted from the Lab This chapter addresses the following evaluation question: What Additional STIP Services do Missions (and their partners) want from the Lab? In asking this question, the evaluation team sought to identify current or new offerings from the Lab in which the Missions and their partners were most interested. Several themes emerged around the desired services, while the specific suggestions were unique to the interviewee, their work, and the Mission context. Overall, staff from both the West Africa Regional and Uganda Missions requested demand-driven services, support in increasing their knowledge of STIP, more of a few services in particular, and some new offerings. Missions request more demand-driven services. Two overarching threads emerged in the requests Mission staff shared around accessing Lab services: 1) reducing the time burden on Mission staff in STIP Performance Evaluation Final Report: September 2017 | 25 working with the Lab, and 2) narrowing the focus of Lab offerings to align with Missions’ strategies and prioritize Missions’ needs. A member of the RPGO at the West Africa Regional Mission articulated this challenge: “One of the obstacles in accessing [the Lab’s] tools is just the increased time it takes to research them, determine if they’re right, and then do the transactional/procurement piece to get them.” The STIP Advisor (in West Africa Regional) and STIP Fellow (in Uganda) play key roles in connecting Mission teams to Lab services. However, PO staff in both Missions are challenged by how much time it takes to funnel the large volume of Lab services and connect these services to the appropriate programmatic needs of technical teams across the Mission, and expressed frustration with continuing to receive offerings from the Lab that do not align with the Missions’ strategies. A member of the West Africa RPO explained: “[The Lab needs] to realize strategically what we’re doing and focus what to do with us. . . . Instead of hitting us with many ideas, link with what we’re doing.” Lab staff will need to maintain an awareness of each Mission’s context, strategy, and needs, and ensure that the services offered align with the Mission’s objectives. This is an essential backdrop for the additional Lab services requested below. Mission staff want to increase their STIP knowledge. Several Mission staff expressed a desire for the Lab to disseminate more learning around STIP. The STIP Repository will be critical to this effort, but for now, the Repository is intended to house all of the Lab’s knowledge products on one online site, and additional synthesis and dissemination of this learning is beyond the current scope. Mission staff suggested a number of effective ways to collect and share STIP learning, including surveys, periodic reports, forums, and open platforms for ongoing collaboration. A few staff at the West Africa Regional Mission suggested that the Lab provide additional capacity building in STIP generally, to enable Mission staff to more effectively integrate STIP as they design their projects and activities. A member of the PO argued that: “STIP 101 is necessary . . . [Mission staff] want the orientation, they just don’t have the time to mosey around the Lab website. It could be short and required.” Several staff requested some type of periodic briefing or report on STIP happenings, by country and by sector, to help them stay up to speed in their field, keep tabs on innovations that could be relevant to their work, and learn how to better integrate STIP. The Lab could also help facilitate peer-to-peer learning around STIP by holding STIP forums for USAID staff, particularly including the FSNs who have participated in the Digital Development or Talent Exchange Fellowships with the Lab. A few Mission staff hoped the Lab could help them better share information and collaborate with partners and stakeholders on projects by creating an open platform to enable this exchange. Staff in both Missions felt that the Lab was well placed to facilitate information-sharing and enable more learning around STIP, across the Agency and its technical areas of work. Some suggestions for how this knowledge exchange could take place, such as through periodic briefings about STIP, could be a relatively light lift for the Lab and provide valuable insights that would support Missions in integrating STIP. STIP Performance Evaluation Final Report: September 2017 | 26 Mission staff want more of particular Lab services and some new offerings. Staff in both Missions requested continued assistance from the Lab with design and analytics training and services that teams found particularly useful for employing STIP in their work. Mission staff also suggested expanding the Lab’s services to conduct more Grand Challenges in the field, encourage experimentation, draw out local talent, and promote technological innovations. Mapping and analytical services from the GeoCenter are valued Lab services; however, some of the GeoCenter’s analyses can only be conducted by Lab staff in Washington (which may be due to limitations with technology or capacity in Missions), so the data soon become outdated. A member of the PO in Uganda requested Lab support in updating assessments periodically. He stated: “Things like vulnerability mapping should not be a one-off thing. We should do it again and see how it has changed, but we don’t have the capacity to do it here on our own.” In the same vein, this staff person requested capacity building from the Lab so he could do more in-house data analysis, in country. Additionally, some Mission staff raised the crucial role the Lab plays in analyzing and linking work across sectors; they felt that more of this support is needed going forward. Many staff in the Uganda and West Africa Regional Missions expressed a desire for services from the Lab that are beyond its current offerings, but could possibly be added in the future. Multiple staff adamantly asserted a need for the Lab to assist Support Offices with employing STIP in operations. This was most prevalent in the West Africa Regional Mission, where staff in the Front Office, Financial Management Office, and PO addressed this need. A staff person in the Financial Management Office stated: “Frankly I think the operations side of the Agency could use a lot of STIP, and it’s a pity that the Lab isn’t focusing more on that as well.” A member of the West Africa Regional PO requested that the Lab help foster innovation and experimentation in country with Grand Challenges or a similar mechanism. The limited flexibility of Agency systems and processes also prompted some Mission staff to call for the Lab’s support with sourcing technical experts who could offer individualized solutions for their work. These experts could provide targeted assistance in addressing a particular development problem by injecting new ideas and helping technical teams take their programs to the next level. A staff person in the RPGO at the West Africa Regional Mission described this need: “What would be useful would be [for the Lab] to source someone who could come give individualized support that’s specific to our needs. . . . We’ve been thinking of the idea of launching a lo-fi mobile game around [CVE]. I’d love to talk with someone who has worked on video games for behavior change. . . . You’re never going to foresee the individualized needs that all these technical offices have throughout the Agency, so don’t try to develop mechanisms to cover all things. Create mechanisms to find an individualized solution.” Some staff in the Uganda Mission requested that the Lab bring STIP closer to the local level by establishing satellite labs at a few strategic Missions. These staff explained that the envisioned satellite labs would enable the Lab in Washington to learn more about STIP in other country contexts, as well as help the Missions facilitate local ownership of STIP approaches in development. STIP Performance Evaluation Final Report: September 2017 | 27 STIP Integration for Implementating Partners This chapter covers the fourth evaluation question: How have the Mission’s IPs Projects and Activities Integrated STIP? The evaluation found that the Missions’ partners integrate STIP into their projects and activities according to their mandates, as well as the appropriateness of STIP approaches for their technical work and local context. These findings are based on the feedback from the West Africa Regional Mission’s three regional partners and four IPs, and 12 of the Uganda Mission’s IPs. See Annex 6 for summaries of the regional partners and IPs who participated in this evaluation. Regional Partners of the West Africa Regional Mission The regional partners of the West Africa Regional Mission primarily integrate STIP in their projects and activities as their mandates and objectives require them to do so. They rely on the technical expertise within their organizations, and as provided by their national member bodies, to enable this integration. STIP integration happens as a result of partners’ technical expertise. The Mission’s regional partners rely on the technical staff they have in their organizations, along with technical expertise provided by certain national bodies from their membership base, to enable STIP integration in their projects. The Executive Director and CEO of Conseil Ouest et Centre Africain pour la Recherche et le Développement Agricoles (CORAF) affirmed that some staff have the skills needed to integrate STIP into their work: “We get technical expertise on STIP from our national research institutions in our consortium. For example, under ECOWAS [Economic Community of West African States], the objective of the West Africa Agricultural Productivity Program [WAAPP] project is to generate and disseminate improved technologies in the participating countries' top priority areas. Under WAAPP, we have different Centers in partner countries, which also bring expertise in STIP to CORAF. Some institutions across the world are also participating in WAAPP and they also provide technical expertise.” The head of the Information and Research Department (IRD) at Agrometeorology, Hydrology, Meteorology Institute (AGRHYMET) in Comité Permanent Inter-Etats de Lutte contre la Sécheresse dans le Sahel (CILSS) said that the work of IRD includes components of STIP, as they conduct scientific research. They also collect and disseminate to their members other research that involves the use of science, technology, and innovations to address issues such as food security, desertification, and water management. He added that his department has a staff of 45 people, and many of them are experts in the fields of agriculture, remote sensing, GIS, etc., who contribute to the research they undertake. The Lab has played a role in integrating STIP. The Head of the HIS at WAHO recognized the involvement of USAID in their work, noting that the West Africa Regional Mission and the Lab (through LWP 1) had recruited and funded two staff members to join their team. One of these staff members was STIP Performance Evaluation Final Report: September 2017 | 28 in charge of database management, and the other was involved in analyzing information and writing reports. Challenges remain to integrate STIP across vast regional organizations. Staff of all the regional partner organizations said that STIP is an essential part of their work, but expressed some challenges in discerning the strategies and methods that are most effective in integrating STIP. The Head of HIS at WAHO raised some larger issues that prevented the organization from realizing some of their STIP objectives: “We have a mission to assist 15 countries to improve their health systems, harmonize their health policies, etc. To do this, WAHO needs health information to make decisions, but at the country level, we are short of this information. Because of this, our orientation is not only to collect data, but also to assist countries to improve their national health information systems. If they can improves their HIS, then we can do our work better and harmonize the health policies . . . but all the countries are at very different levels, so it is not easy to have harmonization.” The West Africa Regional Mission’s objectives entail building the STIP capacity of its regional partners. Several staff in the West Africa Regional Mission suggested that the Lab could offer support by providing technical assistance and STIP capacity building for the Mission’s regional partners. A staff member in the RFO noted: “[I’d like the Lab to spend] as much time or more with our partners — not contractors, but the regional organizations that will be around forever, that we work to support. If we can get our regional local partners to adopt STIP, we’re halfway through our objectives.” Implementing Partners of the West Africa Regional and Uganda Missions IPs of both missions proactively integrate STIP in their projects and activities as their scope of work allows, and as long as it is feasible to do so within their local context. The IPs also rely on their own expertise and support provided by staff in their organizations to integrate STIP in their work. STIP integration is based on Implementing Partners’ technical scope of work and feasibility. Staff of both Missions’ IPs were clear that integrating STIP in their projects and activities depends on the technical scope of their project and the feasibility of STIP approaches and tools for their activities and target areas. Staff from IP organizations also stated that they primarily rely on the technical expertise of their own staff to integrate STIP in their projects and activities. Two Chiefs of Party (CoPs) from the West Africa Regional Mission’s IPs, including the CoP for the family planning (FP) project being implemented by EngenderHealth and the M&E Officer for the Regional HIV/AIDS Prevention and Care Project(PACTE-VIH) implemented by FHI360, said that they start thinking of integrating STIP in their project activities in the design phase. The M&E officer gave an example of how they had integrated an mHealth-related activity in their project. She explained: STIP Performance Evaluation Final Report: September 2017 | 29 “The stigma of homosexuality is very high so there was a hidden population. The research showed most of them were online and technology savvy. So we devised an approach: a social media intervention and sending SMS messages on HIV prevention to gay websites. We also tried to link this population to brick and mortar facilities where they can get tested and get condoms. We also used SMS to reach other beneficiaries on HIV prevention, and another set of messages for those who already have HIV and are on treatment to take their medication.” The CoP of the Commodity Production and Marketing Activity, funded by USAID/Uganda, spoke of innovating in their work, explaining: “Right from the design, we are focused on innovations that could help the farmers. . . . We think about cheaper and more effective innovations. For example, we designed a mobile dryer for crops. USAID funded it, but we designed it and produced it, and now it is being sold to farmers.” Another IP of the Uganda Mission, Management Sciences for Health, implements the Track Tuberculosis Activity (TRACK TB). The CoP and Deputy CoP of TRACK TB affirmed that they innovate to address challenges in realizing their program’s objectives, and spoke to the Mission’s support for this approach: “USAID provides technical support and space to incorporate STIP into our work. We come with a lot of innovative ideas. You get green light on innovative ideas, and USAID has been flexible to allow us to try out new ideas and keep them updated. For us to start thinking of innovations, we think of the challenges we want to answer.” STIP Performance Evaluation Final Report: September 2017 | 30 CONCLUSIONS AND RECOMMENDATIONS Increasing Understanding, Capacity to Use, Application, and Integration of STIP in Missions There are mixed views on the Lab’s role in increasing understanding of and capacity to use STIP. Some staff felt that the Lab had played a role in their understanding of and capacity to use STIP. Staff in both Missions who had taken part in the Lab’s Talent Exchange Fellowship reported an increased understanding and capacity to use STIP as a direct result of the fellowship. Similarly, Mission specialists, such as those in GIS, also benefitted from the support and training received from the Lab. Additionally, senior leadership and several staff in both the Missions recognized the role of the Lab in contributing to this increase. However, they also emphasized that the Mission context, including Mission leadership and office directors’ support of STIP, along with the STIP work already implemented by the technical teams, enabled an increase in understanding and capacity to use STIP. On the other hand, a handful of staff members in both Missions did not believe the Lab’s work had an influence on STIP at their Missions, or thought it was too early to ascertain the Lab’s role. Mission staff want to build their capacity to use STIP. Multiple staff across technical teams in both Missions seek additional training and services from the Lab to help them operationalize STIP in their work, particularly in design and analysis. For some services, staff want additional knowledge transfer and capacity building to enable them to further apply STIP in their work without being reliant on software or staff expertise based in Washington. This echoes findings from the STIP Integration Report, which recommended encouraging STIP professional development opportunities for staff.5 The STIP Integration Report also found that there is no “one way” to build STIP capacity at a Mission; rather, a variety of approaches are needed. Requests for additional support from the Lab with STIP-related design and analytics services included a STIP integration training for Mission staff, extra support for local capacity building for GIS (including more software and training for GIS specialists at the Mission level to run assessments at periodic intervals), and guidance on analyzing and linking STIP opportunities across technical sectors. The Lab’s role in increasing the application of STIP in the Missions’ Program Cycles also varied. With the Uganda Mission, the Lab provided ample support to the Mission in applying STIP in its strategic planning through the STIP Assessment, vulnerability mapping, and the Futures analysis. The evidence and findings generated from these efforts contributed directly to the development of the Mission’s CDCS. Due to the timing of the engagement between the Lab and the West Africa Regional Mission, however, the Lab could not play a role in the Mission’s regional strategic planning. 5 Lessons from the Field: A Report about Integrating Science, Technology, Innovation and Partnerships (STIP). January 2017. STIP Performance Evaluation Final Report: September 2017 | 31 The Lab has also played a role in increasing the application of STIP at the project design- and implementation phases of both Missions’ Program Cycles. For example, the GeoCenter has helped the RHO in the West Africa Regional Mission develop its PAD. In Uganda, the GIS specialist (trained by the GeoCenter) is contributing to the development of all five Mission PADs. FSNs in both Missions who took part in fellowships with the Lab also bring their STIP knowledge and experience to bear in project design work. In the activity design and implementation phases, a variety of Lab services have been employed by different offices in both Missions. For example, in the Uganda Mission, the LWP 2 team integrates a suite of coordinated digital technologies into the FtF initiative; in the West Africa Regional Mission, the Environment Office in REGO obtained $250,000 in funding from the Lab to develop the Land Use Land Cover atlas. The application of STIP in M&E has been limited. While the Health Office at the Uganda Mission has bought into one of the Lab’s MERLIN mechanisms, no Lab services related to M&E have been employed at West Africa Regional Mission. The two Missions (and all operating units across the Agency) do not have separate or specific budget line items for STIP, but this is not a barrier for the application of STIP in the Missions’ projects and activities. The position of the STIP Advisor at the West Africa Regional Mission and the STIP Fellow at the Uganda Mission (who work closely with the Lab), however, do serve as key resources for the Missions to promote STIP integration across their portfolios. USAID’s broader development policies, including the Agency policy framework, USAID Forward, and the ADS, also address and encourage the integration of STIP, but this is not Mission-specific. Lastly, neither Mission is conducting any Mission-wide STIP learning, and hence, it is difficult to demonstrate examples of learning and adapting relative to STIP. A supportive, enabling environment in combination with targeted Lab engagement increased STIP integration across the Missions. Various services and resources provided by the GeoCenter, through LWPs, and research grants, are the most utilized, and have contributed to STIP integration across both Missions. The interviewees believe that STIP integration also occurs when the Lab aligns their targeted engagement with Mission strategy and the technical teams’ projects and activities. Additionally, effective engagement happens when done in person via TDYs. STIP points of contact in both the Mission and the Lab have also been instrumental in promoting STIP integration through their coordination role. As highlighted above, supportive leadership contributed to the increase in STIP integration across both Missions by encouraging staff to address development challenges with STIP solutions. In addition, both Missions’ staff and implementing partners played a pivotal role in STIP integration regardless of the engagements with the Lab. These individuals and organizations have been incorporating STIP approaches into their projects and activities for decades. STIP Performance Evaluation Final Report: September 2017 | 32 Providing Services Aligned with Missions’ Priorities Mission staff request more demand-driven services from the Lab. Mission staff have little time to familiarize themselves with the Lab’s many services and ascertain where and how these might fit into their portfolio of work. Additionally, many Mission staff broached the unfocused offerings coming from the Lab to the Missions, though they noted that this challenge has improved over time. However, Program Office staff in both Missions expressed frustration that the Lab continues to offer services that do not align with the Missions’ strategies. Lab staff must remain aware of each Mission’s context and priorities in order to narrow the focus of their offerings to align with Missions’ strategies and needs. Any efforts to reduce the burden of time on Mission staff in working with the Lab would be welcomed and facilitate further engagement. Mission staff want to learn from other Missions’ experience using STIP approaches in their work. Mission staff want to keep up with “what’s new and exciting” in STIP across development sectors and learn from their peers’ experience, including their achievements and challenges. The Lab Resource Portal (currently in development) will be critical to this effort; however, the portal is intended to house all of the Lab’s knowledge products on one website, while additional synthesis and dissemination of this learning is beyond its current scope. The Lab could facilitate this knowledge exchange around STIP work through any one of a number of methods, such as periodic briefs, online forums, and open platforms for collaboration. Increasing STIP Integration for Mission Partners STIP integration depends on many factors for implementing partners. Staff from both Missions’ IPs were clear that integrating STIP in their projects and activities depends on the technical scope of the work and the pertinence of STIP approaches and tools for their activities and target areas. The West Africa Regional and Uganda Missions also encourage their IPs to incorporate STIP in their project activities, and the IPs praised the Missions for providing them the space to incorporate innovations in their activities. The Lab can support the West Africa Regional Mission in its mandate to build regional partners’ capacity. The regional partners of the West Africa Regional Mission integrate STIP in their projects and activities as their objectives and scope of work require them to do so. The West Africa Regional Mission has a specific objective to support and build the capacity of its regional partners, many of whom have STIP-specific mandates. Therefore, the Mission’s leadership and multiple staff asked for additional support from the Lab in building the STIP capacity of their regional partners. Mission staff acknowledged that, given their limited bandwidth, they are typically not able to provide this level of support to their partners, and several staff felt that the Lab could offer a few discrete, targeted services that would be a great benefit to their regional partner organizations. STIP Performance Evaluation Final Report: September 2017 | 33 Recommendations To increase understanding, capacity to use, and the application and integration of STIP at Missions, the evaluation team recommends that the Lab: ● Create an online module of the STIP integration training to make it easily accessible for Missions around the world. Some Mission staff are not clear about the different components of STIP and how they can incorporate them in their work, so the Lab should also request Missions’ Front Offices to make the STIP integration course compulsory for all staff. ● Solicit feedback from Mission staff regarding additional STIP knowledge gaps. Specifically, the STIP Advisor/Fellow and RMs could help gather feedback across both Missions. Then, Mission-specific trainings can be developed by the relevant Lab staff to address the topics in highest demand. ● Examine where additional GIS expertise might be needed in Missions in order to provide further support given the high demand for this type of service. GIS specialists play important roles in integrating STIP in the Missions, especially in the PADs and project design. ● Continue hosting a STIP forum annually or bi-annually to share lessons learned and best practices around the application of STIP in strategies, projects, and activities. This forum could be held in key regional missions (e.g., Southern Africa Regional) with an open invitation to bilateral missions in the region. ● Explore the possibility of working with relevant Mission teams to encourage STIP learning via a learning agenda or other mechanisms, such as an internal STIP integration competition, brown bags, etc. ● Explore how the Project Development Officer in USAID Missions can obtain a better grasp of the Lab’s services and resources, as well as comprehend where they might provide the most value in their Mission’s Program Cycle. ● Adjust the assessment approach for identifying openings for targeted assistance based on Missions’ needs. Specific Lab teams can then determine which Lab resources might address this request, and offer the Mission options for services based on the Lab team’s resources, bandwidth, and expertise, as well as the Mission’s time frame. ● Establish a plan to capture knowledge before the end of the STIP Advisor/Fellow’s term in order to understand how these positions have supported the engagement between the Mission and the Lab. ● Engage with Mission Directors to explain how Missions can benefit from Lab services and resources to address their needs and priorities. This would be most beneficial if led by Lab leadership. ● Develop a concise memo or brief report for Mission Directors and Deputy Mission Directors that highlights how Missions such as Uganda have integrated STIP in their CDCS, and identify Lab services that have contributed to this integration. ● Engage Missions with strong leadership support and buy-in for STIP, as this is a key enabler for STIP integration. Lab leadership should consider this a key criterion when deciding to engage with future Missions. STIP Performance Evaluation Final Report: September 2017 | 34 To provide services that are aligned with Missions’ priorities, the evaluation team recommends that the Lab: ● Conduct a strategic review to examine the extent to which its services and resources align with Missions’ strategies. This review should involve all of the Lab’s centers and result in a strategic framework for the Lab’s engagement with the Missions, based on the Missions’ priorities and needs. ● Discontinue submitting requests for concurrence on activities that are not clearly aligned with a Mission’s strategy or objectives. Lab leadership must ensure that Lab staff understand the priorities and needs of the Missions with which they work, and confirm that the Lab’s initiatives are in alignment with the Missions’ strategies. ● Disseminate STIP learning products to Missions that highlight specific examples of STIP successes and challenges, with tips that cut across various types of work, development sectors, and countries. Consider how to better utilize Lab Field Notes and other existing dissemination channels for this purpose. To increase STIP integration for Mission Partners, the evaluation team recommends that the Lab: ● Identify specific Lab services that can contribute to addressing the STIP mandates of Missions’ partners. This should be done jointly with the relevant Mission staff. ● Explore whether Lab staff can conduct Temporary Duties to work with Regional Missions’ regional partners, in conjunction with relevant Mission staff. ● Conduct STIP assessments of Missions partners’ work to identify opportunities for increased collaboration with the partners. STIP Performance Evaluation Final Report: September 2017 | 35 ANNEX 1. Evaluation Scope of Work STIP Integration Buy-in Evaluation Activities for FY2017 Expected Period of Performance: October 1, 2016-September 28, 2017 Statement of Work The SOW includes a list of anticipated, illustrative activities to support the CAI work within the Lab and with all USAID OUs. The contractor shall include these activities in their response and Work Plans. The contractor may also propose alternative or additional innovative solutions and/or activities to meet the stated objective/goal. Anticipated period of performance for this SOW is October 1, 2016-September 28, 2017. Because there is already an existing buy­in from the Lab, recruitment and scoping of evaluations may take place prior to October 1, 2016. Activities 1. Conduct Program Performance Evaluations Following USAID policy guidance on evaluation, the contractor will conduct approximately 1­2 program performance evaluations of the Lab. Approximately two of the evaluations will require travel to Missions that the Lab has supported with STIP integration during the Program Cycle. This will likely include one Next Generation partner in Africa and one Knowledge partner in Asia. The evaluations will be linked to the achievement of Agency integration’s objectives and/or its intermediate results. While the evaluations under this SOW will not address this question below, we have noted it for the purpose of understanding the Lab’s theory of change and overarching question about the value-add of STIP: ● Does the Lab’s effort to increase the capacity across the Agency to understand, apply, and integrate STIP tools and resources throughout the Program Cycle contribute to greater development impact than traditional programming? Illustrative Evaluation Questions • LEARN will work with the Lab to determine specific evaluation questions for each evaluation. Illustrative evaluation questions could include: • Has the Lab strategically and operationally contributed to the integration of STIP tools and approaches at the program, strategy and organizational levels across the Agency? STIP Performance Evaluation Final Report: September 2017 | 36 Additional illustrative evaluation questions could link to Lab’s Agency integration intermediate results: Table 1: Intermediate Results and Corresponding Illustrative Evaluation Questions Intermediate Result Illustrative Evaluation Questions IR1 — Agency understanding of STIP tools and resources increased ● How has the Lab contributed to an increased understanding of STIP through the Agency? IR2 — Agency application of STIP throughout the Program Cycle ● How have Lab tools and approaches affected the uptake of STIP tools and approaches utilized by Operating Units throughout the Program Cycle? ● What strategies and/or methods have proved most effective in integrating STIP among our Next Generation partner Missions? In response to the Activity Manager’s written notification of an evaluation task — which identifies technical parameters including evaluation purpose, intended users of the evaluation, key questions for evaluation, background information about the projects or programs to be evaluated, proposed methodology, data collection methods, the intended evaluation team and intended evaluation schedule — the contractor shall develop a written Evaluation Work Plan and Design for each evaluation within [1­2 weeks] of the written notification of an evaluation task. Reporting and Administrative Deliverables All STIP Integration Support Contract written reports and administrative deliverables should be submitted to the USAID PPL/LER LEARN COR, the Lab/CAI director, in addition to the Lab/CAI STIP Integration Support Contract Activity Manager. The written reports and deliverables will be associated with, but distinct from, LEARN’s overall deliverables to include the following: Specific Deliverables under “Activities”: 1. Evaluation Work Plan: The Evaluation Work Plan will include: 1) the anticipated schedule and logistical arrangements; and 2) a list of the members of the evaluation team, delineated by roles and responsibilities. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID ADS in Chapter 200. LEARN anticipates two team members per evaluation working near a full­time level of effort. The evaluations will be sequential, rather STIP Performance Evaluation Final Report: September 2017 | 37 than simultaneous (up to four months per evaluation), enabling both team members to work on each of the (up to) three evaluations. 2. Evaluation Design: The Evaluation Design will include at a minimum: 1) a detailed evaluation design matrix that links the Evaluation Questions in the written notification of an evaluation task to data sources, methods, and the data analysis plan; 2) draft questionnaires and other data collection instruments or their main features; 3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or a sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); 4) known limitation to the evaluation design; and, 5) a dissemination plan. USAID offices and relevant stakeholders are asked to take up to 10 business days to review and consolidate comments through the Lab/CAI STIP Integration Support Contract Activity Manager. Once the evaluation team receives the consolidated comments on the initial evaluation design and Work Plan, they are expected to return with a revised Evaluation Design and Work Plan within 10 business days. 3. In-briefing and Exit Briefing: The evaluation team is expected to provide in-briefing to CAI and, if requested, for any visits to countries with USAID Missions for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and Work Plan, and/or to adjust the SOW, if necessary. The evaluation team is expected to hold a final exit briefing prior to leaving the countries with USAID Missions to discuss the status of data collection and preliminary findings. This presentation will be scheduled as agreed upon during the in-briefing. 4. Mid­term Briefing and Interim Meetings: The evaluation team is expected to hold a mid­term briefing with CAI on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in­briefing. If desired or necessary, biweekly briefings or meetings by phone can be arranged. 5. Final Presentation: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in-briefing. 6. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in the “Final Report Format” section. The report will address each of the questions identified in written notification of the evaluation task and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation Work Plan. Once the initial draft evaluation report is submitted, the CAI will have 10 business days in which to review and comment on the initial STIP Performance Evaluation Final Report: September 2017 | 38 draft, after which the Lab/CAI STIP Integration Support Contract Activity Manager will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the CAI will review and send comments on this final draft report within 10 business days of its submission. 7. Final Evaluation Report: The evaluation team will be asked to take no more than 10 business days to respond/incorporate the final comments from the CAI. The evaluation team leader will then submit the final report. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. Final Report Format The evaluation final report should include: executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report should be formatted according to the evaluation report template. The executive summary should be 3-5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The anticipated length of the evaluation reports, not including annexes, is approximately 30 pages. The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: ● The Evaluation SOW and written notification of the evaluation task; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● Sources of information, properly identified and listed; and ● Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts. In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. STIP Performance Evaluation Final Report: September 2017 | 39 The Lab will be responsible for dissemination of evaluation findings, but LEARN can provide a synthesized, visually appealing 1-2 pager per evaluation to support dissemination efforts. Criteria Ensuring Quality of Evaluation Report Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. ● The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. Evaluation reports shall address all evaluation questions included in the SOW. ● The evaluation report should include the SOW as an annex. All modifications to the SOW — whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline — need to be agreed upon in writing by the Lab/CAI STIP Integration Support Contract Activity Manager. ● The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. Other Requirements All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the written notification of the evaluation task, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the Lab/CAI STIP Integration Support Contract Activity Manager. Any revisions should be updated in the written notification of the evaluation task that is included as an annex to the Evaluation Report. STIP Performance Evaluation Final Report: September 2017 | 40 ANNEX 2. Methodology The evaluation employed a qualitative methodology, and data sources included documentation provided by the Lab and the Missions, with KIIs and GIIs with 96 staff members from the Lab, Missions, and some of the Missions’ partner organizations. A purposive sample of interviewees was compiled for the evaluation team to selectively choose staff in leadership positions and those engaged in STIP activities at both Missions as well as some of the Missions’ partner organizations. Interview protocols for the KIIs and GIIs, as presented in Annexes 2.a-2.d, were semi-structured to allow flexibility in the discussion and a natural flow to the conversation. Detailed notes were taken for all KIIs and GIIs, and the evaluation team analyzed this data by deductively coding themes and querying the data using Nvivo, a qualitative data analysis software. There were no major limitations in the evaluation; however, of note is that more Lab services have been implemented in the Uganda Mission, so most Lab interviewees had engagements with the Uganda Mission with fewer engaged with the West Africa Regional Mission. The performance evaluation team consisted of two members: a senior evaluator and a Monitoring, Evaluation, Research, and Learning specialist. The team was also supported by another senior Monitoring, Evaluation, Learning and Research specialist at LEARN who provided oversight of the activity, reviewing and approving all deliverables. The evaluation team members’ roles and responsibilities are included in Annex 5. Documentation A thorough review and analysis of all pertinent documentation provided by the Lab and Missions (such as strategy documents, Work Plans, programmatic and trip reports, and prior assessments and evaluations) was conducted, to enable the evaluation team to: ● Collect information on the various STIP offerings the Lab has provided to the West Africa Regional and Uganda Missions ● Develop the evaluation design and data collection instruments ● Address the evaluation questions A list of this documentation is included in Annex 3.b. Sample Size, Data Collection, and Analysis Qualitative data was primarily gathered through one-on-one KIIs; GIIs were employed where it was more feasible to talk to some staff at the Lab or at the Missions together. KIIs and GIIs were conducted with a total of 96 staff from the Lab, both Missions, and from some of the Missions’ partner organizations. STIP Performance Evaluation Final Report: September 2017 | 41 CAI and the STIP PoCs at the West Africa Regional and Uganda Missions identified the leadership in the Lab and the Missions, key staff in both the Lab and the Missions who are engaged in STIP activities, and the Missions’ IPs that integrate STIP in their projects and activities. With this information, the evaluation team compiled a purposive sample of interviewees for the evaluation of key staff, and reached out to additional staff at the Lab, Missions, and IPs as was deemed pertinent over the course of the data collection phase. The “Data Sources and Collection Methods” table below highlights key personnel roles identified for interviews, as well as the methods used for collecting data. This list applies to the main evaluation question as well as to all the sub-questions. Table 2. Data Sources and Collection Methods Data Sources Data Collection Methods The Lab ● RMs for West Africa Regional and Uganda Missions ● Managers of LWP areas ● Main team members providing STIP tools, resources, and services to Missions ● Main team members providing STIP for LWP areas ● Pertinent senior staff at the Lab Uganda and West Africa Regional Missions ● STIP PoCs ● Staff engaging with Lab services and activities ● Staff involved in STIP for LWP areas ● FSNs who took part in the Talent Exchange or Digital Development fellowships ● Pertinent senior staff at the Missions Implementing Partner Organizations ● Staff from Missions IPs integrating STIP in their projects and activities ● Lab and Missions document review ● KIIs ● GIIs The evaluation team collected data at three sites from March-June 2017, including: ● Global Development Lab in Washington, DC ● West Africa Regional Mission in Accra, Ghana (visited May 8-19, 2017) ● Uganda Mission in Kampala, Uganda (visited May 22-June 2, 2017) The evaluation team attempted to conduct KIIs and GIIs face-to-face with respondents. However, in cases when this was not feasible, the team used the phone or Skype to speak with the interviewees. The STIP Performance Evaluation Final Report: September 2017 | 42 duration of each KII was approximately 45-60 minutes, and each GII was approximately 60-90 minutes. As mentioned above, in total, KIIs and GIIs were conducted with 96 staff as follows: ● The Lab: 36 ● West Africa Regional Mission: 24 ● West Africa Regional Mission regional partners and IPs: 7 ● Uganda Mission: 17 ● Uganda Mission IPs 12 A list of all persons interviewed is included in Annex 4.a. The KIIs and GIIs were semi-structured to allow flexibility in the discussion and a natural flow to the conversation. Detailed notes were taken for all KIIs and GIIs, and opinions of respondents are characterized as such in this report. For analyzing the interview data, a deductive coding scheme was employed whereby the evaluation questions served as the main source of codes for analysis, and additional codes were included as they emerged during the data collection and/or analysis stages. The evaluation team used Nvivo, a qualitative analysis software, for coding themes and querying the data; and also conducted a broader thematic analysis of the data by reviewing the interview notes several times over. Documented notes on all KIIs and GIIs will be submitted in a readable and organized format, and are owned by USAID, pursuant to USAID’s Open Data policy (ADS 579). STIP Performance Evaluation Final Report: September 2017 | 43 ANNEX 3.a. Data Collection and Analysis Tools: Key Informant Interview Protocol for Lab Leadership Date: Name: Title: Office/Department/Mission: Gender: M F Opening: Evaluation introduction and consent form. Introduction The Lab’s STIP definitions include the following: (When providing your responses, please address any one, or a combination, of the four STIP areas that you are working on in the Mission.) ● Science: Channel the technical expertise of scientists and researchers to build local scientific capacity, empowering people with tools for change, and use the evidence from scientific research to drive new policies and programs. ● Technology: Work to increase access to digital financial services and the Internet, in part by strengthening enabling environments. Increase the use of evidence, data, and analytics for better decision-making. ● Innovation: Identify, test, and accelerate new tools that have evidence of impact, sustainable financing, and reach. Increase the adoption of high-impact solutions and the effective use of innovation methods by the Agency. ● Partnership: Work with impact investors to catalyze private capital for early stage businesses and strengthen the environment for entrepreneurship. Manage USAID’s signature public-private partnership model, the Global Development Alliance, and work to catalyze new models for collective action. 1. What is the value of an Next Generation partnership for the Lab and the Missions/OUs? ● Specifically for the West Africa Regional and Uganda Missions? 2. What do you see as the benefits of having LWPs and working with Missions on them? ● Are there more targeted channels to provide Lab services to Missions? STIP Performance Evaluation Final Report: September 2017 | 44 3. What role is the Lab playing in the integration of STIP in the Program Cycle of the Next Generation Missions? ● Is Lab involvement leading to greater impact of STIP in the missions work? Is it too early to discern? ● With IPs? 4. What strategies and mechanisms are most effective in integrating STIP in a Mission? 5. How do you see the Lab strategically building the STIP ecosystem in the Next Generation Missions countries? ● Does the Lab also see this as its mandate? If yes, then what is it doing to help Next Generation Missions achieve their long-term strategic goals? 6. How do you see the future engagement of the Lab with the Next Generation partners and other Missions and OUs? ● Will the model of current engagement continue? If there are budget issues, what will be the ramifications? 7. Any other challenges and recommendations? STIP Performance Evaluation Final Report: September 2017 | 45 ANNEX 3.b. Data Collection and Analysis Tools: Key Informant Interview Protocol for Lab Staff Providing STIP Services to Missions Date: Name: Title: Office/Department/Mission: Gender: M F Opening: Evaluation introduction and consent form. Introduction The Lab’s STIP definitions include the following: (When providing your responses, please address any one, or a combination, of the four STIP areas that you are working on in the Mission.) ● Science: Channel the technical expertise of scientists and researchers to build local scientific capacity, empowering people with tools for change, and use the evidence from scientific research to drive new policies and programs. ● Technology: Work to increase access to digital financial services and the Internet, in part by strengthening enabling environments. Increase the use of evidence, data, and analytics for better decision-making. ● Innovation: Identify, test, and accelerate new tools that have evidence of impact, sustainable financing, and reach. Increase the adoption of high-impact solutions and the effective use of innovation methods by the Agency. ● Partnership: Work with impact investors to catalyze private capital for early stage businesses and strengthen the environment for entrepreneurship. Manage USAID’s signature public-private partnership model, the Global Development Alliance, and work to catalyze new models for collective action. 1. Briefly, what are your current roles and responsibilities? 2. What STIP-related work did you do at the Mission(s)? ● Do you provide substantial (over 8 hours) or light (less than 8 hours) advisory/Technical Assitance (TA) support per project/activity? STIP Performance Evaluation Final Report: September 2017 | 46 Promoting STIP in the Mission 3. How did the Mission(s) get introduced to your STIP-related work? 4. What type of coordination did you have with the Mission(s) for your work? ● What type of coordination did you have within the Lab for your work? Were you engaged with the Lab’s RM for the Mission(s)? ● Was your STIP-related work part of the integrated Work Plan with the Mission(s) (which is coordinated by the Lab’s RM for the Mission)? 5. A STIP Assessment was done for Uganda. Did your STIP work have any connection with this assessment? (For Uganda only) 6. What type of the Lab’s STIP knowledge products (publications and reports, evaluations, datasets, websites, etc.) were provided to the Mission(s) as a result of your work? ● Are you aware of how they are being used? STIP Integration 7. Can you comment on how the STIP-related work you did was integrated in the Mission(s) Program Cycle? ● At the strategic level? In the CDCS or RDCS? ● At the organizational/structural (i.e., new positions, roles of staff, working groups) level? ● At the project level (design and implementation) in the Mission? ● At the activity level (design and implementation) in the Mission? ● In the Mission-level Performance Management Plan (PMP)? Project-level monitoring, evaluating and learning (MEL) plans? Activity-level MEL plans? 8. Different Lab’s STIP services/offerings are being applied in the Mission. [Show visual of Lab service triangle: Knowledge Sharing, Communities of Practice, Communications; Training, Toolkits, and Exchanges; Advisory and Analytical Services; Co-programming; and Focus Mission] ● Where does the STIP work you did in the Mission(s) fit in the triangle? ● In which of the four STIP areas (Science, Technology, Innovation, and Partnerships) is your work being most applied? 9. What role did the Mission(s) ecosystem (i.e., budget, development policy, resources, leadership) play in your STIP-related work? 10. What strategies and methods have been most effective in aiding your STIP-related work at the Mission(s)? ● Who are the key people and what are the key resources that can further promote your STIP-related work in the Mission(s)? STIP Performance Evaluation Final Report: September 2017 | 47 11. Have you gotten any feedback as to what additional STIP services/offerings the Mission(s) wants from your team/department? From the Lab? 12. Can you comment on whether any non-Lab-led STIP work was being carried out at the Missions(s) that was similar to the STIP work you were doing? ● Was there any overlap with your work? Other 13. Can you comment on how your STIP-related work increased understanding of STIP in the Mission(s)? ● Increased capacity of using STIP in the Mission(s)? ● Increased the application of STIP throughout the Mission(s)? 14. Are there any main lessons that your team/department learned from the STIP-related work you did at the Mission(s)? 15. Do you have any additional comments on any challenges and/or recommendations concerning the STIP-related work you did at the Mission(s)? CLA 16. Can you comment on how the STIP-related work you did is aligned with CLA efforts in the Mission(s)? ● What type of collaborative work (internally and externally with the private sector) was done as part of your work? ● Is evidence and learning being generated based on your work at the Mission? ● Is the evidence and learning based on your work being used to inform decision-making at the Mission? To design and implement project/activities iteratively? For adaptive management? Conclusion ● What other relevant documents/materials can you share with us? ● Are you comfortable being contacted for any follow-up questions? Yes/No ● Do you have any suggestions for any other staff we should interview? ● Are there other final comments that you wish to make? STIP Performance Evaluation Final Report: September 2017 | 48 ANNEX 3.c. Data Collection and Analysis Tools: Key Informant Interview Protocol for Mission Staff Involved in STIP Activities Date: Name: Title: Office/Department/Mission: Gender: M F Opening: Evaluation introduction and consent form. Introduction The Lab’s STIP definitions include the following: (When providing your responses, please address any one, or a combination, of the four STIP areas that you are working on in the Mission.) ● Science: Channel the technical expertise of scientists and researchers to build local scientific capacity, empowering people with tools for change, and use the evidence from scientific research to drive new policies and programs. ● Technology: Work to increase access to digital financial services and the Internet, in part by strengthening enabling environments. Increase the use of evidence, data, and analytics for better decision-making. ● Innovation: Identify, test, and accelerate new tools that have evidence of impact, sustainable financing, and reach. Increase the adoption of high-impact solutions and the effective use of innovation methods by the Agency. ● Partnership: Work with impact investors to catalyze private capital for early stage businesses and strengthen the environment for entrepreneurship. Manage USAID’s signature public-private partnership model, the Global Development Alliance, and work to catalyze new models for collective action. Introduction 1. What are your main roles and responsibilities? STIP Performance Evaluation Final Report: September 2017 | 49 2. What work, projects or activities are you doing with the Lab in “STIP” — any of the following areas: scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? 3. What non-Lab-led work are you doing in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector￾engagement? ● Who are your IPs? 4. What kind of learning are you generating from your STIP work (scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement-related work)? ● How are the Lab’s other knowledge products (publications and reports, evaluations, datasets, websites, etc.) being utilized by your team/the Mission? Promoting STIP in the Mission 5. Was there an STIP Assessment? How useful was it? ● If no assessment, how did you work with the Lab to determine your STIP focus/needs? 6. How is the coordination between the Lab and Mission? ● Do you interact with the STIP Fellow here? How do you see the role of the fellow? ● Is the model of the official/team responsible for this coordination effective? ● How can it be improved? 7. Is your work part of the integrated Work Plan with the Lab? Describe it. How effective is it? ● How can it be improved? STIP Integration 8. How have the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement been integrated in the Program Cycle at the Mission? ● At the project (design and implementation) level? ● At the activity (design and implementation) level? How does/will your team operationalize STIP? ● At the Mission-level PMP? ● At the project- and activity-levels MEL plans? Are there specific STIP indicators? 9. What role has the Lab played in this integration? ● What role have the STIP projects implemented by the Mission’s partner organizations played in this? ● How can this be improved? STIP Performance Evaluation Final Report: September 2017 | 50 10. How have the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement been integrated at the organizational/structural- (i.e., new positions, roles of staff, working groups) level in the Mission? ● Have staff participated in trainings on these issues? ● What role has the targeted engagement by the Lab played in this? ● How can this be improved? 11. Different Lab’s scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement services/offerings are being applied in the Mission. [Show visual of Lab service triangle: Knowledge Sharing, Communities of Practice, Communications; Training, Toolkits, and Exchanges; Advisory and Analytical Services; Co-programming; and, Focus Mission] ● Which types of services have been the most effective in integrating these themes in the Mission? Why is this the case? 12. How effectively is the Mission delivering scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement tools and expertise to other Missions in its region? (For West Africa Regional) 13. The Lab’s scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement with the Mission include all services and offerings coming from the Lab regardless to what team or initiative they are attached (i.e., Next Generation partnership, LWP, etc.). Which services or offerings have been most valuable? Why is this the case? 14. What is the role of the Mission ecosystem (i.e., budget, development policy, resources, leadership) in integrating STIP: areas of scientific research; technology (such as mobile phones, geospatial); innovative approaches to finding solutions; or, private sector-engagement in the Mission? ● What strategies and methods have been most effective in integrating these areas? ● How can the ecosystem be improved? 15. What does the Mission want from the Lab for further integration of “STIP”: new scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement services/offerings? ● What is missing? Other 16. To what extent has targeted engagement by the Lab increased understanding, capacity for using, and application of the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement in the Mission? STIP Performance Evaluation Final Report: September 2017 | 51 ● What role have the STIP projects implemented by the Mission’s partner organizations played in this? ● What more can be done to increase this understanding? 17. What is the most significant change brought to the Mission through the application and integration of the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? 18. What main issues will affect the sustainability of the increased understanding of, capacity to use, and application of the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement in the Mission? 19. Do you have additional comments on challenges and/or recommendations concerning integrating the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement in the Mission? CLA 20. How is the integration of the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement aligned with CLA efforts in the Mission? ● What type of collaborative work (internally and externally with the private sector) has been done to integrate these areas at the Mission? ● Is evidence and learning being generated based on the integration of these areas? How is this evidence and learning being provided to Mission staff? ● What activities (if any) are in place to help support intentional and systematic learning at the mission ● Is the evidence and learning from these areas of integration being used to inform decision-making? To design and implement project/activities iteratively? For adaptive management? ● Is the Mission sharing its evidence/learning from these areas with the Lab and other Missions? Conclusion ● What other relevant documents/materials can you share with us? ● Are you comfortable being contacted for follow-up questions? Yes/No ● Do you have suggestions for other staff we should interview? ● Are there other final comments you wish to make? STIP Performance Evaluation Final Report: September 2017 | 52 ANNEX 3.d. Data Collection and Analysis Tools: Key Informant Interview Protocol for Mission Implementing Partner Organizations Date: Name: Title: Office/Department/Mission: Gender: M F Opening: Evaluation introduction and consent form. Introduction The Lab’s STIP definitions include the following: (When providing your responses, please address any one, or a combination, of the four STIP areas that you are working on in the Mission.) ● Science: Channel the technical expertise of scientists and researchers to build local scientific capacity, empowering people with tools for change, and use the evidence from scientific research to drive new policies and programs. ● Technology: Work to increase access to digital financial services and the Internet, in part by strengthening enabling environments. Increase the use of evidence, data, and analytics for better decision-making. ● Innovation: Identify, test, and accelerate new tools that have evidence of impact, sustainable financing, and reach. Increase the adoption of high-impact solutions and the effective use of innovation methods by the Agency. ● Partnership: Work with impact investors to catalyze private capital for early stage businesses and strengthen the environment for entrepreneurship. Manage USAID’s signature public-private partnership model, the Global Development Alliance, and work to catalyze new models for collective action. Promoting STIP 1. Briefly tell us about your organization. ● What are your main roles and responsibilities? STIP Performance Evaluation Final Report: September 2017 | 53 2. What project and activities are you working on with the Mission in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? ● How is the coordination between your organization and the Mission? 3. What is your thinking around scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? ● Has your organization worked in these areas before? ● Have you worked in these areas before? 4. Where do you and your organization get information, knowledge, and skills in these areas? What role has the Lab or Mission played in this? 5. What knowledge products (publications and reports, evaluations, datasets, websites, etc.) are your projects and activities generating in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector￾engagement? ● How are you making use of these knowledge products? ● Has the Lab or Mission provided you with knowledge products in these areas? STIP Integration 6. What strategies, methods, people, and resources have been most effective in integrating the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement in your projects and activities? ● How is the Lab or Mission aiding in this integration? 7. How have your projects and activities in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector￾engagement contributed to the integration of these areas in your organization’s strategic planning? ● At the organizational/structural- (i.e., new positions, roles of staff, etc.) level? ● Is your staff trained in these areas? ● In the design, implementation, and M&E of other and potentially future projects and activities? 8. What additional scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement services/offerings do you want from the Lab or Mission? ● What additional support/guidance do you need from the Lab or Mission? STIP Performance Evaluation Final Report: September 2017 | 54 Other 9. How have your projects and activities in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector￾engagement increased your organization’s understanding, capacity in using, and application of the areas? 10. What are the most significant changes your projects and activities in these areas have brought to your organization? 11. What main issues will affect the sustainability of your projects and activities in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? 12. Do you have additional comments on challenges and/or recommendations concerning your projects and activities in these areas? 13. Can you comment on how your projects and activities with the Mission have increased the Mission’s understanding and capacity in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector￾engagement? CLA 14. What types of evidence and learning is being generated from your projects and activities in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement? 15. Can you comment on how your projects and activities with the Mission in the areas of scientific research, technology (such as mobile phones, geospatial), innovative approaches to finding solutions, or private sector-engagement are generating learning in the Mission? Conclusion ● What other relevant documents/materials can you share with us? ● Are you comfortable being contacted for follow-up questions? Yes/No ● Do you have suggestions for other staff we should interview? ● Are there other final comments you wish to make? STIP Performance Evaluation Final Report: September 2017 | 55 ANNEX 4.a. Sources of Information: Evaluation Participants 1. USAID/Global Development Lab, Washington, DC Brian Bingham, M&E Specialist, HESN/CDR Chad Blevins, Geographer, Geocenter/CDD Jill Boezwinkle, Senior Program Manager, CDI Chris Burns, Senior Coordinator, LWP 2/CDD Joaquin Carbonell, AOR, CDI Mariam Chaudry, Geospatial Network Coordinator, CDD Matt Corso, Senior Communications and Engagement Advisor, OEC Emmanuella Delva, AOR, HESN/CDR Sabeen Dhanani, Deputy Team Coordinator, LWP 2/CDD Joe Duncan, AOR, CDD Steven Gale, Senior Futures Advisor, CDD Patrick Gault, Senior GIS Analyst, CDD Sarah Glass, Acting Director, LWP 1/CTP Shannon Griswold, Senior Scaling Advisor, EIA Grace Hoerner, Innovation Advisor, CDI Marcus Johnson, Agency Coordinator, Global Development Lab Michelle Jones, AOR, HESN/CDR Joshua Kaufman, Acting Director, CAI/EIA Maurice Kent, AOR, LWP 3 Brian King, Activity Manager for D2FtF, LWP 2/CDD Tom Koutsky, Digital Inclusion Advisor, CDD Maggie Linak, AOR, HESN/CDR Christine Martin, Activity Manager for D2FtF, LWP 2/CDD Ku (Lanakila) McMahan, AOR, CDI Jill Moss, Relationship Manager, CAI Rebecca Obrock, Relationship Manager, CAI Seema Patel, Division Chief for Innovation Design and Advisory, CDI Jason Porter, AOR, PEER/CDR Philip Rihm, Activity Manager, LWP 1 Kathy Rostowski, AOR, LWP 1/CTP Dean Salpini, Field Engagement Division Chief, CAI Rebecca Saxton-Fox, Activity Manager, LWP 1/CDD Rob Schneider, AOR, CTP Thomas Schumacher, AOR, CDI Chason Smith, AOR, CTP Rachel Stallings, Program Analyst, CAI Brit Steiner, Division Chief for Strategy and Learning, CAI Carrie Stokes, Agency Geographer and GeoCenter Director, CDD STIP Performance Evaluation Final Report: September 2017 | 56 Sophia van der Bijl, Senior Impact Assessment Advisor, EIA Kate Weber, Project Development Officer and Communication Advisor, OEC 2. USAID/West Africa Regional Mission, Accra, Ghana Amma Aboagye, STIP Advisor, Regional Program Office Eric Amankwah, Mission Training Coordinator, Regional Executive Office Alfred Amoatwo, Program Management Specialist, Regional Health Office Emmanuel Antoh, GIS Specialist, Regional Program Office Isaac Asenso, Computer Management Assistant, ACE/ICT Noel Bauer, Deputy Director, Regional Health Office Rachel Cintron, Director, Regional Health Office Alex Deprez, Mission Director, Front Office Nikki Enersen, Program Officer, Regional Program Office John Ezell, Acting Regional Controller, Regional Financial Management Office Stephanie Garvey, Director, Regional Peace and Governance Office Catherine Hayford,Financial Management Officer, Chief Financial Office Samuel Kouame, Regional Deputy Controller, Regional Financial Management Office Margaret McMorrow, Environment Officer, Regional Economic Growth Office Melody McNeil, Environment Office Director, Stephen Menard, Director, Regional Program Office Abdou Ndiaye, Deputy XO, Regional Executive Office Leslie-Ann Nwokora, Regional Contracts Office Director, Regional Acquisition & Assistance Office Jorge Oliveira, Food Security Advisor, Regional Economic Growth Office Pape Sow, Regional YALI (Young African Leaders Initiative) Coordinator, Regional PO Crystal Weathersby, Regional Controller, Regional Financial Management Office STIP PoCs at USAID Missions and Offices Supported by West Africa Regional Francois de Paul Bado, Program Officer, USAID/Burkina Faso Stephen Dzizi, Health Officer, USAID/Côte d'Ivoire Corey Hancock, Program Office Director, USAID/Côte d'Ivoire Jennifer Karsner, Agriculture Development Officer, USAID/Niger 3. USAID/Uganda Mission, Kampala, Uganda Oscar Ankunda, Manager, Economic Growth Office Simon Byabagambi, Agronomist, Economic Growth Office Katherina Erxleben, M&E Fellow, Education, Youth, and Child Development Laura Gonzalez, FtF Coordinator, Economic Growth Office Phil Green, Economist, Program Office Patricia Habu, East African Community and Trade Policy Specialist, Economic Growth Office Jo Lesser-Oltheten, Deputy Mission Director, Front Office Mark Meassick, Mission Director, Front Office STIP Performance Evaluation Final Report: September 2017 | 57 Daniel Mutumba, Financial Analyst, Financial Management Office Naomi Natukunda, Program Assistant, Program Office Morris Nsamba, M&E Specialist, Democracy, Rights, and Governance Office Sheila Nyakwezi, Nutrition Specialist, Health Office Richard Okello, GIS Specialist, Program Office Amy Prevatt, STIP Fellow, Program Office Dave Rogers, Manager, Economic Growth Office Solome Sevume, Program Management Specialist, Health Office Jenny VerNooy, Director, Program Office Erin Wroblewski, Program Design Officer, Program Office Leah Wyatt, Learning Fellow, Program Office 4. USAID/West Africa Regional Mission Partners Issoufi Alfari, Head of Information and Research Department, AGRHYMET Center, CILSS Abdourahmane Ba, Chief of Party, ASSESS Tome Ca, Head of Health Information Management Department, WAHO Carla Denizard, Regional Director for West and Central Africa, AfricaLead Marian Honu, M&E Officer, FHI360 Andre Koalaga, Chief of Party, EngenderHealth Abdou Tenkouano, Executive Director and CEO, CORAF 5. USAID/Uganda Mission Partners Barbara Amuron, Chief of Party, Monitoring & Evaluation of Emergency Plan Progress (MEEPP II) Robert Anyang, Chief of Party, FtF Commodity, Production & Marketing activity Souleymane Barry, Deputy Chief of Party, MEEPP II Geri Burkholder, Chief of Party, Literacy Achievement and Retention activity Raymond Byaruhanga, Chief of Party, TRACK TB/Management Sciences for Health Sam Kasozi, Deputy Chief of Party, TRACK TB/Management Sciences for Health Rita Laker-Ojok, Chief of Party, FtF Agricultural Inputs activity Eva Mulema Matsiko, Chief of Party, Governance, Accountability, Participation and Performance (GAPP) program Roy Mayega, Deputy Chief of Party, Resilient Africa Network Marcos Moreno, Chief of Party, Youth Leadership for Agriculture activity Ian Morrison, Chief of Party, USAID Uganda Supporting Access to Justice, Fostering Equity and Peace (SAFE) Program Albert Mugumya, Conflict & Reconciliation Officer, USAID Uganda SAFE Program Charles Mulagwe, Project Director, Uganda Value Added Maize Alliance Peter Muyingo, M&E Officer, Literacy Achievement and Retention activity Anant Singh, Chief of Party, QED Dick Wadada, Project Director, Alur Highlands Coffee Alliance STIP Performance Evaluation Final Report: September 2017 | 58 ANNEX 4.b. Sources of Information: Documents 1. USAID/Global Development Lab, Washington, DC Agency Engagement Plan [AI-Lab] Advisory & TA [AI-Lab] Communities of Practice [AI-Lab] Fellowships & Exchanges [AI-Lab] Knowledge Products [AI-Lab] OUs Using [AI-Lab] Training & Events CAI KM Assessment with Annexes CAI KM Assessment Exec Summary 2-Pager CAI KM Assessment Final Presentation 10-25-16 CAI RF [Results Framework] Draft STIP Integration Research Plan 070516 Exit Interview Questions Final Agenda for October 2016 FSN Cohort FINAL MOA STIP TALENT EXCHANGE 2016-04-21 Good Points for October 2016 Class Lab Activities Overall (last updated September 2015) Lab STIP Talent Exchange Program — Policy and Procedures — Oct 2016 Lab STIP Repository Site Map Lab Strategy Document_Draft 5_5.9.16 Lab-wide AI Indicator Definitions Lessons from the Field: A Report about Integrating Science, Technology, Innovation and Partnerships (STIP) Polaris Clearance Package — Lab Agency Engagement — Approved with Signatures Post Exchange Action Plan Template SA_STIP_Assessment_Letter Size_20161205 “Scenario Planning for Development: It’s About Time,” by S. Gale and R. Williams. New Security Beat: September 28, 2015 STIP Assessment_SA_20161214 STIP Talent Exchange Pre Post STIP Talent Exchange Program — Draft Announcement 4-5-2016 STIP TE Survey Responses Target Countries — Lab Activities USAID STIP assessment — January staff meeting USAID STIP Public goods assessment — Final Report What the Lab Offers 032117_STIP Brief_2_FinalA 20170214 STIP_Lessons_20170324 20170331 STIP INTEGRATION LESSONS FROM THE FIELD for Dojo STIP Performance Evaluation Final Report: September 2017 | 59 2. USAID/West Africa Regional Mission, Accra, Ghana ABP marketing piece 05_16_2016 ACA Fact Sheet August 2015 Accra _ Road to GES Communications Strategy Accra Road to GES_Speaker List Action Memo for Data Visualization Specialist Action Memo for Geospatial Analyst Action Memo for STIP Advisor USPSC Long term 01-11-2016 Action Memo M_E Reporting Specialist 2 13 Activities ADVERT FOR GEOSPATIAL INFORMATION SPECIALIST ADVERT FOR SENIOR DATA VISUALIZATION SPECIALIST After Action Review Guidance March 2013 AFR CDO POC List Agenda _ Road to GES Accra, Impact Hub Approved-Signed Action memo for STIP FSN positions 8.11..16 Bilateral LAB activities Briefer — WA Discussion on Capacity Development of Regional Partners v2 CAI FY16 AAR-Learning Focus Areas Cameroon announcement for STIP CILSS Fact Sheet Oct 2015-AMD Cleared CORAF Fact Sheet Oct 2015 clamaturitymatrix_handout_20160107 Daisy Project Tracker Deliverables Doc02_CORAF and SEED SYSTEMS Doc03_CORAF and WAAPP Doc04-Seed Learning Event Draft Implementation plan and budget for E platform rollout DRAFT REVISED — Suggestions for Potential Lab-WA collaboration and support 03-11-16 version ECOWAS Ag Fact Sheet Dec 2015 Feed the Future PRO-FAB Fact Sheet Nov 2015 Flyer one pager on maize lethal necrosis disease Future vision for STIP in West Africa Gage TDY Update #2 GIS_Inventory Global Shea Alliance Fact Sheet Nov 2015 Infographic case-3-staff-development-final Land Use Fact Sheet May 2016 LOI West Africa Regional Media Advisory Road to GES May 16, 2016 STIP Performance Evaluation Final Report: September 2017 | 60 Media Advisory Road to GES May 19, 2016 Mission PAD presentation June 2016 MLN Concept Note-submitted 26092015 Outbrief-TDY PAPA-MOD#16-AID-624-P-00-08-00060-USGS-Last and end Participating Agency Program Agreement PD FOR PMS — GEOSPATIAL INFORMATION SPECIALIST PD FOR SENIOR DATA VISUALIZATION SPECIALIST SPECIALIST PPL_AfterActionReviewGuidanceMarch2013 public version of the SHER PAD May 19 draft RAO Fact Sheet December 2015 Real Impact Analytics_MoU Regional Analysis — Africa-branded REGO Org Chart-Oct 2016 Road to GES — Accra, Ghana RSVP (Responses) ROECCR Sector Overview Aug 2015 RTIO Fact Sheet Mar 2014 Run of Show_Accra Secy of State Kerry Remarks on Connectivity April 14, 2016 SERVIR W Africa SERVIR West Africa Kickoff Presentation SERVIR West Africa-Ppt South Africa STIP Conference SHER PAD Issues Tracker SSD Fact Sheet Aug 2015 STIP @ West Africa Regional Mission stip-advisor-position STIP coverage_ STIP Interim Update — Daisy 3_8 STIP Recommendations STIP SR one pager STIP TW one pager STIP Update for Alex #1 STIP USAID West Africa Regional Year 2 Strategic Plan TDY Prep The Lab’s Contribution to USAID’s Ebola Response and Recovery 2-pager (Overview) Trade_Investment Hub Nov 2015 USAID_AAR_Guide_0 USAID_DataScientist_Draft Position Description USAID Health Informatics Team_28 March 2016 USAID-WA-RDCS-Public-Version-June 2015 USAID West Africa ASSESS Results framework USAID West Africa Regional FY 2015 Spend Plan Proposal USAID West Africa Regional Mission-STIP FRAMEWORK-Draft USPSCLocalhireLongtermSTIPAdvisor-02.10.16 STIP Performance Evaluation Final Report: September 2017 | 61 WA-BiCC Fact Sheet Aug 2015 WAFP Fact Sheet Oct 2015 Eskaneatze-lokalizazioon WALK Learning Agenda WALK Operation Plan WASP Fact Sheet November 2015 WA-WASH Fact Sheet Jan 2016 Weekly_Project_Progress_Update Oct 7 West Africa Science, Tech, Innovation, Partnership Activity Survey (Responses) - Form Responses West Africa Work Plan AID-624-TO-16-00001 Fully Executed SERVIR Task Order 20160630100721448 3. USAID/Uganda Mission, Kampala, Uganda Appendix_1_Bulleted_summary_of_resilience_factors_7_communities Appendix_2_Detailed_case_studies_7_communities CDR Uganda TDY Trip Report 090616 Digital Inclusion Uganda TDY Report (Koutsky 2017).docx DRG Office Notes DR-TBMIS Training and Enduser manual Nov 2016 EG Office Notes Erin Notes EYCD Office Notes Health Office Meeting Internal_SBU_Uganda_December_2016 Learning Fellow Meeting Notes Next Generation Evaluation IP Interview LIst.xlsx Notes from Discussion w/ Uganda STIP POC Outbrief for the Uganda Front Office Peace Corps Meeting Notes PEER Fact Sheet — 2016 PPL and LAB Uganda TDY Consolidated Report — Feb 2017 Program Office Notes RAN_EA-RILab-Uganda-Community-Consultations-Report-Climate RAN Report on USAID Innovation Bazaar — May2017 RAN Resilience Pathway Analysis for Climate Shocks & Slow Recovery from Conflict Uganda.Position Paper.24.03.17 RAN Uganda Deliberative Poll Report Report to the Uganda People June 2017 REVISED LARA AMELP Jan 2017 — clean copy v24-3-17 Final Steve Gale blog- With New Analytics, a Vision of Alternative Futures for Uganda STIP Activity/DOs Matrix STIP Performance Evaluation Final Report: September 2017 | 62 TB Activity Presentation & Follow Up Notes TDY Trip Report-CDR Research Agenda TA to USAID/Uganda Uganda Active PEER Projects — May 2017 Uganda Endline Final Evaluation Report_REVISED CLEAN_040517 Uganda Mission Director Memo reScaling Off-Grid Work plan 2017 Uganda - October 2016 Country PEER Profile Uganda STI Landscape Assessment — FINAL 3.15.16 Uganda STIP Champions Group Scope.docx Uganda STIP Integration Work Plan Uganda TDY Trip Report 10/17/16-10/20/16 Uganda.2015.LSMS.Report USAID Uganda/Lab STIP Integration Work Plan FY2017 UVAMA_Americas Positive Impact Roadshow UVAMA Overview Final 1_EA RILab_RIC4ACE.Final_Document_30_7_14 1_Full_Report_Context_Analysis_Eastern_Africa 2_Summary Report 4.4 Lab-Uganda STIP Integration Activities 4.6 Letter of Intent for Next Gen Uganda — FINAL SIGNED 12.18.15 4. External Resources “Advancing development in Uganda: evaluating policy choices for 2016-21 and selected impacts to 2040,” by The Frederick S. Pardee Center for International Futures: 2015. “Assessing the evidence base on science, technology, innovation and partnerships for accelerating development outcomes,” by Sabet, SM, Heard, AC, Neilitz, S and Brown, AN. International Initiative for Impact Evaluation (3ie) Scoping Paper 6: 2017. “The Future Can’t Wait: Over-the-horizon views on development,” by the U.S. Department of State, U.S. Agency for International Development, National Defense University, and Woodrow Wilson International Center for Scholars: 2013. “The Role of Science, Technology, Innovation, and Partnerships in the Future of USAID,” by the National Academies of Sciences, Engineering, and Medicine: 2017. Scoping the STIP Evidence Base stip-egm6 3ie+STIP+EGM+(static) STIP Performance Evaluation Final Report: September 2017 | 63 ANNEX 5. Targeted Engagement Targeted Engagement The Lab’s targeted engagement with the West Africa Regional and Uganda Missions explored in this evaluation encompasses support provided through Next Generation partnerships, LWPs, and other activities and assistance. Through this targeted engagement, the Lab worked with the Missions to create key coordinating positions, conducted a STIP assessment for Uganda, and developed integrated Work Plans for both Missions, revised yearly. This section discusses these developments and the state of the coordination between the Lab and the Missions, as well as the role of the STIP Advisor in West Africa Regional and the STIP team in the Uganda Mission. West Africa Regional The current Relationship Manager for West Africa Regional serves as a critical focal point. The RM, who has been in the position for a year, communicates with the STIP Advisor and the supervisor, the RPO director, on a regular basis to make collective decisions about the direction of Lab support. Speaking of this exchange, the RM stated: “The information tends to come to a point between the STIP Advisor and myself. If I’m communicating with my other colleagues at the West Africa Regional Mission who are aware of the Lab, [the STIP Advisor] is always cc’d. That’s where the filtering down happens, and the two of us communicate collaboratively. Often her boss is on the emails, and others are on threads.” When the RM visits the West Africa Regional Mission for TDYs, she meets with staff across the different offices and technical teams. In these meetings, the RM brings information about Lab services and discusses how they could be useful to the Mission’s work, gets into the details of the technical teams’ programs and projects, and explores the feasibility of Lab involvement in their work. The RM also regularly reaches out to the Mission’s technical teams from Washington, as a member of RPGO observed: “The RM is good about forwarding emails frequently and connecting us to people at the Lab. We have ongoing communication with her and she reaches out to us when she hears about opportunities.” Notably, the Lab has had three RMs for West Africa Regional since the commencement of their targeted engagement. CAI’s Division Chief for Strategy and Learning felt that previous RMs might not have been as beneficial as the current RM. She explained: “It was kind of a start and stop with each transition. It was very personality-driven and less cohesive. It was frustrating for the Mission to have to keep reorienting and getting to know a new RM.” Current coordination is seen as beneficial by both sides. The MD and Deputy MD; directors of the RPO, RHO, and DRGO (Democracy, Human Rights and Governance Office), STIP Advisor, and various other staff at the Mission concurred with this view. The RPO director also has a background in GIS and understands the value in integrating STIP in the Mission’s work; hence, his role supervising the STIP Advisor is an additional asset to the coordination with the Lab. The RM further described the STIP Performance Evaluation Final Report: September 2017 | 64 coordination with the Mission as: “We have established a firm, collegial, cooperative arrangement. It’s a two￾way flow.” However, she also acknowledged that the coordination could improve with more face time: “It really helps to have a presence in the country. I’ve heard this several times. You consult, work through the plan, think through potential options for next year’s work . . . and doing all that face-to-face helps. Bringing [the STIP Advisor] to DC was extremely helpful.” Other Lab centers interact directly with the Mission. The technical teams at the Mission also engage directly with the appropriate Lab teams for their specific STIP activities. For example, staff in the RHO work directly with the LWP 1 team on Ebola issues. Lab teams that engage with the Mission generally keep the RM informed about their work, but do not necessarily involve her in the specifics of their technical engagements. The current integrated Work Plan is being finessed. Some of the contents of the STIP Framework report, written by the RPO director and a consultant, were included in the first-year integrated Work Plan for the Mission, but the second-year integrated Work Plan does not rely on this report. The RM explained that parts of the current (second-year) integrated Work Plan had been implemented, and the latter part of this Work Plan is still being finessed. Mission staff request fewer offerings and more demand-driven services. There remain some issues with this coordination, as some Mission staff feel that the CAI continuously offers an inordinate number of services, and the relationship can seem more supply- rather than demand-driven. One staff member in RPO stated: “I wish they were more focused and they wouldn’t kill us with ideas every day. On our end, we’re really busy, and we don’t have time for a lot of that.” Another staff member in the RPO commented: “The relationship is definitely more led from the Lab side. I think that’s not good. They could benefit more by responding to the Mission’s needs as opposed to offering, offering, offering. But that aside, they’re extremely available, responsive, and easy to work with.” The MD, while noting the beneficial relationship with the Lab, also observed that the work with the Lab has to be more demand-driven as opposed to supply-driven, as the Mission is often just responding to the Lab’s initiatives. STIP Advisor The STIP Advisor brings pertinent experience to her role. The STIP Advisor brings valuable experience to her role; prior to coming to West Africa Regional, she worked for a nonprofit in research and policy in Ghana that promotes effective solutions to global poverty. In that role, she coordinated policies and worked on taking information and communication technologies (ICTs) to scale within the Ghanaian government. She also has experience working in radio on youth development issues and in the technology space in Ghana. As part of her SOW, the STIP Advisor acts as a catalyzer by identifying, promoting, and capitalizing on opportunities for STIP in the Mission. The MD sees the STIP Advisor role as serving as the information conduit between the Lab and Mission, and between the Mission and other USAID offices in the region. He also felt that the STIP Advisor has STIP Performance Evaluation Final Report: September 2017 | 65 to play the role of identifying STIP best practices, and disseminating that information within the Lab and to other USAID offices in the region. Some of the STIP Advisor’s responsibilities may not be expedient. The STIP Advisor felt that there were some responsibilities in her SOW that were a bit abstract to implement. For example, while the STIP Advisor’s SOW tasks her with fostering public private partnerships (PPPs), staff in a regional Mission may not be in the most effective position to foster PPPs since they are managed by the Agency in Washington, DC, and are established with IPs rather than regional partners. The STIP Advisor takes the initiative to meet with Mission staff. The STIP Advisor has met with all offices in the Mission, and also regularly attends some of the technical teams’ meetings. Mission staff know the STIP Advisor, and many have talked to her about employing STIP in their work. A staff member from the RHO said: “The STIP Advisor is supposed to take the lead on all STIP activities in the Mission. She has been organizing meetings that I’ve participated in. Before, we engaged a lot with (the RM) when she came here. Now it is common sense that if you have a STIP advisor, you go to her.” The STIP Advisor also understands that she must take the initiative to meet with the various technical teams as she learns about the projects or activities that could potentially incorporate STIP. Staff in other USAID Missions/Offices in the region have limited engagement with the STIP Advisor. Staff from Missions and Offices in Cote D’Ivoire, Niger, and Burkina Faso, with which the West Africa Regional Mission engages regularly, acknowledged that their Mission staff need improved capacity to employ STIP in their work. They reported that they had either met the STIP Advisor in person or exchanged emails with her, but had not had any structured engagement with her so far. One staff member commented: “The STIP Advisor is not very active in [our country office] — I assume she has her hands full. As a limited-presence country, we don’t have a Mission director, controller, CDCS, etc. With these constraints, it’s hard to be up on latest thinking from Washington on STIP.” Uganda The STIP assessment was useful to some but also had some issues. The vast majority of Lab staff did not provide explicit feedback on the Uganda STIP assessment. One exception was from an AOR for a HESN award in Uganda, who observed: “I read the STIP assessment before I went out there as background information, and it was helpful for me to have context of the country and the capacity. In my day-to-day work I didn’t connect back to it much.” Another Lab staff member who was part of the STIP assessment team, however, raised a few issues, explaining: “The Uganda STIP assessment was kind of the first one done. Assessing STIP within a country and Mission is nearly impossible for a team of five who don’t know the country well. The assessment did generate some useful findings. . . . I think our recommendations were too broad and the Mission already knew to a degree what they wanted. The challenge was trying to bring Lab expertise to pair with what they already know.” The STIP Assessment laid the groundwork for the Lab’s relationship with the Mission. According to the RM for Uganda, the STIP Assessment laid the groundwork for the relationship between the STIP Performance Evaluation Final Report: September 2017 | 66 Mission and the Lab. She stated: “We said, ‘This is what’s going on in the country, and what they could do to further integrate STIP in the Mission.’ From there, we were able to create a Work Plan to further integrate STIP.” The PDO at the PO said that the STIP assessment articulated the reasons why the Mission and the Lab had formalized their coordination. She added: “I think the findings from the assessment are still really useful. It helped us identify who the partners are that we can talk to here in country. We’re getting better at engaging partners outside the Mission around design, implementation, and assessment, so we go back to it often to see who’s doing what, even though it changes.” Initial engagement between the Lab and Mission had challenges. Initially, as was noted previously, the Lab was conducting many activities in Uganda of which the Mission was unaware. Furthermore, as the partnership commenced, the Mission leadership had some concerns about the Lab’s approach. The Deputy MD reflected: “They brought us new ideas that were things we were already working on. We wanted them to co-create with us and match our level, instead of coming out with the intention to train us. We beat that message into the ground.” In the first year of the Lab and Mission’s formal engagement, motivated staff on both sides generated many opportunities for collaboration, but there was no system in place to track this work. To enable the STIP PoC at the time (the current PDO in the PO) to better organize and coordinate the activities and relationship between the Lab and Mission, another PO staff member expanded her scope of work and went for an FSN Talent Exchange Fellowship with the Lab. When the FSN returned to the Mission, she became part of the STIP Team, along with the STIP Fellow who was hired in 2017. Coordination between the Lab and Mission has improved significantly over time. The RM has a STIP Team in the PO with which she regularly engages, and having staff working together has contributed greatly to improved collaboration. Discussing this improvement, the PDO in the PO explained: “We’re in a much better place now than we ever were. When we started, we were on a completely parallel track. The Lab had their own strategies and programs in Uganda that were not aligned with the Mission’s work. Now we’ve come to understand each other so much better. The RM helped improve this relationship. The fact that the Lab recognized that so much working was going on in Uganda and we needed a desk officer to coordinate helped us so much. The RM is great about de-escalating conflicts, seeing where OUs are doing the same thing, getting to know us, and answering questions about our Mission at the Lab. This organization is so much better than what was happening before, where folks from the Lab contacted different teams at the Mission and there was no big picture.” Mission leadership also held consultations with Lab leadership to emphasize that the Lab should seek concurrence from the Mission for any activities it wants to conduct in Uganda. The Lab agreed to this process, which was also codified in the revised ADS 201, requiring that Missions formally give STIP Performance Evaluation Final Report: September 2017 | 67 concurrence for activities that Washington-based OUs conduct in country. A staff member in EGO remarked: “It’s clear in the activity approval process that the Lab must work with the Mission on design, concurrence for trips, etc., so the Mission and Lab are clear on what the objectives are. This helps eliminate any coordination gaps and ensures that reporting is connected. Previously there were activities that the Lab managed in Uganda but the Mission didn’t get any progress reports. That has changed a lot and improved.” The PO director echoed this perspective: “When I got here two and a half years ago, the Lab was here — they were all over the place, not coming through the Mission at all. It’s really important that any investment from USAID comes through the Mission and at least aligns with our strategy. There were 70-something independent projects from the Lab in Uganda, and none of us at the Mission knew what they were doing. It’s been the Project Development Officer (in the PO) and the RM from the Lab who have done an amazing job of running everything that comes from the Lab through us. Now there is even a concurrence requirement. That’s changed a lot. The coordination has improved so much.” Mission leadership also felt that as the partnership with the Lab progressed, the Lab started to listen more to what the Mission wanted. When asked if the Mission now has a better relationship with the Lab, the Deputy MD agreed wholeheartedly: “Of all the bureaus that support us, I think we get the most out of the Lab and have the closest relationships with them. The Lab is really on the cutting edge and we like to think of ourselves that way as well, and the Lab listens.” Integrated Work Plans change based on the Mission’s priorities and needs. The first integrated Work Plan between the Lab and the Uganda Mission came out of the STIP Assessment. The RM and other Lab staff developed the Work Plan with inputs from the Mission. It was then reviewed by the PDO, PO director, and STIP Champions group, who reconfigured it to meet the Mission’s needs. The RM further explained that developing the first integrated Work Plan was a big learning experience — at that time, the Work Plan essentially consisted of a spreadsheet outlining the work that the Lab does with the Mission. In contrast, the second-year integrated Work Plan is written out as a document, though the RM felt that it was too early to comment on whether it was effective. The RM also said of these Work Plans: “It’s never done . . . it’s always developing and changing.” A staff member in the PO similarly stated: “The [first] integrated Work Plan was just a list of things. There were too many activities that [didn’t] match well. Under the new plan I think they’re trying to reduce it.” Many staff at the Lab (including from LWP 2, LWP 3, CDD, CDI, HESN, PEER, and EIA) noted that their activities with the Mission were included in the integrated Work Plan. However, their activities do not involve working directly with the RM and the STIP Team in Uganda. STIP Performance Evaluation Final Report: September 2017 | 68 Different Lab teams interact directly with the Mission but keep the Relationship Manager informed. In addition to coordinating with the STIP Fellow, the RM is also responsible for communicating with other staff in the Mission who work on activities with the Lab; she has conducted two TDYs at the Mission to this effect. In addition, different offices and technical teams in the Mission engage directly with the relevant Lab staff for their specific STIP activities. For example, the EGO team has regular communication and interactions with LWP 2 staff at the Lab. As noted previously, the Lab teams who engage directly with teams at the Mission generally keep the RM informed of their work, though not necessarily about the technical specifics. A LWP 2 staff person at the Lab explained: “The RM is in the loop about LWP 2 and knows what is going on . . . she is moderately engaged, but is not part of the technical conversations.” An AOR for a HESN award in CDR provided similar feedback, saying: “Generally we do coordinate with the RM and copy her about what is going on . . . it is more for awareness.” The only exception was an AOR in CTP who stated that his department is not really working with the RM; they engage directly with Mission staff who oversee their partnership work. Mission staff request fewer offerings and more demand-driven services. The coordination between the Lab and Mission has improved significantly; however, some Mission staff raised concerns about the Lab’s constant offerings of services and the ways this taxes the relationship. One staff member in the PO said: “We’ve spoken with the Lab over and over again about the management, having the team in DC understand the work here and make sure it fits in our portfolio. We continue to get bombarded with requests for information, new products, and services to apply. I think it’s gotten better — I give credit to the RM and the Lab for trying to do something about it. And it has been good to bring in a STIP Fellow because we all have other jobs to do.” The PO director also addressed this issue, observing: “Sometimes it feels like there’s more supply than demand.” Internal coordination among Lab teams working in the Mission could be improved. A staff person at the Lab raised a related issue around the internal coordination among the various Lab teams that are engaged with the Uganda Mission. She said that many Lab teams engage with and travel to the Uganda Mission without being aware of what other Lab teams are doing there. She suggested holding periodic meetings for Lab staff across centers who are working with a particular Mission so they can be better informed. STIP Team The STIP Fellow was initially brought on board to work on monitoring and evaluation issues. The STIP Fellow at the Uganda Mission is an MECap fellow who previously worked with the USAID Mission in Nepal in an M&E capacity. She was initially brought on board to work on M&E issues, but the Mission wanted her to focus on STIP, and she became the designated STIP Fellow for her one-year term. According to her SOW, the STIP Fellow primarily serves as the PoC between the Mission’s PO and the Lab. Her role also involves liaising with other Mission offices to incorporate STIP in different areas of STIP Performance Evaluation Final Report: September 2017 | 69 work, including developing the performance management plan (PMP), drafting PADs, and designing projects and activities. For example, the STIP Fellow is currently helping the Health Office put together a STIP health partners assessment for the Mission’s IPs. The STIP Fellow holds dual roles in the Mission. The STIP Fellow raised an important issue about having to wear two hats: She needs to maintain a balance between work with PPL and her role as the Mission’s STIP Fellow, as her position is funded by both entities almost equally. She explained that PPL’s objectives are more focused on learning and M&E, so her role for PPL is to work on assessments that will improve learning and adaptive management at the Mission, while her role as a STIP Fellow has a different set of responsibilities. Nonetheless, she said that she had found a happy medium between her two roles. Of note, though the STIP Advisor did not explicitly state this, STIP and CLA should be mutually reinforcing in the Mission. The STIP Fellow keeps the integrated Work Plan current. After the STIP Fellow commenced her position, she consulted with her supervisor and was able to get clarity on the priorities in the second￾year integrated Work Plan as they pertain to her work. The STIP Fellow has also discussed this Work Plan with the Mission’s offices and technical teams so they can provide updates as needed, to ensure that the Work Plan reflects their current priorities. The STIP Fellow aims to keep the second-year integrated Work Plan current by soliciting periodic input from the technical teams and aligning it with the Mission’s priorities. The role of the STIP Fellow is not clear across the Mission. The STIP Fellow felt that her role is clear, however, within the PO and with the STIP Champions group. She said that she had had interactions with different technical teams, but engaged most with the EGO and DRGO teams. A staff member in the EGO commented on the STIP Fellow’s role: “The engagement we’ve had with her so far is more coordination between the Lab and our bilateral work. She’s also helping us with our coordination across sector portfolios (health, education, etc.) That’s the expertise she brings.” The PDO agreed that the PO was clear about the STIP Fellow’s role, but she was not sure this was the case across the Mission. The Deputy MD had not had any interaction with the STIP Fellow, but stated the following when asked what role the STIP Fellow should play: “That person needs to be visible, in PAD development teams, and asking questions. Her job is to ask the right questions and get people to think, and open up possibilities. Once she’s done that and garnered interest, she can help connect them with people in the Lab who can offer assistance. People may think something is a great idea but not know how to do it. In terms of her Work Plan, there may be some quite specific stopping points, workshops, community of practice meetings, etc., where she should give a presentation, lead a discussion, generate buzz.” The development program assistant in the program office supports the STIP Fellow. As mentioned previously, the STIP Fellow is supported by a Development Program Assistant, an FSN who took part in a fellowship at the Lab. She has different roles in the PO, including assisting with project design, working on Glass (a system to commit funds for grants and contracts), and tracking trainings for the Mission and its IPs. Prior to the arrival of the STIP Fellow, the FSN helped the STIP PoC at the time STIP Performance Evaluation Final Report: September 2017 | 70 (now the PDO) to coordinate tasks between the Lab and Mission. Since the arrival of the STIP Fellow, the FSN is now primarily responsible for coordinating the process of Mission concurrence on Lab activities. She also attends any Lab-related meetings at the Mission. The FSN felt that it makes sense for her to stay in the loop around the STIP Fellow’s work in the Mission, as she will remain after the STIP Fellow’s one-year term ends and could provide some continuity in the work. The project development officer in the program office is an important STIP resource. As the supervisor of the STIP Fellow and FSN, the PDO provides guidance around STIP. Since the PDO served as the initial STIP PoC, she knows the Lab and its services well. Also, her current role requires her to be involved in the design of projects, where she brings a STIP focus to the Mission’s work across technical teams. A technical office staff member explained: “The Project Development Officer is also a STIP PoC — she doubles, and that’s a plus. She gets to be in each and every design conversation, and can make sure that each design walks the talk with STIP, and can help identify elements of STIP that should be included in each different activity design.” A staff person in the Health Office echoed this sentiment saying that when you engaged the PDO and explained your challenges, she offered the Lab as a resource and also connected you with pertinent staff members at the Lab. STIP Performance Evaluation Final Report: September 2017 | 71 ANNEX 6. Description of Missions’ Partners The West Africa Regional Mission’s regional partners that took part in this evaluation include the following: ● CORAF is the West and Central African Council for Agricultural Research and Development, operating under the ECOWAS, and includes a consortium of national research institutions. West Africa Regional provides funding to CORAF to: engage with the private sector; develop crops; conduct scientific research on agricultural methods, improving varieties of cereals, etc.; and innovate and develop new agricultural technologies. An example of a CORAF STIP activity is the electronic exchange marketplace (portal) WASIX.net for seed entrepreneurs set up under the Alliance for Seed Industry in West Africa (ASIWA). ● AGRHYMET is a specialized agency of the CILSS of thirteen countries in West Africa, and engages with the Ministries of Agriculture in these countries. The IRD at AGRHYMET collects data related to food security, agriculture, climate, natural resource management, etc., and conducts analysis of this information. These data are then disseminated through monthly bulletins, mass media, and the Internet. AGRHYMET is currently developing an online platform to rapidly provide analysis to its users on the data they collect. West Africa Regional has supported AGRHYMET through funds for its operations and projects. ● WAHO is a specialized institution of ECOWAS responsible for ensuring coordination of regional health interventions within the ECOWAS region. The Health Information Management (HIM) department of WAHO is responsible for collecting, managing, and making health information available to ECOWAS countries. USAID has funded various WAHO projects and the WAHIT project is currently being managed by RHO at West Africa Regional (along with the LWP1 team at the Lab). West Africa Regional’s IPs that took part in this evaluation include the following: ● The Analytical Support Services and Evaluations for Sustainable Systems (ASSESS) project was launched by USAID in partnership with the U.S. Department of Agriculture’s Foreign Agricultural Service, the University of Rhode Island, Delaware State University, and the Kwame Nkrumah University of Science and Technology. Through independent evaluations, analysis, and the sharing of best practices, ASSESS will support West Africa Regional and its development partners to increase the effectiveness of their programs. ● EngenderHealth is a U.S.-based contractor that is implementing the USAID funded ‘Agir Pour the Planification Familiale’ project in five countries in Western Africa to address FP issues. The project, managed by RHO at West Africa Regional, increases access to quality of and demand for FP services. The project is using technologies to gather data on target audiences to increase access and demand to FP; as well as using mHealth to spread awareness about FP issues. ● FHI360 is a U.S.-based contractor implementing the USAID-funded PACTE-VIH project in two countries in West Africa, managed by the RHO at West Africa Regional. The STIP components of this project include an MS Access database that was developed based on a unique identifier code built around each beneficiary. The project also has an mHealth component that includes an innovative strategy to reach the hidden target population through social media interventions and sending SMS messages on HIV prevention. STIP Performance Evaluation Final Report: September 2017 | 72 ● Africa Lead is a program that helps realize both USAID’s FtF and the African Union’s Comprehensive Africa Agriculture Development Program (CAADP) goals of reduced hunger and poverty. The REGO at West Africa Regional is involved in Africa Lead’s work in the West African region. The Uganda Mission’s IPs that took part in this evaluation include the following: ● The Agricultural Inputs Activity is implemented by Tetra Tech, a U.S.-based contractor, and is managed by the EGO at the Mission and falls under USAID FtF. The activity covers crops (e.g., maize, beans, coffee) and looks at linkages between markets, producers, as well as supply chains and quality of products and counterfeits. The activity incorporates STIP in their work through such mechanisms as e-verification of products, mobile money, and digital solutions for finance. ● The Commodity Production and Marketing Activity under FtF is implemented by Chemonics, a U.S.-based contractor, and managed by the EGO. The activity works to increase the productivity and market access for crops such as coffee, maize, and beans by engaging with the private sector, middle traders, agents, and farmers. Various interventions incorporate STIP, including platforms created for market exchanges and technologies developed for improving farming practices. ● The Youth Leadership for Agriculture Activity under FtF is also implemented by Chemonics and managed by EGO. The activity engages youth and provides them with the required skills to generate income through agriculture; This work encompasses strengthening and leveraging private and public sectors partners. The activity is also looking to streamline cash transfers by digitizing them for target audiences. ● The GAPP program, is implemented by RTI International a U.S.-based contractor, and managed by the DRG office. The project focuses on three main areas: governance program on accountability and influencing policy; strengthening systems; and, community engagement around services. RTI has piloted an innovative activity for inhabitants of a district to increase their engagement with the local government through an SMS platform. ● The Literacy Achievement and Retention activity is also being implemented by RTI International and is managed by the Mission’s education office. The activity focuses on systems strengthening and community engagement to address gender-based violence (GBV) in schools as well as providing teacher training as well as guides and books for children, among other activities. They are utilizing innovative technologies developed by RTI for M&E of their work and also have some partnerships with private companies. ● MEEPP II is implemented by Social and Scientific Solutions, a U.S.-based contractor. They provide M&E support to partners to ensure that they can report on President’s Emergency Plan for Aids Relief (PEPFAR) requirements in Uganda. The project serves different U.S. agencies including USAID, PEPFAR, Peace Corps, and Department of State. They do not have a direct engagement with the Mission, but as part of a technical working group, they have interacted with the Health Office staff. ● The Track TB project is a U.S.-Government flagship project, implemented by Management Sciences for Health (MSH), a U.S.-based contractor, and managed by the Health Office. The project focuses on building capacity in the national TB program and scaling up urban TB services and TB management. As part of their STIP work, MSH has developed a TB information STIP Performance Evaluation Final Report: September 2017 | 73 management system incorporated with mobile technology to report enrollment and treatment data. ● The SAFE program, implemented by a consortium of international contractors, is managed by the DRG office. The project is focused on peace and reconstruction efforts in a region of Uganda and uses SMS and the CrowdMap application to collect data on conflict hotspots. ● QED, a U.S.-based contractor, is the MEL partner of the Uganda Mission and works to strengthen M&E capacity, learning and adaptation, and collaboration. QED does their work through such methods as research and literature reviews, learning events, conducting evaluations, and data quality assessments (DQAs). ● RAN is one of seven development labs within HESN and includes a consortium of 18 universities in Africa and the United States. Their main award involves mobilizing universities in Africa to increase their engagement in developing innovative solutions to development challenges. ● The Alur Highlands Coffee Alliance is a Global Development Alliance (GDA) and a PPP between USAID and Carana that invests in coffee and is managed by Palladium, an international contractor. They provide services to build the capacity of coffee farmers and are going to start work in digital financing shortly. ● Uganda Value-added Maize Alliance is another GDA and a PPP between USAID and AgroWays managed by Palladium. The aim of the Alliance is to bring together small holding maize farmers in the formal sector by establishing formal linkages with them. They are employing various technologies and innovations to help the farmers increase their yields and have partnered with universities to get scientific expertise. STIP Performance Evaluation Final Report: September 2017 | 74 ANNEX 7. Evaluation Team Members: Qualifications, Experience, and Role on the Team The Evaluation Team consisted of a senior evaluator and a Monitoring, Evaluation, Research, and Learning specialist. The team was also supported by another senior Monitoring, Evaluation, Learning and Research specialist at LEARN who provided oversight of the activity, reviewing and approving all deliverables. Amitabh Dabla served as the senior evaluator. His responsibilities included: ● Engage with Lab counterparts to determine evaluation questions and detailed evaluation scope of work. ● Lead the evaluation design and development of the evaluation Work Plan. ● Lead the evaluation data collection (including interviews, focus groups, and other methodologies as defined) and data analysis. ● Provide in-briefings and exit briefings for visits to countries with USAID Missions. ● Participate in mid-term briefing and interim meetings. ● Participate in final presentation to discuss the summary of findings and recommendations to USAID. ● Lead writing of draft evaluation report, which will address each of the questions identified in written notification of the evaluation task and any other issues the team considers to have a bearing on the objectives of the evaluation. ● Lead writing of the final evaluation report by responding to the final comments from the CAI. Meghan Jutras served as the Monitoring, Evaluation, Research, and Learning specialist. Her responsibilities included: ● Contribute to the development of the STIP evaluation Work Plan, design, and methodology. Methods included interviewinging and facilitating meetings with Lab staff already working on specific on STIP activities and integration to ultimately refine the evaluation design. ● Contribute to the completion of the STIP evaluation in coordination with the senior evaluator. This included: reviewing and synthesizing findings from literature and project documentation ahead of field work; arranging and conducting field interviews, focus groups, or other data collection methodologies as defined; capturing, analyzing, and synthesizing data; and assisting in writing the draft and final evaluation reports. ● Participate in and support the senior evaluator with briefings, meetings, and presentations. STIP Performance Evaluation Final Report: September 2017 | 75 ANNEX 8. Conflict of Interest Disclosures STIP Performance Evaluation Final Report: September 2017 | 76