MSTAR PERFORMANCE EVALUATION MECHANISM PERFORMANCE AND LESSONS LEARNED FOR FUTURE USE April 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Beeta Tahmassebi, Kelsey Simmons, and Elisa Knebel of EnCompass LLC. Prepared for the United States Agency for International Development Contract Number AID-OAA-I-15-00021; Order Number AID-OAA-TO-16-00031 April 10, 2017 Cover Photo: “Temporary Education/Home Schooling” (https://flic.kr/p/raJiLL), by Neil Brandvold (for USAID). Used with permission via Creative Commons license CC BY-NC 2.0. Implemented by: EnCompass LLC 1451 Rockville Pike, Suite 600 Rockville, MD 20852 Phone: +1 301-287-8700 Fax: +1 301-685-3720 www.encompassworld.com MSTAR PERFORMANCE EVALUATION MECHANISM PERFORMANCE AND LESSONS LEARNED FOR FUTURE USE DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Policy, Planning, and Learning – Learning, Evaluation, and Research This task order is implemented through the Policy Planning, and Learning – Learning, Evaluation, and Research (PPL-LER) IDIQ, funded by the U.S. Agency for International Development (USAID) under Contract No. AID-OAA-I-15-00021. USAID’s Bureau for Policy, Planning and Learning (PPL) awarded EnCompass LLC the 5-year PPL-LER contract to provide technical and advisory services for performance and impact evaluations, evaluation and performance monitoring capacity building, and performance monitoring activities at the mission, bureau, and agency￾wide levels. PPL-LER task orders design and implementing quantitative and qualitative evaluation studies and assessments, develop and deliver evaluation and performance monitoring training, and provide technical assistance in performance monitoring for USAID development programs worldwide. Recommended Citation Tahmassebi B., Simmons K., and E. Knebel. 2017. mSTAR Performance Evaluation: Mechanism Performance and Lessons Learned for Future Use. Prepared for FHI 360 and the USAID Mobile Solutions Technical Assistance and Research (mSTAR) program. Rockville, MD: EnCompass LLC. April 2017 | mSTAR Performance Evaluation i CONTENTS Acronyms ..........................................................................................................................................................ii Executive Summary ......................................................................................................................................iii Evaluation Questions ................................................................................................................................................iii Evaluation Purpose and Questions ......................................................................................................... 1 Evaluation Purpose .................................................................................................................................................... 1 Evaluation Questions ................................................................................................................................................. 1 Project Background ...................................................................................................................................... 3 Inception ........................................................................................................................................................................ 3 Evolution ........................................................................................................................................................................ 5 methods and limitations ............................................................................................................................. 9 Methods ......................................................................................................................................................................... 9 Limitations .................................................................................................................................................................. 10 Findings ..........................................................................................................................................................11 Appropriateness of the Award Mechanism and Technical Scope ........................................................ 11 Program Performance, Achievements, Challenges, and Lessons Learned ......................................... 17 Lessons Learned for Future Mechanisms ....................................................................................................... 24 Conclusions ...................................................................................................................................................29 Annex 1. Evaluation Statement of Work .............................................................................................32 Annex 2. Evaluation Methods and Limitations .................................................................................34 Annex 3. Data Collection Instruments ................................................................................................. 37 Annex 4. Sources of Information ...........................................................................................................44 Annex 5. Disclosure of Conflicts of Interest .......................................................................................45 April 2017 | mSTAR Performance Evaluation ii ACRONYMS FIELD-Support Financial Integration, Economic Leveraging, Broad-Based Dissemination and Support ICT information and communication technology ICT4D information and communication technology for development LWA leader with associates mSTAR Mobile Solutions Technical Assistance and Research USAID United Stated Agency for International Development April 2017 | mSTAR Performance Evaluation iii EXECUTIVE SUMMARY PURPOSE AND BACKGROUND The Global Development Lab’s Center for Digital Development catalyzes programs and partnerships in digital finance, inclusive access to digital and mobile-based information and service delivery, and the use of data collected by digital devices to improve decision making. As part of this effort, the Lab supports Mobile Solutions Technical Assistance and Research (mSTAR), a broad, flexible, and responsive technical assistance and action learning mechanism that increases the availability of, access to, and use of mobile solutions by USAID missions, operating units, civil society and private sector stakeholders. mSTAR works in three main areas: digital financial services, digital inclusion, and real time data. The mechanism began in 2012, under the leadership of FHI 360, and is scheduled to end in 2019. The Center for Digital Development contracted EnCompass to conduct a performance evaluation of mSTAR in order to identify accomplishments, performance issues, lessons learned, and implementation constraints from mSTAR’s inception until February 2017. The EnCompass evaluation team’s specific tasks were to:  Determine the appropriateness of the project’s original technical scope and award mechanism  Assess overall program performance, achievements, and challenges  Provide recommendations for future mechanisms  Compile lessons learned and areas for improvement. EVALUATION QUESTIONS In a series of meetings in September 2016, EnCompass worked with the Lab and mSTAR to craft a list of evaluation questions (Exhibit 2) that would provide the most useful and relevant data for future planning for USAID. Exhibit 1: Evaluation Questions Evaluation Task Questions 1. Appropriateness of the project’s original a) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for mSTAR at the time of inception? April 2017 | mSTAR Performance Evaluation iv Evaluation Task Questions technical scope and award mechanism b) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? c) Has the prime/sub-partner structure originally envisioned to implement mSTAR activities been effective? d) Has USAID provided adequate guidance to mSTAR regarding desired outcomes? Of the program overall? Of particular projects? 2. Program performance, achievements, challenges, and lessons learned a) How has the prime partner worked with project sub￾partners in terms of communication, decision making, management, delegating tasks, and monitoring? b) What have been the actions taken (by the project) to mitigate risks and course correct during project design and implementation? c) How satisfied have various stakeholders (e.g., USAID, sub-partners) been with the work done by and assistance from mSTAR? d) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate, and timely? e) What have been the projects’ achievements and challenges? 3. Recommendations for future mechanisms a) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from mSTAR need to be continued and what anticipated services are likely to be needed? b) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? METHODS Given the evaluation’s narrow focus and the limited available resources, EnCompass and USAID determined that interviews with key informants and an accompanying document review were the best data collection methods to obtain the information needed to understand mSTAR’s performance. Throughout the evaluation process, the reference group (comprised of USAID and mSTAR staff) worked closely with the evaluation team to provide feedback and guidance. April 2017 | mSTAR Performance Evaluation v Documents reviewed were selected by the reference group and included quarterly and annual reports as well as other documents that could provide insight into the mSTAR mechanism as a whole. Key informants were sampled purposively and in concert with the evaluation reference group, which comprised USAID and mSTAR staff. EnCompass designed structured interview guides, employing an Appreciative Inquiry approach to explore exceptional experiences, values, successes, challenges, and wishes for the future of mSTAR. All data collection tools were reviewed and approved by the reference group before data collection began. The evaluation team conducted 38 interviews from four key stakeholder groups: USAID/Washington, USAID missions, mSTAR staff, and partners that worked on mSTAR projects. Findings APPROPRIATENESS OF THE PROJECT’S ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM The Global Development Lab established mSTAR as an Associate Award with the vision that this type of mechanism would enable the Lab and USAID missions to easily procure funds to quickly tap into a range of technical expertise. However, the Associate Award structure constrained USAID in ways it did not anticipate at the time of inception. mSTAR was not able to engage consortium partners or external technical experts as often as USAID had envisioned. This was due to a variety of factors including: mSTAR’s structure under the FIELD-Support LWA; misalignment of expectations between mSTAR and USAID on what role the Lab wanted mSTAR to play (technical or administrative); low consortium partner interest in executing smaller, shorter tasks; and difficulty in identifying partners with deep technical expertise in a wide-ranging subjects within ICT4D. The broad scope of the mechanism did result in mSTAR offering a range of services to the Lab, Missions, and target beneficaries, but it also resulted in ambiguity about mSTAR’s overall vision and goals. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED Different stakeholders reported different degrees of satisfaction with mSTAR’s performance; with USAID mission staff expressing high degrees of satisfaction, and USAID/Washington staff providing mixed feedback, depending on the expectations and demands of the activity manager in charge. Partners concurred that FHI 360 effectively managed relationships with them in terms of communication, monitoring performance, and compliance with USAID regulations. April 2017 | mSTAR Performance Evaluation vi Collectively, interviewees cited communication, training, logistics, and the support provided to USAID missions as mSTAR’s main achievements. mSTAR faced several challenges. High staff turnover at USAID and within the mSTAR team made it difficult to build strong relationships between the two groups and mSTAR was challenged in responding to a variety of USAID activity managers who had different expectations of what the mechanism should do and limited understanding of how to manage a mechanism. mSTAR was also challenged by the disconnect in the vision coming from the Lab and the requests coming from missions. This was compounded by a lack of clear communication in the missions about who their points of contact were between mSTAR and USAID staff. LESSONS LEARNED FOR MSTAR MANAGEMENT AND STRUCTURE Interviewees generated the following recommendations for the future:  mSTAR’s work in communication and facilitation and its work with the missions should continue to be leveraged and expanded.  USAID and implementing partners can struggle to achieve goals when the mechanism in use is not “fit for purpose”.  Global Development Lab staff must have the necessary skills and understanding of USAID rules and regulations to manage mechanisms.  USAID missions should be engaged in the design and implementation of future mechanisms. CONCLUSIONS The evaluation draws six main conclusions, listed below and mapped to findings in the conclusions section: Regarding the Mechanism  Conclusion 1: Although the Associate Award enabled rapid allocation of funds and a flexible scope of work, the mechanism did not align with USAID’s vision for efficient and cost-effective access to a broad array of technical expertise.  Conclusion 2: Although the mechanism allowed a wide range of projects to be implemented, there was a misalignment of expectations on the overall vision for mSTAR and how it should be managed. Regarding Performance April 2017 | mSTAR Performance Evaluation vii  Conclusion 3: mSTAR effectively managed relationships with partners and missions, although the number of subcontracts awarded was less than anticipated due to constraints of the mechanism.  Conclusion 4: mSTAR is widely recognized by USAID staff as having achieved high levels of success in working with USAID missions. The mSTAR team was also successful in meeting USAID/Washington’s needs in communication, facilitation and logistics. Challenges which arose were related to the provision of certain types of technical assistance to the Lab.  Conclusion 5: High staff turnover and changing USAID/Washington goals challenged the implementation of mSTAR. Regarding Future Needs:  Conclusion 6: mSTAR’s communication and facilitation work should continue, and its successful work with USAID missions should be expanded to engage missions earlier in the process.  Conclusion 7: USAID should build its own project design and management capacity, in order to ensure that future mechanisms are intentionally designed to align with the agency’s goals, while allowing flexibility to adapt to changing priorities. April 2017 | mSTAR Performance Evaluation 1 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The U.S. Global Development Lab’s Center for Digital Development team catalyzes programs and partnerships in digital finance, inclusive access to digital- and mobile-based information and service delivery, and the use of data collected by digital devices to improve decision making. As part of this effort, the Lab supports Mobile Solutions Technical Assistance and Research (mSTAR), a broad, flexible, and responsive technical assistance and action learning mechanism that increases the availability of, access to, and use of mobile solutions by USAID missions, operating units, and civil society and private sector stakeholders. mSTAR works in three main areas: digital financial services, digital inclusion, and real time data. The mechanism began in 2012, under the leadership of FHI 360, and is scheduled to end in 2019. The Center for Digital Development contracted EnCompass to conduct a performance evaluation of mSTAR in order to identify accomplishments, performance issues, lessons learned, and implementation constraints from mSTAR’s inception until February 2017. The EnCompass evaluation team’s specific tasks were to:  Determine the appropriateness of the project’s original technical scope and award mechanism  Assess program performance, achievements, and challenges  Provide recommendations for future mechanisms  Compile lessons learned and areas for improvement. EVALUATION QUESTIONS In a series of meetings in September 2016, EnCompass worked with the Lab and mSTAR to craft a list of evaluation questions (Exhibit 2) that would provide the most useful and relevant data for future planning for USAID. April 2017 | mSTAR Performance Evaluation 2 Exhibit 2: Evaluation Questions Evaluation Task Questions 1. Appropriateness of the project’s original technical scope and award mechanism e) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for mSTAR at the time of inception? f) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? g) Has the prime/sub-partner structure originally envisioned to implement mSTAR activities been effective? h) Has USAID provided adequate guidance to mSTAR regarding desired outcomes? Of the program overall? Of particular projects? 2. Program performance, achievements, challenges, and lessons learned f) How has the prime partner worked with project sub￾partners in terms of communication, decision making, management, delegating tasks, and monitoring? g) What have been the actions taken (by the project) to mitigate risks and course correct during project design and implementation? h) How satisfied have various stakeholders (e.g., USAID, sub-partners) been with the work done by and assistance from mSTAR? i) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate, and timely? j) What have been the projects’ achievements and challenges? 3. Recommendations for future mechanisms c) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from mSTAR need to be continued and what anticipated services are likely to be needed? d) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? April 2017 | mSTAR Performance Evaluation 3 PROJECT BACKGROUND INCEPTION USAID granted mSTAR to FHI 360 as an Associate Award under the Financial Integration, Economic Leveraging, Broad-Based Dissemination and Support Leader with Associates (FIELD￾Support LWA) cooperative agreement in 2012. The mechanism was conceived to increase availability of, access to, and use of mobile solutions by USAID missions, operating units, and other civil society and private sector stakeholders (USAID 2012). The vision for success included significant progress in the three inter-related work streams under mSTAR (USAID 2012):  Digital Financial Services: Mobile money actors in multiple countries are coordinated in a supportive regulatory environment; key payment streams have transitioned to mobile payments in line with USAID’s Commitment to the Better than Cash Alliance; more than 20 percent of the population in participating countries is accessing and using functioning payment systems; and innovative mobile money products and services are being introduced, especially those that support improved opportunities for the poor.  Digital Inclusion: A policy and regulatory environment is fostered that enables increased access and affordability; there is increased mobile broadband infrastructure investment; commercial actors are participants in a competitive ecosystem that extends affordable networks and services to citizens; poor citizens have wider access to mobile broadband connectivity and services and the capacity to use them; and gender gaps in mobile ownership and use are decreased.  Real time data for learning and decision making: Citizens, governments, and development practitioners are increasingly using mobile technologies to collect data to better understand citizens’ needs and concerns more accurately, monitor and evaluate projects, and capture more data to inform decision making about programs and strategy—and are doing this faster and often cheaper. The total estimated amount of the agreement was designated to be $29,790,313. USAID obligated $1,124,000 for the program at the outset (USAID 2012). At mSTAR’s inception, it was conceived that FHI 360 would apply a consortium approach to project management to ensure rapid and appropriate responses to USAID mission demands. As the prime recipient of the award, FHI 360 would be responsible for leading the overall April 2017 | mSTAR Performance Evaluation 4 management and coordination functions for all activities, which would be implemented by a 20- partner mSTAR consortium (Exhibit 2). The consortium comprises FIELD-Support LWA partners and a select group of specialized firms that work in mobile solutions and collectively bring complementary skills in economic strengthening and deployment of mobile technologies that are not otherwise available. Exhibit 3: Envisioned mSTAR Roles and Responsibilities FHI 360  Manage program and provide technical leadership and direction  Serve as prime point of contact with USAID/Mobile Solutions Team and missions  Ensure compliance with terms of the cooperative agreement by all partners  Coordinate short-term technical assistance  Ensure program goals, deliverables, and results are met on time and on budget in the most cost-effective way possible  Facilitate team and stakeholder dialogues to develop appropriate technical assistance teams and provide mobile solutions technical assistance, where appropriate and as needed  Manage collaborative learning and resource strategy and rapid dissemination of learning tools  Provide some technical leadership in mobile solutions  Provide technical leadership in monitoring and evaluation  Participate in the design of appropriate high‐impact resources and tools. Consortium Members  Provide recurrent short-term and long‐term technical assistance and services to support the design, development, and delivery of mobile solutions  Provide technical leadership in mobile money, mobile broadband, and mobile data, as well as financial inclusion and economic and livelihood development  Participate in steering committee or project committee decision making, as requested  Participate in the design of appropriate high-impact resources and tools based on areas of expertise. Source: mSTAR Cooperative Agreement (USAID 2012) The mSTAR program had no specified monitoring and evaluation framework at the outset of the project. Accessing the mechanism was advertised to USAID missions as consisting of the following steps: April 2017 | mSTAR Performance Evaluation 5 1. A mission or operating unit works with the mSTAR agreement officer’s representative to co-create a scope of work. 2. mSTAR works with USAID to refine the scope of work and select a team of specialists to implement it. 3. Once the scope of work is finalized and the buy-in is complete, implementation begins (usually 1 to 2 months). In its first year, the mechanism focused on drafting reports on mobile issues, launching USAID mission-driven activities, and facilitating knowledge sharing events in Mozambique, Bangladesh, the USAID Regional Development Mission for Asia, and the USAID/Afghanistan Mission (FHI 360 2013). EVOLUTION By year 2, mSTAR was defining its areas of work as follows (FHI 360 2014): • Digital finance: Development of robust digital finance ecosystems in targeted countries through increased use of the existing systems and technical assistance to strengthen supply, demand, and regulatory and policy frameworks. • Real-time data for decision making: Improvement of decision making for development problems and citizen empowerment by contributing to the increased use of mobile technologies for data collection and dissemination in the development community. • Digital inclusion: Support for USAID’s broader goals for increasing connectivity of mobile networks, affordability for lower-income populations, and locally relevant content. mSTAR articulated its core services as follows (FHI 360 2014): • Provision of technical assistance and training to increase the effectiveness of mobile ecosystems and actors to serve low-income populations and increase the impact of USAID projects through use of digital tools • Generation of evidence around efficiency, transparency, and safety gains through digital tools, as well as the business case for investing in bringing digital solutions to low-income and underserved populations • Management of platforms for knowledge sharing and dialogue to increase understanding of the development community and mobile ecosystem actors on the use of digital tools for strengthening development outcomes. April 2017 | mSTAR Performance Evaluation 6 FHI 360 drafted a results framework (Exhibit 4 on the next page) that defined its contribution to USAID’s intermediate results (FHI 360 2014). By Year 3, mSTAR had the following core projects: • mSTAR Bangladesh Activity • mSTAR’s Digital Development Training • Data Utilization Study • Assessing the State of Digital Financial Inclusion and Opportunities to Facilitate Sector Growth in Mali • Liberia Ministry of Education Salary Payments activity • Development of Digital Financial Services in Burma activity • Principles for Digital Development activity • mVacciNation assessment in Mozambique • Mozambique Mobile Access Usage Survey. Exhibit 4: mSTAR Results Framework April 2017 | mSTAR Performance Evaluation 7 In Year 4, mSTAR revised its results framework (Exhibit 5) to align with the results framework adopted by the Lab and produced a monitoring and evaluation framework (FHI 360 2016c). April 2017 | mSTAR Performance Evaluation 8 Exhibit 5: Revised Results Framework (2016) April 2017 | mSTAR Performance Evaluation 9 METHODS AND LIMITATIONS METHODS The EnCompass evaluation team held a kick-off meeting with USAID on October 10, 2016, to discuss the evaluation scope of work, to clarify roles, and to establish the timeline of the performance evaluation. The questions which the Lab wanted to explore were set forth in RFTOP (AID-SOL-OAA-16-000177). In the kick-off meeting, USAID stressed that the evaluation was to focus on how mSTAR had functioned overall (and not at the individual activity level). USAID wanted to explore related success factors, challenges, and lessons learned to inform the design of a follow-on mechanism, if needed. EnCompass and USAID jointly determined that given the size and scope of the evaluation, interviews with key informants and an accompanying document review were the best data collection methods to obtain the information needed to understand the mechanism’s achievements and challenges. The team determined that structured interviews with USAID field, regional, and headquarters staff; mSTAR management staff; and consortium partners would allow for exploration of performance factors and recommendations for future mechanisms. A document review was considered essential to gain a better understanding of the mechanism’s evolution over time. A quantitative survey was discussed, but it was deemed that such a survey would not yield data that could be aggregated across all groups. Furthermore the team determined that resources would be best spent conducting a larger number of individual interviews and allowing for a broader range of perspectives to be solicited. After the kick-off meeting with USAID, EnCompass engaged in a preliminary document review and drafted an inception report which articulated the context, purpose, and scope of the evaluation; refined the evaluation questions, and provided an overview of the approaches, methods, and tools that would be used. The evaluation team met with mSTAR on November 11, 2016 to review the draft inception report and to discuss the evaluation plan and proposed sources of data. mSTAR’s suggestions and proposed revisions to the list of key informants were discussed with USAID and approved by the COR. The inception report was finalized and approved by both USAID and mSTAR on November 28, 2016. The evaluation reference group (comprised of representatives from USAID and mSTAR) initially identified 44 stakeholders for interviews, but reduced the number to 41 stakeholders after further reflection. Three of these individuals could not be reached during data collection. Thus, April 2017 | mSTAR Performance Evaluation 10 EnCompass conducted interviews with 38 stakeholders. Every effort was made to obtain a balance of perspectives. Exhibit 6, on the next page, illustrates the demographic makeup of the respondents. Interviews were conducted using a structured interview guide which employed an Appreciative Inquiry approach to explore exceptional experiences, values, and wishes for the future of mSTAR. The evaluation team coded and analyzed all data collected through interviews using Dedoose, a cross-platform application, for both content and thematic analysis. Exhibit 6: Evaluation Interviews by Stakeholder Group and Sex (n=38) Annex 2 provides more background on the determination of sample size and successes and limitations of both methods. Annex 3 provides the structured interview tools used for each stakeholder category. Annex 4 provides the list of documents that were reviewed and the list of all interviewees. LIMITATIONS Although the evaluation approach was deemed appropriate, given time and funding constraints, the team recognized the following limitations: • A lack of quantitative data to triangulate with qualitative data and produce more robust findings: Quantitative analysis of performance indicators was not possible due to the lack of a project-level theory of change, monitoring and evaluation framework, and related monitoring data. • The limited size and scope of the evaluation: The limited budget meant EnCompass could not examine the performance of individual projects to assess the mechanism’s overall performance in a more definitive manner. Although the team attempted to reach different stakeholders and to include as many perspectives as possible, these groups April 2017 | mSTAR Performance Evaluation 11 were small, especially at partner organizations and USAID missions. In addition, EnCompass did not reach all stakeholders (38 out of 41), because some respondents could not be reached during data collection. Thus, the sample could be influenced by non-response bias. This should be kept in mind when interpreting evaluation findings. • The rate at which USAID staff rotate in and out of departments made it difficult to find respondents who could speak to the full life cycle of the mSTAR mechanism, what it had accomplished over time, and how it functioned. Turnover of program staff in 2012 also made it difficult to find mSTAR staff who could speak to the full cycle of the mechanism. FINDINGS The evaluation findings are based on a document review of overarching project documents and interviews with 38 stakeholders from USAID/Washington, USAID missions, and mSTAR staff and partners. This analysis includes direct quotations that represent key themes that emerged during data analysis. APPROPRIATENESS OF THE AWARD MECHANISM AND TECHNICAL SCOPE mSTAR originated in the U.S. Global Development Lab in 2012 (then called the Innovation and Development Alliance, or IDEA). It was awarded as an Associate Award under the FIELD-Support Leader with Associates (LWA). mSTAR is managed by FHI 360. USAID often uses LWAs to access the expertise of consortium partners. For service providers, agreements are structured to allow organizations to work with USAID as thought partners and to co-create programs with greater flexibility. However, mSTAR’s status as an Associate Award under an LWA has implications for how USAID can access sub-partner services and how the overall mechanism is managed. Additionally, the Lab’s activity managers have varying levels of experience managing such a mechanism, which presents a unique situation when examining the success of the project.1 This section explores how the mSTAR mechanism’s structure and technical scope enabled USAID to achieve its desired outcomes. It explores the mechanism’s strengths and weaknesses, how the 1 Activity managers oversee individual activities under an individual buy-in. Activity managers for mSTAR included Lab team leads, Lab program assistants, and mission points of contact. April 2017 | mSTAR Performance Evaluation 12 original prime/sub-partner relationship was utilized, and how USAID/Washington managed the agreement. AWARD MECHANISM AND TECHNICAL SCOPE USAID established mSTAR as an Associate Award to enable the Global Development Lab and USAID missions to quickly tap into a range of technical expertise. However, the Associate Award structure constrained USAID in ways it did not anticipate at the time of inception (both in terms of accessing external technical expertise and in terms of activity management). USAID/Washington staff reported that the reason for choosing an Associate Award at inception was because there was a need to allocate funds quickly through a mechanism that was easily accessible to USAID/Washington and USAID missions. As conceived, an Associate Award—a type of cooperative agreement that falls under a larger LWA—would allow USAID/Washington to utilize a broad consortium of partners, while allowing missions easier buy-in to the mechanism. USAID staff reported that the original vision of the mechanism was to be an “extra arm” of the Lab, enabling the Lab to access additional technical expertise easily and expeditiously. During the first year of mSTAR, it became apparent that there was a misalignment between the Lab’s expectations and mSTAR’s. To play the role of thought partner, as originally envisioned, mSTAR proposed bringing on technical directors for each of the three work streams. The Lab, however, keen to access other experts, suggested that mSTAR focus more on management and allow access to a variety of experts from outside mSTAR for technical support. As an Associate Award, bound by specific rules and regulations for competing work, mSTAR could not simply direct work to external experts without a competition process. Furthermore, this request was at odds with the definition of a cooperative agreement or Associate Award and the original scope of the project. To meet USAID’s needs, mSTAR brought a variety of technical experts into its team. However, from USAID’s perspective, the depth and breadth of expertise still did not meet the need and they continued to push for access to external experts of their choosing. Finding 3 expands on this point. “What they want is to have an extension of themselves … So looking back on it I see why they want a buy-in project but it doesn’t fit within the definition of a cooperative agreement or associate award.” —mSTAR “Originally the idea had been a brain trust in that mSTAR would provide technical support through FHI and we ended up being more of a pass-through.” —mSTAR April 2017 | mSTAR Performance Evaluation 13 Over time, mSTAR and USAID staff reported that USAID activity managers became much more directive and mSTAR staff reported that the award began to be managed more like a contract than a cooperative agreement. mSTAR staff reported that USAID staff did not abide by the limitations in the Automated Directives System’s “substantial involvement clause,” which refers to USAID’s limitations in managing a cooperative agreement. The more directive management approach limited mSTAR’s ability to use its development expertise to find the right partner or project. “We would often treat mSTAR as a contractor, meaning we would be very involved in their thinking and tell them who we want to work with in terms of sub-contractors. But FHI’s position has been that this is a cooperative agreement and you’re not supposed to be telling us what to do. That has caused a big source of tension in terms of management of expectations as well as project management.” —USAID/Washington The Associate Award mechanism allowed for quick procurement of funds and a flexible scope, which helped the Global Development Lab and USAID missions implement a range of projects, but it also resulted in ambiguity of mSTAR’s overall vision. USAID mission staff reported that they appreciated the mechanism because it allowed them to access technical expertise through a quick and efficient process, in comparison with other mechanisms with longer procurement cycles. “To buy in to a Leader with Associates is easy. Sometimes procuring a project can take months, so with this particular mechanism, it saves time.” —USAID mission “From our perspective mSTAR is an easy mechanism to buy into … It greatly alleviates the burden of the mission of having to handle it.” —USAID mission USAID/Washington staff reported that as the Lab continued to grow and mSTAR became known throughout the Lab as the easiest mechanism for allocating funds, the flexible scope of the agreement allowed the addition of more projects to the mechanism. Although this was useful for USAID, it resulted in mSTAR’s being called on to provide a wide range of services and support that were not unified under a common vision. “It was meant to be a mechanism that was easy to procure. It was an associate award which had an opportunity to work with a vast array of consortium partners … It was seen as a one stop shop to allow our team to do what we wanted to do. It meant that everything under the sun could fit under the scope of the project.” —USAID/Washington April 2017 | mSTAR Performance Evaluation 14 “They wanted a flexible mechanism that allowed them to experiment and test things and scale things in the field. They didn’t know what the vision was so they made it really broad.” —mSTAR CONSORTIUM mSTAR was not able to engage consortium partners or external technical experts as easily as USAID had envisioned at the time of the award. When mSTAR was awarded, the agreement allowed for engagement of consortium partners. USAID envisioned that mSTAR’s consortium would provide easy access to a broad set of technical experts who could serve both as thought leaders, working with agency staff, and as implementers, turning their ideas into actions in the field. However, the mSTAR consortium members, which were part of the FIELD-Support LWA, were not necessarily the most appropriate for the Lab’s needs. As time passed and USAID's wishes for the mechanism began to evolve into new directions, mSTAR suggested signing on new pre-approved consortium members, however USAID/OAA did not approve this request. This meant that each time FHI 360 wanted to use a new partner, the process was much slower. When Lab staff felt that the technical expertise they needed could be found only in external experts outside of the mSTAR consortium, USAID used mSTAR as a pass-through. As Finding 1 notes, this process proved problematic, due to competitive rules and regulations related to the Associate Award, cost inefficiencies, and dissatisfaction on the part of mSTAR, whose staff hoped to play the role of thought partner, as originally envisioned. In an effort to respond to USAID’s needs, mSTAR began to hire staff with deeper technical knowledge. Although some staff at USAID collaborated with these new mSTAR team members, many in the Lab still wanted mSTAR to be a pass-through to external experts. USAID respondents acknowledged that this misalignment of expectations was a constraint of the structure. “Great people but I think we got the type of mechanism wrong for the type of work we have given them … I think that falls on us, on how we designed the mechanism and what we expected of them.” —USAID/Washington “I would critique the design from our side and we tried to have a place to park money and that we could have a lot of flexibility and direct a lot of the work and pull on some different consultants and such. A huge limitation is the pool we can pick from with mSTAR, so what I April 2017 | mSTAR Performance Evaluation 15 would love to see is across the center we have a different set of firms who we can go to and we can pull from directly.” —USAID/Washington When the mSTAR consortium did have the expertise USAID was looking for, partner engagement was hindered by the size and scope of projects that emerged from the Lab. mSTAR explained that competition requirements under the Associate Award meant they had to allow partners to bid for the work. As the mechanism was rolled out, the Lab decided to implement a broad array of smaller projects, rather than larger, longer-term projects. Because the bidding process takes consortium partners’ time and resources, these smaller projects provided little incentive for partners to bid. mSTAR staff reported that many partners felt there was no coherent, long-term strategy in which they could find out where they fit. mSTAR staff mentioned one occasion when partners went through the bidding process and USAID ended up not choosing any of them. This experience further hindered the relationship between the prime and sub-partners, discouraging sub-partners’ participation in the mSTAR consortium. “From the perspective of organizations in the consortiums, they spent a lot of time and effort in solicitation and see they don’t get it and then they get disengaged with the process entirely.” —USAID/Washington “The original vision was that we would be doing work through consortium of partners. That did not pan out because the work is small and no one wants to bid on [it]. Anything that was under 150,000 was small, and even for bigger firms, under 500,000, they are not going to bid on it.” —mSTAR “Over time the partners felt they were being asked to provide their technical expertise and insight without any real benefit. There was never a long term partnership that developed … What would have made it better would be for there to be a more defined scope for partners to begin with.” —mSTAR USAID SUPPORT FOR THE AWARD MECHANISM mSTAR was challenged by turnover within the Lab and the need to respond to a variety of USAID activity managers who had different expectations and were mostly unfamiliar with the mechanism’s rules. USAID staff reported that one of their main challenges in supporting mSTAR came from unclear expectations and lack of a shared vision.. USAID staff reported a lack of clarity on the roles and responsibilities of different managers, such as the different roles of the activity managers, who April 2017 | mSTAR Performance Evaluation 16 oversee one project at a time, and the agreement officer’s representatives, who not only oversee projects, but also manage relationships between the partner and the agency. USAID staff also pointed to the multitude of changes happening within the agency, which resulted in shifting expectations for mSTAR. “But there was not a lot of clear guidance in terms of sharing of expectations.” —USAID/Washington “Consistency in expectations—the Lab itself is inconsistent and the Center has gone through so many changes. It is hard to adapt and understand how to accommodate those changes. With those changes, clear expectations from our side were not put out.” — USAID/Washington mSTAR staff reported a lack of guidance from USAID, both in desired outcomes for the mechanism overall and in feedback from different managers within the Lab. mSTAR staff reported varied levels of support that depended on the project and which USAID staff member was managing the project. “It never felt like there was a cohesive long term strategy and we struggled with that. If there had been more of a cohesive message coming from the Lab in general then that would have been really helpful.” —mSTAR “So from every level there has been some frustration with how this has been managed. The expectations for this award were not clear from the outset. It was very broadly written and different people had different expectations for what this could do.” —mSTAR USAID and mSTAR staff also recognized that the Lab was new when mSTAR was conceived and made up of technical experts who mostly came from outside of the international development sphere. Many Lab activity managers therefore had limited experience with the rules and expectations associated with different USAID mechanisms. Due to this inexperience, USAID staff admitted, many of the expectations coming from the Lab about what mSTAR could provide were unrealistic. mSTAR and USAID staff both mentioned that management of the mechanism was often misaligned with what could be expected of a single cooperative agreement. “I feel like there have routinely been people in key point of contact roles who simply don’t have the experience and managing relationships with partners.” —USAID/Washington “USAID should make sure that those who are managing activities and managing as technical leads have previous experience managing government projects and everyone should take at least the project management course that USAID offers, not the AOR/COR April 2017 | mSTAR Performance Evaluation 17 training, but the two training up to that should be require to do that before they are given an activity to manager. —USAID/Washington “The level of knowledge of USAID rules and regulations by lab staff. Impatience by lab staff – they don’t understand that NGOs have overheads.” —mSTAR “So one big issue is the lack of training within USAID and the fact that they don’t know how to work with this mechanism or understanding the history of USAID’s own work in this area.” —mSTAR mSTAR staff reported facing other USAID management challenges, such as delays in finalizing scopes of work, starting projects, and putting monitoring mechanisms in place. mSTAR staff reported that they often needed to go through an extensive and time-consuming process, with inputs from a large number of USAID staff, to finalize scopes of work and deliverables. “It took us 8 months to finalize the scope of work. Because the lab didn’t have a clear vision of what they wanted to achieve and they were still figuring things”. —mSTAR “It took us 2.5-3 years to create a results framework. They didn’t even want an M&E plan. They didn’t want to be held to measurements. The results framework got developed after 4 years. —mSTAR mSTAR staff reported that after recognizing these challenges, they worked to mitigate them by opening formal and informal lines of communication. mSTAR staff spoke of introducing frameworks for monitoring their work and ensuring clear lines of communication between mSTAR and USAID. “Even though we weren’t required to submit an M&E plan or do a results framework we made it an explicit management tool to actually do that. That has been fantastic. In order to course correct we actually use the results framework.” —mSTAR “In some cases when things are challenging, we try to communicate. We track everything. So if we agree with the Lab focal point that we have to agree on 2 or 3 check in points with a contractor. We need to keep that.” —mSTAR PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED As defined by USAID, a performance evaluation is designed to identify accomplishments, performance issues, and constraints in the implementation of a project or program. This section April 2017 | mSTAR Performance Evaluation 18 explores USAID/Washington staff and partners’ perspectives about mSTAR’s performance and management, as well as achievements and challenges in implementing the mechanism. PRIME/SUB-PARTNER MANAGEMENT mSTAR effectively managed relationships with partners in terms of - communication, monitoring performance, and compliance with USAID regulations. Partners gave positive feedback on mSTAR’s communication, reporting that mSTAR was responsive and collaborative throughout the design and implementation of the mechanism. They reported that they enjoyed working with mSTAR colleagues overall. “We found the process to be very collaborative … They were really collaborative. They really care about the work and are solid in their knowledge. —Partner mSTAR staff reported that with its internal structures, FHI 360 had a high capability to monitor partner performance. They reported having a technical monitor and contract monitor for every sub-partner, adjusting their monitoring to local organizations’ structures, and giving partners clear timelines and deliverables to enable performance monitoring at various points throughout the mechanism’s duration. “For every single sub we had a technical monitor and a contract monitor. I was tracking what their obligation is or the milestones, etc. … Whoever was the assigned technical person was to make sure that USAID was happy with the technical quality. There was also a counterpart on the USAID side.” —mSTAR “We will have deliverables and a work plan with a timeline set with deliverables. We will be monitoring it pretty closely. There will a lot of engagement. —mSTAR Partners also pointed to useful support from mSTAR in terms of compliance with USAID regulations. “USAID Lab connected us with mSTAR because they have an expertise in finding different orgs, receiving large grants, USAID requirements, what is required from donors, the M and E expectations of USAID, how to do a theory of change, etc.” —Partner “USAID Lab passes funds through mSTAR to get money to us. They require us to work with them because we have never received such a large grant from USAID.” —Partner April 2017 | mSTAR Performance Evaluation 19 mSTAR’s ability to award contracts to external partners was affected by project scope, required technical expertise, and budget. USAID staff indicated that they wished mSTAR had awarded more sub-awards to implementing partners. However mSTAR’s decisions on awarding were based on strategic decisions related to the scope of the projects—size, technical expertise, timing, funding, and location. In terms of timing, mSTAR staff reported that it depended on which partners were available and how quickly USAID wanted to start implementation, because subcontracting took longer than it would for FHI 360 to take on the work directly. “A lot of times it depends on the money. It’s a lot cheaper to hire a sub than do the work ourselves. We charge a much lower rate to USAID when we use a sub. Expertise is a consideration. If there is no presence of FHI in country then we hire a sub. Time considerations are also big since it takes a lot of time to hire a sub.” —mSTAR “Each activity is so distinct so it’s hard to make a universal. We would determine if we needed US based partners. Everything is on the scope of work and what is required by whom and where. Some things are procured locally which is the right thing to do for certain tasks.” —mSTAR SATISFACTION WITH MSTAR PERFORMANCE Different stakeholders reported different degrees of satisfaction with mSTAR performance; with USAID mission staff expressing high degrees of satisfaction, and USAID/Washington staff providing mixed feedback, depending on the expectations and demands of the activity manager in charge USAID mission staff had overwhelmingly positive feedback about their work with mSTAR. Missions pointed to mSTAR’s cultural competency in managing sensitive in-country relationships. Missions also reported an appreciation for mSTAR’s technical assistance and knowledge in communication and knowledge management. “We have been very satisfied. They are going through the initial activities with MoE. That took a lot of negotiation, training, and awareness raising. They have been very good in understanding cultural barriers and finding ways of working with [the] Ministry.” —Mission “FHI 360 handled that really well, it was a lot to ask of them … FHI 360 was very aware of the relationships and the ministries. There was a lot of drama politically. They handled it really well.” —Mission April 2017 | mSTAR Performance Evaluation 20 “We designed the content of the agenda. He [mSTAR staff member] helped us a lot on the design, the content, brought in guests, marketed the event, built a Google platform—did everything, really.” —Mission USAID/Washington staff reported mixed degrees of satisfaction with mSTAR. Many staff reported that mSTAR was a receptive partner that produced good results in terms of logistics, communication, and work with missions. “Generally the way that I see it and the way my colleagues see it, is that mSTAR has been a value more on logistical things and marketing. They have a great design lab that has been critical to our publications. They want us to use them as technical support but that isn’t a strength of theirs so it has caused tension.” —USAID/Washington A few USAID/Washington staff reported dissatisfaction in getting the desired level of detailed financial records from mSTAR. However, USAID and mSTAR staff acknowledged that the level of reporting requested by these activity managers went beyond what was required under a cooperative agreement. The type of mechanism level budgeting and reporting was a challenge for some USAID managers who wanted detailed financial reports for their activities. mSTAR staff noted that the level of detail requested by USAID was not required under a cooperative agreement and, more importantly, that it was too burdensome and would have hindered their ability to focus on project work. “Getting on the same page around the money has been a challenge.” — USAID/Washington Many USAID/Washington staff reported that their dissatisfaction related to a combination of the lack of technical expertise and the inappropriate design of the mechanism. USAID staff acknowledged that it was unrealistic to anticipate that one prime partner could provide all of the services needed under this mechanism, including high levels of technical expertise, logistics, communication, and knowledge management, while implementing projects in the field. “Part of it is that the instrument is not a good fit for the work we are trying to do so we can’t blame them for that. In terms of responsiveness they can be good but it’s not the same as quality.” —USAID/Washington “I wish they would have had much deeper technical knowledge. That would inform a higher level dialogue. They were not set up to do that … The biggest disappoint in them was as a thought leader which was related to their lack of technical depth.” — USAID/Washington April 2017 | mSTAR Performance Evaluation 21 PROJECT ACHIEVEMENTS mSTAR’s main achievements included its work in communication, training, and logistics and the support it provided to USAID missions. USAID/Washington staff highlighted mSTAR’s achievements in providing the Lab with communication and knowledge management support, specifically in regards to the FHI 360’s Design Lab. The Design Lab provided USAID with a resource to improve their work through graphic design and data visualization. mSTAR reported that a significant amount of time was spent by mSTAR writ large in supporting the Design Lab. Staff reported that almost half of their time working with USAID was spent ensuring overall quality control of USAID deliverables. USAID staff also pointed to different training programs and events run by mSTAR, such as training on gender and information and communication technology (ICT) and monthly global meetings, which received high ratings from participants and were seen as highly effective. “On more than one occasion—the design lab that we tap into a lot has been great. Creative, enabled to deliver on a timely manner and mSTAR has been that facilitator to have access to the design lab especially because we don’t have a graphic designer in house.” —USAID/Washington “They did a good job with facilitating events for us. We had a series of monthly global meetings on a technical subject and they were tasked with organizing USAID implementing partners to host these.” —USAID/Washington “I think some of their achievements have been the creation of the mobile access diagnostic, a gender and ICT training and gender and ICT survey tool that they recently produced to give a snapshot of gender digital divide.” —USAID/Washington “The trainings which we do with them. We send 4-5 staff and they bring 1-2 and one of their people will be the MC of the event. Those get very high ratings pretty consistently and the team takes time to digest the last one and make the next one better. That’s a good rhythm we are in.” —USAID/Washington All respondents pointed to mSTAR’s success in working with USAID missions. Respondents mentioned a variety of successful projects, but the two projects highlighted most often were in Bangladesh and Liberia. In Liberia, mSTAR worked with the mission to map and identify opportunities for the government to use mobile money platforms and sustain progress in existing mobile money payments. The project focused on scaling up e-payments to Ministry of April 2017 | mSTAR Performance Evaluation 22 Education staff and teachers and on mapping payment streams from government to people and people to government. Exhibit 7 expands on the mSTAR project in Bangladesh (FHI 360 2015). “Liberia is an activity that was very successful. Another project is the Bangladesh project that has been super successful. They have done a good job of digitizing payment streams for USAID projects. When you see the videos of what people say about how great it was not to travel with tons of cash, it’s inspiring. Bangladesh does a good job of quantifying their results and telling the story of their impact.” —mSTAR “Again I keep referencing Bangladesh work, good learning that came out of that specifically with gender work and supply chain programs and I am trying to think of something else.” —USAID/Washington mSTAR and USAID mission staff also highlighted mSTAR’s work with the Development Informatics team in developing the “Principles for Digital Development.” These guidelines aim to help development practitioners integrate established best practices in the use of ICT for development projects. mSTAR worked collaboratively with the Development Informatics team to develop the nine overarching principles, which are endorsed by more than 60 development partners. “The dfs [Digital Financial Services] team has become successful. Principles for Digital Development did turn out to be really successful. It’s an industry wide initiative.” —mSTAR “I think they did great. The Principles for Digital Development - they helped drive that forward … resulted in a report and recommendations for donors, implementing partners, Exhibit 7: mSTAR Bangladesh mSTAR worked with USAID/Bangladesh to accelerate the adoption of mobile money and electronic payments in the mission’s agriculture, health, and education programs. Highlights include:  Managing USAID/Bangladesh grantees’ transition from cash to mobile payments  Conducting workshops to increase awareness of mobile money and prepare USAID implementing partners in health, education, and agriculture to introduce mobile payments in their operations  Providing on-demand technical assistance to USAID implementing partners. April 2017 | mSTAR Performance Evaluation 23 and resulted in over 60 organizations now have endorsed the principles was an achievement and they played a big part in that.” —USAID/Washington mSTAR staff noted their role in knowledge management and communication—increasing the Lab’s presence beyond USAID—as a key achievement. “mSTAR has tried to serve as a thought leader so convening and arranging events that are in the interest of USAID to promote them … Their main focus was mainly internal but we are trying to get them more publicity.” —mSTAR mSTAR, USAID missions, and partners all recognized the program’s important achievements in broadening awareness of ICT in the development community. mSTAR staff pointed to their achievements in convening development organizations to introduce them to the benefits of ICT for Development (ICT4D). USAID mission and partner staff spoke of mSTAR’s achievements in broadening awareness of ICT4D and connectivity among key stakeholders from government, nongovernmental organizations, and the private sector. USAID mission staff spoke of the importance of raising awareness in mission offices. “The report we wrote has been influential in USAID’s approach in how they approach connectivity. We got very positive feedback from them, Microsoft, Facebook, and others. It opened their eyes to different ways of connecting people.” —Partner “To show the mission that transitioning to cash from digital is possible and that there are significant benefits to doing so.” —USAID mission PROJECT CHALLENGES This report has already acknowledged several challenges, including inadequate access to technical expertise (see Finding 1), a clear vision for the project (see Finding 2), and USAID’s management experience and capacity (see Finding 4). This section features additional challenges that spoke of or expanded on as challenges that affected project performance. High staff turnover at USAID and on the mSTAR team was a challenge to building relationships and consistency within the mechanism. USAID respondents reported that one of the main challenges for mSTAR was the high staff turnover in the Lab and in the mSTAR team. Turnover within the program created challenges of consistency and relationship-building with the Lab, and turnover at USAID led to a lack of consistency from the Lab in what was expected from mSTAR. April 2017 | mSTAR Performance Evaluation 24 “Staffing is big challenge—a lot of turnover. Lack consistency and building teams. We go into meetings and there is a new person and we say, who is this?” —USAID/Washington “If I was on the other side if I was at mSTAR I would be insanely frustrated and I feel like there has been a lot of changes at the Lab and a lot of lack of clarity from the lab.” —USAID/Washington mSTAR was challenged by the disconnect between missions and the Lab in terms of priorities and lines of communication. mSTAR staff and USAID missions also reported a disconnect between the vision coming from USAID/Washington and the requests coming from USAID missions. Respondents reported that projects felt “top down,” created in Washington without missions’ input. Mission staff also reported a lack of clarity about their points of contact between mSTAR and USAID staff. “If someone only has experience in HQ and doesn’t understand the field that can be a challenge. It makes it difficult for us and them to understand and speak the same language.” —mSTAR “I think people in the mission have the perception that Lab is coming in and giving top down direction that nobody wants and nobody is looking for.” —mSTAR “I do think some clarity would be better so we know who to contact.” —USAID mission LESSONS LEARNED FOR FUTURE MECHANISMS This section reports on feedback from USAID/Washington, missions, mSTAR, and partners on what they would keep, enhance, and add to future mechanisms. It also addresses lessons learned about project management and implementation. WHAT TO MAINTAIN IN THE FUTURE mSTAR’s work in communication and facilitation should be kept as key components of a future mechanism. USAID/Washington staff pointed to mSTAR’s work with the Design Lab as an important resource for them to be able to access in the future. Within this support, mSTAR pointed to their work on editing, formatting and quality control of USAID documents which is a large part of their work with the design lab. In addition, the program’s work with developing toolkits and running training for USAID was seen as important to continue the visibility of ICT4D inside and outside the agency. April 2017 | mSTAR Performance Evaluation 25 “I think the design work, I know the lab has design people but it doesn’t seem to be a service offered to us but somehow we would need access to that kind of a service.” — USAID/Washington “[Regional Development Mission for Asia staff] were very happy with the work we were doing with mSTAR in running their learning series and they also used them to develop toolkits.” —USAID/Washington mSTAR’s work with the missions should continue to be leveraged and expanded in a future mechanism. Interview respondents reported that mSTAR’s work with missions should continue under future mechanisms. USAID/Washington and mission staff pointed to achievements in supporting missions in technical areas (such as e-payments) and in more administrative tasks (such as helping newer implementing partners navigate the USAID system) as important to continue in future mechanisms. USAID/Washington staff pointed to the successes in Bangladesh and Liberia as important models for future projects with missions. “A couple of articles have come out showing that people are excited about e-payment. This is helping the cause of mSTAR. They should have an extra 6 months at a no-cost extension.” —Mission “Acting as the entity that deals with grantees to hand hold them through the system. That is something that will continue to be required. USAID doesn’t have the resources to deal with grantees at this level.” —Mission “Some of the in country work akin to what they did in Bangladesh in terms of what they did with mobile money and as an IP on the ground.” —USAID/Washington Mission staff and partners pointed to a need for projects with flexible scopes that can be adapted to country contexts. Missions also saw future mechanisms as needing to engage a broader array of actors, such as the private sector. mSTAR staff recommended taking a longer￾term approach to looking at the sustainability of projects implemented at the mission level. “Having a broader and more flexible scope might have been useful. Countries vary, needs vary. Make the mechanism as flexible as possible and adapt to the situation.” —USAID mission April 2017 | mSTAR Performance Evaluation 26 “I think there is a real opportunity for USAID to raise the profile of mSTAR type of work and focus on its adoption by missions and bureaus.” —Partner “Very specifically I would love to have the Lab help us create real sustainable plans to continue this work via missions. So kind of the original mission of mSTAR.” —mSTAR “Not only digital payments through mobile systems, but exploring more opportunities to work with private sector, figuring out the most influential players, and convincing them to more involved in digital financial services.” —USAID mission LESSONS LEARNED FROM MSTAR MANAGEMENT AND STRUCTURE When mechanisms are not “fit for purpose,” USAID and implementing partners must work to achieve goals within a system that is constrained. Although USAID staff reported that they were not able to access the technical expertise they wanted from mSTAR, most recognized this as being a result of the mechanism’s design. Staff explained that the Lab required mSTAR to have deep technical expertise on a diverse array of ICT topics and the ability not only to assist with communication, training, conferences, and logistics, but also to provide technical assistance to USAID missions. USAID staff acknowledged that any one organization would be challenged to meet such a broad range of needs. All the while, USAID staff recognized that it was not cost-efficient or effective to use mSTAR as a pass￾through for technical expertise. As a result there was an acknowledged misalignment between what USAID wanted and what mSTAR was able to provide. For future mechanisms, USAID staff suggested using different mechanisms for different purposes and trying to find a more cost-efficient way to access targeted technical expertise. However, USAID staff did not have clear suggestions for specific alternative mechanisms. “We need to bring in people and quickly in some sort of agile way and cost effective way. Problem with mSTAR mechanism is isn’t cost effective to do it like that.” — USAID/Washington “Looking toward one organization to provide thought leadership and implementation is unrealistic.” —USAID/Washington “I think there was a lot of value in having our center have one go to mechanism that we own and use for a lot but then also get creative in and better aware of what other mechanisms around the agency offer … maybe needing a mechanism for staffing – to hire an extension of our team and that might be distinct. Maybe another for product design and April 2017 | mSTAR Performance Evaluation 27 training work but doing the actual on the ground field based work could all be consolidated in one mechanism.” —USAID/Washington “Getting access to independent consultants is important so having a mechanism that allows for that. You can’t expect one organization to be able to do everything. Perhaps just pick one that can do sub-grant making really well.” —USAID/Washington USAID and mSTAR also recognized the need to use mSTAR more effectively in the areas in which the team excels, specifically with regard to their work with missions and in communication and knowledge management. USAID expressed a need to build on mSTAR’s successes, rather than getting caught up in the misalignment of expectations that occurred throughout the project. Very specifically I would love to have the lab help us create real sustainable plans to continue this work via missions. So kind of the original mission of mSTAR. —mSTAR They do well with something in the field that is time bound and project based. — USAID/Washington Global Development Lab staff must have the necessary skills and understanding of USAID rules and regulations to manage mechanisms. mSTAR and USAID staff reported a need for Lab staff to learn how to manage future mechanisms. As a first step, staff recommended ensuring that there is a clear vision for what the mechanism is meant to accomplish, including program goals and objectives and how partners are expected to reach them. mSTAR staff suggested engaging the prime partner in this process to allow their input and ensure that partners are clear on the objectives of the mechanism. mSTAR staff also suggested establishing frameworks to measure the goals of the program and assess whether partners are on track to meet the goals. “I think it would be helpful to have a better understand for people that work with the mechanism… best practices with working with the mechanism and more standard procedures of how things are done in terms of how you obligate and manage funds and manage activities through.” —USAID/Washington “Really do your homework on the types of mechanisms that do exist or can exist. I know that some groups in USAID are doing co-creation workshops. Perhaps a co-creation workshop would have been a better way to go. Think through not only what you want to accomplish but what the limitations are.” —mSTAR April 2017 | mSTAR Performance Evaluation 28 “The biggest thing would be joint project-wide activity planning with Lab/mSTAR to make sure we are being strategic so that it is not piecemeal stuff. What are their ideas, objectives, goals?” —mSTAR USAID missions should be engaged in the design and benefits of future mechanisms. USAID/Washington and mSTAR staff reported that future programs needed to spend more time engaging the missions. Staff reported that USAID needs to invest in marketing these mechanisms to its missions, whose staff often do not know the mechanisms exist. In addition, USAID reported a need to engage missions earlier in the process of designing and rolling out projects. mSTAR staff reported a misalignment between Lab and mission priorities and a need to align those goals in the future. “You really need engagements with the missions. The reason that Bangladesh was a win for mSTAR is because the activity manager got the money, pushed them, championed them. He helped them to move in the mission … you have to have a partner that will be your advocate.” —USAID/Washington “We did a survey and there was a huge disconnect between what the mission is interested in and what the Lab is. The mission is not interested in job aids and toolkits and calling the Lab for help. They want evidence-based experiences and [technical assistance] on the ground.” —mSTAR “Allow for missions to participate... Maybe have a representative body to guide and influence the lab’s work.” —mSTAR April 2017 | mSTAR Performance Evaluation 29 CONCLUSIONS Although the Associate Award enabled rapid allocation of funds and a flexible scope of work, the mechanism did not align with USAID’s vision for efficient and cost-effective access to a broad array of expertise. (Based on findings 1 and 3) Establishing mSTAR as an Associate Award enabled USAID to allocate funds quickly and have a mechanism that allowed the Lab to create projects with flexible scopes of work, which are often needed in the changing ICT landscape. However, the range of projects required deep technical expertise in an array of areas under ICT4D, which mSTAR alone could not always provide. As USAID sought expertise outside the mSTAR consortium, the structure of the Associate Award and USAID/Washington staff’s understanding of how to use the mechanism was constraining. mSTAR was required to competitively bid each project, a process that required time and resources. Moreover, the lack of incentives for sub-partners to buy into smaller projects resulted in FHI 360’s being unable to use the consortium as expected. When partners did not buy in, USAID used mSTAR as a pass-through mechanism, resulting in high overhead costs and frustration within the mSTAR team. Although the mechanism allowed a broad array of projects to be implemented, there was a misalignment of expectations on the overall vision for mSTAR and how it should be managed. (Based on findings 2 and 4) The flexibility of the mechanism allowed USAID to consistently add projects with a range of scopes and technical focus. However, Lab staff were less familiar with managing USAID mechanisms, leading to frequent misalignment of expectations in what mSTAR could provide and how it should be managed. mSTAR staff reported that the mechanism was managed more like a contract than a cooperative agreement. In addition, because the scope of work continued to expand and different activity managers had their own views of the mechanism, there was never a clear vision of the program’s overall purpose. mSTAR effectively managed relationships with partners and missions, although the number of subcontracts awarded was less than anticipated due to constraints of the mechanism. (Based on findings 5 and 6) Partners and missions reported that mSTAR was easy to access and buy into and that the mechanism was effectively managed in terms of communication and performance monitoring. Partners specifically pointed to their appreciation of mSTAR’s ability to help them work within USAID regulations. Missions highlighted the ease with which mSTAR allowed them to access April 2017 | mSTAR Performance Evaluation 30 technical expertise in ICT4D. However, mSTAR was not able to engage the number of consortium partners originally envisioned, due to the small project scopes, the required technical expertise, and the budgets allocated. mSTAR is widely recognized by USAID staff as having achieved high levels of success in working with USAID missions. The mSTAR team was also successful in meeting some of USAID/Washington’s needs as well. Challenges which arose were related to the provision of certain types of technical assistance to the Lab. (Based on findings 7 and 8) All respondents recognized mSTAR’s success in communication, logistics, convening, and training at global and country levels. Respondents also reported a high degree of satisfaction with mSTAR’s achievements with the missions. Missions appreciated mSTAR’s technical expertise and ability to understand country contexts and help them manage difficult situations. Several USAID/Washington staff reported that they were unable to get access to the technical expertise needed from mSTAR . However, they recognized mSTAR’s efforts to fulfill their needs and the challenges of addressing the full range of expectations under a single mechanism. High staff turnover and changing USAID/Washington goals challenged the implementation of mSTAR. (Based on findings 9 and 10) There was a significant amount of staff turnover throughout the 5-year implementation period, in the Lab and within mSTAR, which all respondents reported as a challenge to relationship￾building and consistency. In addition, the USAID/Washington turnover resulted in changing priorities coming out of the Lab which presented challenges for project implementation at the mission level. Stakeholders agreed that mSTAR’s communication and facilitation work should continue, and its successful work with USAID missions should be expanded to engage missions earlier in the process. (Based on findings 11, 12, and 15) mSTAR’s ability to increase awareness of the Lab’s work beyond USAID should continue under future mechanisms. Specifically, mSTAR’s Design Lab allows USAID to produce high-quality reports and deliverables. In addition, mSTAR’s ability to convene and train partners and other offices in the agency will be important in continuing to market the Lab’s work. There was also recognition that future mechanisms need to engage missions earlier in the project design process. mSTAR’s relationships with missions and its success implementing projects in the field should be leveraged to engage missions throughout the process. April 2017 | mSTAR Performance Evaluation 31 Stakeholders reported that USAID should build its own project design and management capacity, in order to ensure that future mechanisms are intentionally designed to align with the agency’s goals, while allowing flexibility to adapt to changing priorities. (Based on findings 13 and 14) While it was widely recognized that the Lab brings a unique group of experts into USAID, respondents suggested spending more time on ensuring that they are properly trained to design and manage mechanisms like mSTAR. Respondents suggested that before award, USAID staff need to be clear about the overall vision, align that vision with the mechanism, and communicate that vision with implementing partners. To avoid management issues, USAID staff also need to be trained on what it means to be an activity manager and how to support implementing partners with different types of mechanisms. April 2017 | mSTAR Performance Evaluation 32 ANNEX 1. EVALUATION STATEMENT OF WORK As stated in the award, the performance evaluation will (1) determine the appropriateness of the project’s original technical scope and award mechanism; (2) assess program performance, achievements, and challenges; (3) provide recommendations for future mechanisms; and (4) compile knowledge of lessons learned and areas for improvement. The evaluation team will interview key informants representing mSTAR project management, NGO, IPO, and private sector partners, and USAID managers at missions and in Washington. The team will also review key project documents, such as the award, workplans, and project reports. Information gathered from key informant interviews and reports will be analyzed and synthesized into an evaluation report that covers three main areas: mSTAR’s achievements and challenges, lessons learned, and recommendations for future programming. In consultation with USAID, the evaluation team included, and then refined, a list of illustrative evaluation questions (Exhibit 8) was included to meet the evaluation’s objectives. Exhibit 8: Original and Revised Evaluation Questions Original Evaluation Question in RFP Revised Evaluation Question 1. Appropriateness of the Project’s Original Technical Scope and Award Mechanism a) To what extent did the projects’ design influence the award mechanism? b) What were the pros and cons of using the award mechanism? c) Was the choice of mechanism appropriate for the activities’ intended outcomes? d) Were the “Prime” and “Sub” partners (and its operational arrangement) appropriate? e) Did mSTAR staff feel they were given adequate guidance from USAID on what to accomplish? a) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for mSTAR at the time of inception? b) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? N/A (removed the original 1c and embedded in 1a) c) Has the prime/sub-partner structure originally envisioned to implement mSTAR activities been effective? d) Has USAID provided adequate guidance to mSTAR regarding desired outcomes? Of the program overall? Of particular projects? April 2017 | mSTAR Performance Evaluation 33 Original Evaluation Question in RFP Revised Evaluation Question 2. Program Performance, Achievements, Challenges, and Lessons Learned a) How did the prime partner work with project sub-partners? b) What were the actions taken (by the project) to mitigate risks and course correct during project design and implementation? c) How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from mSTAR? d) How useful, appropriate, and timely was the projects’ performance? e) What were the programs’ achievements and challenges? a) How has the prime partner worked with project sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? b) What have been the actions taken (by the project) to mitigate risks and course correct during project design and implementation? c) How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from mSTAR? d) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate and timely? e) What have been the programs’ achievements and challenges? 3. Recommendations for Future Mechanisms a) What existing gaps and future technical directions/issues would need to be addressed in a follow-on mechanism that is not currently being addressed by mSTAR? b) What kinds of inputs are no longer needed? c) What additional partners need to be brought on? a) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from mSTAR need to be continued and what anticipated services are likely to be needed? b) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? April 2017 | mSTAR Performance Evaluation 34 ANNEX 2. EVALUATION METHODS AND LIMITATIONS The EnCompass team’s evaluation approach honored the guidance put forth in the USAID Evaluation Policy and drew from their experience and values, exemplified by the following: • Evidence-based evaluation: EnCompass conducted an evaluation that was supported by quantitative and qualitative information that is reliable, valid, and generalizable. • Systems thinking: EnCompass recognized the complexity of the projects and the environment in the countries where mSTAR was implemented and the need to incorporate this context in the evaluation design. • Utilization-focused and participatory methods: EnCompass actively engaged target beneficiaries and key stakeholders in articulating an evaluation frameworks that was meaningful and feasible, and aligned with USAID’s desire that the evaluation directly inform future programming decisions. • Appreciative approach: The evaluation team recognized and built on the strengths of key actors and other resources (e.g. prior evaluations, different partners, and established reporting systems). EnCompass’ appreciative approach discovered examples of success, analyzed best practices, solicited innovative ideas, and ensured that the evaluation was contributing to a shared vision of success. • Integration of gender equality and social inclusion: In line with USAID’s Gender Equality and Women’s Empowerment Policy, our approach engaged a diversity of stakeholders and voices in the evaluation process. EnCompass employed structured interviews and document review as the two main data collection methods. The evaluation team, along with USAID, deemed that these methods corresponded directly to the questions being asked and would generate the highest-quality and most credible evidence possible, taking into consideration time, budget, and other practical considerations. The data collection matrix is featured in Exhibit 9. April 2017 | mSTAR Performance Evaluation 35 Exhibit 9: Data Collection Matrix Evaluation Question Data Collection Method 1. Appropriateness of the Projects’ Original Technical Scope and Award Mechanism a) Was the structure of the award agreement (i.e., the cooperative agreement) appropriate for mSTAR at the time of inception? b) What have been the pros and cons of using the cooperative agreement versus other award mechanisms? N/A (removed the original 1c and embedded in 1a) c) Has the prime/sub-partner structure originally envisioned to implement mSTAR activities been effective? d) Has USAID provided adequate guidance to mSTAR regarding desired outcomes? Of the program overall? Of particular projects?  Document review  Interviews with USAID staff, implementing partner, and sub-partners on technical scope, award mechanism, and partnership roles/management. 2. Program Performance, Achievements, Challenges, and Lessons Learned a) How has the prime partner worked with project sub-partners in terms of communication, decision making, management, delegating tasks, and monitoring? b) What have been the actions taken (by the project) to mitigate risks and course correct during project design and implementation? c) How satisfied have various stakeholders (USAID, sub-partners) been with the work done by and assistance from mSTAR? d) Do USAID stakeholders feel the support they receive from these programs is useful, appropriate and timely? e) What have been the programs’ achievements and challenges?  Interviews with USAID staff, implementing partners and sub-partners on working arrangements and lessons learned  Document review on results 3. Recommendations for Future Mechanisms a) What types of technical assistance should be included in a future support mechanism? What types of existing support(s) from mSTAR need  Interviews with USAID staff, implementing partner and sub-partners on recommendations April 2017 | mSTAR Performance Evaluation 36 Evaluation Question Data Collection Method to be continued and what anticipated services are likely to be needed? b) What lessons have been learned regarding structure, management, and participants/partners that USAID should apply to future mechanisms? The sampling frame for the interviews was selected in coordination USAID, keeping in mind time and funding limitations, geographic reach, and desire for as much representation as possible. Although the EnCompass team wanted to provide more data on the overall performance of the mSTAR in quantitative terms, the lack of any common theory of change, monitoring and evaluation framework, and related monitoring data made that impossible. The evaluation budget did not afford the opportunity for EnCompass to look at the performance of each project on its own. The rate at which USAID staff rotate in and out of departments made it difficult to find interviewees who could speak to the full life cycle of the mechanism, what it had accomplished over time, and how it functioned. Although EnCompass made every effort to balance the perspectives of different stakeholders, only 9 percent of the interviews were conducted with partners. This was due, in part, to the fact that FHI 360 did not end up using partners to the extent envisioned when the mechanism began (see Finding 3). The evaluation team attempted to reach a variety of stakeholders and include as many perspectives as possible; however, the number for each stakeholder was small. Care should be taken in interpreting evaluation findings. April 2017 | mSTAR Performance Evaluation 37 ANNEX 3. DATA COLLECTION INSTRUMENTS To be completed before the interview: Name of person interviewed: Title: Organization and unit: Interviewer: Date: INTRODUCTION Thank you very much for setting aside time to talk with me/us today. I work for EnCompass LLC, a consulting company based in the United States that has been contracted by USAID to conduct a performance evaluation of mSTAR. The purpose of this evaluation is to determine the appropriateness of the project’s original technical scope and award mechanism; assess the project’s performance, achievements, and challenges; provide recommendations for the future; and compile knowledge of lessons learned and areas for improvement. This is not an evaluation of your organization or your work. For this evaluation, we are conducting interviews with USAID in Washington, D.C., and in missions; with mSTAR team members; and with implementing partners. Before I/we begin, I/we want to let you know that any information or examples we discuss during this interview will not be attributed to any specific person or institution. All quotes used in the evaluation report will be attributed to a general stakeholder group, not to an individual, and all identifying information will be removed. The interview will take 45 minutes to 1 hour. Before I begin, do you have any questions about this interview? April 2017 | mSTAR Performance Evaluation 38 MSTAR INTERVIEW PROTOCOL: USAID/WASHINGTON A. APPROPRIATENESS OF THE PROJECT’S ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. What do you do at USAID? 2. What has been your involvement with mSTAR? When did this start? 3. How would you describe the vision mSTAR when it was conceived and awarded? Has that vision changed over time? If so, how? 4. How has being in a cooperative agreement helped or hindered the achievement of mSTAR’s vision? Looking back, could there have been an alternative setup? 5. How did the choice of a consortium for mSTAR come about? What was the rationale? 6. How has that structure helped or hindered mSTAR’s vision? Looking back, could there have been alternatives to this structure? 7. What has been the support given by USAID over time to mSTAR? (Probe for monitoring requirements, vision, reporting, etc.) How could this have improved? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. How satisfied has USAID been with the work done by and assistance from mSTAR? (Probe for utility, appropriateness, and timeliness.) 2. In your view, what have been the project’s main achievements? 3. In your view, what have been the project’s main challenges? 4. Please think of a time when you were really pleased with the support you received from mSTAR. Tell me the story of that time. Who was involved? What happened? What enabled that interaction/project/support to be a success? C. FUTURE DIRECTIONS 1. What are the top three to five services/supports that mSTAR offers to USAID that will continue to be needed in the future? 2. If USAID were to develop a follow-on mechanism to pick up where mSTAR leaves off, are there any relevant services that are not offered by mSTAR but that USAID should be sure to include in future programming? April 2017 | mSTAR Performance Evaluation 39 3. Overall, what lessons have been learned regarding the structure, management, and results of this project that USAID should apply in creating a future mechanism? D. CONCLUSION 1. Is there anything that you’d like to add or discuss that we have not covered in our conversation today? 2. Do you have any questions for me? MSTAR INTERVIEW PROTOCOL: MSTAR TEAM A. APPROPRIATENESS OF THE PROJECT’S ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. What do you do at mSTAR? When did you start? Note: Only ask the remaining questions in Section A to senior leaders of mSTAR or those that have been at the project awhile. Otherwise, go to Section B. 2. How would you describe the vision of mSTAR when it was conceived and awarded? Has that vision changed over time? If so, how? 3. How has being in a cooperative agreement helped or hindered the achievement of mSTAR’s vision? Looking back, could there have been alternatives to a cooperative agreement? (Probe for contract, associate with a leader award.) 4. How did the choice of a consortium for mSTAR come about? What was the rationale? 5. How has that structure helped or hindered mSTAR vision? Looking back, could there have been alternatives to this structure? 6. What has been the support given by USAID over time? (Probe for monitoring requirements, vision, reporting, etc.) How could this have improved? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. Think about a time that you were really proud of your involvement with mSTAR and tell me that story. What was happening? Who was involved? What made this experience a peak experience for you? 2. In your view, what have been the project’s main achievements? 3. In your view, what have been the project’s main challenges? April 2017 | mSTAR Performance Evaluation 40 4. What have been the actions taken by mSTAR to mitigate risks and course-correct to address these challenges? 5. How does FHI 360 decide whether or not to engage a consortium partner or other subcontractor? In what circumstances does FHI 360 decide to do the work themselves? 6. What is the process for engaging partners? 7. What controls does FHI 360 have in place to monitor partner performance and manage quality? 8. What lessons has FHI 360 learned about how to best utilize partners/subs for the mSTAR work. C. FUTURE DIRECTIONS 1. Looking forward, what is the value added of mSTAR for USAID? What could be added that is not there already? 2. Overall, what lessons have been learned regarding the structure, management, and results of this project that USAID should apply to future mechanisms? D. CONCLUSION 1. Is there anything that you’d like to add or discuss that we have not covered in our conversation today? 2. Do you have any questions for me? MSTAR INTERVIEW PROTOCOL: USAID MISSION A. APPROPRIATENESS OF THE PROJECT’S ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. Tell me about the process of accessing mSTAR for the mission from USAID/Washington? (Probe: What was the impetus? How did you hear about it? Was this particularly easier or harder in comparison to other mechanisms? If so, how?) B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. What service has mSTAR provided to your mission? (Probe for how that came about, when it began, what the activity was, and who implemented it.) April 2017 | mSTAR Performance Evaluation 41 2. How satisfied has USAID been with the work done by and assistance from mSTAR? (Probe for ease of doing business, utility, appropriateness, and timeliness.) 3. In your view, what was mSTAR’s main achievement? 4. What wishes might you have to improve the service/support you received? C. FUTURE DIRECTIONS 1. Looking forward, what types of existing support from mSTAR would your mission need? 2. Is there any service that mSTAR is not providing that you wish you could access? D. CONCLUSION 1. Is there anything that you’d like to add or discuss that we have not covered in our conversation today? 2. Do you have any questions for me? MSTAR INTERVIEW PROTOCOL: MSTAR PARTNER A. APPROPRIATENESS OF THE PROJECT’S ORIGINAL TECHNICAL SCOPE AND AWARD MECHANISM 1. Was your organization part of the original consortium of mSTAR partners? 2. How have you been notified and awarded work from mSTAR? What has this process been like? Has it been what you expected as a partner? 3. What have been the strengths and weaknesses of this award process? B. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. Please tell me a bit about a project you did with mSTAR. (Probe for how project was awarded, what it was, and how long it lasted.) 2. How did FHI 360 communicate with you during project implementation? (Probe for methods and frequency.) 3. What did FHI 360 expect from you in terms of performance? (Probe for deliverables and reporting.) 4. What technical or managerial support did you get from FHI 360 during project design and implementation? April 2017 | mSTAR Performance Evaluation 42 5. Overall, what were the strengths of FHI 360’s communication and management? 6. How could FHI 360’s communication and management of your work been improved? C. FUTURE DIRECTIONS 1. Looking forward, what are three to five ways that you think mSTAR could be improved? D. CONCLUSION 1. Is there anything that you’d like to add or discuss that we have not covered in our conversation today? 2. Do you have any questions for me? April 2017 | mSTAR Performance Evaluation 43 MSTAR INTERVIEW PROTOCOL: TECHNICAL ASSISTANCE RECIPIENT A. PROGRAM PERFORMANCE, ACHIEVEMENTS, CHALLENGES, AND LESSONS LEARNED 1. Please tell me a bit about the assistance that FHI 360 provided to you. (Probe for what it was, when it started, how long it lasted, how it was initiated, and whether it was required.) 2. How did FHI 360 work with you during the service period? (Probe for methods, frequency, and communication.) 3. Overall, how satisfied were you with FHI 360’s assistance? (Probe for utility, appropriateness, and timeliness.) 4. In your view, what was FHI 360’s main value added? 5. In your view, what could FHI 360 have done differently or improved? B. FUTURE DIRECTIONS 1. Looking forward for your organization, are there other services/support that FHI 360 offers that you would like to access in the future? C. CONCLUSION 1. Is there anything that you’d like to add or discuss that we have not covered in our conversation today? 2. Do you have any questions for me? April 2017 | mSTAR Performance Evaluation 44 ANNEX 4. SOURCES OF INFORMATION DOCUMENTS REVIEWED FHI 360. 2013. Annual Report Year 1: mSTAR. Arlington, VA: FHI 360. FHI 360. 2014. Annual Report Year 2: mSTAR. Arlington, VA: FHI 360. FHI 360. 2015. Annual Report Year 3: mSTAR. Arlington, VA: FHI 360. FHI 360. 2016a. Annual Report Year 4: mSTAR. Arlington, VA: FHI 360. FHI 360. 2016b. mBangladesh 2-pager. Arlington, VA: FHI 360. FHI 360. 2016c. mSTAR Monitoring and Evaluation Plan. Arlington, VA: FHI 360. USAID. 2012. mSTAR Cooperative Agreement. Washington, D.C.: United States Agency for International Development. April 2017 | mSTAR Performance Evaluation 45 ANNEX 5. DISCLOSURE OF CONFLICTS OF INTEREST Name Elisa Knebel Title Senior Evaluation and Learning Advisor Organization EnCompass LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) mSTAR I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. X April 2017 | mSTAR Performance Evaluation 46 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 8, 2017 Name Kelsey Simmons Title Evaluation Specialist Organization EnCompass LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) mSTAR I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or X April 2017 | mSTAR Performance Evaluation 47 the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date February 15, 2017 Name Beeta Tahmassebi Title Director Organization EnCompass LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) mSTAR I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. April 2017 | mSTAR Performance Evaluation 48 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 10, 2017 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov