December 2020 EVALUATION Final Evaluation for Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Project This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Michael J. Midling of The Mitchell Group, Inc. (TMG). Final Evaluation for Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Project December 30, 2020 Submitted to: Nadia Blackton, Evaluation COR, USAID/Egypt The Mitchell Group, Inc. (TMG) 1816 11th Street, NW Washington, DC 20001 Tel: 202-745-1919 Jenkins Cooper, Vice President/Operations, TMG Email: jenkinsc@the-mitchellgroup.com Phone: +001-202-567-1097 (Mobile) 202-350-0027 (Direct) Walidah Willoughby, Program Manager, TMG Email: walidahw@the-mitchellgroup.com Phone: +001-202-350-0013 Cooperative Agreement No. SINGLE Award IDIQ Contract # AID-263-I-15-00001 Task Order No. AID-OAA-I-15-00028 / 7202632OFOO007 FINAL REPORT: December 30, 2020 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Evaluation of Final Evaluation for Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Task Order No: AID-OAA-I-15-00028 / 7202632OFOO007 USAID Activity Name: Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Cooperative Agreement No: SINGLE Award IDIQ Contract # AID-263-I-15-00001 Total Estimated Amount $ IDIQ Total 24,586,391, CORE Contract $9,586,391, Task Orders Ceiling: $14,913,609 Implementing Partner: The QED Group, LLC Development Objective: N/A Evaluation Coverage: Nationwide Activity Geographic Regions: Egypt Evaluation Geographic Coverage: Egypt Final Evaluation for SIMPLE Project Report i TABLE OF CONTENTS ACRONYMNS ............................................................................................................................................. ii EXECUTIVE SUMMARY............................................................................................................................ ii BACKGROUND ..........................................................................................................................................................iii EVALUATION METHODOLOGY ................................................................................................................................. iii FINDINGS..................................................................................................................................................................iii CONCLUSIONS ........................................................................................................................................................... v RECOMMENDATIONS...............................................................................................................................................vi 1. Introduction............................................................................................................................................ 1 1.1 Background, Context, and Purpose of SIMPLE.................................................................................................... 1 1.2 Evaluation Purpose ............................................................................................................................................. 3 1.3 Evaluation Questions........................................................................................................................................... 3 1.4 Methodology ....................................................................................................................................................... 3 1.4.1 Data Collection............................................................................................................................................. 4 2. Findings, Conclusions, and Recommendations.......................................................................................... 6 2.1 Evaluation Question 1: Effects on USAID Capacity.............................................................................................. 6 2.2 Evaluation Question 2: Effective MEL Tasks and Processes ..............................................................................14 2.3 Evaluation Question 3: Remaining Challenges..................................................................................................18 2.4 Evaluation Question 4: Operational Structure and Co-location........................................................................24 3. Annexes................................................................................................................................................ 29 Annex 1- FCR Table .................................................................................................................................................29 Annex II – Scope of Work .......................................................................................................................................33 Annex III – Background Documents.......................................................................................................................42 Annex IV – Evaluation Team Members..................................................................................................................43 Annex V – Instruments...........................................................................................................................................45 Annex VI – Key Personnel Conflict of Interest (COI) Forms...................................................................................68 Final Evaluation for SIMPLE Project Report ii ACRONYMNS ADS Automated Directives System AMELPs Assessment, Monitoring, Evaluation, and Learning Plans AOR Activity Officer’s Representative BK Beit al Karma CDCS Country Development Cooperation Strategy CIO Chief Information Officer COR Contracting Officer’s Representative DG Democracy and Governance DIS Development Information Solution DO Development Objectives DQA Data Quality Assessment EG Economic Growth ET Evaluation Team FGD Focus Group Discussion GOE Government of Egypt IDIQ Indefinite Duration Indefinite Quantity IP Implementing Partner IR Intermediate Results KII Key Informant Interview KM Knowledge Management M&E Monitoring & Evaluation MEL Monitoring, Evaluation, and Learning MSC Most Significant Change NSCE North South Consultants Exchange PAD Project Appraisal Document PIRS Performance Indicator Reference Sheet PITT Performance Indicators Tracking Tables PMP Performance Monitoring Plan PPR Performance Plan and Reporting SIMPLE Services to Improve Performance Management, Enhancing Learning & Evaluation TMG The Mitchell Group, Inc. USAID United States Agency for International Development Final Evaluation for SIMPLE Project Report iii EXECUTIVE SUMMARY BACKGROUND In 2015, USAID/Egypt entered into a five-year contract, with a period of performance from July 2015 to July 2020, with the QED Group (QED) for the “Services to Improve Performance Management, Enhance Learning and Evaluation” (SIMPLE) Project. Support areas in its design included performance management planning; performance management training of implementing partners (IPs) and staff; field-based monitoring; evaluations; assistance in collecting baseline data; conducting sectoral and geographic assessments; conducting topical and geographic research; and providing assisting with other development design tasks as required. SIMPLE was designed as an Indefinite Delivery Indefinite Quantity (IDIQ) with both core and task order (TO) functions. EVALUATION METHODOLOGY The evaluation addresses the following four questions: 1) How is SIMPLE affecting the capacity of USAID/Egypt to successfully conduct monitoring, and evaluation functions? What are the key factors identified by stakeholders that affect the perceived utility and function of SIMPLE? 2) What are the types of M&E and learning tasks and processes that are addressed effectively through SIMPLE, as reported, and as evident through a selected review of documentation? 3) What M&E and learning tasks and processes remain challenging despite having the Platform in place? 4) How did the IDIQ operational structure and location contribute to the challenges or successes of SIMPLE? The methodological approach for this evaluation combined qualitative and quantitative methods. Evaluators reviewed program documentation, conducted a series of rapid most significant change interviews, conducted key informant and group interviews with key stakeholders, and implemented a short survey for USAID staff. Because of social distancing guidelines and restrictions on travel, evaluators used remote interactive online platforms to conduct the survey and interviews. These materials were then reviewed jointly by team members, coded by theme, and quantized, and triangulated with relevant project documentation. FINDINGS The evaluation found that, because of relations built on trust over time with USAID/Egypt, SIMPLE has contributed to the Mission's capacity to conduct traditional monitoring and evaluation functions. In general, perception of quality of staff and consultants was favorable and USAID interviewees commented on the flexibility and adaptability of the platform. Mission respondents appreciated having the type of assistance in M&E that SIMPLE provided and voiced support for the MEL platform concept. At the core of SIMPLE's monitoring support were data quality assessments (DQAs) and technical reviews of activity monitoring, evaluation, and learning plans (AMELP). At a later stage, beginning in 2019, SIMPLE also developed a Mission-wide performance indicator tracking table (PITT). The evaluation found that DQAs enhanced data consistency and validity among IPs across the technical offices, ensured higher levels of compliance with USAID automated directives system (ADS) requirements, and helped the Mission improve and maintain the quality of data across its IPs. Although there were some reported challenges in DQA quality and consistency, Mission respondents generally noted marked improvements over time in data quality that they used for reporting on performance indicators. Although some CORs reported that they needed to review Program Office comments and SIMPLE comments to AMELPs Final Evaluation for SIMPLE Project Report iv separately, Mission and (non-SIMPLE) IP respondents had a generally favorable view of SIMPLE's contribution to AMELP reviews. Prior to SIMPLE, technical offices did not have a unified system for tracking indicators from IPs. The rollout of the PITT was a useful and well-received step towards enhancing performance data management for reporting. Assessments and evaluations were another area in which SIMPLE generally provided quality services. The two assessments conducted, particularly one on persons with disabilities, were very well received by the Mission and other stakeholders including the Government of Egypt (GOE). Over the course of SIMPLE implementation, evaluations and assessments showed a level of quality and adherence to ADS requirements that met generally accepted standards. However, for some evaluations, USAID staff noted variability among consultant teams, problems with depth of analysis, and to a lesser extent, the ‘actionability’ of recommendations. These challenges were due to several factors which included difficulties in receiving timely necessary approvals from the GOE for data collection and a lengthy approval and procurement cycle for task order SOW development. Such challenges contributed to long timelines from conception to implementation of evaluations, which in turn led to the loss of some qualified evaluation team members before and during evaluations. Inadequate time allocated for evaluation analysis was often cited by evaluation consultants as a problem. For USAID interviewees, these factors were manifested in a lack of adequate contextual understanding on the part of some evaluation teams. Due in part to the impending rollout of USAID/Egypt's Development Information Solution (DIS) in April 2021, some initially envisioned elements of the platform, such as third-party monitoring (TPM) and geographic information systems (GIS) were postponed by USAID during the original period of performance through July 2020. SIMPLE did demonstrate its capacity to work in less permissive areas, but further work may be dependent on a relaxation of current GOE limitations on data collection. Nevertheless, based on some past SIMPLE experiences and deliverables, future MEL platform interventions now have a solid experiential basis for expanding TPM in the future. Learning is now increasingly embraced by the Mission, but opportunities for learning need to be further explored. Training and capacity building for USAID staff have been at a foundational level; USAID interviewees noted that there is high degree of technical skill in the Mission offices, and that training is not always well received, particularly if it is perceived as too basic. Mission respondents expressed the desire for training for Mission staff to both be: 1) more interactive and engaging for experienced technical officers; and 2) more grounded in the key questions that technical staff need to know for site visits and for analyzing IP reports as inputs for performance their reporting. SIMPLE has taken some positive steps toward formal capacity building. In 2017, it conducted a capacity needs assessment and the platform's prime implementing partner indicated that it was in the process of transitioning a full-time staff person to act as an Egypt-based senior learning advisor. However, due to current COVID-19 related travel restrictions, that plan had been put on hold at the time this evaluation was conducted. There is an active task order for capacity building that will provide further training once COVID-19 teleworking measures ease; SIMPLE is currently working with the Mission to further emphasize formal capacity building and knowledge management. SIMPLE interviewees expressed their interest in building a stronger culture of CLA within the Mission and with IPs. CLA is relatively low on the expressed priorities of Mission staff in terms of expressed needs for capacity building. However, although its scope of work (SOW) did not place explicit emphasis on CLA, elements of SIMPLE's approach reflected some key CLA principles in its facilitation work with IPs. Notwithstanding SIMPLE contributions, for IPs, moving from M&E learning into changing the way programs are implemented remains challenging. While several IP interviewees indicated that because of SIMPLE's interventions, they were pleased to now have AMELPs in conformity with USAID regulations, but have yet to see the results of these plans well enough integrated into activity design, or more broadly applied in implementation methods and approaches. Final Evaluation for SIMPLE Project Report v The IDIQ and TO mechanisms were conceived to be efficient and streamlined, and a SIMPLE ordering guide, Mission Delegation of Authority, Mission Order and ADS roles and responsibilities documents for evaluations were put into effect. However, processes related to the IDIQ and TO mechanism were inefficient. Nearly unanimously, interviewees described the bureaucratic burden and costs in terms of staff time and resources, contributing to long timelines for task order products. Most respondents noted that having SIMPLE co-located within the Mission and having an annex office within the security perimeter was a net positive. This allowed for easier communication, relationship-building, and coordination between USAID and SIMPLE staff; this proximity helped both to build informal relationships and increase efficiency. CONCLUSIONS SIMPLE has been largely successful in providing quality traditional monitoring and evaluation services to the Mission. It has addressed a range of these tasks and processes effectively, including monitoring, AMELP reviews, support for performance reporting, and evaluations and assessments. SIMPLE can further build on its record of accomplishment in facilitating Mission-centric processes and on its relations with IPs, which are based on frank and open discussions resulting in technical improvements. Although SIMPLE works on finalizing the SOW with the relevant technical offices and the Program Office, the Mission could further benefit from earlier engagement with the MEL platform in designing evaluation scopes, questions, and methodology. To ensure the sustainability of MEL gains, basic IP capacity on defining indicators and assistance in aligning indicators in AMELPs requires further continued attention. Important gaps continue to exist in terms of understanding of USAID data quality requirements, MEL practices, and CLA among some IPs. Similarly, third-party monitoring (TPM) and integration of GIS data into reporting systems could further contribute to ensuring that IPs are achieving and reporting on planned outcomes. As a demand-driven platform, regular adjustments to MEL platform work plans and targets are key to enabling platforms to match capacity with demand and to integrate evolving Mission priorities. Working closely with the Program Office, SIMPLE was able to meet most of its requests in a timely manner, but often faced difficulties in juggling multiple simultaneous tasks. Moreover, the potential flexibility of the IDIQ and task order mechanisms was not well understood by contract officers and other Mission staff. Further iterations of the MEL platform could be designed to include lessons learned on how to reduce management burdens on USAID and platform staff. Co-location has largely suited the purpose for which it was designed. The decision to locate a nearby SIMPLE annex added additional value in that this space allowed the platform to accommodating SIMPLE consultants and other IPs, whose access to USAID offices was more limited. Evaluators see further room to expand on the annex concept to allow for the creation of an “M&E hub” within the security perimeter. Enlarged meeting spaces could provide a venue for regular meetings on M&E topics, not only for technical officers and IPs, but with the larger M&E community in Egypt. Final Evaluation for SIMPLE Project Report vi RECOMMENDATIONS In terms of capacity, evaluators provided two types of recommendations, some of which have begun to be implemented under a current task order: 1) near term recommendations that can be practically implemented during the remainder of a currently active SIMPLE task order; and 2) next generation MEL platform recommendations, which include components that should be integrated into a follow-on activity or mechanism: Near-term recommendations: ● Integrate key lessons-learned from learning events, CDCS facilitation, and needs assessments, into design of capacity building curriculum for USAID and IP staff. ● Strengthen support for USAID new program design functions through further dissemination of lessons learned (e.g. from cross-evaluation studies). Continue to engage key SIMPLE staff to support brainstorming during early design stage of follow-on activities and evaluation SOWs. ● Pilot monthly remote small-group interactive training, and capacity-building sessions based on its “M&E starter kit” with technical officers and IPs. SIMPLE learning and subject matter M&E specialists should facilitate these sessions. Recommendations for next generation MEL platform: ● USAID should ensure an explicit emphasis on learning for Mission staff and IPs, ensuring that learning and best practices are incorporated into future activity design and further raising awareness on the importance of using data for decision-making and adaptive management. ● The MEL platform should conduct annual learning events focusing on how recommendations from previous learning events and the final performance evaluation can inform future MEL platform activities. ● The MEL platform should conduct extensive and timely team building events with new MEL platform staff and key USAID stakeholders to clarify roles, responsibilities, processes, and expectations. ● The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. As a complement to in-person orientations, materials should also be available online. ● The MEL platform should conduct an annual needs assessment study to inform capacity building and training strategies for USAID and IPs. ● The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored joint training sessions for targeted groups of USAID staff and IPs by subject matter specialty. Individual sessions should be planned to coincide with evaluations, assessments, and AMELP reviews in those specialized areas. ● USAID should ensure an explicit emphasis on collaboration, learning and adaptation in MEL platform. ● The MEL platform should continue with an emphasis on providing M&E support services, including DQAs, AMELP review, assessments, and evaluations, and technical support for using the Development Information Solution (DIS). The platform should expand in scope to include reviews of yearly work plans of selected IPs as directed by the Mission. ● The MEL platform should increase the scope of facilitation services for performance management. In addition to providing support for Mission activities on annual performance reporting, the MEL platform should assist the Mission in dissemination of new and emerging USAID strategies among IPs, donor agencies, and other key stakeholders. ● Streamline processes and strengthen MEL platform collaboration in evaluation SOW design: USAID should reinforce its current collaborative process between the MEL platform and technical offices while ensuring a more streamlined process. Final Evaluation for SIMPLE Project Report vii ● Ensure adequate timelines for evaluations: a) For evaluations, the Mission should ensure that timelines allow for the recruitment and retention of the best evaluation candidates; b) The Mission should consider allowing the platform to submit an initial cost estimate by category of key and non￾key personnel and allow for necessary cost modifications once key personnel are identified; c) and commit to the evaluation. ● Reinforce requirements for evaluation analysis: The Mission and MEL platform should emphasize the importance of depth-of-analysis based on a thorough understanding of contextual issues. ● Institute formal quarterly workplan reviews. To integrate evolving Mission priorities and proactive ideas from MEL partner, the Program Office and MEL platform should conduct collaborative quarterly workplan reviews. ● Require AMELP and DQA training and mentoring for new IPs: The Mission should require that the MEL platform provide all new IPs with AMELP and DQA training, mentoring, and coaching. This will better ensure that IPs understand USAID monitoring requirements before drafting monitoring plans. ● Initiate third-party monitoring (TPM): The Program Office should engage with technical offices to require that the MEL platform conduct pilot monitoring in more remote or less permissive areas in years one and two, progressively rolling out TPM over the life of the activity. Remote channels of implementing TPM should be considered. ● Incorporate GIS functions into DIS performance tracking: The Mission should require that new awards require IPs to incorporate performance reporting by geographic location to prepare for DIS roll-out. The MEL platform should assist the Mission in ensuring technical office and IP understanding and compliance with GIS requirements through training and technical assistance. ● Minimize task order burden: Based on experiences in the contract extension period, which utilizes a single task core mechanism, conduct a robust review to determine if the single task order approach has alleviated burden on program and technical offices and MEL platform. To the extent that this approach has contributed to efficiencies and minimized burden, incorporate it into future MEL platform design. ● Conduct peer-to-peer discussions with other Missions and socialize a revised platform mechanism: To ensure clarity and understanding of mechanism, to inform the new platform design, USAID should promote understanding about the mechanism through peer-to-peer discussions with contract officers in other Missions to discuss options, legal issues, and efficiencies. This should include a discussion of having one platform or multiple task-based platforms under an overarching contract. Results of these discussions should be widely socialized among Mission end users. ● Continue co-location and expand MEL platform capacity to host annex-based events: The MEL platform should continue to be co-located while also expanding its capacity for meetings for IPs, USAID, and other key M&E stakeholders outside of the Mission but within the security perimeter. Final Evaluation for SIMPLE Project Report 1 1.INTRODUCTION The Mitchell Group, Inc. (TMG), and its Egyptian partner, Beit Al Karma Consulting (BK), are pleased to present this final evaluation of the Services to Improve Performance Management, Enhancing Learning & Evaluation (SIMPLE) Activity Monitoring, Evaluation, and Learning (MEL) Platform. TMG was uniquely qualified to undertake this assignment given its capabilities and established record in both successfully implementing and conducting evaluations of MEL platforms globally. TMG was awarded the contract to conduct a final evaluation of the SIMPLE project by USAID/Egypt in February 2020. 1.1 Background, Context, and Purpose of SIMPLE Background. In July 2015, USAID/Egypt entered into a five-year contract (Contract no. AID-263-I-15- 00001), with a period of performance from July 15, 2015 to July 14, 2020, with the QED Group (QED) and its subcontractors, DevTech Systems, Inc., and North South Consultants Exchange (NSCE), to develop and implement the Management, Enhance Learning and Evaluation (SIMPLE) activity. SIMPLE was designed to provide services to the Program Office (PO) and the technical offices of Education and Health, Economic Growth (EG), and Democracy and Governance (DG) to improve performance management and to support program and activity design. This support was intended to assist the Mission in performance management planning; training of implementing partners (IPs) and staff in performance management; field-based monitoring; and evaluations. Support for program and activity design was intended to help the Mission with design and to provide short- and long-term technical assistance in collecting baseline data; conducting sectoral and geographic assessments; topical and geographic research; and assisting with other development design tasks as required. SIMPLE operated under an indefinite duration indefinite quantity (IDIQ) contract developed with both core and task order functions in performance management. Of the total contract value of $24.5, approximately $9.6 million was allocated to core functions and approximately $14.9 million to task order functions. By January 2019, a total of approximately $8.3 million of the contract value for core functions had been obligated.1 Contextual factors. Although the Mission had previously had one small M&E support activity for education evaluations and assessments, SIMPLE was the first comprehensive MEL platform operating across technical areas. As such, it experienced some of the "growing pains" typical of new MEL platforms.2 Moreover, during the past decade, new constraints to operating in Egypt, some of which are described in this report, have contributed to making data collection efforts difficult. SIMPLE design. SIMPLE was developed with both core and task order functions in performance management, with core functions, utilized to support ongoing, routine functions for the Mission such as the development of performance management plans (PMPs and Assessment, Monitoring, Evaluation and Learning Plans (AMELPs), and task orders, which were leveraged to conduct some support functions for IPs such as capacity-building and monitoring and evaluation (M&E) workshops. In practice, however, there was some overlap between core and task order functions, including some core capacity-building efforts requested by USAID, or as part of Mission-requested discussions or early planning for task orders. Below is a detailed breakdown of core and task order tasks. 1 Per SIMPLE contract modification #10. As of July 15 2020, SIMPLE has continued to operate under task order funding. 2 See for example, Laura Arntson, Tonya Giannoni, Nancy Peek, and Kristin Saarlas of Social Solutions International, Inc., for USAID/PPR/LER, 2016: Mission-Based Monitoring, Evaluation and Learning (MEL) Platforms Assessment. Final Evaluation for SIMPLE Project Report 2 CORE TASKS 1.1.1 Performance Management ● Assist the Mission to develop, finalize, and implement a Performance Management Plan (PMP), including revising PMPs at the Development Objective (DO), Intermediate Result (IR), and Sub-IR levels to respond and adhere to standard indicators (Task 1). ● Review and finalize Activity Monitoring, Evaluation and Learning Plans (AMELP) to ensure conformity with USAID Evaluation Policy and other relevant guidance for USAID approval (Task 2). 1.1.2 Monitoring To increase frequency and quality of project monitoring, SIMPLE's core task was to: ● Assist Mission staff to track program results against indicators, targets, and baselines in the PMP and annual Performance Plan and Report (PPR) to analyze data from implementing partners in periodic reports and updates to ensure quality and accuracy (Task 6). 1.1.3 Evaluation, Assessments, Research, and Analysis To promote increased use of analytical results, SIMPLE was to plan and conduct high-quality evaluations, assessments, research papers and other analytical reports, and to disseminate reports and conduct outreach activities as appropriate. Assist the Mission in the development and tracking of an evaluation and assessment plan and in identifying situations which could trigger the need for an evaluation (Task 9). Task Order Tasks 1.1.1 Performance Management – ● Build the capacity of implementing partners to coordinate data collection, reporting requirements and to understand PMP requirements (Task 3). ● Organize M&E workshops with USAID staff, implementing partners, Egyptian non-governmental organizations (NGO), and as necessary with the private sector in support of data collection and reporting (Task 4). 1.1.2 Monitoring ● Provide short and long-term technical assistance for the purpose of conducting third party field￾based monitoring, and related activities (Task 5). ● Per USAID instruction, provide annual analysis of and reporting on the overall trend of the country and by governorate (Task 7). ● Support project implementers and technical teams by conducting data quality assessments and independent analysis at the directive of USAID staff (Task 8). 1.1.3 Evaluation, Assessments, Research, and Analysis ● On an annual basis, perform evaluations described in specific scopes of work. Following ADS guidance including the deliverables and peer review processes, SIMPLE was to provide evaluation teams with the necessary skills and experience and obtain concurrence of the Program Office COR and relevant technical office, facilitate presentations of findings to the Mission, the Government of Egypt (GOE), and other donors and partners, solicit feedback on recommendations and finalize reports (Task 10). 1.1.4 Program and Activity Design Support Through task orders, SIMPLE was to: ● Conduct assessments, analyses, baseline data collection and stakeholder consultations to inform the development of Project Appraisal Documents (PAD) for new programs. Short-term technical assistance was envisioned for conducting studies on a variety of policy and strategic issues (Task 11). Final Evaluation for SIMPLE Project Report 3 1.1.5 Information Management ● Prepare recommendations on how to streamline management and reporting of data, including geographic data and information; prepare recommendations on the use of program information systems, including geographic information systems (Task 12). 1.2 Evaluation Purpose The purpose of this final evaluation is to: (1) Provide evidence and learning to USAID/Egypt to inform the design and use of the next generation of Monitoring, Evaluation and Learning (MEL) support contract; and (2) Assess the quality of products and services provided by SIMPLE. 1.3 Evaluation Questions USAID/Egypt provided the evaluation team (ET) with four questions to answer: 1) How is SIMPLE affecting the capacity of USAID/Egypt to successfully conduct monitoring, evaluation functions? What are the key factors identified by stakeholders that affect the perceived utility and function of SIMPLE? 2) What are the types of M&E and learning tasks and processes that are addressed effectively through SIMPLE, as reported, and as evident through a selected review of documentation? 3) What M&E and learning tasks and processes remain challenging despite having the Platform in place? 4) How did the Indefinite Duration Indefinite Quantity (IDIQ) operational structure and location contribute to the challenges or successes of SIMPLE? 1.4 Methodology TMG’s methodological approach for this evaluation combined qualitative and quantitative methods. Evaluators began with a review of program documentation, including quarterly and annual reports, and then leveraged the most significant change (MSC) method as an entry point to our sequential, mixed-methods approach, which focused on key informant interviews (KIIs) and focus group discussions (FGDs) by themes and sub-themes, supplemented by a survey, then the quantization of qualitative data and triangulation of findings. 1.4.1 Preparatory Activities Desk Review and Document Assessment. The ET began with a desk review of project-related documentation provided by the Mission and the SIMPLE team, including quarterly and annual reports through March 2020 (FY20 Q2), and evaluations assessments and reports conducted by SIMPLE. Most Significant Change (MSC). The team developed a list of 22 key stakeholders and a rapid preliminary exercise was completed with seven respondents to inform instrument design. Stakeholders were asked to provide an answer to two broad questions: Document Review Most Significant Change Mini-Surveys Key Informant Interviews Focus Group Discussions Quantitization of Qualitative Data Triangulation of Findings Figure 1: TMG’s Methodological Approach Final Evaluation for SIMPLE Project Report 4 ● Based on your experience with the SIMPLE MEL Platform, what are the most significant positive effects that you associate with this project? ● Based on your experience with the SIMPLE MEL Platform, what are the most significant negative effects that you have encountered with this project? Instrument Development. Interview and survey protocols were drafted based on document review and MSC responses, and the ET iteratively revised these based on COR and PO comments. Before conducting interviews, and to ensure that data collection was comprehensive and internally consistent, the team developed an online training video and conducted a series of practice exercises and interviews to educate interviewers and note-takers about the level of detail required. Sampling. A total of 60 KIIs and FGDs3 were conducted with: USAID Stakeholders (24), QED, SIMPLE and headquarters (31), and representatives of other USAID/Egypt IPs (5). Of USAID staff, the majority (54%) were from the technical offices, a third (33%) were program office staff, and the remaining 13% were from the Front Office. Because USAID was the main client and beneficiary of SIMPLE’s services, most of the findings and supporting quotes come from USAID staff. The ET has also included quotes from IP representatives where relevant. With limited exceptions, the comments of SIMPLE and QED staff have not been included in discussions of the platform’s usefulness or benefits. A purposive survey was also conducted with only USAID staff; of the 42 who received the survey, sixteen responded. 1.4.1 Data Collection TMG had originally proposed a combination of in-person and remote data collection methods. However, given the restrictions on travel and social distancing guidelines due to the ongoing COVID-19 pandemic, TMG used interactive, online platforms, including Zoom and Google Meet, to conduct its purposive sampling of stakeholders. The ET divided into four sub-teams to conduct the interviews, each with a lead interviewer and a note-taker. Interview teams scheduled their own interviews with assistance from TMG and BK support staff. The ET conducted 57 KIIs and three FGDs, which generally lasted from 60 minutes to 90 minutes. The ET also conducted an online survey, which was sent to 42 USAID staff and completed by sixteen (38% response rate). 1.4.2 Data Analysis Methods All data were iteratively coded and categorized by themes and sub-themes, with approximately 2,500 coded entries across 60 interview transcripts. These were used to generate frequencies of comments by theme and sub-theme across all stakeholder types and later by frequency of comments by stakeholder types. Major analytical focus was first placed on those themes most often discussed by respondents, progressively expanding to a review of other relevant themes and sub-themes to add more nuance and contextual understanding. Preliminary findings were based on an analysis of quantized data that were shared and discussed with the evaluation COR, and further triangulation of data across all datasets was conducted in preparation for a remote presentation to USAID and this evaluation report.4 1.4.3 Research Limitations, Challenges, and Mitigation The team encountered three main challenges during the implementation of the evaluation, namely a pandemic, a change in the lead evaluator, and the timing of data collection. 3 Because of COVID-19 social distancing requirements, most respondents opted for KIIs (n=57). The three (3) FGDs conducted. (1 IP Group, n=4; 2 Consultant Groups n=7) were counted as interviews during quantitization. Results from survey and interviews are as indicative of trends and opinions. Because of small number (n) for survey and interview respondents, tests for statistical significance are not feasible and were not conducted. 4 Quantization is the numerical translation, transformation, or conversion of qualitative data and has become a staple of mixed methods research. Final Evaluation for SIMPLE Project Report 5 ● The COVID-19 global pandemic and the ensuing travel ban meant that in-person data collection could not take place, and fewer group interviews were conducted than expected primarily because many interviewees were working off-site because of social distancing requirements. This was mitigated by primarily focusing on individual interviews conducted remotely. ● The original evaluation lead had to leave the project for medical reasons after data collection had been completed. This was mitigated by the replacement by a seasoned evaluation leader with substantial experience leading MEL platform implementation, who conducted a series of in-depth “handover” discussions with evaluation team members and the TMG senior technical advisor during the transition. ● Data collection took place over the spring and summer of 2020, which coincided with Ramadan and holidays, so fewer USAID staff were available to be interviewed than initially anticipated. This was partially mitigated through an increase in interviews with SIMPLE implementing staff, and through the inclusion of a survey with USAID. Our methodological approach focused primarily on the issues that were most often discussed by USAID interviewees, and the sample of USAID staff was large and diverse enough in terms of respondent type to be generalizable. Final Evaluation for SIMPLE Project Report 6 2. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 2.1 Evaluation Question 1: Effects on USAID Capacity Evaluation Question 1: How is SIMPLE affecting the capacity of USAID/Egypt to successfully conduct monitoring, evaluation and learning functions? What are the key factors identified by stakeholders that affect the perceived utility and function of SIMPLE? 2.1.1 FINDINGS: How is SIMPLE affecting USAID MEL Capacity? SIMPLE’s primary focus was on providing monitoring and evaluation support to the Program Office and technical offices. At the core of its monitoring and evaluation (M&E) support were data quality assessments (DQA) and technical reviews of activity monitoring, evaluation and learning plans (AMELP). SIMPLE did not have a strong focus on formal capacity-building and knowledge management for the Mission or its IPs. SIMPLE was able to increase the Mission’s MEL capacity because it was able to build up a level of trust with them through key deliverables and, particularly during later stages of implementation, greater attention to customer service. SIMPLE had a primary focus on monitoring and evaluation support. Throughout the period of operations, SIMPLE primarily focused on M&E functions, consistently providing timely support for key deliverables including data quality assessments (DQAs) and technical assistance for activity monitoring, evaluation and learning plans (AMELP) in support of technical office staff and their implementing partners (IPs). USAID interviewees consistently cited DQAs as contributing to improvements in meeting Agency reporting requirements, as well as the usefulness of these and evaluation data for evidence-based management decisions. A program officer explained, “DQAs helped us improve and maintain the quality of our data across the IPs, making sure that we are not receiving different products from our IPs across the technical offices and contribute to compliance with ADS requirements.” Another program officer reported on the learning that occurred through monitoring processes: “CORs said that they learned from being involved in the DQA process and how to look at the audit trail and what are the questions.” Technical officers that participated concurred, indicated that they benefited from participation in DQAs through training on checklists for field-based site visits and acquired skills such as reviewing audit trails and learning the types of questions to ask. Mission staff also commented on the utility of SIMPLE in terms of its support, as requested by technical staff for reviewing activity-level MEL plans. SIMPLE also worked with IPs on results frameworks when requested to do so and agreed to reinforce these efforts in the future. According to one program officer, because of feedback from the [2019] learning event, SIMPLE agreed to place a stronger focus on results framework and logical frameworks: “They will have more meetings with the technical offices and when they do a MEL review they will have broader discussion about RF development and how to incorporate a good log frame to be able to provide a good MEL plan.” SIMPLE feedback through DQAs and AMELP reviews also helped to promote greater interaction by technical offices with IPs through recommendations for data quality improvement. For example, in one activity where monitoring data discrepancies were discovered, SIMPLE conducted DQAs and helped to incorporate findings into more robust activity-level MEL plans for specific IPs that helped them think about data collection and monitoring planning in a more intentional way. One technical officer said, “In one project, we focused on the number of trainees and how they were selected. After working with SIMPLE, I began looking at the data more closely to ensure that there is no double-counting and added more validation and references. I also started to think about ways the IP could better collect data and I began to coordinate Final Evaluation for SIMPLE Project Report 7 more with them. It was very hard before SIMPLE. Once SIMPLE started, it helped me to better focus on the MEL plan, the targets, and the PIRS.5 ” Support for improving a range of USAID-required monitoring and data quality deliverables has contributed to Mission capacity. Notwithstanding the favorable accounts of those technical staff that learned through DQAs and AMELPs, many technical staff interviewed focused their comments on SIMPLE's provision of M&E outputs as its main contribution to the Mission’s MEL capacity. For these interview respondents, the key benefit was that SIMPLE helped free up time for other tasks. According to one technical officer, “I think the perceived utility of SIMPLE is to help alleviate some of the management burden placed on our CORs / AORs.” Over the course of implementation, SIMPLE has deepened its use of processes for its monitoring activities, which include startup meetings, debriefing, and validation workshops. Evaluation and assessment findings are disseminated among Mission and other stakeholders through debriefs, and facilitation support for post￾evaluation management plans. One front office respondent noted, “SIMPLE facilitated the senior staff retreats which has worked well for us.” SIMPLE also provided facilitation support for annual performance reviews and, most recently, for the Country Development Cooperation Strategy (CDCS) and other events, which were appreciated by Mission interviewees. Similarly, Mission interview respondents indicated that the performance indicators tracking table (PITT), which began rollout in 2019, as well as SIMPLE's assistance to technical offices on using the table, helped enhance Mission-wide capacity for performance reporting. SIMPLE has also provided logistical and facilitation support, including senior staff retreats. A Front Officer staffer said, “SIMPLE facilitated the senior staff retreats which has worked well for us since they already know us, so it’s not like bringing a stranger into a sensitive and important conversation.” Relations built on improved trust over time. Several issues, including quality concerns and staff turnover, initial gaps in understanding of the platform’s range of services and roles and responsibilities in terms of managing relationships within the Mission, contributed to difficulties in the start-up phase. During this period, SIMPLE did not provide the requisite level of MEL support to the Mission. Over time, this relationship improved. The number of evaluations and assessments increased under a second chief of party (COP), and the current COP has focused on relationship building and management. As one Front Office respondent mentioned, “When SIMPLE changed management three years ago, it significantly improved their performance. Initial management didn’t provide the same quality and timeliness of assessments and evaluations.” However, mid- to late-2018 marked a turning point, as QED strengthened home office support and made some key personnel changes. Responsive staff and the production of high-quality products led to increased trust and confidence in the platform from the Mission and IPs alike. As one program officer indicated in an interview, “There’s a big list of what the offices want SIMPLE to do. It took a while to develop that trust, and it was based on good assessments and evaluations.” Another program officer indicated, that “while it took the Mission a while to see really good assessments, that helped build trust and that trust is 180 degrees from where it was.” A third program officer said a good relationship now exists between SIMPLE and the program office's COR and that it is “as close as it needs to be to ensure that things get done.” SIMPLE's scope of work did not emphasize collaboration, learning, and adaptation (CLA), but elements of SIMPLE's facilitation approach reflected some key CLA principles. Importantly, SIMPLE is increasingly seen as a trusted third party for facilitation of communication between the technical office and IP and other stakeholders. Although there was not a specific emphasis on CLA, according to some USAID respondents, the concepts undergirding CLA are not well understood. SIMPLE has begun some work on 5 The performance indicator reference sheet (PIRS) is the tool that USAID uses to define performance indicators for its IPs. It was very hard before SIMPLE. SIMPLE helped me a lot on how I should look at the MEL plan, the targets, and the PIRS. –Technical Officer Final Evaluation for SIMPLE Project Report 8 CLA, but their staff often pointed to the lack of a concerted CLA approach within the Mission. Because CLA is a relatively new concept in the Agency and the Egypt Mission, the main problem, according to one program officer, is that that “very few people in the Mission understand what CLA actually is.” SIMPLE did however introduce new approaches of facilitating consultation that contained at least some important CLA elements. According to one USAID interviewee from a technical office, “They did well when it comes to the facilitation of processes with stakeholders, for instance, two years ago, we had a meeting with our IPs, and it was part of a consultation session. Then, more recently, we had another meeting and consultation session and SIMPLE facilitated the entire process in a way that is different to what we are used to. They make it more interactive and they encourage the IPs to speak up their minds, even criticizing the way that we are doing things and business models.” SIMPLE did not emphasize formal capacity-building and knowledge management to the degree that it emphasized basic M&E services. SIMPLE’s primary focus was on monitoring and evaluation functions, complemented by facilitation services. Although later training for Mission staff that SIMPLE offered was generally viewed as acceptable, there has not yet been significant formalized knowledge transfer. When SIMPLE has offered trainings, they were sometimes not well-attended. Some feedback from Mission staff indicates that trainings would benefit from being more interactive or better tailored to specific Mission audiences. As one Program Officer noted, “After one of the trainings, the monitoring evaluation and learning session, I felt it wasn’t properly targeted and it wasn’t talking to the CORs/AORs as experts but rather as novices—the training didn’t separate tones in the discussion based on expertise.” Although there is some variability, CORs see themselves as having generally strong monitoring skills, but would appreciate deepening their knowledge in areas such as leveraging indicators for learning, developing MEL plans, complexity-aware monitoring and aligning indicators to results frameworks. Respondents from USAID expressed relatively little interest in training on PIRS or basic M&E.6 SIMPLE reported that it has updated the curriculum based on USAID's “M&E starter pack” and its experiences with USAID in Uganda.7 However, according to SIMPLE interview respondents, the deployment to Egypt of its full-time learning specialist was delayed because of travel restrictions related to COVID-19. A 2019 Learning Event allowed SIMPLE to discuss its contributions and solicit feedback from a wide Mission audience on how they would like to see addressed issues related to training and capacity building. Key takeaways were that SIMPLE would customize the content of its training and consult with technical offices on topics and levels in designing MEL session series.8 Such an approach would ideally address the perception that the capacity-building that was offered was not well-tailored. Additionally, SIMPLE did not conduct much training or focus on learning with the IPs, nor was there always follow-up with IPs to instill learning from work SIMPLE did for the IPs. One respondent representing a USAID implementer shared, “We did not have any debriefing session with SIMPLE on lessons learned, successes and best practices shared from various projects and programs.” Another IP noted how they were not taught to integrate learning into new activity design. “We worked on indicators and the process and everyone was happy because we did it well. What was missing is how to include the learning into the design and the process. This would have contributed to the value.” By the beginning of 2020, SIMPLE had not yet developed a learning strategy using a knowledge management (KM) portal, although according to recent quarterly reports, SIMPLE expressed its intention 6 The survey conducted for this evaluation produced broadly similar results to a larger survey of USAID staff (n=86) conducted by SIMPLE in 2017, which showed a high overall interest (89% of respondents) in performance management training. The USAID Performance and Management Survey Report (2017) noted staff preferences for training on: analyzing and using performance data (96%); establishing baselines (92%); performance management in practice (91%); and developing AMELPs (91%) and DQAs and data verification (86%). 7 Source: SIMPLE quarterly report, Q1 FY 2020. 8 SIMPLE Learning Event Report. February 26, 2019. Final Evaluation for SIMPLE Project Report 9 to do so if requested. In January 2020, SIMPLE submitted a concept note detailing a KM portal, suggesting scoping and development phases, but the Mission informed SIMPLE that it was deferring the idea because of competing priorities, but that it may be revisited in the future.9 2.1.2 FINDINGS: What are the key factors that affect SIMPLE's perceived utility and function? In our interviews, respondents noted several factors that contributed to SIMPLE's perceived utility and function. Experience working with the Mission over time helped SIMPLE better tailor their services to USAID requirements and priorities. Co-location within the Mission (which is discussed under Question 4) and strong support of the platform by the PO and Mission leadership also contributed to SIMPLE's perceived usefulness. Although there are some continuing challenges, which are discussed in the following section, many USAID interviewees commented on the flexibility and adaptability of the platform. Flexibility and adaptability. Over the life of the SIMPLE activity, the platform implementer has increasingly shown flexibility and adaptability within a demand-driven context. Many stakeholders noted SIMPLE’s willingness to fulfill requests within their mandate and adapt to Mission concerns. During the latter years of implementation, SIMPLE adopted a strong customer-service approach that seeks to provide high-quality services in a flexible and adaptable way. There is now a high demand for SIMPLE services, and the platform has managed multiple concurrent evaluations, DQAs, and AMELP reviews, as well as facilitation support for Mission activities such the CDCS. Although it took a long time to solidify relations between SIMPLE and the Mission, interviewees indicated that for the past two years, SIMPLE leadership is now very open to comments for improvement and communicates with staff to ensure that Mission requests for services are met. According to a Mission leader (“front office”), “They are always open to feedback, and we’re in a really good place with them, and lucky to have them.” Technical officers expressed similar sentiments: “SIMPLE leadership are very approachable and very responsive. This leads to a good relationship all around. They were so responsive to what we are asking them to do.” “The fact of being very flexible, they are also open to comments and improvement. The only challenge is timing, but they are always ready and flexible.” Finally, “SIMPLE staff are very cooperative. They understand that they are here to support the mission not to challenge it or to monitor our work.” Quality of staff and consultants. Although there were some issues with the platform leadership’s approach in the beginning, program officers appreciated the level and variety of experience and expertise among SIMPLE staff. According to one program officer, “since the beginning, they have had very highly-qualified staff and the technical expertise is quite rich.” IPs also reported generally good interaction with SIMPLE and the quality of services. One IP respondent stated: “They were professionals and were clear and direct. They assisted us and explained the process to us and were very objective. They provided feedback about what was missing.” SIMPLE did make strong efforts to recruit and maintain a pool of international 9 Cited from SIMPLE quarterly reports, FY 2020, quarters 1 and 2. 0 5 10 15 20 Flexibility & adaptability Quality of staff & consultants Number of USAID interviewees=24; (PO=8; FO=3; TO= 13) Factors most often noted by USAID as contributing to SIMPLE success Program Office Front Office Technical Office They are always open to feedback, and we’re in a really good place with them and lucky to have them. – Front Office interviewee Figure 2: Key Factors Contributing to SIMPLE Success Final Evaluation for SIMPLE Project Report 10 consultants. Also, and in close collaboration with a local partner organization, SIMPLE engaged Egyptian short-term and evaluation consultants, some of whom have been repeatedly recruited, and as a result are familiar with SIMPLE processes. There were, however, several comments related to variability among the consultant pool, particularly in terms of subject matter expertise. According to another program officer, “SIMPLE technical leads are very strong, and this contributed to their success. They have a bench of contracted technical subject experts and some are better than others.” A technical officer indicated that, “I think if they provide more subject matter capacity to improve MEL plans for each sector, project, and program, that would be great, but this did not happen. I would like to recommend if we will have this platform in the future to have SIMPLE's team members with different backgrounds that reflect that different fields representing each office scope of work.” 2.1.3 Conclusions Trust between SIMPLE and the Mission is a tangible gain to be further leveraged for improved collaboration. As trust is critical to the success of any mechanism of this kind which provides critical support, often with access to sensitive data and information, SIMPLE should continue working on its relationship with the Mission, and any new platform should include proactive trust-building activities. SIMPLE played an important role in providing M&E support. SIMPLE's primarily focus was on monitoring and evaluation support and this assistance, which has led to improvements in DQAs, AMELPS, evaluations and assessments. This type of “bread and butter” assistance to the Mission was the cornerstone of SIMPLE's contribution to Mission capacity. Learning is increasingly embraced by Mission but opportunities for learning need to be further explored. Notwithstanding real practical gains in capacity reported by some interviewees as a result in participation in DQAs and AMELP reviews with SIMPLE, mirroring conclusions from a cross-country MEL assessment,10 SIMPLE provided M&E capacity within the Mission to a greater extent than it built Mission M&E staff capacity. To ensure long-term sustainability of SIMPLEs M&E gains, feedback-informed capacity-building can be enhanced and a CLA approach should be strengthened. As discussed in greater detail in later sections, internalization of the M&E gains among USAID can be leveraged through development of knowledge management and knowledge transfer systems. SIMPLE evaluations also provided key lessons learned for future programming and were shared in summary form as an input for strategic development, but there is scope for further sharing and wider dissemination. Also, the recent learning event has proven to be important example of a way in which to 10 Arnston et al. 2016, op cit. The authors found that among the advantage of MEL platforms were the skills and expertise they provided; having a portfolio-wide perspective of a mission’s programming; and dedicated time and resources for MEL to enable USAID staff to do more. The authors noted, however, that "In a time-constrained mission environment, mission staff often focus on fulfilling minimum M&E requirements and compliance over other functions; oftentimes little time remains to dedicate to thoughtful learning and use of M&E data and for providing feedback to appropriate stakeholders. . . [and] that MEL capacity building and CLA are often overlooked in missions and are considered underutilized functions of Platforms." Trust is a tangible gain to be leveraged for improved ll b SIMPLE played an important role in providing M&E The MEL platform concept has proven itself Learning is increasingly embraced by Mission Final Evaluation for SIMPLE Project Report 11 further develop a shared understanding of Mission needs and should be continued. As discussed in greater detail in later sections of this report on current challenges, valuable lessons- learned have the potential to be further absorbed by Mission and IP staff through more institutionalized learning. For this to happen more broadly, the MEL platform, in close collaboration with the Program Office, which can advise on potential training opportunities, must better understand and cater to the capacity-building needs of Mission staff and IPs by providing more formal trainings and informal learning opportunities The MEL platform concept has proven itself. Although there are some remaining challenges, which are further addressed in a later section, Mission respondent overwhelmingly appreciated having the type of assistance in M&E that SIMPLE provided and voiced support for the MEL platform concept. The Mission should leverage the success of SIMPLE and incorporate the critiques of Mission staff and IPs alike to design an even more useful follow-on mechanism. 2.1.4 Recommendations For this section only, evaluators have provided two types of recommendations: 1) near term recommendations that can be practically implemented during the remainder of a currently active SIMPLE task order; and 2) next MEL platform recommendations, which include components that should be integrated into a follow-on activity or mechanism. Recommendation 1.1.Integrate key lessons-learned from learning events, CDCS facilitation, and needs assessments, into design of capacity-building curriculum for USAID and IP staff. Recommendation 1.2. Strengthen support for USAID new program design functions through further dissemination of lessons learned (e.g. from cross-evaluation studies). Continue to engage key SIMPLE staff to support brainstorming during early design stage of follow-on activities. Recommendation 1.3. Pilot monthly remote small-group interactive training and capacity-building sessions based on its "M&E starter kit" with technical officers and IPs. SIMPLE learning, subject matter and M&E specialists should facilitate these sessions. Recommendation 1.4. USAID should ensure an explicit emphasis on learning for Mission staff and IPs, ensuring that learning and best practices are incorporated into future activity design and further raising awareness on the importance of using data for decision-making and adaptive management. Recommendation 1.5. Conduct annual learning events focusing on how recommendations from previous learning events and the final performance evaluation can inform future MEL platform activities. Recommendation 1.6. The MEL platform should conduct extensive and timely team building events with new MEL platform staff and key USAID stakeholders to clarify roles, responsibilities, processes, and expectations. Integrate Integrate key lessons learned for capacity-building. Strengthen Strengthen support for activity design functions. Interact Pilot remote small-group interactive training. Learning Conduct annual learning events. Near-term recommendations Recommendations for Next Gen MEL Final Evaluation for SIMPLE Project Report 12 Recommendation 1.7. The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. As a complement to in-person orientations, materials should also be available online. Recommendation 1.8. The MEL platform should conduct an annual needs assessment study and review of changes in capacity to inform capacity-building and training strategies for USAID and IPs. Recommendation 1.9. The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored joint training sessions for targeted groups of USAID staff and IPs by subject matter specialty. Individual sessions should be planned to coincide with evaluations, assessments, and AMELP reviews in those specialized areas.11 Recommendation 1.10. USAID should ensure an explicit emphasis on collaboration, learning and adaptation in the new MEL platform. Table 1: Q1: Findings Conclusions, and Recommendations (FCR) Table Conclusions Findings Data Source 12 Recommendation To ensure sustainability of M&E gains, feedback￾informed capacity-building should be enhanced. SIMPLE did not have a strong focus on capacity-building and knowledge management. IS 1.1. Integrate key lessons-learned from learning events, CDCS facilitation, and in USAID needs￾assessment into design of capacity-building curriculum. It is important to develop systems and processes so valuable learning is not lost. Incorporating lessons-learned into future programming has not been as systematic as it could have been. IS 1.2. Strengthen support for new USAID program design functions through further dissemination of lessons-learned (e.g. from summary cross￾evaluation studies). The MEL platform should better understand and cater to the training needs of Mission staff and IPs. There is a need for better￾targeted, and more engaging training. IS 1.3. Pilot monthly remote small-group interactive training and capacity-building sessions. Conclusions 1-3 above Findings 1-3 above IS 1.4. Ensure an explicit emphasis on learning for Mission staff and IPs Learning events are important ways to broadly communicate Mission needs and potential platform responses. A previous learning event was useful in communicating Mission needs and potential improvements. ID 1.5. Continue to conduct annual learning events. To engender trust, roles and responsibilities should be explained early and team building should be promoted. Some lack of understanding of the platform’s services, roles and responsibilities led to tension during project start-up. I 1.6. The MEL platform should conduct extensive and timely team-building events with new MEL platform staff and key USAID stakeholders to clarify roles, responsibilities, processes, and expectations. Frequent and targeted orientations will promote understanding of benefits. Trust and understanding of platform’s use and benefits took time to build. I 1.7. The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. Assessing and understanding needs will allow SIMPLE to develop better-targeted capacity building. Trainings were not well￾attended in part because they were often not tailored to most relevant needs. I 1.8. The MEL platform should conduct an annual needs assessment study and review of changes in capacity over time to inform capacity-building and training strategies. 11 As illustrative examples only, these could be focused on M&E-related topics such as early grade reading; sector￾specific agricultural value chains; climate change mitigation; decentralized government initiatives. 12 Key to Data Sources: I = Interviews; D = Document review; S = Survey of USAID staff Final Evaluation for SIMPLE Project Report 13 More frequent training would likely be well-received by Mission and IP staff and engender a culture of learning. Capacity-building was infrequent and Mission staff and IPs both indicated a desire for some regular training. IS 1.9. The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored joint training for USAID staff and IPs. Existing elements of CLA approach should be strengthened. CLA was not an explicit element in core and task order functions for the evaluation. However, elements of CLA exist in SIMPLE's work with USAID and IPs. I 1.10. Ensure explicit emphasis on learning is incorporated into design which include CLA activities for USAID staff and IPs that promotes using data for continuous improvement of processes and products. Higher Impact Lower Impact Short-term 1.1. Integrate key lessons-learned from learning events, CDCS facilitation, and in USAID needs￾assessment into design of capacity￾building curriculum. 1.4. Continue to conduct annual learning events. 1.2. Strengthen support for USAID new program design functions through further dissemination of lessons learned. 1.3. Pilot monthly remote small-group interactive training and capacity-building sessions with technical officers and IPs. 1.5. Begin with team building events to clarify roles, responsibilities, processes, and expectations. 1.6. The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. Long-term 1.8. The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored training for USAID staff and IPs. 1.9. Ensure explicit emphasis on CLA activities. 1.7. The MEL platform should conduct an annual needs assessment study to inform capacity-building and training strategies. Final Evaluation for SIMPLE Project Report 14 2.2 Evaluation Question 2: Effective MEL Tasks and Processes Evaluation Question 2: What are the types of M&E and learning tasks and processes that are addressed effectively through SIMPLE? SIMPLE effectively addressed a range of M&E tasks and processes, including monitoring, activity MEL planning, support for performance reporting, and evaluations and assessments. 2.2.1 Monitoring, AMELPs and Performance Tracking Data Quality Assessments. The evaluators found that DQAs enhanced data consistency and validity among IPs across the technical offices, ensuring higher levels of compliance with the automated directives system (ADS) requirements, and helping the Mission improve and maintain the quality of data across its IPs. Although some interviewees pointed to challenges in DQA quality and consistency and the lack of prior knowledge of data quality requirements among some IPs, Mission respondents generally noted marked improvements in data quality for reporting on performance indicators over time. As technical office interviewee indicated, “DQAs were one of the main tasks they did very well [and] prepared for very professionally. They analyzed and discussed their recommendations with the IPs to help them develop better indicators to measure results.” Program officers also noted that when consultants had a knowledge of the contextual environment and SIMPLE developed relation of trust with IPs, this better ensured DQA consistency. According to one program officer, “SIMPLE does not use their core team for DQAs, but rather brings consultants that can work within the contextual environment of the activity. [Short-term consultants] coming from the U.S. would not know the issues we have with data collection in Egypt. Having local SIMPLE consultant teams ensures that the service is consistent and there is a relationship built between SIMPLE and IP.” Activity Monitoring, Evaluation, and Learning Plans. Although some interviewees discussed challenges with AMELPs related to multiple layers of review, occasionally vague SIMPLE comments were ascribed to a perceived lack of technical expertise. Because USAID has a mediating role in determining the extent to which platforms work with IPs, in some cases, there was a lack of direct interaction and discussion with IPs. Despite these constraints, Mission and IP respondents had a generally favorable view of SIMPLE's contribution to AMELP reviews. To an increasing extent over time, SIMPLE is being viewed as an advisory service to the Mission. When data discrepancies are discovered, SIMPLE works with technical offices and IPs to incorporate findings in more robust MEL plans. IPs also commented that having undergone revisions of their AMELP with SIMPLE support provided them with opportunities to reflect on how to improve 0 3 6 9 12 15 18 AMELPs and DQAs Use of evidence and data Evaluations Improved appreciation of MEL USAID interviewees n=24 (PO=8; FO=3; TO=13) IP interviewees=5; Total n=29 Program Office Front Office Technical Office IPs (Non-SIMPLE) SIMPLE prepared for DQAs very professionally. They analyzed and discussed their recommendations with the IPs to help them develop better indicators to measure results—Technical Officer Figure 3: MEL Tasks Addressed Effectively Through SIMPLE Final Evaluation for SIMPLE Project Report 15 service delivery. “I value SIMPLE services and the effort they are putting on indicators and revision of IPs MEL plans. There were a lot of discussions and negotiations with the IP on the use of indicators. Several have been changed and several have been added to reflect the Scope of Work.” Another said, “We took a step back to accommodate changes to our plan. It was a learning experience because we had to look and ask what was going on throughout the years of the project. This was not something suggested by SIMPLE—we did that on our own to respond to their comments.” Performance Indicator Tracking Tables (PITT). Prior to SIMPLE, technical offices did not have a unified system for tracking indicators from implementing partners. Despite some delays reported by technical office interviewees due to SIMPLE staff turnover and changing approaches, the first use of PITT in mid-2019 by the Economic Growth (EG) and Democracy and Governance (DG) offices and subsequent use by Education and Health by late 2019, and some technical officers have expressed the desire to see the system further rolled out to implementing partners so that they can directly input data. However, as an early step toward performance data management, the PITT proved to be a well-received and useful resource for the Mission for use in reporting. As one technical officer stated, “I work with my technical team on PPRs. The help that SIMPLE provided in creating a platform to record data in a systematic way is excellent. I hope that it will be rolled to our partners so that they can use it as well. It is not rocket science, but it is a way to show data on a quarterly basis. We have not gotten to a point where we are using it as intended. The PITT table is a wonderful tool.” 2.2.2 Evaluation and Assessments Evaluations and assessments were another area in which SIMPLE provided a quality of service and for which USAID and its implementing partners found useful in terms of findings and integration into program design. However, USAID survey respondents and interviewees noted problems with evaluation, however, primarily in terms of depth of analysis. Evaluations. Technical officers reported that over the course of SIMPLE implementation, evaluations have become progressively timelier and more rigorous. They also pointed to good infographics and executive summaries and consistent quality of dissemination. SIMPLE-facilitated post-evaluation management reviews. The Program Office also noted the value of SIMPLE's contribution to determining the evaluability of interventions, pointing to a workforce development initiative for which the government would not give the required data from schools that are not receiving support, making an impact evaluation infeasible. SIMPLE also conducted evaluation synthesis reports that provided important cross-cutting information for CDCS development. MEL plans are now more sound after SIMPLE's contribution – Technical Officer The help that SIMPLE provided in creating a platform to record data in a systematic way is excellent. I hope that it will be rolled out to our partners so that they can use it as well. It is not rocket science, but it is a way to show data on a quarterly basis. The PITT table is a wonderful tool. — Technical Officer The quality of the evaluation products was always very good in terms of the evaluation reports that we produced and the evaluation debriefings that were arranged to share the findings at the end of each evaluation. These are all well received by the Mission and the IPs. The assessments were very helpful—we used the findings and results of these assessments in making decisions and designing new programs. –Program Officer Final Evaluation for SIMPLE Project Report 16 USAID interviewees n=24 (PO=8; FO=3; TO=13); IP interviewees=5; Total n=29 In the survey conducted by the ET, 75% of USAID staff responding to a question on evaluations indicated that the layout and flow of evaluations and efficiency of the evaluation process were good to excellent, which corroborated findings of the ET review team. One program officer described their appreciation of evaluations and evaluation processes as follows: “The quality of the evaluation products was always very good in terms of the evaluation reports that we produced and the evaluation debriefings that were arranged to share the findings at the end of each evaluation. We also used the findings and results of these assessments in making decisions and designing new programs.” Another program officer noted a steady improvement in the process for planning for evaluations: “We have better planning for evaluations and have a consistent timeframe. Now they know and we plan better with the technical team and SIMPLE about process and timing, so we plan earlier for it to be more useful.” However, there were issues identified with some evaluations, particularly in terms of depth of analysis and usefulness of findings and recommendations. Nearly half (46% n=6) of USAID survey respondents indicated that findings and recommendations were of mixed or moderate quality and most (61%) indicated problems with the depth of analysis in reports (poor quality, n=2; mixed to moderate quality n=5), which, according to interviews with USAID was often linked to the choice of consultants. As one technical officer noted, “Some consultants were not very good, so the problem was in the choice of consultant mainly. The [debriefing] sessions delivered by them weren’t very effective.” Technical officers attributed lack of depth in evaluations and relevant recommendations to an insufficient awareness of contextual factors on the part of consultant teams. One technical officer put it this way, “At the end of each evaluation, they should analyze the findings and provide the technical team with recommendations based on findings, even if that means they use data and information from resources external to the project. This would require a lot of work and expertise, but it would save us a lot of time and effort.” However, according to one technical officer, some of these problems are being at least partially addressed, “Evaluations at the beginning lacked the required in-depth analysis, but later products improved. After a recent evaluation, we discussed these issues with them as well as with the PO, and the products became better.” 0 2 4 6 8 10 12 14 16 Use of evidence and data Use of evaluation findings Use of assessments in design Successes USAID and IP interviewees noting usefulness of evaluations and assessments Program Office Front Office Technical Office IPs (Non-SIMPLE) Figure 4: Utility of Evaluations & Assessments Final Evaluation for SIMPLE Project Report 17 Assessments were also consistently cited as providing timely and useful input for identifying knowledge gaps. SIMPLE conducted two that were noted as particularly valuable in helping Mission staff better understand contextual factors and how to design new activities, such as for anti-trafficking. The wide dissemination of findings using multiple channels were useful for Mission strategy and programming. This was particularly true for an assessment in the education sector on people with disabilities (PwD), which had strong support throughout the assessment process from the GOE, and for which special attention was paid to ensuring inclusivity through the dissemination of results through videos and Braille. A program officer explained, “The PwD assessment was so effective that it gave us all pause and we think how to do it better. We integrated finding into our higher education and new basic education design and we will be integrating these into our strategy. In that way, the learning was absorbed and captured and acting on these learning.” Evaluations and assessment support also demonstrated some capacity on the part of the platform to work in non￾permissive areas. Figure 5: Opinion of Evaluations Although third-party monitoring (TPM) was one of the tasks under SIMPLE's IDIQ and SOW it was not developed during the current SIMPLE period. Evaluators are also cognizant of constraints related to GOE approvals, but some past deliverables by SIMPLE can provide an important basis for developing a TPM system in the future. A program officer noted the platform’s work in the non-permissive areas: “SIMPLE did a performance review for a civil society project with El Gora Association. In general, we need to be able to do more of this kind of work. . . and this is something I think we are expanding our portfolio there over the next few years [to increase] visibility in these areas. It is an area of USAID interest and Washington interest. We never asked SIMPLE because we weren’t sure if this is a road that we want to go down, but we would want to do it for the next contract. Monitoring . . . may need a person on the inside [of the platform] who could do the complex context analysis.” According to one program officer, however, this proposition would need to be very carefully discussed within USAID and would require SIMPLE to identify candidates with access and intimate knowledge of the operating context in less permissive areas. 2.2.3 Facilitation Services In the later years of the contract, SIMPLE provided facilitation services – of both Mission-centric processes such as CDCS development and a senior management retreat – as well as IP meetings with Mission staff. These services were well-received for several reasons; the quality of the facilitation, the pre-existing trust comfort existing between SIMPLE and Mission staff, and SIMPLE’s ability to encourage frank interaction between IPs and technical offices. A technical officer explained, “They did well when it comes to the facilitation of processes with stakeholders, for instance, two years ago, we have a meeting with our IPs and it was part of a consultation session, then more recently, we have another meeting and consultation session and SIMPLE facilitated the entire process in a way that is different to what we are used to. They make it more interactive and they encourage the IPs to speak up their minds, even criticizing the way that we are doing things and business models.” 0% 25% 50% 75% 100% Depth of analysis/ Insights gleaned Findings and recommendations Rigorous use of scientific methods Quality of the evaluation team Efficiency of the evaluation process Layout, narrative and flow of report Poor quality - problematic Mixed - moderate quality Good quality Excellent quality Final Evaluation for SIMPLE Project Report 18 2.2.4 Conclusions: EQ2: Effective MEL tasks and processes SIMPLE has been largely successful in providing quality basic M&E services to the Mission. As these services are greatly lessening the burden on Mission staff, leading to better-quality products from IPs, and beginning to inculcate a culture of M&E, SIMPLE should continue to provide high-quality M&E services and excellent customer service. SIMPLE can also further leverage its position as a trusted facilitator between technical offices and their IPs. SIMPLE can further build on its record of accomplishment in facilitating Mission-centric processes and on its relations with IPs, which are based on frank and open discussions resulting in technical improvements in M&E. 2.2.5 Recommendations for Future MEL programming Recommendation 2.1 The MEL platform should continue with an emphasis on providing M&E support services, including DQAs, AMELP review, assessments, and evaluations, and technical support for using the Development Information Solution (DIS). The platform should expand in scope to include reviews of yearly work plans of selected IPs as directed by the Mission. Recommendation 2.2. The MEL platform should increase the scope of facilitation services for performance management. In addition to providing support for Mission activities on annual performance reporting, the MEL platform should assist the Mission in dissemination of new and emerging USAID strategies among IPs, donor agencies, and other key stakeholders. Table 2: Q2: FCR Table and Impact Matrix Conclusions Findings Data Sources 13 Recommendation SIMPLE has been largely successful in providing quality basic M&E services to the Mission. DQAs, assessments, AMELPs, evaluations, and PITT were useful in providing much needed information for performance management. ISD 2.1 The MEL platform should continue with an emphasis on M&E support services. SIMPLE should build on its position as a trusted facilitator between technical offices and their IPs. SIMPLE provided needed logistical and facilitation support and facilitated communication between the technical office and IPs. ISD 2.2. The MEL platform should increase the scope of facilitation services. High Impact Lower Impact Short- & long-term 2.1 The MEL platform should continue with an emphasis on M&E support services. 2.2. The MEL platform should increase the scope of facilitation services. 2.3 Evaluation Question 3: Remaining Challenges Evaluation Question 3: What M&E and learning tasks and processes remain challenging despite having the Platform in place? 2.3.1 FINDINGS: MEL tasks and processes that remain challenging Mission-wide challenges continue to exist in terms of better integrating learning into the program cycle, as well as knowledge management and sharing learning challenges among USAID staff and IPs. Training and capacity-building for USAID and IPs is nascent; there is room for further improvement in evaluations and 13 Key to Data Sources: I = Interviews; D = Document review; S = Survey of USAID staff Final Evaluation for SIMPLE Project Report 19 evaluation processes; and elements of the platform remain underutilized due in part to a limited knowledge of SIMPLE’s range of tasks among some USAID officers. Elements of the platform as originally conceived, such as third-party monitoring and a fuller use of geographic data, have been postponed due to the impending rollout of USAID Egypt's rollout of the Development Information Solution (DIS) and internal USAID discussions on TPM. Training and capacity-building for USAID staff is nascent and a new capacity-building program was under design at the time of our interviews. As of the time that research was conducted for this evaluation capacity building efforts had yet to yield a deep impact. There is an apparent high degree of technical skill, within the Mission with many technical officers having done their jobs for years, and in some cases for decades. For this reason, training is not always well received, particularly if it is perceived as too basic. According to one program officer, the key to deepening capacity lies in the ability to link training with day￾to-day tasks, “The needs are in what questions should we ask during site visits, how can CORs take learning to next level, and help on analyzing quarterly reports and using these as input for performance reporting.” Moreover, USAID interviewees indicated that although some training was good, most of the training activities were not well attended. In general, interviewees believed that trainings needed to be more interactive and engaging. In the words of one program officer, “They need to find a better way to engage this difficult audience in their trainings by providing something super.” Although a capacity-building task order covered both USAID staff and IPs, formal capacity-building for IPs was relatively limited and most learning was indirect through their participation in activities such as DQAs and through AMELP revisions, which were often well received and improved data quality but did not necessarily lead to changes to programming. In this sense, there were missed opportunities for going further with capacity-building that would have better linked monitoring with programmatic change. One IP respondent explained: “SIMPLE helped us design an M&E plan . . . They were always ready to answer questions, to review and they were patient, providing much needed guidance. However, this did not affect the way we conduct programming. We have now a comprehensive MEL plan that better conforms to USAID regulations, but I can’t see the application of a different methodology or implementation approach; we continue doing what we are doing.” However, some program officers expressed concern that IPs might not be receptive to a program training offered through a MEL platform. In the words of one program officer, “IPs really like the trainings they have received, but I think they still fight it because each IP has its own M&E person. When SIMPLE questions them they get upset about it. IPs were very defensive and SIMPLE got better at talking with them over time.” Despite these concerns among USAID staff, IPs themselves were nearly unanimous in maintaining that a MEL platform should have a greater stress on trainings based on global best practices and innovations in MEL, particularly if it could be done in a non-prescriptive way. According to one IP interviewee, “There should be a bigger and more specific learning component that all the partners know is there and that they Figure 6: USAID Staff MEL Skills Needed Final Evaluation for SIMPLE Project Report 20 can access—not just can but almost required. This would help IPs give some thought to and figure out how [activity implementation] could happen better. But in a non-prescriptive way that each IP could design.” Another IP echoed a similar sentiment, “What was missing is how to include the learning into the design and the process. This would have contributed to the value. We shifted our M&E and we made things simpler based on SIMPLE’s feedback, but [these changes] needed to be included in the design and we did not know how to do it.” Participation in the evaluation SOW process. Opportunities for greater SIMPLE input into the SOW development process for task orders has increased as the platform reinforced its competencies and developed increasingly trusting relationships with technical offices. The Mission provides and develops a preliminary design matrix for the methodology following USAID ADS guidance and template and indicates that the MEL platform should propose a methodology that will lead to quality evaluation report as required by the relevant ADS. These processes are, however, not always well understood in technical offices. In the words of one technical officer, “When we design an evaluation scope of work, as the technical office we are asked to write a scope of work in detail including the methodology that we want the contractor to be using. From my point of view, it is the job of the contractor to present me with the methodology. So, I feel that we do a duplicated job, not only with methodology but also with budgeting as the technical office, and it goes through the Program Office and SIMPLE does that as well.” Evaluations: Room for Improvement in Processes. The ET conducted a review of twelve evaluations and two assessments conducted by SIMPLE. Reviewers focused on conformity with ADS and level of quality control, as exemplified by content, flow, and use of graphic information. The review found that evaluations had a level of quality control that met generally accepted evaluation standards and were in conformity with ADS. However, relatively long timelines from conceptualization to evaluation implementation posed difficulties that likely contributed to a perception that some evaluations were lacking in depth of analysis or actionable recommendations.14 Evaluation timelines were long mainly because of the time it took to develop the task order, not due to time dedicated to the actual fieldwork. Many respondents - from USAID program and technical staff to SIMPLE staff and consultants hired to conduct evaluations expressed similar frustrations about the long timelines for preparation and the short time allowed for fieldwork. A program officer explained, “The process for designing and issuing a task order takes time. The technical team think they want to do an evaluation and they think that SIMPLE can mobilize and do the evaluation in two months which is not the case and this requires the procedures and the procurement office and they issue their procedures and it does take time until an evaluation starts. Setting clear expectation with the team [is important].” Especially early in SIMPLE’s lifetime, because the evaluation task order processes took a long time from conception and development to the start of fieldwork, sometimes highly-qualified consultants originally proposed for evaluations had to drop out due to the delay of implementation because of the time it took to prepare the SOWs, RFTOP and clearance process, and technical and cost proposal reviews—all as required by the DOA Mission Order and ordering guide. One consultant noted, “It took a long time getting the evaluation team recruited and set, then they had to be there tomorrow. The lead up and the prep time always took a long time. We had no say in the questions or the methods – they were all prescribed. Everything always needed to be approved and it all needed to be done immediately.” There were also important sensitivities related to working with and receiving approvals from the Egyptian government. In other cases, technical officers attributed lack of depth in evaluations and relevant recommendations to an insufficient awareness of contextual factors on the part of consultant teams. One technical officer put it this way, “At the end of each evaluation, they should analyze the findings and provide the technical team with recommendations based on findings, even if that means they will depend on data and information from external resources or from similar projects in Egypt. It could require a lot of work and expertise, but it would save us a lot of time and effort.” 14 Because of the range of subject matter, evaluators did not review for depth of analysis or actionability of recommendations and relied on interviews and surveys with USAID for these findings. Final Evaluation for SIMPLE Project Report 21 Scheduling MEL processes. Over the life of the activity, SIMPLE has increasingly shown its flexibility and adaptability within a demand-driven context. With high demand for its products and staff, the platform has often managed multiple overlapping activities simultaneously. Geographic Information Systems (GIS). The Mission currently uses GIS and has developed activity maps. However, the IPs are not as far along with integrating GIS. Although the PITT was well received, it does not yet provide for the integration of geographic information, which is suggested by USAID as a best practice, nor do the IPs systematically collect GIS data. In January 2020, the Mission's MIS/GIS specialist, senior economist, SIMPLE CORs, and members of the SIMPLE team held an exploratory meeting to discuss potential support. SIMPLE also shared the draft PITT dashboard with the Program Office's MIS/GIS specialist and subsequently shared samples of QED's data hub related to GIS and data analyses and visualization, but the Mission has deferred GIS implementation because of guidance related to the Development Information Solution (DIS), which is scheduled for rollout of in April 2021. Challenges in the Operating Context. For nearly one decade, USAID/Egypt has been operating within a delicate operating environment. Multiple interviewees noted how these tensions have made both implementation and data collection for USAID and other donor partners difficult. Central Agency for Public Mobilization and Statistics (CAPMAS) approvals for data collection are time-consuming, often taking months and sometimes not granted at all. This often results in an inability to conduct quantitative evaluations and indicator data collection. One SIMPLE implementer described monitoring and data collection as follows: “There are a couple issues that are very sensitive. In working with the GOE, for example, there are times when SIMPLE had to tailor and tone down the language.” Third-Party Monitoring (TPM). The platform demonstrated its ability to work effectively with partners in less permissive environments. Although SIMPLE has had recent discussions with Mission on TPM, work on TPM has been deferred pending internal Mission discussions. Room for Expansion of MEL Platform Utilization. Some of the potential functions of the MEL platform as initially envisaged under the SIMPLE contract remain underutilized. Based on the ET's content analysis of interview responses to a question asking respondents to describe the range of SIMPLE services, a majority (54%) of interviewees demonstrated relatively limited knowledge of the range of SIMPLE tasks. On one hand, some Mission staff view the platform as overly “reactive” rather than “proactive.” In the words of one program officer, SIMPLE is "not being proactive in terms of anticipating issues and needs to change from being reactive to proactive." The current platform implementers, on the other hand, believe that they could undertake an expanded scope of work but, in some cases, such as information management, based on guidance from USAID's Chief Information Officer (CIO), the Mission has not requested further information management support because the CIO has put a hold on Mission M&E web-based systems in anticipation of DIS rollout. According to one SIMPLE implementer, “This is a demand driven project and some things have not been demanded, or the Mission has not requested these. Tasks such as information management have not been touched due to hold-ups in Washington. We did the performance tracking table and we have not yet done more on that. This is a demand-driven project, and this is OK and normal. I don’t want to frame it along “challenges.” SIMPLE is demand-driven and the Mission have their priorities.” 2.3.2 Conclusions and Lessons Learned What M&E and learning tasks and processes remain challenging despite having the Platform in place? 54% 46% Limited Robust Figure 7: % of USAID Staff Demonstrating Robust or Limited Knowledge of SIMPLE’s Range of Tasks Final Evaluation for SIMPLE Project Report 22 The Mission would benefit from a more streamlined process for engaging the MEL platform in designing evaluation scopes, questions, and methodology. Although a process has already been established and implemented, some technical offices need to better understand that their role is to provide broad outlines for sampling and methodological approaches, and that the evaluation implementer will make specific recommendations based on the suggested broad approaches. Evaluations and assessments take time and highly qualified teams. Despite having a level of quality control that met generally accepted evaluation standards and were in conformity with ADS, in some cases, evaluation timelines were not always enough to ensure sufficient depth of analysis. In other cases, evaluation team members did not have the requisite subject matter expertise or contextual awareness to provide actionable recommendations. These problems were exacerbated by long startup times for evaluations. Regular adjustments to MEL platform work plans enable demand-driven platforms to match capacity with demand and to integrate evolving Mission priorities. Working closely with the Program Office, SIMPLE was able to meet most of its requests in a timely way, but often faced difficulties in juggling multiple simultaneous tasks. This led to a perception among technical offices that evaluations take a long time to prepare for and they see a need for the MEL platform to be timelier. Therefore, ensuring that MEL tasks are spread out evenly, considering resources and level of effort required, is critical to avoid overburdening the platform and risking the potential of reducing the quality of SIMPLE services and deliverables. Capacity-building must be more deliberate. To ensure the sustainability of MEL gains in the Mission and with IPs, basic IP capacity on defining indicators in PIRS and aligning indicators in AMELPs requires further continued attention. Important gaps continue to exist in terms of understanding of USAID data quality requirements among some IPs. Third-party monitoring (TPM) could further contribute to ensuring that implementing partners are achieving and reporting on planned outcomes on these outcomes. TPM is most needed in geographic regions that are not accessible to USAID or MEL platform staff. Geographic information systems (GIS) should be expanded upon. GIS science is expanding rapidly and represents another dimension for collecting, analyzing, and visualizing data. The Mission should continue to institutionalize GIS and mandate that IPs collect and use GIS data at the administrative and district levels. 2.3.3 Recommendations for Future MEL programming Recommendation 3.1. Streamline processes and strengthen MEL platform collaboration in evaluation SOW design: USAID should reinforce its current collaborative process between the MEL platform and technical offices while ensuring a more streamlined process. Recommendation 3.2. Ensure adequate timelines for evaluations: For evaluations, the Mission should ensure that timelines allow for the recruitment and retention of the best evaluation candidates. The Mission should consider allowing the platform to submit an initial cost estimate by category of key and non-key personnel and allow for necessary cost modifications once key personnel are identified and commit to the evaluation. Recommendation 3.3. Reinforce requirements for evaluation analysis: The Mission and MEL platform should emphasize the importance of depth-of-analysis based on a thorough understanding of contextual issues. Recommendation 3.4. Institute formal quarterly workplan reviews. To integrate evolving Mission priorities and proactive ideas from MEL partner, the Program Office and MEL platform should conduct collaborative quarterly workplan reviews. Recommendation 3.5. Require AMELP and DQA training and mentoring for new IPs: The Mission should require that the MEL platform provide all new IPs with AMELP and DQA training, mentoring and Final Evaluation for SIMPLE Project Report 23 coaching. This will better ensure that IPs understand USAID monitoring requirements before drafting monitoring plans. Recommendation 3.6. Initiate third-party monitoring. The Program Office should engage with technical offices to require that the MEL platform conduct pilot monitoring in more remote or less permissive areas in years one and two, progressively rolling out TPM over the life of the activity. Remote channels of implementing TPM should be considered. Recommendation 3.7. Incorporate GIS functions into DIS performance tracking. The Mission should require that awards require IPs to incorporate performance reporting by geographic location to prepare for DIS roll-out. The MEL platform should assist the Mission in ensuring technical office and IP understanding and compliance with GIS requirements through training and technical assistance. Table 3: Q3: FCR Table and Impact Matrix Conclusions and Lessons Learned Findings Data Sources Recommendation The Mission would benefit from further MEL platform involvement in design of evaluation scopes and methodology. Consultation in SOW development process introduced unnecessary duplication. I 3.1. Streamline processes and strengthen MEL platform collaboration in evaluation SOW design. Good evaluations and assessments take time and highly qualified teams and evaluation timelines should reflect this. Evaluation task orders had long design delays and trouble getting approvals from GOE, losing qualified staff as a result. IDS 3.2. Ensure adequate timelines for evaluations. In some cases, evaluation implementation timelines were not always long to ensure sufficient depth of analysis. In other cases, some evaluation team members did not have the requisite subject matter expertise or contextual awareness Nearly half of USAID survey respondents indicated that evaluation findings and recommendations were of mixed or moderate quality and over 60% indicated at least some problem with the depth of analysis in reports. I 3.3. Reinforce requirements for evaluations emphasizing the importance of depth-of￾analysis. Regular adjustments to MEL platform work plans enable demand-driven platforms to match capacity with demand and integrate evolving Mission priorities. With high demand, the platform has often managed multiple overlapping activities simultaneously. IS 3.4. Institute formal quarterly workplan reviews. To ensure the sustainability of MEL gains with IPs, basic IP capacity on defining indicators in PIRS and aligning indicators in AMELPs should be built. SIMPLE capacity-building for IPs was indirect and occurred primarily because of basic SIMPLE M&E functions. Gaps in understanding of AMELP design and DQ requirements persist among some IPs. ISD 3.5. Require AMELP and DQA training and for new IPs. Third-party monitoring (TPM) would further contribute to ensuring that implementing partners are achieving and reporting on planned outcomes on these outcomes. The platform demonstrated its ability to work effectively in non-permissive environments. SIMPLE signaled areas in which third-party monitoring was needed. DS 3.6. Initiate third-party monitoring, particularly in non-permissive areas, progressively rolling out TPM to cover all interventions. Geographic information systems (GIS) could provide the Mission with greater ability to better analyze and report on the geographic scope of their interventions. The PITT was well received, but it does yet not provide for integration of GIS data and many IPs do not ID 3.7. Incorporate GIS functions into performance tracking system. Final Evaluation for SIMPLE Project Report 24 systematically collect GIS data. Higher Impact Lower Impact Short-term 3.1. Strengthen and collaboration on evaluation SOW design. 3.2. Ensure adequate timelines for evaluations. 3.3. Reinforce requirements for evaluations Long-term 3.4. Institute formal quarterly workplan reviews. 3.5. Require AMELP training for IPs 3.6. Initiate third-party monitoring. 3.7. Incorporate GIS functions into performance tracking system. 2.4 Evaluation Question 4: Operational Structure and Co-location Evaluation Question 4a: How did the IDIQ operational structure contribute to the challenges/successes faced by SIMPLE? How did the location of SIMPLE contribute to the challenges/successes faced by SIMPLE? 2.4.1 IDIQ Operational Structure The whole-of-portfolio MEL platform was the first of its kind for USAID/Egypt, and its use represented a steep learning curve for USAID staff, particularly for those in technical offices. The potential flexibility of the IDIQ and task order mechanism was not well understood by contract officers and other Mission staff, leading to long timelines and large bureaucratic burdens when developing task orders. Further iterations of the MEL platform could be designed to include lessons learned on whether, and the extent to which, management burdens on USAID and platform staff have been lightened by consolidating tasks into a single task order during the MEL platform's post-July 2020 extension period. The former contracting officer who helped design the platform, who was interviewed by the ET, explained that he should have focused on explaining the simplicity of the mechanism to the contracts and program office staff, spending more time on the ordering guide and working with the Program Office to “decide what is an absolute must [in terms of requirements] and what can we do without to make it nimble and easy.” As it is, the bureaucratic burden on both technical and program staff has been, and still is, very heavy. A technical officer explained, “The Program Office manages task orders and they do not have enough staff to do this. The process is difficult and bureaucratic. You need two or three months to develop a task order and manage it. We are all very busy and hence we have so many pending things. You want things and they are not core functions. The Program Office is overwhelmed and has to go through a very difficult process to develop and award a task order.” A program officer shared their frustration: “Task orders cause delays. We must draft a scope of work, put in together into a package, circulate it within the mission and get clearance; this takes time. So, it is not that we have the scope of work and we just pass it to SIMPLE and that’s it. No, we have to draft the scope of work, with the budget, with the checklist and that goes around the mission for clearances. It would take like two weeks, if not more. So, I am sure that this has The Mission had never used a single award IDIQ and did not understand the simplicity of the mechanism. The SIMPLE ordering guide added unnecessary steps. It shouldn’t take much longer than a month from concept note to award. There is a tendency to over-document and over￾define up front, rather than adjusting as you go along. There does not need to be an independent government estimate for every task order. You need a basic government estimate but not a full-fledged one, setting a ceiling for costs so the budget doesn’t go wild and make adjustments as you go. The procurement process could have been changed to a much more streamlined and simple process but there is a certain amount of bureaucratic inertia. –Former USAID/Egypt Contracting Officer Final Evaluation for SIMPLE Project Report 25 caused them delays in executing the task immediately. They had to wait for all of that, once the package is cleared, then we process it and get it through our procurement offices. It is a long process.” Another program officer explained how the long processes for task orders from concept note to implementation can often exacerbated by seasonal delays. “Sometimes the process causes delays in evaluation process that puts you out of the season (school season, crop season, antiquities exploration season). Some of our activities have a seasonal nature. Because of this internal bureaucracy we miss a lot and we do not conduct the evaluation at the right time. Because of that, the task order structure did not take into consideration the internal bureaucracy.” When asked to respond to a question on how to improve efficiency and lessen the burden of the task order process, Mission survey respondents overwhelmingly (85%; n=11) indicated that reducing layers of technical review was most important, followed by co-creation with the MEL platform on SOWs (62%; n=8) and moving more task orders into core functions (46%; n=6) and better timing in key personnel recruitment (38%; n=5). Mission staff began to perceive that tasks performed under core functions were simpler and quicker, but that task orders, although they were quicker than an open tender, were still time consuming. In the words of a Program Officer, “It takes a lot to do a scope of work, to do a task order.” A SIMPLE staff member indicated their understanding that some level of procurement is necessary, but that the current system does not well serve the needs of technical offices: “The IDIQ has been the biggest problem in this contract. It helps the procurement process and following up on the budget and making it manageable, but it doesn’t serve the technical needs of the Mission.” To some extent, the process appeared to have improved near the mid-point of implementation in late 2018, as layers of oversight and review were progressively streamlined, and the platform implementer reinforced headquarter-based backstopping for the platform. Despite considerable hurdles, some interviewees compared the system favorably to single awards in terms of timing and comparative ease of budgeting. A technical officer explained how the task orders allowed multiple evaluations to take place: “As a single award such as IDIQ and separate evaluations could be issued relatively quickly, and it allowed for many evaluations to be done. Generally, it was a good mechanism for getting several evaluations done.” However, one program officer lamented the hurdles that still remained, “It is still faster than going outside and using a third party, but it needs to be easier. [The mechanism] affected people’s attitudes towards SIMPLE and doing assessments. People would do more evaluations and assessments if they were under core. The initial areas of improvement have to be on the Mission side to figure out how to use them in a more flexible way. We underutilize them because the burden of trying to figure out how to execute a task order. It is burdensome, less burdensome than a third party but still quite burdensome. This is an area for USAID to figure out.” 2.4.2 Co-location One of the architects of the SIMPLE platform explained how it was initially difficult to convince the embassy and RSO of the need for co-location. “It's important to have them in the Mission, but the security team really fought against it. It is done all the time for other federal agencies. It was very difficult to get 38% 46% 62% 85% 0 2 4 6 8 10 12 Better timing in key personnel recruitment Move more task orders to core Allow a co-creation process in SOWs Reduce layers of technical review USAID staff survey (n=16) Figure 8: USAID Staff Survey on Recommendations for Improving Efficiency and Lessening Burden Final Evaluation for SIMPLE Project Report 26 people to wrap their head around this – the whole idea of them adding value . . . they needed to be physically in the building.” Most respondents noted that having SIMPLE co-located within the Mission and having an annex office within the security perimeter was a net positive. It allowed for easier communication, relationship-building, and coordination between USAID and SIMPLE staff; because they were located in the same building, they could see each other frequently and without an advance appointment. SIMPLE staff and USAID staff indicated that proximity helped build informal relations and increase efficiency. SIMPLE staff appreciated the proximity to Mission staff: “Being inside the USAID Mission, the proximity when we do engage with the technical office, makes it extremely easy to meet USAID staff, call them. It makes the workflow very well and it is easier for us to brainstorm and talk in person.” USAID staff, too, were convinced that co￾location benefitted them. A technical officer noted, “They were available and present in all the meetings; they were close to the program office staff. I am sure it improved the efficiency of the services they provided.” A program officer explained how the proximity increased communication and inculcated a culture of M&E within the Mission. “Co-location enabled [SIMPLE staff] to better communicate with people and understand the dynamics. The easy access put an M&E face everywhere people go, in the cafeteria and the elevator.” SIMPLE also rented an annex facility close to USAID, where they could meet regularly with consultants (who were restricted to five meetings per year within USAID premises) where they conducted meetings with staff or conducted dissemination-related activities. Despite the overwhelming benefits of co-location, there were minor challenges related to security clearances. Some SIMPLE staff noted the long processes in obtaining clearances for Mission access. A few (3) USAID respondents indicated some lack of clarity on what types of information could be shared with SIMPLE; some other respondents were concerned that co-location could cause IPs to perceive SIMPLE as USAID staff. Figure 9: Stakeholders’ Perceptions of SIMPLE’s Co-location 2.4.3 Conclusions The potential flexibility of the IDIQ and task order mechanism was not well understood by contract officers and other Mission staff. These mechanisms often contributed to long timelines and bureaucratic burdens for task orders. Final Evaluation for SIMPLE Project Report 27 Co-location has largely suited the purpose for which it was designed and the combination of co￾location and a nearby SIMPLE annex worked well. There is room to further expand on the annex concept to allow for the creation of an M&E hub within the security perimeter. Enlarged meeting spaces would provide a venue for regular meetings on M&E topics, not only for technical officers and IPs, but with the larger M&E community in Egypt. 2.4.4 Recommendations Recommendation 4.1. Minimize administrative burden: Based on experiences in the contract extension period, which utilizes a single task core mechanism, conduct a robust review to determine if the single task order approach has alleviated burden on program and technical offices and MEL platform. To the extent that this approach has contributed to efficiencies and minimized burden, incorporate it into future MEL platform design. Recommendation 4.2. Conduct peer-to-peer discussions with Missions in other and socialize a revised platform mechanism: To ensure clarity and understanding of mechanism, to inform the new platform design, USAID should promote understanding about the mechanism through peer-to-peer discussions with contract officers in other Missions to discuss options, legal issues, and efficiencies. This should include a discussion of having one platform or multiple task-based platforms under an overarching contract. Results of these discussions should be widely socialized among Mission end users. Recommendation 4.3. Continue co-location and expand MEL platform capacity to host annex-based events: The MEL platform should continue to be co-located while also expanding its capacity for external meetings for IPs, USAID, and other key M&E stakeholders outside of the Mission but within the security perimeter. Table 4: Continue Co-location and Expand MEL Platform Capacity to Host Annex-Based Events Conclusions Findings Data Sources Recommendation Task order mechanism places costly burden on staff and MEL implementers. Mission staff do not fully understand the intent of flexibility and simplicity of the mechanism. Task orders were over-designed and over￾bureaucratic. IS 4.1. Minimize task order burden: Conduct a robust review of other MEL and IDIQ task ordering mechanisms to ensure minimal burden on program and technical offices and MEL platform. Revise the current task order process and ordering guide. Socializing the intended ease of the IDIQ might lead to CORs to realize that are over￾complicating the process, thus easing the burden and shortening timelines. The task order was a heavy bureaucratic burden for Mission staff. This was not how the original contract officer designed it. I 4.2. Conduct peer-to-peer discussions with contract officers in other Missions and socialize a revised platform mechanism. Enlarged meeting spaces would provide a venue for regular meetings on M&E topics, not only for technical officers and IPs, but with the larger M&E community in Egypt. SIMPLE also rented an annex facility close to USAID, where they could meet regularly with consultants. IS 4.3. Expand MEL platform capacity to host annex-based events to conduct group trainings and collaborative learning exchanges. Higher Impact Lower Impact Short-term 4.1. Minimize task order burden. 4.2. Socialize the platform mechanism through peer￾to-peer discussions. 4.3. Expand MEL platform capacity to host annex-based events to conduct group trainings and collaborative learning exchanges there. Final Evaluation for SIMPLE Project Report 28 Long-term 4.2. Conduct peer-to-peer discussions and socialize the revised platform mechanism. Final Evaluation for SIMPLE Project Report 29 3.ANNEXES Annex 1- FCR Table Conclusions Findings Data Sourc e Recommendation To ensure sustainability of M&E gains, feedback￾informed capacity-building should be enhanced. SIMPLE did not have a strong focus on capacity-building and knowledge management. IS 1.1. Integrate key lessons-learned from learning events, CDCS facilitation, and in USAID needs￾assessment into design of capacity￾building curriculum. It is important to develop systems and processes so valuable learning is not lost. Incorporating lessons-learned into future programming has not been as systematic as it could have been. IS 1.2. Strengthen support for new USAID program design through further dissemination of lessons learned (e.g. from summary cross￾evaluation studies). The MEL platform should better understand and cater to the training needs of Mission staff and IPs. There is a need for better-targeted, and more engaging training. IS 1.3. Pilot monthly remote small￾group interactive training and capacity-building sessions. Conclusions 1-3 above Findings 1-3 above IS 1.4. Ensure an explicit emphasis on learning for Mission staff and IPs Learning events are important ways to broadly communicate Mission needs and potential platform responses. A previous learning event was useful in communicating Mission needs and potential improvements. ID 1.5. Continue to conduct annual learning events. To engender trust, roles and responsibilities should be explained early and team building should be promoted. Some lack of understanding of the platform’s services, roles and responsibilities led to tension during project start-up. I 1.6. The MEL platform should conduct extensive and timely team building events with new MEL platform staff and key USAID stakeholders to clarify roles, responsibilities, processes, and expectations. Frequent and targeted orientations will promote understanding of benefits. Trust and understanding of platform’s use and benefits took time to build. I 1.7. The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. Assessing and understanding needs will allow SIMPLE to develop better-targeted capacity building. Trainings were not well-attended in part because they were often not tailored to most relevant needs. I 1.8. The MEL platform should conduct an annual needs assessment study and review of changes in capacity over time to inform capacity-building and training strategies. More frequent training would likely be well-received by Mission and IP staff and engender a culture of learning. Capacity-building was infrequent and Mission staff and IPs both indicated a desire for some regular training. IS 1.9. The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored joint training for USAID staff and IPs. Final Evaluation for SIMPLE Project Report 30 Existing elements of CLA approach should be strengthened. CLA was not an explicit element in core and task order functions for the evaluation. However, elements of CLA exist in SIMPLE's work with USAID and IPs. I 1.10. Ensure explicit emphasis on learning is incorporated into design which include CLA activities for USAID staff and IPs that promotes using data for continuous improvement of processes and products. Higher Impact Lower Impact Short-term 1.1. Integrate key lessons-learned from learning events, CDCS facilitation, and in USAID needs￾assessment into design of capacity-building curriculum. 1.4. Continue to conduct annual learning events. 1.2. Strengthen support for USAID new program design functions through further dissemination of lessons learned. 1.3. Pilot monthly remote small-group interactive training and capacity-building sessions with technical officers and IPs. 1.5. Begin with team building events to clarify roles, responsibilities, processes, and expectations. 1.6. The MEL platform should conduct quarterly structured orientations on tasks and services to technical offices and IP staff. Long-term 1.8. The MEL platform should conduct quarterly capacity-building sessions offering high-quality tailored training for USAID staff and IPs. 1.9. Ensure explicit emphasis on CLA activities. 1.7. The MEL platform should conduct an annual needs assessment study to inform capacity-building and training strategies. Conclusions Findings Data Source Recommendation SIMPLE has been largely successful in providing quality basic M&E services to the Mission. DQAs, assessments, AMELPs, evaluations, and PITT were useful in providing much needed information for performance management. ISD 2.1 The MEL platform should continue with an emphasis on M&E support services. SIMPLE should build on its position as a trusted facilitator between technical offices and their IPs. SIMPLE provided needed logistical and facilitation support and facilitated communication between the technical office and IPs. ISD 2.2. The MEL platform should increase the scope of facilitation services. Higher Impact Lower Impact Short- & long-term 2.1 The MEL platform should continue with an emphasis on M&E support services. 2.2. The MEL platform should increase the scope of facilitation services. Conclusions and Lessons Learned Findings Data Source Recommendation The Mission would benefit from further MEL platform involvement in design of evaluation scopes and methodology. Consultation in SOW development process introduced unnecessary duplication. I 3.1. Streamline processes and strengthen MEL platform collaboration in evaluation SOW design. Final Evaluation for SIMPLE Project Report 31 Good evaluations and assessments take time and highly qualified teams and evaluation timelines should reflect this. Evaluation task orders had long design delays and trouble getting approvals from GOE, losing qualified staff as a result. ISD 3.2. Ensure adequate timelines for evaluations. In some cases, evaluation implementation timelines were not always long to ensure sufficient depth of analysis. In other cases, some evaluation team members did not have the requisite subject matter expertise or contextual awareness. Nearly half of USAID survey respondents indicated that evaluation findings and recommendations were of mixed or moderate quality and over 60% indicated at least some problem with the depth of analysis in reports. I 3.3. Reinforce requirements for evaluations emphasizing the importance of depth-of-analysis. Regular adjustments to MEL platform work plans enable demand-driven platforms to match capacity with demand and integrate evolving Mission priorities. With high demand, the platform has often managed multiple overlapping activities simultaneously. IS 3.4. Institute formal quarterly workplan reviews. To ensure the sustainability of MEL gains with IPs, basic IP capacity on defining indicators in PIRS and aligning indicators in AMELPs should be built. SIMPLE capacity-building for IPs was indirect and occurred primarily because of basic SIMPLE M&E functions. Gaps in understanding of AMELP design and DQ requirements persist among some IPs. ISD 3.5. Require AMELP and DQA training and for new IPs. Third-party monitoring (TPM) would further contribute to ensuring that implementing partners are achieving and reporting on planned outcomes on these outcomes. The platform demonstrated its ability to work effectively in non-permissive environments. SIMPLE signaled areas in which third-party monitoring was needed. DS 3.6. Initiate third-party monitoring, particularly in non-permissive areas, progressively rolling out TPM to cover all interventions. Geographic information systems (GIS) could provide the Mission with greater ability to better analyze and report on the geographic scope of their interventions. The PITT was well received, but it does yet not provide for integration of GIS data and many IPs do not systematically collect GIS data. ID 3.7. Incorporate GIS functions into performance tracking system. Higher Impact Lower Impact Short-term 3.1. Strengthen and collaboration on evaluation SOW design. 3.2. Ensure adequate timelines for evaluations. 3.3. Reinforce requirements for evaluations Long-term 3.4. Institute formal quarterly workplan reviews. 3.5. Require AMELP training for IPs 3.6. Initiate third-party monitoring. 3.7. Incorporate GIS functions into performance tracking system. Final Evaluation for SIMPLE Project Report 32 Conclusions Findings Data Source Recommendation Task order mechanism places costly burden on staff and MEL implementers. Mission staff do not fully understand the intent of flexibility and simplicity of the mechanism. Task orders were over-designed and over-bureaucratic. IS 4.1. Minimize task order burden: Conduct a robust review of other MEL and IDIQ task ordering mechanisms to ensure minimal burden on program and technical offices and MEL platform. Revise the current task order process and ordering guide. Socializing the intended ease of the IDIQ might lead to CORs to realize that are over￾complicating the process, thus easing the burden and shortening timelines. The task order was a heavy bureaucratic burden for Mission staff. This was not how the original contract officer designed it. I 4.2. Conduct peer-to-peer discussions with contract officers in other Missions and socialize a revised platform mechanism. Enlarged meeting spaces would provide a venue for regular meetings on M&E topics, not only for technical officers and IPs, but with the larger M&E community in Egypt. SIMPLE also rented an annex facility close to USAID, where they could meet regularly with consultants. I 4.3. Expand MEL platform capacity to host annex-based events to conduct group trainings and collaborative learning exchanges. Higher Impact Lower Impact Short-term 4.1. Minimize task order burden. 4.2. Socialize the platform mechanism through peer￾to-peer discussions. 4.3. Expand MEL platform capacity to host annex-based events to conduct group trainings and collaborative learning exchanges there. Long-term 4.2. Conduct peer-to-peer discussions and socialize the revised platform mechanism. I = Interview, S = Survey, D = Documents Final Evaluation for SIMPLE Project Report 33 Annex II – Scope of Work Egypt SIMPLE – Scope of Work (SOW) SECTION C – DESCRIPTION/SPECIFICATION/STATEMENT OF WORK C.1 CONTEXT AND BACKGROUD The Mitchell Group, Inc. (TMG) and its Egyptian partner, Beit-Al-Karma Consulting (BK), are pleased to present their proposal for the evaluation of the SIMPLE MEL (Monitoring, Evaluation and Learning) platform mechanism. Evaluation Purpose - The purpose of this evaluation is to: (1) provide evidence and learning to USAID/Egypt to inform the design and use of the next generation of MEL support contracts and; (2) assess the quality of SIMPLE’s products and services. TMG’s proposed approach and methodology for this evaluation is tailored to the accomplishment of this two-fold purpose. C.2 TMG’S APPROACH AND UNDERSTANDING OF EVALUATION QUESTIONS TMG and BK plan to conduct a useful and forward-looking evaluation that the Mission can leverage to both assess the performance of the SIMPLE contract and design a future MEL support mechanism. Our proposed approach is based on the following principles and elements: Ensure depth of analysis, questioning assumptions and following compelling leads during the evaluation process that may garner critical insights that can lead to actionable recommendations. Implement a layered analytical framework rooted in an appreciation of the many moving parts of the SIMPLE contract, with its multiple products and services, wide array of stakeholders, different modalities of delivery and its operational environment. Focus on quality over quantity throughout the evaluation. Many evaluations are not able to provide depth and quality of analysis because they attempt to do too much within a limited timeline and often rush important processes. During this evaluation, TMG will allot the time necessary for quality data collection and analysis. Focus on producing practical and actionable recommendations within the USAID and Egyptian contexts. Prioritize local perspectives regarding SIMPLE’s effectiveness and efficiency, particularly in light of local capacity development and building in-country, sustainable, long-term commitment to high-quality MEL services that can facilitate Egypt’s Journey to Self-Reliance. Thus, the evaluation team will conduct in-depth interviews with local consultants and/or organizations contracted by SIMPLE for work. TMG will conduct this evaluation according to USAID standards articulated in USAID’s Evaluation Policy (2011, updated October 2016), adopting a scientific, systematic and sequential approach to implementing the evaluation and integrating a series of feedback loops at various stages to ensure a robust, participatory process. These loops will ensure that the study remains responsive at all times to the client’s needs, provides opportunities for both client and key stakeholders to participate, sustains communication between the evaluation team and the evaluation’s target audiences and assimilates and responds dynamically to data as it flows in. The ET commits to regular and clear communication with USAID staff and participation at strategic junctures is essential for buy-in of the final product and the way forward. TMG’s approach and methodology for this evaluation has been formulated centrally around our understanding of the four evaluation questions and how best to answer them. Thus, TMG feels it is critically important to articulate its understanding and approach to answering each evaluation question below. Final Evaluation for SIMPLE Project Report 34 EQ 1.a: How is SIMPLE affecting the capacity of USAID/Egypt to successfully conduct monitoring, evaluation and learning functions? To best answer this evaluation question, TMG proposes evaluating SIMPLE’s effectiveness in strengthening the capacity of both USAID/Egypt’s internal MEL tools/plans, systems and processes as well as individuals’ level of competence – knowledge, skills, and practices around MEL that are compliant with USAID policy guidance. These SIMPLE services have largely been conducted under the Core Contract tasks in support of the Mission’s performance management and M&E functions. To evaluate these core services, the evaluation team (ET) will explore lines of inquiry about both individual level MEL capacities strengthened amongst USAID Mission staff and also about MEL systems, processes and tools that may have been improved on account of SIMPLE support services. Practically, SIMPLE’s support to the PMP development and implementation process and the Mission’s capacity to monitor and evaluate projects efficiently and effectively will be examined by the ET. EQ 1.b: What are the key factors identified by stakeholders that affect the perceived utility and function of SIMPLE? This aspect of EQ1 leans the approach to this line of inquiry towards an examination of perceived constraints and opportunities relative to SIMPLE’s efforts. This question requires a probing into what key stakeholders perceive as the weaknesses and strengths that impact the utility and function of SIMPLE. TMG’s proposed evaluation design will surface what SIMPLE’s function is perceived to be, how that compares with its intended function, and differences in perception among stakeholders. The reasons respondents give for their perceptions will be probed in-depth as any future design effort should ensure that these factors are addressed. EQ 2: What are the types of M&E and learning tasks and processes that are addressed effectively through SIMPLE, as reported, and as evident through a selected review of documentation? and EQ 3: What M&E and learning tasks and processes remain challenging despite having SIMPLE in place? TMG sees these EQs as two sides of the same coin – essentially, attempting to understand what has worked well and what hasn’t – both in terms of the quality of MEL services provided and the quality of the results/products of those services. EQ2 (what has worked well) presents a challenge, both in terms of obtaining the un-biased truth from staff engaged in close working relationship with SIMPLE and/or USAID staff, and in terms of obtaining the appropriate level of detail required for a meaningful assessment of the products and services provided by SIMPLE. Not only does SIMPLE deliver many different products and services (categorized under 12 technical tasks), but it does so using two different modalities – (1) services under core contract tasks, and (2) services through Task Orders (TO). These modalities of service provision may prove to be significantly different from one other with respect to their management, communication and oversight requirements, resulting in different service experiences for stakeholders. The ET will evaluate SIMPLE’s effectiveness across all 12 technical tasks divided by the two modalities listed above. EQ 4a: How did the IDIQ operational structure contribute to the challenges and successes faced by SIMPLE? And EQ 4.b: How did the location of SIMPLE contribute to the challenges and successes faced by SIMPLE? TMG’s experience of working under similar contractual arrangements means that it is highly cognizant of the difficulties in maintaining and conveying clear lines of authority and communication in TO implementation and managing the TO process in a timely fashion. Internal arrangements for TO management within mechanisms like SIMPLE also bear scrutiny, as these can impact efficiency and effectiveness. All of these aspects will need close examination and probing to answer this evaluation question. C.3 EVALUATION METHODOLOGY TMG’s methodological approach for this evaluation is tailored to answering all evaluation questions in the most robust, scientific and direct manner possible. TMG is proposing a robust methodology which leverages the Most Significant Change (MSC) method as an entry point to our sequential, mixed-methods approach Final Evaluation for SIMPLE Project Report 35 tailored to thoroughly answer all evaluation questions. The sequential steps of TMG’s methodology are clearly articulated chronologically in the following sections. Desk Review and Document Assessment – The ET will begin the evaluation with a thorough desk review of all project related documentation provided by the Mission. During the desk review, the ET will analyze project work plans and performance indicators, periodic reports, MEL products produced, training manuals or PowerPoints and other project relevant documentation. Along with the desk review, the ET will conduct a document assessment to evaluate the quality and compliance of key documents (e.g. SIMPLE IDIQ award and modification documentations, SIMPLE ordering guide, annual and quarterly reports, annual work plans, MEL plans and evaluations and assessment reports) relative to USAID’s own policy and guidance. For example, TMG will take a sample of evaluation reports and their originating Scopes of Work and evaluate their quality against the Agency’s evaluation checklists (ADS201, ADS Ref.201maa, and ADS Ref.201mab). Results from this analysis and other key takeaways from this desk review and document assessment phase will inform instrument development and the team’s analysis throughout the evaluation. Collect Most Significant Change stories – During the first three weeks on the ground, the ET will coordinate with the evaluation COR to create an email list of key stakeholders (see list of Illustrative Stakeholders in Table 1 below) within USAID, IPs and other stakeholders who would be best-positioned to provide an answer to the general opening question, “From your perspective and experience, what was the most significant change brought about by the SIMPLE project with regard to MEL services provision and/or capacity building? Your response can be positive and/or negative.” Leveraging this email list of stakeholders, the ET will send out an anonymous Survey Monkey1 link that asks key stakeholders for brief narrative response (one paragraph) for their MSC story. Using this anonymous method of gaining insights will allow the ET to garner a more focused framing based on frank perspectives right from the start. The responses received in real-time to these questions will help the ET understand the most notable outcomes of SIMPLE implementation and the insights gained from this preliminary exercise (MSC question) will be exploited to better inform and tailor the qualitative instrument development process such that questions asked can most productively surface information that can answer the evaluation questions. Table 1: Illustrative Stakeholders for SIMPLE Evaluation USAID Front Office representatives Implementing Partner M&E specialists USAID Technical Team Leads Team Leaders and technical consultants from evaluations and/or other types of assessments USAID MEL Officers Stakeholders trained by SIMPLE USAID SIMPLE COR and Alt. COR SIMPLE staff and QED Home Office supervisory staff Implementing Partner COPs Host government counterparts at appropriate levels (as Identified and approved by USAID) who received services under SIMPLE Implementing Partner lead technical staff o Qualitative Methods Instrument Development: The ET will leverage insights gained from the MSC responses, the desk review and the document assessment to craft directed probing questions under each evaluation question. These probing questions will form the instruments utilized by the ET during Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs). The ET will ensure that all questions are crafted in both English and Arabic and framed in a way that locals will understand the crux of the question. All questions will also be vetted by local ET members to ensure that they are culturally sensitive and gender appropriate. The ET will share a draft of the questionnaires for KIIs and FGDs with the COR for USAID approval before testing. Instrument testing will be done with the help of our local partner, BK. Final Evaluation for SIMPLE Project Report 36 Qualitative Data Collection and Sampling Methods: TMG will conduct approximately 90 KIIs. This is an estimate; the final number of KIIs to be performed will depend upon the total number of individuals within our sample population, which will be purposive and determined collaboratively between ET, USAID and IPs. Illustrative data sources (stakeholders for the KIIs and FGDs) are listed in the Evaluation Matrix Table 2. While TMG does not know the exact number of relevant stakeholders that have engaged significantly with SIMPLE activities that could be helpful in answering the evaluation questions, we are proposing an estimated population size calculation based on our review of the indicators provided in Annex 2 of the RFTOP and our past experience. We used this study population estimate to propose a representative number of KIIs to conduct, and have concluded the following groups and number of interviews: (1) 20 IPs with 2 key informants (KIs) in each (total 30 KIs); (2) 10 government key informants; (3) 25 KIs from USAID/Egypt staff; 4) 7 KIs from within SIMPLE project staff; 5) 40 KIs from local consultants contracted by SIMPLE (under TOs). This results in a rough total population size of approximately 112 potential stakeholders. If we sample 90 individuals from the population of 112 key stakeholders, the confidence level will be 95% with a 5% margin of error. This high confidence level allows our sample size to be highly representative of the population of stakeholders with in- depth knowledge of and engagement with the SIMPLE project. Since the vast majority of SIMPLE’s work and key stakeholders are located in Cairo, and to allow for more focus on quality and make the best use of our available resources, all data collection will be conducted in Cairo. The ET will split into two sub-teams composed of two ET members in each sub-team plus one notetaker/translator supplied by BK, which will allow the ET to conduct more KIIs and leave time for the two teams to come together daily to discuss their interviews and perform joint analysis, thus creating a space for the cross-fertilization of insights. Prior to splitting up, the ET will conduct KIIs together for two to three days until the TL is assured that all ET members have a shared understanding and common approach to the interviews. This is critical to ensure the comparability of qualitative data collected. Each sub-team will conduct two to three KIIs daily which will allow the ET to conduct five to six KIIs per day over a three week period, totaling 90 KIIs in all. Each KII will last approximately one hour, but may go longer if the informant has more in-depth experience and perspectives to share. The Logistics and Data Collection Specialist (LDCS) will work in close coordination with USAID and the IPs to arrange all meetings with KIs and will also conduct KIIs and FGDs as she has a robust evaluation and M&E background. The ET will also conduct eight (8) FGDs in order to garner unique perspectives that can better be surfaced by group dynamics. FGDs will be composed of five to eight individuals and can be conducted at the office of our local partner. Illustrative stakeholder group types for the FGDs may include COPs from USAID IPs, M&E specialists, government stakeholders, USAID CORs/AORs and local consultants contracted by SIMPLE. The ET members will seek the permission of all respondents to record the KIIs, which will help local notetakers (provided by BK) transcribe interview notes. For stakeholders that speak Arabic, the notetakers will serve as interpreters during the interview and will also translate notes into English transcriptions. All transcribed notes will then be thematically coded during data analysis. Qualitative Data Analysis: The ET will conduct manual thematic coding of all of the qualitative interview notes collected. The Team Lead (TL) will lead thematic coding with support and inputs from other ET members. First, the ET will identify dominant emergent themes and then they will break down sub-themes under each theme. This will allow the ET to quantitize2 the sub-themes. This quantitized data will then be presented using descriptive statistics to create easy-to- understand visual graphs showing frequency response rates for all of the emergent themes and sub-themes identified, effectively turning qualitative narrative into percentages of response themes that emerge from the analysis and thematic coding. Final Evaluation for SIMPLE Project Report 37 o Quantitative Methods Quantitative Instrument Development: After the qualitative analysis is complete, the ET will have a robust idea of: (1) what they know, (2) what may need further validation and (3) what knowledge gaps remain. The final, online quantitative survey will be an opportunity to validate findings where the ET still doesn’t have total certainty and to ask any questions related to any remaining knowledge gaps. The final online survey will be short – 10 to 15 questions – composed mostly of closed-ended (i.e. multiple choice) questions to help with final triangulation around findings where the team doesn’t have robust certainty, and also to help with answering remaining knowledge gaps. The ET will draft the survey using Survey Monkey. All responses will be anonymous in order to allow stakeholders to be as frank as possible in this final round of questions. The evaluation COR will approve all survey questions before implementation. Quantitative Data Collection and Sampling Methods: Data collection for this final online survey will be fairly straightforward. The ET will leverage the original email list of stakeholders for KIIs and FGDs and may even add some additional stakeholders onto the list. Sample size calculations will thus be based on the same methods as described in the qualitative data collection and sampling methods section above. Based upon the qualitative sample size calculation articulated earlier, at least 90 individuals will be surveyed through an email link to the survey sent out to email list of stakeholders. Respondents will be given two weeks to respond to the online survey. All data will be collected by the ET in real-time, and data will be housed in a secure data server at BK and also backed-up at TMG HQ. Quantitative Data Analysis: This basic survey data will be analyzed using descriptive statistics noting response frequencies for each question, and any notable differences amongst disaggregated stakeholders groups (e.g. USAID staff, IP staff, M&E specialists, government representative, SIMPLE staff, SIMPLE consultants). Visual graphs will be created to facilitate understanding and interpretation of quantitative survey findings. Final Triangulation Analysis – After the ET aggregates their findings and insights garnered from their analysis during (1) desk review and document assessment; (2) qualitative data collection; (3) quantitative, online survey data collection, they will have a variety of sources to conduct a final triangulation analysis across all of these data sets. This triangulation analysis will allow the team to separate the most robust findings from the findings that are less certain. The more the evidence across data sources that point in a similar direction, the more confidence the ET will have in the robustness of those findings. The most robust findings will be centrally highlighted in the final evaluation report. Data Management and Study Limitations – All data collected (qualitative and quantitative) by the ET will be examined as it is collected and cleaned where necessary. The ET will also asses data quality as the data is collected and potentially remove outlier data that may be in error. All data sets will be stored on a secure server at BK and also backed up at TMG HQ. One major limitation of the study is the possibility of a social desirability bias, as many of those who will be interviewed may have a close relationship with the SIMPLE project and/or USAID that they may be wary of putting at risk by being too critical. This is also often due to a desire to continue receiving support and funding from USAID and/or SIMPLE. The team will mitigate this issue by emphasizing the anonymity of everything communicated to the team by ensuring an emphasis on being as honest and frank as possible during the interviews. This is also a key reason why the team is using Survey Monkey both at startup to solicit MSC stories and at the end as a final validation survey to anonymously capture honest perspectives. While recording interviews is ideal as an aid to robust notetaking, it may create an environment where key informants feel more reluctant to be frank. Therefore, the ET will not record specific interviews if respondents are reluctant and it is deemed to affect the quality of their responses. Final Evaluation for SIMPLE Project Report 38 Dissemination – The TMG ET will work collaboratively to scope a dissemination plan with USAID which will ensure that all key stakeholders receive a copy of the final evaluation report, and, if desired by the Mission, the ET can conduct a final presentation of dissemination findings with key stakeholders by VTC. Final Evaluation for SIMPLE Project Report 39 Table 2: SIMPLE Evaluation Design Matrix Evaluation Question Data Sources Data Analysis EQ 1.a: How is SIMPLE affecting the capacity of USAID/Egypt to successfully conduct monitoring, evaluation and learning functions? Desk review of project documents Desk Review/Document Assessment: All project related documentation and indicators will be analyzed and key findings/insights extracted. Core Mission performance management and M&E capacity related products (e.g. MEL plans, PMP, PPR, etc.) supported by SIMPLE will be assessed for quality and compliance using USAID guidance and policy documents. Qualitative data: Thematic coding will be conducted to identify dominant emergent themes and sub-themes. Qualitative data will be quantized to produce frequency response graphs, breaking down sub-themes within dominant emergent themes. Short Online Quantitative Surveys: Data will be analyzed using descriptive statistics noting response frequencies for each question and any notable differences amongst disaggregated stakeholders groups (e.g. USAID staff, IP staff, M&E specialists, government representative, SIMPLE staff, SIMPLE consultants). Visual graphs will be created to facilitate understanding and interpretation of quantitative survey findings. Triangulation analysis: At the end of all data collection a triangulation analysis will be conducted across all datasets (desk review/document assessment, qualitative collection, quantitative online survey collection) to identify the most robust key findings. The reliability and robustness of all findings will be assessed and reported based on the level of repeated confirmation across multiple data sources using this triangulation analysis. USAID front office staff USAID technical team leads Illustrative Probing Questions: 1. From a cost-benefit perspective was SIMPLE effective in strengthening Mission MEL capacities as expected? 2. Name a few specific examples that stand out in your mind as a way SIMPLE has strengthened your capacity to perform hi￾quality MEL functions for the Mission? 3. How has SIMPLE impacted the overall Mission capacity to perform MEL functions, particularly in terms of MEL systems or processes that may have improved. 4. What challenges were encountered by SIMPLE in strengthening USAID/Egypt’s capacity to perform MEL functions effectively. 5. Do you feel your opinions about SIMPLE were well considered and taken into account? 6. Are you utilizing any new MEL skills you have gained on account of SIMPLE? USAID MEL officers USAID SIMPLE COR and Alt. COR and other activity CORs/AORs SIMPLE staff and QED home office supervisory staff Data Collection Methods Most Significant Change (MSC) stories collected at start of evaluation; Key Informant Interviews (KIIs); Focus Group Discussions (FGDs); Online Survey EQ 1.b: What are the key factors identified by stakeholders that affect the perceived utility and function of SIMPLE? Illustrative Probing Questions: 1. What do you see as the main function or and utility of SIMPLE? 2. What factors do you think may have impeded or constrained SIMPLE from functioning as effectively as it may have? 3. What key factors were instrumental is the achievement of positive outcomes by SIMPLE? Final Evaluation for SIMPLE Project Report 40 Evaluation Question Data Sources Data Analysis EQ 2: What are the types of M&E and learning tasks and processes that are addressed effectively through SIMPLE, as reported, and as evident through a selected review of documentation? Illustrative Probing Questions: 1. In your opinion what did SIMPLE do best? 2. How well was learning shared across stakeholders from the various SIMPLE evaluations, assessment and other activities conducted? 3. How well did the SIMPLE platform itself adapt and flex its own approach based on learning from various stakeholders? 4. Which SIMPLE processes where notably efficient in certain ways? Explain. 5. What M&E support functions was SIMPLE particularly good at? 6. Were you happy with the quality of short-term technical assistance provided to the Mission by SIMPLE consultants? Why or why not? (specific question for USAID staff) EQ 3: What M&E and learning tasks and processes remain challenging despite having SIMPLE in place? Illustrative Probing Questions: 1. In terms of MEL support and service provision by SIMPLE, where are some possible areas where SIMPLE could improve? 2. Was there any areas where MEL support was needed but SIMPLE was not able to effectively provide that support? Explain. 3. If you were to design a SIMPLE follow-on activity what MEL services or support activities would you focus on and why? Desk review of project documents, USAID front office, USAID technical team leads, USAID CORs/AORs, USAID MEL officers, USAID SIMPLE COR and Alt. COR, Implementing partner COPs, Implementing partner M&E specialists, Technical consultants from services provided under TOs, Stakeholders who participated in SIMPLE capacity building activities, SIMPLE staff and QED home office supervisory staff, Host government representatives (as identified and approved by USAID) Desk Review/Document Assessment: All project related documentation and indicators will be analyzed and key findings/insights extracted. Core deliverable products of SIMPLE (e.g. evaluation reports, DQA reports, AMELPs, etc.) will be assessed for quality and compliance using USAID guidance and policy documents. Qualitative data: Thematic coding will be conducted to identify dominant emergent themes and sub-themes. Qualitative data will be quantized to produce frequency response graphs, breaking down sub-themes within dominant emergent themes. Short Online Quantitative Surveys: Data will be analyzed using descriptive statistics noting response frequencies for each question and any notable differences amongst disaggregated stakeholders groups (e.g. USAID staff, IP staff, M&E specialists, government representative, SIMPLE staff, SIMPLE consultants). Visual graphs will be created to facilitate understanding and interpretation of quantitative survey findings. Triangulation analysis: At the end of all data collection a triangulation analysis will be conducted across all datasets (desk review/document assessment, qualitative collection, quantitative online survey collection) to identify the most robust key findings. The reliability and robustness of all findings will be assessed and reported based on the level of repeated confirmation across multiple data sources using this triangulation analysis. Data Collection Methods Most Significant Change (MSC) stories collected at start of evaluation; Key Informant Interviews (KIIs); Focus Group Discussions (FGDs); Online Survey Final Evaluation for SIMPLE Project Report 41 Evaluation Question Data Sources Data Analysis EQ 4a: How did the IDIQ operational structure contribute to the challenges/successes faced by SIMPLE? Illustrative Probing Questions: 1. What effect did the contractual structure of the mechanisms have on the way SIMPLE delivered its services? 2. How would you recommend the operational structure and processes be improved? 3. What worked well under the IDIQ operational structure and what did not work so well? EQ 4.b: How did the location of SIMPLE contribute to the challenges/successes faced by SIMPLE? Illustrative Probing Questions: 1. What effect did the co-location of SIMPLE with the Mission have on the way it was perceived? The way it operated? 2. What were the major benefits on SIMPLE performance and results because of co-location? 3. What is the biggest challenge faced by SIMPLE on account of co- location? USAID front office, USAID technical team leads, USAID CORs/AORs, USAID MEL officers, USAID SIMPLE COR and Alt. COR, Implementing partner COPs, Implementing partner M&E specialists, Technical consultants from services provided under TOs, Stakeholders who participated in SIMPLE capacity building activities, SIMPLE staff and QED home office supervisory staff, Host government representatives (as identified and approved by USAID) Qualitative data: Thematic coding will be conducted to identify dominant emergent themes and sub-themes. Qualitative data will be quantized to produce frequency response graphs, breaking down sub-themes within dominant emergent themes. Short Online Quantitative Surveys: Data will be analyzed using descriptive statistics noting response frequencies for each question and any notable differences amongst disaggregated stakeholders groups (e.g. USAID staff, IP staff, M&E specialists, government representative, SIMPLE staff, SIMPLE consultants). Visual graphs will be created to facilitate understanding and interpretation of quantitative survey findings. Triangulation analysis: At the end of all data collection a triangulation analysis will be conducted across all datasets (desk review/document assessment, qualitative collection, quantitative online survey collection) to identify the most robust key findings. The reliability and robustness of all findings will be assessed and reported based on the level of repeated confirmation across multiple data sources using this triangulation analysis. Data Collection Methods Most Significant Change (MSC) stories collected at start of evaluation; Key Informant Interviews (KIIs); Focus Group Discussions (FGDs); Online Survey C.4 PERIOD OF PERFORMANCE Overall, the period of performance will start on the award date of the task order and August 24,2020. [END OF SECTION C] Final Evaluation for SIMPLE Project Report 42 Annex III – Background Documents ● DQA Synergy Meeting 2017 Parts 1 and 2 ● ERAS MEL Session ● GSP Task Order Documents o Returning Scholars Outcome o Scholars Satisfaction Survey o Success Stories o Student Report o Report on Outcome on GoE Institutions o Report on Success Stories – Template o Annual Report Template o Quarterly Performance Report Template o Student Report Template o Training Results and TA Report ● PITT & Dashboard ● PPL MEL Platform Assessment 2016 ● SIMPLE Annual Report 2016 ● SIMPLE Annual Report 2017 ● SIMPLE Annual Report 2018 ● SIMPLE Annual Report 2019 ● SIMPLE Evaluation One ● SIMPLE Evaluation Two ● SIMPLE Evaluation Three ● SIMPLE Evaluation Four ● SIMPLE Evaluation Five ● SIMPLE M&E Presentation New IPs ● SIMPLE MEL Plan ● SIMPLE MEL Starter Pack ● SIMPLE Presentation – 5th Synergy Meeting ● SIMPLE Presentation 2 – FTF ERAS Workshop Kick-Off ● SIMPLE Synergy Meeting February 2018 ● SIMPLE Trend Analysis Final Evaluation for SIMPLE Project Report 43 Annex IV – Evaluation Team Members Headquarters Supervision, Support and Quality Assurance: TMG Headquarters Team provides key support for this evaluation. Ms. Walidah Willoughby, Home Office Assessment Backstop Support supports logistics, contracts, administration, and financial management for the contract and is TMG’s point of contact for oversight of the evaluation. Home Office Technical/Project Manager, Ms. Karen Hirschfeld, has 20 years of development experience, including seven working for USAID. She supports survey design, data visualization, and ensures not only rigorous, robust, and comparable analytical methods across all assessments but also the highest level of quality assurance across deliverables. TMG’s Senior Technical Expert, Mr. Jeremy Chevrier, was responsible for quality assurance and technical oversight supporting the team on a pro-bono basis by reviewing all technical deliverables and facilitated tasks requiring home office intervention and response. Mr. Chevrier has worked as the M&E and knowledge management lead in several USAID missions for eight years. The home office staff is robust, highly competent, and has a proven track record of effective communication, organizational, and administrative support. Replacement Team Lead/Senior Evaluation Specialist: Dr. Michael Midling is a highly-experienced monitoring, evaluation, and learning specialist with 25 years of experience in managing complex programs for USAID, ILO, USDOL and other international development organizations. Dr. Midling has managed over 30 complex and large-scale global, regional, and domestic evaluations and directed large evaluation teams in the conduct of surveys, mid-term, and final evaluations in Africa, Asia, Latin America, the Middle East, and Europe. He has served as the Team Leader and Senior Technical Advisor for over 10 USAID evaluations over the past decade. Dr. Midling also served as Chief of Party for large scale Monitoring, Evaluation, and Learning (MEL) support programs for USAID; one in West Africa and one in the Latin America and the Caribbean region. In Nigeria, he oversaw a $20 million portfolio that provided systematic support to all of USAID’s sectoral teams, sectors on performance monitoring, data quality assessment, impact and performance evaluations, and organizational learning, capacity building and knowledge management. Prior to this, Dr. Midling served at USAID’s Global Development Lab as Research Group Lead (GS15); Senior Program Officer for Evaluation at the International Labor Organization’s (ILO) Program on the Elimination of Child Labor (IPEC); and as country director of the first collaborative social sector program with the People’s Republic of China. Dr. Midling has proven technical and administrative leadership expertise in program implementation and supervision, ensuring that all facets of programming achieve or exceed goals. He holds a joint PhD in International Development Education and Administration and Policy Analysis from Stanford University. Former Team Lead/Senior Evaluation Specialist: Dr. Alice Willard has worked in over 40 different countries doing evaluation work, including in the middle east region. Dr. Willard has extensive experience working within USAID’s M&E requirements and served as Team Lead for a large scale final performance evaluation of the SAREL project, a well-known large-scale MEL platform for improving learning and implementation across USAID resilience activities in Niger and Burkina Faso. Subsequently, she worked with the MEL support project for the Sahel Regional Office in reviewing and analyzing Mission data access, analysis, and use. Dr. Willard has more than 20 years’ experience in assessing, reviewing, and designing M&E systems and is well versed in both quantitative and qualitative research methods. Dr. Willard managed the entire evaluation team and during her tenure with the evaluation, assumed responsibility for the work plan, the evaluation design, and the development of the various evaluation products. Senior M&E Capacity Building Specialist: Mr. Alaa Shaheen’s profile combines a deep knowledge of monitoring and evaluation with practical experience gained over 25 years of building Egyptian and regional capacity for results-based management. Not only has he designed and delivered training courses on M&E for hundreds of individual participants, but he has also designed and implemented practical M&E systems for projects and programs, including the Ford Foundation. He has exposure to USAID’s systems from Final Evaluation for SIMPLE Project Report 44 working as a project officer in the early 2000’s and his continued training efforts ensure that he remains current. Added to this, is his deep experience as a team member and at times, team leader, for evaluations in Egypt. These have ranged in subject from human rights issues to local economic development. Mr. Shaheen also has skills in organizing team planning, effective communications, and facilitating knowledge building and information management. Mr. Shaheen has excellent interpersonal skills, and has the ability to access and interact with a wide range of stakeholders, including national, provincial, and local government officials, private sector, CSOs, and target audiences. Mr. Shaheen provided valuable technical input into the design of the evaluation, its tools, review of the SIMPLE products, as well as facilitated interviews. Senior Local M&E Specialist: Ms. Nahla Hassan is TMG’s Senior M&E Specialist who brings exposure to an extensive range of subject areas that she has evaluated, often as Team Leader, and sometimes as a team member. She has rigorous knowledge of evaluation design and research methods, including professional level quantitative and qualitative skills. She has hands-on experience in planning and conducting evaluation field work and will support the team leader in doing so. Her strong analytical skills and ability to handle both quantitative and qualitative data sets will support the ongoing data analysis that is planned and make substantial contributions to the draft and final reports. She has strong oral and written skills in both Arabic and English. Junior Local M&E Specialist: Mr. Michael Fam is a monitoring and evaluation specialist. Upon obtaining his master’s degree in Economics in International Development from The American University in Egypt, Michael Fam joined CARE International in Egypt where he contributed to the management of a research project assessing the learning outcomes of community schools established by CARE and USAID in Upper Egypt. As a Monitoring & Evaluation Officer, he supervised the development and implementation of M&E systems for a number of projects under CARE’s Education and Women’s Rights Programs, which included overseeing the implementation of a number of baseline studies and evaluations. Mr. Fam’s research and M&E skills are coupled with project management experience where he acted as an Initiative Manager for a project funded by The Bureau of Population, Refugees and Migration (PRM). Final Evaluation for SIMPLE Project Report 45 Annex V – Instruments Final Evaluation for SIMPLE Project Report 46 Final Evaluation for SIMPLE Project Report 47 Final Evaluation for SIMPLE Project Report 48 Final Evaluation for SIMPLE Project Report 49 Final Evaluation for SIMPLE Project Report 50 Final Evaluation for SIMPLE Project Report 51 Final Evaluation for SIMPLE Project Report 52 Final Evaluation for SIMPLE Project Report 53 Final Evaluation for SIMPLE Project Report 54 Final Evaluation for SIMPLE Project Report 55 Final Evaluation for SIMPLE Project Report 56 Final Evaluation for SIMPLE Project Report 57 Final Evaluation for SIMPLE Project Report 58 Final Evaluation for SIMPLE Project Report 59 Final Evaluation for SIMPLE Project Report 60 Final Evaluation for SIMPLE Project Report 61 Final Evaluation for SIMPLE Project Report 62 SIMPLE SURVEY FOR USAID STAFF Final Evaluation for SIMPLE Project Report 63 Final Evaluation for SIMPLE Project Report 64 Final Evaluation for SIMPLE Project Report 65 Final Evaluation for SIMPLE Project Report 66 Final Evaluation for SIMPLE Project Report 67 Timeline of SIMPLE core and task order activities Tables 1 and 2 below show a breakdown of core and task order activities, respectively as reported in quarterly and annual reports from Fiscal Year 2016 through early Fiscal Year 2020. Table 7: SIMPLE Core Task Activities Timeline as Reported (FY 16-FY20 Q2) CORE TASKS AS REPORTED IN QUARTERLY/ANNUAL REPORTS Fiscal Year (FY) 2016 2017 2018 2019 2020 Task # Quarter 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 1 CDCS and/or PMP for the Mission 2 AMELP 6 Track program results & analyze data for PMP, PPR, IP reports 9 Assist development of Mission evaluation assessment plan Additional TO Activities implemented as core tasks as requested by Mission Core capacity building of Mission IPs Core workshops for IPs and Mission Discussion on TPM task order Discussion on potential GIS task order Table 8: SIMPLE Task Order Activities Timeline as Reported (FY16-FY20 Q2) Designed as Task Orders As Reported in Quarterly/Annual Reports Fiscal Year (FY) 2016 2017 2018 2019 2020 Task # Quarter 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 Build capacity of Mission's IPs (GSP) 4 Organize M&E workshops for staff, IPs, NGO, private sector 5 Third party field-based monitoring 7 Annual analysis & trend report 8 Data Quality Assessments 10 Evaluations 11 Assessments, analyses, consultations 12 Analyze, recommend, and streamline use of data (e.g., GIS) Final Evaluation for SIMPLE Project Report 68 Annex VI – Key Personnel Conflict of Interest (COI) Forms Michael Midling Final Evaluation for SIMPLE Project Report 69 Final Evaluation for SIMPLE Project Report 70 Alice Willard Final Evaluation for SIMPLE Project Report 71 Alaa Shaheen Final Evaluation for SIMPLE Project Report 72 Nahla Hassan Final Evaluation for SIMPLE Project Report 73 Michael Fam Final Evaluation for SIMPLE Project Report 74 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523