QUARTERLY PROGRESS REPORT HAITI EVALUATION AND SURVEY SERVICES (ESS) PROJECT Quarter 3, Fiscal Year 2018 (April - June 2018) Submitted September 28, 2018 Contract Number: AID-521-C-17-00002 Disclaimer This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Social Impact and do not necessarily reflect the views of USAID or the United States Government. This document was produced for review by the United States Agency for International Development. It was prepared by Social Impact for the USAID/Haiti Evaluation and Survey Services Project. TABLE OF CONTENTS ACRONYMS II INTRODUCTION 1 1 PROJECT STATUS REPORT 1 PROJECT DELIVERABLES 1 TECHNICAL TASKS 2 TECHNICAL TASKS UPDATE 2 PLANNED TASKS FOR NEXT QUARTER 5 5 MAJOR ACCOMPLISHMENTS 6 i | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV ACRONYMS AfT Aid for Trade ATL Assistant Team Leader CO Contracting Officer COP Chief of Party COR Contracting Officer’s Representative DEC Development Experience Clearinghouse DFA Director of Finance and Administration EGRA Early Grade Reading Assessment ERP Early Reading Project ESS Evaluation and Survey Services ET Evaluation Team FGD Focus Group Discussion FY Fiscal Year GOH Government of Haiti HOME Homeownership and Mortgage Expansion HQ Headquarters HR Human Resources IFMS Integrated Financial Management Services IFOS Institut de Formation du Sud IP Implementing Partner IT Information Technology LEVE Local Enterprise and Value Chain Enhancement PE Performance Evaluation Q Quarter SI Social Impact, Inc. SMASH Smallholder Alliance for Sorghum in Haiti SOW Statement of Work TL Team Lead USAID United States Agency for International Development VSFP Voucher School Feeding Pilot USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | ii INTRODUCTION This report updates project activities, particularly achievements and challenges for Quarter 3 (Q3) of Fiscal Year (FY) 2018, covering project activities between April 1, 2018 and June 30, 2018. PROJECT STATUS REPORT PROJECT DELIVERABLES ESS submitted its FY18 Q3 Accrual Report, including information on ESS’s expenditures, disbursements, and accruals on June 14. In addition, ESS submitted the following deliverables: • FY18 Q2 Quarterly Progress Report • FY18 Q2 Quarterly Financial Report 1 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV TECHNICAL TASKS TECHNICAL TASKS UPDATE In Q3 ESS completed the Aid for Trade (AfT) assessment and continued working on the Agriculture Assessment and three evaluations. As noted in the Q2 report, due to the anticipated continued high volume of work coupled with the loss of one of its technical team members, ESS continued to rely on Social Impact (SI) headquarters (HQ) staff support. Specifically, although the ESS Senior Evaluation Specialist is serving as Project Manager on two tasks and its remaining Evaluation Coordinator and Analyst is supporting several tasks in a variety of ways, HQ staff provide task support as Project Assistants and Managers for many of ESS’s ongoing tasks. The progress made and challenges experienced on each ongoing task is detailed below. INTEGRATED FINANCIAL MANAGEMENT SYSTEM FINAL PERFORMANCE EVALUATION Although the Integrated Financial Management System (IFMS) Final Report was approved at the end of Q2, ESS did not upload it to the Development Experience Clearinghouse (DEC) until April 9, about one week late in order to ensure 508 compliance before the report was uploaded. In addition, ESS closed out its work on this task by submitting a folder to USAID containing all the evaluation’s anonymized data, final deliverables, and PDFs of all approval emails. The only remaining task for this evaluation is the After-Action Review. EARLY READING PROGRAM IMPACT EVALUATION In April, USAID requested that ESS conduct a workshop for all Early Reading Program partners, especially the Ministry of National Education and Professional Training. Toward this end, ESS finalized the report’s Executive Summary based on USAID feedback and from that developed a two-page data visualization-oriented summary of key baseline report highlights. Once these were approved, ESS translated them into French to ensure accessibility to the entire audience. USAID held the dissemination workshop, which substituted for the out-briefing, on April 27. Project Manager Andrew Carmona came to Haiti for the presentation. ESS took advantage of his being in Haiti to discuss the next steps in the evaluation, specifically the midline implementation time frame. During these discussions, ESS learned that Save the Children may drop out of implementing the ERP curriculum due to funding issues. If this happens, it will create significant problems for ensuring a statistically reliable impact evaluation. To ensure that this does not happen, ESS will propose some remedies, including possible additional data collection for the beginning of the school year, before the new curriculum is fully in use. By the end of Q3, USAID had not confirmed if Save the Children would be dropping out. LOKAL+ FINAL PERFORMANCE EVALUATION As noted in the Q2 report, the LOKAL+ data collection start was delayed due to challenges finalizing the evaluation design. To compensate for this, the evaluation team compressed the data collection and was able to finish it within 10 days. This enabled them to complete the data analysis on time and conduct the out-briefing on the originally scheduled date, April 16. Unfortunately, this compressed schedule USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 2 resulted in the team needing more time to complete the draft report so that it could conduct more data analysis. Thus, ESS, with USAID concurrence, delivered the draft report one week late, on May 8. ESS received USAID feedback on June 15, indicating substantial revisions were required. To ensure delivery of the highest possible quality report, ESS requested, and received concurrence, for two extra weeks to finalize the report, which was ultimately submitted. AID FOR TRADE ASSESSMENT With a final team in place, including two SI HQ staff, Zuri Linetsky, who served as Team Lead (TL), and Marissa Germain, who was a team member, ESS submitted an updated timeline, which USAID approved. In addition, ESS again revised the AfT Inception Report, which USAID approved on April 10. The team then finalized the Assessment Design, which ESS submitted on April 16 and was on time based on the new approved timeline. The expatriate team members arrived in Haiti the weekend of April 28 so that they could begin to organize the assessment fieldwork. This started with an in-briefing on May 2, after which USAID approved the design on May 4. The team completed data collection on May 27, as planned. They were supported by ESS Evaluation Analyst and Coordinator Louisena Louis, who stepped in to act as a notetaker when the one hired for this task was unable to travel with the team over a holiday weekend. Once the data collection was complete, the team, including Ms. Louis, worked on the data analysis so that it was complete in time for the out-briefing on June 4. The out-briefing was very well received by the Mission, particularly the Economic Growth and Agricultural Development Office, which had requested the assessment. The assessment findings and conclusions provided strong evidence that USAID’s proposal for a trade-hub￾like activity in Haiti would not work. Based on this information, which was further elaborated in the assessment report, ESS submitted on June 24. This due date, which USAID approved, was one week later than anticipated by ESS and USAID’s agreed upon process. However, the AfT TL, a very strong evaluator and SI HQ staff member, made it clear that delivering a high-quality report with all the necessary review and revision was not possible within the two-week timeframe allocated in the process. This, coupled with ESS experience over the first year of implementation, indicated a need to revise the process to allow for three weeks between the out￾briefing and delivery of the draft report. AGRICULTURE ASSESSMENT As noted in the Q2 report, the assessment team identified more than 500 different agricultural projects implemented in Haiti since 2000. Initially the team only obtained useable documentation that addressed the criteria outlined in Assessment Question 1 from about 40 of these. These were the basis for the team’s initial analysis, which it presented to the Mission on April 12, to discuss how to proceed with the assessment. During this meeting, USAID agreed that the team should continue their literature review so that they could include a wider range of documentation. Thus, ESS submitted the Inception Report on April 30. Subsequently, ESS submitted the initial Assessment Design on May 8. USAID sent its feedback on the Inception Report on May 16 and ESS responded with a revised report on May 24, which was almost one week ahead of the two-week turnaround timeframe under the agreed upon ESS process. Unfortunately, in June, because the assessment was taking longer than anticipated to complete, Niels Morel notified ESS that he would have to step down as TL. ESS quickly identified a new TL, Cynthia 3 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV Berning, who they submitted to the ESS Contracting Officer’s Representative (COR) for approval on June 29. KONEKTE FINAL PERFORMANCE EVALUATION Following an intensive recruitment process, ESS finally identified a strong highly qualified and available evaluation team, which was submitted for USAID approval on May 21. USAID approved the team on May 22. With the new team in place, ESS updated its Work Plan, which USAID approved on May 29. ESS moved quickly to onboard the team within one week, in time for the kick-off meeting on May 30. Based on the agreed upon updated timeline, ESS submitted the Inception Report on June 18, two days ahead of schedule. The team spent the rest of June working on the evaluation design. FEED THE FUTURE – VOUCHER SCHOOL FEEDING PROGRAM EVALUATION On April 2, USAID approved the Voucher School Feeding Program (VSFP) Work Plan, which did not include the selected sub-contractor as the procurement process was still underway. In mid-April, ESS completed its selection process, having chosen INURED as its preferred sub-contractor. Unfortunately, INURED would not accept ESS contract terms. Therefore, ESS was obligated to go with the second highest scoring bidder, Institut de Formation du Sud (IFOS). ESS notified USAID that IFOS was sub￾contracted on April 23. With IFOS on board, the kick-off meeting took place on May 4. IFOS then began working on the evaluation design, which ESS submitted one week later, on May 11. Unfortunately, USAID was unhappy with the initial design, which required several rounds of revision before USAID approved it on June 6. The in-briefing took place on May 17 to provide an opportunity for IFOS to speak directly with USAID about the design issues. Due to delays in finalizing the design, data collection did not start until June 13, which raised concerns about IFOS’s ability to complete data collection before school ended. To facilitate timely completion of the data collection, IFOS added additional team members so that it could collect all the necessary data in a compressed timeframe. As a result, they completed the data collection by the end of school, on June 19. They spent the rest of Q3 on the data analysis. RAPID USAID sent ESS a final approved statement of work (SOW) for another “small” evaluation on April 30. ESS submitted the Work Plan for this task on May 29, one day late. Like VSFP, ESS had to conduct a full procurement to identify the sub-contractor for this task. Despite its best efforts to speed the procurement process up as much as possible, it was impossible to do so such that data collection could start by June 6, in time for it to be finished before classes ended. When submitting the Work Plan, ESS proposed two options for addressing this, delaying the data collection to the start of the new school year or completing it during the summer with the Implementing Partner’s agreement that key stakeholders would be available for interviews. USAID agreed to the first option – delaying data collection. This delay would also allow for obtaining Institutional Review Board approval so that children, the real project beneficiaries, could also participate in the evaluation. On June 26, USAID requested an updated Work Plan that included identifying the winning sub-contractor. USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 4 KONBIT As noted in the Q2 report, ESS supported USAID in developing good evaluation purpose and questions for the KONBIT mid-term evaluation. With this process complete, USAID sent ESS the final approved SOW on June 11. Working on the SOW development enabled ESS to start its recruitment process very early and lock-in a well-known and well qualified TL. As such, ESS was able to submit the KONBIT Work Plan on June 22, two weeks early. CHANJE LAVIE PLANTE As with KONBIT, ESS participation in developing the purpose and questions for the Chanje Lavie Plante (CLP) Evaluation positioned it well for starting recruitment early. As such, when USAID sent the final approved SOW on April 2, ESS was able to quickly finalize the Work Plan, submitting it on April 30. Unfortunately, the originally selected TL withdrew from the evaluation. Thus, ESS had to revise and resubmit its Work Plan once an alternate was identified. USAID approved the final Work Plan on June 6. ESS submitted the entire team for approval on June 21. As part of the SOW, USAID requested that ESS identify two research firms to assist with the evaluation data collection, one for a household survey and another for qualitative data collection, particularly focus group discussions. To address this element of the SOW, throughout Q3, ESS conducted two procurement processes. Although the quantitative Request for Proposal closing date was extended by one week to June 4, only one firm submitted a proposal. Fortunately, their proposal was acceptable. Although no bidders were identified as within competitive range for the qualitative work, ESS decided to wait until the evaluation design was finalized before completing the best and final offer process. SOW DEVELOPMENT SUPPORT In addition to its ongoing task implementation, USAID requested ESS support developing the SOW for the Water and Sanitation (WATSAN) Activity baseline. Toward this end, ESS participated in a meeting to discuss the baseline objectives on June 21. PLANNED TASKS FOR NEXT QUARTER In FY18 Q4, ESS hopes to finalize the ERP baseline report, the AfT Assessment, and the LOKAL+, VFSP, and KONEKTE evaluations as well as hold after-action reviews for IFMS, AfT, and LOKAL+. In addition, ESS expects to continue working on the Agriculture Assessment, and the RAPID, KONBIT, and CLP evaluations. ESS will continue to support USAID in finalizing the WATSAN SOW so that it can move forward with that task as well. Finally, based on the Annual Work Plan developed with its COR, ESS expects to receive final approved SOWs for three new tasks, including Homeownership and Mortgage Expansion (HOME) Final Evaluation, Smallholder Alliance for Sorghum in Haiti (SMASH) Final Evaluation, and Local Enterprise and Value Chain Enhancement (LEVE) Final Performance Evaluation. . 5 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV MAJOR ACCOMPLISHMENTS USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 6