QUARTERLY PROGRESS REPORT HAITI EVALUATION AND SURVEY SERVICES (ESS) PROJECT Quarter 1, Fiscal Year 2018 (October - December 2017) Submitted January 31, 2018 Revised September 26, 2018 Contract Number: AID-521-C-17-00002 Disclaimer This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Social Impact and do not necessarily reflect the views of USAID or the United States Government. This document was produced for review by the United States Agency for International Development. It was prepared by Social Impact for the USAID/Haiti Evaluation and Survey Services Project. TABLE OF CONTENTS ACRONYMS II INTRODUCTION 1 1 PROJECT STATUS REPORT 1 PROJECT DELIVERABLES 1 TECHNICAL TASKS 2 TECHNICAL TASKS UPDATE 2 PLANNED TASKS FOR NEXT QUARTER 5 IMPLEMENTATION CHALLENGES 5 MAJOR ACCOMPLISHMENTS 6 ANNEX 1: PROJECT STAFF TABLE 0 ANNEX 2: CONTRACT ISSUES AND PROPOSED SOLUTIONS 1 ANNEX 3: REVISED EVALUATION PROCESS MAP 9 i | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV ACRONYMS ATL Assistant Team Leader CO Contracting Officer COP Chief of Party COR Contracting Officer’s Representative DFA Director of Finance and Administration EGRA Early Grade Reading Assessment EQUI Evaluation Quality, Use, and Impact ERP Early Reading Project ESS Evaluation and Survey Services ET Evaluation Team FGD Focus Group Discussion FY Fiscal Year GOH Government of Haiti HBA Haiti Business Accelerator HQ Headquarters HR Human Resources IE Impact Evaluation IFMS Integrated Financial Management Services IP Implementing Partner PE Performance Evaluation Q Quarter QA Quality Assurance SI Social Impact, Inc. SOW Statement of Work USAID United States Agency for International Development USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | ii INTRODUCTION This report updates project activities, particularly achievements and challenges, for the first quarter of fiscal year (FY) 2018, covering the activity period from October 1, 2017 through December 31, 2017. . PROJECT STATUS REPORT PROJECT DELIVERABLES In mid-December, ESS submitted the FY18 Q1 Accrual Report, including information on ESS’s expenditures, disbursements, and accruals. In addition, during Q1, ESS submitted the following deliverables: • FY18 Annual Workplan • FY17 Q4 Quarterly Progress Report 1 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV • FY17 Q4 Quarterly Financial Report TECHNICAL TASKS TECHNICAL TASKS UPDATE ESS continued its work on two evaluations in Q1: The Integrated Financial Management System (IFMS) final performance evaluation (PE) and Early Reading Project (ERP) impact evaluation (IE). In addition, ESS began work on four new tasks, including two assessments and two evaluations. Through implementation of its initial tasks, ESS is testing and refining its proposed evaluation process to mitigate any weaknesses that could affect its ability to ultimately deliver the high-quality deliverables the Mission expects. Given the volume of work and the newness of ESS staff to Social Impact (SI) processes and procedures, particularly SI’s proprietary quality assurance (QA) process, Evaluation Quality, Use, and Impact (EQUI®), ESS is drawing substantially on SI headquarters (HQ) staff for both project management and QA of these tasks. Throughout the implementation of these tasks, ESS staff are shadowing SI HQ staff to learn and over time assume responsibilities for project management and QA. In addition, ESS field office staff are working directly with consultants to support them throughout the evaluation data collection and analysis phases. A more detailed explanation of the status of these tasks is found below. INTEGRATED FINANCIAL MANAGEMENT SYSTEM FINAL PERFORMANCE EVALUATION USAID approved the complete evaluation team (ET) on October 3. Due to delays in identifying, getting approval for, and onboarding the full ET, ESS submitted the Inception Report and Evaluation Design on the same day, October 16, three days later than anticipated based on the revised evaluation timeline. The team assembled in Haiti and met with the Mission and Chemonics for in-briefings on October 23 and 24. During the in-briefing with the Mission, the team discussed revisions to the Evaluation Design Report. Based on that discussion, ESS submitted a revised Evaluation Design Report on October 27, which was subsequently approved on October 30. The Inception Report was approved November 8. With the Evaluation Design approved, the team began data collection. As documented in ESS weekly updates throughout the IFMS Evaluation fieldwork period, data collection was quite challenging. Despite USAID and Chemonics’ best efforts, the team was unable to meet with everyone who could provide necessary insight. To obtain as many interviews as possible, data collection was extended by one week and, therefore, did not finish until November 20. Another challenge ESS faced was the original Sector Specialist’s inability to contribute productively to the fieldwork. As a result, ESS was forced to terminate his contract. Unfortunately, it was difficult to hire another person to take his place in time to support the team with data collection, so ESS staff filled the void by providing additional note-taker services so that the team could still conduct two interviews simultaneously, if needed. ESS identified another Sector Specialist, and approved and hired him in time for him to support the team with the data analysis. Due to team members’ and USAID Technical Office prior commitments and the extended data collection period, USAID and ESS agreed that rather than conduct the out-briefing immediately following data collection, before the team had fully analyzed all the data, the out-briefing would take place USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 2 following the draft report submission, which took place on December 29. Although this was later than planned, USAID agreed to the delay due to the holidays and the team’s travel. EARLY READING PROGRAM IMPACT EVALUATION USAID approved ESS’s ERP IE Work Plan on October 6. ESS also requested consent to subcontract from the ESS Contracting Officer (CO) on October 13, which was received on October 20. Likewise, ESS submitted the team members for CO rate and Contracting Office Representative (COR) technical approval on October 23, which was received on November 6. Due to the extremely tight timeline for completing data collection for the first baseline by early December when exams started and given the lack of ERP background documents on which to base the design, in agreement with USAID, ESS prioritized completion of the Evaluation Design Report over the Inception Report to contend with the extremely tight timeline. To this end, Andrew Carmona, the ERP IE Project Manager traveled to Haiti for a scoping trip. During this visit, he met with all the ERP Implementing Partners (IPs), the selected research firm, and the Director of the USAID/Haiti Education Office. This trip provided information about where and how the activity would be implemented, which was critical to the design. One key piece of information obtained was the lack of any school listing that could be used for identifying comparison schools. Based on this information, ESS determined that a mapping exercise would be necessary. This exercise identified schools with sufficient numbers of students in grades one and two from which to construct a sampling frame. Based on the information obtained during the scoping trip as well as the limited program documentation available, ESS submitted the Evaluation Design on November 1 and the Inception Report on November 13. Both capture the entire IE, which includes an initial baseline in the fall of 2017, a second baseline in the fall of 2018, and the endline in the spring of 2018. To date, neither have been approved, nor has ESS received any feedback. Following the evaluation design development, the next critical step was validating the evaluation tools. Although the entire team was not on board until November 6, the Team Lead and Early Grade Reading Specialist, Joanie Cohen-Mitchell, was approved and onboarded on October 23. That and the timely onboarding of the ESS Logistics and Procurement Manager enabled ESS to host the Early Grade Reading Assessments (EGRA) Tool Validation Workshop in late October. As USAID requested using EGRA tools that had been used in previous evaluations, but that had not been validated by the Government of Haiti (GOH), this workshop was key to ensuring the GOH’s acceptance of the ERP IE results. On the workshop’s first day, the TL, along with ERP IPs, GOH officials, and reading experts reviewed and revised the tools. The research firm then field tested the revised tools. Finally, the validation workshop participants again reviewed the tools based on the field test results and made final revisions. The resulting tools will be used for the ERP IE. With the ERP IE tools validated, ESS and its subcontractor were able to hold the enumerator training, as planned during the first week of November. The mapping exercise was completed on the final day of training. Based on its results, the sample was adjusted slightly; nonetheless, ESS was able to start data collection on time on November 13. Despite severe rains in northern Haiti that delayed some data 3 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV collection, ESS completed all data collection for the first baseline only one week later than planned, still before the start of exams. The research firm, working closely with SI’s HQ, spent the remainder of Q1 cleaning and preparing the data for analysis. AID FOR TRADE ASSESSMENT On September 26, ESS had its first meeting with the USAID Economic Growth and Agriculture Office (EGAD) to discuss the Aid for Trade Assessment. Through this meeting, and a few others throughout the early part of the quarter, ESS supported USAID in developing the assessment’s Scope of Work (SOW). ESS received the final, approved SOW on November 20. Involvement in the SOW development enabled ESS to start its team recruitment process early, with the first interviews being held on October 31. As a result, ESS was able to submit its Work Plan for this task on December 7, well before the 20- day time frame allowed by the contract. ESS submitted its team for CO rate and COR technical approval on the same day. While the Work Plan was approved on December 13 within USAID’s two￾week review period, the team was not approved until December 21. Unfortunately, due the holiday, ESS was not able to onboard the team immediately. AGRICULTURE ASSESSMENT As with the Aid for Trade Assessment, ESS had its first meeting with EGAD about the Agriculture Assessment on September 26 and immediately began developing a Request for Proposals (RFP) for a research firm to conduct focus group discussions (FGD) with beneficiaries of successful agriculture projects identified through the assessment desk review process. In addition, ESS supported USAID in the SOW development. ESS received the final approved SOW on December 14. To avoid delays in the team approval process, ESS, in agreement with USAID, submitted proposed candidates for the Assessment Team before submitting the Work Plan and Consultant matrix for CO rate and COR technical approval. The first batch of proposed team members was submitted on December 18. Unfortunately, USAID was only satisfied with the proposed ATL and further informed ESS that they strongly preferred a team in which all members were knowledgeable about Haitian agriculture. This ruled out any possible expat team leads. While all those identified as fluent in French, otherwise qualified, and available are experts in tropical agriculture, none had any direct experience with Haiti. Discussions about how best to staff the Agriculture Assessment continued throughout December. LOKAL+ FINAL PERFORMANCE EVALUATION ESS had its first meeting with USAID about the LOKAL+ evaluation in October and a follow-up one that included the IP in November. This early involvement of the IP will facilitate their collaboration and cooperation throughout the evaluation process. As has become customary, ESS supported USAID in the SOW development and began the recruitment process immediately. ESS received the final approved SOW on November 16 and submitted qualified candidates’ CVs to USAID on December 8. ESS subsequently submitted its Work Plan for this task on December 18, four days later than expected because it was waiting for USAID feedback on the proposed team before submitting, and the associated Consultant Matrix for CO rate and COR technical approval on December 22. USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 4 KONEKTE FINAL PERFORMANCE EVALUATION USAID first met with ESS to discuss the KONEKTE evaluation on October 25, and based on this ESS started its recruitment process immediately. ESS supported the SOW development and received the final approved SOW on November 21. ESS subsequently submitted its Work Plan on December 21, a two-day delay to await USAID’s approval of the proposed team. PLANNED TASKS FOR NEXT QUARTER In FY18 Q2, ESS will finalize the IFMS and ERP IE baseline evaluations. ESS will also continue implementing the Aid for Trade and Agriculture Assessments and the LOKAL+ and KONEKTE evaluations. Finally, based on the Annual Work Plan developed with its COR, ESS expects to receive final approved SOWs for four new tasks, including the Feed the Future/Chanje Lavie Plante Final Performance Evaluation; KONBIT Mid-term Evaluation, Homeownership and Mortgage Expansion (HOME) Final Evaluation; and Smallholder Alliance for Sorghum in Haiti (SMASH) Final Evaluation. IMPLEMENTATION CHALLENGES As ESS implementation has ramped up, it has continued to face several important challenges, to which it has sought effective solutions. INCONSISTENCIES IN CONTRACT AND USAID PREFERRED PROCESSES As indicated in earlier quarterly reports, ESS has documented several contractual inconsistencies and challenges, particularly relative to USAID’s preferred processes. SI’s Vice-President for the Program Management Unit, Eric Lundgren, visited Haiti in early December, during which he and the ESS Chief of Party met with USAID to discuss proposed contract modifications to address USAID’s preferred processes. USAID indicated openness to considering these revisions. See Annex 2 for a matrix outlining these proposed solutions. 5 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV MAJOR ACCOMPLISHMENTS . MULTI TASK MANAGEMENT During FY18 Q1, ESS has continued to demonstrate its eagerness to work with the Mission to achieve all its planned activities in a timely fashion and at the highest quality possible. ESS has played an increasingly important role in helping the Mission develop high quality evaluation SOWs. Simultaneously, it has worked proactively to identify qualified teams to ensure timely implementation of all tasks. In Q1, ESS implemented two evaluations and began work on two additional evaluations as well as two assessments. STRENGTHENING CLIENT RELATIONSHIP ESS continues to strengthen its relationship with USAID. During this quarter, ESS identified a series of contract modifications that would better support the Mission’s desired evaluation processes. As part of this effort, ESS also revised the previously developed process map to better reflect USAID’s desired evaluation process to facilitate on-time delivery of high-quality products. See Annex 3 for the revised Process Map. USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 6