QUARTERLY PROGRESS REPORT HAITI EVALUATION AND SURVEY SERVICES (ESS) PROJECT Quarter 4, Fiscal Year 2021 (July – September 2021) Submitted October 31, 2021 Contract Number: AID-521-C-17-00002 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Social Impact, Inc. (SI) and do not necessarily reflect the views of USAID or the United States Government. This document was produced for review by USAID. It was prepared by SI for the USAID/Haiti ESS Project. CONTENTS ACRONYMS II INTRODUCTION 1 1 PROJECT STATUS REPORT 2 PROJECT DELIVERABLES 2 TECHNICAL TASKS 2 TECHNICAL TASKS UPDATE 2 ANNUAL WORK PLAN 8 PLANNED TASKS FOR NEXT QUARTER 8 8 KEY ACCOMPLISHMENTS 9 i | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV ACRONYMS AHJEDD Association Haïtienne de Journalistes Economiques pour le Développement Durable AREA Appui à la Recherche et au Développement Agricole ATL Assistant Team Leader CBO Community-Based Organization CDC The Centers for Disease Control and Prevention CDD Community Driven Development CLP Chanje La Vie Plante CO Contracting Officer COP Chief of Party COR Contracting Officer’s Representative DEC Development Experience Clearinghouse DRG Democracy, Rights, and Governance EGAD USAID Economic Growth and Agriculture Office EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team FCR Findings, Conclusions, and Recommendations FY Fiscal Year GOH Government of Haiti HQ Headquarters IAHBP Increased Accountability in Haiti’s Budgetary Process IFOS Institut de Formation du Sud IP Implementing Partner M&E Monitoring and Evaluation MEDA Mennonite Economic Development Associates PLSO Partner Liaison Security Operations Q Quarter RERA Rapid Education Risk Assessment Santé Service Access, Nationwide, Technology, and Empowerment SI Social Impact, Inc. SOD Statement of Difference SOW Statement of Work TL Team Lead USAID United States Agency for International Development WATSAN Water and Sanitation USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | ii INTRODUCTION This report documents Haiti Evaluation and Survey Services’ (ESS) activities, including achievements and challenges for Quarter 4 (Q4) of Fiscal Year (FY) 2021, covering project activities between July 1, 2021, and September 30, 2021. 1 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV PROJECT STATUS REPORT PROJECT DELIVERABLES ESS submitted its FY21 Q3 Quarterly Progress Report on July 30, 2021. In addition, on September 10, 2021, ESS submitted a quarterly Accrual Report, including information on ESS’s expenditures, disbursements, and accruals. TECHNICAL TASKS TECHNICAL TASKS UPDATE During FY21 Q4, the Increased Accountability in Haiti’s Budgetary Process (IAHBP) final evaluation report was approved on August 3, 2021, finalized, and posted to the Development Experience Clearinghouse (DEC) on September 3, 2021. Likewise, the Reforestation Project Mid-Term Evaluation report was approved on August 24, 2021, finalized, and posted to the DEC on September 3, 2021. As usual, ESS had several ongoing tasks that continued from the previous quarter, including the Santé evaluation, Haiti Rapid Education and Risk Analysis (RERA), the AREA (Appui à la Recherche et au Développement Agricole) final evaluation and the ATTEINDRE/ATTAIN activity baseline. ESS launched two new tasks: the Community Driven Development (CDD) mid-term evaluation and the Water and Sanitation (WATSAN) activity final evaluation. ESS details the progress and challenges related to each task below. SANTÉ MID-TERM EVALUATION ESS submitted the final Santé report on May 7, 2021, addressing all of USAID’s comments. USAID informed ESS that the Agency elected to submit a statement of difference (SOD) on the evaluation. USAID submitted the SOD on August 26, 2021. A meeting was organized with SI, USAID’s Program Office, and Health Team on September 24, 2021 to discuss the best way forward. ESS began preparing its response to the SOD at the end of September, which will be submitted in the next quarter. IAHBP EVALUATION The IAHBP evaluation final report submitted to USAID on June 24, 2021 was approved on August 3, 2021. The Mission-wide out-briefing presentation took place on August 31, 2021. ESS finalized the final report and posted it to the DEC on September 3, 2021. ESS provided the close-out file to USAID on September 8, 2021, including the relevant deidentified datasets. This final evaluation answered the following evaluation questions (EQs): 1. To what extent did the Activity improve civil society participation in Haiti’s budgetary process? 2. How did the Activity contribute to improving public monitoring and analysis of budget execution and budgetary management? USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 2 Key informant interviews and focus group discussions from the evaluation suggested that IAHBP contributed to improving understanding and interest among the segments of civil society that participate in the Government of Haiti (GOH) budget process. The Activity also succeeded in putting in place the Observatoire to serve as a watchdog and catalyst for civil society participation in the budget process. In addition, IAHBP helped establish Association Haïtienne de Journalistes Economiques pour le Développement Durable (AHJEDD), an association of budget-sensitized journalists. This has proven instrumental in increasing media coverage about the budget by expanding the number of journalists trained to report on the budget and potentially eliciting more and better civil society participation in the budget process. However, many structural issues, outside IHABP’s SOW and resources, limit the possibilities for effective civil participation in the budget process. Those issues are related, on one hand, to government agencies’ structural weaknesses and Parliament’s limited role in the process and, on the other, to civil society organizations’ lack of resources and technical capabilities, including those targeted by IHABP. The Activity succeeded in triggering interest from some segments of the population and civil society in the monitoring and analysis of budget execution and management and, to that end, the importance of social audits. The Activity seems to have also sensitized the Ministry of Finance on the need to publish data in a form allowing public monitoring of budgets. However, the lack of access to timely information about budget execution published in a simple, clear, yet detailed format still stands in the way of effective public monitoring. The tools the Activity deployed to address these issues, while they did meet their life of project minimum performance indicators, were not widely adopted by stakeholders and the public. The tools suffered from lack of user-friendliness, promotion, and shareability of contents, and have not been updated with newer information about the budget since mid-2019. The Evaluation Team’s recommendations included the following: 1. To continue raising awareness about fiscal transparency, USAID should continue to support interventions to increase public education and awareness on the budget, especially by expanding information to more segments of Haitian society, like underserved populations, those outside Port-au-Prince, and youth, a group that has been instrumental in the “Petrochallenge” fiscal accountability movement. 2. To facilitate Observatoire becoming fully operational, USAID should continue to sponsor Observatoire until it becomes functional and reaches maturity. Observatoire can become an instrument to empower other CSOs to participate in budget advocacy and fiscal accountability. Future assistance should particularly focus on helping the organization attract member CSOs who have the strength and resources to support Observatoire financially. 3. To expand AHJEDD’s reach throughout Haiti to increase media coverage on the budget, USAID should continue to partner with the organization in order to train more journalists from all regions of the country to report on the budget and public finances. 4. To help Haitian civil society with budget monitoring and budget analysis, USAID should consider providing support to (re)designing the civil society online portal and social media tools to disseminate information on the budget in ways that are detailed, yet clear, simple, and easily understood by non-experts. In that regard, the evaluation team advised that the redesigned tools also publish content in Creole and with infographics that are easily sharable on different mediums. 5. To address obstacles to fiscal accountability due to structural issues and weaknesses in Haiti’s public financial management system, USAID should consider supporting programs to 3 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV improve the GOH’s public financial management and its scores under the 28 indicators of the Public Expenditure and Financial Accountability framework. To address the apparent absence of fiscal transparency at the local government level, where municipalities do not even have a budget, USAID should consider developing a budget transparency program with some municipalities (that have a minimum level of capabilities) and provide help on financial management systems. This program could include Observatoire by developing civil society budget watchdogs at the local level as organization chapters. REFORESTATION MID-TERM EVALUATION On July 6, 2021, ESS presented the out-briefing of the mid-term evaluation findings with the technical office. USAID provided feedback and comments on the draft report on July 13, 2021. ESS addressed all the comments and submitted the final report on July 27, 2021, which USAID approved on August 24, 2021. The Mission-wide out-briefing presentation was organized on August 30, 2021. ESS finalized the final report and posted it to the DEC on September 3, 2021. ESS provided the close-out file to USAID on September 8, 2021, including the relevant de-identified datasets. This mid-term evaluation answered evaluation questions related to: 1) the effectiveness of two implementation mechanisms (direct investment vs. grants); 2) if the Project is improving beneficiaries’ resilience in the face of natural and economic shocks; 3) if the Project is integrating youth and gender; and 4) whether the Project has laid the foundation for sustainability. The evaluation concluded that the Activity's two implementation mechanisms each have their advantages and disadvantages that should be strategically weighed in future project development. It also concluded that the resilience activities have increased income but likely only in the short term, that gender and youth integration has not been a strong focus of the Project, and that few, if any, of the activities will lead to a sustainable reduction in deforestation. The evaluation team made the following recommendations for mid-course corrections and long term for future programming. In the final two years of the Reforestation Project, Chemonics should as a first priority: • Not sign any additional grant agreements at this point, and instead focus on completing what is underway. • Understand why the delivery of materials has been so slow, and prioritize the fulfillment of outstanding promises. • Prioritize finding a way to fence off areas where trees are to be planted to protect them from animals. • Continue to employ as many women as possible in the direct investment activities and pay them at least a standard daily wage for their time. Once these top priorities are met, Chemonics should: USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 4 • Focus remaining efforts on the activities that have shown fast uptake, particularly the livestock￾related ones. • Develop the private sector cooperatives and trade associations like FECCANO and RECOCARNO because they have the best chance at providing market opportunities for agroforestry crops. In future reforestation-focused projects, USAID/Haiti should: • Evaluate how to effectively include both direct investment and sub-grants mechanisms in implementation, likely with direct investment for the main projects but using grants where local organizations exist to support promising local projects. • Require a more logical and cohesive strategy for increasing income to a level high enough to decrease deforestation in poorer areas. • Define the project’s objectives and intended outcomes around youth and gender beyond targeting female participants. • Understand the different motivations for deforestation-causing behaviors among various sub￾groups of the population and attempt to focus messaging and activities more specifically to these different demographics. • Balance a results-based payments approach with realities on the ground. • Focus tree planting efforts on places and species that people have strong incentives to protect. HAITI RAPID EDUCATION AND RISK ANALYSIS During FY21 Q4, ESS submitted a revised timeline on July 9, 2021, and the draft report on July 19, 2021. USAID provided feedback on the draft report as well as an updated evaluation SOW with the need to translate the final report into French for the Ministry of Education. ESS submitted the final report on August 18, 2021. The Mission-wide out-briefing presentation took place on August 23, 2021. USAID approved the final report on September 24, 2021. At the end of FY 21 Q4, ESS started translating the final report and the out-briefing presentation into French, as well as drafting English and French briefers as dissemination materials for the stakeholders. ESS awaits the schedule for the out-briefing presentation with the Education Working Group. AREA During FY21 Q4, ESS submitted the revised timeline on July 7, 2021 and USAID provided an introduction letter for the ET on July 14, 2021. The kick-off meeting with the ET, SI, USAID, and the implementing partner (IP) took place on July 21, 2021, where all parties agreed that the ET needed access to the activity documents to inform the drafting of a comprehensive inception report. Upon receipt of the documents, the ET would resume their work and further revise the timeline. USAID shared project annual reports on July 29, 2021. The ET resumed their work and ESS revised the timeline and submitted it to USAID on July 30, 2021. The IP provided additional documents on August 2-3, 2021. 5 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV ESS submitted the draft inception report three days ahead of schedule on August 13, 2021. USAID provided feedback on the draft inception report on August 30, 2021, and ESS submitted the draft evaluation design report on the same day. ESS selected Institut de Formation du Sud (IFOS) as the local data collection firm. The biweekly call was launched on August 25, 2021. The ET requested access to the short-term trainees as well as other activity research publications to help refine the evaluation design. USAID shared the Kenscoff research presentation with ESS on August 31, 2021. ESS submitted the final inception report on September 6, 2021, which was approved on September 17, 2021. USAID provided feedback on the evaluation design on September 8, 2021. The in-briefing meeting was organized on September 14, 2021, with a follow up meeting on September 15, 2021. ESS submitted the revised and final Design Report, timeline, and budget on September 22, 2021. Per the biweekly call on September 22, the ET was instructed to proceed with all data collection efforts other than the survey, as USAID coordinates the 2,000 short-term trainees’ list for the survey component. USAID approved the final evaluation design report on September 27, 2021, which included comments on the tools. The ET used the activity documents to gather contact information for the qualitative data collection participants; the list was not exhaustive. USAID continued to help gather contact information and followed up with the UF to identify missing contacts. At the end of FY 21 Q4, ESS planned for the qualitative data collection, prepared for the pre-test of the survey tools, and conducted translation and back translation of the tools into French and Creole. ATTAIN/ATTEINDRE During FY21 Q4, USAID approved the revised Work Plan on July 2, 2021. The kick-off meeting with USAID, the evaluation team, and ESS took place on July 15, 2021; at that time USAID shared activity documents. The kick-off meeting with the IP took place on July 17, 2021 and a follow up meeting was held on July 16, 2021. The combined Inception and Baseline Design Report was submitted on August 5, 2021 and USAID provided its feedback on August 19, 2021. USAID shared the introduction letter for the ET on August 5, 2021. The ATL was approved and onboarded, and when the initial TL from SI HQ resigned, another SI HQ TL replacement was approved on August 20, 2021. ESS selected IFOS as the local data collection firm. The biweekly call launched on August 25, 2021, and a follow-up meeting with the IP was held on August 26, 2021. It was determined that ESS should carry on the baseline per the design and coordinate with USAID on security for fieldwork in the south. ESS submitted the revised Inception and Design Report on August 31, 2021, and USAID, alongside implementing partner MEDA, provided their feedback on September 9, 2021. ESS submitted the final inception and design report, updated timeline, and budget on September 17, 2021, and USAID approved these deliverables on September 22, 202, with the exception of the budget. The local internal review board approval was received on August 5, 2021, while the Social Impact, Inc. internal review board was granted on September 24, 2021. The in-briefing meeting took place on September 28, 2021. ESS submitted the revised budget on September 30, 2021, and it was approved the same day. At the end of FY21 Q4, ESS worked to finalize the contract process with IFOS. WATSAN ACTIVITY FINAL EVALUATION The primary purpose of the WATSAN final performance evaluation is to determine the effectiveness of the project in achieving its objectives (USAID Water and Development progress indicators). The evaluation will assess the project’s design and approaches used to achieve its objectives in order to USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 6 inform future USAID/Haiti programming similar in scope or approach. Considering the Theory of Change components, this analysis will consider factors related to partnerships, capacity building, and sustainability. For instance, as a key component of the project, the evaluation will measure whether WATSAN interventions in the 10 communes of implementation have resulted in improved capacity of local utilities and local actors to provide households and communities with sustainable, improved access to water and sanitation services and to what extent the interventions have contributed to improved institutional capacity in the sector. Furthermore, this evaluation will help inform USAID/Haiti’s broader learning questions regarding its strategic approach to engaging both central and local government as well as local actors as key avenues for bolstering capacity to deliver quality services. The primary stakeholders for this evaluation include USAID/Haiti, WATSAN implementing partners DAI and its partners Ayiti Nexus (communications/BCC), Centre de Formation et d’Encadrement Technique (CFET) (capacity development), Zanmi Lasante (health and advocacy), V3 (engineering), and mWater (digital monitoring and reporting), and the National Directorate of Potable Water and Sanitation (DINEPA). Other key stakeholders who will benefit from the evaluation include other donors and entities active in Haiti’s WASH sector (IDB, Spanish Aid, World Bank, etc.) as well as a variety of private entrepreneurs, NGOs, and faith-based organizations devoted to improving the status of Haiti’s water and sanitation service provision. USAID shared the draft SOW for the WATSAN final evaluation on August 20, 2021. Follow up technical meetings were organized between USAID and SI HQ and ESS on August 31, 2021, and September 21, 2021, that led to a final SOW on September 22, 2021. At the end of FY21 Q4, ESS began recruiting the ET members as well as drafting the Work Plan, while USAID prepared activity documents. COMMUNITY DRIVEN DEVELOPMENT (CDD) ACTIVITY In September 2021, USAID/Haiti requested ESS design and implement a mid-term performance evaluation of the Community Driven Development (CDD) activity. The purpose of this performance evaluation is to 1) assess the activity’s effectiveness in achieving its objectives, and 2) assess the sustainability of CDD’s multi-sectoral, community-driven approaches. Best practices, lessons learned, and recommendations provided as part of final reporting should inform USAID/Haiti of possible mid￾course activity corrections. The evaluation will assess CDD’s actions with all key partners, assessing the effectiveness and sustainability of the different activity components and their inter-relations. The primary stakeholders for this evaluation include USAID/Haiti, Organization of American States/Pan-American Development Foundation (OAS/PADF), GOH, local municipalities, community-based organizations, water authorities, private sector entities, and other donors and stakeholders in Haiti. USAID shared the draft SOW for the CDD mid-term evaluation on August 20, 2021. A technical meeting between USAID and ESS was organized on August 30, 2021. ESS provided feedback on the draft SOW on September 3, 2021. USAID provided the final SOW on September 17, 2021. At the end of FY21 Q4, ESS was drafting the Work Plan and recruiting the ET members. 7 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV ANNUAL WORK PLAN As a result of USAID internal consultations and prioritization, the ESS Contracting Officer’s Representative (COR) shared an official list of desired Year 5 Work Plan activities with ESS. WATSAN and CDD are the two new evaluations to be added to the ongoing tasks. Relevant draft SOWs were shared with ESS in August 2021 to ensure completion before the end of the ESS contract. However, these tasks are anticipated to be conducted through May 2022. ESS began preparing a request to USAID for a one month no-cost extension as well as a budget realignment considering the current budget ceiling; this will be submitted next month for USAID feedback or approval. All the tasks are launched and ESS will submit the Year 5 Work Plan due on October 31, 2021. PLANNED TASKS FOR NEXT QUARTER In FY 22 Q1, ESS intends to finalize and upload the final Santé and RERA reports to the DEC, contingent on USAID approval, complete translation of the RERA report and dissemination materials into French, and present the Haiti RERA results to the Education Working Group. In addition, ESS intends to submit draft reports for the AREA and ATTAIN activities. ESS will also continue working on the WATSAN and CDD activities. Finally, ESS will be preparing its close-out plan. IMPLEMENTATION CHALLENGES AND SOLUTIONS ESS is effectively navigating three ongoing implementation challenges. USAID.GOV USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT | 8 KEY ACCOMPLISHMENTS During FY21Q4, ESS had several key accomplishments to highlight. • The IAHBP Final Evaluation and the Reforestation Mid-Term Evaluation were finalized and uploaded to the DEC. − The IAHBP final evaluation provided evidence that showed IAHBP’s contribution to improving civil society’s understanding and interest in the budget process, particularly in Port-au-Prince and among the younger and educated population segments. The final evaluation identified IAHBP’s major contributions, its areas of success, challenges, and lessons learned during the implementation of the activity. In fact, IAHBP also contributed to increased media coverage of the budget, both in quantity and in quality, at least among Port￾au-Prince outlets. However, factors beyond the Activity’s resources and scope of work prevent effective public participation in the budget process. − Similarly, the Activity succeeded in triggering interest in monitoring and analysis of budget execution and management. But lack of access to information stands in the way. Tools the Activity deployed to address this issue met minimum performance indicators but were not adopted by the public due to lack of user-friendliness, promotion, and shareability of content. This evaluation will guide future USAID/Haiti programming in areas such as fiscal transparency, government budgets and spending, and other aspects of good governance. − The Reforestation Mid-Term Evaluation provided detailed information on the effectiveness of two implementation mechanisms (direct investment vs. grants), resilience, gender and youth integration, and sustainability. The Project's two implementation mechanisms each have their advantages and disadvantages that should be strategically weighed in future project development. The resilience activities have increased income but likely only in the short term, and gender and youth integration has not been a strong focus of the Project; few if any of the activities will lead to a sustainable reduction in deforestation. The findings, conclusions, and recommendations suggested USAID/Haiti’s possible mid-course corrections and future program orientation of USAID’s Reforestation Project (URP), whose objective is to reduce the threat to targeted forests and increase tree cover. • All tasks assigned to ESS (Santé, RERA, AREA, ATTAIN, CDD, and the WATSAN final evaluation) are moving forward despite major health and security concerns such as COVID-19, kidnappings, and fuel shortages. 9 | USAID/HAITI EVALUATION AND SURVEY SERVICES REPORT USAID.GOV