PERFORMANCE EVALUATION OF INCREASED ACCOUNTABILITY IN HAITI BUDGET PROCESS (IAHBP)/GROUP CROISSANCE EVALUATION REPORT MAY 2021 CREDIT: GROUP CROISSANCE 2017 PERFORMANCE EVALUATION OF INCREASED ACCOUNTABILITY IN HAITI BUDGET PROCESS (IAHBP)/GROUP CROISSANCE EVALUATION REPORT Date: May 2021 Submitted To: USAID/Haiti Evaluation Mechanism Number: AID-521-A-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. Contact: Salima Mutima, Chief of Party 2300 Clarendon Blvd, Suite 1000 Arlington, VA 22201, USA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV PERFORMANCE EVALUATION OF IAHBP | i ABSTRACT Social Impact, Inc. (SI)’s Haiti Evaluation and Survey Services (ESS) project conducted the final Performance Evaluation (PE) of the Increased Accountability in Haiti Budget Process (IAHBP) Activity, implemented by Group Croissance, S.A., under a cooperative agreement. This evaluation report documents project performance, results, impacts, lessons learned, and best practices from various stakeholder perspectives. It will also guide future USAID/Haiti programming in areas such as fiscal transparency, government budgets and spending, and other aspects of good governance. Evidence showed IAHBP contributed to improving civil society’s understanding and interest in the budget process, particularly in Port-au-Prince and among the younger and educated population segments. IAHBP also contributed to increased media coverage of the budget, both in quantity and in quality, at least among Port-au-Prince outlets. However, factors beyond the Activity’s resources and scope of work prevent effective public participation in budget process. Similarly, the Activity succeeded in triggering interest in monitoring and analysis of budget execution and management. But lack of access to information stands in the way. Tools the Activity deployed to address this issue met minimum performance indicators but were not adopted by the public due to lack of user￾friendliness, promotion, and shareability of contents. ii | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS III EXECUTIVE SUMMARY 1 I. INTRODUCTION 6 Activity Background 6 Activity Description 7 IAHBP Development Hypothesis 7 Result Framework 7 Evaluation Purpose and Audience 8 Evaluation Questions 8 II. EVALUATION DESIGN 8 Data Collection Methods 9 Desk Review 9 Key Informant Interviews 9 Focus Group Discussions 10 Informant Sampling 11 Data Analysis 11 Data Processes 11 Data Analysis Methods 12 Limitations and Mitigation Strategies 12 Biases 12 Other Limitations 13 III. FINDINGS AND CONCLUSIONS 14 EQ1: To what extent did the Activity improve civil society participation in Haiti’s budgetary process? 14 Findings 14 Conclusions 23 EQ2: How did the Activity contribute to improving public monitoring and analysis of budget execution and management? 24 Findings 24 Conclusions 30 IV. RECOMMENDATIONS 31 ANNEXES 32 Annex A: Evaluation Scope of Work 32 Annex B: Evaluation Team Members 42 Annex C: Informed Consent Scripts and Data Collection Protocols 44 Annex D: References and Endnotes 49 Annex E: List of Persons Interviewed 51 Annex F: Evaluation Design Matrix 54 Annex G: Disclosure of Conflict of Interests 56 iii | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ACRONYMS ADS Automated Directive System AHJEDD Association Haïtienne des Journalistes Économiques pour le Développement Durable AMEL Activity Monitoring and Evaluation ATL Assistant Team Leader BRH Central Bank of Haiti (Banque de la République d’Haïti) CNEH National Confederation of Haitian Educators (Conseil National des Éducateurs Haïtiens) COR Contracting Officer’s Representative CSCCA Court of Auditors (Cour Supérieure des Comptes et du Contentieux Administratif) CNMP Commission Nationale des Marchés Publics CSO Civil Society Organization DOS U.S. Department of State DR Document Review EQ Evaluation Question EQUI Evaluation Quality, Use, and Impact ESS Evaluation and Survey Services ET Evaluation Team FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FY Fiscal Year GOH Government of Haiti IAHBP Increased Accountability in Haiti Budget Process IBP International Budget Partnership IP Implementing Partner IS Internet Search KI Key Informants KII Key Informant Interview LOE Level of Effort LOP Life of Project MEF Ministry of Finance OBI Open Budget Index PA Program Assistant PD Program Director PE Performance Evaluation PEFA Public Expenditure and Financial Accountability PI Performance Indicator PM Program Manager PREPOC Programme de Relancement Post Covid QA Quality Assurance SI Social Impact, Inc. SOW Scope of Work TL Team Leader USAID United States Agency for International Development USAID.GOV PERFORMANCE EVALUATION OF IAHBP | 1 EXECUTIVE SUMMARY The United States Agency for International Development (USAID) has requested that Social Impact, Inc. (SI)’s Haiti Evaluation and Survey Services (ESS) project design and implement the final Performance Evaluation (PE) of the Increased Accountability in Haiti Budget Process (IAHBP) implemented by Group Croissance, S.A., under a cooperative agreement. The purpose of this PE report is to present the evaluation team’s (ET) main findings, conclusions, and to make recommendations on next steps. EVALUATION PURPOSE AND EVALUATION QUESTIONS This PE provides an opportunity to do an in-depth investigation and analysis of the issues surrounding the implementation of IAHBP, and to document project performance, results, impact, lessons learned, and best practices from various stakeholders’ perspectives. It will also provide learning for future USAID/Haiti programming in areas such as fiscal transparency, government budgets development and spending, and other aspects of good governance. This evaluation’s findings, conclusions, and recommendations are also relevant for Group Croissance and other civil society organization (CSOs) interested in advancing budget transparency. This final evaluation will answer these evaluation questions (EQs): 1. To what extent did the Activity improve civil society participation in Haiti’s budgetary process? 2. How did the Activity contribute to improving public monitoring and analysis of budget execution and budgetary management? EVALUATION DESIGN, METHODS, AND LIMITATIONS The ET used a qualitative evaluation approach, consisting of document review (DR), Internet search (IS), key informant interviews (KIIs), and focus group discussions (FGDs). Due to the COVID-19 pandemic, the ET performed 48 KIIs and six FGDs remotely through phone calls and technology platforms (Zoom, Microsoft Team, or Google Meet). KIIs and FGDs included 54 males and 30 females, selected from a mixed purposeful and random sampling approach from different stakeholder groups, including Government of Haiti officials (GOH), leaders of CSOs, USAID, the implementing partner (IP), etc. The ET used a rigorous data analysis approach and triangulated data to the greatest extent possible, within the confines of available resources and the allocated time period. Researchers cross-checked results, applied data processes and analytical methodologies explained below to support the evaluation’s final findings and conclusions. The ET took steps to identify possible biases and constraints. 2 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT DID THE ACTIVITY IMPROVE CIVIL SOCIETY PARTICIPATION IN HAITI’S BUDGETARY PROCESS? FINDINGS IAHBP facilitated the creation of a network of journalists1 specialized in reporting on GoH budget process and public finances development. IAHBP identified an existing group of journalists trained by the Central Bank of Haiti (Banque de la République d’Haiti or BRH) on budget and economic reporting. IAHBP spearheaded the formal launch on January 18, 2018 and provided logistical and financial support for the legal registration of the association, called Haitian Association of Economic Journalists for Sustainable Development (Association Haïtienne des Journalistes Économiques pour le Développement Durable or AHJEDD). AHJEDD has quickly established its presence and grown in notoriety as an institution well versed in budget analysis and reporting. AHJEDD legally registered as an association, elected a governing committee and started organizing budget-related events. The newly formed association has demonstrated its sustainability by continuing to train additional journalists on reporting about the budget and the economy, thereby fulfilling part of IHABP’s mandate: to train the trainers. By informing the public on budget matters, AHJEDD has increased interaction between civil society and GOH through regular events, meetings, and discussion sessions. AHJEDD co-publishes, in collaboration with GC and Papyrus, 2 an online periodical called “DevHaiti” focused on informing the public about Haiti’s economy and the budget. AHJEDD has become a mechanism that officials from the Ministry of Finance regularly leverages by holding special information sessions on the budget, especially at the formulation stage. IAHBP also invited different CSOs and individuals to create a civil society entity to serve as budget watchdog. Indeed, on January 8, 2018, IAHBP launched the Civil Society Observatory on Haitian Public Finances (Observatoire de la Société Civile sur les Finances Publiques Haïtiennes, or Observatoire).3 The Observatoire was able to mobilize CSOs and other stakeholders on budget issues quickly after its formation and became a trusted partner by the GOH. Since its establishment, the GOH frequently consults the Observatoire, especially the Ministry of Finance (MEF), during budget formulation stages. An Observatoire member remarked, “MEF always communicated to us the budget orientation letter upon its release.” IHABP’s results framework was designed for Observatoire to be autonomous, financially sustainable, and able to independently organize budget advocacy events. However, due to numerous setbacks, it has not fulfilled this expectation. For example, setbacks in structuring its membership still remain. IHABP reported that the representatives of the initially targeted CSOs did not consistently attend planned meetings or the CSOs would at time send different individuals, less informed about the process, which made it difficult to maintain momentum in organizing it as a consortium of CSOs. To address these challenges, IAHBP switched strategies and launch the Observatoire as a small core group of professionals from different sectors. Observatoire’s legal registration and formalization also remains 1 Activity contract, Attachment B -Program Description (Page 16) referred to those journalists as “journalists sensitive to the National Budget.” 2 A USAID implementing partner in Haiti. 3 Available at https://lenouvelliste.com/public/index.php/article/182110/deux-nouveaux-organismes-pour-promouvoir￾linformation-economique 3 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV uncertain. During the FGDs, most members indicated that they had still not read the draft articles of incorporation, bylaws, mission statement, etc. However, the members will need to approve these documents prior to legal registration. IHABP conducted numerous interventions throughout the country to increase civic education and raise awareness4 about the budget. Nearly all participants across all stakeholder groups reported satisfaction with IHABP’s public education and awareness interventions. Informants also appreciated the quality and relevance of the trainings as well as how those interventions have provided them with a better understanding of the budget, its components, and other elements they did not previously understand. A strong majority of KIIs (29 out of 47) across all stakeholder groups and all the FGDs credited the Activity with increased understanding and interest among the Haitian population about the budget. One economist mentioned “they understand that the budget will have an influence on their quality of life through exchange rate and inflation.” Some informants gave examples of people in street markets and public transportation talking about the budget “to the point you will hear people sometimes: this budget is not good.” Another political party leader noted: “Thanks to the Activity, the budget has been in the political agenda of almost everybody.” Stakeholders 5 indicated there are persistent limitations to civil society participation. CSOs construed “civil society participation” as meaning that the government is both open and responsive to their input. They expressed frustration that despite discussions or recommendations made to the Executive Branch or Parliament, their view was not taken into account. While most stakeholders agree that there are a variety of factors limiting civil society participation in the budget process, their views tend to differ from that of GOH stakeholders. Some GOH officials interviewed highlighted that there are competing demands or even conflict of interest among different CSO groups whose requests regarding the budget are not aligned with each other. Those GoH officials gave examples that private sector CSOs tend to want the GoH to shift strategy towards privatization of public services (like healthcare), whereas other CSOs advocate for more public investments in GoH run clinics and hospitals. A few other GOH officials expressed the view that CSO groups failed to understand that the budget is financed through taxes, which those same groups do not want the government to raise even though current levels of tax revenues are not sufficient to satisfy the competing demands of different sectors. A large majority of informants faulted CSOs’ weaknesses as an impediment to budget advocacy. Informants indicated that most CSOs do not have adequate financial resources and adding budget advocacy in their interventions would burden them with hiring paid experts. Most informants expressed concern that the limited funding will continue to restrict CSOs’ capabilities to intervene in the budget process. For future intervention, some informants proposed improvement in the identification of CSOs with pre-existing strengths and capabilities to ensure their commitment to budget advocacy. Stakeholders point to various structural constraints that limit civil society participation in the budget process and are not amenable to civil society participation. The main element stakeholders cited is the lack of technical capabilities in different ministries to formulate budget proposals for their respective 4 Activity terminology in Program Description. Result framework’s IR2 is: “Increase public participation in the budget process through public education and awareness activities. (Page 15, Attachment B – Program Description). 5 Stakeholders reflect different groups of people who had a stake in the activity. Explicitly including different stakeholder groups during data collection facilitated data triangulation in that it allowed the team to assess the extent to which various groups’ perceptions about key evaluation issues were the same or not. 4 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV agency and sector. A GOH official noted for example that “Most Ministry’s Study and Programming Units [Unité d’Étude et de Programmation] in charge of formulating budget proposals are understaffed.” A near majority of key informants (KIs) said that the budget usually lacks “credibility” (“is not sincere”). The ET corroborated this with results from the World Bank and IMF framework “Public Expenditure and Financial Accountability (PEFA)” 2012 report on Haiti. CONCLUSIONS KIIs and FGDs suggested that IAHBP contributed to improving understanding and interest among the segments of civil society that participate in the GOH budget process. The Activity also succeeded in putting in place the Observatoire to serve as a watchdog and catalyst for civil society participation in the budget process. In addition, IAHBP helped establish an association of budget-sensitized journalists, which has proven instrumental in increasing media coverage about the budget, in expanding the number of journalists trained to report on the budget, and potentially eliciting more and better civil society participation in the budget process. However, many structural issues, outside IHABP’s SOW and resources, limit the possibilities for effective civil participation in the budget process. Those issues are related, on one hand, to government agencies’ structural weaknesses and Parliament’s limited role in the process and, on the other, to CSOs’ lack of resources and technical capabilities, including those IHABP targeted. EQ2: HOW DID THE ACTIVITY CONTRIBUTE TO IMPROVING PUBLIC MONITORING AND ANALYSIS OF BUDGET EXECUTION AND BUDGETARY MANAGEMENT? FINDINGS Some stakeholders think the Activity had some influence on the popular mobilization under the “Petro￾challenge” movement led by young Haitian activists (“Petro-challengers”) demanding accountability on the use of funds Venezuela loaned to Haiti as part of “Petrocaribe” program. Some of those KIs thought that, the Activity’s training on social audits may have influenced some youth activists to use social audits as part of the Petrochallenge. The ET also discussed mobile app and social media tools with different non-IP related informants. None of those KIs said they used the app or the Activity’s social media accounts as sources of information about the budget. In fact, most of those key informants did not even know about the app or website. CSOs and journalists likewise did not say they used the app and social media accounts. Informants’ views on this aspect are similar to their opinion regarding civil society participation in the budget process. KIIs said that there is real interest in the population to monitor budget execution but lots of impediments remain that prevent that from materializing. The main element is the lack of access to information about budget execution. According to some stakeholders, the Ministry of Finance learned to communicate with the public and publish data, and promised to launch a website modeled on the Activity’s tools. However, impediments remained to public monitoring of budget execution and budgetary management. Key informants faulted the CSCCA (Court of Auditors or Cour Supérieure des Comptes et du Contentieux Administratif) and Parliament for not fulfilling their responsibility to monitor the budget execution. 5 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV CONCLUSIONS The Activity succeeded in triggering interests from some segments of the population and civil society in the monitoring and analysis of budget execution and management and, to that end, the importance of social audits. The Activity seems to have also sensitized the Ministry of Finance on the need to publish data in a form allowing public monitoring of budgets. However, the lack of access to timely information about budget execution published in a simple, clear, yet detailed format still stands in the way of effective public monitoring. The tools the Activity deployed to address these issues, while they did meet their Life of Project (LOP) minimum performance indicators, were not widely adopted by stakeholders and the public. The tools suffered from lack of user-friendliness, promotion, and shareability of contents, and have not been updated with newer information about the budget since mid-2019. RECOMMENDATIONS 1. To continue raising awareness about fiscal transparency, USAID should continue to support interventions to increase public education and awareness on the budget, especially by expanding information to more segments of Haitian society, like underserved populations, those outside Port-au-Prince, and youth, a group that has been instrumental in the “Petrochallenge” fiscal accountability movement. 2. To facilitate Observatoire becoming fully operational, USAID should continue to sponsor Observatoire until it becomes functional and reaches maturity. Observatoire can become an instrument to empower other CSOs to participate in budget advocacy and fiscal accountability. Future assistance should particularly focus on helping the organization attract member CSOs who have the strength and resources to support Observatoire financially. 3. To expand AHJEDD’s reach throughout Haiti to increase media coverage on the budget, USAID should continue to partner with AHJEDD to train more journalists from all regions of the country to report on the budget and public finances. 4. To help Haitian civil society with budget monitoring and budget analysis, USAID should consider providing support to (re)designing the civil society online portal and social media tools to disseminate information on the budget in ways that are detailed, yet clear, simple and easily understood by non-experts. In that regard, we advise that the redesigned tools also publish content in Creole and with infographics that are easily sharable on different mediums. 5. To address obstacles to fiscal accountability due to structural issues and weaknesses in Haiti’s public financial management system, USAID should consider supporting programs to improve the GOH’s public financial management and its scores under the 28 indicators of the Public Expenditure and Financial Accountability (PEFA) framework. To address the apparent absence of fiscal transparency at the local government level, where municipalities do not even have a budget, USAID should consider developing a budget transparency program with some municipalities (that have a minimum level of capabilities) and provide help on financial management systems. This program could include Observatoire by developing civil society budget watchdogs at the local level as chapters of the Observatoire. 6 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV I. INTRODUCTION The United States Agency for International Development (USAID) requested that Social Impact, Inc. (SI)’s Haiti Evaluation and Survey Services (ESS) Activity design and implement the final Performance Evaluation of the Increased Accountability in Haiti Budget Process (IAHBP) Activity, also known as Group Croissance (GC), implemented by Group Croissance, S.A., under a cooperative agreement (see Annex A for the evaluation Statement of Work). Prior to fieldwork, the evaluation team (ET) reported its initial findings based on a desk review in an Inception Report followed by an Evaluation Design (both of which USAID approved before data collection). This report presents the ET’s findings, conclusions, and recommendations. ACTIVITY BACKGROUND Fiscal transparency refers to informing the public about how governments raise, spend, and manage resources, often through a series of documents. Fiscal transparency allows and fosters greater government accountability by providing a window through which citizens can monitor government budgets and spending. It helps citizens hold their leaders accountable and facilitates public debate. Fiscal transparency is critical to effective public financial management, builds market confidence, and underpins economic sustainability. Improving fiscal transparency has clear economic and development benefits. Since 2008, the U.S. Department of State (DOS) has conducted fiscal transparency reviews of governments receiving bilateral assistance funded under the Department of State, Foreign Operations, and Related Programs Appropriations Acts. In DOS’s 2016 Fiscal Transparency Report, Haiti did not meet the minimum requirements. Moreover, DOS found no significant progress toward fiscal transparency.6 The ET looked at results of a recent “Public Expenditure and Financial Accountability (PEFA)” report on Haiti. PEFA7 is a World Bank and IMF framework to evaluate the extent to which a country’s budget process is rigorous and allows for transparency, public participation, and accountability. On PEFA’s website, the ET found the last assessment was conducted in 2011 and published in 2012.8 On 28 indicators, “Except for one satisfactory (…) all others are non-satisfactory with a strong majority of D/D+ and some C/C+ scores” (report page 12).9 The U.S. DOS’s Fiscal Transparency Report (June 15, 2020) made similar statements.10 To address this issue, in July 2014, USAID signed a cooperative agreement with Group Croissance S.A. to implement the Civil Society Participation in Haiti’s Budgetary Process Activity, which included several interventions conducted between October 2014 and May 2016. USAID decided to build on the success of this first experience by continuing to support civil society in their efforts to increase government 6 Activity Contract, Attachment B Program Description, p. 13 7 Visit PEFA website for more about the framework: https://www.pefa.org/ 8 Available here: https://www.pefa.org/node/1811, captured April 14, 2021. 9 Scores in Report’s pages 112-114 10 Available here: 2020 Fiscal Transparency Report - United States Department of State, captured on April 14, 2021. The State Department however only presents its findings on its website without explaining its methodology on data collection. 7 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV transparency and accountability in financial matters. IAHBP is a follow up to this previous Activity and is specifically intended to increase citizens’ participation and engagement in Haiti’s national budget process. ACTIVITY DESCRIPTION IHABP was conceived to help Haitians improve their understanding of the mechanisms required to prepare a budget, analyze and evaluate this tool, make recommendations to the national government and Parliament to influence the budget preparation and execution process, and increase transparency through this process. The Activity’s implementation was intended to help find answers to some challenging questions, such as: What are the priorities the budget focuses on? How are the needs of different groups within society taken into account in the budget? What needs to be done to increase transparency in the budgetary process? How can civil society organization (CSO) members help in the national budget preparation and execution process?11 TABLE 1. ACTIVITY INFORMATION SUMMARY ACTIVITY NAME INCREASED ACCOUNTABILITY IN HAITI’S BUDGETARY PROCESS/ Implementer Group Croissance, S.A. Cooperative Agreement/Contract # AID-521-A-17-00002 Total Estimated Ceiling of the Evaluated Activity (TEC) $700,000.00 Life of Activity 1/19/2017– 1/18/2019 Active Geographic Regions All geographic regions Development Objective Support for the program “Increased Accountability in Haiti’s Budgetary Process” USAID Office USAID/Haiti Democracy and Governance Office IAHBP DEVELOPMENT HYPOTHESIS IAHBP’s development hypothesis was based on the economic and development benefits that fiscal transparency provides as a critical condition for market confidence and economic sustainability. The Activity posited that if citizens and, particularly, members of CSOs, understand and are aware of the national budget’s components and start questioning the Government of Haiti (GOH) about budget allocation and execution, this would lead to greater GOH accountability within Haitian society and, therefore, contribute to the country’s fiscal transparency. RESULT FRAMEWORK IAHBP objectives and expected interventions are summarized in the following Results Framework: 11 Ibid, page 14 8 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV FIGURE 1. IAHBP RESULTS FRAMEWORK EVALUATION PURPOSE AND AUDIENCE The purpose of this performance evaluation is to guide future USAID/Haiti programming in areas such as fiscal transparency, government budgets and spending, leadership, financial management, and other aspects of good governance and government accountability. This was based on an in-depth investigation and analysis documenting project performance, results, impacts, lessons learned, and best practices from various stakeholders’ perspectives. In addition to USAID/Haiti, this evaluation’s findings, conclusions, and recommendations are also relevant to Group Croissance and other CSOs interested in advancing budget transparency in Haiti. EVALUATION QUESTIONS This final performance evaluation will answer the evaluation questions (EQs) below. In answering these EQs, the ET considered stakeholder satisfaction, unintended results, and lessons learned. 1. To what extent did the Activity improve civil society participation in Haiti’s budgetary process? 2. How did the Activity contribute to improving public monitoring and analysis of budget execution and budgetary management? II. EVALUATION DESIGN Consistent with the SOW and resource constraints, the ET used a qualitative approach for the evaluation design. The ET first conducted a desk review that facilitated understanding the Activity’s design and implementation, extracting preliminary findings relevant to the EQs and identifying Objective: Increase accountability in Haiti’s budgetary process Intermediate Result 1: Public consultation takes place and broad-based civic input is encouraged during the budget preparation and formulation stage An observatoire on the national budget is created A network of budget-sensitive journalists is launched Advocacy events are organized by the Observatoire Ensure the Observatoire is financially sustainable Intermediate Result 2: Public participation is increased in the budget process as people are educated through awareness activities CSOs are identified, mobilized, and strengthened on budget process Awareness about social audits is increased An Open Budget Index is created for Haiti Intermediate Result 3: The population is aware about public monitoring and analysis of budget execution and budgetary impact An app on the national budget is created Expand the app utilization Content dissemination is made through social media 9 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV information gaps to address through primary data collection. Then the ET developed an Evaluation Design describing fieldwork data collection methods, which consisted of key informant interviews (KIIs) and focus group discussions (FGDs). The purpose of the fieldwork data collection methods were to: 1) supplement and triangulate the findings from the desk review as well as address gaps identified during the desk review to ensure that the evaluation appropriately and sufficiently addresses all key issues, to the extent possible; 2) obtain the most important and relevant information possible from informants; 3) organize information in a form that could be usefully and efficiently analyzed; and 4) ensure, to the extent relevant and possible, that data collected were disaggregated by stakeholder groups, sex, geographic regions, etc. The protocols, which are found in Annex D, consisted of questions that address and derive from the EQs, as well as from the ET’s document review, and initial discussions with USAID. The ET conducted KIIs and FGDs from December 22, 2020 to February 26, 2021. DATA COLLECTION METHODS DESK REVIEW The desk review consisted of document review (DR) followed by an Internet search to identify preliminary findings and gaps. The review included the following categories of documents (see Annex C for a complete list): • Activity contract, description, and modifications • Monitoring, Evaluation, and Learning (MEL) plan • Activity work plans and reports • Media reports (online) The desk review also included Internet searches to assess media coverage of Activity interventions and review relevant websites and media reports, as well as the Activity’s mobile application and social media accounts. The ET used specialized tools to assess the number of downloads and extent of the Activity’s mobile application and social media accounts utilization. KEY INFORMANT INTERVIEWS The ET conducted 48 KIIs, out of 57 planned, with various stakeholder groups. Although some GOH officials, from the Superior Court of Accounts (CSCCA) and the Office of the Prime Minister (Primature), refused to participate, the sampling breadth and diversity was sufficient to ensure the ET gathered accurate, relevant, and triangulated information. Also, most KIIs were conducted in Creole, with a minority of informants choosing to speak in French or switching between Creole and French; just one KII was in English. KII findings contributed to the ET’s responses to all EQs. Table 2 shows the distribution of KIIs by stakeholder group and sex. 10 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Table 2: KIIs by Stakeholder Group and Sex FOCUS GROUP DISCUSSIONS The ET conducted six FGDs, as planned. However, the ET had to replace the planned FGD with followers of IHABP’s social media accounts (Facebook Page, YouTube Channel, and Twitter feed) with an additional FGD with the Activity’s training participants, because it was difficult to contact followers of social media accounts. After Group Croissance provided administrator status to those accounts, the ET sent an invitation through Facebook messenger to more than 18 followers chosen randomly, in the order their name appeared in the followers list. Unfortunately, after an initial schedule where only three briefly connected via Zoom and then abandoned the call due to a poor Internet connection, the ET did not have another opportunity to reconnect with these stakeholders during fieldwork timeline. FGDs were mostly conducted in Creole, with some participants choosing to speak in French or switching between Creole and French. TABLE 3: FGDS BY STAKEHOLDER GROUP AND SEX STAKEHOLDER FEMALE MALE Observatoire Members 4 4 AHJEDD Journalists 0 6 Youth Activists (Petrochallengers) 1 3 University Students 1 3 STAKEHOLDER GROUP FEMALES MALES INTERVIEWS IP Staff (including 1 KI from App Developer) 3 2 5 USAID/Haiti 0 3 3 CSOs 2 6 8 MEF 0 3 3 Ministry of Planning 0 1 1 BRH 1 2 3 Parliamentarians 1 2 3 Ministry of Health 0 2 2 MCDF 3 0 3 Academic & Professionals 2 3 5 Mayors 1 5 6 Political Leaders 1 5 6 TOTAL 14 34 48 11 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV STAKEHOLDER FEMALE MALE Activity’s Female Training Participants 8 0 Activity’s Trainings’ Participants (Male & Female) 2 5 Total 16 21 INFORMANT SAMPLING ET used findings and gaps identified in the document review as the basis for selecting stakeholders for KIIs and FGDs. The ET emphasized selecting informants from all stakeholder groups the Activity touched, as identified in reports and lists of intervention participants, beneficiaries, and partners. The ET selected participants purposefully for the KIIs and two FGDs and randomly for the other FGDs. ET used purposeful selection for interventions that the Activity conducted with specific organizations and representatives of specific stakeholder groups. Purposeful selection allowed the ET to gather information and the perspectives of informants who are directly concerned with a specific issue. The ET deliberately selected informants for KIIs from IAHBP and USAID, CSOs that IAHBP specifically targeted, and GOH officials, including Members of Parliament (MPs). FGDs purposefully included journalists and members of the CSO Observatoire (presented in the findings section). In a few cases, the ET resorted to snowball sampling to replace informants who were unavailable or refused to participate. The ET randomly selected informants in all other cases. For example, for the four other FGDs, informants were drawn from a large pool of potential informants to obtain as broad and diverse perspectives as possible on the Activity’s achievements and challenges, as well as the sustainability of potential outcomes. For random sampling, the ET used an Excel list of nearly 3,900 training participants, which was first randomized using the Excel “Random” function. Afterwards, the list was sorted by stakeholder group. The team selected the first individuals on the resulting list. For the all-female FGD, the ET contacted the first 15 women appearing on the randomized list. This mixed use of purposeful and random sampling allowed the ET to ensure triangulated perspectives about achievements, challenges, outcomes, and lessons learned. DATA ANALYSIS The ET’s systematic and rigorous data analysis approach utilized triangulation to cross-check results, applying the data processes and analysis methods explained below to obtain the evidence that forms the basis for the evaluation’s findings and conclusions. DATA PROCESSES The ET took detailed notes in all KIIs and FGDs and cleaned and shared detailed electronic summaries on a rolling basis throughout fieldwork. Team members conducted regular internal debriefs to discuss progress and any needed adjustments in the evaluation schedule. The team also discussed evidence collected to help answer the evaluation EQs and to identify any discrepancies and emerging patterns and themes that would facilitate developing the subsequent analysis coding scheme. 12 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV DATA ANALYSIS METHODS The ET employed several data analysis methods to identify key findings from the collected data, draw conclusions, and make recommendations for Activity follow-up or potential future programming. Specific analysis methods depended on the data assessed: various qualitative analyses for KII and FGD data; and limited, descriptive statistics for quantitative data (mostly for IAHBP’s social media account followers). For qualitative data, analysis methods included the following: 1. Content Analysis: Content analysis entailed the ET’s intensive review and systematic coding of KII and FGD notes to identify and highlight key themes, and their frequency, within the data. These standardized data were then summarized to better understand the contributions of the Activity’s interventions to its objectives. 2. Trend Analysis: In a few instances, the ET examined different Activity indicators over time to identify patterns of convergence (or divergence) in outputs and outcomes toward the stated objectives. 3. Gap Analysis: The ET analyzed which aspects of the Activity, if any, fell short of anticipated performance, and the likely factors contributing to these gaps. 4. Comparative analysis: The ET analyzed how the informants’ views compared within and across stakeholder groups for purposes of data triangulation. 5. Gender Analysis: All data collected through KIIs and FGDs were disaggregated by sex and analyzed to assess the extent of difference of perspectives among male and female informants, relative to their percentage in the sample. As part of its data analysis process, the ET used a tally sheet matrix based on themes that arose from KIIs and FGDs. The ET tallied the frequency at which each theme occurred by stakeholder group and sex. This enabled the ET to identify trends across stakeholder groups by subgroups and/or sex. The team captured findings, conclusions, and recommendations (FCR) in an Excel-based matrix to categorize findings by EQ. The matrix: 1) ensured that the ET prepared a systematic and thorough response to each EQ; 2) verified that the analysis accounted for sex and social dimensions; 3) identified any gaps where additional clarification or analysis may be necessary; and 4) served as the basis for developing the evaluation report. LIMITATIONS AND MITIGATION STRATEGIES This section highlights possible bias and limitations in data collection and analysis, their implications for the evaluation, and how the ET tried to mitigate them. BIASES 1. Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. The ET 13 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV mitigated theses risk by: 1) reading the evaluation consent before each KII/FGD, which minimized, to the extent possible, the connection between the evaluation, the Activity, and any future opportunities for support, and 2) triangulating informant statements within and across stakeholder groups and with secondary data sources when available. 2. Selection Bias: Selection bias is an inherent risk when implementers help to facilitate contact with Activity recipients. There was a risk that the ET would only reach the most active, responsive, or engaged stakeholders. The ET’s sampling approach mitigated this risk. 3. Gender Bias: Gender bias is a risk because most individuals have a subconscious sense of appropriate roles and behavior for women and men. The ET conducted KIIs with enough female informants, including a female-only FGD (made up of the Activity’s public education participants selected countrywide) to ensure female participants were comfortable expressing their views. The significant imbalance in sex representation in most stakeholder groups was reflected in the sampling. As a result, gender analysis would not have been accurate for most stakeholder groups as female informants were underrepresented and/or were not informed enough to answer questions about some of IHABP’s interventions. Also, the majority (16 out of 30) of female informants were in FGDs. As data from mixed sex FGDs could not be disaggregated by sex, FGDs were not included in gender analysis, except for the female-only FGD. OTHER LIMITATIONS Reticence of government officials to participate. GOH officials from the CSCCA and the Office of the Prime Minister (Primature) refused to participate. However, the sampling breadth and diversity was sufficient to ensure accurate, relevant, and triangulated information from the GOH. COVID-19 pandemic technology limitations. Because of the pandemic, all data collection activities were conducted remotely through conferencing platforms such as Google Meet, Microsoft Teams, or Zoom, or for some KIIs via regular phone calls or WhatsApp. Limited Internet and technology access by some FGD participants posed challenges for the ET. Some FGD participants had poor Internet connections that caused them to be constantly disconnected. In those cases, the ET connected one or two participants through a regular phone call and put them on speakerphone, which allowed them to hear other FGD participants and vice versa. Imbalance in geographic distribution of informants. The fact that the vast majority of informants in all stakeholder groups, except for local authorities, are Port-au-Prince-based posed a challenge for geographic analysis. For example, all CSO leaders targeted for budget advocacy, as well as the budget￾sensitized journalists, are from Port-au-Prince. Therefore, the perspectives of those informants come from their experience in the greater Port-au-Prince area. As a result, the ET had a limited opportunity to obtain the perspectives of stakeholders from outside the capital. Limited possibility to triangulate information for some interventions. During KIIs and FGDs, the ET realized a majority of informants had limited to no information on certain key interventions. Members of four FGDs expressed opinions about the Activity in general, and not on specific interventions. Also, for discussions about the background of some interventions, only IAHBP informants or a few other stakeholder groups could provide information. The report highlights where this is an issue. 14 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Difficulties separating some elements from non-Activity-related circumstances. While the ET found some additional elements as evidence to compare with informants’ statements, they had difficulty isolating elements the evaluation identified from other social or political circumstances or non-Activity￾related interventions. ET limits the extent of the evaluation conclusions. III. FINDINGS AND CONCLUSIONS This section presents the evaluation’s main findings and conclusions by EQ. EQ1: TO WHAT EXTENT DID THE ACTIVITY IMPROVE CIVIL SOCIETY PARTICIPATION IN HAITI’S BUDGETARY PROCESS? In answering this question, the ET reviewed IAHBP’s results framework and identified the Activity’s interventions to improve civil society participation in the GOH budget process. Then, using various data collection methods, the ET assessed the extent to which those interventions actually improved civil society participation and identified factors that may have hindered or limited progress. To improve civil society participation in the budget process, IAHBP conducted interventions such as launching a network of budget-sensitized journalists, creating a civil society budget watchdog, training CSOs about budget advocacy, and conducting civic education and awareness raising about the budget. FINDINGS NETWORK OF BUDGET-SENSITIZED JOURNALISTS IAHBP facilitated the creation of a network of journalists called the Haitian Association of Economic Journalists for Sustainable Dévelopment (Association Haitienne des Journalistes Economique pour le Developpement Durable or AHJEDD), specialized in reporting on the GoH budget and public finances, IAHBP founded the association based on an existing group of journalists trained by the Central Bank of Haiti (Banque de la République d’Haiti or BRH) on budget and economic reporting. On January 18, 2018 IAHBP spearheaded AHJEDD’s formal launch and provided logistical and financial support for its legal registration. AHJEDD started with 17 journalists, of which four were women, all from Port-au-Prince media outlets. AHJEDD’s main objective is “to make economic and financial data more accessible by treating socio-economic topics objectively, and reporting on whether the GOH respects the budget cycle and is meeting its commitments under the Millennium Development goal.” 12 Established Presence and Growing Notoriety AHJEDD has quickly established its presence. On May 5, 2018, less than four months after its creation, it elected a governing committee of seven members for a two-year mandate. In January 2020, the association welcomed 21 additional members, 13 and on December 12, 2020, AHJEDD held its second election, reelecting its governing committee.14 The ET performed Internet searches with AHJEDD’s name and obtained different news articles from a large variety of media outlets. Interviews with 23 informants (IAHBP, USAID, economist [Econ], four GOH, four CSO, two political party [PP]), as well as 12 https://ahjedd.org/ahjedd/communique-de-presse/communique-de-presse/, accessed May 12, 2020 13 https://ahjedd.org/ahjedd/portfolio-view/formation-pour-les-nouveaux-membres-de-lahjedd-remise-des-certificats-jour-7/ 14 https://vantbefinfo.com/un-nouveau-comite-pour-lassociation-haitienne-de-journalistes-economiques-pour-le-developpement￾durable/, accessed May 12, 2020. 15 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV two FGDs (Observatoire [OBS] and AHJEDD) showed that all these stakeholders know of AHJEDD and/or had participated in some of its events. IAHBP leveraged AHJEDD to train journalists throughout the country on reporting on the economy and the budget, which the organization continues to do. The ET reviewed attendance lists for those trainings, which showed the Activity significantly exceeded its AMEL Plan target of 25 journalists over the life of project (LOP) by training a total of 150 journalists (of which 15 were women). Further, the AHJEDD FGD and content review of AHJEDD’s website revealed that the association conducted a seven-day training on budget and economic reporting15 for its new members. That fulfilled one of IHABP’s mandates, that “Group Croissance will conduct some ‘train the trainers’ sessions on the National Budget. Those people who are trained on the National Budget will be able to help spreading information contents about the budget and reaching more people.” 16 Medium of Engagement Between GOH and Civil Society AHJEDD also facilitated engagement between civil society and the GOH about the budget. All KIIs from IAHBP and USAID as well as the AHJEDD and Observatoire FGDs, reported that AHJEDD has, since its inception, been sought-after by Ministry of Finance officials as a medium that officials can use to communicate with the public about the budget. A USAID informant reflected that: “Ministry of Finance officials wants to leverage AHJEDD to get positive media coverage and look good.” Contribution to Increased Media Coverage About the Budget AHJEDD has indeed become a medium for budget awareness. Today, AHJEDD co-publishes, in collaboration with Group Croissance and Papyrus, 17 under an unwritten “gentlemen’s” agreement (according to an IAHBP informant and AHJEDD FGD), an online periodical titled DevHaiti centered on Haiti’s economy and national budget. All issues are available on AHJEDD’s website, 18 and distributed on various WhatsApp groups, which the ET was able to confirm. According to AHJEDD journalists, DevHaiti targets educated professionals. The ET’s review of the existing 23 issues (Number 23 published on April 9, 2021) showed that contents focus mostly on in-depth analysis and perspectives on Haitian economy, at large, by AHJEDD journalists and/or different guest contributors from civil society and even the GOH. The last section of most publications presents summary data about the budget (revenues and spending) in the form of tables and graphs and other data on the Haitian economy. Unfortunately, the ET did not have the opportunity or the tools and resources to assess DevHaiti’s readership. In general, a strong plurality (at least 15 of 47) of KIIs across all stakeholder groups credited IHABP for increased media coverage about the budget, in both quantity and quality. Some of these informants noted the example that previously media coverage and programming mostly covered politics and sports, whereas the economy and budget was only covered by a handful of media outlets, but now many more media include the economy and budget in their programming. Most respondent groups, especially from AHJEDD, economists, GOH, MPs, CSOs, local authorities, and PPs, also noted that journalists talk in a more informed and educated manner about budget topics, they know where to find information, and/or 15 https://ahjedd.org/ahjedd/portfolio-view/formation-pour-les-nouveaux-membres-de-lahjedd-remise-des-certificats-jour-7/ 16 Activity Program Description, p. 20. 17 A USAID implementing partner in Haiti. 18 Available here: https://ahjedd.org/ahjedd/devhaiti/, accessed April 26, 2021. 16 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV are now able to ask more pointed yet professional questions on all aspects of the budget instead of just being a “sounding board” (“caisse de resonance”) for interviewees. The ET used various techniques to assess the extent of increased media coverage about the budget including conducting many Internet searches combining “Haiti budget” with the years “2017–2018,” “2018–2019,” and “2019–2020,” plus search words (in French) like “critics,” “evaluated,” “opinion,” etc. The ET performed the same searches combining these terms with the names of popular Haitian media outlets (e.g., Le Nouvelliste, Le National, Radio Metropole, Radio Vision 2000, Radio Caraibes, RadioTélé Pacific, Radio Kiskeya, Radio Magic 9). The various searches revealed stories from multiple Haitian media outlets that, in total, vastly exceeded the 25-publication LOP target. While the ET did not have the resources, tools, and other secondary data sources to assess media coverage more broadly, especially for media outlets outside the greater Port-au-Prince metropolitan area, by counting media reports on IHABP’s interventions or Activity-related entities like AHJEDD and DevHaiti, the ET found the same result, that the Activity related media coverage surpassed the LOP target of 25. OBSERVATOIRE IAHBP created a civil society entity to serve as a budget watchdog on Haitian public finance (Observatoire de la Société Civile sur les Finances Publiques Haïtiennes, commonly referred to as Observatoire). On January 8, 2018, at the same event in which the Activity launched AHJEDD, IAHBP also presented Observatoire. 19 Observatoire was created with a core group of individual professionals from different sectors. Its mission is to increase civil society participation in the budget process through advocacy events and other types of interventions. Observatoire’s Potential for Civil Society Mobilization After its creation, Observatoire made headway mobilizing CSOs and individuals around the budget. Fifteen informants (four IAHBP, three USAID, three economists) and the AHJEDD and Observatoire FGDs showed Observatoire succeeded in mobilizing a group of at least 30 professionals from different civil society sectors for budget advocacy. During the Observatoire FGD, about half the participants presented themselves as representatives of a CSO. The same Observatoire FGD also revealed that the members are knowledgeable about the budget process, with some members crediting IHABP’s interventions for improving their understanding about the budget and its importance. A Port-au-Prince based professional explained: “By participating in Observatoire’s activities, I have a better understanding of the budget process and the importance of the budget.” KIIs, FGDs, and Internet searches also showed that Observatoire used organized public events and communications, especially by partnering with AHJEDD, to mobilize civil society about the budget. The AHJEDD FGD mentioned Observatoire as co-organizer of multiple events regarding the budget process. One such event was a conference on the importance of good economic and financial governance, co￾organized by AHJEDD and Observatoire, on the occasion of the first anniversary of the two organizations’ creation.20 In this event, the two organizations convened CSOs, economists, and GOH officials, among others, to discuss budget transparency, allocations, and impact. The ET read in AHJEDD’s publication, DevHaiti, issue No. 22 (February 22, 2021), a column penned by two 19 Available at https://lenouvelliste.com/public/index.php/article/182110/deux-nouveaux-organismes-pour-promouvoir￾linformation-economique 20 https://ahjedd.org/ahjedd/qui-sommes-nous/ 17 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Observatoire members about the organization’s inputs to the GOH during the formulation of the post￾COVID special budget (Programme de Relancement Post Covid or PREPOC). Additionally, the ET found a press release dated March 28, 2019 in the newspapers Le National21 and Le Nouvelliste, 22 as well as on IHABP’s Facebook page and Twitter account, where Observatoire expressed concern about the lack of voting on a budget more than six months into FY 2018–2019. Observatoire as vis-à-vis for Ministry of Finance (MEF) Observatoire has become a trusted GOH partner for civil society consultation. The Observatoire and AHJEDD FGDs, as well as 19 KIIs (four GOH, five IP, three economists, three USAID, two MPs, two CSO), showed that the GOH, especially MEF, regularly engages Observatoire during budget formulation stages. Those KIIs and FGDs specifically indicated multiple occasions where MEF officials invited Observatoire to provide their input during the budget formulation stage, in particular for the FY2018- 2019, FY2019-2020, 23 and FY 2020-2021 budgets, as well as the previously mentioned PREPOC, which the Executive Branch adopted by decree in Parliament’s absence. An Observatoire member affirmed: “MEF always communicates to us the budget orientation letter once it is issued.” Observatoire’s Persistent Limitations IHABP’s results framework anticipated that Observatoire would be autonomous, financially sustainable, and able to organize budget advocacy events on its own. However, Observatoire still faces many challenges. Weak CSOs Involvement Observatoire suffered setbacks in structuring its original membership that still remain. IHABP originally envisioned Observatoire would be a consortium of various CSOs, 24 but KIIs with IAHBP staff explained that the Activity had to switch strategy and launch Observatoire with a small core group of professionals from different sectors after IAHBP encountered difficulties organizing it as initially planned. IAHBP informants, as well as the Observatoire FGD revealed that the difficulties ranged from CSO leaders not attending planned meetings and/or sending different people to each meeting, which posed problems of continuity and progress in organizing. During the Observatoire FGD, while some members presented themselves as representatives of a CSO, they did not really explain the extent to which their respective parent CSO was really involved with Observatoire. For example, KIIs with CSO leaders, except for one, showed their respective organizations were not actively involved in Observatoire. Legal Registration Unadvanced Observatoire’s legal registration and formalization remains uncertain. Most members indicated during the FGD that they have not read the draft articles of incorporation, bylaws, mission statement, etc. Nor are they aware of any progress toward registration. The two members that appeared the most active did not give a timeline for the completion of the registration. Members also indicated the existence of a provisional committee, however, apart from the two most active members, its composition was not 21http://lenational.org/post_free.php?elif=1_CONTENUE%2Feconomies&rebmun=2204&fbclid=IwAR0UjaEsmXonCzgO9fQ_V5 Ls2QNNrxVqdOShiZIs2_OhEuZJb2fsRJb5fWg, last retrieved October 1, 2020 22 https://lenouvelliste.com/article/199797/loscfp-consterne-par-labsence-de-budget-2018-2019 23 Parliament returned the proposed budget to Executive branch and adjourned and become non-operational on January 2019 after the terms of the lower house and one third of the Senate ended without elections taking place. 24 Activity FY 2018 Annual report, p. 18 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV indicated. Thus far, Observatoire still operates as a rather informal association that has not yet been legally registered. Nonexistent Financing Strategy The question about Observatoire’s sustainable financing remains unaddressed. The same Observatoire FGD, as well as KIIs with IAHBP and USAID, showed that no plan exists to identify a financing strategy and mechanism for the organization. IAHBP did not allocate funding for Observatoire. Rather, IHABP thought “smart resources mobilization strategies could help do more with less in this project.” 25 The Activity prescribed that: “among the partners and allies, Group Croissance will work with the networks of public universities and key organizations of the civil society.” However, all IAHBP and USAID informants, as well as three economists, and the AHJEDD FGD noted that Observatoire has suffered from or inherited the same funding issues plaguing most Haitian CSOs. The ET asked Observatoire members how they envision solving the financing issue. The main strategy proposed was to ask member CSOs to include Observatoire’s financing in their budget programming. However, it appeared that proposal remained just an idea, as members indicated no practical steps to implement it. Moreover, IAHBP informants and the Observatoire FGD explained that Observatoire relies on Group Croissance for office space to conduct meetings, and some Observatoire members cover its expenses on an as￾needed basis. Finally, it was not clear how Observatoire plans to conduct budget advocacy in the short term. As IAHBP and USAID informants explained, providing inputs on the budget process requires research, analysis, and report writing, among other things. However, as Observatoire has no funding to recruit capable paid staff to perform those tasks, the alternative would be to devise a strategy to use a third￾party to provide such work for free. During the desk review, the ET noted IAHBP universities and academics involved in teaching public finances, who could provide free technical capabilities for budget analysis, were notably absent from the organization’s membership. When presented with that observation, an IAHBP informant explained the lack of culture of research, publication, and advocacy on the part of Haitian universities and academics makes that difficult. AWARENESS RAISING IN THE POPULATION Civic Education and Awareness Raising26 Interventions IHABP conducted numerous interventions throughout the country to increase civic education and raise awareness about the budget. Those interventions included trainings, seminars, and presentations to different stakeholder groups within the population and covered all ten of Haiti’s geographic departments across a large and diverse breadth of sectors. Stakeholder groups reached included ordinary citizens in events open to the general public, CSO representatives, university students, leaders and active political party members, youth activists, MPs from both chambers, local authorities (Mayors as well as municipal section representatives), and even government officials and employees from different Executive Branch agencies, especially from regional offices. IHABP also placed advertisements in different Port-au-Prince media outlets (radio and TV) on the nature and importance of the budget and need for civil society 25 Program Description, Page 20 26 Activity terminology in Program Description. Result framework’s IR2 is: “Increase public participation in the budget process through public education and awareness activities. (Page 15, Attachment B – Program Description) 19 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV participation in the budget process. Activity reports showed that IAHBP provided budget-related training to at least 2,547 beneficiaries, considerably more than the LOP target of 1,625. Nearly all participants across all stakeholder groups reported satisfaction with IHABP’s public education and awareness interventions. Informants appreciated the quality and relevance of the trainings, as well as how those interventions have provided them a better understanding of the budget, its components, and other elements they did not really understand before. For example, a senior employee from a regional government office reported: “Before I thought the figures in the budget were money the government already had in its bank accounts. Thanks to the training, I came to understand the figures are only expected revenues from taxes and other sources.” Most participants believed IHABP succeeded in making more people interested and aware of the budget’s importance. An economist noted: “People are now able to make a link between exchange rate, inflation, and budget deficit.” However, the informants did not provide any specific data from other sources to back up their perception. Limitations of Civic Education A majority of informants, across all stakeholder groups, mentioned some limitations with the budget￾related civic education and awareness raising interventions IHABP conducted. These informants think that the format and medium of communication limited the reach and accessibility of IHABP’s budget￾related civic education interventions to ordinary people, especially the underserved majority of the population and those outside of Port-au-Prince. For example, a plurality of informants mentioned that most media outlets covering the budget are based in Port-au-Prince, and a majority of those are largely followed by middle class and educated people, which are a minority of the population. Identification and Mobilization of Key CSOs IAHBP had to identify, mobilize, and reinforce CSOs so they could be actively involved in budget advocacy. Indeed, the Activity envisioned: “In order to make this action sustainable, Group Croissance will work with their members so those civil society organizations could develop internal capacity to keep their advocacy on the national budget going when this project is over.” 27 The Activity training list showed that IHABP touched many CSOs, mostly Port-au-Prince based, through budget advocacy trainings. The Activity also provided more targeted trainings to a few CSOs. The FY18 Annual Report stated: “In West department only, eight (8) training sessions have been realized for CSOs such as the National Confederation of Haitian Educators (Confédération Nationale des Éducateurs Haïtiens or CNEH), the Medical Association of Haiti (Association Médicale Haïtienne or AMH), and university students…[to] sensitize those groups to advocate for a better budget allocation for health and education services in the next budget considering the urgent needs in those sectors.” A plurality of KIIs credited those interventions as leading more CSO leaders to talk about the budget in the media. However, CSO participation in the budget process remained limited. For example, despite the trainings provided, only CNEH, out of seven CSOs expected over the LOP, with IHABP’s assistance, made a written recommendation to Parliament for the FY 2017–2018 budget. IHABP’S IMPACT Regarding the extent to which IHABP improved public participation in the budget process, stakeholders distinguished between public interest about the budget and real participation in the budget process. 27 Ibid 20 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Noted Improvements A strong majority of KIIs (29 out of 47) across all stakeholder groups and all FGDs credited the Activity’s public education interventions for increased understanding and interest among the population about the budget. Those informants hold the view that regular citizens now better understand the importance of the budget and some budget concepts, like allocations, public revenues, deficit, etc. and more people are interested in hearing about the budget and economy and are eager to find out about budget allocations. One economist mentioned “they understand that the budget will have an influence on their quality of life through exchange rate and inflation.” Some informants gave examples of people in street markets and public transportation talking about the budget “to the point you will hear people sometimes say: this budget is not good” (CSO KII). Another CSO KII noted that, thanks to IHABP, “civil society has more key points and approach in the discussions on the budget.” A political party KII noted: “thanks to IHABP, the budget has been on the political agenda of almost everybody.” A few key informants (one IP, two USAID, one CSO, one GOH) went further and said that IHABP set the stage for increased civil society participation in the budget process, if not yet in a formal, organized way, at least through the media and public demonstrations. One example, most informants cited the widespread public demonstrations against the 2018-2019 budget that was deemed “criminal” by the population. 28 A USAID KII also noted, as an example of headway toward improvement in civil society participation, the fact that MEF now engages AHJEDD and Observatoire. IAHBP and USAID KIIs, as well as the AHJEDD FGD, indicated that current MEF officials now look to engage civil society and the media to “look good.” Persisting Limitations to Civil Society Participation On the other hand, a majority of KIIs (22 out of 47) from most stakeholder groups, and all FGDs, indicated that changes are still limited. In particular, they think increased public interest has not translated into actual participation in the budget process. KIIs made the point that government institutions and officials are still not really committed, open, and responsive to civil society participation in the budget process. One CSO leader stated bluntly that “there is no real citizen participation in budgetary process in Haiti.” This is the main aspect where female informants showed a rather sharp deviation from male informants: many more female informants than male were negative about possible evolution in terms of civil society participation in the budget process. GOH Responsiveness to CSOs’ Input CSOs construed “civil society participation” as meaning that the government is both open and responsive to their inputs. They expressed frustration that despite discussions or recommendations made to the Executive Branch or Parliament, their views were not considered. All CSO leaders interviewed voiced their frustration about the fact that despite IHABP facilitating exchanges and discussions with the Finance Commission of the Chamber of Deputies (Parliament’s Lower House) on the budget, Parliament just disregarded their opinion. The Observatoire FGD echoed this point, with one member noting that: “Observatoire is invited regularly by Ministry of Finance when the letter￾28 “Haïti: violente manifestation contre le budget” (“Haiti: violent demonstration against the budget”), Press dispatch by AFP, published in Journal de Montréal, September 12, 2017, and available at https://www.journaldemontreal.com/2017/09/12/haiti￾violente-manifestation-contre-le-budget. 21 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV circular29 is available, but that does not go any further.” A majority of MPs interviewed tended to share the view that civil society input was not incorporated into the budget. While most stakeholders agreed that there are a variety of factors limiting civil society participation in the budget process, these views tended to differ from that of GOH stakeholders. Some GOH officials interviewed raised the question about CSOs’ failure to understand that there is contradiction among different CSO groups with competing demands or even conflict of interest. Therefore, it is “normal” for each CSO to be frustrated that their input was not considered. One GOH official explained that, for example, for the health or education sectors, private operators are not interested in the government increasing funding for public schools or hospitals, whereas other groups are pushing for that. This GOH official voiced his frustration that at the same time, those CSOs failed to understand that the budget is financed through taxes, which those same groups do not want the government to raise. Other GOH officials agreed that tax revenues are not sufficient to satisfy the competing demands of different sectors. CSOs’ Lack of Technical Capabilities A large majority of informants faulted CSOs’ weaknesses as key impediments to budget advocacy. Indeed, 26 KIIs out of 48, across all stakeholder groups, and the AHJEDD and Observatoire FGDs opined that most CSOs cannot afford the technical staff to conduct budget advocacy. A CSO leader explained that during the last four years, the MEF has invited some CSOs to participate in consultation during the budget formulation stage, but those CSOs did not have the technical capacity to intervene on their own. Those informants indicated that most CSOs do not have enough financial resources as it is, and adding budget advocacy to their interventions would require them to hire additional paid experts. In most informants’ view, the funding issue will remain an impediment to CSOs’ capacity to intervene in the budget process. Some informants offered some possible solutions. A USAID informant pointed out that except for mostly private sector organizations like Chambers of Commerce and trade groups (Bank Association, Petroleum Distributors, etc.), most CSOs depend on international donors awarding grants for specific programs. Six KIIs (two IAHBP, one CSO leader, three PPs) and the AHJEDD and Observatoire FGDs indicated that most CSOs focus on politics or specific sectors like education, health, and human rights, rather than budget advocacy. To address that, five KIIs (two IAHBP, one USAID, one CSO, one GOH) and two FGDs (AHJEDD and Observatoire) indicated that better targeting specific CSOs with pre￾existing strengths and capabilities could have yielded more results in terms of CSO’s engaging in budget advocacy. Three KIIs (two IAHBP and one USAID) took AHJEDD’s success as evidence to support this view, and one IAHBP KII mentioned the Catholic church-linked JILAP (Justice and Peace) as an example of a CSO that could have been a better partner. For this informant, JILAP has more technically competent staff and a network of support through the Catholic Church. Issues Inherent to the Budget Process Stakeholders point to different structural issues in the GOH as impediments to civil society participation in the budget process. 29 Letter-circular sent by the Prime Minister to different government agencies on budget orientation during the budget formulation stage. Article 44 of the Law of May 4, 2016 on Budget Elaboration and Execution. 22 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV When asked about specific examples of the reasons for lack of improvement in civil society participation in the budget process, stakeholders indicated different structural issues making the budget process unamenable to civil society participation. First, at least nine KIIs (one IP, three CSOs, five PPs) and at least two FGDs (Observatoire and Female Participants) explained that Parliament is supposed to be the main forum for civil society participation in the budget process, but that Parliament played limited role in budget formulation, and MPs are not really vested into doing it anyway, if they understand it at all. A CSO leader remarked that: “Parliament has not conducted any public consultation on the budget since 2006!” On the other hand, a majority of KIIs expressed the view that the budget usually lacks “credibility” (“is not sincere”). At least 16 KIIs (one IP, three CSOs, four PPs, three GOH, five economists) and two FGDs (AHJEDD and Observatoire) supported that view. However, MEF officials tended to not agree that the budget, at least at the formulation stage, is not credible or is decided in a non-rigorous fashion without inputs from sectoral Ministries. The International Budget Partnership (IBP) states “A credibly implemented budget has only small deviations from the approved one both in terms of overall expenditure and revenues and in terms of the allocation between programs etc.” When asked to explain how and why the budget is not “credible,” informants offered the following explanations: • Sectoral ministries do not have enough senior employees, tools, and processes to formulate budget proposals for their agency. A GOH official noted for example that “Most Ministry’s Study and Programming Unit [Unité d’Étude et de Programmation] in charge of formulating budget proposal are understaffed.” • The budget is ultimately decided by a few MEF bureaucrats and is not based on sound policy and macro-economic analysis. A GOH official stated that: “I have never seen a serious exercise to do a cost analysis for policy proposal.” Another GOH reflected that: “Often the figures in the budget do not reflect the priorities announced by the government.” An economist remarked that: “Revenue streams and expenditures are in general unrealistic.” • A budget elaborated for one year, is meant to cover many multi-year programs. • An economist took the example of a trivial and odd event during the “investment budget” implementation (the part of the national budget allocated for public investments) to show how the budget is often unrealistic. Both the Ministry of Planning, which has to approve disbursements for this part of the budget, and MEF usually stonewall fund requests by different Ministries because the money is not available, resulting in disbursements starting very late in the fiscal year. Pilot Open Budget Survey IAHBP’s results framework mandated the creation of an Open Budget Index (OBI) for Haiti, which is an index of the level of fiscal transparency in a country. It provides a “score on a 100-point scale using a subset of questions that assesses the amount and timeliness of budget information that governments make publicly available in eight key budget documents in accordance with international good practice standards.” The Activity took steps to make headway toward Haiti’s inclusion in the OBI by reaching out to IBP, the institution that conducts the survey, to see how Haiti could be included. 30 However, IBP was 30 FY18 Report, p. 7-8 23 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV not willing to commit the resources needed, especially in the Haitian context, where reliable data is not available (two KIIs: IP and USAID). IAHBP used the OBI methodology to conduct a pilot survey for Haiti, which produced the following results: • In terms of transparency: Haiti provides the public with scant budget information. • In terms of public participation: Haiti provides few opportunities for the public to engage in the budget process; and • In terms of oversight: the legislature and supreme audit institution in Haiti provide limited oversight of the budget. IBP made a few recommendations to improve these three aspects, including to: • Publish the Executive Budget Proposal on the government website before it is enacted into law; • Ensure that the legislature holds a pre-budget debate and approves recommendations for the upcoming budget; and • Hold legislative hearings on the audit report, during which members of organizations can testify. Despite these efforts, an OBI has not been created for Haiti. IHABP did not have the resources or mandate to create it itself (without IBP). Stakeholders’ Recommendations Informants across all stakeholder groups offered the following recommendations for future civic education and awareness raising interventions: • Continue interventions with more contents in Creole, and in a simpler format. • Focus on the younger population, especially university and high school students. Female informants made this particular recommendation much more strongly than their male counterparts. • Include local CSOs in the national and municipal budget process. • Focus on helping municipalities adopt and manage annual budgets. In that regard, stakeholders recommended that trainings should target municipal employees and technocrats as opposed to the Mayors (who are temporary). • Help Haitian CSOs create an Index or an equivalent. CONCLUSIONS Evidence showed IAHBP contributed to improving understanding and interest among some segments of civil society to participate in the GOH budget process. The Activity also succeeded in establishing 24 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Observatoire to serve as a watchdog and catalyst for civil society participation in the budget process. In addition, IAHBP helped establish an association of budget-sensitized journalists, which has proven instrumental in increasing media coverage about the budget, expanding the number of journalists trained to report on the budget, and potentially eliciting more and better civil society participation in the budget process. However, many structural issues, outside IHABP’s SOW and resources, limit the possibilities for effective civil participation in the budget process. Those issues are related, on one hand, to government agencies’ structural weaknesses and Parliament’s limited role in the process and, on the other, CSOs’ lack of resources and technical capabilities, including those IHABP targeted. EQ2: HOW DID THE ACTIVITY CONTRIBUTE TO IMPROVING PUBLIC MONITORING AND ANALYSIS OF BUDGET EXECUTION AND MANAGEMENT? To answer this question, the ET assessed intervention outcomes to improve public access to information about budget execution and management. The ET also reviewed the extent to which those interventions may have influenced the GOH to provide timely and accurate information about the budget, its execution, and impact to the public. FINDINGS SOCIAL AUDITS IHAPB conducted interventions to increase public awareness about the need for social audits in Haiti. The Activity trained local authorities, political party leaders, and university students on performing social audits. 31 It showed participants the difference between financial and social audits and used different case studies, including how to use social audits to evaluate the Petrocaribe mechanism through which the Government of Venezuela issued certain countries, including Haiti, finance development programs.32 IAHBP also trained 27 students to conduct social audits. The FY2018 report stated that “The social audit has facilitated the participation of the youth in the budget process.” After different GOH agencies that the Activity contacted did not agree to participate in a social audit case study, IHABP launched a social audit contest instead. Sixteen students, all from the Ouest Regional Department, conducted social audits of various GOH projects after IHABP trained them. A plurality of KIIs indicated a link between IHABP’s social audit interventions and anti-corruption movements. Twelve of 47 KIIs (two IAHBP, one USAID, two CSOs, one MP, four PP, one GOH, one economist) and three FGDs (AHJEDD, youth activists, university students) think IHABP also influenced, to some degree, the “Petrochallenge” movement led by young Haitian activists demanding accountability on the use of funds Venezuela loaned to Haiti as part of Petrocaribe program. Some of those KIIs indicated that, in particular, IHABP’s training about social audits may have influenced some youth activists to use social audits as part of the Petrochallenge. 31 FY18 Annual Report, p. 13-16 32 For more on the Petrocaribe program and Haiti’s involvement, consult this Time Magazine article “Haiti Protests: What Is PetroCaribe and Why Is It Fueling Unrest?” https://time.com/5609054/haiti-protests-petrocaribe/, accessed May 12, 2021. 25 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV The ET performed Internet searches with the words “audit collectif Haiti” (“social audit Haiti”) to try to find any connection with the Activity. The search returned various media reports from both Haitian and international outlets showing social audits being performed as part of the movement. A September 16, 2018, article in the online newspaper, LoopHaiti, specifically discussed a campaign for a social audit led by the Petrochallengers: “Social audit: the second phase of #Petrocaribechallenge” 33 (“Audit collectif: la deuxième phase du #Petrocaribechallenge”). Some KIIs who participated in the Petrochallenge (youth and CSO leaders) credited the Activity with training them on social audits. The ET dug deeper into this aspect to find the extent to which IHAPB may have, indeed, contributed to this movement. However, three KIIs, who played direct roles in the movement, did not support a view that the Petrochallenge derived from IHAPB, though they agreed that the Activity’s budget-related civic education and awareness raising interventions served as an amplifying factor. The ET conducted further searches on the wider Petrochallenge to ascertain possible IHABP influence. The team searched for “Petrocaribe Haiti Rapport” in Google and used the tools’ search range to limit results to different years (2015, 2016, 2017) before and during the Activity. The results revealed that the Petrocaribe topic has been part of political discourse since at least 2015. For example, the ET found an article titled: “Haiti-Politique: Rapport Petrocaribe, le moment de vérité” 34 (“Haiti – Politics: Petrocaribe report, the moment of truth”). Further searches showed that the first investigative report on the use of the Petrocaribe fund originated from the Haitian Senate on August 20, 2016. The second Haitian Senate’s “Petrocaribe report” stated: “On August 20, 2016, the Ethics and Anti-Corruption Commission, created by the Senate of the Republic of Haiti in order to investigate the accusations and rumors related to wrongdoings denounced in the management of the Petrocaribe Program, submitted, by virtue of the mandate conferred upon it, its investigative report.” 35 Additional searching returned a piece36 from the online media outlet, AyiboPost, related to the origin of the Petrochallenge, launched by a young Haitian filmmaker, living abroad at the time. TOOLS TO FACILITATE PUBLIC BUDGET EXECUTION MONITORING IHABP developed and deployed certain tools to facilitate public monitoring and analysis of budget execution and budgetary management. The ET reviewed these tools and discussed the extent of their use and effectiveness with stakeholders. Mobile Application On November 12, 2017, IAHBP launched a mobile application (app) dedicated to providing the public with information about the budget and its execution. The app aimed to allow users to obtain information about the national budget, including allocations by entity and public investment projects by department and commune. It also provides information on published articles about the national budget. The app is available on both Google Play (Android) and Apple’s iOS platforms. 33 https://www.loophaiti.com/content/audit-collectif-la-deuxieme-phase-du-petrocaribechallenge, accessed September 6, 2021. 34 https://www.haitilibre.com/article-13289-haiti-politique-rapport-petrocaribe-le-moment-de-verite.html, accessed April 30, 2021. 35 Second Haitian Senate Report of the use of the Petrocaribe Fund, October 2017, Page 12, available at: https://www.scribd.com/document/364094383/HAITI-Rapport-Petro-Caribe-Octobre-2017, accessed April 30, 2021. Also referenced at: https://www.haitilibre.com/article-18361-haiti-flash-rapport-d-enquete-sur-l-utilisation-des-fonds-petrocaribe.html 36 Available at: https://ayibopost.com/un-an-apres-le-petrochallenge-est-encore-en-vie/, captured April 30, 2021. 26 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV The ET performed the following Internet searches: “Google Play budget Haiti,” 37 which led directly to the apps page (where one can download it) on the Google Play Store. That page provided a preview of the apps screens, as well as the following technical information about its Android version: ● designed by CompuConsult, which appeared on Google Play Store as the application’s owner ● last updated May 3, 2019 ● content rated “Everyone” (no age restrictions) ● installed by 500+ people, i.e., more than 500 but fewer than 1,000 people ● reviewed by nine users, who gave it more than four out of a possible five stars ET used an “app view” tool for the iOS (Apple) version of the application, which showed the following information: ● published by CompuConsult ● last updated January 29, 2019 Website In tandem with the app, there is also a full website, www.bidjepeyidayiti.ht, containing more features than the app and that was still in operation as of September 22, 2020. However, the website has not been updated since the last budget, and some information appeared to date from 2018. The FY18 Annual Report mentioned a total of 1,614 visitors to the website. The website appears to not have been updated during FY 2020. The ET also tried to subscribe to the website by entering different email addresses, but received the error message: “Error during email registration.” Infographics on the Budget in DevHaiti and AHJEDD’s Website The last section of each publication presents summary data in the forms of tables and graphs on the budget (revenues and spending) and other data on the Haitian economy. Unfortunately, the ET did not have the opportunity nor the tools and resources to assess DevHaiti’s readership and the extent to which they are tools used by various segments of civil society for actual monitoring of budget execution and budgetary management. Social Media Accounts (YouTube, Facebook, and Twitter) The ET conducted a content review of the different social media platforms to find information on this activity element. The team also conducted Internet searches, consulted secondary sources such as media reports, and reviewed IAHBP’s background documents. IAHBP created both a Facebook group38 and a Facebook page39 titled “Le Budget d’Haiti.” As of October 1, 2020, the Facebook group had 312 members. Any group member can make a new post on 37 The result can be found at https://play.google.com/store/apps/details?id=com.compuconsult.haitibudget&hl=en. 38 https://www.facebook.com/groups/836143253074190. 39 https://www.facebook.com/bidjepeyidayiti/. 27 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV its discussion board. While the page description states that “this group is a discussion board on the budget of the Republic of Haiti,” most posts are not related to the budget, but rather advertising other events and shows. The ET noticed that one member, who joined the group on June 6, 2020, tends to make budget or economic-related posts. On June 11, 2020, this member shared a budget-related article titled (in French) “Haiti-Budget: The Government is still not committed itself to rationalize budget choice.” The Facebook page was followed by 3,340 people as of September 30, 2020. The page includes two icons telling visitors that there is an application available on both Google Play and the iOS App Store. A link directs visitors to the Google Play page for the application. On the content side, as of October 1, 2020, the last post on the page was dated June 16, 2020, sharing a link to an issue of DevHaiti titled (in French) “Health, budget, environment, and the effects of COVID-19 in Haiti.” The ET clicked on the link, but found an error message: “Oops! This page does not exist. This page has been deleted, moved to a different URL, or the URL has been typed in incorrectly.” The previous post was dated March 28, 2019, and shared a Le National article on the press release by Observatoire regarding the absence of a budget more than six months into FY 2018–2019.40 The activity also launched a YouTube channel “Le Budget d’Haiti.” 41 To date, it has posted 12 videos and has a total of 27 permanent subscribers. Individual video viewership ranges from 5 to 558. A Twitter feed, @bidjepeyidayiti, has existed since November 2014 (before IAHBP started). As of September 22, 2020, it has 173 followers and has followed 12 other Twitter users. The last tweet is dated March 28, 2019, sharing the above-mentioned Le National article on Observatoire’s press release. Except for the June 16, 2020, budget-related post on the Facebook page, all social media accounts the ET visited appeared to have been inactive since mid-2019. The Activity’s AMEL Plan targeted 5,000 followers over the LOP for all online and social media tools (Mobile Application, YouTube, Facebook, and Twitter accounts). To assess whether it met this target, the ET calculated the number of active users indicated in reports for the website and app combined with the ET’s direct observation of the numbers of followers on social media accounts (YouTube, Facebook, Twitter). This estimate gave a total of 4,336,42 which fell short of the 5,000 followers targeted. However, the number would reach 5,950 by adding the 1,614 visitors to the website indicated in the FY18 Annual Report. That said, it is worth noting that this assessment method is not precise. For example, the ET cannot assess duplication of users across accounts. The AMEL Plan did not require this level of sophisticated technical inquiry. Furthermore, some people may regularly consult a Facebook page or Twitter account without actually “following” it. IAHBP also disseminated content on budget transparency via other mediums, including television ads focused on civic education, the budget’s importance, and five documentaries on 1) the importance of a 40http://lenational.org/post_free.php?elif=1_CONTENUE%2Feconomies&rebmun=2204&fbclid=IwAR0UjaEsmXonCzgO9fQ_V5 Ls2QNNrxVqdOShiZIs2_OhEuZJb2fsRJb5fWg, last retrieved October 1, 2020. 41 Available at https://www.youtube.com/channel/UCw9GXQ3T6ZIddX8hoWEYJBA. 42 796 (app users, FY18 Annual report) + 3340 (Facebook page followers) + 312 (Facebook Group members) + 27 (YouTube channel subscribers) +173 (Twitter followers). 28 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV national budget, 2) debt, 3) public procurement, 4) social audit, and 5) budget transparency. IAHBP only disseminated the documentary on the importance of a national budget, while “the others would be finalized during the month of November 2018.” 43 STAKEHOLDERS’ PERSPECTIVES ON IAHBP’S BUDGET EXECUTION MONITORING TOOLS All IP key informants (5), including the app developer, expressed mixed satisfaction with the app and tools. While they point to app and website monitoring data showing that the tools exceeded the target fixed during LOP, they indicated that they would have preferred that the tools were vastly more popular and adopted by the public. Two IP informants explained that the app triggered public interest during the first months after its launch, but afterwards interest fizzled due to lack of content updates. As indicated above, the ET’s review of the app and website content showed that, indeed, the contents were all related to the FY2017-2018 budget. The ET also discussed the mobile app and social media tools with different non-IP-related informants. None of those KIIs indicated using the app or IHAPB’s social media accounts as sources of information about the budget. In fact, most of these key informants did not even know about the app or website. However, the two FGDs (AHJEDD and Observatoire) revealed that some of these informants knew about the app and social media accounts, but they, too, indicated not using them as sources of information. Unfortunately, as noted above, ET did not obtain app and social media account users’ perspectives because an FGD with these stakeholders could not be organized. Informants’ Recommendations to Improve the Tools IP informants explained that future interventions deploying apps and social media tools will need input from communications experts. They think these experts in public communication, user interface design, social media, etc. could have helped in both the design of the app, social media content dissemination, and public promotion. These informants pointed out that the content format, etc. could have been more user friendly. They also noted that the app language (French instead of Creole) and relatively low smartphone penetration in Haiti are factors explaining the app and social media accounts’ low penetration and adoption. The ET reviewed IHABP’s staff list, as well as the contract, which showed that the Activity did not include budget provisions for the specialists mentioned by those KIIs. For example, the app developer explained that the development company had to register the app under its name because the IP did not have accounts with Google Play or the Apple Store. Likewise, the IP did not have in-house staff to monitor the app, manage contents, develop updates, etc. As a result, long after contract ended, the development company continued to provide support without additional compensation. Until IHABP ended, the app is still active and remained registered under the developer’s name but not updated. IAHBP’S IMPACT ON PUBLIC MONITORING OF BUDGET EXECUTION The ET discussed their perspectives on improvement, or lack thereof, in public monitoring and analysis of budget execution and budgetary impact with all stakeholder groups. 43 FY18 Annual Report. 29 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Informants’ views on this aspect were similar to their opinion regarding civil society participation in the budget process. A little less than half (22 of 47) of KIIs indicated that the population is really interested in monitoring budget execution, but that there many impediments that prevent actual participation from materializing. The main factor KIIs cited is the lack of access to information about budget execution. When the ET asked them to provide specific examples, 15 KIIs and two FGDs stated that the data on budget execution are not published in a timely manner and/or are not detailed and simplified to allow ordinary citizens to understand. A GOH official stated that “Ordinary citizens need a tutor” to understand those data. As an example, this GOH official said even economists tend to misconstrue the concept of “budget deficit,” Central Bank’s “advance,” or “overdraft” to Public Treasury, which is not automatically a deficit, since it may be paid back later during the fiscal year when the Treasury has collected more revenues. That example forms the basis for why information needs to be communicated in a clearer, more simple manner to the public. Influence on Budget Data Publishing by the GOH According to KIIs with IAHBP and USAID, as well as the Observatoire and AHJEDD FGDs, MEF became sensitized to communicating with the public and publishing data. Some IP and USAID informants further stated that MEF announced that, based on the Activity’s example, they planned to launch a website dedicated to publishing data about the budget. While a new web portal has not yet launched, the ET noticed updated data on MEF’s website at the end of March 2021. National Procurement Board (Commission Nationale des Marchés Publics) The Observatoire and AHJEDD FGDs mentioned the National Procurement Board (CNMP) as regularly publishing and providing data about public procurement. The ET visited the CNMP website and it appeared to present the most up-to-date and organized data on public procurements. In March 2021, data were current through February 2021. One could also search data by public agency. However, the ET did not have the opportunity to assess the extent to which IAHBP contributed to CNMP’s effort in publishing updated procurement data online, and also the extent to which IAHBH integrated these data in the tools they developed for budget monitoring. Nonetheless, the Observatoire and AHJEDD confirmed using CNMP’s website to monitor the public procurement aspects of the budget. Impediments to Public Monitoring of Budget Execution and Budgetary Management Deficiency of Superior Court of Auditors (CSCCA) and Lack of Culture of Accountability Eighteen KIIs (one IP, one USAID, five CSOs, two MPs, four PPs, three GOH, two economists) and two FGDs (AHJEDD and Observatoire) indicated that the CSCCA does not exercise their authority to monitor government budget execution or issue public reports. A journalist stated: “CSCCA remains a black box that does not communicate with public.” The ET visited CSCCA’s website, which only revealed the audit reports about Petrocaribe funds and one report on the effectiveness of public finances. Some KIIs remarked that because of the Petrochallenge, CSCCA has shown a willingness to do its job, but no follow-up reports have been published thus far. Among these informants, 12 also faulted Parliament, whereas 17 thought it will take more time before change could be noted. 30 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV A minority of KIIs (six) noted that Parliament’s voting on only one budget (FY2017-2018) during LOP provided limited opportunity for public monitoring of budget execution and accountability. 44 Finally, 14 KIIs (one IP, three CSOs, one MP, all six PPs, one GOH, two economists) and the female-only FGD supported the view that a pervasive culture of lack of accountability and information retention made it difficult for public monitoring of budget execution. An economist stated: “Lots of people in government just think that they don’t owe any explanation to anybody!” Six KIIs within this group stated that impunity made the situation worse. To corroborate those stakeholders’ statements, the ET went on to assess the extent to which information on budget execution is available on relevant GOH agencies’ websites. This revealed that MEF Tables of State’s Financial Operations, which published budget execution data, were not updated with data for FY2019-2020 until mid-March 2021. Updated information on the last and current fiscal year (2020-2021) were posted at end of March 2021. The data are, in general, in separate Excel or PDF files by quarter. A search on the World Bank “BOOST” standardized Portal for Open Budget presented data for Haiti only for the FY2016-2017 budget. BRH’s website presents aggregated data on government revenues and expenses in its “public finances” section up to FY2018-2019. One needs to download the quarterly “monetary policy notes” for newer data. Data were not found on either the Ministry of Education’s or Ministry of Health’s websites. CONCLUSIONS IHABP succeeded in triggering interest from some segments of the population and civil society in the monitoring and analysis of budget execution and management and the importance of social audits. The Activity seems to have also sensitized MEF to publish data in a form allowing public monitoring of budget execution and management. However, the lack of access to timely information about budget execution published in a simple, clear, yet detailed format still stands in the way of effective public monitoring of budget execution. While some Petrochallengers are likely to be IAHBP social audit training beneficiaries, the ET did not find evidence to support a finding that the Petrochallenge itself originated from the Activity’s interventions. While the tools IHABP deployed to address these issues did meet their LOP performance indicators, they were not widely adopted by stakeholders or the public. The tools suffered from lack of user￾friendliness or promotion, and have not been updated with newer information about the budget since mid-2019. 44 Indeed, for the whole duration of Activity, only the FY2017-2018 budget was voted on by Parliament. For 2018-2019, Parliament rejected the budget submitted by the Executive Branch, and Parliament adjourned the second Monday of 2019 without a budget being voted. The Executive Branch automatically resubmitted the FY2017-2018 budget for FY2018-2019. See Le Nouvelliste article Le Nouvelliste | Jovenel Moïse constate la caducité du Parlement et dispose seul de son budget, accessed April 3, 2021. 31 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV IV. RECOMMENDATIONS 1. To continue raising awareness about fiscal transparency, USAID should continue to support interventions to increase public education and awareness on the budget, especially by expanding information to more segments of Haitian society, like underserved populations, those outside Port-au-Prince, and youth, a group that has been instrumental in the “Petrochallenge” fiscal accountability movement. 2. Findings demonstrated that the Observatoire is not fully operational. Therefore, USAID should continue to sponsor Observatoire until it becomes functional and reaches maturity. Observatoire can become an instrument to empower other CSOs to participate in budget advocacy and fiscal accountability. Future assistance should particularly focus on helping the organization attract member CSOs who have the strength and resources to support Observatoire financially. 3. To expand AHJEDD’s reach throughout Haiti to increase media coverage on the budget, USAID should continue to partner with AHJEDD to train more journalists from all regions of the country to report on the budget and public finances. 4. To help Haitian civil society with budget monitoring and budget analysis, USAID should consider providing support to (re)designing the civil society online portal and social media tools to disseminate information on the budget in ways that are detailed, yet clear, simple and easily understood by non-experts. In that regard, we advise that the redesigned tools also publish contents in Creole and with infographics easily sharable on different mediums. 5. To address obstacles to fiscal accountability due to structural issues and weaknesses in Haiti’s public financial management system, USAID should consider supporting programs to improve the GOH’s public financial management and its scores under the 28 indicators of the Public Expenditure and Financial Accountability (PEFA) framework. 6. To address the apparent absence of fiscal transparency at the local government level, where municipalities do not even have a budget, USAID should consider developing a budget transparency program with some municipalities (that have a minimum level of capabilities) and provide help on financial management systems. This program could include Observatoire by developing civil society budget watchdogs at the local levels as chapters of the Observatoire. 32 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANNEXES ANNEX A: EVALUATION SCOPE OF WORK Final PE of the Project: “Increased Accountability in Haiti’s Budgetary Process/Group Croissance” PURPOSE OF THE EVALUATION The purpose of the evaluation is to provide an opportunity to do an in-depth investigation and analysis of the issues, and to document project performance, results, impacts, lessons learned, and best practices from various stakeholder perspectives. It will also guide future USAID/Haiti programming, in areas such as fiscal transparency, government budgets and spending, leadership, financial management, and other aspects of good governance and government accountability. SUMMARY INFORMATION PROJECT NAME INCREASED ACCOUNTABILITY IN HAITI’S BUDGETARY PROCESS/GROUP CROISSANCE Implementer Group Croissance, S.A. Cooperative agreement number: AID-521-A-17-00002 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $ 700, 000.00 Life of Strategy, Project, or Activity 01/19/2017 to 01/18/2019 Active Geographic Regions N/A Development Objective(s) (DOs) [Insert number and name of the DO that this evaluation relates to] USAID Office Governance Office BACKGROUND Fiscal transparency refers to a series of publications informing about how governments raise, spend and manage public resources worldwide. It allows and fosters therefore greater government accountability by providing a window into which citizens monitor government budgets and spending. By doing so, it helps citizens holding their leadership accountable, and facilitating public debate. Fiscal transparency represents a critical aspect of effective public financial management, builds market confidence, and underpins economic sustainability. Improving fiscal transparency has clear economic and development benefits. Since 2008, the State Department has conducted fiscal transparency reviews of governments receiving bilateral allocations of assistance funded under the State Department Foreign Operations, and Related Programs Appropriations Acts. Since the fiscal year 2012, the State Department has annually published fiscal transparency reports, as required by annual appropriations acts. The release of the report is an 33 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV opportunity for citizens to consider their government’s level of and approach to fiscal transparency, and to engage in discussions about how to improve budget transparency, public participation in the budget process, and fiscal policy. The report describes the minimum requirements of fiscal transparency developed, updated, and strengthened by the Department in consultation with other relevant federal agencies, reviews governments that were originally identified. The minimum requirements of fiscal transparency include having key budget documents that are publicly available, substantially complete, and generally reliable. The review includes an assessment of the transparency of processes for awarding government contracts and licenses for natural resource extraction. Haiti did not meet, in the 2016 report, the minimum requirements of fiscal transparency. There was no significant progress. According to this report, Haiti published its enacted budget, but not its executive budget proposal. The government did not publish end-of-year reports. Budget documents were neither substantially complete nor reliable. Natural resource revenues were not identified by source or type. Allocations to and earnings from state-owned enterprises were not clearly included in the budget. Significant state-owned enterprises did not have audited accounts that were either provided to an oversight body or made publicly available. The supreme audit institution did not make audit reports publicly available. The criteria and process for allocating licenses and contracts for natural resource extraction were outlined in law but have apparently been inconsistently applied. Basic information on natural resource extraction contracts and licenses was rarely publicly available. Haiti’s fiscal transparency would be improved by making its executive budget proposal and end-of-year reports widely and easily accessible to the general public within a reasonable period of time; clearly identifying natural resource revenues and allocations to and earnings from state-owned enterprises in the budget; making audited financial statements publicly available for significant state-owned enterprises; making supreme audit institution audit reports publicly available; consistently adhering to the legal criteria and procedures for natural resource contracting and licensing in practice; and making basic information on natural resource contracts and licenses publicly available. On July 2014, Group Croissance S.A signed a cooperative agreement with USAID to implement the “Civil Society Participation in Haiti’s Budgetary Process” project. A number of activities were conducted through this project from October 2014 to May 2016. USAID has felt the need to build on the success of this first experience and therefore requests from Group Croissance a Technical and Financial Proposal for a second project that will follow up from the first one. And the ““Increased Accountability in Haiti’s Budgetary Process” is born. The primary goal of the “Increased Accountability in Haiti’s Budgetary Process” project was to support the Haitian civil society in its efforts to increase government transparency and accountability in financial management. More specifically, the project focused on increasing citizens’ participation and engagement in Haiti’s national budget process. The project came at a very good timing, considering the significant role the Haitian civil society has to play in helping Haiti to become an emerging country and also considering the need for transparency in the execution of the national budget for a better and fair allocation of resources and for preventing corruption, which can undermine Haiti’s development process. 34 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV The project was implemented under these assumptions: • The political environment is stable; • Funds are available on time; • Civil Society Organizations are motivated to attend the project’s events and to be engaged in the advocacy process; • Journalists are interested in creating a budget-sensitive network and keeping it active; • • The public is responsive through the project’s social media; • • The “Observatoire” will be able to mobilize necessary resources in order to be self-sufficient; • Project’s beneficiaries are actively engaged in the advocacy process and make recommendations regarding the budget; • Funds are available to conduct the Open Budget Survey; • • The population understand the importance of the mobile app and are willing to use it; • • Mayors understand the importance of having local budget and are willing to adopt this new approach in their municipalities; • • The government is willing to observe the budgetary process. Summary Strategy/Project/Activity/Intervention to be Evaluated The “Increased Accountability in Haiti’s Budgetary Process” project was conceived in order to help members of the Haitian society to improve their understanding of the mechanisms to prepare the budget, to analyze, to evaluate this tool and to introduce their recommendations to the National Government and the Parliament with the idea to influence the Budget Preparation and Execution process and the level of transparency in this process. Implementation of the project is supposed help find answers to some challenging questions, such as: what are the priorities the budget focuses on? How the needs of different groups of the society are taken into account in the budget? What needs to be done to increase transparency in the budgetary process? How many changes Civil Society Organization (CSO) members help bring to the National Budget preparation and Execution process? Group Croissance had to focus on developing partnerships with actors from different sectors to conduct activities and to secure additional resources for execution of the project. As it has been in the past, smart resources mobilization strategies could help do more with less in this project. Among the partners and allies Group Croissance had to work with are the networks of public universities, key organizations of the Civil Society, the media, the Federation of mayors and youth organizations. 35 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Group Croissance decided to relate the budget with the Sustainable Development Goals (SDG’s) adopted with their counterparts in September 2015 at the United Nations Indeed, after having worked in the past with an organization called “Gouvernement Jeunesse” to mobilize the population about the engagement taken by the government with 192 other governments to eliminate poverty in the world by 2030, Group Croissance wanted to keep the focus on this movement through the following years. The goal was to get many organizations in the civil society maintaining the pressure on the public authorities to align the SDGs with the investment plans that will be adopted in the budget in the next 15 years. Therefore, Group Croissance decided to work more closely with existed organizations and try to get each other work together in order to have a better impact. Summary of the Project/Activity Monitoring, Evaluation, and Learning (AMEL) Plan The following documents and reports will be made available upon request: the project description and subsequent amendments, annual monitoring and evaluation plans, annual work plans, monthly and annual reports. SCOPE OF WORK FOR THE EVALUATION The result framework indicates the anticipated outputs and outcomes for this activity to help increase accountability in Haiti’s budgetary process. Through planned activities, Group Croissance works to contribute to one of the four pillars of the US government’s Post-Earthquake Haiti strategy, which is Governance and Law. Under this component the project aims to strengthen public administration and financial accountability through sound budget formulation and financial reporting, and remedy underperforming administrative and procurement systems. The objective that derives from this goal and corresponds to the project is: “Increase accountability into Haiti’s budgetary process.” The project helps achieve this goal through activities under three components. Therefore, each of these components constitute an intermediate result: • Public consultation takes place and broad-based civic input is encouraged during the budget preparation and formulation stage • Public participation is increased in the budget process as people are educated through awareness activities • The population is aware about public monitoring and analysis of budget execution and budgetary impact Ultimately, the final PE of “Increased Accountability in Haiti’s Budgetary Process” will allow USAID/Haiti to see to what extent Groupe Croissance has achieved these results. EVALUATION QUESTIONS 1. To what extent did the activity improve civil society participation in Haiti’s budgetary process? 2. How did the activity contribute to improve public monitoring and Analysis of budget execution and budgetary management? 36 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV EVALUATION DESIGN AND METHODOLOGY It is expected that the evaluation team proposes an overall research design to address the evaluation questions and a plan for collecting and analyzing the data. Nevertheless, given the nature of the evaluation questions, the evaluation design should be based solely on a combination of qualitative techniques to address the evaluation questions. Key informant interviews, focus group interviews and in￾depth review of projects reports are among the techniques that the evaluation team should consider when addressing the evaluation questions. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Upon receipt of this Activity Request, IFOS shall submit to Haiti ESS, within seven (7) working days after the contract is fully executed, a Work Plan that will include: (1) anticipated schedule and logistical arrangements; (2) key personnel delineated by roles and responsibilities with their level of effort; (3) deliverable schedule. 2. Inception Report: The evaluation team will have two weeks to review the project documents and produce an inception report or background report that addresses what the evaluation team has learned based on project documents provided to them. The inception report should inform the design of the evaluation. 3. Evaluation Design: Within fifteen (15) working days of the fully executed contract, the sub￾contractor must submit to Haiti ESS an Evaluation Design (which will become an annex to the Evaluation Report). The Evaluation Design will include: (1) description of the project demonstrating understanding of key components and current status; (2) detailed methodology clearly indicating: who will be the key stakeholder groups for data collection, how and why they will be selected, how many of each group will be included, data collection methodology for each participant group and rationale for using it, and a data analysis plan for each data collection methodology; (3) detailed evaluation design matrix that summarizes above information (data sources, methods, and data analysis plan) and links it to the evaluation questions in the SOW; (4) draft questionnaires and other data collection instruments or their main features; (5) a list of potential interviewees and sites to be visited, including the sampling approach; (6) known limitations to the evaluation design; and (7) a dissemination plan. USAID will take up to ten business days to review and consolidate comments. Once the sub-contractor receives the consolidated comments on the initial Evaluation Design and Work Plan, they are expected to revise the reports within one business days. 4. In-briefing: Prior to undertaking fieldwork, the sub-contractor will have an In-briefing with the client (USAID/Democracy and Governance office and ESS COR). The sub-contractor will provide a PowerPoint presentation summarizing the evaluation design. In addition, the meeting will cover the sub-contractor’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and clarify any questions or logistic needs. An In-briefing note with any issues raised and new resolutions resulting from the In-briefing will be submitted to ESS prior to field deployment. 5. Weekly Status Reports: Those reports will cover activities undertaken during the data collection period, challenges faced, strategies adopted to overcome such challenges, programming of upcoming activities for the next reporting period, and any identified risks related to upcoming activities. Weekly updates will also include an updated draft Findings, Conclusions, and Recommendations (Data Analysis) Matrix (Appendix E) outlining what the team has learned to- 37 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV date. The initial weekly version of the matrix should reflect data collected through a review of relevant project reports and other documents. Completing this deliverable weekly (in draft version) will ensure that the final Data Analysis Matrix, due on completion of data analysis, will include all data, and be shared with ESS for review to ensure that the matrix provides relevant information that is clearly grounded in the evidence to fully answer the evaluation questions. 6. Detailed Summary Notes: After each KII and FGD, the team will meet to write a report that consists of the notetaker(s’) combined notes, enriched with details from audio recording. Separately, reflections and general observations from KIIs and FGDs can be written up and included in the summary notes following KII and FGD protocols. The summary notes will, thus, contain the raw qualitative data to inform ESS about findings. They can be in French or English but must stay as close as possible to what was actually said in the KII or FGD and contain as many quotes and explanations as possible. The notes must be anonymized to protect the identities of participants, and the sub-contractor must provide a separate spreadsheet that lists who participated in each KII and FGD with basic demographic data like age, sex, etc. Summary notes must be uploaded daily to the server. 7. Audio Recordings: Audio recordings of each FGD and KII must be submitted to ESS as a backup file that supports the summary notes. Audio recordings must be uploaded daily to the server. 8. Evaluation Out-briefing Presentation: The sub-contractor is expected to hold two final in-person presentations, one with the USAID/Haiti Democracy & Governance Office team and the second with the USAID mission at large, to discuss the summary findings and recommendations after the conclusion of the data collection and analysis. 9. Draft Evaluation Report: The draft evaluation report should follow the USAID Evaluation Report Template, which will be provided. The report will address each of the questions identified in the SOW and any other issues the team considers important to the evaluation objective. Any such issues can be included in the report only after consultation with USAID. Per USAID requirements the report will clearly distinguish between findings, conclusions, and recommendations and the logic among the three must be clearly evident. The submission date for the draft Evaluation Report will be determined in the evaluation work plan, generally two weeks after the mission-wide out briefing. Once the draft Evaluation Report is submitted, USAID/Haiti will have ten business days to review and comment, after which the USAID ESS COR will submit consolidated comments to ESS. ESS will communicate USAID feedback to the sub-contractor. Through ESS, the sub-contractor will then submit a revised final report within three business days, and again USAID/Haiti will review and send comments within five business days. 10. Final Report: The sub-contractor will be asked to take no more than five business days to respond/incorporate the final comments from the ESS COR and the Democracy and Governance Team. ESS will then submit the final report to the ESS COR. All project data and records (FGD and KII summary notes, audio files, etc.) shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITION The Evaluation Team shall be comprised of two Key Personnel positions: (i) a Team Leader, and ii) a Deputy Team Leader served as specialist in Governance. 38 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV The Team Leader (TL) will Lead the evaluation, including desk review, data collection and analysis, and report writing, Manage the team, including delegating responsibilities, training and guiding team members, monitoring progress, and providing feedback on inputs, serves as the primary liaison with USAID/Haiti’s Democracy and Governance Office, facilitate briefings, and lead presentations. The Deputy Team Leader will help the TL in the overall management of the evaluation team and the final products, in conformity with this Statement of Work. He is familiar with the Haitian administration. He has experience in designing and implementing KIIs and FGDs. He leverages sectorial and contextual expertise in the design of the evaluation and data collection instruments, participate in desk review, data collection, and analysis, leads a sub-team during fieldwork and contribute to the production of high￾quality deliverables. Facilitators and Note-takers. They will coordinate all the logistics aspects, such as arranging KIIs and FGDs. Support the ET in conducting data collection in facilitating some KIIs and FGDs as well as taking detailed notes. Assist the ET with coding and data analysis. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. EVALUATION SCHEDULE EVENTS ESS INTERNAL DUE DATE USAID DUE DATE Work plan submitted March 12 Work plan approved April 2020 Inception Report September 21 September 28 Evaluation Design September 25 October 12 In-briefing presentation October 16 October 26 Field data collection begins October 27 Field work completed November 14 Data analysis begins November 16 Data analysis completed November 28 Out-briefing Technical Team November 30 December 07 Out-briefing Mission December 08 December 09 Draft report submitted December 21 January 04 USAID draft report feedback January 18 Final report completed February 01 February 08 39 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV 40 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; and • Signed disclosure of conflict-of-interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.1 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or Activity. 41 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 42 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX B: EVALUATION TEAM MEMBERS POSITION RESPONSIBILITIES Noel Jonathas, Jr. TL/Evaluation Specialist Lead the evaluation, including desk review, data collection and analysis, and report writing. Manage the team, including delegating responsibilities, training and guiding team members, monitoring progress, and providing feedback on input. Serve as the primary liaison with USAID, facilitate briefings, and lead presentations. Ulrick Jean Claude ATL/Sector Specialist Leverage sectoral expertise in the design of the evaluation and data collection instruments. Participate in desk review, data collection, and analysis. Lead sub-team during fieldwork. Contribute to the production of high-quality deliverables. Laurcelie Alcimé and Marie Louise Lafleur Notetakers Coordinate in-country logistics, such as arranging KIIs and FGDs. Support the ET in conducting data collection, primarily though taking detailed notes. Assist with coding and analysis of data. Andrew Carmona PD Provide technical consultation and quality assurance (QA) to the ET. Ensure USAID satisfaction via routine check-ins. Provide high-level guidance on methodology and major deliverables. Implement SI’s EQUI approach in close coordination with the TL, conduct in-depth review of deliverables using QA checkpoints, and review utilization-focused evaluation checkpoints. Offer high-level supervision of contract financial compliance. Louisena Louis PM Promote client satisfaction throughout the evaluation. Ensure fulfillment of SI’s EQUI approach, on-schedule completion of deliverables, and compliance with SI procedures and USAID regulations. Recruit and vet additional team members for evaluation, as required. Onboard the team and train them on SI procedures and quality standards. Facilitate predeparture team planning meetings, check in regularly with the ET, and manage personnel issues. Manage the process of responding to USAID comments on the draft report, using comments matrices to ensure that all feedback is addressed. Supervise the PA. Oversee the evaluation budget, produce projection forecasts, and maintain a tracking system for LOE expenditures. Submit evaluation deliverables in compliance with USAID branding guidelines and SI EQUI standards. Approve invoices and ensure expenses are allowable and consistent with contract requirements. Issue and ensure contractual adherence to partner subcontract. Manage the budget by tracking LOE usage and other spending. 43 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV POSITION RESPONSIBILITIES Alexandra McMullin PA Organize the administrative and logistical procedures for rapid project start up; and manage onboarding, mobilization, and deployment of the evaluation. Mobilize travel arrangements; and process invoices, expense reports, and other administrative documentation. Provide knowledge management support through SI’s online SharePoint intranet system. Copyedit and format deliverables. Liaise with the SI accounting team to ensure timely submission of client invoices. Onboard and orient consultants to SI standard operating procedures and security protocols. Coordinate with local logistician to ensure smooth operations of the ET. 44 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX C: INFORMED CONSENT SCRIPTS AND DATA COLLECTION PROTOCOLS INFORMED CONSENT AGREEMENT—KII Purpose: Thank you for taking the time to speak with us today. My name is [NAME]. I am a researcher from Social Impact, a company that is based in the United States. Our team is conducting a study of the Increased Accountability in Haiti’s Budgetary Process (IAHBP) Activity, also known as IAHBP/Group Croissance, implemented by Group Croissance, S.A., under a cooperative agreement. IAHBP/Group Croissance is a project to support Haitian civil society in its efforts to increase government transparency and accountability in financial management. The activity focused on increasing citizens’ engagement in Haiti’s national budget process. Increased citizen participation in and monitoring of the national budget process has the potential to contribute to addressing corruption and make the Government of Haiti (GOH) better manage and more fairly allocate its resources to reach the country’s development goals. The purpose of the evaluation is to understand the extent IAHBP/Group Croissance was successful meeting its objectives, to assess what were the most important challenges it faces, and identify lessons learned and good practices that might help future similar initiatives in Haiti or elsewhere. We are speaking to you today because you collaborated directly with IAHBP/Group Croissance or participated in an activity conducted by IAHBP/Group Croissance, are involved in Haiti’s budget process or public finances, or because you may have been directly or indirectly affected by IAHBP/Group Croissance interventions. We would like your honest impressions, opinions, and thoughts about various issues related to this project’s implementation and outcomes. We are independent consultants who have no affiliation with Group Croissance nor are we from the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the IAHBP/Group Croissance Activity. The interview will take about one hour of your time. Although we will publish our findings in a public report, all your answers will be kept confidential. Nothing you tell us will be attributed to you by directly name or allow someone to easily identify you in our report. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to improve citizen participation in Haiti’s budget process. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. Do you have any questions? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). 45 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Yes No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ GENERAL DATA COLLECTION PROTOCOL The following data collection protocol will be adjusted for each different stakeholder group and data collection events (KII or FGD) Date: Interviewer Name: Primary Notetaker Name: (Info below to be included in a table for all key informants and FGD participants) Informant Name: Informant Title: Informant Organization: Sex of Respondent: General and Cross-Cutting Issues 1. What do you know about the IAHBP Activity (for people not known to be involved in the Activity)? Or, how long have you been involved in IAHBP, and what were your roles and responsibilities in its implementation? (This question is not for Chief of Party.) 2. What is your view of the Activity’s interventions in terms of whether they were relevant or effective, and potential unintended consequences and lessons learned in general? 3. Overall, how have you seen public authorities (Executive and Parliament) respect of the budget process and commitment to fiscal transparency evolve during the last three years (from 2017 to today)? 46 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV 4. Based on your experiences (with the Activity and otherwise), what are the minimum necessary conditions to successfully achieve budget transparency and accountability in Haiti? 5. There is a constitutional provision on the right to information. What are the main government channels for the public to access accurate information on the national budget and its execution? 6. To what extent have you seen the Activity influence, or not, the public availability of information on the national budget and public procurement? What could have been done differently? 7. How instrumental do you see an institution like the Court of Auditors (Cour des Comptes, CSCCA) for budget transparency and accountability? To what extent has the Activity affected, if at all, the work of this institution? 8. To what extent have the different political crises, civil unrests, and government dysfunction during the last three years (since 2017) affected, or not, the Activity’s performance? On EQ1 Objective 1: Public consultation takes place, and broad-based civic input is encouraged during the budget preparation and formulation stage. Observatoire 9. In your view, what are the main achievements of IAHBP in terms of improving public consultation, with broad-based civic inputs during the budget preparation and formulation stage? 10. The Activity’s AMEL expected some results with specific performance indicators. What are the main challenges faced by the Activity in its efforts to reach planned results? 11. The Activity placed an emphasis on the Observatoire as a key structure to help achieve fiscal transparency in Haiti; however, the Activity reports suggested few successes with it. Specifically, the Activity reports indicated that the Observatoire faced some difficulties operating on its own. What were the main challenges and their root causes? 12. What is the organization’s current legal form and its membership structure (individual or institutional members/CSOs, or both)? 13. Also, the Activity reports did not indicate whether or not outreach efforts were made to leverage academia (university professors) involved in budget and public finance teachings as natural allies/members of the Observatoire. What are your views in that respect? 14. What are the Observatoire’s financing mechanisms and sustainability in the medium and long term? 15. What were Observatoire’s interventions with respect to latest budgets? If none, what were the impediments—apart from internal ones—to get involved in the process? AHJEDD 16. Regarding AHJEDD, what method do you use to trace and count the number of stories on budget published by its members? 17. Can you provide us more information (and documentation if it exists) on AHJEDD’s publication (DevHaiti) in particular? o Is there a Memorandum of Understanding or contract among the three publishers? o How is the paper’s production financed, and what are the planned steps after current financing and support end? 47 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV o What is the distribution strategy and target audience? How do you see that producing an outcome in terms of budget transparency? 18. What general lessons can you draw from AHJEDD’s relative success that may be replicable for other similar activities? Objective 2: Public participation is increased in the budget process as people are educated through awareness activities. On Civil Society Participation in the Budgetary Process 19. The FY 2018 Annual Report indicated that “very few organizations are involved in the fight for budget transparency.” Taking that reality into account, how did the Activity select CSOs most likely to be motivated and engaged in the advocacy process? 20. In what ways did the Activity try to strengthen CSOs’ abilities to engage in the budget process? 21. Activity reports showed only one CSO (CNEH) made a recommendation, and only one regarding a budget, while the Activity’s AMEL targeted seven organizations. o What were the challenges and lessons learned in strengthening and motivating the selected (or your) CSOs to make recommendations on the national budget? 22. What (other) civil society groups could make (additional) contributions to the Activity’s intended outcomes, and how? 23. What are the challenges to motivate those CSOs to be involved in budget recommendations and monitoring? How could those challenges be addressed? Participation of Political Parties and Local Authorities (Mayors) in the Budget Process 24. What outcomes did the Activity’s education and awareness interventions produce in terms of local authorities’ (Mayors’) involvement in the budget process? And in having their own budget? 25. To what extent did the Activity’s budget-related trainings for political parties produce outcomes in terms of increasing their involvement and monitoring the budget adoption and execution? On Social Audit 26. The Activity reports revealed the difficulty in having public authorities participate in social audit exercises. However, the Activity did not mention the Petrocaribe challenge movement and the social audit campaign (audit collectif) led by different youth groups (Petrochallengers) that occurred during the Activity’s period. To what extent did the Activity’s public education intervention influence or not influence this movement? 27. In what ways, and to what extent, could/can CSOs capitalize on the youth-led Petrocaribe challenge to push for greater budget transparency and accountability in Haiti? On the Open Budget Index Survey 28. What is the current status of the process to include Haiti in the Open Budget Index survey? 48 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV 29. Pending the involvement of the IPB in Haiti, to what extent can there be a domestic alternative to IBP’s Open Budget Index? In what ways can a structured, strengthened and reinforced Observatoire take on this task? Activity Objective 3: To improve public monitoring and analysis of the budget. Effectiveness of Public Education Trainings 30. How did you find the budget education trainings conducted by the Activity? And how, if at all, did they affect public participation in the budget process and monitoring? 31. We have data for only FY 2017 on the number of people (84) who completed the trainings and who volunteered to answer the evaluation questionnaire. How do you explain that? 32. To what extent and what outcomes did the public education interventions (trainings, presentations, etc.) produce in terms of public monitoring and analysis of budget execution and management? Activity’s Online Presence (Mobile app, Website, and Social Media Accounts) 33. What was the nature of the contract between CompuConsult and the Activity? 34. What is the current status of the different tools? What institution is responsible for, and how did the Activity obtain accurate budget-related information for publication on the different online tools? 35. What was the Activity’s overall strategy for content dissemination and user acquisition and retention through the online tools (mobile app, website, social media accounts)? 36. How did the Activity envision the financial and operational sustainability of the different online tools (ownership, continuing operation, content management, technical updates, etc.) after the Activity ended? 37. Based on the Activity or otherwise, what are the lessons learned in terms of the “key to success” to increase user acquisition and the retention rate for budget-related online tools? 38. What could make the app and the Activity’s social media tools more successful? 39. Any other thoughts on the Activity or its interventions? 49 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX D: REFERENCES AND ENDNOTES Annual Reports • Project Annual Report FY 2018 (October 1, 2017–September 30, 2018), Group Croissance • Project Annual Report FY 2017 (January 19, 2017–September 30, 2017), Group Croissance Annual Work Plans, IAHBP/Group Croissance Background Documents • Increase Accountability in Haiti’s Budgetary Process, October 2017–September 2018, Work Plan, Year 2, Haiti ESS • Closeout Plan, Increase Accountability in Haiti’s Budgetary Process, Group Croissance Quarterly Reports • Quarterly Report (January 1, 2018–March 31, 2018) • Quarterly Report (April 1, 2018–June 30, 2018) • Quarterly Report (October 1, 2017–December 31, 2017) Evaluation Reports • Internal Mid-Term Evaluation, Action Consult, April 2018 Weekly Reports • Project highlights: May 21 to June 8, 2018, Haiti ESS • Project highlights: July 2 to July 20, 2018, Haiti ESS • Project highlights: August 12 to August 24, 2018, Haiti ESS • Project highlights: August 27 to August 31, 2018, Haiti ESS • Project highlights: September 4 to September 15, 2017, Haiti ESS • Project highlights: July 23 to July 27, 2018, Haiti ESS • Project highlights: August 27 to August 31, 2018, Haiti ESS • Project highlights: September 18 to October 6, 2017, Haiti ESS • Project highlights: October 16 to October 31, 2017, Haiti ESS • Project highlights: August 21 to August 31, 2017, Haiti ESS • Project highlights: September 18 to September 29, 2017, Haiti ESS • Project highlights: May 14 to May 17, 2017, Haiti ESS • Project highlights: August 1 to August 10, 2018, Haiti ESS • Project highlights: September 3 to September 7, 2018, Haiti ESS • Project highlights: September 10 to September 14, 2018, Haiti ESS • Project highlights: September 17 to September 29, 2018, Haiti ESS • Project highlights: October 1 to October 31, 2018, Haiti ESS Monitoring and Evaluation Plans, IAHBP/Group Croissance Background Documents • AMEL Plan, Haiti ESS Project Description and Modifications, IAHBP/Group Croissance Background Documents • Cooperative Agreement No. AID-521-A-17-00002, Increase Accountability in Haiti’s Budgetary Process (including the project description), Group Croissance • AID-521-A-17-00002, Modification of Assistance, Group Croissance • AID-521-A-17-00002-06, Modification of Assistance 50 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV Success Stories • Success story #1 • Success story #2 • Success story #3 • Success story #4 Media Reports (online) • Le Nouvelliste • Le National • LoopHaiti • ToutHaiti • Haitilibre • AyiboPost • AFP • Journal de Montréal • Websites of Radio Metropole, Radio Vision 2000, Radio Caraibes, RadioTélé Pacific, Radio Kiskeya, Radio Magic 9 Websites/Web portals • http://www.bidjepeyidayiti.ht/ • Website of Haiti’s Ministry of Finances (http://www.mef.gouv.ht/index.php?page=Accueil) • Website of the Bank of the Republic of Haiti (www.brh.ht) • Website of the Commission Nationale des Marchés Publics (https://cnmp.gouv.ht/) • Website of Ministry of Education (https://menfp.gouv.ht/#/home) • Website of Ministry of Health (http://mspp.gouv.ht/newsite/) • Website of Superior Court of Accounts (CSCCA) (https://www.cscca.gouv.ht/) • Website of US Department of State (https://www.state.gov/2020-fiscal-transparency-report/) • Website of PEFA (https://www.pefa.org/node/1811) • https://www.facebook.com/groups/836143253074190 • https://www.facebook.com/bidjepeyidayiti • https://www.youtube.com/channel/UCw9GXQ3T6ZIddX8hoWEYJBA • https://www.scribd.com/document/364094383/HAITI-Rapport-Petro-Caribe-Octobre-2017 • https://play.google.com/store/apps/details?id=com.compuconsult.haitibudget&hl=en 51 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX E: LIST OF PERSONS INTERVIEWED NAME OF RESPONDENT DATE GENDER F GENDER M ORGANIZATION STAKEHOLDER GROUP KIIs Kesner Pharel 30-Dec-21 1 Group Croissance IP Judemyre Saint Fort 30-Dec-21 1 Group Croissance IP Evadie Daniel 28-Dec-21 1 Group Croissance IP Mélineda Jean Baptiste 29-Dec-21 1 Group Croissance IP Carl Clermont 30-Dec-21 1 Compu Consult IP Kerns Jean 14-Jan-21 1 USAID (former) USAID/Haiti Alphonse Nkuzimana 14-Jan-21 1 USAID (former) USAID/Haiti Gabriel Joseph 5-Jan-21 1 USAID USAID/Haiti Pierrot Pierre 7-Jan-21 1 CASEC Local authorities/Mayors Dénoil Anténor 6-Jan-21 1 Mayor of Belle Anse Local authorities/Mayors Fritznel Chéry 5-Jan-21 1 Mayor of Roche A Bateau Local authorities/Mayors Marie Yolaine Philippeaux 5-Jan-21 1 Mayor of La Vallée de Jacmel Local authorities/Mayors Monel Timé 22-Jan-21 1 ASEC-Bellevue la Montagne, Petion￾Ville Local authorities/Mayors Frantz Ulysse 31-Jan-21 1 Ex-Mayor of Saint Marc Local authorities/Mayors Magalie Georges 7-Jan-21 1 CNEH CSO Rosny Desroches 7-Jan-21 1 ISC CSO Jean Hugues Henrys 22-Jan-21 1 Association Medicale Haitienne CSO Abdonel Doudou 19-Jan-21 1 Juri Média CSO Katie-Flore Bien-Aimé 28-Jan-21 1 Nou pap dòmi CSO Edouard Paultre 24-Jan-21 1 CONANE CSO Jocelyne Colas 29-Jan-21 1 CJILAP CSO Jean Robert Sylvestre 2-Feb-21 1 Caisse Populaire Saint Anne, Camp￾Perrin CSO 52 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV NAME OF RESPONDENT DATE GENDER F GENDER M ORGANIZATION STAKEHOLDER GROUP Patrice Dumond 8-Jan-21 1 Senate MP Etienne Dieudonne Luma 15-Feb-21 1 Senate (ex) MP Jerry Tardieu 15-Jan-21 1 Chamber of Representatives (ex) MP Jean Hector Anacassis 11-Jan-21 1 LAPEH Political Parties Stevenson Claude 7-Jan-21 1 RDNP Political Parties Clarens Renois 5-Jan-21 1 UNIR Political Parties Evans Beaubrun 15-Jan-21 1 RDNP Political Parties Marie Lourdes Balthazar 17-Jan-21 1 RDNP Political Parties Rosemond Pradel 15-Jan-21 1 Fusion Political Parties Silin Jean Michel 11-Feb-21 1 MEF GOH Denis Junior Bens 10-Feb-21 1 MEF GOH Isaac Muller 12-Feb-21 1 MEF GOH Noel Cassandra Marie Ketia 12-Feb-21 1 MCFDF GOH Marie Pierre Dorce 13-Feb-21 1 MCFDF GOH Patrice Eugene 17-Feb-21 1 Ministry of Planning GOH Philona Jean 20-Feb-21 1 MCFDF GOH Pavel Desrosiers 25-Jan-21 1 Ministry of Health GOH Jean Patrick Alfred 21-Jan-21 1 Ministry of Health GOH Gabriel Ronald 23-Feb-21 1 BRH GOH Benoit Fabiola 25-Feb-21 BRH GOH Mathieu Fortunat 26-Feb-21 1 BRH GOH Etzer Emile 6-Jan-21 1 Independent Economist Jéliel Darius 22-Jan-21 1 Independent Economist Pierre-Marie Boisson 25-Jan-21 1 Group Sogebank Economist Emmanuela Douyon 22-Jan-00 1 Independent Economist Enomy Germain 26-Jan-21 1 Independent Economist Total 14 34 53 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV NAME OF RESPONDENT DATE GENDER F GENDER M ORGANIZATION STAKEHOLDER GROUP FGDs Observatory 9-Jan-21 4 4 OBS Youth Activists 10-Jan-21 1 3 YA University Students 11-Jan-21 1 3 USTD AHJEDD Journalists 13-Jan-21 0 6 AHJEDD Women Participants 7-Feb-21 8 WP General Participants (mixed) FGD 2 5 GP Total 16 21 USAID.GOV PERFORMANCE EVALUATION OF IAHBP | 54 ANNEX F: EVALUATION DESIGN MATRIX TABLE 4 .ACTIVITY PE MATRIX ANALYSIS PLAN ILLUSTRATIVE SUB￾QUESTIONS DATA SOURCES DATA COLLECTION METHOD RELATED ACTIVITY OBJECTIVES/ GAPS EQ Content analysis Gap analysis Quantitative analysis (PI) Sex analysis Questions 1 to 29 IP Observatoire’s members CSOs, including private sector Journalists Independent observers (academic and professional connected to budget and public finances) USAID/Haiti Public officials Parliamentarians Youth groups Objective 1: Public consultation takes KIIs, FGDs place and broad-based civic input is encouraged during the budget preparation and formulation stage Activities organized by Observatoire (PI1) Observatoire’s organization, structure, membership, financing sustainability, and current status AHJEDD/number of stories published, DevHaiti (PI2) Objective 2: Public participation is increased in the budget process as people are educated through awareness activities CNEH/CSO’s participation (recommendations) in the budget process (PI3) CSO capacity-building effort by Activity Creation of Open Budget Index for Haiti: current status of effort (PI4) EQ1: To what extent did the Activity improve civil society participation in Haiti’s budgetary process? 55 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV ANALYSIS PLAN ILLUSTRATIVE SUB￾QUESTIONS DATA SOURCES DATA COLLECTION METHOD RELATED ACTIVITY OBJECTIVES/ GAPS EQ Content analysis Gap analysis Quantitative Analysis (PI) Sex analysis Questions 1 to 8 and 29 to 39 IP Staff Observatoire’s members CSOs, including private sector Journalists Independent observers (academic and professional connected to budget and public finances) USAID/Haiti Public officials Parliamentarians Youth groups App developer (CompuConsult) Objective 3: The population is aware of KIIs, FGDs public monitoring and analysis of budget execution and budgetary impact Effectiveness of the Activity’s public education, outcomes Why so few trainees took evaluation questionnaires (PI5) Number of followers of the Activity’s online presence (PI6) Mobile App: ownership, management, data sources, update, and upgrade App’s retention rate (PI7) App’s active users (PI8) EQ2: How did the Activity contribute to improve public monitoring and analysis of budget execution and budgetary management? Content analysis Gap analysis Sex analysis Questions 1 to 8 Activity’s minimum conditions for KIIs, FGDs success Effectiveness of structural mechanism for budget transparency and accountability Willingness/commitment of public authorities for budget transparency and accountability Cross-cutting issues USAID.GOV PERFORMANCE EVALUATION OF IAHBP | 56 ANNEX G: DISCLOSURE OF CONFLICT OF INTERESTS 57 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV 58 | FINAL PERFORMANCE EVALUATION REPORT USAID.GOV