MALAWI LGAP MIDTERM PERFORMANCE EVALUATION An Evaluation for Learning, Evaluation and Research Activity II (LER II) KIM YI DIONNE This Midterm Evaluation of the Local Government Accountability and Performance (LGAP) was conducted by a team of international and local consultants fielded and managed by The Cloudburst Group, namely Dr. Kim Yi Dionne (International Academic Expert), Christopher Maulidi Phiri (Local Capacity Development Expert), Embry Howell, (Sr. Evaluation Specialist), and Kate Marple-Cantrell (Tasking Coordinator & Evaluation Specialist). This document was produced for review by the United States Agency for International Development, Democracy, Human Rights and Governance Center under the Learning, Evaluation and Research Activity II (LER II) contract: GS10F0218U/7200AA18M00017 Prepared by: The Cloudburst Group 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238 Tel: 301-918-4400 USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | i CONTENTS ACRONYMS III EXECUTIVE SUMMARY 1 EVALUATION PURPOSE AND QUESTIONS 7 PROJECT BACKGROUND 7 METHODS AND LIMITATIONS 9 RESEARCH QUESTION 1 FINDINGS: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION INCREASED MALAWI GOVERNMENT ACCOUNTABILITY AT THE LOCAL LEVEL? 15 LGAP ACTIVITIES TO INCREASE ACCOUNTABILITY 16 FINDINGS ON ACCOUNTABILITY 18 CONCLUSIONS AND RECOMMENDATIONS ON ACCOUNTABILITY 23 RESEARCH QUESTION 2 FINDINGS: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION CONTRIBUTED TO IMPROVED SERVICE DELIVERY ACROSS SECTORS (HEALTH, EDUCATION, AND AGRICULTURE)? 26 FINDINGS ON SERVICE DELIVERY 26 CONCLUSIONS AND RECOMMENDATIONS ON SERVICE DELIVERY 30 RESEARCH QUESTION 3 FINDINGS: WHAT ARE ANY UNINTENDED CONSEQUENCES OR EXTERNALITIES OF LGAP SUPPORT? HOW CAN THESE CONSEQUENCES BE MITIGATED (IF NEGATIVE) OR EXPANDED (IF POSITIVE)? 32 FINDINGS ON UNINTENDED CONSEQUENCES AND EXTERNALITIES 32 CONCLUSIONS AND RECOMMENDATIONS ON UNINTENDED CONSEQUENCES AND EXTERNALITIES 35 RESEARCH QUESTION 4 FINDINGS: HOW HAS THE LGAP AWARD USED THE RESOURCES (FINANCIAL, HUMAN RESOURCES/TECHNICAL ASSISTANCE) EFFICIENTLY AND EFFECTIVELY IN RELATION TO THE PLANNED ACTIVITIES, OUTPUTS AND OUTCOMES, AND SUSTAINABILITY? 36 FINDINGS ON USE OF RESOURCES 36 CONCLUSIONS AND RECOMMENDATIONS ON THE USE OF RESOURCES 39 RESEARCH QUESTION 5 FINDINGS: WHAT LESSONS HAVE BEEN LEARNED TO INFORM FUTURE GOVERNANCE PROGRAMMING? IN PARTICULAR, HOW RELEVANT ARE THE RESULT AREAS AND SUB-RESULT AREAS IN THE LGAP AWARD TO ACHIEVING THE ACTIVITY’S OVERALL OBJECTIVE TO BUILD THE CAPACITY FOR RESPONSIVE, EFFECTIVE, AND ACCOUNTABLE SERVICE DELIVERY THROUGH LOCAL GOVERNMENTS IN MALAWI? 41 FINDINGS ON LESSONS LEARNED 41 CONCLUSIONS AND RECOMMENDATIONS FOLLOWING FROM LESSONS LEARNED 43 REFERENCES 45 ANNEX A: EVALUATION SCOPE OF WORK 47 USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | ii ANNEX B: EVALUATION DESK REPORT AND WORKPLAN, INCLUDING A DESCRIPTION OF THE ASSESSMENT METHODOLOGY USED, A BIBLIOGRAPHY OF DOCUMENTS CONSULTED, DATA COLLECTION TOOLS, AND LIST OF INTERVIEWEES CONSULTED 55 ANNEX C: QUALITATIVE EVENT SUMMARY TABLES 96 ANNEX D: LGAP MIDTERM EVALUATION SURVEY SAMPLING PROTOCOL 105 ANNEX E: SURVEY SUMMARY TABLES 108 ANNEX F: LGAP MID-TERM EVALUATION OUT-BRIEF PRESENTATION 112 USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | iii ACRONYMS ADC Area Development Committee CAPI Computer-assisted personal interview CCAP Church of Central Africa Presbyterian CDF Constituency Development Fund CPS Citizen Perception Survey CSO Civil society organization DAI Development Alternatives Incorporated DDP District Development Plan DEN District Education Network DEYS District Director of Education, Youth, and Sports DHS Demographic and Health Survey FCDO Foreign, Commonwealth and Development Office GOM Government of Malawi HR Human resources HRMIS Human Resources Management and Information System IFMIS Integrated Financial Management Information System IRB Institutional Review Board IT Information technology KII Key informant interview LAPA Local Authority Performance Assessment LGA Local Government Act LGAP Local Government Accountability and Performance MALGA Malawi Local Government Association MEDA Malawi Electoral and Decentralization Activity MG Mothers' group MLGRD Ministry of Local Government and Rural Development MP Member of parliament M&E Monitoring and evaluation NAP National Advocacy Platform NCRSH National Committee on Research in the Social Sciences and Humanities NGO Non-governmental organization NLGFC National Local Government Finance Committee OPC Office of the President and Cabinet PITA Participation, inclusion, transparency, and accountability PTA Parent-Teacher Association SEP Socio-Economic Profile TA Traditional authority TPAS Teacher-Performance Assessment System USAID United States Agency for International Development VDC Village Development Committee USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 1 EXECUTIVE SUMMARY PURPOSE This mid-term performance evaluation of the Local Government Accountability and Performance (LGAP) program is intended to measure how the award has performed and guide the United States Agency for International Development (USAID)/Malawi in its future governance programming. LGAP activities seek to build capacity for responsive, effective, and accountable service delivery through local governments in Malawi. The project takes advantage of a unique policy window in which the Government of Malawi (GOM) is undergoing a reform process to accelerate and deepen ongoing decentralization efforts. To achieve its mission, LGAP strives to improve local government performance and transparency, increase citizen engagement, and strengthen the enabling environment for decentralization in Malawi. Overall, the evaluation found that there was improvement in reported accountability and service delivery at the district level, based on interviews. However, that improvement did not extend down to the village level, where surveys found little improvement. BACKGROUND AND CONTEXT After operating without locally elected leaders for nearly a decade, Malawi held tripartite elections (presidential, parliamentary, and ward councilors) in 2014 to reintroduce the election of local government leaders. Then, the GOM launched several reform initiatives intended to accelerate and deepen an ongoing decentralization process. These reform initiatives created a unique policy window in which USAID/Malawi awarded the LGAP contract to build capacity for responsive, effective, and accountable service delivery through local governments in Malawi. LGAP is a five￾year activity that provided support to eight district councils in Malawi in the initial three years of implementation. In this report, we refer to such districts as “earlier wave LGAP” districts. In September 2019, LGAP expanded to support all 28 councils with additional funding from the United Kingdom’s Foreign, Commonwealth and Development Office (FCDO, formerly DFID). LGAP support is primarily provided via technical assistance to both national and subnational governments, supportive supervision for core local government operations, public financial management and accountability, capacity-building for councilors and District Executive Committee staff, and support for integration with USAID activities. METHODS AND LIMITATIONS In order to address the above research questions, the evaluation relies on a mixed-methods design. The evaluation team conducted desk-based research, individual and group key informant interviews (KIIs), and a survey of LGAP beneficiaries. The team then cross-referenced qualitative and quantitative data to enhance validity and mitigate methodological limitations. This approach offers the most rigorous examination of the data possible, but the conclusions that can be drawn are limited without a counterfactual comparison. The team collected data in 10 of Malawi’s 28 districts, stratified by LGAP implementation wave, region, and local government performance. The team conducted 77 individual and group interviews with key informants and stakeholders. Combined, the views of 172 individuals are reflected in the interview data. An additional 640 surveys were conducted with LGAP beneficiaries, including local officials as well as a random sample of citizens. Due to the purposive sampling approach, the results are not nationally representative of Malawian perspectives. USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 2 A key challenge in conducting this research is its timing during the COVID-19 pandemic. For example, the team followed protocols on physical distancing and face covering, which may have reduced interview rapport, yielding fewer critical perspectives. Preventing infection transmission obviously took priority and the team was fortunate that most fieldwork was conducted while COVID-19 prevalence was very low and/or utilized phone or video conferencing. The compressed timing of fieldwork and the additional coordination requirements, however, made it even more challenging to meet with certain government officials (even by phone), especially those in positions that experience greater turnover. SUMMARY OF FINDINGS AND RECOMMENDATIONS RESEARCH QUESTION 1: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION INCREASED MALAWI GOVERNMENT ACCOUNTABILITY AT THE LOCAL LEVEL? In order to improve government accountability at the local level, LGAP employs both a top-down approach through training and mentoring of district officials and a bottom-up approach by empowering local citizens to hold elected officials and staff accountable. Important activities include training, technical assistance, support to the education sector, support to local governments for citizen outreach and market revenue collection, and coordination between development actors. Interviews showed that there was an improvement in accountability over the implementation period at the district level through coordination of development activities, improved financial accountability, and improved education accountability. However, the survey results suggested that these activities had limited success in improving the view of local government among the public. Although there was an increase in citizen satisfaction with how councilors consult citizens in some districts, there was greater dissatisfaction with how district councils respond to community needs and with the quality of projects implemented by the district council. Consistent with surveys previously conducted in Malawi, this survey found low trust in public officials among respondents, with some suggestive findings that trust of district councils is higher among citizens in earlier wave LGAP districts. In these districts, citizen engagement is higher, which has been linked to greater levels of trust in local government.1 Overall, the survey found high civic engagement at the village level, particularly in community meetings on school social contracts and village action plans. These findings informed the following recommendations: ● USAID should build upon high citizen engagement at the village level to increase trust in the district government through continued LGAP (or similar) programming. The findings suggest that LGAP activities have increased citizen participation in district government activities that should continue to build trust, creating a virtuous cycle of local accountability. ● LGAP and future USAID programming should establish attainable goals for improving outcomes. Focus funding on those activities which have demonstrated success or that provide a foundation on which to build in the future. This includes efforts towards accountability in the education sector, improved coordination across sectors through work with service committees, improved human resources management, and efforts focused on financial management and accountability. 1 Kumagai, Saki and Federica Iorio. 2020. “Building Trust in Government through Citizen Engagement.” World Bank Group, available at: https://openknowledge.worldbank.org/bitstream/handle/10986/33346/Building-Trust-in￾Government-through-Citizen-Engagement.pdf?sequence=5. USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 3 ● LGAP’s future training and technical assistance in the 20 new districts should focus on financial management practices. Provide for the sustainability of previous training through closer coordination with national partners, particularly with the Ministry of Local Government and Rural Development (MLGRD), the National Local Government Finance Committee (NLGFC), and the Malawi Local Government Association (MALGA). ● In its final year, LGAP should further disseminate the citizen budget. Develop new ways to share the citizen budget (e.g., via WhatsApp, similar to Ministry of Health communications about COVID-19) that could then be used to also disseminate district development plans more broadly. ● In designing future programming, USAID should use the particularly effective accountability activities in the education sector as a model for other sectors. For example, these lessons could be used to reform the health or agriculture sectors. RESEARCH QUESTION 2: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION CONTRIBUTED TO IMPROVED SERVICE DELIVERY ACROSS SECTORS (HEALTH, EDUCATION, AND AGRICULTURE)? Similar to the results for accountability, while interviews found perceived improvement at the district level, the survey found citizen views of government service delivery are mixed. When asked to compare current services to five years ago, 41 percent said government services have gotten worse, 22 percent rated them the same, and 36 percent said services have gotten better. However, citizens living in districts with earlier and more intensive LGAP activities provided marginally more positive assessments of government service delivery. Both with respect to overall and specific characteristics of major service sectors, the survey found there is significant room for improvement in service delivery. There was relative perceived success in the education sector, which was viewed more favorably than other sectors. Examples of successful education initiatives that were mentioned include negotiation of contracts between parent groups and teachers and improvements in teacher allocation through automation of human resources records. The improved allocation of staff to the automation of HR records serves as an important example for other sectors, particularly the health sector. In general, respondents had more positive views of workers than of material resources (e.g., views of teachers were more positive than they were about availability of books at school). Improvement in perceptions of local government service delivery did not feature prominently in the KIIs, having been mentioned only 32 times in the 77 interviews (compared to 87 mentions of local governance accountability), though 20 of these mentions occurred in KIIs conducted in the early LGAP districts. Infrastructure projects such as drilling boreholes, while in high demand, were less successful. Finally, due to implementation challenges, analysis of agriculture service delivery was set aside. The following recommendations are based on these findings: ● Given that survey respondents generally had more favorable views of human resources, future USAID programming should focus on ensuring sufficient material resources (with relevant associated governance supports). Even with sufficient staff who are adequately trained and incentivized, challenges will remain due to a lack of resources. ● USAID should undertake the more expensive infrastructure projects, such as boreholes and other construction projects, through coordination with other development partners, as the KIIs identified some of these projects as sources of local discord. USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 4 ● USAID and LGAP’s provision of material support should be implemented with the collaboration of local officials to avoid the potential for envy or suspicion, and ultimately conflict. ● To address sustainability in material resource provision, LGAP and future USAID programing should focus on local revenue generation, which would improve local government self-sufficiency and, in turn, could reduce Malawi’s reliance on foreign aid. RESEARCH QUESTION 3: WHAT ARE ANY UNINTENDED CONSEQUENCES OR EXTERNALITIES OF LGAP SUPPORT? HOW CAN THESE CONSEQUENCES BE MITIGATED (IF NEGATIVE) OR EXPANDED (IF POSITIVE)? The evaluation identified a number of both positive and negative unintended consequences (or externalities) of LGAP’s work. Positive externalities included: stimulation of other funders to also enter the local governance space; spillover from information technology (IT) systems, reporting, and skills into other government functions; increased incentives for staff and committees to work harder, leading to improved coordination between development partners; increased skills of local civil society organizations (CSOs); and, finally, improved public health due to market clean-up activities and better COVID-19 preparedness. The key negative unintended consequence is that some LGAP activities led to unnecessary conflict. Conflict or concern about potential conflict was mentioned in the KIIs (21 times) and these included officials worrying about citizens engaging in boycotts or vigilante justice as well as actual reports of demonstrations that became violent. The KIIs also identified antagonism between development partners at the district level and the national level. The following recommendations are based on these findings: ● To build on the entrance of additional funders in the governance space, the Mission should continue efforts to coordinate activities. ● Because IT interventions have important spillover effects, these activities should be continued and expanded either as part of LGAP or future USAID programming. Examples of successful IT efforts include automation of personnel records, particularly in the education sector, and automation of district financial information. We also suggest an expansion of monitoring and evaluation (M&E) at the district level, to track indicators of progress. ● LGAP or other future programs should continue and expand support to service committees, which have been instrumental in increasing coordination of services and were particularly important for the rapid response to the COVID-19 pandemic. ● Any future program, LGAP or other, should emphasize training in social responsibility of all development partners and program staff aimed at methods for avoiding destructive conflicts. Future efforts should also ensure more close coordination with all the national ministries whose functions overlap with LGAP. RESEARCH QUESTION 4: HOW HAS THE LGAP AWARD USED THE RESOURCES (FINANCIAL, HUMAN RESOURCES/TECHNICAL ASSISTANCE) EFFICIENTLY AND EFFECTIVELY IN RELATION TO THE PLANNED ACTIVITIES, OUTPUTS AND OUTCOMES, AND SUSTAINABILITY? The substantial LGAP funding came at a critical time in the devolution of responsibilities from national to local government and brought significant resources to a huge task that was beyond the resources or political will of national entities. Through its broad-based training program, LGAP revitalized local government structures such as the councils, council staff, council committees, USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 5 school governance committees, CSO networks, and Area Development Committees (ADCs). Interviewees, for the most part, felt that LGAP activities were an efficient and effective use of resources. LGAP also faced challenges due to insufficient time and resources to scale up to all districts in Malawi. There was also high staff turnover among LGAP and district staff and councilors, COVID-19-related delays, and impracticality of village-level work due to scale and distance. Complaints about activities included: incomplete interventions, short and infrequent trainings, insufficient transparency from LGAP on its work plans and exit strategy, and insufficient follow-up. The following recommendations are based on these findings: ● LGAP should implement more comprehensive and more targeted trainings in the new districts if resources permit, particularly training in financial management. ● LGAP or other future programs should improve the tracking of outcomes from LGAP efforts, particularly in the education sector. ● USAID should further develop innovative approaches to local revenue collection to sustain LGAP gains. ● LGAP and future programs can improve the sharing of information and joint planning with all national and local ministries and organizations that share similar goals and activities. RESEARCH QUESTION 5: WHAT LESSONS HAVE BEEN LEARNED TO INFORM FUTURE GOVERNANCE PROGRAMMING? IN PARTICULAR, HOW RELEVANT ARE THE RESULT AREAS AND SUB-RESULT AREAS IN THE LGAP AWARD TO ACHIEVING THE ACTIVITY’S OVERALL OBJECTIVE TO BUILD THE CAPACITY FOR RESPONSIVE, EFFECTIVE, AND ACCOUNTABLE SERVICE DELIVERY THROUGH LOCAL GOVERNMENTS IN MALAWI? Concerning Results Area One (Local Government Performance and Transparency Increased), respondents emphasized that trainings of district councils and district staff, particularly financial staff, were effective in upgrading skills, fostering ownership of roles and responsibilities, and increasing accountability. In Results Area Two (Demand for Accountable Government Strengthened at the Local Level), LGAP training of ADCs was frequently credited with empowering citizens to question their government and become more involved in development activities. However, this training did not disseminate to Village Development Committees (VDCs) and survey data showed a lack of citizen engagement in accountability activities at the district level. Training of CSO and District Education Networks (DEN) representatives further empowered grassroots organizations. For Results Area Three (Decentralization of Policy Environment and Systems Improved), LGAP worked closely with some relevant national-level organizations to foster changes in decentralization policies. Important accomplishments included: work with district councils to update by-laws, work with the Ministry of Finance to develop manuals for district-level financial reporting, and devolution of human resources responsibilities to the local level, including training staff on use of HRMIS (Human Resources Management and Information System) and IFMIS (Integrated Financial Management Information System). However, there was incomplete coordination with all relevant national ministries and organizations, putting some of these gains at risk. The following recommendations are based on these findings: ● A continued focus on the LGAP financial interventions such as automation of financial records and reports through the IFMIS is critical, along with continued emphasis on broad participation in budgeting using the citizen budgets. ● LGAP should continue IT infrastructure development as a sustainability mechanism. USAID.GOV MALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 6 ● As discussed above, improved monitoring and evaluation (M&E) is needed in all districts (particularly in the education sector, where the most effort has been concentrated on service improvements). ● LGAP should continue training in the expansion districts. ● Retraining requires a train-the-trainer or mentoring approach by LGAP to enhance sustainability, with the engagement of national partners. ● LGAP and future programming should continue coordination across development partners and include new partners, such as CSOs. ● LGAP should continue assistance with by-law development in all districts and continue to expand its human resources initiatives to all districts. ● Engagement with national partners, continued partnership with those successfully engaged already, and developing engagement with relevant potential partners, as well as frequent communication with all local development partners, is key to successfully sustaining the success that has resulted from the first four years of the LGAP project. ● A sustainability and exit plan should be a clearly defined task for the LGAP program going forward. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 7 EVALUATION PURPOSE AND QUESTIONS The purpose of this mid-term performance evaluation of the LGAP program is to measure how the award has performed and guide USAID/Malawi regarding future programming. The evaluation team has identified the following five core questions to guide the evaluation: 1. How and to what extent has LGAP implementation increased Malawi Government accountability at the local level? 2. How and to what extent has LGAP implementation contributed to improved service delivery across sectors (health, education, and agriculture)? 3. What are any unintended consequences or externalities of LGAP support? How can these consequences be mitigated (if negative) or expanded (if positive)? 4. How has the LGAP award used the resources (financial, human resources/technical assistance) efficiently and effectively in relation to the planned activities, outputs and outcomes, and sustainability? 5. What lessons have been learned to inform future governance programming? In particular, how relevant are the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build the capacity for responsive, effective, and accountable service delivery through local governments in Malawi? These questions were informed by the desk review during the preparatory phase for this evaluation (available in Annex B—Desk Report). Following each of these core questions, the team has developed a set of sub-questions, which are also listed in Annex B. PROJECT BACKGROUND After operating without locally elected leaders for nearly a decade (Chasukwa et al. 2014), Malawi held tripartite elections (presidential, parliamentary, and ward councilor) in 2014 to reintroduce the election of local government leaders. Then the GOM launched several reform initiatives intended to accelerate and deepen an ongoing decentralization process. These reform initiatives created a unique policy window in which USAID/Malawi awarded the LGAP contract to build capacity for responsive, effective, and accountable service delivery through local governments in Malawi. To achieve this goal, LGAP strives to improve local government performance and transparency, increase citizen engagement, and strengthen the enabling environment for decentralization in Malawi. LGAP is a five-year initiative that runs from October 2016 through September 2021. Development Alternatives Incorporated (DAI), the implementing partner, received a contract that started at $23 million but increased to almost $40 million. In the first three years, LGAP provided support to eight district councils in Malawi. Districts included in the initial LGAP wave were: Balaka, Blantyre, Kasungu, Lilongwe, Machinga, Mulanje, M’mbelwa, and Zomba. In September 2019, LGAP expanded to support all 28 districts with additional funding from the United Kingdom’s FCDO. LGAP support is primarily provided via technical assistance to both national and subnational governments, supportive supervision for core local government operations, public financial management and accountability activities, capacity-building for councilors and District Executive Committee staff as well as ADCs and other grassroots committees, and support for integration with USAID activities. Awards are intended to achieve results in the following five areas: 1. Local government performance and transparency increased; 2. Demand for accountable government strengthened at the local level; USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 8 3. Decentralization policy environment and systems improved; 4. Public service performance management systems improved and implemented; 5. Public financial management related to human resources enhanced. LGAP activities build on the work of a previous democracy and governance initiative by USAID in Malawi in three districts, called the Malawi Electoral and Decentralization Activity (MEDA), which LGAP expanded to eight pilot districts, and eventually to all 28 districts. Earlier decentralization and local governance initiatives were instituted in the wake of multiparty reforms in 1994, but were regularly plagued by inconsistent, ad hoc implementation motivated more by strategic political calculation than development needs (Chinsinga 2008). An important context for LGAP’s work is the return to local elected government in 2014, with a new round of elections in 2019 in which new district councilors were elected (and some councilors re-elected). LGAP is structured around three primary Results Areas: (1) Local Government Performance and Transparency Increased; (2) Demand for Accountable Government Strengthened at the Local Level; and (3) Decentralization Policy Environment and Systems Improved. Each primary Result has several Sub-Results. These drive the activities of the LGAP program, which is multi-faceted and complex. As laid out in the FY 2020 LGAP Annual Report, the theory of change for LGAP activities is as follows: FIGURE 1: LGAP THEORY OF CHANGE There has been significant change in Malawi’s political context from 2014, when LGAP was designed, to 2020, when this evaluation took place. In 2014, Peter Mutharika had just been elected to the presidency, returning the Democratic Progressive Party as the ruling party after two years out of power. Peter Mutharika’s presidency continued a longer period of what political analyst Diana Cammack (2017) termed a “political settlement,” wherein Malawi’s elite would achieve a minimal level of economic performance and political stability in exchange for popular assent to less-than-democratic governance and stalled development. Mutharika was an unpopular president, having only won office with a minority of the vote in 2014 (Dulani and Dionne 2014) and again failing to win majority support in 2019 (Dionne and Dulani 2020a). Nearly a year of mass protests challenged, among other things, the veracity of the 2019 elections (Pensulo 2020). Those demonstrations, the court decision overturning the 2019 elections, and the subsequent election of opposition candidate Lazarus Chakwera offered a unique window of opportunity for governance interventions hoping to build an alternative to Malawi’s political settlement (Dionne and Dulani 2020b). The importance of Malawi’s political opening notwithstanding, scholarly research examining the impact of governance interventions encourages moderating expectations. Transparency and accountability interventions have provided mixed results, with some being highly effective and others having no impact (Kosack and Fung 2014). Often governance challenges require multiple, complementary interventions. For example, transparency interventions that sought to increase citizen knowledge and awareness did not subsequently motivate or inspire action among citizens to hold local governments accountable (see, e.g., Lieberman et al. 2014; Banerjee et al. 2010). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 9 Scholars and development practitioners are still working to design interventions that solve both supply- and demand-side challenges. A recent systematic review of participation, inclusion, transparency, and accountability (PITA) initiatives found that interventions aimed at promoting citizen engagement can stimulate citizen engagement in service delivery, thereby improving access to services and quality of service provision; however, without complementary interventions to address bottlenecks around service provider supply chains and service use, citizen engagement interventions alone may not improve economic or democratic development outcomes (Waddington et al. 2019). Perhaps most relevant to the outcomes of interest in this performance evaluation is the potential impact of an accountability program on service delivery. Rigorous research on transparency and accountability interventions conducted in low-income countries often shows that these interventions fail to significantly impact service delivery and wellbeing outcomes of intended beneficiaries. Although one study heralded as an exemplary model of transparency and accountability under challenging circumstances demonstrated powerful effects on health service delivery and health outcomes (Bjorkman and Svensson 2009), few others have shown similar results. Studies examining similar interventions in the same or similar settings and in the same sector have not found such significant results regarding health outcomes, and weak (or no) results on health service delivery (Raffler, Parkerson, and Posner 2019; Dionne and Camacho 2020). To be sure, some sectors may offer more potential for improvement in response to transparency and accountability interventions (Bauhr and Carlitz 2020) and, thus, it is important to capture information about multiple sectors. In sum, it remains important to modulate optimism about the impact of accountability programming on service delivery. METHODS AND LIMITATIONS This evaluation relies on a mixed-methods design to examine activity outcomes to date. The evaluation team used a combination of desk-based research, individual and group KIIs, and a survey conducted among LGAP beneficiaries. The evaluation team analyzed quantitative and qualitative data, cross-referencing findings against each other to enhance validity and to mitigate any methodological limitations of each method. BACKGROUND RESEARCH As part of the background research, the team conducted a desk review of relevant policy and background documents and considered prior evaluations, studies, and analysis, as well as relevant commentary and analysis. The team also reviewed program work plans, quarterly and annual reports, and M&E records. Please refer to Annex B—Desk Report for the full list of documents reviewed. DISTRICT SAMPLING Qualitative and quantitative data collection for the LGAP performance evaluation took place in 10 of Malawi’s 28 districts. Districts were stratified and then randomly selected to ensure representation across: 1. Wave of LGAP implementation (first or later), to capture the effects of LGAP programmatic decisions on outcomes; 2. Region (north, center, and south), to control for regional effects; and USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 10 3. Performance of local government (as measured in the Local Authority Performance Assessment [LAPA]), to control for “within group” characteristics that may affect LGAP outcomes. The final district sample (see Figure 2 below) includes four districts from LGAP’s first wave and six districts from the later implementation. Regionally, there are three districts from the north, three districts from the center, and four districts from the south. Finally, the evaluation includes six high-performing districts, according to LAPA score, and four low-performing districts. While we were limited by the few districts available to study and thus could not consider additional district characteristics in our research design (e.g., extent of local political unrest), the districts ultimately selected for study in the performance evaluation ranged in the degree of citizen political engagement. Figure 2: LGAP Midterm Evaluation Districts REGION LGAP 1ST WAVE LGAP LATER High LAPA Scores (2018– 2019) North M’mbelwa Nkhata Bay Center Lilongwe Dowa South Mulanje Mangochi Low LAPA Scores (2018– 2019) North - Karonga Center - Nkhotakota South Balaka Chikwawa USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 11 INTERVIEWS AND GROUP DISCUSSIONS The team conducted interviews and group discussions from October 26, 2020–January 13, 2021. There were 77 in-depth interviews and consultations with key informants and stakeholders, including the main stakeholder groups listed below. The interviewees were selected to represent the views of key players, stakeholders, and beneficiaries at all levels of governance. This list includes representatives of the following: ● ADCs, ● District Council Management, ● District Directors of Education, Youth, and Sports (DEYS), ● DENs, ● Council Service Committees, ● National Local Government Finance Committee (NLGFC), ● CSOs, ● Public Service Reform Management Unit, ● Ministry of Local Government and Rural Development (MLGRD), ● Ministry of Finance and Economic Planning, ● Ministry of Education, Science, and Technology, ● Malawi Local Government Association (MALGA), and ● Past and present management personnel from LGAP (USAID and DAI). The team also conducted small group discussions to obtain data on perceptions at specific group levels. Community-based group discussions also provided a better understanding of the normative perceptions of LGAP interventions and outcomes. The evaluation team targeted groups that have received specific LGAP interventions, such as CSOs that received support on advocacy interventions and village leaders in focus districts to understand the roles of ADCs and DENs. Group discussants were grouped by respondent type to ensure groups felt comfortable speaking openly. Accounting for both individual interviews and group discussions, our qualitative data include perspectives shared by 172 individuals. Please refer to Annex C—Qualitative Event Summary Tables for two tables that summarize the number of interviews and interviewees by type of interview (individual or group), location (district and national level), and type of respondent. An illustrative guide of themes these interviews and small group discussions covered, as well as an expanded discussion of the qualitative methodology, can be found in Annex B—Evaluation Desk Report. Tables summarizing findings from qualitative interviews are contained in Annex C. SURVEY Additional quantitative data were collected through a survey of intended beneficiaries in the same sample of 10 districts in order to reach a larger number of stakeholders and further address the research questions. A team of four enumerators conducted face-to-face survey interviews, capturing responses using computer-assisted personal interview (CAPI) software on mobile phones. The CAPI surveys provided efficiencies in post-survey tasks by reducing the time to aggregate, clean, and analyze data. The Cloudburst team trained enumerators via WhatsApp over a period of two days in late October. Enumerators then collected pilot survey data to test the CAPI software and survey instrument. The pilot data were examined to identify needed changes in the survey instrument, but were not USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 12 used in the analysis in this report. Survey data collection occurred over a period of almost six weeks, from October 30–December 8, 2020. The survey instrument is available in Annex B— Evaluation Desk Report. The population of interest in the survey was LGAP’s intended beneficiaries, including local officials, local civil society actors, as well as ordinary citizens – the latter also having a stake in good local governance, even if they may be indirect beneficiaries of LGAP. In each of the 10 districts covered by the performance evaluation, the survey team sampled participants from four primary sampling units: one survey site was the seat of district government (i.e., the boma); and the remaining three sites in each district were randomly selected using data from the 2018 Malawi Population and Housing Census. Enumerators conducted 16 interviews in each site, yielding a sample of 640 survey interviews. Teams were instructed to sample half of the survey respondents in each site (eight respondents) to represent the voices of local officials and civil society (e.g., village headmen, members of school management committees, faith-based organizations, etc.) and the other half (8 respondents) to be drawn from citizens resident in the survey site that were sampled using a random walk procedure, alternating the gender of the respondent (additional details are available in Annex D—LGAP Midterm Evaluation Survey Sampling Protocol). In some cases, the random walk procedure identified residents in the survey site who may have held a leadership or civil society role in the area. Listed below are the areas where surveys were conducted. SURVEY SITES BY DISTRICT: ● Balaka: Balaka Town, Traditional Authority (TA) Msalamala, TA Nkaya, TA Kalembo ● Chikwawa: Chikwawa Boma, TA Katunga, TA Lundu, TA Makhwira ● Dowa: Dowa Boma, TA Chakhaza, TA Chiwere, TA Msakambewa ● Karonga: Karonga Town, TA Kilipula, TA Kyungu, TA Wasambo ● Lilongwe: Area 7 (City), TA Chimutu, TA Kabudula, TA Kalolo ● Mangochi: Mangochi Town, TA Chilipa, TA Jalasi, TA Katuli ● M’mbelwa: Mzimba Boma, TA Jaravikuba Munthali, TA Kampingo Sibande, TA Khosolo Gwaza Jere ● Mulanje: Mulanje Boma, TA Chikumbu, TA Nkanda, TA Nthiramanja ● Nkhata Bay: Nkhata Bay Boma, TA Kabunduli, TA Malanda, TA Malenga Mzoma ● Nkhotakota: Nkhotakota Boma, TA Kanyenda, TA Malenga Chanzi, TA Mwadzama SURVEY RESPONDENT CHARACTERISTICS The team conducted 640 survey interviews; of the 320 interviews targeting citizens resident in the survey sites listed above, some respondents sampled in the random walk procedure identified as local community leaders. Citizens’ understanding of being a local community leader was broad and included, for example, Parent-Teacher Association (PTA) members, faith leaders, and forest committee members. When we refer to the “citizen subsample” in the forthcoming analysis, we are referring to the 244 respondents (of the sample of 640) who did not identify as holding any position of influence in their community. Most (73 percent) of our survey respondents resided in rural areas. While men make up the majority (62 percent) of our respondents in the survey sample, among the citizen subsample, women make up half of the respondents. All of Malawi’s major ethnic and religious groups are represented in the survey sample, though not in the exact proportions that they make up in Malawian society. While the largest ethnic groups represented are Chewa (24 percent), Yao (17 percent), and Tumbuka (13 percent), Lomwe (9 percent) were a much smaller proportion and USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 13 Tonga (10 percent) were a much larger proportion when compared to Malawi’s 2018 Population and Housing Census. Likewise, while Christians make up the majority of our survey sample (as they do in Malawian society), the exact proportions of Church of Central Africa Presbyterian (CCAP) (23 percent) and Roman Catholic (15 percent) did not match their respective proportions in the latest census, and Muslims (18 percent) were overrepresented in our sample. Our survey sample had achieved greater educational attainment than the sample of Malawi’s most recent Demographic and Health Survey (DHS) in 2016, with only 7 percent of the citizen subsample (4 percent men and 9 percent women) reporting having never attended school; only 5 percent of women in the citizen subsample had completed secondary school, which is consistent with the 2016 DHS, but almost 20 percent of men in the citizen sample reported completing secondary school, which is more than double the proportion in the 2016 DHS. These proportions are higher among the broader survey population, as the respondents in the “leader” subsample had higher educational attainment (32 percent of men and 11 percent of women reported having completed secondary school). Survey respondents struggled in the 12 months prior to the survey to meet their basic needs, especially when compared to the most recent wave of the nationally representative Afrobarometer survey. While 53 percent of Malawians surveyed by Afrobarometer in 2018 reported having gone without a cash income either many times or always in the preceding 12 months, that proportion was higher (66 percent) in our survey sample, and even higher (77 percent) in the citizen subsample. We surmise the survey data collected for this performance evaluation reflected a broader economic slowdown in the country made acute by the COVID-19 pandemic. We use questions about meeting basic needs (access to water, food, cooking fuel, essential medicines, and cash income) to create a “lived poverty index” typically used in analyses of survey data collected in African countries in the absence of reliable data on wealth and income (Mattes 2008). Table A.10 provides descriptive statistics comparing the survey sample to the 2018 census and 2016 DHS.2 DATA ANALYSIS After the creation of a cleaned data set, the evaluation team used multiple techniques to obtain findings relevant to the evaluation questions. For the survey data, the team examined descriptive statistics (e.g., means, crosstabs) to obtain statistical profiles of the samples and to identify significant results. To try and identify potential trends or trajectories, the analysis includes comparisons between the survey data collected for this performance evaluation and data collected through the Citizen Perception Survey (CPS) conducted by GeoPoll in 2017.3 When appropriate, the team used more rigorous methods such as conducting t-tests or regression analysis. Multiple regression allowed us to estimate how multiple factors (simultaneously) correlate with the outcomes of interest. For the qualitative data collected through interviews and group discussions, the team organized data into a structured set and coded data to identify common responses and significant themes to answer the evaluation questions. The evaluation team drew from these data patterns as well 2 Of course, there are sampling and reporting differences between the survey conducted for this performance evaluation and both the 2018 census and 2016 DHS. The 2018 census includes the entire population of Malawi, figures reported by the DHS include Malawians aged 15–49 selected in a nationally representative sample, and the survey details reported here are based on a 10-district sample of adult Malawians, aged 18 years or older. When reporting, for example, marital status of the population, the 2018 census includes all Malawians aged 12 years and older. 3 From April 13 through April 25, 2017, GeoPoll conducted 12,000 surveys via short message service in the eight early-wave LGAP districts. These surveys measured citizen satisfaction with decentralized public service and many of the questions appearing on the CPS were repeated in the survey collected for this performance evaluation. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 14 as illustrative quotes and organized findings in tabular structures. These tables are contained in Annex E—Survey Summary Tables. For both qualitative and quantitative data, the team disaggregated findings along relevant dimensions such as gender, location, local affiliation, and assistance received to conduct subgroup analyses and identify significant outcomes among the beneficiaries. ETHICAL CONSIDERATIONS The evaluation followed all ethical practices for human subject research, protecting respondent confidentiality and sensitive information. This included applying for and receiving research ethics and regulatory approval for the study from the primary Institutional Review Board (IRB) in Malawi, the National Committee on Research in the Social Sciences and Humanities (NCRSH), part of the National Commission for Science and Technology (PROTOCOL NO. P.10/20/524) prior to commencing data collection. For in-person fieldwork, all evaluation team members followed current health promotion guidelines on COVID-19. Evaluation team members and participants in the KIIs, interviews, and in-person surveys were instructed to wear masks, maintain two meters of distance, and conduct all interaction outdoors whenever possible. The evaluation team made a sanitizing station available for arriving participants, provided masks to all participants, and advised local stakeholders and participants in advance of precautions in place to protect the participants and researchers. LANGUAGE The English versions of the data collection resources were translated into Chichewa and Chitumbuka, and interviews, discussions, and the survey were conducted in the interviewee’s preferred language (Chichewa, Chitumbuka, or English). Most survey interviews were conducted in Chichewa (89 percent) and the key survey results presented in the analysis did not differ based on interview language. TEAM The evaluation team consisted of Kim Yi Dionne (International Academic Expert); Chris Phiri (Local Capacity Development Expert); Embry Howell (Senior Evaluation Specialist); Kate Marple￾Cantrell, (Tasking Coordinator & Evaluation Specialist); Mordechai Wellish (Research Assistant); and Bright Mleta, Sandra Latif, Eliphaz Kuseni, and Hastings Nthungo (Enumerators/Research Assistants). RISKS & LIMITATIONS The evaluation team also considered several budget and management risks when planning work during the COVID-19 pandemic. The evaluation team closely tracked the changing nature of the local epidemic when coordinating travel and in-person meetings and modified plans according to the latest conditions. The timing of fieldwork and the nature of Malawi’s experience with COVID￾19 allowed for face-to-face interviews, with the exception of a few interviews conducted in early January. The team conducted all survey fieldwork while Malawi was experiencing very few COVID-19 cases and enumerators and respondents adhered to COVID-19 protocols, even when infection rates were very low. While the sampling approach captures beneficiary characteristics based on the LGAP phases, regions, and district performance, the selection of the 10 districts was still based on a purposive approach. The purposeful selection of districts, as opposed to random selection, could have USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 15 introduced bias into the data on measurable results. Therefore, the qualitative and quantitative findings can speak for the sampled group but will not represent a broad range of the Malawian population. The high turnover of personnel in local government offices raised the possibility that the evaluation team would be unable to conduct interviews, discussions, or surveys with LGAP beneficiaries, particularly those who participated in the activity’s early stages. The evaluation team incorporated techniques to address this during the data collection phase, such as applying snowball sampling and requesting interviewed beneficiaries to refer the team to other possible respondents or give their contact information. While the research team had hoped to interview more members of parliament (MPs) and District Commissioners, due to the challenges of scheduling interviews with these stakeholders, we only managed one MP interview and two interviews with District Commissioners. A limitation of the methodology used for this performance evaluation is that findings are often based on stakeholder perceptions, rather than on objective measures. For example, some individuals interviewed were not able to provide data to support their views. What is more, interviewees and survey respondents may not always be forthcoming with their honest views if they feel unsafe in doing so or may respond out of self-interest or tell the evaluators what they think the evaluators want to hear. The ET made every effort to mitigate this risk by triangulating findings with alternative sources of information and interviewing stakeholders with varying motivations and perceptions, establishing rapport with interviewees, and clearly communicating the anonymity of interviews and the survey. The ET found no evidence to suggest that this biased results. RESEARCH QUESTION 1 FINDINGS: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION INCREASED MALAWI GOVERNMENT ACCOUNTABILITY AT THE LOCAL LEVEL? LGAP employs a “three-pronged approach to advocate for, and operationalize, sustainable governance reform—through district council functionality, demand for accountability, and an enabling policy environment.”4 This work is hypothesized to promote greater government accountability as a short-term outcome. Our evaluation approach to understanding the extent to which LGAP implementation increased Malawi government accountability at the local level relied on both qualitative and quantitative data, including comparing our survey data with previously collected survey data. We examined citizen satisfaction with district council response to community needs, trust in public officials, awareness of and engagement in participatory planning and external oversight processes, and transparency and consultation initiatives. In this section, we report on these data measuring accountability issues and identify the factors contributing to LGAP achievements, as well as obstacles confronted. 4 2020 Annual Report, p. 6. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 16 LGAP ACTIVITIES TO INCREASE ACCOUNTABILITY In order to foster accountability, LGAP takes both a top-down approach, with efforts to improve the functioning of government through training and mentoring of elected district councilors, service committees, and staff (Results Area 1), as well as a bottom-up approach, with efforts to empower local citizens to make elected officials and staff accountable (Results Area 2). Engagement with citizens is accomplished through work with ADCs and other citizen groups, such as CSOs/non￾governmental organizations (NGOs) and school committees. While VDCs were considered important as well, resources were not sufficient to work directly with them. LGAP complements these local efforts with work on by-law development and strengthening the enabling environment through work with national actors (Results Area 3). The LAPA was a pre-existing GOM tool that LGAP adopted as a primary data source for M&E because it measures district performance across a variety of functional areas including planning, budgeting, human resources management, and sector service delivery. The LAPA scores for the initial eight first-wave districts reflected a low level of local government performance at the program’s outset. In 2016/2017, none of the scores in the eight first-wave districts were above 2 (“Average Performance”) for the four categories of Planning, Human Resources, Public Financial Management, or Governance, save one in Kasungu District, which scored 3 (“Above Average Performance”) for Human Resources. Adding to the challenges, many citizen committees—for example, ADCs and VDCs—were reported by interviewees as inactive when LGAP began its work. LGAP annual reports document the activities it undertook to foster accountability, and these reports are reinforced by accounts of individuals interviewed during this evaluation (see Table A.3). When asked about the most significant LGAP activities, interviewees mentioned several under Results Areas 1 and 2. Just over half (43) of those regarded some aspect of the trainings that LGAP provided to District development partners (ADCs, councils and staff, CSOs, service/market/school committees, members of parliament [MPs]). The activity was designed with the intention of “Smart” capacity building5 and problem-solving approaches that deemphasize traditional training. Those we interviewed reported that the most important topics of the trainings were as follows: governance (how to conduct meetings and produce minutes); roles and responsibilities of different development partners; financial management; human resources management; ADC empowerment; and other aspects of accountability. These trainings began in the initial eight districts. For example, by the end of FY 2018, LGAP reported that they had completed trainings of the ADCs in all eight pilot districts. The FY 2019 Annual Report gave examples of the number of other trainings provided in three of the eight pilot districts. For example, in that year in M’mbelwa district, LGAP trained 22 councilors, 5 MPs, 6 TAs, 5 representatives of interest groups, and 13 council staff members, for a total of 51 individuals. Patterns were similar in the other two districts. Trainings did not begin in the additional 20 later-wave LGAP districts until late 2019 and then were delayed for several months by the COVID-19 pandemic. By the time our in-depth interviews 5 “Smart” approaches mean designing programs that extend beyond “business as usual.” “Smart” capacity building requires innovative and/or evidence-based methods to strengthen organizational and individual capacity without relying entirely on traditional training approaches. An over-reliance on group training events has questionable effectiveness in building and sustaining capacity. Utilizing the various field offices, a creative, “smart” approach to capacity development includes a multipronged approach that is thoughtful, targeted, and sustainable and does not come at the expense of other services. Approached must extend beyond one-time off-site trainings. In addition, “smart” problem solving involves supporting districts to identify priority gaps so that the Contractor can tailor interventions that have the highest impact and provide feedback to both GOM and USAID partners to address challenges outside of the Activity scope. This approach will respond to the greatest needs of a district and harness the power of innovative technology to maximize the effectiveness, efficiency, sustainability and scale-up-ability of interventions. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 17 began in November 2020, most LGAP trainings had not yet been completed in the later-wave LGAP districts covered by this performance evaluation. Interviewees also highlighted activities LGAP undertook to foster financial accountability, such as training of district staff in financial management and particularly in adopting the IFMIS. As described in the FY 2019 annual report, LGAP used a coaching, mentoring, and peer-to-peer approach in IFMIS training. In FY 2020, LGAP trained 359 council staff in using IFMIS. As of the end of FY 2020, 20 of 28 districts had produced draft year-end financial statements using IFMIS. In addition, by FY 2020 LGAP was working in all districts to develop “citizen budgets,” which are documents understandable to a layperson audience that tracks budgets and expenditures for use by councils and ADCs. Market revenue collectors were also supplied with handheld devices that were designed to improve market revenue collection by generating receipts to prevent the issuance of fake receipts and providing real-time data to the district council so underreporting collections was also automatically detected. Another accountability intervention that our interviewees frequently cited as important was LGAP’s intervention in the education sector. LGAP began this work in late 2019 in the 20 later wave districts, so this work also was delayed due to the pandemic. DENs were revitalized, engaged, and funded to travel to schools in their district and meet with PTAs, mothers’ groups (MGs), school governance committees, and teachers, as well as carry out additional duties for which they are responsible. They then developed social contracts signed by all partners. These are documents used to hold teachers, parents, and pupils accountable. They track how funds are spent and stipulate key activities that schools require of communities in support of pupils’ learning such as making sure that pupils are ready to learn and available for classes, have needed supplies, and complete academic home assignments. The contracts also address areas that communities would like the Senior Primary School Management Team to be responsible for such as increasing teacher-to-pupil contact time and ensuring that schools track curriculum implementation and learners’ outcomes consistently improve across all grades. However, since February 2020, due to the coronavirus pandemic, plans (including monitoring of implementation of the social contracts) were put on hold until the operating environment improved (it is also universally acknowledged that it is too early to talk about improved outcomes). Another LGAP school accountability activity was related to identifying “ghost workers” (persons on payroll who are not actually teaching), and to rationalizing the allocation of teachers and pay to improve productivity and retention. LGAP also provided technical assistance to districts for the development of Socio-Economic Profiles (SEPs), District Development Plans (DDPs), and by-laws. The creation of SEPs and DDPs is part of the devolution of planning and funding development projects to the districts from the national government, and the two documents go hand-in-hand. The SEP acts as a source of baseline data for the design of project proposals that are incorporated in annual investment plans. It also provides an overview of the general physical, social, and economic situation in the district. The DDP lays out a plan for development projects in the district, in order to coordinate planning across funders. One interviewee stated that this was very important work of the LGAP. LGAP helped to finalize the SEP and DDP, including quality control and technical assistance. They attended District council meetings related to the SEP and 5-year DDP. Local leaders now know what is in the DDP and will be able to track project implementation, leading to more accountability. (District Councilor) LGAP activities also sought to improve communication between development partners by sponsoring meetings that bring those partners together. Through these meetings, partners USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 18 learned of each other’s initiatives and the progress of all development projects in the district. Below is one example that an interviewee cited concerning linkages between CSOs and the District Council. Integration meetings helped to group all the CSOs in the district into thematic areas…So there is improved two-way communication between the CSOs and the government. (District Councilor) To increase citizen engagement with local government under Results Area 2, LGAP engaged in outreach using mobile phone messages, radio spots, and billboards. The purpose of these activities was to empower citizens and educate them about local government and the elections that were being held. LGAP reported reaching over a million people in this way. Under Results Area 3, LGAP worked with national ministries and agencies to improve resource management. For example, LGAP supported the Ministry of Finance in developing three manuals to be used in financial management at the district level: an accounting procedure manual, an International Public Sector Accounting Standards manual, and an audit policy manual. Likewise, in improving human resources (HR) management, LGAP developed five HR manuals in consultation with national HR officials, and provided technical assistance in updating personnel files, conducting payroll audits, reallocating staff, and revamping the Human Resources Management and Information System (HRMIS). These were important efforts in the devolution of HR efforts and payroll. Finally, LGAP also worked with the Ministry of Education to develop a Teacher Performance Assessment System (TPAS), revamp the school inspection system to uphold national standards, and develop professional certification for teachers. This national-level work also includes assistance to the MLGRD and GOM on key legislative reforms to further decentralization. Specifically, LGAP supported efforts to revise the Local Government Act (LGA), which currently does not mention ADCs/VDCs, by developing briefing notes and convening consultations with the MLGRD. This includes facilitating local consultations on these issues through local structures (ADCs and VDCs) and working with the National Advocacy Platform (NAP) consortium of CSOs to gather input from organizations across the country. These proposals and comments will be submitted to the Office of the President and Cabinet (OPC) during the legislative drafting process. LGAP has also engaged MPs directly on these issues. FINDINGS ON ACCOUNTABILITY Our survey data offer a few measures of citizen reports of their engagement in holding public officials accountable; we then compared districts with earlier and later LGAP implementation to provide evidence as to LGAP’s impact on citizen engagement in this area. We asked citizens how satisfied they were with how their councilor consults citizens and just over 40 percent reported being satisfied or very satisfied while another 40 percent reported being dissatisfied or very dissatisfied (the remainder were neutral or did not say). There were no statistically significant differences between districts with earlier and later LGAP implementation. Focusing only on those districts with earlier LGAP implementation, a comparison to data collected via citizen participation surveys in 2017 suggests some districts have demonstrated increased satisfaction with councilor consultation, but this was not universal. Table A.11 shows increases in citizen satisfaction between 2017 and 2020 in Lilongwe and Mulanje, but mixed results in Balaka and M’belwa. Overall, citizens in evaluation districts were largely dissatisfied with how their district council responds to community needs (62 percent dissatisfied or very dissatisfied) and with the quality of projects implemented by their district council (53 percent dissatisfied or very dissatisfied). There were no statistically significant differences in these measures between districts with earlier and USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 19 later LGAP implementation. Focusing only on those districts with earlier LGAP implementation, a comparison to data collected via citizen participation surveys in 2017 suggests citizens are generally more dissatisfied today than in 2017 when asked about council responsiveness and projects (see Table A.12 and Table A.13). Trust of public officials is generally low in Malawi, and our survey sample is not exceptional regarding these measures. Respondents were asked how much they trusted their MP, their district council, their traditional leader, CSOs, and their VDC (Figure 3, below). Of these, traditional leaders and VDCs had the most citizen trust, with 65 percent of respondents saying they trust traditional leaders “a lot” and 64 percent saying they trust their VDCs “a lot.” Most challenging for LGAP and any future local governance programming in Malawi is the relatively low trust that citizens have in district councils. Figure 3: Citizen Trust in MPs, District Councils, Traditional Leaders, CSOs, and VDCs The difference in the average response to trusting district councils by LGAP wave does not meet conventional standards of statistical significance, but a graphical comparison suggests that districts that implemented LGAP earlier (and to a greater extent) have greater citizen trust in their councils (Figure 4, below). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 20 Figure 4: Citizen Trust in District Councils by LGAP Wave To measure citizens’ awareness of and participation in participatory planning and external oversight processes, we asked a series of questions about their perceived access to information and their recent participation in community meetings or trainings. Many survey respondents (59 percent) report being dissatisfied or very dissatisfied with access to information from their district council; however, there was some important variation in how they characterized the likelihood they would be able to access certain public information. For example, while half of our survey respondents thought it would be somewhat likely or very likely that they could find out what the local school’s budget was and how funds have been used, only 20 percent thought it was somewhat or very likely that they could find out about the district development plan and budgets if they went to their district council office. Slightly more (27 percent) said it was somewhat or very likely that they could go to the district council to find out who owns a piece of land in their community. There were no statistically significant differences on any of these four measures of citizen access to information related to timing of LGAP programming or to respondent gender. However, respondents in the citizen subsample reported more dissatisfaction with access to information from their district council and even lower likelihood that they would be able to access school budgets, land ownership information, and district development plans; all of these differences were statistically significant. Our survey focused on six different participatory activities: community meetings to discuss the village action plan, district meetings to discuss the district development plan, community meetings to discuss social contracts between the primary school and the community, community meetings to discuss collection and spending of market fees, trainings in the district on managing community resources, and trainings in the district on recordkeeping. Survey respondents reported high participation in community meetings on school social contracts (84 percent) and village action plans (66 percent), but very low participation in meetings about market fees (11 percent) or district development plans (11 percent), which were even lower than reported participation in training activities on managing community resources (22 percent) and recordkeeping (18 percent). There were no statistically significant differences in any of these measures of civic participation related to timing of LGAP programming or respondent gender. However, respondents in the citizen subsample participated at a lower rate for all of these civic activities and all of these differences USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 21 were statistically significant (see Table A.14). Using multiple regression analysis that contains controls for respondent gender, LGAP wave, and the lived poverty index, we find three statistically significant factors associated with civic engagement: respondents in the citizen subsample engage less, more educated respondents engage more, and respondents with higher trust in public officials engage more. In our open-ended interviews, we asked interviewees about perceived effects of LGAP’s activities on district government accountability and learned of numerous ways that LGAP has had such effects on specific areas of accountability (see Table A.4). In approximately equal numbers, interviewees cited: (1) improved accountability through coordination of development activities, (2) improved financial accountability, and (3) improved education accountability as being important effects. Improvement in HR management was also mentioned, though less often. Communication: We learned that ADCs, market committees, and councils are now, following LGAP training, feeling empowered to ask questions about the budget and development progress. As the secretariat, we are now on our toes. There are people who are following what we are doing. It has improved checks and balances. (District Management Team) We were told that both the councilors and district staff (often referred to by interviewees as “the secretariat”) are more responsive to community input and that there is improved communication between partners. Lately, the secretariat has been frantically trying to complete projects funded from the DDF that stalled for some time. This is attributed to the fact that the ADCs are now beginning to follow up on development projects in their area with a newly found sense of ownership and authority. (District Councilor) Financial Accountability: In the area of improved financial accountability, we learned that there is broader participation in planning and budgeting, and that the citizen budget is an important tool in tracking the use of funds. In particular, the Constituency Development Fund (CDF) is a potential source of disagreement concerning development. The MPs make it abundantly clear that the CDF is a no-go zone for the councilors. (District Councilor) However, we were told that some citizens now are more involved in examining that funding, which has been under the purview of MPs. ADCs now have a better understanding of projects in their localities (for example, the CDF projects). When an official has misused funds, he is taken to task. Some MPs have been arrested for this, and one is behind bars. (MP) Improved financial management has led to more timely reporting, which provides a tool for councilors, ADC members, and CSOs to question the use of funds and the progress of development. Until LGAP came to work with the council, getting details about funding for the various sectors was impossible and a very emotive subject. (CSO/DEN member) This increased financial oversight has apparently led to improved accountability, to the point of— in some cases—the involvement of law enforcement. People at the secretariat apparently swindled some money. The communities mobilized and demanded the removal of the directors implicated. (CSO/DEN member) However, interviewees expressed the view that there is more work to be done. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 22 LGAP capacity-building empowered the councilors to be aware of their oversight role; however, the majority of the councilors are still fearful of challenging MPs and the secretariat. (District Councilor) There are reports of gross misuse of funds at the district council, leading to questions about whether LGAP has yet made a difference. (CSO Network) Finally, in the area of financial accountability, respondents indicated that LGAP has improved the security of market revenue collection through mobile phone financial transfers. Consequently, local revenue collection has increased in some districts, although it remains relatively low compared to national revenue transfers and development funding. Education Accountability: The LGAP education work, through training school committees and developing social contracts, was mentioned frequently as a source of increased accountability in schools, especially in the eight first-wave districts. The social contracts between local school governance structures and teachers is leading to improved learning and teaching outcomes. (DEYS official) When people know their roles, they have made the teachers more accountable. They are talking to the teachers, saying ‘We expect you to do ABCD.’ It gives the learners a conviction that their parents are supporting them, and it gives the school personnel the feeling that the community supports them. (DEYS official) Teacher absenteeism is now minimal, and complaints against teachers’ conduct and behavior has been drastically reduced. (DEYS official) Other outcomes mentioned by interviewees include reduced delay in salaries, improved allocation of teachers, and school committee involvement in tracking school improvement funds. LGAP support also has helped revive an inspection system for schools. Learner achievement can only be enhanced if there is a system of monitoring performance, which leads to accountability. Where there is an inspector, there is increased accountability. The inspectors become role models, because of the high standards they hold. (National official) Overall, by 2018/2019, the LAPA scores in the eight first-wave districts had all improved relative to their respective earlier scores, although none scored 4 (“Above Average Performance”) in any of the four categories, and all scored either 1 (“Needs Improvement”) or 2 (“Average Performance”) in the Governance category. Thus, the LGAP activities appear to have helped in these key governance areas, but still with substantial need for more improvement. In our survey, we analyzed open-ended responses to the question “What are the most important improvements needed in local government in your district?”.6 This question was asked early in the fifth year of LGAP, so it reflects the opinions of citizens in two pilot districts where LGAP has been in operation for four years, and four new districts where LGAP has been in operation for only a year (with COVID-19 interruptions). Overall, citizens in the two original districts appear to note better functioning government than citizens in the four newer districts, although there is room for improvement in both (see Table A.5). The perceived need for improved government functioning spanned the range of problems that LGAP has targeted, with notably more mentions (proportionally) in the four new districts. Citizens 6 The 326 citizens in these six districts mentioned 398 improvements that are needed (some citizens mentioned more than one improvement). About three-fourths (77.6 percent) of mentions concerned improvements needed in government functioning, with 8.6 percent mentioning improvements needed to LGAP specifically (particularly the need for more frequent training), and the remainder citing critical development needs not being met in their community. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 23 mentioned needs for: (1) Improved community engagement in development, (2) improved coordination between development partners, (3) less corruption, (4) more monitoring and reporting on progress, (5) less politicization of development funding, and (6) improved quality of staff and officials. This indicates that, while there has been progress, there is still much room for improvement and continued intervention, especially in the later LGAP wave districts. Most of the mentions of the need for more LGAP trainings (25 of 28) were in the later LGAP wave districts, reflecting a prevalent view among citizens there and perhaps a lack of exposure given how new and relatively less substantial LGAP implementation has been in those districts. The responses reflect some disconnect between citizens at the village level and their district governments. Below are some direct quotes from the survey data concerning some of these issues. I am the chair of the ADC, one of the signatories of the Community Development Fund. However, the funds are being released by the district council without my approval. (Male in rural area of expansion district) Members of parliament and the ward councilor have never visited the area since the general elections in 2019. (Female in rural area of expansion district) LGAP should follow up on a project they started at the market. The project has stopped. It may be that someone stole the money meant for the project. (Male in urban area of pilot district) The authorities who receive development funds are corrupt. (Female in urban area of expansion district) Educated people must be elected to the ADCs for easy understanding and taking those in authority to task. (Male in rural area of expansion district) The ADC has not yet been trained. We need to be trained to know how to work with politicians such as members of parliament. (Male in rural area of pilot district) Below are two quotes from citizens reflecting this sentiment when asked for feedback in the evaluation survey: I am hoping for a change, especially for the end of corruption and greed at the [District] Council. (Male in urban area of expansion district) Development in this area is not working. Many things are at a standstill. (Female in rural area of expansion district) CONCLUSIONS AND RECOMMENDATIONS ON ACCOUNTABILITY The survey data illustrate high participation and engagement at the local level (e.g., in school social contract meetings, and meetings about village action plans) and suggest citizens are eager to participate in development activities in which they will be heard. Engagement is greater among citizens who have higher trust in public officials; additionally, the survey data suggest citizens of earlier-wave LGAP districts may have greater trust in district councils than later-wave LGAP districts. Taking these findings together, we might expect that continued LGAP programming (or expanded LGAP programming or programming similar to LGAP in promoting local accountability) could build trust in district councils, which is quite low. That increased trust could then motivate citizens to engage more in district development planning, creating a virtuous cycle of local accountability. To be sure, we must also consider that extending or expanding local governance and accountability programming will not ultimately improve the outcomes donors, officials, and citizens care about, especially when faced with data suggesting a limited impact on outcomes to date. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 24 The prudent way forward would be to consider continued support for programming that has demonstrated success and/or will provide a foundation to build on in the future. For example, the IFMIS trainings were not just popular, but also built a foundation on which local councils can modestly improve revenue generation. More specifically, it was reported in the KIIs that financial technology improvements enhanced local revenue collection. While small in comparison to central government revenue transfers to districts, this local revenue collection would be independent of the whims of national politics and informed and overseen by local civil society and citizens. While perhaps designed to be wholly “smart” capacity building activities, LGAP also offered substantial traditional trainings and technical assistance in multiple districts. Doing so seemed appropriate for, initial IFMIS training for example, especially given the relative success of such activities. There was also more sustainable activity related to “smart” capacity building, including the work with the Ministry of Finance to develop training manuals, which can be utilized (and expanded) even after LGAP activities end. Other “smart” capacity building efforts included mentoring, which were mentioned by respondents but with less frequency than training activities. Trainings and technical assistance related to financial management were highlighted as important and their impact will have follow-on effects through a potential increase in local resources, as a result of improved financial management. To realize these benefits, a future USAID activity should continue this focus on financial management training, with the goal of ensuring that financial management practices are universally adopted across Malawi’s 28 districts. Additionally, future work should include institutionalizing training responsibilities within the national government and building national government capacity and planning for ongoing retraining. The findings do not suggest, however, an endorsement of further decentralized training activities, even if these were mentioned by multiple stakeholders interviewed in the course of this performance evaluation. While interview respondents pointed to the lack of training of VDCs, for example, the survey data show these trainings lacked significant local-level citizen engagement. There are other potential tools to build on existing citizen engagement, such as the citizen budget. If additional trainings are undertaken, stronger monitoring and evaluation of these activities would provide more insight into the efficacy of such trainings for improving government accountability. Pre- and post-training surveys, including three, six, and 12 months after the training, would provide evidence of whether and how government functioning had changed and attribute that to a specific training. The citizen budget was a popular information tool that could be disseminated to and used by a broader audience, since it made financial information understandable to a broader audience such as councilors and ADC members. Future activities should support local government initiatives to share the citizen budget more widely. Districts or development partners could draw upon Malawi’s recent successful experience disseminating reliable information about COVID-19 through new media technologies, especially the Malawi Ministry of Health’s use of WhatsApp. LGAP or its successor could facilitate local government partnerships with CSOs, including community-based organizations, in the production and dissemination of citizen budgets. Given the relatively low likelihood that survey respondents reported they could access district development plans, new methods to disseminate the citizen budget tool could also serve as a model for initiatives to increase access to DDPs as well. Accountability activities in the education sector—especially the HR automation tool—were also highlighted and could serve as models for other sectors. Perhaps what is most appropriate or urgent is improving information and accountability in Malawi’s health sector, which suffers from a serious HR shortage (Palmer 2006) and limited training at the facility level in health information management systems (O’Hagan et al. 2017). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 25 Achieving full accountability requires more than the capacity of local governments to be accountable and citizen capacity to hold them accountable. The long history of lack of government accountability in Malawi will require sustained action to reverse. Even if the aforementioned recommendations are adopted and donors and government seek to expand or enhance accountability initiatives, the scholarly literature referenced above suggests that improving other important outcomes—in the words of LGAP’s theory of change, “improved quality of life for Malawians”—will also require complementary interventions to address bottlenecks in service provider supply chains and service use. We thus turn to service delivery in the next research question. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 26 RESEARCH QUESTION 2 FINDINGS: HOW AND TO WHAT EXTENT HAS LGAP IMPLEMENTATION CONTRIBUTED TO IMPROVED SERVICE DELIVERY ACROSS SECTORS (HEALTH, EDUCATION, AND AGRICULTURE)? LGAP’s theory of change proposes that its combined top-down and bottom-up approach to fostering government accountability will eventually (in the longer term) lead to Malawians experiencing more effective service delivery in line with community needs. However, as discussed above, expectations about LGAP having an impact on service delivery should be tempered given the lack of robust research showing a consistent impact of governance interventions on public service provision and development outcomes. Nevertheless, we collected both qualitative and quantitative data to measure citizen perceptions of and experience with public service delivery and what kinds of assistance or activities initiated or supported by LGAP may be responsible for any improvements in service delivery. FINDINGS ON SERVICE DELIVERY Our surveys asked respondents to evaluate service delivery in their communities broadly and in specific sectors. First, we asked respondents to assess whether service delivery had improved in the last five years, and 41 percent said government services have gotten worse or even much worse, 22 percent rated government services as the same, and 36 percent said government services have gotten better or much better.7 Survey respondents living in districts with earlier and more intensive LGAP activities had marginally more positive assessments of government service delivery, but the differences do not meet conventional standards for statistical significance (in a two-tailed t-test, p=0.10). It is possible an evaluation with greater statistical power (e.g., more survey respondents and/or better balance across LGAP waves) would result in more meaningful differences between districts in different LGAP waves. To measure perceptions of service delivery, we asked respondents about specific characteristics in the major service sectors that are important proxy measures for those respective sectors. For example, in education, we asked respondents to assess teacher availability and attendance as well as the materials available for learning. Likewise, in the health sector, we asked about health worker behavior and medicine availability. The data show significant areas requiring improvement. Perhaps more important, however, is that citizens had distinct views about human resources and material resources. More specifically, citizens had more satisfactory views of health worker, health committee, teacher, and PTA behaviors compared to their views on the availability of material resources like medicines and learning materials. Combined in Figure 5, below, are simple dotplots of respondents’ reported satisfaction with different service characteristics. In the health sector, 48 percent of respondents report being dissatisfied with health worker behavior, but even more (67 percent) are dissatisfied with the availability of medicines in their local healthcare facility. Importantly, only 30 percent of respondents are dissatisfied with the work of their local health advisory committee. Likewise, in the education sector, only 21 percent of survey respondents reported dissatisfaction with the work of their PTA, while 38 percent reported dissatisfaction with teacher availability, and 77 percent reported dissatisfaction with the availability of materials for teaching and learning at their local 7 One percent indicated that they do not know whether service delivery had improved in the last five years. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 27 school. In sum, citizens are more satisfied with characteristics related to human resources and less satisfied with characteristics related to material resources. We asked two questions about the agricultural sector—Malawi’s economic engine—and we see relatively more satisfaction, at least in assessing agricultural extension efforts (a second survey question we asked related to agriculture was about the distribution of fertilizer subsidy coupons, but given serious political challenges in the implementation of the fertilizer subsidy program ongoing during survey fieldwork, we set aside any analysis of data on that question). Figure 5: Service Delivery Satisfaction by Topic USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 28 Examining survey respondents’ evaluations of these service delivery characteristics, we did not find statistically significant differences between respondents in early and late LGAP districts, with one exception: the average satisfaction with availability of materials for teaching and learning was higher in early LGAP districts (in a two-tailed t-test, p=0.015), which have received more sustained support. To be sure, dissatisfaction is still high, though the degree of dissatisfaction is lower and the proportion of respondents satisfied with the learning materials available at their area school is also higher in early LGAP districts. There were no statistically significant differences in satisfaction with these service delivery characteristics based on gender. There were, however, statistically significant differences among the citizen subsample, who expressed higher satisfaction with teaching materials (in a two-tailed t-test, p=0.005) and lower satisfaction with agricultural extension services (in a two-tailed t-test, p=0.03) when compared with the rest of the sample. Figure 6: Satisfaction with Learning and Teaching Materials by LGAP Wave USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 29 Our survey questions evaluating these characteristics of important public sectors mirror those asked by the CPS and thus also allow for some comparison to the early LGAP districts. Comparisons to CPS data from 2017, however, do not suggest there have been significant improvements in service delivery. As Table A.15 shows, satisfaction with service delivery in the early LGAP districts (Balaka, Lilongwe, M’belwa, and Mulanje) has either held steady or declined, with the exception of a slightly higher proportion of respondents in our 2020 survey stating they were satisfied with the work of their health advisory committee. In KIIs, we asked about impressions of service improvements as a result of LGAP. Interviewees were rarely mentioned being aware of any service improvements that could be tied to LGAP interventions (see Table A.6). While there were 87 mentions of interventions tied to improving local governance accountability and evidence of increased accountability (an average of more than one mention per interview), there were only 32 mentions of improved service delivery. Of the service improvements mentioned, about a third (9) concerned improvements in education services, half (15) in health services (almost all in the rapid response to COVID-19 attributed to LGAP assistance), and the remainder (7) in infrastructure improvements, with a single mention of improvement in agriculture. Importantly, we note that most (20) of the mentions were in four of the first-wave districts, even though more interviews (in absolute terms) were conducted in the newer LGAP districts. Most people in the evaluation districts (regardless of LGAP wave) replied that it was too soon to detect any improvements, as in the following quotes: It is too soon to know whether social contracts have improved service delivery in schools. (DEYS Director) It is too early to tell if educational services have been improved, but since the DEN and local school governance structures have been empowered, it is expected that they will ensure improved service delivery. (CSO Network/DEN member) A variety of improvements were mentioned concerning education, including improved teacher performance, improved pupil performance, reduced absenteeism of teachers and pupils, and reduced primary school dropout. Again, these were impressions of the interviewees, since hard data are lacking on these indicators. There has been significant reduction in primary school dropouts due partly to the work of an empowered education service committee. (District Councilor) With increased tracking of the School Improvement Plan and School Improvement Grants, communities are witnessing improved teaching and learning outcomes. This results from increased commitment from teachers, reduced absenteeism by teachers and learners, reduced dropouts, and parents being more involved in the affairs of the schools. (District Councilor) Concerning health services, almost all of the service improvements mentioned (11 of 15 mentions) were about the improved response to the COVID-19 pandemic due to LGAP’s efforts. These increased efforts were in both the first-wave LGAP districts and the later-wave districts where we collected data, even though LGAP initiatives were still quite new in the later-wave districts as the COVID-19 pandemic emerged. This shows that the recent trainings and empowerment of service and school committees was very timely in responding to a totally unanticipated health challenge. LGAP facilitated a multi-stakeholder COVID-19 response. (DEYS Director) LGAP supported coordination of district COVID-19 response. (District Councilor) USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 30 The two other mentions of health service improvements concerned the work of health service committees in coordinating services across the sector, resulting in improved service delivery and public health education. The qualitative data also show presence of nontraditional activities and programs that were tailored to identified needs in subnational areas. For example, small infrastructure projects were initiated after local consultations. Interviewees mentioned the construction of boreholes, market sheds, bridges, and concrete paths with LGAP support. Almost all of these (6 of 7) were in the first-wave districts. These activities were responding to respective districts’ needs and had the potential for sustainability after the program ended but, these were also activities about which respondents provided more negative feedback. For example, some of the boreholes did not function after drilling, and these “dry boreholes” were viewed by some interviewees as a poor investment by LGAP. To be sure, these failed efforts to improve access to an essential public good are largely outside of the control of implementers, but still raise caution to manage expectations about the inevitable variation in activity success. Going forward, the citizens surveyed cited a range of future development needs when asked about improving local government, with the implication that they expect their government to deliver better services, as exemplified by the following response of one of our interviewees: Development in this area is not working. Many things are at a standstill. (Female in rural area of expansion district) In order of number of mentions, the development needs they cited are: (1) health services, (2) infrastructure improvements, (3) market improvements, (4) water access, (5) education, and (6) food security. While LGAP was not set up to address all development needs, the expectation reflected in the theory of change is that LGAP activities, if successful, would lead to service improvements through improved local government functioning and coordination of service delivery. However, consistent with the analysis from the other data sources outlined above, these citizen reports indicate that LGAP activities have not yet succeeded, given that mentions of needed service improvements were as frequent in the pilot districts (proportionately) as in the new districts. CONCLUSIONS AND RECOMMENDATIONS ON SERVICE DELIVERY Given that survey respondents had generally more favorable views of human resources than material resources in evaluating service delivery, future programming and initiatives may need to focus on the deficiencies in material resources (with relevant associated governance supports). This comports with the aforementioned scholarly research on the need for multiple, complementary interventions to improve governance. Even if sufficient staff are adequately trained and incentivized to carry out their responsibilities under the watchful eye of engaged citizens, public servants will remain challenged by the lack of material resources (e.g., medicines, teaching and learning materials) essential to doing their work. The relative success in the education sector identified in the research for this performance evaluation likely reflects, at least to some extent, the additional infusion of financial and technical support.8 8 It is important to note this type of complementarity was a goal in all LGAP first wave districts, as six of those districts (1. Blantyre Rural, 2. Zomba Rural, 3. M'mbelwa, 4. Kasungu, 5. Mulanje, 6. Lilongwe rural) were already serving as GoM hub districts for the HR devolution process, building upon prior World Bank investments in Education Sector devolution. The final two districts (7. Machinga, 8. Balaka) together with Lilongwe rural (which overlapped with 1 of the 6 GoM pilot districts), were the 3 focus districts prioritized under USAID’s 2013 - 2019 CDCS. They were selected based on the integration 3 C's (co-location, coordination, and collaboration) criteria, and were recognized as the districts having the most saturated USAID investments, as well as saturated developmental needs / indicator data - across the missions portfolio. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 31 Our recommendation is not simply for greater complementary material support, but also for donors providing material support for service delivery to seek the perspectives of local officials as they plan budgets and allocations. Donors regularly consult, for example, district-level data aggregating information about health, education, and other development indicators to identify districts with significant development needs as priority areas for programming. When working with national partners, donor agencies can seek input from district officials through and with these national partners and seek information from SEPs and DDPs to better understand the specific resources communities are lacking in order to support local needs. This recommendation would thus require sustaining the production of SEPs and DDPs. Of course, donor interests in the sustainability of governance interventions can limit the potential to improve service delivery. Given the scholarly research suggests governance interventions will be more successful when supported with complementary interventions targeting service provider supply chains, providing financial support for material resources may be necessary. If USAID or FCDO purchase, for example, the essential medicines that local health facilities are lacking in addition to providing trainings to health sector personnel in tracking resources, it would improve service delivery in the healthcare sector. This may be more obvious if compared to a governance intervention where trainings on tracking health sector resources are made available but there are few medicines or other resources in stock to be tracked. Donors and development partners providing financial support meant for complementary interventions aimed at increasing material resources in the public sectors targeted with governance interventions, however, results in – at least – the temporary reliance of the Malawi government on donors in providing for the needs of its citizens. Local revenue generation is one primary avenue to reduce local service providers’ reliance on national budgets, with the potential to reduce Malawi’s reliance on foreign funding to support its service delivery as well. LGAP and/or its successor should support local revenue generation, including following through on our recommendation in the accountability section regarding continued support of IFMIS training as well as other emerging opportunities to engage information and communication technologies in financial management, like automated phone￾based financial transactions to pay for public services. Finally, another reason that interviewees could not assess improvements in service delivery is a lack of valid and reliable data to track outcomes. There is considerable local interest in project M&E information sharing and/or instituting systems to independently monitor service improvements. For example, in the education service delivery area, uniform teacher and pupil attendance data could be tracked and compared across schools and districts. The improvements in the HRMIS should make this tracking more feasible and should be a high priority. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 32 RESEARCH QUESTION 3 FINDINGS: WHAT ARE ANY UNINTENDED CONSEQUENCES OR EXTERNALITIES OF LGAP SUPPORT? HOW CAN THESE CONSEQUENCES BE MITIGATED (IF NEGATIVE) OR EXPANDED (IF POSITIVE)? This evaluation also explored whether LGAP activities led to unintended consequences or externalities, both good and bad. These unintended consequences include spillover effects within the broader government beyond the staff who directly benefited from LGAP, as well as any effects on the other responsibilities of beneficiary personnel in local governments. This spillover affected government functioning and staff effectiveness, thereby impacting civilians. Perceived unintended consequences/ externalities of LGAP activities were grouped into themes reflecting seven positive consequences/ externalities (with 22 mentions—see Table A.7), and 3 negative consequences/externalities (21 mentions). The mentions were almost evenly divided between the 4 pilot districts (15), the 6 other districts (12), and national level interviewees (15). FINDINGS ON UNINTENDED CONSEQUENCES AND EXTERNALITIES POSITIVE UNINTENDED CONSEQUENCES/EXTERNALITIES The evaluation identified positive unintended consequences/externalities from LGAP’s work on a number of levels ranging from donor priorities to national and local government operations. In terms of donor priorities, USAID’s LGAP stimulated other funders—such as FCDO, the World Bank, the Embassy of Ireland, and the United Nations Children's Fund (to a smaller extent)—to enter the local governance space in Malawi. Donors had not been working in this area because of challenges faced by earlier decentralization initiatives and donors’ discouragement concerning the commitment of the GOM towards reform. LGAP changed donor thinking because of its initial progress in revitalizing local government. Some of the significant LGAP activities that paved the way, such as the development of socio-economic profiles and district development plans, are highlighted by one interviewee below. Having the SEP and DDP in each district is critical because donors look for these tools in order for them to be involved. Councils with DDPs and SEPs have better LAPA scores. (District Commissioner) LGAP’s support for automated systems has potentially long-ranging positive consequences. A number of interviewees cited spillover from improved IT systems, reporting, and skills that improve government functioning overall beyond those individuals trained by LGAP. In other words, the LGAP trainings led to improved district staff skills that will spill over into other government functions, as cited below. The introduction of automated systems has helped to speed up processes. For example, where there was no computer and things were done manually, now things are automated and done faster. (MALGA) To reduce corruption, it is necessary to switch from manual processes to automation. It will help monitor accountability. (CSO) Resource materials are now mostly in electronic format and very accessible. (DEN) The existence of LGAP’s activities and reporting structure also appears to have increased incentives for staff and committees broadly to work harder (e.g., produce more timely reports). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 33 There is increased eagerness of the councilors to perform, and better cooperation between councilors and MPs (e.g., councilors filling in their MPs who might have been absent to a particular meeting). There are better working relationships between MPs and councilors of opposing political affiliations. (Councilor) Partners are working harder because they know they must present tangible results during monthly/weekly coordination meetings. (DEN) Additionally, LGAP activities in some cases smoothed the way for development partners to better coordinate their activities and avoid overlap or misunderstandings. Councilors are now able to demand financial reports. (MLGRD) LGAP has funded lots of meetings that would not have happened otherwise, even when it wasn’t in their work plan. (DEYS) The CSOs have the information they need and this has reduced conflicts with the council which, in the past, led to demonstrations. CSOs are now partners in development. (District Commissioner) A positive externality that was also mentioned was the support and mentoring by local CSOs that undertook some of the LGAP education activities, leading to organizational capacity-building successes such as constituency building and greater ability to conduct advocacy. These small and mid-size organizations will remain in the community, and their skills will transfer to other activities. LGAP has given our CSO excellent mentorship on social accountability. It has allowed us to achieve a closer working relationship with the District Commissioner’s office, particularly the Director of Planning and Development. (CSO) LGAP grants to CSOs for training increased these organizations’ profiles in the community. (DEN) When these organizations are brought together, each CSO encourages the others in ways to advocate for their cause. Organizations are free to critique each other and learn from each other. (Chief Education Officer) Finally, two people mentioned positive externalities for improved public health due to market clean-up activities sponsored by LGAP and the addition of toilets in some markets and schools. This was also true for the rapid COVID-19 response facilitated through LGAP support and infrastructure discussed in the prior section, where local governments were better prepared to respond to a completely unanticipated event of major consequence across Malawi. NEGATIVE UNINTENDED CONSEQUENCES/EXTERNALITIES While, as mentioned above, some of the LGAP activities led to reduced conflict, the key negative unintended consequence of LGAP that this evaluation identified is that some LGAP activities led to increased conflict among development partners and citizens, again on several levels ranging from national to local. The first type of conflict was social conflict, including conflicts between local development partners in some districts. Specifically, conflicts sometimes arose between MPs, district councilors and staff, market committees, ADCs, and/or school governance structures on roles and responsibilities for development funding and project implementation. In other cases, citizens empowered with greater information would sometimes respond with demonstrations. Increased conflict was noted in all evaluation districts and by national interviewees (cited 21 times). Below is a selection of the opinions that were expressed on this topic: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 34 The MPs feel they are being overtaken. People think that the MPs bring development funds (funded by the CDF). Where the councilors have political ambition, it becomes difficult to clarify roles and responsibilities for development funding. If not carefully done, the MPs may think that the councilors are taking over their roles. (CSO) It has made people hyper-sensitive and inquisitive, as if they are auditors. This might lead to mistrust and unnecessary antagonism. (District Commissioner) There is conflict between elected officials (councilors) and ADCs concerning roles and responsibilities. (ADC) Because people feel empowered, it’s easier/common for people to boycott and take the law into their own hands; there is a need for understanding of balanced rights and responsibilities (social accountability). (Education Committee) There are incidents of citizens taking the law into their own hands in the name of empowerment and rights. (DEN) Our Traditional Authorities got into conflicts about how to distribute the funds for transportation. (CSO) Feeling empowered, people in the district have resorted to demonstrations to voice their concerns. The demonstrations have been violent, requiring armed police interventions. (CSO/DEN) There were three mentions of LGAP coordination activities reducing conflict through greater coordination (as described in the preceding subsection), but the predominant view was that LGAP activities led to increased conflict. It is understandable that conflicts may arise when new partners are empowered to question the activities and decisions of other partners regarding development plans and funds. And, while uncomfortable, sometimes conflict can be productive. Accountability is increased when citizens have a greater awareness of what CDFs are and how their MP is spending their respective allocation, especially if citizens challenge MPs whom they believe are not serving the public’s interest. The delicate balance, however, is in promoting responsible citizen engagement that does not turn violent. Another source of conflict is the duplication of activities between LGAP and local and national development partners and agencies (7 mentions, 2 from the new districts and 5 from national￾level respondents) arising from inadequate coordination with national-level ministries and organizations. Below are examples of comments on this by respondents: Overlapping agendas with national ministries led to conflict and duplication. Some council staff stopped being accountable to local government and aligned themselves with LGAP staff. Some councils started rejecting LGAP’s presence in their meetings. (MLGRD) There was unanticipated conflict between LGAP and the National Local Government Finance Committee. LGAP destabilized the relationship between the NLGFC and the Ministry of Finance, due to a lack of collaboration around producing a manual for local government. (NLGFC) LGAP appears to compete with NLGFC. We would have expected the two entities to work together so that the council could submit one report that satisfies both entities. (Councilor) The LGAP activities have disrupted the activities of the secretariat. This affects work schedules and plans. (Councilor) USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 35 Some people felt looked down upon by the implementers; if a consultant came in, they would play second fiddle. (HR Official) CONCLUSIONS AND RECOMMENDATIONS ON UNINTENDED CONSEQUENCES AND EXTERNALITIES On the positive externality side, since LGAP has brought new funding into the democracy and governance space in Malawi, efforts should continue to support coordination between funders. This will reduce overlap between development efforts. It is also clear that the LGAP IT interventions have important spillover effects and lead to the sustainability of governance interventions when fully integrated into government operations, by leaving automated systems in place that will carry on after LGAP activities are complete. These should be continued and expanded in future USAID efforts, particularly in the 20 new districts that have not yet fully benefited from the LGAP automation efforts. Without continued support, there is a risk that the gains to date could be lost due to turnover in staff and lack of maintenance of equipment. The work of service committees has been strengthened under LGAP. These committees have led to increased coordination of services and were particularly important during the rapid response to the COVID-19 pandemic. For example, service committees that had been revitalized by LGAP were able to quickly meet and plan a uniform COVID-19 response across the district. Future efforts should continue supporting the work of these committees and their regular convening. Going forward, any future program should emphasize the social responsibility of all development partners and program staff, aimed at avoiding destructive conflict. Governments are at risk of increased protests with the initiation of transparency initiatives (Hollyer et al. 2015). Particularly in the current context—in which Malawian citizens are increasingly participating in demonstrations in order for their voices to be heard (Pensulo 2020; Dionne and Dulani 2020a)—it will be imperative that local officials respond promptly and meaningfully to citizen concerns regarding governance and accountability. All future training programs that target local officials should emphasize the potential for conflict, providing specific historic examples in the local context or from neighboring areas. To avoid potential violence, training programs could incorporate peacebuilding processes (see, e.g., Kudzas et al. 2015), whereby participants actively plan procedures for promptly responding to citizen action so that officials will have these plans in place to enact before citizens take to the streets. Future efforts should also ensure more close coordination with all national ministries and agencies whose functions overlap with LGAP. This is critical for sustainability because, without buy-in, the government entities responsible for implementing improved practices may not retain them. Viewing the national-level conflicts reported in interviews conducted for this evaluation through a political economy analysis prism, we see that a handful of stakeholders are essential to improving local governance and accountability and a lack of buy-in from those stakeholders can challenge governance and accountability initiatives. Any continued work in that improves financial management at the local level, for example, requires coordination with and buy-in from the NLGFC. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 36 RESEARCH QUESTION 4 FINDINGS: HOW HAS THE LGAP AWARD USED THE RESOURCES (FINANCIAL, HUMAN RESOURCES/TECHNICAL ASSISTANCE) EFFICIENTLY AND EFFECTIVELY IN RELATION TO THE PLANNED ACTIVITIES, OUTPUTS AND OUTCOMES, AND SUSTAINABILITY? This evaluation question concerns whether LGAP used its resources effectively and efficiently to accomplish the goals of the program. This includes how resources were distributed across different technical areas (e.g., human resources process improvements, institutional capacity￾building) of focus, as well as the efficiency measures put in place in the distribution of resources. FINDINGS ON USE OF RESOURCES LGAP was implemented through a contract with DAI. According to DAI annual financial reports, the initial contract was for five years (October, 2016 through September, 2021) with a budget of $22,975,909. This amount was increased over time to $39,966,832, in order to expand to 20 new districts and take on new initiatives in the education sector. During the first three years, expenditures averaged approximately $6 million per year. With the addition of new funds, and the expansion into 20 new districts and the education sector, expenditures are averaging almost $11 million per year for the final two years. The project’s large scale can be put into context, by noting that the total contract budget amounts to about 14.6 percent of the total budget for health in Malawi for fiscal year 2020–21. Thus, much was expected from such a large expenditure of resources. While the annual financial reports do not report expenditures by Results Area, they do report that, for example, in FY 2019, fully 97.9 percent of the budget up to that time had been used for either Good Governance activities (84.7 percent) or Civic Participation activities (13.2 percent). These categories are certainly consistent with the LGAP Scope of Work, but unfortunately, financial reports do not provide more specificity about how expenditures match to specific activities. In the FY 2017 Financial Report, an indication of staff functions was provided. At that time (at the end of the first year of the project), in addition to the Chief of Party and Deputy Chief of Party, the central LGAP staff included 11 positions for managers or specialists in the following specialties: public financial management; local government; decentralization; social accountability; organizational capacity; monitoring, evaluation, and learning; grants; finance; procurement; and infrastructure. All of these specialties are consistent with the Scope of Work. In addition to the central LGAP staff, in the first year, LGAP established three regional offices with seven staff positions per office, a program manager, four specialists or coordinators, and two drivers. The number of regional offices was increased when LGAP expanded into new districts. These regional offices gave LGAP the ability to work directly and frequently with the district government, CSOs, and ADCs, and thus are consistent with the emphasis on strengthening local government functions and accountability. Most of those whom we interviewed believed that the trainings, along with coordination meetings and mentoring of local staff, were an efficient and effective use of resources, especially in the new districts, as cited below. The most effective LGAP activities are the trainings. (DEYS) USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 37 The relationship between the councilors and the secretariat has greatly improved. Now the emphasis is on dialogue. Councilors now know how to run council meetings. The conduct of the current cohort of councilors, which were trained by LGAP, cannot be compared to the previous round which was like a mob. Councilors are getting more information from the secretariat, which is now more responsive, such as information on procurements which was previously a source of conflict. (Councilor) The empowerment of the ADCs is the most important component of the LGAP concept. (CSO/DEN) Chairpersons of all local school governance structures have been incorporated into the teams managing the funds. (DEN) The training in financial management was viewed as particularly effective, as there was a marked improvement in financial management due to LGAP training and mentoring of finance staff and councilors. The LGAP Project has done a lot to improve local government accountability. For instance, LGAP has provided training to accounting staff in the councils and also supported the devolution of the human resource (payroll devolution). LGAP has also supported some councils to put in place mechanisms and systems to increase local revenue generation. (MALGA) However, extensive training is expensive, especially when it is not fully targeted to the knowledge base of those trained. Despite LGAP’s stated goal to provide smart capacity-building9 that extends beyond one-time off-site training, some felt the trainings were too brief or infrequent. The training of school stakeholders was initially a mere orientation session; there is a need for it to be proper capacity training. (DEN) All of the training, especially on the IFMIS, was really useful, but it needs to be ongoing. (Councilor) While improving revenue collection is viewed as a critical way to sustain an effective local government, LGAP work in this area yielded mixed results. LGAP attempted to automate market revenue collection through mobile phone banking. In some districts, secure market revenue collection is improving, but this has not taken hold in most places as yet. This is largely due to logistical implementation issues (relating to transportation of and accounting for the cash collected and the cost of the handheld devices). Additionally, the system faced resistance at all levels because it stopped people from accessing "supplementary" income. The electronic revenue collection system is not fully functional. The ideas were brilliant, but it was not done successfully. (MLGRD) However, others gave a more optimistic assessment and expressed the opinion that LGAP’s approach to developing a functional electronic market fee collection system should continue. Indeed, revenue collection is a critical challenge for LGAP and future governance interventions to continue trying to address it. We established baseline targets for revenues from the markets. There has been a deliberate effort to get feedback on how much has been collected and what has been done with the money for development. Our revenue collection has gone up by 50 9 “Smart” capacity-building refers to approaches that rely less on traditional training approaches such as group trainings that have questionable effectiveness and instead would involve multiple field offices that tailor capacity building interventions to the greatest needs in the respective districts. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 38 percent. We used to get a lot of misinformation from the Finance Committee around revenue collection. (Counselor) Also, efforts to improve market infrastructure through new sheds and boreholes occasionally raised expectations that were not met. 80 percent of the boreholes were incomplete; no water was found, but the contractor was paid. (Service Committee) Promised infrastructure development at markets occurred at only a few markets. (Service Committee) It is unclear whether the infrastructure development efforts of LGAP, some of which were unsuccessful, constituted the best use of funds. While it was deemed necessary to show the market vendors that the revenue generated in the markets led to improvement, it might have been more efficient to collaborate with another funder when meeting the districts’ infrastructure needs, which are extensive and expensive to address efficiently and effectively. These findings on infrastructure improvement efforts suggest that it may be a more efficient use of resources going forward if future local governance interventions coordinate with other projects to provide this type of assistance rather than providing it directly. We also heard in KIIs of several other implementation challenges and problems that negatively affected the efficiency and effectiveness of LGAP activities. The interviewees mentioned 90 challenges and problems, as summarized in Table A.8. In terms of the challenges faced by LGAP, interviewees mentioned, in approximately equal numbers: (1) staff turnover of district and LGAP staff, (2) insufficient time and resources to scale up to 28 districts, (3) COVID-19 delays in the schedule, and (4) that village-level work was impractical due to the scale (number of villages) and distance to villages. Of the 36 mentions of challenges, fully 19 were in the six new districts where interviews were conducted. The challenge (weakness), however, is that, midway through the project, LGAP decided to increase its geographical coverage from the initial 8 district councils to cover all the 28 district councils (the whole country). This diluted its impact as the resources were spread thinly. (MALGA) In general, the pilot districts received much more attention than the remaining districts due to time and resource constraints. LGAP reports that this was a deliberate strategy to take a “shorter route to the same destination” in the 20 expansion districts through more targeted investment. In the 20 expansion districts, LGAP sought to apply lessons from the eight pilot districts, as well as focus on the main needs as agreed upon with each district in the fall of 2019, based on an assessment of each council. However, based on interviewee feedback, it appears that this approach may not have been clearly communicated to all stakeholders and some of those in expansion districts still expected to receive what the pilot districts did. At the same time. it is also worth noting that many of those more limited planned activities were also then delayed due to the coronavirus pandemic. Interviewees also pointed out six problems, with 54 mentions (over half in the six new districts), in how LGAP implemented the program. These were, in ranked order of number of mentions: (1) promised interventions were incomplete (for example, all promised trainings were not received, especially in the new districts); (2) there was a lack of engagement or regular communication with key individuals/organizations; (3) allowances or transportation and refreshments were too small given distances; (4) trainings were too short or infrequent; (5) LGAP did not share its work plans or exit strategy; and (6) there was sometimes insufficient follow-up by LGAP staff. These are referenced below. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 39 It was not what we expected, and we do not know how they are using the funds (or how much they have). (CSO/DEN) We never saw the LGAP work plan or budget. (Councilor) LGAP stopped consulting the ministry, and the project changed its brand from a Ministry of Local Government project to an LGAP project. This led to duplication of effort since LGAP hired staff into areas where there were already existing staff. (MLGRD) The ADCs cover huge geographic areas. LGAP did not provide an allowance for transportation and lunch. They were given a Fanta and a biscuit. That does not make for a quality training. (Councilor) After developing the social contracts in schools, there was little follow-up by LGAP staff to assure that what was signed was put into practice. (DEYS) Thinking about whether LGAP’s use of resources to-date may yield sustainable advancements in governance, interviewees shared broad opinions concerning the sustainability of LGAP efforts. Some felt that knowledge had been gained that would be passed on to others by those who had been formally trained. Those with knowledge should be able to continue to use it even if LGAP stops. (Councilor) However, the dominant opinion was that, due to the need for ongoing retraining (especially acute in this environment of high personnel turnover) and lack of complete intervention in the 20 new districts, sustainability was at risk. The problem is that skills will continue to exist only as long as those that have them are still in the system. Elected officials have a five-year tenure, and service committee members have only a one-year tenure. (District Commissioner) The service management committee's term is for one year, so there is a need for continual training of new members. (Service Committee) Another concern with sustainability implications, highlighted above, is that relevant national ministries and organizations have not been fully incorporated into LGAP efforts, and are currently unlikely to be able to continue LGAP initiatives in the future. When projects come from the outside and stand on their own, and do not work with existing institutions, there can be no sustainability. People have been trained in certain business practices, but we don’t know who has been trained in what way. So we cannot fill the gaps when they go away. (NLGFC) CONCLUSIONS AND RECOMMENDATIONS ON THE USE OF RESOURCES LGAP’s substantial funding occurred at a critical time in the devolution of responsibilities from national to local government in Malawi and brought significant resources to a huge task that exceeded the resources of national entities. Especially given the recent absence of elected local governments, the initial participating institutions—if present—were likely oriented around patronage rather than a commitment to serving the community (Chasukwa et al. 2014). Through its broad-based training program, LGAP strived to address this orientation to patronage through revitalized local government structures such as the councils, council staff, council committees, school governance committees, CSO networks, and ADCs. While the program was expensive, LGAP was serving an important role and greatly helping to address a significant gap in governance at the local level. In interviews at the district level, we heard consistent positive USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 40 feedback about their experiences with LGAP, particularly in the four pilot districts studied. While the research conducted for the performance evaluation did not involve a formal cost-benefit analysis, the scale of the challenging initial context and the counterfactual — how might local governance and accountability in Malawi look today in the absence of LGAP? — led the evaluation team to judge the program worthy of significant investment. Going forward, it is clear that there is a demand for greater training of development partners that have been targeted by the LGAP in its first four years. Those we interviewed would like the trainings to be more comprehensive, longer, and more frequent. To be sure, these respondents represent a population that would benefit from additional trainings, as they offer the potential for small financial incentives (e.g., per diems, transport reimbursements, etc.), which have become increasingly scarce in recent years following harmonization among donors of allowance policies (Ngwira and Mayhew 2020). Recognizing the delicate politics surrounding per diems in Malawi in the current period, and with an eye to the sustainability of governance and accountability programming, we recommend that retraining be accomplished through national-level partners such as the MLGRD, the NLGFC, and MALGA. There is still a need in the short term, however, for training in the expansion districts, where the COVID-19 pandemic created delays. These trainings should evolve to incorporate comments of participants tailored to the respective audiences. The evidence does not suggest that longer and more frequent trainings would lead to improved service delivery, even if they would yield greater satisfaction of participants (who, again, stand to financially benefit from longer trainings). There should be more dissemination of existing monitoring data and budget information, such as the citizens' budget, to improve village-level engagement with district-level development planning and implementation. It may also be beneficial for future local governance projects to explore innovative strategies to efficiently expand their geographic presence in the field since a desire for more local offices to improve access to all districts was raised during this evaluation. The education initiatives show promise, but better tracking of outcomes is needed, particularly monitoring of the implementation of social contracts in schools. For example, a list of indicators such as absenteeism of teachers and pupils, pupil/teacher ratios, and school inspection scores could be tracked and publicized by school and district. Despite some setbacks, innovative approaches to local revenue collection are critical to sustaining LGAP gains. LGAP should employ a more nimble strategy to identify and test potentially promising approaches to expand revenue collection of market fees. This has yielded some gains and judicious expansion of it would help to provide funding that is greatly needed for funding local government services. However, any future infrastructure improvements should be undertaken in coordination with other development partners better positioned to succeed. Outsourcing this work to another actor dedicated solely to (or with greater expertise in) the provision of infrastructure will hopefully produce more even results. Finally, for sustainability of the LGAP gains, there is a critical need to improve information sharing and joint planning with all national and local ministries and organizations with similar goals and activities. This kind of coordination is absent from LGAP’s theory of change, even though one of LGAP’s Results Areas is “Decentralization Policy Environment and Systems Improved.” Co￾creation with national stakeholders through continuous coordination, in addition to producing an enabling national policy environment, is necessary for success, and future related programming should be incorporated as part of the theory of change. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 41 RESEARCH QUESTION 5 FINDINGS: WHAT LESSONS HAVE BEEN LEARNED TO INFORM FUTURE GOVERNANCE PROGRAMMING? IN PARTICULAR, HOW RELEVANT ARE THE RESULT AREAS AND SUB-RESULT AREAS IN THE LGAP AWARD TO ACHIEVING THE ACTIVITY’S OVERALL OBJECTIVE TO BUILD THE CAPACITY FOR RESPONSIVE, EFFECTIVE, AND ACCOUNTABLE SERVICE DELIVERY THROUGH LOCAL GOVERNMENTS IN MALAWI? Finally, this evaluation considers what we have learned from LGAP’s experience to date that will inform future governance programming. In particular, this section discusses the relevance of the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build capacity for responsive, effective, and accountable service delivery through local governments in Malawi. FINDINGS ON LESSONS LEARNED In the Statement of Work for the LGAP, USAID outlined three Results Areas: (1) Local Government Performance and Transparency Increased (approximately 50 percent of effort), (2) Demand for Accountable Government Strengthened at the Local Level (approximately 20 percent of effort), and (3) Decentralization Policy Environment and Systems Improved (approximately 30 percent of effort). The Scope of Work provided flexibility for defining specific activities within Results Areas and Sub-Results Areas, as well as “illustrative activities” with suggested indicators for tracking those results. LOCAL GOVERNMENT PERFORMANCE AND TRANSPARENCY INCREASED Interviewees gave their impressions of the most significant LGAP activities (see Table A.3 and discussion above under the four prior findings sections). Concerning Results Area One, respondents emphasized that trainings of district councils and district staff, particularly financial staff, were effective in upgrading skills, fostering ownership of roles and responsibilities, and increasing accountability. The upgrading of IT skills and equipment was especially critical. The financial interventions, particularly the automation of financial reports, increased accountability and reduced corruption in some instances. However, we heard that problems remain with transparency in many places. The majority of the councilors are still fearful of challenging MPs and the Secretariat. (Councilor) There are reports of gross misuse of funds at the District Council, leading to questions about whether LGAP has yet made a difference. (CSO) The audit report shows gross mismanagement of funds. This means that the issues of accountability are far from over. (CSO) The development of DDPs and SEPs in almost all districts through LGAP technical assistance was a critical LGAP accomplishment that improved the ability of local governments to plan and USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 42 implement their development activities. A missing ingredient continues to be the lack of good M&E systems to track development outcomes. For the most part, we heard that districts do not have a robust M&E system in place and, especially that it is not currently possible to compare across districts using common metrics. DEMAND FOR ACCOUNTABLE GOVERNMENT STRENGTHENED AT THE LOCAL LEVEL Concerning Results Area Two, LGAP training of ADCs was frequently credited with helping to empower citizens to question their government and become more involved in development activities. ADCs feel more empowered. For example, we stopped a bridge-building project in a village. The design and quality were poor. (ADC) While there were insufficient resources to directly train VDCs, each ADC contained VDC representatives who were expected to bring their knowledge to their local committee. We were told that this was insufficient to reach most VDCs but, as noted previously in the Accountability Findings section, the evaluation team believes LGAP made a reasonable decision to forego village-level training due to the preexisting robust governance structures at the village level. The membership in VDCs is so huge that it is hard to do training for those members. So skills gained by ADCs cannot be cascaded to the VDCs. (ADC) The training of CSO and DEN representatives further empowered grassroots organizations and other community groups such as market committees and school committees. In particular, the education initiative has led to the training and empowerment of citizens to develop social contracts in schools and monitor the performance of teachers. Further, LGAP sponsored media outreach to citizens, informing them of how to get involved in local government. LGAP sponsored programs on decentralization on the community radio. This has been effective in letting people know their rights, and they are beginning to hold the council to account. (CSO/DEN) Still, data from our citizen survey reflected a continued gap between local citizens at the village level and their district governments that has not been repaired through LGAP, as suggested by some of the quotes from citizens provided in the previous results sections. While training of ADCs and school committees was underway in some of the new districts at the time of our interviews, the process was not complete. So, as with Results Area One, much of LGAP’s effort to empower citizens to hold their government accountable has not occurred, especially in the new districts. Given that this work was greatly appreciated in the pilot districts, this should be continued if resources are available. Now that the foundation from the pilot districts is in place, USAID should incorporate the lessons learned from the pilot districts, particularly that of the importance of training and mentoring in financial management. DECENTRALIZATION POLICY ENVIRONMENT AND SYSTEMS IMPROVED Concerning Results Area Three, LGAP worked closely with some—although not all—relevant national-level organizations to foster changes in decentralization policies. A particularly important accomplishment was to work with district councils to update by-laws, which have the force of law at the local level. LGAP facilitated the development of by-laws, and now local people are aware of the by-laws, making enforcement easier. By-laws included: raising market fees, child USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 43 marriage prohibition, no selling alcohol during school hours, and a ban against selling alcohol to school children. (Councilor) Another important accomplishment was to work with the Ministry of Finance to develop manuals for district-level financial reporting. However, the impact of developing these materials was diminished due to a lack of coordination with other important organizations that can help to sustain effective financial reporting, such as NLGFC and MALGA. LGAP work has also led to a devolution of HR responsibilities to the local level, focusing particularly on the education personnel system. LGAP supported the human resources management teams with revamping the HR Management Information System (HRMIS), including updating personnel files and making available the folders for all the teachers in the district. (District Education Staff) However, given that USAID anticipated that 30 percent of its work would be devoted to this Results Area, there is more work to be done to achieve the results at the national level required for a full devolution of responsibilities to local government. In addition, we were told that the flow of funds to local governments from the national government is insufficient for the responsibilities that have been devolved to them and, to date, the increase in local revenue generation does not yet meet the increased need. Finally, as noted previously in the findings on unintended consequences and externalities, the evaluation recommends expanding the third results area to ensure that the critical coordination among government entities necessary for project sustainability is a core program objective going forward. CONCLUSIONS AND RECOMMENDATIONS FOLLOWING FROM LESSONS LEARNED Concerning Results Area One (Improving Local Government Performance and Transparency), a continued focus on the LGAP financial interventions—such as automation of financial records and reports through the IFMIS—is critical, along with a continued emphasis on broad participation in budgeting using the citizen budgets. IT infrastructure development should also continue as a sustainability mechanism. As discussed above, there is a need for improved M&E in all districts (particularly in the education sector, where the greatest effort has been concentrated on service improvements). While results to date are mixed concerning local revenue enhancement, this is viewed as key to sustaining local government improvement. The SEPs and DDPs are viewed as extremely valuable and efforts to develop them in the new districts and updating those in the pilot districts should continue. Concerning Results Area Two (Accountable Government Strengthened at the Local Level), there is broad support for expanding training of the type offered by LGAP, and we recommend that LGAP training activities in all districts should continue, especially in the 20 new districts. However, this is very expensive. Going forward, after the expansion districts have received targeted training, the retraining could be accomplished using a train-the-trainer/mentoring approach, and through coordination with national partners. LGAP should consider expanding its CSO partners, another method of broadening and sustaining its work to achieve its objectives. Respondents also would like coordination meetings across development partners to continue and be expanded to new partners, including new CSOs. Results from the citizen survey have shown that, while citizens are engaged in accountability activities at the village level, they are considerably less engaged in district-level accountability. More effort is needed to connect those two levels of government through information and other targeted strategies. Some of the interviewees want an expanded focus on infrastructure development, which is a critical development need, although there are USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 44 questions about whether this is reasonable to do under LGAP’s mandate (as discussed above). A better approach would be to engage other development partners in these efforts. Concerning Results Area Three (Decentralization Policy Environment and Systems Improved), LGAP should continue providing assistance with by-law development in all districts and continue to expand its HR initiatives to all districts. LGAP should also improve and expand its communication with national- and district-level partners, and future programs should make this a central to their theories of change. The engagement with national partners, and frequent communication with all local development partners, is key to successfully sustaining the considerable success that has been achieved during the first four years of the LGAP project. Finally, we recommend that the LGAP program develop a clear sustainability and exit plan going forward. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 45 REFERENCES Banerjee, Abhijit V., Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. 2010. Pitfalls of participatory programs: Evidence from a randomized evaluation in education in India. American Economic Journal: Economic Policy 2(1): 1-30. Bauhr, Monika, and Ruth Carlitz. 2020. When does transparency improve public services? Street‐ level discretion, information, and targeting. Public Administration (early view). Björkman, Martina, and Jakob Svensson. 2009. Power to the people: evidence from a randomized field experiment on community-based monitoring in Uganda. The Quarterly Journal of Economics 124(2): 735-769. Cammack, Diana. 2017. Malawi's Political Settlement: Crafting Poverty and Peace, 1994–2014. Journal of International Development 29(5): 661-677. Chasukwa, Michael, Asiyati Chiweza, and Mercy Chikapa-Jamali. 2014. Public Participation in Local Councils in Malawi in the Absence of Local Elected Representatives-Political Eliticism or Pluralism? Journal of Asian and African Studies 49(6): 705–20. Chinsinga, Blessings. 2008. Decentralization and Poverty Reduction in Malawi–A Critical Appraisal. In Decentralisation in Africa: A pathway out of poverty and conflict? Gordon Crawford and Christof Hartmann (eds.). Amsterdam University Press. Dionne, Kim Yi, and Luis Camacho. 2020. The Effects of Capacity Building and Community Building on Health in Eastern Congo. Unpublished working paper. Dionne, Kim Yi, and Boniface Dulani. 2020a. A Malawi court just ordered a do-over presidential election. Here’s what you need to know. The Washington Post (February 4). https://www.washingtonpost.com/politics/2020/02/04/malawi-court-just-ordered-do-over￾presidential-election-heres-what-you-need-know/ Dionne, Kim Yi, and Boniface Dulani. 2020b. Malawians elected a new president in Tuesday’s historic election. The Washington Post (June 28). https://www.washingtonpost.com/politics/2020/06/28/malawians-elected-new-president￾tuesdays-historic-election/ Dulani, Boniface, and Kim Yi Dionne. 2014. Presidential, parliamentary, and local government elections in Malawi, May 2014. Electoral Studies 36: 218-225. Hollyer, James R., B. Peter Rosendorff, and James Raymond Vreeland. 2015. Transparency, protest, and autocratic instability. American Political Science Review 109(4): 764-784. Kosack, Stephen, and Archon Fung. 2014. Does transparency improve governance? Annual Review of Political Science 17: 65-87. Lieberman, Evan S., Daniel N. Posner, and Lily L. Tsai. 2014. Does information lead to more active citizenship? Evidence from an education intervention in rural Kenya. World Development 60: 69-83. Mattes, Robert. 2008. The material and political bases of lived poverty in Africa: Insights from the Afrobarometer. In Barometers of quality of life around the globe, pp. 161-185. Springer, Dordrecht. Ngwira, Chikosa, and Susannah Mayhew. 2020. Donor-driven harmonised payment of allowances policy and NGOs’ community engagement in Malawi. Development in Practice 30(1): 3-14. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 46 O'Hagan, Richael, Melissa A. Marx, Karen E. Finnegan, Patrick Naphini, Kumbukani Ng'ambi, Kingsley Laija, Emily Wilson, et al. 2017. National assessment of data quality and associated systems-level factors in Malawi. Global Health: Science and Practice 5(3): 367- 381. Palmer, Debbie. 2006. Tackling Malawi's human resources crisis. Reproductive Health Matters, 14(27): 27-39. Pensulo, Charles. 2020. “It’s the year of mass protests”: Malawi awaits crucial election ruling. African Arguments (January 30). https://africanarguments.org/2020/01/year-mass￾malawi-protests-election-ruling/ Raffler, Pia, Daniel Posner, and Doug Parkerson. 2020. Can citizen pressure be induced to improve public service provision? Unpublished working paper. Waddington, Hugh, Ada Sonnenfeld, Juliette Finetti, Marie Gaarder, Denny John, and Jennifer Stevenson. 2019. Citizen engagement in public services in low‐and middle‐income countries: A mixed‐methods systematic review of participation, inclusion, transparency and accountability (PITA) initiatives. Campbell Systematic Reviews 15(1-2): e1025. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 47 ANNEX A: EVALUATION SCOPE OF WORK DRAFT STATEMENT OF WORK FOR LOCAL GOVERNMENT ACCOUNTABILITY (LGAP) PERFORMANCE EVALUATION ACTIVITY FEBRUARY 28, 2020 C.1. INTRODUCTION AND BACKGROUND In 2016, USAID/Malawi awarded $39.9 million Local Government Accountability and Performance (LGAP) Contract to DAI to build the capacity for responsive, effective and accountable service delivery through local governments in Malawi. LGAP was designed in response to a reform window created by several Government of Malawi (GOM) policy reform initiatives, following the 2014 election of local government leaders. These reforms were intended to accelerate and deepen the decentralization process which was initiated in 1998. Consequently, USAID/Malawi sought to support the full implementation of GOM’s decentralization reforms to address decades of partial and fragmented policy implementation, which has driven suboptimal public service delivery outcomes. This five-year activity, awarded in August 2016, provided support to eight (8) of the twenty-eight (28) district councils across Malawi in the initial 3 years of implementation. Subsequently, the activity has been expanded to all 28 district councils in September 2019, with additional funding from the United Kingdom’s Department for International Development (DFID). LGAP works to improve local government performance and transparency, increase citizen engagement, and strengthen the enabling environment for decentralization in Malawi. This activity is also designed to assist the USAID/Malawi mission in its efforts to advance the integration of democracy and governance concepts of accountability, representation, and policy reform, with programming in the service sectors, i.e. health, education, etc. LGAP award intends to achieve results in the following five (5) areas. 1. Local government performance and transparency increased (across 28 Local councils) 2. Demand for accountable government strengthened at the local level 3. Decentralization policy environment and systems improved 4. Public service performance management systems improved and implemented 5. Public finance management related to human resources improved LGAP support is provided primarily through technical assistance to the national and subnational governments, which is delivered through five (5) regional offices across Malawi and one (1) head office in Lilongwe. The LGAP award offices are expected to include permanent and short-term field staff, and the offices are either located within the government offices or within the vicinity of government offices. LGAP was also encouraged to co-locate with other projects and or regional offices, in furtherance of USAID’s integration development approach. These offices are expected to provide technical assistance to accomplish the results identified above including providing technical assistance and supportive supervision for core local government operations (i.e. planning, budgeting, administration), public financial management and accountability, capacity building for councilors and District Executive Committee staff, and support for integration with USAID activities, as well as assistance to USAID Mission coordination with district government. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 48 C.2. PURPOSE OF THE LGAP PERFORMANCE EVALUATION USAID/Malawi requires the services of a Contractor to conduct a performance evaluation of the LGAP award. The final Contractor’s report will inform USAID/Malawi on how the LGAP award has performed, and guide USAID/Malawi for future programming. The Contractor will at a minimum be expected to: 1. Assess the LGAP’s performance to date and to determine whether the project’s activities are achieving the intended results as outlined in the agreement 2. Assess the extent to which LGAP award is achieving the overarching goal of supporting GOM in the fulfillment of the decentralization policy, including both changes to the form of government (policies, regulations, organizational structures), as well as function of government through improved performance.10 3. Assess the extent to which the LGAP approach drawing on political economy analysis has increased the effectiveness of the USG/UK investments in decentralization and local governance in Malawi. 4. Document lessons learned, good practices and points for improvement to make possible adjustments for current implementation of LGAP award and to inform future project designs. C.2.1. TARGET AREAS AND GROUPS The Contractor must conduct the performance evaluation at different levels (national, district and community), targeting different stakeholders involved in the activity implementation. LGAP award targets include but are not limited to 28 local councils in Malawi and their structures i.e. Area Development Committees (ADCs), District Executive Committee (DEC), Village Development Committees (VDC), Council Service Committees; Ministry of Local Government and Rural Development; Ministry of Finance and Economic Planning, National Local Government Finance Committee; Malawi Parliament; Civil Society Organizations; Public Service Reform Directorate; Ministry of Education, Science and Technology; Ministry of Health; Ministry of Agriculture; Local Government Service Commission; and Malawi Local Government Association. USAID supports district governments through a variety of programs, and beneficiaries are often unsure which interventions are associated with which programs. The Contractor must explain how it will ensure that the qualitative data collected is attributable to the LGAP activity. If clear attribution is not possible, the contractor must explain how and why LGAP contributed to desired outcomes. C.3.1. LGAP PERFORMANCE EVALUATION QUESTIONS The Contractor must, at a minimum, address the following questions: 1. To what extent has LGAP implementation increased Malawi Government accountability at the local level, if so how? 2. To what extent has LGAP implementation contributed to improved service delivery across sectors (health, education and agriculture), if so how? 3. What are any unintended consequences or externalities of LGAP support? How can these consequences be mitigated (if negative) or expanded (if positive)? 10 [1] See Bridges and Woolcock “How (Not) to Fix Problems that Matter: Assessing and Responding to Malawi’s History of Institutional Reform. http://documents.worldbank.org/curated/en/349361513957973588/pdf/WPS8289.pdf USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 49 4. How has LGAP award used the resources (financial, human resources/technical assistance) efficiently and effectively in relation to the planned activities, outputs and outcomes and sustainability? 5. What lessons have been learned to inform future governance programming? In particular, how relevant are the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build the capacity for responsive, effective and accountable service delivery through local governments in Malawi? C.3.2. TASKS The Contractor must perform the following tasks as part of this scope of work: 1. Draft inception report 2. Develop the performance evaluation methodology 3. Test and verify the performance evaluation methodology 4. Deploy a field team 5. Collect the relevant data to inform the performance evaluation 6. Conduct oral debrief meetings with USAID on the preliminary findings of the performance evaluation 7. Host a stakeholder workshop to present the draft performance evaluation findings for their validation and inputs 8. Draft Final Report C.4. RESULTS: DELIVERABLES AND OUTPUTS The Contractor must furnish the following deliverables and reports: 1. INCEPTION REPORT The inception report must describe the conceptual framework the evaluator will use to undertake the performance evaluation and the justification for selecting this approach. It must detail the performance evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources, and sampling). The Inception report must also contain a work plan, which indicates the phases in the performance evaluation with key deliverables and milestones. USAID/Malawi will review this report and the Contractor must receive COR’s approval of the report before it begins implementing the performance evaluation plan. The Inception Report must at least contain the following: a. A work plan, which indicates the phases in the performance evaluation with key deliverables and milestones and key personnel responsibilities. b. Complete set of performance evaluation questions and elaborate on them as necessary. Any questions added during the contract negotiations must be clearly indicated and any deleted questions must be mentioned with a reason as to their exclusion. c. Discussion of the overall approach of the performance evaluation, highlighting the conceptual model (s) adopted. d. Discussion of risks and limitations that may undermine the reliability and validity of the performance evaluation results. e. Discussion of the data collection and data analysis methods that will be used for each question. State the limitations for each method. Include the level of precision required for quantitative and qualitative methods and value scales or coding used for qualitative USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 50 methods. Standard data collection methods for USAID evaluations are: surveys, questionnaires, interviews, focus groups, document review, and observations. f. Detail key data sources that will be selected to answer each performance evaluation question. g. Explanation of how existing data will be incorporated and used to answer the performance evaluation questions. h. Discussion of the sampling methods and details. Include area and population to be represented, rationale for selection, mechanics of selection, sample size (for each unit of analysis), sample precision, and confidence and limitations. i. Summarized performance evaluation methodology in an evaluation planning matrix that must contain the following column headings: performance evaluation question, measure (s) or indicator(s), data collection method(s), data source, design strategy / framework for each question, sampling methodology, data collection instrument(s) for each question and data analysis methodology on each performance evaluation question. j. Timeline showing the performance evaluation phases (data collection, data analysis and reporting) with their key deliverables and milestones. k. Specific roles and responsibilities of each team member for each performance evaluation phase. Include any changes in the performance evaluation team. l. Discussion of logistics of carrying out the performance evaluation. Include specific assistance that will be required from USAID/Malawi, such as providing arrangements for key contacts within the Mission or Government of Malawi. m. Data collection instruments specific to questions and indicators in the performance evaluation. 2. PERFORMANCE EVALUATION METHODOLOGY The performance evaluation must utilize the mixed-methods research design employing both quantitative and qualitative methods to strengthen the validity of the findings and provide room for data triangulation. The Contractor must describe and document the methodological approach that will be used, and this should follow USAID Evaluation best practices. The approach must include an evaluation framework and assessment tools for each performance evaluation question and highlight the conceptual model (s); specify the measurement criteria to be used to respond to each question. It must discuss any risks and limitations that may undermine the reliability and validity of the performance evaluation results. In order to ensure the maximum value for learning and use, a description of the proposed performance evaluation methodology must include the following, at a minimum: a) Methods of data collection: The Contractor must clearly highlight the different methods and tools that will be utilized to collect both quantitative and qualitative data. b) Sampling: The Contractor must propose how sampling will be done and propose the appropriate sample sizes required to ensure scientific rigor. c) Data analysis: The Contractor must provide the plan for analysis of all qualitative and quantitative data collected. This should include how different secondary sources of data collected during the LGAP implementation (e.g. Perception Surveys; M&E reports; Quarterly narrative reports; and Local Authority Performance Assessment-LAPA) will be utilized to answer the performance evaluation questions. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 51 3. EVALUATION PLAN The Contractor will prepare a detailed work plan in response to SOW requirements and evaluation questions. The purpose of the Evaluation Plan is, in part, for the Contractor to present their evaluation design which includes, in part, research questions, methodology for quantitative and qualitative data collection and data analysis, work plan, timeline and proposed domestic and international travel. The Evaluation Plan must be approved by the USAID before the Contractor can travel and begin their field work. USAID/Malawi will provide approval or request changes. 4. DEBRIEFING MEETING The Contractor must debrief USAID/Malawi on the preliminary findings of the performance evaluation. This debrief meeting must provide a summary of any analytical results; discuss challenges, successes, and the way forward. The Contractor must deliver an oral presentation of the LGAP performance evaluation findings, conclusions, and recommendations for each question to USAID/Malawi, prior to finalizing the draft performance evaluation report. The team leader of the performance evaluation team will be required to update the point of contact at USAID/Malawi on the progress at every phase of the performance evaluation. 5. FINDINGS/VALIDATION WORKSHOP The Contractor must present the key findings, conclusions and recommendations at a half-day stakeholder workshop in Lilongwe. The Contractor shall be responsible for costs, logistics and managing invitations to this workshop. The Contractor shall produce a summary/brief (max. 3 pages) of key findings, conclusions and recommendations to be distributed to stakeholders during the workshop. 6. FINAL PERFORMANCE EVALUATION REPORT The Contractor must submit a final performance evaluation report that is based on analyzed facts and evidence and fully addresses all the performance evaluation questions. The report must be no more than 30 pages in length (excluding annexes). The Contractor must ensure that Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. This includes: ● The performance evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why; ● The performance evaluation report must address all evaluation questions included in the scope of work; ● The performance evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, performance evaluation questions, performance evaluation team composition, performance evaluation methodology or timeline need to be agreed upon in writing by the technical officer at USAID/Malawi; ● Performance evaluation methodology must be explained in detail and all tools used in conducting the performance evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; ● Limitations to the performance evaluation must be disclosed in the report, with particular attention to the limitations associated with the performance evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 52 ● Performance evaluation findings must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; ● Sources of information must be properly identified and listed in an annex; ● Recommendations must be supported by a specific set of findings; and should be action￾oriented, practical and specific, with defined responsibility for the action. The final performance evaluation report must contain the following sections: ● Executive Summary: This section shall be 3-5 pages in length and must summarize the purpose, project background, performance evaluation design and methodologies including performance evaluation questions, key findings, conclusions, and recommendations and lessons learned from the evaluation. ● Background: This section must provide a brief description of the LGAP that highlights the scope of the development hypothesis, activities undertaken in the program, and key performance indicators of the program. ● Methodology: This section must detail the methodology and related research protocols undertaken in conducting the performance evaluation, data collection, analysis, selection criteria/sampling, and related constraints or limitations encountered during the LGAP performance evaluation. ● Findings: Empirical facts collected during the performance evaluation: This section must present findings from the performance evaluation. The performance evaluation findings must assess key outcomes as structured around the organizational framework of the performance evaluation questions. The findings must be specific, concise and supported by strong quantitative and qualitative evidence analyzed through scientifically plausible methodologies. Sources of information used in arriving at the findings must be properly acknowledged and listed in an annex. ● Conclusions: Performance evaluation conclusions must be presented for each key finding. The conclusions must logically follow from the gathered data and findings. These must be explicitly justified. If and when necessary, the evaluator must state his/her assumptions, judgments and value premises so that readers can better understand and assess them. ● Recommendations (Proposed actions for management): This section must precisely and clearly present recommendations that must be drawn from specific findings. The recommendations must be stated in an action-oriented fashion, must be practical, specific, and with defined responsibility for the requisite action. The recommendations presented in this section must follow the performance evaluation questions as the organizational framework. ● References: This section should include all documents reviewed, including background documentation and records of technical data application and decision-making. ● Annexes: These may include, but not limited to, statement of work, tools used in conducting the performance evaluation such as questionnaires, checklists, discussion guides, sources of information, etc. C.5. LGAP PERFORMANCE EVALUATION TEAM COMPOSITION The Contractor must propose a performance evaluation team which should be diverse with expertise in different key components of the LGAP award. As a means of building local capacity to undertake performance evaluations, the team composition should at least have one local expert USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 53 as a key member of the performance evaluation team. The local expert will also help to provide context and linkages to key stakeholders in the agricultural sector. At a minimum, the team should comprise the following key personnel: Expatriate Expert, and Local Capacity Development Expert. Below is a list of the proposed key personnel for the evaluation team and their roles and responsibilities. 1. THE EXPATRIATE EXPERT (TEAM LEADER): KEY DUTIES: ● Oversee all the performance evaluation activities and provide overall oversight and management of the performance evaluation team. ● Ensure quality of performance evaluation outputs and timely submission of each deliverable. ● Plan and coordinate stakeholder meetings, field visits, and be responsible for payments of local logistical needs and local staff working with the performance evaluation team. ● Lead the preparation of the performance evaluation reports and presentation of the key performance evaluation findings and recommendations to the USAID/Malawi team QUALIFICATIONS AND EXPERIENCE: ● Advanced degree in Social Sciences or other relevant field with at least 10 years of experience evaluating international development programs, preferably local governance in Africa. ● In-depth knowledge of USAID evaluation requirements preferably for governance programs. ● Strong program evaluation experience in public sector capacity building programs. ● Strong management and administrative skills with experience of evaluating integrated programs. ● Demonstrated knowledge of political economy and institutional analyses. ● Excellent communication and writing skills, analytical skills, interpersonal skills, team management and leadership skills. ● Demonstrated ability to meet deadlines. 2. LOCAL EXPERT- CAPACITY DEVELOPMENT ● Advanced degree in relevant subject development economics, local development, public administration, or equivalent qualifications with over 7 years of experience. ● Minimum of 7 years of regional or international experience in the field of institutional capacity development and proven track record of working on design of capacity development programs for public institutions. ● Proven experience in conducting studies or evaluations on LCD in Southern Africa, preferably in Malawi. ● Good analytical, writing and presentation skills. ● Demonstrated ability to meet deadlines. The successful performance evaluation team will work closely with Mission’s Democracy Rights and Governance (DRG) and M&E specialists in all tasks such as development of instruments and USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 54 piloting, data collection, data analysis and synthesis of results. The DRG team is also expected to collaborate with the performance evaluation team to provide key documents. C.6. PLACE AND DURATION OF THE LGAP PERFORMANCE EVALUATION The place of the LGAP Performance Evaluation activity is Malawi. The extent of in-country travel and data needs will be determined by USAID/Malawi’s performance evaluation design team and the Contractor. The Contracting Officer’s Representative (COR) is the contact person for the LGAP Performance Evaluation. The COR contact information will be shared with the Contractor in due course. The performance evaluation is expected to last 45 working days and the start date is on/or around the beginning of April 2020. C.7. LOGISTICS The Contractor will be responsible for all logistics including coordinating all travel to districts and local councils, lodging, printing, office space, equipment and car rentals, and managing dissemination of results. The USAID/Malawi’s DRG team will provide support to set up initial meetings with key government officials and all relevant stakeholders. C.8. EXISTING SOURCES OF INFORMATION USAID Mission Malawi will share the following documents with the LGAP Performance Evaluation Team to facilitate the desk review. The applicable documents are listed below: 1. LGAP Scope of Work (SoW) 2. Quarterly Progress Reports (FY2016 – FY2019) 3. Local Authority Performance Assessment Survey Report (1st & 2nd Rounds) 4. LGAP Performance Monitoring and Evaluation Plan (PMEP) 5. LGAP Annual Work Plans (FY2016 – FY2019) 6. Citizen Perception Survey Reports USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 55 ANNEX B: EVALUATION DESK REPORT AND WORKPLAN, INCLUDING A DESCRIPTION OF THE ASSESSMENT METHODOLOGY USED, A BIBLIOGRAPHY OF DOCUMENTS CONSULTED, DATA COLLECTION TOOLS, AND LIST OF INTERVIEWEES CONSULTED MALAWI LGAP MIDTERM PERFORMANCE EVALUATION DESK REPORT AND WORKPLAN INTRODUCTION AND PURPOSE This Evaluation Desk Report and Work Plan includes research questions, updated explanation of the evaluation methodology, proposed desk review document list, list of proposed stakeholders to interview, survey instruments, and a tentative schedule of milestones that includes an illustrative schedule of proposed meetings, groups, and visits. The purpose of this mid-term performance evaluation of the LGAP program is to measure how the award has performed and guide USAID/Malawi for future programming. This will include: 1. Assess LGAP’s performance to date and determine whether the program’s activities are achieving the intended results as outlined in the agreement. 2. Assess the extent to which LGAP is achieving the overarching goal of supporting the GOM in the fulfillment of the decentralization policy, including both changes to the form of government (policies, regulations, organizational structures), as well as the function of government through improved performance. 3. Assess the extent to which the LGAP approach of drawing on political economy analysis has increased the effectiveness of the USG/UK investments in decentralization and local governance in Malawi. 4. Document lessons learned, good practices, and points for improvement to make possible adjustments for current implementation of the LGAP award and to inform future activity designs. PROJECT BACKGROUND After operating without locally elected leaders for nearly a decade, Malawi held tripartite elections in 2014 to reintroduce the election of local government leaders. Then the Government of Malawi (GOM) launched several reform initiatives intended to accelerate and deepen an ongoing decentralization process. These reform initiatives created a unique policy window in which USAID/Malawi awarded the Local Government Accountability and Performance (LGAP) contract to build the capacity for responsive, effective, and accountable service delivery through local governments in Malawi. To achieve this, LGAP works to improve local government performance and transparency, increase citizen engagement, and strengthen the enabling environment for decentralization in Malawi. LGAP is a five-year activity that provided support to eight district councils in Malawi in the initial three years of implementation. In September 2019, LGAP expanded to support all 28 councils with additional funding from the United Kingdom’s Department for International Development (DFID). The overall goal of LGAP is to improve the performance of local government in Malawi USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 56 by improving capacity and accountability of local governments in Malawi’s districts, strengthening citizen engagement and oversight, and improving the broader national policy and systems environment for decentralization. LGAP support is primarily via technical assistance to both national and subnational governments, supportive supervision for core local government operations, public financial management and accountability, capacity building for councilors and District Executive Committee staff, and support for integration with USAID activities. Awards are intended to achieve results in the following five areas: 1. Local government performance and transparency increases; 2. Demand for accountable government strengthened at the local level; 3. Decentralization policy environment and systems improved; 4. Public service performance management systems improved and implemented; 5. Public financial management related to human resources improved. There has been significant change in Malawi’s political context from 2014, when LGAP was designed, to 2020, when this evaluation takes place. In 2014, Peter Mutharika had just been elected to the presidency, returning the Democratic Progressive Party as the ruling party after two years out of power. Peter Mutharika’s presidency continued a longer period of what political analyst Diana Cammack (2017) termed a “political settlement,” wherein Malawi’s elite would achieve a minimal level of economic performance and political stability in exchange for popular assent to less-than-democratic governance and stalled development. The recent popular unrest challenging, among other issues, the veracity of the 2019 elections and subsequent election of opposition candidate Lazarus Chakwera offer a unique window of opportunity for governance interventions hoping to build an alternative to Malawi’s political settlement. RESEARCH QUESTIONS The Evaluation Team has identified the following five core questions to guide the Evaluation: 6. To what extent has LGAP implementation increased Malawi Government accountability at the local level? If so, how? 7. To what extent has LGAP implementation contributed to improved service delivery across sectors (health, education and agriculture)? If so, how? 8. What are any unintended consequences or externalities of LGAP support? How can these consequences be mitigated (if negative) or expanded (if positive)? 9. How has the LGAP award used the resources (financial, human resources/technical assistance) efficiently and effectively in relation to the planned activities, outputs and outcomes, and sustainability? 10. What lessons have been learned to inform future governance programming? In particular, how relevant are the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build the capacity for responsive, effective and accountable service delivery through local governments in Malawi? These questions have been informed by the preparatory phase for this Evaluation. Following each of these core questions, the team has developed a set of sub-questions. Table 1, below, lists proposed evaluation questions and sub-questions, data sources, and analysis methods to explore each question. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 57 Table 1: LGAP Midterm Evaluation Matrix NO. EVALUATION QUESTIONS AND SUB-QUESTIONS DATA SOURCES INDICATORS EQ 1: To what extent has LGAP implementation increased Malawi Government accountability at the local level? If so, how? 1.1 To what extent do local government priorities reflect citizens' priorities? Interviews, discussions, evaluation survey, Citizen Perception Surveys (CPS)11 District alignment of priorities expressed by local officials to priorities expressed by citizens Percentage of respondents who expressed satisfaction with how council/councilor responds to community needs 1.2 To what extent do citizens trust local officials to deliver on their promises? Interviews, discussions, evaluation survey Percentage of respondents who express trust in village/district leaders 1.3 To what extent are citizens aware of and participate in participatory planning and external oversight processes? Interviews, discussions, evaluation survey Percentage of beneficiaries reporting positive perceptions of access to information on local spending Percentage of beneficiaries reporting participation in meetings or training programs in the past year 1.4 To what extent do local governments employ transparency mechanisms such as consultative processes or consensus building forums? Interviews, discussions, evaluation survey, Citizen Perception Surveys (CPS) Number of information sharing / transparency mechanisms implemented Percentage of beneficiary respondents satisfied with access to 11 We will include CPS information in four LGAP Wave 1 districts if it is made available to the evaluation team. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 58 information / level of transparency 1.5 Which assistance has been (relatively) more effective in meeting LGAP targeted results for local government accountability? Interviews, discussions Number of activities reported as relevant/useful for fostering accountability 1.6 What planned milestones (e.g. district plans implemented, policy reforms operationalized) have been achieved since the beginning of LGAP? Interviews, discussions, LGAP reports Number of activities completed, goals/milestones realized 1.7 What are the factors that contributed to the achievement of these milestones? Interviews, discussions, LGAP reports Implementation and context factors that facilitated progress 1.8 What factors made it difficult to achieve these milestones? Interviews, discussions Implementation and contextual factors that hindered progress 1.9 What were the planned milestones that have not been met? Interviews, discussions Number of activities cancelled, goals/milestones not realized EQ 2: To what extent has LGAP implementation contributed to improved service delivery across sectors (health, education and agriculture)? If so, how? 2.1 In which sectors has service delivery improved since LGAP has been implemented? Interviews, discussions, evaluation survey, Citizen Perception Surveys (CPS) Reported change in services overall Reported quality of services by sector Beneficiary satisfaction in public service provision 2.2 Which type(s) of assistance has/have been (relatively) more effective in improving service delivery in LGAP districts? Interviews, discussions Number of activities reported as relevant/useful for facilitating service delivery 2.3 Has service delivery improved due to LGAP? If so, what factors contributed to the improvement? What factors impeded it? Interviews, discussions Implementation and context factors that facilitated and hindered progress EQ 3: What are any unintended consequences or externalities of LGAP support? How can these consequences be mitigated (if negative) or expanded (if positive)? USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 59 3.1 Were there any unintended effects of LGAP? Interviews, discussions Unintended consequences identified 3.2 If so, were those effects positive or negative vis-a-vis the overall goal of the program? How can these consequences be mitigated (if negative) or expanded (if positive)? Interviews, discussions Identified implications of unintended consequences for past/current/future activities 3.3 Were there any spillover effects within the broader government unit/office beyond the staff who directly benefited from LGAP? Interviews, discussions Spillover effects identified 3.4 Did LGAP involvement affect the other responsibilities of beneficiary personnel in local governments? Interviews, discussions Reported changes in personnel responsibilities EQ 4: How has the LGAP award used the resources (financial, human resources/technical assistance) efficiently and effectively in relation to the planned activities, outputs and outcomes, and sustainability? 4.1 How were LGAP resources distributed across the different technical areas (e.g. human resources process improvements, institutional capacity building) of focus? Interviews, discussions, LGAP reports Planned and actual allocation of LGAP resources/effort 4.2 What were the efficiency measures put in place in the distribution of resources? Interviews, discussions, LGAP reports Oversight mechanisms/ course corrections employed by LGAP EQ 5: What lessons have been learned to inform future governance programming? In particular, how relevant are the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build the capacity for responsive, effective and accountable service delivery through local governments in Malawi? 5.1 What have been the success stories on supply-side interventions? What have been the lessons learned? Interviews, discussions, surveys Identified successful supply side activities 5.2 What have been the success stories on demand-side interventions? What have been the lessons learned? Interviews, discussions, surveys Identified successful demand side activities APPROACH The methodology for this evaluation will be based on a mixed-methods design to examine activity outcomes to date. The mixed method approach uses combinations of desk-based research, key￾informant interviews, focus group discussions, and a survey conducted amongst LGAP USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 60 beneficiaries. The evaluation team will examine quantitative and qualitative data, where findings will be cross-referenced against each other to achieve validity and to mitigate any methodological limitations each method might have. BACKGROUND RESEARCH As part of the background research, the team will conduct a desk review of relevant policy and background documents and consider prior evaluations, studies and analysis, as well as relevant commentary and analysis. We will also review program work plans, quarterly and annual reports, and M&E records. Please refer to Appendix 1 for the full list of documents included in this desk review. DISTRICT SAMPLING Qualitative and quantitative data collection for the LGAP performance evaluation will take place in 10 of Malawi’s 28 districts. Districts were stratified and then randomly selected to ensure representation across: 1. Wave of LGAP implementation (first or later), to capture the effects of LGAP programmatic decisions on outcomes; 2. Region (north, center, south), to control for regional effects; and 3. Performance of local government (as measured in the Local Authority Performance Assessment, or LAPA), to control for “within group” characteristics that may affect LGAP outcomes. The final district sample (see Table 2 below) includes 4 districts from LGAP’s first wave and 6 districts from the later implementation. Regionally, there are 3 districts from the north, 3 districts from the center, and 4 districts from the south. Finally, the evaluation includes 6 high-performing districts, according to LAPA score, and 4 low-performing districts. Table 2: LGAP Midterm Evaluation District Selection REGION LGAP 1 ST WAVE LGAP LATER High LAPA Scores (2018- 2019) North M’mbelwa Nkhata Bay Center Lilongwe Dowa South Mulanje Mangochi Low LAPA Scores (2018- 2019) North - Karonga Center - Nkhotakota South Balaka Chikwawa INTERVIEWS AND GROUP DISCUSSIONS Interviews and group discussions will be conducted over four weeks, approximately from October 25-November 23, 2020. The research will consist of a minimum of approximately 50 in-depth interviews. In-depth interviews and consultations will occur with key informants and stakeholders, including the main stakeholder groups listed below. The interviewees will be selected to gather the views of key players and stakeholders within beneficiaries at all levels. This list includes representatives from the following: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 61 ● Area Development Committees (ADCs), ● District Executive Committees (DECs), ● Village Development Committees (VDCs), ● Council Service Committees, ● Chiefs, or officials or representatives in chieftaincies, ● National Local Government Finance Committee, ● Parliament, ● Civil Society Organizations12 , ● Public Service Reform Directorate, ● Ministry of Local Government and Rural Development, ● Ministry of Finance and Economic Planning, ● Ministry of Education, Science and Technology, ● Ministry of Health, ● Ministry of Agriculture, ● Local Government Service Commission, ● Malawi Local Government Association, and ● Past and present management personnel from LGAP (USAID and DAI). The team will conduct small group discussions to obtain data on perceptions at specific group levels. Community-based group discussions also provide an understanding of the normative perceptions of LGAP interventions and outcomes. The evaluation team will target groups that have received specific LGAP interventions, such as CSOs that received support on advocacy interventions or village leaders in focus districts to understand the roles of ADCs and VDCs. Group discussants will be grouped by respondent type to ensure groups feel comfortable speaking openly. Please refer to Appendix 2 for the full list of proposed interviewees and small group discussions. INTERVIEW PROTOCOLS AND GROUP DISCUSSION AGENDAS The evaluation team has developed an interview/discussion protocol for a cross-section of respondents from both the Supply and Demand-Side of the beneficiary group. The interviews/discussions will be structured to explore topics regarding the respondent’s occupation and responsibilities as they relate to LGAP objectives, e.g. district planning, council membership or leadership, and revenue and expenditure oversight. The protocols also explore topics on the specific form(s) of assistance the respondent received under LGAP, such as human resources training. The third component of the interview/discussion protocol gathers data on in-depth context and rich information about felt experiences and unanticipated consequences. While the interviewer will probe on verifiable information of the respondent’s feedback, some responses may be based on perceptions, so it is essential to understand the factors contributing to these perceptions. Group discussions will use the same discussion agenda to structure the event and will explore group-based perceptions on LGAP interventions and outcomes. Discussions can be highly useful to not only verify findings from the quantitative surveys but also to better describe the nuances of 12 The evaluation team has contacted DAI and requested a listing of all CSOs that have participated in LGAP activities. Because of a fixed timetable to conduct the field work, the evaluation team will select a group of CSOs to meet with for group discussions. The evaluation team will strive to achieve adequate representation of all stakeholder groups based on the listing from DAI. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 62 attitudes and behaviors in sampled LGAP areas. Comparing data from all visited locations will also provide valuable information by which USAID can determine how specific program activities are closely aligned with intended results, goals, and objectives in each LGAP location. An illustrative guide of themes these interviews and small group discussions will seek to cover is located in Appendix 3. QUALITATIVE ANALYSIS The evaluation team will code data from interviews and group discussions to organize data into a structured set. The team will code data to a Microsoft Excel analytical matrix to identify common responses and significant themes to answer the evaluation questions. The evaluation team will organize findings in logical and comprehensive tabular structures for a more effective analysis. SURVEY Additional quantitative data will be collected in the form of a survey delivered to beneficiaries in the same sample of up to 10 districts, in order to reach a larger number of stakeholders and further address the research questions, with field plan and sample size to be finalized during the preparatory phase. Surveys will be conducted electronically using mobile phones/tablets. The population of interest for the survey are LGAP beneficiaries, including local officials, local civil society actors, as well as ordinary citizens who have a stake in good local governance. In each selected district, the survey will sample participants from four villages and towns; one survey site will be the seat of district government (i.e., the boma); and the remaining three sites in each district will be randomly selected using listing data from the Malawi Population and Housing Census. Up to 16 survey participants will be interviewed in each site (yielding a sample of up to 640 survey interviews). The survey will be conducted in-person over six weeks, approximately from October 24- December 7, 2020. Over this period, the evaluation team and local enumerators will conduct a beneficiary survey using the data collection resources and procedures described below. SURVEY QUESTIONNAIRE The quantitative survey consists of two sections or modules. The first section includes topical questions to collect information on LGAP programming experiences designed to answer the evaluation questions. For example, the survey asks questions on local government performance and transparency, including inquiries on respondent involvement in and opinions on district planning, expenditure oversight, and public service provision. The second section collects socio￾economic/socio-demographic data on beneficiary respondents, such as the age, education, occupation, and other relevant information. This will allow the evaluation team to create a statistical profile of the beneficiary sample and to control for individual or group characteristics in the analysis that might have an effect on program outcomes. The demographic data will also permit comparison of our sample to the nationally representative sample in the Malawi Demographic and Health Survey (DHS), to speak to the study’s generalizability beyond the 10 districts in which we will collect data. The evaluation team will apply best practices and the following survey design/administration principles: 1. Always begin with an informed consent form 2. Questions should be clear and concise 3. Only one topic per question USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 63 4. For categorical questions, answer options should be mutually exclusive 5. For categorical questions, answer options should be exhaustive 6. Questions/sections should be ordered from easiest to answer to the hardest 7. Questionnaires should not be too long to avoid survey fatigue 8. Always assign a unique identifier for each survey The draft survey instrument is available in Appendix 4. The Cloudburst Field Manager will lead enumerator training and pilot testing in coordination with the in-country survey supervisor. The field manager and supervisor will explain key concepts behind each questionnaire and specific instructions on survey administration. Enumerator teams will have survey protocols, a comprehensive reference document with instructions on all tasks throughout the data collection period. Field enumerators will administer the survey using computer-assisted personal interview (CAPI) resources such as mobile phones and tablets. CAPI surveys provide efficiencies during the data collection period with the ability to correct errors, such as using logic checks and appropriate skip patterns. It also provides efficiencies in post-survey tasks by reducing the time to aggregate, clean, and analyze data. DATA ANALYSIS After the creation of a cleaned data set, the evaluation team will conduct a range of techniques to obtain findings that will answer the evaluation questions. The team will perform descriptive statistics (e.g. means, crosstabs) to obtain statistical profiles of the samples and to identify significant results. The team will also do more rigorous analyses where possible, such as regression analysis or calculating correlation coefficients, to add more depth and rigor in examining results. These approaches estimate the effects of determinants simultaneously, so multiple factors are controlled for and outcome estimations are more valid and precise. For both qualitative and quantitative data, the team will disaggregate findings along relevant dimensions, such as gender, location, local affiliation, and assistance received to conduct sub￾group analyses and identify significant outcomes among the beneficiaries. For quantitative data, it is important to note that sub-group analysis will have to consider sample sizes when stating a particular finding. Where possible, the team will not only disaggregate data to look at differences between sub-groups, e.g. gender disaggregation, but will also determine if findings are addressing “higher level” USAID initiatives, e.g. if findings are meeting Gender Equality and Female Empowerment initiatives as a development objective. ETHICAL CONSIDERATIONS The evaluation will follow all ethical practices for human subject research, protecting respondent confidentiality and sensitive information. This includes applying for and receiving Institutional Review Board (IRB) oversight or exemption for the study. Survey respondents will be guaranteed confidentiality and will be able to refuse to answer any of the questions. No findings will be attributed to respondents by name in the report. The evaluation will take effort to ensure that the process of conducting qualitative interviews/discussions will be as independent as possible. In particular, when conducting interviews/discussions, the representatives of the activity or project management (e.g. LGAP/DAI) will be invited to leave the room. In cases where this will not be possible, the circumstances of such situations will be noted in the final report as limitations/bias of the evaluation. To protect respondent confidentiality, evaluation data and notes containing identifying information will be stored securely and destroyed at the close of the evaluation. The USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 64 evaluation will provide de-identified quantitative data and the final list of qualitative respondents interviewed to USAID at the close of the evaluation. LANGUAGE The English versions of the data collection resources will be translated into Chichewa and Chitumbuka, and interviews, discussions, and the survey will be conducted in the interviewee’s preferred language (Chichewa, Chitumbuka, or English). TEAM The evaluation team consists of Reuben Hermoso (International Team Lead), Kim Yi Dionne (International Academic Expert), Chris Phiri (Local Capacity Development Expert), Kate Marple￾Cantrell, (Tasking Coordinator & Evaluation Specialist), and TBD (Enumerators/Research Assistants). RISKS & LIMITATIONS In addition to general risks such as limited time and the availability of proposed respondents, critical to the quality of the analysis is the willingness of respondents to openly and honestly discuss issues, and to share information and insights. The evaluation team will make every effort to build a strong rapport with respondents by explaining the purpose of the study and choosing a private and comfortable environment to complete the interview. The team will also apply proven field techniques, such as forming male-female pairs within enumerator teams in case respondents have preferences. But if respondents are concerned about how results will be used, despite the full disclosures and guarantees given in the consent statement before a survey, the risk exists that respondents may limit their answers. There are also a number of budget and management risks to consider when planning work during the COVID-19 pandemic. The evaluation team currently believes that travel to selected districts will be possible for fieldwork, but we closely track the feasibility of travel, in-person meetings, and safe transportation needs for the field team and be prepared to modify plans for meetings according to the latest conditions. If there are additional restrictions in place at the time of fieldwork, Cloudburst will work together with USAID/Malawi to decide whether the schedule of activities needs to be modified according to the exact nature of the gathering. While the sampling approach captures beneficiary characteristics along the LGAP phases, regions, and district performance, the selection of the 10 districts was still based on a purposive approach. Therefore, the qualitative and quantitative findings can only speak of the sampled group and should not be taken as representative of any other beneficiary group or the population. The high turnover of personnel in local government offices raises the possibility of the evaluation team’s inability to conduct interviews, discussions, or surveys with LGAP beneficiaries, particularly those who participated in the activity’s early stages. The evaluation team will incorporate techniques to address this possibility during the data collection phase, such as applying snowball sampling and request interviewed beneficiaries to refer the team to other possible respondents or give their contact information. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 65 APPENDIX 1: DESK REVIEW DOCUMENT LIST ACADEMIC SOURCES Brewer, K. R. W. (2014). “Three controversies in the history of survey sampling,” Survey Methodology, (December 2013/January 2014), Vol. 39, No. 2, pp. 249-262. Cammack, Diana. (2017) Malawi's Political Settlement: Crafting Poverty and Peace, 1994–2014. Journal of International Development 29.5, pp. 661-677. Dionne, Kim Yi, and Horowitz, Jeremy. (2016) The political effects of agricultural subsidies in Africa: Evidence from Malawi. World Development 87, pp. 215-226. Dulani, Boniface, and Dionne, Kim Yi. (2014) Presidential, parliamentary, and local government elections in Malawi, May 2014. Electoral Studies 36, pp. 218-225. Ferree, Karen and Horowitz, Jeremy. (2010) Ties that bind? The rise and decline of ethno-regional partisanship in Malawi. Democratization 17.3, pp. 534-563. Posner, Daniel N. (1995) Malawi’s new dawn. Journal of Democracy 6.1, pp. 131-145. Tambulasi, Richard IC, and Kayuni, Happy M. (2007) Decentralization opening a new window for corruption: An accountability assessment of Malawi's four years of democratic local governance. Journal of Asian and African Studies 42.2, pp. 163-183. NEWSPAPER AND WEB RESOURCES Dionne, Kim Yi, and Dulani, Boniface. (2020) Malawians elected a new president in Tuesday’s historic election. The Washington Post (June 28): https://www.washingtonpost.com/politics/2020/06/28/malawians-elected-new-president￾tuesdays-historic-election/ LGAP PROGRAM DOCUMENTS PROJECT DESCRIPTION USAID/Malawi. (n.d.) Amendment of Solicitation/Modification of Contract. USAID. (n.d.) Section C Description/Specification/Statement of work. Malawi Local Government Accountability and Performance (LGAP). WORK PLANS DAI. (2017) Malawi Local Government Accountability and Performance (LGAP) Annual Work Plan-Draft October 1, 2016 – September 30, 2017. USAID. (2017) Local Government Performance and Accountability (LGAP) Work Plan October 1, 2017 to September 30, 2018 (Year 2). USAID. (2018) Local Government Performance and Accountability (LGAP) Work Plan October 1, 2018 to September 30, 2019 (Year 3). USAID. (2019) Local Government Performance and Accountability (LGAP) Work Plan October 1, 2019 to September 30, 2020 (Year 4). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 66 ANNUAL REPORTS DAI/USAID. (2017, October 30). Malawi Local Government Accountability and Performance (LGAP) Annual Report Fiscal Year 2017. DAI/USAID. (2018, October 30). Malawi Local Government Accountability and Performance (LGAP) Annual Report Fiscal Year 2018. DAI/USAID. (2019, October 30). Malawi Local Government Accountability and Performance (LGAP) Annual Report Fiscal Year 2019. QUARTERLY REPORTS USAID/DAI. (2017) Malawi Local Government Accountability and Performance. Quarterly Report Fiscal Year 2017 Quarter I. USAID/DAI. (2017) Malawi Local Government Accountability and Performance. Quarterly Report Fiscal Year 2017 Quarter 2. USAID/DAI. (2017) Malawi Local Government Accountability and Performance. Quarterly Report Fiscal Year 2017 Quarter 3. USAID/DAI. (2018) Malawi Local Government Accountability and Performance. First Quarterly Report Fiscal Year 2018 October 2017 to December 2017. USAID/DAI. (2018) Malawi Local Government Accountability and Performance. Second Quarterly Report Fiscal Year 2018 January 2018 to March 2018. USAID/DAI. (2018) Malawi Local Government Accountability and Performance. Third Quarterly Report Fiscal Year 2018 April 2018 to June 2018. USAID/DAI. (2019) Malawi Local Government Accountability and Performance. Year 3 First Quarter Report October-December 2018. USAID/DAI. (2019) Malawi Local Government Accountability and Performance. FY2019 January￾March Quarterly Report. USAID/DAI. (2019) Malawi Local Government Accountability and Performance. Quarterly Report Third Quarter-Fiscal Year 2019 April 2019-June 2019. USAID/DAI. (2020) Malawi Local Government Accountability and Performance. Quarterly Report First Quarter-Fiscal Year 2020 October 2019-December 2019. CITIZEN PERCEPTION SURVEYS USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Balaka-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Blantyre (Rural)-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Kasungu-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Lilongwe (Rural)-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-M’Belwa-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Machinga-May 2017. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 67 USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Mulanje-May 2017. USAID/LGAP. (2017, May). Citizen Perception Survey: Summary Report-Zomba-May 2017. LOCAL AUTHORITY PERFORMANCE ASSESSMENTS USAID/LGAP. (n.d.) Local Authority Performance Assessment: Summary Findings and Recommendations. MLGRD/DAI. (2020, April) Local Authority Performance Assessment (LAPA) for 2018-2019 Fiscal Year. MONITORING AND EVALUATION PLANS USAID/DAI. (n.d.) Activity Performance Monitoring and Evaluation Plan: Malawi Local Government Accountability and Performance (LGAP) project. Time Period Coverage: August 2016 to August 2021 USAID/DAI. (2019, October) Activity Performance Monitoring and Evaluation Plan (APMEP): Malawi Local Government Accountability and Performance (LGAP). Time Period Covered: October 2019-August 2021 IMPACT EVALUATION Martin, L., & Scim, B. (2020, February 03). Pre-Analysis Plan: Increasing Tax Compliance in Malawian Markets. Martin, L., Seim, B., & Hoellerbauer, S. (2020, January). Impact Evaluation of USAID/Malawi Local Government Accountability and Performance (LGAP) Activity Final Report OTHER LGAP REPORTS DAI and MobileAccord/GeoPoll. (2017) Malawi Local Government Accountability and Performance (LGAP): Measuring Social Accountability and Citizen Perception 2017. USAID/DAI. (2017, July 30) Malawi Local Government Accountability and Performance. Report on the capacity assessment of the ministry of local government and rural development (District Councils) in light of the new establishment warrant. USAID/LGAP. (n.d.) Study on Improving Local Revenues for District Councils in Malawi. Summary of Findings & Recommendations. USAID/LGAP. (n.d.) Institutional Capacity assessment of Local Governments. Summary of Findings & Recommendations. USAID/LGAP. (n.d.) Study of Measuring Social Accountability and Citizen Perception. Summary Findings & Recommendations. USAID/LGAP. (n.d.) Political Economy of Decentralization and Local Government Reforms – Malawi. Summary of Findings & Recommendations. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 68 USAID & GOM REPORTS USAID/Malawi. (2020, March 13). Country Development Cooperation Strategy (CDCS) 2013- 2018. USAID/Malawi. (2020). Country Development Cooperation Strategy (CDCS) 2020-2025. Government of Malawi (n.d.) Guidebook on Local Government System in Malawi. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 69 APPENDIX 2: LIST OF INTERVIEWEES AND FOCUS GROUP DISCUSSION PARTICIPANTS LIST OF STAKEHOLDERS TO INTERVIEW Table 3, below, contains key informants and focus group discussion respondents that the team will prioritize speaking with through approximately 50 interviews and focus group discussions during in-country fieldwork. Within each of the ten selected districts, the team will use group interviews to maximize the number of informants consulted during approximately four qualitative events per district. Table 3: List of Stakeholders to Interview CATEGORY 1: MINISTRIES AND DEPARTMENTS AT CENTRAL GOVERNMENT LEVEL MINISTRY OF LOCAL GOVERNMENT AND RURAL DEVELOPMENT No. Designation Contact Address 1. Director of Local Government Services P.O. Box 30312, Lilongwe 3 2. Deputy Director Decentralization P.O. Box 30312, Lilongwe 3 3. Deputy Director of Planning and Development P.O. Box 30312, Lilongwe 3 MINISTRY OF EDUCATION No. Designation Contact Address 4. Director of Curriculum Development Malawi Institute of Education –Zomba 5. Executive Director Malawi Institute of Education –Zomba 6. Director of Basic Education Capital Hill-Lilongwe 7. Deputy Director of Basic Education Capital Hill-Lilongwe 8. Director of Inspection Capital Hill-Lilongwe 9. Deputy Director of Inspection Capital Hill-Lilongwe 10. Chief School Inspector Capital Hill-Lilongwe 11. Director of Education Planning Capital Hill-Lilongwe MINISTRY OF FINANCE No. Designation Contact Address 12. Secretary to the Treasury Capital Hill, P.O. Box 30049, Lilongwe 13. Director – Public Financial Management Systems Capital Hill, P.O. Box 30049, Lilongwe MINISTRY OF ECONOMIC PLANNING AND DEVELOPMENT No. Designation Contact Address 14. Director of Planning and Development P.O. Box 30136, Lilongwe 3 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 70 DEPARTMENT OF HUMAN RESOURCE MANAGEMENT AND DEVELOPMENT - OFFICE OF THE PRESIDENT AND CABINET No. Designation Contact Address 15. Principle Secretary Capital Hill, P.O. Box 30227, Lilongwe 3 16. Director of HR Capital Hill, P.O. Box 30227, Lilongwe 3 PUBLIC SERVICE REFORM MANAGEMENT UNIT - OFFICE OF THE PRESIDENT AND CABINET No. Designation Contact Address 17. Chief Director Capital Hill, P.O. Box 30227, Lilongwe 3 18. Director (Planning and Management) Capital Hill, P.O. Box 30227, Lilongwe 3 CATEGORY 2: QUASI-GOVERNMENT ORGANIZATION PARLIAMENTARY COMMITTEE ON LOCAL GOVERNMENT AND DECENTRALIZATION No. Designation Contact Address 19. The Chairperson Private Bag B362 Capital City Lilongwe MALAWI LOCAL GOVERNMENT ASSOCIATION (MALGA) No. Designation Contact Address 20. Executive Director Private Bag 34 Lilongwe NATIONAL LOCAL GOVERNMENT FINANCE COMMITTEE No. Designation Contact Address 21. Executive Director Private Bag 352, Lilongwe 3 22. Director of Finance Private Bag 352, Lilongwe 3 23. Director of Infrastructure Private Bag 352, Lilongwe 3 24. Director of Corporate Services Private Bag 352, Lilongwe 3 25. Capacity Development Specialist Private Bag 352, Lilongwe 3 26. Knowledge Management Specialist Private Bag 352, Lilongwe 3 CATEGORY 3: CIVIL SOCIETY ORGANIZATIONS NATIONAL INITIATIVE FOR CIVIC EDUCATION (NICE) No. Designation Contact Address 27. Executive Director P.O. Box 1046, Lilongwe CATHOLIC COMMISSION FOR JUSTICE AND PEACE No. Designation Contact Address 28. National Coordinator P.O. Box 30384 Lilongwe 3 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 71 HUMAN RIGHT DEFENDERS COALITION (HRDC) No. Designation Contact Address 29. The Chairperson Area 47 Sector 3, Ng’ombe Khwawa Road, House No. 61 CATEGORY 4: LOCAL GOVERNMENT AT DISTRICT LEVEL DISTRICT COUNCIL SECRETARIATS (10 DISTRICTS) No. Designation Contact Address 30. District Commissioner Balaka 31. Director of Planning and Development Balaka 32. Director of Finance Balaka 33. District Education Manager Balaka 34. District Commissioner Chikwawa 35. Director of Planning and Development Chikwawa 36. Director of Finance Chikwawa 37. District Education Manager Chikwawa 38. District Commissioner Dowa 39. Director of Planning and Development Dowa 40. Director of Finance Dowa 41. District Education Manager Dowa 42. District Commissioner Karonga 43. Director of Planning and Development Karonga 44. Director of Finance Karonga 45. District Education Manager Karonga 46. District Commissioner Lilongwe 47. Director of Planning and Development Lilongwe 48. Director of Finance Lilongwe 49. District Education Manager Lilongwe 50. District Commissioner M’mbelwa 51. Director of Planning and Development M’mbelwa 52. Director of Finance M’mbelwa 53. District Education Manager M’mbelwa-South 54. Chief Education Officer M’mbelwa-North 55. District Commissioner Mangochi 56. Director of Planning and Development Mangochi 57. Director of Finance Mangochi 58. District Education Manager Mangochi 59. District Commissioner Mulanje 60. Director of Planning and Development Mulanje 61. Director of Finance Mulanje 62. District Education Manager Mulanje 63. District Commissioner Nkhata Bay 64. Director of Planning and Development Nkhata Bay 65. Director of Finance Nkhata Bay 66. District Education Manager Nkhata Bay 67. District Commissioner Nkhotakota 68. Director of Planning and Development Nkhotakota 69. Director of Finance Nkhotakota USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 72 DISTRICT COUNCIL SECRETARIATS (10 DISTRICTS) – CONT’D No. Designation Contact Address 70. District Education Manager Nkhotakota DISTRICT FULL COUNCIL CHAIRS (10 DISTRICTS) No. Designation Contact Address 71. Council Chair Balaka 72. Council Chair Chikwawa 73. Council Chair Dowa 74. Council Chair Karonga 75. Council Chair Lilongwe 76. Council Chair M’mbelwa 77. Council Chair Mangochi 78. Council Chair Mulanje 79. Council Chair Nkhata Bay 80. Council Chair Nkhotakota DISTRICT COUNCIL SERVICE COMITTEES (10 DISTRICTS) No. Designation Contact Address 81. Finance Committee Chair Balaka 82. Development Service Committee Chair Balaka 83. Education Service Committee Chair Balaka 84. Works Service Committee Chair & Human Resources Service Committee Chair Balaka 85. Health and Environment service. Committee Chair Balaka 86. Agriculture Service Committee Chair Balaka 87. Finance Committee Chair Chikwawa 88. Development Service Committee Chair Chikwawa 89. Education Service Committee Chair Chikwawa 90. Works Service Committee Chair Chikwawa 91. Health and Environment service. Committee Chair Chikwawa 92. Human Resources Service Committee Chair Chikwawa 93. Agriculture Service Committee Chair Chikwawa 94. Chair, Finance Committee Dowa 95. Chair, Development Committee Dowa 96. Chair, Education Committee Dowa 97. Chair, Human Resource Committee Dowa 98. Chair, Development Committee Karonga 99. Chair, Education Committee Karonga 100. Chair, Human Resource Committee Karonga 101. Chair, Finance Committee Lilongwe 102. Chair, Development Committee Lilongwe 103. Chair, Education Committee Lilongwe 104. Chair, Human Resource Committee Lilongwe 105. Chair, Finance Committee Mangochi 106. Chair, Development Committee Mangochi 107. Chair, Education Committee Mangochi USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 73 DISTRICT COUNCIL SERVICE COMITTEES (10 DISTRICTS) – CONT’D No. Designation Contact Address 108. Chair, Human Resource Committee Mangochi 109. Chair, Finance Committee Mbelwa 110. Chair, Development Committee Mbelwa 111. Chair, Education Committee Mbelwa 112. Chair, Human Resource Committee Mbelwa 113. Chair, Public Works Mulanje 114. Chair, Health and ENV Mulanje 115. Chair, Agriculture Mulanje 116. Chair, Education Mulanje 117. Chair, Finance Mulanje 118. Chair, Human Resource Mulanje 119. Chair, Development Mulanje 120. Chair, Finance Committee Nkhatabay 121. Chair, Development Committee Nkhatabay 122. Chair, Education Committee Nkhatabay 123. Chair, Human Resource Committee Nkhatabay 124. Chair, Finance Committee Nkhotakota 125. Chair, Development Committee Nkhotakota 126. Chair, Education Committee Nkhotakota 127. Chair, Human Resource Committee Nkhotakota DISTRICT CSO NETWORK (10 DISTRICTS) No. Designation Contact Address 128. CSO Chair MZIDEN – Mzimba Boma 129. CSO Chair FOCUS, Private Bag 17, Karonga 130. CSO Chair YONECO – Nkhata Bay 131. CSO Chair C/o NICE Chikwawa Office 132. CSO Chair Mulanje District 133. CSO Chair Lilongwe District Council 134. CSO Chair Dowa District Council 135. CSO Chair Nkhotakota 136. CSO Chair Balaka 137. CSO Chair P.O. Bix 194, Mangochi 138. DEN Chair Mzimba south 139. DEN Chair Mzimba north 140. DEN Chair Nkhatabay 141. DEN Chair Karonga 142. DEN Chair Balaka DEN 143. DEN Chair Mangochi DEN 144. DEN Chair Dowa 145. DEN Chair Nkhotakota 146. DEN Chair Lilongwe 147. DEN Chair Chikwawa 148. DEN Chair Mulanje USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 74 CATEGORY 5: AREA DEVELOPMENT COMMITTEE (ADC) LEVEL (10 DISTRICTS) No. Designation Contact Address 149. ADC Chair Balaka-Chakanza ADC 150. ADC Chair Balaka-Nsamala ADC 151. ADC Chair Balaka-Mpilisi ADC 152. ADC Chair Balaka-Kapalamula ADC 153. ADC Chair Balaka-Phimbi ADC 154. ADC Chair Balaka-Nkaya ADC 155. ADC Chair Balaka-Chanthunya ADC 156. ADC Chair Balaka-Nyanyala ADC 157. ADC Chair Balaka-Phalula ADC 158. ADC Chair Balaka-Sawali ADC 159. ADC Chair Balaka-Matola ADC 160. ADC Chair Balaka-Toleza ADC 161. ADC Chair Balaka-Mbera ADC 162. ADC Chair Balaka-Kachenga ADC 163. ADC Chair Balaka-Amidu ADC 164. ADC Chair Balaka-Kalembo ADC 165. ADC Chair Balaka-Mpoto ADC 166. ADC Chair Chikwawa-Malemia 167. ADC Chair Chikwawa-Mulolo 168. ADC Chair Chikwawa-Mbenje 169. ADC Chair Chikwawa-Tengani 170. ADC Chair Chikwawa-Ngabu 171. ADC Chair Chikwawa-Makoko 172. ADC Chair Chikwawa-Chimombo 173. ADC Chair Chikwawa-Nyachikazi 174. ADC Chair Chikwawa-Ndamera 175. ADC Chair Dowa-Mponela ADC 176. ADC Chair Dowa-Dzoole B ADC 177. ADC Chair Dowa-Mkukula B ADC 178. ADC Chair Dowa-Chakhaza A ADC 179. ADC Chair Dowa-Msakambewa ADC 180. ADC Chair Dowa-Kayembe ADC 181. ADC Chair Dowa-Dzoole A ADC 182. ADC Chair Dowa-Mkukula A ADC 183. ADC Chair Dowa-Chakhaza B ADC 184. ADC Chair Dowa-Chiwere ADC 185. ADC Chair Karonga-Mbande ADC 186. ADC Chair Karonga-Iponga ADC 187. ADC Chair Karonga-Ngerenge ADC 188. ADC Chair Karonga-Lupembe ADC 189. ADC Chair Lilongwe-Chimutu ADC 190. ADC Chair Lilongwe-Mtema ADC 191. ADC Chair Lilongwe-Chiseka ADC 192. ADC Chair Lilongwe-Chitekwere ADC 193. ADC Chair Lilongwe-Malili ADC 194. ADC Chair Lilongwe-M’bang’ombe ADC 195. ADC Chair Lilongwe-Kalolo North ADC 196. ADC Chair Lilongwe-Kalolo South ADC 197. ADC Chair Lilongwe-Kalumbu Mtenthera ADC USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 75 CATEGORY 5: AREA DEVELOPMENT COMMITTEE (ADC) LEVEL (10 DISTRICTS) – CONT’D No. Designation Contact Address 198. ADC Chair Lilongwe-Kalumbu Nyanja ADC 199. ADC Chair Lilongwe-Njewa ADC 200. ADC Chair Lilongwe-Chitukula ADC 201. ADC Chair Lilongwe-Masumbankhunda ADC 202. ADC Chair Lilongwe-Tsabango ADC 203. ADC Chair Lilongwe-Chadza ADC 204. ADC Chair Lilongwe-Mazengera ADC 205. ADC Chair Lilongwe-Masula ADC 206. ADC Chair Lilongwe-Kabudula ADC 207. ADC Chair Lilongwe-Khongoni ADC 208. ADC Chair Lilongwe-Kalumba ADC 209. ADC Chair Lilongwe-M’bwatalika ADC 210. Chair, ADC Mangochi-Bwananyambi 1 ADC 211. Chair, ADC Mangochi-Majuni ADC 212. Chair, ADC Mangochi-Malombola ADC 213. Chair, ADC Mangochi-Koche ADC 214. Chair, ADC Mangochi-Mbwadzulu ADC 215. Chair, ADC Mangochi-Lulanga ADC 216. Chair, ADC Mangochi-Masanje ADC 217. Chair, ADC Mangochi-Nkope ADC 218. Chair, ADC Mangochi-Katuli ADC 219. Chair, ADC Mangochi-Thumu ADC 220. Chair, ADC Mangochi-Malembo ADC 221. Chair, ADC Mangochi-Majiga ADC 222. Chair, ADC Mangochi-Namisi ADC 223. Chair, ADC Mangochi-Phirilongwe ADC 224. Chair, ADC Mangochi-Mpita ADC 225. Chair, ADC Mangochi-Lutufu ADC 226. Chair, ADC Mangochi-Chimwala ADC 227. Chair, ADC Mangochi-Mponda ADC 228. Chair, ADC Mangochi-Chilipa ADC 229. Chair, ADC Mangochi-Chowe ADC 230. Chair, ADC Mangochi-Makanjira ADC 231. Chair, ADC Mangochi-Chiwunda ADC 232. Chair, ADC Mangochi-Namavi ADC 233. Chair, ADC Mangochi-Malindi ADC 234. Chair, ADC Mangochi-Bwananyambi 2 ADC 235. Chair, ADC Mangochi-Mvumba ADC 236. Chair, ADC Mangochi-Fort Maquire ADC 237. ADC Chair Mbelwa-Chindi ADC 238. ADC Chair Mbelwa-Mzukuzuku ADC 239. ADC Chair Mbelwa-Mmbelwa ADC 240. ADC Chair Mbelwa-Mzikuola ADC 241. ADC Chair Mbelwa-Kampingo Sibande ADC 242. ADC Chair Mbelwa-Khosolo ADC 243. ADC Chair Mulanje-Mulomba ADC 244. ADC Chair Mulanje-Ndanga ADC 245. ADC Chair Mulanje-Nkanda ADC 246. ADC Chair Mulanje-Juma ADC USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 76 CATEGORY 5: AREA DEVELOPMENT COMMITTEE (ADC) LEVEL (10 DISTRICTS) – CONT’D No. Designation Contact Address 247. ADC Chair Mulanje-Chikumbu ADC 248. ADC Chair Mulanje-Mkumbiza ADC 249. ADC Chair Mulanje-Mthiramanga ADC 250. ADC Chair Mulanje-Njema ADC 251. ADC Chair Mulanje-Mimosa ADC 252. ADC Chair Mulanje-Milonde ADC 253. ADC Chair Nkatabay-Zilakoma A ADC 254. ADC Chair Nkatabay-Zilakoma B ADC 255. ADC Chair Nkatabay-Fukambiri ADC 256. ADC Chair Nkatabay-Malengamzoma ADC 257. ADC Chair Nkatabay-Malanda ADC 258. ADC Chair Nkatabay-Fukamalaza ADC 259. ADC Chair Nkatabay-Mankhambira ADC 260. ADC Chair Nkatabay-Mkumbira ADC 261. ADC Chair Nkatabay-Timbiri ADC 262. ADC Chair Nkatabay-Thula ADC 263. ADC Chair Nkatabay-Chipimbininga ADC 264. ADC Chair Nkatabay-Nyaluwanga ADC 265. ADC Chair Nkatabay-M’bwana ADC 266. ADC Chair Nkatabay-Mkondowe ADC 267. ADC Chairperson Nkhotakota-Mwansambo 268. AEC Chairperson Nkhotakota-Mwansambo 269. ADC Chairperson Nkhotakota-Mwadzama 270. AEC Chairperson Nkhotakota-Mwadzama 271. ADC Chairperson Nkhotakota-Nkhanga 272. AEC Chairperson Nkhotakota-Nkhanga 273. ADC Chairperson Nkhotakota-Malengachanzi 274. AEC Chairperson Nkhotakota-Malengachanzi 275. ADC Chairperson Nkhotakota-Mphonde 276. AEC Chairperson Nkhotakota-Mphonde 277. ADC Chairperson Nkhotakota-Kanyenda 278. AEC Chairperson Nkhotakota-Kanyenda 279. ADC Chairperson Nkhotakota-Kafuzira 280. AEC Chairperson Nkhotakota-Kafuzira 281. ADC Chairperson Nkhotakota-Lupachi Sub-ADC 282. AEC Chairperson Nkhotakota-Mwansambo CATEGORY 6: DEVELOPMENT PARTNERS USAID No. Designation Contact Address 283. LGAP Chief of Party DAI 284. Former COR USAID/Malawi 285. COR USAID/Malawi 286. Former Director of DRG / Senior Governance Adviser USAID/Malawi USAID/DG USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 77 USAID – CONT’D No. Designation Contact Address 287. Governance Advisor British High Commission 288. Education Advisor British High Commission CATEGORY 7: GRANTEE PARTNERS (CSOS) USAID No Designation Contact Address 289. Executive Director, Tovwirane M’mbelwa South 290. Executive Director, Foundation for Community and Capacity Development Nkhota Kota 291. Team Leader, Rights Advice Centre Mangochi 292. Executive Director, Nkhadze Youth Alive Balaka 293. Executive Director, Circle for Integrated Community and Rural Development Chikwawa 294. Chief Executive Officer, mHub Area 15 Chiwengo Road, Plot # 46, P.O. Box 1689 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 78 APPENDIX 3: SAMPLE INTERVIEW GUIDE INTERVIEWS (SEMI-STRUCTURED) AND GROUP DISCUSSIONS This guide is presented as a sample or illustrative interview protocol. Please note that, as interviews will be semi-structured, these questions will not necessarily be asked verbatim or in this order. Instead, the guide will act as a memory aide or a checklist for the interviewers, to ensure that relevant topics have been covered to the extent possible. Additionally, successful semi-structured interviewing requires interviewer flexibility to pursue useful themes outside of those listed or to focus on a subset of themes where a source is particularly informative. As such, this guide will serve as a “living” document during fieldwork, and questions may be dropped, added, or revised during fieldwork. INTRODUCTION (ALL) Thank you very much for taking the time to meet with us. Before we start, I’d like to give you a bit more context for the research. I’m an independent consultant engaged by the Cloudburst Group and an expert in local governance. Our Research Assistant, _____________, is also here to take notes. We are conducting a performance evaluation of USAID’s and DIFD’s Local Governance Accountability and Performance (LGAP) project, assessing LGAP’s performance to date and determine whether the program’s activities are achieving the intended results as outlined in the agreement. We hope to learn whether and how LGAP is achieving its goal of supporting the Government of Malawi in the fulfillment of its decentralization policy, including both changes to the form of government (policies, regulations, organizational structures) and the function of government through improved performance. We asked to meet with you both to receive relevant background information and to hear your thoughts on the performance of LGAP in these areas. Your contribution is very important to us and we appreciate your time and input. Results of this interview may be used in assessment reporting, and this report will be made publicly available online. However, we will ask for your permission if we are considering a direct quote and otherwise your responses will be kept confidential. And of course, you do not need to answer every question. Do you have any questions for us before we start? USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 79 Date: Interview No. RESPONDENT INFORMATION Name: Occupation/Title: Organization: DHRMD MFEP&D Other: LGSC MLGRD District Council: OPC NLGFC ADC: MALGH VDC: INTERVIEW QUESTIONS 1 Please provide a brief background about your participation in the LGAP activity. Probes: Activities (trainings, consultations), Sector, Area (HR, PFM, Governance), Years Notes: EQ 1: To what extent has LGAP implementation increased Malawi Government ACCOUNTABILITY at the local level? If so, how? 2 Do you think the LGAP activities were effective in meeting the target to improve local government accountability? Yes ___ Please explain: Notes: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 80 No ___ Skip to 4 3 Which activities were (more) effective? Notes: 4 What factors made it (in)effective? Notes: 5 Were there planned LGAP activities that were not achieved? Yes ___ Please explain: Notes: No ___ Skip to 7 6 What factors made these activities difficult to achieve? Notes: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 81 EQ 2: To what extent has LGAP implementation contributed to improved service delivery across sectors ? If so, how? 7 Has LGAP helped improve service delivery across the targeted sectors? Yes ___ If yes, in which sectors? Notes: No ___ Skip to 11 Public works, health, environment Agriculture and natural resources Education Commerce Gender Cross-cutting 8 Please explain how service delivery improved in this sector. Notes: 9 Please explain which LGAP activities improved service delivery. Notes: 10 Do you think service delivery improved throughout the organization or just in key areas/offices? Probe: e.g., The whole district council or just some offices/sub-units Notes: 11 Why did it (not) help improve service delivery? Notes: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 82 EQ 3: What are any unintended consequences or externalities of LGAP support? How can these consequences be mitigated (if negative) or expanded (if positive)? 12 Did the LGAP result in any positive externalities in your work or organization? Probes: Positive externalities: spillover effects in the organization Yes ___ Please explain: Notes: No ___ Skip to 14 13 Will your organization continue the positive externalities gained? Yes ___ Please explain: Notes: No ___ 14 Did the LGAP result in any unintended consequences in your work or organization? Probes: Unintended consequences: Unexpected added workload or responsibilities Yes ___ Please explain: Notes: No ___ Skip to 17 15 How did you address these unintended consequences? Notes: 16 Did you coordinate with USAID on addressing these unintended consequences? Yes ______ Please explain: Notes: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 83 No ___ EQ 4: How has the LGAP award used the resources (financial, human resources/technical assistance) efficiently and effectively in relation to the planned activities, outputs and outcomes, and sustainability? 17 Do you think LGAP provided enough technical assistance (TA) to you and your organization? Probes: TA refers to trainings, consultations, assessments, mentorships, and all other non-physical support Yes ___ Please explain: Notes: No ___ 18 Would you or your organization be able to maintain the skills gained from LGAP even if, hypothetically, the program would end tomorrow? Yes ___ Please explain: Notes: No ___ 19 Do you think LGAP provided enough manuals, toolkits, and other resource documents even if, hypothetically, the program would end tomorrow? Yes ___ Please explain: Notes: No ___ 20 Do you think LGAP provided enough financial and physical resources for your organization? Probe: Physical resources, e.g. LAN network for IFMIS assistance Yes Financial ___ Please explain: Notes: Yes Physical ___ Skip to No ___ Skip to USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 84 21 Please discuss any measures your organization took to use financial resources efficiently. Notes: EQ 5: What lessons have been learned to inform future governance programming? In particular, how relevant are the result areas and sub-result areas in the LGAP award to achieving the activity’s overall objective to build the capacity for responsive, effective and accountable service delivery through local governments in Malawi? 22 What specific support would you keep if, hypothetically, there will be a new program similar to LGAP? Probes: Specific support, e.g. If planning what specific plan, VAP, SEP? Notes: USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 85 23 What specific support would you remove if, hypothetically, there will be a new program similar to LGAP? Probes: Specific support, e.g. If planning what specific plan, VAP, SEP? Notes: 24 Have you participated in a program similar to LGAP but given by another organization? Probe: GIZ, EU Notes: END OF INTERVIEW. THANK YOU. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 86 APPENDIX 4: SURVEY This draft survey is presented below for USAID review. Before deployment the survey will be piloted and refined to ensure survey questions and flow are easily understandable. INTRODUCTION AND CONSENT Hello. My name is [ENUMERATOR NAME]. I am working with Cloudburst and USAID. We are conducting a study to evaluate the USAID/Malawi Local Government Accountability Performance program. The results of this study may help to improve the programs offered in Malawi in the future. You have been randomly selected to participate in this study if you choose to. If you agree to participate in this study, we would like to ask some survey questions. This interview will take about 45 minutes. You will be asked a few questions about yourself and about your opinions about local services. Your participation is completely voluntary. You can choose not to participate now, or at any time during the survey. All information collected in this study is confidential. Your answers will be grouped with the answers of other people like you and your name will not be used or shared with anyone outside of this study. There are no known risks of participating in this research project other than losing 45 minutes of productive time. There are also no direct benefits to you if you participate other than knowing your information will help USAID understand whether its programs are working or whether they need to improve. Should you feel uncomfortable with any question(s), you may refuse to answer it and I will move on to the next question. If you have any questions or concerns now or in the future, you may contact [LOCAL SUPERVISOR NAME] at [SUPERVISOR MOBILE PHONE NUMBER] or [SUPERVISOR EMAIL ADDRESS]. Or you can contact Kate Marple-Cantrell at Cloudburst: +1-240-582-3638 or kate.marple-cantrell@cloudburstgroup.com. Do you have any questions? PRE-INTERVIEW QUESTIONS TO BE COMPLETED BY ENUMERATOR 1 District Balaka Chikwawa Dowa Karonga Lilongwe M’mbelwa Mangochi Mulanje Nkhata Bay Nkhotakota 2 Urban or rural? Urban Rural 3 Town/Traditional Authority Name (List TK) 4 Village/Neighborhood [Text fill] Are the following facilities present in the area? USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 87 5 School (private, public or both)? Yes No Unable to determine 6 Health clinic (private, public or both)? Yes No Unable to determine 7 Market stalls (selling groceries or clothing)? Yes No Unable to determine 8 Enumerator Name (List TK) 9 Consent to participate? Yes No (End) 10 Language of interview Chichewa Chitumbuka English Other, Specify: [Text fill] 11 Respondent gender Male Female Other 12 Create a unique survey ID using the following format: MMDDID1ID2, where MM corresponds to month, DD corresponds to day, ID1 corresponds to the interviewer’s ID (1, 2, 3, or 4), and ID2 corresponds to the order interview of the day (1st, 2nd, 3rd, or 4th). [Text fill, numbers only] MODULE I: PERCEPTIONS OF LOCAL GOVERNMENT PERFORMANCE I would like to ask you some questions about services in your community. 13 Are government services in your community better or worse than they were five years ago? Much worse Worse The same Better Much better Don’t know 14 Which service requires the most improvement in your community? Health Education Water Agriculture Roads USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 88 Bridges Other, specify: 15 How satisfied were you with the behavior of the health worker during your last visit? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 16 How satisfied were you with the availability of medicines at your facility? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 17 How satisfied are you with the work and involvement of the Health Advisory Committee for your area? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 18 How satisfied are you with the availability and attendance of teachers at the school in your area? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 19 How satisfied are you with the learning and teaching materials available at the school in your area? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 20 How satisfied are you with the work and involvement of your Parent Teacher Association (PTA)? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 21 How satisfied are you with the quality of agricultural extension services near you? Very dissatisfied Dissatisfied Neutral USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 89 Satisfied Very satisfied Don’t know 22 How satisfied were you with the Fertilizer Subsidy Input Programme last year? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 23 Do you know who your councillor is? Yes No Not sure/Don’t know 24 How satisfied are you with how your councillor consults citizens? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 25 How satisfied are you with how your council responds to community needs? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 26 How satisfied are you with the quality of projects implemented by your council? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 27 How satisfied are you with how your council manages funds on projects or services? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Don’t know 28 How satisfied are you with your accessibility to information from your council? Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 90 Don’t know How likely is it that you could get the following information from government or other public institutions? 29 ...if you went to the local school to find out what the school’s budget is and how the funds have been used? Not at all likely Not very likely Somewhat likely Very likely Don’t know 30 ...if you went to the District/City Council to find out who owns a piece of land in your community? Not at all likely Not very likely Somewhat likely Very likely Don’t know 31 ...if you went to the District/City Council to find out about the District Development Plan and budgets? Not at all likely Not very likely Somewhat likely Very likely Don’t know In the last 12 months, have you participated in the following? 32 A meeting in your community to discuss the village action plan? Yes No Not sure/cannot recall 33 A meeting in your district to discuss the district development plan? Yes No Not sure/cannot recall 34 A meeting in your community to discuss a social contract between the primary school and the school community? Yes No Not sure/cannot recall 35 A meeting in your community to discuss how to collect and/or spend market fees collected from vendors? Yes No Not sure/cannot recall 36 A training program in your district to learn about how to implement procedures for managing community resources (e.g., finances, staffing)? Yes No Not sure/cannot recall 37 A training program in your district to learn about how to implement procedures for keeping better or more complete records? Yes No Not sure/cannot recall How much do you trust the following, or haven’t you heard enough about them to say? USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 91 38 Your member of parliament? Not a lot Just a little Somewhat A lot Don’t know/haven’t heard 39 Your district council? Not a lot Just a little Somewhat A lot Don’t know/haven’t heard 40 Your traditional leader? Not a lot Just a little Somewhat A lot Don’t know/haven’t heard 41 Civil society organizations? Not a lot Just a little Somewhat A lot Don’t know/haven’t heard 42 Your Village Development Committee (VDC)? Not a lot Just a little Somewhat A lot Don’t know/haven’t heard 43 Finally, if you could make a suggestion for improving local government accountability and performance in this area, what would it be? [Text fill] MODULE II: DEMOGRAPHIC BACKGROUND Now I would like to ask some questions about you. 44 First Name [Text fill] 45 Last Name/Surname [Text fill] 46 What is your ethnic community, cultural group, or tribe? Tumbuka Nkhonde Lambya Chewa Yao Ngoni USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 92 Lomwe Mang’anja Sena Sukwa Senga Tonga “Malawian only” or “Do not think of myself in those terms” Refused to answer Don’t know 47 What is your religion, if any? Roman Catholic CCAP Muslim Pentecostal or Evangelical Christian Other Christian Other, specify: [Text fill] 48 How long have you been living in your current village/neighborhood? Years: [Text fill, numbers only] 49 What is your present marital status? Married Single Widowed Divorced or Separated 50 How many people live in your household, including yourself? Please include only the people who usually live and eat in your home and not temporary visitors. Also, include the household head even if he or she has not lived in the household for the past 6 months, as long as he/she is still living and supporting this household. Please do not include children who have already moved out or gotten married. [Text fill, numbers only] 51 What is the highest level of education you have completed? None Some primary school Completed primary school Some secondary school Completed secondary school Vocational training or diploma certificate Completed university Post-graduate education Adult literacy program Other, Specify: [Text fill] 52 How long does it take to reach the public primary school nearest to your home? Minutes: [Text fill, numbers only] USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 93 53 How long does it take to reach the market nearest to your home? Minutes: [Text fill, numbers only] 54 How long does it take to reach the clinic nearest to your home? Minutes: [Text fill, numbers only] 55 What is your main occupation? Never had a job Student Housewife/homemaker Agriculture/farming/fishery/forest ry Trader/hawker/vendor Retail/shop Unskilled manual labor (e.g., cleaner, laborer, domestic help, Artisan or skilled manual worker (e.g., electrician, mechanic, machinist) Clerical or secretary Supervisor/foreman/senior manager Security services (police, military, private security) Mid-level professional (e.g. teacher, civil servant) Upper-level professional (e.g., banker, lawyer, doctor, engineer, accountant, senior level government) Other, specify: [Text fill] Are you a member of any of the following committees, councils, or organizations? 56 Area Development Committee (ADC) Yes No 57 District Executive Committees (DEC) Yes No 58 Village Development Committee (VDC) Yes No 59 District Councillor or Council Service Committees Yes No 60 Village headman or village head’s counsellor Yes No 61 Some other local leadership organization? If yes, specify: [Text, fill] USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 94 62 In general, how do you rate your living conditions compared to those of other Malawians? Much worse Worse Same Better Much better Don’t know 63 Looking back, how do you rate economic conditions in this country compared to 12 months ago? Much worse Worse Same Better Much better Don’t know Over the past year, how often, if ever, have you or anyone in your family: 64 ...gone without enough food to eat? Never Just once or twice Several times Many times Always Don’t know 65 ...gone without enough clean water for home use? Never Just once or twice Several times Many times Always Don’t know 66 ...gone without medicines or medical treatment? Never Just once or twice Several times Many times Always Don’t know 67 ...gone without enough fuel to cook your food? Never Just once or twice Several times Many times Always Don’t know 68 ...gone without a cash income? Never Just once or twice Several times Many times Always Don’t know USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 95 APPENDIX 5: TENTATIVE SCHEDULE OF EVALUATION MILESTONES, MEETINGS, VISITS, KIIS, AND FIELDWORK13 TASK TIMELINE PROPOSED DEADLINE Draft and submit desk review and workplan, including desk review of pertinent documents and a detailed performance evaluation methodology September 21, 2020 USAID provides feedback on the desk review and workplan Within one week of receiving the document September 28, 2020 Final desk review and workplan submitted Within three days of receiving comments October 1, 2020 Field preparations, including obtaining IRB and other necessary permissions; programming, translating, testing, and piloting the survey; and preparing introduction letters for interviews Three weeks plus one day after submission of desk review and workplan October 2, 2020 – October 23, 2020 Qualitative data collection, including in-brief meeting with USAID/Malawi 25 interview days and 5 travel days October 25, 2020- November 23, 2020 Quantitative data collection 40 interview days and 5 travel days October 24, 2020- December 7, 2020 Out-brief with USAID on preliminary findings December 9, 2020 Stakeholder workshop to present preliminary findings for validation and inputs December 11, 2020 Draft Performance Evaluation Report with preliminary findings January 22, 2021 Receive feedback on the Performance Evaluation Report Within two weeks of receiving the document February 5, 2021 Final Performance Evaluation Report submitted Within two weeks of receiving comments February 19, 2021 USAID Final Report approval Within two weeks of receiving the document March 5, 2021 Draft Post-Evaluation Action Plan submitted Within two weeks of final report approval March 19, 2021 13 Please note that this work plan timeline assumes that local research review can be completed in approximately five weeks (two weeks during desk report review + three weeks following desk report approval). If this review is delayed, we may need to shift the schedule of fieldwork accordingly to ensure receipt of necessary approvals before the commencement of data collection. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 96 ANNEX C: QUALITATIVE EVENT SUMMARY TABLES Table A.1: Number of Interviews by Location LOCATION NUMBER OF INDIVIDUAL INTERVIEWS NUMBER OF GROUP INTERVIEWS TOTAL NUMBER OF INTERVIEWS NUMBER OF INDIVIDUALS INCLUDED IN INTERVIEWS Balaka 3 3 6 21 Chikwawa 6 1 7 11 Dowa 1 2 3 8 Karonga 2 2 4 7 Lilongwe 3 0 3 3 Mangochi 5 4 9 23 M’mbelwa 6 2 8 10 Mulanje 2 2 4 11 Nkhotakota 3 4 7 12 Nkhata Bay 2 1 3 4 National Level DAI 2 1 3 15 USAID 3 0 3 3 FCDO 0 1 1 2 Other National Level 13 2 15 17 Outbrief 0 1 1 25 Total 51 26 77 172 Note: 21 interviews from pilot districts, and 33 interviews from expansion districts. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 97 Table A.2: Number of Interviews by Type of Respondent TYPE OF RESPONDENT NUMBER OF INDIVIDUAL INTERVIEWS NUMBER OF GROUP INTERVIEWS TOTAL NUMBER OF INTERVIEWS NUMBER OF INDIVIDUALS INCLUDED IN INTERVIEWS District Level Councilors 7 4 11 20 District Commissioners/ Other District Staff 8 9 17 33 ADC Members 4 5 9 36 DEN Members 6 1 7 8 CSO Representatives 8 2 10 13 Sub-Total 33 21 54 110 National Level National Staff 8 2 10 12 Other 5 0 5 5 Sub-total 13 2 15 17 DAI 0 1 1 15 USAID 3 0 3 3 FCDO 0 1 1 2 Outbrief 0 1 1 25 Total 51 26 77 172 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 98 Table A.3: Significant LGAP Activities to Foster Accountability TYPES OF LGAP ACTIVITIES FOSTERING ACCOUNTABILITY AS MENTIONED IN KEY INFORMANT INTERVIEWS NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL Training of District Development Partners (ADCs, Councils, Service/Market/School Committees) 19 26 0 45 ● Governance Training 4 3 - 7 ● Roles and Responsibilities 4 4 - 8 ● Financial Management 3 2 - 5 ● Human Resources Management 1 3 - 4 ● Fostering Accountability 3 2 - 5 ● ADC Needs Assessment/Empowerment 2 10 - 12 ● Training of School Committees 2 2 - 4 Financial Interventions: TA and Training of Finance Staff (including in IFMIS)/Development of Citizen Budgets/Public Expenditures Surveys/ Manuals/ Provision of Computers/Market Fee Initiatives 7 8 3 18 Education Initiatives (Developing Social Contracts/Accrediting and Training Inspectors/Revitalizing District Education Networks/Categorizing Teachers by Location) 6 8 2 16 TA to Develop the District Development Plans, Socio-Economic Plans, and By-Laws 5 3 3 11 TA on Human Resources Management (e.g. Manuals/Updating Personnel Files/HRMIS Revamping) 1 2 1 4 Initiatives to Foster Improved Communication Between Partners (Meetings, Website on Public Sector Reforms) 2 4 2 8 Mobile Phone/Radio/Billboard Outreach to Foster Citizen Awareness 2 1 2 5 Total Mentions 42 52 13 107 Source: Key Informant Interviews with 77 Respondents in 10 Districts and the National Level. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 99 Table A.4: Effects of LGAP Activities on Local Government Accountability TYPES OF ACCOUNTABILITY FOSTERED BY LGAP ACTIVITIES AS MENTIONED IN KEY INFORMANT INTERVIEWS NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL Accountability through Coordination of Development Activities 14 14 1 29 ● ADCs/Market Committees/Councils Empowered to Question Progress and Seek Development Improvements and Assistance 7 8 1 15 ● Council and Secretariat More Responsive to Community Input; Improved Communication Between Partners 1 1 - 2 ● Coordination (e.g. Service Committees and CSOs) Leads to Reduced Duplication and Reduced Overlap of Development Activities 6 3 - 9 ● Participatory Management of Community Development Fund - 2 - 2 Financial Accountability Activities 20 8 3 31 ● Improved and Broader Participation in Budgeting/Citizen Budgets/Councilors and ADCs Empowered to Ask Questions about Budget 3 3 1 7 ● More Timely Financial Reports 5 4 2 11 ● Secure Collection of Market Fees through Electronic Banking 6 1 - 7 ● Anti Corruption Activities (Improved Accounting Systems, Suggestion Boxes, Coordination with Law Enforcement) 6 - - 6 Education Activities: Social Contracts Foster Education Accountability; Reduced Delay in Salaries and Improved Allocation of Teachers; Parents Now Pay for Some School Amenities (e.g. toilets); School Improvement Funds are Tracked; District Education Networks Empowered and More Involved; Inspection System Revitalized 15 9 1 25 Human Resources Management Devolved/Employees More Accountable/Reduced Ghost Workers/Reduced Vacancies and Excess Staff - - 2 2 Total 49 31 7 87 Source: Key Informant Interviews with 70 Respondents in 10 Districts and at the National Level. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 100 Table A.5: Citizen Reports of Improvements Needed in Local Government TYPES OF GOVERNMENT IMPROVEMENTS NEEDED NUMBER OF MENTIONS % OF TOTAL INTERVIEWS 2 Pilot Districts 4 Other Districts Total Government Functioning Improvements Needed Improved ADC/CSO/Community Engagement in Development 12 60 72 22.1 Improved Coordination and Communication Between Development Partners 16 34 50 15.3 Less Corruption/More Transparency and Accountability 12 34 46 14.1 More Monitoring/Reporting on Development Progress 11 27 38 11.7 Less Politicization of Development Funds 8 24 32 9.8 Improved Quality of Staff and Officials/More Monitoring of Staff and Officials/Frequent Elections 7 8 15 4.6 Sub-Total 66 187 253 77.6 LGAP Improvements Needed More Frequent Training at all Levels 3 25 28 8.6 Critical Development Needs Not Being Met Health Services 9 21 30 9.2 Infrastructure (Roads, Bridges, Electricity, etc.) 11 12 23 7.1 Markets (Infrastructure, Revenues) 11 9 20 6.1 Water Access (e.g. Boreholes) 5 13 18 5.5 Education 2 11 13 4.0 Food Security 6 2 8 2.5 Other (Unemployment, Community Policing, COVID￾19, Business Development) 4 1 5 1.5 Sub-Total 48 69 117 35.9 Total 117 281 398 -- Source: Open-ended comments to Evaluation Survey of 326 citizens in 6 districts (Dowa, Lilongwe, M’mbelwa, Mangochi, Nkhata Bay, and Nkhotakota) when asked “What are the most important improvements needed to local government in your district?” Total responses do not add to the total number of interviews, because some respondents mentioned more than one improvement needed. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 101 Table A.6: Service Improvements Due to LGAP TYPE OF SERVICE IMPROVEMENT MENTIONED NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL Education Service Improvements Improved Teacher Performance 3 1 - 4 Improved Pupil Performance 1 - - 1 Reduced Absenteeism of Teachers and Pupils 2 - - 2 Reduced Primary School Dropout 1 - - 1 Improved Pupil/Teacher Ratio - - 1 1 Sub-Total 7 1 1 9 Health Service Improvements Integration of Services through District Health Service Committee Activities 2 1 - 3 Improved Public Health Education 1 - - 1 Rapid District-level COVID-19 Response 4 6 1 11 Sub-Total 7 7 2 15 Infrastructure Improvements Boreholes 1 1 - 2 Market Sheds 3 - - 3 Bridges 1 - - 1 Concrete Paths 1 - - 1 Sub-Total 6 1 - 7 Agricultural Improvements - 1 - 1 Total Improvements Mentioned 20 9 3 32 Source: Based on Key Informant Interviews with 77 individuals. Most individuals, when asked about service improvements due to LGAP, stated that it is too early to see service improvements or that there is a lack of data on service improvements. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 102 Table A.7: LGAP Unintended Consequences UNINTENDED CONSEQUENCES/EXTERNALITIES MENTIONED IN KEY INFORMANT INTERVIEWS NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL Positive Consequences LGAP upgraded IT systems, and improved IT reporting and skills, which spillover to many government functions (such as more regular salaries) - 1 4 5 LGAP activities increased incentives for staff and committees to work harder (e.g. producing timely reports to committees) - 2 1 3 LGAP activities fostered coordination around development funds between council, CSOs, MPs, and school committees in some districts 2 2 - 4 LGAP stimulated other funders of democracy and governance in Malawi 1 - 2 3 LGAP Improved the Functioning of Some Local CSOs 1 2 3 Better Functioning Organizations Have Positive Spillover Effects for Unanticipated Events (e.g. COVID) 1 1 - 2 Market and school upgrades improved public health (e.g. toilets) 2 - - 2 Sub-Total 7 8 7 22 Negative Consequences LGAP activities let to increased conflicts between development partners (e.g. councils, ADCs, market committees, traditional authorities) and/or citizens in some districts 8 3 3 14 LGAP activities led to unintended conflicts/duplication with other national entities concerning roles and responsibilities - 2 5 7 Sub-total 8 5 8 21 Total 15 13 15 43 Source: Key Informant Interviews with 77 Respondents in 10 Districts and at the National Level. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 103 Table A.8: LGAP Implementation Challenges and Problems LGAP IMPLEMENTATION PROBLEMS AND CHALLENGES MENTIONED IN KEY INFORMANT INTERVIEWS NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL LGAP Challenges Staff turnover of district and LGAP staff 4 3 3 10 Insufficient time and resources to scale up to 28 districts 1 7 1 9 COVID delayed schedule 1 4 1 6 Village level work (e.g. VDCs) impractical due to scale and distance 1 5 - 8 Demand for help outside LGAP scope of work - - 1 1 Conflicts over allowances - - 1 1 Councils suffer from lack of funding to sustain gains - - 1 1 Sub-Total 7 19 1 36 LGAP Problems Promised interventions incomplete (e.g. manuals, infrastructure, training, market fee collection) 6 7 5 18 Lack of engagement or regular communication with key individuals/ organizations at the district or national level 1 9 5 15 Allowance for transportation and refreshments too small 3 4 1 8 Trainings too short or infrequent 2 3 1 4 Lack of sharing of work plans or exit strategy - 3 1 4 Insufficient follow-up by LGAP staff - 3 - 3 Sub-Total 12 29 13 54 Total 19 48 23 90 Source: Key Informant Interviews with 77 Respondents in 10 Districts and at the National Level. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 104 Table A.9: Recommendations for Future Governance Programming RECOMMENDATIONS BY RESULT AREA NUMBER OF MENTIONS 4 PILOT DISTRICTS 6 OTHER DISTRICTS NATIONAL LEVEL TOTAL Local Government Performance and Transparency Continue/Expand Support for Financial Management including Support for IFMIS Implementation, Participatory Budgeting, and Expenditure Tracking Surveys 3 1 - 4 Develop/Expand Revenue Enhancement Strategy 1 2 - 3 Develop a Functional/Harmonized M and E Process with Indicators of Development Progress Across District/Monitor School Contracts - 3 1 4 Conduct and Update Socio-Economic Profiles and District Development Plans in All Districts 3 - - 3 Add a Staff Mentorship Component - 1 1 2 Continue/Increase IT Support - 1 - 1 Help Depoliticize Development Decisions - - 1 1 Sub-Total 7 8 3 18 Demand for Accountable Government Strengthened Continue Training and Retraining/Consider VDC Training/Increase Length or Focus of Training/Support Learning Visits Between Districts/Include MPs 12 4 - 16 Continue and Expand Meetings of Development Partners/Expand Number of Development Partners 1 5 1 7 Focus on Infrastructure Development 2 4 - 6 Improve Meeting Space, Refreshments, and Transportation Allowances Based on Distance 2 - 2 4 Sub-Total 17 13 3 33 Decentralization Policy Environment Improved Continue By-Law Development in All Districts 3 - - 3 Continue/Improve Human Resources Initiatives - 1 1 2 Sub-Total 3 1 1 5 LGAP Operations Improve/Expand Communication Between LGAP and National/District Partners 2 4 5 11 Revisit Position and Number of LGAP Offices/Consider Expansion 1 2 - 3 Increase Engagement with Local CSOs 2 - - 2 Sub-Total 5 6 5 16 Total 32 28 12 72 Source: Key Informant Interviews with 77 Respondents in 10 Districts and at the National Level . USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 105 ANNEX D: LGAP MIDTERM EVALUATION SURVEY SAMPLING PROTOCOL The LGAP Midterm Evaluation Survey aims to interview beneficiaries of the Local Government Accountability Performance (LGAP) program. LGAP is meant to benefit all Malawians through improving local governance and accountability, but seeks in particular to improve capacity and accountability of local governments through: supportive supervision for core local government operations, public financial management and accountability, capacity building for councilors and District Executive Committee staff, and strengthening citizen engagement and oversight. In each of the 10 districts of study, survey interviews should be conducted in four areas (which we call primary sampling units) with three different respondent types in each area. PRIMARY SAMPLING UNITS Each district has four primary sampling units (PSUs) from which respondents will be sampled: the district seat (boma) and three TAs that were selected randomly during the research design. The sampled PSUs, by district, are: Balaka: Balaka Town, TA Msalamala, TA Nkaya, TA Kalembo Chikwawa: Chikwawa Boma, TA Katunga, TA Lundu, TA Makhwira Dowa: Dowa Boma, TA Chakhaza, TA Chiwere, TA Msakambewa Karonga: Karonga Town, TA Kilipula, TA Kyungu, TA Wasambo Lilongwe: Area 7 (City), TA Chimutu, TA Kabudula, TA Kalolo Mangochi: Mangochi Town, TA Chilipa, TA Jalasi, TA Katuli M’mbelwa: Mzimba Boma, TA Jaravikuba Munthali, TA Kampingo Sibande, TA Khosolo Gwaza Jere Mulanje: Mulanje Boma, TA Chikumbu, TA Nkanda, TA Nthiramanja Nkhata Bay: Nkhata Bay Boma, TA Kabunduli, TA Malanda, TA Malenga Mzoma Nkhotakota: Nkhotakota Boma, TA Kanyenda, TA Malenga Chanzi, TA Mwadzama The survey team will collect 64 surveys in each district, with 16 surveys from each PSU. RESPONDENT TYPES There are three respondent types: official, civil society, citizen. In each PSU, there will be 8 surveys with officials and civil society leaders and 8 citizen surveys. By officials, we mean District Commissioners; Ward Councillors; people who serve on District Council Service Committees; members of District Executive Committees (DEC) including District Health Officers (DHO) or District Agricultural Development Officers (DADO); members of Area Development Committees (ADC); village headmen (VH) or group village headmen (GVH) or traditional authorities (TAs); and officials or members in public service sectors like education (e.g., District Education Network (DEN) officials, Primary Education Advisor (PEA) officials, members of School Management Committees), health (e.g., members of Hospital Advisory Committees, Health Centre Advisory Committees, Village Health Committees, or Village AIDS Committees), development (e.g., members of Market Committees or Village Development Committees), agriculture (e.g., members of the District Agricultural Extension Coordination Committee), and safety (e.g., members of the Community Policing Committee). USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 106 By civil society leaders, we mean religious leaders; representatives of non-governmental organizations (NGOs) like NICE and YONECO; farmer-based organizations; parent-teacher associations (PTA); and representatives of civil society organizations (CSOs), faith-based organizations (FBOs), or community-based organizations (CBOs). By citizen, we mean any non-institutionalized adult citizen of Malawi who is not an “official” or a “civil society leader,” as enumerated above. SAMPLING OFFICIALS AND CIVIL SOCIETY Officials and civil society respondents will be sampled using a stratified convenience sampling approach. In each PSU, seek interviews with both officials and civil society leaders to meet the completion quota of 8 surveys in each PSU. Among these 8 respondents, there should be a mix of “officials” and “civil society.” In each PSU, there should be at least 2 but no more than 4 civil society interviews. In areas where the survey team is less familiar and will need guidance in identifying potential respondents, enumerators are encouraged to conduct survey interviews early in the day with key informants like District Executive Committee members, Area Development Committee members, and TAs. SAMPLING CITIZENS All 8 citizen surveys in each PSU should be sampled using a random walk procedure in the same area. More specifically, when in an urban or peri-urban PSU, all 8 citizen surveys should come from the same neighborhood; when in a rural PSU, all 8 citizen surveys should come from the same village. The supervisor will select the neighborhood or village from the Area/Town/Boma/TA from which citizens will be sampled. The supervisor should diagram the boundaries of the neighborhood/village and identify starting points and walk directions for each enumerator similar to the diagram below. Different neighborhoods and villages will have different shapes, but the general procedure should be to space enumerators so that there is broad coverage of the neighborhood or village and that one enumerator’s path would not cross another enumerator’s path. Figure 1. Illustration of Spaced Random Walk Starting Points in a PSU USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 107 The first interview for each enumerator will be the household closest to their starting point, walking in the direction specified by the supervisor. Interview either the female or male head in this household, according to gender alternation from the last interview completed; in other words, if your most recent interview was with a male respondent, you should interview the female head of household; if your most recently completed interview was with a female respondent, you should interview the male head of household. Once complete, the enumerator will walk in the direction specified and stop at the fifth household to conduct the next interview. Interview either the female or male head in this household, according to gender alternation from the last interview completed. If a selected household is empty, does not have the corresponding gender respondent available, or the selected respondent refuses to participate in the survey, continue walking in the specified direction and make another attempt at conducting an interview at the fifth household from the previous attempt. Maintain gender alternation according to the most recently completed interview. USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 108 ANNEX E: SURVEY SUMMARY TABLES Table A.10: Evaluation Survey Descriptive Statistics and Comparison to Other Surveys CLOUDBURST 2020 CENSUS 2018 DHS 2016 Demographics Rural 74% 84% 81% Female 42% 51% 52% Marital Status Single/Never married 2% 38% 30% Married/Living together 83% 38% 61% Divorced/Separated 6% 5% 7% Widowed 8% 4% 2% Education None 3% 9% Primary 48% 60% Secondary 45% 27% More than secondary 4% 4% Region North 30% 13% 12% Central 30% 43% 44% Southern 40% 44% 44% Ethnic Identity Chewa 24% 34% 36% Yao 17% 13% 13% Tumbuka 13% 9% 9% Tonga 10% 1% 2% Lomwe 9% 19% 19% Ngoni 9% 10% 12% Mang'anja 7% 3% 3% Sena 4% 4% 3% Nkhonde 4% 1% 1% Other 2% 6% 3% Religion CCAP 23% 14% 18% Catholic 15% 17% 18% Muslim 18% 14% 12% Other Christian 44% 36% 52% USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 109 Table A.11: Findings on Satisfaction with how your councillor consults citizens DISTRICT CPS 2017 PE 2020 CHANGE Balaka Satisfied 27% 30% ↑ Dissatisfied 54% 59% ↑ Lilongwe Satisfied 28% 61% ↑↑ Dissatisfied 39% 24% ↓ M'belwa Satisfied 37% 34% ↓ Dissatisfied 36% 31% ↓ Mulanje Satisfied 25% 49% ↑↑ Dissatisfied 47% 33% ↓↓ Desirable outcome ↑ ↓ Low magnitude of change Undesirable outcome ↑ ↑ ↓ ↓ High magnitude of change Table A.12: Findings on satisfaction with how council responds to community needs DISTRICT CPS 2017 PE 2020 CHANGE Balaka Satisfied 24% 27% ↑ Dissatisfied 57% 61% ↑ Lilongwe Satisfied 26% 28% ↑ Dissatisfied 39% 52% ↑ M'belwa Satisfied 36% 9% ↓ Dissatisfied 36% 66% ↑↑ Mulanje Satisfied 24% 23% ~ Dissatisfied 48% 56% ↑ Desirable outcome ↑ ↓ Low magnitude of change Undesirable outcome ↑ ↑ ↓ ↓ High magnitude of change USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 110 Table A.13: Findings on satisfaction with projects implemented by your council DISTRICT CPS 2017 PE 2020 CHANGE Balaka Satisfied 29% 16% ↓ Dissatisfied 52% 50% ↓ Lilongwe Satisfied 34% 20% ↓ Dissatisfied 37% 44% ↑ M'belwa Satisfied 38% 8% ↓↓ Dissatisfied 37% 63% ↑↑ Mulanje Satisfied 33% 12% ↓ Dissatisfied 42% 53% ↑ Desirable outcome ↑ ↓ Low magnitude of change Undesirable outcome ↑ ↑ ↓ ↓ High magnitude of change Table A.14: Findings on participation in community meetings and trainings in last year LEADER SUBSAMPLE CITIZEN SUBSAMPLE ALL SURVEYS Village Action Plan 73% 54% 66% District Development Plan 13% 8% 11% School Social Contract 89% 75% 84% Market Fees 15% 3% 11% Training: Resources 30% 10% 22% Training: Record keeping 26% 4% 18% USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 111 Table A.15: Findings on satisfaction with services in Balaka, Lilongwe, M'belwa, Mulanje SERVICE CHARACTERISTIC CPS 2017 PE 2020 CHANGE Health worker behavior Satisfied 47% 46% ~ Dissatisfied 37% 46% ↑ Medicines available Satisfied 29% 30% ~ Dissatisfied 48% 63% ↑ Health committee Satisfied 39% 42% ↑ Dissatisfied 31% 31% ~ Teacher attendance Satisfied 53% 51% ~ Dissatisfied 29% 36% ↑ Teaching materials Satisfied 35% 15% ↓↓ Dissatisfied 39% 75% ↑↑ PTA Satisfied 52% 53% ~ Dissatisfied 21% 23% ~ Agricultural extension Satisfied 48% 50% ~ Dissatisfied 26% 35% ↑ Desirable outcome ↑ ↓ Low magnitude of change Undesirable outcome ↑ ↑ ↓ ↓ High magnitude of change USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 112 ANNEX F: LGAP MID-TERM EVALUATION OUT-BRIEF PRESENTATION USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 113 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 114 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 115 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 116 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 117 USAID.GOVMALAWI LGAP MIDTERM PERFORMANCE EVALUATION | 118