December 2019 This publication was produced at the request of the United States Agency for International Development Performance Evaluation Report: Joint USAID/Uganda and DFID Governance, Accountability, Participation and Performance (GAPP) Program PERFORMANCE EVALUATION OF THE JOINT USAID/UGANDA AND DFID GOVERNANCE, ACCOUNTABILITY, PARTICIPATION AND PERFORMANCE (GAPP) PROGRAM EVALUATION REPORT 16th December 2019 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development, the United States Government, the United Kingdom’s Department for International Development, or the United Kingdom Government. ACKNOWLEDGEMENTS This work was commissioned by USAID/Uganda and DFID and was prepared independently by Meg Kinghorn, Team Lead; John Kiyaga-Nsubuga, Local Governance Specialist; Laura Nyirinkindi, Civil Society Specialist; Wolfgang Stuppert, Evaluation Support Specialist; and Gulden Bayaz, VfM Specialist under contract with the QED Group, LLC. Uganda Learning Program (ULA). Additional support was given by JP-Jennifer Pierre, Evaluation Manager; Rachel Santos, Evaluation Support Specialist; Innocent Kiiza, Survey Team Lead; and Aidat Namakula and Genevieve Onyango, Research Assistants. The authors appreciate the technical advice and support from the USAID Mission and ULA staff in designing and conducting this performance evaluation and preparing this evaluation report. Special thanks are extended to the USAID technical team for its input and assistance throughout the final evaluation process, particularly Harriet Muwanga, GAPP Contracting Officer’s Representative and Morris Nsamba, Democracy & Governance M&E Specialist. The evaluation team also appreciates the input and cooperation of the management and staff of the GAPP Activity, in addition to their support during data collection. The GAPP evaluation team appreciates the input from all respondents, including those at national levels, local and lower local governments, and national and district-level civil society organizations. Table of Contents ACRONYMS......................................................................................................................................................vi EXECUTIVE SUMMARY...............................................................................................................................viii 1 BACKGROUND...................................................................................................................................... 1 2 EVALUATION PURPOSE & METHODOLOGY.............................................................................. 4 2.1 DATA COLLECTION AND ANALYSIS.................................................................................... 4 2.2 LIMITATIONS .................................................................................................................................. 5 3 EVALUATION FINDINGS..................................................................................................................... 6 3.1 Q1. GAPP CONTRIBUTIONS TO ACCOUNTABILITY PROCESSES, INSTITUTIONAL STRENGTHENING, CITIZEN PARTICIPATION AND VOICE AT LOCAL AND NATIONAL LEVELS......................................................................................................................... 6 3.1.1 GAPP Contributions to Improved National Local Government Accountability Environment .............................................................................................................................................. 6 3.1.2 GAPP Contributions to Strengthened Local Government Fiscal Management, Accountability, and Decision-Making ................................................................................................... 9 3.1.3 GAPP contributions to district citizen and community participation and voice in local governance ..................................................................................................................................... 14 3.2 Q2. RELEVANCE OF GAPP MANAGEMENT AND IMPLEMENTATION APPROACHES............................................................................................................................................ 16 3.2.1 Participant Recommendations............................................................................................ 17 3.2.2 Appropriateness of GAPP Approach ................................................................................ 18 3.3 Q3. VALUE FOR MONEY ASSESSMENT OF EMBEDDED TECHNICAL SPECIALISTS MODEL, TRIPARTITE OUTREACH MODEL, AND MINISTRY OF HEALTH ACCOUNTABILITY SUPPORT ............................................................................................................. 20 3.3.1 Value for Money of the three models............................................................................... 20 3.3.2 Tripartite Outreach Model (TOM): .................................................................................. 21 3.3.3 Technical Assistance to MOH: ........................................................................................... 22 3.3.4 Embedded Technical Specialists (ETSs): ........................................................................... 23 4 CONCLUSIONS .................................................................................................................................... 25 4.1 LESSONS LEARNED..................................................................................................................... 25 4.2 RECOMMENDATIONS............................................................................................................... 26 List of Tables Table 1: District Status of Context Conditions........................................................................................................11 Table 2: Distribution of GAPP Grants across Focus Areas ...................................................................................15 Table 3: Application of GAPP Support........................................................................................................................17 Table 4: GAPP Outreach as of September 2019 ......................................................................................................23 List of Figures Figure 1: GAPP Theory of Change .................................................................................................................................1 Figure 2: GAPP Expenditures - $33.6 million...............................................................................................................2 Figure 3: Districts Receiving GAPP Support by Donor.............................................................................................2 Figure 4: Process Map of the Audit Cycle ....................................................................................................................7 Figure 5: LG Context Conditions.................................................................................................................................10 Figure 6: Most Significant Type of GAPP Support ....................................................................................................16 Figure 7: Most Significant Aspects of GAPP...............................................................................................................16 Figure 8: Most Useful Aspects of GAPP TA...............................................................................................................17 List of Boxes Box 1: GAPP Evaluation Mix of Methods ............................................................................................................. 4 Box 2: National Level Outcomes............................................................................................................................ 6 Box 3: CSOs Invited to National Decision-making Bodies ............................................................................. 8 Box 4: Context Conditions for High LG Performance ...................................................................................10 Box 5: Critical Coaching from a GAPP ETS.......................................................................................................11 Box 6: District Level Outcomes...........................................................................................................................11 Box 7: Successful first time Councilor.................................................................................................................12 Box 8: Accountability at Work in the Health Sector.......................................................................................13 Box 9: Citizen Level Outcomes............................................................................................................................14 Box 10: Citizens Write Petitions to Improve Service Delivery ....................................................................15 Box 11: Reason for GAPP Effectiveness .............................................................................................................18 Box 12: Four DFID VfM..........................................................................................................................................20 Box 13: Value for Money Savings at the Ministry of Health...........................................................................22 Box 14: Example of Participator District Planning............................................................................................27 List of Annexes ANNEX 1: EVALUATION SCHEDULE......................................................................................................................30 ANNEX 2: EVALUATION SCOPE OF WORK.......................................................................................................32 ANNEX 3: LIST OF DOCUMENTS REVIEWED.....................................................................................................40 ANNEX 4: LIST OF STAKEHOLDERS CONSULTED...........................................................................................47 ANNEX 5: KEY INFORMANT INTERVIEW GUIDE..............................................................................................51 ANNEX 6: FOCUS GROUP DISCUSSION PROTOCOL.....................................................................................53 ANNEX 7: STAKEHOLDER SURVEY TOOL...........................................................................................................54 ANNEX 8: ANALYSIS OF STAKEHOLDER SURVEY............................................................................................61 ANNEX 9: QUALITATIVE COMPARATIVE ANALYSIS DESCRIPTION .........................................................68 ANNEX 10: VALUE FOR MONEY DETAILED ANALYSIS FOR THE THREE MODELS .............................70 ANNEX 11: PROCESS MAPS........................................................................................................................................84 ANNEX 12: OUTCOME SUMMARIES.......................................................................................................................87 GAPP Performance Evaluation Report vi December 2019 ACRONYMS CAO Chief Administrative Officer CBO Community Based Organization CDO Community Development Officer CEPA Centre for Policy Analysis CFO Chief Financial Officer CIDI Community Integrated Development Initiatives CLA Collaborating, Learning, and Adapting CPS Corporate Planning and Strategy CSBAG Civil Society Budget Advocacy Group CSO Civil Society Organization DAI Development Alternatives Inc. DDP District Development Plan DEC District Executive Council DFID Department for International Development DHO District Health Officer DLG District Local Government DOP District Operational Plan DRG Democracy, Human Rights and Governance FGD Focus Group Discussion ETS Embedded Technical Specialist FY Fiscal Year GAPP Governance, Accountability, Participation and Performance Activity GOU Government of Uganda IAD Internal Audit Department IGG Inspector General of Government IP Implementing Partner IPS Institute of Parliamentary Studies KII Key Informant Interview LG Local Government LGAC Local Government Accounts Committee LGDF Local Government Development Framework LGFC Local Government Finance Commission LGPA Local Government Performance Assessment LLG Lower Local Government LRM Local Revenue Mobilization MC Municipal Council M&E Monitoring and Evaluation MEC Municipal Executive Council MEL Monitoring, Evaluation and Learning MEP Monitoring & Evaluation Plan MLG Municipal Local Government MoFPED Ministry of Finance, Planning and Economic Development MOH Ministry of Health MoLG Ministry of Local Government MP Member of Parliament MTE Mid-term Evaluation NAI National Accountability Institution NGO Non-governmental organization NPA National Planning Authority NSA Non-state actor GAPP Performance Evaluation Report vii December 2019 OAG Office of the Auditor General PAC Public Accounts Committee PAC LG Local Government Public Accounts Committee of Parliament PDU Procurement and Disposal Unit PEMCOM Public Expenditure Management Committee PFM Public Finance Management PLOCA Partner Local Organization Capacity Assessment PPDA Public Procurement and Disposal of Public Assets Authority PSA Private Sector Association PSO Private Sector Organization RTI Research Triangle Institute SOW Scope of Work TA Technical Assistance ToC Theory of Change TOM Tripartite Outreach Meeting TOR Terms of Reference ToT Training of Trainers UGX Uganda Shilling VfM Value for Money USAID United States Agency for International Development GAPP Performance Evaluation Report viii December 2019 EXECUTIVE SUMMARY In November 2012, the United States Agency for International Development (USAID) awarded the five￾year Governance, Accountability, Participation and Performance (GAPP) program to Results Triangle International (RTI), in collaboration with Development Alternatives International (DAI) and Palladium. The program was originally valued at $17.7 million and funded solely by USAID. In January 2015, the United Kingdom’s Department for International Development (DFID) joined as the program’s co-funder. As a result, the Program duration was extended to November 28, 2019, with a new program value of US$35.4 million, reflecting a one-to-one matched contribution between DFID and USAID. The Program was initially operational in 25 districts; however, with additional DFID funding in 2015, the program coverage expanded to 35 districts.1 By the end of 2018, GAPP covered 40 districts due to the splitting of some districts. The goal of GAPP was to increase civic participation, accountability and local governance for effective public service delivery. The theory of change states: If Government of Uganda (GOU) and nongovernment￾led efforts are supported to improve democratic governance and accountability, and citizens’ efforts are mobilized to engage and partner with Local Governments (LGs), then service delivery will be more equitable and efficient. The program focused its work to achieve change through three objectives to: 1. Improve the legal, policy, regulatory, and institutional environment to meet demands for more democratic governance; 2. Enhance fiscal management and accountability processes in the delivery of services in targeted districts; 3. Build the capacity of citizens and communities to participate in local governance voice and accountability. The program operated within the context of significant social and political change in Uganda. The assumptions underlying the program’s theory of change all proved to be challenges: 1) political will was shifting from decentralized to more centralized government processes; 2) many district governments were underfunded and understaffed, limiting their ability to assimilate technical support; and 3) democratic space for CSOs was shrinking and relations with government deteriorating. EVALUATION PURPOSE AND METHODOLOGY As GAPP came to its completion, USAID, in consultation with DFID, undertook a performance evaluation of the program for the purpose of providing USAID, DFID, and other stakeholders with 1) lessons learned from project implementation, and 2) recommendations for future programming, on anti￾corruption, accountability and governance interventions in Uganda. The evaluation took place between 3 September and 30 November 2019 (for the evaluation schedule, see Annex 1). The evaluation focused on three questions outlined in the SOW (see Annex 2). The questions were: 1. To what extent did GAPP interventions contribute to accountability processes, institutional strengthening, citizen participation, and voice at the local and national level? 2. To what extent are the GAPP implementation and management approaches relevant to the accountability, transparency, and anti-corruption context in Uganda today? 3. To what extent did GAPP’s Embedded Technical Specialists Model, Tripartite Outreach Model, and Ministry of Health Accountability Support satisfy the principles of Value for Money (VfM)? 1 LGs from the districts of Agago, Adjumani, Alebtong, Amolatar, Amuru, Apac, Arua, Buliisa, Dokolo, Gulu, Hoima, Kalungu, Kayunga, Kibaale, Kiboga, Kiryandongo, Kitgum, Koboko, Kole, Lamwo, Lira, Maracha, Masaka, Masindi, Mityana, Moyo, Mubende, Mukono, Nebbi, Nwoya, Otuke, Oyam, Pader, Yumbe, and Zombo. GAPP Performance Evaluation Report ix December 2019 Due to the complexity of the program and of governmental change, the evaluation team used a multi￾method approach that centers on two complexity-aware methodologies appropriate for this type of evaluation: Qualitative Comparative Analysis (QCA) and Outcome Harvesting to analyze GAPP’s contributions to reported change. The evaluation team collected data in the following ways: • Document review of over 100 secondary documents; • Interviews of 121 people through key informant interviews and focus group discussions; • Polling of 120 participants through a stakeholder survey; • Case study visits to six districts; • Outcome Harvesting Analysis to understand GAPP’s contributions to change; • Qualitative Comparative Analysis; • Value for Money Analysis. EVALUATION FINDINGS The study found that GAPP contributed to significant advances in government performance and accountability in Uganda. The team found evidence that GAPP’s focus on management and policy systems influenced significant shifts in the legal and policy environment for local government accountability and local governance systems. GAPP support to the national accountability institutions, Office of the Auditor General (OAG), Public Procurement Disposal Authority (PPDA), and the Local Government Public Accounts Committee (LG￾PAC) improved the national accountability environment, ensuring that local governments utilize public resources for the intended public services. GAPP contributions to improving the audit review process are particularly notable, as is the facilitation of greater collaboration and engagement of institutional actors with responsibility for this oversight through the Tripartite Outreach Model (TOM). Later in the project, GAPP extended support to the Ministry of Local Government (MoLG), which was key to ensuring sustainability of program accomplishments. In the districts themselves, GAPP contributed to stronger fiscal management, accountability, and decision-making processes. District planning, budgeting, accounting, and auditing functions are now enhanced in program districts. Local governments and local councils are far more skilled at working together and fulfilling their complementary roles. Work remains for better connecting these two bodies to the public for participatory planning and transparent accountability reporting. Senior staff reported a new understanding of leadership through the leadership development interventions, though the evaluation team did not see how these programs had fostered behavior change. GAPP facilitated citizen engagement in demanding improved public service delivery in GAPP-supported districts. As a result of advocacy grants, local governments undertook specific improvements in market infrastructure, health and education services, and public sanitation facilities. The evaluation found examples of citizens organizing petitions and presenting white papers that outline their case. However, these gains were individual instances resulting from the CSO short-term grants funded by the project. The end of this funding source makes the sustainability of these gains fragile. Given the shrinking space for civil society in Uganda today, it is vital to adopt programming strategies that reduce adversarial stances between CSOs and government and increase citizen involvement in public oversight. GAPP responded to a great thirst for information on roles, responsibilities, policies, and procedures to help district professionals do their jobs more effectively. Program stakeholders were very positive about the program’s implementation and management approaches and valued the training and technical assistance they received. What made the support most useful was the technical content, information, tools, and help with problem solving, which participants applied within the internal management systems in their organizations and departments. GAPP’s flexible, hands-on, and systems-oriented approach was instrumental in helping participants transfer the knowledge they gained into ongoing practice. USAID has recognized that building the capacity of a single actor or relationship is insufficient; the focus must be on the system in its entirety (USAID, 2014). The program’s application of this approach in connecting parts of the larger systems to change relationships and information flows was pivotal to the level of change GAPP Performance Evaluation Report x December 2019 that resulted. Unfortunately, the design of the non-state actor (NSA) component did not leverage these approaches to the same degree. GAPP tested the usefulness of the following three models as possible technical assistance innovations in Uganda: i) Embedded Technical Specialists (ETS); ii) Tripartite Outreach Model (TOM); and iii) Support to the Ministry of Health (MoH). While it was not possible to quantify governance outcomes without a measurable VfM framework established at the beginning of the program, the evaluation team found that the three models generated potentially good value. The ETS model can be credited with generating a successful legacy for GAPP, and therefore represents potentially good value for money for the original investment by USAID and DFID. Support to the MoH over the past year generated public savings well beyond the costs incurred from providing support. Finally, TOM can be credited with improving public accountability and citizen participation. CONCLUSIONS Having examined the contributions of the GAPP program towards local governance, accountability and participation, and stakeholder perception of value for money, the evaluation team reflected on findings to develop “lessons learned” and a set of recommendations for consideration in future programming. LESSONS LEARNED There are many lessons to be learned from the GAPP program implementation, especially regarding the systems strengthening approach to address complex change. While much of GAPP emphasized management policies and practices, participants also worked on the process and relationships that connect actors in the system. Lessons learned from GAPP approaches are as follows: Relationship-building is just as important as capacity building - Much of the success underpinning component one came from changing relationships among key institutions, such as the OAG, PPDA, and DPACs, through the tripartite model. Relationship-building also contributed to the effectiveness of component two. ETSs worked to cultivate fruitful working relationships between technical staff and politicians in districts. A lack of focus on changing current relationships between CSOs, PSOs, communities, and government officials at the district level may have hindered similar breakthroughs in component three. (The change in relationships between ministries and national CSOs resulting from GAPP programming illustrate the efficacy of this strategy.) This would require going beyond community needs assessments, which raise expectations, to engaging prioritization and decision￾making within the budgeting cycle which builds understanding. Local leadership is critical - As demonstrated from the QCA analysis, strong local leadership can overcome deficits in context conditions that might otherwise limit local government performance. This requires systems leadership that sets high standards of government responsiveness to citizens and identifies the pieces of the service delivery system. GAPP invested time building trust and confidence with local leaders. By putting local institutions in the lead, GAPP built ownership for change. Additionally, implementation by a largely local staff (Ugandan nationals comprise 65 of 66 staff) who understood the context benefited the program immensely. Feedback, particularly when made public, can motivate behavior change - While GAPP did not introduce the Local Governance Performance Assessment and the Local Government Councils Scorecard, the program funded them in four GAPP-supported districts and made use of their results in the 40 target districts. These publicized annual external assessment tools help to benchmark district performance against expectations and between districts. District officials were aware of their scores and the reasons behind successes or shortcomings, thus increasing motivation to make best use of the capacity strengthening support offered by GAPP. Each spoke of holding a leadership debriefing after the release of assessments to make plans for raising the score in the next year. Conversely, the Partner Local Organization Capacity Assessment (PLOCA)-supported organizations did not benefit from external feedback from stakeholders (e.g., constituents, donors, government officials). That program used only self-assessments. While self-assessments are powerful for internal learning, they do not replace external feedback and measurement tools for gauging organizational performance GAPP Performance Evaluation Report xi December 2019 Citizen-led engagement can significantly contribute to government responsiveness - The program demonstrated that when citizens are engaged, government responds, particularly at the sub￾county level. Some responses resulted from government outreach to citizens, and some from CSOs and PSOs lobbying with program grants. Reporting results achieved from budget expenditures was new for district governments, as lobbying was a departure from the protests and critiques that have typically been used by Ugandan NSAs to voice their views. GAPP demonstrated that both local governments and CSOs have a role to play in informing and engaging citizens. Role of research as a basis for lobbying - GAPP national CSO partners received funding to do research on critical issues affecting decentralization, such as funding. Findings were widely publicized. This demonstrated a depth of knowledge that lent credibility and objectivity to advocacy efforts. Donor coordination and collaboration leads to good value and good practice - USAID and DFID working together on GAPP generated time and cost savings for each partner, thereby improving VfM Economy. It has most likely reduced efforts by Government of Uganda counterparts at national and local levels. This cooperation has arguably extended each development partner’s influence by broadcasting the right messages on donor coordination and effectiveness. Planning sustainability pathways is critical to the outcomes achieved thus far - Sustainability is a subset of VfM Effectiveness, and for many development practitioners it is seen as an additional pillar to 4Es. For example, in designing pathways for sustainability in its last year, GAPP saw that MoLG was emerging as an important actor whose influence needed to be firmly established and whose capacity needed to be strengthened. An ETS who was working in Acholi was moved to the Ministry, recognizing the effectiveness of the program work achieved with LGs by the ETSs. MoLG’s profile as a leader in ‘local government sector’ can benefit from GAPP experience and expertise. RECOMMENDATIONS Founded upon these lessons, the evaluation team makes the following recommendations for USAID, DFID, and other donors to support LG accountability and performance and citizen participation going forward. (See details of these recommendations in section 4.2) 1. Scale up programming innovations - The ETS and TOM were critical to the success of other program activities. The key to their sustainability is to institutionalize these innovations within Ugandan ministries. Future programs would need to enhance MoLG’s District Resource Pools to more closely resemble the ETS model. 2. Expand engagement on service delivery issues to other ministries - GAPP support focused on the three most direct accountability actors in the audit cycle. In the course of project implementation, though, it became clear that other functionaries who were not directly involved played key roles, e.g., the MoLG, MoFPED, Ministry of Public Service (MoPS), and the National Planning Authority (NPA). 3. Invest in leadership, not leaders - GAPP invested considerably in the development of senior leadership in LGs and LCs. However, a focus on systems leadership, not individual leaders, might be more appropriate. This would bring together people from all three sectors (government, private sector, and civil society) who would work together to find solutions to problems that exist in their current context. In monitoring the success of program interventions, future programming should go beyond recording participant reactions to the learning event and a self-assessment tool and use a framework like the Kirkpatrick Model to measure the effectiveness of efforts. 4. Strengthen citizen participation in planning - For citizens to shift from passive recipients of public services to being active in oversight, they must be involved in planning and decision-making for services from the start. Future assistance should engage the NPA to make the planning and monitoring process truly participatory in the way that GAPP was able to introduce true accountability into the audit review process. GAPP Performance Evaluation Report xii December 2019 5. Strengthen councillors in their role of constituent engagement/representation - GAPP gave significant attention to the role and procedures of councillors in passing policy and providing oversight to accountability processes. The role of councillors in constituent engagement and representation was given less attention. Due to resource constraints, councillors often avoid constituents, which contributes to them being unseated in the next election. Strengthening this role could help reverse the high election turnover and improve citizen participation in local governance. 6. Apply systems strengthening strategies to NSA programming - The next round of civil society engagement can increase its emphasis on systems programming in this sector and incorporating this sector as part of governance as a whole. Leverage development innovations from within USAID and lessons from other missions to experiment with different development programs that will engage citizens for “mindset change” i.e., Capacity 2.0, Local Works, and Problem Driven Iterative Adaptation (PDIA). 7. Expand stakeholder feedback mechanisms in programming - Several powerful citizen feedback tools influence government behavior, e.g., community scorecards and social audits. These feedback instruments can be incorporated into government planning processes at the start of the DDP planning cycle and at mid-term review to provide citizen perspective on service quality. Similarly, future programming should support and build upon the Ugandan Quality Assurance Certification Mechanism (QuAM) NGO certification process and constituent feedback techniques to provide CSO partners with external feedback beyond self-assessment reflection processes. 8. Design grant making to strengthen multi-stakeholder linkages and build relationships - CSO grant programs typically start with a call for proposals for projects, which reinforces traditional donor-grantee relationships. Consider cluster approaches that use Collective Impact or solicit program ideas following a ‘whole system’ design event, such as Future Search or Appreciative Inquiry. Plan activities for longer than 1-2 years and provide ETS support to change relations between stakeholders. 9. Integrate citizen engagement and oversight into sectoral programs - In Uganda, common structures exist in service delivery facilities. However, they are often weak or dysfunctional. Donors could strengthen the understanding of citizen oversight for public services by integrating the use of these committees as part of sectoral programs whenever appropriate. Similarly, donors could incorporate actual service delivery challenges and district sectoral development indicators into LG performance management support. 10. Maintain a central ‘VfM Register’ for cost savings to be systematically recorded - GAPP’s VfM reporting included examples of savings, but often without monetary values. As numbers are critical to reporting, future programs should maintain a spreadsheet of savings with monetized values. Sample VfM metrics are explained in Annex 10. 11. Use VfM Case Studies in reporting - Use ‘VfM Case Studies’ for programs where results are hard to monetize. In a case study, costs and benefits from an activity or intervention are isolated for focus. (A template is provided in Annex 10.) For example, MyTax pilot in Masindi, or Buliisa revenue assessment (GAPP, April 2019), could lend themselves to VfM case studies. 12. Track revenue increases and savings through improved procurement as two substantial ways to monetize benefits - Improvements in national surveys and indices (e.g., LGPA) are also useful but changes in those scores are not possible to translate into monetary benefits. Future programming should monitor local revenue increases and procurement savings (through audit processes) to help develop more robust VfM narratives. GAPP Performance Evaluation Report 1 December 2019 1 BACKGROUND Uganda’s shift to decentralized government through the Local Government Act of 1997 was intended to create transformation in the country. The belief was that local governance would bring development, oversight, and accountability closer to the citizens, thus strengthening democratization and public service quality and decreasing corruption. While decentralization was key to Uganda’s historical transformation, it has not yet fully met the needs of citizens for public service quality. Corruption remains an issue, often rooted in the local governance processes. In November 2012, USAID/Uganda awarded the five-year Governance, Accountability, Participation and Performance (GAPP) program to RTI, in collaboration with DAI and Palladium. The GAPP program was designed to strengthen the enabling environment for improved service delivery in local governments. The program’s Theory of Change states: If Government of Uganda (GOU) and nongovernment-led efforts are supported to improve democratic governance and accountability, and citizens’ efforts are mobilized to engage and partner with Local Governments (LGs), then service delivery will be more equitable and efficient. (See Figure 1) Figure 1: GAPP Theory of Change The goal of GAPP was to increase participation, accountability and local governance for effective service delivery. The program intended to achieve this goal through three objectives: 1) to improve the legal, policy, regulatory, and institutional environment to meet demands for more democratic governance; 2) to enhance fiscal management and accountability processes in the delivery of services in targeted districts; and 3) to build the capacity of citizens and communities to participate in local governance voice and accountability. These objectives correspond to the three interdependent GAPP components: Component 1: Supporting and strengthening national-level government accountability institutions in their oversight and interaction with LGs. Component 2: Strengthening local government revenue, procurement, financial, and planning systems. Component 3: Strengthening CSOs and other non-state actors to improve voice and accountability. GAPP Performance Evaluation Report 2 December 2019 GAPP’s original $17.7 million budget was funded solely by USAID. In January 2015, the United Kingdom’s Department for International Development (DFID) joined as the program’s co-funder. As a result, the Program duration was extended to November 28, 2019, with a new program value of US$35.4 million, reflecting a one-to-one matched contribution between DFID and USAID. As of August 2019, the project had expended USD $33.6 million. The largest expenditures were in component 3 (39%) and component 2 (37%). (See Figure 2) The Program initially was operational in 25 districts. However, with additional DFID funding in 2015, the range of technical areas broadened and deepened to also support sub-counties, town councils and municipalities, and ten additional districts in Northern Uganda and Bunyoro sub-region were added. Coverage expanded to 35 districts.2 In August 2017, GAPP’s work plan was amended to include refugee-focused activities, Northern Uganda Regional Forum support, and accountability support to the Ministry of Health (MoH). By the end of 2018, GAPP covered 40 districts due to some districts splitting. (See Figure 3) The program operated within the context of significant social and political change in Uganda. The assumptions underlying the program’s theory of change, all presented significant challenges to the program’s success, namely: 1) political will at the national level to support local governments; 2) the ability of local governments to assimilate the support provided; and 3) open democratic space. • Political support for decentralization – While the GoU officially supports decentralization, it has gradually recentralized key government functions, such as tax collection and revenues (GAPP, 2018). • Local government absorptive capacity – The continued creation of new local government districts and municipalities stretches already scarce financial, material, and human resources and creates a local government structure that is poorly equipped to function effectively (ACODE, 2017). • Democratic space for civil society – The operating environment for civil society continues to shrink and relations with Government continue to deteriorate. CSOs are labeled “agents of governments from the West,” compromising their credibility and effectiveness to mobilize citizens to demand government accountability (GAPP, 2018) (Uganda National NGO Network, 2018). The GAPP program further took place within the context of national and international donor priorities. USAID’s new policy framework of The Journey to Self-Reliance emphasizes national capacity to plan, finance, and implement solutions to local development challenges and a commitment to see these through effectively, inclusively, and with accountability (USAID’s Policy Framework, n.d.). USAID/Uganda is integrating this new policy into its programs by building local ownership and systems thinking into 2 LGs from the districts of Agago, Adjumani, Alebtong, Amolatar, Amuru, Apac, Arua, Buliisa, Dokolo, Gulu, Hoima, Kalungu, Kayunga, Kibaale, Kiboga, Kiryandongo, Kitgum, Koboko, Kole, Lamwo, Lira, Maracha, Masaka, Masindi, Mityana, Moyo, Mubende, Mukono, Nebbi, Nwoya, Otuke, Oyam, Pader, Yumbe, and Zombo. DFID USAID later splits Figure 2: GAPP Expenditures - $33.6 million Figure 3: Districts Receiving GAPP Support by Donor GAPP Performance Evaluation Report 3 December 2019 program design (USAID Uganda, 2017). DFID has been undergoing similar transitions by adopting a Value for Money (VfM) approach that maximizes the impact of development investments to make more informed, evidence-based choices through a better understanding of costs and results. GAPP expanded several innovations at the request of the donors. The first was four regionally based teams of Embedded Technical Specialists (ETS). Team members were specialized in one of three areas of expertise: financial management, procurement and local revenue mobilization (LRM); planning and budgeting; and leadership and council development. Together, these teams of specialists paid regular visits to each of the districts in their regions to give tailored technical assistance to LGs and Local Councils (LCs) as they applied new learnings and solved challenges in their districts. GAPP also introduced the Tripartite Outreach Model (TOM) which brought officials from Public Procurement and Disposal Authority (PPDA), Office of the Auditor General (OAG) and Public Accountability Committee in Parliament on Local Governments (PAC LG) to the districts for full public hearings on a public audit. Before this, LG audit related hearings were not systematically held, the proceedings from the hearings were non-binding, and issues raised in reports were often not brought to the attention of the wider public., The TOM brought the hearings into public scrutiny and resulted in the PAC LG formally tabling a committee report before Parliament. This is a significant development in that it ensured that mismanagement could be investigated and consequences given. Lastly, the introduction of Value for Money (VfM) considerations were introduced with DFID joining the program. The project was able to incorporate VfM thinking into their own work, as well as support the OAG to carry out VfM audits and institutionalize the methodology. These three activities, in combination with the other assistance offered by GAPP, proved to be an effective combination of support for improved local government performance and accountability. GAPP Performance Evaluation Report 4 December 2019 2 EVALUATION PURPOSE & METHODOLOGY As GAPP came to its completion, USAID, in consultation with DFID, undertook a performance evaluation of the program for the purpose of providing USAID and other stakeholders with 1) lessons learned from project implementation, and 2) recommendations for future programming, including newly designed anti-corruption, accountability and governance interventions in Uganda. The evaluation took place between 3 September and 30 November 2019 (for the evaluation schedule, see Annex 1). The evaluation focused on three questions outlined in the SOW (see Annex 2). The questions were: 1. To what extent did GAPP interventions contribute to accountability processes, institutional strengthening, citizen participation and voice at the local and national level? 2. To what extent are the GAPP implementation and management approaches relevant to the accountability, transparency and anti-corruption context in Uganda today? 3. To what extent did GAPP’s Embedded Technical Specialists Model, Tripartite Outreach Model, and Ministry of Health Accountability Support satisfy the principles of VfM? The evaluation team consisted of five primary investigators: Meg Kinghorn, Team Lead; John Kiyaga￾Nsubuga, Local Governance Specialist; Laura Nyirinkindi, Civil Society Specialist; Wolfgang Stuppert, Evaluation Support Specialist; and Gulden Bayaz, VfM Specialist. Additional support was given by JP￾Jennifer Pierre, Evaluation Manager; Rachel Santos, Evaluation Support Specialist; Innocent Kiiza, Survey Team Lead; and Aidat Namakula and Genevieve Onyango, Research Assistants. 2.1 DATA COLLECTION AND ANALYSIS Due to the complexity of the program and of governmental change, the evaluation team used a multi￾method approach (see Box 1) that centers on two complexity-aware methodologies appropriate for this type of evaluation: Qualitative Comparative Analysis (QCA) and Outcome Harvesting to analyze GAPP’s contributions to reported change. The evaluation team collected data in the following ways: ● Document Review of GAPP program documents, results frameworks, quarterly and annual reports, previous evaluations, and publications related to relevant work from other organizations (see Annex 3 for documents reviewed). ● Key Informant Interviews (KIIs) with representatives of stakeholder groups such as local government and local council representatives, civil society organizations, and national government officials directly engaged in or benefiting from the program to learn of changes since the start of GAPP and how the program contributed to those changes (see Annex 4 for the list of stakeholders consulted). Interviews were based on semi-structured guides, using Outcome Harvesting methodology (for an outline of the interview questions, see Annex 5). ● Focus Group Discussions (FGDs) with GAPP staff to gain a detailed understanding of the modes of implementation of GAPP activities (for an outline of the discussion questions, see Annex 6). Box 1: GAPP Evaluation Mix of Methods • Document review of all program reports and publications • 120+ key informant interviews • 4 focus group discussions with GAPP staff • 120 telephone survey interviews • Outcome Harvesting on the national level and in 6 field studies of 4 districts and 2 municipalities (4 GAPP & 2 non￾GAPP). • Qualitative Comparative Analysis (QCA) of 113 districts • Value for Money framework analysis • Participant observation of 3 GAPP Close￾out meetings GAPP Performance Evaluation Report 5 December 2019 ● Stakeholder Survey of program participants regarding their perceptions of GAPP’s effectiveness and their recommendations for similar programs in the future (see Annex 7 for the survey tool used). Data analysis was guided by GAPP’s theory of change and recent research on decentralization in Uganda. The team drew conclusions regarding the contributions of GAPP using the following methods: ● Outcome Harvesting - The team conducted Outcome Harvests on the national level and in six exemplary districts. Outcome Harvesting is an exploratory and strongly participatory approach to analyze contributions of interventions to development outcomes (Wilson￾Grau/Britt 2012). The evaluation team collected (harvested) outcome stories in KIIs and validated them through comparison with other data sources and follow-up interviews. These stories helped the team link GAPP interventions to changes in national policy, LGs, and citizen engagement. ● Qualitative Comparative Analysis - The team used data on the economic, social, and political make-up of 113 LGs in Uganda, including 33 GAPP districts and municipalities. These are all LGs that were evaluated in the Local Governance Performance Assessment (LGPA) and preserved their territorial integrity since 2012. The analysis identified the context-specific requirements in which GAPP contributed to local government performance (for more details on the analysis, see Annex 9). ● Value for Money (VfM) – The team used VfM economic analysis to determine if the GAPP program provided sufficient returns. The methodology combines quantitative analyses (cost benefit, unit costs, break even, cost effectiveness) and qualitative analyses across four sub￾categories (economy, efficiency, effectiveness, and equity). (For details on VfM, see Annex 10.) 2.2 LIMITATIONS ● Limitations in causal analysis – Due to the qualitative nature of the Outcome Harvesting data collection, outcome stories are influenced by differences in perceptions and opinions. Such threats to their internal validity were mitigated through a standardized validation process that involved the triangulation of stakeholder perspectives and of data sources, and, for changes on the local level, the systematic comparison of outcome stories across LGs. ● Limited number of LGs analyzed - The logic of inference used for Outcome Harvesting follows a Most Different Systems Design. That is, it compared very different LGs and tried to identify GAPP-supported mechanisms of change similar across them. If an outcome was linked to GAPP interventions in similar ways in very different contexts, it gave credibility to the claim that GAPP contributed to this change. This means the analysis is most reliable for changes facilitated by well-integrated and frequent district-level GAPP interventions, such as training with councillors/LG staff and ETS support. However, the limited number of LGs the team was able to analyze risks underestimating performance in less integrated and more infrequent district-level GAPP interventions, such as local revenue mobilization and planning in refugee-hosting districts. ● Limited data to support VfM - GAPP was not designed with a VfM strategy or framework. It lacked an overall economic analysis or a set of VfM indicators to be monitored periodically. Instead, GAPP’s VfM reporting often focused on qualitative examples about cost savings achieved by GAPP teams. Moreover, the programming area does not lend itself well to cost-benefit analyses, because the type of benefits it generates are not easy to monetize (e.g. strengthened audit processes or enhanced community voice). Value delivered by GAPP as well as VfM best practices were reviewed and compiled considering these challenges. GAPP Performance Evaluation Report 6 December 2019 3 EVALUATION FINDINGS Through its work, the evaluation team found GAPP to have contributed to significant advances in government performance and accountability in Uganda. This section presents the findings from the evaluation, organized by each evaluation question, before summarizing the exercise in lessons learned and recommendations for future programming. 3.1 Q1. GAPP CONTRIBUTIONS TO ACCOUNTABILITY PROCESSES, INSTITUTIONAL STRENGTHENING, CITIZEN PARTICIPATION AND VOICE AT LOCAL AND NATIONAL LEVELS Overall, the study found that GAPP’s focus on management and policy systems influenced important shifts in the legal and policy environment for local government accountability and local governance systems. Through support to the national government institutions ensuring accountability of local governments, GAPP helped unblock the audit system to become a much more effective oversight process rather than simply a bureaucratic one. By combining training and one-to-one follow-up technical assistance, they helped local councils become functional, where they were previously mired in conflict and minutia. Local government departments improved their planning, budgeting, and local resource mobilization capabilities through the same support combination. And, importantly, the GAPP program helped all three groups (local governments, local councils, and national institutions) strengthen their ability to work together. Though to a lesser degree, GAPP also contributed to advancements in citizen engagement to improve public service delivery in GAPP-supported districts. Through grants to CSOs to lead community advocacy efforts to improve service delivery, many local governments acted to rectify service deficits, positively affecting the lives of many people. Citizens remain largely out of the process of governance, however, and these efforts are likely to remain dependent on donor funding and initiative. Using the Outcome Harvesting method, the evaluation team collected outcomes from various sources (e.g., KIIs, documents, Close-out meetings, district visits), examining those with credible evidence of GAPP contribution. The team then validated these outcomes and contribution claims from other sources, tracing the process of how the outcome originated from the GAPP intervention. The following section summarizes the changes the team found that have occurred because of the GAPP program. Detailed descriptions of the outcomes listed in Box 2 can be found in Annex 12. Because the evaluation question includes a focus on three distinct yet interlocking components, 1) the national accountability environment, 2) district level government performance, and 3) citizen participation and voice, each component is addressed individually. 3.1.1 GAPP Contributions to Improved National Local Government Accountability Environment One primary role of local governments in Uganda is to deliver quality public services (e.g., health, education, water and sanitation) to citizens. This requires the national government to have a strong oversight and accountability environment to ensure scarce resources are directed appropriately. Unfortunately, with gaps in accountability policy and practice, the opportunity for corruption and mismanagement is high. The evaluation found that GAPP made significant contributions to improving this environment, particularly through the audit review process. These contributions included clearing backlogs in the OAG and PPDA, strengthening Parliamentary rules related to audits, and influencing policies in the extractive sector through CSO advocacy. (See Annex 12 for detailed Strong GAPP Contribution 1. Stronger collaboration between tripartite members 2. Coverage of procurement audits significantly increased in GAPP districts Medium GAPP Contribution 3. PAC-LG gives timely consideration of OAG reports 4. Parliamentary rules amended to strengthen treasury memorandum Potential GAPP Contribution 5. CSOs influence transparency and accountability policies in extractives sector 6. CSOs influence thinking on decentralization and LG funding *Details for all the outcomes listed here can be found in Annex 12 Box 2. National Level Outcomes GAPP Performance Evaluation Report 7 December 2019 stories for outcomes listed in Box 2.) GAPP also facilitated greater collaboration and engagement of institutional actors with responsibility for this oversight. It is important to note that these results have national significance, since they affect local government accountability in all Ugandan districts, not only those in the GAPP program. 3.1.1.1 Enhanced oversight of local governments by National Accountability Institutions (NAIs) Due to GAPP support, the two mechanisms instated from the Ugandan central government for local government accountability--annual local government audits from the OAG and PPDA--show signs of being effective. The reports from both are reviewed by the PAC LG, following an investigation of any audit findings. However, the evaluation team found previously that few people fully understood the process (Figure 4), making it easy for negative audit findings to get lost (intentionally or unintentionally), while allowing abuses to escape consequences. Additionally, an increasing number of local government entities created significant demands on the NAIs to investigate audit findings. As a result, the audit investigation backlog in the PAC LG and PPDA had grown to 5+ years. (For the details of each step in the process map in Figure 4, see Annex 11.) Interviews reveal three major challenges that led to the institutional difficulties: 1) shortfalls in personnel with the skills required to conduct timely audits, 2) inadequate funding to conduct comprehensive audits in all districts, and 3) organizational methodological shortcomings that did not allow institutions to complete audits within required timelines. Furthermore, the operational linkages between the national accountability mechanisms were weak and the processes inefficient, as evidenced by the backlog in the compilation and submission of the OAG’s and PPDA reports for consideration by PAC Local Government. Therefore, GAPP focused its technical and financial support on three national accountability institutions, OAG, PPDA, and PAC LG, to eliminate audit report backlogs. This resulted in an increased uptake of accountability reports in Parliament (see outcome #3 detailed in Annex 12). PPDA dramatically reduced the time and money needed to conduct audits by developing their in-house audit capability and introducing an investigation process when an entity is small and a full audit not merited (see outcome #2 detailed in Annex 12). With GAPP support, CEPA advocated for changes to how Treasury Memoranda are referred from Parliament to MoFPED, improving the quality and efficiency of PAC LG deliberations in Parliament (see outcome #4 detailed in Annex 12). At the time of the evaluation, all OAG audit investigations were up to date and PPDA had increased their audit coverage by an estimated 78% in GAPP-supported districts (GAPP, November 2019). As this happened, the number of audits returning unqualified audit opinions soared from 34 percent in 2013 to 80 percent in 2018. Targeted LGs also increased their action on audit recommendations from less than 55 percent in 2013 to 80 percent in 2018. Figure 4: Process Map of the Audit Cycle GAPP Performance Evaluation Report 8 December 2019 3.1.1.2 Strengthened collaboration between key players in the accountability chain for closer LG scrutiny Due to GAPP support, the national institutions involved in local government accountability improved their coordination and adopted an effective mechanism for public oversight. Fragmentation between the key players in accountability oversight created gaps and confusion regarding the overall process. The national institutions did not work together to provide a cohesive environment and relations with local government PACs were tense and counterproductive. GAPP changed those relationships from confrontation to collaboration through two key interventions. GAPP introduced the Tripartite Outreach Model (TOM), changing audit investigations to be conducted in real time at the district level, thus fast tracking the work of the PAC Local Government and increasing stakeholder engagement in the process (outcome #1 detailed in Annex 12). Before the TOM, each institution conducted their own audit investigation in parallel and in Kampala. Few district personnel appeared to respond to queries, causing delays in PAC LG’s work. The TOM reversed this. The PAC LG traveled from Kampala to the districts under investigation. This facilitated direct interactions at the district level between citizens, contractors, District Public Accounts Committee (DPAC) and CSOs. It also helped both DPACs and technical staff better understand the process and their mutual roles, facilitating coordination. As a result, OAG’s reports were considered by PAC Local Government in shorter timelines and with more comprehensive consideration and information gathering. This approach has been so well received in the Legislature that in 2019, Parliament convened a Tripartite approach using government money without GAPP support, and the PAC Local Government has adopted the model as a best practice. 3.1.1.3 National CSOs effectively advocate for sectoral accountability and local government resourcing By involving national CSOs in conducting research and advocacy to improve decentralized government, these organizations earned new levels of credibility and influence in working with government. The level of mistrust between the GoU and leading national CSOs has historically been high with government perceiving CSOs as antagonists. As a result, civil society organizations in Uganda have not been able to constructively engage in policy dialogue and influence. This began to change with GAPP interventions at the national level. GAPP funded several leading national CSOs to undertake research and advocacy to document the challenges facing local governments, particularly with regards to financing. Over the course of several grant cycles, the research papers and advocacy efforts laid an instrumental foundation for increasing government oversight policy reforms in key sectors (outcome #5 detailed in Annex 12) and ensuring that funds designated for local governments are released on time (outcome #6 detailed in Annex 12). One positive unintended result is that the grants gave the CSOs access to high-level officials and demonstrated CSO skills in constructive policy advocacy. Their notable achievements caught the attention of national institutions, resulting in improved credibility and trust between government institutions and the CSOs. Since then, these CSOs have increased their participation in monitoring government decision making processes (see Box 3). The returns on these strategic engagements will likely sustain GAPP results beyond its lifetime. Overall, the evaluation found that GAPP made a strong contribution to the national accountability environment by resolving the institutional blockages that prevented key accountability institutions from executing regulatory compliance. Thus, the accountability wheels started turning, giving impetus to institutions to exercise their roles and responsibilities. It further strengthened the engagement of national CSOs as development partners and advocates for government accountability of public resources and the delivery of quality public services through appropriate support and financing of decentralization. Over the course of several GAPP grants for research-based advocacy, one CSO partner gained access and visibility among government officials for its work. With a stronger reputation as a credible partner, the CSO has been invited to sit on the high-level Public Expenditure Management Committee (PEMCOM). Box 3: CSOs Invited to National Decision-making Bodies GAPP Performance Evaluation Report 9 December 2019 3.1.2 GAPP Contributions to Strengthened Local Government Fiscal Management, Accountability, and Decision-Making For decentralized government to work, it must maximize its resources to effectively deliver public services and build public trust for what it has done. GAPP significantly contributed to strengthening the fiscal management, accountability, and decision-making of local governments and municipalities in program districts. (See Annex 12 for detailed outcome stories for outcomes listed in box 6.) The key success factor was the combined effect of bringing national accountability institutions (i.e. OAG, PPDA and PAC LG) closer to the district level through the introduction of Tripartite Outreach Meetings (TOM). In addition, the targeted training and dedicated ETS technical follow-up support GAPP provided to local government administrative staff and political representatives strengthened systems, skills, and relationships. The three interventions led to improvements in local governance and service delivery as detailed below. As a result of this combined assistance, both the LGs and LCs in GAPP districts are performing their roles. Additionally, they work together to each play their role in the governance and accountability cycle. 3.1.2.1 Where GAPP interventions most contributed to local government performance GAPP-supported local governments perform better on the Local Government Performance Assessment (LGPA) than districts and municipalities that did not receive GAPP support. In the 2018/2019 LGPA, the average performance of GAPP-supported local governments was 67.58 out of a maximum of 100 points, while local governments without GAPP support achieved 63.40 points. A year before, in the first LGPA assessment, the difference was 61.45 to 53.29 points, respectively. The following analysis goes beyond a comparison of average performance. It identifies the conditions under which LGs reach high performance, and the role GAPP support plays in this achievement. It shows that a majority of GAPP-supported local governments performed significantly better on the LGPA than they would have without GAPP support. Districts and municipalities that were not included in the program typically required a much more conducive environment to achieve similar levels of performance. GAPP provided support to local governments with very different assets and needs. Among GAPP￾supported LGs were districts at the tail end of Uganda’s economic development, but also dynamic municipalities with rising levels of local revenue. Our qualitative comparative analysis (QCA) shows that despite such stark differences in context conditions, the GAPP program was able to make essential contributions to governance performance for 52 percent of the LGs that it supported. In these cases, the comparative analysis identified GAPP’s district-level support as indispensable for LGs to achieve high levels of institutional performance. That is, these LGs would not have reached such high levels of performance without the interventions of the GAPP program. This suggests that GAPP’s approach to local government support has proven sufficiently adaptive and targeted to successfully engage with very different local governments. At the same time, not all GAPP-supported local governments would have needed the program’s interventions to achieve high performance. This applies to 15 percent of GAPP￾supported LGs. The QCA suggests that these LGs would have reached high levels of performance even without the support of the program. This points to opportunities in improving program efficiency, by focusing efforts on those LGs where external support appears to be essential. The comparative analysis is based on socioeconomic and performance data of 113 Ugandan LGs. Among these LGs are 30 GAPP-supported districts and municipalities. The team measured the performance of local government institutions based on the LGPA 2018/2019. If LGs received LGPA scores above the national average, they were classified as high performing. The LGPA is the only comprehensive assessment of local governments in Uganda with a coverage that goes beyond individual development programs. It strongly emphasizes administration and service delivery. It measures three dimensions of performance: (1) budget and accountability requirements, (2) crosscutting and sector functional processes, and (3) service delivery results in the areas education, health, and water. In the context of the comparative analysis, “high performing” means that the LG achieved above-average results across these dimensions of performance. Among the LGs we analyzed, 55 percent were high performing (73 percent among GAPP-supported districts, 48 percent among LGs that did not receive GAPP support). Based on GAPP Performance Evaluation Report 10 December 2019 this data, the team asked the question: Under which conditions do LGs reach high performance, and what role does GAPP play in this? The comparative analysis identified five conditions that help explain 78 percent of the cases of high performance among the districts and municipalities in Uganda that were included in the analysis. These conditions emerged from a range of performance-related context conditions that were tested for their ability to differentiate between high and low performing LGs. The five conditions are: (1) high GDP per capita; (2) strong service structure; (3) availability of essential LG staff; (4) high local revenue; and (5) a history of clean audits. Among the discarded context conditions are the party affiliation of the council chairperson, total of central government grants, shortfalls in local revenue mobilization, population size, number of refugees, as well as accessibility of secondary schools, health centers, and the police. As much as the available data allowed, these conditions describe the context of LGs at the start of the GAPP program. The first three of the five context conditions are structural (represented by the darker blue cog wheels in Figure 5). They represent aspects of the local context that are not readily influenced by local leadership. For example, the strength of a service structure refers to the availability and distribution of public service facilities. The other two are managerial in nature. They represent the fruits of a well-organized and motivated workforce in local administration as an indication of successful local leadership. A clean audit, for example, depends on the ability of local leadership to establish a culture of transparency and accountability in local government. These conditions are not necessarily the drivers of local government performance. That is, the analysis does not suggest that in order to improve local governance, interventions should focus on increasing local GDP and building schools to improve the service structure. However, these conditions indicate what constitutes fertile soil for high LG performance. They help us differentiate between contexts in which (1) GAPP support is unnecessary for high local government performance, (2) GAPP support is necessary, and (3) GAPP interventions contributed most to performance improvements. The QCA identified thirteen contexts in which local governments reached high performance (see Annex 9). They feature different combinations of the five context conditions (see Box 4). For example, one context features the presence of a strong service structure, but the absence of essential staff and high local revenue. The analysis shows that whenever more than one structural conditions was in place, GAPP support was not needed for LGs to reach high levels of institutional performance. Districts and municipalities that could rely on, for example, a strong service structure and the availability of essential staff, almost always achieved above-average performance scores, with or without GAPP support. Such a combination of context conditions represents fertile soil for local government performance. When there was no structural context condition in place, however, LGs without GAPP support reached high institutional performance only in rare cases. The GAPP program, in turn, was often able to turn arid land into fertile soil. The analysis indicates that with GAPP’s support, some of the districts and municipalities achieved high performance without any context condition in place. The program made the strongest contributions to local government • High GDP per capita: GDP per capita in the district/municipality above $710. • Strong service structure: Less than 5% of population more than 5 kilometers away from primary school. • Essential staff available: Staffing shortage below 39%. • High local revenue: Local revenue per capita above UGX 750. • Clean Audit: Unqualified Auditors’ Opinion in OAG Audit 2012. Box 4: Context conditions for high LG performance Figure 5: LG Context Conditions GAPP Performance Evaluation Report 11 December 2019 Table 1: District Status of Context Conditions performance when it was planted in soil that provided a basis for growth but was not fertile enough to grow strong institutions without external support. This basis for growth is already provided by either of the managerial context conditions. For LGs with just one of the managerial context conditions in place, GAPP support increased the LGs that reached high performance from 31 percent (for LGs with high local revenue) and 33 percent (for LGs with clean audits) to 100 percent. That is, GAPP made the biggest difference when it supported LGs that showed indications of good local leadership. The role of local leadership features strongly in the stories of change harvested in the three high-performing GAPP-supported LGs (Adjumani, Apac MC, and Kiboga). The outcome stories that formed in a structurally well-equipped non-GAPP district (Mpigi), on the other hand, demonstrate that if structural conditions are in place, institutions develop processes that do not require strong individual leaders or external support to achieve high levels of performance. Table 1 shows the combinations of context conditions that feature in the local governments visited by the evaluation team. The data refers to the situation in 2012. Whenever data for the target year was unavailable, the closest year with available data was chosen. 3.1.2.2 Improved functionality in Local Government Councils Local government councils in GAPP-supported districts are now able to perform their roles in passing budgets and conducting district business. Given the high rates of turnover in elections and the lack of academic requirements for the role, many newly elected councillors did not understand the council’s official role and process in the budget cycle. They did not understand the interdependent functions of the administrative and political branches of local government in raising local revenues, passing budgets, and ensuring fiscal oversight. They did not know the legal limits on their emoluments. Simultaneously, information sharing from administrative departments was often inadequate, preventing councillors from fully understanding local government functions. These factors caused frequent tension and conflicts between the two branches. Through induction and joint trainings, GAPP helped resolve these tensions by assisting councillors and technical staff understand their complementary roles (see Box 5). They distributed copies of the Local Government Act and Local Council Rules of Procedure which gives guidance on the responsibilities and boundaries of each. This support led to attitudinal shifts on both LG High GDP Strong service structure Essential staff High LR Clean audits High performing Adjumani* No No No Yes No Yes Apac MC* No No Yes No No Yes Buliisa* No Yes Yes Yes No No Kapchorwa MC No Yes No Yes No No Kiboga* No No No Yes No Yes Mpigi Yes Yes Yes Yes No Yes Notes: *GAPP LGs One first term district councilor was elected Council Speaker. He received training on his role, but he was still learning what to do. His first council meeting was particularly contentious. But the GAPP ETS observed the meeting proceedings and sent a few critical coaching messages to the Speaker that helped make the meeting a success. One first term district councillor was elected Council Speaker. He received training on his role, but he was still learning what to do. His first council meeting was particularly contentious. But the GAPP ETS observed the meeting proceedings and sent a few critical coaching messages to the Speaker that helped make the meeting a success. Box 5: Critical Coaching from a GAPP ETS Box 5: Critical Coaching from a GAPP ETS Box 5: Critical Coaching from a GAPP ETS Box 5: Critical Coaching from a GAPP ETS Strong GAPP Contribution 7. Local Councils Accomplish Agendas with Broader Participation 8. Local Councils pass annual district budgets and approve annual work plans 9. Relations Between Politicians and Technical Staff Improved 10. Audit Sanctioning of Financial Mismanagement Improved 11. Performance of District Public Accounts Committee Improve 12. Local Government Organizes LRM Process More Efficiently 13. Local Government Planning More Integrated Potential GAPP Contribution 14. Gender based budgeting 15. Local Government Improves Accountability Mechanisms to Citizens *See Annex 12 for outcomes stories of all outcomes listed 16. CSOs influence thinking on decentralization and LG funding *Details for all the outcomes listed here can be found in Annex 12 Box 6: District Level Outcomes Box 6: District Level Outcomes Box 6: District Level Outcomes Box 6: District Level Outcomes GAPP Performance Evaluation Report 12 December 2019 sides so they could jointly focus on service delivery (outcomes #8, #9 listed in Box 6 and detailed in Annex 12). As a result, the relationships between political leaders and technical staff in GAPP-supported districts have remarkably improved. Political leaders better fill their roles in providing oversight of technical officials (room for improvement remains), representing the interests of their electorate, monitoring development and service delivery activities in their constituencies, and formulating by-laws (outcome #9 listed in box 6). In the case of Kiboga district, for example, this led to increased sharing of information between technical staff and political leaders. Political leaders made timely budget and other approvals instead of stalling due to dissatisfaction over their allowances. Political leaders also played an increased role in urging their constituents to pay taxes (some of them had won their elections on an anti-tax platform). In Buliisa District, the improved relationship between political leaders and technical staff greatly reduced conflicts between the two sides. It also improved district operations by making political leaders more receptive to procedure. In Adjumani District, the good relations between technical and political leaders were cited as being instrumental in smoothing and shortening the planning process for FY 2018/19. In GAPP-supported districts, councillors are also now aware of critical policy regulations, such as the 20 percent limit on their allowances as set by revenue collected the previous year. Understanding this correlation has been particularly useful in diffusing the tendency by some councillors to agitate their constituencies against paying taxes. Instead of spending their time trying to pressure departmental staff, local councils are now more focused on their deliberations and agendas. This type of change was not noted in the low-performing counterfactual district visited. 3.1.2.3 Improved local government management Due to GAPP coaching, GAPP-supported districts are now more capable in executing key management functions. Initially, GAPP districts experienced challenges in planning and budgeting, financial management and accounting, internal audit, and local revenue mobilization. Of specific note was that the DPACs did not meet regularly and their investigation reports lacked substance. Over the course of the program, GAPP provided capacity development support in these areas which has improved their performance (see box 7). The program typically offered cluster training to staff or teams on a specific topic, such as local resource mobilization, guidance for the mid-term review for the current district development plan (DDP), or gender and equity budgeting. The trainings helped staff share experiences with others in the same position in different districts. Trainings sometimes included representatives from a variety of departments, if they shared critical roles in the process. Following each training, the ETS teams would follow-up through regular visits to the districts and municipalities. This customized follow-up was critical to helping staff adapt and establish the ideas, tools, and processes within each district and municipal system. As a result, planning and budgeting became more integrated and consultative (outcome #13 listed in Box 6). Some districts with significant development programs now use the improved Technical Planning Committees (TPCs) to integrate the work of implementing partners into their budgets and activity plans for more comprehensive planning. Local government financial statements are usually submitted on time, and internal audit and DPAC reports are more comprehensive in content and coverage. The informational basis of local revenue generation has expanded as business registers have been updated and new sources of revenue have been identified (outcome #12 listed in box 6). This is important, “Before 2013, council meetings took a whole day to discuss what needed to be done, debating what to debate, and reading documents. It was more like a workshop than a meeting”. - Comment from a Closeout Meeting “Before 2013, council meetings took a whole day to discuss what needed to be done, debating what to debate, and reading documents. It was more like a workshop than a meeting”. - Comment from a Closeout Meeting “Before 2013, council meetings took a whole day to discuss what needed to be done, debating what to debate, and reading documents. It was more like a workshop than a meeting”. - Comment from a Closeout Meeting “Before 2013, council meetings took a whole day to discuss what needed to be done, debating what to debate, and reading documents. It was more like a workshop than a meeting”. - Comment from a Closeout Meeting One first term district councilor earned the ranking of the 6th most effective councilor in the country in the ACODE Council Scorecard. He attributes much of his success to what he learned from GAPP. One first term district councillor earned the ranking of the 6th most effective councillor in the country in the ACODE Council Scorecard. He attributes much of his success to what he learned from GAPP. Box 7: Successful First Term Councilor Box 7: Successful First Term Councillor Box 7: Successful First Term Councillor Box 7: Successful First Term Councillor GAPP Performance Evaluation Report 13 December 2019 since sources of local revenue correlate highly with high performance in the LPGA. However, it is also important to note that new government policy on local revenue has recentralized of some key local revenue sources, e.g., taxi parks. In conjunction with these improvements, local governments are beginning to increase their reporting to citizens. GAPP trainings emphasized transparency to citizens on their achievements and officials are learning that citizens want to see actual outputs from allotted resources, not just a reporting of how the money was spent. Many GAPP districts have instituted mechanisms for communicating this information to the public (outcome #15 listed in Box 6). This is building citizen confidence in their government and increasing their willingness to pay taxes. 3.1.2.4 Accountability strengthened through broadened scope of internal audit and DPAC investigations. Local audit processes now work effectively and efficiently due to GAPP training and technical assistance. As seen above, once there is genuine oversight and accountability in the audit process, clean audits tend to increase. Given the linkage between a clean audit and high LGPA score demonstrated by the QCA, this is a significant development. Previously, internal audit and DPAC used to focus only on financial accountability (i.e. tracking expenditure and unaccounted for advances). They were not aware that all local government activities, including service delivery, must be investigated and scrutinized for accountability. Furthermore, DPAC reports were usually filed after being submitted to council or accounting officers instead of being tabled for discussion, thus preventing discussion and action on their findings, thereby negatively impacting service delivery. Through key informant interviews, document review and the cases, the evaluation team traced the steps in the process. (For the details of each step in the DPAC process map, see Annex 11.) Now local governments and national accountability institutions are integrated into a powerful accountability system. GAPP intervention helped broaden the scope of internal audit and the DPAC to cover service delivery. As a result, DPACs have become more assertive and follow up on cases of abuse of office and other malpractices, which were not investigated previously. DPACs now make more specific recommendations using standardized report formats (outcome #10 listed in Box 6). Furthermore, with help from GAPP, District Executive Committees (DECs) and councillors increased their understanding of how to read audit and DPAC reports, increasing follow up on recommendations. The local government accountability systems are comprised of internal audit, DPAC, council and local government technical officials headed by the district Accounting Officer. The national level accountability system is comprised of OAG, PPDA and the Public Accounts Committee on Local Government (PAC LG). Initially, the two systems were not connected, there was minimal information flow between them, and multi-year backlogs of local government internal audit and DPAC reports had accumulated without review by the national accountability institutions. There were no accountability consequences for mismanagement. GAPP helped integrate these two accountability systems into a broader system of accountability that shines a powerful torch on local governance and service delivery issues. One particularly effective GAPP intervention was the introduction of the Tripartite Outreach Model that helped open up the local government system to simultaneous scrutiny from above (i.e. OAG, PPDA, PAC LG, MoLG, and Ministry of Finance, Planning and Economic Development (MoFPED)); laterally at district level (i.e. council, internal audit, DPAC, accounting officers and heads of department, RDC, district security agencies, media, suppliers, etc.); and from citizens (i.e. CSOs, private sector organizations, etc.). Local government political and technical officials are now on “[DPAC meetings] are no longer a torture chamber”. - Key informant interview “[DPAC meetings] are no longer a torture chamber”. - Key informant interview “[DPAC meetings] are no longer a torture chamber”. - Key informant interview “[DPAC meetings] are no longer a torture chamber”. - Key informant interview In one case district, the director of a health facility had “borrowed” eight mattresses and they were not returned. The incident was discovered and promptly investigated. The staff member was found guilty and not only faced disciplinary action but was forced to pay for the procurement of eight new medical-standard mattress. In one case district, the director of a health facility had “borrowed” eight mattresses and Box 8. Accountability at Work in the Health Sector Box 8. Accountability at Work in the Health Sector Box 8. Accountability at Work in the Health Sector Box 8. Accountability at Work in the Health Sector GAPP Performance Evaluation Report 14 December 2019 Strong GAPP Contribution 17. Improvements in public service delivery (health, ed, ag, WASH) due to citizen advocacy 18. Citizens prepare and present petitions to local government regarding service delivery 19. PSAs successfully lobby for service delivery improvements Medium GAPP Contribution 20. CSOs attract funding from stronger organizational systems Potential GAPP Contribution 21. Citizens more willing to pay taxes *See Annex 12 for outcomes stories of all outcomes listed Strong GAPP Contribution 31. Improvements in public service delivery (health, ed, ag, WASH) due to citizen advocacy 32. Citizens prepare and present petitions to local government regarding service delivery 33. PSAs successfully lobby for service delivery improvements Medium GAPP Contribution 34. CSOs attract funding from stronger organizational systems Potential GAPP Contribution 35. Citizens more willing to pay taxes *See Annex 12 for outcomes stories of all outcomes listed Strong GAPP Contribution 36. Improvements in public service delivery (health, ed, ag, WASH) due to citizen advocacy Box 9: Citizen Level Outcomes Box 9: Citizen Level Outcomes Box 9: Citizen Level Outcomes Box 9: Citizen Level Outcomes notice that their actions on local governance and service delivery are under constant watch by all stakeholders. (See example in Box 8.) Now the two systems have been integrated into a more encompassing system that brings the national and local level institutions and units together into an open forum at the district level where issues raised by the local government internal audit and DPAC are discussed openly. This has positively influenced local government accountability. It also promoted increased adherence to regulations and procedures, while simultaneously reducing opportunities for discretionary behavior. This contributed to improved service delivery by broadening DPAC investigations beyond the traditional focus on financial accountability to cover service delivery as well. Service delivery issues raised in DPAC reports formed important points of reference during field inspections by OAG regional auditors, local government councillors, district technical staff, service providers and local citizens. 3.1.3 GAPP Contributions to District Citizen and Community Participation and Voice in Local Governance Public service delivery is meant to benefit citizens equitably in areas of health, agriculture, education, water and sanitation, roads, etc. Citizens play a key role in demanding accountability and responsiveness from the government for the quality of services they receive. Unfortunately, due to historical context, many citizens in Uganda are passive recipients of low-quality services. The absence of citizen participation increases the risk that improvements made in the other two components of the evaluation question may be lost without sustained feedback from citizens and the private sector. The evaluation found numerous “points of light” in citizen oversight for services as a result of the CSO initiative to mobilize communities to identify service delivery gaps and approach government for redress. (See Annex 12 for detailed outcome stories for outcomes listed in Box 9.) While the evaluation team does not negate the immediate positive impact these improvements have on people’s lives, these gains are less systemic and therefore more fragile in terms of representing sustainable change. The strategy for this component relied on the standard approach of offering a combination of grants, capacity assessments, training and tailored technical assistance in response to the assessment results (RTI, 2016). While common in development programs, this strategy has shown itself to be less effective than more systemic approaches that emphasize strengthening adaptive capacities over managerial networking, shared ownership, learning through experimentation, data-driven decision making, etc. (Root Change, 2014). With a different approach, the program may have seen significantly different results. 3.1.3.1 Increased citizen demand for improved local service delivery Having experienced the effectiveness of citizen advocacy through CSO efforts funded by GAPP, citizens are understanding their role and opportunity in demanding government accountability for providing quality services. Throughout implementation, GAPP dispersed US $7 million through 87 grants of 1-2 years duration to 56 grantees (CSOs, PSO, and local governments). With these grants, organizations mobilized stakeholders in identifying gaps in public services and advocating for corrective action from their governments. In addition to community sensitization and outreach, many grantees also offered trainings to government officials on their roles as duty bearers. Grant objectives touched 10 sectors, with health and education receiving the most attention. (See Table 2) GAPP Performance Evaluation Report 15 December 2019 Table 2: Distribution of GAPP Grants across Focus Areas Policy and Governance Education Health Agriculture Gender Refugees Private Sector Local Revenue National Organizations 7 1 1 1 1 1 District Organizations 4 28 26 9 3 2 10 3 Total 11 29 27 9 3 3 11 4 Note: Totals exceed the number of grants since many grants covered several areas As a result, there were many promising examples of citizen groups petitioning their local government for change (see outcome #18 listed in Box 9). Citizens engaged in radio talk shows, roundtables, and community dialogues such as barazas, Accountability Days, and TOMs. Grantees’ reports describe constituents developing petitions or position papers that were presented to their local governments (see box 10). While the evaluation team was unable to determine the specific number of lobbying activities, successful or unsuccessful, as referenced in the grant final reports, the success stories describe numerous instances of improved services (see outcome #17 listed in Box 9) in health clinic and school infrastructure; reduced staff absenteeism; better provision of agricultural inputs; more inclusive practices for women, youth, and persons with disabilities (PWD); and improved market infrastructure through solar lights, to name a few. (See Box 9.) GAPP provided support for mobilizing advocacy efforts but provided a far lesser degree of support on understanding how government works and the points where citizens can best exert influence. Several interviewees noted that momentum was just building when the project came to an end. Without resources, they were forced to turn away community members asking for support. Others reflected that the GAPP program focused on the “technicalities” of governance when what is needed is a “mindset change” in both citizens and government officials on their mutual governance roles. The fault-finding approach of petitioning government for redress where services are failing reinforces perceptions of NSAs as critics and disruptors of government, rather than development partners. They remained largely outside the governance systems of service delivery planning, budgeting, and monitoring, leading to little change in the relationship of NSAs to communities and to government. Evaluators found good examples of engagement in governance, such as writing petitions and presenting white papers. However, since these instances resulted from GAPP grants, evaluators did not find evidence of a sustainable change in the way citizens and communities participate in local governance without donor funding and initiatives by organizing these responses on their own without grant support. 3.1.3.2 Enhanced organizational management systems attract new donor funding Following GAPP organizational development support, CSOs secured new donor funding as a result of improvements they made to management systems (outcome #20 listed in Box 9). National and district CSOs and private sector associations (PSAs) are an important part of the governance process. Due to their familiarity with stakeholders and local conditions they are well positioned to represent communities and association members. However, their effectiveness and sustainability are threatened by the lack of consistent funding, particularly core funding. This is a persistent limitation for district-level CSOs (Uganda National NGO Network, 2018). To overcome these challenges, GAPP offered organizational capacity development support to 46 organizations (12 national and 34 district). Some, but not of all these organizations were GAPP grantees. Organizations were selected following a competitive application process in response to a call for applications from interested CSOs. Selected applicants received a capacity self-assessment using the Partner Local Organizational Capacity Assessment (PLOCA) tool. In response to the self-assessment One GAPP grantee stated that when his organization started working in one GAPP district, no petition had ever been presented by citizens. Citizens did not even know that they could hold their government Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery Box 10: Citizens Write Petitions to Improve Service Delivery GAPP Performance Evaluation Report 16 December 2019 Figure 7: Most Significant Aspects of GAPP results, organizations then received 4-5 days of follow-up GAPP support from consultants and program trainings. A concluding PLOCA assessment was used to measure progress. While four organizations dropped out due to unmet expectations of funding (GAPP, 2019), most that completed the cycle demonstrated an increase in PLOCA scores. While changes in the PLOCA scores cannot be used as a reliable measure of progress, 3 GAPP cites several specific instances where partners received new donor funding because of management improvements (GAPP, August 2019 and GAPP, November 2019). Several noted that the enhanced management systems that resulted from GAPP support increased donor confidence of the CSO’s ability to be accountable for funding. Unfortunately, this outcome cannot be externally validated beyond accounts from GAPP and the CSOs themselves. More rigorous monitoring data and outcome reports would be needed to validate changes in PLOCA scores and CSO anecdotes. In summary, outcomes under component 3 reflect positive instances of meeting immediate needs of organizations and communities in the short-term. Beyond individual successes, the results do not reflect a significant and sustainable change in the ability of CSOs and other non-state actors to improve voice and accountability on the systemic level reflected in the other two components. 3.2 Q2. RELEVANCE OF GAPP MANAGEMENT AND IMPLEMENTATION APPROACHES Program stakeholders were very positive about the program’s implementation and management approaches. They valued the training and technical assistance that they applied to strengthen internal management systems in their organizations and departments. For full survey results, see Annex 8. The GAPP program was unique in many ways, given its flexible management approach, reliance on systems thinking, and the introduction of several innovations. The completion of the GAPP program does not reflect an end to USAID’s commitment towards anti-corruption and decentralization in Uganda and there are several lessons learned. Therefore, it is appropriate to include an exploration of participant perceptions of the effectiveness of GAPP, what made it effective, and recommendations for similar programs in the future. Throughout the close-out meetings, participants expressed strong appreciation for the support they received from the GAPP program. To get a more detailed view of their perceptions, the evaluation team conducted a telephone survey of key program participants from the contact lists provided by program staff. Of the list of persons contacted, 120 respondents participated. (For a full analysis of the survey results, see Annex 8.) Interviewees were told their responses would remain confidential. Of the various types of support received from GAPP (training, technical assistance, publications, materials, financing, etc.), respondents overwhelmingly considered training and technical assistance to be the most significant. CSOs also considered funding support as a significant contribution. (See Figure 6) 3 As is typical with self-assessments, CSO partners were unfamiliar with the process and how results would be used. They tended to inflate their scores and hide shortcomings. These results require triangulation for a more accurate determination of increased capacity. Figure 6: Most Significant Type of GAPP Support GAPP Performance Evaluation Report 17 December 2019 When asked what aspect of the support made the trainings so significant (such as peer learning, contact with experts, materials, GAPP staff, etc.), technical content was of primary importance (76%) followed by useful tools (28%). (See Figure 7.) Participants found the most useful aspect of technical assistance and mentoring to be the technical information they received (54%) followed by problem solving support (39%) and receiving useful tools (33%). (Figure 8.) When asked how important GAPP was in improving their individual job performance, an overwhelming 82 percent said it was highly important, 15 percent said it was important, and only 3 percent said it was somewhat important. The role that GAPP played in the performance of respondents’ organizations or departments was slightly lower, but still impressive. 77 percent said it was highly important, 22 percent said it was important, and less than 1 percent said it was only somewhat important. The evaluation team wanted to know the many ways participants applied what they had gained from GAPP. Most (91%) use their new knowledge to improve systems and procedures. They also trained others (83%) and applied the new tools they received (86%). (Table 3) In response to what effects GAPP support had on their organization, the most common answer (81%) was better management systems followed by enhanced organizational reputation (37%). (Table 3) Finally, the survey asked about the helping and hindering factors in applying new knowledge, skills and tools in the workplace. Respondents said the most common helping factor was GAPP staff (53%) and the greatest obstacle was said to be money (43%). These results are interesting in that they reflect a very similar story across the districts and stakeholder groups. Participants most appreciated the knowledge they received from GAPP. They applied this new knowledge to improve management systems and these systems are showing positive effects through better management and enhanced reputations. There seems to be a great thirst for information on roles, responsibilities, policies, and procedures to help district professionals do their jobs more effectively. GAPP seems to have been unique in bringing this level of information to the districts. 3.2.1 Participant Recommendations During the closeout meetings, there were calls for GAPP to continue for fear of losing the progress that has been made. For example, many called attention to the new group of councillors that will need induction following the next election. Accompanying this sentiment was affirmation that GAPP had done a good job inducting past councillors. However, they were ready to assume responsibility for continued progress themselves. A few respondents suggested that implementation could be improved, specifically in more timely delivery of services (9%) and more follow-up (8%). Overall, survey respondents recommended that future programs continue existing activities (74%). When asked what additional support they suggest, the three most common Improved systems & procedures 90.8 Continued training others 88.3 Applied new tools 85.8 Developed network connections 71.7 Generated additional revenue 69.2 Started new programs & activities 53.3 Other 10.0 “Due to the elections that are going to be held, there will be new leaders coming. I would recommend continuing with similar programs like GAPP so that the new leaders get the technical knowledge in their offices” - Survey respondent The responsibility on us to make this country work! - Closeout Meeting Participant “Due to the elections that are going to be held, there will be new leaders coming. I would recommend continuing with similar programs like GAPP so that the new leaders get the technical knowledge in their offices” Figure 8: Most Useful Aspects of GAPP TA Table 3: Application of GAPP Support GAPP Performance Evaluation Report 18 December 2019 responses were material support (33%), capacity building (25%), and financial support (25%). More technical support was most frequently mentioned for capacity building, reinforcing the need for more information and resources. 3.2.2 Appropriateness of GAPP Approach As seen from the outcomes harvested under question one and the responses to the participant survey above, the evaluation team found that GAPP improved local government performance and accountability. GAPP made a difference in the management policies and practices of national accountability institutions, local governments and councils, and several CSOs. Their flexible, system-focused approach and reiterative technical assistance support were instrumental in leveraging the results. It must be noted, however, that this does not mean that the development goal of supporting decentralization for quality public service delivery is complete. The challenges with local government service delivery and citizen engagement is inherently political, not managerial. Issues of relationships, power dynamics and mindsets played large roles in the context. The assumptions underlying the program’s theory of change and the obstacles they posed underscore this point. Therefore, while GAPP contributed a necessary layer of an overall solution, many of the remaining layers are out of the scope and responsibility of a bi-lateral assistance program alone. The next step is to help the Government of Uganda integrate their gains and build upon them to benefit its citizens. Notwithstanding, the following are elements of the GAPP approach that were particularly effective. While participants found the knowledge, information and tools to be what was useful, the evaluation found that the way the information was customized, delivered, and reinforced was critical to success. Systems approach – USAID recognizes that building the capacity of a single actor or relationship is insufficient; the focus must be on the system in its entirety (USAID, 2014). A systemic approach looks at policies, practices, and resource flows but also goes further to focus on the relationships, power dynamics, and mindsets which keep a system in place (Kania, 2018). GAPP did not focus solely on the skills and processes of individual entities but looked at the system at large to find obstacles to its effectiveness. The best example is the success achieved in unblocking the audit review process, from its start in the local government to its conclusion in Parliament. While a systems-focused approach can take time, be difficult to measure, and be even more challenging to attribute, it offers the potential for greater effectiveness in complex situations such as Ugandan local governance. Adaptive and flexible programming – GAPP was flexible and adaptive in working with local institutions, largely because of the flexibility extended to them by the USAID mission (see box 11). The program adapted to political changes and responded strategically to requests for assistance from the Ministry of Local Government, the Northern Uganda Regional Forum, and the Ministry of Health. The combination of assessments and joint problem-solving helped partners understand system blockages and design long-term solutions to address the root causes. The ETS model allowed flexibility in providing on-demand coaching to partners, which was more effective than training or providing materials alone. Reiterative technical assistance – GAPP provided training and technical assistance to program participants and gave ongoing follow-up support for the new ideas and tools through the ETSs. As will be examined in the next section in detail, the ETS model facilitated greater uptake of technical and financial assistance than would have resulted from training alone. These specialists were responsive to participants, giving support where requested, building a trusting relationship first, and emphasizing local problem-solving rather than adherence to one specific way of doing things. This provided a reiterative support system that was particularly effective over time. Convening the whole, not only building capacity of the parts – Capacity development programs often focus on the capacity of individual people, departments, or organizations instead of the processes The reason GAPP was effective is that they enhanced the performance of existing mechanisms, rather than creating new ones. This built critical ownership for change. They approached [partners] as wanting to support their success, rather than being pushy. They were flexible and responsive, giving assistance where it was needed, rather than promoting what they were offering. - KII Interviewee The reason GAPP was effective is that they enhanced the performance of existing mechanisms, rather than creating new ones. This built critical ownership for change. They approached [partners] as wanting to support their success, rather than being pushy. They were flexible and responsive, giving assistance where it was needed, rather than promoting what they were offering. - KII Interviewee The reason GAPP was effective is that they enhanced the performance of existing mechanisms, rather than creating new ones. This built critical ownership for Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness Box 11: Reasons for GAPP Effectiveness GAPP Performance Evaluation Report 19 December 2019 and relationships that connect actors in the system. The TOM is one example of bringing the different parts of the accountability system together, rather than focusing on the performance of each part. Another is the joint training that GAPP conducted between the political and administrative staff in a district to help them learn their roles and strengthen collaboration. Additionally, support to national CSOs for research and lobbying local government institutions changed the relationship between the two, enhancing collaboration rather than contention. Traditional approach to NSA strengthening – The above points speak specifically to components 1 and 2 to understand the impressive results the program achieved there. Unfortunately, component 3 did not benefit from these programming innovations but relied on more traditional approaches to civil society strengthening. Given the shrinking space for civil society in Uganda today, it is vital to adopt programming strategies that reduce adversarial stances between CSOs and government, and citizen passivity in public oversight. Where there was integration, such as with the work of the national CSOs, systems and relationships showed signs of changing. The shifts in credibility and government collaboration experienced by the national CSOs engaged under component 1 point to the relevance and efficacy of this type of approach in Uganda today. The common strategy for civil society strengthening has been a 1-2-year project grant combined with an organizational capacity assessment and technical assistance based on the results. This strategy is best suited to projects focused on preparing organizations for direct donor funding. This programming does yield project results (e.g., latrines built, staffing levels changed), but does less to change relationships and establish CSOs as development partners to their constituencies, government, and the private sector. Instead, this approach reinforces the description of CSOs as “agents of Western governments”. NSAs are more effective when they understand how government works, how policies and decisions are made, and opportunities to influence the governance process. This could not be achieved through the short-term design of support given to district CSOs and PSOs. CSOs interviewed cited the short-term project funding as a large obstacle. It takes most of the year to establish relationships with constituents and then funding ends and they must turn people away who ask for support, undermining the relationships they established. National CSOs received a series of short-term grants that built off one another which enabled them to go beyond short-term deliverables. About half of the organizations that received a PLOCA assessment also received an advocacy grant. Organizational capacity assessments can be powerful interventions, as reflected by many PLOCA organizations, by building understanding of the systems and structures needed to be considered a mature organization, particularly from a donor perspective. This type of tool, however, has not been found to significantly affect CSO performance but is instead skewed toward readiness for donor compliance (Wetterberg, 2013). Assessment tools that introduce capacities like those recommended by the European Center for Development Policy Management (ECDPM) have been found to be more relevant to organizational performance and sustainability (Baser and Morgan, 2008). A more direct collaboration between grant funding and technical assistance, a longer-term funding engagement, and a focus on bringing CSOs together with other program institutions might have better contributed to the program outcomes. Replication of MoH assistance – As explained in more detail under Section 3.3 below, MOH TA is unlikely to be replicated widely to other GoU ministries, unless they are funded by existing or new donor programs. This is because such TA requires costly expertise and the pay scales of Ugandan civil service may be insufficient to attract such expertise. This has implications for sustainability of these initiatives, as explained below. There is often a good degree of capacity transfer from the experts to their civil servant counterparts, however specific expertise such as legal counsel and civil engineering poses challenges in this regard. A similar set up funded by DFID in Nigeria was placed in the Ministry of Finance, working across all federal ministries and agencies, and recorded substantial savings in a short period of time. Efficiency Unit, GAPP Performance Evaluation Report 20 December 2019 as it was called, had its Director and Advisor paid for by a DFID Nigeria facility. This could be an example to study for future programs. 4 3.3 Q3. VALUE FOR MONEY ASSESSMENT OF EMBEDDED TECHNICAL SPECIALISTS MODEL, TRIPARTITE OUTREACH MODEL, AND MINISTRY OF HEALTH ACCOUNTABILITY SUPPORT In response to evaluation question 3, the three following models were examined to the degree which they represent value for money (VfM): the Embedded Technical Specialist (ETS), Tripartite Outreach Model (TOM), and accountability support to the Ministry of Health (MoH). The evaluation team found GAPP systems in place to underpin delivery of value across the three models. (The systems and related findings are detailed in Annex 10.) Standard VfM sub-categories are derived from the VfM analysis methodologies from DFID and National Audit Office (UK) guidance (see Box 12). For economy, the evaluation team assessed key inputs and processes for their selection, program budget break-down showing key cost items, asset registration and disposal, external audit reports, staff size, and financial controls for grantees. Under efficiency, the team reviewed fund utilization, quality of results reporting, outreach of program interventions, and management of risk. For effectiveness, it discussed the high-level results delivered, costs of results (where possible), coherence of outputs, and sustainability. Under equity, it assessed gender, equity, and social inclusion (GESI) related reporting and GESI results by GAPP. The findings in these categories are applicable to the whole program to a large extent (exceptions where noted), as well as the three models that are in question, namely the tripartite model, ETS model, and MOH technical assistance. 3.3.1 Value For Money of the Three Models This section presents the VfM findings based on 4 VfM categories plus sustainability (4Es+S), identifying attributable best practice actions of the implementers and partners where applicable. It should be noted that GAPP started in late 2012 using USAID financial and programmatic mechanisms. DFID joined later in 2015 and even though it matched USAID funding at par, overall management of the program was left to USAID, with DFID taking the back seat. Many of the requirements that DFID put for programs, such as annual reviews, external auditing of local expenditure annually, and VfM reporting came later and were responded to efficiently by RTI and GAPP staff. The feedback on RTI as the implementing partner was very positive in this sense. RTI and GAPP teams have successfully responded to the requirements of two development partners in a timely and effective manner. It is important to note that GAPP was not set up with a VfM strategy or a framework from the beginning. This means that GAPP program design did not have an overall economic analysis or a set of VfM indicators to be monitored periodically. Similarly, DFID’s business case outlining DFID funding to GAPP did not feature an economic analysis.5 A VfM framework was not adopted during implementation and GAPP’s VfM reporting focused on examples identified at quarterly or annual reporting stages. These were often qualitative examples about cost savings achieved by GAPP teams. GAPP type of programming does not lend itself well to using credible cost benefit analyses, because the type of benefits it generates are not easy to monetize (better procurement, strengthened audit processes, enhanced community voice, etc.) and there are often significant attribution issues. Most often, DFID programs identify a set of VfM indicators similar to costed result indicators in order to be able to report on trends on value. Even then, governance programs pose significant challenges to VfM reporting. Additionally, it is not easy to compare one program’s results to another, as governance issues 4 https://devtracker.dfid.gov.uk/projects/GB-1-203922/documents 5 There were savings estimated of up to £90m from the SUGAR/TAF component of the DFID program, and these were estimated to arise from recovered public funds as a result of anti-corruption interventions. DFID Uganda currently funds a governance program, of which SUGAR and GAPP are two components. • Economy • Efficiency • Effectiveness • Equity • Economy • Efficiency • Effectiveness • Equity • Economy • Efficiency • Effectiveness • Equity • Economy • Efficiency • Effectiveness • Equity • Economy • Efficiency • Effectiveness • Equity • Economy • Efficiency • Effectiveness Box 12. Four DFID VfM Box 52. Four DFID VfM Box 62. Four DFID VfM Box 72. Four DFID VfM Box 82. Four DFID VfM Box 92. Four DFID VfM Box 102. Four DFID VfM Box 112. Four DFID VfM Box 122. Four DFID VfM Box 132. Four DFID VfM Box 142. Four DFID VfM Box 152. Four DFID VfM Box 162. Four DFID VfM GAPP Performance Evaluation Report 21 December 2019 can be very context specific. Value is a relative concept that makes sense in comparisons, and therefore governance interventions face this additional comparative challenge to claim value delivered. Value delivered from the three GAPP models as well as the VfM best practices were reviewed and compiled considering these challenges. The starting point of any VfM analysis is to compare costs and benefits incurred. In the absence of either, qualitative methods and other proxy assessments are used to understand and demonstrate value. These are explained in detail in Annex 10: VfM Detailed Analysis of the Three Models. The systems that underpin delivery of VfM were used uniformly across the GAPP interventions and components. The VfM assessment of the three models on 4Es+S are presented in summary below and in detail in Annex 10. 3.3.2 Tripartite Outreach Model (TOM): Economy: This model was led by technical staff from GAPP and at times used short term TA support. This activity falls under Component 1 work with NAIs and has also had inputs from ETSs. Total cost incurred by GAPP for the TOM meetings was US$ 257,875 (including indirect costs) spanning the program years.6 Efficiency: As for the cost of each meeting, following the cost sharing principle with the government where GAPP paid for venue and accommodation and GoU paid for allowances for attendees, it was reported that each meeting eventually cost a fraction of what it used to be at the outset.7 The model also evolved in a more efficient manner in terms of coverage, such as holding three separate committees for multiple LG units simultaneously in a one-day session. Further, the cost sharing principle provided value also because it indicated a stronger commitment from government counterparts. Effectiveness: The TOM facilitated the GoU to conduct full hearings on a public audit report, as required by law. It is not feasible to quantify the type of outcomes that these meetings have achieved. TOM can be credited with increasing citizen participation and empowerment and potentially strengthening audit processes, chiefly through helping LGs understand their tasks and responsibilities and facilitating closer engagement between the NAIs. However, at this stage these outcomes cannot be credibly monetized. Feedback during the evaluation mission indicated very positive contribution of TOM towards a well-integrated public accountability system and opening accountability discussions inclusive of all stakeholders. Therefore, this model should be maintained for future governance programming. Equity: There are no specific results on VfM Equity that can be reported on TOM. GESI considerations being systematically included in PPDA audits is relevant for work under Component 1, and not specific to TOM. Sustainability: What started as ‘tri-partite’ has in time become ‘multi-partite’. In addition to the triage of PAC-LG, OAG and PPDA, participation has grown to include MPs, civil society, private sector organizations, and media (chiefly radio). In terms of what remains to be achieved for TOM going forward, the GAPP team is of the opinion that service delivery ministries such as health, education, and agriculture can be brought on board. Currently only organizations with ‘watchdog’ mandates take part in the meetings. Considering the balance of 4 Es, it is likely that the tripartite outreach model represents good value for money. The costs incurred are judged to be reasonable and modest. As the evaluation team was unable to ascertain monetary benefits, it used the qualitative evidence generated by GAPP and key feedback during evaluation interviews. These sources indicate change in the positive direction. If the model is sustained beyond GAPP, the value delivered will be more evident, and therefore should be considered as part of future governance programs. 6 The meetings costs have not fluctuated widely over the years, except for a sharp drop in 2016 (a tenth of the cost of other years). The reason for the drop in 2016 was because it was an election year and GAPP scaled back on that activity to avoid meetings to be used for political mileage. The cost was highest for 2018 with over US$66,000. 7 The estimated cost from the Component 1 Technical Lead was UGX 16m ($4,370) for one local government hearing using the tripartite model; following the cost sharing principle this came down to about UGX 1.5m ($410). GAPP Performance Evaluation Report 22 December 2019 3.3.3 Technical Assistance to MOH: This TA activity was implemented as a response to a request from MOH in 2018. Following reports of misappropriation of grant funds from Global Fund and the Vaccine Alliance GAVI, as well as recognizing the need to improve procurement and scrutinize infrastructure development, MOH asked for USAID assistance to address these challenges. Since GAPP has been working on addressing public accountability and governance issues, it was deemed a suitable program to host the proposed TA. Economy: A total of 8 experts were recruited and seconded to MOH, to be supervised by the Permanent Secretary. The team comprised of a legal advisor, civil engineers and a mechanical engineer, and a public finance management specialist in the Planning Department of MOH. At the time of the evaluation review, $ 1,161,837 was spent on the TA according to GAPP reporting.8 Efficiency: In terms of speed of delivery, the unit of experts functioned efficiently, delivering the savings expected and recording a significant savings of UGX 16bn (US$4.36m) in a little over a year. The cost savings originated mainly but not exclusively from the recurrent expenditure budget items of MOH. It was reported that among the experts engaged, the legal advisor and the engineers were the first of their cadres to fulfill those tasks at MOH. It was also reported by the experts that this unit was the first of its kind to be tasked with efficiency savings and other quality checks among the Ugandan line ministries. In this sense, the model represents an innovation. The savings achieved originated from various sources. For instance, through a review of MOH contracts and obligations and improving contract management at the Ministry, MOH saved close to UGX 8bn (> US$ 2m) in avoided fines and out of court settlements. There is now a delegated Contracts Committee in place, which is functioning and there has been no indication of it soliciting bribes from suppliers.9 The civil engineers worked on preparing bid documents and bills of quantities for various infrastructure items for MOH. By carefully reviewing bills of work, they were able to expand the scope of works. For example, They carried out physical upgrades of 308 health facilities rather than the 285 originally planned. Similar savings were achieved through better enforcement of maintenance clauses in contracts, such as for solar installations. Effectiveness: This activity has provided significant returns in the form of public efficiency savings, in a relatively short period of time (see box 13). The current rate of return (benefit to cost ratio) is 3.7510 undiscounted.11 For benefits, the team used public cost savings, with one notable caveat. Ultimately, saving a public agency public funds would signify a benefit to citizens when those savings are used by public authorities for improving service delivery. The team could not follow this chain of causality for the MOH activity. However, it is likely that, for example, servicing of a larger number of health centers will ultimately benefit Ugandans using those facilities and help improve the health outcomes in catchment areas in the future. To put the UGX 16bn of cost savings in perspective, FY2019/20 national budget MOH allocation was studied. This amount corresponds to combined budget allocations of Uganda AIDS Commission 8 This reportedly included indirect expenses. It excludes CLA related costs, which were not assigned to program components. 9Having functioning Contracts Committees by each procuring public agency is one of the key recommendations of the Comprehensive Procurement Assessment Review (CPAR) of World Bank in Uganda. 10Using the savings reported now (UGX 16bn) as the benefit and total GAPP costs on MOH as cost. However, this calculation leaves out the GoU costs which are not known, and a share of CLA costs from GAPP. Estimated CLA costs for MOH TA would be small in this instance. 11Undiscounted because it has only been over a year in terms of benefit duration. For longer periods of time, costs and benefits need to be discounted, most likely at 10% for this context. The unit of MOH experts engaged by GAPP functioned efficiently, delivering savings to the magnitude of around UGX 16bn (US$4.36m) in a little over a year. The cost savings originated mainly but not exclusively from the recurrent expenditure budget items of MOH. It was reported that among the experts engaged, the legal advisor and the engineers were the first of their cadres to fulfil those tasks at MOH. It was also the first unit to be tasked with efficiency savings among the line ministries in Uganda. In this sense, the model represents an innovation. To put the UGX 16bn of cost savings in perspective, the MOH allocation in FY2019/20 national budget was studied. This amount corresponds to combined budget allocations of Uganda AIDS Commission and Kabale Referral Hospital for that year, which is remarkable. The unit of MOH experts engaged by GAPP functioned efficiently, delivering savings to the magnitude of around UGX 16bn (US$4.36m) in a little over a year. The cost savings originated mainly but not exclusively from the recurrent expenditure budget items of MOH. It was reported that among the experts engaged, the legal advisor and the engineers were the first of their cadres to fulfil those tasks at MOH. It was also the Box 13. Value for Money Savings at the Ministry of Health Box 203. Value for Money Savings at the Ministry of Health Box 213. Value for Money Savings at the Ministry of Health Box 223. Value for Money Savings at the Ministry of Health Box 233. Value for Money Savings at the Ministry of Health Box 243. Value for Money Savings at the Ministry of Health Box 253. Value for Money Savings at the Ministry of Health GAPP Performance Evaluation Report 23 December 2019 and Kabale Referral Hospital12 for that year, which is very significant. One additional benefit is extending the benefits of other development partner programs in Uganda. Upgrading of rural health facilities mentioned above is under World Bank’s Intra-Government Fiscal Transfers Program with total funding of US$200m. MOH experts help the program deliver more by expanding the scope of upgrading works for health centers, for the same level of maintenance funds. Equity: In terms of organizational gender equity, two members of the MOH team are female. In terms of outreach, the evaluation team maintains that VfM equity principles are not well-suited to apply to this indirect delivery model. However, scrutinizing procurement processes affirmatively through a gender, disability, or other focus at MOH is something the group of experts can promote going forward. Sustainability: It is understood from the discussions that there is potential to achieve a lot more savings at MOH using the established systems such as the Contracts Committee, better-quality procurement, and contracts enforcement. One civil engineer estimated that they were able to save the MOH up to 50 percent of the infrastructure cost per unit at times. It is unclear how well these outcomes can be sustained beyond the life of the engagement of this small group of experts. The experts often collaborate with their civil servant counterparts and it is likely that there is a good level of knowledge and capacity transfer. However, motivation through a much higher pay scale is potentially something that MOH will not be able to provide for their cadres for the time being. In conclusion, MOH TA most likely represented good returns for the investment by USAID and DFID, with evidence of attribution supported by feedback and analysis of savings. 3.3.4 Embedded Technical Specialists (ETSs): The change to the model of provision of long-term TA support to LGs in GAPP districts, rather than short term and ad hoc TA support as was originally planned, has arguably made a very significant contribution to the achievements of GAPP. Economy: There were 15 ETSs working under GAPP, three for each region, and one representing each category of TA theme. The exact costs incurred for the ETS model were not available (and that would have been a highly difficult task for GAPP teams to do retrospectively). However, it is estimated to be a large proportion of the component 2 costs provided by GAPP finance teams, which was close to US$ 12.4m.13 It is evident, however, that costs of ETSs were much lower than the alternative option of providing local short-term TA. 14 Efficiency: The ETS teams based in regions, rather than in each district, meant that each ETS covered 6- 7 districts per region, signifying a large workload. This also improved efficiency because one ETS served many districts, including lower LG structures. The outreach is shown in Table 4. A total of 7,902 people were reached through various modes of capacity transfers under the ETS model, according to GAPP reporting. ‘Cost of outreach (capacity building per beneficiary)’ can be calculated if total costs of ETS are known. Going forward, in the next phase of governance programming, this metric could be considered as an efficiency indicator. Another outreach metric that can be calculated is ‘program investment in each district’, or ‘cost of GAPP support in each GAPP district’. Using component 2 total costs at around US$12.3m and 40 districts reached, around US$310k was spent on accountability improvements in districts. This is an average, and some districts will have incurred a higher, or a lower 12https://budget.go.ug 13 As at end August 2019. 14 There was no international TA for ETS positions, however even engaging local and national expertise on a short-term basis would have been more costly than retaining the experts at a long-term basis. The short-term fee rate bands used by GAPP can give a good idea about this comparison. Local Government Structure GAPP Outreach Districts 40 Municipalities 14 Sub-counties 317 Divisions 41 Town-Councils 49 Higher Local Gov Leaders 2,865 Lower Local Gov Leaders 5,037 Local Government Structure GAPP Outreach Districts 40 Municipalities 14 Sub-Divisions 317 Divisions 41 Town-Councils 49 Higher Local Gov Leaders 2,865 Lower Local Gov Leaders 5,037 Local Government Structure GAPP Outreach Districts 40 Municipalities 14 Sub-Divisions 317 Table 4: GAPP Outreach as of September 2019 GAPP Performance Evaluation Report 24 December 2019 cost. It would however give an indication of the level of benefits needed to break even per district level investment. Effectiveness: ETS’s work under component 2 contributed to the majority of GAPP’s results in that area. Not many of these results lend themselves to monetization for a VfM analysis. The calculation challenges notwithstanding, ETS model has been instrumental in delivering the summarized results. These were explained also under the outcome harvesting results, and for evaluation question two above, namely: ● Improved public accountability systems through improved audit processes; ● High level of attribution to ETS support for systems improvement; ● Improved revenue collections systems and better local revenue mobilization in several districts; ● Better coordination of refugee response; ● Reduced conflict between elected and technical staff. Equity: In terms of organizational gender equity, one ETS out of 15 was female. For equitable outreach to women, it was reported by GAPP that 356 local women officials were trained in leadership. In addition, support to GAPP districts by ETSs on refugee response coordination and planning can be considered under equity performance, as this dimension of VfM is concerned with all vulnerable groups, not only women. Sustainability: The ETS model is credited with generating a successful legacy for GAPP and therefore representing potentially good value for money for the original investment by USAID and DFID, especially if the outcomes are sustained going forward. The ETS placed by GAPP in MoLG is likely to be an important pathway to sustainability because he will work with this Ministry who is going to lead the local government sector. In the refugee hosting districts, the ETS support will continue during the Extension phase and it is recommended that a new governance program carries on with this modality of support, with the districts to be chosen considering the success criteria explained above in section 2.2. In conclusion, the evaluation team found that the three models examined generated potentially good value. The ETS model can be credited with generating a successful legacy for GAPP and therefore representing potentially good value for money for the original investment by USAID and DFID. For the MOH TA, over the past year, public efficiency savings were well over and above the costs of that TA with apparent potential to deliver more, therefore indicating good value for money. Finally, on the TOM, while it was not possible to quantify the outcomes and limited information is available on the costs, based on key feedback they can be credited to a considerable degree with improving public accountability and citizen participation. All three models represent examples for innovation in public accountability work in Uganda. Several other VfM highlights were noted on local leadership, adaptive and flexible programming, and relationship building. GAPP Performance Evaluation Report 25 December 2019 4 CONCLUSIONS Having examined the contributions of the GAPP program towards local governance, accountability, and participation, as well as stakeholder perception of value for money, the evaluation team reflected on findings to develop “lessons learned” and a set of recommendations for consideration in future programming: 4.1 LESSONS LEARNED There are many lessons to be learned from the GAPP program implementation, particularly regarding the systems strengthening approach to address complex change. Some of the lessons learned from GAPP approaches are as follows: Relationship-building is just as important as capacity building - Much of the success underpinning component 1 came from changing relationships among key institutions, such as the OAG, PPDA, and DPACs, through the tripartite model. Relationship-building also contributed to the effectiveness of component 2. ETSs worked to cultivate fruitful working relationships between technical staff and politicians in districts. Staff developed personal ties with their LG counterparts to effectively perform coaching roles. A lack of focus on changing current relationships between CSOs, PSO, communities, and government officials at the district level may have hindered similar breakthroughs in component 3. Incorporating citizen organizations into governance planning and decision-making could mitigate adversarial relations between government and NSAs and help grow the relationship to that of development partner. (The change in relationships between ministries and national CSOs resulting from GAPP programming illustrate the efficacy of this strategy.) This would require going beyond community needs assessments, which only raises expectations, to engaging them in prioritization and decision￾making within the budgeting cycle, which builds understanding. Local leadership is critical - As demonstrated from the QCA, strong local leadership can overcome deficits in context conditions that might otherwise limit local government performance. This requires systems leadership that can set high standards for government responsiveness to citizens and identify the pieces of the service delivery system. GAPP invested time building trust and confidence with local leaders. Putting local institutions in the lead, GAPP built ownership for change. Another source of local leadership came from the staff. Ugandan nationals comprise 65 of 66 GAPP staff.15 Undoubtedly many sensitive issues were more readily understood and addressed by local staff who understand the local cultural context. Feedback, particularly when made public, can motivate behavior change - While GAPP did not introduce the Local Governance Performance Assessment and the Local Government Councils Scorecard Assessment, the program funded its use in four GAPP-supported districts and made use of their results in the 40 target districts. These publicized annual external assessment tools help benchmark district performance against expectations and between each other. District officials were aware of their scores and the reasons behind successes or shortcomings, thus increasing motivation to make best use of the capacity strengthening support offered by GAPP. Each spoke of holding a leadership debriefing after the release of assessments to make plans for raising the score in the next year. Conversely, the PLOCA-supported organizations did not benefit from external feedback from stakeholders (e.g., constituents, donors, government officials). The program used only self-assessments. While self-assessments are powerful for internal learning, they do not replace external feedback and measurement tools for gaging organizational performance. Citizen-led engagement can significantly contribute to government responsiveness - The program demonstrated that when citizens are engaged, government responds, particularly at the sub￾county level. Some responses resulted from government outreach to citizens, and some from CSOs and 15 Last count in July 2019, and before recent staff attrition, which is routine for programs nearing their end. GAPP Performance Evaluation Report 26 December 2019 PSOs lobbying with program grants. Reporting results achieved from budget expenditures was new for district governments, as lobbying was a departure from the protests and critiques that have typically been used by NSAs to voice their views. GAPP demonstrated that both local governments and CSOs have a role to play in informing and engaging citizens. Role of research as a basis for lobbying - GAPP national CSO partners received funding to do research on critical issues affecting decentralization, such as funding. Findings were widely publicized. This demonstrated a depth of knowledge that lent credibility and objectivity to subsequent advocacy efforts. Donor coordination and collaboration leads to good value and good practice - USAID and DFID working together on GAPP generated time and other cost savings for each partner, thereby improving economy. It has mostly likely reduced efforts by Government of Uganda counterparts, both national and local. This cooperation has arguably extended each development partner’s influence by broadcasting the right messages on donor coordination and effectiveness. Planning sustainability pathways is critical to the outcomes achieved thus far - Sustainability is a subset of VfM Effectiveness, and for many development practitioners it is seen as an additional pillar to 4Es. For example, in designing pathways for sustainability in its last year, GAPP saw that MoLG was emerging as an important actor whose influence needed to be firmly established. An ETS who was working in Acholi was placed in the Ministry, recognizing the program work achieved with LGs by the ETSs, so that MoLG capacity could be strengthened. MoLG’s profile as a leader in ‘local government sector’16 can benefit from GAPP experience and expertise. 4.2 RECOMMENDATIONS Founded upon these lessons, the evaluation team makes the following recommendations for USAID, DFID, and other donors to support LG accountability and performance and citizen participation going forward: 1. Scale-up programming innovations - The ETS and TOM were critical to the success of other program activities. The key to their sustainability is to institutionalize these innovations within Ugandan ministries. Future programs should enhance MoLG’s District Resource Pools to more closely resemble the ETS model. Additionally, future programs may need to engage retired public servants or members of the private sector for support. Lastly, it is advisable to recognize LG or LC members for their excellence and dispatch them to support struggling sub-counties in their district. 2. Expand engagement on service delivery issues to other ministries - GAPP support focused on the three most direct accountability actors in the audit cycle. In the course of project implementation, it became clear that other functionaries who were not directly involved played key roles, e.g., the MoLG, MoFPED, Ministry of Public Service (MoPS), and the National Planning Authority. The role of the MoLG is particularly important to institutionalize the gains of GAPP, such as in-depth induction training for new councillors, tripartite outreach meetings, and coaching assistance through the district resource pool. Lastly, future programs should engage these additional ministries as their interest and resources permit. 3. Invest in leadership, not leaders - GAPP considerably invested in the leadership development of senior leadership of LGs and LCs. However, a focus on systems leadership, not individual leaders, might be more appropriate. Systems leadership works laterally across stakeholder groups to catalyze and empower collective action. Systems leadership use skills like convening, communicating, and metrics monitoring for service delivery (Dreier, 2019). Leadership development with a systems focus would bring together people from within all sectors of society (government, private sector, and civil society) to work together to find solutions to problems that exist in their current context. 16This is a new sector formed by the Cabinet, which will work alongside the 16 other sectors in Uganda, such as health, education, water, agriculture, etc. It will bring together key government departments and development partners working in local government and decentralization. GAPP Performance Evaluation Report 27 December 2019 4. Strengthen citizen participation in planning - This has also been recommended elsewhere, (Uganda National NGO Network, 2018, Kibombo, November 2015). For citizens to shift from passive recipients of public services to being active in oversight, they must be involved in planning and decision-making for services from the start. Guidelines state that Uganda has adopted “bottom-up” planning, yet there is little evidence of this happening beyond narrow consultation. Without vigilance, planning becomes a largely technocratic process. Future assistance should engage the NPA to make the planning, monitoring, and process truly participatory in the way that GAPP was able to introduce true accountability into the audit review process. Some model districts are working this way (see Box 14) and ACODE has its Community Engagement Action Plan (CEAP) model (ACODE, 2017). 5. Strengthen councillors in their role of constituent engagement/representation - GAPP gave significant attention to the role and procedures of councillors in passing policy and giving oversight to accountability processes. The role of councillors in constituent engagement and representation was given less attention. While covered in the workshop material, it did not receive the same application follow-up to ensure that councillors learn how and when to engage constituents. Citizens are not completely versed in the role of the council and councillors and instead expect financial support from representatives. As a result, councillors avoid constituents, contributing to being unseated in the next election. Strengthening this aspect of councillors’ work could help reverse the election turnover and improve citizen participation in local governance. The role of the councillors for special constituencies (women, youth, elderly, PWD) has opportunity for improvement. These councillors tend to underperform their counterparts (ACODE, 2017), which undercuts their potential to bring about palpable change in their stakeholder group. 6. Apply systems strengthening strategies to NSA programming - The next round of civil society capacity building can focus and be programmed in a way that overcomes the challenges for the sector. Increase emphasis on systems programming in this sector and incorporating this sector as part of the governance whole. Leverage development innovations from within USAID and lessons from other missions to experiment with different development programs that will engage citizens for “mindset change” such as Capacity 2.0, Local Works, and Problem Driven Iterative Adaptation (PDIA). Select capacity building strategies and frameworks that strengthen the effectiveness and sustainability of organizations working together through formal or informal networks, not simply individual organizations. Facilitate change in relations and credibility with government, citizens, and the private sector. Encourage national networks to engage government in consultation, present position papers, and form coalitions. Build the understanding in CSOs/PSOs of how government works with cycles, budgets, roles, etc. 7. Expand stakeholder feedback mechanisms in programming - There are several powerful citizen feedback tools that influence government behavior, e.g., community scorecards and social One Ugandan district develops its 5-year development plan with the input of citizens. First, they analyze the past plan to see what they did and did not do. LC1 leaders then mobilize their communities for consultations to identify areas of need (education, water, health, roads, agriculture, etc.) Private sector organizations may present a position paper outlining their needs. Together they identify what they can do for themselves and what the district government can afford to do. Out of 10 priorities, perhaps they can do 6 themselves. For the remaining, they use pair-wise ranking to determine what will benefit them the most. If roads need repair, which roads are the priority? Should they build schools or rehabilitate old ones? Meeting outcomes and attendance lists are documented and sent to the parish for aggregation by the Parish Development Committees. The process continues to the sub-counties and is repeated. District development plans are then based on sub-county plans. Once the development plan is finalized, the district holds an annual public budget conference with stakeholders from the parish and sub-counties. To prepare, they also meet with development partners in the district to see how their programs are contributing to district needs to incorporate into decision-making. At the public meeting, the LG presents what they accomplished the prior year, what they did not accomplish and why. Together, attendees decide their funding priorities for the resources they have. Box 14. Example of Participatory District Planning Box 35. Example of Participatory District Planning Box 36. Example of Participatory District Planning Box 37. Example of Participatory District Planning Box 38. Example of Participatory District Planning Box 39. Example of Participatory District Planning Box 40. Example of Participatory District Planning Box 41. Example of Participatory District Planning Box 42. Example of Participatory District Planning Box 43. Example of Participatory District Planning Box 44. Example of Participatory District Planning Box 45. Example of Participatory District Planning Box 46. Example of Participatory District Planning Box 47. Example of Participatory District Planning Box 48. Example of Participatory District Planning Box 49. Example of Participatory District Planning GAPP Performance Evaluation Report 28 December 2019 audits. These feedback instruments can be incorporated into government planning processes at the start of the DDP planning cycle and at mid-term review to provide citizen perspective on service quality. Support and build upon the Ugandan Quality Assurance Certification Mechanism (QuAM) NGO certification process, rather than outside self-assessments for scoring. Self-assessment tools can be powerful interventions for learning but are not by themselves appropriate for M&E scores or for public benchmarking. Linking CSO capacity benchmarks to QuAM would motivate CSOs to meet these external standards and earn certification, which would build credibility and transparency within the civil society sector. 8. Design grant making to strengthen multi-stakeholder linkages and build relationships - CSO grant programs typically start with a call for proposals for projects, which reinforces traditional donor-grantee relationships and does not facilitate new relationships between CSOs, citizens, or government. Consider cluster approaches that use Collective Impact, (i.e., SACE advocacy project in Nigeria). Or solicit program ideas following a ‘whole system’ design event, such as Future Search or Appreciative Inquiry. Plan activities for longer than 1-2 years and provide ETS support to help stakeholders change the way they relate to one another. 9. Integrate citizen engagement and oversight into sectoral programs and vice versa - In Uganda, common structures exist in service delivery facilities. However, they are often weak or dysfunctional. Donors could strengthen the understanding of citizen oversight for public services by integrating the use of these committees as part of sectoral programs whenever appropriate. Similarly, incorporate actual service delivery challenges and district sectoral development indicators into LG performance management support. 10. Maintain a central ‘VfM Register’ for cost savings to be systematically recorded - GAPP’s VfM reporting included examples of savings, but often without monetary values. As numbers are critical to this type of reporting, future programs should maintain a spreadsheet of savings with monetized values. Some periodic VfM metrics that could be used in future programs are explained in Annex 10. 11. Use VfM Case Studies in reporting - Use VfM Case Studies for programs where results are hard to monetize. In a case study, costs and benefits from an activity or intervention are isolated for focus. (A template is provided in Annex 10.) For example, my Tax pilot in Masindi or Buliisa revenue assessment (GAPP, April 2019) could potentially lend themselves to good VfM case studies. 12. Track revenue increases and savings through improved procurement as two substantial ways to monetize benefits - Improvements in national surveys and indices as in LGPA are also useful, however the changes in those scores are not possible to translate into monetary benefits. Therefore, future programming should monitor local revenue increases and procurement savings (through better audit processes) to help with the development of more robust VfM narratives. 29 | GAPP EVALUATION REPORT ANNEXES USAID.GOV GAPP EVALUATION REPORT ANNEXES List of Annexes ANNEX 1: EVALUATION SCHEDULE......................................................................................................................30 ANNEX 2: EVALUATION SCOPE OF WORK.......................................................................................................32 ANNEX 3: LIST OF DOCUMENTS REVIEWED.....................................................................................................40 ANNEX 4: LIST OF STAKEHOLDERS CONSULTED...........................................................................................47 ANNEX 5: KEY INFORMANT INTERVIEW GUIDE..............................................................................................51 ANNEX 6: FOCUS GROUP DISCUSSION PROTOCOL.....................................................................................53 ANNEX 7: STAKEHOLDER SURVEY TOOL...........................................................................................................54 ANNEX 8: ANALYSIS OF STAKEHOLDER SURVEY............................................................................................61 ANNEX 9: QUALITATIVE COMPARATIVE ANALYSIS DESCRIPTION .........................................................68 ANNEX 10: VALUE FOR MONEY DETAILED ANALYSIS FOR THE THREE MODELS .............................70 ANNEX 11: PROCESS MAPS........................................................................................................................................84 ANNEX 12: OUTCOME SUMMARIES.......................................................................................................................87 30 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 1: EVALUATION SCHEDULE Kampala Based Work Date Complete Weeks Sept Oct Nov 1 2 3 4 5 6 7 8 9 10 11 12 Evaluation Team Lead Mobilized (Team Lead, CS Expert, Methods Expert) 2-Sep Hoima Closeout Event 2-Sep Mukono Closeout Event 5-Sep Desk Review & Methodology Approach 6-Sep In-briefing 6-Sep GAPP Kampala workshops and interviews - data collection 13-Sep Preliminary data analysis 13-Sep Inception Report 16-Sep Phone Based Survey - data collection 3-Oct VfM Kampala based interviews - data collection 4-Oct National level workshops and interviews - data collection 11-Oct Field Work Field work District 1 - data collection 20-Sep Field work District 2 - data collection 4-Oct Field work District 3 - data collection 11-Oct Field work District 4 - data collection 11-Oct Field work District 5 - data collection 18-Oct Field work District 6 - data collection 18-Oct Analysis and Reporting Data aggregation and analysis 25-Oct Presentation of Preliminary Findings 25-Oct Evaluation Report Drafting 8-Nov Stakeholder Debrief 12-Nov 31 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Final Draft Evaluation Report TBD Final Evaluation Report TBD Synthesis Abstract TBD Cleaned and Labeled Data TBD CASE STUDY FIELD VISITS Week Dates in District Date of Travel LG Field team To District To Kampala Week 4 September 26 – 27, 2019 September 25, 2019 September 27, 2019 Apac Municipality ● John Kiyaga ● Wolfgang Stuppert Week 5 September 30 – October 1, 2019 September 29, 2019 October 1, 2019 Kapchorwa Municipality ● John Kiyaga ● Wolfgang Stuppert Week 7 October 15 – 16, 2019 October 15, 2019 October 16, 2019 Buliisa District ● Meg Kinghorn ● John Kiyaga Week 7 October 17 -18, 2019 October 17, 2019 October 19, 2019 Mpigi District ● Meg Kinghorn ● Laura Niyirinkindi Week7 October 17 -18, 2019 October 16, 2019 October 19, 2019 Adjumani District ● John Kiyaga ● Wolfgang Stuppert 32 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 2: EVALUATION SCOPE OF WORK Performance Evaluation of the joint USAID/Uganda and DFID Uganda’s Governance Accountability, Participation and Performance Program (GAPP) Statement of Work I. BACKGROUND Decentralization in Uganda has been described by some as a cornerstone of democratic governance, yet for others it failed to deliver on the promise of effective democratic governance, improved service delivery, increased accountability and citizen engagement at the local level. USAID/Uganda and DFID/Uganda support a seven-year (2012-2019), USD 35.5 Million Governance, Accountability, Participation and Performance (GAPP) activity, implemented by RTI International with its consortium partners Development Alternatives Inc. (DAI) and Palladium17. The goal of GAPP is to increase participation, accountability and local governance for effective service delivery. The objectives of the activity include, a) to enhance accountability within local government institutions; b) to strengthen the Government of Uganda (GOU) accountability institutions; c) to improve local government governance; d) to support civil society organizations (CSOs) and other non-state actors. II. INTRODUCTION In November 2012 USAID/Uganda, awarded a USD 17.7 million contract to RTI International with its consortium partners Development Alternatives Inc. (DAI) and Palladium, to implement the GAPP activity in 25 districts18. The scope of work was expanded through a partnership with DFID which doubled funding, expanded technical breadth to a range of complementary work areas; deepened geographic coverage in existing districts with greater coverage of sub-counties, town councils and municipalities; and expansion to ten additional districts19 in northern Uganda and the Bunyoro sub-region. In addition, the expansion was aimed at enhancing the enabling environment to improve the delivery of health, education and economic growth in target districts from the current 25 to 35 districts. By the close of FY 2017, GAPP covered 39 districts with four additional district local governments (DLGs)—Omoro (carved out of Gulu in 2016), Kagadi and Kakumiro (carved out of Kibaale in 2016), and Pakwach (carved out of Nebbi in 2017). The GAPP’s development hypothesis (theory of change) is “if Government of Uganda (GOU) and nongovernment-led efforts are supported to improve democratic governance and accountability, and citizens’ efforts are mobilized to engage and partner with LGs, then service delivery will be more equitable and efficient”. The GAPP activity assumes that closer government, increased citizen monitoring and transparency are expected to characterize decentralization and should thus be associated with a move toward increased accountability and improved service delivery. Supporting the local governments should strengthen government accountability to citizens by involving citizens in monitoring government performance and demanding corrective actions. The premise is that when accountability is strengthened and improved, the opportunity for corruption to take place is diminished and governance outcomes in health, education and agricultural systems such as responsiveness, equity and efficient use of resources are affected positively. The GAPP is implemented through three major interlinked intervention components: 17 Formerly known as Development and Training Services (dTS) 18 Agago, Amolatar, Amuru, Apac, Kitgum, Gulu, Lamwo, Lira, Nebbi, Nwoya, Oyam, Pader, Koboko, Arua, Buliisa, Hoima, Masindi, Kibaale, Kalungu, Kayunga, Kiboga, Masaka, Mityana, Mubende, Mukono 19 Adjumani, Moyo, Yumbe, Maracha, Zombo, Kole, Otuke, Alebtong, Dokolo, Kiryandongo 33 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 1. Supporting and strengthening national-level government accountability institutions in their oversight and interaction with LGs; 2. Strengthening LG revenue, procurement, financial, and planning systems; 3. Strengthening civil society organizations (CSOs) and other non-state actors (NSAs) to improve voice and accountability. Linked to the above three result areas, GAPP aims at accomplishing the following results: 1. Improved legal, policy, regulatory and institutional environment to meet demands for more democratic governance. 2. Improved fiscal management and accountability processes in the delivery of services in targeted districts. 3. Improved capacity of citizens and communities to participate in local governance. During the implementation, the activity evolved and some areas changed due to the shifting environment, additions were also made by USAID and DFID to make the activity more relevant to changing times. In August 2017, GAPP’s work plan was amended to include substantive refugee-focused activities, Fiscal Transparency Initiative activities, the Northern Uganda Regional Forum activities, as well as Accountability support to the Ministry of Health. The additions are attached as part of the documents for review. III. PURPOSE OF THE EVALUATION The purpose of the final performance evaluation of GAPP is to enable the Government of Uganda, USAID/Uganda and DFID obtain insights and learn about the Activity interventions and approaches that were most effective, with a view to informing the implementation of newly designed anti-corruption, accountability and governance interventions in Uganda. To this end, the evaluation will examine those implementation approaches and adaptations that drove GAPP’s success and/or failure. USAID and DFID are interested in examining the effectiveness, efficiency, equity and economy of the GAPP activity. Using the DFID Value for Money (VfM) approach the evaluation will examine the 4Es to determine the appropriateness of DFID support. Evaluation Questions 1. To what extent did GAPP interventions contribute to accountability processes, institutional strengthening, citizen participation and voice at the local and national level? 2. To what extent are the GAPP implementation and management approaches relevant to the accountability, transparency and anti-corruption context in Uganda today? 3. To what extent did GAPP’s Embedded Technical Specialists Model, Tripartite Outreach Model, and Ministry of Health Accountability Support satisfy the principles of VfM? IV. EVALUATION METHODOLOGY The Evaluation Team will develop and share a detailed methodology with USAID/Uganda and DFID for approval before commencing fieldwork. The proposed methodology should demonstrate the consultants’ understanding of the level of complexity of such an evaluation and how it will be addressed. In answering the evaluation questions, the evaluation team should explore the use of Theory-based approaches to evaluation that will use an explicit theory of change to draw conclusions about whether and how the GAPP Activity contributed to the observed results. While this SOW does not seek to be overly prescriptive on the methodology to adopt, the Evaluation Team is encouraged to consider using a range of mixed methods in light of the depth and complexity of the GAPP activity. The Evaluation Team will justify the choice of the methodology in line with each of the evaluation questions in the SOW. That notwithstanding, the following specific methodological issues should be taken into consideration: 34 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ● The proposed methodology should be explicit on the sampling procedures for identifying key informants and Focus Group Discussion participants among others. When selecting key informants, the evaluation team will pay attention to the following categories: civil society organizations (CSOs), district political and technical leaders, farmers and market associations, USAID/DFID Mission staff and other donors, participating and/or did not participate in GAPP activities. The evaluation team will try as much as possible to use comparative analysis, especially when documenting GAPP successes. Comparative analysis should help the team situate what drove success or not in GAPP districts and non-GAPP districts including noting differences –if any – in GAPP and non-GAPP districts and within GAPP districts. The evaluation team should consider comparison across a variety of units of analysis such as comparison of stakeholders’ perspectives of the GAPP program across districts and at National level, comparisons of performance in GAPP and non-Gap districts. In developing a detailed methodology the team will develop robust criteria for selecting, districts and groups of individuals who have not had a relationship with GAPP will be interviewed as a counterfactual. ● The methodology should outline the sampling frame for activity sites including non- activity sites to be visited by the Evaluation Team. GAPP has a geographical strategy and has implemented different activities at varying levels of intensity over various local governments and entities. In designing the sampling framework for LGs to be visited, the Evaluation Team should, therefore, take into consideration the different treatment packages received by the different LGs, duration and nature of GAPP support to ensure coverage of the breadth of LG treatment offered by GAPP. The evaluation should also attempt to explain differences in performance between GAPP and non-GAPP districts and among GAPP districts. This will help USAID/DFID to know which treatment package is more effective, why and where as much as possible. Note that this is not designed as an impact evaluation, but it is the expectation that evaluation contractor to propose different ways that will provide the required rigor necessary to strengthen the quality of findings and conclusions. ● Given that GAPP engagement in some of the districts has not been continuous the Evaluation Team should specify how they will address recall/response/reporting bias in the evaluation methodology. The consultants should also put forward a detailed plan for how the Evaluation Team skills set structure will handle sensitive and complex questions and approaches in activity sites to be visited. The Contractor is expected to be proficient in using a range of participatory tools for data gathering and analyses, comparative data analyses, quantitative and qualitative evaluation techniques, conducting key informant interviews and leading small group discussions/presentations. ● When answering evaluation question 3, the evaluation team will use the DFID VfM framework. The team will use appropriate tools to gather information on the extent to quality of GAPP financial management, value inputs, price and flexibility in financial delivery; Quality and quantity of outputs delivered. The team will establish if the activity management systems and processes were the most appropriate to deliver the expected outputs efficiently, effectively, equitably and economy. V. DELIVERABLES Deliverables Tentative Due date 1. In-Briefing: Introduction of the Evaluation Team and discussion of the scope of work, initial presentation of the proposed evaluation methodology and work plan, and other emerging issues that may affect the evaluation. September 2, 2019 35 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 2. Inception report: Draft and present an Inception Report to be reviewed by USAID/DFID. The report will include: The evaluation team’s interpretation of the key evaluation questions and their approach to how each question will be addressed. The Consultant shall attach a completed evaluation design matrix using the template attached as Annex 2. Methodology including sampling/selection procedures for selecting respondents/informants to be interviewed, and project sites to be visited. A detailed work plan showing a timeline for each evaluation activity to be undertaken, including the fieldwork and allocation of expertise efforts as well as the methods to be used to answer each evaluation question. Detailed analysis plan/map for each of the evaluation questions, including the proposal data extraction methods for each question as well as a justification of the proposed method. Draft instruments for data collection. September 16, 2019 Fieldwork and interviews. Meetings and interviews with key stakeholders 15 days 1. Presentation of preliminary findings from fieldwork: PowerPoint presentation that at a minimum covers the following major findings, conclusions and recommendations for improvement, based on the evaluation questions Presentations will be made to both the Mission. and external multi-stakeholder group comprising the implementing partner(s) and other Activity stakeholders. The evaluation team will liaise with the Mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. Within 5 working days after completion of fieldwork 2. Draft Evaluation Report: The content should cover all the main elements of the report including major findings, conclusions, lessons learned, and relevant annexes, in line with the USAID/Uganda Evaluation Report standards set out in Annex 3. The draft report is expected within 10 working days after the oral presentation of preliminary findings. 10 working days after oral presentation of preliminary findings 3. Stakeholder debrief: After compiling the draft report, power point presentation will be prepared and presented to both the Mission and external multi-stakeholder group comprising of DFID, Parliamentary committee representative, implementing partner(s) and other Activity stakeholders. The evaluation team will liaise with the Mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. 2 days after draft report 4. Final Draft Evaluation Report: Final Draft Evaluation Report incorporating comments from USAID and other stakeholders submitted within 7 working days of receipt of the comments. The Final Draft Report should not exceed 25 pages of text in the body of the report (excluding the executive summary, table of contents, glossaries, and Within 7 working days after receipt of comments from USAID 36 | GAPP EVALUATION REPORT ANNEXES USAID.GOV annexes) and be submitted in four (4) hard copies and one electronic copy. The final draft report will also include a 2-4 page executive summary that highlights the evaluation findings, conclusions and recommendations presented in an attractive and easy to understand format for the wider public use. and select partners 5.Final Report: A final report (not exceeding 35 pages, excluding the annexes) incorporating final edits within 5 working days of receiving final comments from USAID and other stakeholders. The Final Evaluation Report should be cleared by the USAID/Uganda and DFID. The report will follow the following format: - Cover page- Showing both USAID and DFID Logos Executive summary. An Executive Summary of the main conclusions, lessons learned, and recommendations of the evaluation should be no more than two pages with cross-references to the corresponding page or paragraph numbers in the mail text. Recommendations for future programs and potential scale up: The consultant should outline key programmatic recommendations for improved design and delivery, addressing any issues identified through the evaluation process. Within 5 working days 6.Synthesis abstract A 2-3-page briefer highlighting key findings, conclusions and recommendations from the evaluation for public use. 7. Cleaned, labeled and ready to use electronic copies of data sets: All data records from the evaluation i.e. ready-to-use full data sets from the quantitative data (if any), interview transcripts, and other documented survey and qualitative responses. All reports should be provided in four (4) hard copies and one (1) electronic copy. With the presentation of the final report VI. TEAM COMPOSITION AND QUALIFICATIONS At a minimum the evaluation team will comprise:(a) public financial management expert; (b) local governance and decentralization expert; and (c) civil society/Private sector and citizen engagement expert. The following are Key Personnel: (i) Team Leader: The team leader should be an evaluation expert with technical expertise in one of the programmatic areas of focus. Responsibilities: The team leader will be responsible for ensuring overall quality assurance of the evaluation. This will entail: ● Coordinating the evaluation activities, assigning tasks to team members and providing overall supervision of their performance. S/he will serve as the main point of contact between the evaluation Activity Manager at USAID/Uganda and the Evaluation Team; ● Reviewing all deliverables and be responsible for submitting quality products to USAID on a timely basis; ● Developing, managing and, as needed, communicating updates to USAID on evaluation progress; 37 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Education and Relevant Experience: ● Advanced degree in Social Research Methods, Social Sciences, Public Administration and Management or related fields. ● Significant expertise in conducting evaluations with a minimum of 10 years’ experience in evaluation management including evaluations of local governance activities. ● Played significant role designing, managing or executing a minimum of seven evaluations, at least five of which preferably focused on local government governance issues. ● Must have been a team leader for a minimum of five evaluations, with at least two of similar scale. ● Demonstrable skills and experience in the use of complex quantitative and qualitative methods. ● Must have demonstrated experience producing high-quality reports for evaluations of complex, multi-sector activities. (ii) Local Governance and finance Expert Responsibilities: ● S/he will contribute to the design of relevant tools, collect and synthesize data, prepare and participate in the writing the report. Education and Relevant experience: ● A minimum of a Master’s degree in governance of local governments or Public Administration and Management or Development Studies, Applied Research or related fields. ● Minimum of five years’ experience evaluating or implementing local government development activities. ● Demonstrable experience in local governance finance and financial management systems, preferably in Uganda. ● Technical expertise in strengthening local government systems. ● Must demonstrate knowledge and experience with the functioning of decentralized local governments in Uganda. (iii) Civil society/Private sector and citizen engagement expert Responsibilities: ● S/he will contribute to the design of relevant tools, collect data, prepare and participate in writing the report. Education and Relevant experience: ● Postgraduate degree in public administration and management, development studies, international relations or any other relevant discipline. ● Demonstrable relevant experience in working with civil society organizations, dialogues with civil society and private sector. ● Demonstrable expertise and working knowledge in the following areas: public engagement, civil society, citizen voice and accountability, community participation in local development. ● Proven experience evaluating citizen participation or civil society support activities 38 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ● Good understanding of the civil society sector in Uganda ● Excellent analytical and reporting skills (iv) Value for Money Expert/Economist Value for money analysis assesses whether the program has bought inputs of the right quality and price (economy), how well inputs are converted into outputs (efficiency), what impact the outputs are having (effectiveness), and how much impact the project has had relative to the cost of the inputs (cost￾effectiveness). Responsibilities ● S/he will support the evaluation team in providing concrete evidence on value for money. The consultant will focus primarily on providing value for money assessment for three identified approaches and interventions for the GAPP activity. ● S/he will contribute to the design of relevant tools, collect data, prepare and participate in the writing the report. S/he will take a lead role on the team to conduct a value for money assessment covering the three identified activity approaches, and interventions. Education and Relevant experience: ● Master’s degree in economics, finance, public policy or related disciplines from a recognized university. ● Minimum 5 years of working experience in undertaking cost-benefit/effective/utility analysis and/or value for money studies with international development projects; ● Previous experience of providing management advisory services on value for money at international organizations or development agencies. ● At least 3 years of proven experience with the design and application of diverse VfM approaches is required. ● Previous involvement with and knowledge of donors including DFID or other bilateral agencies that use VfM assessments is an advantage. VII. INSTITUTIONAL RELATIONSHIP AND REPORTING ● Consultants will be answerable to USAID/Uganda and DFID for all deliverables ● USAID/Uganda and DFID will review and approve all deliverables produced by the consultants for this performance evaluation VIII. LEVEL OF EFFORT (LOE) REQUIRED The GAPP performance evaluation is estimated to begin on September 2, 2019 and be completed no later than November 30, 2019. Summary of LOE by Evaluation Team members Team leader 30 Local governance and finance expert 25 CSO and Citizen participation expert 25 Value for Money Expert/Economist 25 39 | GAPP EVALUATION REPORT ANNEXES USAID.GOV DFID/USAID’s roles and responsibilities include: I. Review and approval of the SOW and all deliverables from the consultants II. Procure and supervise consultants III. Review consultants’ work, especially key deliverables, to ensure they respond to the SOW and meet USAID/DFID quality standards IV. Facilitate and supervise fieldwork, including advise the consultants on identifying translators where needed V. Provide recommendations to and clearance on selected evaluators for this assignment GAPP roles and responsibilities are to: I. Provide guidance to the consultants with setting up meetings and interviews, and identification of the District sites to visit II. Review and comment on the scope of work and draft deliverables. IX SCHEDULE AND LOGISTICS Internal Timelines a) Discussions with QED to create shared understanding of expectations and deliverables, July 28, 2019 b) Finalization of the SOW (USAID), July 30, 2019 c) Sourcing and procurement of consultants, by August 20, 2019 d) Finalization of Fieldwork, September 30, 2019 e) Draft Report October 21, 2019 f) Final report, November 30, 2019. 40 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 3: LIST OF DOCUMENTS REVIEWED 1. Acholi Private Sector Development Company Limited. Governance, Accountability, Participation and Performance (GAPP) Program- Annual Performance Report on Enhancing Private Sector Participation in local governance for improved service delivery (March 2017-February 2018). February 2018 2. Advocates Coalition for Development and Environment. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Performance Report on The Local Government Councils’ Scorecard Initiative (LGCSCI) (January 2016- August 2019). August 2019 3. Advocates Coalition for Development and Environment. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Transparency and Accountability in Uganda’s Extractive Industries Sector (March 2018-August 2019). August 2019 4. Africa Youth Development Link. Governance, Accountability, Participation and Performance (GAPP) Program- Quarterly Performance Report on Strengthening Youth Voices for better Local Government Accountability and Service Delivery in Northern Uganda (January 2017-December 2018). December 2018 5. Bainomugisha, A., Muyomba-Tamale, L., Muhwezi W., W., Cunningham, K., Ssemakula, E., G., Bogere, G., Mbabazi, J., Asimo, N., Atukunda, P. Local Government Councils Scorecard Assessment 2016/17: Civic Engagement: Activating the Potentials of Local Governance in Uganda. Kampala: ACODE Policy Research Series No.83. 2017 6. Baser, H. and P. Morgan. 2008. Capacity, Change and Performance Study Report. (ECDPM Discussion Paper 59B). Maastricht: ECDPM. 7. Basheka, B.C. What constrains a transformed Public Service in Uganda? School of Business and Management, Uganda Technology and Management University, East Africa- Special Edition: Volume 49 Number 2.1. June 2014. 8. Build Africa Uganda. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Community Empowerment for Education improvement Project-CEEP (April 2014- March 2016. April 2016 9. Buliisa Initiative for Rural Development Organization. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Voice and Accountability Project. September 2017 10. Centre for Governance and Economic Development. Increasing participation of Micro-Business Enterprises in Local Accountability and Governance processes: Stories from Prise Project. CEGED. 2017 11. Center for Governance for Economic Development. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Performance Report on Private Sector Empowerment (PriSE) Project (February 2017-October 2018). December 2018 12. Centre for Governance and Economic Development. Governance, Accountability, Participation and Performance (GAPP) Program- Rapid Assessment Report on Community Empowerment for Local Accountability. September 2016 13. Centre for Policy Analysis. Governance, Accountability, Participation and Performance (GAPP) Program- Performance report on Improving Local Government Accountability (July 2017- November 2018). November 2018 14. Centre for Policy Analysis. Summary of the Auditor General’s Report for FY. 2014/2015. Kampala: CEPA. 2016. 41 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 15. Child Rights Empowerment and Development Organization. Governance, Accountability, Participation and Performance (GAPP) Program-Project End Report on Building civic competence to enhance democratic governance and social accountability (April 2014- March 2016). March 2016 16. Coalition for Health Promotion and Social Development. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Empowering Communities to Play a Role in Ensuring Access to Quality Health Services in Uganda (January 2017- December 2018). July 2017 17. Coalition for Health Promotion and Social Development. Governance, Accountability, Participation and Performance (GAPP) Program- Project Report on Citizen Engagement to Stop Medicine Stock Outs (COME) Project. (2013/14-2015/16). March 2016 18. Community Awareness and Response on AIDS. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Enhancing good governance, accountability and transparency for improved health service delivery in Kayunga district (May 2015- July 2017). July 2017 19. Community Centre Health Foundation. Governance, Accountability, Participation and Performance (GAPP) Program- Final Report on Program to shift professional norms amongst teachers in Mityana District. August 2018 20. Community Centre Health Foundation. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Promoting Improved Service Delivery in the Health and Education sectors in Uganda (July 2015- April 2017). August 2017 21. Community Driven Network Uganda. Governance, Accountability, Participation and Performance (GAPP) Program- End of project report on increasing participation of communities, CSO’s and PSA’s in budget transparency for improved public oversight of the budgetary process in regard to the extractive sector (March 2018- September 2019). August 2019 22. Community Empowerment for Rural Development. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on [Raising Community Voices, Participation to enhance Accountability in service delivery in Zombo District (December 2017- May 2018). May 2018 23. Community Integrated Development Initiatives. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Contributing towards Private Sector Engagement with LG for better Service Delivery (April 1, 2017 –June 30, 2019). Grant Agreement Number: 0213615-G-2017-013-00. June 2019 24. Community Uplift and Welfare Development. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Citizen Voice and Action for Government Accountability and Improved Service Delivery in Nebbi district (February 2015- March 2017). April 2015 25. Department for International Development. DFID Uganda’s Profile. July 2018. 26. Dichter, Thomas. The Capable Partners Learning Agenda on Local Organization Capacity Development. (February 2014), https://usaidlearninglab.org/sites/default/files/resource/files/LA￾Complete-Final-Report-to-USAID_2.20.14.pdf. 27. Environmental Conservation and Agricultural Enhancement Uganda. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Enhancing Agri-Business Entities’ participation in Local Governance in Hoima district (October 2015- October 2017). October 2017 28. Forum for Women in Democracy. Governance, Accountability, Participation and Performance (GAPP) Program-End of Project Report on Gender Accountability and Participation Project (February 2017- March 2019). April 2019 42 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 29. Global Forum for Development Uganda. Governance, Accountability, Participation and Performance (GAPP) Program- End of project performance report on Youth in Governance and Leadership-YoGAL project (February 2015- January 2017). January 2017 30. Independent Commission for Aid Impact. DFID’s governance work in Nepal and Uganda: A performance review. June 2018 31. Kania, John, Mark Kramer, and Peter Senge. The Water of Systems Change. Boston, MA: FSG. 2018 32. Kayunga District NGO Forum. Final Project Narrative Report. June 2017 33. Kibombo, Richard, Vincent Owarwo, Alfred Nuamanya, Nasreen Jessani, Kenneth Otikal and Patrick Mubangizi. USAID/Uganda Strengthening Decentralization for Sustainability (SDS) Project Evaluation Report. November 2015 34. Koboko Civil Society Network. Governance, Accountability, Participation and Performance (GAPP) Program-Project evaluation report on Strengthening Democratic Governance, Accountability and Participation for improved service delivery in Koboko District (2016-2017). February 2017 35. Krutz, Jeffery Conroy. Perspectives on Politics 9, no4 (2011): 961-63 36. Kwize Development Organization. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Performance Report on Promoting community participation in revenue enhancement, tracking of service delivery and accountability in four sub counties of Mubende District (March 2015- February 2016). February 2016 37. Lake Albert Children/ Women Advocacy and Development Organization. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Buliisa Citizens’ Empowerment Project (August 2015- September 2017). September 2017 38. Lambright, Gina M.S. Decentralization in Uganda: Explaining successes and failures in local governance. Boulder, Colo: First Forum Press. 2011 39. Lambright, Gina M.S. Opposition Politics and Urban Service Delivery in Kampala, Uganda. Development Review Policy, Vol 32, Issue s1, pp. s39-s60. 2014 40. Lewis, Janet I. When Decentralization Leads to Recentralization: Subnational State Transformation in Uganda. Regional & Federal Studies, 24:5, 571-588, DOI: 10.1080/13597566.2014.971771. 2014 41. Masindi NGO Forum. Governance, Accountability, Participation and Performance (GAPP) Program￾End of Project Report on Voice and Accountability Project (August 2015- July 2017). August 2017 42. Mayank Anti-Corruption Coalition. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Social Accountability Action for Community Empowerment (SAACE) Project (March 2015- August 2016). August 2016 43. Mubende Women with Disabilities Association. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Advocacy for a PWD responsive service delivery project in Mubende District (May 2015- April 2017). June 2017 44. Oyugi, Jackson. Challenges faced by the Local Government Public Accounts Committees. National Engagement of the Public Accounts Committee-Local Government with LGPACs. May 2019. 45. Parliament of the Republic of Uganda. A Handbook for Financial Accountability Committees. April 2017 46. Root Change, New Directions in Local Capacity Development: Embracing a Systems Perspective. (LCD Learning Agenda Root Change Findings Report, November 2013), https://usaidlearninglab.org/sites/default/files/resource/files/new_directions_lcd.pdf. 43 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 47. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Progress Results and Lessons Learned. June 2019 48. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (November 2012-March, 2013). April 2013 49. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April-June, 2013). July 2013 50. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (July 1 -September 30 2013). October 2013 51. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (November 29, 2012- September 30, 2013). November 2013 52. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (October 1- December 31, 2013). January 2014 53. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (October 1 2013- September 30, 2014). November 2014 54. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (January 1-March 31, 2014). April 2014 55. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April 1-June 30 2014). July 2014 56. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (July 1- September 30, 2014). October 2014 57. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (October 1- December 31, 2014). January 2015 58. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (January 1- March 31, 2015). April 2015 59. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April 1- June 30, 2015). July 2015 60. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (July 1-September 30, 2015). October 2015 61. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (October 1 2014- September 30, 2015). November 2015 62. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (October 1- December 31, 2015). January 2016 63. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (October 1, 2015 - September 30, 2016). November 2016 64. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April 1- June 30, 2016). July 2016 65. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report, (July 1- September 30, 2016). October 2016 66. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (October 1- December 31, 2016). January 2017 44 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 67. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (October 1 2016 - September 30, 2017). November 2017 68. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (January 1- March 31, 2017). April 2017 69. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April 1- June 30, 2017). July 2017 70. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (July 1- September 30, 2017). October 2017 71. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (January 1- March 31, 2018). April 2018 72. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (April 1- June 30, 2018). July 2018. 73. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (July 1- September 30, 2018). October 2018. 74. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Report (October 1 2017 - September 30, 2018). November 2018 75. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (October 1- December 31, 2018). January 2019 76. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Quarterly Report (January 1- March 31, 2019). April 2019 77. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program￾Technical Proposal. Volume 1, Solicitation No. SOL-617-12-000011, RTI Proposal No. 0281200.228.01. April 2012 78. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Revised Technical Proposal. Volume 1, Solicitation No. SOL-617-12-00001, RTI Proposal No. 0281200.228.01. November 2012 79. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program- Annual Work Plan (October 1, 2018 – November 28, 2019). December 2018 80. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program- Work Plan (March 1, 2013 – September 30, 2013). February 2013. 81. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Work Plan (October 1, 2013 – September 30, 2014). October 2013. 82. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Work Plan (February 1, 2015 – September 30,2015). February 2015. 83. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Revised Work Plan (October 1, 2015 – September 30, 2016). December 2015 84. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Work Plan (October 1, 2016 – September 30, 2017). October 2016 85. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Annual Work Plan (October 1, 2017 – September 30, 2018). February 2018 45 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 86. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Add-On Responses to Cost Questions. RTI Proposal No. 0281400.590.02, Contract No. AID-617-C-13- 00002. July 2014 87. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Baseline Assessment. February 2013 88. RTI International. Governance, Accountability, Participation and Performance (GAPP) Experiences of Capacity Building of Targeted Ugandan Civil Society; triumphs and lessons 2012-2019. August 2019. 89. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Lessons Learned Study, Component 3 – Capacity Building of Ugandan Civil Society. August 2019 90. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-National Consolidated Local Government Development Framework Report, FY 2017. September 2018 91. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program Results: Fact Sheet 2012-2019. November 2019 92. RTI International. Governance, Accountability, Participation and Performance (GAPP) Program-Revised Activity Monitoring and Evaluation Plan. August 2019 93. RTI International. Strengthening Local Capacity in Southeast Asia: Approaches and Experiences. USAID Assessing Local Capacity in Southeast Asia—Task Order, June 2016. 94. RTI International. USAID/ Uganda and DFID Governance, Accountability, Participation and Performance (GAPP) Activity-Mid-Term Evaluation Report. September 2016 95. Rukundo, Drake. Policy framework and policy prepositions for advocacy on Uganda’s decentralization policy. Kampala: CEPA Policy Series Papers Number 4. 2019 96. Samaritan Initiative for Development. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Voice, Information, Participation and Accountability Achievements (VIPAA) project (February 2018- July 2019). July 2019 97. Social Impact, Inc. End line impact Evaluation Report: Can text messages improve local governance? An impact evaluation of the U-bridge program in Uganda. April 2017 98. Southern and Eastern African Trade Information and Negotiations Institute Uganda. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on Contributing to Equitable and Sustainable Revenue Generation for Improved Service Delivery in Kitgum, Lamwo and Pader Districts (July 2016- November 2018). November 2018 99. Southern and Eastern African Trade Information and Negotiations Institute Uganda. Governance, Accountability, Participation and Performance (GAPP) Program - End of Project Report on Enhancing Opportunities for Local Revenue Mobilization and Accountability for Improved Service Delivery in Kitgum, Pader and Lamwo Districts (April 2014- June 2016). June 2016 100. Strategic Organization for Real Action. Governance, Accountability, Participation and Performance (GAPP) Program-End of project report on ‘Created’ and ‘Invited’ advocacy Spaces ‘Created’ and ‘Invited’ advocacy Spaces Participatory Strategies and Processes to accelerate promote and build Citizens’ Capacity and Action to hold LGs accountable for the service delivery. January 2017 101. The Civil Society Budget Advocacy Group. Governance, Accountability, Participation and Performance (GAPP) Program-Monitoring Report. January 2015 102. The Consulting House. Centre for Justice Studies and Innovations. Anti- Corruption Country Threshold Program Final Evaluation Report. USAID. 2010 46 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 103. The Consulting House. Centre for Justice Studies and Innovations. Strengthening Democratic Linkages Program Final Evaluation Report. USAID. 2010 104. Transparency International Uganda. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Enhancing Local Government Service Delivery (September 2016- January 2019). January 2019 105. Uganda Monitoring and Evaluation Management Service Project. Advocacy in the Health and Education Sectors in Uganda – A Situational Analysis. Uganda: USAID. February 2011 106. Uganda National NGO Forum. 2018 State of Civil Society Report: Executive Summary and Infographics, March 2019 107. Uganda Rural Development and Training Program. Governance, Accountability, Participation and Performance (GAPP) Program- End of Project Report on Strengthening Citizen Participation in Decision making processes for Good Governance (August 2015- July 2017). July 2017 108. Uganda Small Scale Industries Association. Governance, Accountability, Participation and Performance (GAPP) Program-End-line Study Report on Promoting Private Sector Participation in Local Governments (March 2017- June 2019). June 2019 109. Uganda Women’s Network. Governance, Accountability, Participation and Performance (GAPP) Program-End of Project Report on Enhancing Capacity and Influence of district women councilors in decision making for Gender, Accountability and Equality (January 2017- December 2018). January 2019 110. UNDP Global Centre for Public Service Excellence. Citizen Engagement in Public Service Delivery￾The critical role of public officials. Singapore, 2016 111. USAID. Award/Contract, RTI International. Contract No. AID-617-C-13-00002. November 2012 112. USAID. Uganda Country Development Cooperation Strategy (2016-2021). 2016 113. USAID. USAID’s Local Systems: A Framework for Supporting Sustained Development. (April 2014), https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf. 114. USAID. USAID Policy Framework: Ending the Need for Foreign Assistance. April 2019 115. USAID. USAID/Uganda Solicitation # SOL-617-12-000011 GAPP Program. February 2012 116. USAID/Uganda. Governance, Accountability, Participation and Performance (GAPP) program A Lessons Learned Study on Tripartite Outreach Model, November 2018 117. Wetterberg, Anna, Derick W. Brinkerhoff, and Jana C. Hertz. Capacity development for local organizations: Findings from the Kinerja Program in Indonesia. Research Triangle Park, NC: RTI International, International Development Group Working Paper No. 2013-03. 2013 118. Wilson-Grau, Ricardo, Heather Britt. Outcome Harvesting. Cairo, Egypt: Ford Foundation. 2012 119. World Bank. Uganda Economic Update, 11th edition: Financing growth and development - Options for raising more domestic revenues: factsheet May 2018 (English). Washington, D.C.: World Bank Group. 2018 47 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 4: LIST OF STAKEHOLDERS CONSULTED The list reflects the breadth of program participations/stakeholders that the evaluation team consulted. The evaluation team also sought impromptu meetings with stakeholders as opportunities arose during field work. # NAME TITLE INSTITUTION / DISTRICT 1 Joyce Ngaiza Governance Advisor DFID 2 Martin Muwaga Senior Monitoring and Evaluation Specialist USAID 3 Morris Nsamba Monitoring, Evaluation& Learning Advisor, DRG USAID 4 Harriet Muwanga Governance Program Specialist, COR USAID 5 Rena Mukasa Finance Specialist USAID 6 Clare Kigozi Finance Specialist USAID 7 Joy Searcie Director DRG Office USAID 8 Felix Kazahura Project Management Specialist, DRG USAID 9 Joseph Hirsch Regional Coordinator, Program office USAID USAID 10 Denis Okwar Regional Development Advisor, Program office USAID 11 Robert Misigi M&E Officer Kiboga NGO Forum 12 Tom Opwonya Executive Director TAAC 13 Aaron Sheldon Project Manager (DAI) DAI HQ 14 Tom Kyakwise Deputy Chief of Party GAPP 15 Eva Matsiko Chief of Party GAPP 16 Deborah Kyamagwa M&E specialist GAPP 17 Sarah Dkwcore CSPS Advisor GAPP 18 Benson Osok ETS, Planning and Budgeting GAPP 19 Robert Opira ETS, Planning and Budgeting GAPP 20 Geofrey Omollo ETS, Leadership and Council Development GAPP 21 Geoffrey Okullo Regional Coordinator GAPP 22 Charity Nazziwa MEL Specialist GAPP 23 Raymond Mutyaba ETS, Planning and Budgeting GAPP 24 Geofrey Mugomba ETS, Leadership and Council Development GAPP 25 Mowica Akot Regional Coordinator GAPP 26 Moses Attan National Accountability and Policy Specialist GAPP 27 Demy Kiyima ETS, Leadership and Council Development GAPP 28 Freddie Kasioye ETS, Finance and LRM GAPP 29 Grace Sekamura ETS, Finance and LRM GAPP 30 Godfrey A Tinka Local Government Program Manager GAPP 31 Elnstham Mugasha ETS, Planning and Budgeting GAPP 48 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 32 Damiel Enku ETS, Finance and LRM GAPP 33 Caroline Joan Oyella Local Government Program Officer GAPP 34 Augistine Atwijukye K. ETS, Leadership and Council Development GAPP 35 Albert Oduman National Allowability and POlicy Advisor GAPP 36 Herbert Aheebwa Regional Coordinator GAPP 37 Barbara Agaba Procurement Officer GAPP 38 Christopher Gumisiriza Grants Manager GAPP 39 Amos Korutaro Grants Officer GAPP 40 John Kauta Technical Assistance for Planning and Budgeting GAPP 41 Joseph Kabaga Finance Officer GAPP 42 Maureen Luyiga Finance Officer GAPP 43 Godfrey Tinka Embedded Technical Specialist GAPP 44 Carol Oyella Embedded Technical Specialist GAPP 45 Peter Wafula Civil Engineer GAPP/MoH 46 Doreen Kakyo Civil Engineer GAPP/MoH 47 Robert Nuwagaba Civil Engineer GAPP/MoH 48 John Paul Ssempebwa Mechanical Engineer GAPP/MoH 49 Brian Luswata Legal Council GAPP/MoH 50 Hector Tibeihaho Team Leader/ MOH Component GAPP/MoH 51 Lisa McGregor Senior Governance Specialist RTI HQ 52 Harriet Mesika Former Clerk to Council LG Adjumani 53 Denis Akuba Ag Internal Auditor LG Adjumani 54 Patrick Tandrupati District Chairperson LG Adjumani 55 Tom Adoko Clerk to Council LG Apac MC 56 Gloria Amuchu Finance Officer LG Apac MC 57 Patrick Eyul Ongom Mayor LG Apac MC 58 Chrizestom Kayise CAO LG Buliisa 59 Simon Agaba Kinene District Vice Chairperson LG Buliisa 60 William Mpagi Internal Auditor LG Buliisa 61 Chris Baruzaliire Ag CFO LG Buliisa 62 Thomasine Tandell Senior Planner LG Buliisa 63 George Toskin Musani Acting Deputy Town Clerk LG Kapchorwa 64 Stephen Mbatya Mayor LG Kapchorwa 65 Roger Mutai Planner LG Kapchorwa 66 Michael Chepkurui CFO LG Kapchorwa 67 Sarah Nakalungi CAO LG Kiboga 49 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 68 Medison Agaba Council Vice Chair LG Kiboga 69 Godfrey Kiiza Assistant Procurement Officer LG Kiboga 70 Olive Atuheire Principal Internal Auditor LG Kiboga 71 David Bwire Principal Human Resources Officer LG Kiboga 72 Steven Sekiwunga Chief Financial Officer LG Kiboga 73 Patrick Wamala Clerk to Council LG Mpigi 74 Andrew Sewagudde Senior Planner LG Mpigi 75 Francis Wamala Assistant CAO LG Mpigi 76 Peter Mutuluuza District Chairperson LG Mpigi 77 Dungu Semata CFO LG Mpigi 78 Godfrey Kyagambidwa Chief Accountant LG Mpigi 79 Charles Nsobya Senior Planner LG Mpigi 80 Allan Sentanu Planner LG Mpigi 81 Benon Mukwaya Assistant LG Mpigi 82 Frances Nayiga Labour Officer / Clerk to Council LG Mpigi 83 Sheila Semugooma Assistant Director of Audit/LG OAG 84 Victor Mukasa Kiddawalime OAG Auditor to Kiboga OAG 85 Lilly Akidi OAG Auditor to Apac MC OAG 86 Josephine Wotera Principal M&E Officer Parliament of Uganda 87 Hilda K. Mwesigwa Manager, Procurement Audit PPDA 88 Sophia Masagazi Chairperson, GESI Committee PPDA 89 John Asimwe Research Coordinator, CEPA CEPA 90 Julius Mukunda Executive Director, CSBAG CSBAG 91 Sam Barteka Speaker of Council LG Kapchorwa 92 Jimmy Yesho Senior Internal Auditor LG Kapchorwa 93 Bob Okai Chairperson LG Apac MC 94 Peter Acuda Obong District Council Speaker LG Apac MC 95 Joseph Ssebude Town Clerk LG Apac MC 96 Proskovia Maliama Acting Clerk to Council LG Adjumani 97 Tokwiny Otiku Procurement Officer LG Adjumani 98 John Bosco Draparaku Senior Planning Officer LG Adjumani 99 Max Martin Mukula Deputy CAO LG Adjumani 100 Jackson Oloya Chairperson DPAC LG Adjumani 101 Wycliffe Magala, Lira-based ETS LG Adjumani 102 Rhoney Nixon Agondua Ag. Deputy CAO/PHRO LG Buliisa 50 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 103 Alfred Asiimwe Chairperson -DPAC LG Buliisa 104 Ronald Waissajja Procurement Officer LG Buliisa 105 Robert Kirungi Speaker LG Buliisa 106 Ishaka Musisi DPAC Chair and Vice Chair LG Mpigi 107 Richard Senoga Mayor LG Mpigi 108 Raymond Mutaawe Town Clerk LG Mpigi 109 Abibu Kagwa Procurement Officer LG Mpigi 110 Gilbert Musinguzui Quality Assurance Manager UDN 111 Jude Odaro Program Manager, Citizen Empowerment and Engagement UDN 112 Andrew Walusimbi Head of Service Delivery Department USSI 113 Onesimus Mugyenyi Deputy ED ACODE 114 Wilsons Muhwezi Director Research ACODE 115 Mark Otile Research Associate ACODE 116 Jonas Mbabazi Project Manager Local Government Score Card Initiative ACODE 117 James Muhindo Research officer ACODE 118 George Mugire Research Fellow ACODE 119 Able Kwikiriza Accountant ACODE 120 Reagan Wamaji ED CEPA 121 Henry Ssemanda Embedded Technical Specialist GAPP 51 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 5: KEY INFORMANT INTERVIEW GUIDE The evaluation team conducted key informant interviews following Outcome Harvesting methodology which helped identify contribution claims. Interviews were semi-structured, using the following protocol, and last between 45 min and 60 min. Outcomes harvested and contribution claims were then validated through a validation interview. Protocols for both types of interviews are attached here. KIIs to Identify Outcomes in Face-to-Face Conversation 1. Change Questions a. What major developments (negative or positive), if any, have you observed/experienced in the last years in… i. If NAI: national accountability processes and institutional strengthening? ii. If LG/CSO: citizen participation, financial management, or decision-making in LG? b. What changes (negative or positive), if any, have you observed/experienced in your work environment? c. Which of these changes were influenced by the GAPP program, if any? 2. Outcome Description (What/ Who/ When/ Where) a. Describe the change. b. What was the situation before? After? c. What were the reasons the ‘before’ situation was present? d. Which social actor made the change? Who adjusted their way of doing things? e. When did the change happen? f. Where did the change take place? g. Unpack any buzzwords (e.g., engagement, support, transparency, accountability). 3. Significance (Why) a. Why is the change important to your respondent? b. In what way does this change make local government service delivery more effective? Be specific. 4. Contribution (What/ How/ When) a. What did GAPP contribute to the change? b. How did GAPP contribute to the change? Describe the chain of events. c. For each activity in the chain of events, list who did the action and when. 5. Counterfactual Probing a. Could the change have taken place without the help of GAPP? b. What would you or your institution have done if there had not been GAPP support? 6. Validation Contacts a. Who else can we speak with to learn more about this? 52 | GAPP EVALUATION REPORT ANNEXES USAID.GOV KIIs to validate Outcomes in Phone Follow-Up Conversations 1. Preparation: a. Check the district outcome list for open questions and verification needs. b. Familiarize yourself with the outcome stories that contain open questions and/or are not yet verified. c. Check the district follow-up list. Distribute open questions and/or validation needs among the different follow-up contacts, according to their potential insight into the outcomes. 2. Interview guide: Good morning/afternoon, my name is [name of interviewer], I am calling from QED on behalf of USAID. We are carrying out a performance evaluation of the GAPP program. Already interviewed or not? [If interviewed during field visits:] Our evaluation team would like to thank you very much for the time you have granted us when we carried out our visit to [name of district] on [date of visit]. [If not previously interviewed:] On our visit to [name of district], one of our interview partners has provided us with your contact details. Are you familiar with the GAPP program? [If not, provide key information on GAPP.] Open questions and/or validation? [If open questions:] We were reviewing the information that we have gathered on the results of the GAPP program in [name of district] and would have some additional question regarding [subject matter of open question]. We hope that you can provide us with some additional information. [Ask open questions.] [If validation needs:] Our interview partners in [name of district] have identified changes in local government performance and citizen participation that they attribute to GAPP. We would be grateful if you could give us your opinion on these changes, and whether GAPP contributed to them. [Repeat the following for each change that needs validation:] 1. [Provide a short description of the change.] To what degree are you in agreement with the description of this change? Do you fully agree, partially agree, or disagree with it? [If partially agree or disagree:] Which parts of the description do you not agree with and why? 2. [Provide a short description of the contribution of GAPP to the change.] How much do you agree with the description of how GAPP contributed to this change? Do you fully agree, partially agree, or disagree with it? [If partially agree or disagree:] Which parts of the description do you not agree with and why? 3. In your opinion, would the change have also come about if GAPP had not contributed to them? 53 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 6: FOCUS GROUP DISCUSSION PROTOCOL Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Introduction a. People b. Evaluation Methodology Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies Select Case Studies People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines c. Menu of Interventions for each line d. Which were most effective People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines c. Menu of Interventions for each line d. Which were most effective People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines c. Menu of Interventions for each line d. Which were most effective People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines c. Menu of Interventions for each line d. Which were most effective People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines c. Menu of Interventions for each line d. Which were most effective People and Process a. ID principle actors by title b. Link actor clusters to sub-IRs by lines Component Story a. How did you start? b. How did you decide i. On the needs to be addressed? ii. On the actors to work with? Explain/ probe the menu of interventions … c. What were the challenges you encountered? d. What were the breakthroughs experienced? e. As the program comes to an end, of what are you most proud? f. What were the regrets or missed opportunities? g. Lessons learned? Component Story a. How did you start? b. How did you decide i. On the needs to be addressed? ii. On the actors to work with? Explain/ probe the menu of interventions … c. What were the challenges you encountered? d. What were the breakthroughs experienced? e. As the program comes to an end, of what are you most proud? f. What were the regrets or missed opportunities? g. Lessons learned? Component Story 54 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 7: STAKEHOLDER SURVEY TOOL The evaluation team will conduct a telephone survey of stakeholders in all GAPP districts to solicit their views on the effectiveness of GAPP interventions, which they perceived as most effective and why, and recommendations for future programming. The survey will be conducted by a team of research assistants specifically trained on the survey tool. The team will reach out to people on the participant stakeholder matrix in Annex 1. The team will focus on reaching one person from each sub-category of stakeholders for each district (e.g., LG technical staff, LG council, and CSOs). The telephone conversations are expected to last 20-30 minutes each. Phone survey tool 1. Introduction text Good morning/afternoon, my name is [name of enumerator], I am calling from QED on behalf of USAID. We are carrying out a performance evaluation of the GAPP program. The information from this evaluation will help us gather the lessons learnt and make recommendations for further programming. We would be grateful if you could grant us 20-30 minutes of your time to participate in this survey. Your response will be kept confidential. You will not be identified by name in any documentation or report. The participation in this survey is voluntary but fundamental for us to understand how future programs could improve their support to you. 2. Personal information Organization Designation District 3. Views on GAPP Interventions QN1: Now I am going to read you a list of kinds of support [LG: the district, CSO: your organization] might have received from the GAPP program. Please respond “Yes” if [LG: the district, CSO: your organization] received the mentioned support or “No” if [LG: the district, CSO: your organization] didn’t. (Read “Did you receive ...?”) Choices Yes No Financial Support – receiving funding, such as a grant Material Support – receiving equipment such as computers, office equipment, generators, etc. Training – attending a course or workshop to learn specific knowledge or skills. Technical Assistance / mentoring / coaching – Onsite support from a project specialist for problem solving or to apply new ideas; Consultants – having additional professional help in 55 | GAPP EVALUATION REPORT ANNEXES USAID.GOV the form of consultants, such as clearing a backlog or producing a manual; Meeting facilitation: receiving budget support for meetings that you organize, such as being reimbursed for petrol, accommodation, or meeting space; Convening meetings – attending a meeting or forum organized by another to hear from experts or for peer learning and exchange; Study tours – traveling away from your site to attend a course or observe how things are done differently to bring back lessons; Publications – receiving financial and/or technical support to produce, print, and distribute a report or manual; Assessment – receiving an organizational or sectoral review of internal operations in order to develop an improvement plan. QN2: So you have mentioned that [LG: the district, CSO: your organization] received the following types of support. Which of these kinds of support was most significant in helping you accomplish your work? (Read out types of support that were mentioned by respondent.) Choices Please choose one Financial Support – receiving funding, such as a grant Material Support – receiving equipment such as computers, office equipment, generators, etc. Training – attending a course or workshop to learn specific knowledge or skills. Technical Assistance / mentoring / coaching – Onsite support from a project specialist for problem solving or to apply new ideas; Consultants – having additional professional help in the form of consultants, such as clearing a backlog or producing a manual; Meeting facilitation: receiving budget support for meetings that you organize, such as being reimbursed for petrol, accommodation, or meeting space; Convening meetings – attending a meeting or forum organized by another to hear from experts or for peer learning and exchange; Study tours – traveling away from your site to attend a course or 56 | GAPP EVALUATION REPORT ANNEXES USAID.GOV observe how things are done differently to bring back lessons; Publications – receiving financial and/or technical support to produce, print, and distribute a report or manual; Assessment – receiving an organizational or sectoral review of internal operations in order to develop an improvement plan. 4. Branching Section (depending on the response to QN2) TRAININGS A1: What was the most useful aspect of the training? Please choose one from the list I am going to read to you. A2: What was the second useful aspect of the training? Please choose one from the list I am going to read to you. Choice Most Useful Second Most Useful Technical content Learning methods Contact with experts and presenters Learning from others Receiving useful tools, materials, or contacts Follow-up support from GAPP Relationship with the GAPP team Something else Specify “something else”: A3: How did [LG: the district, CSO: your organization] apply the knowledge or skills that were learned? MENTORING AND TECHNICAL ASSISTANCE B1: What was the most useful aspect of the mentoring and technical assistance? Please choose one from the list I am going to read to you. B2: What was the second most useful aspect of the mentoring and technical assistance? Please choose one from the list I am going to read to you. Choice Most Useful Second Most Useful Support with problem solving; 57 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Receiving technical information; Receiving help in applying learning to my specific context; Access to networking / new contacts; Receiving useful tools, materials, or contacts Relationship with the staff; Something else Specify “something else”: B3. How did [LG: the district, CSO: your organization] apply the technical assistance you received? ORGANIZATIONAL ASSESSMENT C1: What was the most useful aspect of the organizational capacity assessment? Please choose one from the list I am going to read to you. C2: What was the second most useful aspect of the organizational capacity assessment? Please choose one from the list I am going to read to you. Choices Most Useful Second Most Useful Learning the functions of a good organization (e.g., governance) Learning the needed policies and procedures (e.g., procurement policy) The participatory, self-assessment process Time away with staff to reflect and strategize Relationship with the GAPP staff Something else Specify “something else”: C3: How did [CSO: your organization] apply the results of the organizational assessment? CONSULTANT, MATERIAL, PUBLICATION or FACILITATION SUPPORT D1: What are you able to do differently because of the support you received? CONVENING MEETINGS E1: What is the most useful aspect of the meetings [that you attended]? Please choose one from the list I 58 | GAPP EVALUATION REPORT ANNEXES USAID.GOV am going to read to you. E2: What is the second most useful aspect of the meetings [that you attended]? Please choose one from the list I am going to read to you. Choice Most Useful Second Most Useful Content of the presentations; Meeting peers in similar roles from other organizations or districts Meeting participants from other work sectors (e.g., central government, district government, or civil society) Time away to reflect and strategize Something else Specify “something else”: E3: What was [LG: the district, CSO: your organization] able to do differently because of these meetings. *END OF BRANCHING* QN3: How important was GAPP in improving your individual job performance? Choices Please mark choice Highly important Important Somewhat important Not at all important QN4: How significant was GAPP in strengthening your [department’s/organization’s] work performance? Choices Please mark choice Highly significant Significant Somewhat significant Not at all significant 59 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 5. Application of Program Support QN5a: Has [LG: the district, CSO: your organization] done any of the following because of the support it received from GAPP? Choices Please mark all that apply Continued to train others on knowledge & skills learned at a GAPP training Applied new tools or procedures developed through GAPP Made improvements in management systems and procedures Started new programs or activities Developed network connections Worked to generate additional revenue Something else Specify “something else”: QN5b: How did GAPP contribute to this? QN6: Now I would like to know who or what helped [LG: the district, CSO: your organization] the most to do that. I am going to read a list for you to select from. QN7: And now - who or what was the greatest obstacle in doing that? Choices Qn6. Qn7. My [Superior/Immediate Boss/Supervisor] Other colleagues GAPP staff Money Organizational or committee priorities External accountability processes Community Participation Something Else Specify “something else”: QN8: Finally, before we get to your recommendations, we would like to ask what has been the most significant effect of GAPP to [LG: the district, CSO: your organization]. Again, I am reading to you a list. 60 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Please choose only one of the options. Is it that you now have… QN9: What has been the second most significant result of GAPP support? Choices Most Significant Second most Significant Better management systems Better working infrastructure (computers, generators, etc.) More funding for programs and services New activities or services to clients or beneficiaries; Enhanced the reputation of my organization, department, or committee; Increased personal job responsibilities Personal promotion Other Please specify “other”: 6. Recommendations for Future QN10: What recommendations do you have for improving programs such as GAPP in the future? QN11: What additional support should be included? QN12: What other comments/suggestions do you have that have not been addressed already? 61 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 8: ANALYSIS OF STAKEHOLDER SURVEY 1. Description of sample Stakeholder type Count Percent Local government (admin.) 42 35.0 Local council (LC4/5) 41 34.2 CBO (local CSO) 37 30.8 Total 120 100.0 Region Count Percent Acholi 28 23.5 Bunyoro 26 21.8 Central 24 20.2 Lango 25 21.0 Westnile 16 13.4 Total 119 100.0 Region Count Percent Northern 69 58.0 Western 26 21.8 Central 24 20.2 Total 119 100.0 Region LG (%) LC (%) CSO (%) Total (%) Northern 33.3 34.8 31.9 100.0 Western 38.5 34.6 26.9 100.0 Central 37.5 33.3 29.2 100.0 Total 35.0 34.2 30.8 100.0 2. Support from the GAPP program Type of support Count Percent Training 119 99.2 Technical Assistance 114 95 Meeting facilitation 114 95 Assessment 98 81.7 Consultants 93 77.5 Convening meetings 91 75.8 Publications 75 62.5 Study tours 70 58.3 Material 51 42.5 Financial 35 29.2 Type of support Northern (% of Northern) Western (% of Western) Central (% of Central) All regions (% of respondents) Financial 26.09 34.62 29.17 28.57 Material 40.58 61.54 25.00 42.02 Training 100.00 96.15 100.00 99.16 TA 92.75 96.15 100.00 94.96 Consultants 76.81 76.92 79.17 77.31 Facilitation 91.30 100.00 100.00 94.96 62 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Convening 75.36 69.23 83.33 75.63 Study tours 42.03 76.92 87.50 58.82 Publication 59.42 57.69 75.00 62.18 Assessment 78.26 84.62 87.50 81.51 Notes: First line reads “Whereas only 26.09% of respondents in the Northern Region have received financial support from GAPP, in the Central Region 29.17% have received financial support.” Type of support LG (% of LG) LC (% of LC) CSO (% of CSO) Total (%) Financial 23.81 14.63 51.35 29.17 Material 40.48 39.02 48.65 42.50 Training 100.00 100.00 97.30 99.17 TA 95.24 92.68 97.30 95.00 Consultants 66.67 75.61 91.89 77.50 Facilitation 95.24 92.68 97.30 95.00 Convening 78.57 68.29 81.08 75.83 Study tours 52.38 65.85 56.76 58.33 Publication 61.90 58.54 67.57 62.50 Assessment 83.33 73.17 89.19 81.67 Notes: Reads the same as above. Most significant type of support Count Percent Technical Assistance 48 40.0 Training 48 40.0 Assessments 7 5.8 Financial 7 5.8 Consultants 5 4.2 Material 2 1.7 Publications 2 1.7 Facilitation 1 0.8 Convening meetings 0 0.0 Study tours 0 0.0 Total 120 100.0 Most significant type of support LG (% of LG) LC (% of LC) CSO (% of CSO) TA 45.24 34.15 40.54 Training 45.24 60.98 10.81 Assessments 4.76 0.00 13.51 Material 2.38 2.44 0.00 Publications 2.38 0.00 2.70 Consultants 0.00 0.00 13.51 Facilitation 0.00 2.44 0.00 Financial 0.00 0.00 18.92 Total 100.0 100.0 100.0 3. Most useful aspects of different types of support Most useful aspects of trainings Overall (%) First (%) Second (%) Technical content 76 64 12.8 Receiving useful tools 28 6 23.4 63 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Follow-up support 20 2 19.1 Learning from others 20 10 10.6 Relationship with GAPP team 20 6 14.9 Learning methods 16 4 12.8 Contact with experts and presenters 14 8 6.4 Total 194.0 100.0 100.0 N = 50 Most useful aspects of mentoring/TA Overall (%) First (%) Second (%) Receiving technical information 53.8 48.1 6.4 Support with problem solving 38.5 26.9 12.8 Receiving useful tools 32.7 5.8 29.8 Receiving help in applying learning 28.8 11.5 19.1 Access to networking 17.3 7.7 10.6 Relationship with GAPP staff 17.3 0.0 19.1 Other 1.9 0.0 2.1 Total 190.3 100.0 100.0 N = 52 Most useful aspects of assessments Overall (%) First (%) Second (%) Learning needed policies & processes 75 50 28.6 Learning functions of good org. 62.5 50 14.3 Relationship with GAPP staff 12.5 0 14.3 Participatory/self-assessment process 37.5 0 42.9 Total 187.5 100.0 100.0 N = 8 4. Overall assessment of GAPP Importance of GAPP for individual job performance Count Percent Highly important 98 81.7 Important 18 15.0 Somewhat important 4 3.3 Not at all important 0 0.0 Total 120 100.0 Significance of GAPP for performance of org./depart. Count Percent Highly significant 93 77.5 Significant 26 21.7 Somewhat significant 1 0.8 Not at all significant 0 0.0 Total 120 100.0 Application of GAPP support Count Percent Improved systems & procedures 109 90.8 Continued training others 106 88.3 Applied new tools 103 85.8 Developed network connections 86 71.7 Generated additional revenue 83 69.2 Started new programs & activities 64 53.3 Other 12 10.0 Most significant effect of GAPP Overall (%) First (%) Second (%) 64 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Better management systems 80.5 67.8 12.8 Enhanced reputation of organization 36.4 10.2 26.5 Increased personal job responsibilities 31.4 6.8 24.8 New activities 16.9 5.1 12.0 More funding for program & services 15.3 3.4 12.0 Better working infrastructure 10.2 4.2 6.0 Personal promotion 8.5 2.5 6.0 Total 199.2 100.0 100.0 Most significant effect of GAPP LG (% of LG)* LC (% of LC)* CSO (% of CSO)* Better management systems 80.5 85.4 75.0 Enhanced reputation of organization 41.5 20.0 50.0 Increased personal job responsibilities 34.1 45.0 13.9 Personal promotion 17.1 5.0 2.8 Better working infrastructure 12.2 7.5 11.1 More funding for program & services 9.8 25.0 11.1 New activities 4.9 12.5 36.1 Total 100.0 100.0 100.0 *Mentioned as first or second most significant effect. 5. Enabling/hindering factors Helped most (in application) Count Percent GAPP staff 62 52.5 Other colleagues 19 16.1 Community participation 11 9.3 Superior 10 8.5 Organizational priorities 10 8.5 External accountability process 3 2.5 Money 3 2.5 Total 118 100.0 Helped most (in application) LG (% of LG) LC (% of LC) CSO (% of CSO) GAPP staff 51.2 58.5 47.2 Other colleagues 17.1 19.5 11.1 Superior 12.2 4.9 8.3 Organizational priorities 12.2 4.9 8.3 Community participation 4.9 7.3 16.7 Money 2.4 0.0 5.6 External accountability process 0.0 4.9 2.8 Total 100.0 100.0 100.0 Greatest obstacle (for application) Count Percent Money 50 43.1 Other 18 15.5 Other colleagues 14 12.1 Community participation 11 9.5 Organizational priorities 10 8.6 Superior 6 5.2 External accountability process 4 3.4 GAPP staff 3 2.6 65 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Total 116 100.0 Other: ● Authorities ● Excluding some stake holders like vendors from the training as it limited the knowledge to only technical people. ● Lack of adequate infrastructure in the district. ● Lengthy, bureaucratic process, for example for petitions made by the community members on education, change in policies, budgets etc. ● Limited information flow (2x), with one respondent explaining “there was slow/no information flow channel from the top to the operational level”. ● Most departments were resistant to change. ● Newly appointed councillors. This often means that there is staff turnover which keeps the district in frequent orientations and trainings than implementation thus an obstacle. ● Political interference (4x), with one respondent explaining “because this program had an element of taxation and performance”. ● Security officials looked at the organization as spies. ● Technical staff ● Lack of time (2x): o The greatest obstacle was the time that was allocated for the GAPP program, this was too short given the activities that were attached to it. o The time factor of the GAPP Program was the greatest obstacle. This is because the duration of the program was short compared to the scope. ● The timing is lacking, we always experienced cases of activities colliding. ● There is a tendency of late sharing of timelines. The greatest obstacle was the late sharing of when something has to be delivered, this is majorly because of the bureaucracy. Greatest obstacle (for application) LG (% of LG) LC (% of LC) CSO (% of CSO) Money 37.5 37.5 55.6 Other colleagues 20.0 10.0 16.7 Superior 15.0 12.5 8.3 Other 12.5 7.5 5.6 Community participation 7.5 17.5 2.8 External accountability process 2.5 0.0 8.3 GAPP staff 2.5 2.5 2.8 Organizational priorities 2.5 12.5 0.0 Total 100.0 100.0 100.0 6. Recommendations Type of recommendation Count Percentage (of respondents) More of existing activities 89 74.2 Improvement of implementation 39 32.5 New topics/activities 20 16.7 Recommended improvements Count Percentage (of respondents) More timely delivery of services 11 9.2 More follow-through & follow-up 10 8.3 Stronger local presence 9 7.5 More inclusive design of trainings 7 5.8 66 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Conduct local needs assessments 6 5.0 Improved communication 5 4.2 More timely delivery of services: ● "Giving Technical Support has been very significant, however GAPP/similar programs should be able to give their support in a more timely manner. Various districts have different needs, therefore for the future programs, there is need to first conduct a needs assessment.” ● "Trainings should be for all stake holders like vendors because sometimes the knowledge is not passed on. And arranging different sessions of meeting that can favor all people because sometimes we miss out due to our time schedules.” ● "Due to the elections that are going to be held, there will be new leaders coming; so there I would recommend to continue with similar programs like GAPP so that the new leaders get the technical knowledge in their offices." ● "GAPP or future programs should change the training environment for staff get comfortable in their workspace and exposure to another environment to perform better. Future programs should work on enhancing this support and also conduct them on a regular basis because learning doesn't stop." ● "I recommend that needs assessment should be conducted prior of any project/program intervention. Additionally, there should be an element of complexity for all the started tasks. For example, some GAPP staff would start a new task without finishing the earlier one. Lastly there should earlier communication of the planned work to be conducted in the community. This helps us to include it in our work plan so that cases of collusion are minimized." ● "Future Programs should push the government to come up with the minimum qualification for councillors so that we limit the number of people that cannot read nor speak at the lower council level. The trainings should at least take 4 to 5 days to help us fully understand the concepts and ask all the questions we may be having. District chairpersons should be taken to Uganda Management Institute to learn leadership skills for at least one week." ● "I recommend that the trainings should be conducted at least once in every two months not just a one off, the way GAPP was doing. All organizations should be given the similar treatment of support, not giving to some much and to the others less." ● "I recommend that the training duration is between 2-3 days, this allows understanding of concept and ideas, less of that it will be just for formality purposes other than understanding." ● "Future programs should improve the method of calling for trainings and meetings to a timely manner. And administrative explanations should be made clear such as facilitations to avoid conflicts." ● "The process of getting a consultant is too long yet at times we need urgent response, I think it’s because consultants are from Kampala. So I recommend future programs should get the consultants from within the region." ● "The program should ensure that the other activities to be started, the earlier ones started should first be completed." Proposed new topics/activities Count Percentage (of respondents) Material support 6 5.0 International exchange 6 5.0 Organizational exchange 5 4.2 Economic development 1 0.8 Infrastructural support 1 0.8 Formation of AIDS committees 1 0.8 Support to saving groups 1 0.8 Monitoring LG activities 1 0.8 67 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Types of additional support proposed Count Percentage (of respondents) Financial 30 25.0 Capacity building 32 26.7 Material support 39 32.5 Building infrastructure 5 4.2 Other 6 5.0 Types of proposed capacity building Count Percentage (of respondents) Trainings: Grant writing 4 3.3 Trainings: Economic empowerment 3 2.5 Trainings: Accounting 3 2.5 Trainings: HIV/AIDS 1 0.8 Trainings: Data collection 1 0.8 Trainings: Customer care 1 0.8 Technical support 10 8.3 Learning exchange: District 7 5.8 Learning exchange: International 3 2.5 Learning exchange: Organizations 3 2.5 Review & Feedback 1 0.8 68 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 9: QUALITATIVE COMPARATIVE ANALYSIS DESCRIPTION The QCA identified thirteen contexts in which local governments consistently reached high performance. These contexts are either found in specific regions in Uganda (Central & East, North & West) or across the country. They apply either to district or municipalities or two both types of local government. Essentially, these contexts consist of combinations of the five context conditions (1) high GDP, (2) strong service structure, (3) availability of essential staff, (4) high local revenue, and (5) a history of clean audits. Together, they explain 78% of cases of high performance among all analyzed districts and municipalities in Uganda. Context conditions for high local government performance (1) High GDP per capita. Annual GDP per capita in the district or municipality is considered high if it is more than 710$. According to this threshold, 9.7% of LGs have a high GDP per capita. Data taken from Rafa et al. (2017) and the 2014 National Population and Housing Census. (2) Strong service structure. A strong service structure is present if less than 5% of the population in the district or municipality live more than five kilometers away from a primary school. 29.9% of LGs have such a strong service structure. Data taken from the 2014 National Population and Housing Census. (3) High local revenue. Local revenue per capita is considered high if it is more than UGX 750. According to this threshold, 52.1% of LGs have a high local revenue. Data taken from Annual Reports of the Auditor General (2010–2014) and the 2014 National Population and Housing Census. (4) Essential staff available. Essential staff is available if staffing shortages in the district or municipality are below 39%. This is the case in 33.8% of LGs. Data taken from Annual Reports of the Auditor General (2010–2014). (5) Clean Audit. A district or municipality has a “Clean Audit” if the Auditors’ Opinion in the OAG Audit 2012 was unqualified. This was the case in 41.7% LGs. Data taken from Annual Reports of Auditor General (2010-14). The diagram on the next page shows the thirteen contexts for high LG performance. With regard to GAPP support, we differentiated between district-level and strong national-level support. GAPP district￾level support includes activities in the district or municipality under components 2 and 3. GAPP national￾level support to the district or municipality is considered strong if NAI outreach activities took place in the district and GAPP supported two or more procurement audits. For the first six contexts, GAPP support is a necessary component. This means that in these cases, GAPP made an indispensable contribution to high LG performance. The remaining seven contexts represent ways in which LGs achieved high performance without GAPP support. How to read the diagram Contexts represent parallel ways in which LGs achieve high performance. Contexts are numbered and listed in separate rows. The two columns on the right side of the diagram with a grey background provide information about the outcome under study and the percentage of all high-performing LGs with the given context that reached the outcome. The plus in the first of the two grey columns indicates that the presence of the outcome was analyzed. The columns with a colored background represent the conditions that were employed in the analysis. A plus in the column of a condition means that that condition is present in the given context. A minus means that in the given context the condition is not present. If in one of the rows there is neither a plus nor a minus for a given condition, it means that the condition is irrelevant for that context. For example, the first row indicates that 3% of high performing LGs achieved this outcome because they received district-level GAPP support and strong national-level GAPP support. All other conditions were irrelevant for 69 | GAPP EVALUATION REPORT ANNEXES USAID.GOV achieving high performance in this context. Cont ext Geography and LG type Context condition GAPP support High￾perfor￾ming LG Cove￾Regions rage District or Mun. High GDP Strong Service Structure Essential Staff High LR Clean Audit District Level Strong National Level 1 Across Both 3% 2 CE & EA Both 14% 3 Across Both 10% 4 Across Both 9% 5 Across Both 7% 6 Across Both 9% 7 Across Mun. 9% 8 Across Mun. 5% 9 Across Both 5% 10 NO & WE Both 5% 11 CE & EA Both 14% 12 CE & EA District 5% 13 CE & EA District 17% Notes: Solution for all Ugandan LGs that retained the territorial integrity since 2012 (n = 113). Explains 78% of cases of high LG performance (consistency: 94%). The paths show that GAPP leveled the playing field for LGs. Where GAPP provided support, it decisively improved the chances of less well-equipped LGs reaching high performance. For some types of LGs that received GAPP support, context conditions lost importance for achieving high performance altogether. If a district or municipality received strong national-level support under GAPP component 1 or is in the Central and Eastern Region, performance conditions were irrelevant for high performance (contexts 1 and 2). Results indicate that under these circumstances, any GAPP-supported LG can reach high performance. Other GAPP-supported LGs outside Central or Eastern Uganda needed at least one context condition in place to achieve high performance (contexts 3 to 6). The biggest difference GAPP made when it supported LGs that showed some indication of good local leadership (paths 3 and 4). For LGs with just one of the managerial context conditions in place, GAPP support increased the share of LGs that reached high performance from 31% (context 3) and 33% (context 4) to 100%. For these cases, the presence of structural context conditions was irrelevant. When districts in Northern and Western Uganda were not supported by GAPP, they could only achieve high performance if they could rely on a strong service structure (see paths 9 and 10). 70 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 10: VALUE FOR MONEY DETAILED ANALYSIS FOR THE THREE MODELS The purpose of a VfM strategy is for accountability to taxpayers, to ensure that performance is maximized at minimum costs and supports real time learning so that the program is adaptive, and course correction takes place with costs factored in. It allows high cost low performing activities to be identified and improved throughout program implementation. If programs are designed at the outset with an economic analysis that estimates benefits or has unit cost targets for various outreach levels, VfM strategies or frameworks usually follow that analysis and include metrics to monitor progress towards achieving that defined value. VfM metrics are costed results where we produce information on how much units of delivery cost, and monitor the trends internally, or compared to other similar programs (external benchmarking). This type of comparison is essential, because value is a relative concept that makes sense when compared to things. Activity-based costing method provides the foundations for VfM analysis. It is a method of assigning costs to actual activities, based on the resources that they consume. At its essence, activity-based budgeting begins by looking at results and the activities that created them, as opposed to cost and input-based budgeting, which often begins with inputs and works outward. Within activity-based costing, it is necessary to estimate the proportion of fixed costs and overheads across each activity. This does not mean VfM analyses are not possible without this costing method, however they are more challenging, and implementers should design VfM indicators that can be calculated using total expenditure, or outputs related costs. Analysis of GAPP Expenditure and Key Cost Drivers: In order to understand the main costs incurred at GAPP, the information in the table 1 below is provided based on GAPP expenditure until the end of August 2019. The component costs provided are inclusive of labor, indirect costs, direct costs, and fees related to that component. MOH activity is provided as a separate line item. Finally, CLA was also provided as a separate item, inclusive of reflections sessions, learning, lead MEL specialist’s inputs. GAPP Components/ Activity Total Expenditure (FY 2013-Aug 2019) Work stream with NAIs (including work with OAG, PPDA, tripartite meetings, etc) $ 4,782,927 LGs work stream (including ETSs, GAPP and RTI staff work on component 2, etc) $ 12,380,381 Work with civil society (including PLOCA work by RTI/ GAPP teams, GAPP team lead on component 3, work with private sector, advocacy trainings, etc) $ 6,765,408 Grants to civil society (all 87 grants, and GAPP grants manager time, etc) $ 6,344,291 Ministry of Health TA $ 1,161,837 CLA (reflections sessions, learning, lead MEL specialist’s inputs) $ 2,120,013 Total $ 33,554,856 There will reportedly be close to full utilization by the end of GAPP, which is November 2019, and a small variance might be incurred due to exchange rate fluctuations. 71 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Analyzing the above expenditure, we find that voice and accountability related work with civil society consumed the biggest share of funds at around US$13.1m. This was followed closely by local government work stream, at around US$12.3m. The GAPP interventions working with national institutions (NAIs) spent considerably less, and perhaps in line with the main objectives of the program, of working more with local government levels rather than centrally. NAIs and LGs related work streams contributed more significantly to GAPP results than the work with civil society, despite the latter spending more of program funds. These are discussed elsewhere in this report. Chart 1: GAPP Funds Expended to Sep 2019 In terms of the overall make of the GAPP budget and key cost drivers, they are shown in table 2 below. The utilized amounts and therefore percentages may have slightly changed20, however it is useful to explain what the main cost categories were. As seen below, direct staff costs and indirect costs make up of a large proportion, and this is common across programs which use technical assistance as main mode of delivery. Trainings and workshop related costs are also in relation to this delivery. Further, grants to CSOs made up of over a fifth of the overall budget. GAPP Budget Item Share in Overall Budget (%) Labour 19% Fringe Benefits 6% Travel and transport 5% Training, workshop, service delivery 6% Equipment and materials <1% Grants 22% Other direct costs (activities) 15% Indirect costs (home office related) 22% Fixed fees 5% Total 100% VfM Questions for Evaluation Report: Evaluation Question #3: To what extent did GAPP’s Embedded Technical Specialists Model, Tripartite Outreach Model, and Ministry of Health (MOH) Accountability Support 20 As this was taken from the DFID Annual Review of November 2018. 72 | GAPP EVALUATION REPORT ANNEXES USAID.GOV satisfy the principles of VfM? 3.1 To what extent do these models represent VfM and are appropriate systems in place to secure and assess this? The 3 models examined were assessed to represent VfM to a considerable extent. In terms of systems being in place to deliver and assess VfM, the evaluation team found that such systems that underpin delivery of value were in place for GAPP. The systems and related findings are summarized below. They concern all three models that are in question, namely the tripartite model, ETSs and MOH Technical Assistance. Table 3 summarizes the systems reviewed in order to respond to the above question. The existence and utilization of those systems apply across the board for the GAPP activity, and to a large extent for the 3 models examined here. Where there were exceptions, these are noted and explained. The VfM sub-categories are derived from the VfM analysis methodology that was provided in Inception Report for this evaluation. This type of reporting is common in DFID-funded programs. VfM Related System/ Mechanism GAPP Experience/ Practice/ Result Overall A VFM Strategy/ Framework designed at the outset GAPP did not have this. An economic analysis that defines expected benefits in monetary terms GAPP did not have this. Economy Expenditure statements that define cost categories GAPP provided these to the Evaluation team. Budget break-down showing key cost items GAPP has these and provided them periodically to USAID and DFID. Having a finance manual which outlines procurement processes to follow GAPP used USAID processes and RTI Procurement& Administrative Desk Guide Having an asset registration system and asset disposal plan (at the end of the program) GAPP has these in place, including the disposal plan. External audits carried out on program expenditure GAPP was audited once (covering two years 2014-2016), and at the time of writing, going through one more audit. Those audits covered locally incurred expenditure. This was done as a result of DFID requirement. Majority of the issues raised by the previous audit were regarding grants handling by CSOs. Appropriate staff size GAPP had 66 members of staff as at July 2019. This number included ETSs, and MOH experts as well. For a program that operated in 5 regions, 40 districts and across 3 workstreams, this is considered appropriate. Selection of inputs Out of the routinely procured inputs, workshop and training costs such as venues and accommodation were important. For these, GAPP used pre-qualification process and lists. Similarly for TA, a round of pre-qualification was done for short term TA, covering 117 individual consultants and their fee rates. For fee rates, a market survey by SDS (a sister program by USAID) was used as benchmark. For items of higher threshold (US$35,000), RTI Nairobi office carried out procurements. All the rest of the procurement was done out of Kampala office, and not from GAPP regional offices. 73 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Financial controls for grantees Compliance checks and quality assurance were done by regional offices and GAPP head office. Engagement before and after grants for financial reporting and controls, financial trainings. PLOCA tool was used as the basis of assessments. Two grants were cancelled after irregularities were discovered. Majority of the grants (56 out of 87) were below the $150,000 level (simplified cost reimbursement) and 7 grants were above that (standard cost reimbursement). The remaining were in-kind asset transfers to LGs or fixed amount awards to CSOs. Adherence with USAID and DFID financial regulations Both agencies reported positive feedback on RTI performance in this regard, and provided evidence. Documenting VfM cost savings examples GAPP provided qualitative examples in quarterly and annual reporting. Further, USAID reported that RTI complied well with cost controls related criteria in the Contractor Performance Assessment, where such examples were provided. Efficiency Quality of results reporting Follows USAID processes, CLA and FQAs, etc. Extensive results framework with mostly numeric and percentage type results reported. Tracking outreach (head count, etc) Over 14k people trained, audits that were undertaken and investigations that were supported, as well as outreach from grants implemented by CSOs. Other data on RHDs, TOM meetings, etc were also tracked. Risks monitoring Risks were carefully monitored and significant changes in risk levels or likelihoods of occurrence were reported in quarterly and annual reports. One example was scaling back on TOM meetings by GAPP during 2016 as it was an election year and the meetings would carry the risk to be politicised. Variances between budgets and actual fund utilisation USAID reported that there were not significant deviations from work plans and budgets. Fund utilisation will be close to full by the end of the program. Effectiveness Costs of results (cost effectiveness) GAPP has not tracked costs of delivery. Costs were not apportioned to units of results delivered, which is a requirement for VfM reporting. For this evaluation, costs of main program components, as well as MOH TA were prepared and shared with the evaluation team. However, we do not have detailed information on exactly what was under each category, and what assumptions were used for apportioning of overheads. Monetised or quantified benefits from program interventions This was not done at the outset, and not done during implementation. However, for MOH TA it is available, as explained below. For revenue work under component 2, a number of examples of quantification were given in results reporting. This is also provided below. Sustainability Explained under each model, as well as prominent examples are highlighted. For example, placing an ETS in MoLG, and cost sharing arrangements for tripartite meetings, can be mentioned. DRP is also a good example, especially for sustaining ETSs work, however there is need for a better mechanism to compensate the experts in the pool in order to sustain quality of delivery. Equity 74 | GAPP EVALUATION REPORT ANNEXES USAID.GOV GESI results and performance GAPP type of programming can be classified as ‘indirect delivery’, and therefore does not necessarily feature the modalities and interventions to directly impact on women or other disadvantaged groups on a program-wide and systematic manner. For example, an education, health or a cash transfers program would be better suited to target and reach vulnerable groups. We can conclude that GAPP possibly utilised opportunities to address GESI content in their work, however hard to conclude that there were any outstanding achievements in VfM Equity. This is valid for all three models investigated in this section. Reporting of GESI outreach The reporting on GESI included the number of women councillors under Component 2. A number of CSOs’ reports featured disaggregated headcounts of outreach, including for gender and disability. 3.2 To what extent do the actions and processes taken by the implementers and partners in implementing these three models represent best practice to maximize economy, efficiency, effectiveness and equity? It should be noted that GAPP started in late 2012 using USAID financial and programmatic mechanisms. DFID joined later in 2015 and even though it matched USAID funding at par, overall management of the program was left to USAID, with DFID taking the back seat. Many of the requirements that DFID put for programs, such as annual reviews, external auditing of local expenditure annually, VfM reporting, came later on and were responded to efficiently by RTI and GAPP staff. The feedback on RTI as the implementing partner was very positive in this sense. RTI has successfully responded to the requirements of two development partners in a timely and effective manner. GAPP was not set up with a VfM strategy or a framework. This means that GAPP program design did not have an overall economic analysis, or a set of VfM indicators to be monitored periodically. Similarly, DFID’s Business Case outlining DFID funding to GAPP did not feature an economic analysis21. A VfM framework was not adopted during implementation, and GAPP’s VfM reporting has focused on examples identified at quarterly or annual reporting stages. These were often qualitative examples about cost savings achieved by GAPP teams. GAPP type of programming does not lend itself well to using credible CBAs, because the type of benefits it generates are not easy to monetize (better procurement, strengthened audit processes, or enhanced community voice, etc.) and there are often significant attribution issues (governance is very context specific and often the program is one of the many actors in an intervention area). Most often, DFID programs are identifying a set of VfM indicators which are like costed result indicators, to be able to report on trends on value. Even then, governance programs pose significant problems to VfM reporting. This was also recognized by DFID watchdog ICAI in a recent review looking at DFID’s governance work in Nepal and Uganda22. Finally, it is not easy to compare one program’s results to another, as governance issues can be very context-specific. Value is a relative concept that makes sense in comparisons, and therefore governance interventions face this additional comparative challenge. Value delivered from the 3 GAPP models, as well as the VfM best practices were reviewed and compiled considering these challenges. The above table for systems, VfM categories and criteria apply to a large extent to all of GAPP, as well as to the three models explored for the VfM questions. For each of the three models investigated for this Evaluation, we combined evaluation Questions 3.1 and 3.2, and will explain the VfM findings based on 4 21 There were savings estimated of up to £90m from the SUGAR/ TAF component of the DFID program, and these were estimated to arise from recovered public funds as a result of anti-corruption interventions. 22 DFID’s Governance Work in Nepal and Uganda: A Performance Review, ICAI, June 2018, p.30. 75 | GAPP EVALUATION REPORT ANNEXES USAID.GOV VfM categories, plus Sustainability, (4Es+S) identifying attributable actions of the implementers and partners where applicable. Tripartite Outreach Model: Economy: The model used GAPP processes for procurement, as explained above. It was led by technical staff from GAPP, and at times, used short-term TA support. This activity falls under Component 1 work with NAIs, and has also had inputs from ETSs. Total cost incurred by GAPP for the tripartite meetings was provided by GAPP teams upon request by evaluation team, and it amounts to US$ 257,875 (including indirect costs) spanning over the program years23. Efficiency: As for the cost of each meeting, following the cost sharing principle with the government, where GAPP paid for venue and accommodation and GoU paid for allowances for attendees, it was reported that each meeting eventually cost a fraction of what it used to be at the outset. The estimate from the Component 1 Technical Lead was that it cost around UGX 16m ($4,370) for one local government hearing using the tripartite model, and following the cost sharing principle this came down to about UGX 1.5m ($410). The model also evolved in a more efficient manner in terms of coverage: holding 3 separate committees for multiple LG units simultaneously in a one-day session. Further, the cost sharing principle provided value also because it indicated a stronger commitment from the Government counterparts. Effectiveness: The essence of TOM activity was to conduct full hearings following a public audit, as required by law. Before GAPP, LG audit related hearings were not periodic and systematic, and the issues raised in reports were often not brought to the attention of the wider public. A significant change that GAPP brought about, according to key feedback, was bringing PPDA, OAG and PAC LG to work more closely, because this has helped each organization carry out their functions more efficiently and effectively. Similar to carrying out of actual LG audits, post-audit hearings which are required by law, were not being conducted. The proceedings compiled after hearings used to be non-binding, like a workshop report and this also changed to a formal committee report written and tabled at the Parliament. It is not feasible to quantify the type of outcomes that these meetings have achieved. TOM model can be credited with increasing citizen participation, empowerment and potentially strengthening audit processes, chiefly through helping LGs understand their tasks and responsibilities and getting the NAIs to work more closely. However, at this stage these outcomes cannot be credibly monetized. It is feasible however, to deduct that better public audits can eventually lead to better procurement and avoidance of fraud and leakage of public funds. The feedback during the evaluation mission indicated very positive contribution of TOM model towards a well-integrated public accountability system in Uganda, and opening up accountability discussions inclusive of all stakeholders. This model should be maintained for future governance programming. Equity: There are no specific results on VfM Equity that can be reported on TOM model. GESI considerations being systematically included in PPDA audits is relevant for work under Component 1, and not specific to TOM. Sustainability: What started as ‘tri-partite’ has in time become ‘multi-partite’. In addition to the triage of PAC-LG, OAG and PPDA, the participation has grown to include MPs, civil society, private sector organizations, and media (chiefly radio). In terms of what remains to be achieved for TOM going forward, GAPP team are of the opinion that as the current TOM structure does not include the service delivery ministries such as health, education and agriculture, they can be brought on board. Currently only organizations with ‘watchdog’ mandates take part in the meetings. 23 The meetings costs have not fluctuated widely over the years, except a sharp drop in 2016 (a tenth of the cost of other years). The cost was highest for 2018 with over US$ 66k. The reason for the drop in 2016 was because it was an election year and GAPP scaled back on that activity to avoid meetings to be used for political mileage. 76 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Considering the balance of 4 Es, it is likely that TOMs have represented good value for money. The costs incurred are judged to be reasonable and modest. As we are unable to ascertain monetary benefits, we used the qualitative evidence generated by GAPP, and key feedback during evaluation interviews. They all indicated change in the positive direction. If the model is sustained beyond GAPP, the value delivered will be more evident, and therefore should be considered as part of future governance programs to be funded. Technical Assistance to MOH: This TA activity was implemented as a response to a request from MOH. Following reports of misappropriation of grant funds from Global Fund and GAVI, as well as recognizing the need to improve and scrutinise infrastructure development, MOH asked for USAID assistance to address these challenges. Since GAPP has been working on addressing public accountability and governance issues, it was deemed a suitable program to host the proposed TA. The following assessment is based on a focused group discussion held with the MOH experts in Kampala during the evaluation interviews, as well as a review of the documents and spreadsheets the experts have produced to record the cost savings. Economy: MOH TA activity started in 2018 upon request from MOH. A total of 8 experts were recruited and seconded to MOH, to be supervised by the Permanent Secretary. The team comprised of a legal advisor, civil engineers and a mechanical engineer, as well as a public finance management specialist in the Planning Department of MOH. 2 of the experts finalized their work at the end of September 2019, and it is understood that the remaining 6 will continue to be engaged during the 17 month extension to be funded by DFID. To date, $ 1,161,837 was spent on the TA according to GAPP reporting24. Efficiency: In terms of speed of delivery, the unit of experts has functioned efficiently, delivering the savings expected, and recording a significant UGX 16bn (US$4.36m) in a little over a year. The cost savings originated mainly, but not exclusively from the recurrent expenditure budget items of MOH. It was reported that among the experts engaged, the legal advisor and the engineers were the first of their cadres to fulfil those tasks at MOH. It was also reported by the experts that this unit was the first of its kind to be tasked with efficiency savings and other quality checks, among the line ministries in Uganda. In this sense, the model represents an innovation. The savings achieved originated from various sources. For instance, through a review of MOH contracts and obligations, and improving contracts management at the Ministry, MOH saved close to UGX 8bn (> US$ 2m) in avoided fines and out of court settlements. There is now a delegated Contracts Committee in place, which is functioning and there has been no indication of it soliciting bribes from suppliers25. The civil engineers have worked on preparing bid documents and bills of quantities for various infrastructure items for MOH. By carefully reviewing bills of work, they were able to expand the scope of works, for example, carrying out physical upgrades of 308 health facilities across Uganda, rather than the 285 planned originally. Similar savings were achieved through better enforcing of maintenance clauses in contracts, for example for solar installations. Effectiveness: This activity has provided significant returns in the form of public efficiency savings, in a relatively short period of time. The current rate of return (Benefit to Cost Ratio) would be 3.7526 24 This reportedly included indirect expenses. It excludes CLA related costs, which were not assigned to program components. 25 Having functioning Contracts Committees by each procuring public agency is one of the key recommendations of the Comprehensive Procurement Assessment Review (CPAR) of World Bank in Uganda. 26 Using the savings reported now (UGX 16bn) as the benefit and total GAPP costs as cost, however this calculation leaves out the GoU costs which are not known, and a share of CLA costs from GAPP. We estimate the CLA costs for MOH TA would be small in this instance. 77 | GAPP EVALUATION REPORT ANNEXES USAID.GOV undiscounted27. For benefits, we used public cost savings, however a caveat needs to be presented. Ultimately, saving a public agency public funds would signify a benefit to citizens when those savings are used, by public authorities, for improving service delivery, with the objective of providing better health, education or similar wellbeing outcomes. We cannot yet follow this chain of causality for the MOH activity. However, it is likely that for example, servicing of a larger number of health centers will ultimately benefit Ugandans using those facilities and help improve the health outcomes in catchment areas. To put the UGX 16bn of cost savings in perspective, FY2019/20 national budget MOH allocation was studied. This amount corresponds to combined budget allocations of Uganda AIDS Commission and Kabale Referral Hospital28 for that year, which is very significant. One additional benefit that can be considered here is extending the benefits of other development partner programs in Uganda. Upgrading of rural health facilities mentioned above is concerning World Bank’s Intra-Government Fiscal Transfers Program with total funding of US$200m. MOH experts help that program deliver more by expanding the scope of upgrading works for health centers, for the same level of maintenance funds. Equity: In terms of organizational gender equity, two members of the MOH team are female. In terms of outreach, we maintain that VfM equity principles are not well-suited to apply to this indirect delivery model. However, scrutinizing procurement processes affirmatively through a gender, disability or other focus at MOH is something the group of experts can promote going forward. Sustainability: It is understood from the discussions that there is potential to achieve a lot more savings at MOH through the use of the established systems such as the Contracts Committee, and better quality procurement and contracts enforcement. One civil engineer estimated that they were able to save the MOH at times up to 50% of the infrastructure costs per unit. It is unclear how well these outcomes can be sustained beyond the life of the engagement of this small group of experts. The experts often collaborate with their civil servant counterparts and it is likely that there is a good level of knowledge and capacity transfer. However, motivation through a much higher pay scale is potentially something that MOH will not be able to provide for their cadres for the time being. In conclusion, MOH TA most likely represented good returns for the investment by USAID and DFID, with evidence of attribution supported by feedback and analysis of savings. Embedded Technical Specialists (ETSs): The model of provision of long term TA support to LGs in GAPP districts, rather than short term and ad hoc TA support as was originally planned, has arguably made a very significant contribution to the achievements of GAPP. The details of ETS support and related outcomes in governance systems of LGs are also discussed elsewhere in this report, in response to other EQs. Here we will focus on the ETS model specifically for the 4 VfM principles. Economy: There were 15 ETSs working under GAPP, 3 for each region, and 1 representing each category of TA themes of leadership, financial management and local revenue, planning and budgeting. The total number of staff working on LG component of GAPP (component 2) was 18 and included a dedicated M&E specialist. Currently one of the former ETSs for Acholi region is seconded to the MoLG, working on equipping MoLG with the oversight, technical support and policy direction required to fulfil its mandate. We do not have the exact isolated costs for the ETS model (and that would have been a 27 Undiscounted because it has only been over a year in terms of benefit duration. For longer periods of time, costs and benefits need to be discounted, most likely at 10% for Uganda context. 28 https://budget.go.ug 78 | GAPP EVALUATION REPORT ANNEXES USAID.GOV highly difficult task to do retrospectively) however it is estimated to be a large proportion of the component 2 costs provided by GAPP finance teams, which was close to US$ 12,4m29. Efficiency: The ETS model being based on regions, rather than each district, meant that each ETS would need to cover 6-7 districts in one region, signifying a large workload. It also meant that this improved efficiency because one ETS served many districts, including lower LG structures. The outreach is shown below: Local Government Structure GAPP Outreach (September 2019) Districts 40 Municipalities 14 Sub-counties 317 Divisions 41 Town Councils 49 Higher Local Gov Leaders 2,865 Lower Local Gov Leaders 5,037 A total of 7,902 people were reached through various modes of capacity transfers under the ETS model, according to GAPP reporting. ‘Cost of outreach (capacity building beneficiary)’ can be calculated if total costs of ETS are known. Going forward, in the next phase of governance programming, this metric could be considered as an efficiency indicator. Tracking this metric would make sense also because similar capacity building support is likely to be provided over time, giving opportunities to compare trends internally, as well as with other governance programs as it is a commonly used metric. Finally, number of people trained is one of the USAID key global indicators and will be tracked for each program. Another outreach metric that can be calculated is ‘program investment in each district’, or ‘cost of GAPP support in each GAPP district’. Using component 2 total costs at around US$12.3m and 40 districts reached, we find that around US$310k was spent on accountability improvements in districts. This is an average, and some districts will have incurred a higher, or a lower cost. It would however give an indication of the level of benefits needed to break even per district level investment. Effectiveness: ETS’s work under the component 2 contributed to majority of GAPP’s results in that area. Not many of these results lend themselves to monetisation for a VfM analysis. The recorded revenue increases for the target districts amounted to US$1m over a period of 3 years until 2016. It is potentially challenging to attribute revenue increases exclusively to GAPP work, however GAPP was the only external support that the districts received in revenue and accountability area, as confirmed by majority of the interviews and the Evaluation survey results. This period corresponds to before the ETS model was introduced, so it could be attributable to GAPP and not to ETSs. GAPP have not since tracked revenue increase results, as there was a reform introduced, putting URA in charge of local revenues. In any case, the $1m revenue increase would likely not have been sufficient for the investment in that component to break even30. However, it is arguably a substantial amount in its own right, considering the narrow base of local revenues in a low-income country. Revenue increases and potential savings through improved procurement are two substantial ways that benefits from component 2 work could be monetised. Improvements in national surveys and indices as in LGPA are also useful, however the changes in those scores are not possible to translate into monetary 29 As at end August, 2019. 30 Considering the total LG component cost of $12.3m over 7 years. 79 | GAPP EVALUATION REPORT ANNEXES USAID.GOV benefits. Therefore future programming should also consider monitoring local revenue increases or procurement savings in order to help with more robust VfM narratives. The calculation challenges notwithstanding, ETS model has been instrumental in delivering the summarized results. These were explained also under the outcome harvesting results, and for questions 2.1 and 2.2 above. o Improved public accountability systems (through improved audit processes- for example a significant increase in unqualified audit opinions in GAPP LGs, from 34% to 80%, as well as through more functional LGPACs, from around 5% in 2013 to 74% in 201831). o The survey results indicate a high level of attribution for systems improvement to ETS work. What would be explicitly attributable to GAPP is that ETSs made trainings more effective through on the job coaching and mentoring. o Improved revenue collections systems (through better tax assessments and collection) and better local revenue mobilization, if not in all districts, in a number of them. o Better coordination of refugee response. o Reduced conflict between elected and technical staff. This was also a clear attribute to ETSs work, evident from the survey and interviews during evaluation. Equity: In terms of organizational gender equity, one ETS out of 15 was female. For equitable gender outreach, it was reported by GAPP that 356 local women officials were trained in leadership. In addition, support to GAPP districts by ETSs on refugee response coordination and planning can be considered under equity performance, as this dimension of VfM is concerned with reach on all vulnerable groups, and not only women. Sustainability: None of the results above is possible to monetize at the moment, however this does not mean that they did not provide value. On the contrary, the ETS model can be credited with generating a successful legacy for GAPP and therefore representing potentially good value for money for the original investment by USAID and DFID, especially if the outcomes are sustained going forward. The ETS placed by GAPP in MoLG is likely to be an important pathway to sustainability because he will work with this Ministry who is going to lead the ‘Local Government Sector’ in Uganda. In the refugee hosting districts, the ETS support will continue during the Extension phase, and it is recommended that a new governance program carries on with this modality of support, with the districts to be chosen, considering the success criteria explained above in 2.2. 3.3 What are the potential VfM highlights and lessons regarding these three models learned for similar follow-on programs and wider audiences? Even though GAPP has operated without a VfM strategy or a framework, there have been a number of VfM related lessons learned. GAPP has done limited reporting on VfM, focusing on cost savings through procurement or other cost-conscious behavior, as explained above. The VfM highlights below are derived from key stakeholder and beneficiary interviews, as well as program documentation. GAPP has demonstrated good examples of adaptive programming and flexibility, which speaks to VfM Efficiency and Effectiveness. Key stakeholder feedback indicated that even though the MOH TA seemed as an add-on, and disconnected from the program Theory of Change, GAPP was efficient in its response to taking on this work stream, which provided a distinct example of adaptive programming. MOH technical assistance was implemented with a high degree of success and with evidence of cost savings of public funds, as discussed above. 31 GAPP Program Results: Fact Sheet 2012-2019, p.3. 80 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Relationship building is just as important as capacity building, as demonstrated by the tripartite model. Much of the positive feedback on GAPP’s work with NAIs emphasised the fact that improving the relationship between OAG and PPDA underpinned the results achieved under component 1. This also relates to effectiveness because after GAPP or any other development activity comes to an end, the institutions will still carry on their work. Also important to note from a standpoint of nurturing close working relationships, was the ETS experience. ETSs relationships with their LG counterparts and the careful balance they have managed to strike between professional delivery and courteous mentorship over long periods of time have enabled the results under Component 2. Development partners planning, implementing and delivering together constitutes a VfM good practice. USAID and DFID working together on GAPP have meant time savings and other cost savings for each partner, therefore improving Economy. It has mostly likely saved efforts also to the Government of Uganda counterparts, both national and local. It has arguably extended each development partner’s influence further, by giving the right messages on donor coordination and effectiveness. The two development partners are planning the next phase of governance program together in Uganda, which underscores this collaboration as a possible win-win. It is crucial for GAPP to consider and plan sustainability pathways for the outcomes achieved so far. Sustainability is a sub-set of VfM Effectiveness, and for some development practitioners increasingly seen as an additional pillar to 4Es. As an example, for designing pathways for sustainability of achievements, over its last year, GAPP saw that MoLG was emerging as an important actor whose influence needed to be enhanced. An ETS who was working in Acholi was placed in the Ministry over the past year, recognizing that for sustainability of the program work thus far with LGs by the ETSs, MoLG capacity needed to be strengthened. Because what GAPP do with local government level accountability and revenue systems is in fact MoLG’s mandate. Further, MoLG’s profile as a leader in ‘local government sector’32 can benefit from GAPP experience and expertise. GAPP has potentially brought innovation in public accountability work in Uganda. In some ways, all three GAPP models explored here provide examples of innovation, which is relevant for VfM Efficiency. Tripartite meetings that were never held before and that brought main national accountability institutions closer; using ETSs for TA delivery on long term basis; and finally MOH TA each constitute an innovation. MOH experts told the evaluation team that they are the only unit working exclusively on cost savings among the line ministries they knew of. In this sense, it sets an example to be followed by other public agencies. Having local staff working in highly local content interventions has arguably enhanced GAPP’s VfM effectiveness. 65 out of GAPP’s 66 members of staff33 are Ugandan, with RTI staff supporting from Kenya and the US. The COP and all technical leaders of program components are Ugandan. There was high praise for GAPP senior management, as well as for the lower level technical staff from a high number of respondents during evaluation interviews. There are undoubtedly universal aspects of public accountability work, however many of the issues are context-specific, and arguably better understood by local staff. Furthermore, the program manager on USAID side, as well as the Senior Responsible Officer for GAPP from DFID were also Ugandan and had been working on local governance issues in Uganda in general, and GAPP in particular, for a long time. Conclusions: 1. GAPP did not start with a VfM framework and did not have one during implementation. There are no periodic indicators that we can analyse by trends, or a cost benefit analysis we can perform overall on results, using programmatic data. In order to answer VfM questions, we 32 This is a new sector formed by the Cabinet, and will work alongside the 16 other sectors in Uganda, such as health, education, water, agriculture, etc. It will bring together key government departments and development partners working in local government and decentralization. 33 Last count at July 2019, and before recent staff attrition, which is routine for programs nearing their end. 81 | GAPP EVALUATION REPORT ANNEXES USAID.GOV examined sub-categories of the main 4 VfM areas and sought answers as to whether GAPP had the systems to deliver on the VfM principles. Further, triangulating program costs and results data, feedback from interviews and survey findings, we reached the VfM conclusions. 2. We find that on the 3 models examined here, there was potentially value generated by GAPP. The ETS model can be credited with generating a successful legacy for GAPP and therefore representing potentially good value for money for the original investment by USAID and DFID, especially if the outcomes are sustained going forward. For the MOH TA, over the past year, public efficiency savings well over and above the costs of that TA have been delivered, with apparent potential to deliver more, and therefore indicating good value for money. Finally on the TOM, while we cannot quantify the outcomes and have limited information on the costs, based on key feedback we find that they can be credited to a considerable degree, with improving public accountability and citizen participation. Commitment from government counterparts in the form of cost sharing of the meetings is considered evidence to recognition of the model. Finally, all three models represented examples for innovation in public accountability work in Uganda context, as discussed under Q 3.3. 3. For overall program, we assessed and verified the existence of systems, however find it more challenging to make program-wide and confident VfM conclusions. This is due to lack of detailed understanding on the costs and benefits levels to compare. GAPP posed additional difficulties in this regard because voice and accountability sphere are universally challenging to use quantified economic analyses in. Recommendations: It is advisable to maintain a central ‘VfM Register’ for cost savings to be recorded more systematically. GAPP’s VfM reporting included examples of savings but often without monetary values. As numbers are more influential in this type of reporting, it is recommended that future programs maintain a spreadsheet of such savings, which various teams (and not just finance team) can contribute to. Finally, it is not only DFID who are interested in showcasing ‘bang for buck’ for their programs, and USAID are also increasingly exploring ways to capture and report value. Value does not have to be only in the form of costs saved, but also in benefits generated. It is important to attempt to understand and demonstrate value, in monetary terms where possible, for all national and international audiences. VfM Case Studies should be used in reporting. Governance and accountability type of programming is often not suitable for conducting overall economic analyses. In addition, voice and citizen participation interventions feature outcomes that are not necessarily credible to quantify or monetize. Unless benefits are monetized, it poses challenges to compare costs and reach VfM conclusions. What would be a good way to address VfM for these programs could be ‘VfM Case Studies’ where the costs and benefits of one activity or intervention in the program are isolated and focused on. A template is provided in table 4 for future programs. The premise is that by telling a VfM story in the case study format, value generation and VfM effectiveness is inferred on a case basis, if not for the whole program. For example, MyTax pilot in Masindi, or Buliisa revenue assessment34 examples would potentially lend themselves to a good VfM Case Study. In each case, costs incurred would need to be determined, as well as the estimated or actual monetary benefits, with evidence of source. MOH TA also offers this opportunity, and a simple calculation for this was provided above under Question 3.1. Activity Headline VfM Finding Total costs incurred How does this intervention factor in 34 GAPP Quarterly Report for Dec 2018-March 2019, April 2019, p.13. 82 | GAPP EVALUATION REPORT ANNEXES USAID.GOV sustainability? Is there an exit strategy? How does this inform development partners about future resource allocation? How can lessons be used in wider country/ regional context? Who has benefitted from the activity/ project? Are there indirect beneficiaries? What are the economic benefits realised? What are the social benefits realised? Are there any unintended/ negative consequences? What made this activity successful? Use of VfM Metrics: A number of examples were provided above in text, and others were added here to demonstrate what type of metrics can be used for VfM reporting by programs such as GAPP. They are presented in a tabular format to show sources and responsibilities for reporting. The finance teams will have to assume responsibility for calculating costs against the results proposed, and they will be expected to work in collaboration with MEL team and program management. Finally, the metrics were chosen among those suitable for governance programming. VfM Domain Indicator Reporting Responsibility Remarks/ Evidence/ Source Economy Cost savings (tracked and monetised) examples though procurement or other cost-conscious behaviour. Program management/ finance lead Quantified or monetary records of how much the savings amount to. Economy Average Daily Fee Rates on Consultancy/ Technical Assistance Finance team Charge out rates to USAID/ DFID. Economy % of management or administrative costs on overall spend Finance team Actuals to be reported on program management, to include all costs related. Efficiency Percentage variance of actual spend from budgeted figures. Finance team This is a proxy VfM indicator and it stipulates that the lower the difference, the more efficient financial management. Efficiency Overall fund utilisation to date (or utilisation vs time lapsed on the programme). Finance team As above, as an indication of efficient programme management. Efficiency Cost of Outreach (capacity building activities) MEL/ Finance team As a proxy for outreach efficiency. Trends to be analysed for conclusion on efficiency performance. 83 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Efficiency Leveraging of other funding (Government or Development Partner) MEL team Quantified and monetised results. Efficiency Resource used by or Program investment in each district Finance team This can be a good initial step in comparing the benefits and results achieved in a particular district supported. Effectiveness Efficiency savings or additional revenues generated (quantified or monetised) MEL team Such as from saved expenditure, modernisation and IT related improvements, etc. Examples provided above in the body of report. Effectiveness Evidence of enhanced Capacities for counterparts. MEL team Can be qualitative. Examples of these are behaviour changes in people, perceptions, direct beneficiaries, etc. Equity Evidence or case studies of equitable reach/ contributions. MEL team VfM Good Practices (all 4 Es) Examples to be compiled and documented. 84 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 11: PROCESS MAPS DISTRICT LG AUDIT REVIEW PROCESS 85 | GAPP EVALUATION REPORT ANNEXES USAID.GOV NATIONAL LG AUDIT REVIEW PROCESS 86 | GAPP EVALUATION REPORT ANNEXES USAID.GOV 87 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ANNEX 12: OUTCOME SUMMARIES LIST OF EVALUATION OUTCOMES Outcome #1. Stronger Collaboration Among Tripartite Members (OAG, PAC LG, PPDA, and DPAC) to Promote Accountability 88 Outcome #2. Greater Percentage of GAPP Supported Districts Receive Annual PPDA Audit 90 Outcome #3. Parliamentary Accountability Committee (PAC LG) Gives Timely Consideration of OAG Reports 923 Outcome #4. Parliamentary Rules Amended to Strengthen Treasury Memorandum Mechanism 944 Outcome #5. CSOS Influence Transparency and Accountability Policies in the Extractive Sector 95 Outcome #6. Funds Earmarked for Local Governments Withheld by Central Government Released Due to CSOs Research 97 Outcome #7. Local Councils Accomplish their Agendas with Broader Participation 988 Outcome #8. Local Councils Pass Annual District Budgets and Approve Annual Work Plans 100100 Outcome #9. Relationship between Politicians and Technical Staff Improved 101100 Outcome #10. Internal Audit Sanctioning of Financial Mismanagement Improved 1033 Outcome #11. Performance of District Public Accounts Committee Improved 1045 Outcome #12. Local Government Organizes the LRM Process more Efficiently 1077 Outcome #13. Local Government Planning More Integrated 1078 Outcome #14. Gender Budgeting Perspectives Included 109 Outcome #15. Local Government Improves Mechanisms for Accountability to Citizens 11010 Outcome #16. Improvements In Health, Education, Agriculture and Wash Public Services due to Citizen Advocacy 1111 Outcome #17. Citizens Formulate and Present Petitions to Local Government Regarding Service Delivery Improvements 1133 Outcome #18. Private Sector Associations Lobby for Service Delivery Improvements 1144 Outcome #19. CSOS Attract New Funding due to Strengthened Organizations Systems 1166 Outcome #20. Citizens More Willing to Pay Taxes 1167 88 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #1. STRONGER COLLABORATION AMONG TRIPARTITE MEMBERS (OAG, PAC LG, PPDA, and DPAC) TO PROMOTE ACCOUNTABILITY Outcome Description: Strong collaborations have been forged among the key stakeholders in the accountability chain; PPDA, OAG, Parliament, DPACs and Ministry of Finance collaborated through the tripartite review of local government audit reports. Previously, these departments implemented related procedures in parallel. Parliamentary hearings were held in Kampala and attracted very little attention and oversight. As a result, this accountability system provided very little accountability. GAPP began its support to these departments in 2013 by introducing the tripartite model. Over the course of this engagement, the model has changed and become established within the ministries as a valuable model to continue. It has helped both DPACs and technical staff better understand the process and the roles of others, thereby facilitating coordination among the bodies. It also created a better flow of information and cooperation between national and district government bodies. In the last two meetings with PAC LG in 2020, the number of Parliament LGAC MPs going to the districts increased. DPACs would also attend committee meetings of Parliament in the districts, funded by GAPP. The LGAC raised issues within and outside of the Committee report and even send members to verify concerns relating to service delivery at the district level. The DPACs would also raise issues on accountability. This joint approach to business built the capacity of DPACs, seeing how the LCCAC operates in its sittings, and offered an invaluable coaching/mentoring process. Non￾PAC MPs from the regions convening would attend and were helped to understand the accountability issues from their region. Significance: Improving the relationship between PPDA and OAG resulted in them working better together to deliver more integrated reports to Parliament on the status of accountability of LGs. Tripartite meetings helped pass on knowledge and skills to DPACs and also improve the environment of accountability at LG level. Citizen participation in the tripartite meetings was achieved through their sitting in and reporting/whistleblowing on the areas of lack of accountability. GAPP Contribution: HIGH GAPP initially introduced this model early in the program as a way for district and national governments to work together. The model evolved over time to involve public hearings in the districts where all parties and the public could participate. GAPP covered the expenses and logistics of travel for officials from Kampala to attend, as it was not in their budgets. They also funded the technical assistance support provided by CEPA in facilitating the meetings. At the end of the project, the OAG has committed to including these meetings in its budget going forward. One interviewee noted that the underlying level of change is that PPDA’s relationships changed – this is most significant; capacity and policies already existed – but the new relationships shifted execution. 89 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Other Contribution Claims GAPP partner CEPA played a significant role in providing full time technical assistance where one member staff sat in on all the PAC LG meetings in Parliament and in the districts where the Tripartite convened meetings. CEPA coordinated the tripartite meetings with GAPP, planning with the PAC committees, working on schedules, and writing internal tripartite program reports. Staff from CEPA was the equivalent of the ETS technical support to PAC LG; she would sit in planning meetings with Clerk to Committee, Chair and plan for excursions, submit Reflections, and attend meetings with third parties, e.g. OAG. Sources of Evidence: ● GAPP, Program Results: Fact Sheets 2012 – 2019 ● GAPP, Strengthening Collaboration amongst National Accountability Institutions for Greater Local Government Accountability ● Interviews with PPDA and OAG staff ● Interview with CEPA staff 90 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #2. GREATER PERCENTAGE OF GAPP SUPPORTED DISTRICTS RECEIVE ANNUAL PPDA AUDIT Outcome Description The Public Procurement and Disposal of Public Assets Authority (PPDA) is responsible for holding 386 entities (of which 134 local governments are part) accountable for spending of public money through annual procurement audits. With inadequate resources to conduct the audits (funding and staff), the Authority relied on outsourcing audits to consultants; it was not able to keep up with responsibilities. PPDA had fallen 5 years behind in audits and could not give adequate follow-up and capacity support to districts. GAPP started its assistance to PPDA in 2013 in 25 districts. This was later increased as the program expanded coverage. With support from GAPP, PPDA undertook two improvement strategies: 1. They built a cadre of staff able to complete audits in-house while ensuring quality of series delivered. PPDA now has fully trained and qualified staff that carryout audits and inspections directly. They also have funds in their budget to support each audit (transport, accommodation, materials); 2. They documented and increased use of Procurement Inspections (lower scale and scope than audits) for entities with smaller budgets. Inspections were previously underutilized, as the practice was not well documented and promoted. Currently, PPDA performs either an audit or inspection in every entity every year or two years. In addition to more frequent audits, PPDA is also able to give more follow up support by producing and following-up on an action plan to support [RS1] improvement. Additionally, according to PPDA, the cost of conducting PPDA procurement audits reduced from UGX 35 million to an average of UGX 4 million, and UGX 2 million for inspections. Significance As a result of increased scrutiny, local government personnel are becoming more accountable for spending public money on services. With less corruption and/or mismanagement, public monies are being better used for the intended purpose of providing public services as evidenced by the following two indicators (from GAPP): ● PPDA procurement audits increased from 28% in FY 2011/12 (17 of 25 LGs) to 78% in FY 2018/19 (31 of 40 LGs) in GAPP supported districts; ● Percentage of LGs returning unqualified audit opinions increased from 34% FY 13 to 80% FY 18. GAPP Contribution: HIGH The project initially conducted an assessment to determine where capacity and efficiency blockages were located. They then provided funding and ongoing technical assistance through coaching for: ● Training PPDA staff to do in-house audits; and, 91 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ● Documenting and rolling-out the alternate inspection model. Other Contribution Claims None. No other donor supports PPDA, so these improvements would not have been achieved. Sources of Evidence: ● Interview with Manager, Procurement Audit, PPDA 9/16/2019 ● GAPP staff focus group discussion ● GAPP Annual Report 2017 92 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #3. PARLIAMENTARY ACCOUNTABILITY COMMITTEE (PAC LG) GIVES TIMELY CONSIDERATION OF OAG REPORTS Outcome statement: The Parliamentary Accountability Committee now gives timely consideration and tabling of reports within 6 months, as constitutionally mandated. (However, substantial backlogs still exist in COSASE.) Before GAPP, the OAG was conducting audits and presenting reports to parliament which were being considered out of designated timelines creating a big backlog. Seven years would pass without PAC LG, PAC central government and COSASE looking at the OAG reports. As a result, whatever happened in the LGs would largely go unnoticed. A CEPA research study recommended working through omnibus reporting. Now accountability committees are staffed with two to three clerks. Splitting the PAC LG Committee into sub committees produced one report on districts and municipalities in 6 months in 2017 for the FY 2014/14-year report. From 2016 to Feb 2019 PAC met the mandate and cleared the backlog from 2010; now PAC can review the OAG report within 6 months from commencing work With GAPP funding in 2013, CEPA conducted research to analyze and understand the backlog in each committee, and causes and solutions, i.e., how to unblock issues at the national level referencing the GAPP theory of change. The assessment revealed that the OAG reports were huge and complex and there were no standard procedures for the PACs to assess those reports. Though the OAG reports were directed to the 3 specific committees mentioned, the PACs were working randomly in an ad hoc manner with no standard approach; some committees had only looked at 5 out of 100 reports while COSASE had not produced any report for 2.5 years in the 9th Parliament and had a backlog of reports of about 10 years in 2013. CEPA advocated for the 14 recommendations in the study, holding meetings with PAC’s Chairs to implement recommendations. PAC central government picked one component of the 3 CEPA recommendations on the consideration of the OAG report based on the opinion of the OAG (disclaimer, adverse). COSASE adopted the omnibus reporting and LG PACs adopted splitting into sub committees. The recommendations were captured in the Handbook. CEPA met the clerk of parliament to lobby for an increase of PAC clerks, and when Parliament increased its staffing, the priority went to accountability committees, each of which got 2 clerks. The mid-term review noted that the work of parliamentary committees was improved through better business processes and resulted in more work being done, e.g., ● PAC-LG work was conducted in sub committees, increasing the volume of outputs ● The COSASE and PAC began focusing on key outstanding audit issues rather than considering every query; i.e., PAC prioritized the most critical audit issues ● All PACs started enforcing rules to discourage diversion of committee business to issues outside the OAG report ● Committees enforced rules to discourage diversion of committee business to issues outside the OAG report Significance 93 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Clearing outstanding reports by the PAC LG was critical for the PACs to be able to flag past and current areas of accountability flaws in service delivery at LG and central level. The backlog situation had blocked the ability to address salient action points in OAG reports for the fiscal year in which they were identified. One interviewee noted that the underlying level of change is that PPDA’s relationships changed – this is most significant; capacity and policies existed – the new relationships shifted implementation. A validation source said that the GAPP supported support to the OAG through CEPA was instrumental to progress. Arranging the working modalities of the PAC LG committee and prioritizing areas of focus while increasing the number of clerks was significant in that PAL LG was able to work in a timely fashion and attend to current and past outstanding OAG reports. GAPP Contribution: MEDIUM GAPP gave funding for technical support and research to clear the 7-year backlog of report reviews. DAI sent an employee to help work with parliament to analyze blockages in the PACs and strengthen the internal Parliamentary Rules of Procedure over the course of 3 years. GAPP funded CEPA to research (2013) improvements to PAC processes and provide a consultant to develop a Handbook for Parliamentary Accountability for MPs. CEPA then mentored the 10th parliament and 3 PACs with GAPP funding. CEPA additionally funded media training for journalists to improve their understanding of accountability issues in relation to service delivery. Other Contribution Claims Parliamentary members received orientation from other actors hoping to influence accountability in the House. Therefore, there were additional organizations contributing to these significant results and it is difficult to isolate GAPP’s contribution. Sources of Evidence: ● Interview with CEPA ● Interview with Ugandan journalist ● GAPP staff focus group discussion ● GAPP, GAPP Program Results: Fact Sheet 2012 – 2019 ● Mid-term evaluation of USAID/Uganda and DFID governance, accountability, participation and performance (GAPP) activity 94 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #4. PARLIAMENTARY RULES AMENDED TO STRENGTHEN TREASURY MEMORANDUM MECHANISM Outcome statement: CEPA advocacy in 2015 prompted amendments to the rules for how Treasury Memoranda are referred from Parliament to MoFPED, improving the quality and efficiency of PAC LG queries and deliberations in Parliament. Under the PFMA 2015, a Treasury memorandum is an action report by MoFPED detailing the actions taken on the recommendations of Parliament arising out of the report of the Auditor￾General. By law, the Minister of Finance is to submit a Treasury memorandum to Parliament within six months from the date of Parliament’s consideration of the OAG’s report, indicating measures taken to implement the recommendations of Parliament in respect to OAG’s report of the preceding financial year. However, before GAPP, several Treasury Memoranda queries debated from the floor and the ensuing Parliamentary resolutions to MoFPED were not being directed to MoFPED by Parliament and no action was being taken on them; neither would OAG audit them. Also, Treasury Memorandum that were submitted by MoFPED were found by CEPA to be vague and not reflecting what parliament had recommended. CEPA lobbied the clerk to start submitting accountability resolutions passed on the floor to MoFPED. CEPA urged the Speaker and Accountant General, who explained that they were overwhelmed and understaffed, to verify accountability issues unless huge monetary sums were involved. CEPA appeared before the Rules committee and explained that there was no clear procedure for how Treasury Memoranda are referred from Parliament to MoFPED; this flaw was amended and is now reflected in the current rules of Parliament. Now, Treasury Memoranda are being audited as is the Treasury. Significance Strengthening the Treasury Memorandum mechanism was important in ensuring that PAC LG’s oversight function over accountability for service delivery is not undermined by failure of the executive to act on Parliamentary resolutions. GAPP Contribution: MEDIUM GAPP funded CEPA to research the issue and advocate for improved governance procedures. Funding was later extended for CEPA to support parliament and OAG, building capacities of LG PACs, OAG, and Parliament. Other Contribution Claims The work done by CEPA under GAPP funding and support was significant to this outcome. However, the exact contribution and its significance could not be verified by OAG or MoFED. Sources of Evidence: ● Interview with CEPA staff ● GAPP staff focus group discussion 95 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #5. CSOS INFLUENCE TRANSPARENCY AND ACCOUNTABILITY POLICIES IN THE EXTRACTIVE SECTOR Outcome statement: Due to advocacy efforts of a coalition of CSOs organized by ACODE, MoE&MD and MoFPED incorporated recommended provisions into the bill to promote transparency and accountability regarding licensing and mining. Originally, laws overseeing extractive industries in Uganda lacked provisions on transparency and accountability (T&A). For example, the 2003 Mining Act lacked any provision on transparency in mining licenses. CSOs such as ACODE had been working on decentralization issues for years but had not engaged MoLG. With funding and support from GAPP, ACODE lead an effort of intense research and organized CSO advocacy and lobbying efforts in 2017-18. They engaged organizations that traditionally had not worked specifically on T&A in the extractives sector, e.g. Uganda Law Society, and involved LGs, too. The Civil Society Coalition on Oil and Gas also came on board, and all these actors reinvigorated the debate on T&A. As a result, ● In January 2019, government took the initial steps to join EITI, announced the appointment of a national coordinator, and established a multi stakeholder group to oversee implementation of EITI ● Commitments were made by MoE&MD and MoFPED to operationalize legal and policy provisions to promote transparency and accountability in the extractives sector. In 2018 MoFPED developed a cabinet memo advocating for the provisions of the National Oil and Gas Policy, and Oil and Gas Revenue Policy and in January 2019 the government agreed to start operationalizing these provisions ● The Minister of Energy and Minerals incorporated ACODE’s draft provisions on the 2019 Mining and Minerals Bill regarding provisions on competitive licensing of mining blocks and this was incorporated in the Bill; ACODE urged the government to launch the Mining and Minerals Bill on 9th August 2019 ● Government and the Joint Venture Partners in the oil and gas sector committed to more space for stakeholder involvement and to involve a wide category of stakeholders in information sharing sessions An additional result was that ACODE was nominated by CSOs as one of 3 CSOs to join the multi￾stakeholder group and has successfully expanded the group from 3 CSO reps to 5 CSO reps, including representatives from artisanal miners. The multi stakeholder group is now representative of citizen interests as well as those of government and oil companies. CSOs, including ACODE, are now integrated in critical government decision making structures and processes for accountability in the extractives sector. When MOFPED was developing the TOR for the multi-stakeholder group, they wrote to ACODE for guidance on EITI. ACODE sent them a draft TOR for review and input in July 2019. When the global conference on EITI was convening in June 2019, ACODE was included in the national delegation for the first time. Significance: CSOs are the citizenry in action demanding transparency in service delivery and fiscal management. In the extractives sector, especially with the discovery of oil and gas deposits, there had been 96 | GAPP EVALUATION REPORT ANNEXES USAID.GOV challenges with transparency in the award of licenses. By working closely with government and being co-opted into certain spheres of influence and decision making, CSOs were able to persuade government to adopt strategies, reforms, and actions that would promote transparency and accountability. GAPP Contribution: Unverified GAPP funded ACODE to conduct research, organize dialogues, and engage media in a campaign, and engaged and trained CSOs to advocate for the proposed revisions to the bill. GAPP also funded the National Decentralization Conference of 13 August 2019. ACODE co-organized events with the MoLG that led to the Bank of Uganda directive. Funded by GAPP, ACODE organized dialogues with private sector organizations, MoFPED, McEnery and Mineral Development (MoE&MD), the Petroleum Authority, the National Chamber of mines and 3 major oil companies, Total, Tullow, CNOOC, and MPs and leaders of LGs. The debate was on operationalizing the policies on extractives in a transparent manner. Press conferences were held by ACODE that largely focused on the demand for T&A, with CSOs mobilized by ACODE making a call for government to adhere to these principles. These events were from 2018 to 2019. ACODE twice convened CSOs and trained them on how to organize around EITI using consultants paid for by GAPP. ACODE also shared information with CSOs and convened them around EITI advocacy issues. ACODE mapped out other organizations not necessarily working on 0il and mining, e.g. The Anti-Corruption Coalition, and CS BAG and others, to flag T&A issues in early 2019. From July to October, November 2018 and July 2019 ACODE conducted media campaigns which included radio talk shows, press conferences and publication and print funded by GAPP. They highlighted issues that would otherwise have passed without public engagement, e.g. when Government passed the resolution to join EITI, the media never flagged it. Other Contribution Claims This contribution is significant, however, the contribution of ACODE, and GAPP for providing funding, could not be externally verified beyond those two players. Sources of Evidence: ● Interview with ACODE ● ACODE, Transparency and Accountability in Uganda’s Extractive Industries Sector End of Project Report 97 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #6. FUNDS EARMARKED FOR LOCAL GOVERNMENTS WITHHELD BY CENTRAL GOVERNMENT RELEASED DUE TO CSO RESEARCH Outcome statement: A research study done by ACODE revealed that funds earmarked for local governments were held in the MoFPED. Two weeks after the Permanent Secretary, Secretary of the Treasury, and two other ministries promised to act in response to a presentation of the study at a 2019 conference, the directive to release the funds was issued. Significance: The availability of funding is a major challenge for performance of local governments. The successful release of funding from central government due to intervention from high levels is quite significant. GAPP Contribution: UNVERIFIED GAPP gave funding to ACODE for the funding and organization of the decentralization conference where the findings were presented. Neither would likely have taken place without this support. Other Contribution Claims This contribution is significant, however, the contribution of ACODE and to GAPP for providing funding could not be externally verified beyond those two players. Sources of Evidence: ● Interview with ACODE ● ACODE final report 98 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #7. LOCAL COUNCILS ACCOMPLISH THEIR AGENDAS WITH BROADER PARTICIPATION Outcome statement: With support from GAPP, local councils are more effective and efficient in carrying out their role. Before 2013, council meetings often took one to three days. Councillors would spend time discussing what needs to be done, reading documents, and debating allowances and when they would get compensated. With GAPP training and support, council meeting times reported in case districts reduced to two to four hours. Rather than the previous open discussion and debate, meetings now follow official process protocols. The agenda is set in advance by the Clerk to Council, who also organizes any reports to be reviewed (e.g., DPAC reports). Councillors know their roles and responsibilities and come prepared to deliberate and make decisions having already read the reports and reviewed the agenda. Simultaneously, the substance of their meetings has changed from insubstantial (attendance at events, personal cars, etc.) to fulfilling their duties - passing budgets, approving workplans, approving reports, and addressing relevant district business. Significance LC is now more efficient in their work, enabling the technical departments to carry out their responsibilities and providing them better oversight. This strengthens the systems checks and balances, rather than the technical departments working in an accountability vacuum. Since meetings are shorter, less money is needed per meeting, allowing councils to conduct six meetings per year within their allotted budget. Meetings are less costly and more effective. GAPP Contribution: HIGH Council members: After the election in 2016, GAPP conducted intensive 2-day induction training jointly with the MoLG for all council members in GAPP districts on their roles and responsibilities, and rules for Council proceedings. Trainings involved mock council meetings during which time Council members and Technical staff role played actual meetings. Following the trainings, the regional ETS visited districts to observe council proceedings and give feedback on performance to reinforce learning. GAPP printed and distributed and trained Councillors on Local Government Rules and Procedures and the Local Government Act. These rules were previously unknown and unavailable to Councillors. Council Speaker – GAPP gave training to all Council Speakers in target districts on how to prepare an agenda in advance, etc. ETS gave close follow-up coaching as requested. In one instance, the ETS gave coaching to the Speaker during an initial council meeting by passing a note of recommended actions, which changed the course of the meeting. Other Contribution Claims In 2012 (after elections in 2011) SDS financed inductions for politicians across the country. So, every district had an induction. In 2016, it was done by GAPP, but only in their districts. The GAPP training was more participatory (i.e., mock meetings) and involved close follow-up coaching through the ETS. In Mpigi, which was formerly involved in the SDS project, precursor to GAPP, Councillors with experience coach new ones. Technical staff also coach new Councillors on their role so that they 99 | GAPP EVALUATION REPORT ANNEXES USAID.GOV learn together during the first year. While change was noted in GAPP-supported districts, there have been other contributions to this effort through other projects. Sources of Evidence: ● LC, Kiboga District ● LC, Buliisa District ● LG, Mpigi District ● LC, Mpigi District 100 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #8. LOCAL COUNCILS PASS ANNUAL DISTRICT BUDGETS AND APPROVE ANNUAL WORK PLANS Outcome statement: Since 2017-2018, Councils supported by GAPP can routinely pass a district budget, based upon the draft submitted by the Planning unit. Previously, councils understood the budget to be solely for their operations and support. They would refuse to pass the budget until they received their allowances. Now they are clear that this is the budget for the entire district government. They know they must pass the budget before they can receive payment. They also know that their allowances must not exceed 20 percent of the revenue the district collected the previous year, contingent on the realization of revenue forecasts. If the district is unable to collect all planned local revenue due, the Council must scale back planned expenditures. For example, in Kiboga, the district projected a revenue of GX 400 million in FY 2018/19 but collected only UGX 300 due to out-break of foot and mouth disease in the district. Councillors accepted the consequent reduction in allowances without conflict. In Kiboga district, the Council refused to approve the district budget for FY 2017/18 until their allowances were paid. This was largely due to lack of understanding by the councillors of their roles and functions, exacerbated by poor communication and information sharing on the part of the technical side. Significance Most importantly, district departments now have approved budgets and workplans against which to make expenditures. Secondarily, tension between the Council and the Technical units has reduced greatly, shifting the relationship from adversarial to collaborative. Politicians also spend less time asking technical departments for money, since they know the regulations. GAPP Contribution: HIGH After the election in 2016, GAPP conducted intensive 2-day induction trainings for council members in GAPP districts, often including LG technical staff to ensure role clarification. Council members were provided with copies of the Local Government Act and the council rules of procedure for continuous reference. Additionally, GAPP ETS regularly visited the districts to help resolve misunderstandings between council and technical officials arising from role confusion, or to coach specific council organs (e.g. DEC, council committees) to enable them perform better. Other Contribution Claims In Mpigi, which was formerly involved in the SDS project, precursor to GAPP, the District and Town Councils have both approved the annual budget and work plan for longer than the project duration. No change was noted. So, while the change was noted in GAPP-supported districts, there have been other contributions to this effort through other projects. Sources of Evidence: Interviews with Council Chairs, Council Speaker, CAOs and GAPP staff in all case districts. 101 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #9. RELATIONSHIP BETWEEN POLITICIANS AND TECHNICAL STAFF IMPROVED Outcome statement: Since the start of the program the relationship between politicians and technical staff in GAPP supported districts has improved remarkably. Now, the councillors have better understanding of their roles and responsibilities with respect to representing the interests of their electorate, monitoring development and service delivery projects and activities in their constituencies, and formulating policies. For example, when resources (central grants and local revenue) are received, the CAO/ Town Clerk provides details to all relevant officials (e.g., District Chairperson, Resident District Commissioner, District Intelligence Officer, etc.) so that they can monitor how the money is used. Before the start of the GAPP project, conflicts between politicians and technical staff were very common due to misunderstanding of roles. Counselors would demand to review key communication sent from the executive or made decisions that were not within their mandate. This often led to disagreements, standoffs, and refusal by Council to perform its duties. For example, one council member declared that taxes would not be collected from constituents in his/her parish. Councillors were frequently in department offices asking for money for allowances or programs. Councils routinely pass district budgets, based upon the draft submitted by the Planning unit. Previously, councils understood the budget to be solely for their operations and support. They would refuse to pass the budget until they received their allowances. Now they are clear that the budget is for the entire district government. They know they must pass the budget before they can receive payment for sitting allowance. They also know that their allowances must not exceed 20 percent of the revenue the district collected the previous year, contingent on the realization of revenue forecasts. If the district is unable to collect all planned local revenue due, the Council must scale back planned expenditures. For example, in Kiboga, the district projected revenue of GX 400 million in FY 2018/19 but collected only UGX 300 due to out-break of foot and mouth disease in the district. Councillors accepted the consequent reduction in allowances without conflict. Tensions still remain and flare at times, though, but are generally at manageable levels. “50/50” as one Clerk to Council described it. Significance Reduced conflicts meant that council and departments no longer spend time fighting over ‘who is boss’, but in fulfilling their mutual roles and responsibilities. GAPP Contribution: HIGH The 2-day GAPP induction training and follow-up coaching by ETSs gave councillors an understanding of the responsibilities and limits of their role. This has been instrumental in generating the required attitudinal change in bringing the two sides closer together. GAPP also held ongoing trainings for other key staff, such as CAOs. Other Contribution Claims Mpigi district has similar levels of conflict that they work out with patience and discussion. So, while the change was noted in GAPP-supported districts, there have been other contributions through other projects. Sources of Evidence: 102 | GAPP EVALUATION REPORT ANNEXES USAID.GOV - LG / LC leadership, Kiboga District - LG leadership, Apac District - LG leadership, Kapchorwa Municipal Council - LG / LC leadership, Buliisa District - LG / LC leadership, Mpigi Town Council 103 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #10. INTERNAL AUDIT SANCTIONING OF FINANCIAL MISMANAGEMENT IMPROVED Outcome statement: The conducting and review of audits on local government process is a critical accountability mechanism to mitigate against corruption and financial mismanagement. However, this process was not well understood or monitored. Should an unfavorable report be issued, it was easy for an official to file the report and ensure it was not reviewed. Few audit reports were making their way through a complete review process, meaning merited corrective actions were not taken. GAPP helped clarify and strengthen this process: The internal audit report is sent to DPAC, to provide the opportunity for LG officials implicated to exonerate themselves. Issues that are satisfactorily clarified are dropped and DPAC sends its evidence- based report to council with clear recommendations (e.g., recoveries, disciplinary action). Copies are sent to the LCV chairperson, MoLG Minister, MoFPED, OAG, Accountant General, IGG, RDC and all LGs appearing in the report. Following deliberation, council sends the report to the accounting officer with instructions to take action on recommendations. Action not taken by the accounting officer will feature in the OAG report, triggering involvement of the PAC LG with serious implications for the officer. Following GAPP support, there has been significantly increased accountability for funds. For example, field activity reports are more detailed - including photos of activities and stakeholders/ end-users. This has in turn corresponded with the increase in the percentage of clean audits from 34% in FY 2013 to 80% in FY 2018. The key factor in this has been the increased effectiveness of the link between internal audit and District Public Accounts Committee (DPAC) processes. This externalization of accountability issues, together with scrutiny of final accounts by the OAG, largely accounts for the significant improvement in financial management in most GAPP districts. Significance The presence of an active and competent oversight body has put pressure on accounting officers in all local governments visited (Kiboga, Apac, Buliisa, Adjumani), and by extension other GAPP districts, to manage finances in compliance with the Public Financial Management Act (2015). This increases district financial accountability, ensuring public money is directed toward public services. GAPP Contribution: HIGH GAPP ETS provided training and technical support to finance departments, internal auditors and DPACs. District local government councils were guided on their roles and functions, especially in financial management. OAG teams from regional offices traveled to confirm the work was done. Other Contribution Claims No program provides similar support. It is unlikely this outcome would happen without GAPP. Sources of Evidence: ● LG/LC in Kiboga District ● CFO, DPAC Chairperson, Buliisa District ● Auditors, Hoima and Gulu OAG Branch ● Municipal Finance Officer, Internal Auditor, Apac Municipality 104 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ● Ag. Internal Auditor, Chairperson DPAC, Adjumani District ● ETS Lira regional officer 105 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #11. PERFORMANCE OF DISTRICT PUBLIC ACCOUNTS COMMITTEE IMPROVED Outcome statement: This District Public Accounts Committee is critical to local government oversight for accountability. This appointed body reviews all audit reports and investigates findings as needed. Previously, the DPACs in most local governments were considered peripheral and their summons often ignored by local government technical officials. This was because most DPAC officials did not understand their roles and functions and technical officials did not fully understand the legal basis of the DPACs. Accounting officers and council chairpersons could refuse to act on DPAC reports if they felt threatened by their contents. There is ample evidence that GAPP contributed to the significant improvement that has occurred in the performance of District Public Accounts Committees (DPACs). Beginning in 2014, they trained committee members on their roles and strengthened their abilities to read and understand audit reports. The District Executive Committee (DEC) and the Finance Committee of Council were trained on ‘dummy’ DPAC reports to prepare them for handling the real reports. Local government staff were also sensitized that respecting DPAC summons was in their best interest. As a result, most GAPP-supported DPACs have been empowered and have become more assertive. A good example is the Adjumani local government DPAC. Staff with issues are given one week to interface with DPAC to resolve those issues. Previously such staff would refuse to appear before DPAC claiming that they did not receive the summons. Now they must sign the delivery book when the summons are issued to them. Significance: Effective performance of the DPAC is crucial for promoting accountability in local governments. Adjumani DPAC provides an example of the seriousness with which accountability is being enforced. Whistle-blowers informed the DPAC about two cases, one of embezzlement and another of abuse of office by public officials. The DPAC took up both matters and police cases were brought against both culprits. GAPP Contribution: HIGH GAPP ETS inducted councils, technical staff and DPACs on their different but complementary roles in running local governments. This clarification made a big difference in raising the stature of DPACs in the local government system, especially when the linkage between internal audit, DPACs the OAG and the Local Government Public Accounts Committee of Parliament was articulated by GAPP ETS during various inductions. Furthermore, DPACs were provided with targeted training and guidance to improve their performance: this included how to analyze audit reports; how to detect fraud; the structure of DPAC reports (the report format was shared with DPACs); how to make recommendations; how to withstand harassment from technical officials that are being investigated; the legal requirement for the accounting officer to implement council resolutions arising from DPAC recommendations; the requirement for the chairperson of DPAC to attend council sessions; the need for DPAC to use a friendly rather than a confrontational approach in their investigations; the importance of giving a fair hearing to the accused; and the importance of covering all aspects of a local government and not just accountability issues. Other Contribution Claims No program provides similar support. It is unlikely this outcome would happen without GAPP. 106 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Sources of Evidence: ● LC leadership, Adjumani District. ● ETS Lira regional office ● LG leadership, Adjumani District ● LC leadership, Buliisa District. ● LC leadership, Apac Municipal Council. ● Auditor, OAG Gulu Regional Office ● Auditor, Gulu Regional Office 107 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #12. LOCAL GOVERNMENT ORGANIZES THE LRM PROCESS MORE EFFICIENTLY Outcome statement: Since most revenues come from tied grants from the central government, local governments have little discretionary funding to finance their activities and address citizen priorities for services unless they raise their own local revenues through taxation. However, many districts have very poor infrastructure to assess and collect individual property and business taxes. Targets for revenue collection were set, but only based on previous experience. Lists of taxpayers were generally kept in hard copy only. Since parish chiefs were not regularly collecting information, potential taxpayers were identified every year from scratch. These taxpayers were issued several demand notes, one for each tax. Many sent staff to go door to door to enumerate, assess, and collect taxes in one step. There was no process to handle complaints. All this instilled a very low level of citizen confidence that the process was either equitable or transparent. GAPP supported local governments to increase locally mobilized revenue by standardizing their procedures, making the assessment and collection process transparent, and communicating how the revenues are used. The process was designed to ensure regular data collection, and regular updates of the database. Some of the case districts had established a digital taxpayer database. Others had established an appeal process. In one of the visited municipalities, the tasks of enumeration, assessment, and tax collection were separated in phases for different teams. These steps have contributed significantly to reduce local taxpayers’ resistance to taxation. GAPP claims that urban sub-national governments registered a 38% increase in locally generated revenues. It increased from UGX 8.3 billion in FY2012 to UGX 11.4 billion in FY2015. Case districts Adjumani and Apac both reported an increase in local revenue. Kiboga partially offset large revenue losses from cattle disease through improved local resource procedures. Significance Increasing LRM is critical for LGs to increase their ability to react to local grievances and developmental needs. And LRM depends on proper data management, especially a comprehensive register of all taxpayers. GAPP Contribution: HIGH GAPP provided training on LRM and helped to identify additional sources of local revenue. With coaching from the ETSs, local governments organized their LRM process more efficiently. Other Contribution Claims In the visited town council, the current municipality did not have enforcement officers. These were hired in 2013. Enforcement officers enforce local court orders, also in relation to tax arrears. While there were other interventions, such as from the Finance Commission, GAPP enhanced the process and made it functional on the ground. Sources of Evidence: ● LC leadership, Apac Municipal Council and Adjumani Council ● LG leadership, Kiboga District ● GAPP, GAPP Program Results, Fact Sheet 2012 – 2019 108 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #13. LOCAL GOVERNMENT PLANNING MORE INTEGRATED Outcome statement: The planning process has improved due to increased coordination among district/ municipal local governments. Local governments develop 5-year development plans using templates provided by the National Planning Authority (NPA). The templates are sent down to sub-counties (in the case of districts) or divisions (in the case of municipalities) to capture priorities from below. This is done by Parish Chiefs and Community Development Officers. The plans from below are integrated into district or municipal 5-year development plans, which are then broken down into annual workplans. Districts used to find developing annual workplans to be a big challenge due to inadequate guidance and backstopping from the NPA. However, support from the GAPP ETS helped them to overcome this challenge. Kiboga district provides a good example. Previously, the Kiboga district planning process for the annual workplans was not well coordinated due to inadequate skills and lack of technical support. As a result, the annual work plan for FY 2018/19 did not capture the key outputs of the education, health, and water departments - the 3 departments with the biggest percentage (60%) of the district conditional & sector grants. However, this was addressed in the work plan for FY 2019/20. There was more integration across sectors, more stakeholders were incorporated, and increased use of data led to better prioritization and forecasting. The key success factors were: 1) close coordination and teamwork among the 12 departments, and 2) close involvement and demonstration by the GAPP ETS during the planning process. The senior planner and heads of department improved their skills in evaluating the current district development plan, identifying key outputs for each department and working as a team to develop the next district development plan. It is likely that the annual workplan for FY 2020/21 will reflect the district’s service delivery needs better and will be implemented with district commitment. Significance Ability by the local government to develop development and annual workplans was crucial for prioritizing among competing demands in order to utilize the limited resources available. GAPP Contribution: HIGH GAPP ETS provided training, backstopping and hands-on guidance to heads of department during the planning process. The planning unit was also assisted to prepare quarterly reports. Technical guidance and support provided by GAPP ETS enabled the district’s 12 departments to generate and incorporate their data into the annual workplan for FY 2019/20. Previously, the district used to experience difficulties in prioritizing competing demands during the budgeting process. The GAPP ETS helped by training the district’s technical planning committee and attending the district budgeting conference every October to provide technical advice to the process. Other Contribution Claims Some CSO partners agree that the government has become more consultative in its planning. However, others maintain that it is still largely a top-down process with narrow consultation given. Sources of Evidence: ● LG leadership, Kiboga District ● CSO partners, Adjumani, Kiboga, Buliisa districts 109 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #14. GENDER BUDGETING PERSPECTIVES INCLUDED Outcome statement: Local government technical staff and councillors now have stronger capacity of in Gender Responsive service delivery. Often budgeting focuses on high-profile infrastructure projects, such as roads and markets. Little attention is given to services such as sanitation, accessible education, and lighting which disproportionately affect women and girls. Gender-responsive budgeting provides a lens to ensure that the collection and allocation of public resources are directed to advancing gender equality and women’s empowerment. These services include: construction, renovation and equipping of maternity wards, upgrading Health Centers, procurement delivery beds and mama kits, girl child school sponsorship, construction of girls changing rooms in schools, procurement of sanitary pads and recruitment of female teachers in primary schools, retention of girls at school and equal opportunities for girls and women’s livelihood among others. GAPP introduced gender-responsive budgeting through its two CSO partners, FOWODE and UWONET. The organizations trained 180 district leaders and 132 citizens in Gender Responsive Budgeting, planning and service delivery monitoring. As a result, the district plans and budgets of the three districts have shown improved inclusion of gender and equity issues. Women, youth and PWD councillors’ caucuses contributed to enhanced local government performance through improved oversight of service delivery. Citizens in the six sub-counties have demonstrated increased agency by participating in government planning and budgeting processes for improved gender sensitive service delivery in the education and health sectors. Additionally, there is an increased level of debate in council seen by the number of motions fronted by the women caucuses. Over 25 of motions were moved by women caucuses pertaining to gender equality, women and girls’ rights and social issues in the respective districts. Significance Increasing the ability of citizens and government representatives to recognize the needs of the genders in budgeting is critical to advancing social equity. Additionally, since women are known to have influence on many development indicators (population, education, family income, etc.), supporting women generates wider community impacts. GAPP Contribution: UNVALIDATED GAPP funded the two CSO partners to bring gender-responsive budgeting to select districts. There is evidence of increased gender-sensitive considerations by women caucus members. However, these claims could not be substantiated beyond GAPP and the partners themselves. Other Contribution Claims None noted. Sources of Evidence: ● Final reports form FOWODE and UWONET ● Validation interview with FOWODE field staff 110 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #15. LOCAL GOVERNMENT IMPROVES MECHANISMS FOR ACCOUNTABILITY TO CITIZENS Outcome statement: Prior to GAPP, there was very little accountability reporting to the citizens on what money was received by the local government, how it was spent, and what was achieved. This led to continued passivity regarding ownership, engagement, and oversight of funds designated for public services. Now in many GAPP districts established mechanisms are being used to communicate this information to the public. This includes: ● Community barazas are a common mechanism for facilitating community dialogue with government officials; ● Accountability Day in Kiboga where local council leaders (LCI, 2 & 3) and citizens meet at the district headquarters for reporting of ongoing development and service delivery interventions in the district. Citizens then make ask questions and voice concerns on the results. Up to 250 participants attended the meeting in 2019. Issues raised are taken up by technical officials for action and are reported on the following year; ● Kiboga also prepared a printed Accountability Report that was distributed through the Parish Chiefs to explain the contents to citizens; ● Kiboga also erects sign-posts on roads that were built or rehabilitated with public money; ● Districts commonly post their budgets and other key information on bulletin boards outside of public offices (Although the budget posted in the Buliisa District offices was from 2013/14); ● Other districts use regular radio programs to communicate this information. Significance Citizen understanding and engagement is a necessary input for public accountability and oversight for government services. It is also increasing tax payment rates when citizens understand and see value in the taxes they pay. GAPP Contribution: LOW Kiboga local government credits GAPP with inspiring a spirit of accountability that has taken root in their work because of the training and mentoring that they received. Other Contribution Claims Kiboga accountability initiatives were their own ideas. Barazas are an initiative of the Office of the Prime Minister. Action Aid held a project a few years ago which constructed display boards for government information. CSO partners maintain that while reporting has improved, there are still gaps in the process to make it fully participatory, e.g., holding events at the sub-county level. Sources of Evidence: ● LG / LC leadership, Kiboga ● CSO partner, Kiboga 111 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #16. IMPROVEMENTS IN HEALTH, EDUCATION, AGRICULTURE AND WASH PUBLIC SERVICES DUE TO CITIZEN ADVOCACY Outcome Description Most of the population in Uganda lives in rural areas and are predominantly dependent upon the government for public services such as education, health, water and sanitation, and agricultural extension. However, quality of these services is low, due to insufficient infrastructure, shortage of human resources, and low management and technical capabilities. With negligible citizen demand for improved services, poor conditions continue. Through the GAPP program, CSOs and PSAs engaged citizens to lobby government to bring deficient services to the attention of their district councils for redress. In total, GAPP supported 49 organizations for district-level projects in education, health, agriculture, private sector development, local resource mobilization, and issues of gender, refugees, and oil and gas. Some projects focused on several of these. While their projects varied in strategy and sector, many of the organizations did outreach, media publicity through radio programs, and trained government officials on their sensibilities as duty bearers for public services. As a result of the projects, government addressed the specific issues raised and improvements were made to service delivery. Illustrative examples of improvements in targeted facilities include: ● Reduced health worker absenteeism; ● Drug stock-outs reduced; ● Improved sanitation facilitations (latrines and safe water) at schools, markets, and health facilities; ● Renovation of facilities, especially maternity wards (placenta pits built, maternity wards renovated, etc.); ● New schools constructed or renovated; ● Reduced teacher absenteeism; ● Enforcement of laws protecting producers from fake agricultural inputs. (Specific details of results can be found in the individual end-of-project reports for the grants. However, many reports provide more detail on activities than results, so exact results are difficult to determine.) Significance: These improvements have direct bearing the lives of many citizens in the districts. Women have safer and cleaner deliveries, patients receive drugs when they need them, children receive better education, illness and safety risks from poor sanitation are reduced, and agricultural producers get a better return on their effort. These benefits will be in place if staff continue to fulfill their job responsibilities and the constructed/rehabilitated facilities are maintained. GAPP Contribution: High The GAPP program provided funding for the proposals, which were selected competitively, based on an initial concept paper. Grantees also received some training on how to do advocacy to support achievement of their objectives. Therefore, the contribution of GAPP to these results is high. Other Contribution Claims 112 | GAPP EVALUATION REPORT ANNEXES USAID.GOV There are other development organizations providing health and education services in rural Uganda to complement services provided by the government and strengthen government systems to deliver better services. These programs certainly complement the efforts made under GAPP. However, support for citizens advocacy is unique. Most achievements would not have happened without GAPP support. Sources of Evidence: ● GAPP Program Results: Fact Sheet 2012 – 2019 ● Final project reports for grantees (x49) ● Close-out meeting testimonies 113 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #17. CITIZENS FORMULATE AND PRESENT PETITIONS TO LOCAL GOVERNMENT REGARDING SERVICE DELIVERY IMPROVEMENTS Outcome statement: Due to Uganda’s development history, citizens rarely engage their governments on service delivery because of their low level of awareness on rights and standards of effective service delivery. Involvement of citizens in monitoring service delivery was non-existent. GAPP funded national level CSO ACODE and its partner NGOs in the district to train citizens in four districts on how to write letters and petitions. As a result of the project, citizens increased their monitoring in LG service points. This includes private sector organizations who used to complain about poor structures and services. Through ACODE’s support they had information and tools to demand services. Citizens started engaging their leaders through regular platforms between service users and providers. This would lead to responses from leaders and debates which would validate these issues. Significance: HIGH Politicians are increasingly sensitive to citizen demands because they are the electorate. Experience shows that when demands come directly from the citizenry, it captures attention of LG leaders. Furthermore, citizens best know what challenges they face in service delivery and can demand accountability for services and corrective action be taken when needed. This is a critical pressure point for progress. GAPP Contribution: HIGH GAPP gave ACODE a grant to train citizens in four districts through its partner CSOs in the district. The increase in citizen activities in presenting petitions started shortly thereafter. Other Contribution Claims No other programs support this type of activity in Uganda. Sources of Evidence: ● GAPP Lessons Learnt Study for Component 3 ● ACODE grant report ● Local Government Councils’ Scorecard Initiative (LGCSCI) End of Project Performance Report January 2016 – August 201E ● Making Public Finance Sustainable and Accountable- CSBAG story on Strengthening Local Government Financing and Accountability (Undated) ● GAPP Mid-term Evaluation report, 2015 ● Interviews with Uganda Debt Network, CSBAG, and CEPA 114 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #18. PRIVATE SECTOR ASSOCIATIONS LOBBY FOR SERVICE DELIVERY IMPROVEMENTS Outcome statement: There are now good examples of private sector associations organizing to constructively engage their local governments to improve public services. Like the general citizenry, most businesspeople know very little about holding government accountable for providing quality public services to the people. They knew even less about governance cycles and how to approach government to act in response to needs. PSAs were traditionally confrontational with local governments, demanding quick solutions. GAPP helped increase understanding of the processes for citizens to get better response to their demands. GAAP’s partner USSIA developed public private platforms in all the 3 districts of operation which were important for assessing service needs in those districts and lobbying for improved services. These platforms then engaged government officials (CAO, URA, CDOs, etc.). Several advocates successfully organized to formulate and present written position papers to document the real challenges affecting them on the ground and ask for a response. Many were able to influence policy actions at the district level, resulting in better relations and improved service delivery. Significance: When businesspeople see their local governments being responsive to their needs and supportive of their ability to make profit in their work, they become constructive development partners within the district. Through the progress made in the GAPP districts in helping business associations more constructively engage government, many were able to better link tax money that they pay and the role of the local government in providing better public services that support their interests. GAPP Contribution: LOW GAPP realized that PSAs hardly understood the LGs governance structure, so they were trained on understanding LG. The program provided funds for USSIA to work with PSOs and PSAs on advocacy and lobbying. Trainings were not one offs but would be accompanied by follow up mentoring and coaching by GAPP. GAPP staff also supported USSIA through coaching, problem solving, and reporting. GAPP gave direct support to 7 PSAs through grant awards. GAPP grantees mapped out existing PSAs, discovering those business groups that existed that it could work with, particularly those not normally supported (e.g., women, youth, PWDs and the general community). Other Contribution Claims There are no other development programs working to support private sector advocacy in the districts. This outcome would likely not have happened without GAPP support. Sources of Evidence: ● Interview with USSIA staff ● End of Project Report on Contributing towards Private Sector Engagement with LG for better Service Delivery; April 1, 2017 – June 30, 2019 115 | GAPP EVALUATION REPORT ANNEXES USAID.GOV ● USSIA End-Line Study Report Kiboga, Mityana & Mukono Districts ● CIDI End of Project Report on Contributing towards Private Sector Engagement with LG for better Service Delivery; April 1, 2017 – June 30, 2019 ● GAPP program Stories of Change, 2019 116 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #19. CSOS ATTRACT NEW FUNDING DUE TO STRENGTHENED ORGANIZATIONS SYSTEMS Outcome statement: Several CSOs in GAPP-supported districts have been able to attract new funding as a result of the organizational capacity development support they received from GAPP. Many local CSOs struggle with stable funding for short-term projects and with donor reluctance to fund core-costs or capacity development. GAPP assisted organizations interested in support for institutional capacity building. In response to interest from advertisements, support was agreed for either one CSO or one PSO in each district. Some organizations were also grant recipients, but there was no direct link, since the calls for institutional strengthening support were made differently at different times than those for grants. Originally, 25 organizations received support in 25 districts. With arrival of DFID partnership, another 10 district organizations and 14 national level organizations were added. The Participatory Local Organizational Capacity Assessment (PLOCA) assessment was done once in 2 years, resulting in an action plan to address gaps in areas such as strategic planning, institutional governance, M&E, financial management and fundraising and sustainability. Follow-up support was given on action plans through consultants (a 4-5 day of on-site technical assistance) or training. Fundraising was a priority for all. Several partners achieved their resource mobilization and proposal writing milestones and were successful in attracting new funding. Significance Organizational sustainability of local CSOs is threatened by reliance on a small number of donors to fund activities through multi-year projects. Organizations with strong systems and processes are not only able to more effectively carry out their work, but also attract funding from a diversity of donors through more efficient and transparent systems (e.g., financial management). GAPP Contribution: MEDIUM A PLOCA assessment would be conducted in the first year for new partners and technical assistance and training provided to all 46 PLOCA organizations in the next. GAPP hired consultants that carried out the PLOCA assessments and provided follow-up support. The GAPP program also provided the consultants with standard manuals and guides that were adapted to suit local contexts. Other Contribution Claims While overall PLOCA scores show an increase, self-assessment scores are important process tools, but not objective indicators of capacity. Additionally, with the low level of technical assistance, is it difficult to verify the specific contribution made by the GAPP program. Therefore, this outcome contribution is considered medium. Sources of Evidence: ● GAPP staff focus group discussion ● Interview with GAPP National CSO grantees and in case districts ● GAPP. GAPP’s Experiences of Capacity Building of Targeted Ugandan Civil Society; Triumphs and Lessons 2012 – 2019 117 | GAPP EVALUATION REPORT ANNEXES USAID.GOV Outcome #20. CITIZENS MORE WILLING TO PAY TAXES Outcome statement: Because people do not see a clear connection between the taxes they pay and the public services they receive, many citizens, particularly the local business community, traditionally resist paying taxes. Additionally, some district councillors made campaign promises that constituents from his/her parish would not pay taxes. GAPP interventions helped to decrease resistance to taxation by citizens, especially among the local business community. Local governments increased their reporting to citizens on what they accomplished with resources annually through Accountability Days, barazas, and posting information on bulletin boards. The increased efficiency and transparency of tax assessments and collections helped improve confidence in the process. (Citizens saw that others were being assessed similarly and they could pay their taxes through the bank, thus mitigating corruption.) Training councillors helped them see the importance of local revenues and they became supporters, rather than resisters, of tax collection. This helped local governments increase locally mobilized revenue. An increase in LRM is reported in Adjumani and Apac. In Kiboga, an otherwise bigger loss in LR due to cattle disease affecting market activity was partially offset by improving the LRM process. GAPP claims that a 38% increase in locally generated revenues by supported urban sub-national governments was registered. It increased from UGX 8.3 billion in FY2012 to UGX 11.4 billion in FY2015. Significance: HIGH Increasing LRM is critical for LGs to increase their ability to react to local grievances and developmental needs. Citizens understanding that their taxes fund local services has increased because of the information now being provided by LGs on how tax money is utilized. This provides districts with much needed revenue for operations and services. GAPP Contribution: WEAK Some measures that increased transparency in local public spending were reportedly inspired by GAPP trainings and workshops. One LG organized two meetings with business owners where GAPP ETS talked to the business community about the importance of LR. The GAPP contribution to these meetings was important because GAPP is perceived as a neutral third party. While this outcome is significant, the direct contribution of GAPP to this outcome could not be clearly verified by the evaluation team. Other Contribution Claims Barazas are the initiative of the Office of the Prime Minister and Accountability Days are the initiative of the local government in Kiboga. Sources of Evidence: ● Kiboga: KIB2 (CAO), KIB3 (PIA), KIB4 (PHRO), KIB5 (CFO), KIB9 (OAG Auditor) ● Apac MC: APA1 (MFO), APA2 (IA) GAPP Performance Evaluation Report 118 December 2019 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523