1 FINAL EVALUATION OF USAID/UGANDA’S DISTRICT OPERATIONAL PLAN APPROACH April 19, 2016 This publication was produced for review by the United States Agency for International Development. It was submitted by the USAID/Uganda Monitoring, Evaluation and Learning Program, AID-617-C-13-00007 It was prepared by SoCha, LLC under subcontract to QED Group, LLC. i DISTRICT OPERATIONAL PLAN FINAL EVALUATION Submitted to: USAID/Uganda Prepared by: SoCha, LLC, under subcontract to QED Group, LLC Disclaimer: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii Acknowledgements The SoCha team would like to thank the multitude of district officials, implementing partner staff and USAID/Uganda DOP representatives who took time out of their busy schedules to provide valuable feedback to improve a process they’ve deeply cared about and contributed. We would also like to thank the QEDGroup,LLC management staff, especially Andrew Reuter, Paul King, Jennifer Dahnke and Anant Singh, for giving us the opportunity to demonstrate our ability to successfully complete this unique evaluation, as well as use of their office, staff and resources along the way. Wolfgang Richert, the Outcome Harvesting trainer, also provided valuable insights on what needed to be done, improved our skills as evaluators and weathered our interrogations of the method. We all benefited from his patience. A special thank you is reserved for the USAID/Uganda Mission. As we point out in the evaluation, the complexity of the DOP process required an approach that moved beyond linear logframe methods and instead embraced uncertainty without sacrificing rigor. The Mission’s careful guidance, collaborative suggestions on various methods, and overall willingness to explore new innovations undoubtedly contributed to any strong points in this evaluation. We nevertheless bear full responsibility for any shortcomings in the report. -The SoCha Team iii Acronyms BFP Budget Framework Paper CACI Coordination, Alignment, Collaboration and Integration CAO Chief Administrative Officer CDCS Country Development Cooperation Strategy CLA Collaborating, Learning and Adapting COP Chief of Party csQCA Crisp Set Qualitative Comparative Analysis DBTA District-Based Technical Assistance DCOP Deputy Chief of Party DDP District Development Plan DEC District Executive Committee DHMIS District Health Management Information System DHO District Health Officer DLG District Local Government DMC District Management Committee DMIP District Management Improvement Plans DO Development Objective DOP District Operational Plan DS District Staff DTO District Technical Officer DTPC District Technical Planning Committee eDTPC Extended District Technical Planning Committee fsQCA Fuzzy Set Qualitative Comparative Analysis FtF Feed the Future GAPP Governance, Accountability, Participation and Performance GoU Government of Uganda IP Implementing Partner LC5 Local Council 5, which sits at the district level LGA Local Government Act LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MCD Mission Comparison District ME&L Monitoring, Evaluation and Learning MFD Mission Focus District MoFPED Ministry of Finance, Planning and Economic Development MoH Ministry of Health MoLG Ministry of Local Government MoU Memorandum of Understanding MSC Most Significant Change NDP National Development Plan iv NUFO O/H ONA OPM PAD PEA PMP QCA QED RDC RFA RFP SDS SoCha SoW ToC USAID Northern Uganda Field Office Outcome Harvesting Organizational Network Analysis Office of the Prime Minister Project Appraisal Document Political Economy Analysis Performance Monitoring Plan Qualitative Comparative Analysis QED Group, LLC Resident District Coordinator Request for Agreement Request for Proposal Strengthening Decentralization for Sustainability SoCha, LLC Scope of Work Theory of Change United States Agency for International Development v Table of Contents Acknowledgements............................................................................................................................................................. ii Acronyms............................................................................................................................................................................ iii Table of Contents............................................................................................................................................................... v Executive Summary..........................................................................................................................................................vii Introduction......................................................................................................................................................................... 1 Background ..................................................................................................................................................................... 1 Evolution of the Scope of Work ................................................................................................................................. 2 Research Design and Methodology............................................................................................................................. 2 Is DOP Relevant?............................................................................................................................................................... 7 Introduction .................................................................................................................................................................... 7 DOP Relevance: The GoU Perspective .................................................................................................................... 7 LGA and MoLG........................................................................................................................................................ 7 NPA and District Development Plans................................................................................................................... 7 MoFPED and Local Government Finance........................................................................................................... 8 OPM and the Partnership Policy ............................................................................................................................ 9 MIA and the NGO Policy........................................................................................................................................ 9 USAID Contractors: Exempt or Obliged?.........................................................................................................10 Conclusion: Is the DOP Relevant for the GoU? ...............................................................................................10 DOP relevance at the District Level.....................................................................................................................11 Conclusion: Is DOP Relevant at the District Level?........................................................................................14 Geo-Focusing................................................................................................................................................................14 Conclusion: To What Extent did the Mission Articulate and Implement a Consistent Geo-Focusing Approach?................................................................................................................................................................. 16 DOP Relevance: The USAID Perspective and the DOP Theory of Change...................................................16 Is DOP Efficient?.............................................................................................................................................................19 DOP Implementation and SDS................................................................................................................................. 19 NUFO .......................................................................................................................................................................20 DOP Joint Results Framework and ME&L Plan...............................................................................................20 DOP Performance and Perceptions .........................................................................................................................22 Previous DOP Learning and Recurring Issues...................................................................................................22 IP Performance and Perceptions ..........................................................................................................................23 USAID Performance and Perceptions.................................................................................................................25 DMC Performance and Perceptions....................................................................................................................27 DMC Meeting Costs....................................................................................................................................................29 Conclusion.....................................................................................................................................................................31 Is DOP Effective?............................................................................................................................................................32 vi Introduction ..................................................................................................................................................................32 Proposed Framework for a Revised DOP Theory of Change .............................................................................32 Measuring DOP Outcomes: Outcome Harvesting ...............................................................................................33 Outcome Findings........................................................................................................................................................35 The Evidence Supports the Revised Theory of Change ...................................................................................37 Breaking Down the Evidence by Bucket: IP and District Drivers of Change .............................................38 The Nature and Frequency of Potential Outcomes Suggests SDS Support is Crucial ................................39 No Evidence of the DMC fostering IP collaboration across DOs................................................................. 39 Comparing Outcomes in DOP and Non-DOP Districts: No Difference.....................................................40 Explaining the Variation in Outcomes.....................................................................................................................40 Results: Success=Substantive Action Items AND Adequate Grant A Disbursement Rates.....................43 Conclusion.....................................................................................................................................................................44 Should DOP Continue?...................................................................................................................................................46 Yes…..............................................................................................................................................................................46 …But not in its current form.....................................................................................................................................47 Annex 1: Scope of Work ................................................................................................................................................... i Annex 2: Design Phase Report....................................................................................................................................... ix Annex 3: List of People Interviewed and Documents Reviewed .........................................................................xxiv Annex 4: Perceived Success of the DMC Meetings and District Outcomes.......................................................xxx Annex 5: DOP Interview Protocol............................................................................................................................xxxi Annex 6: Outcome Harvesting Database ................................................................................................................... xlii Annex 7: Outcome Harvesting Reference Sheets....................................................................................................xliii Annex 8: Action Item Database...................................................................................................................................xlv Annex 9: DOP Dossiers and Field Notes.................................................................................................................xlvi Annex 10: QCA Factors and FSQCA Outputs.......................................................................................................xlvii Annex 11: DOP Truth Tables....................................................................................................................................... liii Annex 12: IP Survey........................................................................................................................................................ liv Annex 13: Cost Model and Instructions...................................................................................................................... lvi Annex 14: Evaluation Team Experience with Outcome Harvesting..................................................................... lxii Annex 15: DOP Cost Model .......................................................................................................................................lxiv vii Executive Summary In 2011, USAID/Uganda introduced the District Operational Plan (DOP) process to mitigate anticipated coordination and operational issues that might arise from having an increased concentration of program activities in 19 “Mission Focused Districts” (MFDs). The DOP is a formal agreement between USAID, district government, and implementing partners (IPs) that provides a framework for planning and coordinating USAID assistance with district-level governments to achieve shared development objectives (DOs) through a more effective and efficient approach. Under the DOP, each district establishes a District Management Committee (DMC), chaired by the Chief Administrative Officer (CAO) of the district, and relevant district technical offices meet quarterly with USAID and its implementing partners in that district to discuss and resolve coordination, alignment, and operational issues. Overtime, the DOP has evolved to merge with local government structures; in particular, most DMC meetings have now been combined with District Technical Planning Committees (DTPC) to become “extended DTPCs” (eDTPC). The DOP process has been primarily facilitated by the Strengthening Decentralization Systems (SDS) activity, as well as through direct support by USAID staff in northern districts. As the DOP process draws to a close, USAID/Uganda has commissioned a final evaluation to determine the effectiveness of the DOP approach and its potential contribution to USAID/Uganda’s development results. The evaluation should contribute to building a picture of the DOP process by not only looking at the causal chain of effects that lead to results, but also considering the ‘web’ of interactions between different actors, forces and trends, and their effect on the results USAID and its partners are striving to achieve. The evaluation should also highlight key drivers of change across different districts and stakeholders. As USAID/Uganda comes to the end of its first Country Development and Cooperation Strategy (CDCS), this evaluation will provide feedback on the overall effectiveness of the geo-focusing approach, the selectivity principle, and its implications for more efficient and effective programming going forward. Four evaluative sets of questions were addressed. Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? DOP relevance for the Government of Uganda (GoU) needs to be understood at two levels: the national and the district. The evaluation team found that most central government ministries have some form of regulation – a law, circular, policy, directive, etc. – in place that stipulates donors and their IPs should not only inform the districts in which they operate of their activities but also ensure that the central authorities also remain aware. But with the lack of any credible mechanism, these regulations tend to exist in name only and cannot adequately address the central government’s intention to understand what foreign service providers are doing on its sovereign soil. As such, the DOP process is an extremely relevant coordination mechanism for the GoU, even if it is not of its own design or directly supported by it. Although the DOP process still does not ensure that information submitted to the districts travels back up to the center, the DOP process does in fact meet the various central government requirements in ways that its ministries cannot achieve. At the district level, the evaluation team found that the DOP process - although extremely relevant as the main inter-sectoral coordination body to bring together district technical officers, IPs and USAID reps – is constrained by the extensive and expensive GoU decision-making process defined by decentralization. Overtime, DOP relevance has grown as SDS has moved to integrate DMC meetings with DTPC ones, and political participation has also helped reduce transaction costs. Nevertheless, DOP relevance will remain limited in terms of its decision making capability as long as political decision makers are not formally brought into the process (even if they may be currently involved informally). USAID should carefully weigh the efficiency gains associated with this move against the inevitable rise in costs associated with sitting allowances and other decision making supports. viii Question 2: To what extent did USAID/Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? Despite an explicit commitment to geo-focusing in the 2011 CDCS, USAID/Uganda’s mid-level procurement, compliance and GIS systems did not advance fast enough to adequately implement the approach and test its efficacy. As such, the central hypothesis of the CDCS – that development results for all three Development Objectives (DOs) will be improved when Health and Economic Growth projects work in the same place as Democracy and Governance projects – could not be adequately tested. Nevertheless, the upcoming CDCS 2.0 does offer the opportunity to take the original central hypothesis forward and explore the benefits of a more integrated approach, assuming substantial improvements to the Mission’s current GIS capability and IP compliance. The evaluation team also found that the DOP process played a crucial role in elaborating many of the geo￾focusing assumptions. Understood both as a mitigating as well as enabling mechanism, the DOP theory of change emphasized its own pathway to improving aid effectiveness; mostly involving improvements to processes that were assumed to subsequently lead to improved development impacts. Although a good start, the DOP theory of change required a less precise formulation, and a more realistic recognition of how the DOP may or may not contribute to development outcomes via a variety of uncertain pathways that cannot be identified beforehand. Moreover, the DOP theory of change was characterized by a notable lack of appreciation for USAID/District government relationship building as an end in and of itself. These issues aside, the DOP theory of change clearly established its relevance to the goals of the CDCS and offered a new modality for USAID to monitor its investments across Uganda. Question 3: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? Overall, the evaluation team found that the efficiency of the DOP process cannot be separated from the wider local government capacity building activities of SDS. As many observed, coordination without additional capacity building does not improve USAID/GoU relationships nor does it improve development effectiveness. Moreover, SDS’s implementation of the DOP process, originally in 13 districts but then expanded to all 19, went through considerable learning overtime on how to improve its efficiency. As the facilitating activity, SDS performed relatively well. Views on DOP efficiency are decisively mixed. The evaluation team found that much deeper challenges to the IP/District Official/USAID relationship persisted throughout the period of implementation, such as persistent problems with attendance, meaningful action items, meeting organization, decision making, documentation and leadership. IPs in particular hold the most negative attitude of the DOP process, and the opportunity costs of their attendance is high. USAID representatives, on the other hand, have gained considerably from their attendance and have improved both their relationship with district officials as well as their own situational awareness of implementation quality. Finally, district representatives appear to have gained the most from the DOP process in terms of their understanding of how USAID supports improvements in their districts. Estimated costs for these gains run about $7,500 USD per DMC meeting. Although seemingly expensive, considering that there currently is no other intersectoral modality in place to bring together USAID reps, IPs and district officials, we conclude that the DOP process is commensurate with the financial costs, but retain concerns over the opportunity costs incurred by IPs. Question 4: Has the DOP process contributed to changes to coordination, alignment, collaboration and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? Are these relationships different in districts where the DOP has not been implemented? What factors drove variations in its implementation? After a significant departure from the linear logframe/results frame approach to development outcomes, the evaluation team found that a revised framework of stochastic outcomes better conceptualized how the DOP process contributed to outcomes that occurred outside of, but in connection with, DMC meetings. Consequently, an alternative approach to capturing these outcomes was required, and the evaluation team adopted the “outcome harvesting” approach. Although not without its own set of biases, O/H nevertheless ix enabled the evaluation team to collect a wider array of outcomes than previously allowed for in the earlier DOP results framework or the revised ME&L plan. The evaluation team found 149 outcomes that could be in some way associated with the DOP process. From this, we were able to make the following five conclusions: 1. The evidence supports the revised DOP theory of change; 2. There is evidence that IPs and Districts are driving their own changes, but less evidence that they are instituting changes driven by the other; 3. The nature and frequency of potential outcomes suggests that DOP effectiveness would be limited without wider SDS support. 4. We found no examples of the DMC fostering IP collaboration across DOs, and thus also found no evidence to support a main assumption of the central hypothesis. Put differently, inter-sector co￾location and coordination does not organically lead to inter-sector integration through the DMC. 5. When comparing DOP to non-DOP outcomes, we do not find any significant differences in the average number of outcomes per district, i.e. implementing the DMC does not yield more outcomes than if IPs were left to establish their own bi-lateral relations. Conclusion 5 is the most serious, as the evidence suggests that, in terms of measurable outcomes, the DOP process has not made any significant differences than if the DOP had not occurred. Leaving these concerns aside, the evaluation team also ran a Qualitative Comparative Analysis (QCA) to identify the essential factors behind DOP contribution to the various outcomes harvested. The evaluation team found that District success - defined in terms of achieving at least seven DOP-related outcomes - was achieved when districts generated substantive action items at least one third of the time AND disbursed at least an average 80% of SDS Grant A funds every quarter. Notably, all other variables – Perceived Levels of Success, USAID and IP Attendance Rates, Average Questionable Cost Rates, Central Government Grant Disbursement Rates and Quarterly Sector Meeting Performance Rates – dropped out. These findings make intuitive sense. First, the ability of DMC participants to come together and collaboratively agree upon substantive points of action can be interpreted as an indication of a shared commitment to move past coordination and into more meaningful forms of working together and problem solving. Yet, as the evaluation team heard time and time again, coordination in and of itself is insufficient for action, but instead also requires significant capacity and resources to move forward. Second, the ability of Districts to consistently disburse high levels of SDS grant assistance may in fact fulfill this second requirement. What is also interesting is that QCA found that equivalent levels of central government grant disbursement were not relevant to success, suggesting two additional interpretations: 1. GoU grant money is seldom used to support USAID partner interactions and progress; districts would rather wait for USAID to provide this money directly; and 2. Success in the DOP Process was most likely contingent upon participation in the wider SDS activity (another point consistently articulated to the evaluation team by various stakeholders). The findings of the QCA suggest that there is no one size-fits-all to producing successful outcomes and that DOP-sponsored pathways are also valid. Even so, we must reiterate that success in the DOP process was most likely contingent upon participation in the wider SDS activity. Should the DOP Continue? In lieu of a list of follow-on recommendations, the evaluation team was advised instead to conclude with a discussion on whether or not the DOP should continue and, if so, which aspects should be brought forward and which should be dropped. In doing so, we suggest that, despite the above listed issues, the DOP process should continue. Although we found that the DOP may not be the most effective modality in terms of contributing to development outcomes, the DOP process is still necessary to help reinforce both x diplomatic and developmental functions. Both elements converge on the DOP process, which still stands as the key entry point for USAID representatives to monitor USG investments in both social and physical capital. However, its design should be modified significantly. Our evaluation suggests the following modifications: Expand the Web of DOP Stakeholders One of the most straight forward changes that the Mission may wish to consider is to revise the DOP MoU to formally embrace political decision makers in the process. Doing so will potentially increase the efficiency of the district decision making process and reduce the number of procedural steps before decisions can be taken by the district chairperson. We anticipate that more substantive decisions will also be taken as a result, which will potentially lead to a higher number of development outcomes. Consolidate USAID Representation The Mission may wish to consider creating a new, full-time “relationship manager” position to replace the current volunteer-based approach to DOP representation. These relationship managers would fulfill the three main responsibilities currently identified by DOP rep volunteers: Enforcement, Policy Clarification and Arbitration/Follow-up. Training modules on each of these functions should be developed accordingly. Explore the Costs of Expanding the DOP to Sub-District Officials and the Community Should the Mission consider advancing a more integrated development objective approach and, in doing so, involve sub-district officials in the DOP process, the evaluation team strongly recommends that the Mission first conduct a bureaucratic decision making study to identify the variety of allowances that must be paid to in order to expand the process. By posing the question “how much would it cost to administer a given district development plan down to the community level,” the study would most likely reveal the portion of bureaucratic “capture” through these allowances, as well as where potential opportunities to consolidate the decision making process can be found. Cross-cutting Changes Finally, given the lack of evidence that the DOP facilitates cross-sectoral outcomes, the Mission may wish to consider reducing the number of DOP meetings to twice a year. One of these meetings could further align to the annual budget and planning conference, while at the same time, IPs should still be required to attend quarterly sector meetings specific to their activity. This recommendation is based upon the need to balance IP concerns and opportunity costs with USAID’s own relationship building imperative and GoU compliance requirements. Analytical methods used to support this evaluation included: Outcome Harvesting (O/H): O/H is an approach useful for mapping out outcomes that cannot be predicted at the beginning of an activity and don’t follow linear pathways. The method was well suited for this evaluation. Under O/H, an evaluator analyzes a known outcome and works backwards to trace the actors, steps, factors, and key moments in time which led to a certain project outcome. The evaluation team used O/H to map out various outcomes for each district throughout the life of the DOP to better understand which outcomes are common and which are unique. Qualitative Comparative Analysis (QCA): QCA was used to understand how various combinations of implementation activities work together to contribute to DOP outcomes in and across the 19 districts. This method modeled the DOP experience according to a “truth table” that identified which sets of activities each district received, external factors that are relevant to the development results DOP hopes to achieve, and the actual development results that occurred under the program. Limitations: Both O/H and QCA came with limitations. O/H appears to have a strong positive outcome bias. Despite explicitly asking for negative and unintended outcomes from respondents, none were offered. xi IPs, despite an overall negative view of the DMCs, could also not provide specific examples of a negative change in condition in which the DOP contributed. Second, like most respondent-based approaches, O/H suffered from time proximity bias, in which more recent outcomes were easier to describe in more detail than those emerging in 2013 or earlier. To balance these, the evaluation team drew upon a large amount of performance data to tell a more holistic story, which partially compensates for these limitations. In regards to QCA, the analysis and conclusions were limited to only explaining factors that could be modeled. QCA requires data that is both comprehensive (it applies to all cases) and diverse (we can’t use averages across districts, because the average does not vary). Moreover, other relevant factors, such as political relationships, cultural factors, etc., were not modeled and therefore the results run the risk of advancing explanations that have no causal relationship to the outcomes observed. Fortunately, the QCA results are intuitive and the causal explanations relatively straightforward. 1 Introduction Background In 2011, USAID/Uganda introduced the District Operational Plan (DOP) process to mitigate anticipated coordination and operational issues that might arise from having an increased concentration of program activities in 19 “Mission Focused Districts” (MFDs). The DOP is a formal agreement between USAID, district government, and implementing partners (IPs) that provides a framework for planning and coordinating USAID assistance with district-level governments to achieve shared development objectives (DOs) through a more effective and efficient approach. Under the DOP, each district establishes a District Management Committee (DMC), chaired by the Chief Administrative Officer (CAO) of the district, and relevant district technical offices meet quarterly with USAID and its implementing partners in that district to discuss and resolve coordination, alignment, and operational issues. Objectives 1. Ensure that USAID programs are aligned with district development plans. 2. Eliminate duplication and improve complementarity among USAID implementing partners; improve collaboration and communication with local governments and other stakeholders working at the district level. 3. Strengthen partner districts and USAID’s joint coordination, implementation, monitoring and evaluation of activities within the district. 4. Make it easier for district governments to understand USAID’s portfolio and for them to provide feedback to USAID on project performance. Main Features Key signatories of the DOP include USAID, district government leadership, and all USAID implementing partner Chiefs of Party with a presence in the district. The DOP establishes a USAID Assistance District Management Committee comprised of all signing parties. To date, the Committee: 1) monitors and reviews progress of USAID-supported activities; 2) raises and discusses procedural, financial or administrative concerns that may affect the pace and quality of partner implementation; and 3) suggests and designs improvements to the implementation process. The USAID Assistance District Management Committee meets periodically, and to the extent possible, utilizes an existing district forum such as the Technical Planning Committee. As an activity, the DOP diverged according to two models of implementation: through direct technical cooperation with the USAID Northern Uganda Field Office (NUFO) and through the SDS activity as a subset of its Objective 1: To coordinate activities at district and sub county level. Both modalities underwent their own respective evolution, faced unique sets of challenges, required different types of resources and inputs, and thus offer different lessons to be learned. As the DOP process draws to a close, USAID/Uganda has commissioned a final evaluation to determine the effectiveness of the DOP approach and its potential contribution to USAID/Uganda development results. The evaluation should contribute to building a picture of the DOP process by not only looking at the causal chain of effects that lead to results, but also considering the ‘web’ of interactions between different actors, forces and trends, and their effect on the results USAID and its partners are striving to achieve. The evaluation should also highlight key drivers of change across different districts and stakeholders. USAID/Uganda, implementing and local government partners will use key lessons learned and recommendations from this evaluation to develop better program implementation strategies and build stronger partnerships for greater development results. As USAID/Uganda comes to the end of its first Country Development and Cooperation Strategy (CDCS), this evaluation will provide feedback on the overall effectiveness of the geo-focusing approach, the selectivity principle, and its implications for more efficient and effective programming going forward. QED Group, LLC (hereafter QED) has subcontracted SoCha, LLC (hereafter SoCha) through the USAID/Uganda Monitoring, Evaluation and Learning Contract (aka The Learning Contract) mechanism to perform the work associated with completing the above listed objective. 2 Evolution of the Scope of Work As an expression of USAID/Uganda’s commitment to Collaboration, Learning and Adaptation (CLA), this evaluation was guided by principles of adaptive management and divided according to a two-phased approach: an initial “Design” phase and the subsequent “Implementation” Phase. During the design phase, an original set of illustrative evaluation questions were offered in the scope of work and the evaluation team explored their feasibility. At the end of the design phase, there were some revisions to these questions as well as an elaboration on the methods used and overall analytical strategy of the evaluation. The original scope of work, with modified questions, is included in Annex 1, and the Design Phase Report is included in Annex 2. This evaluation is designed to address the follow four questions1: • Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? • Question 2: To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? • Question 3: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? • Question 4: Has the DOP process contributed to changes to coordination, alignment, collaboration and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? Are these relationships different in districts where the DOP has not been implemented? What factors drove variations in its implementation? Additionally, the evaluation team should conclude with a discussion on whether or not the DOP should continue, and, if so, which aspects should be taken forward and which should be left behind. Research Design and Methodology The evaluation team adopted a comparative research design to address the above listed questions. These comparisons occurred across a variety of units of analysis. We compared a. various stakeholder perspectives (IPs, District Officials and USAID Representatives) of the DMC meetings; b. comparisons across DOP districts, including regional comparisons as well as NUFO vs. SDS comparisons; and c. comparisons of various national government views on the relevance and importance of the DOP. Additionally, the evaluation was also designed to address the question: “what happens when the DOP is not implemented in districts with a high concentration of activities?”2 “High concentration” refers to districts that host activities from all 3 Development Objectives (Economic Growth, Democracy & Governance, and Public Health), but lack a coordinating mechanism, such as DOP, other SDS activities and/or GAPP. These comparison districts thus offered the opportunity to observe what forms of organic coordination and cooperation may have emerged without the DOP. The districts selected were Jinja, Kaberamaido, Mbarara, Rubirizi, Tororo and Wakiso.3 1 These evaluation questions are modified versions of those the original scope of work. Their approval was given when the Phase 1 Protocol report was accepted. 2 It should be noted that previously, USAID/Uganda had selected six comparison districts as part of its geo-focusing strategy. However, subsequent analysis determined that some of these districts differed in significant ways that prevented legitimate comparisons. See See Kisekka-Ntale, F. and Kibombo, R. (2013). “Reviewing the Sample Design for USAID/Uganda’s Evaluation of the Effect of Integration of Activities on Development Results in Selected Districts” Final Report. Unpublished USAID Document. 3 Since the selection of the original 19 MFDs, around 22 additional districts have received activity support from all three DOs (but no DOPs). Out of these, only 9 (not including Kampala) host all three DOs but do not receive additional support under SDS or another district government capacity building project known as GAPP. When selecting the six districts, the evaluation team considered how geography may also be a factor in DOP effectiveness; therefore, some degree of regional representation was maintained. Ideally, two districts from the North and East should also be selected, as these regions have the highest concentration of USAID-supported districts. However, there is only one high concentration, non-DOP district in the North – Kaberamaido. For the East, Tororo and Jinja were selected. Wakiso District is the only option for the Central Region. Finally, Mbarara and Rubirizi districts were selected 3 During Phase 1, USAID/Uganda encouraged the evaluation team to consider a number of evaluation methods and techniques to support the analysis. In doing so, the team distinguished between analytical methods and data collection approaches. An analytical method applies a clear set of rules to the data to reach certain conclusions. Conversely, data collection approaches are those which present a series of sequential steps and interview tactics to gathering data, but do not present clear rules for analysis. These are laid out below. Areas of Inquiry and Data Sources The main research design of the evaluation was rooted in the need for a comprehensive catalogue of the DOP experience to adequately address the evaluation questions. Previous conceptions of the DOP had been highly influenced by limited exposure to DMC meetings. Observers who happened to witness a well-run DMC meeting tended to have strong views of DOP efficacy and its potential. Conversely, those who witnessed poorly run meetings tended to conclude that the DOP process was a waste of time and resources. Given the high level of anticipated variation across the districts and that there was no logical basis for drawing a “representative” sample, the evaluation team analyzed all 19 districts as the central focus of the evaluation.4 This design was further guided by two considerations: First, the DOP process needed to be understood in terms of a multi-stakeholder approach, i.e. SDS/NUFO representatives, USAID DOP representatives, other USAID staff, relevant IPs and District Level technical staff (i.e. the CAO, technical department heads and the District Planner) – both operating within a previously-established GoU legal and institutional framework, as well as with the goals and objectives of the CDCS. Second, evaluation inquiries must remain cognizant of the previous work already done and build upon it (as opposed to replicating it). This led to the following areas of inquiry: • SDS Representatives: SDS, the IP tasked with facilitating the DOP, first in 13 districts and then later across all 19, periodically sent staff representatives to prepare district officials and attend the DMC meetings. The evaluation team held extensive conversations with SDS DOP staff, collected information regarding their specific financial and human resource inputs, systematically discussed, defined and catalogued their own evidence of DOP outcomes and noted their lessons learned/recommendations. • USAID Representatives: USAID/Uganda Representatives also regularly attended DMC meetings. Documentation of the USAID DOP representative experience was mostly found in various DMC trip reports. Additionally, as part of the DOP process, staff from the Program Office held periodic reflection and review sessions with DOP representatives to understand how the process was proceeding. Building upon this work, the evaluation team conducted two focus group discussions to identify definitions of DOP success according to USAID representatives’ perspectives, and understand USAID representatives’ activities and level of engagement with the DOP process. We also interviewed various members of the Program Office and former DOP representatives who were not longer stationed in Uganda. • Implementing Partners: For this evaluation, the team collected additional IP information through in-depth interviews and an IP-wide DOP survey. Out of the total 33 IPs participating, eight IPs were interviewed and another eight responded to the survey. These efforts build upon previous efforts. from the West as Kabarole has been the source of intense coordination under various DO3 activities due to its status as a regional referral center for the area.The remaining districts were Kabarole, Moroto and Soroti. 4 The Mission made it clear to the evaluation team that the evidence produced by this evaluation goes beyond the anecdotal, and therefore any sampling strategy must be able to demonstrate its level of representativeness. Yet doing so is difficult for a population universe of this size (i.e. 19). First, a “representative” random sample according to the commonly accepted goal of 95% confidence with a +/-5% margin of error would require a sample size of 18, i.e. the differences between a representative random sample and a full case review are minimal. Second, the amount of diversity across each district varies considerably and we currently have no basis for commonality to conduct a stratified sample, i.e. we cannot determine how representative the sample would be if we stratified. Thus, the evaluation team has identified a comprehensive catalogue of district experiences as the best course of action. 4 As part of a previous effort to understand how IPs were responding to the DOP process, USAID/Uganda commissioned a survey of IPs and District staff in 2013. The survey asked a variety of questions on the DOP process and did explore the question of DOP outcomes. However, these questions were posed at a very general level (e.g. “has the DOP process improved collaboration) and did not generate concrete evidence on DOP effectiveness. Also, in March 2015 the Mission conducted a CoP feedback session on ways to improve the DOP. Although useful, the session did not systematically collect evidence on outcomes and effectiveness. • District Technical Officials: District officials who attended DMC meetings were primarily technical representatives who fell under the leadership of the CAO. A central focus of the evaluation was to interview these individuals – close to one hundred in total – using the outcome harvesting methodology. This built upon the previous DOP survey, which included district technical representatives in the sample, but did not yield concrete information that could be used to confirm/reject the DOP theory of change and effectiveness. The Learning Contract also hosted a “DOP Learning Event” in November 2013, which also generated useful feedback on the way forward, but did not garner the evidence needed for adequate hypothesis testing. Therefore, the bulk of evaluative efforts of the evaluation focused on building a comprehensive catalogue of all 19 districts to systematically build evidence of DOP outcomes. • National Government Officials: In order to understand the relevance of the DOP, the evaluation team also visited various representatives from the Ministry of Local Government, Ministry of Local Government, Ministry of Health, Office of the Prime Minister, and the National Planning Authority. • Documents and Datasets: The evaluation team compiled and reviewed a considerable trove of documents and data. Annex 3 lists 83 different types of documents, most of which were collected at the district. Of special note, the evaluation team 266 USAID DOP Representative field reports, more than 200 attendance sheets, and more than 150 action item lists. The evaluation team also analyzed datasets from previous DOP perception surveys. Analytical Methods Prior to commissioning the DOP evaluation, the Mission was well aware that the DOP process was likely to unfold in ways that defied traditional M&E methods. Indeed, the DOP could be viewed as a “complex process,” in which the relationship between cause and effect is unknown at the outset of the cause, and is only better understood in retrospect (although seldom repeats).5 Therefore, the evaluation scope required more innovative approaches better-suited for interventions operating in dynamic, uncertain situations. The Mission and the evaluation team worked through a number of methods, and Mission staff played a valuable role linking the team to external expertise when/where necessary. Annex 2 – the Design Phase Report – contains a more detailed discussion of the previous analytical methods that were considered by the evaluation team but not endorsed. Here, we summarize the two main analytical methods used for this evaluation: Outcome Harvesting (O/H): O/H is an approach useful for mapping out outcomes that cannot be predicted at the beginning of an activity and don’t follow linear pathways. The method was well suited for this evaluation. Under O/H, an evaluator analyzes a known outcome and works backwards to trace the actors, steps, factors, and key moments in time which led to a certain project outcome. The relationships between these components can then be mapped out into a visual narrative of the key changes which led to the outcome in question. The evaluation team used O/H to map out various outcomes for each district throughout the life of the DOP to better understand which outcomes are common and which are unique. The overarching O/H protocol is found in Annex 5, the database of all outcomes harvested in found in Annex 6, and the O/H reference sheets used during the interviews is found in Annex 7. Qualitative Comparative Analysis (QCA): QCA is especially useful for complex programs that seek to understand how various combinations of implementation activities work together to contribute to program outcomes in 5 PPL (2013). “Complexity Aware Monitoring Discussion Brief,” USAID Online Learning Lab publication. Found at: http://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief. 5 and across a variety of settings.6 This method models the DOP experience according to a “truth table” that identifies which sets of activities each district received, external factors that are relevant to the development results DOP hopes to achieve, and the actual development results that occurred under the program (see Annex 12). As a rigorous analytical technique, QCA uses a Boolean Algebra-based algorithm to identify which combinations of DOP activities, events and conditions are necessary for success, which ones may not be necessary but are still sufficient for success, and which ones are superfluous relative to the other more essential factors. Finally, the evaluation team had originally committed to exploring the use of a previous Organizational Network Analysis (ONA) of DOP collaboration as part of its analytical toolkit. Unfortunately, we were unable to use the ONA for the following reasons: • The unit of inquiry was blurred between individuals and organizations, i.e. individuals are the respondents but represented as organizations. This creates problems in larger organizations, as they appear several times in top broker categories even though they are separate individuals and distort many network indicators; • The ONA unit of analysis focuses on networks and not districts. This is problematic for the IP data as an IP may appear to be strongly “networked” due solely to the fact that it appears across multiple districts (i.e. a structural cause) instead of any collaborative properties it may possess. This “structural bias” could potentially overcome IPs more limited in geographic scope but more capable in terms of collaborative efforts. As a result, direct comparisons between IP and District networks are not reliable. • Positive social response bias: Some of the questions in the ONA survey are inherently loaded with a positive social response bias that predisposes higher levels of collaboration in the results. For example, when IPs were asked if they felt engaging the DOPs was important, they were not given the option to indicate engagement was “not important.” Another example is that district officials were asked if they had collaborated on cost sharing agreements (34.7% said they had), but were not asked if these collaborations involved IPs (most likely they didn’t). • Inadequate sample size: For some districts, there were only two respondents, which was inadequate to make generalizable statements about various levels of connectedness in those districts. Limitations Both O/H and QCA come with limitations as methods. For O/H, we found that it has a strong positive outcome bias. Despite explicitly asking for negative and unintended outcomes from respondents, none were offered. IPs, despite an overall negative view of the DMCs, could also not provide specific examples of a negative change in condition in which the DOP contributed. Second, like most respondent-based approaches, O/H suffers from time proximity bias, in which more recent outcomes were easier to describe in more detail than those emerging in 2013 or earlier. This type of bias specifically involves a respondent’s inability to recall details and/or the fact that earlier staff have left/activities are no longer active. The implications of both of these biases are that there may be additional outcomes that we failed to capture, as well as a systematic inability to capture any unintended negative outcomes connected to the DOP. To balance this, we also drew upon a large amount of performance data to tell a more holistic story, which partially compensates for this limitation. Annex 15 includes a more detailed description of the evaluation team’s experience with O/H. In regards to QCA, our analysis and conclusions were limited to only explaining factors we could model. QCA requires data that is both comprehensive (it applies to all cases) and diverse (we can’t use averages across districts, because the average does not vary). Moreover, other factors we know are relevant, such as 6 The main premise of QCA is that of multiple conjunctural causation, which means that (1) most often not one factor but a combination of activities lead to development results; (2) different combinations of activities can produce the same outcome; (3) one activity can have different impacts on the outcome, depending on its combination with other factors and the context; and (4) the absence of an activity may be just as important as its presence. 6 political relationships, cultural factors, etc., were not modeled and therefore we run the risk of advancing explanations that have no causal relationship to the outcomes observed. Fortunately, the QCA results are intuitive and the causal explanations relatively straightforward. Our largest data limitation involved a limited response by the IPs. Out of the 33 IPs we identified as relevant to the DOP process, 15 were contacted. Seven responded to an online IP survey (28 did not), and eight were interviewed face to face. This accounts for 45% of all IPs. Comparatively speaking, IPs did not receive as much attention as the District Officials. However, the evaluation team did have a considerable amount of data on IP perceptions based upon the previous works already conducted. We also cross checked about 10% of the outcomes reported by the Districts with the IPs to confirm. Evaluation Layout In what follows, the evaluation questions are divided into three main sections: Is the DOP relevant?, Is the DOP efficient?, and Is the DOP effective?. In lieu of a list of follow-on recommendations, the evaluation team was advised instead to conclude with a discussion on whether or not the DOP should continue and, if so, which aspects should be brought forward and which should be dropped. We conclude accordingly. At the end of the report, we attached a number of databases based upon the data we collected, as well as comprehensive district Dossiers that contain additional relevant information for each district. 7 Is DOP Relevant? Introduction This section addresses Questions 1 and 2 of the Scope of Work, which we interpret as questions regarding the larger concept of DOP relevance. Relevance questions revolve around how an activity reinforces national policies and frameworks, as well as coherency of theories of change.7 For this evaluation, these questions are: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? This section is divided according to two, inter-related perspectives on DOP relevance: the GoU perspective and the USAID perspective. In addition to identifying the national frameworks and regulations in which the DOP is relevant, this section highlights the degree to which the DMC meeting fits in with district-level administrative bodies. However, the evaluation team found that in practice, district level decision making processes are hampered by inefficiencies built into the division of technocratic and political spheres. This discussion is relevant for the subsequent section on USAID’s perspective, as it reveals both the potential and limits of what the DOP can realistically achieve as well as the limits of the DOP theory of change. DOP Relevance: The GoU Perspective LGA and MoLG The question of DOP relevance for the GoU is found in the context of its push for decentralization, which began as soon as the National Resistance Movement assumed control and was consolidated under the March 1997 Local Government Act (LGA). LGA not only lays the legal foundation for the transfer of power to district and subdistrict level structures, it also empowers district bodies – primarily through the District Executive Committee (DEC) and technical representation headed by the Chief Administrative Officer (CAO) - to register, oversee, monitor, and coordinate NGO functions and activities. These functions are replicated at lower government levels all the way down to the Parish. LGA’s implementation falls under the mandate of the Ministry of Local Government (MoLG), which is responsible for promoting decentralization, updating policy and legal instruments, coordinating implementation of sector policies at the local level, and setting policies and service standards.8 In practice, MoLG serves more as a coordination body than as a central resource for consolidation. The implications for USAID IPs is that there is no central, one-stop shop mechanism for registration and reporting within MoLG, and IPs are thus legally obliged to establish separate relationships with each district they may find themselves working in. Moreover, there is also no mechanism for formalizing intra-district communications in that relations established at one district provide little or no support to an IP who must establish additional relations in another district. Finally, there is no district to MoLG reporting mechanism through which the Ministry is able to monitor which USAID IPs have established relationships in any of the 111 districts. Instead, MoLG representatives only become aware of these relationships through ad-hoc field visits and informal connections. NPA and District Development Plans LGA, under Section 35.3, requires that each district “prepare a comprehensive and integrated development plan incorporating plans of lower level local governments for submission to the National Planning Authority” (NPA). These “District Development Plans” (DDP) are drafted once every five years through what is supposed to be a widely participatory process that dips all the way down to the Parish and community levels. Oftentimes, DDPs include detailed activity lists that span from 100 to 150 pages containing two types of items: higher level development objectives, usually formulated by the district technical officers, and more 7 The use of standard OECD categories in this evaluation – such as Relevance, Effectiveness and Efficiency, are only for organizational purposes. The substantive effort of the evaluation is dedicated to directly answering the questions in line with USAID standards. 8 Taken from MoLG website, accessed on 12/16/15. https://molg.go.ug/ministry/ 8 specific resource allocation items (such as x number of hospital beds at x facility), usually formulated by political officials and directed towards their constituencies.9 The DDP also serves as the main document to align local development priorities with national ones, and every DDP must obtain written sign off from NPA’s Executive Director. Once the DDPs are completed, NPA is tasked to oversee the implementation of each DDP, but in fact the office does not have the resources or the systems in place to regularly monitor implementation NPA’s “Local Government Development Planning Guidelines” suggest that donors and their IPs should come into Step 15 of 27 (analyze and compile development resources) of the DDP drafting process.10 Yet donors and their IPs should, in theory, contribute to the DDP process at the beginning of the planning stage, and are required by the NPA to ensure that their own workplans and objectives don’t contradict those of the District’s DP in which they operate.11 The evaluation team reviewed all nineteen 2011-2016 DDPs in the DOP districts and found that there were no Donors and IPs – USAID/Uganda or otherwise – who were explicitly mentioned in these documents. Although these DDPs were drafted prior to the DOP, it could be that districts are reluctant to include these commitments in their plans, as MoFPED could potentially use this information as a justification to reduce the annual grant budgets the GoU provides to the district for conditional development grants.12 Moreover, we also found no instances in which a DDP was modified, which would require official sign off by the NPA Executive Director. To avoid this process, districts allegedly instead make changes or modifications to their annual development workplans, but the evaluation team also found no evidence of this occurring either. Given these gaps in substantive modification, a few IPs have cynically concluded that DDPs act more as show pieces to attract donor resources than as strategic guideposts for local officials to actually use to advance their district’s social progress. It should also be noted that the evaluation team found no clear guidelines on how an IP could in practice fall “outside of the district priorities.” DDP objectives were broad enough to cover all potential activities that might emerge under the USAID/Uganda 2011 CDCS, and no district official across all 19 districts could identify a single IP activity that fell outside of its corresponding DDP. A notable exception came in the form of SDS’ attempt to create “District Management Improvement Plans” (DMIP), a concerted effort to reconcile district service delivery priorities with central government funding allocations and district capacity to then fill the gaps with outcome-based programmatic planning. These DMIPs required continuous involvement on the part of district technical officials and substantial technical support via SDS’ District￾Based Technical Assistance (DBTA) teams, but the process also revealed which USAID IPs were directly supporting District service delivery priorities and which of their activities fell to the periphery. However, the DMIP effort did not garner the level of commitment it needed from IPs and the district technical teams, and it subsequently dropped out.13 MoFPED and Local Government Finance Not surprisingly, the Ministry of Finance, Planning and Economic Development (MoFPED) still maintain the strongest connection to district implementation of the DDP, as the Ministry has a mandate for resource mobilization and budget approval. In line with the Paris Declaration (to which the USG is a signatory), MoFPED should also ensure that both on- and off-budget support provided by donor funding related to local government plans is disbursed and reported on using modalities consistent with Local Governments Financial and Accounting Regulations of 1998. This, of course, requires that donors, including USAID, submit annual planned and actual budget expenditures, as well as use the Office of Audit General for the audit of projects they support. As suggested above, both on- and off-budget resources earmarked by foreign 9 Interview with NPA Executive Director, November 26, 2015. 10 See the Guidelines at http://npa.ug/wp-content/uploads/LG-PLANNING-GUIDELINES.pdf 11 During our meeting, NPA’s Executive Director mentioned that he had recently drafted a circular that explicitly required all IPs, regardless of Donor, to submit their workplans and budgets every quarter to the district planning officer. However, after repeated requests, he could not provide a copy of the circular nor provide its reference. The circular may still be in draft form. 12 Interview with the Chief Economist at the Office of the Prime Minister, November 23, 2015. 13 Interview with former SDS Chief of Party, December 7, 2015. 9 donors for district development should be deducted from central government funds earmarked for the same purpose to avoid funding duplications and waste. Despite these regulations, the evaluation team did not find a single example in which MoFPED was aware of duplicative IP resources in a given district and adjusted their grant disbursements accordingly. To be sure, USAID/Uganda does regularly submit annual budgets to MoFPED in a timely manner.14 Yet these estimates cannot be broken down to the district level, nor by IP. Moreover, USAID/Uganda in general does not provide on-budget support, and, in the few cases it has, has done so through the districts and not through MoFPED. To overcome this problem, MoFPED also requires IPs to submit their budgets and planned disbursements directly to the Ministry’s Aid Liaison office via the Aid Management Database. However, the evaluation team was informed that, to date, only SDS has reportedly directly submitted this information.15 Finally, we heard confidentially from one IP that, although annual budget conferences should seek to avoid duplication, districts benefit from duplication and are incentivized to mask it. A similar observation was made by the Executive Director of the National Planning Authority. Although these claims are hard to substantiate, they do reinforce the argument that building stronger links between MoFPED and IP district spending forecasts will be difficult. The only remaining potential mechanism for MoFPED to track to what extent IP resources flow into a given district is through the annual budget and planning cycle, in which districts are supposed to capture IP contributions at the Annual Budget Conference and submit these to MoFPED as part of its annual Budget Framework Paper (BFP). To assist, SDS developed an Integrated Budgetary and Planning Tool and invested considerable TA into building the capacity of district officials to use the tool. However, IP compliance with the tool in terms of submitting budgets has remained relatively low. Yet even if all IPs were to comply, MoFPED doesn’t have the capacity and monitoring system to extract, compile and analyze this information across all 111 districts, let alone adjust accordingly. It should also be noted that MoFPED in practice provides very little resources to most districts for the implementation of the DDP. One recent study estimated that the bulk (around 80%) of central government revenue transfers come in the form of “conditional grants,” of which a substantial percentage is used to pay salaries and associated allowances.16 While this study wasn’t the definitive work on the subject, it does raise significant questions that most likely could benefit from additional research. OPM and the Partnership Policy Although it was originally formulated in partnership with MoFPED, the Ugandan Partnership Policy currently sits within the purview of the Office of the Prime Minister. Partially created in response to previous aid corruption scandals in 2012, the 2013 Partnership Policy is an effort to increase the effectiveness of development assistance and ensure transparency and accountability among the GoU and its main development donor partners. Although there are no specific clauses in the policy that require IPs to submit workplans and budgets, the “spirit of the policy” implies that IPs and NGOs make a “good faith” effort to keep districts informed of their activities and objectives.17 MIA and the NGO Policy In 2012, in cooperation with OPM, the Ministry of Internal Affairs (MIA) launched the National NGO Policy. The Policy requires all NGOs operating in any given district to register with a newly-created District NGO Monitoring Committee, which is chaired by the Residential District Coordinator (RDC – typically personally appointed by the President to oversee national programs and preserve the national interest), with close attendance by the district security officer and tracked by the District Community Development Officer. 14 It should be noted, however, that USAID, as well as the European Commission, do not use the Office of the Audit General to audit their projects (the other major donors do). See, for example, the annual Public Financial Management Performance Reports 15 Interview with SDS CoP, on December 7, 2015, respectively. We were unable to obtain access to this database to confirm. 16 For a good overview of the fiscal transfer system, see the LEARN (2015), “Ugandan Decentralization Policy and Issues Arising in the Health and Education Sectors: A Political Economy Study,” unpublished USAID report 17 Interview with Chief Economist at OPM, November 23, 2015. 10 The Policy requires all NGOs to sign a MoU with the District, attend quarterly monitoring meetings, submit their budgets and workplans prior to the meeting, and work closely community leaders to ensure their needs are being met. At first glance, the District NGO Monitoring Committee appears to closely resemble the quarterly DMC meeting. However, critiques of the National NGO Policy decry the initiative as a political move to silence independent criticism of the government, deter accountability and transparency, and punish various advocacy groups, such as members of LGBT rights groups, for their views. Indeed, a recent revised draft version of the Policy contained provisions for substantial penalties and even prison sentences for those NGOs who were deemed to “prejudice the dignity of the Ugandan People.”18 Here, the difference between the NGO Monitoring Board and the DMC comes into sharp focus, as the latter maintains a strict technical focus on the implementation of the DDP, while the former expands developmental concerns to broadly interpreted political censorship. Despite their similarity, the two coordination mechanisms serve entirely different purposes. In spite of this saber rattling, the National NGO Policy in practice has by and large failed to emerge as many had feared, due primarily to a lack of resources to fund its implementation. Its main directive arm, the NGO Board (not to be confused with the Uganda National NGO Forum) has barely met since its inception in 2012 (members do meet centrally but limit their activities to issuing NGO licenses at $200 USD per license). Ambassador Kangwagye, the Chairman of the Board, stated the Shs200 million annual funding “cannot help the staff to carry out technical work of monitoring and checking’ on a nationwide basis”19, suggesting the budget would not even stretch to facilitate the monthly meeting of the 11 board members who require sitting allowance, transport refund and per diems. The more contentious aspects of the revised draft National NGO Policy have also been removed. As such, it would be a mistake to view the NGO monitoring committee as a viable district coordination mechanism in which IPs and district officials come together. USAID Contractors: Exempt or Obliged? During the various meetings with national representatives,20 the evaluation team discovered strong confusion regarding terms such as NGO, CBO, civil society, implementing partner and contractor in reference to the applicability of various laws and circulars that speculated NGO compliance. One of the clearest confusions was around for-profit contracting firms (who implement a substantial portion of USAID/Uganda’s portfolio), which are not governed by Uganda’s NGO legislation but instead register under the Ugandan Companies Act. Some officials, such as the Acting Commissioner of Local Councils under MoLG, expressed that USAID-funded NGOs, aka IPs, are exempt from the requirements of certain laws, such as the NGO Act, which instead referred to small scale CBOs with private funding. Others, such as the Director of NPA, argued strongly that all national regulations that reference NGOs clearly refer to USAID IPs, because they “provide services to the community.” Regardless of the correct legal interpretation, what these divergent understandings do reveal is a notable lack of experience with actually implementing the various laws and directives cited above when it comes to demand donors, IPs and NGOs comply with reporting and coordination requirements. Conclusion: Is the DOP Relevant for the GoU? Despite its predicted benefits, the decentralization process has severed crucial feedback loops between district and center, and has limited the ability of the GoU to “see like a state,”21 i.e. better understand where, from 18 See for example, Human’s Rights Watch at https://www.hrw.org/el/node/284454 19 http://web.monitor.co.ug/brochures/NGOboard15122012.pdf 20 The evaluation team interviewed officials from the Ministry of Local Government (MoLG), the National Planning Authority (NPA), the Office of the Prime Minister (OPM) and the Ministry of Health (MoH), as well as reviewed various national laws and policies. Unfortunately, no representatives from the Ministry of Finance, Planning and Economic Development (MoFPED) or the Local Government Finance Commission (LGFC) were available for discussion. 21 The term comes from James C. Scotts 1998 book Seeing Like a State, a comparative study into the efforts of various central governments to produce modernization and development. 11 whom, and to which districts development resources flow so that the central government can make more informed budget allocation decisions. To be sure, most central government ministries have some form of regulation – a law, circular, policy, directive, etc. – in place that stipulates donors and their IPs should not only inform the districts in which they operate of their activities but also ensure that the central authorities also remain aware. But with the lack of any credible mechanism, these regulations tend to exist in name only and cannot adequately address the central government’s intention to understand what foreign services providers are doing on its sovereign soil. As such, the DOP process is an extremely relevant coordination mechanism for the GoU, what was reaffirmed by MoLG representatives during our interviews and at DOP events, even if it is not of its own design or directly supported by it. Although the DOP process still does not ensure that information submitted to the districts travels back up to the center, the DOP process does in fact meet the various central government requirements in ways that its ministries cannot achieve. DOP relevance at the District Level Understanding DOP relevance at the district level requires a brief review of the decision making process and the division of labor between technical and political bodies. Under decentralization, power was devolved from the center and reconstituted at the district level. Although lower level bodies are given substantial authority (esp. at the sub-country level), the bulk of decision-making power resides at the district level under the elected Local Council 5 (LC5), headed by the District Chairperson. Underneath the LC5 is the District Executive Committee (DEC), which is nominated by the Chairperson and approved by the Council and composed of various elected secretaries. The DEC initiates and formulates policy for approval by the Council, oversees policy and DDP implementation, and receives and reviews requests made by the District Technical Planning Committee (DTPC). The DTPC is the highest level technical planning body at the district and is headed by the Chief Administrative Officer (CAO), who is the highest level technical representative and appointed by the central government. The DTPC supports the DDP by coordinating and integrating the sectoral plans of the lower local governments for presentation to the District Council, and is responsible for implementation of these plans. It is important to note that CAOs have no formal decision making power on their own, but instead must always seek approval from the DEC and ultimately the Council for any substantive change to take effect. However, this is not to say that the CAO has no substantive power in the district. On the contrary, the CAOs are responsible for the preparation and technical endorsement of the annual budget, has the power to convene DEC and Council member meetings for approval, may direct implementation resources to various locations in the district (as long as this does not conflict with specific earmarks already called for in the DDP), can deny IPs the right to operate in the district and informally wield the influence of the central authorities. Although elected leaders oversee the CAO’s work and wield authority over them, CAOs retain a significant level of autonomy that can be used to cripple the implementation of service delivery, which can have profound political consequences. The division of labor between the technical and electoral bodies assumes a high degree of collaboration and interdependency. When working in unison, the DTPC can make technical recommendations that are quickly endorsed by the DEC and authorized by the Council. However, as Golooba-Mutebi states “contradictions and tensions inherent in the co-existence of the two sides frequently arise for a variety of reasons, including arrogance (perceived or real) on the part of well-educated civil servants toward less-educated elected officials; confusion over rules; central government officials’ interference in local matters, including politics; local councils attempting to overstep their authority over civil servants; and civil servants’ rejection of arbitrary decisions by the elected leaders” (LEARN 2015, p. 4). Political confrontation between NRM appointed CAOs and elected opposition officials can also paralyze a district decision making structure, such as recently in Masaka where the Democratic Party representatives refused to work with the DEC and the annual 12 development plan was rejected.22 Finally, CAO’s appointments are usually for around two years before they are reassigned to another district elsewhere. Districts are then forced to frequently deal with turnover effects and discontinuity in leadership. Suffice it to say, effective cooperation is a variable in district service delivery and should not be an assumption. Perhaps the largest barrier to effective decision making and efficient service delivery is the way in which decision making is incentivized. Academic observers have long criticized the district planning system in terms of its dysfunctional pre-occupation with extended systems of managerialism and drawn-out process at the expense of public action.23 Uganda’s participatory-based approach, many argue, subjects any decision to undue scrutiny and builds in inefficiencies. Considering just the sheer number of Ugandan elected officials across the country is a daunting task. Although the evaluation team was unable to locate reliable figures on this, one article on Ugandan government salaries when there were only 80 districts explained that the country has administrative units across 45,000 local councils, 5,500 parishes, 1026 sub-counties, and 80 districts. Each unit is supposed to hold 10 members, which takes the total to an estimated half a million council members who constitute the decision making process across its various levels and geographies.24 With the expansion of the number of districts, the number of councilmembers today has also significantly increased. The potential for inefficiency across such an expansive system is compounded when the local compensation system behind decision making is factored in. For any decision that involves elected officials, a variety of allowances are paid to facilitate the process, including sitting allowances, facilitation fees, per diems, transport allowances, fuel costs, “safari day” allowances and “appreciation fees.” According to one senior GAPP representative, LC5 members by law are entitled to sitting allowances for the execution of their work. These allowances are set at 20% of locally￾raised revenues. Sitting allowances alone can be more than 300,000 UGS per council member per meeting, and one decision meeting may involve up to 10 members a sitting. For council members who do not receive salaries/emolument (councilors who sit below the level of the DEC),25 sitting and other associated allowances typically serve as their own official source of income and thus are unlikely to be waived. Central Government Conditional Grants are used to cover most of these expenses, but when these resources are depleted, decision making can come to a halt.26 The GAPP representative also noted that many councils fail to sit, especially at sub county level, due to lack of allowances. Although no formal study has yet been done, the percentage of service delivery resources that are captured by these supplements to facilitate any decision must be considerable. As such, CAO decisions to convene council members are limited by the degree to which these resources are available. The question of DOP relevance must be understood within these institutional constraints. The DOP process originally conceptualized the DMC meeting as a purely technical coordination body that sat alongside the DTPC. Although the District Chairperson and USAID Mission Director are ex-officio members of the DMC, the established MoUs explicitly designate the CAO and his/her technical officers as the main representatives on the district side.27 As such, the DMC meeting, by design, does not make high level 22 See http://ugandaradionetwork.com/story/delegates-denounce-dp-district-executive-committee-in-masaka 23 See, for example, Green, E. (2013), “The Rise and Fall of Decentralization in Contemporary Uganda,” Wider Working Paper No. 2013/087; Stelman, U. (2012). “Understanding Organizational Performance in the City of Kampala,” Africa Power and Politics Working Paper No. 27; and Kasozi-Mulina, S. (2013). “Process and Outcomes of Participatory Budgeting in a Decentralised Local Government Framework: A Case in Uganda,” Unpublished Thesis, University of Birmingham. 24 See http://www.independent.co.ug/society/society/3633-uganda-a-sleeping-giant-of-too-many-leaders-laws-policies￾and-no-implementation; and http://buzzkenya.com/latest-salaries-top-ugandan-politicians-revealed/ 25 The Local Government Act, Chapter 243, section 19 states that LC5 and DEC councilors are in full service of their councils and are entitled to emolument. 26 See, for example, http://ugandaradionetwork.com/a/story.php?s=34515; http://www.monitor.co.ug/News/National/District￾leaders-demand-allowances/-/688334/1628934/-/xjwbhv/-/index.html; and http://ugandaradionetwork.com/story/kiboga-district￾councillors-request-higher-sitting-allowances 27 See any District MoU, which all contained the same content. 13 decisions on behalf of the district side and recommendations reached should first be deliberated upon and reaffirmed at the DTPC, then forwarded for review and endorsement to the DEC, before actual authorization can be granted by the Council. Nevertheless, the DMC does initiate conversations that may not have started in the DTPC, as well as reach lower level decisions that don’t require Council approval. For example, a CAO at a DTPC can raise the issue of health worker absenteeism and recommend a strict punitive policy action in response, but it becomes trickier when it comes to the actual decision around adopting the new policy and should involve the Council.28 The graphic below shows where the DMC meeting fits into the regular district process: 28 This example was provided to the evaluation team by the Learning Contract CoR through email correspondence. 14 Figure 1: Local Government Structure Although the DMC meeting appears to be a parallel and potentially duplicative coordination body vis-à-vis the DTPC, it is important to emphasize that DTPC meetings tend to be closed door sessions that have not traditionally involved external donors and their IPs. Moreover, the DMC also occupies a privileged cross￾sectional position in which IPs and districts representatives across technical offices come together to coordinate issues that potentially go beyond the more sector and development objective focused working groups, such as the DWSCC WASH working group. Over time, the space for duplication with the DTPC also disappeared as SDS was able to convince eighteen district CAOs that combining the DMC with the DTPC to form an “extended DTPC” would reduce transaction costs (although some IPs noted that including sub-district officials was just as likely to led to more inefficient meetings). Finally, as the popularity of the DOP process spread to council members, their attendance at the DMC at times enabled CAOs to obtain de facto DEC endorsement and substantive decisions could head immediately to the Council for consideration.29 Conclusion: Is DOP Relevant at the District Level? In summary, the DOP process at the district level, while still relevant as the main inter-sectoral body to bring together district technical officers, IPs and USAID reps, has still been constrained by the extensive and expensive decision-making process as defined by decentralization. Overtime, DOP relevance has grown as SDS has moved to integrate DMC meetings with DTPC ones, and political participation has also helped reduce transaction costs. Nevertheless, DOP relevance will remain limited in terms of its decision making capability as long as political decision makers are not formally brought into the process. USAID should carefully weigh the efficiency gains associated with this move against the inevitable rise in costs associated with sitting allowances and other decision making supports. Geo-Focusing USAID’s articulation of Geo-focusing has its roots in the 2010 Presidential Policy Directive Six, which lays out the vision for the US government’s international development initiatives as a “core pillar” of America’s power, and emphasizes the need to implement development in a more strategic, cohesive, and efficient way by being more selective, focusing investments, and conducting development through “partnerships” to 29 The evaluation team found examples of this happening in Mbala, Bugiri and Oyam DMC meetings. 15 cultivate long term sustainability. The wording of the directive left substantial room for interpretation on exactly how to make these improvements, and USAID/Uganda responded with the concept of Geo-focusing. Although the Mission never formally defined geo-focusing, it was first discussed in the CDCS in regards to the “central hypothesis” that “development results for all three Development Objectives (DOs) will be improved when Health and Economic Growth projects work in the same place as Democracy and Governance projects” (CDCS, p.31). Although there were other concerns driving activity geography (such as Presidential Initiatives like Feed the Future), geo-focusing was to be the core pillar of how USAID/Uganda operationalized PPD-6 and more strategically focused its resources. Geo-focusing was further elaborated in a 2011 Policy Action Memorandum. There, 19 geo-focused districts, aka Mission Focus Districts (MFDs), were chosen because they contained an overlap of all three DOs. Note: the presence of SDS in a district was considered as representative of DO2 (Democracy and Governance) due to its systems strengthening activities. Although the choice of district was more a product of coincidental colocation, the memo stipulated that, moving forward, future activities were to be concentrated in these 19 districts to test the central hypothesis. Yet a few exceptions remained for activities that had: a national focus; a focus on oil and protected areas; DO3 Population based programs; and Karamoja Special Objectives. Throughout the life of the first CDCS, the number of activities across the 19 MFDs continued to grow in moderation. As of January 2013, there were around 33 different IPs operating in the 19 MFDs. By mid￾2014, the number had increased by four to 37; and, at the time of this evaluation, the number appeared to be around 44 IPs.30 Outside of the CDCS and Policy Action Memo, the evaluation team found limited evidence that geo-focusing was institutionalized within the Mission. For example, the evaluation team found no mention of geo￾focusing in subsequent Project Appraisal Documents (PAD) (mid-level strategy documents that further elaborate specific procurements that address higher level goals and objectives). The 2013 Sustainable Health Services instead identifies 61 priority districts it will address, while the 2012 FTF PAD remained implemented in its 34 focus districts. The 2013 D&G Rights and Accountability PAD lacked any district focus. The lack of a MFD discussion in these documents may reflect a lack of autonomy of the part of USAID/Uganda to solely decide where to implement, and instead reveal that programs, especially Presidential Directives such as FtF, have a different conception of geographic focusing. With this in mind, the evaluation team then focused on new competitive procurements that were posted from the USAID/Uganda Mission on fbo.gov and grants.gov. A random search of archived Request for Proposals (RFPs) and Request for Agreements (RFAs) does not yield many tenders that explicitly mention the 19 MFDs until three years after the Memo was published. For example, the 2012 NU-HITES RFA identifies 15 Northern Districts, including the 6 Northern MFDs, but these are not mentioned as such. It is not until 2013 that explicit mention of the 19 MFDs starts to emerge – such as in the 2014 Uganda Social Marketing Activity RFA and the more recent 2015 Youth Leadership for Agriculture RFP. This suggests a lag time of approximately three years from CDCS commitment until MFD geographic focusing became a procurement consideration. The above references 2011 Policy Action Memo recognized that improvements to the Mission’s GIS system would have to be made to accommodate the geo-focusing strategy (see p. 7). Yet despite the geographic emphasis of the CDCS and the Policy Memo, USAID’s GIS failed to keep pace. At any given point since the CDCS, USAID/Uganda did not have reliable information on the specific locations of all of its IPs in Uganda. To be sure, there were a number of efforts made to improve the Mission’s GIS functionality. At the beginning of the geo-focusing exercise, Mission staff produced district briefers that identified which IPs were operating per district. They also changed the format of the activity database to include geo-focus data and 30 The above estimates were provided by SDS and may not be accurate. Unfortunately, the evaluation team was not able to find precise numbers regarding how many IPs there were across Uganda’s district at any given time. 16 allow searches by district. But unfortunately these efforts were not sustained over time and there was very little adherence by IPs to enter geo-coded data. A/CORs were regularly asked by the USAID GIS Officer via a quarterly email to identify in which districts they operate, but these emails were seldom acknowledged. There has been no enforcement mechanism to require IPs to report this information.31 It was only in July 2015, after the Program’s Office received the support of a summer intern, that substantial efforts were made to understand where USAID IPs were operating. Although it was not USAID/Uganda’s stated intention, a more sophisticated geo-systems tracking, such as IP budget breakdowns by district, sub-district areas of operation and even household-level tracking, appear to be beyond reach of the current GIS system, and a substantial restructuring of roles, responsibilities and resources would facilitate a more comprehensive geo￾focused strategy. USAID/Uganda’s ME&L “Learning Contract” mechanism could potentially play a role in filling this gap. Yet even though the Mission appeared to lack a formal geo-focusing emphasis until circa 2014, the recent updating of the CDCS has once again raised the question of where the Mission should implement. Recognizing the inherent complexity of adopting a “single-tiered” geo-focusing approach, drafts of the new CDCS have identified a more limited space of programming autonomy in terms of three tiers of integration. The first tier has been assigned to those activities that have a much broader focus than what a specific MFD approach can accommodate (e.g. Presidential Initiatives, etc.), while a second tier of “co-location” remains similar to the original geo-focused approach of de facto alignment of different DOs. Unlike the previous CDCS, however, a new focus of deliberate “areas of intensity” allows for a highly integrated effort of cross objective activities focused at community or even household level. As of this evaluation, the criteria for selecting which areas/communities had not been finalized, but a set of “vulnerability assessment” tools may potentially enable the Mission to take geo-focusing to a new level. It must be reiterated, however, that this will require substantial investment in improving the current GIS capability. Conclusion: To What Extent did the Mission Articulate and Implement a Consistent Geo￾Focusing Approach? In conclusion, despite an explicit commitment to geo-focusing in the 2011 CDCS, USAID/Uganda’s mid￾level strategic, procurement and GIS systems did not advance fast enough to adequately implement the approach and test its efficacy. Nevertheless, the upcoming CDCS 2.0 does offer the opportunity to take the original central hypothesis forward and explore the benefits of a more integrated approach, assuming substantial improvements to the Mission’s current GIS capability. DOP Relevance: The USAID Perspective and the DOP Theory of Change As already mentioned, geo-focusing was defined in terms of the “central hypothesis,” i.e. the notion that development results will be greater in districts where all three DOs are present, which identified 19 MFDs. Yet proponents of the central hypothesis recognized that intensely concentrating activities in select districts could potentially lead to negative outcomes, such as increased/unreasonable demands on district representatives’ time, duplicative/conflicting interventions, and poorly aligned implementation cycles that might miss opportunities for greater synergy. Out of this, the notion of the DOP was created.32 In other words, the DOP process was partially conceived as a preventative mechanism designed to mitigate the potential unintended consequences of greater activity concentration, and thus initial DOP relevance was/is found in the Mission’s own understanding of how to realize the goals and objectives of the first CDCS. The DOP was also hypothesized to have various positive effects. According to the July 2012 DOP Paper, the DOP would instigate five major effects that would contribute to more effective aid. These were systems effects, efficiency gains, strategic coherence, increased mutual accountability, and lower transaction costs for IPs and districts. The major assumption behind these predictions seems to be that a much deeper level of 31 Interview with USAID GIS Officer, July 15, 2015 32 In fact, the DOP found its initial roots in the public health section, which explains why there were an 22 districts carried by SDS that had biannual meetings for health partners very similar to the DOP. 17 integration among IPs and District service delivery would emerge as the result of attendance at the DMC coordination meetings. To be sure, the DOP paper identified a series of tools to help foster that integration, e.g. a joint results framework, IP mapping tools, etc., but nowhere does the DOP theory of change specify that USAID will take a more proactive role in integrating its own activities for the Central Hypothesis to be realized. Instead, co-location of various DO activities, combined with the DOP coordination mechanism, is considered sufficient for development outcomes to improve. Mid-way into the DOP process, USAID staff took another look at that the DOP theory of change and provided additional elaboration. As the result of a participatory workshop, team members came together to create the following DOP Causal Pathway model: Figure 2: Illustration of DOP Causal Pathways Note here that the Central Hypothesis has been modified to some extent. Rather than just assume that some level of coordination to integration will organically emerge as a result of DO co-location, here the hypothesis explicitly states that local government is the hypothesized actor to coordinate DO activities, which lead to a variety of positive outcomes. Another assumption here was SDS and GAPP for the North would provide local government strengthening support to assist with that coordination. The Causal Pathway model was an improvement upon the previous DOP theory of change, in part because USAID staff had more experience and information in understanding how the DOP played out in practice. However, two critical gaps in the Pathway model remained. First, many of the subsequent outcomes envisioned on the graphic “bleed” into other outcomes; i.e. they are not discrete instances and can potentially fall under multiple boxes in the graphic. Further refinement with empirical evidence would be necessary to eliminate, combine and/or introduce new outcome categories as time went on. Second, the majority of outcomes connected to the DOP and DMC meetings are procedural, but their connection to the much wider “enhanced development impact” box at the end is anything but clear. Sometimes referred to as the “missing ILLUSTRATION OF DOP CAUSAL PATHWAYS Signed DOPs with 19 focus districts District capacity strengthening activities Implementation of district capacity building activities by SDS and GAPP in the areas of • planning & budgeting • leadership, management and decision making structures /processes • accountability and transparency • engagement of citizens • service delivery Coordination of USAID activities in the district • Quarterly DMCs • Follow up on key issues from DMCs • Sharing work plans and reports • Periodic performance review Engagement of district officials in USAID activities increased Integration of USAID activities in district plans Relationships between USAID and the districts strengthened Improved capacity of LGs to plan & provide selected services Rationalization of USAID activities within districts improved (duplication eliminated, complementarity increased) More responsive and targeted investments Availability of quality services improved (USAID supported sectors) DO activities implemented in the districts Alignment of USAID activities with district plans and priorities CDCS Uganda Geo-focusing – strategic decision to focus activities in specific districts Collocation of USAID DO activities PPD-6: Selectivity and focus for greater results Paris Declaration on partnership and alignment with local systems Awareness about USAID programs and activities increased LG ownership & influence over development activities increased Mutual accountability and transparency between USAID, IPs and districts Collaboration between different USAID DO activities in the district enhanced District led coordination of other donor activities Enhanced development impact Proposed Development Hypothesis If district local governments coordinate USAID assistance from all three DOs, then greater alignment and integration of USAID investments will yield greater development impact. 18 middle” or the “logical leap,” the DOP theory of change in this form could not demonstrate how this initiative was going to tangibly contribute to the goals of the CDCS and validate its Central Hypothesis. As will be discussed in the following sections, the DOP theory of change requires some revisions to render it more “evaluatable.” Specifically, a less precise formulation but more realistic recognition of how the DOP may or may not contribute to development outcomes, through a variety of uncertain pathways that cannot be identified beforehand, is needed. Moreover, the DOP theory of change is characterized by a notable lack of appreciation for USAID/District government relationship building as an end in and of itself. These issues aside, the DOP theory of change clearly established its relevance to the goals of the CDCS and offered a new modality for USAID to monitor its investments across Uganda. 19 Is DOP Efficient? Introduction This section addresses Question 3 of the Scope of Work, which we interpret as a question regarding DOP efficiency. By efficiency, we refer to the relationship between the outputs - both qualitative and quantitative - in relation to the inputs (costs and human resources). This also opens the door to a discussion of how the DOP process was implemented. The evaluative question guiding this section is: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? In answering this question, the evaluation team found it useful to view the DOP process as a relationship among USAID Representatives, District Officials and various IPs, in which SDS, itself an IP, played a special role in facilitating the process among the other three. Graphically, this is shown below: Figure 3: DOP’s Tripartite Relationship The graphic helps provide a conceptual guide for the following section. First, we review DOP implementation and how the DOP process moved from two separate implementation models to a single approach implemented by SDS (noting that SDS had only been in NUFO for less than 6 months at the time of the evaluation). We then turn to a discussion on how each member of the DOP relationship perceived the DOP and their performance on various output measures. We then conclude this section with a model to determine costs. Throughout the discussion, we find incredibly mixed views on the efficiency of the DOP, how it has been handled and what can be improved. DOP Implementation and SDS The DOP process officially kicked off with the first MoU signings in February 2012 in Mbale and Kamuli. At the signing ceremony, high level USAID representatives made statements about how the DOP represented a new phase which put the District in the driver’s seat of IP coordination and implementation in support of its own DDP.1 Yet in practice, SDS played a strong facilitation role, not only preparing the official ceremony, but also in linking the DMC to its wider technical assistance activities in the districts in which it operated. SDS was originally required to support 13 of the 19 MFDs, while the six districts in Northern Uganda were supported the Northern Uganda Field Office (NUFO). Although the six Northern districts signed MoUs around the same time as the rest (Amuru first signed on March 2, 2012), they did not benefit from the additional IP coordination efforts provided by SDS and NUFO was expected to play a stronger role to fill the gap. More importantly, Northern districts also did not receive equivalent levels of technical assistance and capacity building, which put them at a natural disadvantage.2 Indeed, SDS and several other stakeholders repeatedly expressed the view that DMC coordination meetings alone would not function without the additional capacity building support SDS provided. The evaluation team found evidence to support this, as discussed in the Effectiveness section. 1 Interview with former USAID/Uganda Program’s Office representative. 2 Note the difference between the Northern and remaining DOP District MoUs, whereas the latter makes special mention of SDS as a coordination, while the former only makes reference to IPs providing technical assistance to their respective sectoral areas. 20 NUFO The six northern districts were directly implemented by a USAID representative from the Northern Uganda Field Office (NUFO).3 Unable to provide the same level of capacity building as SDS, the USAID representative focused more on joint coordination aspects with various IPs and the Districts. The representative also had a strong relationship with UNOCHA, who had established similar humanitarian coordination meetings with various districts to avoid duplication, promote synergies and inform the district local government leaders of the contribution of the development partners during humanitarian situation in their districts.4 However, as the post-conflict efforts and the coordination meetings waned, so did the first iteration of the NUFO DOP “model” until 2014, when the Northern DMCs received additional support from the DOP Coordinator, a unique role facilitated by the USAID Development Advisors program. USAID/Uganda also considered directing the Governance, Accountability, Participation and Performance (GAPP) activity to assume responsibility for coordination of the Northern DMC meetings, but after a delayed start, lack of authority on the part of the USAID representative to direct GAPP, and reluctance on the part of GAPP to assume these functions, the idea was dropped.5 The Northern model continued until the NUFO office closed in Dec 2014, after which time SDS conducted a preliminary assessment to expand its scope of work to include these districts. They found that the Northern DMCs lagged considerably behind the other DOP districts, and noted, inter alia, that • DOP secretariats do not sit before the DOP meetings – agendas not developed in time, minutes not circulated; • Reports are presented by the individual IPs – not consolidated, heads of department never presented their activities in some districts; • Amuru district had last had a DOP meeting on 16th April 2014 (skipping the Q4 2014 and Q1 2015 meetings); and • Many district officials reported that USAID IPs never attended the budget conferences and they never shared information with the district planning units.6 SDS then assumed responsibility for the DMC meetings in the northern Districts and also provided additional technical assistance and capacity building support that NUFO could not deliver. DOP Joint Results Framework and ME&L Plan Originally conceived, the DOP Process was supposed to contribute to progress on 13 “standard indicators” that were identified in the 2012 “Common Indicator Policy” Action Memorandum. The indicators were derived from a consultative process involving various members of the DOs and the Program’s office. They emerged in various Joint Result Frameworks for each DOP district, and IPs as well as district officials were also free to include additional custom indicators they felt appropriate (Kamuli, Mayuge, Isingiro, Kasese, and Mbale all did so). The 13 indicators reflected a balanced mix of impact and intermediate result measurements, incorporated MDG measurements, and included indicators already endorsed by the GoU under its National Development Plan. The Policy also identified potential data sources, such as ongoing Lot Quality Assurance Sampling (LQAS) surveys, the Afro-barometer Survey, Feed the Future Surveys and Local Government Performance Assessments. The indicators were to be applied to the 19 MFDs as well as six “Mission Comparison Districts” (MCDs) in anticipation of a “difference in differences” research design to explore DOP attribution. Even so, the Mission anticipated that additional resources would be needed to manage the 13 indicators. 3 NUFO was an attempt by USAID/Uganda to establish a regional presence in Northern Uganda. In 2007, USAID opened a field office in Gulu in the Bank of Uganda premises, primarily to support the implementation of USAID's Office of Transition Initiatives'-OTI worth USD $14-million under the Northern Uganda Transitional Initiative (NUTI) program. NUTI was designed to aid internally displaced persons (IDPs) to resettle back into their communities from the IDP camps. The office was closed in December 2014. 4 Email correspondence with said representative. See also the “Joint United Nations Agencies and United States Agency for International Development Coordination Meeting” Report, July 29, 2014, held at the UNICEF Gulu Office. 5 Interview with GAPP CoP on July 16, 2015. Internal USAID Publication. 6 See the SDS “NUFO Concept Paper February 2015.” Internal USAID Publication. 21 The DOP Joint Results Framework failed to take hold in practice for three main reasons. First, the Framework had no clear owner who was tasked with ongoing maintenance and analysis. SDS’ revised February 2012 scope of work stated that SDS was to “Include DOP indicators (to be approved by USAID) in recipient Performance Monitoring Plan (PMP) and district-based DOPs that will assist USAID, the contractor, districts and IPs in monitoring district progress toward achieving improved sustainability of decentralized services supported.” According to SDS, USAID later agreed in 2013 to directly take ownership of the indicators while SDS would make sure they were disseminated,7 but USAID never completed this process. Second, the original indicators themselves were inappropriate measures of DOP progress. For example, a review of the sample design not only found that the six MCDs were not sufficiently equivalent to the 19 MFDs (they shouldn’t be compared) but that some of the indicators (e.g. those found in the Afro barometer) were not valid.8 Third and finally, the choice of assigning 13 standard indicators across a diversity of districts – each of which contained its own unique blend of IP activities, baseline conditions, political dynamics, other donor programs, etc. – was most likely not the best in retrospect. Rather, how the DOP contributed to development results was probably better conceptualized on a district by district basis. Standard indicators should only be adopted across contexts when either those contexts are sufficiently comparable, or when there is a research interest in contrasting their differences despite a limited degree of commonality. Those conditions didn’t clearly apply in this situation. To be sure, had the Mission developed 19 separate theories of change (one for each district), it may have found that many indicators would have appeared in several districts. Moreover, this would have been a very labor intensive exercise that might have stretched the capacity of Mission staff. Nevertheless, the common indicators still needed to be more adequately mapped to the specific contributions the DMC process was expected to add to the variety of activities unfolding across the MFDs. Ultimately, a better course of action would have been to dedicate more effort to design specific causal pathways of change that were reflective of the conditions in each district. The Joint Results Framework was abandoned in 2015 in favor of a DOP “Monitoring, Evaluation and Learning Plan,” (ME&L) developed by the Learning Contract, which contained 24 indicators and was approved in May 2015. The process for its creation was again consultative and was again flawed in terms of its attempts to impose a common set of indicators upon a diversity of conditions, interventions and contexts. Unlike the Joint Results Framework, the ME&L plan also contained a number of DMC process indicators, such as the proportion of districts conducting DMCs that meet agreed quality standards. Also unlike the previous Framework, the Plan clearly identified roles and responsibilities for data collection and collation, in which SDS and the Learning Contract were to take the strongest roles. Despite these advances, little progress had been made on the collection of this information as of the conduct of this evaluation. The Learning Contract had enlisted the aid of an intern to compile certain indicators, but, as of December 2015, 15 indicators still had data gaps, required further computation, and/or had no source whatsoever. More importantly, it is unclear who is to use this information and for what purpose. The indicators and district sampling strategy are underpowered to make any rigorous conclusions regarding DMC contribution to any witnessed changes, but, more to the point, there is no clear hypothesis with corresponding causal pathway that these indicators are testing. Conclusion Overall, the evaluation team concludes that SDS facilitation of the DOP process was adequate. The activity increasingly assumed more responsibility for facilitating the DMC meetings and the evaluation team found no major criticism of their efforts from any stakeholder we interviewed. If there is a critique to be launched against SDS, it is that the DOP’s evolution was to some extent slowed by SDS’s own lack of internal synchronization of its various components. For example, a number of tools, such as the Integrated Budget and Planning Tool, could’ve directly benefited the DOP process had they been incorporated into the DMC meetings early on. Yet this observation should not downplay the impressive effort SDS put into establishing and sustaining the DOP process. As $50 million + USD program with many moving parts, internal silo effects are to some extent inevitable, and, although it is out of the SoW of this evaluation to review SDS as a whole, 7 See USAID-SDS meeting minute summaries for April 16, May 29 and September 18, 2013 for confirmation. These minutes were distributed to USAID for approval within a week of each meeting. 8 See Kisekka-Ntale, F. and Kibombo, R. (2013). “Reviewing the Sample Design for USAID/Uganda’s Evaluation of the Effect of Integration of Activities on Development Results in Selected Districts” Final Report. Unpublished USAID Document. 22 our impression is that SDS management was very proactive in breaking down these silo effects whenever possible. In our view, SDS implementation should be considered efficient. DOP Performance and Perceptions Previous DOP Learning and Recurring Issues Since the inception of the DOP, there has been considerable effort made to encourage feedback and learning to improve the efficacy of the DOP process. While the DOP constitutes just one part of the strategy for SDS’ objective of increasing coordination between IPs and Local Governments at a district level, it has been the focus of a number of individual learning events between IPs, DMCs and USAID representatives. One of these, a “360 review” conducted by USAID in the Fall of 2013, voiced a number of reoccurring themes that other learning events and reviews continued to stress, such as the need for USAID reps to attend all DMC meetings, the need to assign specific reps to each district, the need to align activities with the district financial year and the need to participate in district planning and budgeting processes and conferences. While useful, the 360 did little to systematically examine USAID’s inputs into the process, or to hold the content of USAID trip reports to the same standard as those of the DMC. Additional reviews voiced similar themes, such as those found in overarching SDS Annual Reports, more focused CLA DOP Analysis Reports, as well as a large body of meeting minutes and reports from various learning events. The main finding from reviewing these documents is that, despite the emphasis on learning, certain issues remained unaddressed despite multiple calls for their resolution. More precisely, these are: Scheduling issues The most consistent complaint since the first DOP feedback event in December 2012 concerns poor scheduling. The quarterly frequency of the meetings is agreed during the introductory meeting in each district, but both the setting of a date and the notification of the relevant parties involved has been a struggle. Still in February 2015, at the Chief of Party roundtable, participants highlighted scheduling issues as one of the main deterrents to attendance. The two main features are that dates are set with insufficient time to prepare the necessary reports, and that changes are made to the schedule and not properly communicated. And even when the date is decided there can be delays from district officials – one DOP in Lira was rescheduled to the following day, on the morning of the agreed day. Lack of Preparation A lack of preparation or non-attendance have an obviously detrimental effect on the DMC, however the inability to overcome this logistical hurdle has the more serious effect of undermining trust in the DOP as an institution. The Chief of Party of SUSTAIN complained of waiting 6 hours for a DOP to start in Gulu, which proceeded to be of little value. Because of this poor performance, he now feels justified in delegating attendance there to a junior team member. However, a common complaint from district officials is the insufficient authority of IP representatives to enable effective decision-making during DMC discussions. Lack of Attendance One ongoing complaint made on all sides was the lack of consistent attendance by all stakeholders. District Officials complained consistently whenever perfect IP attendance wasn’t achieved; IPs complained that district decision makers were often absent; and both complained the USAID representatives were not present enough to reinforce their own viewpoints and seriousness of the meeting. This is clearly a self-perpetuating problem and can lead to, over time, a “downward spiral” in which high level officials who first attend lose interest as their counter parts send lower level officials with little authority and/or awareness to represent their respective functions. Poorly managed meetings An on-going challenge faced by SDS has been trying to maintain quality of the meetings, even though this is by and large outside of the activity’s control. The DTPC meeting format already exists, but the inclusion of IPs in the DOP expands both the scope and volume of the meeting. From the inception of the DOP, these changes were going to be twinned with capacity building and technical assistance to try and improve the abilities of the CAO in making the meeting efficient. The complaints about 5-hour long meetings with meandering or overly specific subject matter suggest the management of the meetings requires more work. 23 Overemphasis on Leadership Complaints about leadership remain one of the key impediments to a successful DMC. Even if this cannot be measured, there is a lack of focus on solving the issue of leadership. The CLA report presented a bar graph showing 100% satisfaction with CAO/LC5 effectiveness, which should spark more questions about the culture of questioning leadership in a district than it does answer them.9 Even so, the issue of leadership quality varies across the districts, and is fundamentally personality driven. The clearest reason for this is the variation in personalities of the CAOs themselves. For example, Felix Osuku, former CAO of Kamuli, was a very avid proponent of the DOP, and his own belief in the importance of the meeting led to greater participation. Clearly this kind of leadership will have a positive effect on the DOP. However, successful development policy cannot rely on exceptional personalities. Indeed, there is a pitfall in placing too much emphasis on the presence of good leaders, rather than building structures that can function regardless of the capability of the notional chairperson. IP Performance and Perceptions IP performance and perceptions here are discussed in terms of a comprehensive view of their actual attendance, and overarching discussion of IP perceptions of the benefits and disadvantages of attendance at these meetings. In doing so, the evaluation team reviewed several previous IP surveys and feedback sessions, conducted seven face to face IP interviews and administered an email based survey to 34 IPs (see Annex 13 for the survey). Unfortunately, IP response to the survey was low (only eight of the 34 responded), and their input into the questionnaire was minimal. Nevertheless, the evaluation team did compile a considerable amount of information to provide a general picture of the IP perspective. IP Performance: Attendance The evaluation team compiled all of the attendance records of all DMC meetings. In practice, there were actually two sets of attendance lists, one maintained by SDS and the other by each individual district office. The district office records were by far more comprehensive, and their records tended to better capture non￾USAID IP attendance as well. The following chart summarizes IP attendance over the past three years: Chart 1: IP Average DMC Attendance Scores, 2013-2015 IP presence in this case signifies the total number of IPs that are implementing their activity in a district, and therefore the number we can reasonably consider to represent ‘full attendance’. Some of these IPs will have not 9 See SDS (August 2014), “Collaborating, Learning, And Adapting Impact Study.” 0 2 4 6 8 10 12 14 16 18 20 0 2 4 6 8 10 12 Bugiri Bushenyi Ibanda Iganga Isingiro Kamuli Kamwenge Kapchorwa Kasese Luwero Mayuge Mbale Sironko Amuru Dokolo Gulu Lira Oyam Pader 2013 AVG IP Attendance 2014 AVG IP Attendance 2015 AVG IP Attendance IP Presence Total non-IP attendees 24 been signatories of the initial MoU, but in theory will have been contractually ‘encouraged’ to go by AOR/CORs. Interestingly, we found a great deal of variation in IP attendance trends that defy easy generalization. Some districts, such as Bugiri and Mayuge, started out with strong IP attendance but subsequently declined. Others, such as in Ibanda, Iganga and most famously Mbale, grew in strength as time went on. But by far the most common pattern was strong initial attendance, followed by a subsequent dip in 2014, and only to return to higher levels of attendance in 2015. IP attendance does seem to grow as a whole in 2015 compared to previous years, but this trend cannot be fully attributable to DMC success. As mentioned above, while attendance records are the best measure we have of IP performance, we should not equate attendance with buy-in on the part of IPs. The ‘downward spiral’ effect of fulfilling contractual obligations to attend DMCs by sending junior representation would not be immediately apparent through attendance records alone. Moreover, non-USAID IP attendance also wildly varies, and the only clear pattern that seems to emerge is a pronounced lack of non-USAID IPs attending DMC meetings in the north, despite the assertion that these were designed to layer on top of already existing intersectoral emergency response meetings coordinated by UNOHCA (see above). A further notable finding across all MFDs is that we saw no examples of sustained attendance at a DMC by a non-USAID IP. IP Perceptions of the DOP Variation in IP attendance may reflect the generally negative view by IPs of the DOP process. According to the survey administered by the evaluation team, there were mixed responses to the concept of further DOP integration that we have read in past reports from SDS evaluations. The misalignment of budgeting cycles between the district and USAID mean the IPs are limited in what they really can commit for all but the 2 month overlap (September – October). Increased involvement in the district planning process is seen as helpful, particularly for aligning with the District’s priorities, but this does not necessarily contribute to substantive change. Structural inflexibility on both sides means that ‘sharing’ does not necessarily result in integration, despite best intentions. Sharing of budget information seems to be the most complex, particularly due to the confusion about the details of IP spending. Staff from both the Gender Roles Equality and Transformation (GREAT) and Strengthening Uganda's Systems for Treating AIDS Nationally (SUSTAIN) activities suggested that the district officials were mainly focused on how to control resources that are outlined in the budget. While IPs acknowledge the meager resources of the district are inhibiting to action, the desire to funnel more IP spending through the district is considered to be risky due to weak accountability. However, through being more closed about their spending, IPs increase the suspicion around their project which adversely affects their standing in the District. Despite the extended effort of SDS to facilitate this budget information sharing, the IP responses seem largely negative about their experiences. On the positive side, there was unanimous agreement that the increased visibility afforded by representation at the DMC is beneficial for IPs implementation in the district. CARANA (Uganda Value-Added Maize Alliance) were keen to state they already had good relations with DPOs, DAOs and DCDOs in the districts where they implement, however the value of the DMC is the ability to reach actors in the district outside of the particular department heads. IPs did also see particular benefit in garnering political support both at a district and sub￾county level when the project relies on community involvement. The increased understanding of the aim and objectives of an activity by district leaders is considered fundamentally beneficial to IPs ability to implement. NUCAFE also suggested that through developing relationships with the district leadership they have also improved the outputs of their project – particularly through incorporating district officials in their organized activities. Through their involvement in the DMC, these IPs felt that they increased the representation of their farmers to the district’s policy makers. The best examples of where the DMC has directly influenced IPs implementation are through the opportunities for collaboration that arise through creating the forum to share plans and problems. Direction about where to implement seems to be particularly well received – in particular knowing the details of other IPs activities in the district. Most notably from the responses was how SUSTAIN benefited from the DMC in Mbale in working 25 out with other IPs where best to spread the coverage of their hub and laboratory to avoid overlap with other IPs. Similarly, through joint monitoring with the DHO in Mbale they were able to avoid double employment of health center staff. There is of course communication outside of the DMC, but the increased awareness of activities that DMC offers leads to a better understanding of potential pitfalls. The most positive examples we found where situations in which IPs managed to collaborate as a result of the DMC. The potential modularity of activities means that while coordination with the district is quite common, building relationships with other IPs can be more complicated and hence the DMC’s role in facilitating this is appreciated. STAR-E in particular sites the benefits of using Communication for Healthy Communities’ (FHI360) IEC materials for their health education, as well as the improvement in referral pathways for OVCs through coordination with SUNRISE, SCORE and ChildFund. The responses from the IP survey suggest that this IP-IP collaboration is one of the most tangible benefits of the DMC meeting. The question in the survey to which we got the most collective response from IPs was related to their negative experience of the DMC meeting itself. All the IPs highlighted the issue of poor organization on the part of the district, where changes to the schedule are not communicated properly and meetings start late when they finally agree a date. In particular, IPs highlighted that they have to make considerable effort to attend the DMC, which often involves adjusting their planned activities as the meetings can be scheduled at short notice. The frustration is notable, therefore, when the district will reschedule or delay a meeting at short notice – sometimes even on the morning of the meeting. The other main complaint we found, consistent with previous reports’ findings, was poor timekeeping by the districts. Meetings will start late due to the delay of necessary members (CAO/department heads) and then the meetings themselves can proceed to be long and unfocussed. SAFE (NCSC) in particular claim that one of the key limits to the DMC is that the district does not seem to own the process – even where the meetings are conducted in a timely fashion, the key issues are not followed up. The universal complaints about timekeeping suggest that the DMC is not a priority in the district, and while USAID can keep IPs accountable there is little the IPs can do to coerce the districts to hold up their end of the agreement. USAID Performance and Perceptions USAID representative performance and perceptions here are discussed in terms of a comprehensive review of USAID field trip reports and insights garnered from two focus group discussions. The focus groups were useful mostly in terms of capturing USAID’s self-defined role in the DOP process. Unfortunately, focus group turnout was low, and we suspect our responses suffer from a positive social response bias. The evaluation team sought a measure for determining USAID performance at DMC meetings, and turned to an analysis of trip reports. The trip reports also tell an incomplete picture, but nevertheless do reveal to what extent USAID representatives share their DMC experiences with the rest of the Mission. USAID Representative Performance: Trip Reports Officially, each USAID representative was required to both sign into DOP attendance sheet as well as submit a field trip report. In practice, we found evidence that many USAID reps had attended without signing the sheets – so much so that we were unable to construct a reliable USAID rep attendance report. As a result, we were only able to analyze trip reports. We also found that field trip reports were compiled in a variety of different places, e.g. on the USAID/Uganda “P drive” shared folder, on the hard drives of various USAID reps, and on the files of the USAID DOP Coordinator. No one had compiled a comprehensive database of these reports, which may suggest that Mission saw these reports as procedural rather than meaningful. In terms of use, the trip reports seemed to have served two main purposes: 1. To confirm representatives attended, and 2. To inform a 2013 “360 Review” of how well the DOP was working (see above). As such, the evaluation team used the trip reports as an opportunity to review the content of USAID Representative reports on the DOP experience. We conducted a review of all USAID representative trip reports from the first DMC meeting until the last wave of Fall 2015 meetings. For each meeting that took place, USAID had committed to send a delegate who would be responsible for taking notes and submitting a field report that would summarize discussion points and recommend action items. Had this been executed at 100% USAID attendance, there would have been 266 field reports (19 districts participating in a maximum of 26 14 DMCs each). However, only 160, or 60%, of the required reports were submitted. Of the 160 reports that were submitted, 37.5% only contained content on procedural items. Reports that were rated this way tended to simply state the names and/or titles of speakers, and contain a few words about what that person spoke about. There was little to no analysis or critical thinking involved in these field reports and even summaries were incomplete and contained little information that was valuable or actionable. The remainder of the field reports (62.5%) contained information that was substantive or actionable, though only half of these were both substantive and actionable. In all, 66 of the field reports (or 41%) included recommendations of some sort for action items or next steps. Unfortunately, few reports included action items that related to project activities or to improvement of the implementation landscape. In all, across 3+ years of DMC attendance by USAID representatives, 51 DMC field reports included recommendations/action items for improvement of actual project implementation. Most recommendations revolved around how to make the DOP itself run more efficiently or other procedural elements. Recurring recommendations included: “start DMC meeting on time,” “Enhance DOP engagement with IPs,” “Ensure IPs invited to DMC meetings,” “stress to IPs that DMC meetings are important,” and “improve attendance at DMC.” Twenty eight reports from following quarters demonstrated progress, or at least follow-up, which represented only 17.5% of the total reports submitted saw any sort of follow-up as a result. Further, when considering that there should have been a total of 266 field reports, this implies that just 10.5% of DMC meetings directly resulted in any sort of documented action or follow up. USAID Representative Perceptions In contrast to the negative views harbored by most IPs, the evaluation team found a decisively more positive view among the USAID representatives contacted. The evaluation team conducted two focus group discussions with USAID representatives during the evaluation. There was obviously a positive bias in these focus groups, as they were the views of only those most motivated to come to the discussion and were also the most regular attendees at the DMCs. Yet this bias was unsurprising, as USAID representatives were selected through a volunteer model in which no representative was required to attend any DMC. The most useful finding of the discussions revolved around USAID’s perceived role in the DMC meetings. Overall, this can be understood in facilitation terms, but three important responsibilities were unanimously agreed upon: 1. Enforcement: All agreed, as did district officials and IPs, that the presence of a USAID representative meant it was more likely CAOs would attend, IPs would send higher level representatives, and the meeting would be better prepared. 2. Clarification: District officials tended to take what USAID representatives said more seriously than if the same information came from IPs. For example, many districts are confused by how IP budgets are structured and why they don’t see more resources flow into their district. Also, one respondent gave a powerful example of how she threatened to walk out of the DMC if sitting allowances were brought up again, and this topic hasn’t been discussed since (IPs reinforced the value of this role, and stated how now it was easier for them to use USAID-set per diem rates with the USAID representative there). 3. Arbitration and Follow-up: At time, issues emerged in the DMC meetings in which various IPs needed USAID authority to find a solution. Even if the USAID representative present at the meeting didn’t have direct responsibility, s/he could take the information back to Kampala to be addressed through the proper processes in place. Additionally, USAID representatives that met with the evaluation team felt strongly that their experience attending the DMC meetings was productive and worthwhile. Respondents placed particular emphasis on how attendance gave them an opportunity to hear alternative viewpoints regarding the quality of IP implementation; viewpoints seldom captured in regular IP reports. In Kasese, this awareness was even more profound as the DMC began to include field visits in their agenda. One of the challenges with being a USAID rep was its volunteer nature. Only those most dedicated could regularly attend the DMC meetings, as attendance was not part of their job description and they oftentimes had 27 to struggle with supervisors to obtain permission for field visits. They suggested that the Mission consider modifying their own key performance indicators to include DMC attendance. USAID representatives also expressed a general level of skepticism regarding the capacity of most districts to cope with the extra demands of the DOP. To be sure, there were sporadic examples of where the CAO or LC5 seems to provide impetus to the USAID-ideal of having a District taking ownership over its relationship with IPs (Mbale being the most cited). However, even in these cases where the political will was present, the massive gap in capacity of the districts compared to the kind of processes IPs/SDS/USAID require means technical assistance was critical. Finally, the act of participating in the focus group itself was informative. Despite the fact that these were the believers who had previously filled out trip reports, some stories were being shared for the first time about examples of how they have held districts to account to prove they are fully engaged in the process (e.g. the example mentioned above where a representative threatened to walk out of the meeting if the subject of sitting allowances came up again). Some focus group participants suggested that USAID internally should conduct similar exercises as an additional avenue for information from the field to reach the Kampala office. DMC Performance and Perceptions The bulk of district outcomes connected to the DOP process are discussed in the Effectiveness section of this evaluation. Here, we instead turn to internal DMC performance through a comprehensive review of all action items that emerged across every DMC meeting. We also briefly discuss district perceptions of the process; mostly confirming what previous learning exercises discovered. DMC Performance: Action Items The evaluation team aggregated DMC meeting minutes, action items and presentations from all 19 DOP districts, USAID and SDS. Through this process the team identified a total of 1428 action points across all districts. Of these, a little over half (57%) were procedural items, dealing with the logistics of the meeting, reporting formats and attendance. The remaining 43% were substantive items dealing with activities, or community, IP and/or district implementation concerns (and were not procedural). In total, only 21% of these items were resolved, with the majority (44%) disappearing entirely. Table 1: Action Item Summary Type Total Percentage Total number of action items 1428 Total procedural action items 811 56.79% Total substantive action items 617 43.21% Resolution Status Total resolved action items (as of October 2015) 297 20.80% Resolved procedural items 186 22.93% Resolved substantive items 111 17.99% Unresolved action items 325 22.76% Disappeared action items 631 44.19% In Process 175 Many respondents in the districts point to the importance of the DMC as being an open space for problem solving and collaboration, as well as a forum that emphasizes accountability. They frequently pointed to the action items as a strong mechanism to reinforce both the problem solving and accountability. They stated that the DMC action items mechanism holds people accountable by ensuring that items are not lost over time, that 28 each meeting the item is brought up until it is resolved and creative solutions are discussed to address arising problems. The evaluation team attempted to collect all relevant DMC items during their meetings with district officials. However, it was rare that a district maintained a complete record of all DMC material. Oftentimes records were spotty at best, located on numerous computers and hard copy locations. This created obvious data gaps for the evaluation team and action item calculations. But more importantly, it points to larger documentation issues at the district level, suggesting that from one meeting to the next, meeting minutes and action items potentially get lost, which could explain the high disappearance rate. Additionally, the reporting format changes depending on the secretariat for the meeting. At times action items were imbedded in meeting minute discussion, other times they were broken out into a chart whereby the secretariat could quickly check in with the person responsible and measure progress against previous action items. The shifting format also accounts for the high number of lost action items. Looking deeper into the substantive action items, only 18% (111 items across around 170 meetings in 19 districts) of all substantive items were actually resolved over time. It is fair to assume that meeting discussion and action items would center on procedural elements of the meeting at the onset to create a uniform reporting format and get all stakeholders onto the same page. However, the action items highlight that DMC leadership was at times stuck in the minutia of the meeting and reporting rather than the substantial information sharing, which is the stated aim of the DMC, with 57% of all action items falling into the procedural category. The figure below of action item trends suggests that action items have not moved away from the procedural in three years. Line 1 depicts the quarterly average of all action items that are substantive in nature (the procedural can be understood as the remaining amount of action items, i.e. 100%-the substantive). This line does not change much from quarter to quarter, and we actually see a downward trend in recent quarters. Line 2 represents the percentage of substantive items that are resolved from one meeting to the next. There is a bit of variability, however no clear trend, suggesting that DMC leadership and members have not progressed toward accomplishing more substantive actions during meetings. In most districts, substantive items were largely line department specific and often dealt with health, nutrition, or OVC work. Many of the action items are from the health or community based services department presenting their goals for the coming quarter and targets met in the last reporting period. Important to note is that it is rare for an action item to cross line departments, or to involve more than one IP. Most action items were simply a to do list for district staff, and on occasion they involved an IP interaction. Finally, action items were oftentimes inactionable; they were either vague actions and/or the resources to accomplish the action were not available. There were few instances of action items leading to more substantial, significant DOP outcomes. This is largely because action items are treated akin to a task list. DOP outcomes tend to be more substantial, requiring more actors and more frequent follow through. Many action items that had the potential to go this direction disappeared completely. Chart 2: Action Item Trends 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% L1: % Substantive Items L2: % Resolved Substantive Items 29 District Perceptions District perceptions of the DOP process, relatively speaking, have been the most positive and favorable. Undeniably, District officials now have an improved understanding of USAID’s contribution to their population’s social improvement. As was commonly stated, officials would previously observe a variety of white Land Cruisers, branded with the USAID logo, but have little idea what they were doing. Now, as a result of the DMC meetings, officials are much more confident of what is happening in their district. Many district planners, statisticians and community development officers also expressed gratitude for the parallel capacity building efforts of SDS. In connection with the DOP, they indicated that they now approach bureaucratic challenges more methodologically, analytically and with increased confidence. Finally, most CAOs find the discussion section of the DMC meetings particularly useful, and indicated that they are able to openly discuss and debate issues at these meetings in ways that they can’t at the regular DTPC meetings. The most negative view expressed of the DOP process by various districts was that it wasn’t “DOP enough.” Most CAOs were quick to criticize anything short of perfect attendance from both the USAID representatives and the IPs. Another common complaint was IP representation, as many CAOs felt that only IP CoPs should attend and were slighted when, say, M&E officers were the only representatives. A substantial number of officials also felt that IPs as well as USAID reps should attend annual budget conferences. By far the biggest complaint revolved around IPs who failed to submit workplans and budgets. Despite earlier claims that the district was in the driver’s seat, district officials, especially planners, felt a borderline sense of resentment that they were required to continuously pester IPs for this information when they were, according to the law, obliged to submit this information of their own accord. Some district officials also asked the evaluation team why USAID didn’t submit this information directly to the districts. Finally, despite the improvements of moving from the DMC to the eDTPC, some CAOs expressed reservations regarding the legality of this move. In Luwero, the CAO reportedly refuses to combine the two precisely for this reason. Other CAOs suggested that for this to continue, they would feel more comfortable if MoLG issued a circular authorizing this move. DMC Meeting Costs As part of the evaluation, the evaluation team committed to developing a cost model to project cost estimates of running the DOP meeting, not only in the 19 MFDs but also in other districts in the country. The main purpose of the cost model is to establish the unit cost of delivering the quarterly meeting using historical spending data, as well as identify projected costs for all Ugandan districts, based upon varying transport, Per Diem and lodging costs. The model includes an instruction manual, which is found in Annex 14. The reason for the cost model was partly connected to previous work done on understanding meeting costs. SDS had estimated meeting costs as follows: Table 2: SDS Estimated Costs per DMC Meeting SDS direct cost per DMC Meeting Description Cost (USD) Labor (15 days @ $120/day) 1,800 Per Diem (3 trips, 3 days each) 450 Transport 835 Stationary 50 Total $3,135/meeting These estimates are more limited and do not include management costs, IP attendance costs or USAID attendance costs. In response, the evaluation team collected IP cost estimates through its survey, worked with SDS to estimate management costs and made additional estimates regarding USAID attendance. 30 The full model is attached in Annex 16, is fully customizable and presents cost estimates for running a DMC meeting in every district in Uganda. However, the model makes three important omissions that could not be captured but nonetheless underestimate the total costs associated with DMCs. These are: • Indirect Costs: IP NICRAs, G&A, Overhead, Fringe and other indirect rates are not included in the model, as these are procurement sensitive pieces of information. These rates should be added by USAID when making a decision about the efficacy of DOP costs. • Capacity Building Costs: The model assumes that some type of capacity building activity is present, e.g. SDS, which undoubtedly reduces meeting preparation time and improves the quality of the DMC meeting. The cost estimates should not be understood to remain valid should additional capacity building efforts not be in place. • IP Opportunity Costs: These primarily refer to IP senior management opportunity costs to travel to and attend DMC meetings instead of fulfilling other important functions. According to IP interviews, these are substantial, esp. for IPs who operate in multiple districts. For example, for the COP to attend all DMC meetings if it operated in all districts, it would require nearly 40% level of effort per quarter and would most likely require the creation of a DCOP specifically to fulfill that position. Cost Estimates The model presents three different attendance scenarios: Low, Medium and High. The model also estimates start-up costs, which include pre-signing meetings and the actual MoU signing ceremony. For the 19 districts, the estimates are as follows: Table 3: DMC Meeting Cost Estimates As can be seen, the estimated costs per DMC meeting are substantially higher than those estimated by SDS when looking only at their direct costs. Depending on levels of attendance, these average costs can range from Low Attendance Medium Attendance High Attendance Start Up Amuru $ 5,962.00 $ 6,813.16 $ 7,915.49 $ 1,032.33 Bugiri $ 5,300.95 $ 6,041.94 $ 6,923.92 $ 811.98 Bushenyi $ 5,943.32 $ 6,791.38 $ 7,887.48 $ 1,026.11 Dokolo $ 6,081.51 $ 6,952.59 $ 8,094.76 $ 1,072.17 Gulu $ 5,999.34 $ 6,856.73 $ 7,971.51 $ 1,044.78 Ibanda $ 5,887.30 $ 6,726.02 $ 7,803.45 $ 1,007.43 Iganga $ 5,181.43 $ 5,902.51 $ 6,744.65 $ 772.14 Isingiro $ 5,816.34 $ 6,643.23 $ 7,697.01 $ 983.78 Kamuli $ 5,282.27 $ 6,020.15 $ 6,895.91 $ 805.76 Kamwenge $ 5,917.18 $ 6,760.88 $ 7,848.27 $ 1,017.39 Kapchorwa $ 5,808.87 $ 6,634.52 $ 7,685.81 $ 981.29 Kasese $ 6,133.79 $ 7,013.59 $ 8,173.19 $ 1,089.60 Lira $ 5,935.85 $ 6,782.66 $ 7,876.28 $ 1,023.62 Luwero $ 5,040.82 $ 5,738.46 $ 6,533.74 $ 725.27 Mayuge $ 5,286.01 $ 6,024.51 $ 6,901.51 $ 807.00 Mbale $ 5,573.58 $ 6,360.01 $ 7,332.87 $ 902.86 Oyam $ 5,838.75 $ 6,669.37 $ 7,730.62 $ 991.25 Pader $ 6,316.80 $ 7,227.10 $ 8,447.69 $ 1,150.60 Sironko $ 5,659.48 $ 6,460.23 $ 7,461.72 $ 931.49 Average Cost Per Meeting: $ 5,735.03 $ 6,548.37 $ 7,575.05 $ 956.68 DMC Meeting Cost Estimates, Per District (in USD) 31 around $5700 to $7500 per meeting. Spread out over the course of around 170 DMC meetings, the total estimated costs to date range from about $988,000 to $1,292,000 USD. Conclusion Overall, the evaluation team found that the efficiency of the DOP process cannot be separated from the wider local government capacity building activities of SDS. As many observed, coordination without additional capacity building does not improve USAID/GoU relationships nor does it improve development effectiveness. Moreover, SDS’s implementation of the DOP process, originally in 13 districts but then expanded to all 19, went through considerable learning overtime on how to improve its efficiency. As the facilitating activity, SDS performed relatively well. Views on DOP efficiency are decisively mixed. The evaluation team found that much deeper challenges to the IP/District Official/USAID relationship persisted throughout the period of implementation, such as persistent problems with attendance, meaningful action items, meeting organization, decision making, documentation and leadership. IPs in particular hold the most negative attitude of the DOP process, and the opportunity costs of their attendance is high. USAID representatives, on the other hand, have gained considerably from their attendance and have improved both their relationship with district officials as well as their own situational awareness of implementation quality. Finally, district representatives appear to have gained the most from the DOP process in terms of their understanding of how USAID supports improvements in their districts. Estimated costs for these gains run about $7500 USD per DMC meeting. Although seemingly expensive, there currently is no other intersectoral modality in place to bring together USAID reps, IPs and district officials. In the next chapter, we explore if the outcomes attributed to the DOP process are commensurate with the financial costs, but retain concerns over the opportunity costs incurred by IPs (which we can’t quantify). 32 Is DOP Effective? Introduction This section addresses Question 3 of the Scope of Work, which we interpret as a question set squarely focused on DOP effectiveness. Effectiveness sections seek to measure of the extent to which an aid activity attains its objectives and the major factors influencing the achievement or non-achievement of the objectives. For this scope, the question set is: Has the DOP process contributed to changes to collaboration, alignment, coordination and integration (CACI) of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? What explains variations of these across the DOP and non DOP districts? The most pressing agenda item facing the DOP evaluation revolves the question of its effectiveness. In order to justify its existence, the DMC meeting, regardless of its efficiency, cannot be an end in and of itself, but must contribute to wider processes of social change and/or relationship strengthening. Yet, as discussed in the Relevance section, previous iterations of the DOP theory of change remained anchored in a classic performance monitoring framework of moving from output (DMC coordination meetings) to outcome (improvements to process) to impact (development objectives improved), even though the nature of the DOP process suggested a more nuanced, “complexity-aware” framework in which cause and effect relationships are less understood.42 The earlier results framework and subsequent ME&L plan only exacerbated this problem, as they both suggested that progress across a variety of complex and indeterminate processes could be sufficiently gauged through a basket of common indicators without clearly delineating the process pathways for these outcomes to be achieved. This poses a significant challenge for the DOP evaluation, as we lack an adequate framework to assess DOP’s contribution to the CACI changes and relationship improvements sought by this evaluation question. Proposed Framework for a Revised DOP Theory of Change To assist, this section offers an alternative evaluative framework that better captures what is referred to as the “stochastic” (i.e. random) nature of DOP outcomes.43 Rather than imply DOP outcomes follow linear pathways, a stochastic framework explicitly recognizes that elements of complexity and uncertainty increasingly come into play as we move farther away from the DMC meeting and closer to the development outcomes the DOP ultimately seeks to achieve. As a result, our ability to predict these outcomes becomes increasingly obscured by a growing level of randomness the farther we move down the results chain, correlated to the increase in external factors affecting outcomes. Yet this is not to suggest that predicting DOP outcomes is an entirely arbitrary and designless exercise. Rather, a stochastic DOP theory of change recognizes that by increasing the number, frequency and intensity of decision makers who come together to exchange information and discuss solutions at the DMC, the likelihood of subsequent processual outcomes also increases, as does the likelihood that these outcomes may also contribute to more tangible outcomes further down the chain. This revised theory of change has been inductively derived from the empirical evidence collected by the evaluation team (see below). More precisely, DOP outcomes are conceptualized to unfold across a spectrum of coordination, collaboration, formal action and hard (development) outcomes. Coordination outcomes, as the name implies, involve additional coordination meetings, information exchanges, awareness raising and update practices that occur outside the DMC meeting. Collaboration outcomes imply working together jointly towards the same goal, but usually independent of each other and in ad hoc/one-off fashion. Formal actions are those that usually imply some kind of written regulatory action, such as a district circular or policy change, and are more systematic and/or regular in character. All three are procedural types of outcomes. Finally, “hard outcomes” are those associated with development indicators, improvements to the lives of beneficiaries outside of the bureaucratic structure and social change. This is graphically depicted as follows (note: the data points are hypothetical and not based upon actual data): 42 PPL (2013). “Complexity Aware Monitoring Discussion Brief,” USAID Online Learning Lab publication. Found at: http://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief. 43 Random generally means unrecognizable, not adhering to a pattern, although some mathematicians seem to use "random" to also mean uniformly distributed. Random variables are subsets of stochastic processes. Stochastic processes here are defined as a collection of both predictable and random variables strung together to lead to multiple outcomes in a variety of different ways. 33 Figure 4: Revised DOP Theory of Change As suggested above, we expect to find a higher concentration of coordination and collaboration outcomes connected to the DMC meeting than formal actions and hard outcomes. Note also that, because the DMC meeting is primarily a forum for coordination, we expect to find a higher concentration of collaboration outcomes than coordination ones (coordination within the DMC is not an outcome). Moreover, the scattered distribution of the outcomes suggests that, although randomly falling across the spectrum, they are contained within certain boundaries of complexity connected to the DOP process. These boundaries grow wider over time, but in doing so, the influence of the DOP grows weaker and weaker as an increasing number of factors outside of the DMC’s control come into play in complex fashion. The evaluation team understands that moving from the intentional to stochastic approach is difficult in high stakes development contexts. Theories of change with the clearest objectives that offer the most accessible, tangible pathways to change (irrespective of their feasibility) tend to trump those that recognize uncertainty exists and can’t offer upfront promises of return. Indeed, despite various calls in international development to move to a post-logframe era, M&E experts still struggle to offer an alternative to the deceptive simplicity of moving from outputs to outcomes to impacts.44 Nevertheless, in cases like the DOP process and policy initiatives, we maintain that assuming linear approaches are not appropriate and do more harm than good. Measuring DOP Outcomes: Outcome Harvesting To address the needs of the evaluation scope of work, USAID/Uganda instructed the evaluation team to review a number of methodologies and approaches, including Outcome Harvesting (O/H).45 The O/H approach was emphasized primarily due to its goodness of fit with the DOP stochastic theory of change and its emphasis on capturing a wide variety of outcome types that would otherwise go unnoticed by more traditional 44 See, for example, Hummelbrunner, R. (2010) Beyond logframe: Critique, Variations and Alternatives, in ‘Beyond Logframe; Using Systems Concepts in Evaluation’, Tokyo, FASID, found at https://www.fasid.or.jp/_files/publication/oda_21/h21-3.pdf; Grove, N and Zwi, A, (2008). “Beyond the Log frame,” Development in Practice, Vol 18, #1, February; SIDA (2005), “The use and abuse of the logical framework approach,” SEKA, found at http://www.intrac.org/data/files/resources/518/The-Use-and-Abuse-of-the-Logical￾Framework-Approach.pdf; Sigsgaard, P. (2008). “Logical Frameworks: Problems and Potentials,” online post at http://www.petersigsgaard.dk/PDFfiler/gasper_logical_framework_problems.pdf ; and Davies, R. (2005). “Moving from Logical to Network Frameworks,” found at http://www.mande.co.uk/docs/MMA.htm 45 See the Phase 1 DOP Protocol Report for a discussion of the methodologies reviewed. 34 approaches.46 O/H has proven to be especially useful in complex situations when it is not possible to define concretely most of what an intervention aims to achieve, or even, what specific actions will be taken over a multi-year period.47 The evaluation team applied O/H to all 19 DOP districts, primarily visiting technical staff members such as the CAO, DHO, DPO and others. Whenever possible, council members were interviewed. The evaluation team also visited six additional non-DOP districts that also were not SDS-supported. The non-DOP districts were selected to answer the questions around the DOP counter-factual: what would happen if DOP or a similar intersectoral coordination mechanism was not in place? Would we see similar outcomes emerging with the same frequency? Non￾SDS districts were selected because SDS also facilitated a bi-annual intersectoral coordination meeting (under its Objective 1 mandate) in non-DOP districts similar to the DMC meeting (had we selected these districts, we’d be addressing questions around what is the optimal frequency for intersectoral coordination meetings, e.g. 2 vs. 4 times a year?). During the evaluation period of performance, representatives at one of the six districts could not be interviewed as they were attending a DTPC meeting (despite their confirmation that they would be available and free). We also applied it to 33 IPs via an electronic questionnaire, as well as through direct interviews to seven IPs. The approach was less rigorously applied to two focus groups involving USAID representatives, primarily because focus group forums were less conducive to the specificity requirements of the approach. For the purposes of this evaluation, we define a DOP outcome as an observable change in the relationships, agenda, policy or practices of one or more social actors in which attendance at a DMC meeting or participation in other aspects of the DOP process has contributed. Crucially, these outcomes cannot be contained within the DMC meeting, but must have occurred subsequently and outside of the meeting. Broken down, we applied three criteria for an outcome to be considered a DOP outcome: 1. Clear description of what exactly has changed: Outcomes that could only be described in terms of gerunds or participles (e.g. further strengthening, improving, etc.) with no clear change in condition or change in state were not considered outcomes. 2. Specificity: Respondents must be able to answer specific questions about when the outcome occurred, how it occurred, who was involved, where it occurred, and what were the specific chain of events leading up to the outcome, as well as the subsequent chain of events that can be connected to the outcome. If the outcome could not pass these qualifications, it was labeled “fluffy” and was dropped from the analysis. 3. Contribution: Respondents were then asked to work backward to link the change to participation in the DMC meeting or some other aspect of the DOP process. This was often done by posing three different scenarios to the respondent: a. if the outcome still would’ve occurred had the DOP process not been in place, b. if the outcome was also connected to SDS’ and/or another IP’s wider scope of work, and c. if the outcome could have also been realized at the DTPC. If the respondent could plausibly argue that the outcome would not have occurred without the DOP process and in any of the previous three scenarios, it was scored as a concrete DOP outcome. If the outcome could’ve still occurred under the three scenarios but the respondent felt that the DOP process still contributed (by, say, speeding up the process), then it was scored as a potential DOP outcome. Special attention was paid to outcomes involving non-USAID IPs and other stakeholders. These were scored as non-IP DOP outcomes, and only concrete non-IP outcomes were captured in DOP districts (no potential non-IP outcomes were logged). If the outcome had no connection to the DOP process in a DOP district, or if it was harvested in non-DOP districts, it was scored as a non-DOP outcome. Finally, a random set of outcomes were “substantiated” in terms of confirming their existence and accuracy by cross checking with separate IPs, district officials, USAID representatives, other stakeholders as well as various documents (trip reports, workplans, annual reports, meeting minutes, etc.) 46 Heather Britt, one of authors of the O/H manual, also authored the USAID paper on the complexity awareness monitoring. Thus, it should come as no surprise that the approach is compatible with the framework. 47 Taken verbatim from the O/H page on http://betterevaluation.org/plan/approach/outcome_harvesting 35 Respondents were asked to provide as many examples of outcomes as they could. When the respondent ran out of examples of outcomes that could be harvested, the interview was completed. In this sense, the outcomes we harvested are said to represent a “comprehensive” catalogue of what the DOP process has produced. However, at times it was inevitable that some key respondents were not available and could not be revisited during the evaluation period of performance. As such, it is still possible that additional outcomes connected to the DOP process were not captured by the evaluation team. Yet in the Ugandan context in which district official absenteeism and unexpected schedule changes are both quite common, any approach adopted would face similar barriers no matter what time of the year it was launched. As such, we maintain that our approach is still the most comprehensive attempt to capture DOP outcomes to date and is unlikely to be surpassed by alternative reasonable efforts. Outcome Findings Overall, the evaluation team uncovered and harvested 211 outcomes. Of these, 75 are concretely connected to the DOP process (and include seven DOP outcomes that involve external stakeholders); 74 are potentially connected to the DOP process, but probably would’ve occurred anyway; and 62 are outcomes that cannot be connected to the DOP in that DOP district (24 outcomes) or occurred in a non-DOP district (38 outcomes). Note that it is highly likely that there are an event greater number of Non-DOP outcomes in DOP districts than what is presented here (esp. hard outcomes involving IPs), but these are systematically under-represented as they fell outside the scope of work. Instead, this category is better understood as potential DOP outcomes that were subsequently proven to have no relationship to the DMC, but we cataloged anyway for reference purposes. Once all outcomes were harvested, the evaluation team came together to categorize the outcomes into various “buckets.” In doing so, we first tested the goodness of fit with the original DOP theory of change buckets (e.g. systems effect, efficiency gains, etc.) as well as the second “DOP causal pathway” buckets offered in the second version of the DOP theory of change. Neither fully captured the range of outcomes we had harvested, and many of those buckets were not discrete but instead bled into each other. We were then forced to build our own bucket list, but did manage to take some of the previous buckets forward. This was an iterative process, and involved several revisions, removals and re-conceptualizations. The result was 15 discrete buckets, ranging from five to 29 outcomes in each one. Each bucket is listed below, with the number of instances per bucket included in parentheses, as well as descriptions of each bucket and examples. The descriptions are as follows: 1. Reporting and information sharing (14): New types of document reporting and/or compliance with previous reporting requirements are created. IPs report to district outside of DMC/DOP requirements. For example, in response to the knowledge gap identified in July 2015 DMC in Bugiri, Straight Talk volunteered to share locations of hotspots of new HIV infections to assist the DDHS in targeting activities. 2. Duplication (19): District and/or IPs gain information through the DOP process that enables them to avoid/eliminate duplication. This applies to both IP and/or Non-IP. For example, in Kamwenge in 2015, the DHO avoided duplicating HMIS training at many private health facilities that were already offered by PACE due to DMC reporting. 3. District-led coordination of involving IPs into an existing district process and regular events (6): For example, in Kapchorwa in 2015, the CAO advised IPs attending the DMC meetings to attend and participate in the the Annual Budget Conference including three non- USAID partners (KACSOA, KPHA, and Action Aid) as well as STAR-E, and SDS. 4. IP to IP collaboration (9): IPs within the same sector collaborate when they normally would not have. For example, in Bushenyi in 2013, Red Cross reported plans for the upcoming Day of the African Child and four IPs joined and supported the activity. 5. Monitoring (7): New/expanded site monitoring visits involving IPs and District Officials. For example, in 2013 Community Connector and the district production team Ibanda coordinated field visits to strengthen monitoring and project sustainability. 6. Change in/addition to IP implementation based on district need (20): Here, need means a specific, ad￾hoc problem has been identified or district has asked for assistance). For example, in the December 2014 DMC meeting in Iganga, Musana and the district agreed to share the 300,000 UGX cost of resettling a child as an emergency response when the district did not have available funds. 36 7. District and IP collaborate/conduct field activities/trainings (12): Note, this is not the same as monitoring visits. For example, in October 2015 in Mbale, the production department’s annual agricultural show was implemented for the first time through partnerships with IPs. 8. Departments advise IP where to implement (4): For example, in Bushenyi in 2014, based on referral linkages established in the DMCs, the CBS department coordinated with STAR‐SW to deliver services in an OVC case. 9. District coordinates IP activities (7): For example, in Dokolo, the DHO raised the DDP priority of reaching national rate of health center births (50%). In response and stated at the DMC, NU-HITES hired and paid additional midwives in order to make deliveries possible in more HC2 health centers. 10. District arbitrates IP conflict (7): This usually occurs when USAID staff are not aware of it, and requires District authority to solve. For example, in Kapchorwa in 2014, there was negative community perception regarding how KACSOA was delivering services at the Kapewa Health Center. The DDHS followed up on contracting issues and sensitized both community members and staff on proper drug procurement procedures to solve the conflict. 11. New district process instituted (24): New district process area developed to respond to identified issue. Here, district changes the way they do things and their behavior. For example, recently in Kapchorwa, as a result of discussions in the DMC, the CAO was given a new role of directly following up on rape charges that fall off of the docket before proceeding to court. The CAO now does this regularly. 12. Political reporting/outcomes policy changes (6): For example, in Lira, the education sector working group raised the issue of prompt attendance at school in the DMC last year. The CAO then agreed to draft an ordinance that compelled parents to have children to school by 8am, which was passed by the LC5 within a month. 13. District offers new/expands existing service provision (6): For example, following a resolution proposed by an IP in the Oct-2015 DMC in Oyam, the district upgraded an additional number (beyond the legal requirement) of its HC2 and HC3 health facilities by relocating staff and medical supplies. 14. Availability and/or quality of services improved (hard outcomes) (7): For example, recently in Gulu, the issue of the current Malaria outbreak was raised and actions were discussed. In response, IPs coordinated on communication strategy and dissemination of info about malaria outbreak in cooperation with the district. Through radio broadcasts and posters, malaria cases dropped significantly with no deaths recorded for 2 months despite outbreak. 15. More responsive and targeted investments (1): Here, investments must be lasting, such as infrastructure, and strategically targeted. For example, in Kapchowra, KACSOA’s assessment presented in the DMC prompted the District Office to construct a medical staff facility in Tumboboi, Kaptanya in March 2015 as well as began the construction of a maternity ward this year. The buckets were further categorized according to four higher level concepts. We had originally attempted to define these in terms of the CACI framework offered by the Scope of Work. However, we found no instances of true integration and the alignment concept is better understood as a phenomenon that sits at the strategic level and not the outcome level. Rather, we found that the 15 buckets were better represented by the higher order concepts of coordination, collaboration, formal action and hard outcomes. The overall distribution of outcomes, according to type, bucket, and master concept, is as follows: 37 Table 4: Outcome Distribution by Bucket and Concep From this data, the evaluation team is able to draw five conclusions that will be discussed below: 1. The evidence supports the revised DOP theory of change; 2. There is evidence that IPs and Districts are driving their own changes, but less evidence that they are instituting changes driven by the other; 3. The nature and frequency of potential outcomes suggests that DOP effectiveness would be limited without wider SDS support. 4. We found no examples of the DMC fostering IP collaboration across DOs, and thus also found no evidence to support a main assumption of the central hypothesis. Put differently, inter-sector co￾location and coordination does not organically lead to inter-sector integration through the DMC. 5. When comparing DOP to non-DOP outcomes, we do not find any significant differences in the average number of outcomes per district, i.e. implementing the DMC does not yield more outcomes than if IPs were left to establish their own bi-lateral relations. Each of these conclusions is discussed in more detail below. The Evidence Supports the Revised Theory of Change When we examine the distribution of master-buckets for potential and concrete outcomes, we find a general pattern that falls in line with the revised theory of change: Chart 3: Distribution of Outcomes by Master Concept Note that coordination outcomes are significantly lower than collaboration outcomes, which makes sense as most of the coordination work is performed at the DMC meetings and collaboration outcomes occur outside. Then, as the revised ToC suggests, formal action and hard outcomes noticeably decline, as they interact with greater levels of complexity and tracing the DMC contribution becomes more difficult. Concept Outcome Bucket Total Concrete: DOP Contributed Potential: May have occurred without DOP Non DOP Outcomes in DOP Districts Comparison District Outcomes Coordination 1=Outside DMC Reporting 26 9 5 6 6 2=Duplication Avoided 24 8 11 1 4 3=District led coordination 11 4 2 0 5 4=Collaboration between IPs (not across DOs) 10 7 2 0 1 5=Monitoring (Joint Field Visits) 14 4 3 3 4 6=IPs Change Implementation Based Upon Need 24 7 13 2 2 7=District and IPs collaborate to conduct field activities or trainings 18 5 7 2 4 8=Cross sectoral collaboration (not IP to IP) 7 5 2 0 0 9=District directs IPs where to implement 8 0 4 4 0 10=District arbitrates stakeholder conflict 8 4 3 0 1 11=District changes their process 29 12 12 2 3 12=Policy (New or Changed) 11 2 4 1 4 13=District offers new/expands existing series 7 4 2 0 1 14=Availability/quality of services 9 3 4 1 1 15=More responsive/targeted investments 5 1 0 2 2 Total 211 75 74 24 38 Collaboration Formal Action Hard Outcome 38 Another interpretation raised by this downward trend is that the respondents – primarily district officials and higher level IP staff – themselves lack the situational awareness to fully grasp how their efforts play out on the ground. It could be that many more collaboration outcomes subsequently have been realized as hard outcomes, but tracing these progressions would require much deeper monitoring systems than what is currently in place in most districts and activities. Breaking Down the Evidence by Bucket: IP and District Drivers of Change Broken down even further, we find that the distribution of outcomes across the 15 buckets is unevenly spread: Chart 4: Distribution of Buckets for Total and Concrete Outcomes The evidence collected by the evaluation team shows that participation in the DOP process oftentimes leads to outcomes in which IPs make changes themselves, such as outcomes where duplication is avoided and IPs adjust their implementation based upon district needs (not driven by the district). Districts also seem to be learning from their interactions at the DMC, and we find a high number of outcomes in which districts make some kind of change to their current process or introduce new processes. By contrast, we see fewer outcomes in which the district is in the driver’s seat of IP implementation, although districts do sometimes arbitrate IP conflicts. As stated before, we find very few hard outcomes, especially more targeted investments. The finding that both IPs and Districts have the ability to change their own activities and processes but less of an ability to change the other’s activities should be intuitive. Given that the DMC has no actual authority vested within it, and that Districts do not have a formal mandate to change what an IP is doing that supersedes A/CoR authority, meaningful changes must then occur through collaborations and persuasions. Although not to be discredited, implementation changes through persuasion are almost always fewer than those driven by authoritative and punitive action. Bringing in the DMC/meeting cost data discussed in the previous chapter, the evaluation team costed out the distribution of various master concepts, both in terms of total outcomes and concrete ones. The results are presented below: 39 Table 4: Unit Cost per Outcome, Cumulative The table is meant to be illustrative in the sense that it presents the average unit cost of each type of outcome. The unit costs were derived by dividing the number of outcomes for each bucket into the total number of DMC meetings (175) to yield the average number of DMC meetings it takes to achieve that type of outcome. This number was then multiplied by the average cost per meeting (we used the moderate estimate of scenario 2 - see above) to give the average cost for each type of outcome. The table divides avg. cost per outcome into two types: overall outcomes (which include both potential and concrete) and only concrete. Note, because there was only one “more responsive/targeted investment” outcome, it assumes the entire cost of all 175 DMC meetings. The Nature and Frequency of Potential Outcomes Suggests SDS Support is Crucial We find there are almost as many potential outcomes as there are concrete outcomes, suggesting that other coordination processes and IP activities within the district are able to generate as many outcomes as the DOP alone. This observation especially applies to activities associated with SDS, such as additional sector specific coordination meetings, better placement of health workers at facilities, etc. This finding then reinforces the proposition suggested in the efficiency chapter that the DOP process would not be as effective without the wider support provided by the SDS activity. No Evidence of the DMC fostering IP collaboration across DOs Prior to the field visits, the evaluation team was instructed to remain especially attentive to any outcomes associated with IPs from different development objectives (i.e. economic growth, democracy & governance, and public health) that could be connected to the DMC meetings. Recall that original Central Hypothesis assumed that some level of inter-DO collaboration and integration was originally expected to emerge without further action taken by USAID to do so, and that the revised DOP hypothesis assumed that this type of cross￾DO collaboration would be driven by the district. In fact, the evaluation team found no evidence that any of these types of outcomes had occurred.48 Thus, we must conclude that, in its current form, the evaluation team found no evidence to support the Central Hypothesis. It may be that the evaluation team missed a few inter-DO collaboration outcomes because a select handful of IPs or District representatives were not present during our interviews. Even if this were the case, the numbers would be very few, and thus we are relatively confident that our lack of support for the Central Hypothesis does not suffer from a Type II error.49 48 A few potential exceptions included the OVC referral system, which sought to refer orphans to activities that improved household nutrition. However, these were not counted, as they were already included in the design of many IP activities and would have occurred regardless of the DOP’s existence. 49 A Type II error is a term used in statistical hypothesis testing that suggests that the evidence present fails to detect an effect that is present, even though the sample collected is reasonably representative. Concept Bucket # of Outcomes/Bucket (potential and concrete) Avg. # of Meetings to Achieve Outcome* Avg. Cost Per Outcome** # of Concrete Outcomes/Buc ket Avg. # of Meetings to Achieve Concrete Outcome* Avg. Cost Per Concrete Outcome** Coordination Outside DMC Reporting 14 12.5 $ 81,850.00 9 19.4 $ 127,322.22 Duplication Avoided 19 9.2 $ 60,310.53 8 21.9 $ 143,237.50 District led Collaboration 6 29.2 $ 190,983.33 4 43.8 $ 286,475.00 Collaboration between IPs (not across DOs) 9 19.4 $ 127,322.22 7 25.0 $ 163,700.00 Monitoring (Joint Field Visits) 7 25.0 $ 163,700.00 4 43.8 $ 286,475.00 IPs Change Implementation Based Upon Need 20 8.8 $ 57,295.00 7 25.0 $ 163,700.00 District and IPs collaborate on field activities/ trainings 12 14.6 $ 95,491.67 5 35.0 $ 229,180.00 Cross sectoral collaboration (not IP to IP) 7 25.0 $ 163,700.00 5 35.0 $ 229,180.00 District directs IPs where to implement 4 43.8 $ 286,475.00 0 - District arbitrates stakeholder conflict 7 25.0 $ 163,700.00 4 43.8 $ 286,475.00 District changes their process 24 7.3 $ 47,745.83 12 14.6 $ 95,491.67 Policy (New or Changed) 6 29.2 $ 190,983.33 2 87.5 $ 572,950.00 District offers new/expands existing series 6 29.2 $ 190,983.33 4 43.8 $ 286,475.00 Availability/quality of services 7 25.0 $ 163,700.00 3 58.3 $ 381,966.67 More responsive/targeted investments 1 175.0 1,145,900.00 $ 1 175.0 1,145,900.00 $ Collaboration Formal Action Hard Outcome *Calculated by dividing the number of outcomes in each bucket by the total number of meetings (175) **Based upon Scenario 2 estimates of $6,548 per DMC meeting Unit Cost per Outcome (based upon avg. # of meetings for outcome to occur) 40 Comparing Outcomes in DOP and Non-DOP Districts: No Difference Recall that the evaluation team visited non-DOP districts to build evidence for what might have happened had the DOP process not occurred. Put differently, do the “organic” (i.e. non-DOP) relationships between districts and IPs lead to a fewer number of outcomes than in DOP districts? At first glance, we find that DOP outcomes, both potential and concrete, appear to occur more often than non￾DOP outcomes. However, this would be misleading for two reasons. First, the non-DOP outcomes in comparison districts fell across a much smaller number of districts. Second, because these were not mission focus districts, the number of IPs was also noticeably smaller, which could potentially reduce the number of IP related outcomes. When we divide these numbers by the total number of applicable districts, the averages are much different: Table 5: Outcomes across Districts Over Three Years Outcomes Across Districts Over Three Years Total DOP outcomes (Concrete & Potential) Of which are Concrete Comparison District Outcomes Number of Outcomes 149 75 38 Number of Districts 19 19 5 Avg. Outcomes/District 7.8 3.9 7.6 The comparison with non-DOP districts is does not bode wellfor proponents of the DOP process who link success only to the types of outcomes we harvested, as we fail to find any significant differences in outcomes between the two. Even worse, when we focus solely on concrete outcomes (those that would not have occurred without the DOP), we find an even lower average of outcomes per district, potentially suggesting that the DOP process is less effective and less conducive to generating outcomes than leaving IPs to “organically” coordinate with district officials on their own. Further analysis of the various types of outcomes fails to significantly adjust the overall conclusion. Although there is a much higher concentration of Collaboration and Formal Action outcomes in DOP districts than in non-DOP ones, we also find a slightly higher concentration of Hard outcomes in non-DOP districts (.42 in DOP vs. .6 in non-DOP). We do see, however, noticeable differences in the number of coordination outcomes; primarily because coordination within the DMC meeting is not counted as an outcome in DOP districts, while coordination between the district and IPs in non-DOP districts is. This naturally biases non￾DOP districts to have higher coordination outcomes, as well as increases their average outcomes per district score. Yet when we take out coordination outcomes from both sets of districts, the average outcome per district slightly changes more in favor of the DOP districts, i.e. 7.1 outcomes per DOP district vs. 6.4 outcomes per non-DOP district. Even under these more favorable adjustments, DOP districts still only yield around 10% more favorable outcomes (both potential and concrete) than non-DOP districts, many of which probably would have occurred anyway. It must be noted that the lower number of IPs in the comparison districts may in fact be more conducive to more outcomes, as it is easier to establish and maintain bi-lateral relations that lead to outcomes. Also, because we had no means for identifying how similar DOP and non-DOP districts were prior to the DMC meetings, we also cannot say for certain if the differences can be attributed to the DOP process. These qualifications aside, the comparison between DOP and Non-DOP outcomes are still useful, and the evaluation team remains confident in concluding that the differences between the two are noticeably less than a DOP proponent would expect. Incidentally, answering the question in the evaluation scope of work: “What explains variations of these across the DOP and non DOP districts?” is no longer necessary as we found vary little variation to explain save what is offered above. In lieu of this question, we offer to explain outcome variation within DOP districts. Explaining the Variation in Outcomes The DMC Leadership Thesis 41 Throughout the evaluation, the team encountered a wide-spread belief that leadership provided by the district CAO (or assistant CAO in some cases) was the key variable that explained why a DMC performed well, which, in turn, would explain why some DMCs generated more outcomes than others. The evaluation team tested this thesis by holding collaborative consultations with the USAID DOP Coordinator, SDS district cluster leads (the main focal points for the preparation of the DMCs) and the SDS DOP manager to create a “perceived level of DMC performance.” Through two days of workshops, team members agreed upon the scale to be used, ranked the DMCs accordingly, discussed the rankings and identified key reasons for them. These perceptions were cross-checked and reconfirmed with various other stakeholders, including USAID representative focus group members and IP Chiefs of Party.50 To then test the relationship between perceived success and number of outcomes, we ran Pearson’s coefficients for both total (concrete + potential) and just concrete outcomes. The results are as follows: Table 6: Correlations between Perceived Success and Outcomes What this suggests is that we find a fairly strong, but not perfect, positive relationship between the perception of a successful DMC and the number of outcomes (both potential and concrete). When we focus solely on concrete outcomes, the correlation weakens to a more moderate relationship. Loosely interpreted, these findings suggest that DMCs that appear to be successful will yield higher levels of total and concrete outcomes around 75% and 50% of the time, respectively. Yet given that these are not perfect correlations, nor are they explanations, the results do suggest that other factors, or, more precisely, a combination of other factors, may also be at play. Explaining the Variation: QCA To better explain the variation across districts, the evaluation team conducted a Qualitative Comparative Analysis (QCA) of contributing factors that may be associated with DMC-connected outcomes. QCA is especially suited for this task, as it emphasizes how various combinations of factors come together to explain the outcome across a variety of settings.51 In doing so, QCA uses a Boolean Algebra-based algorithm to identify which combinations are necessary for success, which ones may not be necessary but are still sufficient for success, and which ones are superfluous relative to the other more essential factors. The evaluation team used QCA to model the DOP experience according to a “truth table” that identifies which sets of activities each district received, external factors that are relevant to the development results DOP hopes to achieve, and the outcomes we found associated with attendance at the DOP. As with any model, the factors and variables included are greatly limited by the types of data that is available. QCA also has three important data requirements for the model to operate: 1. the variables used must have data points for all 19 districts and 2. the variables must be structured according to binary scale of membership/non-membership (coded as 1 and 0, respectively), which in turn requires that we “anchor” each variable according to a threshold that defines where the cut off for membership lies.52 For this evaluation, we were able to collect a variety of district capacity variables connected to the wider SDS intervention, as well as generate our own variables based upon the attendance and action item analysis we collected. These variables, and their corresponding thresholds, are listed below: 50 The rankings fell across a spread of 0=Perceived Failure, .333=Perceived Partial Failure, .666=Perceived Partial Success, and 1=Success. The scores and justifications by district are found in Annex 4. 51 The main premise of QCA is that of multiple conjunctural causation, which means that (1) most often not one factor but a combination of activities lead to development results; (2) different combinations of activities can produce the same outcome; (3) one activity can have different impacts on the outcome, depending on its combination with other factors and the context; and (4) the absence of an activity may be just as important as its presence. 52 More nuanced versions of QCA, such as “fuzzy set” QCA (fsQCA), allow for more fine-grained assignment and partial membership scores, but even these still require that we still draw a threshold of membership. Correlation of Perception to Outcome Total Concrete Pearson's 0.762 0.514 Significance 0.001 0.024 42 Dependent Variable • Success: The main dependent variable to be explained, we defined “success” as a district which generated at least 7 outcomes, which is the same number of outcomes generated in non-DOP districts. Conversely, districts with fewer than 7 outcomes, i.e. less than those in non-DOP districts, were not scored as successful. According to this definition, there were 12 successful DOP districts. Independent Variables • Overall District Action Item Performance: Action item performance may serve as a predictor of outcome performance based upon the assumption that meaningful items raised in the DMC meeting should manifest as outcomes outside of the DMC meetings if the DOP process is working. Based upon the data compiled in the action item database, we ranked a district as “performing” if there was at least one substantive action item generated for every two procedural items, i.e. at least 33% of action items were substantive. Conversely, districts with higher levels of procedural action items (above 66%) were scored as non-performing. While this may appear to be a low threshold for acceptable levels of substantive action item performance, it may be that many substantive action items cannot occur until various procedural items are first addressed. According to this definition, there were 13 action item performing districts. • Average Quarterly IP Attendance: IP attendance is another logical predictor of outcome performance based upon the assumption that for the DOP process to be effective, IPs should regularly attend and participate in the DMC meetings. Based upon the data compiled in the attendance database, we ranked a district as “performing” if at least 50% of IPs who were supposed to regularly attend DMC meetings actually attended over time. According to this definition, there were only 9 districts in which at least 50% of IPs regularly attended. • Average Quarterly USAID Representative Attendance: USIAD representative attendance is another logical predictor of outcome performance based upon the assumption that for the DOP process to be effective, USAID reps should regularly attend and participate in the DMC meetings. Based upon the data compiled in the USAID rep attendance database, we ranked a district as “performing” if USAID reps were present at DMC meetings at least 33% of the time (no district would’ve met the 50% attendance threshold as in the case of IPs). According to this definition, there were only 10 districts in which a USAID representative regularly attended at least 33% of DMC meetings. • Overall Perceived Success: As already discussed above, we found a fairly strong correlation between perceived levels of success and outcome performance. We took this variable forward into the QCA model, and designated districts as either successful or unsuccessful according to DMC meeting performance, as per the observations made by IPs, USAID Reps and SDS staff. According to this definition, there were 10 successful districts. The following indicators are derived from SDS Grant A Quarterly Performance Review Sheets • Average Quarterly Rate of SDS Grant A Disbursement: Throughout the life of the DOP process, SDS has also administered grants to each district. These grants, falling under SDS category “A”, were cash resources provided to the district to support district technical assistance, cover communications costs, pay “safari” and potentially other allowances, provide reimbursements for the delivery of drugs and visit communities. The rate of disbursement of these grants serves as a proxy for district capacity to administer funds to act upon district priorities. SDS set the threshold for average Grant A quarterly rates of disbursement at least 80% for acceptable performance, and we followed suit. Sixteen districts met these standards. • Average Quarterly Questionable Costs of Grant A Disbursements: Questionable costs are the percentage of SDS Grant A expenditures that cannot be accounted for according to accepted accounting practices. While the previous indicator is a proxy for how much money a district can 43 deliver, this indicator is a proxy for how well they are doing it. SDS set the threshold for questionable costs at no more than 20% of quarterly disbursements, and we followed suit. Thirteen districts met these standards. • Average Quarterly Rate of Central Government Disbursement: In addition to tracking Grant A disbursement rates, SDS also tracked overall central government disbursement rates. These grants, falling under GoU category “conditional” were cash resources provided to the district to support service delivery. The rate of disbursement of these grants serves as a proxy for district capacity to administer funds to provide services to its population. SDS set the threshold for average quarterly rates of disbursement at least 80% for acceptable performance, and we followed suit. Ten districts met these standards. • Average Quarterly Sector Meeting Performance: In addition to the DOP and eDTPC, SDS also supported various quarterly sector meetings, including district health management team and district orphans and vulnerable children meetings. As part of their grant performance metric, SDS captured various performance aspects of each sector meeting, such as evidence of technical assistance, % of action items resolved, and meeting functionality. The evaluation team combined the performance scores on each sector meeting to obtain a master sector performance score for each DOP district. SDS set the threshold for overall sector performance to at least 80% for acceptable performance, and we followed suit. Six districts met these standards. • Average IP quarterly report/ work plan shared in the eDTPC/ DOP meetings: This indicator gauges the level of IP sharing of quarterly and workplan reports with the eDTPC and DOP meetings. It was collected quarterly by SDS. The sharing of information was a major assumption of the DOP theory of change, and was key to the coordination function it played. Following SDS, we set the threshold of acceptable information sharing at 80% of IPs sharing their workplans/reports every quarter. Performance on this indicator was very low, and only four districts met these requirements. Before proceeding, a two caveats on the nature of the must be made. First, the data derived from the SDS grant program was extremely unique. Not only do most grant giving programs for building district capacity lack such detailed information, but SDS also imposed unusually high compliance standards on its grantees that are not typically found in similar programs. This level of compliance should be seen as a unique capacity building exercise in and of itself. Second, although all NUFO districts had much less experience with SDS’ approach to capacity building and none attained success, we included them in the model for purposes of diversity. The QCA algorithm works according to logical comparisons of both pathways to success and lack thereof, and thus including NUFO districts provided valuable counter factual evidence to sharpen the pathway to success discussed below. Results: Success=Substantive Action Items AND Adequate Grant A Disbursement Rates After running several iterations of the QCA model, the evaluation team was able to drop several variables because 1. QCA eliminated them through its minimization process, or 2. retaining them did not improve the model’s explanatory power. What remained was a singular pathway to success: Pathway 1: Substantive Action Items AND Adequate Grant A Disbursement Rates This can read as: District success, defined in terms of achieving at least seven DOP-related outcomes, was achieved when districts generated substantive action items at least one third of time AND disbursed at least an average 80% of SDS Grant A funds every quarter. Notably, all other variables – Perceived Levels of Success, USAID and IP Attendance Rates, Average Questionable Cost Rates, Central Government Grant Disbursement Rates and Quarterly Sector Meeting Performance Rates – dropped out of the model. We ran two versions of the QCA model: crisp set, which codes the variables into dichotomous categories of full membership or full non-membership, and fuzzy set, which allows for more nuanced analysis based upon partial membership in 44 each category. The results of both versions were more or less equivalent,53 and are presented in more detail in Annex 11. How to interpret these findings? First, the fact that many of the variables dropped out of the model does not mean that they do not have a positive effect on district performance nor does it mean that are not valid indicators of district strength. Rather, QCA has found evidence to suggest that, logically, they cannot be part of the causal explanation for higher number of outcomes across districts because often enough there is ‘success’ when these factors are not present. Second, it may be the case that other variables not introduced into the model may better explain the outcomes. Third and finally, QCA is only able to display relations between variables - whether or not these relations can be read as causal needs to be determined by theory and causal explanation.54 As such, the above listed solution set is better viewed as a causal suggestion than as a causal affirmation. Explaining the Findings Nevertheless, the combination of factors suggested above does make logical sense. First, the ability of DMC participants to come together and collaboratively agree upon substantive points of action can be interpreted as an indication of a shared commitment to move past coordination and into more meaningful forms of working together and problem solving. Yet, as the evaluation team heard time and time again, coordination in and of itself is insufficient for action, but instead also requires significant capacity and resources to move forward. Second, the ability of Districts to consistently disburse high levels of SDS grant assistance may in fact fulfill this second requirement. What is also interesting is that QCA found that equivalent levels of central government grant disbursement were not relevant to success, suggesting two additional interpretations: 1. GoU grant money is seldom used to support USAID partner interactions and progress; as districts would rather wait for USAID to provide this money directly; and 2. Success in the DOP Process was most likely contingent upon participation in the wider SDS activity (another point consistently articulated to the evaluation team by various stakeholders). Conclusion After a significant departure from the linear logframe/results frame approach to development outcomes, the evaluation team found that a revised framework of stochastic outcomes better conceptualized how the DOP process contributed to outcomes that occurred outside of, but in connection with, DMC meetings. Consequently, USAID/Uganda recognized that an alternative approach to capturing these outcomes was required, and guided the evaluation team to adopt the “outcome harvesting” approach. Although not without its own set of biases, O/H nevertheless enable the evaluation team to collect a wider array of outcomes than previously allowed for in the earlier DOP results framework or the revised ME&L plan. The evaluation team found 149 outcomes that could be in some way associated with the DOP process. From this, we were able to make a number of conclusions regarding DOP effectiveness; the most unfavorable of which is that, in terms of measurable outcomes, the DOP process has not made any significant differences than if the DOP had not occurred. Nevertheless, one of the main findings of the QCA - that the ability of DMC participants to come together and collaboratively agree upon substantive points of action forms a key factor 53 For both models, QCA yields corresponding consistency and coverage scores. Coverage is the proportion of the total number of cases covered by the causal expression, i.e. a proportional measure of the extent to which the solution ‘explains’ the outcome. Consistency is the degree to which there is a relationship of necessity or sufficiency between a causal condition (or combination of conditions) and an outcome is met within a given data set. For the crisp set model, we find a consistency of 1 (which is always the case for crisp sets) and a coverage of .916, which means that the substantive action item condition, in combination with the grant A rate of disbursement condition, covers almost 92% of the successful districts in the model. The only exception is Amuru – which was a borderline success that did witness higher levels of grant A disbursement but lacked adequate levels of substantive action items. In the fuzzy set model, we did not find a model that met acceptable consistency and coverage score standards, save a reduced Grant A disbursement and substantive action item one in which Grant A disbursement drops out. Here, the consistency score drops to .852 (which still meets best practice standards but is not considered very strong), and coverage increases to .929. QCA also tests if any of the variables are necessary conditions, i.e. they must always be present for success to occur. In the case of the crisp set, substantive action items are close to being a necessary condition, with a score of .917, which can be seen as almost always necessary. In the fuzzy set, both conditions pass the test for necessity at .92 for substantive action items and .953 for disbursement rates. In combination, they yield a perfect necessity score of 1. 54 The evaluation team was unable to conduct a rigorous experimental design impact evaluation (e.g. RCT) to establish that the outcome can be definitively attributed to the presence of these variables. 45 contributing to success – does suggest that there is no one size-fits-all to producing successful outcomes and that DOP-sponsored pathways are also valid. Even so, we must reiterate that success in the DOP Process was most likely contingent upon participation in the wider SDS activity. 46 Should DOP Continue? In lieu of a traditional list of follow-on recommendations, the evaluation team was advised at the USAID/Uganda Preliminary Findings Presentation to instead to conclude the evaluation with a discussion on whether or not the DOP should continue and, if so, which aspects should be brought forward and which should be dropped.55 These recommendations are made in the context of the various paths facing the future CDCS. Yes…. We suggest that, despite the lack of solid evidence that it is the most effective modality for contributing to cross-sectoral development outcomes, the DOP process should continue. When assessing the efficiency of DOP relationships, it is also necessary to recognize that the USAID representative does simultaneously act as both diplomat and development manager.56 To the extent the Mission embraces the notion that communicating its development expertise and objectives serves diplomatic functions, we must then explicitly recognize that the USAID to District Official relationship, in and of itself, is also a meaningful end and does not always need to be justified in terms of its direct contribution to development outcomes. Put differently, building and sustaining relationships matter. The following graphic illustrates how USAID dialogue via the DOP helps reinforce both diplomatic and developmental functions: Figure 5: Convergence of Investments in Diplomacy and Development Given that the district is the center of both local power and the realization of the DDP, USAID’s relationship with district officials then contains both political and the developmental elements. As such, the DOP process remains the key for USAID/Uganda to monitor this relationship. The question then arises “how do district officials benefit from this relationship?” The efficiency section of this evaluation has found that, undeniably, district officials now have an improved understanding of USAID’s contribution to their population’s social improvement. Officials also felt a sense of satisfaction of compliance 55 This comment was made by the Mission Director. The presentation was made on December 16, 2015. 56 Although implicit in the day to day interactions of USAID officials and explicit in higher level forums such as the Quadrennial Diplomacy and Development Review, most USAID strategy documents do not explicitly discuss USAID staff member’s diplomatic role. For example, the previous 2011 USAID/Uganda CDCS mostly referenced diplomatic efforts in the context of much wider USG risk mitigation efforts against massive systemic shocks (e.g. cross border conflict, growing instability, etc.). An explicit discussion of how USAID’s communication mission (based upon the Foreign Assistance Act of 1961) is to inform host country audiences about U.S. development assistance is lacking. Contrast this with the 2007 GAO report, "U.S. Public Diplomacy: Actions Needed to Improve Strategic Use and Coordination", which found that "... USAID use program-specific research to design, implement, and evaluate the impact of thematic communication efforts created to influence the attitudes and behaviors of target audiences,” and that “that USAID use actionable research to support a campaign-style approach to communications.” Irrespective of a formal discussion in the CDCS, USAID representatives do perform a diplomatic function through their interactions with other stakeholders. 47 with the law and with their directives when IPs attended these meetings, and expressed regular dissatisfaction when they weren’t. Finally, in some districts, district officials have started to see the value of incorporating IP workplans and budgets into their annual budget cycles, even if this hasn’t significantly influenced their own budget requests from the central authorities. …But not in its current form Recognizing that the DOP process probably will not survive without continued SDS or similar types of district capacity building support, substantial changes still need to be made. These changes depend upon the future strategic direction the Mission decides to take, as currently considered through the process of creating a second CDCS. Based upon the evaluation teams own understanding of potential developments, we propose three potential scenarios of change should the DOP to continue: Dopping Out: Expand the Web of DOP Stakeholders Scenario 1 assumes that the new CDCS will assign high importance to the cultivation of Ugandan electoral leadership at the district level. If so, one of the most straight forward changes that the Mission may wish to consider is to revise the DOP MoU to formally embrace political decision makers in the process. Doing so will potentially increase the efficiency of the district decision making process and reduce the number of procedural steps before decisions can be taken by the district chairperson. We anticipate that more substantive decisions will also be taken as a result, which will potentially lead to a higher number of development outcomes. However, we must stress the potential pitfalls associated with this course of action. First, a more formal incorporation of political actors into the DOP process will invariably introduce more narrow political interests. DOP facilitators may quickly find themselves enmeshed in political controversies as elected officials could potentially seek to redirect IP resources to reward their constituencies at the expense of other beneficiaries. Second, the question of various council member supports, e.g. sitting allowances, per diems, etc., will invariably arise and require a disciplined, consistent policy on the part of USAID as well as potentially in continued support with other donors. Third and finally, USAID/Uganda may wish to consider exploring how changes to the procedural process – both in terms of the current eDTPC hybrid as well as a more expanded political coordination forum – could be formalized at the national level, potentially in partnership with MoLG. Without such action, DOP process gains are most likely unsustainable. Again, these efforts must work in tandem with wider capacity building activities, which, under this scenario, should also include more direct support to political leadership. Dopping Up: Consolidating USAID Representation Scenario 2 assumes that under the new CDCS, the Mission will seek to further strengthen relations with the GoU through the creation of a USAID “relationship manager.” In doing so, these individuals should replace the current volunteer-based model for DOP representation. Although volunteers tend to exhibit higher levels of commitment in the short term; their continued availability cannot be counted on in the medium to longer term. Instead, full-time relationship managers with specific job descriptions should be hired to fulfill the three main responsibilities currently identified by DOP rep volunteers: Enforcement, Policy Clarification and Arbitration/Follow-up (see the Efficiency section for a more indepth discussion). Training modules on each of these functions should be developed accordingly. Given the constraints associated with both hiring and finding office space for new USAID personal, the Mission may wish to consider procuring USAID relationship managers through a third party recruitment mechanism (as well as continue to draw upon the USAID Development Advisors Fellow program). Yet even if these individuals are not full-fledged USAID employees, the Mission should make it clear to IPs and District officials that these relationship managers act on USAID’s behalf and thus should be given the deference and respect afforded to full-time USAID employees. As such, they need to be viewed differently than as contract personal. Also, to deal with the lack of desk space within the Mission, these relationship managers should spend the majority of their time based in the field and be tasked to keep a much closer watch on USAID investments than what current staff experience. However, the Mission should also remain cognizant of the need to organize 48 wider focus group discussions and informal information sharing forums within Kampala in which these relationship managers regularly return to the Mission to share experience and improve Mission situational awareness. Dopping Down: Involving Sub-District Officials and the Community The third and final scenario assumes that the Mission will seek to establish a new Tier-3 “area of intensity” within a specific regional focus to pilot a highly integrated effort of cross objective activities focused at community or even household level. Leaving aside the substantial improvements to the Mission’s GIS and beneficiary tracking capabilities, the number of stakeholders within the DOP would expand dramatically to at least include County and Parish level representatives in a much more participatory process. The question of to what extent USAID would seek to incorporate subdistrict political leadership would also need to be addressed, thus opening the door to some of the concerns raised above under scenario 1. Evaluating the efficacy of this scenario is beyond the scope of this evaluation, but the evaluation team does strongly recommend that USAID/Uganda first explore the bureaucratic costs of a much deeper DOP and decision making process. In line with the previous discussion on the way in which decision making is incentivized (see the Relevance section), we strong urge the Mission to consider first conducting a “costs of decision-making” study to identify the variety of allowances that must be paid to in order to expand the process. By posing the question “how much would it cost to administer a given district development plan down to the community level,” the study would most likely reveal the portion of bureaucratic “capture” through these allowances, as well as where potential opportunities to consolidate the decision making process can be found. Cross-cutting Changes Regardless of scenario, the evaluation team would like to conclude with a seemingly cosmetic yet important recommended change to the DOP, should it continue. Given the lack of evidence that the DOP facilitates cross-sectoral outcomes, the Mission may wish to consider reducing the number of DOP meetings to twice a year. One of these meetings could further align to the annual budget and planning conference, while at the same time, IPs should still be required to attend quarterly sector meetings specific to their activity. This recommendation is based upon the need to balance IP concerns and opportunity costs with USAID’s own relationship building imperative and GoU compliance requirements. Finally, our experience with O/H, while positive, suggests that it would be a fruitful approach that is better suited for inhouse staff to administer than consultants. The costs associated with training and sustaining mid￾level consultants to meet the method’s intense labor requirements may be prohibitive. Conversely, USAID/Uganda, either inhouse or potentially through such mechanisms as the Learning Contract, would be able to regularly deploy the method as an effective means of monitoring stochastic outcomes more effectively than through the traditional results framework approach. As such, the evaluation team recommends USAID/Uganda consider adopting O/H into its toolkit of M&E capabilities. i Annex 1: Scope of Work STATEMENT OF WORK FOR AN INDEPENDENT EVALUATION OF USAID UGANDA’S DISTRICT OPERATIONAL PLAN APPROACH (With Modified Questions Included) I. Overview USAID intends to conduct an independent evaluation of USAID Uganda’s District Operational Plan (DOP) approach and its contribution to USAID/Uganda and Uganda Districts’ development results. USAID Uganda, implementing and local government partners will use key lessons learned and recommendations from this evaluation to develop better program implementation strategies and build stronger partnerships for greater development results. The evaluation will be conducted in February through March 2015 to feed into the incoming Country Development and Cooperation Strategy (CDCS) process. II. Background USAID/Uganda’s 2011 CDCS implements key elements of the Presidential Policy Directive on U.S. Global Development (2010) i.e. greater selectivity, country ownership, and rigor to evaluate impact to improve development results. USAID/Uganda implemented the selectivity principle in its strategy by using a geographic focusing approach. The underlying development hypothesis to this approach as defined in the CDCS states that development results would be greater in areas where all three development objectives i.e. Health, Economic Growth, and Democracy and Governance, are present. The geo-focusing policy provides a framework for concentrating USAID activities in 19 mission focus districts (MFDs). An attendant common indicator policy provides the framework for testing this development hypothesis. The Mission introduced the DOP concept in 2011 to mitigate coordination and operational issues that could arise from having increased USAID presence in the MFDs. The DOP is a formal agreement between USAID, district government, and implementing partners (IPs) that provides a framework for planning and coordinating USAID assistance with district-level governments to achieve shared development objectives through a more effective and efficient approach. Under the DOP, each district establishes a District Management Committee chaired by the Chief Administrative Officer of the district, and relevant district technical offices meet quarterly with USAID and its implementing partners in that district to discuss and resolve coordination, alignment, and operational issues. The agreement also lays out a set of indicators, measured at the district level, to track improvements in the sectors the district and USAID are working in. These indicators are tracked in both the MFDs and a set on non DOP districts i.e. mission comparison districts (MCDs). The DOP approach aligns well with the Paris declaration. Likewise, at the district level the local governments are expected in lead the process of development planning and coordination of all development assistance in their jurisdiction towards common development priorities. USAID DOP approach and other local government strengthening activities are geared at strengthening the LGs leadership role. USAID facilitates and supports the DOP process via two different models: • Model 1 - the USAID Strengthening Decentralization for Sustainability (SDS) project, coordinates and supports districts to organize DMC meetings and follow up on issues raised as part of its core local government strengthening activities. SDS supports 13 of the 19 MFDs57; • Model 2 – USAID’s Northern Uganda Field Office (NUFO), based in Gulu, provides support and oversees the DOP process in six districts in Northern Uganda58. 57 SDS supports similar activities in an additional 22 districts with a focus on coordination of health partners. 58 Though the USAID NUFO was formally closed in December 2014, different modalities remain in place to provide DOP assistance. ii Subsequent mission reflections on the development hypotheses and initial implementation of the DOP revealed that the development hypothesis was somewhat vague thereby becoming necessary to become much more specific about the causal pathways regarding integration, program intensity, and the DOP approach and how each relates to improved development results. In 2013-14, USAID in consultation with her implementing partners and local governments, re-articulated the theory of change, causal pathways that can better demonstrate how DOPs can lead to greater development results. We have re-formulated both intermediate level and impact level indicators that would demonstrate results and success. Over the past years, the Mission has conducted various activities to monitor progress, results and identify emerging lessons. Some data has been collected on the different indicators by the district, process monitoring information by SDS, other data by other implementing partners and the Mission. Various learning events have been conducted at the regional and national level to share emerging good practices, lessons and determine improvement priorities. As USAID comes to the end of 2011-15 CDCS, the Mission desires to get an independent opinion on the extent of implementation of this approach and identify results achieved so far, opportunities and lessons learnt so far. III. Purpose and Use of the Evaluation The purpose of this evaluation is to seek an independent opinion on the effectiveness of the DOP approach and its potential for contributing to better development results. The evaluation will contribute to building a picture of the DOP process work effectiveness which doesn’t look only at the causal results chain but the ‘web’ of interactions between different actors, forces and trends and their effect on the results USAID and its partners are striving to achieve. The evaluation will highlight key drivers of change across different districts and stakeholders. USAID Uganda, implementing and local government partners will use key lessons learned and recommendations from this evaluation to develop better program implementation strategies and build stronger partnerships for greater development results. As USAID Uganda comes to end of its first CDCS, this evaluation will provide feedback on the overall effectiveness of the geo-focusing approach, the selectivity principle and its implications for more efficient and effective programming going forward. IV. Evaluation Questions 1. To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo￾focusing approach? The evaluation will also investigate the factors driving variations in implementation. 2. How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? 3. How have behaviors and relationships within and between USAID, implementing partners, and districts changed as a result of the DOP process? 4. To what extent is the DOP process supporting and or hindering incremental improvements in coordination, collaboration, integration and alignment of USAID programs? What factors explain the differences in results among different DOP districts, between MFDs and MCDs, implementing partners, and development objectives? 5. What evidence exists to suggest that long term outcomes and higher-level development results in USAID supported sectors are better in the districts where DOPs are being implemented? What potential is there to achieve such results? What other factors exist that could better support the achievement of these results? iii MODIFICATION TO SOW Based upon the conclusion of the design phase, the evaluation questions have been modified. Below is the evaluation question matrix that captures the changes to the original evaluation questions. Note, the original ordering of the questions changed for purposes of style and logical flow. Proposed Revision of DOP Evaluation Questions Question 1 To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo￾focusing approach? The evaluation will also investigate the factors driving variations in implementation. The first sentence opens the door to a discussion implementation, what was done well and potential missed opportunities. The second sentence fits better below regarding the more analytical approach to variation Question 2: To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? Question 2 How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? No comment on this question. Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? Question 3 How have behaviors and relationships within and between USAID, implementing partners, and districts changed as a result of the DOP process? These two questions strongly overlap, as the behaviors and relationships among the various parties can be described in terms of coordination, collaboration, integration and alignment. The questions are ones of DOP effectiveness that go beyond DMC inputs and outputs. The last sentence about difference in results suggests comparative analyses of these factors, potentially with other districts. Question 4: Has the DOP process contributed to changes to coordination, alignment, collaboration and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? Are these relationships different in districts where the DOP has not been implemented? What factors drove variations in its implementation? Question 4 To what extent is the DOP process supporting and or hindering incremental improvements in coordination, collaboration, integration and alignment of USAID programs? What factors explain the differences in results among different DOP districts, between MFDs and MCDs, implementing partners, and development objectives? Question 5 What evidence exists to suggest that long term outcomes and higher-level development results in USAID supported sectors are better in the districts where DOPs are being implemented? What potential is there to achieve such results? What other factors exist that could better support the achievement of these results? Phase 1 of this evaluation revealed that there is little quantitative evidence to directly answer this question and move beyond only a general level commentary. This question also again suggests a comparative analysis with non-DOP, USAID supported districts, which is addressed above. This discussion could also occur under Question 1. Dropped New Question In the exploratory phase, we've learned there is an appetite to discuss the costs/ effectiveness of the DOP process to understand if it was value for money. We suggest a new question accordingly. Question 3: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? The modification of these questions went into effect when the DOP Phase 1 Inception Report was approved. V. Evaluation Phases and Deliverables iv Design Phase: The firm/individuals will kick off this process by developing an appropriate evaluation design and protocol to get to the answers of the key evaluation questions. Given the complexity of the whole geo-focusing and DOP approach i.e. involvement of multiple actors, programs, differences in districts, implementing partners and their activities, interests and incentives for different stakeholders, what has been an evolving nature of the approach, un-certainties among others, the successful firm/individuals is expected to propose an appropriate evaluation design that appreciates these multiple elements and yet strives for rigor where appropriate. Based on initial documents review59 and stakeholder interaction, the firm/ individuals will submit for discussion an evaluation design for review and approval by USAID and its partners. A quasi￾experimental design was proposed as part of the initial measurement framework but subsequent review revealed significant weaknesses and limitations. Though the firm/ individuals are expected to consider the issues raised by the review in the development of this design, feasible proposals on how comparisons can be made between mission focus districts and comparison districts are expected. While this SOW does not seek to be overly prescriptive in terms of what methodology is adequate for the task – the firm/individuals are encouraged to consider different methods like ‘Outcome Harvesting60, Rapid Outcome Mapping Approach61, qualitative comparative analysis, social network analysis methods amongst others. Integrating views and perspectives of different collaborators/stakeholders in this DOP process including voices from different units within USAID, critical USAID support implementing partners like SDS, Governance, Accountability, Participation and Performance (GAPP), Uganda Monitoring and Evaluation and Learning program (Learning Contract) and other USAID implementing partners, different stakeholders in the local governments, other development partners, Ministry of Finance, Planning and Economic Development and Ministry of Local Government, Office of the Prime Minister will be critical to building an understanding of the value of the DOP process and overall USAID/Uganda’s development work. Thereby use of other qualitative and quantitative methods as well as other kinds of innovations are anticipated. The evaluation design should also propose an appropriate sampling and data analysis strategy. In data analysis, care must be taken to ensure ability to compare results across DOP districts, between MFDs and MCDs, development objectives, and SDS and NUFO supported areas as possible. In examination of development results, differential impacts between men and women youth will also be pursued. The evaluation team shall describe the type of software for quantitative and qualitative data analysis they propose to use. Expected deliverables under the Design Phase: 1. In-country Briefing: Introduction of the evaluation team, discussion and clarification of the key elements of the SOW upon contract award. 2. An Inception report and implementation plan for the design phase detailing the Contractor’s interpretations of the assignment, approach to the development of the evaluation design and work schedule to be submitted within 5 working days of the in-country briefing. 3. Evaluation design and protocol: This will include a collaboratively revised and or re-articulated theory of change, evaluation questions including any revisions to the original questions, key evaluation design and methodologies, data collection tools, instruments and detailed survey protocol where appropriate, sampling strategy, detailed data analysis strategy and plan, outline and structure for the final evaluation report, proposals for other communication products and ways for presenting and sharing findings with 59 Annex 2 for relevant documents 60 Outcome Harvesting is a utilization-focused, highly participatory tool that enables evaluators, grant makers, and managers to identify, formulate, verify, and make sense of outcomes they have influenced when relationships of cause-effect are unknown. Unlike some evaluation methods, Outcome Harvesting does not measure progress towards predetermined outcomes or objectives, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change. Ricardo Wlison Grau and Heather Britt ‘Outcome Harvesting” Ford Foundation, May 2012. 61 John Young, Louise Shaxson, Harry Jones, Simon Hearn, Ajoy Datta, and Caroline Cassidy “RAPID Outcome Mapping Approach” Overseas Development Institute, June 2014 v different stakeholder groups, detailed evaluation work plan, proposed evaluation team. This evaluation design and protocol will submitted within 30 working days of the in-country briefing. Implementation Phase: This phase kicks off upon approval of the evaluation design and protocol. During this phase, the firm/individuals will conduct the evaluation to assess the effectiveness of the DOP approach evaluation. This will include the actual data collection, analysis, and preparation of the reports as per the approved evaluation design. At the start of this phase, the firm/individuals may present a revised implementation plan for discussion and approval by USAID. Expected deliverables under the Implementation Phase: 4. Revised implementation plan: The evaluation team will revise the previously submitted implementation plan to reflect any changes following approval of the evaluation design and protocol. The revised implementation plan will be submitted 5 days after approval is received. 5. Bi-weekly Progress Reports: Brief informal written reports summarizing progress, challenges and constraints and describing evaluation team’s response. The mode i.e. whether as an email, phone actual report and or phone call will be agreed upon at the in brief. 6. Oral Presentation: Power Point presentation (including handouts). The oral presentation should, at a minimum, cover the major findings, conclusions, recommendations, and key lessons. The evaluation team will liaise with the mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. The audience will include Mission management and staff, implementing partners and representatives of local government amongst others. The presentation shall be held within 30 days following the approval of the evaluation design. 7. Draft Evaluation Report: The report should comply with the USAID’s Evaluation Report standards. The report is expected within 10 working days after the oral presentation. 8. Final Draft Report: Complete report incorporating comments from USAID and other stakeholders submitted within 5 days of receipt of comments. Evaluation Close out Phase: This will be the final phase of the evaluation during which the final report will be submitted, other communication products and dissemination events held. During this phase, the firm/individuals will also participate in after action review of the process together with USAID – the modalities i.e. whether in person/virtually will be decided upon with USAID. Expected deliverables under the Close-Out Phase: 9. Final Report: The contractor will submit a final report incorporating final edits for wider dissemination within two days. The approved final report should be cleared by USAID before submission to the DEC. Draft and Final Evaluation Report should be provided in four (4) hard copies and one (1) electronic copy. 10. Evaluation Brief/Summary contains user-friendly summary of the evaluation findings, conclusions and recommendations that can be used for public dissemination. The evaluation firm/individuals will explore use of infographics and other modes of powerfully communicating results. This will be submitted alongside the final report. 11. Other products as determined in the final approved evaluation design. 12. A “Knowledge Repository” of all documents, studies, and data generated by the evaluation. This will become organized and accessible to interested parties. This will include cleaned data if quantitative survey is conducted, or data sets used for analysis, analytical models used, transcripts and recordings of interviews. VI. Key Personnel vi Given the diverse and complex nature of this evaluation, the Offeror should propose a multi-disciplinary team composed of both local and international experts. The evaluation team will have complementary expertise and experience in the following areas: evaluation design, management, and implementation, public administration, local governance and decentralization management, systems thinking, social research methods, and data analysis skills. The following positions are considered key personnel: International Evaluation Team Leader • Responsibilities: o The Team Leader will be responsible for coordinating the activities, assigning tasks to team members and supervising performance. S/he will be the main point of contact between the evaluation activity manager in QED, USAID and the evaluation team. o Developing, managing and, as needed, communicating updates to USAID on evaluation progress o Ensuring that the tasks within the evaluation work plan are those best suited (and most efficient) to achieving the objective of the evaluation. o The Team Leader will review all plans and outputs and be responsible for delivering quality products to USAID on a timely basis. o The Team Leader will be responsible for ensuring a draft report is submitted to USAID/Uganda prior to departure from the country, followed by a final report, which incorporates USAID/Uganda and key stakeholders’ feedback. • Education and Relevant experience: o Advanced degree in Social Research Methods, Social Sciences, Public Administration and Management or related fields. A PHD is desirable. o Ten years’ experience in conducting development evaluations of complex cross-sectoral development programs. o Must have played significant roles in evaluation of more than seven activities, five of them preferable in local governance and/or the administration sector. o Must have been a team leader for a minimum of five evaluations. o Must have demonstrated success in interacting and communicating effectively with a broad range of stakeholders, including international organizations, host country government officials, and civil society organizations. o Must have demonstrated experience producing high-quality reports for evaluations of complex, multi-sector programs. o Experience in application of systems thinking in past evaluations is desirable. Local Expert (Local Governance Specialist) • Responsibilities: o S/he will contribute to the design of relevant tools, collect data, prepare and participate in the writing the report. • Education and Relevant experience: o Master’s Degree in Public Administration and Management, Development Studies, Organizational Development or related fields. o Minimum of seven years’ experience implementing and evaluating local government development programs. o Must demonstrate knowledge and experience with the functioning of decentralized local governments in Uganda. vii VII. Management Roles and Responsibilities The Learning Contract / QED have overall responsibility for the coordination and management of this evaluation. This will include the day-to-day management of the evaluation team, logistical support, quality assurance and submission of relevant deliverables to USAID. QED will also organize relevant stakeholder in briefs and de-briefs and overall ensuring that the key requirements of the SOW are being met. Within USAID, the Program Office will be responsible for managing the evaluation. This will entail coordination of inputs of different offices and stakeholders coordinating necessary inputs, approval of evaluation products and deliverables in line with the requirements of the SOW. The DOP Monitoring, Evaluation and Learning group composed of USAID, SDS, and the Learning contract will be involved in the review and clearance of evaluation products. SDS, as a key player in this process take an active role in provision of data, support the preparations and other guidance to the evaluation Contractor while in the field in their districts of support. SDS will assist with providing relevant data, setting up appointments as requested by the evaluation team. MAP OF THE USAID UGANDA MISSION FOCUS AND COMPARISON DISTRICTS viii ix Annex 2: Design Phase Report Introduction In 2011, USAID/Uganda introduced the District Operational Plan (DOP) process to mitigate anticipated coordination and operational issues that might arise from having an increased concentration of program activities in 19 “Mission Focused Districts” (MFDs). The DOP is a formal agreement between USAID, district government, and implementing partners (IPs) that provides a framework for planning and coordinating USAID assistance with district-level governments to achieve shared development objectives (DOs) through a more effective and efficient approach. Under the DOP, each district establishes a District Management Committee (DMC), chaired by the Chief Administrative Officer (CAO) of the district, and relevant district technical offices meet quarterly with USAID and its implementing partners in that district to discuss and resolve coordination, alignment, and operational issues. As the DOP process draws to a close, USAID/Uganda has commissioned a final evaluation to determine the effectiveness of the DOP approach and its potential contribution to USAID/Uganda development results. The evaluation will contribute to building a picture of the DOP process by not only looking at the causal chain of effects that lead to results, but also considering the ‘web’ of interactions between different actors, forces and trends, and their effect on the results USAID and its partners are striving to achieve. The evaluation will highlight key drivers of change across different districts and stakeholders. USAID/Uganda, implementing and local government partners will use key lessons learned and recommendations from this evaluation to develop better program implementation strategies and build stronger partnerships for greater development results. As USAID/Uganda comes to the end of its first Country Development and Cooperation Strategy (CDCS), this evaluation will provide feedback on the overall effectiveness of the geo￾focusing approach, the selectivity principle, and its implications for more efficient and effective programming going forward. QED Group, LLC (hereafter QED) has subcontracted SoCha, LLC (hereafter SoCha) through the USAID/Uganda Monitoring, Evaluation and Learning Contract (aka The Learning Contract) mechanism to perform the work associated with completing the above listed objective. In line with SoCha’s proposal, the evaluation team has advanced a two-phased approach: an initial “Design” phase and a subsequent “Implementation” Phase.62 This “Evaluation Design and Protocol.” report is the final product of the Phase 1 Design. It includes a discussion of what the evaluation team learned during Phase 1 regarding the evaluative needs of the Mission, revised evaluation questions, the key evaluation design and methodologies, data collection tools, instruments and detailed survey protocol, and the Phase 2 implementation workplan. DOP Evaluation Learning Agenda This section is presented as a summary discussion of information gathered during Phase 1 around priority learning agenda items as expressed through the evaluation questions. Specifically, it is based upon the following efforts to understand and confirm the evaluation’s learning objectives: • Ongoing meetings with the main implementer of the DOP activity, Strengthening Decentralized Systems (SDS), including a one-day “success” workshop held at their office in Kampala; discussions with the previous SDS Chief of Party (CoP), additional discussions with staff members from the “Governance, Accountability; Participation and Performance (GAPP) activity, and ongoing conversations with Learning Contract staff; • Observations of four DMCs and discussions with corresponding district level representatives (Kamwenge, Mbale, Sironko, and Kapchorwa); 62 The original Scope of Work (SoW) anticipated three phases: design, implementation and close out phase. This closeout phase has not disappeared, but has been combined with the implementation phase under the same contract mechanism. The expected deliverables from each phase have also remained the same. x • Several meetings with USAID/Uganda personnel in the Program Office, Front Office and select Development Objective (DO) representatives; three former staff members who were previously involved in the DOP process but have since transferred; and ongoing interactions with the current DOP Coordinator/USAID Resident Fellow; • Extensive data collection and document review; • Construction of a USAID/Uganda IP district database, in cooperation with the Mission’s GIS team; • Consultations with two separate Ugandan local governance experts; • Meetings with three data collection/survey firms and a collection of local governance data roundtable with the Research Evaluation Learning and Monitoring (RELM) working group; • A “DOP Update and Confirmation” session/presentation held at the Mission to discuss and confirm the evaluation’s priority learning agenda items and evaluation questions; and • Comments submitted by evaluation stakeholders on the initial Phase 1 inception report, as well as subsequent comments made after the confirmation session regarding the final wording of the evaluation questions. The goal of this section is not to present preliminary findings or offer early evaluative diagnoses. Rather, it is to introduce relevant information that shapes the current learning agenda and justify the selection of the evaluation questions. The Question of DOP Relevance The DOP process was born in the context of Presidential Policy Directive 6 (PPD-6), which calls upon USAID Missions worldwide to develop in a more strategic, cohesive, and efficient approach to focusing investments and conducting development through “partnerships” to cultivate long term sustainability. USAID/Uganda, as the first mission to successfully author a CDCS, interpreted PPD-6 through the lens of “geo-focusing” and subsequently defined it in terms of a requirement that at least one activity from each DO, i.e. Health, Economic Growth, and Democracy and Governance, be present in a district for it to count as “focused.” Geo-focusing was further defined in terms of the “central hypothesis,” i.e. the notion that development results will be greater in districts where all three DOs are present, which identified 19 MFDs. Yet proponents of the central hypothesis recognized that intensely concentrating activities in select districts could potentially lead to negative outcomes, such as increased/unreasonable demands on district representatives’ time, duplicative/conflicting interventions, and poorly aligned implementation cycles that might miss opportunities for greater synergy. In other words, the DOP process was partially conceived as a preventative mechanism designed to mitigate the potential unintended consequences of greater activity concentration, and thus initial DOP relevance was/is found in the Mission’s own understanding of how to realize the goals and objectives of the first CDCS. Additionally, the DOP process was also relevant to how the Government of Uganda (GoU) conceptualized its own “partnership policy” with USAID and other donors. The partnership policy reaffirmed principles already expressed in the Paris Declaration on Aid Effectiveness, i.e. country ownership, alignment, harmonization, results, and mutual accountability, and was also designed to specifically complement the GoU’s own Vision 2040 strategy, which is expressed through the ongoing National Development Plan (NDP) and the lower level District Development Plans (DDPs) that flow out of the NDP. Here, the DOP process served as an entry point for the Mission link up and complement these frameworks, as the initial DOP MoU included provisions for the alignment of USAID development priorities with those of each respective DDP. The above listed discussion has already been captured in summary form in the 2012 DOP theory of change document.63 However, this was an internal document that was not circulated for public consumption. Nor 63 See “District Operational Plans in Uganda: What Are They, and Why They Should Result in Better Development”, July 2012, authored by Jeremiah Carew. xi have subsequent efforts been made to bring DOP relevance up to date with, interalia, changes to the NDP, aid effectiveness initiatives instigated by the Ugandan Ministry of Finance, Planning and Economic Development (MoFPED), and evolving interpretations of the 2010 Local Government Act, which is overseen by the Ministry of Local Government (MoLG). Indeed, there remains a need to capture and document MoLG’s views on how the DOP process, not only in theory but also in practice, is relevant to the GoU; as well as cross-check these views with independent Ugandan local governance experts to gain a fuller picture. Such documentation is necessary not only in terms of potentially justifying whether or not the DOP process should continue, but also to serve as a piece of institutional memory for subsequent Mission staff to understand how fits into a wider engagement strategy with the GoU. Therefore, the first evaluation question is: Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? The Question of DOP Implementation This evaluation makes two important, yet inter-related, distinctions around what is widely referred to as geo￾focusing and the DOPs. First, geo-focusing must be understood within the wider context of PPD-6 and the CDCS. There is nothing inherent in geo-focusing that necessarily implies a DOP modality. Indeed, the DOP first and foremost should be seen as only one of many potential modalities to support sustained USAID/GoU engagement, and is therefore not the basis for that engagement. Second, we make a distinction between the DOP as a coordinating activity (implemented by a partner or through direct implementation), and the wider DOP process that involves ongoing engagement among IPs, USAID representatives and district level officials. Although they are inter-related, the DOP process is best understood as an expression of the wider geo-focusing approach. How this strategy was communicated and consistently sustained throughout implementation is an important first step in understanding how well the DOP unfolded as an activity, what kind of incentives were in place to drive key actors, and if the key decisions made were the right ones. Should the DOP process move forward into the next CDCS cycle, documenting how it was implemented as an activity will enable the Mission to avoid making the same mistakes twice, as well as take forward implementation successes that are worth replicating. As an activity, the DOP diverged according to two models of implementation: through direct technical cooperation with the USAID Northern Uganda Field Office (NUFO) and through the SDS activity as a subset of its Objective 1: To coordinate activities at district and sub county level. Both modalities underwent their own respective evolution, faced unique sets of challenges, required different types of resources and inputs, and thus offer different lessons to be learned. These variations hold enormous learning potential to strengthen the DOP proof of concept, and could lay the ground work for a more flexible approach that is better tailored according to varying levels of district-level commitment on the one hand, and varying levels of Mission and IP resources on the other. Understood in the form of the second evaluation question, it is: Question 2: To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? What factors drove variations in its implementation? The Question of DOP Effectiveness The most pressing agenda item facing the DOP evaluation revolves is the question of DOP effectiveness. Answering this question starts with a focus on the DOP theory of change and its predicted results. Previously it was mentioned that the DOP process partially emerged as a preventive mechanism designed to mitigate the potential unintended consequences of greater activity concentration. Additionally, DOP proponents also predicted that the process may facilitate various positive “systems” and “spillover” effects, numerous efficiency gains, increased strategic coherence, increased mutual accountability, and lower transaction costs. Subsequent efforts to develop the DOP theory of change yielded the DOP causal pathway and various interpretations of the “DOP effect” offered by different stakeholders. The evaluation team classified these various outcomes according to a typology of improved Coordination, Alignment, Collaboration and Integration (CACI) of IP/District relations. CACI improvements were also expected to xii result in better targeting of investments, increased capacity and ability to deliver services, and increased efficiency, which then lead to better development outcomes. The evaluation team has already learned that USAID/Uganda made sufficient efforts to define and elaborate the DOP theory of change (although there is always room for improvement). What remains is to collect sufficient evidence to empirically test the DOP hypothesis in terms of the actual outcomes it produced (or at least contributed to). One particular challenge, however, with the DOP theory of change is that the single intervention (mostly focused around the conduct of quarterly DMC meetings) was hypothesized to lead to multiple outcomes (vs. a more traditional development hypothesis in which the intervention is associated with only one outcome). As such, the evaluation team must cast a wide conceptual net to empirically capture the inherent complexity of the DOP theory of change. Capturing DOP effectiveness must be a comprehensive exercise. The evaluation team discovered a myriad of views regarding DOP effectiveness based upon anecdotal evidence and limited exposure to DMC meetings. These views were highly dependent upon small sample size biases. When observers witnessed relatively well￾functioning DMCs, their overall view of DOP effectiveness tended to be quite favorable. Conversely, observers who witnessed poorly functioning DMCs tend to pose tough questions around its usefulness and view the exercise as more driven by ritual than result. Similarly, the evaluation team found anecdotal evidence of various “DOP effects,” but these need to be interrogated and more evidence compiled to determine if the outcomes are more systematic in nature and not one-off or random. Therefore, there is a strong demand for the evaluation team to build a comprehensive catalogue of all the incidences of effectiveness that can be connected to the DOP process. Incidences of effectiveness are defined in terms of interactions and evidences of CACI that fall outside of the DMC meetings, e.g. are part of the preparation process that go beyond the normal procedure and/or subsequent interactions that occur after (and as a result of) the meeting. Given both the wide variation of factors that influence DOP function across districts as well as the relatively manageable size of the DOP population, the catalogue should include a review of all 19 DOPs in which ever incidence of CACI should be catalogued. Doing so will not only provide a complete picture of DOP effectiveness, but will also enable the evaluation team to link this evidence with the cost model above to assess the wide gamut of benefits and their costs. The answer to the question of DOP effectiveness will be strengthened through counter-factual comparisons of non-DOP districts. But this raises an additional question of what is to be compared? The answer should be found in the priority learning areas the Mission would like explored. As part of Phase 1, the evaluation team hosted an interactive confirmation session to pose this question and offered three comparative possibilities: 1. Compare DOP districts vs. other SDS-supported (as well as GAPP-supported) districts to answer questions around how much district level support makes sense to improve CACI and/or can we get by with less? 2. Compare DOP districts (all 3 DOs present) to Non-DOP Districts that contain lighter concentrations of activities (2 DOs or fewer) to answer questions around should the DOPs be expanded to non-MFDs? 3. Compare DOP districts with Non-DOP Districts that have high numbers of IPs (all 3 DOs present) to answer what happens when the DOPs are not present? The evaluation team recommended and the USAID representatives concurred that comparison 3 was the priority learning question. In line with the six previous Mission Comparison Districts (MCDs), it was agreed that 6 districts would be selected to answer this question.64 64 The evaluation team deemed a comparison with the six original MCDs inappropriate for this type of analysis, as five of the six also received support from SDS or GAPP. Moreover, a previous analysis conducted by Dr. Fred Kisekka-Ntale suggested that, depending upon the indicator, some of the original six MFDs differed significantly from the MFDs and therefore could not be seen as equivalent xiii However, during the discussion, a fourth comparison was suggested: Compare DOP districts that received additional SDS system strengthening support (e.g. capacity building grants) with DOP districts that did not receive these additional SDS resources. This comparison would help the Mission understand if in fact, district willingness and DOP effectiveness were more of a byproduct of a larger support package by SDS, which would suggest that DOP effect outcomes are, at best, spurious correlations. Although this comparison would be compelling, the empirical reality is that it cannot be tested, as there are no DOP districts that have not received additional SDS or GAPP support. The exception may be found in the NUFO DOP districts if they began the process prior to additional systems strengthening, and the evaluation team will have to explore further to understand if a sound comparison can be made. Finally, in certain SDS districts, the DMCs have started to merge with the Technical Planning Committee (TPC). The TPC at times operates through closed door sessions consisting primarily of district and sub￾district representatives. However, some CAOs have seen the value of integrating the two modalities, which leads to an additional sub-question around improved effectiveness. As the evaluation team plans to conduct a visit the entire DOP universe, answering this sub-question does not require any additional case studies. In summary, this evaluation offers the opportunity to not only test if the predictions of negative unintended consequences of geo-focusing have come to pass (in high concentration districts with no DOP), but also if the corresponding positive “DOP effects” emerged as anticipated. Understood in the form of an evaluation question set, it is as follows: Question 3: Has the DOP process contributed to changes to coordination, alignment, collaboration and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? Are these relationships different in districts where the DOP has not been implemented? The Question of DOP Costs When faced with the decision of whether or not to move forward, the question of how much an activity costs snap to the front and center. In the case of the DOPs, there has not been a dedicated effort to disentangle DOP expenditures from the wider Objective 1 and SDS spending, quantify the amount of effort and resources dedicated by Mission staff to engage and support the DOP process, and estimate the level of effort put forward by district level technical staff. The purpose of doing so would not be to hold implementers accountable to the disbursement decisions they made throughout the life of the activity, but instead would be to collect enough information from them to build a cost model that could project estimated costs across the districts of Uganda should the Mission decide to expand the DOPs elsewhere. It should be noted that the utility of any activity cost model rests upon both the quality of its data and the soundness of its assumptions. The evaluation team has already collected sufficient expenditure information from SDS and has compiled staff levels of effort according to start up and ongoing costs. However, the evaluation team may face challenges collecting reliable cost information at the districts regarding levels of effort and at the Mission-level regarding potentially procurement sensitive information such as security transport costs. These challenges will be mitigated by building a fully transparent cost model in which all assumptions and cost inputs are clearly defined and customizable so that it easily can be updated if Mission staff supply more reliable information. Once completed, the model can be used not only to inform the Mission’s decision on the next phase of the DOP, but also can be used to estimate the cost of the potential benefits that are captured above in the effectiveness section of the evaluation. Understood in the form of the second evaluation question, it is: Question 4: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? Additional Questions Potentially Not Addressed comparators. See Kisekka-Ntale (2013), “Reviewing the Sample Design for USAID/Uganda’s Evaluation of the Effect of Integration of Activities on Development Results in Selected Districts.” Internal Report. xiv The original SoW for this evaluation also included a question regarding the existence of evidence of longer term outcomes and improvements to higher-level development results. Phase 1’s preliminary review found that, in fact, there is little to no evidence of improvements that could be systematically linked to the DOPs. Therefore it was suggested that this question be dropped and that the discussion over why the evidence doesn’t exist fall under Question 2 regarding implementation. Views on this recommendation are divided, as some still see the merit of a standalone “impact” section in the report. The evaluation team is currently not including this question as part of the Phase 2 analysis, but includes it here should the Mission decide to move forward with in as per the comments received on this draft. The question is as follows: Is there evidence to suggest that the long term outcomes and higher-level development results in USAID supported sectors are better in the districts where DOPs are being implemented? If there is no evidence, why not and what could've been done? Summarized in terms of a matrix, the changes to the DOP evaluation questions are presented below: Revised Evaluation Questions Original Questions Comments Proposed Revised Questions How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? No comment on this question. Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo￾focusing approach? The evaluation will also investigate the factors driving variations in implementation. The first sentence opens the door to a discussion implementation, what was done well and potential missed opportunities. The second sentence fits better below regarding the more analytical approach to variation Question 2: To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo-focusing approach? What factors drove variations in its implementation? How have behaviors and relationships within and between USAID, implementing partners, and districts changed as a result of the DOP process? These two questions strongly overlap, as the behaviors and relationships among the various parties can be described in terms of coordination, collaboration, integration and alignment. The questions are ones of DOP effectiveness that go beyond DMC inputs and outputs. The last sentence about difference in results suggests comparative analyses of these factors, potentially with other districts. Question 3: Has the DOP process contributed to changes to coordination, alignment, collaboration and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? Are these relationships different in districts where the DOP has not been implemented? To what extent is the DOP process supporting and or hindering incremental improvements in coordination, collaboration, integration and alignment of USAID programs? What factors explain the differences in results among different DOP districts, between MFDs and MCDs, implementing partners, and development objectives? Proposed New Question In the exploratory phase, we've learned there is an appetite to discuss the costs/costs effectiveness of the DOP process to understand if it was value for money. We suggest a new question. Question 4: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? What evidence exists to suggest that long term outcomes and higher-level development results in USAID supported sectors are better in the districts where DOPs are being implemented? What potential is there to achieve such results? What other factors exist that could better support the achievement of these results? Phase 1 of this evaluation revealed that there is little quantitative evidence to directly answer this question and move beyond only a general level commentary. However, we can instead discuss what kind of approach and evidence was needed to directly address this question and offer some discussion of why this didn't happen/what happened instead. This question also again suggests a comparative analysis with non-DOP, USAID supported districts, which is addressed As per COR suggestion, this question is removed. xv above. The Approach The main research design of the evaluation is based upon an attempt to build a comprehensive catalogue of the DOP experience to adequately address the evaluation questions. This design will be further guided by two factors: First, the DOP process must be understood in terms of a multi-stakeholder approach - primarily SDS/NUFO representatives, USAID DOP representatives, relevant IPs and District Level technical staff (i.e. the CAO, technical department heads and the District Planner) – operating within a previously-established GoU legal and institutional framework. Second, evaluation inquiries must remain cognizant of the previous work already done and build upon it (as opposed to replicating it). Below we elaborate on this approach. SDS/NUFO representatives The evaluation team has already held extensive conversations with SDS DOP staff, collected information regarding their specific financial and human resource inputs, systematically discussed, defined and catalogued their own evidence of CACI outcomes and noted their lessons learned/recommendations. The evaluation team will conduct a similar process for the NUFO DOPS (i.e. prior to SDS support) with the current DOP coordinator and, if possible, the previous USAID staff representative (the representative has left the Mission but is still potentially available to provide information in Uganda). USAID Representatives Documentation of the USAID DOP representative experience is mostly found in various DMC trip reports. Additionally, as part of the DOP process, staff from the Program Office held periodic reflection and review sessions with DOP representatives to understand how the process was proceeding. Building upon this work, the evaluation team will attempt to meet with all DOP representatives (most likely through 2-3 focus group workshops) to: • identify definitions of DOP success according to USAID representatives’ perspectives; • understand USAID representatives’ activities and level of engagement with the DOP process; and • develop a timeline, per DOP district, of key moments which indicate success as mapped to intermediate results listed in the DOP causal pathway. Implementing Partners As part of a previous effort to understand how IPs were responding to the DOP process, USAID/Uganda commissioned a survey of IPs and District staff in 2013. The survey asked a variety of questions on the DOP process and did explore the question of DOP outcomes. However, these questions were posed at a very general level (e.g. “has the DOP process improved collaboration) and did not generate concrete evidence on DOP effectiveness. Also, in March 2015 the Mission conducted a CoP feedback session on ways to improve the DOP. Although useful, the session did not systematically collect evidence on outcomes and effectiveness. For this evaluation, the team will collect additional IP information to better understand DOP effectiveness. Through a survey and a select number of key informant interviews, the evaluation team will document: A. The “Continuum of Flexibility” a. One of the key stated objectives of the DOP process is better implementing partner coordination with each other, local government, and other implementers at the district level. The ability of implementing partners to fulfill this objective requires that they are able to adapt to shifting priorities at the district level. However, due to constraints such as organizational mandates and defined scopes of work (as well as a lack of willingness), implementing partners may not always have the authority or the desire to accommodate all such requests. xvi b. The evaluation team will include questions to solicit IPs experiences within a “continuum of flexibility,” to understand instances in which implementing partners have or have not accommodated requests for activity changes, and why. B. Targeting investments a. Better targeting of investments, in alignment with district government priorities, is another key stated objective of the DOP process. The evaluation team seeks to understand how implementing partners target their activities in a given district. The survey will include a question to the implementing partners to assess whether their activities are aligned with district priorities as a result of: i. Co incidence ii. Deliberate effort - pre planning (blank slate) iii. Adjustment iv. Unable to adjust v. No alignment C. Outlier examples: a. An open space in which Implementing Partners can contribute other examples from their experiences which point to coordination or lack of coordination with district government and other stakeholders. District Technical Officials As already mentioned, the previous DOP survey included district technical representatives in the sample, but did not yield concrete information that could be used to confirm/reject the DOP theory of change and effectiveness. The Learning Contract also hosted a “DOP Learning Event” in November 2013, which also generated useful feedback on the way forward, but did not garner the evidence needed for adequate hypothesis testing. Therefore, the bulk of evaluative efforts will come from a comprehensive sample of all 19 districts to systematically collect evidence of CACI experiences and outcomes. The evaluation team will use a variety of techniques to collect this information (outlined below in the Tool section). Other Donors Preliminary evidence already shows that implementing partners funded by other donors have attended some DMC meetings. For Phase 2, the evaluation team will continue to compile the list of non-USAID representatives funded by other donors and also include them in the interview discussions. Questions will range from preliminary descriptions to more detailed inquires around evidence that similar CACI outcomes may have also been shared by these groups. Depending on their location and availability, these interviews may be conducted remotely. Comparison Districts As discussed, the evaluation team will sample six comparison districts to address the question of “what happens when the DOP is not implemented in districts with a high concentration of activities?” “High concentration” refers to districts that host all 3 DOs. Since the selection of the original 19 MFDs, around 22 additional districts have received activity support from all three DOs (but no DOPs). Out of these, only 9 (not including Kampala) host all three DOs but do not receive additional support under SDS or GAPP. These are: Districts with all 3DOs present by no GAPP or SDS support Region District Selected? East Jinja Yes West Kabarole xvii North Kaberamaido Yes West Mbarara Yes North Moroto West Rubirizi Yes East Soroti East Tororo Yes Central Wakiso yes When selecting the six districts, it is important to consider how geography may also be a factor in DOP effectiveness; therefore, some degree of regional representation should be maintained. Ideally, two districts from the North and East should also be selected, as these regions have the highest concentration of USAID￾supported districts. However, there is only one high concentration, non-DOP district in the North – Moroto, so the evaluation team will select two districts from the West. For the East, Kaberamaido and Tororo were selected, as the former has received little evaluative attention in the past, while Tororo was originally selected as a MCD and also found to lack significant differences from MFDs on select indicators. Wakiso District is the only option for the Central Region. Finally, Mbarara and Rubirizi districts were selected from the West as Kabarole has been the source of intense coordination under various DO3 activities due to its status as a regional referral center for the area. Additional Sources While the bulk of Phase 2 will be directed to the efforts above, the evaluation team will also continue to compile, review and synthesize previous DOP documentation, interview Key Informants and other stakeholders, review DOP policy relevance and collect information needed to build the cost model. To assist, the evaluation team will also hire a local governance advisor throughout Phase 2 to provide technical guidance on the institutional and policy framework, especially in regards to the Law on Local Government in reference to DOP relevance, on the appropriateness of district level tools, and lead the analysis of DOP relevance in the national policy context. Most of the essential expenditure data has already been collected from SDS. To transform this data into a cost model, the evaluation team will collect additional human and potential budget resource information from USAID, and combine this information with previously-established per diem and transport allowance costs to build a model that can estimate costs across all districts in Uganda. The model will also include various operational scenarios, which will allow the Mission to plan should some of the current assumptions change. Sampling Strategy As suggested above, the evaluation team will conduct a comprehensive sample of the DOP districts. This is primarily because the Mission has made it clear to the evaluation team that the evidence produced by this evaluation go beyond the anecdotal, and therefore any sampling strategy must be able to demonstrate its level of representativeness. Yet doing so is difficult for a population universe of this size (i.e. 19). First, a “representative” random sample according to the commonly accepted goal of 95% confidence with a +/-5% margin of error would require a sample size of 18, i.e. the differences between a representative random sample and a full case review are minimal. Second, the amount of diversity across each district varies considerably and we currently have no basis for commonality to conduct a stratified sample, i.e. we cannot determine how representative the sample would be if we stratified. Finally, the large number of comparative questions, e.g. NUFO vs. SDS; DOP w/TPC vs. DOP w/o TPC; DOP vs. Non-DOP; etc. demands that we have a robust collection of districts to adequately address all of these areas. Thus, the evaluation team has identified a comprehensive catalogue of district experiences as the best course of action. Methods and Analytical Strategy xviii The evaluation matrix presented below defines the boundaries of the evaluation scope of work, questions to be answered, analytical terrain to be explored, corresponding data requirements and operating assumptions. Some of the items listed below have already been discussed and/or are self-explanatory. This section primarily focuses on the choice of established method and analytical strategy behind each evaluation question: Revised Evaluation Matrix Evaluation Question Set Type of Answer Method Data Required Sample Strategy Coordination Assumptions Question 1: How does the DOP approach reinforce or not national government policies and strategies on country ownership, partnership, coordination of development assistance and building of stronger local governments? Institutional Analytical Review of Policy Documents and Logical Testing Key Informant Interview data; Document Review of national policies and strategies, etc. Purposive, based upon ID of key informants GoU remain cooperative and supportive; District Development Plans are accessible Question 2: To what extent did USAID Uganda articulate clearly and thereafter consistently implement the geo￾focusing approach? Confirmational, Narrative Loose Comparisons and Descriptive Activity M&E data; Interview data, Program Documents and Regular Reports, Reviews, etc. Purposive, based upon Mission guidance SDS continues to provide support, NUFO USAID representative provides input Question 3: Has the DOP process contributed to changes to collaboration, alignment, coordination and integration of USAID programs as well as other types of relationships among USAID/Uganda representatives, implementing partners and DOP representatives? What explains variations of these across the DOP and non DOP districts? Formal comparisons of Changes to Practices within DOP and with non-DOP districts; Catalogue of Outcomes that fall outside DMC meetings Outcome Harvesting, potentially ONA and QCA (tbd. in week 4 of the evaluation) ONA data (and transformed for district analysis); IP Perception Survey data; FGD data from USAID Reps; outcome data from District Officials; SDS and NUFO activity data; additional context data Comprehensive plus 6 comparator districts ONA data and findings are usable, SoCha staff understand O/H methods, District Officials respond to O/H methods, IPs respond to survey; USAID Reps are available; outcomes are clearly defined and there is adequate variation to run QCA Question 4: Are the costs (both in terms of human and financial resources) of the DOP process commensurate with the benefits? Value for Money Cost Modeling Activity actual cost data, USAID cost estimates; district fuel and per diem rates Enough market data to build the model M&E Data is available, 3rd Party Firm collects Outcome Data During Phase 1, the evaluation team considered a number of evaluation methods and techniques to support the analysis. In doing so, the team distinguished between analytical methods and data collection approaches. An analytical method applies a clear set of rules to the data to reach certain conclusions. For examples, in statistics there are clear rules about what levels of significance are acceptable and unacceptable, or to what extent the differences between two means are significant when taking into account random error. Conversely, data collection approaches are those which present a series of sequential steps and interview tactics to gathering data, but do not present clear rules for analysis. Another useful distinction made by the team concerns formal and informal analysis. Formal analysis refers to analytical methods that have clear criteria that always yield the same result regardless of who is conducting the analysis, e.g. through a software program. Informal analysis refers to methods that may yield different results depending on who is doing the analysis, e.g. expert determination. Data Collection Methods that were considered but not endorsed: xix Most Significant Change (MSC): MSC is a qualitative, conversational method that is used to identify how/if projects leave more memorable (as opposed to average) changes to the lives of beneficiaries. MSC captures these changes through various domains of change, what types of changes are more significant and varying means of verification. As an interview tactic, MSC’s search for “meaningful” changes is a promising approach, but was not selected due to its inherent bias against outcomes that are not as provocative. MSC also lacks formal criteria for determining what is significant and what isn’t, and previously left the decision to loosely structured panel groups (the results of which varied according to the group). However, MSC’s founder, Rick Davies, has moved more towards applying formal comparative criteria to MSC data through decision-tree (dtree) and Qualitative Comparative Analysis (QCA)-based algorithms (indeed, Mr. Davies has/is working with SoCha on similar exercises in Indonesia and for the Comic Relief Foundation across four African cities). The merits of using each analytical approach are addressed below. Outcome Mapping (O/M): O/M is a potentially very interesting technique for documenting the types of changes in bureaucratic behavior and administrative practices the DOP evaluation would like to discover. It focuses on the “missing middle” of the results chain by identify what happens after an implementer provides the support as the beneficiary experiments with new types of behaviors and practices. Unfortunately, however, O/M’s primary data input usually comes from beneficiary diaries that not only require a great deal of time and resources to maintain, but also need to be introduced at the beginning of an activity to effectively document change. As this was not done at the onset of the DOP process, the window of opportunity to use O/M has closed. Indepth Case Studies: Although not typically defined in terms of procedural steps and guidelines, in-depth case studies are useful for understanding how different elements fit together and how different elements (implementation, context and other factors) have produced the observed impacts. Indeed, it is quite common for evaluations of this nature to sample only a few DOP cases for in-depth exploration to make wider claims about the DOP experience as a whole. However, as already mentioned, the wide variety of district contexts in which the DOP process has operated makes any in-depth study highly vulnerable to sampling bias; thus, the evaluation team has avoided this type of method for this evaluation. Data Collection Method Chosen Outcome Harvesting (O/H): O/H is an approach useful for mapping out the events, over time, that led to certain outcomes in a project. Under O/H, an evaluator analyzes a known outcome and works backwards to trace the actors, steps, factors, and key moments in time which led to a certain project outcome. The relationships between these components can then be mapped out into a visual narrative of the key changes which led to the outcome in question. The evaluation team will use O/H to build “outcome maps” for each district to identify various pathways taken by each district throughout the life of the DOP to better understand which outcomes are common and which are unique. Annex 1 provides a illustrative example of what an effective O/H map may look like in this evaluation. As discussed below, this type of temporally￾sensitive data may also be used to conduct more formal analysis of which events are necessary and/or sufficient for the outcomes to occur using QCA. Formal Analytical Methods Not Chosen Statistical Modeling: By far the most common formal method used in evaluations today involves some type of statistical modeling of program and outcome data. However, statistical modeling itself requires that a number of data quality, data consistency/fidelity, and sample size considerations are met. For this evaluation, the main unit of analysis is the district, which yields a total population size of 25 (19 DOPs plus 6 comparators). As most statistical models assume at least 30 data points, this type of formal method is not appropriate and any results produced would most likely be highly unstable. dTree: As suggested above, dTree analysis is a potentially useful formal method for modeling qualitative data and drawing rigorous conclusions. dTree analysis presents variables across linearized decision rules where the outcome sits at the end of the tree, and the conditions behind the outcome define the branches. In this way, xx multiple pathways to success can be identified and mapped; an approach especially useful for context-specific outcomes with smaller sample sizes. Rick Davies and others have compared the results using dTree with those of QCA using the same dataset and found that: (1) a decision tree analysis yielded fewer paths to achieving success (i.e. it managed complexity better), and that (2) those paths yield levels of accuracy that are equivalent to QCA.65 While the results have been subject to considerable criticism,66 the explanation for fewer pathways and improved accuracy is found in the linear nature of dTree in that the algorithm’s comparison rules automatically eliminate simultaneous comparisons of the same pathway elsewhere, and instead fixate only on the results of the first comparison. This inherently limits the number of potential pathways as part of the solution set. As such, the results of dTree are highly dependent upon the order in which the evidence is presented. Formal Analytical Methods Chosen Organizational Network Analysis: ONAs help visualize the connections across functions and between organizations. It can capture the sharing of information, collaborations, and information flows. An ONA for key DOP stakeholders in the MFDs, led by the Learning Contract, is already underway. The evaluation team will build upon this analysis to better understand the relationship, if any, between different actors’ networks and the effectiveness of the DOP mechanism in each district. When pinned against the outcome maps identified under O/H, the ONA will demonstrate how events may increase/intensify networks of collaboration over time. However, there are a few challenges with the ONA data, which may prevent a formal incorporation into this evaluation. These are: • The unit of inquiry is blurred between individuals and organizations, i.e. individuals are the respondents but represented as organizations. This creates problems in larger organizations, as they appear several times in top broker categories even though they are separate individuals and distort many network indicators; • The ONA unit of analysis focuses on networks and not districts. This is problematic for the IP data as an IP may appear to be strongly “networked” due solely to the fact that it appears across multiple districts (i.e. a structural cause) instead of any collaborative properties it possess. This “structural bias” could potentially overcome IPs more limited in geographic scope but more capable in terms of collaborative efforts. As a result, direct comparisons between IP and District networks are not reliable. • Positive social response bias: Some of the questions in the ONA survey are inherently loaded with a positive social response bias that predisposes higher levels of collaboration in the results. For example, when IPs were asked if they felt engaging the DOPs was important, they were not given the option to indicate engagement was “not important.” Another example is that district officials were asked if they had collaborated on cost sharing agreements (34.7% said they had), but were not asked if these collaborations involved IPs (most likely they didn’t). While some of these challenges can be mitigated through careful interpretation, the remaining question behind using the ONA data for this evaluation revolves around our ability to convert the network properties of the dataset to district properties. To do this, the evaluation team will need to identify some criteria for membership in each district network as well as account for interdependence between districts. The ONA consultant has agreed to work with the evaluation team on this. QCA: This method is especially useful for complex programs that seek to understand how various combinations of implementation activities work together to contribute to program outcomes in and across a 65 The study was presented at the biannual European Evaluation Session in Dublin in 2014. See the presentation here: https://www.youtube.com/watch?v=dHhSQHX01L0 66 See http://vawreview.blogspot.co.uk/2014/10/qualifying-rick-daviess-findings-from.html xxi variety of settings.67 This method models the DOP experience according to a “truth table” that identifies which sets of activities each district received, external factors that are relevant to the development results DOP hopes to achieve, and the actual development results that occurred under the program. As a rigorous analytical technique, QCA uses a Boolean Algebra-based algorithm to identify which combinations of DOP activities, events and conditions are necessary for success, which ones may not be necessary but are still sufficient for success, and which ones are superfluous relative to the other more essential factors. We anticipate that this will yield various “collaborative pathways,” which can help guide the Mission throughout the implementation lifecycle should it move forward with another iteration of the DOP process. However, the use of QCA for this evaluation is not guaranteed. One of the main prerequisites to using this method is that it requires a clear definition of success. As discussed above, the DOP ToC in fact suggests multiple types of success and that these variations definitions may themselves be only applicable to certain contexts but not to others. By the end of the data collection period (the fourth week of Phase 2), the evaluation team will compile the data and discuss with the evaluation team management which, if any, of the outcome variables can/should be used as the definition of success. In summary, due to the inherent challenges of the data and methods mentioned above, a formal analysis of the evaluation findings using ONA techniques and QCA may not be possible for Question 3. If this turns out to be true, the evaluation team will be limited to the informal techniques mentioned above, which will still be useful and sufficient to answer the main evaluation questions. DOP Evaluation Implementation Workplan Phase 2 of the evaluation is expected to take six weeks and is laid out according to the workplan flow chart listed below. The workplan assumes that the team will visit all 19 DOP districts as well as six comparator districts across five data collection teams. Each team will be composed of two members: a lead facilitator who will guide the inquiries and an administrative support member who will take notes and capture responses. The field teams will be supported by a logistics officer during the field visits. Additionally, the local governance advisor will provide continuous technical support throughout the evaluation. The team leader will manage the evaluation team as well as conduct field visits throughout the evaluation. Pre- Phase 2 Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 1. Set-up and training SDS document request District Development plan copies Ministry of Local Government meeting request Hire Enumerators Hire Local Governance Advisor Hire Administrative staff Train Enumerators Start developing district profiles 2. District Send communication to district office Organize logistics 67 The main premise of QCA is that of multiple conjunctural causation, which means that (1) most often not one factor but a combination of activities lead to development results; (2) different combinations of activities can produce the same outcome; (3) one activity can have different impacts on the outcome, depending on its combination with other factors and the context; and (4) the absence of an activity may be just as important as its presence. xxii Interviews at district 3. Implementing partners Develop survey Implement survey Schedule IP Interviews Conduct IP interviews 4. USAID Schedule USAID workshops with DOP reps Conduct success workshops 5. Finalize report Analysis and write-up Submission Staffing (all in-country)** Team Leader - Corey Patterson Evaluation Expert - Zoe So 4 Enumerators (Facilitators - mid level) 4 Enumerators (Support - junior level) Logistics Officer (1) Local Governance Expert (1) Evaluation Report Write Up and Outline Throughout the conduct of the evaluation, the evaluation team will be guided by the principles of “Utilization focused evaluations” (UFEs),68 which focus on providing information for specific users, uses and decisions. As such, our main strategy will be to ensure that the intended uses of the evaluation by the primary users guide all other decisions that are made during the evaluation process. Yet the write up will conform to the standards and guidelines set forth under the OECD’s DAC Criteria for Evaluating Development Assistance. Specifically, the evaluation report will be drafted according to the following outline: I. Front Matter (title page, acronyms, etc.) II. Executive Summary, including a findings and recommendations III. Introduction a. Background b. Scope of Work c. Level of Effort d. Methods and Data Collection e. Challenges and Limitations IV. Question of DOP Relevance (Question #1) a. Intro b. Findings c. Recommendations V. Question of DOP Implementation (Question #2) a. Intro b. Findings c. Recommendations 68 The term, and its approach, are well articulated in Michael Quinn Patton’s well-known book Utilization-Focused Evaluation, now in its 4th Edition. xxiii VI. Question of DOP Effectiveness (Question #3) a. Intro b. Findings c. Recommendations VII. Question of DOP Costs (Question #4) a. Intro b. Findings c. Recommendations VIII. Overall Conclusions, based upon various scenarios of the CDCS IX. Back Matter (Annexes) Proposed Innovation for Disseminating the Findings of the Evaluation As per the requirements of the SoW, once the evaluation is completed, SoCha will search for innovative ways to disseminate the findings of the evaluation. In addition to the inherently participatory approach to disseminating the findings mentioned above and to the formal submissions made to the DEC, SoCha will also seek to disseminate the findings through: • Sub-regional DOP learning events (as appropriate) • National Conferences and Events, such as the Ugandan Evaluation Week • Regional Conference on Local Governance and Aid Coordination Effectiveness • Specialized Conferences and Forums that focus on the use of QCA for Evaluation (such as through COMPASSS); • USAID’s Learning Lab • Other venues as appropriate. Note, SoCha understands that all disseminations are subject to USAID prior approval. xxiv Annex 3: List of People Interviewed and Documents Reviewed List of People Interviewed Name Position Organization/District Onen Anthony Population Officer/ DOP FP Amuru District Office JB Okello Olum DCDO Amuru District Office Kyonda Muhamudu SDS DFP/ D-Planner Bugiri George William Omuge CAO Bugiri Vacant LCV -Chairperson Bugiri Omuge George William CAO Bugiri District Office Bateganya Edith DFP/Gender Officer Bugiri District Office Mwebaze Charles SDS DFP/ D-Planner Bushenyi Lilian Nakamatte CAO Bushenyi Willis Bashaasha LCV -Chairperson Bushenyi Watti Simon Peter DCAO Bushenyi District Office Mwebaze Charles DFP/District Planner Bushenyi District Office Christopher Sande Kyomya CAO Dokolo District Office Ogwal Alfred Population Officer Dokolo District Office Rita Laker-Ojok Chief of Party FTF Agricultural Inputs (Tetra￾Tech) Patrick Rader Chief of Party FTF Commodity Production and Marketing Activity (Chemonics) Tom Kyakwise DCOP GAPP Eva Matsiko COP GAPP Oruut Jimmy DCDO/ DOP FP Gulu District Office Robert Ongom Ag. DHO Gulu District Office Sylvia Magezi Chief of Party HarvestPlus Sam Wekesa Masaba SDS DFP/ D-Planner Ibanda Alice Assimwe Rushure CAO Ibanda Melichiadis Kazwengye LCV -Chairperson Ibanda Bakesiima Patrick District Planner/ SDS FP Ibanda District Office Nsubuga Zirimenye DCAO Ibanda District Office Jonah Kayemba SDS DFP/ D-Planner Iganga Joseph Maira Mukasa CAO Iganga Shaban Sadiq Nkuutu LCV -Chairperson Iganga Maira Joseph Chief Administrative Officer Iganga District Office xxv Mukasa Kayemba Jonah District Planner Iganga District Office Fred Godoba Political Economy Specialist Independent Consultant James Mugisha Senior Health planner Intersectoral Committee/ MOH Stephen Besiga SDS DFP/ D-Planner Isingiro Alex Kwizera CAO Isingiro Ignatius Byaruhanga LCV -Chairperson Isingiro Mugarura Edward DCDO Isingiro District Office Byamazima Innocent Stastician Isingiro District Office Robert Banafamu SDS DFP/ D-Planner Kamuli Ben Otim CAO Kamuli Proscovia Salaamu Musumba LCV -Chairperson Kamuli Ben Ogwette Otim CAO Kamuli District Office Banfamu Robert District Planner Kamuli District Office Tuhairwe Getrude SDS DFP /District CDO Kamwenge Owen Rujumba CAO Kamwenge Robert Itwara Kamasaka, LCV -Chairperson Kamwenge Masereka Amis CAO Kamwenge District Office Okumbuke Shaban SDS Focal person Kamwenge District Office Teko Andrew SDS DFP/ D-Planner Kapchorwa Joyce Loyce Nambozo CAO Kapchorwa Sam Cheptoris LCV -Chairperson Kapchorwa Teko Andrew District Planner Kapchorwa District Office Cherotich Dan. Z District Vice Chairperson Kapchorwa District Office Sowedi Kitanywa SDS DFP /Sen. Probation and Welfare Officer Kasese William Kanyesigye CAO Kasese Lt Col. Mawa Muhindo LCV -Chairperson Kasese William M. Kanyesigye CAO Kasese District Office Kitanywa Soweddi DCDO Kasese District Office Brian Ramsey CDCS Secretariat Learning Contract Charlotta Sandin CDCS Secretariat Learning Contract Stuart Belle Knowledge Management & Organizaitonal Learning Advisor, USAID Monitoring, Evaluation and Learning Contract xxvi Learning Program Elias Byamungu CAO Lira District Office Olao Ronny Population Officer Lira District Office Johnson Gumisiriza Principal Economist Local government finance commission David Adriance Chief of Party Long Term Family Planning (Marie Stopes International) Mr. Eustace Gakwandi CAO Luwero James Oketayot SDS DFP / DCAO Luwero Haji Abdul Nadduli LCV -Chairperson Luwero Oketayot C.J DCAO Luwero District Office Kisakye Joseph SDS Focal person Luwero District Office Frederick Kisekka Ntale Research Fellow Makerere Institute of Social Research Gyaviira Dhikusooka SDS DFP/ D-Planner Mayuge Mr. Yusuf Senteza CAO Mayuge Omar Muwaya Bongo LCV -Chairperson Mayuge Dhikusooka Gyaviira SDS Focal person Mayuge District Office Muzige Paul ACAO Mayuge District Office Margarate Duca SDS DFP/ D-Planner Mbale Paul Walakira CAO Mbale Bernard E.M.Mujasi LCV -Chairperson Mbale Dr. John Baptist Waniaye District Health Officer Mbale District Office Margaret Duca Population Officer Mbale District Office James Mugisha Senior Health Planner Ministry of Health Swizin Kinga Mugyema Acting Commissioner Local Councils Development Ministry of Local Government Swizin Mugyema Assistant Commissioner of Local Councils Development MLG Dr. Joeseph Muvawala Executive Director National Planning Authority Boaz Musimenta Director Office of the Prime Minister Boaz Musimenta Principal policy analyst Office of the Prime Minister Laker Allen District Physical Planner/DOP FP Oyam District Office Agaro Caroline DHO Oyam District Office Leru Andrew CAO Pader District Office Catherine Amony Otto Population Officer Pader District Office Peter Alani Kamuli, Iganga SDS Godfrey Wabwire Mbale, Sironko SDS xxvii Moses Omara Kapchorwa SDS Francis Abwaimo Mayuge, Bugiri, Kasese, Kamwenge SDS Michael Ayebazibwe Ibanda SDS Robert Kalemba Bushenyi, Isingiro,Luwero SDS Johan Kleinhans Finance Director SDS (Cardno) Ella Hoxha Former CoP, SDS SDS (Cardno) Reilly Ross Communication Manager (Former USAID DOP Coordinator) SDS (Cardno) Anita Regional Grants Manager SDS (Cardno) Henry Kamau Kuria Deputy Chief of Party - Grants and Programmes SDS (Cardno) James Kakooza M&E Consultant SDS / Cardno Vincent Wanzala SDS DFP/ Ag.D-Planner Sironko Joseph Lomongin CAO Sironko Nabende James LCV -Chairperson Sironko Gidongo Peter Deputy CAO Sironko District Office Wozisi Fred District Planner Sironko District Office Augustin Muhwezi Chief of Party SUSTAIN (URC) Reilly Ross Fellow USAID Uganda Lane Pollack Organizational Learning Advisor USAID Uganda Jennifer VerNooy Deputy Team Leader, Program Office USAID Uganda Phil Greene Economist USAID Uganda Leslie Reed USAID/Uganda Mission Director USAID Uganda Harriet Busingye Muwanga COR for GAPP USAID Uganda Mark Messick USAID/Uganda Deputy Mission Director USAID Uganda May Mwaka Monitoring and Evaluation Specialist/Program Office USAID Uganda Xavier Ejoyi USAID DOP Rep - Mbale USAID Uganda Charles Egu USAID DOP Rep - Mbale USAID Uganda Richard Okello USAID DOP Rep - Mbale Geo-Information/Database Management Specialist USAID Uganda Kevin N USAID DOP Rep - Sironko USAID Uganda Carol Ssekandi USAID DOP Rep - Bugiri, Kasese USAID Uganda Zdenek Suda Team Leader, Programs Office USAID Uganda Simon Byabagambi USAID DOP Rep - Bushenyi, Ibanda USAID Uganda Angelina A￾Mpyisi USAID DOP Rep - Luwero USAID Uganda Peter Birigenda USAID DOP Rep - Iganga USAID Uganda xxviii List of Documents Reviewed Categories Documents Received CDCS & Geo￾focusing • Uganda CDCS 2011-2015 • PPD 6 • Uganda USAID Standard Indicator Policy • DOP Theory of Change • CDCS geo-focusing policy memo 2011 • Final report for matching USAID MFD • DOP Causal pathways document DOP Documentation • Signed DOP agreements • All meeting minutes, action items, presentations and attendance lists, collected at SDS and each District • District Development Plans • Organizational Network Analysis and data set • All USAID Rep DOP Trip Reports DOP Guides • April 16 USAID meeting summary.docx • DOP Handbook 2013.docx • DOP Implementation Strategy Oct 2013 FINAL.docx • DOP MOU Amendment_Sironko.pdf • DOP progress.xlsx • Executive leadership and facilitation skills workshop Participant Handbook.pdf • Executive leadership participant resource handbook.pdf • May 29 USAID - QED - SDS meeting summary.docx • September 18 USAID meeting summary-2.docx • SOW Objective 1.docx • Steps in the DOP journey.docx • Support to DOP Implementation.doc • TA Strategy Menu.pdf • USAID one pager on DOP Support.docx DOP Learning • COP Meetings feedback (Feb 2015) • DOP - Aid Effectiveness and Partnership Strengthening Tool_Cardno.pdf • DOPEvent_VideoScript.docx • CLA DOP Report • Grant A - Grants Management Case Study_Cardno.pdf • HSS - HRH Private Sector Partnership_Cardno.pdf • SDS CLA Competition - Grants improve service delivery_FINAL.docx • SDS CLA Competition - IPB strengthens DOPs_FINAL.docx • TA Impact Analysis Case Study_Cardno.pdf • 2013 DOP survey.pdf • Biannual IP Eastern region meeting report- Dec 12 2012.docx • December 2013 DOP Learning Event Report.pdf • Discussion during IP central region meeting.docx • DOP - Aid Effectiveness and Partnership Stregthening Tool_Cardno.pdf • DOP Analytical Report 2013 - Final.pdf • DOP and Grants Consolidated Jan 21.doc • DOP Annual Progress Report, Feb 2013.pdf • DOP Closing Remarks Kamuli CAO.pdf • DOP Progress graphs.docx xxix • DOP ROUNDTABLE MEETING NOTES NOVEMBER (2).docx • DOP survey for districts FINAL.doc • DOP _ Annual Progress Report _ Feb 2012 - Feb 2013-.doc • DOPEvent_VideoScript.docx • dopfolderonpdrive.zip • Eastern region DMC meeting attendance-Jan 13 to Sept 2013.docx • Francis Abwaimo -- Recent DOP Developments.docx • Grant A - Grants Management Case Study_Cardno.pdf • HSS - HRH Private Sector Partnership_Cardno.pdf • Minutes DOP MEL Working Group Meeting_23 Sep 2015.docx • Nov 7 2012 - initial DOP Feedback discussion.docx • Regional IPs Meeting Western Region -11th Dec 2102 (5).docx • Responsibility to Coordinate_SDS Training.ppt • SDS CLA Competition - Grants improve service delivery_FINAL.docx • SDS CLA Competition - IPB strengthens DOPs_FINAL.docx • SDS program - Year 5 Q3 report .docx“Joint United Nations Agencies and United States Agency for International Development Coordination Meeting” Report, July 29, 2014, held at the UNICEF Gulu Office.. • See the SDS “NUFO Concept Paper February 2015.” Internal USAID Publication. • USAID-SDS meeting minute summaries for April 16, May 29 and September 18, 2013 for confirmation. • Kisekka-Ntale, F. and Kibombo, R. (2013). “Reviewing the Sample Design for USAID/Uganda’s Evaluation of the Effect of Integration of Activities on Development Results in Selected Districts” Final Report. Unpublished USAID Document. Local Government and Uganda Context • MoLG website, accessed on 12/16/15. https://molg.go.ug/ministry/ • Local Government Planning Guidelines, at http://npa.ug/wp-content/uploads/LG￾PLANNING-GUIDELINES.pdf • LEARN (2015), “Ugandan Decentralization Policy and Issues Arising in the Health and Education Sectors: A Political Economy Study,” unpublished USAID report • Green, E. (2013), “The Rise and Fall of Decentralization in Contemporary Uganda,” Wider Working Paper No. 2013/087; • Stelman, U. (2012). “Understanding Organizational Performance in the City of Kampala,” Africa Power and Politics Working Paper No. 27; • Kasozi-Mulina, S. (2013). “Process and Outcomes of Participatory Budgeting in a Decentralised Local Government Framework: A Case in Uganda.” MEL • DOP MEL plan (May 2015) • Joint Results Framework • Common Indicator Policy • Data for common indicators, 2011, 2012, 2013 • Complexity Aware Monitoring SDS • SDS quarterly and annual reports • DOP Manual • DMC Meeting Notes & Attendance lists • SDS PMP • SDS Modification • DOP Trip reports GAPP • Scope of Work • Quarterly Reports xxx Annex 4: Perceived Success of the DMC Meetings and District Outcomes District Total Outcomes Concrete Outcomes Status Score Explanation for Score Amuru 7 4 Partial Failure 0.3 Focal person does use Google groups, but CAO is not supportive and DCAO slightly involved. IPs and Heads don't show up, IPs don't care, Action Points are procedural Bugiri 16 13 Success 1 Good preparation for DMC / tech meetings. Reports are summarized; High quality discussion not dominated by 1 person. IP's and government engaged; Concrete actions Bushenyi 12 5 Success 1 CAO is strong, LC5 attends, Secretariat is functional and well prepared, good outputs and action papers, active participation, RDC attends Dokolo 3 0 Partial Failure 0.3 CAO not involved, DCAO is more active, but no one supports her, DHO not involved, very IPs show up but resist submission of workplans Gulu 3 2 Failure 0 CAO never attended, limited to 5-7 people, low IP attendance, no functioning focal person Ibanda 10 5 Success 1 Strong CAO leadership holds participants accountable; District monitors IPs; IPs required to report or will be summoned by CAO. Meetings start on time Iganga 5 3 Partial Success 0.7 Active participations, IPs prepare reports; Strong CAO leadership; but meetings don't start on time Isingiro 7 6 Partial Success 0.7 CAO is not supportive downward decline, little district capacity, lots of participation but low IP attendance Kamuli 11 9 Success 1 DMC Secretariat functional - plan and hold meetings, do minutes and reports; Department heads actively participate; Strong CAO leadership and political leadership Kamwenge 6 3 Partial Failure 0.3 Poor but improving b/c of new CAO. Previous CAO was very week, little department head attendance, no follow-ups Kapchorwa 11 8 Partial Success 0.7 Started strong but declined due to internal conflicts. Old CAU was strong, new CAO does not get along with LC5, well prepared, never postponed Kasese 7 3 Success 1 Strong CAO, Non USAID IPs attend, use SDS templates, present evidence and data, no have field monitoring Lira 4 0 Partial Failure 0.3 Political rep is there, old CAO was not involved, Secretariat doesn't deliver, nothing gets accomplished, but good IP participation; no one is prepared Luwero 5 3 Partial Failure 0.3 CAO does not support DMC, departments don't prepare, only two district members participate, IPs don't attend, send low level reps Mayuge 8 4 Partial Failure 0.3 Leadership weak; no control over staff; Conflict affects participants; Neighbors Bugiri, so has similar IPS; Strong start Mbale 13 3 Success 1 Very strong CAO, first DMC pilot, strong IP attendance, strong outputs, very good action items Oyam 8 1 Partial Failure 0.3 CAO is strong when there, but frequently absent, political wing does attend, no action papers, no outputs, Dept. heads don't prepare, IP attendance is 50/50 Pader 4 2 Partial Failure 0.3 CAO talks well, but does not follow up or drive action; very few outputs, IPs irregularly attend, strong non-IP presence, no action items Sironko 9 1 Success 1 CAO is strong, 100% participation, all sub county chiefs and NGOs attend, strong follow up; district provides lunch Total/Average 149 75 Partial Success 0.61 Perceived Success of the DMC Meetings and District Outcomes xxxi Annex 5: DOP Interview Protocol USAID/UGANDA DISTRICT OPERATIONAL PLAN EVALUATION INTERVIEW PROTOCOL Overview Our overall goal is to develop District Operational Plan case studies (4 to 5 pages, plus annexes) for each of the 19 Mission Focus Districts. We will start building the case studies using available documents. The team will then use a variety of tools and analytical methods, detailed below, to incorporate input from district, implementing partners, and USAID/Uganda stakeholders. Each district will have a dedicated enumerator who will be responsible for ensuring the case study is completed. Each case study should capture: ● History: A chronology of DOP engagement, including: signing of the MOU, trainings, dates of each DMC meeting, and any other key events. ● Stakeholders: A list of key past and present actors from the district government, implementing partners, USAID representatives, lower level government, etc. ● Experiences: A catalog of instances with indications of collaboration, alignment, cooperation, integration. These examples should include a detailed summary of the process leading up to any action taken. ● Context: Relevant district and regional economic, social and political context. ● Way forward: Recommendations from all stakeholders on the future of the DOP Forms Internal Indices 1. Attendance analysis (excel) 2. Document inventory per district and meeting meta-analysis (excel) 3. Action paper - analysis (excel) 4. Analysis of minutes? External Data Collection tools 5. Action papers, compiled + top 20 (for district) 6. Action paper ranking - per district (for district) 7. IP intermediate results - alignment with DDPO (for district) 8. IP survey (for IPs) 9. “Modeling Success” workshop xxxii presentation and guideline 10. District Focus Group Discussion agenda 11. Courtesy Call to CAO - Talking points Logistics 12. Draft letters to District / CAO District Tool – Document Review and Analysis INTERVIEW PROTOCOL: DISTRICT Document Review and Analysis To prepare for meetings with district government stakeholders, the responsible enumerator(s) will first analyze the available documents for each district to start building the case study and to prepare for district interviews. Step 1: Document Inventory 1. Review and catalog documents using the Document Analysis form. 2. Identify gaps in documents. 3. Prepare requests for missing documents. Document Source Attendance sheets SDS, District DMC Presentations (+IP reports if easily available) District Action Items SDS, District District Development Plans District District Workplans District District Monthly Reports District SDS Trip Reports SDS Complete tracker data SDS USAID Trip Reports USAID NUFO Trip Reports USAID Step 2: Catalog of Experiences The action paper from each DMC meeting is a potential source for concrete examples of collaboration between district government, implementing partners, and other district level stakeholders. The team will undertake the following steps: xxxiii 1. Code Action Item Team members will review the action paper and code all action items in the Action Paper Analysis form according to category. The enumerator can refer to previous DMC meeting minutes for additional context and the discussions which led to a specific action item. 2. List of joint action items a. Enumerators will copy all action items for the district into one document, using the Action Item List template. These examples will later be used during the district interview process. b. The action items should then be coded using the Action Item Analysis form. Categories for analysis include: ● Solo - Action items which appear to be proposed, undertaken, and completed by the same party. ● Joint - Action items which appear to be proposed, undertaken, and completed by different parties. Examples of joint actions could include but are not limited to: ○ Joint monitoring ○ Joint workplans and budgets ○ Referrals to other service providers 3. Analyze Minutes The minutes from each DMC meeting can be a source of “experiences” which can be harvested. They may also provide context for the action items identified in the Action Papers. However, the depth, format, and thoroughness of minutes may vary from district to district and meeting to meeting. Enumerators will therefore perform two types of analysis on meeting minutes. ● Quality rating (to be determined) ● Content analysis of indications or experiences (outcome harvesting) Step 3: Attendance Analysis The DMC meeting serves as a forum for various stakeholders in the district to share information, raise and discuss issues, and decide and follow up on joint actions. Both the coordinator (SDS, NUFO) and the district government circulate registration sheets at the meetings. These attendance records will be analyzed to understand the volume and diversity of stakeholder attendance in districts, across time. 1. Code attendance records: Enumerators will use SDS or District registers to code the presence of categories of participants using the Attendance Analysis form. Categories include: a. Chief Administrative Officer b. Assistant or District Administrative Officer c. Technical Heads d. Lower Level Local Government (for example - sub county chiefs) e. District Chairperson f. Other representative from District Chairperson’s office g. USAID representative xxxiv h. USAID Implementing Partners (the unit is the organization, not the participant. For example, if three representatives from Marie Stopes are at the same meeting, we will only count this as one “unit” in attendance.) i. Non-USAID Implementing Partners j. Other Civil Society Organizations k. Other Stakeholders (specify) District Visit Guide District Visits and Interviews Step 1: Preparations While reviewing DMC meeting and other documents, the SoCha enumerators will prepare for their in￾person visits to the District Headquarters. For each district, the responsible team, with the help of the Administrative Assistant: 1. Draft and send a letter of request to the Chief Administrative Officer (adapted from a template by the Team Leader and Evaluation Expert). The letter should include: a. Introduction of the DOP Evaluation and team b. Purpose of meeting c. Proposed date of meeting d. Invited participants: This list will most likely include the District Planner and the Heads of Departments, but it may vary depending on past attendance records. e. Inquiry about availability of venue f. Brief agenda g. Summary of additional documents to be requested upon arrival 2. Arrange logistics: a. car hire b. lodgings c. facilitation materials d. USB modem, flash disk, photo device e. draft budget of anticipated expenses f. arrange with Team Leaders for cash advances as needed STEP 2: District interviews SoCha will field teams of two enumerators to each district to conduct interviews and collect additional documents. The district teams will be comprised of: 1) the enumerator responsible for the district, and 2) a junior-level enumerator who will assist with facilitation and note-taking. District teams should expect to spend two full days at each District Headquarters. On Day 1, the team will become more familiar with the District Headquarters; collect any additional documents, especially Action Papers; and complete preparations for the workshop on Day 2. On Day 2, the team will conduct a two hour focus group discussion with the DMC Secretariat and any other key district participants. In the case a focus group discussion is not possible, the team should try to meet as many of the DMC Secretariat members as possible. xxxv After the focus group discussion, enumerators can try to follow up individual participants, depending on need and availability. The following tables are general guidelines for enumerators for the two days in district. DAY 1 Introductions and data collection Activity Details Courtesy Call to the Chief Administrative Office ● Introduce team and purpose of visit ● Go over visit agenda ● Request CAO’s thoughts on DOP process Collect any available missing documents, as identified from the Document Analysis exercise. The District Planner is likely the best source for DMC meeting materials. If he or she does not have a complete set, the Heads of Departments may have many of the individual technical reports, and IP reports. ● Try to get soft copies when possible. ● If soft copies are not possible, request to photocopy / scan / take photos of the hard copy documents. Enumerators may need to use IT centers in town or at their hotels. ● Collate any additional Action Items. Select up to twenty action items to be reviewed with district officials on Day 2. Copy the action items into the Action Item Ranking Form. DAY 2 Focus group discussion Guidelines Potential participants may vary by district, but will likely include: ● DMC Chair and Secretariat ● Technical heads and district planner (if any members are not part of the Secretariat) ● District Chairperson (political) ● Optional: ○ Lower Level Government (sub-county chief) ○ Implementing Partners Suggested agenda for the focus group discussion: 1. Introduction 2. Goals: a. Harvest outcomes b. Catalog as tangibly as possible all outcomes that can be associated with DOP xxxvi process 3. Part 1 - Structured: Enumerators present the Action Ranking Paper, with no more than 20 action items listed. The group will engage in: a. Action Item Ranking: Review action items, and reach a group consensus on the top three and bottom three which have (or have not) contributed to: i. Achieving the district development plan ii. Sustaining relationship with Implementing Partners b. Outcome harvesting: After ranking the Top and Bottom 3, the group will engage in a collaborative outcome harvesting exercise. Enumerators will facilitate a discussion to understand each action, and answer the following questions: i. Who were the main actors? ii. What where the issues at stake iii. When did key actions take place? iv. Where v. How vi. Conclusion 4. Part 2: Semi-structured: The enumerators will facilitate semi-structured discussions on the following: a. Other examples: Participants can contribute additional examples of key actions which may not have been captured in the Action Item analysis. b. Results Framework / current indicator reporting: i. How has the DOP process has or has not contributed to the result framework? ii. Are the indicators aligned with the District Development Plan? (Which ones are aligned and which ones are not applicable?) c. Examples and assessment of the effectiveness of other components of the DOP, including: i. Training ii. Other example of alignment iii. Changes in district relationship with USAID / IPs iv. Removing duplication d. Suggestions for improving USAID and IP cooperation, collaboration, alignment and integration with District priorities. e. Sustainability i. Will you (district) continue this process without funding? If yes, which components? ii. Do you view this as a pilot and is there any potential if evidence is good you could re-structure the way you manage this? (this question may be more appropriate for Ministry of Local Government) Implementing Partner Tool INTERVIEW PROTOCOL: IMPLEMENTING PARTNERS xxxvii Document Review Step 1: Document collection USAID/Uganda has collected feedback from Implementing Partner representatives on the DOP process on several occasions. The resulting reports are a good source of IP perspectives on the DOP process. Thus far, the evaluation team has collected the following such reports: ● Report from the DOP Learning Event - December 2013 ● “Analysis of Data from District Operational Plan Survey” (MMC) - October 2013 ● Chief of Party Feedback Meeting notes - February 2015 In addition, USAID often collects other documentation as part of the IP compliance processes. The evaluation team will seek to identify if there are other relevant documents which should be considered within the scope of this evaluation. For example, the evaluation team learned from an interview during Phase 1 that in some districts, a USAID staff member emailed IPs who did not attend a DMC meeting for an explanation of absence. Step 2: Analysis Document collection will be an ongoing process, as the team may learn of additional documents as we continue to meet with key stakeholders. However, the team will simultaneously begin an initial review of these documents in order to develop and apply the best tools for analysis. Implementing Partner Survey Survey In addition to mining already available documentation, the evaluation team will develop a questionnaire to better understand 1) the variations in implementing partners’ experiences with the DOP process, and 2) what value the DOP process adds for USAID implementing partners as compared to districts where there is no DOP. Step 1: Develop survey tool Within these broad concepts, the evaluation team will specifically seek to understand implementing partners’ experiences using a survey tool, to be implemented via email. The survey will cover the following: 1. The “Continuum of Flexibility” One of the key stated objectives of the DOP process is better implementing partner coordination with each other, local government, and other implementers at the district level. The ability of implementing partners to fulfill this objective requires that they are able to adapt to shifting priorities at the district level. However, due to constraints such as organizational mandates and defined scopes of work, implementing partners may not always have the authority or the desire to accommodate all such requests. The evaluation team will include questions to solicit IPs experiences within a “continuum of flexibility,” to understand instances in which implementing partners have or have not accommodated requests for activity changes, and why. xxxviii 2. Targeting investments Better targeting of investments, in alignment with district government priorities, is another key stated objective of the DOP process. The evaluation team seeks to understand how implementing partners target their activities in a given district. The survey will include a question to the implementing partners to assess whether their activities are aligned with district priorities as a result of: a. Coincidence b. Deliberate effort - pre planning (blank slate) c. Adjustment d. Unable to adjust e. No alignment 3. Outlier examples: An open space in which Implementing Partners can contribute other examples from their experiences which point to coordination or lack of coordination with district government and other stakeholders. Step 2: Implement survey The anticipated respondents will be USAID implementing partners, in the following categories: DOP districts Non-DOP districts (for comparative purposes) IPs who are highly engaged (for example, high attendance at meetings, timely submission of district reports) IPs who also work in DOP districts IPs who have not been engaged (for example, consistently miss DMC meetings) IPs who have no operations in DOP districts Step 3: Key Informant Interviews Based on the document review and survey data, the evaluation team may select a few Implementing Partners for key informant interviews. T USAID Representative Tool INTERVIEW PROTOCOL: USAID DISTRICT OPERATIONAL PLAN REPRESENTATIVES “Modeling Success” Mini-Workshops The evaluation team will conduct “Modeling Success” mini-workshops with USAID District Operational Plan representatives. This tool was developed in Phase 1, and used to generate discussion and feedback with a group of SDS district cluster leads. The goal of the “modeling success” mini-workshop is to 1) identify definitions of DOP success according to USAID representatives’ perspectives; xxxix 2) understand USAID representatives’ activities and level of engagement with the DOP process, and 3) develop a timeline, per DOP district, of key moments which indicate success as mapped to intermediate results listed in the DOP causal pathway A general outline of the workshop follows. The presentation, with detailed guidelines, is also attached. Modeling Success Mini Workshop Guidelines Materials: Flip charts, projectors, markers, tape, post-its in different colors Time: 3 hours Agenda 1. Introduction 2. Learn our modeling approach 3. Exercise 1: Establish broad definitions of DOP success 4. Exercise 2: Identify what exactly has been done 5. Exercise 3: Identify and timeline indications of intermediate results Introduction: Our approach: modeling DOPS 1. Expert vs Model 2. Model all districts vs sampling 3. Define the model collaboratively – model your activities, model definition of success a. Use a truth table b. Explain truth table concepts: i. Variation ii. Membership iii. Introduce concept: Indications vs indicators Exercise 1: Definitions of success 1. List all DOP districts we’ll discuss 2. Rank based on scale 0, 0.3, 0.5, 1 3. List all definitions of success Exercise 2: What have you done? (Model USAID Representatives) 1. List of activities (flip chart) 2. Categorize activities per DOP (projector) 3. List all DOPs in first column Exercise 3: Identify and timeline indications of intermediate results 1. Introduce the results (Green boxes in causal pathway - 6 terms) - we prepare the flip chart in advance with headings – success / unsuccess indications 2. Go around the room - solicit as many examples as possible 3. Look for duplicates - eliminate duplicates 4. Each participant uses color coded post-it to indicate whether activity took place in their xl district 5. Discuss color coding visual 6. Participants take post-its to place on a timeline for their district FLIP CHARTS to prepare: 1. Exercise 1: List of all DOPS + Rank success 2. Exercise 1: List all definitions of success 3. Exercise 2: List of activities 4. Exercise 2: Organize activities into types 5. Exercise 3: Box 1: District led coordination 6. Exercise 3: Box 2: Collaboration between different USAID DOs 7. Exercise 3: Box 3: Integration of USAID 8. Exercise 3: Box 4: Local government ownership 9. Exercise 3 Box 5: Mutual accountability and transparency 10. Exercise 3: Box 6: Availability of quality services improved 11. Exercise 3: Timelines: 1 for each district. TABLES to prepare: 1. List of districts and DOPs 2. Categories of activities Outcome Harvesting Templates The evaluation team used four different template forms connected to outcome harvesting. These are: • Interview Checklist (used in preparation for the interviews) • Outcome Reporting Template (used to capture Outcomes) • Field Outcome Substantiation Form (used to verify outcomes with other parties while in the field) • Home Office Substantiation Form (used to verify outcomes with other parties by the Kampala team) These are attached below with the spaces between entries removed. A. Interview Checklist Interview Checklist Done Introduce selves, methodology, and objective of interviews Obtain permission to record interviews and/or take photos Document request (DDP and any records kept) Open-ended Question(s): What was the most significant change? Go through a form for each outcome - Description (What / Who / When / Where) o Describe the change. What specifically was done differently? o Describe the social actor who made the change. Who adjusted their way of doing things? o Describe when the change happened. o Describe where the change took place - Contribution (What / How / When) xli o What specific aspect/piece of the DMC meeting/DOP process and activities of social actors outside of the DMC meeting contributed to the changes? o How did the DMC meeting/DOP process contribute to the change? Describe the chain of events. o For each item/activity in the chain of events, list who did the action and when. - Significance (Why) – Why is the outcome important to your respondent? o Given the aim of the DOP process - strengthening the relationship between the district and USAID IPs, why is this outcome significant? Be specific. o Unpack any buzzwords (e.g. Engagement / Alignment / Change / Transparency / Accountability) QUESTIONS TO ASK IF TIME PERMITS Causal Pathway Questions - Can you point to any evidence that district officials have increased their engagement with USAID activities outside of the DMC meeting - How have USAID activities aligned with district plans and priorities as a result of the DMC? - How have relationships between USAID and the districts strengthened? - How has the DOP changed the way your LG plans and provides services? - Has there been district led coordination of other donor activities? - Are there any observable changes about how different USAID district office activities collaborate within your district? - Has there been integration of USAID activities in district plans? - Have you been involved in any IP projects as a result of the DOP? - In your view, does the presence of a USAID representative make a difference or not? - Has there been any change in how you are investing resources? Ask about action items [time permitting] Exit question: Are there any challenges you’d like to highlight or recommendations you’d like to make? [time permitting] xlii Annex 6: Outcome Harvesting Database Please see Excel Sheet embedded. Annex 6 - Master Outcome Spreadshe xliii Annex 7: Outcome Harvesting Reference Sheets DISTRICT: [district where interview conducted] DATE: [interview date] / / RESPONDENT(S): [Give Names, Titles] OUTCOME SERIAL NO.: [Give unique number.] INVOLVED/KNOWLEDGEABLE: TIME: [If recording time] Outcome Details Recording Sheet DISTRICT: [district where interview conducted] DATE: [interview date] / / RESPONDENT(S): [Give Names, Titles] OUTCOME SERIAL NO.: [Give unique number.] INVOLVED/KNOWLEDGEABLE: TIME: [If recording time] A. Outcome Description WHAT: Describe the change. What specifically was done differently? WHO: Describe the social actor who made the change. Who adjusted their way of doing things? WHEN: Describe when the change happened. When did the actor adjust their way of doing things? WHERE: Describe where the change took place. Where was it that the actor adjusted their way of doing things? Date Outcome Occurred: / / B. Significance WHY: Why is the outcome important to your respondent? Given the aim of the DOP process - strengthening the relationship between the district and USAID IPs, why is this outcome significant? Be specific. C. Contribution WHAT: What specific aspect/piece of the DMC meeting/DOP process and activities of social actors outside of the DMC meeting contributed to the changes? HOW: How did the DMC meeting/DOP process contribute to the change? Describe the chain of events. WHEN: For each item/activity in the chain of events, list who did the action and when. A. Field Outcome Substantiation Form DISTRICT: [Note what district this pertains to.] WHO: [Note who the party is that you are speaking with.] Outcome Serial No. Outcome Significance Contribution Agreement Comments Agreement Comments Agreement Comments xliv B. Home Office Substantiation Form Outcome Serial No. Outcome Significance Contribution Person(s) to Be Contacted and Contacts Please put your comments in capital letters. xlv Annex 8: Action Item Database Please see Excel Sheets embedded. Action Points Kamuli RKS.xlsx Action Points Kapchorwa RKS.xlsx Action Points Mayuge RKS.xlsx Action Item Summary ALL.xlsx Action Items - IGANGA RKS.xlsx Action Items - KASESE RKS.xlsx Action Items - MBALE RKS.xlsx Action Items Amuru RKS.xlsx Action Items Bushenyi RKS.xlsx Action Items Dokolo RKS.xlsx Action Items Gulu - 12-17-15 EFY RKS.xlsx Action Items Ibanda RKS.xlsx Action Items Isingiro 12-17-15 EFY Action Items Kamwenge RKS.xlsx Action Items Lira RKS.xlsx Action Items Luwero RKS.xlsx Action Items Oyam - 12-17-15 EFY RKS.xlsx Action Items Pader - 12-17-15 EFY RKS.xlsx Action Items Sironko RKS.xlsx Action Points Bugiri RKS.xlsx xlvi Annex 9: DOP Dossiers and Field Notes Please see MS Word File embedded. Annex 9 - DOP DOSSIERS.docx xlvii Annex 10: QCA Factors and FSQCA Outputs Listed below are the output files associated with running the QCA models. QCA best practice requires that these outputs be attached when reporting results. We used fsQCA 2.5 software to run the model. To reproduce the results: Instructions for running the model 1. Open fsQCA 2. Under File, Open Data - open the "doptt.csv" file 3. Under Analyze, chose "Crisp Truth Table Algorithm" ( 4. Under select variables, set “outcome” to Outcome. 5. Then add the following variable to analyze: act.it, per.suc, ip.attend, quest.cost, sect.per, ip.workplan, core.spend, sds.spend, usaid.attend. It is good to click show solution cases in output. 6. Hit Run 7. Review the truth table 8. Go to Edit - select "Delete and Code", using the default values (1 and .8). 9. Hit Standard Analyses 10. For the first round of PI Chart, hit "Mark All" then "OK" 11. Do this for the second and third as well. 12. You should obtain the same results below. We’ve only included the parsimonious solution below. FSQCA Outputs Crisp Set Analysis ********************** *TRUTH TABLE ANALYSIS* ********************** File: C:/Users/PMP8EZ/Documents/DOPS/QCA/doptt.csv Model: outcome = f(act.it, per.suc, ip.attend, quest.cost, sect.per, ip.workplan, core.spend, sds.spend, usaid.attend) Rows: 17 Algorithm: Quine-McCluskey True: 1 --- COMPLEX SOLUTION --- frequency cutoff: 1.000000 consistency cutoff: 1.000000 raw unique coverage coverage consistency ---------- ---------- ---------- act.it*per.suc*ip.attend*~quest.cost*~ip.workplan*core.spend*sds.spend*~usaid.attend 0.166667 0.166667 1.000000 act.it*per.suc*ip.attend*sect.per*~ip.workplan*~core.spend*sds.spend*usaid.attend 0.166667 0.166667 1.000000 act.it*per.suc*ip.attend*quest.cost*~sect.per*core.spend*sds.spend*usaid.attend 0.166667 0.166667 1.000000 act.it*~per.suc*~ip.attend*~quest.cost*sect.per*~ip.workplan*core.spend*sds.spend*~usaid.attend 0.083333 0.083333 1.000000 xlviii act.it*per.suc*ip.attend*quest.cost*~sect.per*~ip.workplan*~core.spend*sds.spend*~usaid.attend 0.083333 0.083333 1.000000 act.it*per.suc*~ip.attend*quest.cost*~sect.per*ip.workplan*~core.spend*sds.spend*~usaid.attend 0.083333 0.083333 1.000000 act.it*~per.suc*~ip.attend*quest.cost*~sect.per*~ip.workplan*core.spend*sds.spend*usaid.attend 0.083333 0.083333 1.000000 act.it*per.suc*~ip.attend*~quest.cost*sect.per*ip.workplan*core.spend*sds.spend*~usaid.attend 0.083333 0.083333 1.000000 solution coverage: 0.916667 solution consistency: 1.000000 Cases with greater than 0.5 membership in term act.it*per.suc*ip.attend*~quest.cost*~ip.workplan*core.spend*sds.spend*~usaid.attend: Ibanda (1,1), Kasese (1,1) Cases with greater than 0.5 membership in term act.it*per.suc*ip.attend*sect.per*~ip.workplan*~core.spend*sds.spend*usaid.attend: Kamuli (1,1), Sironko (1,1) Cases with greater than 0.5 membership in term act.it*per.suc*ip.attend*quest.cost*~sect.per*core.spend*sds.spend*usaid.attend: Bushenyi (1,1), Kapchorwa (1,1) Cases with greater than 0.5 membership in term act.it*~per.suc*~ip.attend*~quest.cost*sect.per*~ip.workplan*core.spend*sds.spend*~usaid.attend: Mayuge (1,1) Cases with greater than 0.5 membership in term act.it*per.suc*ip.attend*quest.cost*~sect.per*~ip.workplan*~core.spend*sds.spend*~usaid.attend: Mbale (1,1) Cases with greater than 0.5 membership in term act.it*per.suc*~ip.attend*quest.cost*~sect.per*ip.workplan*~core.spend*sds.spend*~usaid.attend: Bugiri (1,1) Cases with greater than 0.5 membership in term act.it*~per.suc*~ip.attend*quest.cost*~sect.per*~ip.workplan*core.spend*sds.spend*usaid.attend: Oyam (1,1) Cases with greater than 0.5 membership in term act.it*per.suc*~ip.attend*~quest.cost*sect.per*ip.workplan*core.spend*sds.spend*~usaid.attend: Isingiro (1,1) ********************** *TRUTH TABLE ANALYSIS* ********************** File: C:/Users/PMP8EZ/Documents/DOPS/QCA/doptt.csv Model: outcome = f(act.it, per.suc, ip.attend, quest.cost, sect.per, ip.workplan, core.spend, sds.spend, usaid.attend) Rows: 17 Algorithm: Quine-McCluskey True: 1-L --- PARSIMONIOUS SOLUTION --- frequency cutoff: 1.000000 consistency cutoff: 1.000000 xlix raw unique coverage coverage consistency ---------- ---------- ---------- act.it*sds.spend 0.916667 0.916667 1.000000 solution coverage: 0.916667 solution consistency: 1.000000 Cases with greater than 0.5 membership in term act.it*sds.spend: Bugiri (1,1), Bushenyi (1,1), Ibanda (1,1), Isingiro (1,1), Kamuli (1,1), Kapchorwa (1,1), Kasese (1,1), Mayuge (1,1), Mbale (1,1), Oyam (1,1), Sironko (1,1) ********************** *TRUTH TABLE ANALYSIS* ********************** File: C:/Users/PMP8EZ/Documents/DOPS/QCA/doptt.csv Model: outcome = f(usaid.attend, sds.spend, core.spend, ip.workplan, sect.per, quest.cost, ip.attend, per.suc, act.it) Rows: 11 Algorithm: Quine-McCluskey True: 1 0 Matrix: 0L Don't Care: - --- INTERMEDIATE SOLUTION --- frequency cutoff: 1.000000 consistency cutoff: 1.000000 Assumptions: raw unique coverage coverage consistency ---------- ---------- ---------- ~usaid.attend*sds.spend*core.spend*~ip.workplan*~quest.cost*ip.attend*per.suc*act.it 0.166667 0.166667 1.000000 usaid.attend*sds.spend*~core.spend*~ip.workplan*sect.per*ip.attend*per.suc*act.it 0.166667 0.166667 1.000000 usaid.attend*sds.spend*core.spend*~sect.per*quest.cost*ip.attend*per.suc*act.it 0.166667 0.166667 1.000000 ~usaid.attend*sds.spend*core.spend*~ip.workplan*sect.per*~quest.cost*~ip.attend*~per.suc*act.it 0.083333 0.083333 1.000000 usaid.attend*sds.spend*core.spend*~ip.workplan*~sect.per*quest.cost*~ip.attend*~per.suc*act.it 0.083333 0.083333 1.000000 ~usaid.attend*sds.spend*~core.spend*ip.workplan*~sect.per*quest.cost*~ip.attend*per.suc*act.it 0.083333 0.083333 1.000000 l ~usaid.attend*sds.spend*~core.spend*~ip.workplan*~sect.per*quest.cost*ip.attend*per.suc*act.it 0.083333 0.083333 1.000000 ~usaid.attend*sds.spend*core.spend*ip.workplan*sect.per*~quest.cost*~ip.attend*per.suc*act.it 0.083333 0.083333 1.000000 solution coverage: 0.916667 solution consistency: 1.000000 Cases with greater than 0.5 membership in term ~usaid.attend*sds.spend*core.spend*~ip.workplan*~quest.cost*ip.attend*per.suc*act.it: Ibanda (1,1), Kasese (1,1) Cases with greater than 0.5 membership in term usaid.attend*sds.spend*~core.spend*~ip.workplan*sect.per*ip.attend*per.suc*act.it: Kamuli (1,1), Sironko (1,1) Cases with greater than 0.5 membership in term usaid.attend*sds.spend*core.spend*~sect.per*quest.cost*ip.attend*per.suc*act.it: Bushenyi (1,1), Kapchorwa (1,1) Cases with greater than 0.5 membership in term ~usaid.attend*sds.spend*core.spend*~ip.workplan*sect.per*~quest.cost*~ip.attend*~per.suc*act.it: Mayuge (1,1) Cases with greater than 0.5 membership in term usaid.attend*sds.spend*core.spend*~ip.workplan*~sect.per*quest.cost*~ip.attend*~per.suc*act.it: Oyam (1,1) Cases with greater than 0.5 membership in term ~usaid.attend*sds.spend*~core.spend*ip.workplan*~sect.per*quest.cost*~ip.attend*per.suc*act.it: Bugiri (1,1) Cases with greater than 0.5 membership in term ~usaid.attend*sds.spend*~core.spend*~ip.workplan*~sect.per*quest.cost*ip.attend*per.suc*act.it: Mbale (1,1) Cases with greater than 0.5 membership in term ~usaid.attend*sds.spend*core.spend*ip.workplan*sect.per*~quest.cost*~ip.attend*per.suc*act.it: Isingiro (1,1) Analysis of Necessary Conditions Outcome variable: outcome Conditions tested: Consistency Coverage act.it+sds.spend 1.000000 0.666667 act.it 0.916667 0.846154 sds.spend 1.000000 0.750000 Fuzzy Set Analysis ********************** *TRUTH TABLE ANALYSIS* ********************** File: C:/Users/PMP8EZ/Documents/DOPS/QCA/fsqca.csv Model: outcome.fz = f(act.sub.fz, sds.burn.fz) Rows: 4 Algorithm: Quine-McCluskey True: 1 --- COMPLEX SOLUTION --- li frequency cutoff: 1.000000 consistency cutoff: 0.824713 raw unique coverage coverage consistency ---------- ---------- ---------- act.sub.fz 0.928924 0.928924 0.852355 solution coverage: 0.928924 solution consistency: 0.852355 Cases with greater than 0.5 membership in term act.sub.fz: Bugiri (0.95,0.95), Bushenyi (0.94,0.84), Kapchorwa (0.92,0.79), Kamwenge (0.84,0.4), Ibanda (0.83,0.74), Mbale (0.8,0.88), Sironko (0.79,0.67), Mayuge (0.75,0.59), Kasese (0.71,0.51), Oyam (0.7,0.59), Kamuli (0.6,0.79), Isingiro (0.56,0.51), Iganga (0.52,0.3) ********************** *TRUTH TABLE ANALYSIS* ********************** File: C:/Users/PMP8EZ/Documents/DOPS/QCA/fsqca.csv Model: outcome.fz = f(act.sub.fz, sds.burn.fz) Rows: 4 Algorithm: Quine-McCluskey True: 1-L --- PARSIMONIOUS SOLUTION --- frequency cutoff: 1.000000 consistency cutoff: 0.824713 raw unique coverage coverage consistency ---------- ---------- ---------- act.sub.fz 0.928924 0.928924 0.852355 solution coverage: 0.928924 solution consistency: 0.852355 Cases with greater than 0.5 membership in term act.sub.fz: Bugiri (0.95,0.95), Bushenyi (0.94,0.84), Kapchorwa (0.92,0.79), Kamwenge (0.84,0.4), Ibanda (0.83,0.74), Mbale (0.8,0.88), Sironko (0.79,0.67), Mayuge (0.75,0.59), Kasese (0.71,0.51), Oyam (0.7,0.59), Kamuli (0.6,0.79), Isingiro (0.56,0.51), Iganga (0.52,0.3) ********************** *TRUTH TABLE ANALYSIS* ********************** lii File: C:/Users/PMP8EZ/Documents/DOPS/QCA/fsqca.csv Model: outcome.fz = f(sds.burn.fz, act.sub.fz) Rows: 2 Algorithm: Quine-McCluskey True: 1 0 Matrix: 0L Don't Care: - --- INTERMEDIATE SOLUTION --- frequency cutoff: 1.000000 consistency cutoff: 0.824713 Assumptions: raw unique coverage coverage consistency ---------- ---------- ---------- act.sub.fz 0.928924 0.928924 0.852355 solution coverage: 0.928924 solution consistency: 0.852355 Cases with greater than 0.5 membership in term act.sub.fz: Bugiri (0.95,0.95), Bushenyi (0.94,0.84), Kapchorwa (0.92,0.79), Kamwenge (0.84,0.4), Ibanda (0.83,0.74), Mbale (0.8,0.88), Sironko (0.79,0.67), Mayuge (0.75,0.59), Kasese (0.71,0.51), Oyam (0.7,0.59), Kamuli (0.6,0.79), Isingiro (0.56,0.51), Iganga (0.52,0.3) Outcome variable: outcome.fz Conditions tested: Consistency Coverage act.sub.fz+sds.burn.fz1.000000 0.638714 sds.burn.fz 0.953603 0.655360 act.sub.fz 0.928924 0.852355 liii Annex 11: DOP Truth Tables Please see Excel Sheet embedded. Annex 11 - QCA Truth Table and Fact liv Annex 12: IP Survey Implementing Partner DOP Experience Questionnaire Dear Valued Implementing Partner, SoCha, in cooperation with QED’s Learning Contract, is conducting an evaluation of the quarterly District Management Committee (DMC) meeting and associated District Operational Plan (DOP) process across the nineteen USAID Mission Focus Districts (MFDs). As part of this exercise, we would like to capture the actual experiences of implementing partners (IPs) who have participated (or have not participated but were requested to do so) in the DMC meetings. Your activity has been included on a master list of IPs who, at one point or another, were requested to attend the DMC meetings. Please note: USAID has already obtained valuable feedback on IP perceptions of the DOP through a survey in 2013, as well as on challenges facing the DOP through a Chief of Party (CoP) roundtable held in early February of this year. Both exercises were focused on dynamics and relationships inside and in preparation for the DMC meetings. By contrast, our evaluation is trying to understand: • Potential/perceived risks and benefits of greater integration of IP activities into district level budget and planning processes • If, how and to what extent the issues, action items and interactions raised within the DMC meetings have contributed to outcomes (both positive and negative) outside of the DMC meetings. • Estimate costs incurred by IPs to attend the DMC meetings. Below you will find six (6) substantive question areas we’d like to ask you and/or your team to address, plus additional questions regarding costs associated with attending the DMC. Our focus here is on actual experiences that have occurred, and we request that you be as specific as possible when sharing these experiences (e.g. identify the who, what, when, where and how of the experience). NOTE: If no one from your activity has attended a DMC meeting, but have been requested to do so (by the district, the Strengthening Decentralization Systems (SDS) activity and/or a USAID representative), do not fill out the questionnaire. Instead, you are kindly requested to please elaborate on why you/your team made this decision (please note: it is extremely important we understand why some IPs are not attending the DMC meetings, so please share). Your responses will be kept strictly confidential and we will not reveal any details that can be used to identify your activity without your explicit authorization. We are primarily concerned with overall DOP trends and not concerned with the performance of individual activities. As a matter of protocol, we are addressing this questionnaire to the activity Chief of Party. However, we understand that you may not have attended the DMC and instead delegated this task to members of your team. Please forward this questionnaire to them. Please review and complete the questionnaire by Wednesday, December 9, 2015. Please send to Carroll Patterson at cpatterson@socha.net. If you have any questions about this exercise, require further clarification and/or feel an additional discussion would be useful, please do not hesitate to contact either Carroll Patterson at the above email or 0783 636 905, or Luke Gormley at lgormley@socha.net or 0752 537 280 Your feedback is greatly appreciated. Thank you. Corey Patterson, PhD Evaluation Team Leader, DOP Evaluation lv Questions about Outcomes that have occurred outside a DMC meeting as a direct result of attendance at a DMC meeting (use as much space as you need) 1. Please share specific changes (up to 3) to your activity’s implementation (e.g. changes to your work plan, changes to your budget, changes in where you implement, etc.) that have occurred as a result of your attendance at the DMC meetings 2. Has your attendance at DMC meetings led to any subsequent outcomes that have improved your project’s ability to implement (e.g. the district has removed some kind of impediment, the community better supports your activity, etc.) 3. Please describe any forms of ongoing collaboration/cooperation (up to 3) with other IPs, district representatives and/or third party stakeholders that emerged as a direct result of your attendance at the DMC meeting (excluding those that occur within the DMC meeting). 4. Please describe any negative experiences (up to 3) you have had in connection with your attendance at the DMC meeting 5. Please describe any other experiences you may have had that occurred outside of the DMC meeting but were a direct result of attending that meeting. Question about your views on deeper IP integration into district work and budget planning 1. The DOP process ultimately seeks to encourage IPs to integrate their work plans and budgets into district-level processes, such as the district development plan and annual budget conference. What do you see are the potential benefits and risks of further integration into these and other processes? Please provide specific examples if you have any Level of Effort and Cost Estimates Please provide us with estimates of the following figures: a. In how many districts do you and/or your staff members attend DMC meetings every quarter? _______ b. On Average, how many of the four quarterly DMC meetings do you/your team attend annually (1, 2, 3 or 4). __________________ c. Average number of staff that attend each DMC meeting ______ d. Average preparation and round trip travel time dedicated to each DMC meeting _______(days) e. Average Transport Costs associated with attending each meeting __________(UGS) f. Average Per Diem and Lodging Costs associated with attending each meeting __________(UGS) g. Any other costs associated with attending a DMC Meeting ______(UGS) Please explain __________ lvi Annex 13: Cost Model and Instructions Please find the cost model attached in the corresponding Excel Sheet. Costing Model Manual Introduction USAID/Uganda commissioned SoCha, LLC to conduct an evaluation of the District Operational Plan (DOP) process, which is currently being implemented by Strengthening Decentralization for Sustainability (SDS). The DOP entails a quarterly meeting with district technical officials, a USAID representative and USAID-supported implementing partners. The DOP has unfolded across 19 districts in Uganda over the past two years. As part of the evaluation, SoCha committed to developing a cost model to project cost estimates of running the DOP meeting in other districts in the country. The main purpose of the cost model is to establish the unit cost of delivering the quarterly meeting using historical spending data, as well as identify projected costs for all Ugandan districts, based upon varying transport, Per Diem and lodging costs. This document describes the cost model. It is a user manual, not a report on the costing. Model Overview and Basics Basic, but important, rules The formatting of the model is simple. The following shades are used to guide the user: The rest of the worksheets are grey or white. It is important that users only change information in the bright blue or lighter blue cells, as indicated above. Note that some cells will change colour when a drop down option is changed. The above rules will continue to apply. Overview of the model The figure below shows the worksheets in the workbook that are important to the user. Once the model has been populated with cost estimates in the Assumptions and Districts worksheets, the user should only need to visit the Scenarios & Results worksheet. Yellow cells contain formulae and should not be touched Bright blue cells contain drop down menus, users must choose one of the options on the drop down Ligher blue cells are where numbers can be entered in Scenarios & Results Enter number of meeting attendees per scenario Select regions and districts you want to see results for Select Currency Districts Enter details about districts: - per diem paid in district (only if it varies by district) - number of people attending training (only if it varies by district) - venue hire (Yes/No - default is No) Assumptions Overheads Oversight Preparation Training MOU Finalistion Input Prices Fuel per diem car hire venue catering exchange rate Maths Results by District Results per scenario Totals per region and district per meeting and per year. Also shows set up, training and management costs. (Information only) The worksheets are described below. The Scenarios & Results Worksheet The results of the costing are summarised on the left side of this worksheet, as shown below: Regions and Districts can be selected using the drop down menus on the left. On the right hand side of this worksheet is the following table: The cost of different scenarios can be tested here, as this table allows the user to compare the cost of different attendance levels at the meetings. Certain assumptions around travel and Per Diem are set in the assumptions worksheet, discussed below. The user can choose which currency (USD or UGX) the results should be in. The exchange rate is set in the Assumptions worksheet. As mentioned above, if the assumptions are correct, this is the only worksheet the user needs to use in the model. The Results by District Worksheet Meeting Costs Scenario One Scenario Two Scenario Three Set up Costs Select Regions to see costs by Region Central 89 124 100 493 111 861 18 280 Eastern 126 017 143 543 161 069 28 971 Northern 128 818 148 808 168 797 34 279 Western 106 570 122 203 137 836 26 326 Total costs of selected regions 450 529 515 046 579 564 107 857 Meeting Costs Select Districts to see costs by District Scenario One Scenario Two Scenario Three Set up Costs Buikwe 3 667 4 124 4 582 725 Gomba 3 705 4 174 4 644 750 Luwero 3 668 4 126 4 583 725 Masaka 3 764 4 254 4 744 790 Mityana 3 679 4 140 4 601 732 Mpigi 3 703 4 172 4 641 748 Mubende 3 800 4 301 4 803 813 Mukono 3 627 4 071 4 515 698 Nakaseke 3 777 4 272 4 766 798 Nakasongola 3 736 4 217 4 697 771 Rakai 3 863 4 386 4 909 856 Sembabule 3 822 4 331 4 840 828 Kampala 3 609 4 047 4 485 686 Total costs of selected districts 48 420 54 616 60 811 9 920 Number of People Attending Meetings Scenario One Two Three SDS 1 1 1 Implementing Partners 3 4 5 USAID 4 6 8 Technical District Staff 2 3 5 10 14 19 Choose currency: USD This worksheet is for information purposes only. The user cannot change anything in it. The figure below is a snapshot of the information provided. The results of all districts are shown. They are also summarised by region, as can be seen below. Note: the costs per meeting shown in the Scenarios & Results worksheet include the preparation and Management & Admin costs. The one-off costs are incurred when a district joins the DOP process. The Districts Worksheet The Districts worksheet allows the user to change cost assumptions by district. The figure below is a snapshot of the information sheet. All the districts are listed. The options in this worksheet are as follows: • Per Diem in District: o In the District worksheet, the user can select whether the Per Diem in a district is “High”, “Medium” or “Low”. The values of the three levels is set in Assumptions. o Also in Assumptions is the option to set the Per Diem to be the same in all districts, and that value is also set in Assumptions. If this option is chosen, the heading “Per Diem in District” is changed to “Per Diem is fixed, Ignore”. • Km Travelled (one way): this is the number of kilometres from Kampala to the town in the district where the meetings take place. The correct values have already been entered for all districts. • Extra Travel Costs: In some districts there is a fee for a ferry ride per car, and/or for each passenger. These columns capture these costs. The correct values have been entered for all districts. • Does a venue need to be hired for meetings in this district? In the event that a meeting needs to be held at a venue outside of the district offices, the model allows the user to indicate this. The default Amounts are in USD Costs per meeting (excluding preparation and overheads) Cost per meeting Once Off Costs One Two Three Preparation Costs Management & Admin Set up Costs Training Region Central 30 184 41 553 52 922 51 740 7 200 18 280 8 276 Eastern 45 131 62 657 80 183 71 286 9 600 28 971 13 174 Northern 49 429 69 418 89 408 70 389 9 000 34 279 15 909 Western 39 456 55 090 70 723 59 313 7 800 26 326 12 097 District Name One Two Three Preparation Costs Management & Admin Set up Costs Training Buikwe 1 230 1 687 2 144 2 137 300 725 321 Bukomansimbi 1 332 1 840 2 349 2 188 300 827 372 Butambala 1 235 1 696 2 156 2 140 300 730 324 Buvuma 1 239 1 701 2 164 2 142 300 734 326 Extra Travel Costs District Name Per Diem in District Km travelled (one way) Per Car Per Person Does a venue need to be hired for meetings in this district? How many staff attend once-off leadership training? Buikwe High 63 No Bukomansimbi High 163 No Butambala High 72 No Buvuma High 78 No for all districts is “No”, as it is assumed there are meeting venues at all districts and therefore no need to hire a venue. • How many staff attend once-off leadership training? Users can enter the number of people from each district who attend the once-off leadership training in this column. There is an option in Assumptions to change this to be the same for all districts. If that option is chosen, then this heading changes to “Number of people attending training fixed, ignore”. The Assumptions Worksheet The Assumptions worksheet includes two main sections: 1. Administrative and Set Up Costs and Assumptions (on the left) 2. Input Prices and Operational Assumptions (on the right) Note that all prices are in USD initially. There is the option in the Results & Scenarios worksheet to show the results, and the prices, in UGX. Administrative and Set Up Costs and Assumptions Four separate activities are costed here: • Management and administration support • Preparation for quarterly meetings • Leadership training • MoU Finalisation The first two activities are costs incurred per meeting. The last two are set up costs incurred once per district. Administration and Overhead Costs At the top of these assumptions are the administration costs. These are per meeting costs. These cost estimates are simple, and capture the amount of time managers spend to oversee the preparation for, and implementation of, meetings. Provision is also made for the time of administrative assistants who assist with processing payments, organising transport and vouchers and so on. Note that these are days per meeting, and the user can select less than a full day if need be (e.g. 0.5 or 0.2). Preparation per quarterly meeting The next level on this side of the worksheet is the costing of the preparatory meetings: It is assumed that the officials travelling to the meetings hire a car, and that the cost of travel (discussed below) applies. SDS officials will receive Per Diems if this is selected in the “Per Diems and Travel” section discussed below. Leadership training Administrative/Overhead Costs - Head Office Number of days per quarterly meeting: No. Days USD/Day) Management time (overseeing SDS activities) 1 150 Administrative Assistant (payments, bookings etc) 2 75 Preparation per quarterly meeting Days spent by SDS Cluster lead per meeting 15 Preparatory Trips per meeting* 1 Number of Staff travelling to preparatory meetings 2 SDS Cluster Daily Fee (USD) 120 It is assumed one car is hired per prep trip; distance travelled = km on district sheet The model provides for the costing of providing leadership training to officials in districts. As for preparatory meetings, it is assumed the course trainer hires a car, and the cost of car hire and fuel consumption are relevant here too. It is important to note that the user can select between a fixed number of officials per district or entering in the number attending per district. In the picture below, the option to enter the number of officials attending training in the districts sheet has been chosen, While in the picture below, by contrast, the option to have a fixed number (in this case, eight) of officials attend has been chosen. MoU finalisation The model makes provision for costing the finalisation of the MoU with districts. As for other parts of the model, the travel and Per Diem assumptions discussed below apply. Input Prices and Operational Assumptions (on the right) Per Diems There are two ways in which the Per Diems can be estimated. Firstly, the rate can vary between districts. This option is chosen below: This is the more realistic option. The user can choose three different amounts – from high to low. The corresponding Per Diem for each district must be selected in Districts. The other, less realistic option is to select a fixed Per Diem per district as shown below: Leadership Training People attending varies by district ←Select option Enter numbers in District sheet 8 Number of days per training course 1 Number of facilitators per course 1 Per Diem/Daily Fee per facilitator (USD) 120 Material costs per person attending the course 12 Catering costs per person/day (USD) 15 Leadership Training People attending varies by district ←Select option Enter numbers in District sheet 8 Leadership Training The same number attend in all districts ←Select option Enter number of people that attend here 8 MoU Finalisation Number of USAID officials that travel to district 2 Number of times they travel to district 2 Costing of Signing Ceremony (USD) 50 Enter values for Per Diem per level Per Diems - select an option below↓ High Medium Low Per Diem (USD/Day) vary by district 120 75 60 Enter per diem here: Use drop dow n on District sheet If selected, the appearance of this section will change and the user will be able to enter a Per Diem value that is the same for every district. Travel The cost of travel is a combination of the cost of fuel and car hire. The cost of fuel depends on the distance travelled (which is shown in the Districts worksheet), the fuel efficiency of the cars and price of fuel per litre. These last two are shown below. The cost of car hire is a combination of a fixed charge and cost per kilometre if a certain distance is exceeded, which is also shown below. Note that the above travel assumptions are used to estimate the cost of travel for trainers, SDS officials travelling to preparatory meetings, and officials travelling to MoU signing ceremonies. Who gets Per Diems and who travels? The cost of Per Diems and transport depends on how many people receive Per Diems, and how many travel. In the Scenarios & Results worksheet, the number of people attending the meetings is entered. In this section, who gets Per Diems, and who travels, is indicated. The “Yes” for “Do they travel to meetings” must be selected for people who travel from Kampala to the meeting. It is not relevant for officials who travel as part of their normal daily commute. The model estimates the number of cars that travel by dividing the total number of people who travel by the number indicated next to “How many representatives travel in each car?”If the SDS Cluster Lead and USAID staff get Per Diems, then the model will include this cost in travel for preparatory meetings and MoU signings. Sundry costs The following costs appear near the bottom of the worksheet. The UGX per USD exchange rate can be updated here as well. Note that these meeting costs are cost per meeting, not per person. Enter values for Per Diem per level Per Diems - select an option below↓ High Medium Low The Per Diem (USD/Day) is the same in all districts 120 75 60 Enter per diem here: Fuel Effiency - litres/100 km 7 Fuel Price (USD/Litre) 1.14 Vehicle Hire - cost per day 77 USD/Day Vehicle Hire - additional cost if distance > 200km 0.15 USD/KM Who gets per diems and who travels? Do they receive per diems? Do they travel to meetings SDS Cluster Lead Yes Yes USAID Yes Yes Implementing Partners Yes Yes Technical District Staff No No How many representatives travel in each car? 4 Catering, venue hire and stationery costs Costs per meeting (USD per meeting) Stationery 50 Catering (Grant A) 25 Venue Hire (Default is 0 unless selected in Districts) 40 UGX per USD 3315 lxii Annex 14: Evaluation Team Experience with Outcome Harvesting Overall, our experience with Outcome Harvesting was very positive and we think it was the appropriate tool for evaluating the DOP process. As PPL notes, O/H is one of a limited number of tools promoted by USAID for complexity monitoring. We believe this is connected to the way O/H approaches outcomes – enumerators approach the sites with no preconceptions on what the outcomes should be and instead create spaces for respondents to define outcomes empirically in their own terms. We feel that this removes a lot of the implementation blinders imposed by the log frame. In this sense, O/H is a refreshing addition to a new wave of M&E thinking that pushes us past linear logics and untenable assumptions regarding our industry’s ability to cause positive social change. O/H is not rigorous is the sense that evaluators can follow clear sets of rules that would guide two different evaluators to the exact same result. As such, it is difficult to call it an analytical method. But O/H does advocate a set of concepts based upon the logic of comparison, specificity and process tracing. Our enumerators were well trained in these concepts and approached DOP stakeholders armed with a toolkit that allowed them to quickly assess the evaluative value of stakeholder responses, even if the topics fell out of the enumerators’ substantive expertise. Our enumerators also commented on how their training took stakeholder conversations to new levels of content neither the enumerator nor the respondent initially expected. To the extent enumerators apply O/H’s concepts with discipline, we consider it a pseudo-rigorous method, i.e. subjective bias can still be high, but this is interrogated enough to have a solid foundation. As we mention in the main report, Outcome Harvesting, esp. with its emphasis on specificity and concreteness, in many ways involved training the respondent to respond with more precision and content than they are used to. This did meet some resistance, but this was a necessary burden for the evaluation to move beyond impressionistic assessments and into deliberate policy analysis. Moreover, when respondents couldn’t provide enough details to convince the enumerator, the events were not recorded as outcomes. Although this may appear too judgmental, we had to ensure O/H was applied with high standards so as to not discredit the evaluation and the approach. Indeed, enumerators were comforted by these standards, because they detected a lot of “fluff” in the conversations. By pointing to the requirements of O/H as a justification, it allowed them to focus more on the tangible. O/H is not without challenges. The biggest involve costs and level of effort. Unlike closed surveys and checklist approaches, where reliability requires enumerators to eliminate their judgments and follow standardized protocols, O/H requires enumerators to think on their feet and apply logical principles. This type of enumerator is extremely hard to find for extended field based assignments like the DOP. They are typically mid to senior level, possess strong writing and analytical skills and have an ability to unpack causal relationships through dialogue. They typically come at a price. Moreover, the labor intensive nature of O/H means that costs can quickly become prohibitively high during short term assignments involving a number of site visits and verification. Also, although doing so would’ve been unfeasible for budget reasons, it would’ve been better to space the visits across 2-3 day visits. For these reasons, O/H may not be the most cost effective approach for short terms evaluations conducted within 1-2 month timeframes. Instead, a far more cost effective approach would be to build the inhouse capacity of talented M&E and other staff to implement the method and include it as part of their job description. This would also allow O/H’ers to more comprehensively cultivate outcomes over time. Specific to the DOP, our enumerators also faced a fast learning curve in terms of understanding the activities of other IPs. It was quite common for DOP respondents to take credit for activities and outcomes that would have happened anyway because they were built into the scopes of separate activities. Indeed, many respondents were understandably confused by SDS DOP and non-DOP tasks, and saw no reason to distinguish between the two. Thankfully, our enumerators caught these tendencies early on and we provided constant clarifications on what were other IP-assigned tasks so as not to confuse them as DOP outcomes. Nevertheless, a lesson learned here is that O/H enumerators really need to include and offer alternative explanations for the outcomes to avoid making spurious connections. Finally, as we mentioned in the report, we experienced a strong positive social response bias with the method. Although we asked, no respondents offered negative outcomes. To be sure, there were complaints by IPs about attending the DOPs, but none could translate these into negative outcomes (they were inconveniences). It may be that there were no negative outcomes, but subsequent users of O/H should be wary of this. Also, like most methods, O/H suffers from time proximity bias, in which more recent outcomes were easier to describe in more detail than those emerging in 2013 or earlier. This type of bias specifically involves a respondent’s inability to recall details and/or the fact that earlier staff have left/activities are no longer active. lxiv Annex 15: DOP Cost Model Please see Excel Sheet embedded. Annex 15 - DOP Cost Model.xlsx