Midterm Evaluation: SUDAN BRIDGE-WINROCK BUILDING RESPONSIBILITY FOR THE DELIVERY OF GOVERNMENT SERVICES (BRIDGE) PROJECT MIDTERM EVALUATION MARCH 14, 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Dr. Nancy J. Allen, Yoseph Endeshaw, and Meron Tesfamichael, Management Systems International. BUILDING RESPONSIBILITY FOR THE DELIVERY OF GOVERNMENT SERVICES (BRIDGE) PROJECT MIDTERM EVALUATION Contracted under DFD-I-00–05–00251–00, Task Order No. 2 Services Under Program and Program Offices for Results Tracking (SUPPORT) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 i CONTENTS PROJECT DATA TABLE ............................................................................................................ ii ACKNOWLEDGEMENTS ......................................................................................................... iii ACRONYMS ................................................................................................................................. iv EXECUTIVE SUMMARY ........................................................................................................... v I. Introduction ................................................................................................................................ 1 II. Purpose and Methodology of Evaluation ................................................................................ 4 III. Major Findings and Conclusions ........................................................................................... 7 3.1 Program Design ............................................................................................................ 7 3.2 Program Implementation: Effectiveness, Impact and Sustainability .......................... 13 3.3 Program Management ................................................................................................. 36 3.4 Evaluation of Crosscutting Issues ............................................................................... 42 IV. Summary Recommendations ................................................................................................ 46 Annex 1: Scope of Work ............................................................................................................. 49 Annex 2: List of Site Visits ......................................................................................................... 65 Annex 3: List of Documents Consulted ..................................................................................... 66 Annex 4: List of Key Informant Interviews.............................................................................. 69 Annex 5: BRIDGE Winrock Results Framework .................................................................... 71 Annex 6: Summary Table of Achievements ............................................................................. 72 Annex 7: Training Assessment Guideline ................................................................................. 82 Annex 8: Adoption of Agricultural Technology ....................................................................... 83 Annex 9: Partial List of LGDF Proposals ................................................................................. 88 Annex 10: List of indicators ....................................................................................................... 94 Annex 11: Diagram of M&E System ......................................................................................... 95 Annex 12: Summary Overview of Citizens’ Perception Survey ............................................. 96 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 ii PROJECT DATA TABLE Program Name Building Responsibility for the Delivery of Government Services (BRIDGE-Winrock) Agreement Information Cooperative Agreement Award No. 650-A-00–09–00003–00 Modifications: None Region/Country Africa/Southern Sudan Program Funding USD $52, 499,991 from USAID with $846,369 from Winrock-led Consortium Prime Managing Contractor WI (Winrock International) Other Implementing Partners RTI International ACDI/VOCA GOAL Windle Trust—Added at beginning of implementation PACT—Added at beginning of implementation Key Program Dates BRIDGE-Winrock Program Beginning and End Dates January 22, 2009– January 22, 2012 BRIDGE-Winrock Program Mid-Term Evaluation Mission Period September 20, 2010– October 28, 2010 Evaluation Provider MSI (Management Systems International) MTE Team Members: Dr. Nancy J. Allen (External Evaluator/Team Leader) Mr. Yoseph Endeshaw (External Evaluator) Miss Meron Tesfamichael (External Evaluator) Mr. Martin Maciek Malual (Local Government Board, GOSS) MTE Mission Site Visits Southern Sudan States of Northern Bahr-el-Ghazal, Warrap, and Unity Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 iii ACKNOWLEDGEMENTS Management Systems International (MSI) Services Under Program and Project Offices for Results Tracking (SUPPORT) project contracted with independent evaluators, Nancy J. Allen, Yoseph Endeshaw and Meron Tesfamichael, to conduct a midterm evaluation of the United States Agency for International Development (USAID) mission in Sudan‘s Bringing Responsibility to the Delivery of Government Services (BRIDGE) program. The BRIDGE program, led by Winrock International, has been implemented in Northern Bahr-el-Ghazal, Warrap, and Unity States in Southern Sudan since 2009. The team would like to thank everyone that contributed to our work, in particular Martin Maciek Malual of the government of Southern Sudan‘s Local Government Board. Mr. Maciek was an integral member of the evaluation team and an invaluable guide to the political and social complexities of the developing state in Southern Sudan. We would also like to thank the staff of BRIDGE-Winrock, whose deep commitment to program objectives was clearly evident in the considerable time and thought given to all of our questions. In particular, we would like to thank Lutfiyah Ahmed, Deputy Chief of Party for Programs, who coordinated the evaluation with a thoroughness and graciousness that greatly improved the quality of the evaluation. Finally, we extend our thanks to outgoing Chief of Party, John Palmucci, and incoming Chief of Party, Eveline Viegas, for their committed support throughout the evaluation. The team would also like to thank MSI and the SUPPORT staff for their able assistance, especially Monitoring and Evaluation Consultant, Valerie Haugen; SUPPORT Assessment, Monitoring, and Evaluation Advisor, Ingrid Orvedal; and Chief of Party, Ami Henson. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 iv ACRONYMS AOTR Agreement Officer Technical Representative APS Annual Program Statement AWP Annual Work Plan BRIDGE Building Responsibility for the Delivery of Government Services CA Cooperative Agreement CAG Community Action Group CBD Community-based Development CBO Community-Based Organizations COP Chief of Party CPA Comprehensive Peace Agreement CSO Civil Society Organization DG Director General D&G Democracy and Governance FY Fiscal Year IDP Internally Displaced Person GOSS Government of Southern Sudan LGA Local Government Act LGB Local Government Board LGDF Local Government Development Fund LG Local Government M&E Monitoring and Evaluation MAF Ministry of Agriculture and Forestry MARF Ministry of Animal Resources and Fishery MOEST Ministry of Education, Science, and Technology MOFEP Ministry of Finance and Economic Planning MOLPSHRD Ministry of Labor, Public Service, and Human Resources Development MSI Management Systems International MTE Midterm Evaluation NBG Northern Bahr el Ghazal NGO Nongovernmental Organization PTA Parent-Teacher Association RTI Research Triangle Institute SMOAF State Ministry of Animals and Fisheries SOW Scope of Work SPLM Sudanese People‘s Liberation Movement STTA Short Term Technical Assistant TOT Training-of-Trainers UNDP United Nations Development Program USAID United States Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene Education WSG Women‘s Support Group Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 v EXECUTIVE SUMMARY This report provides an analysis of the Building Responsibility for Delivery of Government Services (BRIDGE)-Winrock program based on a midterm evaluation conducted from September 20–October 28, 2010. The overarching conclusions of this evaluation are that:  BRIDGE-Winrock has met and exceeded targets in a large number of areas, representing a remarkable level of foundational activity, achieved in an extremely challenging environment. A solid foundation has been established for increasing impact over the remaining life of the program.  There are substantial risks to continuing progress and sustainability. Some of these risks are a consequence of working in the context of an exceedingly fragile state, and others reflect implementation challenges being experienced in the field.  BRIDGE-Winrock’s experience suggests the program design is viable and the model replicable as an approach to area-based local government capacity building and service delivery if: (1) certain critical success factors can be maintained; (2) substantive program modifications are introduced; and (3) programmatic integration is more fully operationalized. Program Description The BRIDGE-Winrock program was funded as an integral component of USAID‘s strategy to support the Comprehensive Peace Agreement (CPA). The program‘s dual aim of building the capacity of government and creating tangible peace dividends is implemented via five program components: i. Democracy and Governance. Strengthen the capacity of government to deliver social services and promote economic growth (led by Winrock with technical support from RTI). ii. Community Development. Strengthen the capacity of communities and government to work together to identify, prioritize, and address community needs (led by Winrock). iii. Agriculture and Livelihoods. Increase food security and promote rapid, broad-based, and self-sustaining economic growth through development of the agriculture sector and livelihood opportunities (led by ACDI/VOCA and supported by Winrock). iv. Water, Sanitation, and Hygiene. Increase access to safe water and sanitation (led by PACT with support from GOAL in Warrap State). v. Education. Expand access to quality education (led by Winrock). The BRIDGE-Winrock program targets three of the most vulnerable states in Southern Sudan: Northern Bahr-el-Ghazal (NBG), Warrap, and Unity. Purpose and Methodology of Evaluation The purpose of the midterm evaluation was to: (1) assess the program‘s achievements against performance targets; (2) provide USAID with an informed discussion regarding the likelihood of the program‘s impact; and, (3) provide USAID with an analysis of the program‘s design. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 vi The methodology of this evaluation was fundamentally qualitative, with close attention given to quantitative information wherever such data was available. Ongoing document reviews were supplemented by research and interview questions aimed at probing well beyond reported accomplishments. The team interviewed over 250 individuals and visited more than 20 communities in seven counties across the three states. The team sought to introduce methodological rigor where possible by going beyond showcase project sites. Major Findings and Conclusions Program Design The program design has a sound logic rooted in a strategic model of integrated community-based and area-based local government development, commonly advocated for in post-conflict and conflict￾affected settings. The strong strategic logic was not fully operationalized in programmatic terms, particularly with regard to integration across components. The program is responsive in design and implementation to the interests of both the Government of Southern Sudan (GOSS) and the needs of communities. BRIDGE-Winrock is deeply committed to working with and through local government to deliver services and build capacity. Staff throughout the program regularly strive to ―put the face of government‖ on all activities. Indeed this dedication to and notable success at working with and through government is a hallmark of the program. There was, however, one outstanding complaint, namely that BRIDGE-Winrock operates in only three counties in each state. This concern is so widely and strongly held at the state level that it must be addressed in any consideration of program extension or expansion. Limiting implementation to selected counties has the potential to create conflict that overshadows the contributions of the program. Program Implementation BRIDGE-Winrock has met and exceeded targets in a large number of areas, representing a remarkable level of foundational activity, achieved in an extremely challenging environment. Program reporting routinely understates the enormous implementation challenges being experienced in the field. The report analyzes each of the five components individually and then discusses the value added and tensions associated with integration across components. i.) Democracy and Governance (D&G) Component The strategy for this component has been developed more formally and comprehensively than that for any of the other components, specifying work in four areas: financial management; planning and budgeting; tax administration; and human resources development (HRD). The bulk of capacity￾building activity in the first three of the four thematic areas has been provided through training in classroom and workshop settings. Evaluation team members concluded that training is contributing to improved administrative capacity. In the area of Planning and Budgeting, BRIDGE-Winrock has facilitated the production of Draft County Plans and Budgets in all three states. The production of Draft County Plans and Budgets is the most solidly identifiable outcome of the D&G component, as well as being critical to its success. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 vii However, evaluators remain concerned that the expanding scope of the program at the state level poses risks. In particular, increasing involvement in Tax Administration seems premature given the embryonic state of tax legislation. There was little evidence of likely impact of human resource development activities in the near or long term, and activities in the HRD area appeared particularly removed in time and content from the service delivery aims of the program. ii.) Community Development Component The Community-Based Development (CBD) Component works to develop the capacity of communities and government to work together to identify, prioritize, and address community needs. In this effort the CBD component has conducted a number of activities, concentrating heavily on the formation of over 65 Community Action Groups (CAGs). CAGs play a vital role in the implementation of BRIDGE￾Winrock activities, but do not yet exhibit independent capacity to perform the broader democratic and economic functions anticipated in community-driven models of decentralized development. With respect to sustainability of BRIDGE–Winrock efforts, it is almost certain that CAGs would not survive the departure of BRIDGE-Winrock. The limited capacity of CAGs for independent collective action is largely a reflection of the magnitude of the problem, not a weakness of BRIDGE-Winrock‘s efforts. As is so often true, it will take time and sustained, focused efforts to empower communities ravaged by decades of war to engage effectively with a still extremely limited government. iii.) Agriculture and Livelihoods Component The project design for this component is based in a strong model for market-based sustainability with which the consortium partner ACDI/VOCA has substantial and successful experience in other countries. However, the enabling environment for agriculture in Southern Sudan is poor from the perspective of both markets and government. Due in large part to the weakness of the enabling environment, but also to poor implementation, very few of the agricultural and livelihood activities are performing as anticipated. Sustained impact from current activities does not seem likely without substantial improvements to the design and implementation of the agriculture and livelihoods component. iv.) Water, Sanitation and Hygiene Promotion (WASH) Component The WASH component contributes the most tangible peace dividends of the BRIDGE-Winrock program, meeting the most universally and clearly identified need of communities—namely, access to clean water. Furthermore, the program appears to be implementing this activity in a manner that has potential for sustained impact. The WASH component has been more successful in increasing access to water than it has in improving sanitation services and practices. v.) Education Component The needs of government and communities for educational support are enormous, and while BRIDGE-Winrock is making very positive contributions to filling those needs, it was difficult to reach firm conclusions about likely impact. It was similarly difficult to discern strategies being used to ensure sustainability beyond the inherent and successful commitment to close cooperation with government. The strategic clarity implied by the two focus areas is somewhat belied by a lack of programmatic coherence on the ground and a near absence of any discussion of intended outcomes. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 viii Benefits and Tensions of Integrated Goals and Activities There was no formal effort to develop program integration or measure the value-added of integration at the start of the program. Nonetheless, there are many points of intersection or integration that clearly add value to the BRIDGE-Winrock program. The strongest benefits occur as a consequence of activities that serve important integrated goals, namely those that (1) connect communities with local and state government and/or (2) connect capacity building efforts with service delivery. There are a number of program efforts through which these connections are forged—almost all require the integrated efforts of the CBD and D&G components in coordination with other sectoral components. There are also some areas in which increasing distance between component activities may compromise the achievement of fundamental integrated goals, namely in the tension between ‘building the capacity to deliver services’ and ‘ensuring that services are effectively delivered.’ Staff throughout the organization strongly emphasize that BRIDGE-Winrock is first and foremost a ―capacity building project.‖ This conceptualization of BRIDGE-Winrock focuses organizational attention on administrative capacity building at the possible expense of attention to the service delivery components. There are some indications that the quality of service delivery interventions has received less managerial and strategic attention than have D&G capacity building efforts. Cost Effectiveness The evaluation team was asked to look at the question of value for money. No matter how successful the program has been in terms of targets achieved at least, two questions always remain: ―Were these achievements ‗worth the money‘ spent?‖ and ―Are there better alternatives for these funds?‖ The expenditure of $53 million is undeniably expensive and the proportion allocated to salaries and travel is indisputably large. Only a very crude analysis was possible, but rough comparisons with similar programs suggest that BRIDGE-Winrock‘s foundational achievements are comparatively strong and costs not out of line. The evaluation team therefore cautiously concludes (given the severe limitations of available data) that BRIDGE-Winrock is reasonably cost-effective. Program Management Organizational Assessment BRIDGE-Winrock is a well-managed and disciplined organization, committed to delivering results. The level of accomplishment in terms of quality staffing and the establishment of sound organizational structures and systems (including well-functioning logistical support at compounds in each of the states) is remarkable. The sense of shared purpose was profound, owing a great deal to strong leadership and disciplined management. Management and Functionality of the Consortium BRIDGE-Winrock was structured as a consortium for sound technical and logistical reasons. In practice, however, the consortium structure has contributed little to the success of BRIDGE￾Winrock. Despite the obvious technical expertise of consortium partners, there has been substantial restructuring of consortium relationships over the course of the first 18 months, with Winrock International increasingly assuming more direct control over project implementation. Given the Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 ix challenging context it must contend with and the pressing time frames it faces, the BRIDGE-Winrock program has been well served by this tighter control. Monitoring and Evaluation Monitoring, narrowly defined as the routine tracking of inputs and outputs, is well executed by BRIDGE-Winrock. However, the system provides little information for evaluation purposes beyond verification that agreed-upon inputs and outputs have been achieved. The BRIDGE-Winrock team is well aware of the weaknesses of the M&E indicators for evaluation purposes, and in internal deliberations has suggested adding other indicators. In an attempt to inform future evaluations of BRIDGE-Winrock, the program has developed three ―impact indicators.‖ This attempt to assess impact is commendable, and these indicators may produce useful results for soft trend analysis1 , though they are not likely to yield statistically significant results. Evaluation of Crosscutting Issues Gender Equity BRIDGE-Winrock exhibits a strong and successful commitment to improving gender equity in all areas of organizational activity. This commitment was abundantly evident in interviews, and most convincingly apparent in both formal and informal observations of staff interactions and discussions. Environmental Protection BRIDGE-Winrock‘s attention to environmental issues is almost exclusively compliance-focused. Outside of the compliance orientation, there was little evidence of pro-active environmental sensitivity or consideration of longer-term environmental issues. Conflict Sensitivity The core features of BRIDGE-Winrock‘s program design reflect conflict sensitivity: USAID selected the three states in which BRIDGE-Winrock operates because of their geopolitical importance in the Sudanese conflict, and BRIDGE-Winrock‘s emphasis on community based development (CBD) also reflects best practice in program design for conflict-affected areas. In the context of implementation, however, there was remarkably little attention given to conflict dynamics beyond planning for likely interruptions to work. Summary Recommendations The evaluation team concludes that a solid foundation has been established for increasing impact over the remaining life of the program. However, this report does highlight a number of risks to continuing progress—some of which are a consequence of working in the context of an exceedingly fragile developing state and some of which reflect weaknesses in implementation. Recommendations in the report seek to address these risks, suggesting improvements in program design and implementation. 1 Comparative analysis of project performance over time. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 x For the remaining life of program, the evaluation team recommends that BRIDGE-Winrock focus more closely on quality of outputs, with careful attention to service delivery components of the program:  Maintain focus on those capacity building efforts that have a strong link and well-defined outcome pathway to local service delivery improvements.  Exercise caution with respect to expanding mandate towards state and national capacity building efforts.  Give special attention to possible M&E changes for immediate and improved measurement of emerging outcomes so that stronger evidence will be in place for continued funding decisions With respect to individual components, the team recommends: Democracy and Governance:  Review the efficacy of HRD activities (i.e., job descriptions and organograms) with special attention to the integrated and downstream value of these activities in the time frame shared by service delivery demands and capacity building requirements.  Focus tax administration activities on basic principles and operations that would survive substantial changes in the tax code following the referendum. Community Development:  Focus on strengthening already existing CAGs/WSGs rather than facilitating the formation of new ones.  Clarify the vision for the civic function of CAGs and develop clear strategies to realize those functions. Agriculture and Livelihoods:  Conduct a thorough examination of work in the agriculture and livelihoods area with a view to correcting serious design and implementation issues and challenges. Water, Sanitation and Hygiene:  Continue strong focus on sustainability of WASH efforts taking care to ensure that those activities (e.g., fee-based supply, accessibility of pump mechanics) produce intended outcomes.  Request changes to targets for household latrines while increasing activities directed at increasing the awareness and acceptance of sanitation and hygiene practices. Education:  Conduct an internal review of the education component with a view to improved programmatic coherence across activities directed towards identifiable outcomes. The model has potential for replication or expansion based on the implementation experience of BRIDGE-Winrock. Successful replication, however, is far from assured. The relative success of BRIDGE-Winrock can in part be attributed to strong leadership accompanied by disciplined Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 xi organizational and managerial focus on short-term deliverables. These critical success factors need to be maintained while also improving attention to the quality of outputs and sustainability of outcomes. The decision to replicate the BRIDGE program to other states must:  Be grounded in additional formative evaluation that needs to begin almost immediately within BRIDGE-Winrock;  Include substantial program modifications addressing weaknesses identified in this report, and as suggested in the component recommendations above;  Be accompanied by much-improved M&E in any future programming, with special attention to realistic time frames and baseline measurement, perhaps including a baseline household panel survey in appropriately sampled counties; and  Consider rollout of all or a subset of program activities across all counties in a state to avoid potential conflict. USAID funding of multi-sectoral, area-based programs would be well served by additional and creative efforts at operationalizing integrated program design. This would entail, among other activities:  Achieving definitional clarity around programmatic versus strategic integration;  Clarifying the degree of programmatic integration necessary to achieve strategically integrated goals;  Clarifying USAID expectations of the contribution of multi-sectoral, area-based programs to country-wide sectoral goals;  Understanding how component activities contribute separately, but additively, to integrated outcomes and/or how they might be integrated to achieve more sustainable outcomes; and  Understanding how risks or failures in one area of activity do or do not threaten results in another area. However, based on the internationally well-documented challenges of partnership and consortium management, over-specification of integration in the form of unrealistic integrated monitoring and evaluation indicators or in the form of tightly woven joint work planning is not recommended. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 1 I. INTRODUCTION 11. Country Context At roughly one-quarter the size of the United States, Sudan is the largest country in Africa and shares a border with nine others: Eritrea and Ethiopia in the East; Kenya, Uganda, and the Democratic Republic of Congo in the South; the Central African Republic, Chad, and Libya in the West; and Egypt in the North. Since its independence in 1956, Sudan has suffered from two prolonged civil wars, with only a decade of troubled peace from 1972 to 1983. Since independence, military regimes favoring Islamic-oriented governments have dominated national politics, concentrating wealth and power in the mostly Arab North at the expense of the marginalized majority elsewhere in the country. The first civil war between the South and the North began just five months before independence and ended 17 years later with the signing of the 1972 Addis Ababa Peace Accord. The second Sudanese war began in 1983 and ended some two decades later on January 9, 2005, with the official signing of the Comprehensive Peace Agreement (CPA). The 2005 CPA granted a six￾year interim period of autonomy to Southern Sudan that was to be followed by an internationally monitored referendum in 2011 that would determine the future borders of the nation and the sharing of common wealth, which includes both substantial oil reserves and significant water resources. The CPA and the Sudan Interim National Constitution established an asymmetrical federal system, with the Government of Southern Sudan (GOSS) acting as a buffer between the central government and southern states. The Interim Constitution of Southern Sudan provides for three levels of government in the South: national (GOSS), state and local. Accordingly, the Southern Sudan is decentralized into ten states: Central Equatoria, Eastern Equatoria, Jonglei, Lakes State, Upper Nile, Warrap, Western Bahr al￾Ghazal, Western Equatoria, Western Upper Nile, and Northern Bahr al-Ghazal. The states are further subdivided into counties, payams (district), and bomas (village), which collectively constitute local government (LG). Sudan has a total population estimated at 40 million. Population dynamics in Southern Sudan are highly fluid as mobile pastoralist communities and large refugee and internally displaced person (IDP) flows merge within sometimes ambiguous borders. As a result, population estimates for Southern Sudan differ enormously, ranging from 5 to 10 million. The most recent official published source of information, the Statistical Yearbook for Southern Sudan (2009), puts the total population in Southern Sudan at 8,260,490. The decades of war and underdevelopment as well as frequent drought, floods, and famine have had a devastating impact on social, political, and economic structures in Southern Sudan. Southern Sudan currently has the lowest Human Development Indicators in the world. Although some progress has been made in fostering social, political, and economic stabilization since the signing of the CPA, numerous challenges persist. The CPA created high expectations among citizens for tangible peace dividends, transparent power and wealth sharing, and an end to insecurity. However, the government‘s ability to deliver basic services such as education, health, and infrastructure is extremely limited, Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 2 hampered by a cavernous gap in human and institutional capacity at all levels of government as well as by continuing security concerns throughout Southern Sudan. 1.2 Program Description and Context The Building Responsibility for Delivery of Government Services (BRIDGE) program was funded as an integral component of USAID‘s strategy to support the Comprehensive Peace Agreement (CPA).2 As originally stated by USAID in the Annual Program Statement: ―The objective of the Building Responsibility for the Delivery of Government Services (BRIDGE) Program is to support the CPA by delivering tangible peace dividends through building capacity of State and County governments to engage with communities to meet their needs and increase their incomes.‖ BRIDGE-Winrock is managed under a Cooperative Agreement with a budget of $52.5 million and is implemented by a consortium of partners, led by Winrock International. The three-year program was launched in January 2009, and at the time of this evaluation the staff was in the process of developing the work plan for the final year of operation.3 The objective (stated above) embodies two simultaneous aims, to build the capacity of government and to create tangible dividends in terms of meeting basic needs and increased incomes. The dual aim is implemented via five program components: 1. Democracy and Governance: Strengthen the capacity of government to deliver social services and promote economic growth (led by Winrock with technical support from RTI). 2. Community Development. Strengthen the capacity of communities and government to work together to identify, prioritize, and address community needs (led by Winrock). 3. Agriculture and Livelihoods. Increase food security and promote rapid, broad-based, and self-sustaining economic growth through development of the agriculture sector and livelihood opportunities (led by ACDI/VOCA and supported by Winrock). 4. Water, Sanitation, and Hygiene. Increase access to safe water and sanitation (led by PACT with support from GOAL in Warrap State). 5. Education. Expand access to quality education (led by Winrock). The USAID/Sudan‘s strategy statement (2006–2008) focuses on ―supporting the implementation of and reducing threats to the CPA, including the provision of peace dividends that … address the root factors that fuel conflict.‖ Accordingly, one of the chief strategic commitments of USAID‘s conflict-mitigating strategy is to focus on key geographic areas—those states on the North-South border that have been highly affected by and/or are prone to conflict as well as the Three Areas (Abyei Area, Blue Nile State and Southern Kordofan State). The BRIDGE-Winrock program targets three of those vulnerable states: 2 USAID/Sudan has another project called ―BRIDGE,‖ implemented by Mercy Corps in other parts of Southern Sudan. For clarity, this activity will be called BRIDGE-Winrock and the other will be referred to as BRIDGE-Mercy Corps. 3 Consortium partners include ACDI/VOCA, GOAL Sudan, PACT Sudan, RTI International, and Windle Trust. Consortium management is discussed in Section 3.3.3 as part of the discussion of program management. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 3 Northern Bahr-el-Ghazal (NBG), Warrap, and Unity. In each of these states, BRIDGE-Winrock operates in three counties and a subset of payams: Northern Bahr-el-Ghazal: Aweil East, Aweil West, Aweil North Warrap: Gogrial West, Tonj North, Twic Unity: Rubkona, Mayom, Guit All three states are characterized by weak government institutions with few skilled staff in critical areas such as economic planning, agriculture, health, and engineering. Personnel capacity at the county level is generally even weaker than at the state level. The revenue base for states and counties is low and dependent on GOSS transfers that meet basic payroll commitments with little or no operating funds remaining to conduct the business of government. Agricultural productivity is low in all three states, and the level of food insecurity remains extremely high. The 2009/2010 Annual Needs and Livelihoods Assessment (ANLA) concluded that in NBG over 50 percent of all households were either moderately or severely food insecure; similarly, in Warrap the number of severely food insecure households was estimated at 23 percent, with moderately food insecure household estimated at 26 percent.4 The food security situation in Unity is somewhat less dire, but still a challenge both for resident pastoralists and the very large numbers of returnees in that state. Harvests in October/November 2010 were expected to ameliorate household food status, but not to fundamentally alter the situation. The health profile of the three states is dismal, with infant and under-five mortality rates in NBG and Warrap among the highest in Southern Sudan. Contaminated water and poor sanitation systems contribute to high incidence of disease in all three states. In education, these three states have some of the highest rates of illiteracy in all of Southern Sudan and, correspondingly, the lowest rates of primary school intake. Since the signing of the CPA, all three states have continued to receive a high number of displaced persons returning from neighboring countries as well as northern Sudan, with estimates of returnees and internally displaced persons at just over 50 percent in NBG and approximately 20 percent in Warrap. With limited economic opportunity and weak availability of social services, integration of IDPs has been a challenge to state and local government officials. The number of returnees was growing rapidly as BRIDGE-Winrock neared its third year of operation. The challenges of operating in this environment cannot be overstated. They manifest in a variety of ways, most immediately in the constraints that limited human and institutional capacity of government and the compelling need for basic services impose on program implementation. The sustainability of program efforts are challenged further by (a) a ubiquitous ―relief mentality‖ that defines working relationships with communities and local government around immediate and often unrealistic demands; (b) a near absence of government funding to extend and sustain program efforts; and (c) an extremely limited private sector to sustain program efforts through market-based mechanisms. 4 2009/2010 ANLA (February 2010) is a collaborative assessment by the Ministry of Agriculture and Forestry, the World Food Programme, World Vision International, the Food and Agriculture Organization, the Southern Sudan Relief and Rehabilitation Commission, and the South Sudan Commission for Census, Statistics and Evaluation. Data was not available for Unity state for this time period. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 4 II. PURPOSE AND METHODOLOGY OF EVALUATION 2.1 Purpose and Scope of the MTE Evaluation The purpose of the midterm evaluation was to (1) assess the program‘s achievements against performance targets at the midterm point in the implementation period; (2) provide USAID with an informed discussion regarding the likelihood of the program‘s impact in the medium to longer term; and (3) provide USAID with an analysis of the program‘s design, particularly with respect to the program‘s multi-sectoral structure and potential for integrated outcomes. In support of this purpose, a series of questions were posed in the Scope of Work (SOW).5 Fundamentally, these questions sought to determine the:  Effectiveness and efficiency  Impact and sustainability  Viability and replicability of:  Program design  Program implementation  Program management. This SOW requested that the team take a special look at the management and organization of the program. The team leader regularly included questions to ascertain organizational efficiency and effectiveness using McKinsey‘s 7S template as a guiding framework.6 Additionally, the evaluators were asked to address cross-cutting issues of gender equity, conflict-sensitive programming, and environment, paying particular attention to USAID‘s strategic commitments in these areas. 2.2 Evaluation Methodology The methodology of this evaluation was fundamentally qualitative, with close attention given to quantitative information wherever such data was available. Ongoing document reviews were supplemented by research and interview questions aimed at probing well beyond reported accomplishments. The complexity of the integrated program design and breadth of the program required:7 5 See Annex 1 6 This is a framework for analyzing the strand weaknesses an organization. The seven S’s are: Strategy, structure and systems (considered the "hardware" of success), style, staff, skills and shared values (are the "software"). Read more at: http://strategiccoffee.chriscfox.com/2008/06/mckinsey-7-s.html 7 Annex 2 contains a detailed site visit schedule, Annex 3 contains a full list of literature and documents reviewed, and Annex 4 contains a full list of persons interviewed. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 5 (1) Extensive site visits in NBG, Warrap, and Unity states: a. Exploratory and focused discussions with Community Action Groups, Women‘s Support Groups, Water Management Committees, PTAs, Vegetable Producer Groups and so on. b. Focused interviews with trainees in multiple areas (e.g., ox-plow farmers, government accountants, English Language trainees) c. Observational visits to vegetable demonstration gardens, fishery groups, a vocational school, boreholes, household latrines, and other program activities. (2) Extensive review of relevant literature in a number of areas, including: a. Community-based development b. Capacity-building in state and local government c. Development in conflict-affected areas d. Integrated programming and rural development e. Fiscal and administrative decentralization (3) Intensive examination of internal and external program-related documents (4) Key informant and focus group interviews,: a. Government officials at national, state, and local levels b. Program staff c. Collaborating organizations d. Community members and other targeted beneficiaries During the fieldwork for this evaluation (conducted between September 20, 2010 and October 28, 2010 as Southern Sudanese began the 100-day countdown towards the 2011 referendum), the team interviewed over 280 individuals and visited over 20 communities in seven counties across the three states. The team sought to introduce methodological rigor where possible by going beyond showcase project sites, insisting on visits to sites selected in a quasi-random process. 8 The evaluation team is confident that within the limitations of time and geography a thorough evaluation was conducted.9 The evaluation process was guided by Management Systems International‘s (MSI) quality assurance process, which included a two-day Team Planning Meeting (TPM) at the outset of the evaluation in which Winrock and USAID staff participated. The SOW was finalized at the TPM and was followed by the development of an evaluation plan premised on MSI‘s ―Getting To Answers‖ matrix, extracts of which appear below by way of example: 8 Complete lists of communities/CAGS were provided to the evaluation team. The BRIDGE-Winrock staff selected approximately one-third of the field sites. The evaluators identified two-thirds of communities for visits on the basis of a random number generator. If, however, the community was determined to be inaccessible due to flooding or the distance was deemed prohibitive in the available time, the next community on the list was selected and so on. 9 The team recognizes weaknesses in the extent and range of interviews with USAID program staff. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 6 Table 1: Getting to Answers Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling or Selection Approach Data Analysis Method Data Source Methods Effectiveness and Efficiency How effective has the program been in achieving the performance targets/indicators set out for each of its five components? Description Statistics Document reviews Interviews Site visits Observation Program staff Targeted GOSS line ministries State and LG institutions Community representatives Purposive sampling Trend analysis Basic statistical analysis Cross-Cutting Issues: Conflict Sensitivity, Gender and Other Equity Considerations, Environment To what extent does a conflict-sensitive lens inform program strategy and implementation? Description Comparison Document reviews Interviews Observations Project documents USAID Program staff Literature on programming in post-conflict context Purposive sampling Content analysis Project Design and Future Directions What is the development hypothesis of the program and to what extent it is relevant and coherent with the country context as well as the needs and priorities of the targeted States and LG? Description Document reviews Interview Observation Program and USAID staff State and LG representatives Community members Program Documents USAID country strategy Sudan data Purposive sampling Content and trend analysis This process was supplemented by interview protocols and other evaluation guidelines prepared for fieldwork and analysis. On return from the final field visits, the evaluation team conducted (as specified in the MSI guidelines) an abbreviated ―Findings, Conclusions, and Recommendations ‖ workshop in preparation for debriefs of the evaluation to USAID and the GOSS.10 10 It is important to note that the evaluation team believes strongly that, given the multi-sectoral nature of the program as well as the geographic distances and travel time required to conduct the evaluation, the time required for analysis of the data was substantially underestimated. As such, the evaluation debrief process was less a summation of findings, conclusions, and recommendations than a significant part of the analytic process itself. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 7 III. MAJOR FINDINGS AND CONCLUSIONS 3.1 Program Design 3.1.1 Relevance of the Program to the Needs and Interests of Beneficiaries The objectives of the program are well directed to the interests of both the government of Southern Sudan and the needs of communities in which the program is implemented. Maintaining democratic alignment between the two—the needs of communities and the interests of government—is a central objective of decentralized governance and, by extension, the BRIDGE-Winrock program. Responsive to the Interests of Government BRIDGE–Winrock has implemented its activities in close cooperation with government at the national, state, and local level and in thoughtful extension of the legal provisions established by the Local Government Act (LGA) of 2009. The strategy and programmatic priorities of BRIDGE-Winrock were developed in regular dialogue with the GOSS, especially the Local Government Board (LGB). This central relationship has been strengthened with the recent hire of a senior staff member who has worked very closely with the LGB for more than three years. Government officials at national, state, and local levels praised BRIDGE-Winrock‘s efforts and success at working with government, in multiple instances attesting to uniquely close cooperation between BRIDGE-Winrock and government as exemplified in statements such as, ―BRIDGE [Winrock] is one of the agencies closest to us [more] than any other agency.‖ Despite the program staff‘s clear and demonstrated effort to work directly with government, there are (and will continue to be) complaints of insufficient consultation and coordination, just as there are between most donor-funded programs and the GOSS. Aligned with National Policy Priorities The Local Government Act (LGA), promulgated in 2009, constitutes the guiding legislation for the design of the BRIDGE-Winrock program. The LGA enacts the commitment to administrative and fiscal decentralization established in the CPA and the Interim Constitution of Southern Sudan. The CPA calls for a decentralized system ―with significant devolution of power‖ to local levels of government. Similarly, the Interim Constitution of Southern Sudan has reaffirmed the decentralized system of government in Southern Sudan (Article 50), entrusting local governments with a wide range of roles— including the promotion of self-governance, public participation, economic and social development, self-reliance, and service provision—and gives them the power to levy, charge, collect, and appropriate fees and taxes in accordance with the law (Article 173). In line with the Interim Constitution, the LGA assigns a wide range of roles and responsibilities to local governments, which include (a) regulation and maintenance of law and public order; (b) regulation, provision, and maintenance of services to the people; (c) land administration and environmental management; (d) encouragement and promotion of local development; (e) provision of access and opportunities for the people to engage in the development of their communities; and (f) protection of the rights of the people and their interests. Financially, the LGA provides that local governments are funded by a combination of government grants (i.e., conditional grants and block grants), locally generated revenues (i.e., council property, social service, and council land taxes), community contributions (e.g. labor contributions and monetary contributions), grants and donations from Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 8 organizations and individuals, and loans. According to the LGA, Local Government Councils are required to prepare an annual plan and budget based on an integrated and participatory approach. Aligned with Sectoral Policy Priorities Evaluation team members met with representatives of each of the relevant line ministries in the agriculture, education, and water and sanitation sectors at the national level and in each of the three states to determine whether component priorities and activities were appropriately aligned with GOSS sectoral policy priorities. With very modest and appropriate adjustments, this proved to be the case. In the WASH component, BRIDGE-Winrock implements activities within the outlines of the now well￾established WASH framework promoted over the past decade by WHO and UNICEF, a policy framework to which the GOSS also adheres. In the education component, BRIDGE-Winrock initially prioritized improving teacher quality whereas the government has identified school construction as its number one priority (and teacher quality as the second priority). This appears well justified in the context of USAID‘s sectoral and strategic commitments in the education field, although pressure to contribute to school reconstruction efforts is strong and certain to continue.11 In the agriculture and livelihoods component, the national policy framework is less well articulated than it is in other sectors, and BRIDGE-Winrock has struggled to implement its agricultural activities in the absence of clear policies for agricultural extension, pesticide use, livestock pharmaceuticals distribution, etc., focusing appropriately on market-based solutions. BRIDGE-Winrock consistently received high praise from government officials at the state line ministries for the level of cooperation. It is important to note also that program activities closely accord with the priorities identified in USAID‘s 2009 Government of Southern Sudan Functional Capacity Prioritization Study.12 That study prioritized the functional areas of financial resource management, human resource management, and equitable social service access, which are also priority areas of BRIDGE-Winrock‘s capacity building and service delivery interventions. Coordinated with Local Government At the local level, BRIDGE-Winrock has sought to maintain close cooperation with county, payam, and boma administrators. This can be difficult, as turnover among local government officers is high due to mandated two-year rotations. Maintaining these relationships can sometimes require artful negotiation. County commissioners insist vehemently that all NGO activity be coordinated through county offices, in line with the vision of the Local Government Act and expectations that in the future NGO funds will be administered strictly according to priorities established in county plans and budgets. This can lead to some occasional tension. For example, BRIDGE-Winrock staff assert that county administrators have sought, on occasion, to maneuver the siting of boreholes to meet their political, and sometimes personal interests. County administrators likewise contend that some community action group (CAG) leaders have sited boreholes according to their personal interests and not those of the larger community, insisting that the county commissioner‘s staff is in the best position to make such judgments. The weight of anecdotal evidence favors BRIDGE-Winrock‘s assertion although there do appear to be isolated cases where interests of 11 BRIDGE-Winrock is increasing support to school construction and rehabilitation through LGDF funds. 12 USAID (December 2009). Government of Southern Sudan: Functional Capacity Prioritization Study. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 9 individual CAG members may have influenced the location of a borehole. Still, in the majority of counties, BRIDGE-Winrock was acknowledged as the NGO most closely coordinating its activities with local government. Advancing the Priorities of Communities The program is designed at its core to advance the voice of communities in the developing state. The community development component is organized to assist communities in identifying and then communicating priorities to local government. A number of activities, including the formation of community action groups and facilitation of town hall meetings, are directed at this goal. Town hall meetings have been facilitated by BRIDGE-Winrock to bring together county planning unit representatives, payam and boma officials, CAGs, and community members. BRIDGE-Winrock has facilitated 12 town hall meetings in each state. According to a number of participants, town hall meetings were an empowering experience and also provided an opportunity to interact with community members from other bomas and payams. Outstanding and Critical Concern Remains There is an outstanding concern that was raised repeatedly and forcefully by the majority of senior government officials in virtually every evaluation interview, as well as at the GOSS evaluation debriefing. This concern, often voiced as a strong complaint, is the fact that BRIDGE-Winrock operates in only three counties in each state (NBG has five counties; Warrap has six counties; Unity has nine counties). In just one day of interviews, content analysis of evaluation notes revealed over 15 instances in which this topic was raised. BRIDGE-Winrock staff members find themselves regularly responding to this issue. For the time being, program staff have been able to explain that the choice of counties was dictated by a number of piloting criteria and that currently available resources limit expansion to additional counties. In a compromise attempt to address this issue, the LGB, BRIDGE-Winrock, and UNDP reached an agreement whereby UNDP would engage in local government capacity building in those counties in which BRIDGE-Winrock was not operating. However, this concern persists and is so widely and strongly held at the state level that it must be addressed in any consideration of program extension or expansion. Limiting implementation to selected counties has the potential to create conflict that overshadows the contributions of the program. Additionally, it has the potential to create tension between the GOSS and USAID. Unfortunately, the issue is likely to become more complex before it is resolved, as state governments are actively lobbying the GOSS for the addition of multiple counties in each state. Recommendations:  Careful consideration should be given to expansion of activities to additional counties based on assessment of program capacity and a formal estimation of costs of expansion.  In any future replication of BRIDGE-Winrock to other states, consideration should be given to a carefully planned rollout of project activities from one county to another, until full coverage is reached. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 10 3.1.2 Validity of Development Hypothesis and Programming Approach Development Hypothesis Evaluation of a program is greatly enabled by a clear statement of the development hypothesis (―causal framework‖ or a ―theory of change‖) that guided initial program design and informs ongoing implementation. Formulating a succinct statement of the development hypothesis underlying the BRIDGE-Winrock program proved to be an unusual challenge for the evaluation team. The challenge centered primarily on the issue of whether the program was designed and/or is managed as an ―integrated‖ program or, alternately, whether the five components are managed as separate elements. The Cooperative Agreement (CA) states only that the components are ―mutually supportive‖ and makes no formal commitment to integration. Soon after the signing of the CA, however, program documents begin to highlight the integrated nature of the program. Indeed, a BRIDGE-Winrock strategic planning workshop held in July–August 2009 argued for the development of indicators that demonstrate the value-added of integration and requiring that team leaders demonstrate how component results add value to each other.13 Documents on Winrock International‘s website regularly promote the BRIDGE￾Winrock program as being integrated. However, in developing the SOW for this midterm evaluation, the issue of whether the program was to be evaluated as an ―integrated program‖ remained elusive, with the Chief of Party (COP) asserting at one point that ―BRIDGE [Winrock] is essentially five projects that enjoy programmatic and operational synergies and cost savings because they share teams, compounds, and costs.‖14 There are multiple reasons for the ambiguity. First, although there has been an evolving commitment to program integration over the course of the first 18 months of implementation, there was no formal effort at ―logical framing‖ of integration at the outset of the program, and work plans have been framed around individual component tasks.15 Second, while there is substantial verbal commitment and reference to ―integration‖ by all staff, this term is understood differently across components and organizational levels, sometimes referring to coordinated activities and sometimes only to integrated goals. Third, sectoral funding provided by other USAID units was treated as a single pot of money by BRIDGE-Winrock, which paradoxically increases the appearance of integration, while at the same time making it exceedingly difficult to analyze the contribution of individual components to integrated outcomes or sectoral objectives. Finally, in the context of an evaluation, analyzing a program as five separate projects could allow management to isolate failures and successes of a program at the level of a project or partner, rather than at the program level. The evaluation team was asked to research the meaning of ―integrated programming‖ within USAID and to examine the literature on integrated rural development in preparation for the evaluation. Upon doing so, the evaluation team ultimately concluded that although integration had not been operationalized, either in the context of program logic (i.e., mapping of inputs/outputs across components for integrated outcomes) or in the broader understanding of the USAID/Sudan mission 13 ―Include Indicators that demonstrate Components really are more than the sum of their parts. Program-wide, a limited number of Integration Indicators should be considered for inclusion in the monitoring process. Component Managers and Team Leaders should be required to demonstrate how Component results add value to each other.‖ (Jeremy Condor. Strategy Review Facilitation Report. July/August 2009, p. 6) 14 Email correspondence with MTE team dated September 22, 2010. 15 Logical framing of integration would, for example, entail outcome mapping that indicated how outputs of one component acted as inputs to another component or contributed additively to joint outcomes. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 11 (e.g., sectoral outcome or efficiency expectations of multi-sectoral funding), there was a clearly stated development hypothesis, asserting in direct accord with USAID strategic objectives that: ―Increasing government legitimacy by improving the perception citizens have of their government‘s ability to meet their needs contributes to peace and stability‖16 This logic has a strong foundation in the literature on democracy and governance in post-conflict settings. Studies indicate that the instability associated with fragile states is the product of ineffective and illegitimate governance.17 In other words, the existence of a legitimate government that effectively provides public goods and services contributes significantly to promoting peace and stability. As such, the approach is well grounded in current governance literature that promotes an integrated strategy, which is:  Local government focused;  Area-based;  Multi-sectoral; and  Community-based. BRIDGE-Winrock sits firmly in that familiar strategic framework. It is, however, less common to see that strategy articulated into the architecture of a single program.18 Programming The clearest articulation of this strategy appeared in early documents built around the results framework diagrammed below: 16 Sudan BRIDGE Cooperative Agreement between USAID and Winrock International (January 22, 2009) 17 USAID, Fragile State Strategy, 2005, p.3. 18 There is a ―grey literature‖ (mostly by UNDP) that discusses efforts to articulate this strategy on the ground, much of it calling for improved operationalization of integration. Operationalizing this strategy on the ground is a challenge for a variety of reasons not the least of which, according to one UNDP analysis, is the risk associated with investment in the high cost of the start-up phase in area-based development. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 12 This results framework was programmed as a series of tasks with associated targets and indicators built around these three results (see Annex 5 for the full results framework as it appeared in the first-year work plan). Formal program documents have increasingly separated the planning and measurement of BRIDGE-Winrock activities into the five components, while discussion of the program alludes increasingly to integration. BRIDGE-Winrock began operations with almost identical program designs and targets for each of the three states (e.g., identical number of Community Action Groups targeted for formation, identical number of agricultural extension workers trained, identical latrine design and construction approaches, etc.). As time has progressed, however, program managers have become increasingly aware of the unique physical, political, and social landscapes of each state. Accordingly, there has been a growing awareness of the need to adapt program activities to the needs and circumstances of the particular area, and this recognition is likely to be reflected to a modest degree in the work plan for the third year of the program. The need for adaptation is most obvious in Unity state. There is no question that Unity state has posed the greatest implementation challenges for BRIDGE-Winrock. The ecology of Unity was described by one staff member as a ―green hell‖ of teeming flora and fauna, with impacts for program activities as well as the health and morale of staff. The black cotton soils in much of Unity state are difficult to traverse in the rainy season, and at the time of the evaluation, Rubkona County had been inaccessible for more than three months. Unity also poses the greatest risk for continuing and emerging conflict. In summary, the program design has a sound logic rooted in a strategic model of community and area￾based local government development. However, the integrated logic was not carefully mapped for implementation purposes. Integration across components was sometimes overstated and generally underspecified. Program targets were replicated across states with little consideration of the need for variation or adaptation. The team concluded it was necessary to evaluate the performance of the five Figure 1: BRIDGE-Winrock Results Framework Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 13 components individually and seek to identify the points of intersection and coordination that add value to achieving the integrated goal of improving government capacity and ability to deliver services as a means to increasing citizens‘ perception of government. Recommendation:  USAID funding of multi-sectoral, area-based programs would be well served by additional and creative efforts at operationalizing integrated program design. This would entail, among other things:  Achieving definitional clarity around programmatic versus strategic integration  Clarifying the degree of programmatic integration necessary to achieve strategically integrated goals  Understanding how component activities contribute separately, but additively, to integrated outcomes  Understanding how component activities can be integrated to achieve more sustainable outcomes  Understanding how risks or failures in one area of activity do or do not threaten results in another area  Understanding and clarifying expectations of the contribution of multi-sectoral, area￾based programs to country-wide sectoral goals However, based on the internationally well-documented challenges of partnership and consortium management, specifying integration in the form of unrealistic integrated monitoring and evaluation indicators or in the form of tightly woven joint work planning is not recommended. Such efforts are highly likely to be bureaucratically burdensome and of limited value, but some improved operationalization of integration seems in order in any future funding of area-based multi-sectoral programs. 3.2 Program Implementation: Effectiveness, Impact and Sustainability In this section, achievements for each of the components are examined separately, followed by a review of integrated efforts and results. This section on program implementation closes with a brief discussion of cost considerations. For each component, the progress made in achieving performance targets is identified. More importantly, the evaluation team examined the quality of implementation of the leading interventions analyzing the particular challenges faced in that component area. Additionally, the team assessed the probable impact of these efforts and the strategies being taken to ensure sustainability. It is very important to report that BRIDGE-Winrock has implemented an impressive number of activities, reporting regularly and reliably on over 35 targeted indicators. The narrative accounts and data presented in those reports clearly indicate, and this evaluation has verified, that BRIDGE-Winrock has been remarkably successful with respect to short-term deliverables. (See Annex 6 for summary tables of those targeted achievements.) It is not useful, however, to present much of this data in the body of this report. For purposes of substantive evaluation, much of the data provides little meaningful information and the presentation of Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 14 achievements against targets can be misleading. The primary task of the evaluation team has been to go beyond those long lists of inputs and outputs and, where possible, analyze the probable outcomes and impacts of these activities. Table 2: Farming as a Business Training, Quarter 3 2010 State Male Trainees Female Trainees NBG 94 152 Warrap 181 71 Unity 126 36 Totals 401 259 For example, Table 2 appears in the Quarter 3 Report (FY 2010). This table implies that 660 individuals, having received training in Farming as a Business (FaaB), are now engaging, or have the capacity to engage, in farming with a ―business sensibility.‖19 These individuals received training in FaaB from 62 trainers, trained earlier by BRIDGE-Winrock (and also counted as an output). In fact, on the ground, the team heard no evidence that FaaB training was being employed actively in farming decisions. Moreover, the consultant‘s report on the training of trainers (TOT) workshop advised that: ―It is the belief of the consultant that the participants who attended this FaaB training workshop will not engage in significant extension of the FaaB concepts without assistance. The participants as a whole did not exhibit a strong comprehension level of the material, and they had a lack of demonstrated ability to lead training sessions during the TOT workshops.‖ The questionable success of the FaaB training effort is not the point here. The point is that the presentation of such data (and there is much data of this sort)—while underscoring BRIDGE-Winrock‘s commendable success in achieving program targets—can easily give a false impression of substantive outcomes when the project may in fact be encountering significant challenges to sustainable impact. This evaluation takes a critical look at the actual experience with a view to probable impact and sustainability. 19 Farming as a Business (FaaB) trains farmers and agricultural extension workers to assign a monetary (or proxy) value to the costs of farming inputs including labor, equipment, seeds, and fertilizer in order to inculcate a sense of business profitability, with a long-run view to market-oriented agricultural development. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 15 3.2.1 Component 1: Democracy and Governance DEMOCRACY AND GOVERNANCE Purpose: Strengthen the capacity of government to deliver services and promote economic growth. Activities: (a) English Language Training; (b) Capacity Building in Accounting, Financial Management, Planning and Budgeting, Human Resource Development and Tax Administration; (c) Policy Dissemination; (d) Tendering and Procurement. Progress against Targets: BRIDGE-Winrock has met targets in six out of seven indicators, reporting for Indicator 1.2 (Number of Sub-National Government Entities Receiving USG Assistance) that over 29 government entities have received assistance to improve operational effectiveness, and for Indicator 1.3 (Number of Individuals Who Received USG-Assisted Training, Including Management Skills and Fiscal Management, to Strengthen Local Government and/or Decentralization) that over 800 individuals have received training. BRIDGE-Winrock has assisted with the production of county plans and budgets in all three states. The stated purpose of the D&G component is to ―strengthen the capacity of government to deliver services and promote economic growth.‖ This purpose is consonant with the overarching objective of the entire program, and as such the D&G unit maintains a strategically dominant role in the program, defining to a large extent the scope of BRIDGE-Winrock‘s involvement at the county and state level. The D&G unit performs the core administrative capacity building function for the program. 20 Initially, this component was led by RTI International (RTI), but the consortium agreement was amended after the first year and now RTI performs a narrower technical advisory function for D&G, concentrating most recently in the area of human resources development (HRD). Institutional Capacity Assessment The work in the D&G area has been informed by an extensive Institutional Capacity Assessment (ICA), the completion of which is a major accomplishment in its own right. The results of the ICA confirmed the near absence of governmental capacity in many critical areas, providing a detailed baseline for continuing articulation of ongoing activities. The strategy for this component has been developed more formally and comprehensively than any of the other components, including the identification of multiple collaborative opportunities with external partners and identification of an evaluation framework. The strategic outline emerging from the governance gap analysis specifies work in four areas:  Financial Management  Planning and Budgeting  Tax Administration  Human Resources Development The bulk of capacity-building activity in the first three of the four thematic areas has been provided through training in classroom and workshop settings at both the state and county level. 20 We make a distinction in this evaluation between administrative and technical capacity building. The D&G unit performs the bulk of administrative capacity building, while the Education, Agriculture, and WASH components for the most part build technical capacity. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 16 Training The evaluation team reviewed training content and delivery and explored outcomes in greater depth than the brief coverage here suggests (see Annex 7 for training assessment guideline). The training materials appear sound, and the finance-related training content well delivered by Tribal Helm.21 English Language Training (ELT) has been conducted by Windle Trust, an internationally respected ELT provider with special expertise in language training in conflict-affected communities. Liaison work with government agencies for the facilitation of training has been professionally managed by BRIDGE￾Winrock. The training is well regarded by partner organizations and was accorded high praise, as is evident in statements such as that from USAID-funded Deloitte-Touche, asserting that in implementing the FMIS, ―…it is not so much that we have benefitted from the training provided by Winrock, but that we depend on it.‖ Very large numbers of staff at the state and local level have received training with BRIDGE-Winrock support—indeed to the point of saturation for basic training in some subject areas. Although, generally speaking, training modules were developed in response to expressed needs of government, there were clearly cases, particularly in initial trainings, where the material exceeded the absorptive capacity of individuals and of particular state ministries. Outcomes of classroom training were limited also in a number of cases where government staff are unable to utilize their training on the job due to the absence of computers, obstruction of supervisors, or other reasons. Despite these relatively common limitations, the evaluation team concluded that training provided by BRIDGE-Winrock is producing positive outcomes. Additionally, there are very few other NGOs providing equally comprehensive training at the state or county level. Training, particularly in financial administration as well as English Language Training, has contributed to improved individual career prospects and is having some scattered but clearly identifiable performance benefits at the organizational level. This conclusion was reached on the basis of multiple and repeated questions exploring the impact of training on individual and unit job performance. While it is difficult to quantify or confirm the results with any degree of rigor, evaluation team members were convinced that the training is contributing to improved administrative capacity. This was most evident in the newly acquired ability to perform often basic, yet critical, financial management tasks. Planning and Budgeting BRIDGE-Winrock has facilitated the production of Draft County Plans and Budgets in all three states, building on a series of earlier trainings and the facilitation of dialogue between communities and local government at the county, payam, and boma levels. These draft budgets will feed into the 2011 state and national budget cycle, pending establishment and approval by legislative councils at the county level and continuing cooperation at the state level, especially at the state Ministry of Finance and Economic Planning (SMOFEP). The production of these draft plans and budgets is the most solidly identifiable outcome of the D&G component as well as a powerful point of intersection across all the other components. Moreover, these draft plans are the quintessential result of a critical process in the 21 Tribal Helm is a regionally based training organization approved by GOSS. A chief complaint about Tribal Helm inside BRIDGE-Winrock and other organizations is that it is reluctant to share the full content of its training materials, which would allow others to determine what additional training might or might not be necessary. Additionally, Tribal Helm is considered to be ―expensive.‖ At the time of this evaluation, BRIDGE-Winrock was actively pursuing alternative options. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 17 democratic alignment of community needs with government financial planning. The production of draft budgets stands in contrast to the absence of any such budgets in those counties in which UNDP was designated to facilitate the planning and budgeting process. Although county administrations remain very dependent on the facilitation efforts of BRIDGE-Winrock to produce these drafts, there was a strong sense of ownership over the plans in almost every county. LG programs typically seek multiple outcomes in planning and budgeting efforts. One set of outcomes is associated with the benefits of participation in terms of democratic citizenship and the improved accountability and transparency afforded by the planning and budgeting process. Another set of outcomes is related to the material benefits of participation reflected in a shift in priorities and resources to the identified needs of poor communities. Like most community-based LG programs, the BRIDGE￾Winrock case provides more evidence of participatory benefits rather than material outcomes. This is particularly true in the BRIDGE-Winrock case where county plans and budgets have been developed, but there are still no assurances that the next step in the process will be completed or funds will be made available to implement plans. Tax Administration BRIDGE-Winrock has provided some training to county and state officials in the proper use of tax forms and recording of tax revenues. This basic training appears to have provided some useful knowledge to individual accountants at the county and state level with some possible, but uncertain, improvements in overall tax administration. This has occurred even though the tax system lacks clarity, exhibits obvious elements of corruption, and remains reliant on outdated (and sometimes non-existent) forms produced in North Sudan. Additionally, BRIDGE-Winrock facilitated a tax administration workshop in Juba to which government officials from all three states were invited in order to learn from NBG‘s experience in the establishment of a State Revenue Authority. NBG‘s approach was described as being more advanced in many aspects of tax administration than the Ministry of Finance (MOF) at the GOSS level. Although basic taxation authority is laid out in the LGA, there is little consensus on how the future tax codes and systems might develop. As such, the evaluation team remains concerned that extensive involvement in tax administration by BRIDGE-Winrock may be premature, particularly in light of almost certain changes to tax administration after the referendum. Human Resources Development Efforts to improve HRD in state and local government has involved material support to the State Ministry of Labor, Public Service and Human Resource Development (SMOLPSHRD) in the form of equipment (there are also plans to assist with office construction); dissemination of public administration policy frameworks; the development of organograms for state ministries; and the creation of job descriptions for civil servants. These activities are being advised and largely conducted by RTI on an intermittent short-term technical assistance (STTA) basis. There was little evidence of any likely impact of HRD activities in the near or long term. These conclusions are based largely on observations in Unity state and could be belied on closer examination of the process in other states. In every state, the SMOLPSHRD appeared the least capable, among all state ministries, of performing its administrative functions, and ministry staff could provide no evidence that training received from BRIDGE-Winrock had been of any individual or organizational value. This ministry was newly created following the April 2010 election and lacks a solid legislative foundation on which to base its work. This could be a strong argument for much-needed capacity building, and some Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 18 capacity building within the ministry itself is perhaps warranted. However, it is unclear that existing legislation provides a sufficient basis on which to build a programmatic effort that reaches outside the SMOLPSHRD. Additionally, struggles for control over civil service administration between the governor‘s office, the SMOFEP and SMOLPSHRD are almost inevitable. Various projects, including Booz & Co‘s electronic payroll implementation project, appear indirectly to be having greater influence over civil service rationalization than is the BRIDGE-Winrock effort. RTI consultants did not appear to be working very closely with D&G staff in the field, nor did RTI staff appear to be as well informed about state and local government relations as might be expected. Finally, this activity seems distant from the service delivery elements of the program both in terms of activities and with regard to the timeframe in which related activities are conducted. It is very difficult to see, for example, how developing job descriptions at the SMOLPSHRD internally or for other ministries will produce results that will strengthen service delivery or strengthen the capacity of communities to work with government in the remaining year of the program, especially when largely managed on an STTA basis. Strategic Risks The initial mandate of the program to focus on local government, especially county-level capacity building, expanded soon after the program was launched to include an increasing number of activities at the state level and some limited involvement in national policy dissemination. The pressure to expand the scope of the D&G unit has continued as the program enters the third year of implementation. This pressure is fostered by the program‘s relative success; by the absence of other area-based administrative capacity building organizations with BRIDGE-Winrock‘s resources and established relationships; and by positive demand of state and local government. The challenge for capacity-building NGOs is to lay the building blocks of a service delivery system that is economically sound, fully owned by government, and likely to gradually scale up government capacity. Accordingly, capacity is built to plan and to monitor and in the longer-term to directly implement service provision. The overall effectiveness of government institutions is, in part, a function of how well they undertake their financial management, planning and budgeting, human resource management, and (in the long run) tax administration operations. Therefore, most LG development projects focus efforts, as does BRIDGE-Winrock, in these same four thematic areas. There is strong systems logic to this design (i.e., government needs a well-defined and skilled civil service sector and financial resources via taxation to properly implement plans and strong financial management to ensure resources are spent according to plans and budgets). However, results from capacity building efforts in HRD and to a lesser extent tax administration, are not likely to develop in the same time frame as other elements, particularly given the embryonic status of both tax administration and labor law in Southern Sudan. The expanding mandate of the D&G component poses some risk to sustainable program outcomes and to the integrated goal of improving the perceptions of government by citizens through the delivery of tangible peace dividends. The vital link between service delivery and capacity building risks becoming increasingly attenuated as involvement expands to more distant and abstract administrative functions. More importantly, an upward expanding mandate risks a shift in focus away from community-based interests towards governmentally defined interests, thereby potentially weakening this critical link between the people and government in the decentralized governance model. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 19 Additionally, a number of governance experts suggest that it is likely that following the referendum there will be a centralizing shift in fiscal and administrative control as the GOSS gains control over additional resources. In such cases, it is the state level functions that are most often squeezed out of the administrative system. Local government remains to deliver services whether those funds are mediated through state administration or central government, and the capacity of local government will continue to require capacity building. Recommendations:  BRIDGE-Winrock should exercise caution with respect to the broadening mandate of the D&G unit given the risks discussed above. Specifically, the evaluation team recommends:  Review of the efficacy of HRD activities with special attention to the integrated and downstream value of these activities in the timeframe shared by service delivery demands and capacity building requirements. Consider narrowing the scope of activities in HRD until such time as the MOLPSHRD and respective labor legislation are better developed.  Focus activities on those functions that will remain operationally viable even in the event of possible centralizing shift in fiscal and public service administration following the referendum. This would include, for example, those county-level administrative and technical functions (e.g., county education supervision) that will continue to exist regardless of whether future funding is received on a conditional sectoral basis or as unrestricted block grants.  Focus tax administration activities on basic principles and operations that would survive substantial changes in the tax code following the referendum.  Strengthen and complement training by increased on-the-job coaching and mentoring. This should not require large numbers of additional staff but would likely require some internal BRIDGE-Winrock staff training. 3.2.2 Component 2: Community Development COMMUNITY DEVELOPMENT Purpose: Strengthen the capacity of communities and government to work together to identify, prioritize, and address community needs Activities: (a) Formation of Community Action Groups (CAGs) and Women‘s Support Groups (WSGs); (b) Facilitation of town hall meetings; (c) Facilitation of community-driven LGDF/micro￾grant projects Progress against Targets: BRIDGE-Winrock has exceeded targets in this component, forming over 150 CAGs and WSGs. CAGS perform a vital integrating function for many of BRIDGE-Winrock‘s activities. Community-based development (CBD) programs and projects are a central feature of local government development models in conflict-affected and post-conflict environments. As stated in a 2007 USAID report: Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 20 ―CBD programs are often the first development programs on the ground in conflict-affected areas. They can reinforce a sense of community, generate an improvement in livelihoods, help improve the community‘s ability to realize its goals, (attempt to) build transparent governance, and empower marginalized groups.22 Moreover, strengthened communities can act as a vitalizing safeguard for decentralization in the face of possible centralizing tendencies that are likely as the GOSS matures. Community Action Groups BRIDGE-Winrock‘s community development efforts have focused heavily on the formation of Community Action Groups (CAGs). BRIDGE-Winrock has assisted in the establishment of over 165 CAGs across the three states. These CAGs sit at the logistical center of BRIDGE-Winrock program architecture and project implementation. CAGs serve as an entry point for a range of BRIDGE￾Winrock interventions (e.g., identification of farmers for agricultural training, identification of sites for borehole construction, and identification of community members for literacy training) and act as a locus for a number of other community-based activities, including sanitation and hygiene training. The formation of CAGs followed standard community mobilization practices in international development, augmented by an ―appreciative inquiry‖ approach.23 CAG members were elected in an open process conducted in close consultation with local chiefs. Local chiefs were described as the primary traditional political and social structural unit, which has survived the many years of civil war. These elections were described in glowing terms by BRIDGE-Winrock staff in all states as an inspiring and empowering exercise in democracy. CAG members were subsequently trained on their roles and responsibilities, the development of bylaws, and meeting management, as well as on other topics such as gender sensitization. The enormous effort involved in forming CAGs de novo raises the question: ―Was it necessary for BRIDGE-Winrock to create CAGs, or were other civil society organizations (CSOs) already in existence that could have been mobilized more quickly?‖ The answer, for the most part, is that there appear to be few if any CSO structures on which to build. In Warrap, some individuals have had experience with community organizations in areas where World Vision has had a strong presence. In Unity, some individuals have been exposed to community-based projects implemented by other NGOs. But in general, there do not appear to have been extant CSOs that could have been readily mobilized. The clearest evidence appears in the ICA report prepared by BRIDGE-Winrock in which state and local government officials were asked if activities of government were conducted in cooperation with local CBOs. The answer was routinely no, with the exception of the DG in the State Ministry of Animals and Fisheries (SMOAF) in NBG who reported that ―years ago there had been active farmer cooperatives with which the SMOAF worked directly, but these cooperatives fell apart during the war years.‖ There has been extreme attention on meeting targets for formation of CAGs. Community Development project officers were added in Unity in recent months, specifically to speed the process of CAG formation. The original plan in all states was to form CAGs in only three payams in each county. In 22 USAID (2007). Community-Based Development in Conflict-Affected Areas. 23 In its broadest focus, appreciative inquiry involves systematic discovery, the art and practice of asking questions that strengthen a system‘s capacity. It is the theory and practice for approaching change from a holistic framework. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 21 Unity, Community Development staff determined that if CAGs were implemented in just three counties and targets were met (and management has insisted that they be met) that this would result in more CAGs in each payam than could be absorbed by population numbers and densities. Accordingly, the BRIDGE-Winrock program is being implemented in all payams in each of the counties. Community Action Plans BRIDGE-Winrock guides CAGs in the development of Community Action Plans (CAPs) through which community needs are identified and prioritized. Community Development officers work closely with CAGs to identify and implement community development activities. There was some evidence that those needs identified by communities are shaped by perceptions of what the BRIDGE-Winrock project can deliver. It was evident in almost all site visits that participatory methods had been used to encourage a sense of ownership among the community members. However, at this stage of the program, there is little evidence to suggest that the participatory approach has led to improved delivery of services or a strong sense of ownership over project results. Engagement with Government BRIDGE-Winrock uses CAGs to strengthen the quality of interaction between communities and local governments in a variety of ways. BRIDGE-Winrock strives to ―put the face of government‖ on all interactions with communities by using, for example, County WASH coordinators to promote hygiene and sanitation awareness or agricultural extension agents to train farmers. BRIDGE-Winrock facilitates town hall meetings bringing payam/boma officials and CAGs together to provide communities the opportunity to be heard, thus reinforcing the participatory processes while increasing the transparency of local government. Equity Considerations CAG members routinely insisted that they represent all community members and disavowed any assertions that they might receive special privileges as members or leaders in a CAG. These protestations were not always convincing as CAG members dismissed the possibility of any differences of opinion between CAG and other community members. The language used to describe community relations sometimes had the ring of the ―participatory‖ jargon likely learned in CAG training. It was also abundantly evident that CAG members received a disproportionate share of training benefits, most apparent in the case of highly coveted English Language Training. There was also some evidence that CAG members and leaders were among the more educated members of their communities, as it is almost inevitable that those who can effectively communicate with outsiders, read documents, and keep accounts and records will assume dominant positions in community organizations. As CAGs mature, it would be wise for BRIDGE-Winrock to monitor patterns of elite dominance. CAGs clearly play a vital role in the implementation of BRIDGE-Winrock activities. CAGs do not yet exhibit independent capacity to perform the broader democratic and economic functions anticipated in most community-driven models of decentralized development. Most importantly with respect to sustainability of BRIDGE-Winrock efforts, it is almost certain that CAGs would not survive the departure of BRIDGE-Winrock anytime in the near future. The limited capacity of CAGS for independent collective action is largely a reflection of the magnitude of the problem, not a weakness of Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 22 BRIDGE-Winrock. As is so often true it will take time and sustained, focused effort to empower communities ravaged by decades of war to engage effectively with a still extremely limited government. Recommendations:  Focus on strengthening already existing CAGs/WSGs rather than facilitating formation of new ones. In doing so, a candid review must be conducted to identify the functionality of current CAGs/WSGs, reconfiguring and/or strengthening groups as needed. Additional CAGs/WSGs should be formed only where a clear need is well established.  Further consideration must be given to the longer-term civic function of CAGs in Sudan‘s decentralized state model. BRIDGE-Winrock needs to clarify its vision for the future role of CAGs and develop clear strategies to realize those functions.  Closer examination of conflict dynamics in beneficiary communities must be conducted both for existing CAGs and as part of determining the location and make-up of future CAGs. Additional attention needs to be given to ensuring the equitable distribution of benefits. 3.2.3 Component 3: Agriculture and Livelihoods AGRICULTURE AND LIVELIHOODS Purpose: Increase food security and promote rapid, broad-based, and self-sustaining economic growth through the development of the agriculture sector and livelihood opportunities Activities: (a) Extension worker training and TOT for FaaB; (b) Transport for extension workers; (c) Vegetable gardens; (d) Gum acacia producer groups; (e) Beekeeping groups; (f) Sorghum production training; (g) Ox-plow training; (h) Blacksmith training; (i) Merchant grants; (j) Fisheries training and technologies; (k) Vocational training Progress against Targets: BRIDGE-Winrock has exceeded targets for its primary indicator (3.1 Number of Individuals Who Have Received USG-Supported Short-Term Agricultural Sector Productivity Training), training more than 3000 individuals in various areas. The project design for this component is based in a strong model for market-based sustainability with which the consortium partner ACDI/VOCA has substantial and successful experience in other countries. However, the enabling environment for agriculture in Southern Sudan is poor from the perspective of both markets and government. Due in large part to the weakness of the enabling environment, but also to poor implementation, very few of the agricultural and livelihood activities are performing as anticipated. Expectations remained high at the time of the evaluation for strong results from ox-plow and crop production training for the upcoming harvest season. As highlighted in the box above, BRIDGE-Winrock has provided extensive training in agricultural methods and other livelihoods areas (Indicator 3.1: Number of Individuals Who Have Received USG￾Supported Short Term Agricultural Sector Productivity Training). Additionally, Indicator 3.2 (Number of Farmers, Processors, and Others Who Have Adopted New Technologies or Management Practices as a Result of USG Assistance) suggests that approximately 30 percent to 40 percent of all trainees have adopted all agricultural technologies on which they have been trained. This result is highly suspect on a Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 23 number of grounds—not least being the unusual uniformity with which the data are reported (see Annex 8). This data was not substantiated on examination in the field. It was abundantly clear that many of these technologies have not, in fact, been adopted nor is there much progress being made for second season impact. Agricultural Extension BRIDGE-Winrock has trained agricultural extension workers in a number of basic agronomic practices as well as FaaB and value chain analysis. Sixty extension workers were identified (20 in each of three states) and provided extended training at the National Agricultural Training Institute at Yei. It was confirmed at state ministries and county commissioner‘s offices and by Winrock staff that only a very small subset of these agricultural extension workers remain actively engaged in extension work. There are multiple reasons for this, including a lack of motivation on the part of extension workers in the absence of regular or assured salaries at the county or state level. Some of these agricultural extension workers have taken their new skills and become gainfully employed elsewhere. Additionally, even where some motivation exists on the part of extension workers, they often have no means of transport to conduct extension activities. BRIDGE-Winrock is well aware of these challenges and has engaged with state ministries to discuss improving incentives for agricultural extension workers. The program has also started to provide some transport in the form of bicycles and motorcycles, but it is too soon to tell whether these are being regularly and usefully employed in extension activities. This is a huge challenge for BRIDGE-Winrock, as a number of the agriculture activities depend on extending technologies through training by extension workers. Vegetable Gardens A very large number of the recipients of agricultural technology listed in Annex 8 are community members who have received training on BRIDGE-Winrock vegetable garden demonstration plots (and been provided with seeds, small tools, and community wheelbarrows). On the basis of discussions with members of over 10 vegetable producer groups, the evaluation team concluded firmly that vegetable projects have failed to produce expected or reported results. Vegetable gardens have been beset by a range of challenges from poor siting (i.e., too much water or too little water) and poor seed germination to devastating pest infestations; yields for the most part were reported to be very low with very modest contributions to household consumption and little or no surplus available for market sales. There have been a few isolated success stories that have been highly promoted in BRIDGE-Winrock reports and in statements such as: ―tomatoes are on the market because of BRIDGE [Winrock].‖ Community members profess great optimism regarding the market potential of vegetable plots, but this hope appeared based more on BRIDGE-Winrock promises (and assumption of continuing free seed supply) than on demonstrated success.24 There were also a number of inherent contradictions in the design and implementation of the projects. Some BRIDGE-Winrock staff members emphasized that these were demonstration plots and the intent is that farmers will adopt the technology, putting it in practice at individual homesteads or other plots. Other staff members clearly conceived of the plots as community projects (within which farmers tend individual rows) with a treasurer selected to manage prospective income from vegetable sales. With a 24 In a somewhat disturbing incident from an evaluation perspective, community members asserted that BRIDGE-Winrock staff members had requested community members to start clearing land for a vegetable garden on the morning of the evaluation team‘s arrival with the promise of seeds in the near future. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 24 small number of individual exceptions, vegetable producer groups have treated vegetable gardens as a community endeavor with requests for shared fencing (to protect vegetables from roaming cattle and goats)—one of the more frequent requests in interviews and the subject of a number of local government development fund (LGDF) grant requests. In Unity state, the evaluation team encountered a handful of successful vegetable producers who had obtained seeds from the BRIDGE-Winrock demonstration plots. On closer examination, following a visit to the bountiful garden of one such vegetable producer, team members learned that Agro-Action, a German NGO, had been active in vegetable garden training in the region for some years, having left Unity only a few months earlier; the particular farmer whose homestead plot the team visited had been the coordinator for Agro-Action. It is imminently sensible for a project to take advantage of existing knowledge in the community to extend its own efforts. It is not clear, however, that BRIDGE-Winrock was aware of the existing knowledge base or had made any direct effort to utilize the expertise in the community. Fisheries BRIDGE-Winrock has supported fishery groups in both Warrap and Unity states, providing fisher folk with basic supply kits (nets, lines) and cooler boxes, along with training in post-catch handling (e.g., drying, salting) intended to teach fisher folk how to prolong the salable life of fish. The training responds to the central problem experienced by fisher folk, which is that fish often rot before they can be delivered to or sold in the market. In a further effort to extend the salable life of fish, BRIDGE￾Winrock supplied two fishery groups with gasoline-fueled fish drying machines. The project assumed that improved sales would provide the funds to sustain use of the fish dryers. In fact, neither of these is currently operating, and the drying machines are sitting idle at the SMOAF. Only one of the fishery groups visited by the evaluation team appeared to be thriving economically: that group was located in Mangga, Unity state in an area that is readily and regularly accessed by northern traders. Other fisheries in Unity state have not met the same level of success due largely to the difficulties of bringing fresh or dried fish to markets. Staff members discussed the possibility of identifying local merchants who might be supplied with a truck or other means of transport for the delivery of fish to local markets. While the idea of developing local traders addresses a central economic problem, the history of success for such agribusiness endeavors is limited, especially in areas such as Southern Sudan where there appears to be limited local entrepreneurial capacity.25 Externally identified and inexperienced local merchants rarely exhibit the entrepreneurial profile necessary to take on the capital risks regularly assumed by experienced middlemen in the region. Beekeeping and Gum Acacia BRIDGE-Winrock obtained the services of a consultant in March 2010 to provide training in beekeeping, including in the construction of wooden beehive boxes. The same consultant provided training in the harvesting of gum acacia, training the farmers primarily in the identification of additional tree species from which gum acacia can be extracted. Investment in both of these livelihoods activities is 25 USAID has invested in innovate agribusiness approaches for improving farm-to-market access for smallholder farmers and supported successful development of a range of non-traditional agricultural enterprises in a number of developing countries. Similarly, ACDI/VOCA and other agencies have successfully identified critical leverage points in local and global supply chains that can be developed to improve outcomes for smallholder farmers. One of the groups most resistant to change or substitution in most developing country supply chains have been local regional middlemen, particularly those who provide critical transport functions. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 25 premised in the strong market potential for these products, although the evidence for that potential appeared largely anecdotal and not always consistent. Both activities have been very positively reported in program documents, even though limited activity has resulted from the training provided in April 2010. Both the colonization of beehives and the harvesting of high-quality gum occur during the dry season, which had not yet begun at the time of the evaluation. The need to improve the seasonal timing of all BRIDGE-Winrock interventions was well appreciated by program management across a wide range of activities, particularly in agriculture. Vocational Training BRIDGE-Winrock has expanded its focus in this component to include a greater emphasis on livelihood opportunities beyond agriculture. At the time of the MTE, the program had recently identified community members for vocational training in masonry, tailoring, carpentry, and other skills. These courses were just beginning at the time of the midterm evaluation, and it is too soon to reach solid conclusions. Crop Production BRIDGE-Winrock has high expectations for improved harvests in late 2010 following sorghum and maize production training, provision of seed kits, and planned training in post-harvest handling. These training efforts build on an innovative project to increase the sustainable use of ox-plows. Sustained use of ox-plows has the potential for greatly expanding the area of land under cultivation. Yet, the adoption of ox-plow technology has been poor in Southern Sudan despite similar efforts in the past. Such efforts have failed, in part, because there is considerable cultural resistance to the practice of animal traction, but also because there has been no commercially sustainable source of ox-plows and system for repair of ox-plows. In an effort to address the issue, BRIDGE-Winrock has introduced a voucher-based method for the subsidized purchase of ox-plows, intended to sustain continuing production of ox-plows. Additionally, BRIDGE-Winrock has trained more than 20 blacksmiths in the repair of ox-plows. The evaluation team asked to meet with some blacksmiths while in Warrap; only one of the original trainees was found in the town and he had not been active as a blacksmith for four months, and the remaining four blacksmiths trained in Warrap had moved to Wau. The team did not have the opportunity to similarly track blacksmiths in other states, although the team did speak to a few farmers in NBG whose ox-plows were already in disrepair. Still, a number of farmers have eagerly adopted ox plow technology, and it is expected that yields will increase by as much as 25 percent among those farmers. Although there are reasons to be optimistic about production increases in this first year of crop training, the truer test will come in subsequent years. Success seems far from assured. As the descriptions above indicate, sustained impact of current activities is not likely without substantial improvements to the design and implementation of the agriculture and livelihoods component. Recommendations:  Undertake a more thorough evaluation of this component than was possible in this multi￾sectoral evaluation, with close attention being given to the quality of implementation.  Reconsider the feasibility of a number of activities, particularly those that are dependent on the identification and training of vendors and manufacturers who are lacking in any prior market experience.  Increase efforts to strengthen intermediate links in value chain (e.g., blacksmith training). Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 26  Substantively redesign vegetable garden activities with application of lessons learned from prior experience of German Agro-Action as well as BRIDGE-Winrock‘s own experience. 3.2.4 Component 4: Water, Sanitation and Hygiene (WASH) WATER, SANITATION, AND HYGIENE PROMOTION Purpose: Increase access to water and sanitation. Activities: (a) New and rehabilitated water points; (b) Formation and training of Water Management Committees (WMCs); (c) Hygiene education; (d) School latrines; (e) Household latrines; (f) Spare parts supply; (g) Building government capacity to monitor rural water Progress against Targets: BRIDGE-Winrock has met most targets in this component, with the exception of targets for household latrines. The program has established or repaired 167 water points, providing an estimated 83,500 people with access to improved drinking water. The water, sanitation, and hygiene (WASH) component contributes the most tangible peace dividends of the BRIDGE-Winrock program, meeting the most universally and clearly identified need of communities— namely, access to water. Furthermore, the program appears to be implementing this activity in a manner that has potential for sustained impact. The WASH component has been more successful in increasing access to water than it has been in improving sanitation services and practices. Access to Clean Water As highlighted in the box above, BRIDGE-Winrock has established or repaired 167 water points, providing an estimated 83,500 people with access to improved drinking water. The estimate of people served is calculated (as is standard practice in WASH programming) by multiplying the number of water points by the standard ―load‖ of 500 persons per water point. In fact, the number of users may be far greater at some water points. Impact is assumed in part from consistent anecdotal reports from community members who attest to improved health, but is based also on established science, which permits the relatively conservative conclusion that access to clean water alone reduces diarrheal disease by approximately 15 to 20 percent.26 No data were available to confirm other benefits typically ascribed to improved access to water (e.g., decreased time involved in collection of water by girls and women); such benefits can be assumed, although these were less obvious at the end of the rainy season than they might be at another time of year. 26 The evaluation team understands the complexity of the literature on the relative impact of different WASH interventions, both singly and in combination. See, for example, International Initiative for impact Evaluation‘s August 2009 review of WASH impact entitled ―Water Sanitation and Hygiene Interventions to Combat Childhood Diarrhoea in Developing Countries‖ compiled by Hugh Waddington, Birte Sniltsveit, Howard White and Lorna Fewtrell. This synthetic review provides a thorough examination of a series of earlier meta-analyses of WASH interventions. At risk of oversimplification, this review confirms the greater impact and cost-effectiveness of a number of interventions (i.e., hand-washing and certain point-of-use (POU) water quality management approaches) over construction and rehabilitation of water points, especially in the absence of special attention to water quality maintenance from source to POU. However, this same study and others support the conservative conclusion that access to clean water will reduce diarrheal disease by approximately 15 to 20 percent, even in the absence of additional interventions. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 27 Potential sustainability derives from five program elements: (1) Establishment of Water Management Committees (WMC). WWMCs have been formed at the community level to monitor each newly constructed or rehabilitated borehole. Committee members appeared well trained, and all water points visited were clean and well maintained. WMCs in most communities had successfully collected funds from users for maintenance and repair. (2) Training of Pump Mechanics. BRIDGE-Winrock has trained more than 40 government￾employed pump mechanics who can be contacted by communities in the event that repairs are needed. The team did not investigate the capacity or availability of pump mechanics, but it is a critical link in the effort to sustain pumps in good working order. (3) Development of a Fee-Based Parts Supply System. UNICEF has provided spare parts for borehole maintenance free of charge throughout the civil war period and since the signing of the CPA. Most experts agree that this is not a sustainable approach. BRIDGE-Winrock has encouraged the introduction of a more sustainable distribution system, opening discussions with state governments and UNICEF to initiate a fee-based supply of parts for water point maintenance. To date, this effort has resulted in the acquisition of one warehousing container, and the system is not yet operable. It is not yet clear that this laudable effort for sustainable supply will succeed. (4) Technical capacity building at the State Ministries of Physical Infrastructure (Rural Water Department). BRIDGE-Winrock has engaged in a number of capacity building activities (e.g., database management for tracking and maintenance of water points, procurement, and contract administration) that will contribute to improved management of water supply and sanitation services at the state level if adequate operating support is maintained going forward. (5) Technical capacity building of County WASH Supervisors/Coordinators. County WASH coordinators have been trained by BRIDGE-Winrock and facilitated in the delivery of sanitation and hygiene messaging to communities. The effectiveness of the training is questionable, but it is a step in the direction of sustainable government delivery of basic services. Results from these efforts are difficult to verify, and some are likely to devolve, but all of the above are important elements for sustainable clean water supply. Access to Sanitation Less than seven percent of Southern Sudanese citizens use improved sanitation facilities.27 BRIDGE￾Winrock prepared to address this enormous gap with extremely ambitious targets of 300 household latrines in each state for 2010, with an additional 700 across all three states in 2011. At the time of the evaluation, only a few dozen household latrines had been installed, although good progress had been made on school-based latrines. There are at least three possible explanations for the delay in installation of household latrines. First, PACT Sudan failed to fulfill its obligations as a member of the consortium. Second, households and communities were not interested in, nor amenable to, using household latrines. Third, delays were 27 Southern Sudan Statistical Yearbook, 2009. Available at: http://ssccse.org/statistical-year-book/ Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 28 caused by technical challenges associated with particular soil formations and characteristics of the surrounding water table. Somewhat surprising has been the unanticipated technical challenges, experienced most acutely in Unity state, where (during the rainy season) seepage from latrines threatens contamination of the water supply. PACT Sudan and a number of other relief NGOs have substantial experience in WASH in Southern Sudan and some of the technical challenges should have been anticipated. Not surprising at all is the resistance of communities to the use of household latrines. Numerous assessments of the water and sanitation situation in Southern Sudan have concluded that cultural barriers to the use of latrines are formidable, and that extensive sanitation awareness and promotion efforts will be required to create demand for household latrines. Additionally, although GOSS has embraced the broad WASH framework promoted for a decade or more by UNICEF and WHO, they have not articulated a strategy for promotion of sanitation. BRIDGE-Winrock staff asserted that an all-out effort on both the supply and demand side of the sanitation equation will allow them to meet stated targets. Departing PACT Sudan staff were skeptical, stating that, while they are confident Winrock could hire contractors to successfully install 1000 latrines, they were equally convinced that the effort would be wasted, and that a visit in a year‘s time would find that these household latrines were not being used. PACT‘s position is well supported by their prior experience in Sudan, particularly their experience with Sudan‘s Water for Recovery and Peace Program (WRAPP). A 2008 external evaluation of WRAPP concluded that the planned implementation of 100 latrines would ―not bear any considerable impact‖ as ―it did not correspond with current international experience that sanitation and hygiene interventions are only successful if practiced by at least 80 percent of the population.‖ That evaluation recommended that PACT consider the construction of low-cost demonstration models of latrines that can realistically be replicated by individual households. Recommendations:  Continue strong focus on the sustainability of WASH efforts, taking care to ensure that those activities produce the intended outcomes (e.g., fee-based supply, accessibility of pump mechanics).  Consideration must be given to revising targets for household latrine implementation while laying improved groundwork on the demand side of sanitation and addressing substantial technical challenges.  Conduct K-A-P survey 28of community members who received WASH training in 2009 to ascertain the efficacy of earlier activities and the sustainability of outcomes. 28 A K-A-P Survey is one that looks at the target populations knowledge, attitudes and practices relating to intended behavior change. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 29 3.2.5 Component 5: Education EDUCATION Purpose: Expand access to quality education Activities: (a) Education management training; (b) ELT for teachers; (c) In-service teacher training; (d) Scholarships for female teachers; (e) Material support for schools; (f) Bicycles and motorcycles; (g) PTA Training; (h) Adult literacy Progress against Targets: Progress has been slow against targets in the education component, especially in the first year of operations (2009), but BRIDGE-Winrock has been gaining ground in the second year, and in-service teacher training efforts could begin to produce results in schools in the final year of the program. The education component conducts activities in two broad focus areas: strengthening government capacity to provide quality education and strengthening the role of communities in support of education. The needs of government and communities for educational support are enormous, and while BRIDGE￾Winrock is making very positive contributions to filling those needs, it was difficult to reach firm conclusions about likely impact. It was similarly difficult to discern strategies being used to ensure sustainability beyond the inherent commitment to close cooperation with government. The strategic clarity implied by the two focus areas is somewhat belied by a lack of programmatic coherence on the ground. Broad Range of Valuable Activities Priority activities are, in principle, teacher training, administrative capacity building and the revitalization and training of PTAs. Activities are, in theory, directed towards eight schools identified for support in each county. In practice, the education component engages in a broad range of activities—some on a regular basis, others as one-off responses to requests from state ministries of education. Many of these activities are indeed focused on the eight BRIDGE-Winrock supported schools in each county, while others are directed more broadly at the state level. These activities have included, for example:  Renovation and provision of dormitories at a vocational school  Adult mother-tongue literacy training  Provision of two-year teacher certification scholarships for six female teachers  Provision of blackboards and other equipment  Facilitation of teacher competency examinations for 1400 teachers in Unity state  Training of County Education Directors, Payam Education Supervisors, County Education Inspectors, Head Teachers, Deputies, and School Inspectors in education management  Printing of textbooks  Transportation for the distribution of science lab equipment  School-based sanitation and hygiene training  Facilitation of and participation in Girl‘s Education Day Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 30  Outreach to cattle camp schools  Provision of bicycles and motorcycles to county and payam education officials, including head teachers Most of these initiatives have discrete value and are deeply appreciated by state ministries of education and county education staff. Some are important precisely because they cement relationships with state and county education administrators. Others, such as the provision of bicycles and motorcycles, appear promising, contributing to improved supervision of distant schools and educational facilities. The BRIDGE-Winrock program undertakes all of its education activities in close collaboration with state and country education offices and in strict adherence to national curriculum standards. Still, the range of activities prompts questions about strategic fit and the comparative advantage of BRIDGE-Winrock‘s participation in conducting some of these activities. Teacher Training The two major teacher-training activities are English Language Training (a vital competency in support of the English-based national curriculum) and a phased series of activities directed towards in-service teacher training. ELT is well underway and is credited with identifiable improvements in teachers‘ ability to deliver the national curriculum. In-service teacher training was entering its second phase during the time of the evaluation. In-service teacher training began in the first year with assistance from BRIDGE￾Winrock for completion of a national curriculum for in-service teacher training. Curriculum development was followed by the training of ―tutors,‖ who in turn trained 63 TOTs. These 63 TOTs were just beginning in-service training of primary school teachers at the time of the evaluation and a number of questions remain about rollout of the in-service teacher training efforts. There were no plans discussed for measuring outcomes from in-service teacher training. Parent Teacher Associations PTAs are pivotal structures in Southern Sudan‘s education strategy, and the roles and responsibilities of PTAs are well-defined. Building on this structural foundation, BRIDGE-Winrock has provided training and support to more than 67 PTAs, with a target of 143 PTAs by the end of the project. Parent members of PTAs reported active involvement in school administration and mobilization of community support. Teacher members of PTAs reported on valuable communication with parents through PTAs. County Education Officers reported that schoolyards and school latrines were cleaner in those schools supported by PTAs trained by BRIDGE-Winrock. Program staff reported positively on the active participation of PTAs in improving school function. Experience and knowledge gained by membership in PTAs has value that is likely to survive beyond BRIDGE-Winrock support, extended and sustained by government‘s structural commitment to PTAs. In short, BRIDGE-Winrock‘s support to PTAs appears, by all accounts, to be producing positive results. Administrative Capacity Building BRIDGE-Winrock assists with building administrative capacity for education in a variety of ways, primarily through training and workshops targeting a wide range of educational officers, including county education directors, payam education supervisors, gender focal officers, head teachers, school board members, and others. Training has focused on planning and management, data collection, recordkeeping, report writing, and other topics based on curriculum and guidelines developed by the Ministry of Education, Science, and Technology. BRIDGE-Winrock receives high praise for their Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 31 support in this area, and MOEST officials insisted that there were no other agencies, outside of UNICEF, providing such strong support to the ministry. School Construction Over time, BRIDGE-Winrock has also become increasingly involved in the rehabilitation and expansion of school buildings and classrooms under continuing pressure from State Ministries of Education and in response to ardent requests of communities. BRIDGE-Winrock has targeted the rehabilitation of 60 classrooms in the second year, and an additional 60 classrooms for the final year of the program. Multiple LGDF proposals for school construction are now in process, although very few had been completed at the time of the evaluation (see Annex 9). The costs of support to school construction are extremely high. For example, the cost of a proposed secondary school construction project in Guit County, Unity is estimated at $132,000; the cost of a classroom renovation project at a Girl‘s Primary School in Gogrial West County, Warrap is estimated at $123,000; and the cost of roofing alone at one school in Gogrial West is estimated at $41,920. This prompts strategic questions about alternative use of these funds for teacher training or other activities that might produce more sustainable outcomes at a lower cost.29 There was little discussion about the costs of maintaining these schools and classrooms in functional condition over the long term, either among government officials or BRIDGE-Wirnock staff. There is significant risk in assuming that these schools will be well maintained either by government or communities. BRIDGE-Winrock may wish to revisit the sound reasoning that informed the initial decision to prioritize teacher quality over physical support to schools. Difficult to Determine Probable Impact It is difficult to map the connection between the many activities and assess probable impact on children, targeted schools, regional education outcomes, or sectoral education priorities. In the absence of detailed component budgets, it is also difficult to determine which activities are receiving the largest financial or staff support. This evaluation challenge is compounded by the absence of meaningful indicators for this component. BRIDGE-Winrock records data on school enrollment for girls and boys in the relevant counties as a proxy indicator for improved access to quality education. This is not a useful indicator of BRIDGE-Winrock contributions. Demand for education is extremely high, and school enrollments are increasing rapidly independent of any activities by BRIDGE-Winrock or government. The landscape in Warrap, for example, is dotted with the brick walls of school buildings constructed by communities in eager, and perhaps naïve, anticipation of government or NGOs providing roofs and assigning teachers. Meanwhile, large numbers of children continue to study on rough benches under the shade of large trees. More meaningful indicators that measure, for example, student attendance or student learning outcomes are difficult to gather, and it is likely too early in the overall development of the education sector to expect significant results. Still, some consideration needs to be given to developing better indicators going forward. There is some hope that implementation of a national EMIS will improve the availability of data for such assessments in the future, and BRIDGE-Winrock is encouraged to think more carefully about how that data might be used. Recommendations:  Conduct an internal review of the education component with a view to improved programmatic coherence across activities. This strategic review should ask, among other things. 29 Costs of teacher training are assumed to be much lower, but those financial figures are not available in BRIDGE‘s financial statements. The cost of school construction is known, as they are financed through the LGDF process. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 32  Does this or will this activity contribute to improving educational outcomes in the counties in which BRIDGE-Winrock implements its program?  Does BRIDGE-Winrock have a comparative advantage in performing this activity? Would this activity be more effectively implemented by another program or project?  Are there activities that promise more sustainable impact for lower costs?  Where does this activity sit on outcome pathway to improved learning outcomes for children? How might that be measured?  Identify indicators other than school enrollment that could be used in the future either as direct or proxy measures of improved access to education. 3.2.6 Benefits and Tensions of Integrated Goals and Activities In this section, significant points of integration across components that clearly add value to the program are identified. Additionally, points where increasing separation across components could compromise integrated goals are discussed. As noted earlier, there was no formal effort to promote program integration or measure the value-added of integration at the start of the program. However, over time, BRIDGE-Winrock staff and documents have increasingly identified the program as ―integrated,‖ and, according to field staff, significant attention in meetings is devoted to ―improving integration.‖ Benefits There are many points of intersection or integration that clearly add value to the BRIDGE-Winrock program. BRIDGE-Winrock provides the following example in the 2010 second-quarter report: ―In an excellent example of program integration across components and collaboration with the State and Local Government, the Team Leader, Deputy Team Leader, Sector Specialists, payam administrators, and local governmental representatives from sector ministries visited candidate payams to determine fit for Local Government and BRIDGE [Winrock] objectives across all five program components.‖ Some examples appear relatively simple, such as the decision to channel all communications with CAGs through a Community Development officer in order to better manage community expectations. Early in implementation it was recognized that in the context of the extreme and pronounced ―relief mentality‖ exhibited in many communities, visits from multiple BRIDGE-Winrock staff members from each of the components led to exaggerated expectations. This coordination of multi-sectoral interventions and concomitant influence on citizens‘ perceptions is extremely important, and the value should not be underestimated. Other straightforward examples of integration are those like school gardens or vocational training facilitation that are implemented by the joint efforts of two or more components, education and agriculture in these two examples. There are, of course, cost advantages of sharing vehicles and M&E staff that are difficult to estimate, but that are certainly not trivial. The clearest benefits result from activities that serve important integrated goals, namely those which: (1) Connect communities with local and state government; and/or (2) Connect capacity building efforts with service delivery. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 33 There are a number of program efforts through which these connections are forged—almost all require the integrated efforts of the Community Development and D&G components in coordination with sectoral components. The most obvious effort is the use of CAGs as the locus for facilitating interaction between communities and local government. In the process, LG gains understanding of constituents‘ priorities and communities learn to negotiate for scarce resources. The interaction between communities and government has been most concretely facilitated through the county planning and budgeting process (discussed earlier), which required the coordinated effort of D&G and Community Development staff. Local Government Development Fund There is another significant activity—the Local Government Development Fund (LGDF)—that both connects communities with local government and very directly builds capacity in the process of service delivery improvements. Implementation of the activity requires the integrated efforts of BRIDGE￾Winrock staff across components. The LGDF (sometimes referred to as the micro-grants program) finances local projects, initiated either by CAGs or government. BRIDGE-Winrock has allocated $3.76 million over the life of the program to finance these local initiatives. Although the LGDF funds originate from BRIDGE-Winrock, the process is designed to appear as if administered by government, thereby both building government capacity and improving the perception of government‘s capacity to deliver services. The most obvious capacity building gains from LGDF have occurred as a result of the tender and procurement standards that BRIDGE-Winrock seeks to inculcate through the LGDF process. It was evident both at State Education Ministries and Departments of Rural Water that the process was instrumental in enhancing the contract administration capacity of individual staff and departments. BRIDGE-Winrock adheres strictly to the existing procurement regulations of GOSS as it models professional standards. BRIDGE-Winrock sectoral program staff work closely with government officials on a committee of five persons—helping develop selection criteria, reviewing bids, etc.—but BRIDGE￾Winrock staff do not participate in the final selection of the grants. BRIDGE-Winrock releases the funds to the state ministry upon the execution of contracts administered by government. The importance of the LGDF process has been somewhat compromised by the weak implementation of this activity to date. BRIDGE-Winrock has an enormous backlog of LGDF proposals in various stages of approval and implementation. Community members complain that months go by without any communication about proposals in which they have invested great hopes and significant time. The backlog is due to a number of issues, including an attenuated proposal preparation and approval process. BRIDGE-Winrock has a two-stage approval process, which importantly assesses the viability and sustainability of projects. Project proposals require an in-kind or direct financial contribution by communities and/or government, which can take time to negotiate. BRIDGE-Winrock approval is followed by USAID review and approval of accompanying Environmental Management and Mitigation Plans (EMMP). BRIDGE-Winrock is making a concerted effort to manage the backlog and rationalize the entire LGDF process with the recent hire of a new senior program officer member to manage this process. BRIDGE-Winrock has generated over 119 LGDF grants funding a range of activities from school construction to bicycle purchases (see Annex 9 for a list of approved LGDF projects). The following table provides a summary of LGDF projects approved through the third quarter of 2010. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 34 Table 3: Summary of LGDF Projects, Third Quarter 2010 Approved/ Identified In Progress Completed Total Number of Projects 36 53 30 119 Cost (BRIDGE-Winrock) $697,008 $698,051 $352,141 $1,747,200 In-Kind Contribution $126,881 $76,625 $110,007 $313,513 Tensions There are also some aspects of the program in which increasing separation between components(as they are implemented) may compromise BRIDGE-Winrock‘s ability to achieve its goals. The tension between ‗building the capacity to deliver services‘ and ‗ensuring that services are effectively delivered‘ is a common tension in all capacity-building programs. BRIDGE-Winrock is deeply committed to working with and through local government to deliver services and build capacity. Staff regularly discuss efforts to ―put the face of government‖ on all activities. Indeed this dedication to working with and through government is a notable hallmark of the program. The level of commitment to government capacity building, expressed and actual, is remarkable in Southern Sudan as well as in comparison with international development projects in many other countries. Program managers consistently and repeatedly describe BRIDGE-Winrock, sometimes quite insistently so, as a ―governance project.‖ Staff throughout the organization echo this conceptualization, frequently emphasizing that BRIDGE [Winrock] is first and foremost a ―capacity building project.‖ This conceptualization of BRIDGE-Winrock focuses organizational attention on capacity building at the possible expense of attention to the service delivery components. There is not strong evidence that service provision has been compromised by this focal lens, but there is a clear indication that the quality of service delivery interventions has received less managerial and strategic attention than have D&G capacity building efforts. The challenge going forward is to maintain an appropriate balance, ensuring that all capacity building efforts have clear administrative and technical links to downstream service delivery efforts within an integrated time frame. 3.2.7 Cost Effectiveness The evaluation team was asked to look at the question of ―value for money.‖ No matter how successful the program has been in terms of targets achieved, two questions always remain:  ―Were these achievements worth the money spent?‖  ―Are there better alternatives for the use of these funds?‖ These are always difficult questions to answer, particularly in the absence of a rigorous monitoring and evaluation system or clear methodology for assigning value to nascent outcomes at the midterm point in Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 35 a program.30 In the case of BRIDGE-Winrock, cost analysis proved particularly challenging, as the Cooperative Agreement requires virtually no financial reporting at any level of useful analytic detail. BRIDGE-Winrock is not required to report expenditures on a sectoral basis, a process that, if adopted, might provide some minimal ability to assign costs to component activities.31 Additionally, a true cost effectiveness analysis would require a better understanding of the opportunity costs of these funds in Southern Sudan; no such analysis was readily possible either as a matter of strategic discussion or in terms of specific program alternatives. The expenditure of $53 million is undeniably expensive, or as one USAID official asserted, ―hideously expensive.‖ Prima facie, this would appear to be the case but, in fact, the costs may not be out of alignment with similar programs. The only other program of comparable range and content in Southern Sudan is the BRIDGE-Mercy Corps program, which was funded at an almost identical level. According to anecdotal reports, BRIDGE-Mercy Corps has encountered substantial challenges in implementation and therefore a comparison is not currently possible except at the crudest of levels. This exceedingly crude analysis implies that BRIDGE-Winrock is likely to prove to be more cost effective than its BRIDGE counterpart, although such a conclusion can only be confirmed following a formal evaluation of BRIDGE-Mercy Corps. The evaluation team continued to seek some rough method to better understand the cost structure of BRIDGE-Winrock relative to the program outputs. The team made some limited inquiries about two other projects that also engage in substantive capacity building efforts at the state level in Southern Sudan: Deloitte‘s computerized Financial Management Information System (FMIS) project and Booz & Co.‘s Electronic Payroll implementation project. By most accounts, Booz & Co.‘s project is operated very effectively and efficiently. One senior observer, however, asserted that Booz & Co.‘s capacity to deploy and motivate staff with limited operating budgets is unique and is not comparable to BRIDGE￾Winrock or most USAID projects. The anecdotal information received regarding Deloitte was not sufficiently reliable to report with any certainty, but suggests that BRIDGE-Winrock‘s expenditures on training and workshops have been both more effective and efficient. The team identified a few projects outside Sudan that might offer some insight into the program‘s relative cost-effectiveness. Chief among these was USAID‘s ―Local Governance and Community Development (LGCD) Project in Southern and Eastern Regions of Afghanistan.‖ Although the context is vastly different, the programs were remarkably similar in concept and core design features.32 The Afghan LGCD project was awarded $95 million in October 2006 for a three-year period (contract ceiling was increased to $164 million in November 2007 and later extended through 2011). There are 30 Cost-benefit (CB) analyses generally assign cost-benefit values at an economy-wide level, which is premature in the Southern Sudan context. Additionally, such CB analyses often value training as an externality, which is not consonant with the unit-based capacity building evaluation criteria being promoted as appropriate for current capacity-building efforts in Southern Sudan. Cost-effectiveness analysis requires a sound comparison with similar efforts or comparison with alternative uses of funds. 31 Efforts to obtain more detailed financial data were denied by the implementing partner, and in the absence of a sectoral reporting requirement, USAID chose not to explore the issue at any level of further detail. 32 The program was designed to build the capacity of local governments, both through direct capacity building and by providing opportunities to engage the community and local government to work together for the implementation of community development projects. The program‘s overarching aims are to strengthen the legitimacy of the government of Afghanistan, increase constituent confidence in the government, and promote stability. The LGCD program has four components, identified in the Task Order as: i) strengthening local governance; ii) promoting community development; iii) implementing local stability initiatives; and iv) providing support to the Provincial Reconstruction Teams. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 36 clearly significant differences in budget lines, including substantial start-up analysis costs and construction commitments not found in BRIDGE-Winrock. According to the 2009 midterm evaluation of the Afghan LGCD, progress was limited, and yet expenditures had exceeded $50 million. BRIDGE￾Winrock‘s foundational achievements look comparatively strong and, within the limitations of this comparison, therefore reasonably cost-effective. Another source of comparison is a range of UNDP LG development projects in a number of conflict￾affected areas (e.g., Afghanistan, Bosnia and Herzegovina, Democratic Republic of Congo, Iraq, Palestinian Territory, Sierra Leone, Timor-Leste). These projects were funded at much lower levels than BRIDGE-Winrock, but have also produced somewhat ambiguous outcomes. UNDP analyses and evaluations suggest a reluctance and/or capacity to take on the risk of high-cost investment in area￾based development. These UNDP comparisons provide limited insight into the cost-effectiveness of BRIDGE-Winrock, but do suggest that is unlikely that BRIDGE-Winrock can justify the capital and operational costs of establishing and managing area-based compounds for capacity building purposes alone. For this and other reasons, it is therefore incumbent upon BRIDGE-Winrock to maintain a balanced focus on both capacity building and service delivery Recommendations:  USAID should require implementing agencies of LG development projects to provide financial data that would allow analyses of:  Respective costs of service delivery and capacity building activities. Such cost assignments (especially those with high salary structures) are never perfect, but even with +/-20 percent error in assignment of costs the information is exceedingly valuable for future programming decision.  Sectoral analysis of expenditures  USAID should consider funding a comparative evaluation or comparative case study of BRIDGE-Winrock and BRIDGE-Mercy Corps in addition to or in conjunction with the midterm evaluation of BRIDGE-Mercy Corps. 3.3 Program Management 3.3.1 Organizational Assessment BRIDGE-Winrock is a well-managed and disciplined organization, committed to delivering results.33 The level of accomplishment in terms of quality staffing and the establishment of sound organizational structures and systems (including well-functioning logistical support at compounds in each of the states) is remarkable. Early delays experienced in hiring and equipping of offices appear within the normal 33 Conclusions from a 2009 strategy document: ―Require senior staff to take a firm corporate line on the program purpose and the interdependence of program Components. The leadership should, conditional on the support of home office, remove those senior staff and managers who are unwilling or unable to deliver, or who fail to demonstrate a high level of commitment to the program and its strategic purpose.‖ Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 37 boundaries of operating in the Southern Sudanese context. Indeed, in comparison to some other projects, start-up could be judged as fast—with the program reported as fully operational in NBG and Warrap within the first year and by the end of the first quarter of the second year in Unity. Shared Purpose and Strong Leadership Conversations with staff (with the instructive exception of the most recent hires) were unusually consistent, seeming almost scripted at times. Even these newest staff members were more cautious in sharing their opinions than is the case in many organizations. Although, it is clear that there was an established ―corporate message,‖ individuals conveyed remarkable conviction in their comments. The staff shares a strong commitment to capacity building, delivering on targets as specified in the work plan, capacity building, working through local government; and, albeit with less clarity, to ―integrating‖ efforts across components. Staff at all levels report excellent communication between staff in the field and in Juba; and between Juba and U.S. headquarters. Staff in the field praise the responsiveness of management with considerable respect expressed for management‘s decisions. The sense of shared purpose was profound, owing a great deal to strong leadership and disciplined management. The risk in such a disciplined approach is that at times the program can become indicator-driven at the expense of flexibility in implementation. Commitment to meeting targets in the face of questionable effectiveness has been evident in vegetable garden activities, CAG formation, and household latrine construction. Competent Staff and Rational Structure. BRIDGE-Winrock management expresses concern regularly about the challenges of hiring qualified staff and coping with high turnover, especially for field-based assignments. BRIDGE-Winrock has not yet reached its ideal staff levels, currently employing 160 individuals with an employee roster of approximately 190 planned for the third year of operations. This is, by most measures, a large number of direct hires and reflects a preference for direct control over implementation and a dearth of CBOs or other private sector providers capable of providing equivalent services. Turnover is indeed high by international standards, but is relatively normal in the Sudanese context. And while operations would benefit from lower turnover and higher skill levels, the evaluation team did not feel results were compromised in any significant way by these challenges. A unified and qualified senior leadership team based in Juba has worked together since the program‘s inception, providing consistent vision and management. Transition from one Chief of Party to another appeared to be proceeding smoothly (the former Deputy COP had been promoted on the planned departure of the former COP). Team Leaders in the field have experienced high turnover due to the extraordinary demands of the job and challenges of living circumstances, and retaining qualified team leaders will continue to be a challenge. As compounds become increasingly well established and operating systems more defined, it may be possible to delegate some tasks to compound managers, freeing team leaders to attend more closely to implementation of activities off the compound. At that time, consideration needs to be given to team leaders with more development expertise for improved management of component activity. The professional capacity and skill levels of local staff appeared very comparable to NGO staff in neighboring countries. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 38 BRIDGE-Winrock operates according to a very rational organizational chart, with remarkably uniform job titles and descriptions across all components. Decision-making authority is well defined, with little tension reported at the matrix of program and operations. Operationally Strong Systems. Finance, human resources, information technology (IT), and other systems appear well managed, for operational and logistical control purposes, by an experienced Deputy Chief of Operations. The financial system appears very strong from a control standpoint. Cash is managed securely, and accounts are reconciled with low levels of error and in a timely fashion. It is likely that Winrock International and USAID auditors would grant the BRIDGE-Winrock program a positive and unqualified rating. On the other hand, the financial system is not managed or used to provide input to substantive financial decision-making. As is very typical in development projects, program management at all levels understand their financial role as one of executing budgets—not determining best use of funds once a budget has been established. For example, sectoral analysis of expenditures could (with effort) almost certainly be carried out, but the value of such analysis is not appreciated. The absence of attention to cost was pronounced. On only one occasion did the evaluator‘s question solicit an answer that showed thoughtful consideration of the relative value of one activity over another in terms of cost. The HR system likewise is well managed as an administrative support function, with careful attention to compliance with labor law and salary and benefits administration. The IT system is critical to the smooth functioning of the entire organization and is well supported for continuing improvement in an environment where IT maintenance is enormously challenging. 3.3.2 Management and Functionality of the Consortium34 BRIDGE-Winrock was structured as a consortium for both pragmatic and professional reasons. Pragmatically, it is difficult for any single implementing partner to hire the necessary numbers of staff for a program of this size and range in the timeframe available between the typical project funding date and project start date. Professionally, the range of activities (governance, agriculture, WASH, education, community development) suggests a need for specialized practitioners, with substantive technical knowledge and abundant experience in the different areas of operation. In practice, however, the consortium structure has contributed little to the success of BRIDGE￾Winrock. There is some evidence that consortium partners were somewhat hastily incorporated into the final proposal, as the project expanded from one state in the initial proposal to three states in the final submission. Original consortium partners, Norwegian Church Aid and the United Methodist Committee on Relief both elected to withdraw from the consortium before implementation began. The current consortium partners are: 34 Analysis of the consortium arrangements reflects Winrock International‘s assessment of relationships to a greater extent than is ideal for evaluation purposes. The evaluation team conducted one interview with RTI staff, but that interview concentrated on HRD activities rather than the consortium relationship. Evaluators did conduct a full interview on the consortium relationship with PACT Sudan and Windle Trust. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 39 RTI International (RTI) RTI, based in North Carolina, is an independent, nonprofit organization that engages in research and development across a large number of fields and serves a wide range of clients. RTI identifies its fields of expertise in the international development arena as ―public health, education, governance and management, urban development, environmental resource management, and public finance and economic growth assistance for developing and democratizing countries.‖ RTI cites extensive experience and innovative practice in cross-sectoral local governance development. RTI was a logical choice to provide leadership and technical support for LG development at this critical stage in Southern Sudan‘s transition from civil war to decentralized statehood. In the initial program design, RTI was tasked with leading the D&G component, with significant responsibilities for hiring staff and providing strategic direction for the unit. However, RTI (according to Winrock International) was not staffed to make timely managerial and strategic contributions to the D&G program, and RTI and Winrock consequently restructured the relationship, reframing RTI‘s role as a technical advisory function. RTI provides support to BRIDGE-Winrock through a series of STTA visits; in that role, RTI contributed substantively to the ICA effort and is now concentrating in human resources development. The STTA approach limits RTI‘s time in the field, with clear impacts on the depth of knowledge and strength of relationships that the organization brings to programmatic and strategic development of the D&G component. PACT Sudan (PACT) Pact Sudan is a capacity building international NGO that has been operating in Sudan since 2002. PACT Sudan leads a number of programs in Sudan, including the Water Resources for Peace Program, and was accordingly an obvious choice to lead BRIDGE-Winrock‘s WASH activities. At the time of the evaluation, Winrock and PACT were in the process of dissolving the consortium relationship. The relationship was dissolved officially on grounds of PACT‘s failure to meet WASH targets in the area of household latrines. The relationship was also characterized by substantive differences in the approach to capacity building. Winrock International, in the context of BRIDGE￾Winrock, defines capacity building around the strengthening of government capacity to deliver WASH services. Sudan PACT defines capacity building in the WASH sector and other areas around strengthening of local CBOs to facilitate public service delivery. Additionally, PACT and Winrock have different approaches to staffing at the field level, with PACT operating with limited staff in the field, relying on CBOs for many activities. A number of PACT employees have joined the Winrock staff with a very positive commitment to meeting targets for sanitation. ACDI/VOCA ACDI/VOCA based out of Washington, D.C., specializes in agribusiness, food security, enterprise development, financial services, and community development. Winrock International and ACDI/VOCA have a strong base of shared expertise in agriculture and food security and have acted as partners in a number of other consortia in the past. The relationship between Winrock International and ACDI/VOCA in Southern Sudan is currently proceeding well after a shaky start that was rooted in individual rather than organizational differences. Despite a very positive relationship between ACDI/VOCA and Winrock, given some of the implementation challenges discussed earlier, it is not evident that the agriculture component is benefitting from a strategic partnership between ACDI/VOCA and Winrock. GOAL Sudan GOAL Sudan is part of an international humanitarian organization headquartered in Ireland and has substantial experience in relief operations in Sudan. GOAL Sudan receives support from BRIDGE- Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 40 Winrock for WASH activities in just one county, Twic, in Warrap. The BRIDGE-Winrock activities are conducted as part of GOAL‘s broader public health program in Twic County. The evaluation team did not explore this relationship in depth, visiting only one community in Twic where no significant issues were identified. Windle Trust Windle Trust is a non-profit English-language training institute headquartered in the U.K. with extensive experience in East Africa, particularly among refugee populations. Windle Trust is regarded by Winrock International as a full strategic partner, although it more closely resembles a service-delivery contract relationship than a strategic partnership. The management of BRIDGE-Winrock, as was discussed above, has been particularly disciplined in its effort to meet targets and to present a unified message. This is a management style better suited to the administration of well-defined contractual arrangements than the collaborative or strategic leadership style often required to motivate complex consortiums and partnerships. That said, the BRIDGE￾Winrock program has been well served by the more disciplined approach considering the challenging context and pressing time frames in which it is currently operating 3.3.3 Monitoring and Evaluation Monitoring Well Executed Monitoring, narrowly defined as the routine tracking of inputs and outputs, is well executed by BRIDGE-Winrock. A senior monitoring and evaluation officer in Juba is supported by an M&E officer in each state, managing a paperwork-intensive system that verifies and tabulates data for purposes of reporting on more than 35 indicators agreed upon by USAID and Winrock International in design of the program (see Annex 10 for a list of indicators). Data is collected with appropriate controls to ensure field-based M&E officers are conducting site visits as required by the M&E work plan. Original documentation is carefully transmitted and registered for purposes of systems integrity. The system confirms that targeted activities have been conducted and serves the needs of USAID‘s results framework and reporting requirements, with some repeat counting of the same base activities (see Annex 11 for a diagram of the M&E system). The system provides little or no useful information for evaluation purposes beyond verification that agreed-upon inputs and outputs (e.g., training sessions, workshops) have been completed. Only two or three of the indicators hold out the possibility of assessing outcomes of activities. Current M&E indicators provide almost no information on quality or basic functionality of activities. Soon after beginning evaluation fieldwork in Sudan, it became abundantly clear that, while BRIDGE-Winrock had in most areas met (and sometimes exceeded) established targets, the reported data were of limited value as indicators of current functioning or quality of activities—much less as predictors of longer-term impact. The highly positive narrative of the first-year annual report and second-year quarterly reports are accompanied by long lists of activities, for which outcomes are not clear. The BRIDGE-Winrock team is well aware of the weaknesses of the M&E indicators for evaluation purposes and in internal deliberations has suggested adding other indicators.35 35 ―Revise Indicators to reflect the fact that the program amounts to more than a series of short-term deliverables. Consider the inclusion of process, quality and/or KAP (Knowledge, Attitude and Practice) Indicators, even if these are not Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 41 Important Evaluation Efforts In an attempt to inform future evaluations of BRIDGE-Winrock, the program has developed three ―impact indicators.‖ This attempt to assess impact is commendable. These indicators may produce useful results for soft trend analysis but are not likely to yield statistically significant results. These three impact indicators were: Institutional Capacity Assessment. The Institutional Capacity Assessment (ICA) is an excellent instrument for internal planning and assessment of progress. It is unlikely to yield results that have external validity for impact assessment. Citizens Perception Survey The Citizens Perception Survey uses three-point scales to establish whether citizens have positive perceptions of government. While a three-point scale can be a very valid measure of perceptions at a single point in time across demographic groups, it is less likely to yield statistically significant results on a time series basis unless there are substantial changes in perceptions. Still it may provide useful data for some soft trend analysis. (See Annex 12 for summary description of Citizens Perception Survey) Food Security BRIDGE-Winrock chose ―food security,‖ actually defined as an increase in crop yields, as an indicator of impact for service delivery components and proxy indicator for peace dividends. This is very unlikely to produce reliable results for two reasons. First, BRIDGE-Winrock agricultural activities are not on a scale to expect substantive impact on food security beyond a small number of targeted households. Second, BRIDGE-Winrock has chosen to purposively sample ―lead‖ farmers who are known to have adopted crop production technology. Purposive sampling of lead farmers is a one-time test of training effectiveness—not of food security impact more broadly. 3.3.4 Coordination and Management by USAID BRIDGE-Winrock enjoyed a close relationship with the original Agreement Officer‘s Technical Representative (AOTR), who was actively involved in the design, public launch, and mission oversight of the program for the first 18 months of implementation. In that period, financial approvals and program modifications were readily obtained, contributing to BRIDGE-Winrock‘s operational flexibility and success. On the other hand, the close relationship also made it easier for USAID to impose new demands on Winrock International, including expanded capacity-building activities at the state level. For example, as explained in BRIDGE-Winrock‘s annual report for 2009: ―At USAID‘s request and based on the findings of its ICAs, BRIDGE [Winrock] has modified its approach. BRIDGE [Winrock] will now build good governance capacity in four areas: planning and budgeting, financial management, tax administration and human required within the USAID Cooperative Agreement or Project Document. Introduce such indicators gradually and progressively across the entire program. If USAID does not ascribe value to these Indicators, use them internally for program strengthening.‖ (BRIDGE-Winrock) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 42 resource management. While it will follow through on original intentions to work at the county level and continue to support the same specified technical counterparts at the state level, it will now also direct substantial support to the state ministries of finance and collaborate closely with the state Ministries of Local Government and the state Directorates of Labor, Public Service and Human Resource Development. This will likely mean that BRIDGE [Winrock] will not be able to implement every activity with the same level of support in every state.‖ This expanded mandate of BRIDGE-Winrock to a broader set of D&G activities, including HRD, has (as discussed earlier) focused programmatic attention to state-level activities at some risk to service delivery and county level activities. Programmatic attention to both tax administration and HR activities also establishes goals that are unlikely to be fully obtained within the three-year timeframe. The departure of the former AOTR and some uncertainty over who was to assume that role going forward has been a difficult adjustment for program managers. Winrock International management reported substantial delays in obtaining approvals for straightforward items that were speedily approved in the past. Such decisions by the acting or new AOTR are complicated by the nature of the Cooperative Agreement (CA) that informs USAID‘s decision-making. The CA requires very limited financial reporting on the part of Winrock International, providing little information on which to base certain types of decisions. BRIDGE-Winrock management expressed appreciation for recent visits to the field by the Deputy Chief of Mission and other USAID staff, but complained, as do other USAID-program leaders, that such visits are infrequent and usually quite abbreviated, creating little opportunity for real learning. Additionally, high USAID staff turnover in Southern Sudan, coupled with frequent R&R travel, contribute to notable and frequent absence of key USAID staff and regular arrival of newcomers unfamiliar with projects in Sudan. Implementing agencies contend that that they are perpetually in the process of bringing USAID staff ―up to speed‖ on projects or programs and emerging complexities of the Sudanese context in which they operate. BRIDGE-Winrock expressed particular concern that too few USAID staff members appreciate the effort, importance, or nuances of establishing collaborative relationships with government. 3.4 Evaluation of Crosscutting Issues 3.4.1 Gender Equity USAID has established a clear commitment to improving gender equity in its Sudan programs. As stated in USAID‘s Strategy Statement for Sudan (2006–2008): ―Gender issues will be addressed under each strategic objective and women will equitably participate in all USAID-funded activities. The SPLM is committed to addressing women‘s political participation and has instituted a quota system to ensure representation in government. USAID will work to help women fulfill their potential by providing leadership training and integrating gender into leadership, public administration, and financial management training so that a more balanced view of women‘s roles emerge within civil administration and civil society. Women are fundamental to building peace and resolving South-South conflict.‖ Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 43 Winrock International extends this commitment effectively in BRIDGE programming. BRIDGE￾Winrock exhibits a strong commitment to improving gender equity in all areas of organizational activity. This commitment was abundantly evident in interviews, and most convincingly was heard in both formal and informal observations of staff interactions and discussions. Staff, without prompting from the evaluation team, frequently and expressively raised concerns about gender inequity in the Southern Sudanese context. This is a topic to which Diaspora Sudanese, including influential members of the BRIDGE-Winrock staff, bring new sensibilities and model new behavior. BRIDGE-Winrock contributes to improving gender equity in various ways:  Assisting with the formation of a Women‘s Support Group (WSG) in parallel with each Community Action Group (CAG). The purpose of the WSGs is to raise the profile of women and women‘s issues in community action and planning, as well as provide an arena for the development of women‘s leadership and public communication skills. There was almost no evidence that these WSGs were contributing to any positive outcomes for women at this time; but their very existence has the potential to raise awareness in an environment of pervasive disregard for the contribution of women in civic and economic fora.  Making concerted efforts to improve the numbers of women in decision-making positions and leadership roles within CAGs, PTAs, Vegetable Producer Groups, and other community endeavors. By design, 30 percent of all CAG members are women and a smaller number, approximately 10 percent, hold leadership positions.  Working to increase the number of women trained in government and within communities. Despite this commitment, BRIDGE-Winrock has found it extremely difficult to meet USAID expectations for women‘s participation in a number of training activities, most notably in the area of financial management and agricultural extension where few women exist on the civil service payroll. Similarly, BRIDGE-Winrock has found it challenging to identify livelihoods projects and vocational training opportunities for women, expressing concern that typical female-focused efforts could lead to a proliferation of food kiosks and handicrafts projects for which there is little market demand. In some instances, women have been included in training efforts even where the expectation of their continuing participation in the activity is limited. Both results, over-representation and under-representation of women, raise the usual questions about the efficacy of ―affirmative action‖ in training. Still, forward-looking commitment is exhibited in developing the capacity and participation of women in government service and economic activity.  Modeling women‘s leadership capacity very effectively by the example of its own staff. Senior management at BRIDGE-Winrock in Juba (including the incoming Chief of Party, the Deputy Chief of Party-Programs, senior D&G program managers, and others) are dynamic, and in almost all cases, African women. Capable African women also hold highly visible leadership positions in NBG and Warrap and at a more junior level in Unity. As could be expected, these women maintain a keen focus on gender issues wherever and whenever they arise.  Providing sex-disaggregated data and reporting for all training and other activities as required in USAID results frameworks. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 44  Providing both gender-based budgeting and gender sensitivity training to community groups and local government officials. This commendable attention to gender issues has occurred in the absence of an explicit gender-equity strategy. Still, some formal attention to gender equity outside of the routine planning process could be productive. A strategic exploration of the formidable challenges of gender equity in Sudan, using best practice tools, might lead to a more creative set of responses than are currently being implemented. For example, USAID‘s Livelihoods Tool Kit for programming in conflict-affected environments explores how ―changes in gender roles and responsibilities brought about by conflict‖ can lead to the identification of new livelihoods opportunities. Formal strategic attention to gender equity is likely to widen the scope of attention to a broader range of issues affecting women, such as the particular challenges faced by female-headed households, and simultaneously widen the scope of the equity lens to embrace challenges faced by other vulnerable groups, including the disabled, orphans, and unemployed youth. 3.4.2 Environmental Protection BRIDGE-Winrock‘s attention to environmental issues is almost exclusively compliance-focused and the project acts in compliance with USAID‘s strict environmental management and mitigation practices, seeking compliance also with any relevant standards established by GOSS.36 The compliance process is detailed and can consume significant amounts of managerial and staff time in the preparation, review, and approval of often necessary EMMPs. All construction and related activities are subjected to a baseline screening that identifies the proposed activity as ―very low risk,‖ ―moderate or unknown risk,‖ or ―high risk‖ for environmental consequences. For any activities determined to be of moderate or high risk, an environmental review process then determines whether the activity will have ―no significant adverse impact,‖ ―no significant adverse impact with mitigation efforts,‖ or ―significant adverse impact.‖ The environmental review is then followed as needed by development of the EMMP (including monitoring requirements) for which USAID approval must be received. Attention to environmental compliance is a significant task for the senior program manager for the LGDF, who was hired in the second year of operations to manage the growing backlog of LGDF proposals. As a result of the commitment to compliance—and in the absence of any contrary evidence—the evaluation team feels it is safe to conclude that BRIDGE-Winrock activities are having ―no significant adverse impacts‖ on the environment. Outside of the compliance orientation, there was little evidence of a proactive environmental sensitivity or consideration of longer-term environmental issues. This conclusion is reached on the basis of the relative absence of any discussion of environmental issues informally among staff or in formal program documents. Environmental issues were raised in only a few discussions. The issue of pesticide use arose on at least three occasions in the context of challenges of pest infestation of vegetable gardens. The primary concern was that in the absence of a national policy 36 Such standards include the guidelines for school construction contained in the South Sudan National Education Policy complemented by Environmental Guidelines for Small-Scale Activities in Africa, available at www.encapafrica.org/SmallScaleGuidelines.htm. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 45 for chemical/inorganic pesticide use that BRIDGE-Winrock was forced to rely on what was assumed to be much less effective organic approaches to pest management. No concern was raised about the environmental health consequences of inorganic pesticide use, and little knowledge of integrated pest management approaches was evident among field staff. Again, in the context of vegetable gardens, one project officer lamented that because of USAID‘s environmental policy, BRIDGE-Winrock was unable to participate in the construction of feeder roads, which would greatly facilitate the transport of vegetables to local markets. In that case, the same officer did acknowledge that the unregulated and indiscriminate placement and construction of such feeder roads by earlier NGOs had contributed adversely to flooding in the area. In the WASH arena, there was extensive discussion of the immediate environmental health consequences (and the need for appropriate technical solutions) to address seepage of fecal matter from latrines into the high water table. However, in the WASH arena, there was also a notable absence of any deeper discussion of the longer-term environmental issues surrounding water supply and little evidence of any watershed-level orientation; such ecological awareness has increasingly become part of the international dialogue among water and sanitation experts. 3.4.2 Conflict Sensitive Lens Southern Sudan has endured decades of war and continues to experience intermittent conflict internally and across the north-south divide. The possibility of renewed conflict looms menacingly on the horizon as the country awaits the outcome of the 2011 referendum. The risk of conflict is heightened by chronic food insecurity, weak social services, and high potential of floods or droughts. The evaluation team was most aware of troubling conflict dynamics in Unity State where there is historical and still acute animosity between Nuer and Dinka; patent distrust between northern Sudanese traders/merchants and local communities; easily ignited hostility among resident and IDP/returnee communities; and, more recently, political divisions heightened by a contested April 2010 gubernatorial election. Conflict sensitivity enters programming at two critical junctures: first, in program design and second, in ongoing attention to and management of existing and emerging conflict. The core features of BRIDGE￾Winrock‘s program design reflect conflict sensitivity. USAID selected the three states where BRIDGE￾Winrock operates because of their geopolitical importance in terms of the Sudanese conflict. BRIDGE￾Winrock‘s emphasis on CBD also reflects best practice in program design for conflict-affected areas. The BRIDGE-Winrock program was funded, in part, because of its conflict-mitigating potential. In the context of implementation, however, there was remarkably little discussion of conflict-sensitive programming beyond planning for likely interruptions to work expected around the time of the 2011 referendum. The evaluation team may have failed to ask the right questions expecting the topic to emerge easily on the basis of a few simple questions. However, on the basis of a somewhat limited examination of the issue, it does not appear that BRIDGE-Winrock plans flexibly or thinks creatively about alternative approaches on the basis of ongoing conflict analysis and/or program risk analysis. There are clearly program activities that are threatened by emerging conflicts. The most immediate example comes from Unity state, where it is feared that northern Sudanese traders and merchants will soon choose to leave (or be forced to leave) due to conflict between them and local communities Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 46 following the referendum. The economy in Unity is greatly dependent on commerce facilitated by northern Sudanese, and their departure will have severe implications for continuing economic activity. Formal Peace and Conflict Assessment tools and guidelines can provide insights and suggest solutions such as proactively ―identifying entrepreneurs who are willing to be among the first to forge links with people on the other side of the conflict.‖ The use of conflict early warning indicators and conflict￾sensitive programming tools are recommended in order to plan creatively for a year in which conflict is not unlikely. IV. SUMMARY RECOMMENDATIONS BRIDGE-Winrock has met and exceeded targets in a large number of areas, representing a remarkable level of foundational activity, achieved in an extremely challenging environment. The evaluators are convinced that a necessary foundation has been established for increasing impact over the remaining life of the program. This evaluation also describes a number of risks to continuing progress, some of which are a consequence of working in the context of an exceedingly fragile developing state and some of which reflect weaknesses in implementation. The following recommendations seek to address these risks, suggesting improvements in program design and implementation. The recommendations below are divided in two categories. First are those recommendations directed primarily at the implementing partner and which, for the most part, address implementation issues for the remaining life of the program. Second are those recommendations that provide guidance for possible replication and expansion of the BRIDGE-Winrock program and, to a large extent, address issues with strategic and programmatic implications for USAID. 4.1 Remaining Life of Program For the remaining life of program, the evaluation team recommends that BRIDGE-Winrock focus more closely on quality of outputs, with careful attention to service delivery components of the program. Generally:  Maintain focus on those capacity building efforts that have a strong link and well-defined outcome pathway to local service delivery improvements.  Exercise caution with respect to an expanding mandate towards state and national capacity building efforts.  Give special attention to possible M&E changes for immediate and improved measurement of emerging outcomes so that stronger evidence will be in place for continued funding decisions. Specific and more detailed recommendations for each component appear at the end of the respective sections earlier in this report. Below are summary highlights from those recommendations: Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 47 Democracy & Governance  Review of the efficacy of HRD activities (i.e., job descriptions and organograms) with special attention to the integrated and downstream value of these activities in the time frame shared by service delivery demands and capacity building requirements. Consider narrowing the scope of activities in HRD until such time as the MOLPSHRD and respective labor legislation are better developed.  Focus tax administration activities on basic principles and operations that would survive substantial changes in the tax code following the referendum. Community Development  Focus on strengthening already existing CAGs/WSGs rather than facilitating formation of new ones.  Strategically consider the longer-term civic function of CAGs in Sudan‘s decentralized state model. BRIDGE-Winrock needs to clarify its vision for the future role of CAGs and develop clear strategies to realize those functions. Agriculture and Livelihoods  ACDI/VOCA senior management is encouraged to conduct a thorough examination of work in the agriculture and livelihoods area with a view to addressing serious design and implementation issues and challenges. Water, Sanitation and Hygiene  Continue the strong focus on the sustainability of WASH efforts, taking care to ensure that those activities (e.g., fee-based supply, accessibility of pump mechanics) produce the intended outcomes.  Request changes to targets for household latrines while increasing activities directed at increasing awareness and acceptance of sanitation and hygiene practices. Education  Conduct an internal review of the education component with a view to improving programmatic coherence across activities. 4.2 Replication The model has potential for scale-out replication based on the implementation experience of BRIDGE￾Winrock. Successful replication, however, is far from assured. The relative success of BRIDGE￾Winrock can in part be attributed to strong leadership accompanied by disciplined organizational and managerial focus on short-term deliverables. This discipline needs to be maintained while also improving attention to the quality of outputs and continuing emphasis on sustainability of outcomes. The decision to replicate the BRIDGE-Winrock program to other states needs to: Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 48  Be grounded in additional formative evaluation that needs to begin almost immediately within BRIDGE-Winrock.  Include substantial program modifications reflecting criticisms identified in the report and suggested in the component recommendations above.  Be accompanied by much-improved M&E in any future programming, with special attention to realistic timeframes and baseline measurement, perhaps a baseline household panel survey in appropriately sampled counties.  Consider rollout of all or a subset of program activities across all counties in a state to avoid potential conflict. USAID funding of multi-sectoral, area-based programs would be well served by additional and creative efforts at operationalizing integrated program design. This would entail, among other things:  Achieving definitional clarity around programmatic versus strategic integration;  Clarifying the degree of programmatic integration necessary to achieve strategically integrated goals;  Understanding how component activities contribute separately, but additively, to integrated outcomes;  Understanding how component activities can be integrated to achieve more sustainable outcomes;  Understanding how risks or failures in one area of activity do or do not threaten results in another area; and  Understanding and clarifying expectations of the contribution of multi-sectoral, area-based programs to country-wide sectoral goals. However, based on the internationally well-documented challenges of partnership and consortium management, over-specification of integration in the form of unrealistic integrated monitoring and evaluation indicators or in the form of tightly woven joint work planning is not recommended. Such efforts are highly likely to be bureaucratically burdensome and of limited value, but some improved operationalization of integration seems in order in any future funding of area-based multi-sectoral programs. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 49 ANNEX 1: SCOPE OF WORK Scope of Work Management Systems International (MSI) SUPPORT Project with USAID/Sudan37 Mid-Term Evaluation of Building Responsibility for Delivery of Government Services (BRIDGE/Winrock) Program – as Implemented by Winrock International (Evaluation In-Country Start and End Dates: September 20-October 24, 2010) 1. Program Details Program Identification: Name: Building Responsibility for Delivery of Government Services (BRIDGE/Winrock) Program NOTE: USAID/Sudan has another project called ―BRIDGE‖, implemented by Mercy Corps in other parts of Southern Sudan. For clarity, this activity will be called ―BRIDGE/Winrock‖ and the other will be referred to as BRIDGE/Mercy Corps. Contracting Vehicle: Cooperative Agreement Award No. 650-A-00-09-00003-00 Program Funding: $52, 499,991 from USAID with $846,369 from Winrock-led Consortium Program Beginning/End Dates: 22 January 2009 to 22 January 2012 Key Agreement/Contract Modifications: Modifications: None Implementing Partners(s): Prime: Winrock International RTI International ACDI/VOCA GOAL United Methodist Committee on Relief (UMCOR) – dropped before implementation Norwegian Church Aid (NCA) – dropped prior to implementation Windel Trust – added at beginning of implementation Pact – added at beginning of implementation USAID/Sudan Technical Office: Program Office David Schroder, Program Officer BRIDGE Team:  Carmelita Maness, Economic Governance Activity Manager 37 MSI holds a 3-year contract to provide Mission-wide support to USAID/Sudan in program and project evaluation and designs, MIS management, translation services, logistics support, facilities management, VIP hosting, and research. An in￾country team, based in Juba, provides these services, supplemented by short-term technical assistance. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 50  Victor Lako, Democracy and Governance Activity Manager  Pia Philip, Education Activity Manager  Martin Swaka, Health Activity Manager 2. Evaluation Rationale and Type BRIDGE implementation is at the half-way mark in the Program‘s life-cycle. USAID/Sudan typically undertakes an assessment of the effectiveness of the implementation of a project at this point in time. Therefore, a formative, collaborative mid-term evaluation is being carried out for the BRIDGE Program. 3. Background A. Country Context Sudan is the largest country in Africa, borders nine countries, and has a population estimated at 40 million. Since independence in 1956, Sudan has suffered from civil war, with only a decade of troubled peace from 1972 to 1983. Southern Sudan and the critical border areas (consisting of the northern states of Southern Kordofan and Blue Nile, plus Abyei – a commonly referred to as the Three Areas) are characterized by years of underdevelopment, war, famine, drought and flood, producing a crisis of enormous proportions across the region and resulting in the devastation of economic, political and social structures. In addition to the loss of lives, opportunities and infrastructure, the war displaced families and divided communities. In consequence, the health, education and infrastructure status of the Sudanese people are among the poorest globally. After decades of civil war, Sudan's warring parties signed a Comprehensive Peace Agreement (CPA) in January of 2005. Since that time the country has taken steps toward peace, reconciliation and good governance, although the pace has been slower than expected or desired. Despite the signing of the CPA, Sudan remains a vulnerable state. Its children, many of whom are orphans, returning refugees and ex-combatants, are particularly at risk - especially in the volatile north/south border areas. It is essential that displaced and other affected people, including orphans and ex-combatant youth, be safely reintegrated into their communities. In the case of the youth, affected by the many conflicts and tensions during the past 21 years, the provision of basic education is critical to providing a solid foundation upon which their future success and contribution to society can be based. The provision of education can also be seen as a tangible result of the "peace dividends" expected by Sudanese citizens and, in turn, will contribute to stabilization in the region. Durable stability is contingent upon demonstrative and observable change "on the ground" and education, highly valued by the Sudanese, is both a necessary and visible symbol of that change. In many areas, primary health and education services have been almost exclusively externally funded. Non-governmental organizations (NGOs), faith-based organizations (FBOs), and multilateral and bilateral aid agencies offering humanitarian relief became the prime providers of an array of much needed services. As peace is consolidated, USAID will continue to support a responsible transition from emergency to development assistance that seeks to improve access to and quality of basic education. Education and health activities are reinforced by investment in other essential services, such as water and sanitation, in an effort to rebuild local communities, reduce tensions, and provide the much sought-after peace dividends. B. Sector Context Functional capacity for governance, administration, and delivery of public services Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 51 Public support for the CPA can be secured only through legitimate, democratic governance that delivers the peace dividends sought by Sudanese citizens. The limited evidence to date of ―development dividends‖ in many parts of Southern Sudan and the Three Areas has dampened enthusiasm for the CPA, resulted in criticism and discontent from citizens, and hindered resolution of local conflicts. Failure of the regional Government of Southern Sudan (GOSS) to establish effective, accountable institutions and of local governments to deliver basic social services, could lead to a resumption of conflict among local or regional groups and call into question the viability of Southern Sudan as an independent state if the referendum leads to independence in 2011. In the federal system established under the Sudan Interim National Constitution and the Interim Constitution of Southern Sudan, state and county governments have the primary responsibility for delivering core public services to citizens (e.g. education, health and basic infrastructure). However, as a region emerging from protracted decades of war where government institutions were non-existent and delivery of social services were nil before state and county governments could fulfill their mandate in the interim constitutions, the GOSS regional government had to be established. For southern Sudanese, core services, such as education, health, infrastructure, and an environment conducive for economic growth, are some of the primary anticipated dividends of peace. Unfortunately, delivery of these services has lagged due to a stark lack of human and institutional capacity at all level of government, limited institutional and physical infrastructure, and continuing security concerns throughout Southern Sudan. While tremendous progress has been made in establishing ministries and other government institutions where there were none, major challenges remain. The structure of government is in place; but ministries are not yet truly functional. Although revenues are coming in and payments are being made, the GOSS is heavily reliant on oil revenue without having sufficiently diversified revenue sources which would help insulate the government revenue flows from abrupt changes in oil prices. In addition, many governance fundamentals need to be implemented or improved. Stronger and more consistent linkages among government priorities, policies, legislation, budget execution, service delivery, and project implementation need to be forged. The civil service rosters at all levels of government in the south remain bloated, and staff lack appropriate technical skills and tools to perform adequately. On the finance side, procedures to enable transparent financial management are not yet fully established and auditing and oversight functions remain weak at the GOSS, State and County levels. Underlying governance concerns are continuing along with multiple security challenges. During the war, conflict fed into a cycle of retribution and violence. After the war, there continues to a number of issues related to reconciliation and power sharing in the South which, combined with limited options to enhance livelihoods which could mitigate community tensions, continue to fuel violent clashes among ethnic groups. Through several different programs, USAID is currently working with a number of GOSS institutions, including the Ministries of Health; Education; Finance and Economic Planning; Labor, Public Services, and Human Resources Development; Transport; Agriculture, Wildlife and Legal Affairs and Constitutional Development as well as commissions such as the Land Tenure Commission and the Anti￾Corruption Commission and Center for Statistics and Evaluation, among others. Given this troubling context, GOSS faces tremendous challenges in 2011. An independent Southern Sudan would be a landlocked country with no easy access for trade outside its immediate neighbors. GOSS does not have a diversified economic base and is dangerously dependent on oil, with 97% of its budgetary revenue resulting from oil sales. Possible separation in 2011 raises the question of economic, political, and security uncertainty. Constraints among capacity-building institutions USAID recognizes that the GOSS faces a massive capacity building challenge. While citizens increasingly demand services and higher levels of performance (and the GOSS strives to become increasingly professional), there are few training service providers accessible to Southern Sudanese civil servants. A total of sixty-five percent of the civil service have achieved a primary education or less. This impedes the ability of government to provide effective services and threatens delivery of peace dividends. It is also an Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 52 indirect threat to the stability of Southern Sudan because the GOSS will only achieve legitimacy in the eyes of the citizens through performance. C. State Context Together, Northern Bahr el Ghazal (NBG), Unity, and Warrap states have an estimated population of 2.3 million people.38 Relative to bordering states, the governments of all three states have low numbers of staff in areas such as economic planning, local government, agriculture, health, and engineering. Staffing, however, is bloated with individuals who do not work or are ―ghost‖ employees. Many of those who work lack the necessary skills to perform their duties, with many being illiterate. Operational linkages between state and county governments are weak. Personnel capacity at the county level is generally even weaker than at the state level. The revenue base for states and counties is low and dependent on GOSS transfers. Corruption is present at all levels; staff motivation is generally very low. The people of NBG and Warrap are agro-pastoralists. In NBG, household wealth is held in cattle and is vulnerable to poor standards in animal husbandry, nutrition, and veterinary care. Although attitudes are changing, enthusiasm for engaging in agricultural production has been low. Farmers do not produce enough staple crops to feed their families year round. Vegetables that could be grown locally are imported. Crops are often decimated by floods or drought. While NBG has one of Southern Sudan‘s highest levels of food insecurity, there is reason for optimism. Cereal yields rose by 28 percent from 2005 to 2006. Good potential exists in the livestock sector for dairy products, meat processing, and hides and skins. The people of NBG have relatively easy access to local markets and transportation, as well as market linkages in northern Sudan. In Warrap, the major crops are sorghum, groundnuts, and sesame. There is demand for these products, along with sesame oil, in Twic and potential markets in Wau and Western Equatoria. Productivity is low, however, due to a lack of technical capacity, flooding and drought, fungal diseases, and limited access to inputs and services. Moreover, with high livestock prices, incentives are limited for farmers to upgrade agricultural practices or introduce new crops. Warrrap has one of Southern Sudan‘s highest levels of food insecurity as well as high levels of inter-tribal conflict over grazing land and access to water. Livelihood development is stunted by a lack of cash to invest in business start-up and construction. Unity is the most oil-rich state in Southern Sudan. The benefit of this is that the state receives 2 percent of the total revenue from oil drilled in the state. The downside, of course, is that it creates yet another source of tension and instability for the state to overcome due to inequitable distribution of this revenue. Livestock, basic staple crop production, and fishing are the main agricultural-based livelihood activities in Unity but productivity remains low due to limited technical capacity, frequent flooding and cyclical droughts, conflict, and inadequate access to inputs and services. Relatively few small farm holders engage in agriculture for commercial purposes. However, the political will to support agriculture does seem to exist. In 2006, the state purchased 39 tractors, 100 tons of sorghum seed, and 40 tons of maize seed, along with various other inputs, with the goal of helping farmers to sow 40,000 feddans of land. There is considerable potential for developing non-forestry agricultural products, such as gum Arabic, lulu, and lalob. Markets in South Kordofan and Khartoum are accessible and there is potential for supplying agricultural produce to cafeterias for oil industry workers. 39 The health profile of the three states is not encouraging. Contaminated water and poor sanitation systems and practices contribute to the high incidence of disease. Limited access to water is a frequent cause of conflict; disputes arise over both human and animal consumption. In Unity and Warrap States only 60 percent of the population has access to an improved water source. In NBG, 50 percent of the population has access to an improved water source. In addition, access to sanitation is very low of all three states, being 5.5, 1.9, and 5.3 (as a percentage of the population) in Unity, Warrap, and NBG, respectively.40 The long distances that people, particularly women, must often travel to collect water 38 Statistical Yearbook for Southern Sudan (2009) Southern Sudan Centre for Census Statistics and Evaluation 39 Statistics about state economies taken from Winrock Sudan BRIDGE Award, signed 01-26-2009 40 Statistical Yearbook for Southern Sudan (2009) Southern Sudan Centre for Census Statistics and Evaluation Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 53 results in time lost and increased risk to personal security. Easing access to water and improving sanitation will have an enormous impact on development prospects in the states. The water sector presents opportunities for spurring private sector growth, such as the repair of small pumps. The situation in the three states‘ education sector is dismal. In NBG, one percent of children attend school41, and girls account for less than 10 percent of the students. Teacher-pupil ratios are on the order of 1:50 to 70, and hundreds of the teachers are classified as untrained. Drop-out rates for boys and girls are at 18 and 32 percent respectively. Cultural and economic factors put girls at a particular disadvantage. Most classes are conducted under trees or in makeshift shelters, exposing children to rain and extreme heat. In a survey of 145 schools, only 20 had permanent buildings. Positive developments are underway, however. Since 2006, teachers have been registered and are receiving salaries when funds are available. A new unified curriculum has been developed for primary grades one through five, and the government has coordinated with UNICEF and other partners to deliver scholastic materials to as many schools as possible. Warrap has 337 primary schools and only nine secondary schools. Of the secondary schools, seven are in the counties selected for support from BRIDGE Winrock. Schools have high drop-out rates. Parent Teacher Association (PTA) training has been provided to primary schools throughout the state but further strengthening of PTAs is needed to make them fully functioning bodies. No Boards of Governors have yet been trained. Warrap is believed to have the highest number of households hosting refugees. Unity has 240 primary schools. Only 44 of the schools are housed in permanent structures. Many of the teachers are not qualified to teach their subject areas and some are teaching at levels beyond their own levels of completion. The state uses a portion of oil revenues to hire teachers from Uganda and Kenya but, rather than stay in the villages, many of these teachers move to the state‘s capital city and run small businesses while collecting their salaries. Because of immediate needs, families often have incentives to direct children‘s time away from school toward endeavors that are more productive in the short-term. Many children go with their families to cattle camps.42 Since the signing of the CPA, all three states have continued to receive a high number of displaced persons returning from northern Sudan. With limited economic opportunity and weak availability of social services, integration of internally displaced persons (IDPs) has been a challenge to state and local government officials and to receiving communities. Thousands of young ex-combatants are without a meaningful livelihood, constituting a large and volatile population. D. Program Description BRIDGE/Winrock is implemented by a consortium of NGOs led by Winrock International. The program operates in three states of Southern Sudan: Northern Bahr el Ghazal (NBG), Unity, and Warrap. Activities are targeted in three counties within each state to build state and local government capacity to deliver services addressing key governance and technical issues. Activities are also expected to have impacts beyond BRIDGE counties because targeted state line ministries (education, water, agriculture, and health which was added in 2010) and Ministries of Local Government, Labor and Finance, will systemically reach out to other non-BRIDGE counties. The three-state approach is intended to create synergies among states that will allow for joint trainings and other opportunities to scale up the program‘s impact. The program has five components: 1. Strengthen the capacity of government to deliver social services and promote economic growth (led by Winrock with technical support from RTI); 2. Strengthen the capacity of communities and government to work together to identify, prioritize, and address community needs (led by Winrock); 41 Statistical Yearbook for Southern Sudan (2009) Southern Sudan Centre for Census Statistics and Evaluation 42 Other education statistics taken from Winrock Sudan BRIDGE Award, signed 01-26-2009 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 54 3. Increase food security and promote rapid, broad-based, and self-sustaining economic growth through development of the agriculture sector (led by ACDI/VOCA and supported by Winrock); 4. Increase access to safe water and sanitation in a sustainable manner (led by PACT with support from GOAL in Warrap State); and 5. Expand access to quality education (led by Winrock). Each of the five goals is served by a separate program component. While each component can stand independently, the five are also mutually supportive. For example, by building state and local government capacity in component 1 (Governance Capacity Building), the program buttresses the supply side of the good governance equation, making it more effective, responsive, transparent, and inclusive. By empowering communities and strengthening their relationships with local government in component 2 (Community Development), the program improves the absorptive quality, self-reliance, and democratic values of the equation‘s demand side. The three areas of sector support addressed in components 3 to 5 – agriculture, water and sanitation, and education – provides the substance necessary for making the relationship between government and communities meaningful. Together the five components enable the Winrock team to catalyze two types of value chains: the governance value chain, which links state and local governments and the communities they serve; and private sector value chains, which link the producers, processors, and numerous providers of related goods and services needed to reach the market. By fortifying the linkages up and down (and, when appropriate, between) the public and private sector value chains in the agriculture, water and sanitation, and education sectors (and to a lesser extent in the health and small infrastructure sectors), the program assures a positive and sustainable impact on the livelihoods of the people of NBG, Unity and Warrap and the political stability of a fragile region. Building Sudanese capacity and sparking Sudanese initiative is at the heart of the program. To build capacity, the program focuses on strengthening knowledge and skills, facilitating systemic reform, and deepening democratic practices. To spark initiative, whenever possible, BRIDGE/Winrock offers its counterparts menus of activities from which to choose. This is intended to promote local ownership of the activities and ensure consonance between their priorities and program objectives. BRIDGE/Winrock also collaborates with counterparts to design activities that minimize dependence, maximize self-reliance, and promote market mechanisms. Working in these rural areas of Southern Sudan presents a host of logistical and other challenges in housing, water, transportation, communication, and obtaining qualified staff. E. Linkage to USAID/Sudan Strategy and USG Foreign Assistance Framework The underlying hypothesis of the BRIDGE program is that increasing government legitimacy by improving the perception citizens have of their government‘s ability to meet their needs contributes to peace and stability. The ability of GOSS to coordinate and provide peace dividends in the form of services is also critical to support implementation of the CPA. BRIDGE contributes to two of USAID/Sudan‘s strategic objectives: SO 9 Avert & Resolve Conflict and SO 10 Manage Crisis, Promote Stability, Recovery & Democratic Reform. Under the Foreign Assistance Framework BRIDGE includes activities which fall under the Governing Justly and Democratically, Investing in People and Economic Growth areas of the USG foreign assistance framework. In addition, gender equality, transition from relief to development and conflict mitigation are cross-cutting themes. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 55 F. Geographic Orientation As of Year 2, BRIDGE/Winrock is working in the following counties: State Original two counties per state in Year 1 County added in Year 2 Northern Bahr el Ghazal Aweil West Aweil North Aweil East* Unity Rubkona Mayom Guit Warrap Gogrial West Twic Tonj North *While the Year 1 focus was on two counties per state, some initial work was undertaken in Northern Bahr el Ghazal in Aweil East as well. 4. Available Information to Support the Evaluation The following information will be provided to the Evaluators prior to their arrival in Juba.. 1. BRIDGE/Winrock-Winrock Award (including Project Description) 2. BRIDGE/Winrock-Winrock PMP as of October 2009 3. Annual Work Plans 4. Quarterly and Annual Reports 5. NDI surveys of citizen attitudes 6. BRIDGE/Winrock Citizens Perception Survey 7. MSI trip reports for BRIDGE/Winrock evaluations conducted in Warrap (April 2010) and Unity (July 2010) 8. BRIDGE/Winrock Institutional Capacity Assessment Report (February 2009) 9. Democracy and governance impact assessment tool (June 2010) 10. Democracy and governance gap analysis report (September/November 2009) 11. Trip notes, Diana Swain, July 2009 12. Trip notes from John Marks and other USAID field visits. 13. Unity field visit report, Mike K (August/September 2009) 14. FAAB report 15. Animal traction assessment report (April 2009) 16. Supply Chain Development for Ox-Plow Spare Parts (April 2009) 17. USAID Evaluation Guidelines 18. MSI Evaluation and Special Study Guide 5. Evaluation Purpose and Questions Purpose The purpose of the mid-term evaluation is to: (1) assess the Program‘s achievements against performance targets at the mid-term point in the implementation period, (2) provide USAID with an informed discussion regarding the likelihood of the Program‘s impact in the medium to longer term and, (3) provide USAID with a thorough analysis of the Program‘s design. On the basis of the evaluation team‘s analysis and recommendations, USAID and the Implementing Partners will be able to: (1) modify implementation in order to expand Program achievements in the Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 56 short-term, (2) help ensure the sustainability of any short-term benefits being derived from the Program and, (3) strengthen the foundations needed to help ensure the impact of the investment in the medium to long-term. Evaluation Questions Effectiveness and Efficiency 1. How effective and efficient has the Program been in achieving its performance targets to date and whether the achievements are worth the cost of the investment? (That is, is the Program achieving what it is supposed to be achieving and is it doing so in a timely manner and demonstrates value for money?)  In addition to output targets, please consider the extent to which the Program is achieving positive changes in: (1) organizational and institutional development in the areas of financial management, participatory planning and budgeting, tax administration, human resource development and transparency and accountability in targeted government agencies, and (2) strengthened vertical linkages between communities and local governments to jointly identify and address community priorities in agriculture, education, water and sanitation.  Please consider, among other things, strategic decisions that the IPs have made and the extent to which these have been informed by a sound analytical process. 2. To what extent has USAID‘s capacity to coordinate and manage the Assistance Agreement affected the IP‘s management of the Program and the Program‘s performance to date?  Please consider at a minimum whether USAID, in its management of the Assistance Agreement of a complex program, is meeting the needs of the Implementing Partner. 3. How effectively and efficiently have the management aspects of the Program been undertaken to date?  Please consider at a minimum Program management aspects such as communication and coordination, human resources management, Head Office interface and support, reporting, risk assessment and mitigation, the management and functionality of the consortium, monitoring and evaluation and whether there are financial or other cost implications to any of these aspects.  Please consider management aspects that demonstrate or fail to demonstrate efficient Program management and whether there are financial or other cost implications.  When analyzing the Program‘s monitoring and evaluation approach, please consider whether the Program‘s M&E approach meets the needs of the Implementing Partner (IP), individual partners and USAID in the following areas: o Reporting against outputs indicators and outcome (i.e., improved livelihoods, water supply/sanitation, and basic education services) indicators linked to peace dividends. o Data capture and disaggregation by relevant variables including sex, ethnicity, SES, for example. o Decision-making by the IP and its consortium partners, IP oversight and coordination of the consortium, USAID oversight and reporting needs. 4. What evidence is there of quality, both in terms of Program management and Program achievements? Cross-Cutting Issues: Conflict Sensitivity, Gender and Other Equity Considerations, Environment 5. To what extent does a conflict-sensitive lens inform Program strategy and implementation? 6. How well does the Program address equity issues, particularly gender? Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 57  Please consider both direct and indirect beneficiaries. 7. To what extent does the Program take environmental consequences into account? Sustainability and Impact 8. What short-term benefits have come about as a result of the Program and how likely is it that these current benefits will be sustained post-Program?  Please consider both the direct and indirect beneficiaries.  To the extent possible, given available data, please consider any aggregated benefits related to development results such as increases in income, wellness, savings due to payroll rationalization, etc. For example, a proxy indicator for improvements in the quality of education resulting from BRIDGE‘s teacher training activities might be students‘ improved scores on achievement tests. 9. What is the medium- longer-term impact that is expected and what is the likelihood that this impact will be realized post-Program? Project Design and Future Directions 10. Are the Program‘s development hypothesis and program logic sound and still viable for the context?  Please consider, among other things, whether the Program is relevant to state and local government needs and demands. 11. How well does the Program‘s design reflect best practice and lessons learned about development programs in conflict-affected contexts?  Among other things, please locate the BRIDGE approach within the literature on integrated rural development programs (IRDP) including community-driven development (CDD) and community-based approaches (CBA).  Please consider current knowledge about capacity development and peace and conflict. 12. How appropriate is the Program‘s conceptual framework and implementation approach in the current Sudanese context in general and particularly with respect to increasing government capacity?  Please consider, among other things, the extent to which a clear approach to capacity building at the individual, organizational and institutional levels is articulated and able to be monitored and evaluated.  Please pay particular attention to the CPA and the expected peace dividends. 13. How viable is the current design and implementation approach in the event that USAID wishes to replicate the Program in (a) new counties and payams in the three targeted states and/or (b) in new states?  Please consider at a minimum the following: o The benefits and challenges of replicating the Program‘s conceptual framework and implementation approach within the three targeted states and/or in new states. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 58 o Re-design considerations if replication of the approach is deemed to be appropriate and viable within new counties and payams in the current targeted and/or new states. Lessons Learned 14. What are critical lessons from Program experience to date? 6. Evaluation Methods and Procedures The External Evaluators will be provided the information provided in Section 3, above, before arriving in Sudan. They will be expected to be familiar with this information prior to arriving in Juba. A Team Planning Meeting (TPM) will be held upon arrival in Juba to agree on how team members will work together, how they will interact with the client (USAID) and other stakeholders, and to develop a work plan and finalize a Travel Schedule. The team will conduct meetings in Juba with USAID/Sudan, key GOSS institutions and implementing agencies. The team will need to visit project site(s) so some travel arrangements will be arranged prior to the team‘s arrival. At a minimum the team will need to travel to Northern Bahr el Ghazal, Warrap, and Unity states. During the TPM the team will finalize the methodology to be used and produce the evaluative instruments to be employed. The team will use the ―Getting to Answers‖ approach detailed in Annex II of the MSI Evaluation and Special Study Guide to develop detailed qualitative and quantitative methodological approaches to meeting the terms of this Scope of Work. We expect that in addition to basing the evaluation‘s findings on interviews and review of project documents, the team will also utilize the following simple approaches:  Development of an interview guide to ensure that the correct evaluation questions are being addressed the appropriate individuals and that they are being posed and recorded consistently.  Interviews with state/local government counterparts, other key stakeholders, and beneficiaries.  Indicate other basic methodological approaches appropriate to the task, such as focus group discussions, key informant interviews etc.  It is expected that the evaluation team will make every effort to identify any commonalities in findings, conclusions and recommendations across the sectors in which the Program is working and also to discriminate between the respective sectors where there are clear distinctions in findings, conclusions and recommendations between the sectors.  It is also expected that the evaluation team will have a clear understanding of CDD/CBA program design and implementation in conflict-affected contexts and use this knowledge to inform the evaluation process.  It is expected that the evaluation team will have a clear analytical framework that it will use to make the evaluation of this complex, multi-faceted Program manageable, so as to yield the best possible findings, conclusions and recommendations. Once the methodology has been finalized at the TPM it will be shared with USAID as part of the work plan approval process. 7. Team Composition and Participation Team Composition USAID/Sudan is conducting the Mid-Term Review in a collaborative manner to maximize USAID, GOSS and Implementing Partners‘ learning opportunities. Accordingly, the team will consist of the following individuals: Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 59  Three External Evaluators (two senior and one mid level - skill sets detailed below) contracted by MSI  One representative from GOSS (Juba-based)  Two representatives of the Implementing Partner. All team members are expected to participate on a full-time basis throughout the evaluation period. Additional inputs may come from other staff from these agencies when and where possible. MSI will facilitate the participation of any GOSS and State Officials. The Implementing Partner will make recommendations and preliminary contact with these team members. Team Member Roles and Responsibilities USAID, GOSS and the IP will provide historical, contextual and programmatic background information that will inform the assessment. MTE Team members are expected to participate in the Team Planning Meeting (TPM), field visits, interviews, the Findings, Conclusions, and Recommendations Workshop and frequent reflection sessions on evaluation learning (that often occur at the end of a day in the field). USAID, GOSS and IP team members participate as representatives of their respective organizations and are expected to share their learning with their home organizations so that all three key organizations are kept abreast of progress. There will be three external, independent Evaluators--an Evaluation Team Leader and two other Evaluators. The Team Leader will be the formal representative of the team and will arrange for updates regarding progress against the evaluation work plan to the AOTR (or his/her delegate) and MSI‘s Chief of Party (COP) or Assessment, Monitoring and Evaluation Specialist (AME), as determined at the TPM. The Evaluation Team Leader will take full responsibility for managing the team, organizing its work, and ensuring quality control and delivery of a final report acceptable to USAID. The Evaluators will take the lead in conducting the evaluation, leading interviews, framing the analysis, facilitating group discussion and consensus, preparing for the debriefing, and producing a draft and a final evaluation report. Precise division of labor between the Evaluators will be determined at the TPM. The Evaluators may ask USAID, GOSS or IP representatives to be absent from some interviews in order to ensure candid responses from those individuals being interviewed and to avoid any conflicts of interest. The following capacities must be present among the Evaluators: 1. Strong skills in evaluation, assessment, and analysis of USAID projects (preferably 10 years or more) 2. Experience in the design, management, or implementation of integrated development projects in conflict-affected contexts (preferably 5 year or more) 3. Strong research and writing skills 4. Extensive experience working in East Africa and in Sudan and/or similar post conflict environments 5. Facilitation experience, experience leading participatory evaluations, or at least evaluations where evaluation teams include critical stakeholders as active participants 6. Experience arranging meetings, setting up travel schedules for field visits, reporting on meeting outcomes, and generally managing the logistics of the evaluation (although significant logistical assistance will be provided by the SUPPORT team in Juba) 7. Experience in implementing or evaluating the following: a. Local governance capacity strengthening b. Managing remote field projects c. Education d. Community-based projects e. Agriculture/food security f. Water and sanitation Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 60 8. Activities, Logistics, and Timing Prior to arriving in Juba, the External Evaluators will have familiarized themselves with the background material provided to them, as referenced in Section 3, above. All team members should be present for the TPM and for initial briefings and discussions with USAID‘s Economic Growth Office and other Mission officers, as well as IP and GOSS officials. A Work Plan and travel program for the in-country visit as well as the subsequent report writing period will be submitted to USAID for approval during the first few days of work in Juba. The Work Plan will also include a schedule for periodic MSI and USAID progress reports and possible submissions of specific work products, as determined by the parties. Approximately four days prior to departure the Evaluation Team will present to USAID, Implementing Partners, other development partners and the GoSS an out-briefing, with succinct supporting documents. The Draft Evaluation Report will be submitted prior to the External Evaluators‘ departure from Juba. There may be more than one exit briefing. The implementing partner will provide a lsit of GoSS and development partner representatives with whom they have been working to be invited. In addition members of the capacity enhancement working group should be issued an invitation. The Mission and the IP will each submit its comments on the draft report within ten work days of receipt the draft report. The Draft Final Report will be submitted to USAID ten work days after the Team Leader‘s receipt of USAID‘s and the IP‘s final written comments on the draft. It is envisioned that all External Evaluators will be in Sudan the entire duration of the evaluation‘s in￾country component (six-day work weeks are authorized), including the TPM, a debriefing, and submission of a draft report to MSI‘s COP or AME prior to departure from Sudan. In addition to travel days, additional days are provided for the External Evaluators to complete reading and processing all background information prior to departure for Sudan. Additional days are provided to finalize the report. (See graphic presentation in Section 9, below.) MSI‘s field office in Juba will be responsible for travel arrangements (travel, housing in the field, etc.) for the USAID and GOSS team members. MSI will fund travel-related costs for GOSS team member(s), but not for IP or USAID team member(s).43 MSI and the Implementing Partners will jointly arrange all meetings for the team, in coordination with GOSS. The team will be provided office and meeting space, as needed, at SUPPORT‘s Juba Office Compound. 9. Projected Level of Effort (LOE) Tasks (Each External Evaluator, unless otherwise noted) Work Days (6-day weeks in Sudan; 5 outside Sudan) Initial Preparation Review documents, complete desk review on integrated programs, draft interview guide and proposed methodology, finalize travel schedule and travel days to Juba 9 Team Planning Meeting 3 43 If the USAID representative is an Institutionally-Contracted Staff member provided by MSI, his/her travel costs will be provided by MSI separately. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 61 Methodology, work plan development/finalization In-Country Evaluation Initial briefings, meetings, field visits, draft report preparation and debriefings 18 Draft Report and Debriefings 9 Return Travel 2 Final Report Preparation in home country Incorporate collective Sudan feedback, complete final report, and submit to MSI. 3 Total for Evaluation Team Leader 44 44 44 The Team leader will have an additional 2 days of LOE for final editing of the report. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 62 10. Report Production and Format The team will present for approval by USAID a draft outline of the report during its first week in country. The report must:  Distinguish clearly between findings, conclusions (based strictly on findings) and recommendations (based clearly on the report‘s findings and conclusions);  Comply with all instructions of the SUPPORT Project‘s ―Evaluation/Special Study Quality Management Guide‖ and meet the specific requirements of the ―Evaluation Report Review – Score Sheet‖, contained therein;  Include a Table of Contents; a list of acronyms, an Executive Summary of no more than three pages; a section describing the project to be evaluated and purpose of the evaluation; a section on the methodology employed, including relevant skill sets of the evaluators;  Include any annexes the team considers useful to the reader; and  A copy of this SOW as an Annex. A formal debriefing will be provided to USAID, the IP and the GOSS, as scheduled during the TPM and recorded in the evaluation work plan. The team will present key Findings, Conclusions and Recommendations for comment from the stakeholders. The team will record all relevant feedback from the meeting and will respond to all comments in completing its draft reports. The External Evaluators need not include all suggestions in the report, but must consider such suggestions in finalizing the Draft Report. An electronic (in MS Word) version of the Draft Report will be presented to the IP and USAID in Juba with four hard copies being provided to the USAID/Sudan Mission and one hard copy to the IP prior to the departure of the Team Leader. The document will not exceed 40 pages, excluding annexes and Executive Summary. The Mission and the IP will each submit its respective comments on the draft report electronically to MSI‘s COP – using the ―track changes‖ and ―comments‖ functions in MS WORD as much as possible – within ten work days. Each organization will combine internal comments, resulting in a unified set of comments from USAID and the IP. The external evaluators will then incorporate the final feedback into a final report, which will be branded. The Mission will receive ten paper copies of the final report as well as an electronic version, once the Mission has accepted the product. 11. Deliverables  A draft work plan, ensuring that all aspects of Getting to Answers (from the TPM) are addressed  A schedule of travel and key activities  Interim progress briefings to MSI and the Mission, as determined during the TPM  Preliminary report outline  Draft Findings, Conclusions and Recommendations to MSI prior to completion of the first Draft Report  Out-briefing, with supporting documents  Draft report  Final report 12. Compliance with USAID Regulations The Evaluation Team will ensure that the evaluation is fully compliant with the terms for Project Evaluations contained in the USAID Automated Directives System (ADS) Series 203 and other relevant regulatory requirements, as may be determined by USAID. Additionally, the Team will utilize MSI‘s SUPPORT Project‘s ―Evaluation/Special Study Quality Management Guide.‖ The Guide will be presented to the Team members prior to their arrival in-country. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 63 ADDENDUM 1: Specific Questions that Emerged in Discussions to Produce the SOW Program Management 1. Are there any issues with respect to program design and assumptions (documented or implied) affecting program management that should be reconsidered based on experience to date? For example: a. What are the advantages and disadvantages of the consortium? Is the consortium approach leading to efficiencies and synergies? Are there ways that the structure and management systems could be improved, considering USAID and IP‘s contracting requirements? b. Has USAID been able to manage the internal coordination and contracting requirements of the program successfully and in a manner that maximizes synergies? c. Given the program locations, what are some of the inherent human resources challenges affecting program implementation (e.g., recruiting and retaining staff)? How has the IP creatively addressed them? Are there any other potential solutions to these constraints that the Evaluation Team could suggest? Program Design 2. Are there any issues with respect to program design and assumptions (documented or implied) that should be reconsidered based on experience to date? a. Does the development hypothesis behind the program remain valid? b. Is the program design (see USAID/Sudan APS) in line with what is feasible in the time frame provided? Are the expectations of USAID reasonable? c. Is the program design appropriate to transition from relief to development in targeted areas? 3. Was the program designed to bring about changes in state and local government capacity to deliver services? Is the modality and focus correct? Has the program addressed all critical constraints for the program to succeed? 4. Should the balance between delivering peace dividends and improving government capacity to deliver services be revisited? a. Is the program design in line with state and local government needs and demands? How willing is the state and local government to receive assistance? Please gauge the effectiveness among technical assistance, training, and mentoring. 5. What is the value-added of an integrated program for implementation? Performance to Date 6. Are the key governance systems addressed by BRIDGE/Winrock (improvements in financial management, more participatory planning and budgeting, tax administration and human resource development), likely to be adopted by state and local government? Is sufficient progress being made towards improving transparency and accountability within targeted government institutions? 7. Can the key government capacity building interventions promote increased access to peace dividends and improved service delivery? 8. Is training provided being applied. 9. Is BRIDGE/Winrock‘s approach to community development/mobilization enabling communities to interact with local government and jointly identify and address their needs and priorities (agriculture, education, water and sanitation)? 10. Does the program address gender and social imbalance? If so, how? 11. Does the program directly or indirectly address peace building and conflict management and mitigation? 12. Has implementation progress been acceptable? What factors have led to success, or lack thereof? Program Monitoring 13. BRIDGE/Winrock is a complex multi-institutional, multi-disciplinary program operating in multiple areas. Please indicate if the monitoring and evaluation system is adequate for the following (and if not provide recommendations for improvements): Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 64 a. Capturing program outputs and peace dividends (i.e., improved livelihoods, water supply/sanitation, and basic education services) b. Is the program setting up systems to capture program impact – for example, are citizen‘s increasingly satisfied with government service delivery as a result of BRIDGE? What if any additional impact indicators should be developed? c. Does the M&E system capture the extent to which training received is changing the way in which government officials operate and do business? Is it possible to determine the impact on service delivery? d. Does the monitoring system capture data disaggregated by gender in a way that can inform gender￾conscious programming? e. Individual partner decision-making f. Prime oversight and internal consortium coordination needs g. USAID oversight needs h. USAID reporting needs Sustainability and Future direction 14. Can/should the program expand to new regions (counties and payams) within existing states? If it is not yet possible to make this determination, what issues must first be resolved? 15. Can/should the program be replicated in new states? If it is not yet possible to make this determination, what issues must first be resolved? 16. Is the program prepared to address improved health service delivery? If it is not yet possible to make this determination, what issues must first be resolved? 17. What changes in the current structure or program approach, if any, would be required to justify an extension/expansion of the program? 18. What improvements brought about by the program are sustainable? Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 65 ANNEX 2: LIST OF SITE VISITS Northern Bahr-el-Ghazal Aweil West: 1. Akuakngap Boma CAG/WSG, Guomjuer Center Payam 2. Akewich Payam CAG, Guomjuer Center Payam 3. Wedwil Primary School PTA 4. Wedwil Boma CAG, Goumjuer East Payam Aweil North - 1. Pamet Boma CAG/WSG, Malual Center Payam 2. Majok Ngor Boma Veg Group, Ariatt Payam 3. Majok Ngor Boma PTA/ Fishing Group 4. Mayom Adhal Boma CAG, Malual North Payam Aweil East - 1. Mangok Boma Vegetable Producers Group, Mangok Payam Warrap Gogorial West (day 1) 1. Ngapkuot-Jok WMC, Madeng Boma, Gogrial Payam 2. Gogrial Boma CAG, Gogrial Boma, Gogrial Payam Gogrial West (day 2) 1. Kuac South CAG, Kuac South Payam, Gogrial West County. 2. Wunkuel dit Boma CAG, Kuac South Payam, Gogrial West County 3. Angui CAG/WSG, Angui Boma, Kuac North Payam, Gogrial west County Twic 1. Maper CAG/WSG, Maper Boma, Aweeng Payam, Twic county 2. Ajong (CAG/WSG) Ajong Boma, Wunrok Payam, Twic county. Unity Rubkona County – 1. Dhorbor Payam, CAG/WSG and Veg. Producers Group 2. Bimrouk Boma, Fruit Tree community 3. Tong Boma, CAG/WSG/ beehive 4. Pakur Boma Guit County 1. Watonyon Payam – CAG and Fishing Group Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 66 ANNEX 3: LIST OF DOCUMENTS CONSULTED BRIDGE-Winrock Program Documents BRIDGE Cooperative Agreement Award No. 650-A-00-09-00003-0 BRIDGE APS – Program Description BRIDGE Quarterly Progress Report Year 2010 Q3 BRIDGE Quarterly Progress Report Year 2010 Q2 BRIDGE Quarterly Progress Report Year 2010 Q1 BRIDGE Year 2010 Work Plan BRIDGE Annual Report 2009 BRIDGE Year 2009 Work Plan BRIDGE ICA February 2009 BRIDGE Governance Gap Analysis June 2010 BRIDGE State Capacity Building Needs Assessment 2009 Consultant‘s Report: Farming as a Business (FaaB) Training of Trainers’ (TOTs) Workshops. Summary of Proceedings and Recommendations, prepared by Adam Norikane, Consultant, December 2009 Multiple Trip Reports Government (GOSS) Documents The Taxation Act, 2009, Laws of Southern Sudan Laws of Southern Sudan, The Local Government Act, 2009 (Acts Supplement No. 6) 15th October 2009 Work plan for implementation of the Local Government Act 2009 Local Government Training Needs Assessment 2009 (An initiative of the Local Government Board of Southern Sudan with support from the German Technical Cooperation.) Guidelines for Integrated State and County Planning and Budgeting (Ministry of Finance and Economic Planning Government of Southern Sudan) May 2010 Public Financial Management Operational Manual, GOSS, July 2010. Framework on State Public Financial Management Reform, GOSS, June 2010 Internal Audit Manual, GOSS Public Service Reforms Implementation Framework Manual, Ministry of Labour, Public Service, and Human Resource Development August 2008. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 67 Workshop Resource Guide: GOSS Public Service Reform Implementation Workshop, June-August 2010. Sudan BRIDGE Program: Human Resource Management Training Manual, GOSS, July 2010 County Development Plan and Budget 2011 Aweil East County, Northern Bahr el Ghazal, (June 2010) Government of South Sudan Functional Capacity Prioritization Study, USAID/Sudan December 2009. Training Materials Information Technology (Foundational Computer Skills) Contract No. 2-330-0212110, by Tribal Helm, BRIDGE Sudan, August 29, 2010 Information Training in Accounting (Government Accounting) Contract No. 2-330-0212110, by Tribal Helm, BRIDGE Sudan, August 29, 2010 Intermediate Training in Accounting (Principles of Accounting) Contract No. 2-330-0212110, by Tribal Helm, BRIDGE Sudan, July 4, 2010 Foundation Training in Finance, Contract No. 2-330-0212110, by Tribal Helm, BRIDGE Sudan, May 27, 2010 Community Action Groups Training Module 2009, BRIDGE Sudan Women Support Groups Training Module 2009, BRIDGE Sudan Gender Sensitivity Workshop Module 2009, BRIDGE Sudan USAID Documents and other related academic papers: Monitoring and Evaluation in Post Conflict Settings, USAID, March 15, 2009 Community-Driven Development in the Context of Conflict-Affected Countries: Challenges and Opportunities, World Bank, Report No. 36245, June 20, 2006 CDD Impact Assessment Study: Optimizing Evaluation Design Under Constraints. Social Development Papers by Paul Wassenich and Katherine Whiteside, No. 51 February 2004 State Report Northern Bahr el Ghazal: Village Assessments and Returnee Monitoring Analytical Report, Maps and Statistical Tables 2009. Prepared by Internatioal Organiation for Migration (IOM) State Report Warrap: Village Assessments and Returnee Monitoring Analytical Report, Maps and Statistical Tables 2009. Prepared by International Organization for Migration (IOM) Workshop on Local Government in Post-Conflict Situations: Challenges for Improving Local Decision Making and Service Delivery Capacity Report, Oslo Norway, 28–29 November 2007 http://www.undp.org/oslocentre/docs08/oslo1107/Workshop_%252520Report.pdf (accessed October 10, 2010) Management Systems International (2010) Government of Southern Sudan: Strategic Capacity Building Study, United States Agency for International Development. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 68 Management Systems International (2009) Government of Southern Sudan: Functional Prioritization Study, United States Agency for International Development. Manuela Leonhard ―Providing Aid Agencies with Tools for Conflict-Sensitive Practice: Lessons Learned from Peace and Conflict Impact Assessment (PCIA)‖ Journal of Peacebuilding & Develeopment, Vol. 1, No. 1, 2002 Hugh Waddington, Birte Snilstveit, and Howard White ―Water, Sanitation and hygiene interventions to combat childhood diarrhea in developing countries‖ International Initiative for Impact Evaluation (3ie) (2009) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 69 ANNEX 4: LIST OF KEY INFORMANT INTERVIEWS JUBA NAME TITLE INSTITUION DATE (2010) INTERVIEW LOCATION James H. Flock Sr. Project Coordinator ACDI/VOCA and WI/BRIDGE Sept 23 BRIDGE office Virginia Chitanda Local Government Advisor WI/BRIDGE Sept. 23 BRIDGE office Jackie Manche Governance Advisor WI/BRIDGE Sept. 23 BRIDGE office Mary Mogga Community Development Advisor WI/BRIDGE Sept. 23 BRIDGE office Wesley Sigal Water & Sanitation Advisor WI/BRIDGE Sept. 23 BRIDGE Office John Ephraim Jaile Education Advisor WI/BRIDGE Sept. 23 BRIDGE office Thomas Hart ADB Advisor MOF, GOSS Sept. 24 MOF, Juba John Chuol Dho DG for Agricultural Production, MAF, GOSS Sept. 24 MAF, Juba Prof Dr. Scopas Dima Project Coordinator and Advisor Agricultural Development MAF, GOSS Sept. 24 MAF, Juba Manhiem Bol Malek Director Rural Water Supply Development MWRI Sept. 24 MWRI, Juba William Hemmink, Michael Eddy, David Schroeder USAID Commission USAID/Juba Sept. 24 (briefing) USAID Residence compound Juba David Masua Operational Manager Windle Trust Sept. 25 Windle Trust/Juba Steward F. Kutiyote Education program manager, Windle Trust Sept. 25 Windle Trust/Juba Erasmus Financial Manager Windle Trust Sept. 25 Windle Trust/Juba John Palmucci COP WI/BRIDGE Sept. 26 BRIDGE Advisors Meeting 10 Advisors WI/BRIDGE Sept. 27 Observation of a planning meeting James L. Jones FMIS Team Leader Deloitte Sept. 27 Deloitte Office, Juba Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 70 JUBA NAME TITLE INSTITUION DATE (2010) INTERVIEW LOCATION Edward Kokole Juma Director General, Quality Promotion and Innovation MoE, GOSS Sept. 27 MOE, Juba Dr. Utam Watba Director of Development MoE, GOSS Sept. 28 MOE, Juba Name Missing Dir. General of Alternative Education MoE, GOSS Sept. 28 MOE, Juba Hasan al￾Shammari Team Leader, Unity State BRIDGE-Winrock Sept. 28 BRIDGE Office, Juba Demetria Arvanitis Managing Director, VTA Winrock International Sept. 28 BRIDGE Office, Juba Edward Wright Program Manager, BRIDGE, Pact Sudan Pact Sudan Oct. 11 Sudan BRIDGE Compound, Juba Mengistu Teklemariam Program Manager Pact Sudan Oct. 11 Sudan BRIDGE Compound, Juba Askari Jafari Deputy Chief of Operations Sudan BRIDGE Oct. 11 Sudan BRIDGE Compound, Juba Evelyn Viegas John Palmucci Incoming Chief of Party Outgoing Chief of Party Sudan BRIDGE Oct. 17 Sudan BRIDGE Compound, Juba John Marks South Sudan Country Program Advisor USAID Oct. MSI Compound, Juba Tressen Sullivan USAID Governance Advisor MSI Oct. MSI Compound, Juba Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 71 Increase Citizen Satisfaction with State and Local Public Administration Systems Overall Indicators: Ind 1: Citizen Perceptions poll Ind 2: ??? Ind 3: ??? Result 1 Strengthen capacity of State and County/Locality Government to plan, manage and/or deliver services to address community needs Indicators: Ind 1.1: [institutional development matrix] Ind 1.2: [# sub national government entities receiving assistance to improve their performance] Ind 1.3: [# performance improvement plans completed] Ind 1.3: [# supervision visits] 2.3 Local Government and Decentralization 2.4 Anti Corruption Reforms 3.1 Consensus Building Processes Result 2 Increase engagement of communities with government in planning for and sustaining service delivery Indicators: Ind 2.1: [# of local mechanisms supported for citizens to engage their sub￾national governments] Ind 2.2: [# of mechanisms for external oversight of public resource use] 1nd 2.3 [# of consensus building processes assisted] Ind 2.4: [in kind contributions for social investment funds (community/government) Results 1.6 MCH 2.1 Basic Education 1.7 FP/RH 1.8 WASH Result 3.2 Increase access to safe water and sanitation (and in MC areas basic health care) Indicators: Ind 3.1:# water committees with operations and maintenance plans Ind 3.2: # people increased access to safe water Ind3.3 # people increased access to sanitation Result 3.3 Increase access to education Indicators: Ind 3.1: # students enrolled in USG supported in primary school Ind 3.2: # teachers trained Ind 3.3 # PTAs supported Ind 3.4 # administrators trained Ind 3.5 # adult learners enrolled in USG supported programs Development Context:  Some of the worst human development indicators in the world  Nascent institutional capacity Risks to Program:  Renewed conflict  Slow implementation of CPA  Restricted access to three areas  NGOs are accountable for promoting the transition to longer term development and working with government structures Illustrative Activity Types See Component 1 See Component 2 See Component 3 See Component 4 See Component 5 Elements and Sub- Elements Result 3: SERVICES DELIVERED (by/with LG) Ind 3.1: Measure of increased incomes Ind 3.2: Number of individuals trained in good health and hygiene practices MC only # & % children under 12 months receiving DPT3 MC only CYP and or # FP/RH counseling visits Result 3.1 Increase food security/income Indicators: Ind 3.1: # producer associations assisted Ind 3.3 # rural hh benefiting Ind 3.4 # receiving short term productivity training Ind 3.5 # gaining emp as a result of workforce development Ind 3.6 # firms receiving assistance 6.2 Private sector productivity 6.3 Workforce development 5.2 Ag sector productivity ANNEX 5: BRIDGE WINROCK RESULTS FRAMEWORK Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 72 ANNEX 6: SUMMARY TABLE OF ACHIEVEMENTS 1.1: Institutional Capacity Assessments, Measured for Key Ministry/Office Structures Results achieved with USG assistance by FY 2009 Baseline FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 0.5 NA 7.5 NA NBG 0.5 NA 7 NA Warrap 0.1 NA 7.5 NA Unity 1 NA 8 NA 1.2: Number of Sub-National Government Entities Receiving USG Assistance to Improve Their Performance Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 27 29 49 51 NBG 9 13 16 17 Warrap 9 8 17 17 Unity 9 8 16 17 1.3: Number of Individuals who Received USG-Assisted Training, including Management Skills and Fiscal Management, to Strengthen LG and/or Decentralization Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 51 379 673 823 Men 39 357 505 617 Women 12 22 168 206 NBG 17 149 214 264 Men 13 144 161 198 Women 4 5 54 66 Unity 17 111 228 278 Men 13 98 171 209 Women 4 13 57 70 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 73 Warrap 17 119 231 281 Men 13 115 173 211 Women 4 4 58 70 70 1.4: Number of Local Mechanisms Supported with USG Assistance for Citizens to Engage their Sub￾National Government Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 7 15 96 183 NBG 2 0 32 56 Warrap 4 5 32 61 Unity 1 10 32 66 1.5: Number of Consensus-Building Processes Assisted By USG Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP actual Total 12 15 NA NA 15 NBG 4 0 NA NA 0 Warrap 4 5 NA NA 5 Unity 4 10 NA NA 10 Other notes: This indicator will not be included in FY 2010 & 2011. It was mistakenly included because of USAID funds obligation originally planned for BRIDGE Mercy Corps activities in the 3 areas. 1.7: Number of Government Officials Receiving USG-Supported Anti-Corruption Training Notes on Baselines/Targets: This indicator was added at the end of FY 2009. FY 2010 numbers address activities related to audit and tax administration training and technical assistance: 16 officials including audit officers per state. FY 2011 numbers will be readjusted based on government commitment and willingness to participate in and promote these activities. Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target Total 0 0 48 30 78 Men 0 0 24 15 39 Women 0 0 24 15 39 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 74 NBG 0 0 16 10 26 Men 0 0 8 5 13 Women 0 0 8 5 13 Unity 0 0 16 10 26 Men 0 0 8 5 13 Women 0 0 8 5 13 Warrap 0 0 16 10 26 Men 0 0 8 5 13 Women 0 0 8 5 13 3.1: Number of Individuals Who have Received USG Supported Short Term Agricultural Sector Productivity Training Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 534 839 2,415 5,186 Men 223 477 690 1,719 Women 311 362 1,725 3,467 NBG 178 244 805 1,693 Men 134 132 230 546 Women 44 112 575 1,147 Unity 178 273 805 1,722 Men 44 106 230 520 Women 134 167 575 1,202 Warrap 178 322 805 1,771 Men 45 239 230 653 Women 133 83 575 1,118 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 75 3.2: Number of Farmers, Processors, and Others who have Adopted New Technologies or Management Practices as a Result of USG Assistance Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 EOP target Total 249 336 1,449 2,944 Men 187 195 414 940 Women 62 141 1,035 2,004 NBG 83 98 483 967 Men 62 53 138 301 Women 21 45 345 666 Unity 83 109 483 978 Men 62 41 138 289 Women 21 68 345 689 Warrap 83 129 483 998 Men 63 101 138 349 Women 20 28 345 649 3.3: Number of Producers Organizations, Water Users Associations, Trade and Business Associations, and Community-Based Organizations (CBOs) Receiving USG Assistance Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target Total 39 104 282 226 612 NBG 13 15 100 80 195 Unity 13 43 112 90 245 Warrap 13 46 70 56 172 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 76 3.4: Number of Women's Organizations/Associations Assisted as a Result of USG Supported Intervention Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target Total 42 50 159 127 336 NBG 14 13 53 42 108 Unity 14 21 53 42 116 Warrap 14 16 53 42 111 3.5: Number of Rural Households Benefiting Directly from USG Interventions Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 600 693 13,203 10,562 24,458 NBG 200 213 4,881 3,905 8,999 Unity 200 241 4,881 3,905 9,027 Warrap 200 239 3,441 2,753 6,433 3.6: Percentage Change in Sorghum Productivity per Feddan for Target-Community Farmers over Baseline Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 0% 0% 25% 25% 25% NBG 0% 0% 25% 25% 25% Unity 0% 0% 25% 25% 25% Warrap 0% 0% 25% 25% 25% 3.7: Number of New and Expanded Businesses Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 90 8 180 144 332 8 NBG 30 0 60 48 108 0 Unity 30 1 60 48 109 1 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 77 Warrap 30 7 60 48 115 7 3.8: Number of Firms Receiving USG Supported Assistance to Improve their Management Practices Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 0 0 195 156 351 NBG 0 0 65 52 117 Unity 0 0 65 52 117 Warrap 0 0 65 52 117 3.9 Number of Persons Participating in USG-Funded Workforce Development Programs Results achieved with USG assistance by FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 0 18 330 264 612 Men 0 18 300 240 558 Women 0 0 30 24 54 NBG 0 3 110 88 201 Men 0 3 100 80 183 Women 0 0 10 8 18 Unity 0 1 110 88 199 Men 0 1 100 80 181 Women 0 0 10 8 18 Warrap 0 14 110 88 212 Men 0 14 100 80 194 Women 0 0 10 8 18 Location of Data Storage: BRIDGE Program office, Juba Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 78 4.5: Number of Individuals Trained in Good Health and Hygiene Practices Results achieved with USG assistance by state and gender FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target Total 786 31,112 34,000 27,200 92,312 Men 588 13,006 13,000 10,400 36,406 Women 198 18,106 21,000 16,800 55,906 NBG 262 0 5,000 4,000 9,000 Men 196 0 2,000 1,600 3,600 Women 66 0 3,000 2,400 5,400 Unity 262 200 24,000 19,200 43,400 Men 196 125 9,000 7,200 16,325 Women 66 75 15,000 12,000 27,075 Warrap 262 30,912 5,000 4,000 39,912 Men 196 12,881 2,000 1,600 16,481 Women 66 18,031 3,000 2,400 19,956 4.6: Number of People in Target Areas with Access to Improved Drinking Water Supply as a Result of USG Assistance Results achieved with USG assistance by state FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP Actual Total 90,500 83,500 129,600 - 103,680 - 316,780 Male 36,200 33,400 48,600 38,880 120,880 Female 54,300 50,100 81,000 64,800 195,900 NBG 30,167 24,000 48,000 38,400 110,400 Male 12,067 9,600 18,000 14,400 42,000 Female 18,100 14,400 30,000 24,000 68,400 Unity 30,166 21,500 33,600 26,880 81,980 Male 12,066 8,600 12,600 10,080 31,280 Female 18,100 12,900 21,000 16,800 50,700 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 79 Warrap 30,167 38,000 48,000 38,400 124,400 Male 12,067 15,200 18,000 14,400 47,600 Female 18,100 22,800 30,000 24,000 76,800 4.7: Number of People in Target Areas with Access to Improved Sanitation Facilities as a Result of USG Assistance Results achieved with USG assistance by state FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 6,250 - 4,500 3,600 8,100 Male 3,125 - 1,800 1,440 3,240 Female 3,125 - 2,700 2,160 4,860 NBG 1,250 - 1,500 1,200 2,700 Male 625 - 600 480 1,080 Female 625 - 900 720 1,620 Unity 1,250 - 1,500 1,200 2,700 Male 625 - 600 480 1,080 Female 625 - 900 720 1,620 Warrap 3,750 1,500 1,200 2,700 Male 1,875 600 480 1,080 Female 1,875 900 720 1,620 5.1: Number of Learners Enrolled in USG-Supported Primary Schools or Equivalent Non-School Based Settings Results achieved with USG assistance by state and gender FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 4,800 11,209 11,192 11,192 33,593 Men 3,600 7,507 8,394 5,596 21,497 Women 1,200 3,702 2,798 5,596 12,096 NBG 1,600 2,792 2,792 2,792 8,376 Men 1,200 1,704 2,094 1,396 5,194 Women 400 1,088 698 1,396 3,182 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 80 Unity 1,600 4,548 4,544 4,544 13,636 Men 1,200 3,173 3,408 2,272 8,853 Women 400 1,375 1,136 2,272 4,783 Warrap 1,600 3,869 3,856 3,856 11,581 Men 1,200 2,630 2,892 1,928 7,450 Women 400 1,239 964 1,928 4,131 5.2: Number of Parent-Teacher Association or Similar ‘School’ Governance Structures Supported Results achieved with USG assistance by state and type FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2 010 FY 2011 target FY 2011 actual EOP target EOP actual Total 48 29 57 57 143 NBG 16 16 19 19 54 Unity 16 8 19 19 46 Warrap 16 5 19 19 43 5.3: Number of Teachers/Educators Trained with USG Support Results achieved with USG assistance by state and school FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 288 187 523 228 938 Male 259 174 444 114 732 Female 29 13 79 114 206 NBG 96 58 169 76 303 Male 86 52 142 38 232 Female 10 6 27 38 71 Unity 96 79 191 76 346 Male 86 77 167 38 282 Female 10 2 24 38 64 Warrap 96 50 163 76 289 Male 86 45 135 38 218 Female 10 5 28 38 71 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 81 5.4: Number of Classrooms Repaired with USG Assistance Results achieved with USG assistance by state and school FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2 010 FY 2011 target FY 2011 actual EOP target EOP actual Total 0 0 60 60 120 NBG 0 0 20 20 40 Unity 0 0 20 20 40 Warrap 0 0 20 20 40 5.5 Number of Adult Learners Enrolled in USG-Supported Schools or Equivalent Non-School-Based Settings Results achieved with USG assistance by state and school FY 2009 target FY 2009 actual FY 2010 target FY 2010 through 09/30/2010 FY 2011 target FY 2011 actual EOP target EOP actual Total 540 540 1,080 Male 405 270 675 Female 135 270 405 NBG 180 180 360 Male 135 90 225 Female 45 90 135 Unity 180 180 360 Male 135 90 225 Female 45 90 135 Warrap 180 180 360 Male 135 90 225 Female 45 90 135 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 82 ANNEX 7: TRAINING ASSESSMENT GUIDELINE 1. Was this training (as designed, delivered and evaluated) likely to fill hopes and expectations for improving unit performance? 2. Did this training improve unit performance? Pre-Training Needs Assessment: Was this training designed to fill:  Gap determined by formal needs assessment?  Gap determined by BRIDGE-Winrock staff based on___________________________?  Request of GoSS recognized by B-W as strongly needed and appropriate?  Request of GoSS understood by B-W to be perhaps __________________(e.g., premature, not appropriate)  Other ________________________________________________________ Selection of Trainees: How were trainees selected? Gender considerations? Were these the appropriate trainees in terms of organization placement, absorptive capacity, etc.? What pre-training skills testing was used to inform training content? Training Content: Best Practice? Developed in conjunction with/direct support of GoSS ? In accordance with available (if any) national curriculum standards? In-Training Training Delivery Quality of Trainers Gender Considerations? Post-Training Formal Evaluation of Trainees (knowledge, competencies) Tests? Trainer Qualitative Assessments of students? Gender considerations? Organizational Usefulness Is trainee able to use the training on the job (e.g., has materials & equipment, within mandate of unit , useful according to individual job description) Organizational and Institutional Impact Is this (unit) able to/doing a better job because of this training? Evidence? Training is/was likely to fulfill expectations of improved performance at/within SO ___________ THAT Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 83 ANNEX 8: ADOPTION OF AGRICULTURAL TECHNOLOGY The following table presents data recorded for two indicators: 3,1 Number Of Individuals Who Have Received USG Supported Short Term Agricultural Sector Productivity Training (TRAINING) 3.2 Number Of Farmers, Processors, And Others Who Have Adopted New Technologies Or Management Practices As A Result Of USG Assistance (ADOPTION) 3.1 TRAINING 3.2 ADOPTION Geographic Location Event Date W M Sub￾total W M Sub-total 1 Western Bahr el Ghazal State, Wau Ox plow Parts Manufacture Training – Warrap Participants 08– 20/ Apr/09 0 3 3 0 1 1 (33%) 2 Ox plow Parts Manufacture Training – NBG Participants 08 – 20/ Apr/09 0 3 3 0 1 1 (33%) 3 Agriculture Extension Methodology Training of Trainers – NBG Participants 26 May – 09 Jun/09 0 13 13 0 5 5 (38%) 4 Agriculture Extension Methodology Training of Trainers – Warrap Participants 26 May – 09 Jun/09 1 11 12 0 4 4 (33%) 5 Central Equatoria State, Juba Value Chain Analysis Training – Unity Participants 22 – 24/ Jun/09 0 5 5 0 2 2 (40%) 6 Value Chain Analysis Training – Warrap Participants 22 – 24/ Jun/09 1 4 5 0 2 2 (40%) 7 Value Chain Analysis Training – NBG Participants 22 – 24/ Jun/09 0 6 6 0 2 2 (33%) 8 NBG State Extension Officer Fruit Tree Seedling Production Training 18- 22/Aug/09 0 12 12 0 5 5 (42%) 9 NBG State, Aweil North County, Malual North Payam, Mayom-Adhal Boma Karnhom Vegetable Producer Group Training (Ag micro￾grant) 24- 28/Aug/09 22 9 31 9 4 13 (42%) 10 NBG State, Aweil West County, Gomjuer Center Payam, Akewic Village Akewic Vegetable Producer Group Training (Ag micro￾17- 19/Sep/09 25 6 31 1 0 2 12 (39%) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 84 3.1 TRAINING 3.2 ADOPTION Geographic Location Event Date W M Sub￾total W M Sub-total grant) 11 NBG State, Aweil North County, Malual North Payam, Malual Loch Boma Malual Loch Vegetable Producer Group Training (Ag micro￾grant) 17- 19/Sep/09 14 15 29 6 6 12 (41%) 12 NBG State, Aweil North County, Malual North Payam, Mayom-Adha Boma Mayom Adhal Vegetable Producer Group Training (Ag micro￾grant) 21- 23/Sep/09 17 17 34 8 4 12 (36%) 13 NBG State, Aweil North County, Malual North Payam, Mayom-Adhal Boma Mayom-Adhal Fishing Group Training (Ag micro-grant) 21- 23/Sep/09 0 4 4 0 2 2 (50%) 14 NBG State, Aweil West County, Ayat East Payam, Majok-Adim Boma Majok-Adim Vegetable Producer Group Training (Ag micro￾grant) 21- 23/Sep/09 21 9 30 7 7 14 (47%) 15 NBG State, Aweil West County, Gomjuer Center Payam, Anghol Village Anghol Vegetable Producer Group Training (Ag micro￾grant) 24- 26/Sep/09 13 17 30 5 7 12 (40%) 16 NBG State, Aweil North County, Ariath Payam, Akwem Village Ariath Fishing Group Training (Ag micro￾grant) 25- 27/Sep/09 0 8 8 0 3 3 (38%) 17 NBG State, Aweil North County, Malual Centre Payam, Pamat Boma Pamat Fishing Group Training (Ag micro￾grant) 25- 27/Sep/09 0 5 5 0 2 2 (40%) 18 NBG State, Aweil North County, Malual West Payam, Majak Baai Boma Majang Bai Fishing Group Training (Ag micro-grant) 25- 27/Sep/09 0 8 8 0 3 3 (38%) 19 Unity State, Rubkona County Agriculture Extension Methodology Training of Trainers (Unity) 26 May – 09 Jun/09 4 15 19 2 6 8 (53%) 20 Unity State, Rubkona County Kuerliel Vegetable Producer Group Training (Ag micro￾grant) 12- 14/Aug/09 11 1 12 4 0 4 (33%) 21 Unity State, Rubkona County Kier Kakier Vegetable Production Group Training (Ag micro￾grant) 12- 14/Aug/09 17 2 19 7 1 8 (42%) 22 Unity State, Rubkona County Thor Koam Vegetable Producer Group Training (Ag micro￾grant) 12- 14/Aug/09 11 0 11 4 0 4 (36%) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 85 3.1 TRAINING 3.2 ADOPTION Geographic Location Event Date W M Sub￾total W M Sub-total 23 Unity State, Rubkona County, Bentiu Town, Bimrouk Village Babanous Fruit Tree Producer Group Training 18- 20/Aug/09 4 10 14 2 4 6 (43%) 24 Unity State, Rubkona County, Bentiu Town, Bimrouk Village Wigpual Fruit Tree Producer Group Training 18- 20/Aug/09 6 4 10 2 2 4 (40%) 25 Unity State, Mayom County, Kueryiek Payam, Kuoy Village Kueryiek Vegetable Group Training (Ag micro-grant) 22- 24/Sep/09 9 8 17 4 3 7 (41%) 26 Unity State, Mayom County, Ngop Payam, Juom Village Ngop Vegetable Producer Group Training (Ag micro￾grant) 22- 24/Sep/09 27 1 28 1 1 0 11 (39%) 27 Unity State, Mayom County, Pup Payam, Tam Village Tam Vegetable Producer Group Training (Ag micro-grant) 22- 24/Sep/09 15 1 16 6 0 6 (38%) 28 Unity State, Mayom County, Pup Payam, Tam Village Tam Fisher Group Training (Ag micro￾grant) 22- 24/Sep/09 0 15 15 0 6 6 (40%) 29 Unity State, Rubkona County, Bentiu Town Extension Methods and Crop Production Training 24- 27/Aug/09 3 6 9 1 2 3 (33%) 30 Unity State, Mayom County, Riak Payam, Ngoang Village Riak Vegetable Producer Group Training (Ag micro-grant) 25- 27/Aug/09 29 0 29 1 2 0 12 (41%) 31 Unity State, Mayom County, Wangbuor_1 Payam, Pibor Village Pibor Vegetable Group Training (Ag micro￾grant) 25- 27/Aug/09 15 2 17 6 1 7 (41%) 32 Unity State, Mayom County, Wangbuor_1 Payam, Pibor Village Pibor Fishing Group Training (Ag micro￾grant) 25- 27/Aug/09 0 15 15 0 6 6 (40%) 33 Unity State, Mayom County, Kuerbuone Payam Agriculture Vegetable Producer Group (Ag micro-grant) 25- 27/Aug/09 14 3 17 6 1 7 (41%) 34 Unity State, Mayom County, Kuerbuone Payam, Tam Village Fishing Group Training - Kuerbuone (Ag micro￾grant) 25- 27/Aug/09 2 18 20 1 7 8 (40%) 35 Warrap State, Gogrial West County, Gogrial Payam Ox Plow Training for Farmers 10 Apr – 14 May/09 0 70 70 0 3 7 37 (53%) 36 Warrap State, Kuajok Ox Plow Parts Manufacture Training 04- 11/Jun/09 0 5 5 0 2 2 (40%) 37 Warrap State, Twic County Fish Drier Training 30 Jun – 02 Jul/09 3 9 12 1 4 5 (42%) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 86 3.1 TRAINING 3.2 ADOPTION Geographic Location Event Date W M Sub￾total W M Sub-total 38 Warrap State, Twic County, Wunrok Payam, Mon Village Mon Village Vegetable Producer Group Training (Group II) 14- 16/Jul/09 3 3 6 1 1 2 (33%) 39 Warrap State, Twic County, Wunrok Payam, Mon Village Mon Village Vegetable Producer Group Training (Group I) 14- 16/Jul/09 5 10 15 2 4 6 (40%) 40 Warrap State, Gogrial West County, Kuac North Payam, Waralel Village Training of Mathiang Vegetable Producer Group on Vegetable Production as a Business (Ag micro-grant) 17- 19/Aug/09 6 13 19 2 5 7 (37%) 41 Warrap State, Gogrial West County, Kuac North Payam, Angui Boma Angui Vegetable Producer Group Training (Ag micro￾grant) 21- 23/Sep/09 12 13 25 5 5 10 (40%) 42 Warrap State, Gogrial West County, Kuac South Payam, Kharic Village Yienlit Vegetable Producer Group Training (Ag micro￾grant) 21- 23/Sep/09 2 18 20 1 7 8 (40%) 43 Western Bahr el Ghazal State, Wau Blacksmith Training on Modified Fire Blower – Warrap participants 25- 27/Aug/09 0 5 5 N A N A NA 44 Warrap State, Gogrial West County, Kuac North Payam, Monyjoc Boma, Mangar Village Mangar Kuel Vegetable Producer Group Training (Ag micro￾grant) 21- 23/Sep/09 17 3 20 7 1 8 (40%) 45 Warrap State, Gogrial West County, Kuac North Payam, Mathiang Village Mathiang Vegetable Producer Group Training on Business Development (Ag micro-grant) 21- 23/Sep/09 8 11 19 N A N A NA Warrap State, Gogrial West County, Akon South Payam, Ayien Boma Ayien Vegetable Producer Group Training (Ag micro￾grant) 28- 30/Sep/09 1 22 23 0 9 9 (39%) Warrap State, Gogrial West County, Akon North Payam, Ameth Boma Ameth Vegetable Producer Group Training (Ag micro￾grant) 28- 30/Sep/09 3 21 24 1 8 9 (38%) Warrap State, Gogrial West County, Gogrial Payam, Wath Gogrial Village Gogrial Payam Vegetable Producer Group Training (Ag micro-grant) 28- 30/Sep/09 5 7 12 2 3 5 (42%) Warrap State, Gogrial West County, Kuac North Payam, Luk-Luk Village Kuac Area Producer Group Training (Ag micro-grant) 28- 30/Sep/09 16 11 27 6 4 10 (37%) Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 87 Below: No corresponding training reported under 3.1 Geographic Location Event Date W M Sub-total Western Bahr el Ghazal State, Wau Ox-plow Parts Manufacture Training – Warrap participants 08- 20/Apr/09 0 4 4 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 88 ANNEX 9: PARTIAL LIST OF LGDF PROPOSALS Approved LGDF Projects for Education Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost 1 AES multi￾purpose hall roofing project Aweil Center NBG $3,440 $1,800 $0 $5,240 2 Renovation and Furnishing of WAU Vocational Training Center Dormitories Unity $21,170 $4,590 $0 $25,760 3 Expansion of Classroom at Liech Primary School Rubkona Unity $80,000 $17,000 $0 $97,000 4 Secondary school construction in Guit County Unity $120,000 $10,000 $2,000 $132,000 5 Renovation of Six Classroom Kuajok Boy's Primary School Gogrial West Warrap $57,083 $0 $8,333 $65,416 6 Kuajok Primary Health Care Center Fencing Project Warrap $35,833 $6,541 4 $0 $42,37 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 89 Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost Gogrial West 7 Mangar Primary School Roof Project Gogrial West Warrap $32,900 $0 $9,020 $41,920 8 Ameth School Improvement Project Gogrial West Warrap $19,660 $11,960 $0 $31,620 9 Kuajok Girl's Primary School Classroom Renovation Project Warrap $110,600 $12,500 $0 $123,100 10 Fence for Mon Village Vegetable Producer Group Kuajok Warrap $3,200 $0 $600 $3,800 11 Pariang Primary School Project Gogrial West Warrap $29,881 $0 $6,540 $36,421 12 Maluil-tit primary school expansion project Gogrial West Warrap $29,610 $0 $6,540 $36,150 13 Nyiel Abyer Improvement Project Twic Warrap $31,600 $0 $6,540 $38,140 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 90 Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost 14 Lukluk Primary School Rehabilitation Project Gogrial West Warrap $57,410 $0 $12,830 $70,240 TOTAL $632,387 $64,391 $52,403 $749,181 Approved LGDF Water and Sanitation: Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost 1 Keurbuohne Borehole Unity $17,188 $2,604 $0 $19,792 2 Latrine & Water Pump for Wangbour Payam School Unity $25,000 $3,000 $3,240 $31,240 3 Borehole for Duargatluaktan gtony's Community Unity $14,000 $400 $2,100 $16,500 4 Borehole Koikoi Unity $14,000 $400 $2,100 $16,500 5 Borehole for Kuernguene Community Unity $14,000 $400 $2,100 $16,500 6 Borehole for Lathtang Unity $14,000 $400 $2,100 $16,500 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 91 Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost Community 7 borehole for Ngop Community Unity $14,000 $400 $2,100 $16,500 8 Borehole for Voting Center in Alhap Village Warrap $13,240 $1,825 $0 $15,065 9 Borehole for Ngapkuot-Jok Warrap $14,000 $0 $1,825 $15,825 10 Borehole Malou Warrap $14,000 $0 $1,825 $15,825 11 Borehole Rumrak Warrap $14,000 $0 $1,825 $15,825 12 Borehole Wun Manyang Warrap $14,000 $0 $1,825 $15,825 13 Borehole Wunkeldit Primary School Warrap $14,000 $0 $1,825 $15,825 14 Construction of VIP Latrines at Kuajock PHCC Warrap $9,890 $3,360 $0 $13,250 15 Public Pit Latrine for Majok Ngor Market NBG $13,774 $0 $1,313 $15,087 16 Clinic Pit Latrine in Majok Adim Boma NBG $8,500 $0 $885 $9,385 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 92 Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost 17 Public Pit Latrine for Pamat Market NBG $13,774 $0 $1,313 $15,087 18 Drilling of one borehole to Lueth Malual Community NBG $13,500 $350 $1,600 $15,450 19 Drilling of one borehole to Marol Deng Geng Community NBG $13,500 $400 $2,100 $16,000 20 Drilling of one borehole to Community NBG $13,500 $400 $2,250 $16,150 21 Drilling of one borehole to War Pach Community NBG $13,500 $400 $2,100 $16,000 22 Drilling of one borehole to Wathok Community NBG $13,500 $400 $2,100 $16,000 23 Drum pit latrine liners for Aweil East County community - Baach NBG $4,600 $1,900 $0 $6,500 24 Drum pit latrine liners for Aweil East County community – NBG $4,600 $1,900 $0 $6,500 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 93 Project Name State BRIDGE Estimated Cost Estimated In-Kind Contribution Government Estimated In-Kind Contribution Community Total Project Cost Malual Baii 25 Rehabilitation of Rupdhal Borehole NBG $750 $0 $0 $750 TOTAL $318,816 $18,539 $36,526 $373,881 Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 94 ANNEX 10: LIST OF INDICATORS 1.1 Institutional capacity assessments, measured for key Ministry/office structures 1.2 Number of sub-national government entities receiving USG assistance to improve their performance 1.3 Number of individuals who received USG-assisted training, including management skills and fiscal management, to strengthen local government and/or decentralization 1.4 Number of Local Government Employees Who Receive USG Assisted Gender Training 1.5 Number of local mechanisms supported with USG assistance for citizens to engage their sub-national government 1.6 Number of consensus-building processes assisted by USG 3.1 Number of individuals who have received short term agricultural productivity training 3.2 Number of farmers, processors, and others who have adopted new technologies or management practices as a result of USG assistance 3.3 Number of producers‘ organizations, water users associations, trade and business associations, and community-based organizations (CBOs) receiving USG assistance 3.4 Number of women's organizations/associations assisted as a result of USG supported intervention 3.5 Number of rural households benefiting directly from USG interventions 3.6 Percentage change in sorghum productivity per feddan for target-community farmers over baseline 3.7 Number of new and expanded businesses 4.1 Number of individuals trained in good health and hygiene practices 4.2 Number of people in target areas with access to improved drinking water supply as a result of USG assistance 4.3 Number of people in target areas with access to improved sanitation facilities s a result of USG assistance 5.1 Number of learners enrolled in USG-supported primary schools or equivalent non￾school-based settings 5.2 Number of parent-teacher associations or similar ‗school‘ governance structures supported 5.3 Number of classrooms repaired with USG assistance Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 95 ANNEX 11: DIAGRAM OF M&E SYSTEM Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 96 ANNEX 12: SUMMARY OVERVIEW OF CITIZENS’ PERCEPTION SURVEY REPORT ON CITIZENS’ PERCEPTION SURVEY (MARCH 2010) In March 2010, a BRIDGE monitoring and evaluation team designed and implemented a survey on citizens‘ perceptions of local government‘s ability to deliver essential services. The survey was specifically designed to measure community perceptions, with the assumption that, over time, communities‘ perceptions of local government in BRIDGE-targeted areas will improve. The survey is a tool to measure the impact of BRIDGE programs. Methodology: This Citizens‘ Perception Assessment will be conducted annually and will focus on communities that have at least six months of interaction with BRIDGE and BRIDGE-supported local government. For 2010, we have included a control group that has experienced no interaction with BRIDGE. However, we do not recommend including a control group for 2011. The assessment will be carried out again in March 2011 using the BRIDGE communities that were interviewed for the 2010 assessment, and six additional CAGs that have had at least six months of BRIDGE interventions. The CAGs interviewed in year one for the baseline will be surveyed again, but the respondents will not necessarily be the same. Sample Size: The total sample size used for this study was 136 people from BRIDGE facilitated CAGs/WSGs, and an additional 34 people were interviewed from communities in non-BRIDGE intervention areas. Out of a total CAG/WSG population of 2,465, a sample size of at least 92 was needed in order for the data to be meaningful. This sample size was calculated using a confidence rate of 95% and a confidence interval of 10. Other factors such as feasibility and logistics within a week‘s time periods for each state were also considered, and so the final number of 136 was reached.45 Survey Instrument: Impact Category Perception Survey Question Confidence Does local government understand the needs of your community? Are decisions made by local government important to your community? Do community members say positive things about local government? Does local government work hard to help communities like yours? By next March, do you think local government will help your community more, the same, or less? Responsiveness Are local government decisions and actions known to your community? 45 Creative Research Systems, ―Sample Size Calculator,‖ http://www.surveysystem.com/sscalc.htm, (accessed 25-Mar-10). This site was used to calculate the sample size. Building Responsibility for the Delivery of Government Services (Bridge) Project Midterm Evaluation March 14, 2011 97 Does local government want to listen to community needs? Is local government more interested in helping your community today than it was last March? Effectiveness Is the local government capable of making your life better? Since last March, have local government services to your community improved, stayed the same, or worsened? Each of the 10 perception survey questions corresponds to three categories by which we measure perception: effectiveness, responsiveness, or confidence. Each question has three possible answers. The negative answer corresponds to 0 points. The neutral or slightly positive answer corresponds to 1 point. The positive answer corresponds to 2 points. Data Summary